Bad lights on maps

[Grum]

(moving to Programming, since the following is more appropriate here)

 

[20:21:13] io/map_io.c.load_map:270 - Bad light (number 6) when loading './maps/map5nf.elm'; co-ords [277116301511919403008.000000 4306482138230070181888.000000 274039735486251008.000000] colour [289450130210097640681504768.000000 215645259835638173004055183360.000000 0.000000]

[20:21:13] io/map_io.c.load_map:270 - Bad light (number 7) when loading './maps/map5nf.elm'; co-ords [277116301511919403008.000000 4306482138230070181888.000000 274039735486251008.000000] colour [71547309178549094580224.000000 4742886487500329025650168954880.000000 215906350033743069948630007808.000000]

Please explain this more. Also those coords #'s don't mean a thing to me, they're not the coords we use to find a place in the map editor.

Unfortunately, they are. Back when we were going through the Intel darkness bugtesting, I found that some lights were in invalid places or had invalid colour/brightness settings. The above is a sample of the output when the client loads the map files (they're no longer used in the client, and (I think I added the code for this )should be moved to near (0,0) when you open the map file in the map editor (as soon as it uses the new code, anyway).

 

There are actually a lot of lights set in the map files that have corrupted data

I've started on a small python script to try and fix the broken lights (and possibly other issues). It's not finished yet, but while working on it I encountered something interesting. Here's the output of running the script on map5nf.elm.

Checking maps/map5nf.elm
Invalid position for light 6:
{'pos_z': 2.7403973548625101e+17, 'pos_x': 2.771163015119194e+20, 'pos_y': 4.3064821382300702e+21, 'g': 2.1564525983563817e+29, 'r': 2.8945013021009764e+26, 'b': 1.0693948974660876e-38}
'.\\particles\\smoke2.part\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
Invalid position for light 7:
{'pos_z': 2.7403973548625101e+17, 'pos_x': 2.771163015119194e+20, 'pos_y': 4.3064821382300702e+21, 'g': 4.742886487500329e+30, 'r': 7.1547309178549095e+22, 'b': 2.1590635003374307e+29}
'.\\particles\\fire_torch.p\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

What you see is for each light with a weird position a dump of the data that was read, followed by the actual bytes that were read (as a python string). You may notice that this data bears a striking resemblance to ASCII text, and very much looks like the beginning of a particle system definition.

 

I'll probably investigate this further, but if anyone has ideas on how and where the map editor may mess up lights and particle defs (or how it might have done so in the past, and how it was fixed... hopefully ), I'd love to hear them.

 

Some final notes: reading and writing maps is a bit of a mess currently. The data structures are simply written to disk without any consideration for portability (padding bytes, byte order, ...). If the internal structure of the arrays ever changes, we'll be in for a world of hurt.

[Roja]

There are lights attached to some particles, like the fires for example. They come included in the particle file some how.

[Grum]

I've finished the script to remove lights and objects at invalid positions from the maps. Unix use:

python checkmap.py /path/to/maps/*.elm

to fix them all. The script will copy the old maps with a .bak suffix for backup purposes, so if it does mess up you can easily copy the old map back. Please let me know if it doesn't work as expected, though.

 

Roja, I'm sure Ent can get this running for you, but I'd be happy to do it for you if you can give me access to the development maps somehow.

 

The script can be download from http://www.ghnet.nl/~ge/downloads/checkmap.py

Edited July 3, 2007 by Grum

[Beaverhunter]

Nice, seems like a good script.

 

However this might not be completely on topic but it's still in the same matter:

When trying to fix "bad" lights, it should also be considered to fix "missing" lights, some maps lack lighting badly, especially caves. Using the New Lighting, which I know isn't completely approved and done yet, it reveals these kind of problems. Currently the client relies on the "flat" lighting system that barely requires any proper lighting sources placed. And I do assume we are leaving this kind of lighting soon so it would be nice if someone started to check this aswell.

Sorry if it's abit off topic.

[Roja]

Beaverhunter, lights are placed in the map editor. It's up to map makers to properly place the lighting.

 

Grum..I'll have to get you all the maps to process eventually(don't know when but certainly before the next update).

[Florian]

After the script ran:

 

Bad light (number 1) when loading './maps/cont2map18.elm'; co-ords [-0.764134 91.490128 6.899997] colour [1.000000 1.000000 2.999999]

[Grum]

Possible, ttl's check complains always when the light is outside the map, and it just id (x=-0.764134). I gathered you'd still be able to see that light though, so I gave it a bit more leeway (the script removes it when x or y is less than -10). Not sure what's the best thing to do here, but this actually seems a valid light.

[ttlanhil]

If I recall correctly, Ent said to use the map bounds; no light should be outside of the map. However, yes, a light that close to the border will probably have useful lighting; it'll change angles and cut-offs depending on how far out of the map it is, so being less than 0 doesn't always make it bad.

I would think it's unlikely that this is actually a corrupt light, as the values seem to be valid (compared to the range of values that you can see with the corrupt data; at least as light values).

Unless there's a good reason to cut-off at 0, I think it's probably best to extend the check in the code to -10 or so

[Entropy]

BTW, we are going to change the light format soon, so that script will be useful, with some modifications.

More details soon.

[Xaphier]

I would like to change the light structure that is saved in the maps from:

typedef struct
{
float pos_x;
float pos_y;
float pos_z;
float r;
float g;
float b;
char reserved[16];
}light_io;

to:

typedef struct
{
float pos[3];
float diffuse_colorr[3];
Uint8 specular_color[3];
Uint16 quadric_attenuation;
Uint16 range;
Sint16 cutoff;
Sint16 exponent;
Sint16 light_dir_x;
Sint16 light_dir_y;
Sint8 light_dir_z_sign;
}light_io;

I only used the reserved space for the new informations, so no change at the server is needed. That's why no float formats are used. After reading the light_io, the new values are scaled so they can be used. Specular color is scaled with 1/255 (that gives three floats from 0.0 to 1.0). For quadric attenuation, range and exponent a look up table could be used to give a non uniform scaling. Cutof, light_dir_x and light_dir_y are scaled with 1 / 32767 to give values from -1.0 to 1.0. Light dir z is reconstructed doing:

light_dir_X: the converted float value of light_dir_x

light_dir_Y: the converted float value of light_dir_y

light_dir_Z = light_dir_z_sign * sqrt(1.0 – light_dir_X * light_dir_X -  light_dir_Y *  light_dir_Y)

That way, we could use:

1. quadric light attenuation (more realistic)
   
2. specular lightning with differenct color for specular than diffuse
   
3. different quadric attenuation per light
   
4. spot lights
   
5. a distance after that attenuation starts (range, only with shaders)
   

To get all of this working with "new" and old maps, we should introduce a version to the map format. We could use one of the reserved chars in the header, because they should be zero for all maps. With the new format we also could fix some lightning bugs. To do so, we need to ensure that the diffuse color values are between 0.0 and 1.0. The values needed to get scaled to be in this range and the quadric attenuation needs adaption to the scaling factor so lights with color values greater than one have smaller attenuation. It would be great if this and the version adaption could be done with the python script that removes invalid lights.

[Grum]

I would like to change the light structure that is saved in the maps from:

typedef struct
{
float pos_x;
float pos_y;
float pos_z;
float r;
float g;
float b;
char reserved[16];
}light_io;

to:

typedef struct
{
float pos[3];
float diffuse_colorr[3];
Uint8 specular_color[3];
Uint16 quadric_attenuation;
Uint16 range;
Sint16 cutoff;
Sint16 exponent;
Sint16 light_dir_x;
Sint16 light_dir_y;
Sint8 light_dir_z_sign;
}light_io;

I only used the reserved space for the new informations, so no change at the server is needed.

You'll have to put light_dir_z_sign after specular_color then, otherwise the compiler will insert a padding byte after specular_color, and 3 after bytes after light_dir_z_sign. Yet another reason to get rid of those io structs...

It would be great if this and the version adaption could be done with the python script that removes invalid lights.

Sure, I'll update the script (there's some new stuff in there anyway), and put up a new version. Since we'll be inserting new values, any preferences for what they should be?

[Xaphier]

I know the problem with packing, I only choose this order for clarity

range, exponent, light_dir_x, light_dir_y and light_dir_z_sign should be zero as default value. cutoff should be -32768 as default and specular_color should be {255, 255, 255} as default. quadric_attenuation needs a bit calculation if the diffuse color values are bigger than one, but I am not done with that. At least, the map format version should be greater than zero after the update, so why not starting with one?

[Grum]

I know the problem with packing, I only choose this order for clarity

Ok, as long as we're clear on where that byte is actually located in the file

 

A new version of the script is available from http://www.ghnet.nl/~ge/downloads/checkmap.py . I've used the first unused byte in the header for the version number (currently byte res_2 in the io_struct). If we want to use a short int for version numbering i'll probably use bytes res_3 and res_4 instead, but we'll see. I've left the quadrice attenuation at 0 for now until we know what it should be. To write out a map with new lights and version number 1, use

checkmap.py -lw <map_name> ...

or if you like verbosity

checkmap.py --newlights --write <map_name> ...

Note that nothing is written unless you add the -w flag.

 

The new script also contains options for displaying a graphical representation of the tile and height maps. Play with it if you like, but it's not particularly useful yet. I'm using this script too to see if I can code up something simple that will allow us to make rooms other than the room you're in on inside maps invisible, which is why those options are there

Edited July 20, 2007 by Grum

[Xaphier]

The new script also contains options for displaying a graphical representation of the tile and height maps. Play with it if you like, but it's not particularly useful yet. I'm using this script too to see if I can code up something simple that will allow us to make rooms other than the room you're in on inside maps invisible, which is why those options are there

You could use a "group" id for every object. Then while loading the map, only objects with the group id we currently use are loaded. btw. you could use the bouding boxes of the objects etc. to build the PVS data

[Grum]

You could use a "group" id for every object. Then while loading the map, only objects with the group id we currently use are loaded.

That was the idea

btw. you could use the bouding boxes of the objects etc. to build the PVS data

*googles for PVS, no luck. More with wikipedia...ah. Potentially Visible Set*

 

Yup, something like that. We probably also have to take the height map into account, for things like the invisible maze in VC. For now, I'm just toying

[Roja]

Hi,

 

I pm'd Grum but have yet to hear from him. Can someone please help me to run this python script on windows? Or can volunteer to do the map conversion for me(I'll send you the maps).

Thanks!

[Grum]

Hmm...odd. I was sure I did reply.

 

Anyway, I'd be happy to convert them for you, just send the maps or let me know where to get them.

[Roja]

Ok I emailed them to you..if you can get them back to me ASAP today that'd be really great cause we're doing the test client today.

[Grum]

<Converted maps have been sent to Roja>

 

The map editor has been updated to insert default values for the new light format when it is compiled with NEW_LIGHT_FORMAT. The parameters are the same as checkmap.py inserts, so the map format should now be consistent again. The new parameters aren't used in the client, so the change shouldn't have any effect, but some testing (i.e. compile the map editor with NEW_LIGHT_FORMAT, load and save a few maps, and make sure nothing weird happens, esp. with the lights) would be much appreciated.

[Entropy]

We didn't include the maps in the client ebcause they were way bigger than the original. Any idea what caused that size increase?

[Grum]

Yes, the cluster maps need space, especially for larger maps (2 bytes for every height tile). When you zip the maps, the effect is limited, though (7.1 vs. 6.5 MB).

[Entropy]

Ah, ok, great.

The client has default support for the cluster maps?

They are used for not seeing stuff that you can't walk to from the current position, correct?

If so, maybe only the inside maps should have that? The ones tagged as dungeon.

[Learner]

Ah, ok, great.

The client has default support for the cluster maps?

They are used for not seeing stuff that you can't walk to from the current position, correct?

If so, maybe only the inside maps should have that? The ones tagged as dungeon.

Some 'indoor' spaces use empty space on outdoor maps. One example I can think of is the inside of the huts in MM I think are part of Grubani outside? I could be mistaken, but I think I saw that when the ants kept getting stuck in those huts.

[Roja]

 

Some 'indoor' spaces use empty space on outdoor maps. One example I can think of is the inside of the huts in MM I think are part of Grubani outside? I could be mistaken, but I think I saw that when the ants kept getting stuck in those huts.

 

No they're not. We only do that with guildmaps.

[Roja]

Grum, using your maps and the client from RC2, I still get a bad light error:

 

[07:53:23] Bad light (number 4) when loading './maps/cont2map20.elm'; co-ords [-1.317150 21.556053 7.299996] colour [5.099998 5.099998 5.099998]