e3d format

[KarenRei]

I had an issue that I initially posted on this thread:

 

http://www.eternal-lands.com/forum/index.p...mp;#entry401055

 

It got no responses. So, I looked up who made the changes (Xaphier) and mailed him directly. Again, no responses. So, I'm starting a new thread just for this issue.

 

As mentioned first over a year ago, a fair number of EL objects have defective normals. You don't notice them that much in EL as is because EL is incredibly flat, visually. However, whenever you make EL not look flat (a stated goal long ago, which has been repeated since by Roja, and one of the motives behind me working on NEW_LIGHTING, and actually the main motive behind me joining the programming team in general), these make things look bad wherever they're found. As a consequence, I wrote FIX_NORMALS -- a very simple, very fast function that computes the normals on all objects when the object is loaded and ignores the loaded normals. In an ideal situation, the normals aren't stored in the object at all, because computing them is much faster than loading them from disk. Even if they're loaded and then computed, the computing is still time-insignificant.

 

Writing FIX_NORMALS was easy. There were these nice neat arrays -- an array for vertices, an array for normals, an array for face indices, and so on, and the calculations for computing normals are very simple. But almost six months ago, the object loading code and how things were stored changed. load_3d_object_detail() moved to io/e3d_io.c and became radically different. Unbeknownst to me, FIX_NORMALS was not included in this migration.

 

I attempted to re-implement it, but there have been problems. Unlike the nice clean arrays before, now, the data is kind of jumbled all together, and I'm trying to wrap my head around it. To follow, open up io/e3d_io.c and scroll down to FIX_NORMALS in load_3d_object_detail. I attempted to base it on how the gl drawing array functions were being provided pointers to what they needed to draw.

 

In a given object, I'm assuming all of the normals in an object are ((float*)(normals_ptr + v_size * j))[0] through [2], where normals_ptr is cur_object->vertex_data + get_normal_offset(cur_object->vertex_options), and j ranges from 0 to cur_object->vertex_no. I'm picking faces by:

 

int face_list[VERTEX_FLOAT_COUNT];

if (indicies_size == 2)

{

for (k = 0; k < VERTEX_FLOAT_COUNT; k++)

face_list[k] = short_list[j * VERTEX_FLOAT_COUNT + k];

}

else

{

for (k = 0; k < VERTEX_FLOAT_COUNT; k++)

face_list[k] = int_list[j * VERTEX_FLOAT_COUNT + k];

}

 

where j ranges from 0 to cur_object->index_no / VERTEX_FLOAT_COUNT. The results from this typically look correct. I'm then assuming that the vertices at each face in the same range is ((float*)(vertex_ptr + v_size * face_list[0] through [2]))[0] through [2], where vertex_ptr is cur_object->vertex_data + get_vertex_offset(cur_object->vertex_options);.

 

This works on most objects, but not on all of them. For example, with feather1.e3d, it works fine, and when normals are computed with these vertices and faces, we get the same results as feather1.e3d's (properly set) normals. But with arrow1.e3d, which also had properly set normals, the vertex data gotten by the above assumptions is bonkers. Vertex 0 is identical to vertex 1, vertex 2 is the same as 3, 4=5, 6=7, and so on. And I have no clue why. In arrow1.e3d's case, it thinks the coordinates of vertices 0 through 4 are:

 

0) 0.066864, 0.003510, 0.006104

1) 0.066864, 0.003510, 0.006104

2) -0.443512, 0.003510, 0.038361

3) -0.443512, 0.003510, 0.005890

 

And so on. Needless to say, this produces completely invalid normals, as it doesn't make any sense. So, if anyone has a clue what I'm misunderstanding that's causing things to break, I'd be much obliged.

Edited December 4, 2007 by KarenRei

[Florian]

FIX_NORMALS causes a crash for me

 

http://el.beplacid.net/bugs/view.php?id=12

[alvieboy]

0) 0.066864, 0.003510, 0.006104

1) 0.066864, 0.003510, 0.006104

2) -0.443512, 0.003510, 0.038361

3) -0.443512, 0.003510, 0.005890

 

And so on. Needless to say, this produces completely invalid normals, as it doesn't make any sense. So, if anyone has a clue what I'm misunderstanding that's causing things to break, I'd be much obliged.

 

I wonder if you could skip vertices if distance from previous one is lower than some threshold (i.e., 1e-10). That way you could spot those duplicates, and end up with only 2 vertices.

 

This is not proper way to do it, though. The file should be fixed imho.

Maybe we can hack some script to check those files.

 

 

Álvaro

[Entropy]

I am not sure (Xaphier is busy for the time being), but I think the idea was to actually fix the normals in the exporting script, and/or make a script to fix all the normals altogether, rather than have a runtime thing that fixes them all the time.

[Xaphier]

Sorry for my late reply. Yes, I am working at fixing the im- & exporter scripts for blender. I also try to reuse the python code to be able to fix all files without to im- & export them in blender. The blender scripts are working, but I am no python expert and I have some problems getting them working without blender (I am not shure how to handle modules ).

[KarenRei]

Thanks, Xaphier If you have any python questions, let me know. I use it extensively at work. What are your module problems? If it's just how to use them in general, consider them to be like C++ namespaces. There are two common ways to import:

 

import modulename <= equivalent to including the include file that defines the namespace and its functions in C++

from modulename import * <= equivalent to "using namespace modulename" in C++

 

You can also import things other than "*" in the second case (i.e., a limited subset). Basically, if you did:

>>> import os

 

You could do:

>>> print os.environ

 

But if you did:

>>> from os import *

Or...

>>> from os import environ

 

All you'd need to do is:

>>> print environ

 

If your problems are figuring out how to use a specific module, import it, then do a dir on it as such:

>>> import os

>>> dir(os)

 

That'll list its contents. In python, the standard for documentation is to have a member variable called __doc__. So, for example:

>>> import os

>>> print os.close.__doc__

close(fd)

 

Close a file descriptor (for low level IO).

>>>

 

If your problems are more specific than this, let me know.

Edited December 4, 2007 by KarenRei

[KarenRei]

Since this is being worked on, when I get a chance this evening, I'll remove FIX_NORMALS from the code.

[Xaphier]

I will commit the code later

Edited December 7, 2007 by Xaphier

[Xaphier]

Now the python code for the converter and the blender im- & exporter scripts is in cvs. The module dir is python-tools

[KarenRei]

Wonderful. I'll give them a try tonight.

[KarenRei]

Ok, I ran into some errors. First off without any special arguments, just a -o to specify the output file and the input file:

 

Converting file: eternallands/3dobjects/ground_objs/wheat1.e3d

Traceback (most recent call last):

File "./converter.py", line 165, in <module>

main()

File "./converter.py", line 162, in main

convert(infile, outfile, check, uv_range, size)

File "./converter.py", line 25, in convert

load_el3d.load(file)

File "/home/meme/code/c/eternallands/python-tools/el3d-file-tools/el3d/el3d_file_obj.py", line 349, in load

self.options = self.header.load(file)

File "/home/meme/code/c/eternallands/python-tools/el3d-file-tools/el3d/el3d_file_obj.py", line 253, in load

data = file.read(self.size())

File "/home/meme/code/c/eternallands/python-tools/el3d-file-tools/el3d/el3d_file_obj.py", line 316, in size

return struct.calcsize(self.binary_format)

File "/usr/lib/python2.5/struct.py", line 51, in calcsize

o = _compile(fmt)

File "/usr/lib/python2.5/struct.py", line 39, in _compile

s = Struct(fmt)

struct.error: bad char in struct format

 

Indeed, that is an invalid struct string -- "<4c<4b<16c<10i<4B". The way the struct module works, either the whole struct is in little endian or it's not; you don't need to prefix each entry with a "<". The proper string is "<4c4b16c10i4B".

 

Also, I get errors earlier on when I try the -d option, which I take to mean it'll convert all e3d files recursively:

 

Converting file: /usr/games/eternallands/

Traceback (most recent call last):

File "./converter.py", line 165, in <module>

main()

File "./converter.py", line 162, in main

convert(infile, outfile, check, uv_range, size)

File "./converter.py", line 36, in convert

file.close()

UnboundLocalError: local variable 'file' referenced before assignment

 

This is correct. If open fails on line 34, the fallback case in the exception isn't going to have file assigned. Of course, the real problem here is that it's trying to run open(infile, 'rb'), and that's not going to work if infile is a directory So, am I just misunderstanding the -d option, or is this a bug?

[Xaphier]

I committed a fix for the struct strings and the vector calculation. Converting hole dirs work like this:

convert -d -o outdir indir

or for in place conversion:

convert -d indir

[KarenRei]

Aww, so close! Got a good portion of the way though, and then:

 

Converting file: /usr/games/eternallands/3dobjects/structures/door14.e3d

Converting file: /usr/games/eternallands/3dobjects/structures/insidehouse_snow4.e3d

Converting file: /usr/games/eternallands/3dobjects/structures/fence4.e3d

Converting file: /usr/games/eternallands/3dobjects/structures/door2.e3d

Traceback (most recent call last):

File "./converter.py", line 162, in <module>

main()

File "./converter.py", line 157, in main

convert_dirs(infile, outfile, check, uv_range, size)

File "./converter.py", line 105, in convert_dirs

convert_dir(file_name, in_dir, out_dir, check, uv_range, size)

File "./converter.py", line 95, in convert_dir

convert_file(name, in_dir, out_dir, check, uv_range, size)

File "./converter.py", line 74, in convert_file

name, ext = file_name.rsplit('.', 1)

ValueError: need more than 1 value to unpack

 

I know the meaning of that to be that there's no . in the path. I'm not sure what file_name.

 

Update: hmm, looks like you're not checking to see if what's being converted is an e3d, if I'm not mistaken. Well, at the very least, I added a try/except block to catch that error gracefully. And I was able to work around this bug anyways by converting /usr/games/eternallands/3dobjects instead of /usr/games/eternallands.

 

 

Eeek, bad! It frotzed all of my objects

Hopefully I've got a backup somewhere around here...

Edited December 8, 2007 by KarenRei

[Entropy]

You can redownload the Linux data from the download page.

[KarenRei]

Update: Restored, tried again, same thing happened So, something's wrong. If you need me to send you files, let me know.

[Xaphier]

Yes, please send me some of the files so I can try to find out what's wrong

[Entropy]

Can you convert just one object and see if it works or not?

[Xaphier]

I think I found the problem

It was no bug in the python code, it was a bug in the client. Is fixed in cvs and should work now.

[KarenRei]

Fresh checkout:

 

gcc -march=i686 -Wall -Wdeclaration-after-statement -O0 -ggdb -pipe -DLINUX -DELC -DAFK_FIX -DALPHA_ACTORS -DATI_9200_FIX -DAUTO_UPDATE -DCLICKABLE_CONTINENT_MAP -DCLUSTER_INSIDES -DCOUNTERS -DCUSTOM_LOOK -DCUSTOM_UPDATE -DCXX_MISC -DDEBUG_TIME -DEYE_CANDY -DFONTS_FIX -DFUZZY_PATHS -DIDLE_FIX -DMASKING -DMINES -DMINIMAP -DNEW_ACTOR_ANIMATION -DNEW_ACTOR_SCALE -DNEW_ALPHA -DNEW_FILE_IO -DNEW_LIGHTING -DNEW_SOUND -DNEW_TEX -DNOTEPAD -DOGG_VORBIS -DOPTIONS_I18N -DPNG_SCREENSHOT -DPOPUP -DSFX -DSIMPLE_LOD -DUSE_INLINE -DUSE_SEND_VIDEO_INFO -DZLIB -I/usr/include/SDL -D_GNU_SOURCE=1 -D_REENTRANT -I/usr/include/libxml2 -fno-strict-aliasing -c -o 2d_objects.o 2d_objects.c

In file included from 2d_objects.c:9:

load_gl_extensions.h:281: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix2x3fv’

load_gl_extensions.h:282: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix2x4fv’

load_gl_extensions.h:283: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix3x2fv’

load_gl_extensions.h:284: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix3x4fv’

load_gl_extensions.h:285: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix4x2fv’

load_gl_extensions.h:286: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix4x3fv’

make: *** [2d_objects.o] Error 1

[Florian]

Fresh checkout:

 

gcc -march=i686 -Wall -Wdeclaration-after-statement -O0 -ggdb -pipe -DLINUX -DELC -DAFK_FIX -DALPHA_ACTORS -DATI_9200_FIX -DAUTO_UPDATE -DCLICKABLE_CONTINENT_MAP -DCLUSTER_INSIDES -DCOUNTERS -DCUSTOM_LOOK -DCUSTOM_UPDATE -DCXX_MISC -DDEBUG_TIME -DEYE_CANDY -DFONTS_FIX -DFUZZY_PATHS -DIDLE_FIX -DMASKING -DMINES -DMINIMAP -DNEW_ACTOR_ANIMATION -DNEW_ACTOR_SCALE -DNEW_ALPHA -DNEW_FILE_IO -DNEW_LIGHTING -DNEW_SOUND -DNEW_TEX -DNOTEPAD -DOGG_VORBIS -DOPTIONS_I18N -DPNG_SCREENSHOT -DPOPUP -DSFX -DSIMPLE_LOD -DUSE_INLINE -DUSE_SEND_VIDEO_INFO -DZLIB -I/usr/include/SDL -D_GNU_SOURCE=1 -D_REENTRANT -I/usr/include/libxml2 -fno-strict-aliasing -c -o 2d_objects.o 2d_objects.c

In file included from 2d_objects.c:9:

load_gl_extensions.h:281: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix2x3fv’

load_gl_extensions.h:282: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix2x4fv’

load_gl_extensions.h:283: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix3x2fv’

load_gl_extensions.h:284: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix3x4fv’

load_gl_extensions.h:285: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix4x2fv’

load_gl_extensions.h:286: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘ELglUniformMatrix4x3fv’

make: *** [2d_objects.o] Error 1

 

See here: http://www.eternal-lands.com/forum/index.p...5028&st=380

[KarenRei]

Xaphier:

 

I was able to compile again, and the normals look messed up, not fixed. Things that were right before look all wrong now. Do you need a file sent?

[Xaphier]

Yes, please send me a file with the wrong normals and a screen shot, because I didn't see any problems . Also you can please look at the code that builds the normals, perhaps you can see a bug.

[KarenRei]

I looked at the normal code, and at first glance, it looked right, although I haven't looked at how it saves, loads, or uses normals because I'm not familiar with your format.

 

Screenshot, original models, lighting contrast increased:

http://www.daughtersoftiresias.org/progs/el/good.jpg

 

Screenshot, new models, same:

http://www.daughtersoftiresias.org/progs/el/bad.jpg

 

Model, original:

http://www.daughtersoftiresias.org/progs/e...k_big1.e3d.good

 

Model, new:

http://www.daughtersoftiresias.org/progs/e...ck_big1.e3d.bad

Edited December 12, 2007 by KarenRei

[Xaphier]

There was a bug in the normal and tangent calculation. I thought I could use the "+=" operator for my vector calculation, but that way a mistake . Now the normals aren't all zeros. New version is in cvs

[KarenRei]

Thanks I'll try it out again this evening.