Jump to content
Eternal Lands Official Forums
ttlanhil

New file handling routines

Recommended Posts

I've developed, with some help from Xaphier, some additional file handling routines. By doing the file opening in one place, we have some advantages. With just the one call (which means it's already shorter code, as there's no safe_snprintf() stuff to build the file name for each file that will be opened), we can have:

- Check for files in configdir/updates (both read and write. If you don't have write permissions for your datadir, this means you can finally use AUTO_UPDATE).

- Better multi-platform support. Currently, you need at least an #ifdef WINDOWS in many places. Now it's just the one call.

- Easier multilingual support. Just the one call will try `lang` as well as "en" (and for those who have seen this combined with the #ifdef WINDOWS, you'll know it wasn't that pretty).

- Automatic checking for a file in several places. In the existing code, in a number of places, there's a check for a file in the configdir, followed by a check in the datadir. This now happens automatically for all files.

- ~ on Windows. Yes, now Windows users will have a separate configdir as well. Where exactly this will be varies depending on which version of Windows you have, but it will be something like:

C:\Documents and Settings\username\Application Data\elc\el.ini

Which may be quite an advantage for households where a computer is shared and several people play EL.

 

Initially, this will mean the code is larger, and there are more #ifdef's... however, once this becomes standard, the code size will drop, as well as becoming more readable as you can skip over a dozen lines of #ifdef WINDOWS try `lang` etc

 

Still to be done:

- Version support in the use of configdir/updates, so that a client update won't be broken by files AUTO_UPDATE'd into /updates. This will likely be just a case of appending the client version onto "updates".

- Direct access to a specific file. This should only be needed by AUTO_UPDATE, which can check to see if the file in datadir is current. If so, and there's one in /updates, the one in /updates can be removed. This checking will also need to be added.

- Addition of error logging in some places. Where my_fopen() used to be used, a new error log will have to be used on failure.

- Removal of the remaining fopen or my_fopen calls, as well as the my_fopen and related functions. I think most of what's remaining is binary files, which Xaphier's file loaders are designed for; the loading of keys.ini is a trickier case than just replacing a fopen() call... However I plan to work on the use of hotkeys when I have some spare time anyway :)

 

Now, this will use the NEW_FILE_IO define, which Xaphier's existing file handling also uses. If you want to test it out, you'll need to enable this flag (otherwise you get exactly the same behaviour as without the patch). Go to Patches to download it. You'll probably need to

patch -p0 -i filename [--dry-run]

Without the -p0 it may fail to find the existing files in /io/

 

Note also, if you do try this, some of your files will be saved in a new place (that includes on *nix, all the *elm.txt files (mapmarks) and *.xm (exploration data for minimap) are now in configdir/maps/ instead of configdir/ ), where the old client will not find them.

If you swap back and forth, you'll need to manually move files.

The new client shouldn't have any problems opening files in the old locations, but it will only check in the old place if files are not found in the new place. In other words, it will work the first time (per file that gets saved), and after that you will have two separate sets of files.

 

I've played with this for a while myself, and have had a few other people use the code, but now I need wider testing and bug reporting. :P

Share this post


Link to post
Share on other sites

Wow. That is a lot of new code. Makes me a little hesitant to do the patch. =)

Some notes and questions. If I understand this correctly, each user will get their own "maps" directory into which they can put custom maps. Right? Or is this just for the *.elm.txt files? I have not read the patch too closely yet, but will the files be separated into new directories by playername? This would be also nice for separating chat_logs (I know others have suggested chat_log_playername.txt in the past).

 

I have an issue with elpathwrapper.c's handling of the OSX code. You have tried to maintain the path settings as I had previously in the basecode (as we discussed in PM). By default on OS X, (HOME)"Library/Application\ Support/" will exist, so you don't need to create it.

+	strcpy(locbuffer, getenv("HOME"));
+	if(pwd[0] != '\0'){
+		chdir(locbuffer);
+#ifdef OSX
+		mkdir("Library/Application\ Support/Eternal\ Lands/", 0700);
+		chdir(pwd);
+	}
+	strcat(locbuffer, "Library/Application\ Support/Eternal\ Lands/");
+#else // *nix!

Although I guess it cannot hurt to be safe. However, we should probably check to make sure the directory creation was successful.

 

If others are okay with the code, go ahead and commit. I will play catchup. :P

Share this post


Link to post
Share on other sites

If we are doing this, we might as well add some support for Vista.

AFAIK (and I might be wrong), in Vista you can't put stuff in Program Files, you need to add the stuff somewhere like document and settings/users/some directory

So this would help with the auto update.

 

Nevertheless, we need to get rid of the two el.ini for Linux, where you need (sometimes) to edit it in two places to get stuff to work.

 

So what I think we should do is come up with a predefined folder for each OS, where the dynamic stuff (logs, update things, etc.) goes.

Share this post


Link to post
Share on other sites

If we are doing this, we might as well add some support for Vista.

AFAIK (and I might be wrong), in Vista you can't put stuff in Program Files, you need to add the stuff somewhere like document and settings/users/some directory

So this would help with the auto update.

 

Nevertheless, we need to get rid of the two el.ini for Linux, where you need (sometimes) to edit it in two places to get stuff to work.

 

So what I think we should do is come up with a predefined folder for each OS, where the dynamic stuff (logs, update things, etc.) goes.

The linux el.ini in the install directory is only used for when a new user is running the game for the first time. After that is always uses $HOME/.elc/el.ini. It's not that hard to remember, but many people don't know about that other directory where chat logs, etc are kep.

Share this post


Link to post
Share on other sites
Wow. That is a lot of new code. Makes me a little hesitant to do the patch. =)
It is large, but in most cases the changes are just small chunks replacing some string manipulation and a fopen call with a call to the new functions (or sometimes the string manipulation is still there)
Some notes and questions. If I understand this correctly, each user will get their own "maps" directory into which they can put custom maps. Right? Or is this just for the *.elm.txt files?
the configdir/maps/ folder is only used for .elm.txt and .xm, that's no different from hiw it is at the moment except that there's a new folder instead of sticking these all in /configdir (which is what happens now, and it makes it a bit harder to see relevant files).
I have not read the patch too closely yet, but will the files be separated into new directories by playername? This would be also nice for separating chat_logs (I know others have suggested chat_log_playername.txt in the past).
It's not; it could be, but I think splitting by player name is only a good idea for some files, such as the logs. In most cases, files should be shared between EL accounts
Although I guess it cannot hurt to be safe. However, we should probably check to make sure the directory creation was successful.
True and true. I'll add the checks soon
If we are doing this, we might as well add some support for Vista.

AFAIK (and I might be wrong), in Vista you can't put stuff in Program Files, you need to add the stuff somewhere like document and settings/users/some directory

So this would help with the auto update.

In Vista, the call I use to get the user's home directory is deprecated but remains as a wrapper around the new functions. As such, I didn't think it was necessary to add more code to deal with both access types.

That said, I don't see why Vista should be any different to *nix, now, as EL on all OSes will use data_dir for data, and a place in $HOME for config.

When AUTO_UPDATE runs, it will try to update data_dir if it can, and then try updating in config_dir if it couldn't open in data_dir.

This means if you have permissions to update the data_dir, it will update for all users and save disk space. This should be a win. The only problem I can forsee is if Vista is so annoying as to ask the user for permission to write to data_dir when the code is intended to failover

Nevertheless, we need to get rid of the two el.ini for Linux, where you need (sometimes) to edit it in two places to get stuff to work.
We can't, and you don't. I haven't changed how this works, you edit the one in $HOME to take effect, and it's only if you don't have one in $HOME that the one in the data_dir is looked at at all
So what I think we should do is come up with a predefined folder for each OS, where the dynamic stuff (logs, update things, etc.) goes.
That's what this does. Except that on OSX and *nix you know exactly where it will be; on Windows it depends on which version you have, as we use a Windows call to get the user's application data directory (which should become the roaming application data directory on Vista, according to MSDN)
It's not that hard to remember, but many people don't know about that other directory where chat logs, etc are kep.
Unfortunately, my patch won't help this at all. But saving anywhere else is wrong... the one alternative is not using a dot-directory, as the elc data is not just config files that the application will manage itself, but also files that the user may edit.

Other than that, we need to document the use of home directories so users know where to find files if they need to (perhaps even on the download page?)

 

If desired, I can provide a C program of a dozen lines or so to print where the config dir will be, I currently have one for the windows directory (which is by far the hardest of the bunch, and thanks to LabRat for testing this for me so I could develop the use of $HOME on windows)

Share this post


Link to post
Share on other sites

Other than that, we need to document the use of home directories so users know where to find files if they need to (perhaps even on the download page?)

What about putting something on startup (like the graphics checks) that lists what el.ini file was used. Something like:

 

Looking for ini in <configdir>. Not found.

Looking for ini in default data dir <curr_dir>. Found, using.

Creating config dir <configdir>. Done. el.ini and log files will be saved to this directory.

 

That way people can see where the directory is when the client loads, and know where to find the files.

It could also help debugging a little as they don't need to find the error log files. Alternatively if we think it could confuse people, output that info into the error log.

Share this post


Link to post
Share on other sites

Printing the location of configdir on startup sounds reasonable, I'll add that.

However, I don't think printing it to the error_log will be a good idea; as by the time they find the error_log, they know where configdir is. ;)

Share this post


Link to post
Share on other sites
Printing the location of configdir on startup sounds reasonable, I'll add that.
Done. It'll print "Your personal settings and logs will be saved in %s" after the OpenGL checks, and before "Connecting to server".

I've also appended VER_MAJOR and VER_MINOR to the configdir/updates folder name.

Otherwise I haven't changed anything in my tree; and I haven't updated the patch.

 

Are there any other questions/suggestions/requests?

ed: And bug reports, of course :(

Edited by ttlanhil

Share this post


Link to post
Share on other sites
Printing the location of configdir on startup sounds reasonable, I'll add that.
Done. It'll print "Your personal settings and logs will be saved in %s" after the OpenGL checks, and before "Connecting to server".

I've also appended VER_MAJOR and VER_MINOR to the configdir/updates folder name.

Otherwise I haven't changed anything in my tree; and I haven't updated the patch.

 

Are there any other questions/suggestions/requests?

ed: And bug reports, of course :icon4:

Please don't append the version for config/update automatically. For the majority of the players you don't want that, about the only people that would use that without messing up there systems are the Devs & some testers. Otherwise we will get flooded with complaints about losing configs etc everything there is a new client released.

Share this post


Link to post
Share on other sites

I think you've missed the point of what /updates is for... that's where it puts AUTO_UPDATE stuff if there's no write permissions for the data dir. Only auto-updated (or custom-updated) files will go there.

the configdir and all config files are untouched by that change.

 

configdir - all configuration files and logs and misc other files

configdir/maps - all mapmarks and minimap exploration data

configdir/updates - overrides for files in data_dir, for use when data_dir doesn't give you write permissions

Edited by ttlanhil

Share this post


Link to post
Share on other sites

I think you've missed the point of what /updates is for... that's where it puts AUTO_UPDATE stuff if there's no write permissions for the data dir. Only auto-updated (or custom-updated) files will go there.

the configdir and all config files are untouched by that change.

 

configdir - all configuration files and logs and misc other files

configdir/maps - all mapmarks and minimap exploration data

configdir/updates - overrides for files in data_dir, for use when data_dir doesn't give you write permissions

The problem is you will have people with 20+ sets of directories! Thats why it shouldn't do that for the normal client!

Share this post


Link to post
Share on other sites
The problem is you will have people with 20+ sets of directories! Thats why it shouldn't do that for the normal client!
How? One configdir per computer-user-account, with two subdirectories, and possibly more in /updates (created as needed).

The client version number will only be used in the name of the updates dir, which gives overrides to the datadir. The only way we don't do this is if you can guarantee that old auto-updated files won't break anything during a full client update. Otherwise we need to be certain we don't use old auto-update'd files.

And since the files are no longer in the place where they'll get wiped automatically during the update, it takes some protection against using old files.

 

Semimythical/trimmed directory listing

user@bugs:~/build/elc$ ls -R ~/.elc
~/.elc:
error_log.txt
chat_log.txt
commands.lst
counters_ttlanhil.dat
spells_ttlanhil.dat
local_ignores.txt
screenshots/
el.cfg
maps/
el.ini
notes.xml
updates14/

~/.elc/maps:
anitora.elm.txt
cont2map1_caves.elm.txt
cont2map2.elm.txt
map13.elm.txt
map7_cave.xm
map13.xm

~/.elc/screenshots:
elscreen001.png  elscreen003.png  elscreen012.png  elscreen051.png
elscreen002.png  elscreen004.png  elscreen013.png  elscreen060.png

~/.elc/updates14:
particles/
textures/

~/.elc/updates14/particles:
waterfall.part

~/.elc/updates14/textures:
book1.bmp

ed: adjusted for the new updatesVER scheme

Edited by ttlanhil

Share this post


Link to post
Share on other sites
The problem is you will have people with 20+ sets of directories! Thats why it shouldn't do that for the normal client!
How? One configdir per computer-user-account, with two subdirectories, and possibly more in /updates (created as needed).

The client version number will only be used in the name of the updates dir, which gives overrides to the datadir. The only way we don't do this is if you can guarantee that old auto-updated files won't break anything during a full client update. Otherwise we need to be certain we don't use old auto-update'd files.

And since the files are no longer in the place where they'll get wiped automatically during the update, it takes some protection against using old files.

 

Semimythical/trimmed directory listing

user@bugs:~/build/elc$ ls -R ~/.elc
~/.elc:
error_log.txt
chat_log.txt
commands.lst
counters_ttlanhil.dat
spells_ttlanhil.dat
local_ignores.txt
screenshots/
el.cfg
maps/
el.ini
notes.xml
updates14/

~/.elc/maps:
anitora.elm.txt
cont2map1_caves.elm.txt
cont2map2.elm.txt
map13.elm.txt
map7_cave.xm
map13.xm

~/.elc/screenshots:
elscreen001.png  elscreen003.png  elscreen012.png  elscreen051.png
elscreen002.png  elscreen004.png  elscreen013.png  elscreen060.png

~/.elc/updates14:
particles/
textures/

~/.elc/updates14/particles:
waterfall.part

~/.elc/updates14/textures:
book1.bmp

ed: adjusted for the new updatesVER scheme

If the version number is applied only to the updates folder, then you still will have 10+ directory trees very quickly for players who wont even knowabout them. Also remember that ANYTHING except the executable could be included in an auto update, even a map. Just so far we've been avoiding doing that.

Share this post


Link to post
Share on other sites
If the version number is applied only to the updates folder, then you still will have 10+ directory trees very quickly for players who wont even knowabout them.
No you won't. Those numbers only change on a client update (at least that I've seen, and there's also why I avoided the 3rd digit of the version number). So far that seems to be a coupe of times per year.

And even if that were true, you're assuming I'll leave old update directories lying around, which I won't (what use are they?).

If there were a safe way to wipe a directory when you go to a new client version, I could get away with one updates/ dir, but I'd rather not trust that.

Also remember that ANYTHING except the executable could be included in an auto update, even a map. Just so far we've been avoiding doing that.
That's fine.

All the file handling will check for updates properly, because it all goes through one of a few functions set up for this. This isn't a problem.

Apart from the file loaders, none of ELC has any clue where the file comes from; you can load a map from datadir and some of the textures it needs from configdir/updates.

Share this post


Link to post
Share on other sites
If the version number is applied only to the updates folder, then you still will have 10+ directory trees very quickly for players who wont even knowabout them.
No you won't. Those numbers only change on a client update (at least that I've seen, and there's also why I avoided the 3rd digit of the version number). So far that seems to be a coupe of times per year.

And even if that were true, you're assuming I'll leave old update directories lying around, which I won't (what use are they?).

If there were a safe way to wipe a directory when you go to a new client version, I could get away with one updates/ dir, but I'd rather not trust that.

Also remember that ANYTHING except the executable could be included in an auto update, even a map. Just so far we've been avoiding doing that.
That's fine.

All the file handling will check for updates properly, because it all goes through one of a few functions set up for this. This isn't a problem.

Apart from the file loaders, none of ELC has any clue where the file comes from; you can load a map from datadir and some of the textures it needs from configdir/updates.

You're going to run into having to wipe that dir with ANY client update related to all the numbers (and they aren't limited to a single digit). Or the version has to include all the numbers, or you will have problems.

Share this post


Link to post
Share on other sites
You're going to run into having to wipe that dir with ANY client update related to all the numbers (and they aren't limited to a single digit).
Okay, I changed that part. It's now:
snprintf(updatepath, sizeof(updatepath), "updates_%d_%d_%d_%d", VER_MAJOR, VER_MINOR, VER_RELEASE, VER_BUILD);

Share this post


Link to post
Share on other sites
You're going to run into having to wipe that dir with ANY client update related to all the numbers (and they aren't limited to a single digit).
Okay, I changed that part. It's now:
snprintf(updatepath, sizeof(updatepath), "updates_%d_%d_%d_%d", VER_MAJOR, VER_MINOR, VER_RELEASE, VER_BUILD);

That still doesn't solve my concern of a player getting 20 directories of autoupdate stuff and not realizing it.

Share this post


Link to post
Share on other sites

No, but what I said before about removing old ones when the client version changes does.

ed: Oh, and another thing, any files in /updates that aren't listed in the auto-update file list will be removed. Any files where the one in datadir matches the checksum (hebce datadir is updated) will have the one in /updates deleted. It won't be useful for player-made files, but it won't be a disk hog.

Edited by ttlanhil

Share this post


Link to post
Share on other sites

Would be better if instead of

C:\Documents and Settings\username\Application Data\elc\el.ini

The path is

C:\Documents and Settings\username\Application Data\[name of el folder in program files]\el.ini

This way there may be a couple el.inis on the same account. Personally i'd like to have at least 2, one for the main and one for the test server

Share this post


Link to post
Share on other sites

You can't assume that the datadir is in Program Files... It's entirely possible that someone will install EL as, say, "K:\".

As for multiple el.ini's, the game supports setting options on the command line, so you can always do something like

testserv.bat:
start elc.exe -sp=2001 --othersetting=value

Then simply double-click whichever batch file has the settings you want for that run of the client.

If you find that setting options on the command line is a hassle because of the number you need to change, you can also make a bunch of different el.ini files, and copy the one you want to 'el.ini' on startup in a different batch file.

Share this post


Link to post
Share on other sites

I've uploaded a new version to Berlios.

The version number stuff is used on configdir/updates, I've added more logging for failing to open files, syscall return checking, etc.

All the auto-update support should now work.

 

Still needed:

- keys.c needs work. But I intend to work on that separately later.

- A recursive rmdir() function. I can come up with one of these for linux without too much work, but it will need work for other platforms as well. On the up side, this isn't needed yet (it will only come into play when the client version changes. And Since it will clear all non-current updates, it can be added 3 updates down the track and all that happens is we use up some extra disk space in the mean time)

- Removal of the last few (my_)fopen() calls, and the my_fopen() function itself. This will happen a bit later on as the function is used in a few places still.

 

Please test, where possible. If there are no problems or requests reported in a while, I'll put it in CVS. :)

 

ed: Actually, I do have something else to adjust already... I don't think there a reason why we can't have an option like

-DCONFIGDIR=.elfolder

Which would make all config files in (for the *nix example) ~/.elfolder/

So unless someone comes up with a reason not to do this, I'll add it to my next version (which will likely be the one checked in)

Edited by ttlanhil

Share this post


Link to post
Share on other sites

- A recursive rmdir() function. I can come up with one of these for linux without too much work, but it will need work for other platforms as well.

something like

int remove_directory (const char* name)
{
 DIR* dir = opendir (name);
 struct dirent* entry;
 int res = 1;

 if (!dir)
return 0;

 // clear the directory
 while ( (entry = readdir (dir)) )
 {
char buf[256];
safe_snprintf (buf, sizeof (buf), "%s/%s", name, entry->d_name);

if (entry->d_type == DT_DIR)
{
  if (strcmp (entry->d_name, ".") != 0 && strcmp (entry->d_name, "..") != 0)
  {
	res &= remove_directory (buf);
  }
}
else
{
  if (remove (buf))
	res = 0;
}
 }

 // now remove the directory
 if (res) 
if (remove (name))
  res = 0;

 return res;
}

should be reasonably portable, I guess.

 

EDIT: of course, this is entirly untested and not guaranteed to compile :)

EDIT 2: some fixes.

Edited by Grum

Share this post


Link to post
Share on other sites
EDIT: of course, this is entirly untested and not guaranteed to compile :)
Check out font.c, with the FONTS_FIX stuff, about reading the directory... I'd be quite surprised if this worked on windows like it should on a real OS.

Also, unless we have many files in datadir it probably won't be a problem, but you're opening up many directories while you recurse. One of the things I saw when looking into those was `man ftw`.

Of course, with your code, one could simply closedir() before you do the recursive call and open/read again later, if it turned out to be a problem.

Share this post


Link to post
Share on other sites

Sigh... I read POSIX in the man pages and assumed that MSVC would implement that. Silly me...

 

As for the nr of open directories being a problem: I'd be surprised if we'd create a directory tree where that becomes a problem. But adding a closedir () before the directory remove call would be a good idea :)

Share this post


Link to post
Share on other sites
Sigh... I read POSIX in the man pages and assumed that MSVC would implement that. Silly me...
Yup! :)
As for the nr of open directories being a problem: I'd be surprised if we'd create a directory tree where that becomes a problem.
Me too. For this I'd merely put a note about it at the start of the function, in case someone else grabs the code for some other job (if we do run out of depth, it'd probably be because people are doing funky things with symlink loops, and deserve the problems). Of course, checking the return value of the syscalls isn't a bad idea.
But adding a closedir () before the directory remove call would be a good idea :D
Ooh, yes, I'd say so.

 

I've added the ability to do:

FEATURES += CONFIGDIR=\".elc2/\"

in my local tree, with appropriate warnings about not misusing it (the short form is, unless you have a good reason to do it, just leave it alone :) ). As an added advantage, this reduced the number of hardcoded strings as well as removing an #ifdef/#else/#endif set (to be replaced by another in a different place, but at least that makes it more readable).

So I think the change in general is a positive one even if no-one uses the feature.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

×