ttlanhil Report post Posted June 17, 2007 I've developed, with some help from Xaphier, some additional file handling routines. By doing the file opening in one place, we have some advantages. With just the one call (which means it's already shorter code, as there's no safe_snprintf() stuff to build the file name for each file that will be opened), we can have: - Check for files in configdir/updates (both read and write. If you don't have write permissions for your datadir, this means you can finally use AUTO_UPDATE). - Better multi-platform support. Currently, you need at least an #ifdef WINDOWS in many places. Now it's just the one call. - Easier multilingual support. Just the one call will try `lang` as well as "en" (and for those who have seen this combined with the #ifdef WINDOWS, you'll know it wasn't that pretty). - Automatic checking for a file in several places. In the existing code, in a number of places, there's a check for a file in the configdir, followed by a check in the datadir. This now happens automatically for all files. - ~ on Windows. Yes, now Windows users will have a separate configdir as well. Where exactly this will be varies depending on which version of Windows you have, but it will be something like: C:\Documents and Settings\username\Application Data\elc\el.ini Which may be quite an advantage for households where a computer is shared and several people play EL. Initially, this will mean the code is larger, and there are more #ifdef's... however, once this becomes standard, the code size will drop, as well as becoming more readable as you can skip over a dozen lines of #ifdef WINDOWS try `lang` etc Still to be done: - Version support in the use of configdir/updates, so that a client update won't be broken by files AUTO_UPDATE'd into /updates. This will likely be just a case of appending the client version onto "updates". - Direct access to a specific file. This should only be needed by AUTO_UPDATE, which can check to see if the file in datadir is current. If so, and there's one in /updates, the one in /updates can be removed. This checking will also need to be added. - Addition of error logging in some places. Where my_fopen() used to be used, a new error log will have to be used on failure. - Removal of the remaining fopen or my_fopen calls, as well as the my_fopen and related functions. I think most of what's remaining is binary files, which Xaphier's file loaders are designed for; the loading of keys.ini is a trickier case than just replacing a fopen() call... However I plan to work on the use of hotkeys when I have some spare time anyway Now, this will use the NEW_FILE_IO define, which Xaphier's existing file handling also uses. If you want to test it out, you'll need to enable this flag (otherwise you get exactly the same behaviour as without the patch). Go to Patches to download it. You'll probably need to patch -p0 -i filename [--dry-run] Without the -p0 it may fail to find the existing files in /io/ Note also, if you do try this, some of your files will be saved in a new place (that includes on *nix, all the *elm.txt files (mapmarks) and *.xm (exploration data for minimap) are now in configdir/maps/ instead of configdir/ ), where the old client will not find them. If you swap back and forth, you'll need to manually move files. The new client shouldn't have any problems opening files in the old locations, but it will only check in the old place if files are not found in the new place. In other words, it will work the first time (per file that gets saved), and after that you will have two separate sets of files. I've played with this for a while myself, and have had a few other people use the code, but now I need wider testing and bug reporting. Share this post Link to post Share on other sites
0ctane Report post Posted June 17, 2007 Wow. That is a lot of new code. Makes me a little hesitant to do the patch. =) Some notes and questions. If I understand this correctly, each user will get their own "maps" directory into which they can put custom maps. Right? Or is this just for the *.elm.txt files? I have not read the patch too closely yet, but will the files be separated into new directories by playername? This would be also nice for separating chat_logs (I know others have suggested chat_log_playername.txt in the past). I have an issue with elpathwrapper.c's handling of the OSX code. You have tried to maintain the path settings as I had previously in the basecode (as we discussed in PM). By default on OS X, (HOME)"Library/Application\ Support/" will exist, so you don't need to create it. + strcpy(locbuffer, getenv("HOME")); + if(pwd[0] != '\0'){ + chdir(locbuffer); +#ifdef OSX + mkdir("Library/Application\ Support/Eternal\ Lands/", 0700); + chdir(pwd); + } + strcat(locbuffer, "Library/Application\ Support/Eternal\ Lands/"); +#else // *nix! Although I guess it cannot hurt to be safe. However, we should probably check to make sure the directory creation was successful. If others are okay with the code, go ahead and commit. I will play catchup. Share this post Link to post Share on other sites
Entropy Report post Posted June 17, 2007 If we are doing this, we might as well add some support for Vista. AFAIK (and I might be wrong), in Vista you can't put stuff in Program Files, you need to add the stuff somewhere like document and settings/users/some directory So this would help with the auto update. Nevertheless, we need to get rid of the two el.ini for Linux, where you need (sometimes) to edit it in two places to get stuff to work. So what I think we should do is come up with a predefined folder for each OS, where the dynamic stuff (logs, update things, etc.) goes. Share this post Link to post Share on other sites
Learner Report post Posted June 17, 2007 If we are doing this, we might as well add some support for Vista. AFAIK (and I might be wrong), in Vista you can't put stuff in Program Files, you need to add the stuff somewhere like document and settings/users/some directory So this would help with the auto update. Nevertheless, we need to get rid of the two el.ini for Linux, where you need (sometimes) to edit it in two places to get stuff to work. So what I think we should do is come up with a predefined folder for each OS, where the dynamic stuff (logs, update things, etc.) goes. The linux el.ini in the install directory is only used for when a new user is running the game for the first time. After that is always uses $HOME/.elc/el.ini. It's not that hard to remember, but many people don't know about that other directory where chat logs, etc are kep. Share this post Link to post Share on other sites
ttlanhil Report post Posted June 18, 2007 Wow. That is a lot of new code. Makes me a little hesitant to do the patch. =) It is large, but in most cases the changes are just small chunks replacing some string manipulation and a fopen call with a call to the new functions (or sometimes the string manipulation is still there)Some notes and questions. If I understand this correctly, each user will get their own "maps" directory into which they can put custom maps. Right? Or is this just for the *.elm.txt files?the configdir/maps/ folder is only used for .elm.txt and .xm, that's no different from hiw it is at the moment except that there's a new folder instead of sticking these all in /configdir (which is what happens now, and it makes it a bit harder to see relevant files).I have not read the patch too closely yet, but will the files be separated into new directories by playername? This would be also nice for separating chat_logs (I know others have suggested chat_log_playername.txt in the past).It's not; it could be, but I think splitting by player name is only a good idea for some files, such as the logs. In most cases, files should be shared between EL accountsAlthough I guess it cannot hurt to be safe. However, we should probably check to make sure the directory creation was successful.True and true. I'll add the checks soonIf we are doing this, we might as well add some support for Vista.AFAIK (and I might be wrong), in Vista you can't put stuff in Program Files, you need to add the stuff somewhere like document and settings/users/some directory So this would help with the auto update. In Vista, the call I use to get the user's home directory is deprecated but remains as a wrapper around the new functions. As such, I didn't think it was necessary to add more code to deal with both access types.That said, I don't see why Vista should be any different to *nix, now, as EL on all OSes will use data_dir for data, and a place in $HOME for config. When AUTO_UPDATE runs, it will try to update data_dir if it can, and then try updating in config_dir if it couldn't open in data_dir. This means if you have permissions to update the data_dir, it will update for all users and save disk space. This should be a win. The only problem I can forsee is if Vista is so annoying as to ask the user for permission to write to data_dir when the code is intended to failover Nevertheless, we need to get rid of the two el.ini for Linux, where you need (sometimes) to edit it in two places to get stuff to work.We can't, and you don't. I haven't changed how this works, you edit the one in $HOME to take effect, and it's only if you don't have one in $HOME that the one in the data_dir is looked at at allSo what I think we should do is come up with a predefined folder for each OS, where the dynamic stuff (logs, update things, etc.) goes.That's what this does. Except that on OSX and *nix you know exactly where it will be; on Windows it depends on which version you have, as we use a Windows call to get the user's application data directory (which should become the roaming application data directory on Vista, according to MSDN)It's not that hard to remember, but many people don't know about that other directory where chat logs, etc are kep.Unfortunately, my patch won't help this at all. But saving anywhere else is wrong... the one alternative is not using a dot-directory, as the elc data is not just config files that the application will manage itself, but also files that the user may edit.Other than that, we need to document the use of home directories so users know where to find files if they need to (perhaps even on the download page?) If desired, I can provide a C program of a dozen lines or so to print where the config dir will be, I currently have one for the windows directory (which is by far the hardest of the bunch, and thanks to LabRat for testing this for me so I could develop the use of $HOME on windows) Share this post Link to post Share on other sites
Torg Report post Posted June 18, 2007 Other than that, we need to document the use of home directories so users know where to find files if they need to (perhaps even on the download page?) What about putting something on startup (like the graphics checks) that lists what el.ini file was used. Something like: Looking for ini in <configdir>. Not found. Looking for ini in default data dir <curr_dir>. Found, using. Creating config dir <configdir>. Done. el.ini and log files will be saved to this directory. That way people can see where the directory is when the client loads, and know where to find the files. It could also help debugging a little as they don't need to find the error log files. Alternatively if we think it could confuse people, output that info into the error log. Share this post Link to post Share on other sites
ttlanhil Report post Posted June 18, 2007 Printing the location of configdir on startup sounds reasonable, I'll add that. However, I don't think printing it to the error_log will be a good idea; as by the time they find the error_log, they know where configdir is. Share this post Link to post Share on other sites
ttlanhil Report post Posted June 18, 2007 (edited) Printing the location of configdir on startup sounds reasonable, I'll add that.Done. It'll print "Your personal settings and logs will be saved in %s" after the OpenGL checks, and before "Connecting to server".I've also appended VER_MAJOR and VER_MINOR to the configdir/updates folder name. Otherwise I haven't changed anything in my tree; and I haven't updated the patch. Are there any other questions/suggestions/requests? ed: And bug reports, of course Edited June 18, 2007 by ttlanhil Share this post Link to post Share on other sites
Learner Report post Posted June 18, 2007 Printing the location of configdir on startup sounds reasonable, I'll add that.Done. It'll print "Your personal settings and logs will be saved in %s" after the OpenGL checks, and before "Connecting to server".I've also appended VER_MAJOR and VER_MINOR to the configdir/updates folder name. Otherwise I haven't changed anything in my tree; and I haven't updated the patch. Are there any other questions/suggestions/requests? ed: And bug reports, of course Please don't append the version for config/update automatically. For the majority of the players you don't want that, about the only people that would use that without messing up there systems are the Devs & some testers. Otherwise we will get flooded with complaints about losing configs etc everything there is a new client released. Share this post Link to post Share on other sites
ttlanhil Report post Posted June 18, 2007 (edited) I think you've missed the point of what /updates is for... that's where it puts AUTO_UPDATE stuff if there's no write permissions for the data dir. Only auto-updated (or custom-updated) files will go there. the configdir and all config files are untouched by that change. configdir - all configuration files and logs and misc other files configdir/maps - all mapmarks and minimap exploration data configdir/updates - overrides for files in data_dir, for use when data_dir doesn't give you write permissions Edited June 18, 2007 by ttlanhil Share this post Link to post Share on other sites
Learner Report post Posted June 18, 2007 I think you've missed the point of what /updates is for... that's where it puts AUTO_UPDATE stuff if there's no write permissions for the data dir. Only auto-updated (or custom-updated) files will go there. the configdir and all config files are untouched by that change. configdir - all configuration files and logs and misc other files configdir/maps - all mapmarks and minimap exploration data configdir/updates - overrides for files in data_dir, for use when data_dir doesn't give you write permissions The problem is you will have people with 20+ sets of directories! Thats why it shouldn't do that for the normal client! Share this post Link to post Share on other sites
ttlanhil Report post Posted June 18, 2007 (edited) The problem is you will have people with 20+ sets of directories! Thats why it shouldn't do that for the normal client!How? One configdir per computer-user-account, with two subdirectories, and possibly more in /updates (created as needed).The client version number will only be used in the name of the updates dir, which gives overrides to the datadir. The only way we don't do this is if you can guarantee that old auto-updated files won't break anything during a full client update. Otherwise we need to be certain we don't use old auto-update'd files. And since the files are no longer in the place where they'll get wiped automatically during the update, it takes some protection against using old files. Semimythical/trimmed directory listing user@bugs:~/build/elc$ ls -R ~/.elc ~/.elc: error_log.txt chat_log.txt commands.lst counters_ttlanhil.dat spells_ttlanhil.dat local_ignores.txt screenshots/ el.cfg maps/ el.ini notes.xml updates14/ ~/.elc/maps: anitora.elm.txt cont2map1_caves.elm.txt cont2map2.elm.txt map13.elm.txt map7_cave.xm map13.xm ~/.elc/screenshots: elscreen001.png elscreen003.png elscreen012.png elscreen051.png elscreen002.png elscreen004.png elscreen013.png elscreen060.png ~/.elc/updates14: particles/ textures/ ~/.elc/updates14/particles: waterfall.part ~/.elc/updates14/textures: book1.bmp ed: adjusted for the new updatesVER scheme Edited June 18, 2007 by ttlanhil Share this post Link to post Share on other sites
Learner Report post Posted June 18, 2007 The problem is you will have people with 20+ sets of directories! Thats why it shouldn't do that for the normal client!How? One configdir per computer-user-account, with two subdirectories, and possibly more in /updates (created as needed).The client version number will only be used in the name of the updates dir, which gives overrides to the datadir. The only way we don't do this is if you can guarantee that old auto-updated files won't break anything during a full client update. Otherwise we need to be certain we don't use old auto-update'd files. And since the files are no longer in the place where they'll get wiped automatically during the update, it takes some protection against using old files. Semimythical/trimmed directory listing user@bugs:~/build/elc$ ls -R ~/.elc ~/.elc: error_log.txt chat_log.txt commands.lst counters_ttlanhil.dat spells_ttlanhil.dat local_ignores.txt screenshots/ el.cfg maps/ el.ini notes.xml updates14/ ~/.elc/maps: anitora.elm.txt cont2map1_caves.elm.txt cont2map2.elm.txt map13.elm.txt map7_cave.xm map13.xm ~/.elc/screenshots: elscreen001.png elscreen003.png elscreen012.png elscreen051.png elscreen002.png elscreen004.png elscreen013.png elscreen060.png ~/.elc/updates14: particles/ textures/ ~/.elc/updates14/particles: waterfall.part ~/.elc/updates14/textures: book1.bmp ed: adjusted for the new updatesVER scheme If the version number is applied only to the updates folder, then you still will have 10+ directory trees very quickly for players who wont even knowabout them. Also remember that ANYTHING except the executable could be included in an auto update, even a map. Just so far we've been avoiding doing that. Share this post Link to post Share on other sites
ttlanhil Report post Posted June 18, 2007 If the version number is applied only to the updates folder, then you still will have 10+ directory trees very quickly for players who wont even knowabout them.No you won't. Those numbers only change on a client update (at least that I've seen, and there's also why I avoided the 3rd digit of the version number). So far that seems to be a coupe of times per year.And even if that were true, you're assuming I'll leave old update directories lying around, which I won't (what use are they?). If there were a safe way to wipe a directory when you go to a new client version, I could get away with one updates/ dir, but I'd rather not trust that. Also remember that ANYTHING except the executable could be included in an auto update, even a map. Just so far we've been avoiding doing that.That's fine.All the file handling will check for updates properly, because it all goes through one of a few functions set up for this. This isn't a problem. Apart from the file loaders, none of ELC has any clue where the file comes from; you can load a map from datadir and some of the textures it needs from configdir/updates. Share this post Link to post Share on other sites
Learner Report post Posted June 18, 2007 If the version number is applied only to the updates folder, then you still will have 10+ directory trees very quickly for players who wont even knowabout them.No you won't. Those numbers only change on a client update (at least that I've seen, and there's also why I avoided the 3rd digit of the version number). So far that seems to be a coupe of times per year.And even if that were true, you're assuming I'll leave old update directories lying around, which I won't (what use are they?). If there were a safe way to wipe a directory when you go to a new client version, I could get away with one updates/ dir, but I'd rather not trust that. Also remember that ANYTHING except the executable could be included in an auto update, even a map. Just so far we've been avoiding doing that.That's fine.All the file handling will check for updates properly, because it all goes through one of a few functions set up for this. This isn't a problem. Apart from the file loaders, none of ELC has any clue where the file comes from; you can load a map from datadir and some of the textures it needs from configdir/updates. You're going to run into having to wipe that dir with ANY client update related to all the numbers (and they aren't limited to a single digit). Or the version has to include all the numbers, or you will have problems. Share this post Link to post Share on other sites
ttlanhil Report post Posted June 18, 2007 You're going to run into having to wipe that dir with ANY client update related to all the numbers (and they aren't limited to a single digit).Okay, I changed that part. It's now:snprintf(updatepath, sizeof(updatepath), "updates_%d_%d_%d_%d", VER_MAJOR, VER_MINOR, VER_RELEASE, VER_BUILD); Share this post Link to post Share on other sites
Learner Report post Posted June 18, 2007 You're going to run into having to wipe that dir with ANY client update related to all the numbers (and they aren't limited to a single digit).Okay, I changed that part. It's now:snprintf(updatepath, sizeof(updatepath), "updates_%d_%d_%d_%d", VER_MAJOR, VER_MINOR, VER_RELEASE, VER_BUILD); That still doesn't solve my concern of a player getting 20 directories of autoupdate stuff and not realizing it. Share this post Link to post Share on other sites
ttlanhil Report post Posted June 18, 2007 (edited) No, but what I said before about removing old ones when the client version changes does. ed: Oh, and another thing, any files in /updates that aren't listed in the auto-update file list will be removed. Any files where the one in datadir matches the checksum (hebce datadir is updated) will have the one in /updates deleted. It won't be useful for player-made files, but it won't be a disk hog. Edited June 18, 2007 by ttlanhil Share this post Link to post Share on other sites
kibora Report post Posted June 19, 2007 Would be better if instead of C:\Documents and Settings\username\Application Data\elc\el.ini The path is C:\Documents and Settings\username\Application Data\[name of el folder in program files]\el.ini This way there may be a couple el.inis on the same account. Personally i'd like to have at least 2, one for the main and one for the test server Share this post Link to post Share on other sites
ttlanhil Report post Posted June 19, 2007 You can't assume that the datadir is in Program Files... It's entirely possible that someone will install EL as, say, "K:\". As for multiple el.ini's, the game supports setting options on the command line, so you can always do something like testserv.bat: start elc.exe -sp=2001 --othersetting=value Then simply double-click whichever batch file has the settings you want for that run of the client. If you find that setting options on the command line is a hassle because of the number you need to change, you can also make a bunch of different el.ini files, and copy the one you want to 'el.ini' on startup in a different batch file. Share this post Link to post Share on other sites
ttlanhil Report post Posted June 21, 2007 (edited) I've uploaded a new version to Berlios. The version number stuff is used on configdir/updates, I've added more logging for failing to open files, syscall return checking, etc. All the auto-update support should now work. Still needed: - keys.c needs work. But I intend to work on that separately later. - A recursive rmdir() function. I can come up with one of these for linux without too much work, but it will need work for other platforms as well. On the up side, this isn't needed yet (it will only come into play when the client version changes. And Since it will clear all non-current updates, it can be added 3 updates down the track and all that happens is we use up some extra disk space in the mean time) - Removal of the last few (my_)fopen() calls, and the my_fopen() function itself. This will happen a bit later on as the function is used in a few places still. Please test, where possible. If there are no problems or requests reported in a while, I'll put it in CVS. ed: Actually, I do have something else to adjust already... I don't think there a reason why we can't have an option like -DCONFIGDIR=.elfolder Which would make all config files in (for the *nix example) ~/.elfolder/ So unless someone comes up with a reason not to do this, I'll add it to my next version (which will likely be the one checked in) Edited June 21, 2007 by ttlanhil Share this post Link to post Share on other sites
Grum Report post Posted June 21, 2007 (edited) - A recursive rmdir() function. I can come up with one of these for linux without too much work, but it will need work for other platforms as well. something like int remove_directory (const char* name) { DIR* dir = opendir (name); struct dirent* entry; int res = 1; if (!dir) return 0; // clear the directory while ( (entry = readdir (dir)) ) { char buf[256]; safe_snprintf (buf, sizeof (buf), "%s/%s", name, entry->d_name); if (entry->d_type == DT_DIR) { if (strcmp (entry->d_name, ".") != 0 && strcmp (entry->d_name, "..") != 0) { res &= remove_directory (buf); } } else { if (remove (buf)) res = 0; } } // now remove the directory if (res) if (remove (name)) res = 0; return res; } should be reasonably portable, I guess. EDIT: of course, this is entirly untested and not guaranteed to compile EDIT 2: some fixes. Edited June 21, 2007 by Grum Share this post Link to post Share on other sites
ttlanhil Report post Posted June 21, 2007 EDIT: of course, this is entirly untested and not guaranteed to compile Check out font.c, with the FONTS_FIX stuff, about reading the directory... I'd be quite surprised if this worked on windows like it should on a real OS.Also, unless we have many files in datadir it probably won't be a problem, but you're opening up many directories while you recurse. One of the things I saw when looking into those was `man ftw`. Of course, with your code, one could simply closedir() before you do the recursive call and open/read again later, if it turned out to be a problem. Share this post Link to post Share on other sites
Grum Report post Posted June 21, 2007 Sigh... I read POSIX in the man pages and assumed that MSVC would implement that. Silly me... As for the nr of open directories being a problem: I'd be surprised if we'd create a directory tree where that becomes a problem. But adding a closedir () before the directory remove call would be a good idea Share this post Link to post Share on other sites
ttlanhil Report post Posted June 21, 2007 Sigh... I read POSIX in the man pages and assumed that MSVC would implement that. Silly me...Yup! As for the nr of open directories being a problem: I'd be surprised if we'd create a directory tree where that becomes a problem.Me too. For this I'd merely put a note about it at the start of the function, in case someone else grabs the code for some other job (if we do run out of depth, it'd probably be because people are doing funky things with symlink loops, and deserve the problems). Of course, checking the return value of the syscalls isn't a bad idea.But adding a closedir () before the directory remove call would be a good idea Ooh, yes, I'd say so. I've added the ability to do: FEATURES += CONFIGDIR=\".elc2/\" in my local tree, with appropriate warnings about not misusing it (the short form is, unless you have a good reason to do it, just leave it alone ). As an added advantage, this reduced the number of hardcoded strings as well as removing an #ifdef/#else/#endif set (to be replaced by another in a different place, but at least that makes it more readable). So I think the change in general is a positive one even if no-one uses the feature. Share this post Link to post Share on other sites