bkc56 Report post Posted January 15, 2009 (edited) Enhanced: This method now supports both Windoze and Linux. Warning: The following is for computer literate folks. If you don't understand some/most/all of the description below, then it's probably not something you can use. Prerequisite: You'll need gzip on your computer. You can get it here: http://www.gzip.org/ (but if you couldn't find it yourself, you probably should quit while you're ahead). Notation: This is posted with Entropy's permission. Disclaimer: If you mess this up and your computer explodes riddling your body with shards of metal, plastic, and bits/bytes/words of data, don't come crying to me. Final Warning: Run you fool. Go, save yourself. Get away while you still can! OK, here we go... A while back the EL Client was modified to be able to read in compressed data files. They of course have to be expanded in memory, but disk I/O is very slow compared to code execution so the trade off is a significant net gain. What having compressed data files means is that (for example) the Whitestone .elm date file that's over 4M in size is only 287K compressed. My EL installation directory went from 280M to 88M after compression. And that all was in data files that the client needs to read in. While I haven't tried to take any actual before/after measurements, I have experienced a noticeable improvement in client start-up and map change times. For me it is well worth the time I took to write the script below and run it. And any time we get a client update or new version, I can simply run the script again to compress everything. For Windoze: skip down to post #6. For Linux: keep reading. So here's what you need to do: 1. Copy the code below and paste it into a text file and save it as a shell script. I saved it as zip-el.ksh, but do something appropriate for your setup (remember, you're suppose to know how to do this sort of thing or you shouldn't still be reading). 2. Verify that the EL environment variable correctly points to your EL installation. The script does a quick check to make sure the directory exists, but that's it. 3. Run it. You should see it print out several hundred lines about compressing various files. When it's done, fire up your client. Finally, here's the script source. Good luck, and if you take any timing measurement please post them so we can get a feeling for what sort of improvement this makes for people. #!/bin/ksh # This script will walk the Eternal Lands directory structure provided, # finding all the files of the specified extension types, # and compress them using gzip. # Directory where Eternal Lands in installed EL="/program files/eternal lands" # File times to compress with gzip TYPES=".bmp .elm .e3d .2d0 .caf" if [ ! -d "$EL" ] then echo "Error: directory does not exist: '$EL'" exit fi echo "Processing '$EL'..." cd "$EL" for t in $TYPES do echo "===========================" echo "Processing type '$t'..." echo "===========================" # find the file, grep for the extension but only at the end of the # string, xargs to process one at a time, gzip with parameters: # -9 = max compression, -f = force, -v = verbose find . -type f -print | grep "$t\$" | xargs -l1 -I{} gzip -9 -f -v "{}" done echo "Compression complete." Edited January 15, 2009 by bkc56 Share this post Link to post Share on other sites
baddude Report post Posted January 15, 2009 i made a backup of el and gziped the .bmp and .elm files in the maps dir a few days ago and works good. i made a new backup and gziped the .e3d, .2d0, .caf and .part files in the other dirs. i got the file list from reading the script above. error log is complaining of not being able to load .part files now so maybe those cant be gziped or its just me so maybe someone else can confirm this -x~~ Share this post Link to post Share on other sites
bkc56 Report post Posted January 15, 2009 OK, I've removed .part from the list until we can get confirmation. I hadn't seen any problems before, but checking my error_log tonight shows the same errors. Share this post Link to post Share on other sites
Gampa Report post Posted January 15, 2009 Thank you for the script. I always used to zip all those files by hand. I've been too lazy to do it for 1.8.0 until now. It's so much easier now with the script. I replaced ksh with bash in the script, still works flawlessly. Needless to say that the client loads much faster with the compressed files. Share this post Link to post Share on other sites
baddude Report post Posted January 15, 2009 tyvm bkc56 for the script, very fast and easy Share this post Link to post Share on other sites
Gilrain Report post Posted January 15, 2009 For Windoze users you'll need to have MKS-Toolkit, Cygwin, or something similar installed; or be willing to learn some new things about linux-type tools in order to make use of this. not at all just get the windows binary of gzip (currently http://www.gzip.org/gzip124xN.zip), extract the 'gzip.exe' somewhere to (i use c:\ in my example, you might need to modify that in the script) and use the following batch. 1) Copy the code to your editor 2) adjust EL folder and path to GZIP where indicated. 3) save it as 'zip-el.bat' 4) then open a 'Command Prompt' and goto the directory where the batch is in and start it. (you might even doubleclick it, but then you might miss some output messages) If you didnt adjust the EL folder, it will quit with an error. If you didnt adjust the path to your GZIP executable, it will complain for each file. @echo off REM This script will walk the Eternal Lands directory structure provided, REM finding all the files of the specified extension types, REM and compress them using gzip. REM tested with WinXP (command line extensions need be enabled (default)) REM Version 0.9 (15.1.2009) REM Directory where Eternal Lands in installed (NO surrounding quotes pls) set ELDIR=c:\program files\eternal lands REM File times to compress with gzip REM TYPES=".bmp .elm .e3d .2d0 .caf" - just for the record echo "Processing %ELDIR%..." cd "%ELDIR%" IF ERRORLEVEL 1 GOTO NOSUCHDIR REM note that you need to specify the gzip executable here with full path, if its not in PATH for /R "%ELDIR%" %%i IN (*.bmp,*.elm,*.e3d,*.2d0,*.caf) DO @c:\gzip -9 -f -v "%%i" REM -9 = max compression, -f = force, -v = verbose echo "Compression complete." GOTO END :NOSUCHDIR echo "ELDIR not found. (%ELDIR%)" :END remarks: - i hope the posting permission extends to this one - i didnt test it with Vista and i wont - if you are not familiar with command lines and pathes under Windows: dont use it - use it at your own risk, this comes with no warranty at all - dont flame me, i use linux _and_ windows - thx to bkc56 for the linux script Share this post Link to post Share on other sites
SleepyJo Report post Posted January 15, 2009 Tested on Ubuntu 8.10. - Definitely faster game loading a felt 50% at least - A lot faster map loading. Another at least 50% - and I think I gained a minimum of 10 to 12fps. Thanks for this great little script. Share this post Link to post Share on other sites
bkc56 Report post Posted January 15, 2009 (edited) not at all Great, thanks! I realized after thinking about it that creating a .bat file shouldn't be that hard, and I was going to do it today. You saved me the effort. I'll edit the base-note, remove some of the warnings/requirements, and point to your script for the Windoze users. Edit: modification done. Edited January 15, 2009 by bkc56 Share this post Link to post Share on other sites
BloodSucker Report post Posted January 15, 2009 (edited) Works fine under Vista 64bit. Client size: Before: 440 MB (with unofficial hi-res maps) After: 212 MB #Edit: BTW, since the gz maps works fine, I'll post new mappack, with gzipped files Edited January 15, 2009 by BloodSucker Share this post Link to post Share on other sites
SolarStar Report post Posted January 15, 2009 Because I would like to do this in a native windows environment i want to point out that 7zip 4.64 (LGPL) has a command line version. I will try to find some code (i have never done this) although it may take some time. When i'm done i will post it here. If you find this an easy exercise then go ahead and post it before it do. Share this post Link to post Share on other sites
LabRat Report post Posted January 15, 2009 the client can only read .gz files not .7z Share this post Link to post Share on other sites
SolarStar Report post Posted January 15, 2009 7zip supports many more formats, including gzip. Share this post Link to post Share on other sites
ProHibited Report post Posted January 15, 2009 (edited) Ive just gzipped everything using your code (from post #6) running 3 clients logged on in white stone gives a 75-85% cpu usage (also have some other things open) but as far as I know that always used to be 100% unless the max framerate was met. So I guess my CPU is no longer the limiting factor. Ill try to try measure some stuff, I think I see an improvement but that may be because I want to see an improvement ;p Note: normally keep clients in console mode, now I don't and my fan kicks in Anyway, great work tnx. Edit: with one client open in UW I get ~20-30% cpu usage but hardly use any page file thing Edited January 16, 2009 by ProHibited Share this post Link to post Share on other sites
Entropy Report post Posted January 16, 2009 This will NOT speed up the client or use less CPU, it will just, in some cases, load maps and 3d objects faster (but this depends on how much RAM you have an other factors). With 3GB of ram, I do not notice any significant delay when changing maps (without compressed files). Share this post Link to post Share on other sites
Link Report post Posted January 16, 2009 - i didnt test it with Vista and i wont Tested with Vista Home Premium 64-Bit, works fine. Thanks for the script Share this post Link to post Share on other sites
Walt-Her Report post Posted January 16, 2009 I did the .elm files before already but now its defenatly faster. I had to take out the BMP files tho, if i compressed them i got a broken screen with all kind of coloured bars, names of players (& monsters) where only a white bar etc. compressing the " .elm .e3d .2d0 .caf " files is a bit slower but atleast doesn't break the client. Share this post Link to post Share on other sites
Gilrain Report post Posted January 16, 2009 Because I would like to do this in a native windows environment .. err, you might have missed, that i already posted the code for a native windows environment If you download 7zip or gzip is no difference here if you really prefer 7zip to make your .gz files, just alter the command at the end of the FOR loop. And for improvement: i have WinXP with 2 GB RAM and mapchanging was definitely faster (using bloodsuckers maps here). It might just be 1 or 2 seconds, but i think its a real improvement regards, Gilrain Share this post Link to post Share on other sites
BloodSucker Report post Posted January 16, 2009 Just a side note: the map editor wont work properly any more (after using gzip). Share this post Link to post Share on other sites
SleepyJo Report post Posted January 16, 2009 This will NOT speed up the client or use less CPU, it will just, in some cases, load maps and 3d objects faster (but this depends on how much RAM you have an other factors).With 3GB of ram, I do not notice any significant delay when changing maps (without compressed files). True. I have to take back my statement about the improved FPS. It is not the case. Share this post Link to post Share on other sites
bkc56 Report post Posted January 16, 2009 Just a side note: the map editor wont work properly any more (after using gzip).I guess when they enhanced the client to read gzip files, no one did the map editor. Not a big problem though, just means a backup copy of the map directory (uncompressed) for use with the map editor. Plus it's a very few individuals who use the map editor and would need to do that. Share this post Link to post Share on other sites
BloodSucker Report post Posted January 16, 2009 Just a side note: the map editor wont work properly any more (after using gzip).I guess when they enhanced the client to read gzip files, no one did the map editor. Not a big problem though, just means a backup copy of the map directory (uncompressed) for use with the map editor. Plus it's a very few individuals who use the map editor and would need to do that. Not only the map directory, but also textures and stuff. As a matter of fact, better to keep separate EL installation with plain files to use with map editor. Otherwise everything is messed up (textures, windows, buttons etc. etc.). But that's important mostly for the map makers. Regular EL players never use map editor. Share this post Link to post Share on other sites
blar32 Report post Posted January 18, 2009 Just a minor issue the icon.bmp doesn't show the EL icon when running the client if zipped. Once I unzipped it came back. --blar332 Share this post Link to post Share on other sites
Shujral Report post Posted January 21, 2009 worked fine for me, thanks. Share this post Link to post Share on other sites
Canabtzedec Report post Posted October 2, 2009 Running Win XP Pro 2002 SP3 on a P4 900Mhz. My 'net is a blazing 28.8Kbps... wireless... fastest pings I ever see are around 350ms... avg 500... Holy Smoke!!! I tried this patch and E.L. (and my whole system) stopped locking up!!! Thought I better mention this because I've quit playing a few times because of the locking and system crashes... I know Ent. made the game and said it won't speed up, the SPEED (fps) is the same, just dosen't lock my system every 2-3 minutes anymore. http://www.eternal-lands.com/forum/style_e...ault/lurker.gif I REALLY REALLY think the files should come 'pre-compressed' in future updates. I also should mention that I noticed the faster load times and map changes, as noted in prior posts. Thanks to bkc56 for the post!!! Share this post Link to post Share on other sites
Learner Report post Posted October 2, 2009 Running Win XP Pro 2002 SP3 on a P4 900Mhz.My 'net is a blazing 28.8Kbps... wireless... fastest pings I ever see are around 350ms... avg 500... Holy Smoke!!! I tried this patch and E.L. (and my whole system) stopped locking up!!! Thought I better mention this because I've quit playing a few times because of the locking and system crashes... I know Ent. made the game and said it won't speed up, the SPEED (fps) is the same, just dosen't lock my system every 2-3 minutes anymore. http://www.eternal-lands.com/forum/style_e...ault/lurker.gif I REALLY REALLY think the files should come 'pre-compressed' in future updates. I also should mention that I noticed the faster load times and map changes, as noted in prior posts. Thanks to bkc56 for the post!!! One of the issues is that if the files are compressed before the bundles are made, the downloads are larger. In order to compress after the install, the compression program needs to be in included and run after the install. Also, mixing compressed & uncompressed files (such as downloading via autoupdate) can confuse people as to which file is being used or using old files unintentionally. My default the client will look for compressed files first, and auto update in the past has supplied uncompressed files. Share this post Link to post Share on other sites