Jump to content
Eternal Lands Official Forums
bkc56

Client Performance Improvement

Recommended Posts

Enhanced: This method now supports both Windoze and Linux.

 

Warning: The following is for computer literate folks. If you don't understand some/most/all of the description below, then it's probably not something you can use.

 

Prerequisite: You'll need gzip on your computer. You can get it here: http://www.gzip.org/ (but if you couldn't find it yourself, you probably should quit while you're ahead).

 

Notation: This is posted with Entropy's permission.

 

Disclaimer: If you mess this up and your computer explodes riddling your body with shards of metal, plastic, and bits/bytes/words of data, don't come crying to me.

 

Final Warning: Run you fool. Go, save yourself. Get away while you still can!

 

OK, here we go...

 

A while back the EL Client was modified to be able to read in compressed data files. They of course have to be expanded in memory, but disk I/O is very slow compared to code execution so the trade off is a significant net gain.

 

What having compressed data files means is that (for example) the Whitestone .elm date file that's over 4M in size is only 287K compressed. My EL installation directory went from 280M to 88M after compression. And that all was in data files that the client needs to read in.

 

While I haven't tried to take any actual before/after measurements, I have experienced a noticeable improvement in client start-up and map change times. For me it is well worth the time I took to write the script below and run it. And any time we get a client update or new version, I can simply run the script again to compress everything.

 

For Windoze: skip down to post #6.

 

For Linux: keep reading.

 

So here's what you need to do:

 

1. Copy the code below and paste it into a text file and save it as a shell script. I saved it as zip-el.ksh, but do something appropriate for your setup (remember, you're suppose to know how to do this sort of thing or you shouldn't still be reading).

 

2. Verify that the EL environment variable correctly points to your EL installation. The script does a quick check to make sure the directory exists, but that's it.

 

3. Run it. You should see it print out several hundred lines about compressing various files. When it's done, fire up your client.

 

Finally, here's the script source. Good luck, and if you take any timing measurement please post them so we can get a feeling for what sort of improvement this makes for people.

 

#!/bin/ksh

# This script will walk the Eternal Lands directory structure provided,
# finding all the files of the specified extension types,
# and compress them using gzip.

# Directory where Eternal Lands in installed
EL="/program files/eternal lands"

# File times to compress with gzip
TYPES=".bmp .elm .e3d .2d0 .caf"

if [ ! -d "$EL" ]
then
echo "Error:  directory does not exist: '$EL'"
exit
fi

echo "Processing '$EL'..."
cd "$EL"
for t in $TYPES
do
echo "==========================="
echo "Processing type '$t'..."
echo "==========================="
# find the file, grep for the extension but only at the end of the
# string, xargs to process one at a time, gzip with parameters:
# -9 = max compression, -f = force, -v = verbose
find . -type f -print		|
grep "$t\$"			|
xargs -l1 -I{} gzip -9 -f -v "{}"
done

echo "Compression complete."

Edited by bkc56

Share this post


Link to post
Share on other sites

i made a backup of el and gziped the .bmp and .elm files in the maps dir a few days ago and works good. i made a new backup and gziped the .e3d, .2d0, .caf and .part files in the other dirs. i got the file list from reading the script above. error log is complaining of not being able to load .part files now so maybe those cant be gziped or its just me so maybe someone else can confirm this :rolleyes:

-x~~

Share this post


Link to post
Share on other sites

OK, I've removed .part from the list until we can get confirmation.

 

I hadn't seen any problems before, but checking my error_log tonight shows the same errors.

Share this post


Link to post
Share on other sites

Thank you for the script. I always used to zip all those files by hand. I've been too lazy to do it for 1.8.0 until now.

It's so much easier now with the script. :confused: I replaced ksh with bash in the script, still works flawlessly.

 

Needless to say that the client loads much faster with the compressed files. :brooding:

Share this post


Link to post
Share on other sites
For Windoze users you'll need to have MKS-Toolkit, Cygwin, or something similar installed; or be willing to learn some new things about linux-type tools in order to make use of this.

 

not at all ;)

 

just get the windows binary of gzip (currently http://www.gzip.org/gzip124xN.zip), extract the 'gzip.exe' somewhere to (i use c:\ in my example, you might need to modify that in the script) and use the following batch.

1) Copy the code to your editor

2) adjust EL folder and path to GZIP where indicated.

3) save it as 'zip-el.bat'

4) then open a 'Command Prompt' and goto the directory where the batch is in and start it. (you might even doubleclick it, but then you might miss some output messages)

If you didnt adjust the EL folder, it will quit with an error.

If you didnt adjust the path to your GZIP executable, it will complain for each file.

 

@echo off
REM This script will walk the Eternal Lands directory structure provided,
REM finding all the files of the specified extension types,
REM and compress them using gzip.
REM tested with WinXP (command line extensions need be enabled (default))
REM Version 0.9 (15.1.2009)

REM Directory where Eternal Lands in installed (NO surrounding quotes pls)
set ELDIR=c:\program files\eternal lands

REM File times to compress with gzip
REM TYPES=".bmp .elm .e3d .2d0 .caf" - just for the record

echo "Processing %ELDIR%..."
cd "%ELDIR%"
IF ERRORLEVEL 1 GOTO NOSUCHDIR

REM note that you need to specify the gzip executable here with full path, if its not in PATH

for /R "%ELDIR%" %%i IN (*.bmp,*.elm,*.e3d,*.2d0,*.caf) DO @c:\gzip -9 -f -v "%%i"

REM  -9 = max compression, -f = force, -v = verbose

echo "Compression complete."
GOTO END

:NOSUCHDIR
echo "ELDIR not found. (%ELDIR%)"

:END

 

remarks:

- i hope the posting permission extends to this one

- i didnt test it with Vista and i wont

- if you are not familiar with command lines and pathes under Windows: dont use it

- use it at your own risk, this comes with no warranty at all

- dont flame me, i use linux _and_ windows

- thx to bkc56 for the linux script :)

;)

Share this post


Link to post
Share on other sites

Tested on Ubuntu 8.10.

 

- Definitely faster game loading a felt 50% at least

- A lot faster map loading. Another at least 50%

- and I think I gained a minimum of 10 to 12fps.

 

Thanks for this great little script.

Share this post


Link to post
Share on other sites
not at all ;)
Great, thanks!

 

I realized after thinking about it that creating a .bat file shouldn't be that hard, and I was going to do it today. You saved me the effort.

 

I'll edit the base-note, remove some of the warnings/requirements, and point to your script for the Windoze users.

 

Edit: modification done.

Edited by bkc56

Share this post


Link to post
Share on other sites

Works fine under Vista 64bit.

 

Client size:

Before: 440 MB (with unofficial hi-res maps)

After: 212 MB

 

#Edit:

BTW, since the gz maps works fine, I'll post new mappack, with gzipped files ;)

Edited by BloodSucker

Share this post


Link to post
Share on other sites

Because I would like to do this in a native windows environment i want to point out that 7zip 4.64 (LGPL) has a command line version. I will try to find some code (i have never done this) although it may take some time. When i'm done i will post it here. If you find this an easy exercise then go ahead and post it before it do. ;)

Share this post


Link to post
Share on other sites

the client can only read .gz files not .7z

Share this post


Link to post
Share on other sites

Ive just gzipped everything using your code (from post #6) running 3 clients logged on in white stone gives a 75-85% cpu usage (also have some other things open) but as far as I know that always used to be 100% unless the max framerate was met. So I guess my CPU is no longer the limiting factor.

 

Ill try to try measure some stuff, I think I see an improvement but that may be because I want to see an improvement ;p

 

Note: normally keep clients in console mode, now I don't and my fan kicks in :) Anyway, great work tnx.

 

Edit: with one client open in UW I get ~20-30% cpu usage but hardly use any page file thing

Edited by ProHibited

Share this post


Link to post
Share on other sites

This will NOT speed up the client or use less CPU, it will just, in some cases, load maps and 3d objects faster (but this depends on how much RAM you have an other factors).

With 3GB of ram, I do not notice any significant delay when changing maps (without compressed files).

Share this post


Link to post
Share on other sites

I did the .elm files before already but now its defenatly faster.

I had to take out the BMP files tho, if i compressed them i got a broken screen with all kind of coloured bars, names of players (& monsters) where only a white bar etc. compressing the " .elm .e3d .2d0 .caf " files is a bit slower but atleast doesn't break the client. :)

Share this post


Link to post
Share on other sites
Because I would like to do this in a native windows environment ..

 

err, you might have missed, that i already posted the code for a native windows environment :)

If you download 7zip or gzip is no difference here ;)

if you really prefer 7zip to make your .gz files, just alter the command at the end of the FOR loop.

 

And for improvement:

i have WinXP with 2 GB RAM and mapchanging was definitely faster (using bloodsuckers maps here). It might just be 1 or 2 seconds, but i think its a real improvement :)

 

regards,

Gilrain

Share this post


Link to post
Share on other sites
This will NOT speed up the client or use less CPU, it will just, in some cases, load maps and 3d objects faster (but this depends on how much RAM you have an other factors).

With 3GB of ram, I do not notice any significant delay when changing maps (without compressed files).

True. I have to take back my statement about the improved FPS. It is not the case.

Share this post


Link to post
Share on other sites
Just a side note: the map editor wont work properly any more (after using gzip).
I guess when they enhanced the client to read gzip files, no one did the map editor. Not a big problem though, just means a backup copy of the map directory (uncompressed) for use with the map editor. Plus it's a very few individuals who use the map editor and would need to do that.

Share this post


Link to post
Share on other sites
Just a side note: the map editor wont work properly any more (after using gzip).
I guess when they enhanced the client to read gzip files, no one did the map editor. Not a big problem though, just means a backup copy of the map directory (uncompressed) for use with the map editor. Plus it's a very few individuals who use the map editor and would need to do that.

 

Not only the map directory, but also textures and stuff. As a matter of fact, better to keep separate EL installation with plain files to use with map editor. Otherwise everything is messed up (textures, windows, buttons etc. etc.).

 

But that's important mostly for the map makers. Regular EL players never use map editor.

Share this post


Link to post
Share on other sites

Running Win XP Pro 2002 SP3 on a P4 900Mhz.

My 'net is a blazing 28.8Kbps... wireless... fastest pings I ever see are around 350ms... avg 500...

 

Holy Smoke!!! I tried this patch and E.L. (and my whole system) stopped locking up!!!

 

Thought I better mention this because I've quit playing a few times because of the locking and

system crashes...

I know Ent. made the game and said it won't speed up, the SPEED (fps) is the same, just

dosen't lock my system every 2-3 minutes anymore.

http://www.eternal-lands.com/forum/style_e...ault/lurker.gif

I REALLY REALLY think the files should come 'pre-compressed' in future updates.

 

I also should mention that I noticed the faster load times and map changes, as noted in prior posts.

 

Thanks to bkc56 for the post!!!

Share this post


Link to post
Share on other sites
Running Win XP Pro 2002 SP3 on a P4 900Mhz.

My 'net is a blazing 28.8Kbps... wireless... fastest pings I ever see are around 350ms... avg 500...

 

Holy Smoke!!! I tried this patch and E.L. (and my whole system) stopped locking up!!!

 

Thought I better mention this because I've quit playing a few times because of the locking and

system crashes...

I know Ent. made the game and said it won't speed up, the SPEED (fps) is the same, just

dosen't lock my system every 2-3 minutes anymore.

http://www.eternal-lands.com/forum/style_e...ault/lurker.gif

I REALLY REALLY think the files should come 'pre-compressed' in future updates.

 

I also should mention that I noticed the faster load times and map changes, as noted in prior posts.

 

Thanks to bkc56 for the post!!!

One of the issues is that if the files are compressed before the bundles are made, the downloads are larger. In order to compress after the install, the compression program needs to be in included and run after the install.

 

Also, mixing compressed & uncompressed files (such as downloading via autoupdate) can confuse people as to which file is being used or using old files unintentionally. My default the client will look for compressed files first, and auto update in the past has supplied uncompressed files.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×