#md5check command

[bluap]

After a discussion about map .elm files yesterday, I thought it might be useful to build in a #command for doing md5 sums on client files. This would allow easy checks to be done when people are trying to help others with problems. I'll explain later how this might be extended so people can easily help themselves.

 

As it happens, the md5 computation code is already built into the client so the new command was relatively easy, I have a patch coded up and would be more than happy to commit it to the client. Here's what the new command does:

 

Entering "#ckdata" displays the md5sum of the current map .elm file:

client: 360bdb294217a98642fc1e19e5b1d0c7 ./maps/map5nf.elm

 

Entering "#ckdata textures/compass.bmp" displays the md5sum of the specified file:

client: ecca300936d5a2efe067da2828b420b1 textures/compass.bmp

 

So what about that extension. I was thinking that if this new command string was allowed to pass though to the server. The server could return a message displaying the expected version of the md5sum for the specified file or current map. Obviously, it would be easy to have this in a lookup table to make it faster. I would have though that this method gives an easy way to cross check files rather than some more complex client/server exchange. In any case, IMHO the command is already useful without the server part.

 

Any thoughts?

 

Edit: The command name was changed to "#chdata"

Edited December 21, 2008 by bluap

[Usl]

Very good idea (from a helpdesk / @1 / @2 point of view).

 

If there is a desire not to make any change to the server, it is also possible to implement this entirely client-side (it could easily go into your patch).

It would be enough to provide a file "checksums.txt", with the checksums of all the right .elm for the current version, and distribute it to clients with the Automatic Update feature. Then, the client could read this file on startup, and warn the user about modified (illegal) .elm files immediately, or provide a command to check them explicitly. When a modified .elm is distributed through Auto update (if ever), the accompayning checksums.txt can be distributed as well, so we are sure the two of them will be in sync. Adding the checksum.txt should require no change to the code and no recompilation, just a change to a configuration file. Moreover, the versioning system (1_7_0/...) we already have in place would work seamlessly with the checksums.txt system (we only need to put the checksums.txt file for a set of maps in the same directory as the maps themselves).

 

If a user disables Auto update, we can always instruct them to turn it on and restart in case they report map-related problems.

 

Of course, a malicious user could alter both the map and the checksum.txt, or the code in the client to check it, but our purpose is just to simplify the helpdesk part, so even a client-only solution seems sufficient.

[Learner]

Very good idea (from a helpdesk / @1 / @2 point of view).

 

If there is a desire not to make any change to the server, it is also possible to implement this entirely client-side (it could easily go into your patch).

It would be enough to provide a file "checksums.txt", with the checksums of all the right .elm for the current version, and distribute it to clients with the Automatic Update feature. Then, the client could read this file on startup, and warn the user about modified (illegal) .elm files immediately, or provide a command to check them explicitly. When a modified .elm is distributed through Auto update (if ever), the accompayning checksums.txt can be distributed as well, so we are sure the two of them will be in sync. Adding the checksum.txt should require no change to the code and no recompilation, just a change to a configuration file. Moreover, the versioning system (1_7_0/...) we already have in place would work seamlessly with the checksums.txt system (we only need to put the checksums.txt file for a set of maps in the same directory as the maps themselves).

 

If a user disables Auto update, we can always instruct them to turn it on and restart in case they report map-related problems.

 

Of course, a malicious user could alter both the map and the checksum.txt, or the code in the client to check it, but our purpose is just to simplify the helpdesk part, so even a client-only solution seems sufficient.

Keep in mind that *nix systems already have command line md5 tools in place already ... which is how the server builds the update list to begin with. I'm concerned with lag of an automated checksum tool will add to low end systems if it's in the client (even if it's in a seperate thread). Having a seperate external tool for that would probably be better.

[bluap]

If there is a desire not to make any change to the server, it is also possible to implement this entirely client-side (it could easily go into your patch)....

All that could be done. The only thing I don't like is checking all the .elm files at start-up; that could take quite some time. I was thinking of a simple use where someone reports say a map bug, they are asked to post/pm the result of this command, we can easily tell if it's the incorrect map file. With the server part, people can check themselves and save others involvement. The definitive md5sums could even be posted on a web page rather than returned by the server.

 

Thanks for the feedback though.

[Roja]

This would allow easy checks to be done when people are trying to help others with problems. I'll explain later how this might be extended so people can easily help themselves.

 

Maybe first explain to us what "#md5check" means, if this is to be used to help players with problems

[bluap]

Maybe first explain to us what "#md5check" means, if this is to be used to help players with problems

Sorry

You could have a look here. However, an md5sum just a way of getting a reasonably unique number from an arbitrary sized file. Any difference, no matter how small produces a different number. False matches for different file contents are very unlikely. md5sums are frequently used to check a downloaded file is not different from the original. When you download the file, the website gives the md5sum number on the download page. When you have finished downloading the file, you run a program such as md5sum on the downloaded file and compare the numbers. If they are the same, you can reasonably expect the download was good.

[The_Piper]

MD5 is an algorithm to calculate checksums, any other would be fine too.

 

http://en.wikipedia.org/wiki/MD5

 

Asking the server for a checksum of a given map and then compare it with the checksum of the local/installed map is a good idea IMO. But dont do it at client startup time, just do it once when you enter a map which you havent checked already.

 

The benefit of checking *.elm files via checksums is simple:

 

Every n00b who copies custom maps PLUS old *.elm files (aka the whole maps folder) will get an error message that the *.elm files are not up to date and s/he should reinstall the game.

 

Same with those tricky guys who install modified *.elm files, for example, VOTD without trees and such. That wont work then anymore too.

 

Pro: You won't be able to use different *.elm files than the server does. No more *false* bug reports because someone uses outdated/wrong *.elm files, and it stops cheaters too.

 

Contra: If the server uses new *.elm files, you must make sure, that the client gets those files too. But this can be solved using the auto update feature.

 

Piper

[Usl]

If there is a desire not to make any change to the server, it is also possible to implement this entirely client-side (it could easily go into your patch)....

All that could be done. The only thing I don't like is checking all the .elm files at start-up; that could take quite some time. ...

 

You are right, checking them at startup could be overkill on slow systems, but the same system can be used to do the checking locally with the # command.

 

Just as a reference, these are the times to check the MD5 for all the maps (including guild maps) on my laptop, 1.73GHz Intel Pentium M processor.

real	0m1.206s
user	0m0.572s
sys	 0m0.132s

On low-end systems, it could take 3-5 secs, which is indeed very significant at that point.

[bluap]

Keep in mind that *nix systems already have command line md5 tools in place already ... which is how the server builds the update list to begin with. I'm concerned with lag of an automated checksum tool will add to low end systems if it's in the client (even if it's in a seperate thread). Having a seperate external tool for that would probably be better.

A separate tool would be OK but you can never tell if its run on the correct files. Building the command into the client ensures the checked files are the ones being used. The proposed command only checks a single file. On my vastly underpowered laptop which can barely play EL, the md5sum for the WS map is computed in much less than a second.

[bluap]

Asking the server for a checksum of a given map and then compare it with the checksum of the local/installed map is a good idea IMO. But dont do it at client startup time, just do it once when you enter a map which you havent checked already.

Yes, that would I think, be a good way of doing the check. However, a server change would have to complete with Radu's other priorities. Having variable speed features for example would be higher on my list.

 

What I have done so far is all client side and does, I think, give some benefit to solving map and other file problems.

 

Thanks again for the feedback, this has generate more discussion than I expected.

[Aislinn]

Oh I like this a lot!

 

Edit: Would it be possible to get some sort of command that would compare two md5sums?

[Entropy]

I think the MD5 is overkill, a CRC32 should do it I think (and it is less data to send to the server, in the event we'll want to do it server side as well). But for now, I think it should only be a client implementation.

[Roja]

Pro: You won't be able to use different *.elm files than the server does. No more *false* bug reports because someone uses outdated/wrong *.elm files, and it stops cheaters too.

 

Contra: If the server uses new *.elm files, you must make sure, that the client gets those files too. But this can be solved using the auto update feature.

 

Although this is a very good thing, it really would suck for map developers...and anyone who wants to use the test server. I upload maps to the test server at any given time-to test them.

[Torg]

I think the MD5 is overkill, a CRC32 should do it I think (and it is less data to send to the server, in the event we'll want to do it server side as well). But for now, I think it should only be a client implementation.

Given that MD5 is already implemented in the client (and has been for a while - for updates iirc), I'd use that over CRC32 (which I don't think is implemented in the client yet) until it is needed for cross-checking with the server.

[belton]

Just a small note: If you allow arbitrary filenames, make sure you 'sanitize' them and not allow '..' or anything similar in the filename, to not allow going 'up' from the EL folder (people can/will be asking people to type these commands)

[Entropy]

Just a small note: If you allow arbitrary filenames, make sure you 'sanitize' them and not allow '..' or anything similar in the filename, to not allow going 'up' from the EL folder (people can/will be asking people to type these commands)

 

Yeah, so you would get the md5 of some secret file, have fun with that.

[belton]

Just a small note: If you allow arbitrary filenames, make sure you 'sanitize' them and not allow '..' or anything similar in the filename, to not allow going 'up' from the EL folder (people can/will be asking people to type these commands)

 

Yeah, so you would get the md5 of some secret file, have fun with that.

or of /dev/urandom

/dev/zero

C:\pagefile.swp

and yeah, you can probably glean information from an MD5sum of *certain* files

[bluap]

There seemed to be general approval for this command or something like it so I have committed the change; it is a very small addition anyway. I have renamed the command to "#ckdata" so we can change to using a simpler CRC if required without changing the command name. Without parameters, the check defaults to the current map .elm file. Any specified parameter is used a the name of the file to check. As the standard io functions are used, only files within the EL datadir paths are valid. Other files or non existing files return an error message. e.g. "#ckdata /dev/zero" returns "ckdata: invalid file or filename.". Paths containing "../../" are also safe.

[Entropy]

and yeah, you can probably glean information from an MD5sum of *certain* files

 

Such as?

[Alia]

and yeah, you can probably glean information from an MD5sum of *certain* files

 

Such as?

 

For example:

Lots of files in /proc have quite standard content, so you can easily look through all possible

variants, compute it's md5 and compare with md5 you coaxed from unsuspecting player,

and learn this way something about his computer.

Edited October 13, 2008 by Alia

[Entropy]

Or you can coax the player to just do a cat /proc and e-mail you the output

I don't think that this command can be used to get any meaningfull information about someone's computer.

[bluap]

Or you can coax the player to just do a cat /proc and e-mail you the output

I don't think that this command can be used to get any meaningfull information about someone's computer.

And as I said, only files in the datadir path can be accessed anyway.

[bluap]

Would it be possible to get some sort of command that would compare two md5sums?

Rather late in the day I realise there would be a really easy way to do this. The current command "#ckdata" either checks the current map if no parameters are specified, or the specified file if a parameter is specified. What I could do is accept an optional second parameter, the expected checksum value. If this is specified, the command would simply give a OK/Fail type message.

 

It is probably a bit late for the current release but I would be happy to code this up to be included if Entropy gave me the go ahead. The code change would be limited to the new #chdata function, it would be a simple change.....

[Entropy]

Ok, so long as only the code for this function is changed.

[bluap]

Ok, so long as only the code for this function is changed.

Thanks. I've committed the change, only the #ckdata function was modified.

 

"#ckdata" will show the sum for the current map elm file.

"#ckdata file_name" will show the sum for the specified file.

"#ckdata expected_checksum file_name" will compare the specified sum with the sum for the specified file. If the sums match, you get a match message, otherwise you get a no match message.

 

The output from the show sum versions can be directly copy/paste to use in the compare version.

If the file does not exist or the specified sum is not the correct length, you get an error message which includes the above help.