dns_server Report post Posted November 11, 2005 (edited) i created this little script to parse my log files and retrieve some simple statistics from them. the log files are in a text file with the format: day (as a long) space message ie 1128599852564: [PM from person: message] #!/bin/bash echo parseing logs rm test cat *.txt|grep "PM from"|awk '{print $4 " "$5 " " $6}'|sed -e 's/\]//g' -e 's/\://g'|tr 'a-z' 'A-Z'>test echo createing users list, to the file 'users' rm users cat test|awk '{print $1}'| sort | uniq -c | sort -nr>users echo createing a command list to the file 'command' rm command cat test|awk '{print $2}'| sort | uniq -c | sort -nr>command echo createing file 'stats' to store the player with the most requested stats rm stats cat test |awk '{print $2 " " $3}'|grep "STATS"|awk '{print $2}'|sort | uniq -c | sort -nr>stats echo createing file 'skills' to store the player with the most requested skills rm skills cat test |awk '{print $2 " " $3}'|grep "SKILLS"|awk '{print $2}'|sort | uniq -c | sort -nr>skills here's the top 10 from each file with the links to the full files. There where 75416 pm's to iknow over several months. ==> users <== 3631 DEADFACE 1707 ASGNNY 1168 GRYFLET 1080 RAEN 958 YASERHAMEED 910 ECLYPSE 842 MASKA007PL 751 DAMAGEDTMX 740 BERLYSAM 685 JORMANE ==> command <== 53889 STATS 6123 SKILLS 2122 PONG 1362 SKILL 1116 PING 1085 HELP 942 ON 898 STAT 728 SEEN 627 JOKER ==> stats <== 522 CRUS 374 SHIVAR 303 DONPEDRO 294 BELALI 283 ALCHMASTER 279 MATESS 277 ZIARA 275 WEXY 271 BENHE 270 TOOMASS ==> skills <== 46 DONPEDRO 45 KIBORA 34 ENKI 34 MASTERPITER 33 FELO 31 BEP 27 SKILLS 26 ANYONE 24 FRASSE 23 MASA Edited November 11, 2005 by dns_server Share this post Link to post Share on other sites
Placid Report post Posted November 11, 2005 Hahahaeh, nice one DNS. Actually quite interesting to see :-) I especially enjoyed reading the stats bit. Share this post Link to post Share on other sites
crus Report post Posted November 11, 2005 OMG lol.. ==> stats <== 522 CRUS what does this means?.. That ppl wanted to see my stats 522 times?.. *faints* Share this post Link to post Share on other sites
LadyWolf Report post Posted November 11, 2005 OMG lol.. ==> stats <== 522 CRUS what does this means?.. That ppl wanted to see my stats 522 times?.. *faints* You feel special now dont you crus W00t! i made the list Share this post Link to post Share on other sites
crus Report post Posted November 11, 2005 I kind of feel wanted .. not so special )).. Share this post Link to post Share on other sites
WoodeH Report post Posted November 11, 2005 Hehe. That's clever, nice dns Share this post Link to post Share on other sites
Algorn Report post Posted November 14, 2005 if anyone knows how to convert the Unix long time to a human readable date that can be passed thru grep --count, please chime in. i have been wracking my brain over how to do it(i am not a scripting guru), and i would like to be able to parse the stats by month, since the above stats were totals for all logged months from iknow. if you have any questions about what i'm wanting to do, please feel free to drop me a PM here at the forum(i won't be ingame until at least around xmas due to some hardware failure in my linux box ) <sidenote> some of you may have noticed some delay in iknow's response around 8am GMT (iknow's server is set to GMT time) to aprox 9am GMT. this is due to the server's backup script running at that time and it seems to occupy quite a bit of the processor time. over the next few days or so i will try to tweak the backup script so it doesn't consume so much of the processor's resources<sidenote> Share this post Link to post Share on other sites
trollson Report post Posted November 14, 2005 (edited) if anyone knows how to convert the Unix long time to a human readable date that can be passed thru grep --count, please chime in. i have been wracking my brain over how to do it(i am not a scripting guru), and i would like to be able to parse the stats by month, since the above stats were totals for all logged months from iknow. On GNU/Linux, something like: date '+%s' | awk '{ print strftime ("%F-$T",$1) }' "date '+%s'" is just for demostration, and returns the current time as seconds since the epoch (GNU extension to 'date', you'll need to check whether this is the same value you have). The first argument of 'strftime' is a format string, as per the C API strftime (man 3 strftime). You can change this to something to suit your requirements. For example, "%m" should just print out the month number (01-12). Edited November 14, 2005 by trollson Share this post Link to post Share on other sites
Algorn Report post Posted November 15, 2005 On GNU/Linux, something like: date '+%s' | awk '{ print strftime ("%F-$T",$1) }' "date '+%s'" is just for demostration, and returns the current time as seconds since the epoch (GNU extension to 'date', you'll need to check whether this is the same value you have). The first argument of 'strftime' is a format string, as per the C API strftime (man 3 strftime). You can change this to something to suit your requirements. For example, "%m" should just print out the month number (01-12). okay, i don't currently have a linux box running to test this with, however iknow runs on a NetBSD 2.0 server. now the quandry i am having is when i perform the above command as you have it, it returns the current and correct date(with th exception of attaching "-$T" at the end of the date string. now, if i pass one of the long times from one of the logs thru | awk '{ print strftime ("%F-$T",$1) }', all dates return as 1901-12-13-$T (the $T being consistent with previous behavior mentioned and therefore is easily removable once i figure out how to get the long times converted correctly). example: bash-3.00# echo 1132031808843 | awk '{ print strftime ("%F-$T",$1) }' 1901-12-13-$T this behavior is also persistent if i cat $log_name | awk '{ print strftime ("%F-$T",$1) }' whereby all dates from within the log are returned as the previously mentioned 1901 date. any suggestions on why this would be? Share this post Link to post Share on other sites
trollson Report post Posted November 15, 2005 Your time stamp is in milliseconds since the epoch. The 'strftime' function expects it to be in seconds since the epoch. Try: echo 1132031808843 | awk '{ print strftime ("%F-%T",$1/1000) }' Typo: The '$T' my previous post should have been a '%T'. If you just want to get the month out, and dont care about the rest of the timestamp, change the format to "%b", or "%B", or "%m". Format strings are as given the man page for 'strftime' (C API, "man 3 strftime" or similar). Share this post Link to post Share on other sites
tuXico Report post Posted November 15, 2005 (edited) Hi, since I enjoy shell scripts, I dont want you to stop to play with date but i think it would be better to rotate the log (I dont know if logrotate is available on others linux than debian by default...) so you ll get tarballs of previous month (checkable with zgrep, zcat...) and your script would have less info to parse (only those of the current (or last if you addapt your script with 'z' commands) month. Thanks for iknow btw =) CitizenKane Edited November 15, 2005 by tuXico Share this post Link to post Share on other sites
Algorn Report post Posted November 16, 2005 (edited) Hi, since I enjoy shell scripts, I dont want you to stop to play with date but i think it would be better to rotate the log (I dont know if logrotate is available on others linux than debian by default...) so you ll get tarballs of previous month (checkable with zgrep, zcat...) and your script would have less info to parse (only those of the current (or last if you addapt your script with 'z' commands) month. Thanks for iknow btw =) CitizenKane iknow's logs are rotated as part of his inherent functions (ie, dns coded him to rotate his own logs). we're only interested in extracting usage stats from those logs and trying to break it down by monthly usage. trollson, thanx for the help. i believe i can work with the advice you've given me, i just need to find an efficient way to parse the logs and get the monthly counts. <update> and with the wonderful assisstance provided to me from trollson, i present to you some usage stats for iknow as of 2005-11-16 06:20am GMT (adjust to your timezone): Number of Occurrences of 'PM from' 83868 Number of Occurrences of 'PM from deadface' 7266 Total Number of PMs per Month(includes PMs from deadface) 43410 oct 22575 sep 13264 nov Total Number of PMs from deadface per month 3582 sep 49 oct trollson, again thank you very much for your assisstance. this geek couldn't have done this without your help. Edited November 16, 2005 by Algorn Share this post Link to post Share on other sites