Page 1 of 1 1
Topic Options
#58821 - 2001-09-05 01:14 PM Log Parsing and redirection to CSV
Anonymous
Unregistered


Well, I'm stumped. I'm trying find a way in KIX to parse through multiple log files '@wksta.@domain.log' (which were generated via a KIX inventory script), and aggregate the reults to CSV format for import to Excel...

Here is an example of one of the Log files:

--------------------------------------------

Opened 'QRTP50001.QRTP.log' at 05:58:42

--------------------------------------------
System Information
--------------------------------------------
DOMAIN = QRTP
LOGON_SERVER = \\QRTPNTPDC1
WORKSTATION = QRTP50001
USERID = user
FULLNAME = Test User
PRIVILIDGE = USER
IPADDRESS = 10. 1. 37. 52
MAC_ADDRESS = 00B0D056B084
LOCATION = CHELSEA
OS_VERSION = W2K
CPU_SPEED(MHz) = 498
FREE_C:\(Bytes) = 1401796
KIX_VERSION = 3.62
KIX_Directory = \\qrtpntbdc1\netlogon\logon
AMO_INSTALLED = Yes
SDO_INSTALLED = Yes
LAST_LOGIN = 2001/09/05 05:58:43

--------------------------------------------
Software Information
--------------------------------------------
Software Entry 1 Adobe Acrobat 4.0
Software Entry 2 AOL Instant Messenger (SM)
Software Entry 3 Branding
Software Entry 4 Unicenter TNG Software Delivery Option
Software Entry 5 Citrix ICA Client
Software Entry 6 Compaq Insight Manager XE Client Files
Software Entry 7 Internet Explorer Exception pack
Software Entry 8 Microsoft Internet Explorer 6
Software Entry 9 Internet Explorer ReadMe
Software Entry 10 J-Bots 2000m
Software Entry 11 LiveUpdate
Software Entry 12 Lotus Notes
Software Entry 13 MSN Gaming Zone
Software Entry 14 Microsoft PhotoDraw 2000
Software Entry 15 Microsoft Windows Critical Update Notification
Software Entry 16 Motocross Madness 1.0
Software Entry 17 Norton AntiVirus 5.0 for Windows NT
Software Entry 18 OutlookExpress
Software Entry 19 Proxy Remote Control Master
Software Entry 20 Windows 2000 Hotfix (Pre-SP3) [See Q293826 for more information]
Software Entry 21 Internet Explorer Update Q295106
Software Entry 22 Windows 2000 Hotfix (Pre-SP3) [See Q298012 for more information]
Software Entry 23 Internet Explorer Update Q299618
Software Entry 24 Windows 2000 Hotfix (Pre-SP3) [See Q300972 for more information]
Software Entry 25 Windows 2000 Hotfix (Pre-SP3) [See Q301625 for more information]
Software Entry 26 Remedy User 4.0
Software Entry 27 Remote Control Option
Software Entry 28 Shockwave Flash
Software Entry 29 SPX
Software Entry 30 Visual Kixtart Editor
Software Entry 31 WildTangent Updater
Software Entry 32 Windows 2000 Service Pack 2
Software Entry 33 WinRAR archiver
Software Entry 34 WildTangent Web Driver
Software Entry 35 Microsoft Office 2000 SR-1 Premium
Software Entry 36 Reflection X 8.0.1
Software Entry 37 Microsoft Windows 2000 Professional Resource Kit
Software Entry 38 WebFldrs
Software Entry 39 Windows 2000 Application Compatibility Update
Software Entry 40 DameWare NT Utilities
Software Entry 41 Macromedia Extension Manager
Software Entry 42 Macromedia Dreamweaver UltraDev 4
Software Entry 43 Windows 2000 Administration Tools
Software Entry 44 Extranet Access Client
------------------------------------------

There are approx. 1500 of these that I would like to parse and concatonate into a single CSV file.

Any help would be appreciated, I have a temporary solution in Unix Kornshell using Awk, but would like to keep the solution in pure KIX if possible.

Thanks for any pointers...

Chuck,

[ 05 September 2001: Message edited by: chas31av ]

[ 05 September 2001: Message edited by: chas31av ]

Top
#58822 - 2001-09-05 02:51 PM Re: Log Parsing and redirection to CSV
mvdw Offline
Starting to like KiXtart

Registered: 2001-05-01
Posts: 124
Loc: Voorburg, Netherlands
Hi Chuck,

as i read it, you want to read 1500 files (approx) with that exact same format and then for each file read write one new line with ; delimited.

if so, start with making a loop that opens up all of these files one by one (this all depends on where they are located an how the naming scheme is...)

before you enter the loop, make kix open (if necessary create) a file called output.csv, use something like :
if open (5, "$outputfile", 4) = 0

the 4 tells kix to create the file if it doesn't exist.

the in the loop, open a new loop reading all the lines and extracting some variable like :

While @error = 0
$line = Readline(5)
if $line <> "" (this checks for empty lines)
do something

endif
loop

then at the do something you should think of some way to extract the data you need into a variable, using string manipulations.

check the help file for the different functions you can use. (split, len, ltrim, rtrim etc etc)

If there is a variable length number of lines to be read you could also perform a loop with readlines first to determine the numer of lines and then use that number to define an array to catch them in.

as son as you get all the variables filled, perform a writeline in the outputfile like:

writeline(5, "$var1 + ";" + $var2 + ";" + .. etc")

you could start off (after the creation of the outputfile) with a line that defines all fields in a similar way.

the code would look like :

code:
 break on

if open (5, "$outputfile", 4) = 0
endif

if writeline (5, "Computername" + ";" + "cpuspeed" + ";" + etc)
;writes the line at the top of the file that explains the data..
endif

for each "file to read" ;think of something here to address all the files you want to read

While @error = 0
$line = readline (5)
IF $line <> ""
====================================
extract data from the $line variable
====================================
endif
loop

writeline(5, "$var1 + ";" + $var2 + ";" + .. etc")

next

if close(5) = 0
endif

exit


looking at your inputfile it is a shame that the software enbtries do not have a "=" sparating the description from the value, that way it would have been real easy to use a split on the read line usoing the "=" to split on and extracting the description and value in a similar way for each line. Maybe that can b changed ??

did that help ??

Ciao,
MvdW

_________________________
rgrds, Maarten

Top
#58823 - 2001-09-05 03:16 PM Re: Log Parsing and redirection to CSV
Anonymous
Unregistered


Many thanks... I was stuck at the sub-loop logic, and I think this will definately help. I will add the = as a delimeter going forward, as well as add an EOF statement at the end.

I'll pound around with this and get back to you, I appreciate your response!

Chuck,

[ 05 September 2001: Message edited by: chas31av ]

Top
#58824 - 2001-09-05 05:44 PM Re: Log Parsing and redirection to CSV
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
The following script will collate the information for you.
code:
BREAK ON CLS
;
; logparse.kix

; Amendment history:
; 11 September 2001 RMH File is now CLOSE() when input exhausted
; 5 September 2001 RMH Initial release. Created with VIM.

; Parse logfiles and convert to CSV format.
; Change the parameters on the lines marked <<CHANGE ME>> to fit your local
; setup.

DIM $TokenName[64]
DIM $TokenValue[64]
$TokenLimit=64

$DEBUG=0 ; <<CHANGE ME>> 1=Debug info on, 0=off
$SourcePattern="d:\kixdev\*.log" ; <<CHANGE ME>> File path/pattern for log source files
$TargetCSVFile="d:\kixdev\output.csv" ; <<CHANGE ME>> Consolidated output file
$MaximumSoftware=0 ; Most number of software packages

$TokenCount=0
$SystemTokens="~"
$Token = "DOMAIN" GOSUB SUBAddToken
$Token = "LOGON_SERVER" GOSUB SUBAddToken
$Token = "WORKSTATION" GOSUB SUBAddToken
$Token = "USERID" GOSUB SUBAddToken
$Token = "FULLNAME" GOSUB SUBAddToken
$Token = "PRIVILIDGE" GOSUB SUBAddToken
$Token = "IPADDRESS" GOSUB SUBAddToken
$Token = "MAC_ADDRESS" GOSUB SUBAddToken
$Token = "LOCATION" GOSUB SUBAddToken
$Token = "OS_VERSION" GOSUB SUBAddToken
$Token = "CPU_SPEED(MHz)" GOSUB SUBAddToken
$Token = "FREE_C:\(Bytes)" GOSUB SUBAddToken
$Token = "KIX_VERSION" GOSUB SUBAddToken
$Token = "KIX_Directory" GOSUB SUBAddToken
$Token = "AMO_INSTALLED" GOSUB SUBAddToken
$Token = "SDO_INSTALLED" GOSUB SUBAddToken
$Token = "LAST_LOGIN" GOSUB SUBAddToken

$=REDIRECTOUTPUT($TargetCSVFile,1)
$Index=1 WHILE $Index <= $TokenCount GOSUB SUBGetValue '"$Token",' $TokenValue[$Index]="" $Index=$Index+1 LOOP
"Software..." ?
$=REDIRECTOUTPUT("")

$SourceFile=DIR($SourcePattern)
WHILE $SourceFile <> "" AND @ERROR = 0
GOSUB SUBProcessFile
$SourceFile=DIR()
LOOP

? "Processing complete. Outputfile is " $TargetCSVFile ?

EXIT

:SUBProcessFile
"Processing input file " $SourceFile ?
$Section=0
IF OPEN(1,$SourceFile,2)
"Error opening " $SourceFile " for reading" ?
"Additional info: " @ERROR @SERROR ?
RETURN
ENDIF
$LineCount=0
$Input=READLINE(1)
WHILE @ERROR=0
$LineCount=$LineCount+1
IF $DEBUG " $LineCount >> " $Input ? ENDIF
gosub SUBParseLine
$Input=READLINE(1)
LOOP
$=CLOSE(1)
$=REDIRECTOUTPUT($TargetCSVFile,0) ? $=REDIRECTOUTPUT("")
RETURN

:SUBParseLine
if $Input = "" RETURN ENDIF
if substr($Input,1,1) = "-" RETURN ENDIF
IF $Input = "System Information"
$Section=1
$SectionName=$Input
" Processing " $SectionName "..." ?
RETURN
ENDIF
IF $Input = "Software Information"
$=REDIRECTOUTPUT($TargetCSVFile,0)
GOSUB SUBPrintSystem
$=REDIRECTOUTPUT("")
$Section=2
$SectionName=$Input
" Processing " $SectionName "..." ?
RETURN
ENDIF
if $Section ELSE
IF $Debug " Discarding unknown Line" ? ENDIF
RETURN
ENDIF
$Routine="SUBSection" + $Section
GOSUB $Routine
RETURN

:SUBSection1
$TokenIN=$Input
GOSUB SUBGetToken $SystemElement=$Token
GOSUB SUBGetIndex
IF $Index ELSE " INVALID TOKEN " GOSUB SUBParseError RETURN ENDIF
GOSUB SUBGetToken
IF $Token <> "=" " INVALID LINE FORMAT " GOSUB SUBParseError RETURN ENDIF
IF $Debug " System element " $SystemElement " has value '" $TokenIn "'" ? ENDIF
$TokenValue[$Index]=$TokenIn
RETURN

:SUBSection2
$TokenIN=$Input
GOSUB SUBGetToken IF $Token <> "Software" " Not a software entry " GOSUB SUBParseError RETURN ENDIF
GOSUB SUBGetToken IF $Token <> "Entry" " Not a software entry " GOSUB SUBParseError RETURN ENDIF
GOSUB SUBGetToken
$SoftwareElement=VAL($TOKEN)
IF $SoftwareElement ELSE " No software entry number " GOSUB SUBParseError RETURN ENDIF
IF $SoftwareElement > $MaximumSoftware $MaximumSoftware=$SoftwareElement ENDIF
IF $Debug " Software element " $SoftwareElement " has value '" $TokenIn "'" ? ENDIF
$=REDIRECTOUTPUT($TargetCSVFile,0)
'"$TokenIn",'
$=REDIRECTOUTPUT("")
RETURN

:SUBGetToken
$Space=INSTR($TokenIN," ")
IF $Space
$Token=substr($TokenIn,1,$Space-1)
$TokenIn=substr($TokenIn,$Space+1,9999)
ELSE
$Token=$TokenIn
ENDIF
RETURN

:SUBParseError
"ERROR parsing line " $LineCount " in section " $SectionName ?
" >> $Input" ?
RETURN

:SUBAddToken
$TokenCount=$TokenCount + 1
$Key=CHR($TokenCount+ASC("A")-1)
IF $TokenCount >= $TokenLimit "TOO MANY TOKENS" EXIT ENDIF
$SystemTokens="$SystemTokens$Key~$Token~"
RETURN

:SUBGetIndex
$Index=INSTR($SystemTokens,"~" + $Token + "~")
if $Index ELSE RETURN ENDIF
$Index=ASC(substr($SystemTokens,$Index-1,1))-ASC("A")+1
RETURN

:SUBGetValue
$Token=""
$Key="~" + CHR($Index+ASC("A")-1) + "~"
$MyIndex=INSTR($SystemTokens,$Key)
if $MyIndex ELSE RETURN ENDIF
$Token=substr($SystemTokens,$MyIndex+3,999)
$Token=substr($Token,1,INSTR($Token,"~")-1)
RETURN

:SUBPrintSystem
$MyIndex=1
WHILE $MyIndex <= $TokenCount
'"' $TokenValue[$MyIndex] '",'
$TokenValue[$MyIndex]=""
$MyIndex=$MyIndex+1
LOOP
RETURN


<EDIT>I haven't put any code in to deal with speech marks in the data - these will fubar the CSV file. If you need this functionality, the easiest thing is to strip speechmarks as eash line if read from the file.</EDIT>

[ 05 September 2001: Message edited by: Richard Howarth ]

<EDIT2>
Added a CLOSE after the input exhausted to allow the file stream to be reopened for the next file - avoids a -3 error (Thanks Chas ;))
<EDIT2>

[ 11 September 2001: Message edited by: Richard Howarth ]

Top
#58825 - 2001-09-11 06:24 PM Re: Log Parsing and redirection to CSV
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Bug fixed in code. See previous posting for details.
Top
Page 1 of 1 1


Moderator:  Glenn Barnas, NTDOC, Arend_, Jochen, Radimus, Allen, ShaneEP, Ruud van Velsen, Mart 
Hop to:
Shout Box

Who's Online
0 registered and 2220 anonymous users online.
Newest Members
Viginette, ManuvdWielNL, Sir_Barrington, batdk82, StuTheCoder
17888 Registered Users

Generated in 0.057 seconds in which 0.03 seconds were spent on a total of 12 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org