Page 1 of 1 1
Topic Options
#170169 - 2006-11-09 12:14 AM Kix is too fast...
7LayerJeff Offline
Lurker

Registered: 2006-10-20
Posts: 3
My script gets a list of text files (standard formatted audit files with extension *.aud), uses ReadLine to parse each file line-by-line, extracts strings to variables, then jams the data into a MS SQL 2000 table so the audit data is actually useful to us. Here's the odd thing: if I run this in debug mode holding down my return key to complete the script, it works flawlessly. If I try to execute the script at regular speed it only writes about 60% of the lines to the SQL database. Which records get written is random each time it runs. There are no errors.

I'm looking for help figuring out where the problem is when I run at full speed. One suspicion I have is that maybe the SQL server is choking on the speed data is being sent to it. Everything is running directly on the SQL server so no network is involved--even the files being parsed are on it. The server itself is extremely powerful and very underutilized so, I don't know, maybe I'm off on that. The files being parsed are rarely larger than 60K but are occasionally as large as 700 K. Any ideas are appreciated. Yes, I know the code is kludgy, horribly structured and supremely ugly...

Code:
 ; Declare global Variables
Dim $AudFile
Dim $Server
Dim $Database
Dim $OraclePN
Dim $UnixPID
Dim $DateTime
Dim $Action
Dim $DBUser
Dim $Priv
Dim $CliUser
Dim $Terminal
Dim $Status
Dim $LineStr
Dim $FileLines
Dim $CurrLine
Dim $RecordTime

; Open connection to the SQL database
$objConn = DBConnOpen("Driver={SQL Server};Server=Kaos;Database=OraAudit;Trusted_Connection=yes")

; Gather a list of all *.aud files
$FileList = DirPlus("c:\oraaudit\current\","/S /A-D /F aud")
$TotFiles = 0
For Each $FileElem in $FileList
$TotFiles = $TotFiles+1
Next
; Just leave if there's nothing to do
If $TotFiles = 0
Goto LEAVE
EndIf
$FileCount = 0
$TotFiles = $TotFiles-1

:NEXTFILE
$AudFile = $FileList[$FileCount]

; Open the current Oracle Audit file being processed
Open(1, $AudFile, 0)

:NEXTREC
; Read current line into string variable
$LineStr = ReadLine(1)
If @ERROR <> 0
Goto CLOSEFILE
EndIf

; Decide whether to change session or action variables
Select
Case Left ("$LineStr", 10) = "Audit File"
Gosub SETSESS
Case Left ("$LineStr", 3) = "Mon" Or
Left ("$LineStr", 3) = "Tue" Or
Left ("$LineStr", 3) = "Wed" Or
Left ("$LineStr", 3) = "Thu" Or
Left ("$LineStr", 3) = "Fri" Or
Left ("$LineStr", 3) = "Sat" Or
Left ("$LineStr", 3) = "Sun"
Gosub SETACTION
EndSelect

:CLOSEFILE
; Closes the current Oracle Audit file being processed
Close(1)

; Move on to process next file in the list
If $FileCount < $TotFiles
$FileCount = $FileCount+1
Goto NEXTFILE
EndIf

:LEAVE
; Close connection to sql database
DBConnClose($objConn)

Quit

:SETSESS
; Subroutine to parse session-level lines and assign to variables
Select
Case Left ("$LineStr", 10) = "Audit File"
$AudFile = Right ("$LineStr", -11)
Case Left ("$LineStr", 10) = "Node name:"
$Server = Right ("$LineStr", -11)
Case Left ("$LineStr", 14) = "Instance name:"
$Database = Right ("$LineStr", -15)
Case Left ("$LineStr", 14) = "Oracle process"
$OraclePN = Right ("$LineStr", -23)
Case Left ("$LineStr", 12) = "Unix process"
$UTempPID = Right ("$LineStr", -18)
Gosub UNIXPID
$LineStr = ReadLine(1)
$LineStr = ReadLine(1)
Goto SETACTION
EndSelect
$LineStr = ReadLine(1)
If @ERROR <> 0
Goto CLOSEFILE
EndIf
Goto SETSESS
Return

:UNIXPID
For $i = 1 to 7
If SubStr("$UTempPID", $i,1) = ","
$UnixPID = Left("$UTempPID", $i-1)
EndIf
Next
Return

:SETACTION
; Subroutine to parse action-level lines and assign to variables
Select
Case Left ("$LineStr", 3) = "Mon" Or
Left ("$LineStr", 3) = "Tue" Or
Left ("$LineStr", 3) = "Wed" Or
Left ("$LineStr", 3) = "Thu" Or
Left ("$LineStr", 3) = "Fri" Or
Left ("$LineStr", 3) = "Sat" Or
Left ("$LineStr", 3) = "Sun"
$DateTime = SubStr("$LineStr", 5, 20)
Case Left ("$LineStr", 8) = "ACTION :"
Gosub ACTION
Case Left ("$LineStr", 14) = "DATABASE USER:"
$DBUser = Right ("$LineStr", -15)
Case Left ("$LineStr", 11) = "PRIVILEGE :"
$Priv = Right ("$LineStr", -12)
Case Left ("$LineStr", 12) = "CLIENT USER:"
$CliUser = Right ("$LineStr", -13)
Case Left ("$LineStr", 16) = "CLIENT TERMINAL:"
$Terminal = Right ("$LineStr", -17)
Case Left ("$LineStr", 7) = "STATUS:"
$Status = Right ("$LineStr", -8)
Goto WRITE
EndSelect
$LineStr = ReadLine(1)
If @ERROR <> 0
Goto CLOSEFILE
EndIf
Goto SETACTION
Return

:ACTION
; Combines ACTION lines if it spans multiple lines
$LineStr = Right ("$LineStr", -9)
$J = Len($LineStr)-2
$LineStr = SubStr("$LineStr",2,$J)
If InStr($LineStr, "'")
Gosub QUOTES
Else $Action = $LineStr
EndIf
$LineStr = ReadLine(1)
If @ERROR<> 0
Goto CLOSEFILE
EndIf
If Left ("$LineStr", 14) = "DATABASE USER:"
$DBUser = Right ("$LineStr", -15)
$LineStr = ReadLine(1)
If @ERROR<> 0
Goto CLOSEFILE
EndIf
Goto SETACTION
EndIf
While Left ("$LineStr", 14) <> "DATABASE USER:"
If InStr("$LineStr", "'")
Gosub QUOTES
Else
$Action = $Action + $LineStr
EndIf
$LineStr = ReadLine(1)
If @ERROR<> 0
Goto CLOSEFILE
EndIf
Loop
Goto SETACTION

:QUOTES
; Change single'quotes to two single-quotes so SQL statement handles correctly
For $k = 1 to (Len($LineStr))
If SubStr($LineStr, $k, 1) <> Chr(39)
$Action=$Action+SubStr($LineStr, $k, 1)
Else
$Action=$Action+Chr(39)+Chr(39)
EndIf
Next
Return

:WRITE
; Timestamp each record for ordering, and because nothing else is unique
$RecMon = @MONTHNO
$RecMDay = @MDAYNO
$RecTime = @TIME
$RecMsec = @MSECS
$RecYr = @YEAR
$RecordTime = Chr(39)+'$RecYr'+Chr(45)+'$RecMon'+Chr(45)+'$RecMDay'+Chr(32)+'$RecTime'+Chr(46)+'$RecMsec'+Chr(39)

;Write record to SQL database
$sql = "INSERT DBAAudit VALUES ("+Chr(39)+$AudFile+Chr(39)+","+Chr(39)+$Server+Chr(39)
+","+Chr(39)+$Database+Chr(39)+","+$OraclePN+","+$UnixPID+","+Chr(39)+$DateTime+Chr(39)
+","+Chr(39)+$Action+Chr(39)+","+$DBUser+","+Chr(39)+$Priv+Chr(39)+","+Chr(39)+$CliUser+Chr(39)
+","+Chr(39)+$Terminal+Chr(39)+","+$Status+","+$RecordTime+")"
$retcode = DBExecuteSQL($objConn, $sql)

; skip the blank line after STATUS and then cleanup action variable
$LineStr = ReadLine(1)
If @ERROR <> 0
Goto CLOSEFILE
EndIf
$action = ''

Goto NEXTREC

Function DBConnOpen($ConnDSN, optional $ConnTimeout, optional $CmdTimeout)
Dim $objConn, $adStateOpen

$adStateOpen=Val('&00000001')

If VarType($ConnTimeout)
$ConnTimeout=Val($ConnTimeout)
Else
$ConnTimeout=15
EndIf

If VarType($CmdTimeout)
$CmdTimeout=Val($CmdTimeout)
Else
$CmdTimeout=30
EndIf

$ConnDSN=Trim($ConnDSN)
If Not $ConnDSN
Exit 87
EndIf

$objConn = CreateObject("ADODB.Connection")
If @ERROR
Exit @ERROR
EndIf

$objConn.ConnectionTimeout = $ConnTimeout
If @ERROR
Exit @ERROR
EndIf

$objConn.CommandTimeout = $CmdTimeout
If @ERROR
Exit @ERROR
EndIf

$objConn.Open($ConnDSN)
If @ERROR
Exit @ERROR
EndIf

If Not $objConn.State=$adStateOpen
$objComm=''
$DBConnOpen=''
Exit @ERROR
EndIf
$DBConnOpen=$objConn

EndFunction

Function DirPlus($path,optional $Options, optional $f, optional $sfflag)
If Not VarType($f) Dim $f EndIf
If Not VarType($sfflag) Dim $sfflag EndIf

Dim $file, $i, $temp, $item, $ex1, $mask,$mask1,$maskArray,$maskarray1,
$ex2, $code, $CodeWeight, $targetWeight, $weight, $masktrue
Dim $tarray[0]

$ex1 = SetOption(Explicit,on)
$ex2 = SetOption(NoVarsInStrings,on)
$codeWeight = 0

If Not Exist($path)
$temp = SetOption(Explicit,$ex1)
$temp = SetOption(NoVarsInStrings,$ex2)
Exit @ERROR
EndIf

If Not VarType($f)
$f = CreateObject("Scripting.FileSystemObject").getfolder($path)
EndIf
If @ERROR
$temp = SetOption(Explicit,$ex1)
$temp = SetOption(NoVarsInStrings,$ex2)
Exit @ERROR
EndIf

For Each $temp in Split($options,"/")
$temp=Trim($temp)
Select
Case Left($temp,1) = "s"
If Not VarType($sfflag)
If Val(Right($temp,-1)) = 0
$sfflag = -1
Else
$sfflag = Val(Right($temp,-1))
EndIf
EndIf
Case Left($temp,1) = "a"
Select
Case Right($temp,-1)="d"
$codeWeight = $codeWeight + 1
$temp = "if $file.attributes & 16 " ;"if $file.type = 'File Folder' "
Case Right($temp,-1)="-d"
$codeWeight = $codeWeight + 1
$temp = "if ($file.attributes & 16)=0 " ;"if $file.type <> 'File Folder' "
Case Right($temp,-1)="s"
$codeWeight = $codeWeight + 1
$temp = "if $file.attributes & 4 "
Case Right($temp,-1)="-s"
$codeWeight = $codeWeight + 1
$temp = "if ($file.attributes & 4)=0 "
Case Right($temp,-1)="h"
$codeWeight = $codeWeight + 1
$temp = "if $file.attributes & 2 "
Case Right($temp,-1)="-h"
$codeWeight = $codeWeight + 1
$temp = "if ($file.attributes & 2)=0 "
Case Right($temp,-1)="r"
$codeWeight = $codeWeight + 1
$temp = "if $file.attributes & 1 "
Case Right($temp,-1)="-r"
$codeWeight = $codeWeight + 1
$temp = "if ($file.attributes & 1)=0 "
Case Right($temp,-1)="a"
$codeWeight = $codeWeight + 1
$temp = "if $file.attributes & 32 "
Case Right($temp,-1)="-a"
$codeWeight = $codeWeight + 1
$temp = "if ($file.attributes & 32)=0 "
EndSelect
$code = $temp + "$weight=$weight+1 endif" +@CRLF + $code

Case Left($temp,1) = "m"
$maskarray = Split(Right($temp,-2),"|")
$codeweight = $codeweight + 1
$code = "$masktrue=0 for Each $mask in $maskarray if instr($file.name,$mask) $masktrue=1 " +
"EndIf Next If $masktrue $weight=$weight+1 endif" + @CRLF +$code
Case Left($temp,1) = "f"
$maskarray1 = Split(Right($temp,-2)," ")
$codeweight = $codeweight + 1
$code = "$masktrue=0 for Each $mask1 in $maskarray1 if substr($file.name,Instrrev($file.name,'.')+1)" +
"=$mask1 $masktrue=1 EndIf Next If $masktrue $weight=$weight+1 endif" + @CRLF +$code

EndSelect
Next
$code = "$weight = 0 $targetWeight = " + $codeweight + @CRLF + $code
$code = $code + "if $weight = $targetweight Exit 1 endif"

For Each $file in $f.subfolders
If Execute($code)
$tarray[$i] = $file
$i = $i + 1
ReDim preserve $tarray[$i]
EndIf
If $sfflag
$temp = dirplus($file,$options,$file,$sfflag-1)
For Each $item in $temp
$tarray[$i] = $item
$i = $i + 1
ReDim preserve $tarray[$i]
Next
EndIf
Next
For Each $file in $f.files
If Execute($code)
$tarray[$i] = $file
$i = $i + 1

ReDim preserve $tarray[$i]
EndIf
Next

If $i
ReDim preserve $tarray[$i-1]
$i=0
Else
$tarray = 0
EndIf

$dirplus = $tarray
$temp = SetOption(Explicit,$ex1)
$temp = SetOption(NoVarsInStrings,$ex2)
Exit @ERROR
EndFunction

Function ArrayDim($array, optional $dims)
Dim $a

If VarType($array) & 8192
$a=0
Do
$a=$a+1
Until UBound($array,$a)=-1
$arraydim=$a-1
If Val($dims)
Dim $b[$arraydim-1]
For $a=1 to $arraydim
$b[$a-1]=UBound($array,$a)
Next
$arraydim=$b
EndIf
Else
$arraydim=-1
EndIf
EndFunction

Function PadStr($Input, $Pad, $Length, optional $PadSide)
Dim $i, $x
$PadStr = ""
$Input = "" + $Input
$Pad = "" + $Pad
$Length = 0 + $Length
If $PadSide="" Or Len($PadSide)>1 Or InStr("LR",$PadSide)= 0
$PadSide = "L"
EndIf

$x = Len($Input)

For $i=$x to $Length - 1 Step Len($Pad)
If $PadSide = "L"
$Input = $Pad + $Input
Else
$Input = $Input + $Pad
EndIf
Next
If $PadSide = "L"
$Input = Right($Input, $Length)
Else
$Input = Left($Input, $Length)
EndIf
$PadStr = $Input
Exit 0
EndFunction

Function DBExecuteSQL($objConn, $sql, optional $cmdType)
Dim $cmdCommand, $rsRecordset
Dim $adCmdUnspecified, $adCmdText, $adCmdTable, $adCmdStoredProc, $adCmdUnknown, $adCmdFile, $adCmdTableDirect

$adCmdUnspecified = -1
$adCmdText = 1
$adCmdTable = 2
$adCmdStoredProc = 4
$adCmdUnknown = 8
$adCmdFile = 256
$adCmdTableDirect = 512

$cmdType=IIf(VarType($cmdType),Val($cmdType),$adCmdText)

If VarType($objConn)<>9 Or $sql=''
Exit 87
EndIf

$cmdCommand = CreateObject('ADODB.Command')
If @ERROR
Exit @ERROR
EndIf

$cmdCommand.ActiveConnection = $objConn
If @ERROR
Exit @ERROR
EndIf

$cmdCommand.CommandType = $cmdType
If @ERROR
Exit @ERROR
EndIf

$cmdCommand.CommandText = $sql
If @ERROR
Exit @ERROR
EndIf

$rsRecordset=$cmdCommand.Execute()
$rsRecordset=''
$cmdCommand=''
$DBExecuteSQL=@ERROR
Exit @ERROR
EndFunction

Function DBConnClose($objConn)
Dim $adStateOpen

$adStateOpen = 1

If VarType($objConn)=9
If $objConn.State = $adStateOpen
$objConn.Close()
If @ERROR
Exit @ERROR
EndIf
EndIf
$objConn=''
Else
Exit 87
EndIf

$DBConnClose=0
Exit 0
EndFunction



**Moderator Edit - Richard H. - Split long line to enhance readability **


Edited by Richard H. (2006-11-09 08:56 AM)

Top
#170170 - 2006-11-09 01:21 AM Re: Kix is too fast...
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11629
Loc: CA
Please edit your post and remove the LONG LINES so that we can
actually attempt to read your code.

Thanks

Top
#170171 - 2006-11-09 10:12 AM Re: Kix is too fast...
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4673
Loc: The Netherlands
I have to admit that I did not read the entire script.

What happens if you put a sleep after each SQL command so the SQL server has some time to process the stuff?

Code:

SQL command here
sleep 2
next SQL command



Edited by Mart (2006-11-09 12:19 PM)
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
#170172 - 2006-11-09 05:25 PM Re: Kix is too fast...
7LayerJeff Offline
Lurker

Registered: 2006-10-20
Posts: 3
That does it! I decreased the delay incrementally until I got down to:
Code:
sleep .01


At that point it still runs plenty fast and completes successfully. I'll leave it there. Thanks for the tip!

Turns out Kix is too fast. Rare that I ever have that problem with anything.

Top
#170173 - 2006-11-09 07:02 PM Re: Kix is too fast...
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11629
Loc: CA
I think there must be some other issue at play here as SQL can take
publishing and transfers natively without such delays.

I'm not really sure what mechanism might be at play here but I don't think
it is speed.
 
Glad it's working for you though and thank you for correcting the long lines.
 

Top
#170174 - 2006-11-10 05:28 AM Re: Kix is too fast...
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11165
Loc: Boston, MA, USA
Feels like the SQL statements are being executed asynchronously.
_________________________
There are two types of vessels, submarines and targets.

Top
#170175 - 2006-11-10 04:08 PM Re: Kix is too fast...
7LayerJeff Offline
Lurker

Registered: 2006-10-20
Posts: 3
Two things:

1) The sleep command I described above should have said "sleep 0.01" not "sleep .01". The latter is not correct syntax and won't work.

2) I noticed that if I run the script nice and slow the records are written to the sql table in the exact order they are sent. As I gradually speed up the script (by decreasing the sleep time) the record writes to the table become more and more disordered, and at some point I just start losing records. I'm thinking now it's a SQL issue--at full speed the script writes about 150 records/sec, which is nothing. I'll have our DBAs look into it. If they find anything I'll post it for other's benefit. Thanks to all for your input.

Top
#170176 - 2006-11-21 07:39 PM Re: Kix is too fast...
Kdyer Offline
KiX Supporter
*****

Registered: 2001-01-03
Posts: 6241
Loc: Tigard, OR
Quote:

Two things:

1) The sleep command I described above should have said "sleep 0.01" not "sleep .01". The latter is not correct syntax and won't work.

2) I noticed that if I run the script nice and slow the records are written to the sql table in the exact order they are sent. As I gradually speed up the script (by decreasing the sleep time) the record writes to the table become more and more disordered, and at some point I just start losing records. I'm thinking now it's a SQL issue--at full speed the script writes about 150 records/sec, which is nothing. I'll have our DBAs look into it. If they find anything I'll post it for other's benefit. Thanks to all for your input.




IMHO, I would remove the GOTOs from your script.. You can probably decrease the size of the script by removing them and incorporate them as FUNCTIONS..

Kent
_________________________
Utilize these resources:
UDFs (Full List)
KiXtart FAQ & How to's

Top
Page 1 of 1 1


Moderator:  Glenn Barnas, NTDOC, Arend_, Jochen, Radimus, Allen, ShaneEP, Ruud van Velsen, Mart 
Hop to:
Shout Box

Who's Online
0 registered and 820 anonymous users online.
Newest Members
batdk82, StuTheCoder, M_Moore, BeeEm, min_seow
17885 Registered Users

Generated in 0.134 seconds in which 0.093 seconds were spent on a total of 12 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org