annie
(Fresh Scripter)
2005-03-16 09:25 PM
how to scrub a file?

I need to 'scrub' a log file that's being placed on my server and will grow continually unless I purge records > say 72 hours old. Anyone know a decent way to do this? help??

NTDOCAdministrator
(KiX Master)
2005-03-16 09:26 PM
Re: how to scrub a file?

Try searching here on the board using different keywords.

Plenty of code and UDF stuff to do what you want.

Sorry don't have time to do the search for you right now.

Richard H.Administrator
(KiX Supporter)
2005-03-16 09:33 PM
Re: how to scrub a file?

Normally with log files you rotate them rather than delete records.

Pick a period (hourly, daily) and when it is triggered MOVE the file to a new name - keep a few generations.

The only tricky thing is that the process writing to the file needs to understand what is happening when the file moves. You may need to stop the process, move the file and then re-start the process.

For example, assuming your file is called "mylogfile.txt" and you want to keep a weeks worth then at midnight each day you would do the following in a batch file:
Code:
move mylogfile.006 mylogfile.007
move mylogfile.005 mylogfile.006
move mylogfile.004 mylogfile.005
move mylogfile.003 mylogfile.004
move mylogfile.002 mylogfile.003
move mylogfile.001 mylogfile.001
move mylogfile.txt mylogfile.001

annie
(Fresh Scripter)
2005-03-18 07:11 PM
Re: how to scrub a file?

I did search, and didn't find anything that removed records, only things that removed files or folders. I don't really wnat to copy the file, as my goal is to reduce the amount of space the logs are taking up, not increase by generating multiple copies. all I want to do is shave the oldest records off the bottom.

LonkeroAdministrator
(KiX Master Guru)
2005-03-18 07:32 PM
Re: how to scrub a file?

so, why not just use readfile() and then writefile()

nah, even easier, readline() + drop-the-last-lines-code + writeline()

Bryce
(KiX Supporter)
2005-03-18 08:40 PM
Re: how to scrub a file?

here is some code, using FSO to return a files creation date, and then compairing the age of that date vs the current date and time.

i ma using the UDF's FlipCTime() and TimeConvert()

Code:

$f = 'logfile.log'
$f = CreateObject("Scripting.FileSystemObject").GetFile($f)

$date = Split($f.DateCreated)[0]
$date = Split($date,"/")
$date = $date[2]+"/"+$date[0]+"/"+$date[1]

$time = timeconvert(Split($f.DateCreated)[1] +" "+ Split($f.DateCreated)[2])

$72h = 259200

If flipctime(@DATE,@TIME) - flipctime($date,$time) > $72h
	? 'the file ' + $f.name + ' was created 72+ hours ago'
EndIf


Bryce

Les
(KiX Master)
2005-03-18 09:23 PM
Re: how to scrub a file?

You guys are missing the req's. It is just one file. Within this file are dated records and the ojective is to purge "older than" records. AFAIK, since KiX cannot edit files in place, the only option is to open the existing log AND create/open a new file, recursively parse and skip or copy to new until EOF, close both files and delete the original log, keeping the new smaller log.

NTDOCAdministrator
(KiX Master)
2005-03-18 09:28 PM
Re: how to scrub a file?

Questions:

1. How many lines in the log file or how large in MB
2. Do you want to purge from the TOP or BOTTOM of log?
3. Is this log file open by another process, any issues there?
4. Entries older then x How are old entries determined?

Richard's solution would trim down the file size for you automatically. If you don't want to keep the older files, then delete them instead of saving them.

LonkeroAdministrator
(KiX Master Guru)
2005-03-18 10:12 PM
Re: how to scrub a file?

isn't that just what les and someone else already said...

Les
(KiX Master)
2005-03-18 10:19 PM
Re: how to scrub a file?

While I don't think Annie is ready to accept Richard's suggestion yet, I believe it is the most logical approach. Rather than let hte log file grow too large, move it to another file on a regular schedule and purge the old files once they reach a certain age.

Also, this topic should have been called "How to prune a file?"

NTDOCAdministrator
(KiX Master)
2005-03-18 10:40 PM
Re: how to scrub a file?

Here are some UDFs that might help if you do want to edit the file

UDFs from Waltz
ReadFileHead() - Delivers the first #n lines of a file

ReadFileTail() - Delivers the last #n lines of a file

UDF from Bryce
Tail() - return the last Line or Lines of a file

Richard H.Administrator
(KiX Supporter)
2005-03-21 09:29 AM
Re: how to scrub a file?

Annie,

You are not keeping multiple copies by rotating the file. You set the rotation period so that you keep the same number of records, but simply split up into several files. The rotation automatically truncates the file.

Let's say you want to keep no more than 100 megabytes of log file data. In this case check the file every so often and when it reaches 10MB rotate it. Keep 9 instance of the rotated files.

You will never have much more than 100MB of log files stored providing you set the check interval correctly.

Richard H.Administrator
(KiX Supporter)
2005-03-21 10:18 AM
Re: how to scrub a file?

There is an extensive log maintenance "kit" here: http://www.kixtart.org/ubbthreads/showflat.php?Cat=&Number=75703&page=&view=&sb=5&o=&vc=1

However, I think that this is a bit more fully featured than you need.

Here is a simple example of how you could do the size limit thing in KiXtart:
Code:
Function udfRotateLog($sPath,$iSize,Optional $iGenerations,Optional $bInitLog)



	Dim $sMask,$i,$fd



	If Not CInt($iGenerations) $iGenerations=1 EndIf

	If Not CInt($bInitLog) $bInitLog=0 EndIf

	$iSize=CDBL($iSize)



	While Len($sMask)<Len($iGenerations) $sMask=CSTR($sMask)+"0" Loop



	If GetFileSize($sPath)>=$iSize

		; Sanity check - Ignore

		;	1	Read Only

		;	4	System

		;	16	Direcroty

		;	256	Temporary

		;	4096	Offline

		If (1+4+16+256+4096) & GetFileAttr($sPath) Exit 1 EndIf

		If @ERROR Exit @ERROR EndIf

		; Remove oldest generation

		Del $sPath+"."+$iGenerations

		; Rotate existing log files

		For $i=$iGenerations To 2 Step -1

			Move $sPath+"."+Right($sMask+($i-1),Len($iGenerations)) $sPath+"."+Right($sMask+$i,Len($iGenerations))

			; Ignore "file not found"

			If Not(@ERROR=0 OR @ERROR=2) Exit @ERROR EndIf

		Next

		; Now move primary log file

		Move $sPath $sPath+"."+Right($sMask+1,Len($iGenerations))

		If @ERROR Exit @ERROR EndIf

		; Finally, create an empty log file if needed

		If $bInitLog

			$fd=FreeFileHandle()

			If @ERROR Exit @ERROR EndIf

			If Open($fd,$sPath,1+4) Exit @ERROR EndIf

			$=Close($fd)

		EndIf

	EndIf

	Exit 0

EndFunction


Parameters are:
  • $sPath - Path to your log file.
  • $iSize - When file reaches this size (in bytes) it is rotated. If you set this to zero, the file is always rotated.
  • $iGenerations - How many generations of the file to keep
  • $bInitLog - Set to true if you want to create an empty file once the log file has been rotated.


So to keep 50 MB of history, and create an empty file after the rotation:
Code:
$sLogFile="c:\foo\mylogfile.txt"

$iMaxSize=10*1024*1024 ; 10 Megabytes

$iMaxHistory=5

 

udfRotateLog($sLogFile,$sMaxSize,$iMaxHistory,1)