Page 3 of 3 <123
Topic Options
#97363 - 2003-01-02 03:32 PM Re: KiXGolf: Part 2: The Code: Burrows-Wheeler Transform
BrianTX Offline
Korg Regular

Registered: 2002-04-01
Posts: 895
Actually, speed does matter, at least in applications here at work. We deal with log files of about 50 Gigabytes per month. Some compression algorithms would take days to compress the log files, so we had to find a utility with good speed and excellent text compression. So, with WinRar we are able to compress the files down to about 3 or 4 GB per month. It still takes several hours.

Brian

Top
#97364 - 2003-01-02 03:33 PM Re: KiXGolf: Part 2: The Code: Burrows-Wheeler Transform
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11165
Loc: Boston, MA, USA
I don't think you want to use KiXtart for that [Wink]
_________________________
There are two types of vessels, submarines and targets.

Top
#97365 - 2003-01-02 03:50 PM Re: KiXGolf: Part 2: The Code: Burrows-Wheeler Transform
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
brian, try to copy the same amount to different hd.
then extract that copy time from the total.

what greater files, that greater is also the meaning of hd-speed
_________________________
!

download KiXnet

Top
#97366 - 2003-01-02 03:59 PM Re: KiXGolf: Part 2: The Code: Burrows-Wheeler Transform
BrianTX Offline
Korg Regular

Registered: 2002-04-01
Posts: 895
I don't see that much will be gained by copying files.. The files are already on an HP NetRaid controller with fast SCSI drives (Configured for RAID-5). What's slowing us down is our Pentium III 500Mhz quad-processor system running NT 4.0. What I end up doing is running four compressions at once so that I can fully use the processing power. A dual Pentium 4 at 2 GHz would be much faster, I believe, but it still takes time!

Brian

Top
#97367 - 2003-01-02 04:01 PM Re: KiXGolf: Part 2: The Code: Burrows-Wheeler Transform
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
would, have not tested?
I didn't mean that copying will help anything. it shows you how slow it actually is to move files.

when you have that huge files, the moving to your compressor and the compressed to somewhere takes time too.
_________________________
!

download KiXnet

Top
#97368 - 2003-01-06 02:03 AM Re: KiXGolf: Part 2: The Code: Burrows-Wheeler Transform
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
shawn?
not sure you are out there but I would like some speed info/comparizon between your and the winning code.

if the increase is more than 10% then it's worth of investigating...
_________________________
!

download KiXnet

Top
#97369 - 2003-01-06 03:10 AM Re: KiXGolf: Part 2: The Code: Burrows-Wheeler Transform
Shawn Administrator Offline
Administrator
*****

Registered: 1999-08-13
Posts: 8611
Its much more than 10% - Erik (kholm) did a quick speed comparison and posted the results but I cant find it [Frown] ... the comparison was something like the following (for the 749 byte string):

Winning code: 2.5 seconds
Fast code: 30 msecs

Its no big mystery or anything - the fast code is fast because it only needs to perform two passes of the data, and one pass of the "alphabet" 0-255 ... no matter how big the string is.

-Shawn

Top
#97370 - 2003-01-06 03:12 AM Re: KiXGolf: Part 2: The Code: Burrows-Wheeler Transform
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
thanks, so I'll play with it then.

man, the difference is huge! [Eek!]
_________________________
!

download KiXnet

Top
#97371 - 2003-01-06 03:12 AM Re: KiXGolf: Part 2: The Code: Burrows-Wheeler Transform
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11165
Loc: Boston, MA, USA
Part two of the KiXGolf tournament "Burrows-Wheeler-Transform" is now closed. Winners will be announced soon in the "Part Three" thread.
_________________________
There are two types of vessels, submarines and targets.

Top
Page 3 of 3 <123


Moderator:  Arend_, Allen, Jochen, Radimus, Glenn Barnas, ShaneEP, Ruud van Velsen, Mart 
Hop to:
Shout Box

Who's Online
0 registered and 657 anonymous users online.
Newest Members
M_Moore, BeeEm, min_seow, Audio, Hoschi
17883 Registered Users

Generated in 0.068 seconds in which 0.034 seconds were spent on a total of 11 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org