Page 1 of 1 1
Topic Options
#100724 - 2003-05-05 10:37 PM KixGolf Suggestion
New Mexico Mark Offline
Hey THIS is FUN
****

Registered: 2002-01-03
Posts: 223
Loc: Columbia, SC
There are many flavors of CSV. Most are just subsets of the de facto standard. I would love to see one or two UDF's that could handle reading/writing a file in CSV format (or subset, so long as none of the rules were violated). I'm assuming that:

1. To write, the UDF would accept a filespec or open file object and 2D array of values to write in CSV format.

2. To read, the UDF would accept a filespec or open file object and return a 2D array of values.

Here is the best summary I've personally seen on CSV.

http://www.creativyst.com/Doc/Articles/CSV/CSV01.htm

I've started something like this more than once, but the logic is a little dizzying. However, I'm sure many of the intrepid coders on this board could knock something like this out in a day or so. [Big Grin]

NMM

Top
#100725 - 2003-05-05 10:57 PM Re: KixGolf Suggestion
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
done that been there [Wink]

don't remember anymore what part I coded or was it totally done into beta state...
but then came the lack of need as I used it as DB and we already had our SQL UDFs...
_________________________
!

download KiXnet

Top
#100726 - 2003-05-05 11:11 PM Re: KixGolf Suggestion
New Mexico Mark Offline
Hey THIS is FUN
****

Registered: 2002-01-03
Posts: 223
Loc: Columbia, SC
I've had occasion recently to do some data analysis on various Excel spreadsheets and other data flat-files, sometimes in CSV format. Usually, I can get away with just splitting a CSV file into an array based on a comma delimiter. However, I would love to have a tool to really deal with CSV files properly.

It just struck me that this might be a particularly good candidate for a KixGolf tournament. The standard to which I provided the link is pretty clear, and the scope is limited. I'm estimating that less than 100 lines of code would be needed. The best coders on this board could probably do it in under 50 lines.

Top
#100727 - 2003-05-06 04:23 AM Re: KixGolf Suggestion
Bryce Offline
KiX Supporter
*****

Registered: 2000-02-29
Posts: 3167
Loc: Houston TX
this might be a deal breaker for kix...

quote:

Each record is one line ...but
A record separator may consist of a line feed (ASCII/LF=0x0A), or a carriage return and line feed pair (ASCII/CRLF=0x0D 0x0A).
...but: fields may contain embedded line-breaks (see below) so a record may span more than one line.

kix's native own file reading commands may come up short in being able to read a entry that covers multiple lines...

only way around this that i can think of right now, is if you enter into the CSV reading UDF knowing the total number of data fields before hand, or if your CSV file always has a header line.

Top
#100728 - 2003-05-06 04:37 AM Re: KixGolf Suggestion
Howard Bullock Offline
KiX Supporter
*****

Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
Then the challenge is more interesting...isn't it?

[ 06. May 2003, 04:37: Message edited by: Howard Bullock ]
_________________________
Home page: http://www.kixhelp.com/hb/

Top
#100729 - 2003-05-06 08:55 AM Re: KixGolf Suggestion
MightyR1 Offline
MM club member
*****

Registered: 1999-09-09
Posts: 1264
Loc: The Netherlands
Think not...

Later in the article a demo of this multiline is displayed. In the first and last line, notice the single "...
_________________________
Greetz,
Patrick Rutten

- We'll either find a way or make one...
- Knowledge is power; knowing how to find it is more powerful...
- Problems don't exist; they are challenges...

Top
#100730 - 2003-05-07 12:21 AM Re: KixGolf Suggestion
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
quote:
Later in the article a demo of this multiline is displayed. In the first and last line, notice the single "...
The problem is that KiXtart (with ReadLine() at least) will interpret both 0x0A and 0x0D 0x0A as line terminators.

This means that when you come to reconstruct the data, you don't known which characters to join the field parts back together with.

You would need to use the FSO calls or similar, to get the binary data.

While in practice you may not care what the embedded line termination characters were, you cannot contstruct a valid general CSV parser without handling them.

You could kludge it by counting the number of characters read and comparing to the file length. Where the record line terminators and embedded line terminators are the same characters you can calculate the number of characters missing and determine which line termination characters were used.

Where the embedded line termination is different from the record line termination you may still be able to calculate which to use based on the number of lines read, however there will be occasions when it is impossible to determine (number of record line terminators = number of embedded line terminators).

Top
#100731 - 2003-05-06 01:10 PM Re: KixGolf Suggestion
MightyR1 Offline
MM club member
*****

Registered: 1999-09-09
Posts: 1264
Loc: The Netherlands
Point taken...

Is thougher than it looks [Wink]
_________________________
Greetz,
Patrick Rutten

- We'll either find a way or make one...
- Knowledge is power; knowing how to find it is more powerful...
- Problems don't exist; they are challenges...

Top
#100732 - 2003-05-06 02:17 PM Re: KixGolf Suggestion
Howard Bullock Offline
KiX Supporter
*****

Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
Richard, shame on you for giving away the solution. [Wink]

[ 06. May 2003, 14:18: Message edited by: Howard Bullock ]
_________________________
Home page: http://www.kixhelp.com/hb/

Top
#100733 - 2003-05-06 03:09 PM Re: KixGolf Suggestion
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11165
Loc: Boston, MA, USA
There might be a very elegant solution to the "Read CSV file". One of the standard ODBC drivers is a TXT/CSV-driver. Thus, one should be able to utilize the DB-UDFs in combination with a properly defined CSV-ODBC-driver in order to read/write CSV-text-files.

BTW, I'm already working on a new KiXgolf challenge, but cannot yet give you an ETA for it (maybe two weeks?). I'm currently landscaping the course [Wink]
_________________________
There are two types of vessels, submarines and targets.

Top
#100734 - 2003-05-06 03:12 PM Re: KixGolf Suggestion
MightyR1 Offline
MM club member
*****

Registered: 1999-09-09
Posts: 1264
Loc: The Netherlands
Using the ODBC stuff isn't pure KiX... The Lonk will not like this I guess [Big Grin]

Looking forward to the new Golg Tournament...
_________________________
Greetz,
Patrick Rutten

- We'll either find a way or make one...
- Knowledge is power; knowing how to find it is more powerful...
- Problems don't exist; they are challenges...

Top
#100735 - 2003-05-06 05:18 PM Re: KixGolf Suggestion
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
yeah, me waits for the new golg too [Razz]

not pure, there you are sure but as kix is limited in it's read-write, so that can't be done purely...
_________________________
!

download KiXnet

Top
#100736 - 2003-05-07 02:53 PM Re: KixGolf Suggestion
New Mexico Mark Offline
Hey THIS is FUN
****

Registered: 2002-01-03
Posts: 223
Loc: Columbia, SC
Fields containing embedded reporter-er-line feeds threw me for a loop. The logic of handling embedded quotes is tricky as well.

Personally, I don't see any way to do this without using the FS object. But I really don't see that as a bad thing. Since we are dealing with data, CSV files can get big pretty quickly, which could cause problems with KiXtart's all-in-memory file handling as well.

What makes the standard I referenced so intriguing is that the rules are spelled out clearly. The trick is making a script follow those rules without becoming bloated and slow.

While I felt I could write code that would support the rules, doing it clearly and efficiently is a different matter, and it requires some pretty sophisticated reasoning/coding skills.

That's why I thought it would be a good challenge to this group. FWIW, I'm willing to play too, but I see no chance of me winning this one. I'll be lucky to actually get the code to work, much less make it an object of beauty.

Top
#100737 - 2003-05-07 06:48 PM Re: KixGolf Suggestion
New Mexico Mark Offline
Hey THIS is FUN
****

Registered: 2002-01-03
Posts: 223
Loc: Columbia, SC
I'm starting to play with this a little and I'm seeing problems already.

1. The ideal way (IMO) to read a CSV file would be to read the entire file into a string, then parse the string character-by-character. (This would be drastically faster than reading a file byte-by-byte.) In VBScript this would be no problem as a string may contain 2^31 characters.

This makes reading a file byte-by-byte likely the best option.

2. Processing X records with Y fields each really lends itself to a recursive function. That said, I hate writing/debugging recursive code. I'm going to avoid that for now... but this is where really good scripters can shine.

3. The potential for embedded new lines and quotes within a field makes this a real bear.

4. For now, I'm just playing with a CSV(File, optional Array) UDF. Whether or not an optional array of data is supplied as the second parameter controls whether it reads or writes CSV data.

NMM

Top
#100738 - 2003-05-07 06:52 PM Re: KixGolf Suggestion
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
me thinks that the easiest way to handle this is to read the whole file to memory.

even though your file is 500M in size, that is easily diggable into swap-space [Wink]

that would also be a lot easy to control...
_________________________
!

download KiXnet

Top
#100739 - 2003-05-07 11:28 PM Re: KixGolf Suggestion
kholm Offline
Korg Regular
*****

Registered: 2000-06-19
Posts: 714
Loc: Randers, Denmark
Mark,

Here is a clean KiX code example for reading the CSV-file into a 2D Array.

The errorchecking is not fully made, but the code is functioning. (KiX 4.20/4.21 RC2)

Sorry, I can't keep the Golf score under 100 lines [Wink]

code:
Break On
$RC = SetOption(Explicit,On)
Dim $File,$CsvArr

$File = @CurDir + '\csvtest.txt'
$CsvArr = CsvArray($File)
If @Error = 0
CsvShow($CsvArr) ; Display array on screen
EndIf

Return


; ********** Convert a CSV - file to 2D array **********
Function CsvArray($File)
Dim $Rec,$Row,$NumFields,$Handle,$i

$Handle = FreeFileHandle()
If $Handle
$RC = Open($Handle,$File)
Else
Exit 9 ; No free filehandles
EndIf

; Read first record, and determine number of fields
$Rec = ReadRec($Handle)
If @Error
Exit 1 ; File not found or empty
EndIf
$NumFields = UBound($Rec)
ReDim $CsvArray[$NumFields,0]
$Row = 0
For $i = 0 To $NumFields
$CsvArray[$i,0] = $Rec[$i]
Next

; Read the rest of records in the file
Do
$Rec = ReadRec(1) ; Read next record
If Not @Error
If UBound($Rec) <> $NumFields
Exit 2 ; Error in CSV-file, wrong number of fields in a record
EndIf
$Row = $Row + 1
ReDim Preserve $CsvArray[$NumFields,$Row]
For $i = 0 To $NumFields
$CsvArray[$i,$Row] = $Rec[$i]
Next
EndIf
Until @Error
$RC = Close($Handle)
EndFunction


; Supportfunction for CsvArray() - Read a record from the CSV-file
Function ReadRec($Handle)
Dim $Line,$Tmp,$Start,$End,$i

$Line = ReadLine($Handle)
If @Error = 0
$Line = Join(Split($Line,'""'),Chr(1))
If InStr($Line,'"')
While (UBound(Split($Line,'"')) & 1) And (@Error = 0)
$Tmp = ReadLine($Handle)
If @Error = 0
$Line = $Line + @CRLF + Join(Split($Tmp,'""'),Chr(1))
EndIf
Loop
EndIf
$Start = Left($Line,1) = '"'
$End = Right($Line,1) = '"'
$Tmp = Split($Line,'"')
$Line = ''
For $i = $Start To Ubound($Tmp) - $End
If $i & 1
$Tmp[$i] = '"' + Join(Split($Tmp[$i],','),Chr(2)) + '"'
EndIf
$Line = $Line + $Tmp[$i]
Next
$Tmp = Split($Line,',')
For $i = 0 To Ubound($Tmp)
$Tmp[$i] = Trim(Join(Split($Tmp[$i],Chr(1)),'""'))
$Tmp[$i] = Join(Split($Tmp[$i],Chr(2)),',')
Next
$ReadRec = $Tmp
Else
Exit 1
EndIf
EndFunction


; Testfunction for CsvArray() - Display the array
Function CsvShow($CsvRec)
Dim $i,$j

For $i = 0 To UBound($CsvRec,2)
For $j = 0 To UBound($CsvRec,1)
'Index: ' + $i + ',' $j ?
$CsvRec[$j,$i] ?
Next
?
Next
EndFunction

Testfile: csvtest.txt
quote:
John,Doe,120 jefferson st.,Riverside, NJ, 08075
"John ""Da Man""",Repici,"120,
121 Jefferson St.",Riverside, NJ,08075
Stephen,Tyler,"7452 Terrace ""At the Plaza"" road",SomeTown,SD, 91234
,Blankman,,SomeTown, SD, 00298
"Joan ""the bone"", Anne",Jet,"9th, at Terrace plc",Desert City,CO,00123

-Erik

{Edit}
Corrected missing DIM's

[ 08. May 2003, 01:34: Message edited by: kholm ]

Top
#100740 - 2003-05-07 11:52 PM Re: KixGolf Suggestion
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
this long code and devider only as ","?

think we can do better [Wink]
_________________________
!

download KiXnet

Top
#100741 - 2003-05-08 04:26 AM Re: KixGolf Suggestion
New Mexico Mark Offline
Hey THIS is FUN
****

Registered: 2002-01-03
Posts: 223
Loc: Columbia, SC
Thanks for the posted code. It gives me some ideas for some of the (non-tested so far) code I'm working on. One of my goals is to keep everything in one (preferable) or two functions for both reading and writing CSV format.

My approach has been a byte-by-byte read with a several boolean control variables to control whether we are at the end of a field or record, and to toggle quote mode on or off.

The writing (CSV data) part is easy. Though I haven't tested that, I think I can get it down to about fifteen lines code or less.

I'll post tomorrow after doing some testing and debugging.

NMM

[ 08. May 2003, 13:36: Message edited by: New Mexico Mark ]

Top
Page 1 of 1 1


Moderator:  Arend_, Allen, Jochen, Radimus, Glenn Barnas, ShaneEP, Ruud van Velsen, Mart 
Hop to:
Shout Box

Who's Online
0 registered and 2220 anonymous users online.
Newest Members
Viginette, ManuvdWielNL, Sir_Barrington, batdk82, StuTheCoder
17888 Registered Users

Generated in 0.086 seconds in which 0.039 seconds were spent on a total of 12 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org