Page 1 of 1 1
Topic Options
#66359 - 2002-06-08 05:57 AM Unicode Converter UDF
Howard Bullock Offline
KiX Supporter
*****

Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
I was attempting to write a UDF that would convert a (unicode file to an ascii file format) and an (ascii file format to a unicode file).

I have been successful in going from unicode to Ascii, but seem to have hit a limitation with KiXtart going from Ascii to unicode. KiXtart doesn't seem to handle Chr(0) at all when adding it to a string. Does anyone have any insight?

I know this isn't a true unicode converter but it works for single byte languages.

code:
UnicodeConvert("c:\data\scripts\unicode.sql")

Function UnicodeConvert($File)
Dim $FH, $x, $null, $rc

$FH = OpenFile($File, "R")
$x = ReadLine($FH)
if @error = 0
if instr($x, Chr(255)+Chr(254))
$Data = Right($x, 1)
While (@error=0)
$x = ReadLine(1)
if @error = 0
if Asc($x)<>0
$Data = $Data + $x
else
$null = $null + 1
if $null = 4
$Data = $Data + @CRLF
$null = 0
endif
endif
endif
Loop
else
$Data = Chr(255)+Chr(254)
Gosub "ProcessLine"
$x = ReadLine($FH)
While (@error=0)
Gosub "ProcessLine"
$x = ReadLine(1)
Loop
endif
else
exit 1
endif

$rc = close($FH)
Del $File
$FH = OpenFile($File, "W", 1, 1)
$rc = WriteLine($FH, $Data)
$rc = close($FH)
?
For $i=1 to len($Data)
Asc(substr($Data,$i,1)) " "
Next


exit 0

:ProcessLine
For Each $x IN Split($x,"")
$Data = $Data + $x + Chr(0)
Next
$Data = $Data + Chr(13) + Chr(0) + Chr(10) + Chr(0)

return

Endfunction



;FUNCTION OpenFile()
;
;AUTHOR Howard A. Bullock (hbullock@tycoelectronics.com)
;
;ACTION Opens file based on specified parameters. Supports overwriting a file.
;
;SYNTAX OpenFile(FileName, R|W, [0|1], [0|1])
;
;PARAMETERS $FileName (Required) - String value
; $ReadWrite (Required) - String value
; $Create (Optional) Default(0) Do not create a new file. (1 or 0)
; $OverWrite (Optional) - Default(0) Append (1 or 0)
;
;
;REMARKS This function attempts to open the specified file. @error is set on exit.
; The UDF will search for the first available file number.
;
;RETURNS File handle (integer 1-10) if successful; nothing if error
;
;DEPENDENCIES None
;
;EXAMPLES
; Open file for write; do not create the file; append
; $FH = OpenFile("junk.txt", "W")
; if @error = 0
; Writeline($FH, "Some text")
; endif
;
; Open file for write; create the file; delete old file
; $FH2 = OpenFile("junk2.txt", "W", 1, 1)
; if @error = 0
; Writeline($FH2, "Some text")
; endif;
;
Function OpenFile($FileName, $ReadWrite, optional $Create, optional $Overwrite)
Dim $FileName, $ReadWrite, $Create, $Overwrite
Dim $FH, $RC, $Mode

; Validate input parameters
if VarType($Create) = 0
$Create = 0
endif
if VarType($Overwrite) = 0
$Overwrite = 0
endif
if not (VarType($FileName) = 8 and
($ReadWrite = "R" or $ReadWrite = "W") and
($Create = 0 or $Create = 1) and
($Overwrite = 0 or $Overwrite = 1) )
exit 1
endif
if $Create < $OverWrite
exit 1
endif

; Build Mode value
if $Create = 0
$Mode = 0
else
$Mode = 1
endif
if $ReadWrite = "R"
$Mode = $Mode + 2
else
$Mode = $Mode + 4
endif

;Handle OverWrite option
if $OverWrite = 1
if exist ($Filename)
del $Filename
endif
endif

; Open file
$FH=1
$RC=Open ($FH, $FileName, $Mode)
? "@error @serror"
if $RC <> 0
select
case $RC=2
exit $RC
case $RC=-3
while $RC = -3 and $FH < 11
$RC=Open ($FH, $FileName, $Mode)
if $RC = -3
$FH=$FH +1
endif
Loop
if $RC = -3
exit $RC
endif
case $RC=-2
exit $RC
case $RC=-1
exit $RC
case 1
exit $RC
endselect
endif

$OpenFile = $FH
exit 0
Endfunction



[ 08 June 2002, 15:24: Message edited by: Howard Bullock ]
_________________________
Home page: http://www.kixhelp.com/hb/

Top
#66360 - 2002-06-08 03:09 PM Re: Unicode Converter UDF
Shawn Administrator Offline
Administrator
*****

Registered: 1999-08-13
Posts: 8611
Hey Howard, you think this is some kind of bug ? Can't seem to insert a CHR(0) into anything ...

eee-ahhh-eee-ahhh-eee-ahhh :

this is the "long line" police, pull over to the side of the road please Howard ...

[ 08 June 2002, 15:10: Message edited by: Shawn ]

Top
#66361 - 2002-06-08 03:18 PM Re: Unicode Converter UDF
Howard Bullock Offline
KiX Supporter
*****

Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
Yeah I did some other testing this morning and found that chr(0) doesn't seem to work at all. Bug in my book.

Sorry for the long lines. I was tired and forgot to change TABS to spaces.

{edit} OK officier, I have paid my fine and performed my community service by removing the long lines. [Razz]

[ 08 June 2002, 15:25: Message edited by: Howard Bullock ]
_________________________
Home page: http://www.kixhelp.com/hb/

Top
#66362 - 2002-06-08 03:47 PM Re: Unicode Converter UDF
Anonymous
Unregistered


The Ascii zero is in the manual under 'General Syntax Rules', I quote:
Strings can contain any characters, except the \0 (NULL) and \x1a (end of file) characters.

With FSO you can open a file as unicode for read or write.

I knocked up the following to read and write to/from an array
code:
$files = @ScriptDir+"\output.uni",@ScriptDir+"\output.txt"
$txt = "hello there","how are you","today?","I am "
WriteFile($files[0],$txt,"u")
AppendFile($files[0],"Unicode"+@CRLF+"so there")
WriteFile($files[1],$txt)
AppendFile($files[1],"Ascii"+@CRLF+"ending with EOL"+@CRLF)
;
FOR EACH $fn IN $files
$arr = ReadFile($fn)
? $fn +', Lines = '+Ubound($arr)
FOR $ajh = 0 to Ubound($arr)
? ""+$ajh+" : "+$arr[$ajh]
NEXT ?
NEXT
GETS $rc
EXIT
;
;========================================
; Creates a text file from $text (can be an array)
; If $type = "u" then a Unicode text file is created.
; New-line is NOT forced after text
; (to force New-Line $text[n] should be an empty element
; or if $text is a string it should end with @CRLF)
;
FUNCTION WriteFile($fn,$text,OPTIONAL $type)
IF Ubound($text) < 1 $text =Split($text,@CRLF) ENDIF
IF $type = "U" $type = -1 ELSE $type = 0 ENDIF
DIM $oFSO,$oFile,$i
$oFSO = CreateObject("Scripting.FileSystemObject")
IF @Error EXIT @Error ENDIF
$oFile = $oFSO.OpenTextFile($fn,2,-1,$type)
IF @Error EXIT @Error ENDIF
FOR $i = 0 TO UBound($text) -1
$oFile.WriteLine($text[$i])
NEXT
$oFile.Write($text[UBound($text)])
$oFile.Close
ENDFUNCTION
;
;========================================
; Appends $text (can be an array) onto an existing text file
; New-line is NOT forced before or after text
; (to force New-Line $text[n] should be an empty element
; or if $text is a string it should begin/end with @CRLF)
;
FUNCTION AppendFile($fn,$text)
IF Ubound($text) < 1 $text =Split($text,@CRLF) ENDIF
DIM $oFSO,$oFile,$type
$oFSO = CreateObject("Scripting.FileSystemObject")
IF @Error EXIT @Error ENDIF
$oFile = $oFSO.OpenTextFile($fn,1,0,0) ;Ascii
IF @Error EXIT @Error ENDIF
$type = $oFile.Read(2) $oFile.Close
IF $type = CHR(255)+CHR(254) $type = -1 ELSE $type = 0 ENDIF
$oFile = $oFSO.OpenTextFile($fn,8,0,$type)
FOR $i = 0 TO UBound($text) -1
$oFile.WriteLine($text[$i])
NEXT
$oFile.Write($text[UBound($text)])
$oFile.Close
ENDFUNCTION
;
;========================================
; Reads contents of file into an array.
; Ascii / Unicode is detected automatically.
;
FUNCTION ReadFile($fn)
DIM $oFSO,$oFile,$type
$oFSO = CreateObject("Scripting.FileSystemObject")
IF @Error EXIT @Error ENDIF
$oFile = $oFSO.OpenTextFile($fn,1,0,0) ;Ascii
IF @Error EXIT @Error ENDIF
$type = $oFile.Read(2) $oFile.Close
IF $type = CHR(255)+CHR(254) $type = -1 ELSE $type = 0 ENDIF
$oFile = $oFSO.OpenTextFile($fn,1,0,$type)
$ReadFile = Split($oFile.ReadAll,@CRLF)
$oFile.Close
? "Type "+$type
ENDFUNCTION


Top
#66363 - 2002-06-08 03:56 PM Re: Unicode Converter UDF
Howard Bullock Offline
KiX Supporter
*****

Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
Thanks for setting us straight. That's what happens when you're tired and assume things should work the way you want them to.

Should have read the doc again... I hope that Ruud permits Nulls in strings because we have to sometimes use null delimited string in registry values.

Wait a minute... The docs say null (\0) can be in a string yet when I read the unicode file the null chartacters around chr(13) and chr(10) were included in the string and processed properly. That is how I determined the end of line. [Confused] Why do some work?

[ 08 June 2002, 15:57: Message edited by: Howard Bullock ]
_________________________
Home page: http://www.kixhelp.com/hb/

Top
#66364 - 2002-06-10 10:39 AM Re: Unicode Converter UDF
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
I don't know why the DOS end of file character cannot be included, but the CHR(0) makes sense.

The problem is that because strings are variable length you need a character to determine where the string ends in memory, and CHR(0) is the one normally used.

If you want to handle CHR(0) then you have to use a "binary" storage method, commonly an array of characters with an additional variable for the data length.

This all adds significant complexity to the coding. You would need new ways of reading, writing, printing, substringing, catenating and comparing "binary strings" which would require length parameters to determine how much of your storage object to consider.

So I don't think we'll see CHR(0) in general strings.

AJH's use of arrays and objects which handle binary data is the way to go.

Top
#66365 - 2002-06-10 03:53 PM Re: Unicode Converter UDF
BrianTX Offline
Korg Regular

Registered: 2002-04-01
Posts: 895
There is a utility called uconvert.exe included with Microsoft's Platform SDK.

It converts any code page to unicode...

As a side note.. would it be possible to call an object/method in internet explorer to convert to unicode? just a thought..

Brian

Top
#66366 - 2002-06-10 04:13 PM Re: Unicode Converter UDF
Howard Bullock Offline
KiX Supporter
*****

Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
Yeah, I saw that. This was just a KiXtart exercise. [Wink]

I just wish (second one today) that all ASCII characters could be placed into strings like other languages.

[ 14 July 2002, 16:04: Message edited by: Howard Bullock ]
_________________________
Home page: http://www.kixhelp.com/hb/

Top
Page 1 of 1 1


Moderator:  Glenn Barnas, NTDOC, Arend_, Jochen, Radimus, Allen, ShaneEP, Ruud van Velsen, Mart 
Hop to:
Shout Box

Who's Online
0 registered and 447 anonymous users online.
Newest Members
Timothy, Jojo67, MaikSimon, kvn317, kixtarts2025
17874 Registered Users

Generated in 0.036 seconds in which 0.013 seconds were spent on a total of 13 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org