Page 3 of 4 <1234>
Topic Options
#106271 - 2003-10-30 02:05 AM Re: POLL: String Expressions
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11623
Loc: CA
So many different scripts. Well if someone will post (and agree) on a valid script to really test this, then I'll run it on one of our 6BG RAM Quad 2.0GHz Xeon systems and see what I get.

May take a couple days to get to this though.

Top
#106272 - 2003-10-30 04:43 AM Re: POLL: String Expressions
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11164
Loc: Boston, MA, USA
Interesting, isn't it?

I'm currently running a test script on one of our work computers and it's at
code:
String length = 133,300,000 = 130,175 kB = 127 MB

while adding 100,000 characters to the string on each iteration.

Personally, I think that the limit might be the same number as the number of elements in an array (according tot he KiXtart v4.22 RC1 Manual)

I was actually interested to see if anyone can fidn the solution before asking Ruud ;-)

And based on the poll, only two people have voted for the number I suspect is the winning one. Most went for the incorrectly documented 32,000 character limit. [Big Grin]
_________________________
There are two types of vessels, submarines and targets.

Top
#106273 - 2003-10-30 06:14 AM Re: POLL: String Expressions
Jack Lothian Offline
MM club member
*****

Registered: 1999-10-22
Posts: 1169
Loc: Ottawa,Ontario, Canada
The last script I posted was looping through using base 4. It succeeded in creating a string of 67,108,864 characters but it died trying to add 201,326,592 characters to the string. Unfortunately it did not die nicely. It seemed to just bring the OS to a halt. If it died nicely, I figured I would alter my script so it iteratively converged on the appropriate number by lowering the base & resetting the starting point to the last successful size. It would have been reasonably fast too but it required the loop to end with some kind of error condition.

My machine at home only has 256M but I have it set so the virtual memory is 1GB. When the system hung most of this virtual memory was being used.
_________________________
Jack

Top
#106274 - 2003-10-30 06:28 AM Re: POLL: String Expressions
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11623
Loc: CA
Thanks Jack,

Still waiting for a VALID script to use for testing on Server. [Wink]

Top
#106275 - 2003-10-30 05:08 PM Re: POLL: String Expressions
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11164
Loc: Boston, MA, USA
NTDOC: Here's my test script

code:
cls
break on

$rc=setoption('NoVarsInStrings','on')
$rc=setoption('Explicit','on')

dim $StartStringLength, $FinalStringLength, $StringSegmentLength, $StringSegmentsToAdd
dim $SegmentCount, $CurrentStringLength, $CurrentSize, $StringSegment

$StartStringLength=0
$FinalStringLength=25000000000
$StringSegmentLength=10000000

$StringSegmentsToAdd=cdbl($FinalStringLength)/$StringSegmentLength

dim $StringSegmentPart[$StringSegmentLength/1000], $String[cdbl($StartStringLength)/$StringSegmentLength]
$StringSegmentPart=join($StringSegmentPart,'abcdefghij')
$StringSegment=''
for $SegmentCount=1 to 100
$StringSegment=$StringSegment+$StringSegmentPart
next
$SegmentCount = 0

if ubound($StartStringLength)
$String=join($String,'abcdefghij')
endif
;$rc=redirectoutput(@SCRIPTDIR+'\stringtest.txt')

? 'String Length Start = ' + formatnumber(len($String),0,-1)
? 'String Length Final = ' + formatnumber($FinalStringLength,0,-1)
? 'String Segment Length = ' + formatnumber(len($StringSegment),0,-1)
? '# Segments To Add = ' + formatnumber($StringSegmentsToAdd,0,-1)

while $SegmentCount<$StringSegmentsToAdd and not @ERROR
$String=$String+$StringSegment
$SegmentCount=$SegmentCount+1
$CurrentStringLength=cdbl(len($String))
$CurrentSize=val($CurrentStringLength/1024/1024)
? 'Current String Length ('+formatnumber($SegmentCount,0,-1)+') = ' + formatnumber($CurrentStringLength,0,-1) + ' = ' +formatnumber($CurrentSize,0,-1) +' MB'
loop
if not @ERROR
? 'Final String Length ('+formatnumber($SegmentCount,0,-1)+') = ' + formatnumber($CurrentStringLength,0,-1) + ' = ' +formatnumber($CurrentSize,0,-1) +' MB'
else
? 'Error at step '+$SegmentCount+' = ' + formatnumber(len($String),0,-1)+ ' = '+formatnumber(cdbl(len($String))/1024/1024,0,-1)+' MB'
endif

exit 0



[ 30. October 2003, 17:09: Message edited by: sealeopard ]
_________________________
There are two types of vessels, submarines and targets.

Top
#106276 - 2003-10-30 09:50 PM Re: POLL: String Expressions
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
k, I will start testing too.
running on Pentium pro (200MHz) but with like 5,2G of mem.
_________________________
!

download KiXnet

Top
#106277 - 2003-10-30 09:57 PM Re: POLL: String Expressions
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
pat, how come your messenger is logged off? [Wink]
_________________________
!

download KiXnet

Top
#106278 - 2003-10-31 07:22 AM Re: POLL: String Expressions
MightyR1 Offline
MM club member
*****

Registered: 1999-09-09
Posts: 1264
Loc: The Netherlands
Had to go do a little fitness [Wink] Therefore pc not turned on

BTW I now get KiX errors telling me I'm 'Out of memory'...

Jens, your script gave me:

String Length Start = 0
String Length Final = -769,803,776
String Segment Length = 10,000,000
# Segments To Add = -77
Final String Length (0) = 0 = 0 MB

Huh???

[ 31. October 2003, 07:27: Message edited by: MightyR1 ]
_________________________
Greetz,
Patrick Rutten

- We'll either find a way or make one...
- Knowledge is power; knowing how to find it is more powerful...
- Problems don't exist; they are challenges...

Top
#106279 - 2003-10-31 11:41 AM Re: POLL: String Expressions
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
pretty effective mem usage with nul-string!
I got 35M mem usage with 21.000.000 long string...
_________________________
!

download KiXnet

Top
#106280 - 2003-11-01 12:42 AM Re: POLL: String Expressions
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
btw, noticed now how much I hate wintosh.

anyway, going from physical mem to virtual (with w2k and 64MB it's pretty quickly) slows things down quite a lot.
CPU is at 0-5% all the time and most of the time even HD's are quite silent (scsi-drives).

started crawling at 20M and now the speed... well, there is none.

not that HD is slow but windows makes the virtual management so slow that everything is sloow.
if someone has a machine with 6-7G of ram, please shoot in.
disable virtual mem and kick in with FAST results.
_________________________
!

download KiXnet

Top
#106281 - 2003-11-01 12:53 AM Re: POLL: String Expressions
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Just another little wrinkle to add in as this is a mostly academic exercise.

When catenating strings you will be creating copies (expressions on internal stacks, temporary copies for assigning).

This means that unless you are very careful the limitation (memory/machine stability) will not be the string size, but the total size of the string(s) in the expression and the temporary copies.

It also means that KiXtart may be able to handle different string sizes, depending on what you are doing with them.

Top
#106282 - 2003-10-31 02:51 PM Re: POLL: String Expressions
MightyR1 Offline
MM club member
*****

Registered: 1999-09-09
Posts: 1264
Loc: The Netherlands
268.499.168 and counting...
_________________________
Greetz,
Patrick Rutten

- We'll either find a way or make one...
- Knowledge is power; knowing how to find it is more powerful...
- Problems don't exist; they are challenges...

Top
#106283 - 2003-10-31 03:24 PM Re: POLL: String Expressions
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11164
Loc: Boston, MA, USA
310,000,000 and counting ;-)

Personally, I think the max string length is 2,147,483,647.
_________________________
There are two types of vessels, submarines and targets.

Top
#106284 - 2003-10-31 03:28 PM Re: POLL: String Expressions
MightyR1 Offline
MM club member
*****

Registered: 1999-09-09
Posts: 1264
Loc: The Netherlands
Think not...

adding 268435456 char string with 65536 char string gave "out of memory"...
_________________________
Greetz,
Patrick Rutten

- We'll either find a way or make one...
- Knowledge is power; knowing how to find it is more powerful...
- Problems don't exist; they are challenges...

Top
#106285 - 2003-10-31 03:34 PM Re: POLL: String Expressions
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
jens, making the poll now would make much more sofisticated results as most of us would know what you really asked...

as a BTW...

ps, pat, you still offline...
_________________________
!

download KiXnet

Top
#106286 - 2003-11-04 01:29 AM Re: POLL: String Expressions
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11623
Loc: CA
Think I found the limit (at least according to Jens script).

Script ran on
HP/Compaq DL580-G3
Quad Xeon 2.0GHz CPU w/2MB L3 Cache
Windows 2003 Enterprise Server
6GB RAM


The script starts to rebuild at 448MB

quote:
Current String Length (87)=390,000,000=371 MB
Current String Length (88)=400,000,000=381 MB
Current String Length (89)=410,000,000=391 MB
Current String Length (90)=420,000,000=400 MB
Current String Length (91)=430,000,000=410 MB
Current String Length (92)=440,000,000=419 MB
Current String Length (93)=450,000,000=429 MB
Current String Length (94)=460,000,000=438 MB
Current String Length (95)=470,000,000=448 MB
Current String Length (96)=0=0 MB
Current String Length (97)=10,000,000=9 MB
Current String Length (98)=20,000,000=19 MB
Current String Length (99)=30,000,000=28 MB
Current String Length (100)=40,000,000=38 MB
Current String Length (101)=50,000,000=47 MB
Current String Length (102)=60,000,000=57 MB
Current String Length (103)=70,000,000=66 MB
Current String Length (104)=80,000,000=76 MB
Current String Length (105)=90,000,000=85 MB
Current String Length (106)=100,000,000=95 MB
Current String Length (107)=110,000,000=104 MB
Current String Length (108)=120,000,000=114 MB
Current String Length (109)=130,000,000=123 MB
Current String Length (110)=140,000,000=133 MB
Current String Length (111)=150,000,000=143 MB
Current String Length (112)=160,000,000=152 MB
Current String Length (113)=170,000,000=162 MB
Current String Length (114)=180,000,000=171 MB
Current String Length (115)=190,000,000=181 MB
Current String Length (116)=200,000,000=190 MB
Current String Length (117)=210,000,000=200 MB
Current String Length (118)=220,000,000=209 MB
Current String Length (119)=230,000,000=219 MB
Current String Length (120)=240,000,000=228 MB
Current String Length (121)=250,000,000=238 MB


 -

[ 04. November 2003, 01:32: Message edited by: NTDOC ]

Top
#106287 - 2003-11-04 03:02 AM Re: POLL: String Expressions
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11164
Loc: Boston, MA, USA
Now, that would be an unexpected value. [Confused]

Still waiting for my script to finish on a slow P4-3.06 with 1GB RAM. [Frown]
_________________________
There are two types of vessels, submarines and targets.

Top
#106288 - 2003-11-04 03:34 AM Re: POLL: String Expressions
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
with this task, the processor speed indeed is not a matter.
only the physical ram amount does a difference.
_________________________
!

download KiXnet

Top
#106289 - 2003-11-04 04:06 PM Re: POLL: String Expressions
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11164
Loc: Boston, MA, USA
Not really. There's a lot of memory duplication going on, thus the RAM speed is also a factor as well as the HD speed in case the swapfile needs ot be used.

Anyway, got it up to 450MB and will not try smaller increments at a baseline of 450MB.

[ 04. November 2003, 16:07: Message edited by: sealeopard ]
_________________________
There are two types of vessels, submarines and targets.

Top
#106290 - 2003-11-04 05:48 PM Re: POLL: String Expressions
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4396
Loc: New Jersey
Interesting..

Here's a quick test - the string length doubles each iteration. It was run on a server with 2G of RAM. Obviously it's more than 32767 bytes! My script ran out of RAM (and 100% CPU cycles!) when going from 512Meg to 1Gig string length. I suppose I could trap this and grow the string incrementally after 512Meg, but hey - it's a scripting language! Why would I need more than 512Meg long strings?? [Big Grin]
(actually, I started increasing the string length by 32K right after 512M, and only got one segment of 32K beyond the 512M point..)

Glenn

code:
 
String Length: 1
String Length: 2
String Length: 4
String Length: 8
String Length: 16
String Length: 32
String Length: 64
String Length: 128
String Length: 256
String Length: 512
String Length: 1024
String Length: 2048
String Length: 4096
String Length: 8192
String Length: 16384
String Length: 32768 <= exceeds spec in manual here
String Length: 65536
String Length: 131072
String Length: 262144
String Length: 524288
String Length: 1048576
String Length: 2097152
String Length: 4194304
String Length: 8388608
String Length: 16777216
String Length: 33554432
String Length: 67108864
String Length: 134217728
String Length: 268435456
String Length: 536870912

ERROR : out of memory!
Script: C:\temp\s.kix
Line : 11

It took about 2:36 to get to this point

Here's the script:
code:
 
Break On

Dim $L, $S
$L = 0.0
$S = 'X'

$L = Len($S)
'String Length: $L' ?

While @ERROR = 0
$S = $S + $S
$L = Len($S)
'String Length: $L' ?
Loop

_________________________
Actually I am a Rocket Scientist! \:D

Top
Page 3 of 4 <1234>


Moderator:  Arend_, Allen, Jochen, Radimus, Glenn Barnas, ShaneEP, Ruud van Velsen, Mart 
Hop to:
Shout Box

Who's Online
0 registered and 262 anonymous users online.
Newest Members
gespanntleuchten, DaveatAdvanced, Paulo_Alves, UsTaaa, xxJJxx
17864 Registered Users

Generated in 0.081 seconds in which 0.026 seconds were spent on a total of 13 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org