Page 1 of 1 1
Topic Options
#191058 - 2008-12-12 03:08 PM Read a file and randomly pick entries
Willmaykett Offline
Fresh Scripter

Registered: 2007-01-07
Posts: 21
Hi all,

I hope some talented person can help me.

I have an application the generates a text file with a series of entries in it and I need to generate a file that randomly picks x amount of entries (where x is a number I need). For example

Widget1
widget2
Widget5
Widget9
Widget10

Now, the number of entries, ie widgets, can vary, from zero to 20 or more in it.

What I need to do is be able to RANDOMLY pick a predetermined number of widget entries. So that if I say pick 2 entries, I might get Widget5 and 9, then next time it might be 1 and 10.

I think I need to:
1. Read the file and create an array.
2. Work out how many entries are in the array.
3. Pick a predetermined number that I want.
4. Randomly pick some of the entries based on the number I want.
5. Then generate a new file with those random entries.

Make sense.

I need some serious help in making some code, and would seriously appreciate the help!

Thanks,
Wil

Top
#191063 - 2008-12-12 03:56 PM Re: Read a file and randomly pick entries [Re: Willmaykett]
eriqjaffe Offline
Hey THIS is FUN

Registered: 2004-06-24
Posts: 214
Loc: Arlington Heights, IL USA
quick & dirty:

 Code:
break on

global $RandomCallCounter

$Array = RandomizeArray(ReadFile("\\path\to\infile.txt"))

$arraysize = ubound($array) + 1

? "The array contains "+$arraysize+" elements.  How many do you want to pick? "
gets $howmany

$ = open(1,"\\path\to\newfile.txt",5)

for $x = 1 to ($howmany)
    $ = WRITELINE(1,$Array[$x]+@CRLF)
next

$ = close(1)


This relies on the ReadFile() and RandomizeArray() UDFs, both of which can be found in the UDF forum. May not be the neatest way, but it'll get the job done.


Edited by eriqjaffe (2008-12-12 03:57 PM)

Top
#191064 - 2008-12-12 03:57 PM Re: Read a file and randomly pick entries [Re: Willmaykett]
Gargoyle Offline
MM club member
*****

Registered: 2004-03-09
Posts: 1597
Loc: Valley of the Sun (Arizona, US...
 Code:
srnd(@msec)
$Pick = 4 ; how many do you want
$Counter = 0
Open(1,"c:\file.txt)
$Line = Readline(1)
While @error = 0
  redim preserve $Array[$Counter]
  $Array[$Counter] = $Line
  $Counter = $Counter + 1
  $Line = Readline(1)
Loop

$Size = Ubound($Array)

For $Output = 0 to $Pick
 ? $Array[Rnd[$size]
Next
_________________________
Today is the tomorrow you worried about yesterday.

Top
#191076 - 2008-12-12 11:48 PM Re: Read a file and randomly pick entries [Re: Gargoyle]
Willmaykett Offline
Fresh Scripter

Registered: 2007-01-07
Posts: 21
Hi guys,

Wow, thanks for the tips. Would never have been able to do what I wanted in such a small amount of code.

Thanks guys, really appreciate it !

I ran the code from eriqjaffe and it worked a treat. What extra would I need to do so that if 3 entries existed in the file, and I picked 5, that all 3 entries will be used? Currently it throws an error about being out of bounds.

The code from Gargoyle threw an error all the time, and I think I worked out it was some syntax errors. But then it generated only a series of numbers, not a list of what was in the file. Further than that, I got stumped .

Thanks heaps for your work so far!


Edited by Willmaykett (2008-12-13 01:00 AM)

Top
#191085 - 2008-12-13 04:02 AM Re: Read a file and randomly pick entries [Re: Willmaykett]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4402
Loc: New Jersey
There's a small syntax error in Gargoyle's code - change
? $Array[Rnd[$size]
to
$Array[Rnd($size)] ?

The first "[" should actually be a "(", and the closing ")" was missing.

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#191086 - 2008-12-13 04:21 AM Re: Read a file and randomly pick entries [Re: Glenn Barnas]
Gargoyle Offline
MM club member
*****

Registered: 2004-03-09
Posts: 1597
Loc: Valley of the Sun (Arizona, US...
Typing way to fast Oh well nice catch.
_________________________
Today is the tomorrow you worried about yesterday.

Top
#191109 - 2008-12-13 08:59 AM Re: Read a file and randomly pick entries [Re: Gargoyle]
Willmaykett Offline
Fresh Scripter

Registered: 2007-01-07
Posts: 21
Hi guys,

Interestingly, it still didn't work. I had to change all the [] to () then it worked, BUT, it still isn't giving me what I want.

When I create a file with the following entries:
Line1
Line2
Line3
Line4
Line5

and I run the code

 Code:
srnd(@msec)
$Pick = 4 ; how many do you want
$Counter = 0
Open(1,"c:\data\file.txt")
$Line = Readline(1)
While @error = 0
  redim preserve $Array[$Counter]
  $Array[$Counter] = $Line
  ? $Line 
  $Counter = $Counter + 1
  $Line = Readline(1)
Loop

$Size = Ubound($Array)

For $Output = 0 to $Pick
 $Array(RND($size)) ?
Next
Sleep 3


the first part of the array gives me the list of all the lines, but there is a 0 in it, and the random part gives me 5 random numbers, not 4 as expected, and not the actual data

Is something a bit amiss?

Thanks

Top
#191112 - 2008-12-13 09:11 AM Re: Read a file and randomly pick entries [Re: Willmaykett]
Gargoyle Offline
MM club member
*****

Registered: 2004-03-09
Posts: 1597
Loc: Valley of the Sun (Arizona, US...
You must remember that Array's are 0 based so...

 Code:
0  1  2  3  4  5

Is actually 6 entries

The For $OUTPUT takes this into account and starts at 0
You will have to adjust the $Pick to be 1 less than you actually want

Change the Output loop to the following

 Code:
For $OUTPUT = 0 to $Pick
 $Selection = RND($size)
 $RandomLine = $Array[$Selection] 
 ?"The Random Number Generated was " + $Selection 
 ?"The Line Associated with the number is " + $RandomLine
Next


Edited by Gargoyle (2008-12-13 09:12 AM)
Edit Reason: forgot code tags
_________________________
Today is the tomorrow you worried about yesterday.

Top
#191117 - 2008-12-13 01:36 PM Re: Read a file and randomly pick entries [Re: Gargoyle]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4402
Loc: New Jersey
This one works - the corrections and enhancements are commented. Note that there is nothing preventing the same data from being displayed more than once - if duplicate selections is a problem, you can modify the code slightly to select an array element pointer. If the element contains data, you output it and clear the element. If the element is empty, it's been used, so reduce the counter by one instead of outputting the value.
 Code:
break on

Dim $Array[0]				; Must initialize the array first!

srnd(@msec)
$Pick = 4 ; how many do you want

$Counter = -1				; alternate, pre-increment method uses -1 init

If Open(1,"c:\data\file.txt") = 0	; trapped the result code
  $Line = Readline(1)
  While @error = 0
    $Counter = $Counter + 1		; moved to pre-increment
    redim preserve $Array[$Counter]
    $Array[$Counter] = $Line
    $Line ?
    $Line = Readline(1)
  Loop
  $ = Close(1)				; closed the file
EndIf

$Size = Ubound($Array)
$Size + 1 ' elements were loaded' ?	; show element count

For $Output = 1 to $Pick		; counter, not array index, so 1-based
  $Array[RND($size)] ?			; corrected d"(" and "[" 
Next

If you simply add the element count to the original code, you always get 0 (-1 ubound). That's because the array was not declared before being ReDim'd. This resulted in no data in the array.

I modified the loop to a pre-increment model (personal preference). Both work, but in pre-increment, the array index pointer always accurately reflects the array count, which could eliminate the need for the UBound in certain code applications.

I added an If to the Open to trap the result code, preventing the extraneous zero from appearing on the console, as well as added the Close call.

The Element Count message pointed to the original problem of an uninitialized array.

Finally, the parens and square braces were messed up again. I also changed the For loop to be 1-based. This is why you were getting an extra item. The $Output value is a counter of the number of selections, so should be 1-based. The Rnd function returns a value between 0 and the max value (array size) and is the array element index, which is properly zero-based.

Here's an alternate loop to prevent duplicate selections:
 Code:
For $Output = 1 to $Pick
  $Index = RND($size)            ; select a random element
  If $Array[$Index]              ; does it contain data?
    $Array[$Index] ?             ; then display it
    $Array[$Index] = ''          ; and clear it to prevent re-use
  Else
    $Output = $Output - 1        ; reduce the output count so we try again
  EndIf
Next

This assumes you don't need the array data. If you need to maintain the array data, you can duplicate the array and use the duplicate for controlling the output.

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#191180 - 2008-12-15 07:43 AM Re: Read a file and randomly pick entries [Re: Glenn Barnas]
Willmaykett Offline
Fresh Scripter

Registered: 2007-01-07
Posts: 21
Thanks Glenn, really appreciate the help so far!

What I intend doing is running this script as a scheduled task once a week to randonly pull out the required number of entries from my file. What sometimes may happen is that the extract may contain only 4 entries, but my "picker" will normally be set for 5.

I am going to use the "alternate loop" code above, and make the following additions:
 Code:
$Size = Ubound($Array)
$Size + 1 ' elements were loaded' ?	; show element count
$SizeTotal = $Size + 1
$SizeTotal ?

If $SizeTotal =< $Pick
	"Entries in file less than or equal to $PICK" ?
	Sleep 3
	Exit
EndIf


What I also noticed is that the entries picked don't appear to be that "random"

for example,
My file with:
Widget1
Wigdet2
Widget3
Widget4
Widget5

and pick of 2 always gives me
Widget1
Widget2

Thought it would be a bit more random than that...

Thanks heaps for all you help so far. \:\)


Edited by Willmaykett (2008-12-15 09:29 AM)
Edit Reason: Made changes to code

Top
#191182 - 2008-12-15 10:32 AM Re: Read a file and randomly pick entries [Re: Willmaykett]
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
 Quote:
What I also noticed is that the entries picked don't appear to be that "random"


Are you seeding the random number using SRND()?

The RNG generates a pseudo-random sequence. If you don't seed it you will always get exactly the same sequence.

It is also good practice to discard the first random number. There is an old feature which caused the very first random number to be predictable even with different seeds.

Something like:
 Code:
$sDiscard=SRND(@MSECS) ; Pick a seed for the RNG.  @MSECS is OK for most applications.
$sDiscard=RND() ; Discard first random number as some versions return a predictable first result.

Top
#191184 - 2008-12-15 12:11 PM Re: Read a file and randomly pick entries [Re: Richard H.]
Willmaykett Offline
Fresh Scripter

Registered: 2007-01-07
Posts: 21
This is my (mostly Glenn's actually) code so far.

 Code:
break on

Dim $Array[0]				; Must initialize the array first!
SRND(@msec)
$Pick = 2 ; how many do you want

$Counter = -1				; alternate, pre-increment method uses -1 init

If Open(1,"c:\data\file.txt") = 0	; trapped the result code
  $Line = Readline(1)
  While @error = 0
    $Counter = $Counter + 1		; moved to pre-increment
    redim preserve $Array[$Counter]
    $Array[$Counter] = $Line
    $Line = ReadLine(1)
  Loop
  $ = Close(1)				; closed the file
EndIf

$Size = Ubound($Array)
$Size + 1 ' elements were loaded' ?	; show element count
$SizeTotal = $Size + 1
$SizeTotal ?

If $SizeTotal =< $Pick
	"Entries in file less than or equal to $PICK" ?
	Sleep 3
	Exit
EndIf

For $Output = 1 to $Pick
  $Index = RND($size)            ; select a random element
  If $Array[$Index]              ; does it contain data?
    $Array[$Index] ?             ; then display it
    $Array[$Index] = ''          ; and clear it to prevent re-use
  Else
    $Output = $Output - 1        ; reduce the output count so we try again
  EndIf
Next
Sleep 3


So I am not sure where I am supposed to put what you suggest.
With my test file, it always returns:
Widget1
Widget2

and never a "random" selection with any of the other text in the file?

Thanks

Top
#191188 - 2008-12-15 03:17 PM Re: Read a file and randomly pick entries [Re: Willmaykett]
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Ok, here is the way I would do it:
 Code:
Break ON

$fso = CreateObject("Scripting.FileSystemObject")
$txt = $fso.OpenTextFile("C:\data\file.txt",Not 0)
$Array=Split($txt.ReadAll,@CRLF)
$txt.Close

$=SRND(@MSECS)
$=RND(0)

$Pick=2

For $Output=1 to $Pick
	$Array=Split(udfSqueeze(Join($Array,@CRLF),@CRLF),@CRLF)
	$iChose=IIf(UBound($Array)>0,RND(UBound($Array)),0)
	"Selection "+$Output+": "+$Array[$iChose]+@CRLF
	$Array[$iChose]=""
Next

Function udfSqueeze($s,$q)
	$udfSqueeze=$q+$s+$q
	While InStr($udfSqueeze,$q+$q)
		$udfSqueeze=Join(Split($udfSqueeze,$q+$q),$q)
	Loop
	$udfSqueeze=Split($q+$udfSqueeze+$q,$q+$q)[1]
EndFunction


The method strips array elements out once they have been selected, meaning that you won't hit a blank element and you won't need to retry.

Retrying is not a huge problem when your select set is a small proportion of the entire set, but as the proportion grows you will get more and more hits on an empty array element.

Of course you need to balance this against the work needed by all the split/joins.

In this example if you have 4 lines in the file and you attempt to pick 10 lines then the script will not fail, it will report null values for picks 5-10.


Edited by Richard H. (2008-12-15 03:18 PM)
Edit Reason: Removed debugging code

Top
#191192 - 2008-12-15 08:55 PM Re: Read a file and randomly pick entries [Re: Richard H.]
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11631
Loc: CA
Also of note which Richard and others pointed out to me a while back. Even though they may not seem random at times they are. Just because there are duplicates doesn't make it not random. Using a very large number set might help to reduce duplicates from being seen as often, but even a large number set can produce duplicates and as Glenn discussed you would need to code appropriately to remove duplicates if not wanted.

Richard wrote an excellent Random Password Generator in KiX that I really had high hopes of creating in a KiXforms script but it turned out to be too much coding for me to do on my own and the developer of KiXforms (Shawn Tassie) didn't have the time to assist me in finishing it up so I never completed it (Sorry Richard, great script though on your part).

Top
#191194 - 2008-12-16 04:15 AM Re: Read a file and randomly pick entries [Re: NTDOC]
Willmaykett Offline
Fresh Scripter

Registered: 2007-01-07
Posts: 21
Hi guys,

Well, it looks like I got something working, using Richard's code and I added in the writeline stuff to generate the required file. I have tested it and it works a treat!

Thanks to all you guys who have helped me out so much. I will try an sit down and understand what it really is doing....

For a total newbie like me, with NO programming experience, any recommendations on resources for "scripting for dummies" like me..that starts from the beginning??

Once again, Thanks heaps you guys.

Top
#191195 - 2008-12-16 05:26 AM Re: Read a file and randomly pick entries [Re: Willmaykett]
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11165
Loc: Boston, MA, USA
I'm sure that Jack lothian can give you a better run-down on random numbers. However, the rND() function si sufficiently random when using a large sample size. You will alwasy have local patterns that look repeatable or a lot if identical patterns if you generate random numbers over only a small range.
_________________________
There are two types of vessels, submarines and targets.

Top
#191198 - 2008-12-16 10:37 AM Re: Read a file and randomly pick entries [Re: Willmaykett]
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
 Originally Posted By: Willmaykett
For a total newbie like me, with NO programming experience, any recommendations on resources for "scripting for dummies" like me..that starts from the beginning?


Well, for KiXtart specific material you can take a look at our other forums - there is a wealth of information and places like the FAQ and KiXtart Learning forums may be a good place to look around.

For more formal material there a KiXtart book has been published: http://www.amazon.com/Start-Finish-Guide-Scripting-KiXtart/dp/B0000W86HC

You are always welcome to post questions about KiXtart in the Basic Scripting forum, even if the questions don't relate to a specific script.

If you have a more general scripting question or a question about another scripting language then you may post a question to the Lounge forum.

Top
Page 1 of 1 1


Moderator:  Jochen, Allen, Radimus, Glenn Barnas, ShaneEP, Ruud van Velsen, Arend_, Mart 
Hop to:
Shout Box

Who's Online
0 registered and 837 anonymous users online.
Newest Members
ManuvdWielNL, Sir_Barrington, batdk82, StuTheCoder, M_Moore
17887 Registered Users

Generated in 0.072 seconds in which 0.026 seconds were spent on a total of 13 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org