Page 1 of 1 1
Topic Options
#143338 - 2005-07-11 01:06 PM Pre-Tokenized Scripts - Performance Impact?
Skatterbrainz Offline
Starting to like KiXtart

Registered: 2002-10-17
Posts: 172
Loc: Virginia, USA
Is there as known impact on performance when pre-tokenizing scripts? In particular, using an example of having one kixtart.kix script that would use "call" on five "child" scripts, but now changing them to "Include" and then pre-tokenizing the parent (rolling up all the included scripts). Does this particular situation present a performance impact of any noticable kind? I'm seeing some test scripts like this that seem to run slower now using this example scenario. I'm wondering if that's normal or do I need to rework my code.
_________________________
silence is golden, but duct tape is silver

Top
#143339 - 2005-07-11 01:31 PM Re: Pre-Tokenized Scripts - Performance Impact?
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
well, quess the problem is that there is no example scripts with which to try.
the example script should not be affectable by other slowdown factors like network lag.
_________________________
!

download KiXnet

Top
#143340 - 2005-07-11 02:37 PM Re: Pre-Tokenized Scripts - Performance Impact?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Code:
Does this particular situation present a performance impact of any noticable kind?



The opposite should be the case, unless your INCLUDEed scripts are truly massive and are causing local code branches to become very slow.

Is it possible that you have introduced a subtle variable scope issue by using INCLUDE?

Top
#143341 - 2005-07-11 04:20 PM Re: Pre-Tokenized Scripts - Performance Impact?
Skatterbrainz Offline
Starting to like KiXtart

Registered: 2002-10-17
Posts: 172
Loc: Virginia, USA
That's possible. From what I'm "hearing" I am guessing it's most likely due to coding issues than pre-tokenizing. I wasn't sure if when the tokenized script is executed whether there was a processing overhead to decrypt the tokenized code first.
_________________________
silence is golden, but duct tape is silver

Top
#143342 - 2005-07-11 04:35 PM Re: Pre-Tokenized Scripts - Performance Impact?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Quote:

Is it possible that you have introduced a subtle variable scope issue by using INCLUDE?




Ignore that comment - INCLUDED files are imported as a type of UDF, so scope shouldn't be an issue.

Top
#143343 - 2005-07-11 04:59 PM Re: Pre-Tokenized Scripts - Performance Impact?
Les Offline
KiX Master
*****

Registered: 2001-06-11
Posts: 12734
Loc: fortfrances.on.ca
Huh?
Included files are as if the files were concatenated into one large file before tokenization so the result would be as if it were one large script. It is entirely possible for there to be var scope issues.

CALL and INCLUDE cannot be interchanged with impunity.
_________________________
Give a man a fish and he will be back for more. Slap him with a fish and he will go away forever.

Top
#143344 - 2005-07-11 05:30 PM Re: Pre-Tokenized Scripts - Performance Impact?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Quote:

Huh?
Included files are as if the files were concatenated into one large file before tokenization so the result would be as if it were one large script. It is entirely possible for there to be var scope issues.

CALL and INCLUDE cannot be interchanged with impunity.




That's what I assumed when I first posted, but then I did some tests just to make sure that I wasn't talking bollocks.

Turns out I was talking bollocks.

Much to my surprise, INCLUDE does *not* just include the file as if it was in-line code when the main script is tokenized. Try INCLUDEing the same file twice with "Explicit" on - you will get a very revealing error message when you try to run the tokenized script

If you define (DIM) a local variable within an INCLUDED file you will see that it is not available to the script that INCLUDEd the subscript, so the scope is restricted tothe subscript as it would be in a CALL.

Strangely though, an EXIT in an INCLUDEd script still causes the main script to exit

Top
#143345 - 2005-07-11 06:46 PM Re: Pre-Tokenized Scripts - Performance Impact?
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11634
Loc: CA
And what a "can of worms" Ruud has seemed to open with INCLUDE and TOKENIZATION

I agree with you Richard in that there are some small differences going on under the covers so to speak. I had muliple larger scripts that worked well with calls but was a royal pain trying to clear up the scope when using INCLUDE

However, to remain on topic. The Tokenized script should run faster unless there is some type of scope issue (normally might even error for you) or odd data fragmentation on your network maybe that is allowing this to surface in which case you'd need to look further into your packets and see what's going on at a router/switch level.
 

Perhaps Ruud can layout in more technical terms all of the details of what/how things are really happening with the INCLUDE - the details are way too generic in the manual.
 

Top
#143346 - 2005-07-11 07:14 PM Re: Pre-Tokenized Scripts - Performance Impact?
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
eh, nothing you said here makes me believe what you suggest.
quess I need to try it out myself too.
_________________________
!

download KiXnet

Top
#143347 - 2005-07-11 07:45 PM Re: Pre-Tokenized Scripts - Performance Impact?
Skatterbrainz Offline
Starting to like KiXtart

Registered: 2002-10-17
Posts: 172
Loc: Virginia, USA
What I had "before" was a kixtart.kix file, a UDF file with all my global functions, and five separate .kix files that were executed upon "Call" statements. I modified them to wrap each inside a Function block and now I include them from the Kixtart.kix file and execute using the function names instead. The size of code is about the same (prior to tokenizing) and all my vars are either explicitly Global or DIM'd locally in their function scopes. I try to not let any variables leak in between. The tokenized version seems to show a longer delay before it starts kicking off (bad pun), so I was assuming (incorrectly it seems) that it was decrypting the code to run it. I will work on my code to see where the chokepoints are.
_________________________
silence is golden, but duct tape is silver

Top
#143348 - 2005-07-12 09:59 AM Re: Pre-Tokenized Scripts - Performance Impact?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Quote:

eh, nothing you said here makes me believe what you suggest.
quess I need to try it out myself too.




Indeed you should.

Here is a very simple proof. Two scripts ScriptA.kix and ScriptB.kix:
ScriptA.Kix
Code:
$=SetOption("Explicit","ON")
"KiXtart version: "+@KIX+@CRLF
INCLUDE "ScriptB.Kix"
"$$VAR IsDeclared In ScriptA.kix="+IsDeclared($VAR)+@CRLF
Exit 0



ScriptB.kix
Code:
DIM $VAR
"$$VAR IsDeclared In ScriptB.kix="+IsDeclared($VAR)+@CRLF



Now watch what happens...
Quote:

C:\Temp>kix32 scripta.kix
KiXtart version: 4.50
$VAR IsDeclared In ScriptB.kix=1
$VAR IsDeclared In ScriptA.kix=0

C:\Temp>kix32 -t scripta.kix

C:\Temp>kix32 scripta.kx
KiXtart version: 4.50
$VAR IsDeclared In ScriptB.kix=1
$VAR IsDeclared In ScriptA.kix=0




In both cases $VAR is undeclared after the INCLUDE. If the code was simply in-lined then the local variable $VAR would be declared after the INCLUDE.

Now, try changing ScriptA to:
Code:
$=SetOption("Explicit","ON")
"KiXtart version: "+@KIX+@CRLF
INCLUDE "ScriptB.Kix"
"$$VAR IsDeclared In ScriptA.kix="+IsDeclared($VAR)+@CRLF
INCLUDE "ScriptB.Kix"
Exit 0



Now you get:
Quote:

C:\Temp>kix32 scripta.kix
KiXtart version: 4.50
$VAR IsDeclared In ScriptB.kix=1
$VAR IsDeclared In ScriptA.kix=0
$VAR IsDeclared In ScriptB.kix=1

C:\Temp>kix32 -t scripta.kix

C:\Temp>kix32 scripta.kx

ERROR : doubly defined UDF [scriptb.kix]!




Interesting, huh?

Top
#143349 - 2005-07-12 12:53 PM Re: Pre-Tokenized Scripts - Performance Impact?
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
well...
it indeed looks odd. just like you said.
it looks like if Ruud has reused some UDF code stuff.

anyways, this seems to be a problem of explicit as well as include.
nah, will stop speculating and shall wait for his reply.


Edited by Lonkero (2005-07-12 12:55 PM)
_________________________
!

download KiXnet

Top
#143350 - 2005-07-12 04:37 PM Re: Pre-Tokenized Scripts - Performance Impact?
Ruud van Velsen Moderator Offline
Developer
*****

Registered: 1999-05-06
Posts: 391
Loc: Amsterdam, The Netherlands
In short: there is indeed an issue with INCLUDing a file multiple times.

This will be fixed in the next update.

As for performance: running a pre-tokenized script created from multiple source scripts will (should) be faster than running the clear text versions. Especially if the scripts reside on a network, because it saves network roundtrips.

If you find otherwise, please post sample scripts that demonstrate the issue.

Regards,

Ruud

Top
#143351 - 2005-07-12 04:47 PM Re: Pre-Tokenized Scripts - Performance Impact?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Ruud,

Is there any likelihood that using "Exit" in a tokenised-included sub-script will be changed to function the same way as a CALLed script? i.e. it will exit only the INCLUDEd scope rather than exiting the calling scope?

Top
#143352 - 2005-07-12 04:54 PM Re: Pre-Tokenized Scripts - Performance Impact?
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
I hope not.
I still insist that included script should not have own scope but share the global scope of the including script.
just like it was intended in the first place.

if one wants to have subscopes, there are methods for doing that.
_________________________
!

download KiXnet

Top
#143353 - 2005-07-12 06:22 PM Re: Pre-Tokenized Scripts - Performance Impact?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Quote:

I hope not.
I still insist that included script should not have own scope but share the global scope of the including script.
just like it was intended in the first place.

if one wants to have subscopes, there are methods for doing that.




Tricky. Either it should have it's own scope in which case DIMmed variables are local to the included script (as it is at the moment) and exit will cause an exit only from the included script (which it doesn't) or it is simply merged in-line in which case locally dimmed variables are visible in the including parent script and exit causes an exit from the parent script.

It should definately do one or the other, and not mix the two.

As to which is preferable, well that's up for debate.

If INCLUDEd files have their own scope then sub-scripts which have been developed using CALL will work with INCLUDE with no change. I can't think of any major benefit of having INCLUDEd code in the same scope as the parent script.

Top
#143354 - 2005-07-12 06:30 PM Re: Pre-Tokenized Scripts - Performance Impact?
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
or, there is no major benefit of having them any other scope.
and again, that is doable already.
so, I think include should stay as it was intended.

the problem currently is that it does not work right.
so all is needed is the fixing.
_________________________
!

download KiXnet

Top
#143355 - 2005-08-24 09:01 AM Re: Pre-Tokenized Scripts - Performance Impact?
Ruud van Velsen Moderator Offline
Developer
*****

Registered: 1999-05-06
Posts: 391
Loc: Amsterdam, The Netherlands
Just to let you the issue with including a file multiple times is fixed in 4.51.

Ruud

Top
Page 1 of 1 1


Moderator:  Jochen, Allen, Radimus, Glenn Barnas, ShaneEP, Ruud van Velsen, Arend_, Mart 
Hop to:
Shout Box

Who's Online
0 registered and 661 anonymous users online.
Newest Members
Viginette, ManuvdWielNL, Sir_Barrington, batdk82, StuTheCoder
17888 Registered Users

Generated in 0.151 seconds in which 0.121 seconds were spent on a total of 12 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org