Skatterbrainz
(Starting to like KiXtart)
2004-02-15 01:03 AM
Fetching HTML from a Form Feed?

I'm trying to fetch the HTML result of a page that would normally be called from a form page. I found the GetURL UDF function and it is perfect for my needs, but I still need to be able to send a form-request to the page I want to get the resulting HTML from. In other words, the resulting HTML is actually a page that returns a result from a form entry on the HTML page that normally calls it in the UI. Does anyone know how I can submit the form request to get that using an object backend (like microsoft.xmlhttp). I hope this makes sense.

Sealeopard
(KiX Master)
2004-02-15 01:18 AM
Re: Fetching HTML from a Form Feed?

See for example Is it possible to update a web page with a KiXtart script?

LonkeroAdministrator
(KiX Master Guru)
2004-02-15 01:33 AM
Re: Fetching HTML from a Form Feed?

form does not have it's own requests.
they do have actions and it's contents (vars) are sent when submitted.
depending what amount of data you are to submit, depends also the request (either GET or PUT)
anyway, what you need to send in you can actually do with the addressfield (see this threads url)

for more information on forms, you should lookie at http://w3.org under html 4.01 specs

Skatterbrainz
(Starting to like KiXtart)
2004-02-15 01:33 AM
Re: Fetching HTML from a Form Feed?

Thanks! That helps, but I'm still a bit confused. The current scheme is that an ordinary HTML page with a form and a single input box, submits the "empno" value entry to another ASP page which returns employee info in a one-row table. The form will not work with GET, only POST, and inline params do not work either (obviously, the ASP page is not checking for Request.QueryString, only Request.Form).

There is no javascript or vbscript code in the form page, only plan HTML. How can I submit a request to the ASP page programmatically so that it thinks I'm submitting the "empno" form value from a real HTML form?

Sealeopard
(KiX Master)
2004-02-15 01:36 AM
Re: Fetching HTML from a Form Feed?

The link I posted above shows in detail how to accomplish this. Forms don't need VBScript/JavaScript in order to function, just a submit button and an appropriately defined form action as coded in the HTML page.

Skatterbrainz
(Starting to like KiXtart)
2004-02-15 01:53 AM
Re: Fetching HTML from a Form Feed?

THanks! I don't want to wear out my stay on this thread. Some of the links in the thread you point me to are dead and I cannot get the info they refer to. In any case, I see that you mention not using IE at all, and that would be fine if I can do it that way. All I need to do is fetch the HTML result, like GetURL does, but pass in a form entry.

The form page has one input box named "empno" and the form tag calls "empinfo.asp" using "post". I tried the code below, but it doesn't work and causes the result to show an error due to failed SQL query.

Code:

Break On
Do
	?"Enter search term> "
	Gets $term
	Until $term
	$URL = "http://intranet.mycompany.com/empinfo.asp"
	$IE = CreateObject("InternetExplorer.Application")
	$IE.Visible = 1
	$IE.Navigate($URL)
	While $IE.Busy and $IE.ReadyState <> 4 and @ERROR = 0
Loop
$IE.Document.getElementById("empno").Value = $term
$IE.Document.getElementById("submit").Click()
Exit 0

I enter the the string value emp number and hit Enter and it does open the requested page but isn't passing in the parameter value properly. What am I doing wrong?

LonkeroAdministrator
(KiX Master Guru)
2004-02-15 02:05 AM
Re: Fetching HTML from a Form Feed?

k, as you seem little off and didn't bother checking on that topic, shoot this (if it works you can buy me a cup of coffee or something):
Code:

$term="whatever value you want to put there :P"

$web=createobject("microsoft.xmlhttp")

$web.open("POST","http://intranet.mycompany.com/empinfo.asp", not 1)

$web.setRequestHeader = "Content-Type", "application/x-www-form-urlencoded"

$web.send("empno="+$term)

"reply: " ? $web.responsebody



Skatterbrainz
(Starting to like KiXtart)
2004-02-15 02:14 AM
Re: Fetching HTML from a Form Feed?

Closer! (I'll send a cup of coffee anyway if I can just get it to work). The result still says "no matching records" but when I enter the same value '54321' in the form HTML page and pick Submit, it returns a valid result.

Here is the actual form HTML code pasted in below. In case it helps. (And yes, I did read through the thread you pointed me to. Sorry if I seem anxious or obnoxious - not my intent ) Just that the discussion seemed to go into a different area than where I'm trying to go.

Code:
<HTML>

<HEAD><TITLE>Employee User Name List</TITLE></HEAD>



<BODY BGCOLOR="#B2BCE9"><FONT FACE="ARIAL,HELVETICA">

<H3>Employee Standard User Name List</H3>

<b>

<FONT FACE="Comic Sans MS" SIZE=2 COLOR=black>

<P>   

<FORM METHOD="POST" ACTION="http://intranet.mycompany.com/empinfo.asp">

Employee #: <INPUT NAME="empno" SIZE="12"> <INPUT TYPE=SUBMIT>

</FORM>

</FONT>

</BODY>



</HTML>

LonkeroAdministrator
(KiX Master Guru)
2004-02-15 02:43 AM
Re: Fetching HTML from a Form Feed?

that indeed does not help anymore.
I know that you are playing with that weird server... was it MS eaSy?

anyways, normally servers do accept both ways, post and get.
the only difference is that post'ing allows lot larger data to be transmitted.
now, the above code does the sending just correctly.
but seems that the request-header is not 100% what IIS expects and thus it thinks it's wrong.
the request-header is something I can't help you with.

except you could try with it commented out.
and think there is more on the subject at that w3.org page discussing forms.
maybe at some MS page too where ppl cry about that too.

Skatterbrainz
(Starting to like KiXtart)
2004-02-15 02:50 AM
Re: Fetching HTML from a Form Feed?

Thank you for being patient with me. I will do as you suggest and try the header change, as well as researching w3.org (or w3c.org?). Thanks for pointing me in the right direction!

LonkeroAdministrator
(KiX Master Guru)
2004-02-15 03:00 AM
Re: Fetching HTML from a Form Feed?

ja, and once you find something, please note it here.
I have currently only 1 IIS server (as intranet) which is going down once I get the time to uninstall it and set up apache on the w2k server.
that doesn't mean all others do the same and I believe there is benefit in this struggle.

oh, btw, is there a way for you to list what the server sees in the request?
do you have the control over the server or any page on it?

Sealeopard
(KiX Master)
2004-02-15 03:04 AM
Re: Fetching HTML from a Form Feed?

Maybe this is a stupid question, but why bother with the HTML/ASP code at all?

As it is a simple SQL query you're performing you can do this all from within KiXtart.

Example:
Code:

;DBCommand() = http://www.kixtart.org/ubbthreads/showflat.php?Cat=&Board=UBB12&Number=83050
$searchfor='whatever'
$dsn='DRIVER={SQL Server};SERVER=servername;UID=user;PWD=password;DATABASE=mydatabase'
$sql="SELECT Field1, Field2 FROM Table1 WHERE Field3='"+$searchfor+"'"
$recordset = DBCommand($dsn,$sql)
? 'Error = '+@ERROR+' - '+@SERROR
for $row=0 to ubound($recordset,1)
  for $column=0 to ubound($recordset,2)
    ? 'Field(row='+$row+', column='+$column+') ='+$recordset[$row,$column]
  next
next

Skatterbrainz
(Starting to like KiXtart)
2004-02-15 03:05 AM
Re: Fetching HTML from a Form Feed?

I don't have any control over anything on the target server. The issue is that this is our "parent" company and they don't share with us "peons". This particular page takes an employee ID number and returns a table row with their network userid, email address and first, last and middle name. When people are hired, they are entered into their system first and assigned a userid. We want to be able to query that and use the return data to feed a system to create their AD user accounts automatically (or stage them in a holding table until reviewed and then kicked off).

They won't let us directly query their SQL backend, and their ASP page won't accept inline parameters (like using GET instead of POST, etc.) We're stuck with an HTML form and submit button to retrieve the info. I'm just trying to find a way to automate the process better. Everything for automating the user account creation is in place and ready. The only hold-up is getting the userid value from the empid number.

Sealeopard
(KiX Master)
2004-02-15 03:16 AM
Re: Fetching HTML from a Form Feed?

If nothing else works, then open up IE, use SETFOCUS to IE, use SENDKEYS to enter info and submit page, then scrape the returning HTML page from the temporary internet folders folder. Not an elegant solution but doable.

Skatterbrainz
(Starting to like KiXtart)
2004-02-15 03:21 AM
Re: Fetching HTML from a Form Feed?

Eek-Gads! That's a scary thought, but I understand your reasoning. The xmlhttp approach is so close it's killing me. If I could only get the request to spoof a form input so that the ASP page would see it via Request.Form() it seems like it should work. That would give me a direct text result as well (like your examples show) and would solve my dilema. I'll post here if I discover a solution that works.

Sealeopard
(KiX Master)
2004-02-15 03:34 AM
Re: Fetching HTML from a Form Feed?

Can you post the code you're using?

Skatterbrainz
(Starting to like KiXtart)
2004-02-15 03:47 AM
Re: Fetching HTML from a Form Feed?

I'm using the xmlhttp object code Lonkero posted earlier on this thread (up above). Here it is again...

Code:
$term="54321"
$web=CreateObject("microsoft.xmlhttp")
$web.Open("POST","http://intranet.mycompany.com/empinfo.asp", not 1)
$web.setRequestHeader = "Content-Type", "application/x-www-form-urlencoded"
$web.Send("empno="+$term)
$response = $web.ResponseBody
? $response

Sealeopard
(KiX Master)
2004-02-15 03:52 AM
Re: Fetching HTML from a Form Feed?

And what's the server response or error message you get? The KiXtart code itself looks correct.

Skatterbrainz
(Starting to like KiXtart)
2004-02-15 04:00 AM
Re: Fetching HTML from a Form Feed?

Not an "error" but simply that the request failed to return a valid matching record. The code is returned as it shows if I enter a bad value in the search form, only that it comes back in raw text form through the xmlhttp object. It seems to indicate that the empno value is not submitted like a true form('empno') entry, and is therefore causing the ASP page to fail on the query.

LonkeroAdministrator
(KiX Master Guru)
2004-02-15 04:07 AM
Re: Fetching HTML from a Form Feed?

k, lets try another one.
disable all trusts to that page (like disable all cookies for it etc) and be sure there are no cached stuff about the server in your HD.
go again there and try to do it again.
if not successfull, you have faced the same sh*t the korg board has.
a webcoder that thinks all browsers are cookie aware and all users do want to let the server save files on their HD.
this also means that the browser object can't do or that you should fight to get rid of the cookie need on that page.

if still succesfull, then it should be doable with the object and there is something else little wrong...

Sealeopard
(KiX Master)
2004-02-15 04:10 AM
Re: Fetching HTML from a Form Feed?

Maybe they're also filtering the HTTP headers to ferret out invalid requests e.g. via browsertype or similar.

Skatterbrainz
(Starting to like KiXtart)
2004-02-15 05:13 AM
Re: Fetching HTML from a Form Feed?

I don't know. I copied their search form page to my laptop and cleaned out everything but what I posted earlier. It works fine. Trying the cache and cookie thing didn't seem to make a difference. Nor did trying it from a previously unused machine. Just curious, does the "send" method for xmlhttp send as though it's a form collection object or as though it is a GET request?

LonkeroAdministrator
(KiX Master Guru)
2004-02-15 05:18 AM
Re: Fetching HTML from a Form Feed?

if you read the code I pasted, it clearly says POST.
like the http and html specs do say, there is only 2 types of requests.
and with the code you are using the post one.
forms can use both.

but I must no quess there is something else too in that webpage that you mistakenly left out or didn't notice to have significance.

Skatterbrainz
(Starting to like KiXtart)
2004-02-15 05:05 PM
Re: Fetching HTML from a Form Feed?

I wasn't doubting you at all, only asking a question out of curiosity. I apologize for dragging this on. I posted everything I have in my form page. I cannot post the ASP target page because I do not have access to the server or the source code. It may be that they are indeed doing more on their end than I am aware of. All I know is that I can send a form entry from my own form page and their ASP page accepts it and returns the desired output. When I try to call the page using script code it fails to return the desired result. I think I'm done trying for now. Thanks for all your help.

LonkeroAdministrator
(KiX Master Guru)
2004-02-15 09:47 PM
Re: Fetching HTML from a Form Feed?

well, doh.
sure, but like said before, the things like cookies and browser checks and alike are things that can affect.
if you said you did with clean machine did it have cookies disabled?
did you try with other browsers?

anyway, your only ways are to either search for the problem, or ask them.

btw, they also might check for referring page.
if so, they have specifically disabled automating the process.

Sealeopard
(KiX Master)
2004-02-16 01:27 AM
Re: Fetching HTML from a Form Feed?

However, then you can start spoofing heder variables, e.g.
Code:

$web.setRequestHeader = "Referer", "http://url.to.com/emp.html"

might work, however, according to http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q178066& it would not. YMMV

This might also help: http://www.webmasterworld.com/forum24/396.htm

Richard H.Administrator
(KiX Supporter)
2004-02-16 03:48 PM
Re: Fetching HTML from a Form Feed?

Here is an old-school method, controlling the IE object.

The $sGT weirdness is just to get around posting problems.

Don't forget to change the form URL to the real comany name:


Break On
 
; vim600: ai sw=3 ts=3
 
$sGT="<"
 
; Form HTML
$sHTML=''+$sGT+'HTML>'
+' '+$sGT+'HEAD>'+$sGT+'TITLE>Employee User Name List'+$sGT+'/TITLE>'+$sGT+'/HEAD>'
+' '+$sGT+'BODY BGCOLOR="#B2BCE9">'+$sGT+'FONT FACE="ARIAL,HELVETICA">'
+'  '+$sGT+'H3>Employee Standard User Name List'+$sGT+'/H3>'
+'  '+$sGT+'b>'
+'  '+$sGT+'FONT FACE="Comic Sans MS" SIZE=2 COLOR=black>'
+'   '+$sGT+'P>   '
+'   '+$sGT+'FORM Name=frmEmpInfo METHOD="POST" ACTION="http://intranet.mycompany.com/empinfo.asp">'
+'    Employee #:'
+'    '+$sGT+'INPUT Name=empno SIZE="12">'
+'   '+$sGT+'/FORM>'
+'  '+$sGT+'/FONT>'
+' '+$sGT+'/BODY>'
+''+$sGT+'/HTML>'
 
; Create Internet Explorer object
$oIE = CreateObject("InternetExplorer.Application")
 
; The following lines tidy up the screen a bit, giving a semi-kiosk mode.
$oIE.toolbar=0
$oIE.addressbar=0
$oIE.statusbar=0
 
$oIE.Navigate("about:blank")
; Wait for page to stop loading...
While $oIE.busy AND $oIE.readystate <> 4 AND @ERROR = 0 Loop
$oDoc=$oIE.document
 
; Now, add the starting form...
$oDoc.Write($sHTML)
While $oIE.busy AND $oIE.readystate <> 4 AND @ERROR = 0 Loop
 
; Comment out the next line if you don't want the form to appear (stealth mode)
$oIE.Visible=1
 
; The next line "autofills" the employee field.
; Note, we could just as well have coded the value in the HTML above, but this
; is an example of how to do control the form from within KiXtart
$oDoc.frmEmpInfo.empno.value="12345"
 
; This sleep not required - it's just here to give the viewer an opportunity
; to see what is going on.
Sleep 2
 
; Here is the magic - submit the form.
$oDoc.frmEmpInfo.submit()
 
; And wait for it to refresh...
While $oIE.busy AND $oIE.readystate <> 4 AND @ERROR = 0 Loop
 
; Uncomment the next two line to close IE and clean up.
;$oIE.quit()
;$oIE=0