Web scraping in KiX - KiXtart.org

You are not logged in. [Log In] KiXtart.org website » Forums » KiXtart » COM Scripting » Web scraping in KiX

Page 2 of 3

Topic Options

#162344 - 2006-05-27 01:23 AM

Re: Web scraping in KiX

Lonkero Offline

KiX Master Guru

Registered: 2001-06-05
Posts: 22346
Loc: OK

add some error output:
Code:


$HTML = CreateObject("microsoft.XMLhttp")
if @error exit @error endif
        $HTML.Open("GET", $URL, Not 1)
        if @error exit @error endif
        $HTML.Send
        if @error exit @error endif
        If $HTML.Status = 200
                $GetPage = $HTML.Responsebody
                if @error exit @error endif

and what the error is:
Code:


GetPage("http://10.10.10.1/kix/kix64.dll?Login") ? ?
if @error
 "error occured." ?
 "error: " @error ?
 "descr: " @serror ?
else
 "exited without errors." ?
endif

_________________________
!

download KiXnet

Top

#162345 - 2006-05-27 02:16 AM Re: Web scraping in KiX
pearly pearly Getting the hang of it Registered: 2004-02-04 Posts: 92	Unfortunately, console returned "exited without errors."
Top

#162346 - 2006-05-27 02:31 AM Re: Web scraping in KiX
Lonkero Lonkero KiX Master Guru Registered: 2001-06-05 Posts: 22346 Loc: OK	k, download wget and try to pull the same addy with verbose output. what does it give you? _________________________ ! download KiXnet
Top

#162347 - 2006-05-27 03:47 AM Re: Web scraping in KiX
pearly pearly Getting the hang of it Registered: 2004-02-04 Posts: 92	excellent! it returned the html text to a file. what does this do that COM doesn't?
Top

#162348 - 2006-05-27 03:53 AM Re: Web scraping in KiX
NTDOC NTDOC Administrator Registered: 2000-07-28 Posts: 11624 Loc: CA	It makes COM calls and API calls too, it's just more purpose built than the exposed elements of the COM object you're using here via KiXtart.
Top

#162349 - 2006-05-27 04:03 AM Re: Web scraping in KiX
pearly pearly Getting the hang of it Registered: 2004-02-04 Posts: 92	so I'm guessing there is no solution in KiXtart? I'm satisfied with wget, but if there is a solution in KiXtart, I'm more than welcome to hear it. Thanks!
Top

#162350 - 2006-05-27 04:12 AM Re: Web scraping in KiX
NTDOC NTDOC Administrator Registered: 2000-07-28 Posts: 11624 Loc: CA	Well we don't work there where you do, so we don't know the inner workings of your network and what all might be causing an issue. There are some things that KiXtart just can't natively do and socket manipulation is one of them. If WGET is doing what you want, then nothing wrong with it until such time as maybe KiXtart does support it. I think KiXforms has some support but how much I'm not sure.
Top

#162351 - 2006-05-27 04:49 PM Re: Web scraping in KiX
Lonkero Lonkero KiX Master Guru Registered: 2001-06-05 Posts: 22346 Loc: OK	why I suggested wget is that if you pull a page with it in verbose mode, you will see if the server forces a page redirect or otherwise some odd behavior. oh, and xp security fix or your antivirus both can block the script as virus. _________________________ ! download KiXnet
Top

#162352 - 2006-05-27 06:01 PM Re: Web scraping in KiX
Lonkero Lonkero KiX Master Guru Registered: 2001-06-05 Posts: 22346 Loc: OK	oh, and did you try only one of responsebody and responsetext? or both and neither returned anything? don't remember what the one was with which you can return the response headers. _________________________ ! download KiXnet
Top

#162353 - 2006-05-30 11:26 PM Re: Web scraping in KiX
pearly pearly Getting the hang of it Registered: 2004-02-04 Posts: 92	I used both and it returned nothing back. Will I be notified if XP or anti-virus blocked a script?
Top

#162354 - 2006-05-31 07:56 AM Re: Web scraping in KiX
Lonkero Lonkero KiX Master Guru Registered: 2001-06-05 Posts: 22346 Loc: OK	usually not. xp and some antiviruses may block stuff silently as being part of their ultimate security feature. _________________________ ! download KiXnet
Top

#162355 - 2006-05-31 11:03 AM Re: Web scraping in KiX
NTDOC NTDOC Administrator Registered: 2000-07-28 Posts: 11624 Loc: CA	Yep, rarely will a product accurately notify you why you can't remotely access it. It could be many things. On some of my systems with the Symantec VPN installed I can't remotely connect unless the driver is disabled. It just tells me the "Network path can not be found" Most firewalls are designed to not respond either on purpose so that the querying system won't send new or varied queries trying to find something open which is the right thing for them to do.
Top

#162356 - 2006-05-31 06:55 PM Re: Web scraping in KiX
pearly pearly Getting the hang of it Registered: 2004-02-04 Posts: 92	sigh how frustrating. alas, i will use wget to extract the page. thanks for your help guys!
Top

#162357 - 2006-06-01 06:04 PM Re: Web scraping in KiX
Lonkero Lonkero KiX Master Guru Registered: 2001-06-05 Posts: 22346 Loc: OK	hey, pearly... thought I asked it already, but... does wget verbose mode give you some insight what is happening? is there some redirection or something going on? _________________________ ! download KiXnet
Top

#162358 - 2006-06-01 07:15 PM

Re: Web scraping in KiX

pearly Offline

Getting the hang of it

Registered: 2004-02-04
Posts: 92

This is what it says :

Code:

--10:13:08--  http://11.111.111.11/testing/xyz.dll?login
           => `xyz.dll@login.1'
Connecting to 11.111.111.11:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified

    [  <=>

10:13:08 (58.24 KB/s) - `xyz.dll@login.1' saved [27169]

Top

#162359 - 2006-06-02 05:40 PM Re: Web scraping in KiX
Lonkero Lonkero KiX Master Guru Registered: 2001-06-05 Posts: 22346 Loc: OK	how is your commandline for wget? the interesting part on the reply is: "=> `xyz.dll@login.1'" _________________________ ! download KiXnet
Top

#162360 - 2006-06-02 06:20 PM Re: Web scraping in KiX
pearly pearly Getting the hang of it Registered: 2004-02-04 Posts: 92	it's nothing unique. wget --verbose http://11.111.111.11/testing/xyz.dll?login
Top

#162361 - 2006-06-02 07:33 PM Re: Web scraping in KiX
Lonkero Lonkero KiX Master Guru Registered: 2001-06-05 Posts: 22346 Loc: OK	so, do you get something with: http://11.111.111.11/testing/xyz.dll@login.1 _________________________ ! download KiXnet
Top

#162362 - 2006-06-02 07:45 PM Re: Web scraping in KiX
pearly pearly Getting the hang of it Registered: 2004-02-04 Posts: 92	'xyz.dll@login.1' is saved to the same location as the wget program and in it, is the full source code (of the login page). i tried wget --verbose http://11.111.111.11/testing/xyz.dll@login.1 but it gives me the following : Code: --10:37:50-- http://11.111.111.11/testing/xyz.dll@login.1 => `xyz.dll@login.1.1' Connecting to 11.111.111.11:80... connected. HTTP request sent, awaiting response... 404 Object Not Found 10:37:50 ERROR 404: Object Not Found. also i couldn't get past the login screen with this program. i'm not sure how to enter in the login information so i can access the other pages.
Top

#162363 - 2006-06-02 11:05 PM Re: Web scraping in KiX
Lonkero Lonkero KiX Master Guru Registered: 2001-06-05 Posts: 22346 Loc: OK	ja, I think I ask in the very beginning of this thread what you want to really achieve. getting the html does not help if you in fact want to pass information. anyway, logically, if that addy didn't, did you try the obvious: http://11.111.111.11/testing/xyz.dll@login _________________________ ! download KiXnet
Top

Page 2 of 3

Previous Topic View All Topics

Index Next Topic

Moderator: Shawn, ShaneEP, Ruud van Velsen, Arend_, Jochen, Radimus, Glenn Barnas, Allen, Mart

Hop to:

Shout Box

Search the board with:
superb Board Search
or try with google:

	Web kixtart.org