Pulling data from a website

[lawsonm1]

I'm thinking of trying to come up with a program that would pull data from a website, and use it. Specifically, get numerical data on stocks, futures, commodities, etc. from some website, and use that data in calculations, etc. within my program. I'm wondering is that is even possible; and/or how to go about it. Any suggestions? Thanks, Mike

[Pete]

If you're more interested in the results than building the entire program, download WGET or cURL web crawling utilities. You can make QB64 SHELL calls to either of those programs, and pass the info on what sites you want to crawl. I know someplace I have a QB64 app I made, and I' thinking it just used the QB64 _OPENCLIENT statement for that one...

http://www.qb64.org/wiki/OPENCLIENT

Anyway, you can search some threads here using WGET and/or cURL, and you'll find some coding examples, I'm sure. You cam also search _OPENCLIENT.

Oh, you will probably also want to code a parsing program, to extract that data from the downloaded webpage. What I do is load the entire page as a string variable, using a BINARY file read operation, and then a lot of INSTR() work to parse through the html tags to find the data, and then cut that section out to display it in my QB64 app.

I hope this gives you some ideas on how to proceed, and maybe some others will share their coding experiences, too and present some other alternatives.

Pete

[SMcNeill]

https://www.qb64.org/forum/index.php?topic=756.msg6455#msg6455

The above is a program which pulls information from our wiki and saves it to a file.  It may be useful as an example for your needs.

[FellippeHeitor]

If you do decide to write your own solution, keep in mind that QB64 won't work with secure connections (https). If the website you're going to fetch data from uses a secure connection, you'll need an external utility, like the ones Pete outlined above.

[doppler]

If you do decide to write your own solution, keep in mind that QB64 won't work with secure connections (https).

Well that bites.  Everybody who knows, the move to https is needed and happening.  Is it too much of a bitch to get https going ?

[lawsonm1]

Pete/Fellippe, I looked at wGet. Not too bad, and it only took a few minutes, and a few brain cells, and I already grabbed the website (https). The I ran it through the DOS find command, and got the output to a reasonably few lines. From there I can read it into my QB64 code and pull the data I need. Now I have to figure out the QB64 code and convert the string data to numeric (MID$ and VAL). Thanks again for the help, Mike

[Pete]

Well that bites.  Everybody who knows, the move to https is needed and happening.  Is it too much of a bitch to get https going ?

Grasshopper, like the ancient Chinese saying goes... Sometimes it is better to write a batch file than it is to make a bitch post. And by ancient, I mean circa 1990. That's 2000 years ago in BASIC years.

Hey, I tend to agree, but I don't know if it is all that doable work-wise. I recall WGET didn't support it just a few years ago, which is why I switched to cURL. Now I'd say WGET has caught up well with the times. I can't even remember all the gimmicky posts I read years ago, at WGET and/or cURL forus on ways to try and get around the the https issue. I thin bak then I hitched cURL to FireFox, and that allowed FireFox to broker the secure connection. So maybe this would have been accomplished already if it wasn't difficult for the developers to implement?

Pete :D

[lawsonm1]

Pete, well, I ran the following with no issues. I only had to do a few tests, and as I encountered failures, the program pretty much said what the error was. I know there will be other websites that may be a little harder to get data from, but for this first test, it seems to work.

wget -v --secure-protocol=TLSv1 --no-check-certificate --output-document=c:\temp\bb.txt https://www.thebullionbank.com

find "ask-price" c:\temp\bb.txt > c:\temp\bb1.txt

[Pete]

That's a good start Mike. The documentation is fairly well written for both resources, WGET and cURL, and both do a good job of error reporting. I experienced similar results when working through my project. Good luck with the parsing; it's the real grunt work but it's also the payoff.

Pete

[lawsonm1]

So, I have gotten a small program (that seems to work), that will pull data from a website, parse out what I need into a file, then take the data and put it into a date & time stamped array, and save it to disk. Then it goes to sleep for 5 minutes and repeats the cycle for ten hours, and then the program closes/exits. I'm thinking of putting the .exe as part of the Windows Scheduled Tasks to run once a day starting right before 8am. Does anyone know of any issues related to having the Windows Task Scheduler kick off QB64 generated .exe files? Thanks, Mike

[FellippeHeitor]

After generated, QB64 executables are no different from any other executable. You should have no issues.

[Pete]

The only issue I can see is with some anti-virus software, but those would have flagged your exe builds. I had one that required me to whitelist my QB64 folder, so it would ignore checking exe files launched from that folder. At worst, if I moved the file to some other directory, I could just whitelist the exe file itself.

Pete