Text Only | Text with Attachments

QB64.org Forum

Active Forums => QB64 Discussion => Topic started by: IronMan on March 20, 2020, 08:56:50 pm

Title: Downloading TEXT file from internet
Post by: IronMan on March 20, 2020, 08:56:50 pm
I am obtaining several text files from the NOAA weather site. I'm currently using curl from a shell command from within my QB64 app like:

SHELL "curl https://tgftp.nws.noaa.gov/data/forecasts/zone/ms/msz024.txt >forecast.txt"

to obtain a text file from this site. I however would like to do this within the QB64 itself rather than having to SHELL and run a external application like Curl to fetch the file. I know there is a Download command in QB64, but I'm not understanding how to use it. Does anyone have experience on how to do this? Perhaps a small piece of code for an example?

Any help is appreciated.

Thank you.
Kent
Title: Re: Downloading TEXT file from internet
Post by: Real2Two on March 20, 2020, 09:00:16 pm
Try doing some get of GET request of QB64, which I remember seeing code with the GET with QB64, but I forgot about it.
Title: Re: Downloading TEXT file from internet
Post by: Real2Two on March 20, 2020, 09:02:54 pm
Check out how this works https://github.com/FellippeHeitor/Snippets/blob/master/scoreboard.bas
Title: Re: Downloading TEXT file from internet
Post by: FellippeHeitor on March 20, 2020, 09:47:03 pm
QB64 won’t fetch from an https address.
Title: Re: Downloading TEXT file from internet
Post by: IronMan on March 20, 2020, 09:54:06 pm
Quote from: FellippeHeitor on March 20, 2020, 09:47:03 pm
QB64 won’t fetch from an https address.


Ahhh, didn't know that.

But it will a non-secure FTP or HTTP? I am downloading some images form within QB64 but they're coming off a FTP server.
That must be why I'm having problems when trying to get filed form SSL sites.
Title: Re: Downloading TEXT file from internet
Post by: lawsonm1 on March 20, 2020, 10:06:10 pm
I posted a similar question back on 2/6/2020 (Pulling data from a website) https://www.qb64.org/forum/index.php?topic=2158.msg114021#msg114021. wGet was one of the suggestions. That worked out great for what I was doing. I grabbed the website data I wanted, and just parsed out the parts I needed. Mike
Title: Re: Downloading TEXT file from internet
Post by: EricE on March 21, 2020, 04:25:34 am
This posting is for those finding this thread looking for information about using cURL to download files.

Using curl to automate HTTP jobs
https://curl.haxx.se/docs/httpscripting.html (https://curl.haxx.se/docs/httpscripting.html)

libcurl - the multiprotocol file transfer library
https://curl.haxx.se/libcurl/ (https://curl.haxx.se/libcurl/)

Everything curl - the book
https://curl.haxx.se/book.html (https://curl.haxx.se/book.html)


Title: Re: Downloading TEXT file from internet
Post by: EricE on March 21, 2020, 04:29:16 am
And here is the manual for Wget

GNU Wget 1.20 Manual
https://www.gnu.org/software/wget/manual/wget.html (https://www.gnu.org/software/wget/manual/wget.html)
Title: Re: Downloading TEXT file from internet
Post by: IronMan on March 21, 2020, 05:48:51 pm
Curl is doing exactly what I need since you can't download files from HTTPS sites in QB64.

Example:
SHELL "curl -k https://tgftp.nws.noaa.gov/data/forecasts/zone/ms/msz024.txt > forecast.txt"

The only thing is, when the SHELL is executed in the program, it opens a black command windows and shows the Curl being ran in it. This is especially isn't good when using inForm in a Windows environment.

Does anyone know if there is a way to not allow the command windows to be shown when SHELL is used?

Thank you.
Kent
Title: Re: Downloading TEXT file from internet
Post by: FellippeHeitor on March 21, 2020, 06:19:44 pm
Code: QB64: [Select]
  1. SHELL _HIDE "curl -k https://tgftp.nws.noaa.gov/data/forecasts/zone/ms/msz024.txt > forecast.txt"
  2.  
Title: Re: Downloading TEXT file from internet
Post by: IronMan on March 21, 2020, 06:56:22 pm
Quote from: FellippeHeitor on March 21, 2020, 06:19:44 pm
Code: QB64: [Select]
  1. SHELL _HIDE "curl -k https://tgftp.nws.noaa.gov/data/forecasts/zone/ms/msz024.txt > forecast.txt"
  2.  

That got it.

Thanks
Title: Re: Downloading TEXT file from internet
Post by: IronMan on March 21, 2020, 10:12:07 pm
Quote from: FellippeHeitor on March 21, 2020, 06:19:44 pm
Code: QB64: [Select]
  1. SHELL _HIDE "curl -k https://tgftp.nws.noaa.gov/data/forecasts/zone/ms/msz024.txt > forecast.txt"
  2.  

The SHELL _HIDE fixed the command window from showing up. I'm now getting a "Please Wait" showing up while the application is after about 3-4 seconds once it starts. Not really causing a problem, but I can't find this anywhere in the code so I assume it's in the inForm code somewhere. Any way to avoid this?
Title: Re: Downloading TEXT file from internet
Post by: FellippeHeitor on March 21, 2020, 10:20:30 pm
InForm will show that when execution is halted for too long. It’ll happen when a SHELL call takes too long or when you do a loop somewhere that hogs the input routines. It is by design.

You cannot disable it, but you can set a custom message:

Code: QB64: [Select]
  1.  __UI_WaitMessage = "Fetching data"
Title: Re: Downloading TEXT file from internet
Post by: Dav on March 21, 2020, 11:10:11 pm
Here's another method without using SHELL.  Just thought I'd throw it into the mix!  Works for me in Windows 7.  Download from https sites. Uses a built-in Windows Library function.   

- Dav

Code: QB64: [Select]
  2.     FUNCTION URLDownloadToFileA% (BYVAL pCaller AS LONG, szURL AS STRING, szFileName AS STRING, BYVAL dwReserved AS LONG, BYVAL lpfnCB AS LONG)
  4.  
  5. url$ = "https://tgftp.nws.noaa.gov/data/forecasts/zone/ms/msz024.txt"
  6. urlfile$ = "forecast.txt"
  7.  
  8. a% = URLDownloadToFileA%(0, url$, urlfile$, 0, 0)
  9.  
  10. SLEEP 5   'pause to allow the download to complete
  11.  
  12. IF a% = 0 THEN
  13.     PRINT "Downloaded!"
  15.     PRINT "NOT Downloaded!"
  17.  
Title: Re: Downloading TEXT file from internet
Post by: TerryRitchie on March 21, 2020, 11:44:25 pm
Quote from: Dav on March 21, 2020, 11:10:11 pm
Here's another method without using SHELL.  Just thought I'd throw it into the mix!  Works for me in Windows 7.  Download from https sites. Uses a built-in Windows Library function.   

Dav, the code snippets you post are always great, and this one is no exception. Added to my tool bag. Thank you.
Title: Re: Downloading TEXT file from internet
Post by: Dav on March 22, 2020, 02:08:49 pm
You're very welcome, Terry.  Now that I'm out of work (most musicians here currently are due to the virus) I'll probably be participating more in the coding forums.  There sure is a lot of good programs people are posting here.  I'm enjoying just reading the forum.

- Dav
Title: Re: Downloading TEXT file from internet
Post by: IronMan on March 22, 2020, 05:28:14 pm
Quote from: Dav on March 21, 2020, 11:10:11 pm
Here's another method without using SHELL.  Just thought I'd throw it into the mix!  Works for me in Windows 7.  Download from https sites. Uses a built-in Windows Library function.   

- Dav

This works perfect when running in QB64. However, when I use it with the InFORM add on for QB64, it does some crazy things and locks up my computer. It may just be my computer.

Thanks for the code snippet.

- Kent

Code: QB64: [Select]
  2.     FUNCTION URLDownloadToFileA% (BYVAL pCaller AS LONG, szURL AS STRING, szFileName AS STRING, BYVAL dwReserved AS LONG, BYVAL lpfnCB AS LONG)
  4.  
  5. url$ = "https://tgftp.nws.noaa.gov/data/forecasts/zone/ms/msz024.txt"
  6. urlfile$ = "forecast.txt"
  7.  
  8. a% = URLDownloadToFileA%(0, url$, urlfile$, 0, 0)
  9.  
  10. SLEEP 5   'pause to allow the download to complete
  11.  
  12. IF a% = 0 THEN
  13.     PRINT "Downloaded!"
  15.     PRINT "NOT Downloaded!"
  17.  
Title: Re: Downloading TEXT file from internet
Post by: EricE on March 22, 2020, 06:15:02 pm
Question: Is urlmon part of Microsoft's Internet Explorer, or of the Win32 API?
Title: Re: Downloading TEXT file from internet
Post by: Dav on March 22, 2020, 06:24:47 pm
Sorry that is happening, Ironman.  Not sure why, using it hasn't locked up my pc before (I'm still on Win7 btw). 

EricE, urlmon.dll is an Internet Explorer library included with Windows.

- Dav

Title: Re: Downloading TEXT file from internet
Post by: EricE on March 22, 2020, 06:43:49 pm
Thank you Dav.

Here is the Microsoft web page for the URLDownloadToFile function.

URLDownloadToFile function
https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/ms775123(v%3Dvs.85) (https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/ms775123(v%3Dvs.85))

Here is a web page containing much information about the urlmon dll.

Windows 7 DLL File Information - urlmon.dll
https://www.win7dll.info/urlmon_dll.html (https://www.win7dll.info/urlmon_dll.html)
Title: Re: Downloading TEXT file from internet
Post by: FellippeHeitor on March 22, 2020, 07:15:07 pm
Can't use SLEEP with InForm.
Text Only | Text with Attachments