Background Downloading from the net

[SMcNeill]

Code: QB64: [Select]

1. CONST NumberOfFilesToDownload = 1 'Set These for the files you want to continiously download in the background
2. DIM SHARED DownloadList(NumberOfFilesToDownload) AS STRING
3. DownloadList(1) = "http://rss.wunderground.com/q/zmw:24138.1.99999"
4.  
5.  
6. DIM SHARED Maintimer AS LONG, Downloadtimer AS LONG
7. DIM SHARED KeepDownloading(NumberOfFilesToDownload) AS LONG
8. DIM SHARED DownloadHandle(NumberOfFilesToDownload) AS LONG
9. DIM SHARED DownloadData(NumberOfFilesToDownload) AS STRING
10.  
11.  
12.  
13.  
14.  
15.  
16. Maintimer = _FREETIMER
17. Downloadtimer = _FREETIMER
18.  
19. ON TIMER(Maintimer, 6) RestartDownload 'This is the time between background refreshes that we want from our webpages/RSS feeds
20. ON TIMER(Downloadtimer, 0.05) BackgroundUpdate 'This is the timer so we download our information in the background.  It's currently set to poll a remote server 20 times per second
21.  
22.  
23. TIMER(Maintimer) ON 'NOTE, when a timer is turned ON, it won't start until AFTER the specified amount of time has passed
24. InitDownloads 'To bypass this initial wait, we manually call InitDownloads ourselves to start the process
25.  
26.  
27. DO
28.     _LIMIT 60
29.     IF LEN(DownloadData(1)) = 0 THEN TimesRefreshed = TimesRefreshed + 1
30.     'Notice TimesRefreshed WON'T increase by one each refresh.
31.     'Our mainloop is making 60 passes per second (as per our LIMIT)
32.     'and the download process only runs 20 times a second.
33.     'There's also the small lag to account for from when we first send data to open the remote server
34.     'to when we get our first data back.
35.     '
36.     'For my test on my PC, this number increments by about 20 (+/-5) each refresh
37.     LOCATE 1, 1: PRINT LEN(DownloadData(1)), TimesRefreshed
38. LOOP
39.  
40.  
41.  
42. SUB RestartDownload 'restart the download timer every 60 seconds to refresh download$ in the background.
43.     FOR i = 1 TO NumberOfFilesToDownload 'clear the old files
44.         KeepDownloading(i) = 0
45.         DownloadData(i) = ""
46.         CLOSE DownloadHandle(i) 'close any old connections (if the remote server hasn't already)
47.     NEXT
48.     InitDownloads 'reopen all those closed handles for fresh transfers
49. END SUB
50.  
51. SUB BackgroundUpdate
52.     FOR i = 1 TO NumberOfFilesToDownload
53.         GET #DownloadHandle(i), , a$
54.         IF a$ <> "" THEN
55.             DownloadData(i) = DownloadData(i) + a$
56.         ELSE
57.             KeepDownloading(i) = KeepDownloading(i) + 1 'Let's track our number of blank responses
58.         END IF
59.         IF KeepDownloading(i) > 40 THEN CloseConnection = CloseConnection + 1 'if we get no data from anything for 2 seconds, we stop trying
60.     NEXT
61.     IF CloseConnection = NumberOfFilesToDownload THEN TIMER(Downloadtimer) OFF
62. END SUB
63.  
64. SUB InitDownloads
65.     FOR i = 1 TO NumberOfFilesToDownload
66.         DownloadHandle(i) = Opendownload(DownloadList(i))
67.     NEXT
68.     BackgroundUpdate 'manually call the background service once so we don't wait for the timer
69.     TIMER(Downloadtimer) ON
70. END SUB
71.  
72. FUNCTION Opendownload (url$)
73.     link$ = url$
74.     url2$ = RTRIM$(LTRIM$(link$))
75.     url4$ = RTRIM$(LTRIM$(link$))
76.     IF LEFT$(UCASE$(url2$), 7) = "HTTP://" THEN url4$ = MID$(url2$, 8)
77.     x = INSTR(url4$, "/")
78.     IF x THEN url2$ = LEFT$(url4$, x - 1)
79.     NewsClient = _OPENCLIENT("TCP/IP:80:" + url2$)
80.     IF NewsClient = 0 THEN EXIT FUNCTION
81.     e$ = CHR$(13) + CHR$(10) ' end of line characters
82.     url3$ = RIGHT$(url4$, LEN(url4$) - x + 1)
83.     x$ = "GET " + url3$ + " HTTP/1.1" + e$
84.     x$ = x$ + "Host: " + url2$ + e$ + e$
85.     PUT #NewsClient, , x$
86.     Opendownload = NewsClient
87. END FUNCTION

Something which I'm always needing to do and yet have never actually sat down and worried about actually working out all the details before:  a method to download files as a background process inside another program.

My latest little clock program grabs information from a RSS feed from the internet -- https://www.qb64.org/forum/index.php?topic=1095.0

In it, the website which I get information from drops a long bomb of information which takes several seconds to download.  Since the program is a clock, it NEEDS to update the screen so the second dial moves correctly, without a visible pause every time the download routine is called.  The current routine inside that program is insufficient for background downloading, and thus I wrote up this little code so I can plug it into it and not have any issues.

I think this little program comments itself well enough so nobody would have any issues understanding how it works, but if you're the least bit confused about any part of it, feel free to ask and I'll help explain whatever I need to.

The demo is set to poll the file information 20 times a second, and do a complete refresh of data every 6 seconds.  (Which is REALLY insane for constantly polling a website for information, like from a RSS feed, which normally only needs to update once every fifteen minutes or so...  Some servers might flag your IP for a DoS and refuse connections if you poll them too often, so be careful with that.  For my needs, currently, I'm just going to refresh the weather data every 10 minutes which my clock runs.  Nobody nowhere should object to that small amount of automated connections...)

[Pete]

I usually slave downloads out to wget or curl with shell calls, but I have used _openclient in the past. Neat stuff!

So...

IF LEFT$(UCASE$(url2$), 7) = "HTTP://" THEN url4$ = MID$(url2$, 8)

How about this instead?

IF LEFT$(UCASE$(url2$), 4) = "HTTP" THEN url4$ = MID$(url2$, INSTR(url2$, "://") + 3)

That way, you take care of https and http urls. ;)

Pete

[SMcNeill]

Https won’t work without jumping through hoops much more complicated than this.  (For one thing, you need to use a different port to start with...)

A quick read on it here: https://biztechmagazine.com/article/2007/07/http-vs-https

[Pete]

Ah, it has been awhile. That's probably the reason why I started using Curl and Wget. Wget finally started supporting https and Curl has for some time. It would be great if QB64 could be updated at some point, too. A lot of newer sites are https now... including this one!

Pete