Print Page - Background Downloading from the net

Code: QB64: [Select]

CONST NumberOfFilesToDownload = 1 'Set These for the files you want to continiously download in the background
DIM SHARED DownloadList(NumberOfFilesToDownload) AS STRING
DownloadList(1) = "http://rss.wunderground.com/q/zmw:24138.1.99999"
 
 
DIM SHARED Maintimer AS LONG, Downloadtimer AS LONG
DIM SHARED KeepDownloading(NumberOfFilesToDownload) AS LONG
DIM SHARED DownloadHandle(NumberOfFilesToDownload) AS LONG
DIM SHARED DownloadData(NumberOfFilesToDownload) AS STRING
 
 
 
 
 
 
Maintimer = _FREETIMER
Downloadtimer = _FREETIMER
 
ON TIMER(Maintimer, 6) RestartDownload 'This is the time between background refreshes that we want from our webpages/RSS feeds
ON TIMER(Downloadtimer, 0.05) BackgroundUpdate 'This is the timer so we download our information in the background.  It's currently set to poll a remote server 20 times per second
 
 
TIMER(Maintimer) ON 'NOTE, when a timer is turned ON, it won't start until AFTER the specified amount of time has passed
InitDownloads 'To bypass this initial wait, we manually call InitDownloads ourselves to start the process
 
 
DO
    _LIMIT 60
    IF LEN(DownloadData(1)) = 0 THEN TimesRefreshed = TimesRefreshed + 1
    'Notice TimesRefreshed WON'T increase by one each refresh.
    'Our mainloop is making 60 passes per second (as per our LIMIT)
    'and the download process only runs 20 times a second.
    'There's also the small lag to account for from when we first send data to open the remote server
    'to when we get our first data back.
    '
    'For my test on my PC, this number increments by about 20 (+/-5) each refresh
    LOCATE 1, 1: PRINT LEN(DownloadData(1)), TimesRefreshed
LOOP
 
 
 
SUB RestartDownload 'restart the download timer every 60 seconds to refresh download$ in the background.
    FOR i = 1 TO NumberOfFilesToDownload 'clear the old files
        KeepDownloading(i) = 0
        DownloadData(i) = ""
        CLOSE DownloadHandle(i) 'close any old connections (if the remote server hasn't already)
    NEXT
    InitDownloads 'reopen all those closed handles for fresh transfers
END SUB
 
SUB BackgroundUpdate
    FOR i = 1 TO NumberOfFilesToDownload
        GET #DownloadHandle(i), , a$
        IF a$ <> "" THEN
            DownloadData(i) = DownloadData(i) + a$
        ELSE
            KeepDownloading(i) = KeepDownloading(i) + 1 'Let's track our number of blank responses
        END IF
        IF KeepDownloading(i) > 40 THEN CloseConnection = CloseConnection + 1 'if we get no data from anything for 2 seconds, we stop trying
    NEXT
    IF CloseConnection = NumberOfFilesToDownload THEN TIMER(Downloadtimer) OFF
END SUB
 
SUB InitDownloads
    FOR i = 1 TO NumberOfFilesToDownload
        DownloadHandle(i) = Opendownload(DownloadList(i))
    NEXT
    BackgroundUpdate 'manually call the background service once so we don't wait for the timer
    TIMER(Downloadtimer) ON
END SUB
 
FUNCTION Opendownload (url$)
    link$ = url$
    url2$ = RTRIM$(LTRIM$(link$))
    url4$ = RTRIM$(LTRIM$(link$))
    IF LEFT$(UCASE$(url2$), 7) = "HTTP://" THEN url4$ = MID$(url2$, 8)
    x = INSTR(url4$, "/")
    IF x THEN url2$ = LEFT$(url4$, x - 1)
    NewsClient = _OPENCLIENT("TCP/IP:80:" + url2$)
    IF NewsClient = 0 THEN EXIT FUNCTION
    e$ = CHR$(13) + CHR$(10) ' end of line characters
    url3$ = RIGHT$(url4$, LEN(url4$) - x + 1)
    x$ = "GET " + url3$ + " HTTP/1.1" + e$
    x$ = x$ + "Host: " + url2$ + e$ + e$
    PUT #NewsClient, , x$
    Opendownload = NewsClient
END FUNCTION

Something which I'm always needing to do and yet have never actually sat down and worried about actually working out all the details before: a method to download files as a background process inside another program.

My latest little clock program grabs information from a RSS feed from the internet -- https://www.qb64.org/forum/index.php?topic=1095.0

In it, the website which I get information from drops a long bomb of information which takes several seconds to download. Since the program is a clock, it NEEDS to update the screen so the second dial moves correctly, without a visible pause every time the download routine is called. The current routine inside that program is insufficient for background downloading, and thus I wrote up this little code so I can plug it into it and not have any issues.

I think this little program comments itself well enough so nobody would have any issues understanding how it works, but if you're the least bit confused about any part of it, feel free to ask and I'll help explain whatever I need to.

The demo is set to poll the file information 20 times a second, and do a complete refresh of data every 6 seconds. (Which is REALLY insane for constantly polling a website for information, like from a RSS feed, which normally only needs to update once every fifteen minutes or so... Some servers might flag your IP for a DoS and refuse connections if you poll them too often, so be careful with that. For my needs, currently, I'm just going to refresh the weather data every 10 minutes which my clock runs. Nobody nowhere should object to that small amount of automated connections...)

I usually slave downloads out to wget or curl with shell calls, but I have used _openclient in the past. Neat stuff!

So...

IF LEFT$(UCASE$(url2$), 7) = "HTTP://" THEN url4$ = MID$(url2$, 8)

How about this instead?

IF LEFT$(UCASE$(url2$), 4) = "HTTP" THEN url4$ = MID$(url2$, INSTR(url2$, "://") + 3)

That way, you take care of https and http urls. ;)

Pete

Https won’t work without jumping through hoops much more complicated than this. (For one thing, you need to use a different port to start with...)

A quick read on it here: https://biztechmagazine.com/article/2007/07/http-vs-https

Ah, it has been awhile. That's probably the reason why I started using Curl and Wget. Wget finally started supporting https and Curl has for some time. It would be great if QB64 could be updated at some point, too. A lot of newer sites are https now... including this one!

Pete

QB64.org Forum

Active Forums => Programs => Topic started by: SMcNeill on February 24, 2019, 05:22:47 am