reading remote text file

[xra7en]

I thought I saw a thread here awhile ago regarding this, might have been a different language.

can qb64 read a remote txtfile?

example mycoolsite.com/textfile.txt

was trying (of course unsuccessfully) the open command.

thanks guys

ps
Its the weekend, working on another goofy project.

[191Brian]

Pretty sure the answer is no, open only works on files local to you either on your PC or network share.

[SMcNeill]

You can look at the QB64 wiki download program I have here, as an example, if you’re just transfering over http and not https: https://www.qb64.org/forum/index.php?topic=756.msg6455#msg6455

Basically, use _OPENCLIENT and GET to do the work:

SUB DownloadRSS (url$, file$, timelimit)
    link$ = url$
    DIM l AS _INTEGER64, lod AS _INTEGER64
 
    url2$ = RTRIM$(LTRIM$(link$))
    url4$ = RTRIM$(LTRIM$(link$))
    IF LEFT$(UCASE$(url2$), 7) = "HTTP://" THEN url4$ = MID$(url2$, 8)
    x = INSTR(url4$, "/")
    IF x THEN url2$ = LEFT$(url4$, x - 1)
    NewsClient = _OPENCLIENT("TCP/IP:80:" + url2$)
    IF NewsClient = 0 THEN EXIT FUNCTION
    e$ = CHR$(13) + CHR$(10) ' end of line characters
    url3$ = RIGHT$(url4$, LEN(url4$) - x + 1)
    x$ = "GET " + url3$ + " HTTP/1.1" + e$
    x$ = x$ + "Host: " + url2$ + e$ + e$
    PUT #NewsClient, , x$
 
    OPEN file$ FOR OUTPUT AS #1: CLOSE #1
    OPEN file$ FOR BINARY AS #1
 
    t! = TIMER ' start time
    head$ = ""
    cont_type$ = ""
    DO
        _LIMIT 20
        GET #NewsClient, , a$
        IF LTRIM$(a$) > "" THEN PUT #1, , a$
    LOOP UNTIL TIMER > t! + timelimit AND timelimit > 0 ' (in seconds)
    CLOSE #NewsClient
    CLOSE #1
END SUB

[xra7en]

I checked the wiki with no luck, but only because I had no direction - "open" was all I knew.

This is good, I will part it out and see how it ticks!!

what I am doing is pulling a simple text file from one of my sites, I could just turn off the SSL and from what you said it should work.

Out of necessity, I wrote a "chocolatey" alternative (qb64), but much more simpler, been using it for last few years - I wanted something simple without complexity. works great. however I want a centralized DB instead of locally. So trying the old fashion text method - it works, and does not need something advanced as SQL(programing sql for remote access is an overkill for what I want). I basically just keep track of appname and link to it directly.



thanks!

I'll let you know if this works... and post a code here.

[zaadstra]

I use(d) to do this with a shell and wget command, then open the downloaded file.
But the example from SMcNeill looks very cool! Especially the timeout will be helpful too.
Looking forward to more examples here! :-)

[SpriggsySpriggs]

I would just use the WinAPI function sets. The easiest one is URLDownloadToFile

[xra7en]

actually I ended up using curl like the rest of my app does. der!!!
 wget I get all kinds of security errors when I use it. Curl works good with the following command:

Code: QB64: [Select]

1. curl -LO -k [url]

I like what McNiel has, but I cannot get it to work. returns a 0 every time. I'll have to tinker with that later. I like it better than curl or wget as it is part of qb64.

[SpriggsySpriggs]

This doesn't require curl or any other external shell program.

Code: QB64: [Select]

1. Declare Dynamic Library "urlmon"
2.     Function URLDownloadToFileA%& (ByVal pCaller As Long, szURL As String, szFileName As String, Byval dwReserved As Long, Byval lpfnCB As Long)
3. End Declare
4.  
5. Function FileDownload%& (URL As String, File As String)
6.     FileDownload = URLDownloadToFileA(0, URL + Chr$(0), File + Chr$(0), 0, 0)
7. End Function

[xra7en]

well that worked amazing simple!! Wish I knew more about API's and how to use them in qb64

I started a progress bar question here: https://www.qb64.org/forum/index.php?topic=3913.0
obviously on larger files I would like to display something... so I know it is not locked up

[xra7en]

hmmm doesn't work all the time

example
using

Code: QB64: [Select]

1. fn$="Git-2.31.1-64-bit.exe"
2. url$ = "https://github.com/git-for-windows/git/releases/download/v2.31.1.windows.1/Git-2.31.1-64-bit.exe"
3. filedownload(url$,fn$)

does nothing. no download

but works flawlessly when downloading a text file.

[SpriggsySpriggs]

If that's the case then you probably need to use a more complex API function set. I have code for this but I'm currently on my phone. I have code that tracks progress as well as downloads any file. It will be a few hours before I can get it to you, though.

[SpriggsySpriggs]

@xra7en

This code allows for downloading of any file type and will track the progress of the download as long as it can query the webpage for the file size. You can edit it however you like for your own needs.

Code: QB64: [Select]

1. Option _Explicit
2.  
3. Const INTERNET_OPEN_TYPE_DIRECT = 1
4.  
5. Const INTERNET_DEFAULT_HTTP_PORT = 80
6. Const INTERNET_DEFAULT_HTTPS_PORT = 443
7.  
8. Const INTERNET_SERVICE_HTTP = 3
9.  
10. 'Flags
11. Const INTERNET_FLAG_SECURE = &H00800000
12. Const INTERNET_FLAG_RELOAD = &H80000000
13.  
14. Const HTTP_QUERY_CONTENT_LENGTH = 5
15.  
16. Const TRUE = 1
17. 'CONST FALSE = 0
18.  
19. Declare Dynamic Library "Wininet"
20.     Function InternetOpen%& Alias "InternetOpenA" (ByVal lpszAgent As _Offset, Byval dwAccessType As Long, Byval lpszProxy As _Offset, Byval lpszProxyBypass As _Offset, Byval dwFlags As Long)
21.     Function InternetConnect%& Alias "InternetConnectA" (ByVal hInternet As _Offset, Byval lpszServerName As _Offset, Byval nServerPort As Integer, Byval lpszUserName As _Offset, Byval lpszPassword As _Offset, Byval dwService As Long, Byval dwFlags As Long, Byval dwContext As _Offset)
22.     Function HTTPOpenRequest%& Alias "HttpOpenRequestA" (ByVal hConnect As _Offset, Byval lpszVerb As _Offset, Byval lpszObjectName As _Offset, Byval lpszVersion As _Offset, Byval lpszReferrer As _Offset, Byval lpszAcceptTypes As _Offset, Byval dwFlags As Long, Byval dwContext As _Offset)
23.     Function HTTPSendRequest%% Alias "HttpSendRequestA" (ByVal hRequest As _Offset, Byval lpszHeaders As _Offset, Byval dwHeadersLength As Long, Byval lpOptional As _Offset, Byval dwOptionalLength As Long)
24.     Sub InternetCloseHandle (ByVal hInternet As _Offset)
25.     Function InternetReadFile%% (ByVal hFile As _Offset, Byval lpBuffer As _Offset, Byval dwNumberOfBytesToRead As Long, Byval lpdwNumberOfBytesRead As _Offset)
26.     Function HTTPQueryInfo%% Alias "HttpQueryInfoA" (ByVal hRequest As _Offset, Byval dwInfoLevel As Long, Byval lpBuffer As _Offset, Byval lpdwBufferLength As _Offset, Byval lpdwIndex As _Offset)
27. End Declare
28.  
29. Declare Dynamic Library "Kernel32"
30.     Function GetLastError& ()
31.     Sub SetLastError (ByVal dwErrCode As Long)
32.     Function FormatMessage& Alias "FormatMessageA" (ByVal dwFlags As Long, Byval lpSource As Long, Byval dwMessageId As Long, Byval dwLanguageId As Long, Byval lpBuffer As _Offset, Byval nSize As Long, Byval Arguments As _Offset)
33. End Declare
34.  
35. Declare CustomType Library ".\offsettostring"
36.     Function offset_to_string$ Alias "offset_to_offset" (ByVal offset As _Offset)
37. End Declare
38.  
39. Declare Library
40.     Function MAKELANGID& (ByVal p As Long, Byval s As Long)
41. End Declare
42.  
43.  
44. Screen _NewImage(480, 80, 32)
45. $ScreenHide
46. _ScreenHide
47. $Console
48. _Dest _Console
49.  
50. _Title "URL Downloader"
51. _ConsoleTitle "Enter Link"
52. Dim link As String
53. Dim filename As String
54.  
55. Do
56.     Cls
57.     Line Input "Link: ", link
58.     Line Input "File Name : ", filename
59. Loop Until link <> "" And filename <> ""
60. _ScreenShow
61. _Title _Title$ + " - " + Mid$(filename, _InStrRev(filename, "\") + 1)
62.  
63. _Console Off
64. _Dest 0
65.  
66. DownloadLink link, filename
67.  
68. Sub DownloadLink (URL As String, File As String)
69.     Dim As String URLFile
70.     URLFile = URL
71.     Dim As _Offset hsession
72.     hsession = InternetOpen(0, INTERNET_OPEN_TYPE_DIRECT, 0, 0, 0)
73.     If hsession = 0 Then
74.         Cls
75.         Print "Error : InternetOpen", ErrorMessage(GetLastError)
76.         InternetCloseHandle hsession
77.         Exit Sub
78.     End If
79.  
80.     Dim As _Offset httpsession
81.     URL = Mid$(URL, InStr(URL, "/") + 2)
82.     URL = Mid$(URL, 1, InStr(URL, "/") - 1)
83.  
84.     httpsession = InternetConnect(hsession, _Offset(URL), INTERNET_DEFAULT_HTTPS_PORT, 0, 0, INTERNET_SERVICE_HTTP, 0, 0)
85.     If httpsession = 0 Then
86.         Cls
87.         Print "Error : Internet Connect", ErrorMessage(GetLastError)
88.         InternetCloseHandle hsession
89.         Exit Sub
90.     End If
91.  
92.     Dim As _Offset httpRequest
93.     Dim As String sessiontype, location, accepttypes
94.     sessiontype = "GET" + Chr$(0)
95.     location = Mid$(URLFile, InStr(URLFile, URL) + Len(URL)) + Chr$(0)
96.     accepttypes = "*/*" + Chr$(0)
97.     httpRequest = HTTPOpenRequest(httpsession, _Offset(sessiontype), _Offset(location), 0, 0, _Offset(accepttypes), INTERNET_FLAG_RELOAD Or INTERNET_FLAG_SECURE, 0)
98.     If httpRequest = 0 Then
99.         Cls
100.         Print "Error : HTTPOpenRequest", ErrorMessage(GetLastError)
101.         InternetCloseHandle hsession
102.         Exit Sub
103.     End If
104.  
105.     Dim As Long sendrequest
106.     Dim As String headers
107.     headers = ""
108.     sendrequest = HTTPSendRequest(httpRequest, 0, 0, 0, 0)
109.     If sendrequest <> TRUE Then
110.         Cls
111.         Print "Error : HTTPSendRequest", ErrorMessage(GetLastError)
112.         InternetCloseHandle hsession
113.         Exit Sub
114.     End If
115.  
116.  
117.     Dim As _Byte query
118.     Dim As String queryinfo
119.     queryinfo = Space$(1024)
120.     Dim As Long querylen
121.     querylen = Len(queryinfo) - 1
122.  
123.     query = HTTPQueryInfo(httpRequest, HTTP_QUERY_CONTENT_LENGTH, _Offset(queryinfo), _Offset(querylen), 0)
124.     If query <> TRUE Then
125.         Cls
126.         Print "Error : HTTPQueryInfo", ErrorMessage(GetLastError)
127.         InternetCloseHandle hsession
128.     End If
129.  
130.     Dim As Long bytesToRead
131.     bytesToRead = Val(queryinfo)
132.  
133.     Dim As String szBuffer
134.     szBuffer = Space$(4097)
135.     Dim As _Unsigned Long dwRead, bytesRead
136.     If _FileExists(File) Then
137.         Kill File
138.     End If
139.     Open File For Binary As #1
140.     Dim As _Byte a
141.     Dim As String filedownload
142.     Dim As Long errr, bytesForRate
143.     Dim x!
144.     Dim y!
145.     Dim Rate!
146.     Dim As Single ratetime
147.     Do
148.         x! = Timer
149.         a = InternetReadFile(httpRequest, _Offset(szBuffer), Len(szBuffer) - 1, _Offset(dwRead))
150.         errr = GetLastError
151.         If dwRead > 0 Then
152.             filedownload = Mid$(szBuffer, 1, dwRead)
153.             Put #1, , filedownload
154.             bytesRead = bytesRead + dwRead
155.             bytesForRate = bytesForRate + dwRead
156.             ratetime = timeElapsedSince(x!)
157.             If _Round(ratetime) >= 1 Then
158.                 Rate! = (bytesForRate / ratetime) / 1024
159.                 bytesForRate = 0
160.             End If
161.             Cls
162.             Print "Downloading to " + File
163.             If bytesToRead <> 0 Then
164.                 Print bytesRead; "bytes downloaded of"; bytesToRead
165.                 Print Using "###.##%"; bytesRead / bytesToRead * 100
166.             Else
167.                 Select Case bytesRead
168.                     Case Is < 1024
169.                         Print Using "   ####  B downloaded"; bytesRead
170.                     Case Is < (1024 ^ 2) And bytesRead >= 1024
171.                         Print Using "####.## KB downloaded"; (bytesRead / 1024)
172.                     Case Is < (1024 ^ 3) And bytesRead >= (1024 ^ 2)
173.                         Print Using "####.## MB downloaded"; (bytesRead / (1024 ^ 2))
174.                     Case Is < (1024 ^ 4) And bytesRead >= (1024 ^ 3)
175.                         Print Using "####.## GB downloaded"; (bytesRead / (1024 ^ 3))
176.                 End Select
177.             End If
178.             Print "Rate="; _Round(Rate!); "KBps"
179.             _Display
180.         End If
181.     Loop Until bytesRead = bytesToRead Or errr <> 0
182.     If errr Then
183.         Print "Error downloading file:"; errr
184.         Close #1
185.         InternetCloseHandle hsession
186.         Kill File
187.         Exit Sub
188.     Else
189.     End If
190.     Close #1
191.     InternetCloseHandle hsession
192.     Cls
193.     Print bytesRead; "bytes downloaded of"; bytesToRead
194.     Print Using "###.##%"; bytesRead / bytesToRead * 100
195.     Print "Downloaded to " + File
196. End Sub
197.  
198. Function timeElapsedSince! (startTime!)
199.     If startTime! > Timer Then startTime! = startTime! - 86400
200.     timeElapsedSince! = Timer - startTime!
201. End Function
202.  
203. Function ErrorMessage$ (errCode As Long)
204.     Const FORMAT_MESSAGE_ALLOCATE_BUFFER = &H00000100
205.     Const FORMAT_MESSAGE_FROM_SYSTEM = &H00001000
206.     Const FORMAT_MESSAGE_IGNORE_INSERTS = &H00000200
207.  
208.     Const LANG_NEUTRAL = &H00
209.     Const SUBLANG_DEFAULT = &H01
210.  
211.     Dim As _Offset lpMsgBuf
212.     Dim As Long msg
213.  
214. msg = FormatMessage(FORMAT_MESSAGE_ALLOCATE_BUFFER Or _
215.                     FORMAT_MESSAGE_FROM_SYSTEM Or _
216.                     FORMAT_MESSAGE_IGNORE_INSERTS, _
217.                    0, _
218.                     errCode, _
219.                     MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), _
220.                     _Offset(lpMsgBuf), _
221.                    0, 0)
222.  
223.  
224.     ErrorMessage = offset_to_string(lpMsgBuf)
225. End Function

Code: C++: [Select]

1.  //save as offsettostring.h
2. void *offset_to_offset(void* offset){
3.    return offset;
4. }

[xra7en]

YOU KNOW you are giving a new meaning to "BASIC" LOL
Im starting to feel like I am programming in c++



thanks I'll tinker with that next week - stuck on call this week grrrrr

[SpriggsySpriggs]

@xra7en Hahaha yeah it's basic to me but not everyone :)

It is very similar to programming in C++ because all of those functions are from C DLLs and that header file is a C++ file.

[xra7en]

ok. been playing around with this api, fun.

keep getting this error o.O

Code: QB64: [Select]

1. Error: HTTPOpenRequest  The data area passed to a system  call is too small

filename: npp.7.9.5.Installer.x64.exe
link: https://github.com/notepad-plus-plus/notepad-plus-plus/releases/download/v7.9.5/npp.7.9.5.Installer.x64.exe

Now this is all greek to me, however like the internal ability. so learning