Text Only | Text with Attachments

QB64.org Forum

Active Forums => QB64 Discussion => Topic started by: xra7en on May 15, 2021, 11:24:26 am

Title: reading remote text file
Post by: xra7en on May 15, 2021, 11:24:26 am
I thought I saw a thread here awhile ago regarding this, might have been a different language.

can qb64 read a remote txtfile?

example mycoolsite.com/textfile.txt

was trying (of course unsuccessfully) the open command.

thanks guys

ps
Its the weekend, working on another goofy project.

Title: Re: reading remote text file
Post by: 191Brian on May 15, 2021, 03:36:28 pm
Pretty sure the answer is no, open only works on files local to you either on your PC or network share.
Title: Re: reading remote text file
Post by: SMcNeill on May 15, 2021, 03:55:06 pm
You can look at the QB64 wiki download program I have here, as an example, if you’re just transfering over http and not https: https://www.qb64.org/forum/index.php?topic=756.msg6455#msg6455

Basically, use _OPENCLIENT and GET to do the work:

SUB DownloadRSS (url$, file$, timelimit)
    link$ = url$
    DIM l AS _INTEGER64, lod AS _INTEGER64
 
    url2$ = RTRIM$(LTRIM$(link$))
    url4$ = RTRIM$(LTRIM$(link$))
    IF LEFT$(UCASE$(url2$), 7) = "HTTP://" THEN url4$ = MID$(url2$, 8)
    x = INSTR(url4$, "/")
    IF x THEN url2$ = LEFT$(url4$, x - 1)
    NewsClient = _OPENCLIENT("TCP/IP:80:" + url2$)
    IF NewsClient = 0 THEN EXIT FUNCTION
    e$ = CHR$(13) + CHR$(10) ' end of line characters
    url3$ = RIGHT$(url4$, LEN(url4$) - x + 1)
    x$ = "GET " + url3$ + " HTTP/1.1" + e$
    x$ = x$ + "Host: " + url2$ + e$ + e$
    PUT #NewsClient, , x$
 
    OPEN file$ FOR OUTPUT AS #1: CLOSE #1
    OPEN file$ FOR BINARY AS #1
 
    t! = TIMER ' start time
    head$ = ""
    cont_type$ = ""
    DO
        _LIMIT 20
        GET #NewsClient, , a$
        IF LTRIM$(a$) > "" THEN PUT #1, , a$
    LOOP UNTIL TIMER > t! + timelimit AND timelimit > 0 ' (in seconds)
    CLOSE #NewsClient
    CLOSE #1
END SUB
Title: Re: reading remote text file
Post by: xra7en on May 15, 2021, 04:41:17 pm
I checked the wiki with no luck, but only because I had no direction - "open" was all I knew.

This is good, I will part it out and see how it ticks!!

what I am doing is pulling a simple text file from one of my sites, I could just turn off the SSL and from what you said it should work.

Out of necessity, I wrote a "chocolatey" alternative (qb64), but much more simpler, been using it for last few years - I wanted something simple without complexity. works great. however I want a centralized DB instead of locally. So trying the old fashion text method - it works, and does not need something advanced as SQL(programing sql for remote access is an overkill for what I want). I basically just keep track of appname and link to it directly.



thanks!

I'll let you know if this works... and post a code here.
Title: Re: reading remote text file
Post by: zaadstra on May 15, 2021, 05:00:00 pm
I use(d) to do this with a shell and wget command, then open the downloaded file.
But the example from SMcNeill looks very cool! Especially the timeout will be helpful too.
Looking forward to more examples here! :-)
Title: Re: reading remote text file
Post by: SpriggsySpriggs on May 15, 2021, 05:02:43 pm
I would just use the WinAPI function sets. The easiest one is URLDownloadToFile
Title: Re: reading remote text file
Post by: xra7en on May 15, 2021, 05:18:13 pm
actually I ended up using curl like the rest of my app does. der!!!
 wget I get all kinds of security errors when I use it. Curl works good with the following command:

Code: QB64: [Select]
  1. curl -LO -k [url]

I like what McNiel has, but I cannot get it to work. returns a 0 every time. I'll have to tinker with that later. I like it better than curl or wget as it is part of qb64.
Title: Re: reading remote text file
Post by: SpriggsySpriggs on May 15, 2021, 06:29:25 pm
This doesn't require curl or any other external shell program.

Code: QB64: [Select]
  2.     Function URLDownloadToFileA%& (ByVal pCaller As Long, szURL As String, szFileName As String, Byval dwReserved As Long, Byval lpfnCB As Long)
  4.  
  5. Function FileDownload%& (URL As String, File As String)
  6.     FileDownload = URLDownloadToFileA(0, URL + Chr$(0), File + Chr$(0), 0, 0)
Title: Re: reading remote text file
Post by: xra7en on May 16, 2021, 07:40:35 am
well that worked amazing simple!! Wish I knew more about API's and how to use them in qb64

I started a progress bar question here: https://www.qb64.org/forum/index.php?topic=3913.0
obviously on larger files I would like to display something... so I know it is not locked up
Title: Re: reading remote text file
Post by: xra7en on May 16, 2021, 10:44:47 am
hmmm doesn't work all the time

example
using

Code: QB64: [Select]
  1. fn$="Git-2.31.1-64-bit.exe"
  2. url$ = "https://github.com/git-for-windows/git/releases/download/v2.31.1.windows.1/Git-2.31.1-64-bit.exe"
  3. filedownload(url$,fn$)

does nothing. no download

but works flawlessly when downloading a text file.
Title: Re: reading remote text file
Post by: SpriggsySpriggs on May 16, 2021, 10:59:54 am
If that's the case then you probably need to use a more complex API function set. I have code for this but I'm currently on my phone. I have code that tracks progress as well as downloads any file. It will be a few hours before I can get it to you, though.
Title: Re: reading remote text file
Post by: SpriggsySpriggs on May 16, 2021, 03:39:03 pm
@xra7en

This code allows for downloading of any file type and will track the progress of the download as long as it can query the webpage for the file size. You can edit it however you like for your own needs.

Code: QB64: [Select]
  2.  
  3. Const INTERNET_OPEN_TYPE_DIRECT = 1
  4.  
  5. Const INTERNET_DEFAULT_HTTP_PORT = 80
  6. Const INTERNET_DEFAULT_HTTPS_PORT = 443
  7.  
  8. Const INTERNET_SERVICE_HTTP = 3
  9.  
  10. 'Flags
  11. Const INTERNET_FLAG_SECURE = &H00800000
  12. Const INTERNET_FLAG_RELOAD = &H80000000
  13.  
  14. Const HTTP_QUERY_CONTENT_LENGTH = 5
  15.  
  16. Const TRUE = 1
  17. 'CONST FALSE = 0
  18.  
  20.     Function InternetOpen%& Alias "InternetOpenA" (ByVal lpszAgent As _Offset, Byval dwAccessType As Long, Byval lpszProxy As _Offset, Byval lpszProxyBypass As _Offset, Byval dwFlags As Long)
  21.     Function InternetConnect%& Alias "InternetConnectA" (ByVal hInternet As _Offset, Byval lpszServerName As _Offset, Byval nServerPort As Integer, Byval lpszUserName As _Offset, Byval lpszPassword As _Offset, Byval dwService As Long, Byval dwFlags As Long, Byval dwContext As _Offset)
  22.     Function HTTPOpenRequest%& Alias "HttpOpenRequestA" (ByVal hConnect As _Offset, Byval lpszVerb As _Offset, Byval lpszObjectName As _Offset, Byval lpszVersion As _Offset, Byval lpszReferrer As _Offset, Byval lpszAcceptTypes As _Offset, Byval dwFlags As Long, Byval dwContext As _Offset)
  23.     Function HTTPSendRequest%% Alias "HttpSendRequestA" (ByVal hRequest As _Offset, Byval lpszHeaders As _Offset, Byval dwHeadersLength As Long, Byval lpOptional As _Offset, Byval dwOptionalLength As Long)
  24.     Sub InternetCloseHandle (ByVal hInternet As _Offset)
  25.     Function InternetReadFile%% (ByVal hFile As _Offset, Byval lpBuffer As _Offset, Byval dwNumberOfBytesToRead As Long, Byval lpdwNumberOfBytesRead As _Offset)
  26.     Function HTTPQueryInfo%% Alias "HttpQueryInfoA" (ByVal hRequest As _Offset, Byval dwInfoLevel As Long, Byval lpBuffer As _Offset, Byval lpdwBufferLength As _Offset, Byval lpdwIndex As _Offset)
  28.  
  30.     Function GetLastError& ()
  31.     Sub SetLastError (ByVal dwErrCode As Long)
  32.     Function FormatMessage& Alias "FormatMessageA" (ByVal dwFlags As Long, Byval lpSource As Long, Byval dwMessageId As Long, Byval dwLanguageId As Long, Byval lpBuffer As _Offset, Byval nSize As Long, Byval Arguments As _Offset)
  34.  
  35. Declare CustomType Library ".\offsettostring"
  36.     Function offset_to_string$ Alias "offset_to_offset" (ByVal offset As _Offset)
  38.  
  40.     Function MAKELANGID& (ByVal p As Long, Byval s As Long)
  42.  
  43.  
  44. Screen _NewImage(480, 80, 32)
  49.  
  50. _Title "URL Downloader"
  51. _ConsoleTitle "Enter Link"
  53. Dim filename As String
  54.  
  56.     Cls
  57.     Line Input "Link: ", link
  58.     Line Input "File Name : ", filename
  59. Loop Until link <> "" And filename <> ""
  61. _Title _Title$ + " - " + Mid$(filename, _InStrRev(filename, "\") + 1)
  62.  
  65.  
  66. DownloadLink link, filename
  67.  
  68. Sub DownloadLink (URL As String, File As String)
  69.     Dim As String URLFile
  70.     URLFile = URL
  71.     Dim As _Offset hsession
  72.     hsession = InternetOpen(0, INTERNET_OPEN_TYPE_DIRECT, 0, 0, 0)
  73.     If hsession = 0 Then
  74.         Cls
  75.         Print "Error : InternetOpen", ErrorMessage(GetLastError)
  76.         InternetCloseHandle hsession
  77.         Exit Sub
  78.     End If
  79.  
  80.     Dim As _Offset httpsession
  81.     URL = Mid$(URL, InStr(URL, "/") + 2)
  82.     URL = Mid$(URL, 1, InStr(URL, "/") - 1)
  83.  
  84.     httpsession = InternetConnect(hsession, _Offset(URL), INTERNET_DEFAULT_HTTPS_PORT, 0, 0, INTERNET_SERVICE_HTTP, 0, 0)
  85.     If httpsession = 0 Then
  86.         Cls
  87.         Print "Error : Internet Connect", ErrorMessage(GetLastError)
  88.         InternetCloseHandle hsession
  89.         Exit Sub
  90.     End If
  91.  
  92.     Dim As _Offset httpRequest
  93.     Dim As String sessiontype, location, accepttypes
  94.     sessiontype = "GET" + Chr$(0)
  95.     location = Mid$(URLFile, InStr(URLFile, URL) + Len(URL)) + Chr$(0)
  96.     accepttypes = "*/*" + Chr$(0)
  97.     httpRequest = HTTPOpenRequest(httpsession, _Offset(sessiontype), _Offset(location), 0, 0, _Offset(accepttypes), INTERNET_FLAG_RELOAD Or INTERNET_FLAG_SECURE, 0)
  98.     If httpRequest = 0 Then
  99.         Cls
  100.         Print "Error : HTTPOpenRequest", ErrorMessage(GetLastError)
  101.         InternetCloseHandle hsession
  102.         Exit Sub
  103.     End If
  104.  
  105.     Dim As Long sendrequest
  106.     Dim As String headers
  107.     headers = ""
  108.     sendrequest = HTTPSendRequest(httpRequest, 0, 0, 0, 0)
  109.     If sendrequest <> TRUE Then
  110.         Cls
  111.         Print "Error : HTTPSendRequest", ErrorMessage(GetLastError)
  112.         InternetCloseHandle hsession
  113.         Exit Sub
  114.     End If
  115.  
  116.  
  117.     Dim As _Byte query
  118.     Dim As String queryinfo
  119.     queryinfo = Space$(1024)
  120.     Dim As Long querylen
  121.     querylen = Len(queryinfo) - 1
  122.  
  123.     query = HTTPQueryInfo(httpRequest, HTTP_QUERY_CONTENT_LENGTH, _Offset(queryinfo), _Offset(querylen), 0)
  124.     If query <> TRUE Then
  125.         Cls
  126.         Print "Error : HTTPQueryInfo", ErrorMessage(GetLastError)
  127.         InternetCloseHandle hsession
  128.     End If
  129.  
  130.     Dim As Long bytesToRead
  131.     bytesToRead = Val(queryinfo)
  132.  
  133.     Dim As String szBuffer
  134.     szBuffer = Space$(4097)
  135.     Dim As _Unsigned Long dwRead, bytesRead
  136.     If _FileExists(File) Then
  137.         Kill File
  138.     End If
  139.     Open File For Binary As #1
  140.     Dim As _Byte a
  141.     Dim As String filedownload
  142.     Dim As Long errr, bytesForRate
  143.     Dim x!
  144.     Dim y!
  145.     Dim Rate!
  146.     Dim As Single ratetime
  147.     Do
  148.         x! = Timer
  149.         a = InternetReadFile(httpRequest, _Offset(szBuffer), Len(szBuffer) - 1, _Offset(dwRead))
  150.         errr = GetLastError
  151.         If dwRead > 0 Then
  152.             filedownload = Mid$(szBuffer, 1, dwRead)
  153.             Put #1, , filedownload
  154.             bytesRead = bytesRead + dwRead
  155.             bytesForRate = bytesForRate + dwRead
  156.             ratetime = timeElapsedSince(x!)
  157.             If _Round(ratetime) >= 1 Then
  158.                 Rate! = (bytesForRate / ratetime) / 1024
  159.                 bytesForRate = 0
  160.             End If
  161.             Cls
  162.             Print "Downloading to " + File
  163.             If bytesToRead <> 0 Then
  164.                 Print bytesRead; "bytes downloaded of"; bytesToRead
  165.                 Print Using "###.##%"; bytesRead / bytesToRead * 100
  166.             Else
  167.                 Select Case bytesRead
  168.                     Case Is < 1024
  169.                         Print Using "   ####  B downloaded"; bytesRead
  170.                     Case Is < (1024 ^ 2) And bytesRead >= 1024
  171.                         Print Using "####.## KB downloaded"; (bytesRead / 1024)
  172.                     Case Is < (1024 ^ 3) And bytesRead >= (1024 ^ 2)
  173.                         Print Using "####.## MB downloaded"; (bytesRead / (1024 ^ 2))
  174.                     Case Is < (1024 ^ 4) And bytesRead >= (1024 ^ 3)
  175.                         Print Using "####.## GB downloaded"; (bytesRead / (1024 ^ 3))
  176.                 End Select
  177.             End If
  178.             Print "Rate="; _Round(Rate!); "KBps"
  179.             _Display
  180.         End If
  181.     Loop Until bytesRead = bytesToRead Or errr <> 0
  182.     If errr Then
  183.         Print "Error downloading file:"; errr
  184.         Close #1
  185.         InternetCloseHandle hsession
  186.         Kill File
  187.         Exit Sub
  188.     Else
  189.     End If
  190.     Close #1
  191.     InternetCloseHandle hsession
  192.     Cls
  193.     Print bytesRead; "bytes downloaded of"; bytesToRead
  194.     Print Using "###.##%"; bytesRead / bytesToRead * 100
  195.     Print "Downloaded to " + File
  197.  
  198. Function timeElapsedSince! (startTime!)
  199.     If startTime! > Timer Then startTime! = startTime! - 86400
  200.     timeElapsedSince! = Timer - startTime!
  202.  
  203. Function ErrorMessage$ (errCode As Long)
  204.     Const FORMAT_MESSAGE_ALLOCATE_BUFFER = &H00000100
  205.     Const FORMAT_MESSAGE_FROM_SYSTEM = &H00001000
  206.     Const FORMAT_MESSAGE_IGNORE_INSERTS = &H00000200
  207.  
  208.     Const LANG_NEUTRAL = &H00
  209.     Const SUBLANG_DEFAULT = &H01
  210.  
  211.     Dim As _Offset lpMsgBuf
  212.     Dim As Long msg
  213.  
  214. msg = FormatMessage(FORMAT_MESSAGE_ALLOCATE_BUFFER Or _
  215.                     FORMAT_MESSAGE_FROM_SYSTEM Or _
  216.                     FORMAT_MESSAGE_IGNORE_INSERTS, _
  217.                    0, _
  218.                     errCode, _
  219.                     MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), _
  220.                     _Offset(lpMsgBuf), _
  221.                    0, 0)
  222.  
  223.  
  224.     ErrorMessage = offset_to_string(lpMsgBuf)

Code: C++: [Select]
  1.  //save as offsettostring.h
  2. void *offset_to_offset(void* offset){
  3.    return offset;
  4. }
Title: Re: reading remote text file
Post by: xra7en on May 16, 2021, 08:32:12 pm
YOU KNOW you are giving a new meaning to "BASIC" LOL
Im starting to feel like I am programming in c++



thanks I'll tinker with that next week - stuck on call this week grrrrr
Title: Re: reading remote text file
Post by: SpriggsySpriggs on May 16, 2021, 08:49:08 pm
@xra7en Hahaha yeah it's basic to me but not everyone :)

It is very similar to programming in C++ because all of those functions are from C DLLs and that header file is a C++ file.
Title: Re: reading remote text file
Post by: xra7en on May 22, 2021, 04:15:43 pm
ok. been playing around with this api, fun.

keep getting this error o.O
Code: QB64: [Select]
  1. Error: HTTPOpenRequest  The data area passed to a system  call is too small


Quote
filename: npp.7.9.5.Installer.x64.exe
link: https://github.com/notepad-plus-plus/notepad-plus-plus/releases/download/v7.9.5/npp.7.9.5.Installer.x64.exe

Now this is all greek to me, however like the internal ability. so learning
Title: Re: reading remote text file
Post by: xra7en on May 22, 2021, 04:19:26 pm
trying some research:
found that this error needs to be addressed by using the
InternetCrackUrlA()

https://docs.microsoft.com/en-us/windows/win32/api/wininet/nf-wininet-internetcrackurla

apparently I have to break down the url before sending it?
Title: Re: reading remote text file
Post by: SpriggsySpriggs on May 22, 2021, 04:19:39 pm
@xra7en Just for me to do some testing:

Title: Re: reading remote text file
Post by: SpriggsySpriggs on May 22, 2021, 04:21:33 pm
In my code I gave you, my code already breaks the URL down into location and base URL. You'll see some MID$ stuff before I do some calls
Title: Re: reading remote text file
Post by: xra7en on May 22, 2021, 04:29:46 pm
oops I did not do any editing...

lemme tinker with that. I see you did break it down. I want to see if I can solved this (hints are helpful - sorta like playing portal.. point the direction but not what portal to what wall haha)
Title: Re: reading remote text file
Post by: xra7en on May 23, 2021, 04:45:06 pm
Ok dont see anything I need to change. I actually found your video on youtube - LOL. and your download worked fine. so it might be just how I am parsing the download

Which is the correct way:

download https://www.cooldownloadsite.com/download/filename.zip, filename.zip
or
https://www.cooldownloadsite.com/download, filename.zip

My curent setup is the first option.

Text Only | Text with Attachments