Need help using QB64 to access a webservice

[FilipeEstima]

Hey fellows.

I can work myself around QB64 on doing trivial things, but now I need to learn if it's possible to make online requests to a webservice (and how to do it). The webservice in question is provided by the brazilian mail company Correios and is used by online stores to send information about ZIP origin and destination, weight, dimensions etc; in return the e-commerce gets estimated delivery days and prices for the services asked. The results come in XML format, which I should be able to interpret.

My main problem is, how do go I about connecting my QB64 software to the webservice? I don't know anything about http protocols.

[FellippeHeitor]

What web address must you query?

[FilipeEstima]

Hi Fellippe, how are you?

This one:

http://ws.correios.com.br/calculador/CalcPrecoPrazo.asmx?op=CalcPrecoPrazo

And this one:

http://ws.correios.com.br/calculador/CalcPrecoPrazo.asmx?op=CalcPrazo

I can only get you an example of the second, as I don't yet understand how to put all the needed arguments on the first one.

http://ws.correios.com.br/calculador/CalcPrecoPrazo.asmx/CalcPrazo?nCdServico=10154&sCepOrigem=69910000&sCepDestino=99920000

[FilipeEstima]

Here's the manual on how to make such queries:

https://correios.com.br/solucoes-empresariais/comercio-eletronico/palestras-correios-1/pdf/ManualdeImplementacaodoCalculoRemotodePrecosePrazos.pdf

[FilipeEstima]

I got it. To make queries for price and delivery days:

http://ws.correios.com.br/calculador/CalcPrecoPrazo.aspx?sCepOrigem=70002900&sCepDestino=71939360&nVlPeso=1&nCdFormato=1&nVlComprimento=20&nVlAltura=20&nVlLargura=20&sCdMaoPropria=n&nVlValorDeclarado=0&sCdAvisoRecebimento=n&nCdServico=04510,04014&nVlDiametro=0&StrRetorno=xml

Still I don't understand how to make such queries from within a QB64 program, and receive the result in XML, so I can make a routine to gather the useful data.

[FellippeHeitor]

You can use the download function Galleon wrote (I'm pasting it here from the wiki, adapted to your case):

Code: QB64: [Select]

1. server$ = "ws.correios.com.br"
2. query$ = "/calculador/CalcPrecoPrazo.asmx/CalcPrazo?nCdServico=10154&sCepOrigem=69910000&sCepDestino=99920000"
3.  
4. PRINT "Attempting to fetch data..."
5.  
6. IF Download(server$ + query$, "correios.xml", 10) THEN ' timelimit = 10 seconds
7.     COLOR 14: PRINT "Success!": COLOR 15
8.     OPEN "correios.xml" FOR BINARY AS #1
9.     DO
10.         IF EOF(1) THEN EXIT DO
11.         LINE INPUT #1, dataLine$
12.         PRINT dataLine$
13.     LOOP
14.     CLOSE #1
15. ELSE
16.     PRINT "Download failed."
17. END IF
18.  
19. FUNCTION Download (url$, file$, timelimit) ' returns -1 if successful, 0 if not
20.     url2$ = url$
21.     x = INSTR(url2$, "/")
22.     IF x THEN url2$ = LEFT$(url$, x - 1)
23.     client = _OPENCLIENT("TCP/IP:80:" + url2$)
24.     IF client = 0 THEN EXIT FUNCTION
25.     e$ = CHR$(13) + CHR$(10) ' end of line characters
26.     url3$ = RIGHT$(url$, LEN(url$) - x + 1)
27.     x$ = "GET " + url3$ + " HTTP/1.1" + e$
28.     x$ = x$ + "Host: " + url2$ + e$ + e$
29.     PUT #client, , x$
30.     t! = TIMER ' start time
31.     DO
32.         _DELAY 0.05 ' 50ms delay (20 checks per second)
33.         GET #client, , a2$
34.         a$ = a$ + a2$
35.         i = INSTR(a$, "Content-Length:")
36.         IF i THEN
37.             i2 = INSTR(i, a$, e$)
38.             IF i2 THEN
39.                 l = VAL(MID$(a$, i + 15, i2 - i - 14))
40.                 i3 = INSTR(i2, a$, e$ + e$)
41.                 IF i3 THEN
42.                     i3 = i3 + 4 'move i3 to start of data
43.                     IF (LEN(a$) - i3 + 1) = l THEN
44.                         CLOSE client ' CLOSE CLIENT
45.                         d$ = MID$(a$, i3, l)
46.                         fh = FREEFILE
47.                         OPEN file$ FOR OUTPUT AS #fh: CLOSE #fh 'Warning! Clears data from existing file
48.                         OPEN file$ FOR BINARY AS #fh
49.                         PUT #fh, , d$
50.                         CLOSE #fh
51.                         Download = -1 'indicates download was successfull
52.                         EXIT FUNCTION
53.                     END IF ' availabledata = l
54.                 END IF ' i3
55.             END IF ' i2
56.         END IF ' i
57.     LOOP UNTIL TIMER > t! + timelimit ' (in seconds)
58.     CLOSE client
59. END FUNCTION

Then all you need to do is build the query$ string and send the download request. The resulting file is your XML to parse.

[FilipeEstima]

Thank you, I'll  give it a try and let you know if I got it right.

[FilipeEstima]

Hi Fellippe, while I waited for a reply here, I found that curl could be used as well. With curl I must direct the result to a file, then I have to open that file and examine it. Since I'm going to make a lot of queries, this approach isn't optimal.

The solution you provided is much better, because if I simply make d$ shared, I can comment the lines that save the query result to a file and work directly with d$, therefore much faster because it doesn't involve the OS saving a new file over and over again. Thanks!

[FellippeHeitor]

Good to hear you managed to improve the solution. Best of luck with your new program!

[FilipeEstima]

I finally made the program I wanted, thanks to your help. When running it, I noticed the time it takes to finish the job is quite long. It has to make about 6.000 queries and it takes almost 4 hours to finish. So I divided the list in smaller parts and made 8 versions of the program, each one responsible for a part of the total list.

That's a hell of an ugly solution; now I'm back to ask for ideas. I suppose there's no way for a single instance of the program to make many parallel queries, right? By many I mean 8 or more in parallel, instead of serial queries.

Another solution would be to launch one program ("master") that would call many copies of another program ("servant"), providing each one with "start" and "end". In this case the "servant" program contains all the necessary information, and each instance would use only the portion between "start" and "end" as determined by the "master". So instead of 6.000 serialized queries, I'd have 750 queries in parallel (8 at the same time).

Maybe I made it sound more confusing that it actually is LOL

[FellippeHeitor]

I don't know what the limit for _OPENCLIENTs is, but you surely could tweak the Download function to work with more than one connection at once. I imagine it'd take an array to store client numbers and separate variables to store incoming data, but it sure sounds doable.

If you get the latest development build you can have TYPEs with variable width strings. Something like

TYPE Connection
    Client AS LONG
    ReceivedData AS STRING
END TYPE

DIM SHARED connection(100) AS Connection

Then work the function to keep track of those.

[Cobalt]

If you get the latest development build you can have TYPEs with variable width strings. Something like

Did Luke get that ironed out? last I knew we couldn't save it correctly directly with PUT ##,,UDT_array
or do you mean for him to reassign the UDT string to another string and then save?

[FellippeHeitor]

This is direct run time manipulation, no file saving involved.

Due to the nature of variable width types, QB64 will prevent you from using them with Get and Put, as previously discussed.

[luke]

If you get the latest development build you can have TYPEs with variable width strings. Something like

Did Luke get that ironed out? last I knew we couldn't save it correctly directly with PUT ##,,UDT_array
or do you mean for him to reassign the UDT string to another string and then save?

For now you can't use UDT's with variable length strings in GET or PUT operations, but they're perfectly functional otherwise (I just fixed the issue it had with LEN).