Active Forums => QB64 Discussion => Topic started by: Parkland on April 15, 2020, 12:52:54 am
Title: put / get buffer binary mode tcp
Post by: Parkland on April 15, 2020, 12:52:54 am
Ok, I made a program which is a transfer program for what I'm making, a peer to peer ap. so the program so far (not 100% done) can copy files disk to disk, download from network to disk, upload from disk to network. It has 1000 transfer command queue capacity, but the program only runs 32 at once and advances the queue as they complete.
So far, I am using binary mode, and get and put, 1 byte at a time.
I am curious; the beginning of the file I transfer sometimes is missing data in the beginning, I made the program wait for 1 second before the data starts transferring. Other than the beginning, it's all intact.
I am wondering though, is this data protected by tcp/ip for errors and buffering? If the connection drops or slows down, does the sending computer pause at the put command until a value is read? Or does it just keep sending bytes and they're lost?
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 15, 2020, 01:23:52 am
I guess I'm not even sure what I'm asking. I'm curious about how the qb64 interfaces with the network protocol I guess. I have 1 computer with qb64 and a program running, sending a file, byte by byte using put #a, ,x$
and another computer with qb64 and another program using get #a,, x$ ; and adding x$ to a file in binary mode.
So, what happens when the one computer sends faster than the other recieves? does qb64 automatically slow down the sending program to not overload the tcp/ip stack? Or does data start getting lost?
And is this the right way to send data? Is the output bytes filling a buffer and sending as a stream of packets? Or is it triggering a packet for every single byte and wasting tons of network bandwidth?
Title: Re: put / get buffer binary mode tcp
Post by: Unseen Machine on April 15, 2020, 04:43:47 am
TCP/IP gurantees the data and the data order but doesnt work very fast. UDP is WAYYYYYYY faster bu you have to manage the data as the order an actual data transfer is not guranteed (hence it's nickname being the Usless Datagram Protocol!)
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 15, 2020, 11:44:07 pm
OK interesting.
so using TCP/IP in qb64 (the only internet connectivity method available), a person shouldn't have to worry about error correction etc.
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 16, 2020, 12:23:04 am
[ This attachment cannot be displayed inline in 'Print Page' view ] [ This attachment cannot be displayed inline in 'Print Page' view ]
Ok did some experiments over a wireless connection to my laptop, and a desktop computer sending the data. I tried sending 1 byte at a time, 4 bytes, then 8. Sending 1 byte per PUT instruction, you can see that the sending computer is slower than the wifi speed. Sending 4 bytes per instruction is faster. Over 8 bytes per PUT on the sending computer makes no difference.
Interesting I was wondering if sending 1 byte at a time was wasteful and inefficient, but it seems to have rather low network overhead regardless.
Unfortunately, qb64 really uses a lot of CPU to move data at just 3 megabytes per second. Maybe because of the PUT and GET instructions as opposed to using INPUT and PRINT with larger strings.
** the first pic is 1 byte at a time, 2nd pic is 4 bytes at a time. *** I used 1 kb=1024 bytes, 1 MB=1024kb
Sending computer just sends a character using out. Recieving computer adds the byte to a string, then displays string length every second. (and erases string)
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 16, 2020, 04:08:14 am
Ok, I found some things out sending and receiving data in binary modes over tcpip.
The string size can be increased to raise data rate and lower CPU usage. However, there seems to be data lost sometimes :(
When i add delay in the sending computer program, the file is the right size (240mb) When I remove the delay instruction , the file on the receive computer ends up being a random (smaller) size.
So, I'm not sure. Could have 2 connections open? 1 to transfer data and the other one to link back whats received? In the system task monitor, it is uploading data while downloading, so it *seems* like qb64 should already be doing this, but maybe not?
Title: Re: put / get buffer binary mode tcp
Post by: FellippeHeitor on April 16, 2020, 07:29:30 am
Connections are bidirectional, you can get and put simultaneously.
Amount of data is controlled by the system, just send away. The receiving end should get the data sequentially until an agreed end marker is received.
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 16, 2020, 07:45:39 am
.... How come by slowing the sending computer down , the receiving computer is getting all the information, and if I let the sending computer run full speed, information is lost :(
...I'm still experimenting, maybe I'll figure it out yet
Title: Re: put / get buffer binary mode tcp
Post by: FellippeHeitor on April 16, 2020, 08:01:27 am
I suspect your GET side is not storing things properly in a buffer:
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 16, 2020, 08:18:03 am
This gets repeated until lpc = length of file, which is exchanged before the file is transffered.
GET fn1(x), , i$ (reading from network) PUT fn2(x), , i$ (saving to file) lpc= lpc + LEN(i$)
The sending computer is a desktop with ethernet to switch. Laptop is the reciever on WIFI. The problem seems to be that this works at low data rate (1 byte per PUT) it works as it should. I can increase bytes per PUT until the network backs up, then I seem to recieve less than is being sent. Adding delay in sending computer program fixes it.
Also the send program is very simple, recieve program is huge and slower. I checked, and the recieve string is sometimes up over 1000 bytes, so lots slower than the sending program.
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 16, 2020, 08:37:19 am
Could I be doing something wrong on the sending computer? It looks something like:
DIM a as STRING * (2 works, playing with higher numbers) bt=2 (or same as A above)
open "file" for binary as #1 open "net" for binary as #2
for x = 0 to 10000 step bt get #1,x,a put #2,,a next
Title: Re: put / get buffer binary mode tcp
Post by: FellippeHeitor on April 16, 2020, 10:08:17 am
What are the actual commands you're using for this bit of pseudocode?
Title: Re: put / get buffer binary mode tcp
Post by: SMcNeill on April 16, 2020, 10:23:42 am
DIM a as STRING, L AS LONG bt=2 (or same as A above)
open "file" for binary as #1 open "net" for binary as #2
L = LOF(1) a$ = SPACE$(L) get #1,1,a$ PUT #2,,L put #2,,a$
Then read the file size as a long, and DO... GET.... LOOP UNTIL the info received = that size.
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 16, 2020, 10:29:07 am
[ This attachment cannot be displayed inline in 'Print Page' view ]
SCREEN 12 DIM i AS STRING * 32 ' set length of string to get from file/ send on net bl = 32 ' for program calculation
server& = _OPENHOST("TCP/IP:59999") PRINT "Waiting for connection..."
DO connection& = _OPENCONNECTION(server&) LOOP UNTIL connection& <> 0
PRINT "recieved connection"
GAME: PRINT "sending bytes" _DELAY .5 OPEN "beavis.avi" FOR BINARY AS #1
' get file length, make it be 32 bytes p$ = STR$(LOF(1)) PRINT p$ x = LEN(p$) q = 32 - x FOR z = 1 TO q o$ = o$ + " " NEXT p$ = o$ + p$ PRINT LEN(p$) PRINT p$ FOR z = 1 TO 32 a$ = MID$(p$, z, 1) PUT #connection&, , a$ NEXT
PRINT "Waiting for send instruction..." ' wait for "!" from client wfi: GET #connection&, , a$ IF a$ <> "!" THEN GOTO wfi PRINT "Sending data ..." _DELAY 0.2
' send file data
lf& = LOF(1) PRINT lf& PRINT "Sending bulk data..."; bl; "bbyte legnth" OPEN "beavis2.avi" FOR BINARY AS #3 ' for testing, write to local disk the data being sent over net
FOR x& = 1 TO lf& STEP bl ' repeat until end of file, increment of bytes sent each time GET #1, x&, i PUT #connection&, , i PUT #3, , i ' to local disk, for testing ' _DELAY (.0001) ' Playing around with this number changes the data recieved on other end NEXT
dtft:
CLOSE CLEAR END
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 16, 2020, 10:38:15 am
The program works; the source file data is read, sent to other computer proper, then data starts transferring, and saving. Everything works, just that at a certain speed, it seems data starts getting lost.
The more I slow down the sending computer/ program, the more data is recieved on the recieving computer / program.
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 16, 2020, 10:54:57 am
For EG, This "beavis.avi" file, is 240mb If I leave the "_delay" command commented out so it runs full speed, the receive computer ends up with a ~75mb file.
If I use the delay command, the received file is larger.
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 16, 2020, 12:34:59 pm
[ This attachment cannot be displayed inline in 'Print Page' view ]
This is a receive program I just made, still same issue, not recording all data to disk, as if not all data from send program is making it over the network.
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 16, 2020, 01:22:21 pm
I made 2 more little programs, The send one sends a 10 byte string 100,000 times,
The receive one uses GET to read it, in the form;
X: Get connection,,d$ data$=data$+d$ locate 1,1 : print len(data$) goto x
and I ran it 4 times, all 4 times the receive end string stopped around half what was sent, but never the exact same.
What the heck is going on? Is there something incompatible? I have linux mint on desktop and laptop, is there some kind of buffer issue with qb64?
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 16, 2020, 03:40:11 pm
I'm going to take a wild guess and it seems like the buffer is 64kb. In the screenshots, I used just plain string data, ignore message about file, that data isn't used.
Sending 64 bytes 1000 times works perfect. sending 64 bytes 2000 times data is lost. Sending 64 bytes 2000 times but with a time delay on the sending computer, perfect again.
Does anyone know if this is a bug? Or just the way it's supposed to work? I guess I can make the file transfer be 64kb at a time, and the receiving computer asks for more data whenever it saves the last 64kb.
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 18, 2020, 12:20:19 am
There must be a buffer / stack overflow issue. I modified the programs.
Now, the computer that's downloading data and saving it on disk, sends "total received data" to the sending computer whenever it receives data. The sending computer, keeps the record of the total received data on the receiving computer, and won't send data exceeding that number plus 64000.
I'm not sure what size the buffer could be, I'm guessing 64kb, but things happen so fast that it's hard to say. I can say for this set up / speed etc, when I keep the sending computer from sending more than 64kb that the receiving one hasn't received, I get the file 100%. If I change my buffer size to 80kb, I only got 99% of my file.
So it really seems that using the PUT command, you can PUT up to 64kb (?) of data and it will make it to the other computer, but over 64kb(?) it just disappears.
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 18, 2020, 02:07:28 am
This definitely complicates things. With such a small buffer with the TCP/IP, the sending program has to only keep sending data while receiving data from receiving computer of the received data, to insure a 64kb gap isn't exceeded. This isn't too terrible in the house, latency is low. If latency became high, like over the internet, speed would drop way off.
Or-
The sending computer sends data as packets made in QB64, the receiving computer sends a message back to indicate if it was received, then the sending computer if it stops getting received reports, re sends the lost information, and slows down sending rate, so that hopefully the buffer doesn't get exceeded.
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 18, 2020, 02:59:14 am
[ This attachment cannot be displayed inline in 'Print Page' view ] [ This attachment cannot be displayed inline in 'Print Page' view ]
This literature seems confusing 🤔 So get and put read and write "raw" data.... so IS that udp protocol then? Or just a tcp/ip stream?
From reading that, it doesn't seem like get and put are a reliable way to transfer data without designing some kind of packet programming 🤢
Title: Re: put / get buffer binary mode tcp
Post by: Parkland on April 18, 2020, 03:45:23 am
Upon further reading, the linux UDP transmit buffer is 65536 bytes, exactly the limit I thought I was hitting. So, I'm pretty convinced that get and put are using UDP stream not tcp/ip