Steve's Data Transfer Protocol

[SMcNeill]

As I've recently been playing around with various programs which communicate with each other via TCP/IP, I've decided that I needed some sort of simple protocol to make certain that the data I send and receive from computer to computer is correct and not corrupted.  Here's the routine I've basically set up so far:

Code: QB64: [Select]

1. DIM SHARED Host AS LONG
2.  
3. SCREEN _NEWIMAGE(800, 600, 32)
4. COLOR &HFFFFFFFF, &HFF000000
5. Host = _OPENHOST("TCP/IP:7990") ' this will be the host code
6.  
7. DO
8.     Player = GetClient 'Not
9.     IF Player THEN
10.         PRINT "New Player connected"
11.         'Do stuff
12.         UserData$ = In$(Player)
13.         'do stuff with the data the user sent
14.  
15.         'and close the connection
16.         CLOSE Player
17.         Player = 0
18.     END IF
19.     _LIMIT 30
20. LOOP
21.  
22.  
23. FUNCTION GetClient
24.     GetClient = _OPENCONNECTION(Host) ' receive any new connection
25. END FUNCTION
26.  
27. FUNCTION In$ (who)
28.     DIM b AS _UNSIGNED _BYTE
29.     'CHR$(2) = Start of Text
30.     'CHR$(3) = End of Text
31.     'CHR$(4) = End of Transmission (It's what we use to tell the client, "We give up!  Closing connection!"
32.     'CHR$(6) = Acknowledge
33.     'CHR$(15) = Not Acknowledge
34.  
35.     GET #who, , b 'just check for a single byte from each connection
36.  
37.     IF b <> 2 THEN
38.         'If we get something which isn't a CHR$(2) to start communication, send back a failure notice.
39.         SendError who
40.         EXIT FUNCTION 'Exit so we can move on to the next connection to check for that leading chr$(2)
41.     END IF
42.  
43.     'Only if that initial byte is CHR$(2), do we acknowledge receipt and await further messages.
44.  
45.     SendConfirmation who 'we send ACKnowledgement back to tell the client we're ready for them to talk to us.
46.  
47.     DO
48.         count = count + 1
49.         timeout## = ExtendedTimer + 5
50.         DO
51.             _LIMIT 100 'no need to check for incoming information more than 100 times a second!
52.             GET #who, , a$
53.             IF a$ <> "" THEN tempIn$ = tempIn$ + a$ 'tempIn$ should never be more than 105 bytes
54.             ETX = INSTR(a$, CHR$(3)) '       chr$(3) is our ETX character (End of TeXt)
55.             IF ETX THEN EXIT DO
56.             IF ExtendedTimer > timeout## THEN 'If it takes over 5 seconds to send 100 bytes (or less) of info
57.                 SendError who '              something is wrong.  Terminate the attempt, but be nice, and let
58.                 EXIT FUNCTION '              the other client know something went wrong, so they can try again,
59.             END IF '                         if they want to.
60.         LOOP UNTIL LEN(tempIn$) > 105 'If we have over 105 bytes with our string, we didn't send the data properly.
61.         IF LEN(tempIn$) > 105 THEN
62.             SendError who 'send the client an error message
63.             EXIT FUNCTION
64.         END IF
65.         tempIn$ = _TRIM$(LEFT$(tempIn$, ETX - 1)) 'strip off the ETX character and check to make certain data is valid.
66.  
67.         c$ = RIGHT$(tempIn$, 4) 'these 4 bytes are the checksum
68.  
69.         CheckSum = CVL(c$) 'Check to make certain the data apprears valid.
70.         FOR i = 1 TO LEN(l$): Check = Check + ASC(l$, i): NEXT
71.         IF CheckSum <> Check THEN '            Our data is not what we expected.  Part may be lost, or corrupted.
72.             SendError who
73.             EXIT FUNCTION
74.         ELSE
75.             SendConfirmation who
76.             EXIT DO
77.         END IF
78.     LOOP UNTIL count = 5
79.     'If we get bad data 5 times in a row, something is wrong.  We're just going to close the connection.
80.     IF count = 5 THEN
81.         SendError who
82.         EXIT FUNCTION
83.     END IF
84.  
85.     'and if we're down this far, our data has been recieved, verified, and is now good to use.
86.     In$ = LEFT$(tempIn$, 4) 'left part of the string is the data the user is sending us
87. END FUNCTION
88.  
89. SUB SendError (who)
90.     DIM b AS _UNSIGNED _BYTE
91.     b = 4
92.     PUT #who, , b
93. END SUB
94.  
95. SUB SendConfirmation (who)
96.     DIM b AS _UNSIGNED _BYTE
97.     b = 6
98.     PUT #who, , b
99. END SUB
100.  
101. FUNCTION ExtendedTimer##
102.     DIM m AS INTEGER, d AS INTEGER, y AS INTEGER
103.     DIM s AS _FLOAT, day AS STRING
104.     day = DATE$
105.     m = VAL(LEFT$(day, 2))
106.     d = VAL(MID$(day, 4, 2))
107.     y = VAL(RIGHT$(day, 4)) - 1970
108.     SELECT CASE m 'Add the number of days for each previous month passed
109.         CASE 2: d = d + 31
110.         CASE 3: d = d + 59
111.         CASE 4: d = d + 90
112.         CASE 5: d = d + 120
113.         CASE 6: d = d + 151
114.         CASE 7: d = d + 181
115.         CASE 8: d = d + 212
116.         CASE 9: d = d + 243
117.         CASE 10: d = d + 273
118.         CASE 11: d = d + 304
119.         CASE 12: d = d + 334
120.     END SELECT
121.     IF (y MOD 4) = 2 AND m > 2 THEN d = d + 1 'add a day if this is leap year and we're past february
122.     d = (d - 1) + 365 * y 'current month days passed + 365 days per each standard year
123.     d = d + (y + 2) \ 4 'add in days for leap years passed
124.     s = d * 24 * 60 * 60 'Seconds are days * 24 hours * 60 minutes * 60 seconds
125.     ExtendedTimer## = (s + TIMER)
126. END FUNCTION
127.  

Now this isn't going to work to send binary files, as I'm using some of the ASCII characters as reserved command codes, but since this isn't meant to be a file transfer protocol, I don't think it should be a problem.  My command codes are as follows:
    'CHR$(2) = Start of Text
    'CHR$(3) = End of Text
    'CHR$(4) = End of Transmission (It's what we use to tell the client, "We give up!  Closing connection!"
    'CHR$(6) = Acknowledge
    'CHR$(15) = Not Acknowledge

The idea the behind the process is this one:

First, we simply wait for a CHR$(2) character to come in, as a request from a client saying they want to send us data.  If we get anything else before that, we send them an error message.  All messages start with chr$(2), and when we get it, we send a confirmation back to the client so they know we're all set to receive their data (CHR$(6)).

At this point, they send us the data, which is limited to being 105 bytes or less.  This 105 byte structure consists of up to 100 bytes of data, 4 bytes for a checksum of the data sent, and then the termination code.  (CHR$(3))

Once we verify that everything is correct, we either send back a success, or failure signal, to the client.  If we fail, they can try to resend the data, otherwise all is golden.

I tried to comment the process here so that it'd be easily understood by anyone who looks it over, but if anyone has any questions, just ask them.  If there appears to be something wrong with my logic, feel free to tell me about that as well.  I haven't actually tested this in a working program yet (my test game is still in development and hasn't gotten to the point where it's trying to talk back and forth to other games yet), but I don't see anything that looks wrong with it.  Unless I just made a common typo, or other silly mistake, it should work as intended here...

Take a look at it.  See if it looks like a process that will hold up to general usage to send plain text back and forth between computers.  And, if you see something that I goofed on, or overlooked, kindly point it out to me.  If all works as intended, this will end up going into a transfer library later for me, so I can just plug it into any project and use it to send and receive data between devices.  :)

[FellippeHeitor]

I also use network messages to have InForm's modules communicate in real time during execution, and had to come up with a custom protocol as well. I began as you have, with single low-ascii characters as markers, but then I came across your "how to send binary data" issue as well. What I ended up with (as can be seen in the non-runnable snippet below, just to showcase my local solution) was using actual text messages to mark what type of information is being shared, along with the MKL$() representation of the length of the data about to be sent. Kind of what HTTP does with GET and Content-length.

Code: QB64: [Select]

1.     GET #Client, , incomingData$
2.     Stream$ = Stream$ + incomingData$
3.     'STATIC bytesIn~&&, refreshes~&
4.     'refreshes~& = refreshes~& + 1
5.     'bytesIn~&& = bytesIn~&& + LEN(incomingData$)
6.     'Caption(StatusBar) = "Received:" + STR$(bytesIn~&&) + " bytes | Sent:" + STR$(totalBytesSent) + " bytes"
7.  
8.     $IF WIN THEN
9.         IF PreviewAttached THEN
10.             IF prevScreenX <> _SCREENX OR prevScreenY <> _SCREENY THEN
11.                 prevScreenX = _SCREENX
12.                 prevScreenY = _SCREENY
13.                 b$ = "WINDOWPOSITION>" + MKI$(_SCREENX) + MKI$(_SCREENY) + "<END>"
14.                 Send Client, b$
15.             END IF
16.         ELSE
17.             IF prevScreenX <> -32001 OR prevScreenY <> -32001 THEN
18.                 prevScreenX = -32001
19.                 prevScreenY = -32001
20.                 b$ = "WINDOWPOSITION>" + MKI$(-32001) + MKI$(-32001) + "<END>"
21.                 Send Client, b$
22.             END IF
23.         END IF
24.     $ELSE
25.         IF PreviewAttached = True THEN
26.         PreviewAttached = False
27.         SaveSettings
28.         END IF
29.         Control(ViewMenuPreviewDetach).Disabled = True
30.         Control(ViewMenuPreviewDetach).Value = False
31.     $END IF
32.  
33.     STATIC prevAutoName AS _BYTE, prevMouseSwap AS _BYTE
34.     STATIC prevShowPos AS _BYTE, prevSnapLines AS _BYTE
35.     STATIC prevShowInvisible AS _BYTE, SignalsFirstSent AS _BYTE
36.  
37.     IF prevAutoName <> AutoNameControls OR SignalsFirstSent = False THEN
38.         prevAutoName = AutoNameControls
39.         b$ = "AUTONAME>" + MKI$(AutoNameControls) + "<END>"
40.         Send Client, b$
41.     END IF
42.  
43.     IF prevMouseSwap <> __UI_MouseButtonsSwap OR SignalsFirstSent = False THEN
44.         prevMouseSwap = __UI_MouseButtonsSwap
45.         b$ = "MOUSESWAP>" + MKI$(__UI_MouseButtonsSwap) + "<END>"
46.         Send Client, b$
47.     END IF
48.  
49.     IF prevShowPos <> __UI_ShowPositionAndSize OR SignalsFirstSent = False THEN
50.         prevShowPos = __UI_ShowPositionAndSize
51.         b$ = "SHOWPOSSIZE>" + MKI$(__UI_ShowPositionAndSize) + "<END>"
52.         Send Client, b$
53.     END IF
54.  
55.     IF prevShowInvisible <> __UI_ShowInvisibleControls OR SignalsFirstSent = False THEN
56.         prevShowInvisible = __UI_ShowInvisibleControls
57.         b$ = "SHOWINVISIBLECONTROLS>" + MKI$(__UI_ShowInvisibleControls) + "<END>"
58.         Send Client, b$
59.     END IF
60.  
61.     IF prevSnapLines <> __UI_SnapLines OR SignalsFirstSent = False THEN
62.         prevSnapLines = __UI_SnapLines
63.         b$ = "SNAPLINES>" + MKI$(__UI_SnapLines) + "<END>"
64.         Send Client, b$
65.     END IF
66.     SignalsFirstSent = True
67.  
68.     DO WHILE INSTR(Stream$, "<END>") > 0
69.         thisData$ = LEFT$(Stream$, INSTR(Stream$, "<END>") - 1)
70.         Stream$ = MID$(Stream$, INSTR(Stream$, "<END>") + 5)
71.         thisCommand$ = LEFT$(thisData$, INSTR(thisData$, ">") - 1)
72.         thisData$ = MID$(thisData$, LEN(thisCommand$) + 2)
73.         SELECT CASE UCASE$(thisCommand$)
74.             CASE "TOTALSELECTEDCONTROLS"
75.                 TotalSelected = CVL(thisData$)
76.             CASE "FORMID"
77.                 PreviewFormID = CVL(thisData$)
78.             CASE "FIRSTSELECTED"
79.                 FirstSelected = CVL(thisData$)
80.             CASE "DEFAULTBUTTONID"
81.                 PreviewDefaultButtonID = CVL(thisData$)
82.             CASE "SHOWINVISIBLECONTROLS"
83.                 __UI_ShowInvisibleControls = CVI(thisData$)
84.                 Control(ViewMenuShowInvisibleControls).Value = __UI_ShowInvisibleControls
85.             CASE "ORIGINALIMAGEWIDTH"
86.                 OriginalImageWidth = CVI(thisData$)
87.             CASE "ORIGINALIMAGEHEIGHT"
88.                 OriginalImageHeight = CVI(thisData$)
89.             CASE "TURNSINTO"
90.                 ThisControlTurnsInto = CVI(thisData$)
91.             CASE "SELECTIONRECTANGLE"
92.                 PreviewSelectionRectangle = CVI(thisData$)
93.                 LoseFocus
94.             CASE "MENUPANELACTIVE"
95.                 PreviewHasMenuActive = CVI(thisData$)
96.             CASE "SIGNAL"
97.                 Signal$ = Signal$ + thisData$
98.             CASE "FORMDATA"
99.                 LastFormData$ = thisData$
100.                 LoadPreview
101.                 IF NOT FormDataReceived THEN
102.                     FormDataReceived = True
103.                 ELSE
104.                     Edited = True
105.                     IF __UI_Focus > 0 THEN
106.                         IF PropertySent THEN PropertySent = False ELSE LoseFocus
107.                     END IF
108.                 END IF
109.             CASE "UNDOPOINTER"
110.                 UndoPointer = CVI(thisData$)
111.             CASE "TOTALUNDOIMAGES"
112.                 TotalUndoImages = CVI(thisData$)
113.         END SELECT
114.     LOOP
115.  

It's convoluted, I know (why am I even sharing this then, right?), but what I do here:

Code: QB64: [Select]

1. thisData$ = LEFT$(Stream$, INSTR(Stream$, "<END>") - 1)
2.         Stream$ = MID$(Stream$, INSTR(Stream$, "<END>") + 5)
3.         thisCommand$ = LEFT$(thisData$, INSTR(thisData$, ">") - 1)
4.         thisData$ = MID$(thisData$, LEN(thisCommand$) + 2)

is extract the next command (thisCommand$) from the incoming stream, then whatever data (thisData$) comes after the command (here included the length of the data itself), then I process it with the SELECT CASE block below.

Maybe it's helpful. If it's not, it was free to post anyway!

😂

[Petr]

Thank you. Both ideas are very useful for me.