Text Only | Text with Attachments

QB64.org Forum

Active Forums => Programs => Topic started by: SMcNeill on November 21, 2019, 10:54:15 am

Title: Steve's Data Transfer Protocol
Post by: SMcNeill on November 21, 2019, 10:54:15 am
As I've recently been playing around with various programs which communicate with each other via TCP/IP, I've decided that I needed some sort of simple protocol to make certain that the data I send and receive from computer to computer is correct and not corrupted.  Here's the routine I've basically set up so far:

Code: QB64: [Select]
  2.  
  3. SCREEN _NEWIMAGE(800, 600, 32)
  4. COLOR &HFFFFFFFF, &HFF000000
  5. Host = _OPENHOST("TCP/IP:7990") ' this will be the host code
  6.  
  8.     Player = GetClient 'Not
  9.     IF Player THEN
  10.         PRINT "New Player connected"
  11.         'Do stuff
  12.         UserData$ = In$(Player)
  13.         'do stuff with the data the user sent
  14.  
  15.         'and close the connection
  16.         CLOSE Player
  17.         Player = 0
  18.     END IF
  19.     _LIMIT 30
  21.  
  22.  
  23. FUNCTION GetClient
  24.     GetClient = _OPENCONNECTION(Host) ' receive any new connection
  26.  
  27. FUNCTION In$ (who)
  29.     'CHR$(2) = Start of Text
  30.     'CHR$(3) = End of Text
  31.     'CHR$(4) = End of Transmission (It's what we use to tell the client, "We give up!  Closing connection!"
  32.     'CHR$(6) = Acknowledge
  33.     'CHR$(15) = Not Acknowledge
  34.  
  35.     GET #who, , b 'just check for a single byte from each connection
  36.  
  37.     IF b <> 2 THEN
  38.         'If we get something which isn't a CHR$(2) to start communication, send back a failure notice.
  39.         SendError who
  40.         EXIT FUNCTION 'Exit so we can move on to the next connection to check for that leading chr$(2)
  41.     END IF
  42.  
  43.     'Only if that initial byte is CHR$(2), do we acknowledge receipt and await further messages.
  44.  
  45.     SendConfirmation who 'we send ACKnowledgement back to tell the client we're ready for them to talk to us.
  46.  
  47.     DO
  48.         count = count + 1
  49.         timeout## = ExtendedTimer + 5
  50.         DO
  51.             _LIMIT 100 'no need to check for incoming information more than 100 times a second!
  52.             GET #who, , a$
  53.             IF a$ <> "" THEN tempIn$ = tempIn$ + a$ 'tempIn$ should never be more than 105 bytes
  54.             ETX = INSTR(a$, CHR$(3)) '       chr$(3) is our ETX character (End of TeXt)
  55.             IF ETX THEN EXIT DO
  56.             IF ExtendedTimer > timeout## THEN 'If it takes over 5 seconds to send 100 bytes (or less) of info
  57.                 SendError who '              something is wrong.  Terminate the attempt, but be nice, and let
  58.                 EXIT FUNCTION '              the other client know something went wrong, so they can try again,
  59.             END IF '                         if they want to.
  60.         LOOP UNTIL LEN(tempIn$) > 105 'If we have over 105 bytes with our string, we didn't send the data properly.
  61.         IF LEN(tempIn$) > 105 THEN
  62.             SendError who 'send the client an error message
  63.             EXIT FUNCTION
  64.         END IF
  65.         tempIn$ = _TRIM$(LEFT$(tempIn$, ETX - 1)) 'strip off the ETX character and check to make certain data is valid.
  66.  
  67.         c$ = RIGHT$(tempIn$, 4) 'these 4 bytes are the checksum
  68.  
  69.         CheckSum = CVL(c$) 'Check to make certain the data apprears valid.
  70.         FOR i = 1 TO LEN(l$): Check = Check + ASC(l$, i): NEXT
  71.         IF CheckSum <> Check THEN '            Our data is not what we expected.  Part may be lost, or corrupted.
  72.             SendError who
  73.             EXIT FUNCTION
  74.         ELSE
  75.             SendConfirmation who
  76.             EXIT DO
  77.         END IF
  78.     LOOP UNTIL count = 5
  79.     'If we get bad data 5 times in a row, something is wrong.  We're just going to close the connection.
  80.     IF count = 5 THEN
  81.         SendError who
  82.         EXIT FUNCTION
  83.     END IF
  84.  
  85.     'and if we're down this far, our data has been recieved, verified, and is now good to use.
  86.     In$ = LEFT$(tempIn$, 4) 'left part of the string is the data the user is sending us
  88.  
  89. SUB SendError (who)
  91.     b = 4
  92.     PUT #who, , b
  94.  
  95. SUB SendConfirmation (who)
  97.     b = 6
  98.     PUT #who, , b
  100.  
  101. FUNCTION ExtendedTimer##
  102.     DIM m AS INTEGER, d AS INTEGER, y AS INTEGER
  103.     DIM s AS _FLOAT, day AS STRING
  104.     day = DATE$
  105.     m = VAL(LEFT$(day, 2))
  106.     d = VAL(MID$(day, 4, 2))
  107.     y = VAL(RIGHT$(day, 4)) - 1970
  108.     SELECT CASE m 'Add the number of days for each previous month passed
  109.         CASE 2: d = d + 31
  110.         CASE 3: d = d + 59
  111.         CASE 4: d = d + 90
  112.         CASE 5: d = d + 120
  113.         CASE 6: d = d + 151
  114.         CASE 7: d = d + 181
  115.         CASE 8: d = d + 212
  116.         CASE 9: d = d + 243
  117.         CASE 10: d = d + 273
  118.         CASE 11: d = d + 304
  119.         CASE 12: d = d + 334
  120.     END SELECT
  121.     IF (y MOD 4) = 2 AND m > 2 THEN d = d + 1 'add a day if this is leap year and we're past february
  122.     d = (d - 1) + 365 * y 'current month days passed + 365 days per each standard year
  123.     d = d + (y + 2) \ 4 'add in days for leap years passed
  124.     s = d * 24 * 60 * 60 'Seconds are days * 24 hours * 60 minutes * 60 seconds
  125.     ExtendedTimer## = (s + TIMER)
  127.  

Now this isn't going to work to send binary files, as I'm using some of the ASCII characters as reserved command codes, but since this isn't meant to be a file transfer protocol, I don't think it should be a problem.  My command codes are as follows:
    'CHR$(2) = Start of Text
    'CHR$(3) = End of Text
    'CHR$(4) = End of Transmission (It's what we use to tell the client, "We give up!  Closing connection!"
    'CHR$(6) = Acknowledge
    'CHR$(15) = Not Acknowledge

The idea the behind the process is this one:

First, we simply wait for a CHR$(2) character to come in, as a request from a client saying they want to send us data.  If we get anything else before that, we send them an error message.  All messages start with chr$(2), and when we get it, we send a confirmation back to the client so they know we're all set to receive their data (CHR$(6)).

At this point, they send us the data, which is limited to being 105 bytes or less.  This 105 byte structure consists of up to 100 bytes of data, 4 bytes for a checksum of the data sent, and then the termination code.  (CHR$(3))

Once we verify that everything is correct, we either send back a success, or failure signal, to the client.  If we fail, they can try to resend the data, otherwise all is golden.

I tried to comment the process here so that it'd be easily understood by anyone who looks it over, but if anyone has any questions, just ask them.  If there appears to be something wrong with my logic, feel free to tell me about that as well.  I haven't actually tested this in a working program yet (my test game is still in development and hasn't gotten to the point where it's trying to talk back and forth to other games yet), but I don't see anything that looks wrong with it.  Unless I just made a common typo, or other silly mistake, it should work as intended here...

Take a look at it.  See if it looks like a process that will hold up to general usage to send plain text back and forth between computers.  And, if you see something that I goofed on, or overlooked, kindly point it out to me.  If all works as intended, this will end up going into a transfer library later for me, so I can just plug it into any project and use it to send and receive data between devices.  :)
Title: Re: Steve's Data Transfer Protocol
Post by: FellippeHeitor on November 21, 2019, 11:38:03 am
I also use network messages to have InForm's modules communicate in real time during execution, and had to come up with a custom protocol as well. I began as you have, with single low-ascii characters as markers, but then I came across your "how to send binary data" issue as well. What I ended up with (as can be seen in the non-runnable snippet below, just to showcase my local solution) was using actual text messages to mark what type of information is being shared, along with the MKL$() representation of the length of the data about to be sent. Kind of what HTTP does with GET and Content-length.

Code: QB64: [Select]
  1.     GET #Client, , incomingData$
  2.     Stream$ = Stream$ + incomingData$
  3.     'STATIC bytesIn~&&, refreshes~&
  4.     'refreshes~& = refreshes~& + 1
  5.     'bytesIn~&& = bytesIn~&& + LEN(incomingData$)
  6.     'Caption(StatusBar) = "Received:" + STR$(bytesIn~&&) + " bytes | Sent:" + STR$(totalBytesSent) + " bytes"
  7.  
  8.     $IF WIN THEN
  9.         IF PreviewAttached THEN
  10.             IF prevScreenX <> _SCREENX OR prevScreenY <> _SCREENY THEN
  11.                 prevScreenX = _SCREENX
  12.                 prevScreenY = _SCREENY
  13.                 b$ = "WINDOWPOSITION>" + MKI$(_SCREENX) + MKI$(_SCREENY) + "<END>"
  14.                 Send Client, b$
  15.             END IF
  16.         ELSE
  17.             IF prevScreenX <> -32001 OR prevScreenY <> -32001 THEN
  18.                 prevScreenX = -32001
  19.                 prevScreenY = -32001
  20.                 b$ = "WINDOWPOSITION>" + MKI$(-32001) + MKI$(-32001) + "<END>"
  21.                 Send Client, b$
  22.             END IF
  23.         END IF
  24.     $ELSE
  25.         IF PreviewAttached = True THEN
  26.         PreviewAttached = False
  27.         SaveSettings
  28.         END IF
  29.         Control(ViewMenuPreviewDetach).Disabled = True
  30.         Control(ViewMenuPreviewDetach).Value = False
  31.     $END IF
  32.  
  33.     STATIC prevAutoName AS _BYTE, prevMouseSwap AS _BYTE
  34.     STATIC prevShowPos AS _BYTE, prevSnapLines AS _BYTE
  35.     STATIC prevShowInvisible AS _BYTE, SignalsFirstSent AS _BYTE
  36.  
  37.     IF prevAutoName <> AutoNameControls OR SignalsFirstSent = False THEN
  38.         prevAutoName = AutoNameControls
  39.         b$ = "AUTONAME>" + MKI$(AutoNameControls) + "<END>"
  40.         Send Client, b$
  41.     END IF
  42.  
  43.     IF prevMouseSwap <> __UI_MouseButtonsSwap OR SignalsFirstSent = False THEN
  44.         prevMouseSwap = __UI_MouseButtonsSwap
  45.         b$ = "MOUSESWAP>" + MKI$(__UI_MouseButtonsSwap) + "<END>"
  46.         Send Client, b$
  47.     END IF
  48.  
  49.     IF prevShowPos <> __UI_ShowPositionAndSize OR SignalsFirstSent = False THEN
  50.         prevShowPos = __UI_ShowPositionAndSize
  51.         b$ = "SHOWPOSSIZE>" + MKI$(__UI_ShowPositionAndSize) + "<END>"
  52.         Send Client, b$
  53.     END IF
  54.  
  55.     IF prevShowInvisible <> __UI_ShowInvisibleControls OR SignalsFirstSent = False THEN
  56.         prevShowInvisible = __UI_ShowInvisibleControls
  57.         b$ = "SHOWINVISIBLECONTROLS>" + MKI$(__UI_ShowInvisibleControls) + "<END>"
  58.         Send Client, b$
  59.     END IF
  60.  
  61.     IF prevSnapLines <> __UI_SnapLines OR SignalsFirstSent = False THEN
  62.         prevSnapLines = __UI_SnapLines
  63.         b$ = "SNAPLINES>" + MKI$(__UI_SnapLines) + "<END>"
  64.         Send Client, b$
  65.     END IF
  66.     SignalsFirstSent = True
  67.  
  68.     DO WHILE INSTR(Stream$, "<END>") > 0
  69.         thisData$ = LEFT$(Stream$, INSTR(Stream$, "<END>") - 1)
  70.         Stream$ = MID$(Stream$, INSTR(Stream$, "<END>") + 5)
  71.         thisCommand$ = LEFT$(thisData$, INSTR(thisData$, ">") - 1)
  72.         thisData$ = MID$(thisData$, LEN(thisCommand$) + 2)
  73.         SELECT CASE UCASE$(thisCommand$)
  74.             CASE "TOTALSELECTEDCONTROLS"
  75.                 TotalSelected = CVL(thisData$)
  76.             CASE "FORMID"
  77.                 PreviewFormID = CVL(thisData$)
  78.             CASE "FIRSTSELECTED"
  79.                 FirstSelected = CVL(thisData$)
  80.             CASE "DEFAULTBUTTONID"
  81.                 PreviewDefaultButtonID = CVL(thisData$)
  82.             CASE "SHOWINVISIBLECONTROLS"
  83.                 __UI_ShowInvisibleControls = CVI(thisData$)
  84.                 Control(ViewMenuShowInvisibleControls).Value = __UI_ShowInvisibleControls
  85.             CASE "ORIGINALIMAGEWIDTH"
  86.                 OriginalImageWidth = CVI(thisData$)
  87.             CASE "ORIGINALIMAGEHEIGHT"
  88.                 OriginalImageHeight = CVI(thisData$)
  89.             CASE "TURNSINTO"
  90.                 ThisControlTurnsInto = CVI(thisData$)
  91.             CASE "SELECTIONRECTANGLE"
  92.                 PreviewSelectionRectangle = CVI(thisData$)
  93.                 LoseFocus
  94.             CASE "MENUPANELACTIVE"
  95.                 PreviewHasMenuActive = CVI(thisData$)
  96.             CASE "SIGNAL"
  97.                 Signal$ = Signal$ + thisData$
  98.             CASE "FORMDATA"
  99.                 LastFormData$ = thisData$
  100.                 LoadPreview
  101.                 IF NOT FormDataReceived THEN
  102.                     FormDataReceived = True
  103.                 ELSE
  104.                     Edited = True
  105.                     IF __UI_Focus > 0 THEN
  106.                         IF PropertySent THEN PropertySent = False ELSE LoseFocus
  107.                     END IF
  108.                 END IF
  109.             CASE "UNDOPOINTER"
  110.                 UndoPointer = CVI(thisData$)
  111.             CASE "TOTALUNDOIMAGES"
  112.                 TotalUndoImages = CVI(thisData$)
  113.         END SELECT
  114.     LOOP
  115.  

It's convoluted, I know (why am I even sharing this then, right?), but what I do here:
Code: QB64: [Select]
  1. thisData$ = LEFT$(Stream$, INSTR(Stream$, "<END>") - 1)
  2.         Stream$ = MID$(Stream$, INSTR(Stream$, "<END>") + 5)
  3.         thisCommand$ = LEFT$(thisData$, INSTR(thisData$, ">") - 1)
  4.         thisData$ = MID$(thisData$, LEN(thisCommand$) + 2)

is extract the next command (thisCommand$) from the incoming stream, then whatever data (thisData$) comes after the command (here included the length of the data itself), then I process it with the SELECT CASE block below.

Maybe it's helpful. If it's not, it was free to post anyway!

😂
Title: Re: Steve's Data Transfer Protocol
Post by: Petr on November 21, 2019, 02:02:02 pm
Thank you. Both ideas are very useful for me.
Text Only | Text with Attachments