Text Only | Text with Attachments

QB64.org Forum

Active Forums => Programs => Topic started by: bplus on February 01, 2022, 11:33:24 am

Title: Split Versus Tokenize
Post by: bplus on February 01, 2022, 11:33:24 am
https://qb64forum.alephc.xyz/index.php?topic=4618.msg140260#msg140260
Quote
I think my way of splitting a string into an array is slightly better:

I say Split has at least two advantages!
1. Less LOC
2. Preserves blank lines from a .bas file.

And I am pretty sure for small files there is no significant time differences, not even sure if Tokenize is faster and how big a file you need to see it. I leave that to believers of Tokenize. ;-))
Code: QB64: [Select]
  1. _Title "Split versus Tokenize" ' b+ 2022-02-01
  2.  
  3. ' Is Tokenize really worth the extra LOC compared to Split?
  4. ' No! if you want to preserve blank lines.
  5.  
  6. '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  7.  
  8. '            save this file as "Split versus Tokenize.bas"
  9.  
  10. '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  11.  
  12. Open "Split versus Tokenize.bas" For Binary As #1
  13. buf$ = Space$(LOF(1))
  14.  
  15. Get #1, , buf$
  16.  
  18.  
  19. deli$ = Chr$(13) + Chr$(10)
  20.  
  21. ReDim LoadMe$(1 To 1)
  22.  
  23. startSplit = Timer(.001)
  24. Split buf$, deli$, LoadMe$()
  25. splitTime = Timer(.001) - startSplit
  26.  
  27. For i = LBound(LoadMe$) To 10
  28.     Print i, LoadMe$(i)
  30. 'For i = UBound(LoadMe$) - 5 To UBound(LoadMe$)
  31. '    Print i, LoadMe$(i)
  32. 'Next
  33. Print "Time for Split was:"; splitTime
  34.  
  35. ' reset for tokenize
  36. ReDim LoadMe$(1 To 1)
  37. startTokenize = Timer(.001)
  38. tokenize buf$, deli$, LoadMe$()
  39. TokenizeTime = Timer(.001) - startTokenize
  40. For i = LBound(LoadMe$) To 10
  41.     Print i, LoadMe$(i)
  43. 'For i = UBound(LoadMe$) - 5 To UBound(LoadMe$)
  44. '    Print i, LoadMe$(i)
  45. 'Next
  46. Print "Time for Tokenize was:"; TokenizeTime  ' << edit: had wrong variable in here
  47.  
  48.  
  49.  
  50. ' note: I buggered this twice now, FOR base 1 array REDIM MyArray (1 to 1) AS ... the (1 to 1) is not same as (1) which was the Blunder!!!
  51. 'notes: REDIM the array(0) to be loaded before calling Split '<<<< IMPORTANT dynamic array and empty, can use any lbound though
  52. 'This SUB will take a given N delimited string, and delimiter$ and create an array of N+1 strings using the LBOUND of the given dynamic array to load.
  53. 'notes: the loadMeArray() needs to be dynamic string array and will not change the LBOUND of the array it is given.  rev 2019-08-27
  54. Sub Split (SplitMeString As String, delim As String, loadMeArray() As String)
  55.     Dim curpos As Long, arrpos As Long, LD As Long, dpos As Long 'fix use the Lbound the array already has
  56.     curpos = 1: arrpos = LBound(loadMeArray): LD = Len(delim)
  57.     dpos = InStr(curpos, SplitMeString, delim)
  58.     Do Until dpos = 0
  59.         loadMeArray(arrpos) = Mid$(SplitMeString, curpos, dpos - curpos)
  60.         arrpos = arrpos + 1
  61.         If arrpos > UBound(loadMeArray) Then ReDim _Preserve loadMeArray(LBound(loadMeArray) To UBound(loadMeArray) + 1000) As String
  62.         curpos = dpos + LD
  63.         dpos = InStr(curpos, SplitMeString, delim)
  64.     Loop
  65.     loadMeArray(arrpos) = Mid$(SplitMeString, curpos)
  66.     ReDim _Preserve loadMeArray(LBound(loadMeArray) To arrpos) As String 'get the ubound correct
  68.  
  69. Sub tokenize (toTokenize As String, delimiters As String, StorageArray() As String)
  71.         Function strtok%& (ByVal str As _Offset, delimiters As String)
  72.     End Declare
  73.     Dim As _Offset tokenized
  74.     Dim As String tokCopy: tokCopy = toTokenize + Chr$(0)
  75.     Dim As String delCopy: delCopy = delimiters + Chr$(0)
  76.     Dim As _Unsigned Long lowerbound: lowerbound = LBound(StorageArray)
  77.     Dim As _Unsigned Long i: i = lowerbound
  78.     tokenized = strtok(_Offset(tokCopy), delCopy)
  79.     While tokenized <> 0
  80.         ReDim _Preserve StorageArray(lowerbound To UBound(StorageArray) + 1)
  81.         StorageArray(i) = pointerToString(tokenized)
  82.         tokenized = strtok(0, delCopy)
  83.         i = i + 1
  84.     Wend
  85.     ReDim _Preserve StorageArray(UBound(StorageArray) - 1)
  87.  
  88. Function pointerToString$ (pointer As _Offset)
  90.         Function strlen%& (ByVal ptr As _Unsigned _Offset)
  91.     End Declare
  92.     Dim As _Offset length: length = strlen(pointer)
  93.     If length Then
  94.         Dim As _MEM pString: pString = _Mem(pointer, length)
  95.         Dim As String ret: ret = Space$(length)
  96.         _MemGet pString, pString.OFFSET, ret
  97.         _MemFree pString
  98.     End If
  99.     pointerToString = ret
  101.  
  102.  
  103.  

Edit: had wrong Time variable in showing Tokenize Time. Ha! I was wondering how they were coming exactly the same each time tested.
Title: Re: Split Versus Tokenize
Post by: _vince on February 01, 2022, 11:58:49 am
nice, bplus.  Those tokenize and pointertostring functions should be banned.  Looks like a C compiler swallowed VB.NET -- not BASIC at all!
Title: Re: Split Versus Tokenize
Post by: bplus on February 01, 2022, 12:02:22 pm
Quote from: _vince on February 01, 2022, 11:58:49 am
Looks like a C compiler swallowed VB.NET

LOL

I will bet if we really need a speed bump for some app, we might have to resort to this stuff?
Title: Re: Split Versus Tokenize
Post by: Cobalt on February 01, 2022, 12:28:25 pm
Quote from: bplus on February 01, 2022, 12:02:22 pm
LOL

I will bet if we really need a speed bump for some app, we might have to resort to this stuff?

At that point its time to go to a non-translated language. If speed is needed to that point better start learning machine code and using ASM.
Title: Re: Split Versus Tokenize
Post by: bplus on February 01, 2022, 01:16:56 pm
Also weird is how some times are negative!?!

Does that mean if we run the program allot we will go back in time?
Title: Re: Split Versus Tokenize
Post by: SpriggsySpriggs on February 01, 2022, 01:37:21 pm
The reason you weren't preserving blank lines is because you are using tokenize incorrectly. You should pass it either CHR$(13) or CHR$(10) (can't remember which). If you pass both, it'll split by both. It can take a list of delimiters.
Title: Re: Split Versus Tokenize
Post by: bplus on February 01, 2022, 02:48:17 pm
Quote from: SpriggsySpriggs on February 01, 2022, 01:37:21 pm
The reason you weren't preserving blank lines is because you are using tokenize incorrectly. You should pass it either CHR$(13) or CHR$(10) (can't remember which). If you pass both, it'll split by both. It can take a list of delimiters.

Both are used as one delimiter for .bas files.

So you are saying I cannot use ", " as 1 delimiter for Tokenize either, I must only use only single char delimiters?

Advantage #3 for Split = variable length delimiters. :-))
Title: Re: Split Versus Tokenize
Post by: SpriggsySpriggs on February 01, 2022, 02:49:57 pm
You're intentionally misinterpreting what I'm saying but that's ok. If you prefer to remain ignorant then by all means, please do.
Title: Re: Split Versus Tokenize
Post by: bplus on February 01, 2022, 02:51:50 pm
Quote from: SpriggsySpriggs on February 01, 2022, 02:49:57 pm
You're intentionally misinterpreting what I'm saying but that's ok. If you prefer to remain ignorant then by all means, please do.

What did I misinterpret, .bas files are really split by Chr$(13) + Chr$(10) you need both in one delimiter.
Title: Re: Split Versus Tokenize
Post by: bplus on February 01, 2022, 02:58:20 pm
@SpriggsySpriggs

If you want, I will say Tokenize has advantage of multiple single char delimiters, that must be how you "list" them in a single delimiter string...

Actually if that were true Tokenize should be splitting into even more blank lines?

I suspect Tokenize ignores blank lines? Maybe?

Easy enough to test....
Title: Re: Split Versus Tokenize
Post by: bplus on February 01, 2022, 03:13:40 pm
4th advantage of Split = it preserves the Lbound of the storage array.

Title: Re: Split Versus Tokenize
Post by: SMcNeill on February 01, 2022, 03:35:36 pm
Quote from: bplus on February 01, 2022, 02:51:50 pm
What did I misinterpret, .bas files are really split by Chr$(13) + Chr$(10) you need both in one delimiter.

This isn't quite true.  :P

Files are delimited according to the CRLF line endings, which vary from OS to OS.

Old versions of Windows used CHR$(13) + CHR$(10) as the CRLF character.
Linux uses CHR$(10) for line endings.
Old Macs use CHR$(13) for line endings.
New Macs have swapped over to CHR$(10) line endings, if I remember correctly.
New versions of Windows have gotten with the program and generally default to the old CHR$(13) + CHR$(10) line endings, but they also now allow you to import and export your line endings however the heck you want!

It's your OS which determines what types of line endings your files have; not the BAS extension itself.  At the end of the day, all a BAS file is, is a TXT file with a little more descriptive extension on it.  ;)
Title: Re: Split Versus Tokenize
Post by: bplus on February 01, 2022, 04:04:54 pm
Screw files then here is a test that proves tokenize ignores blank lines and changes the lbound of the load array to 0:
Code: QB64: [Select]
  1. _Title "Tokenize blank line test" ' b+ 2022-02-01
  2. b$ = "Line 1"
  3. For i = 2 To 10
  4.     If i = 4 Or i = 7 Or i = 10 Then
  5.         b$ = b$ + Chr$(10) + "Line" + Str$(i)
  6.     Else
  7.         b$ = b$ + Chr$(10)
  8.     End If
  10.  
  11. deli$ = Chr$(10)
  12.  
  13. ReDim LoadMe$(1 To 1)
  14. startSplit = Timer(.001)
  15. Split b$, deli$, LoadMe$()
  16. splitTime = Timer(.001) - startSplit
  17. For i = LBound(LoadMe$) To UBound(LoadMe$)
  18.     Print i, LoadMe$(i)
  20. Print "Time for Split was:"; splitTime
  22.  
  23. ' reset for tokenize
  24. ReDim LoadMe$(1 To 1)
  25. startTokenize = Timer(.001)
  26. tokenize b$, deli$, LoadMe$()
  27. TokenizeTime = Timer(.001) - startTokenize
  28. For i = LBound(LoadMe$) To UBound(LoadMe$)
  29.     Print i, LoadMe$(i)
  31. Print "Time for Tokenize was:"; TokenizeTime
  32.  
  33.  
  34.  
  35. ' note: I buggered this twice now, FOR base 1 array REDIM MyArray (1 to 1) AS ... the (1 to 1) is not same as (1) which was the Blunder!!!
  36. 'notes: REDIM the array(0) to be loaded before calling Split '<<<< IMPORTANT dynamic array and empty, can use any lbound though
  37. 'This SUB will take a given N delimited string, and delimiter$ and create an array of N+1 strings using the LBOUND of the given dynamic array to load.
  38. 'notes: the loadMeArray() needs to be dynamic string array and will not change the LBOUND of the array it is given.  rev 2019-08-27
  39. Sub Split (SplitMeString As String, delim As String, loadMeArray() As String)
  40.     Dim curpos As Long, arrpos As Long, LD As Long, dpos As Long 'fix use the Lbound the array already has
  41.     curpos = 1: arrpos = LBound(loadMeArray): LD = Len(delim)
  42.     dpos = InStr(curpos, SplitMeString, delim)
  43.     Do Until dpos = 0
  44.         loadMeArray(arrpos) = Mid$(SplitMeString, curpos, dpos - curpos)
  45.         arrpos = arrpos + 1
  46.         If arrpos > UBound(loadMeArray) Then ReDim _Preserve loadMeArray(LBound(loadMeArray) To UBound(loadMeArray) + 1000) As String
  47.         curpos = dpos + LD
  48.         dpos = InStr(curpos, SplitMeString, delim)
  49.     Loop
  50.     loadMeArray(arrpos) = Mid$(SplitMeString, curpos)
  51.     ReDim _Preserve loadMeArray(LBound(loadMeArray) To arrpos) As String 'get the ubound correct
  53.  
  54. Sub tokenize (toTokenize As String, delimiters As String, StorageArray() As String)
  56.         Function strtok%& (ByVal str As _Offset, delimiters As String)
  57.     End Declare
  58.     Dim As _Offset tokenized
  59.     Dim As String tokCopy: tokCopy = toTokenize + Chr$(0)
  60.     Dim As String delCopy: delCopy = delimiters + Chr$(0)
  61.     Dim As _Unsigned Long lowerbound: lowerbound = LBound(StorageArray)
  62.     Dim As _Unsigned Long i: i = lowerbound
  63.     tokenized = strtok(_Offset(tokCopy), delCopy)
  64.     While tokenized <> 0
  65.         ReDim _Preserve StorageArray(lowerbound To UBound(StorageArray) + 1)
  66.         StorageArray(i) = pointerToString(tokenized)
  67.         tokenized = strtok(0, delCopy)
  68.         i = i + 1
  69.     Wend
  70.     ReDim _Preserve StorageArray(UBound(StorageArray) - 1)
  72.  
  73. Function pointerToString$ (pointer As _Offset)
  75.         Function strlen%& (ByVal ptr As _Unsigned _Offset)
  76.     End Declare
  77.     Dim As _Offset length: length = strlen(pointer)
  78.     If length Then
  79.         Dim As _MEM pString: pString = _Mem(pointer, length)
  80.         Dim As String ret: ret = Space$(length)
  81.         _MemGet pString, pString.OFFSET, ret
  82.         _MemFree pString
  83.     End If
  84.     pointerToString = ret
  86.  
  87.  
Title: Re: Split Versus Tokenize
Post by: SMcNeill on February 01, 2022, 04:17:53 pm
I'd say the redim issue is probably here:

        ReDim _Preserve StorageArray(lowerbound To UBound(StorageArray) + 1)
        StorageArray(i) = pointerToString(tokenized)
        tokenized = strtok(0, delCopy)
        i = i + 1
    Wend
    ReDim _Preserve StorageArray(UBound(StorageArray) - 1)


Notice the difference n the first and last lines?  The last line doesn't have the lowerbound set in the ReDim statement.
Title: Re: Split Versus Tokenize
Post by: bplus on February 01, 2022, 04:30:20 pm
Yep! Fixed with lowerbound on line 70:
Code: QB64: [Select]
  1. _Title "Tokenize blank line test" ' b+ 2022-02-01
  2. b$ = "Line 1"
  3. For i = 2 To 10
  4.     If i = 4 Or i = 7 Or i = 10 Then
  5.         b$ = b$ + Chr$(10) + "Line" + Str$(i)
  6.     Else
  7.         b$ = b$ + Chr$(10)
  8.     End If
  10.  
  11. deli$ = Chr$(10)
  12.  
  13. ReDim LoadMe$(1 To 1)
  14. startSplit = Timer(.001)
  15. Split b$, deli$, LoadMe$()
  16. splitTime = Timer(.001) - startSplit
  17. For i = LBound(LoadMe$) To UBound(LoadMe$)
  18.     Print i, LoadMe$(i)
  20. Print "Time for Split was:"; splitTime
  22.  
  23. ' reset for tokenize
  24. ReDim LoadMe$(1 To 1)
  25. startTokenize = Timer(.001)
  26. tokenize b$, deli$, LoadMe$()
  27. TokenizeTime = Timer(.001) - startTokenize
  28. For i = LBound(LoadMe$) To UBound(LoadMe$)
  29.     Print i, LoadMe$(i)
  31. Print "Time for Tokenize was:"; TokenizeTime
  32.  
  33.  
  34.  
  35. ' note: I buggered this twice now, FOR base 1 array REDIM MyArray (1 to 1) AS ... the (1 to 1) is not same as (1) which was the Blunder!!!
  36. 'notes: REDIM the array(0) to be loaded before calling Split '<<<< IMPORTANT dynamic array and empty, can use any lbound though
  37. 'This SUB will take a given N delimited string, and delimiter$ and create an array of N+1 strings using the LBOUND of the given dynamic array to load.
  38. 'notes: the loadMeArray() needs to be dynamic string array and will not change the LBOUND of the array it is given.  rev 2019-08-27
  39. Sub Split (SplitMeString As String, delim As String, loadMeArray() As String)
  40.     Dim curpos As Long, arrpos As Long, LD As Long, dpos As Long 'fix use the Lbound the array already has
  41.     curpos = 1: arrpos = LBound(loadMeArray): LD = Len(delim)
  42.     dpos = InStr(curpos, SplitMeString, delim)
  43.     Do Until dpos = 0
  44.         loadMeArray(arrpos) = Mid$(SplitMeString, curpos, dpos - curpos)
  45.         arrpos = arrpos + 1
  46.         If arrpos > UBound(loadMeArray) Then ReDim _Preserve loadMeArray(LBound(loadMeArray) To UBound(loadMeArray) + 1000) As String
  47.         curpos = dpos + LD
  48.         dpos = InStr(curpos, SplitMeString, delim)
  49.     Loop
  50.     loadMeArray(arrpos) = Mid$(SplitMeString, curpos)
  51.     ReDim _Preserve loadMeArray(LBound(loadMeArray) To arrpos) As String 'get the ubound correct
  53.  
  54. Sub tokenize (toTokenize As String, delimiters As String, StorageArray() As String)
  56.         Function strtok%& (ByVal str As _Offset, delimiters As String)
  57.     End Declare
  58.     Dim As _Offset tokenized
  59.     Dim As String tokCopy: tokCopy = toTokenize + Chr$(0)
  60.     Dim As String delCopy: delCopy = delimiters + Chr$(0)
  61.     Dim As _Unsigned Long lowerbound: lowerbound = LBound(StorageArray)
  62.     Dim As _Unsigned Long i: i = lowerbound
  63.     tokenized = strtok(_Offset(tokCopy), delCopy)
  64.     While tokenized <> 0
  65.         ReDim _Preserve StorageArray(lowerbound To UBound(StorageArray) + 1)
  66.         StorageArray(i) = pointerToString(tokenized)
  67.         tokenized = strtok(0, delCopy)
  68.         i = i + 1
  69.     Wend
  70.     ReDim _Preserve StorageArray(lowerbound To UBound(StorageArray) - 1) ' <<< fix with lowerbound
  72.  
  73. Function pointerToString$ (pointer As _Offset)
  75.         Function strlen%& (ByVal ptr As _Unsigned _Offset)
  76.     End Declare
  77.     Dim As _Offset length: length = strlen(pointer)
  78.     If length Then
  79.         Dim As _MEM pString: pString = _Mem(pointer, length)
  80.         Dim As String ret: ret = Space$(length)
  81.         _MemGet pString, pString.OFFSET, ret
  82.         _MemFree pString
  83.     End If
  84.     pointerToString = ret
  86.  
  87.  
  88.  

Score is now Split 3 and Tokenize 1  :)
Text Only | Text with Attachments