Split Versus Tokenize

[bplus]

https://qb64forum.alephc.xyz/index.php?topic=4618.msg140260#msg140260

I think my way of splitting a string into an array is slightly better:

I say Split has at least two advantages!
1. Less LOC
2. Preserves blank lines from a .bas file.

And I am pretty sure for small files there is no significant time differences, not even sure if Tokenize is faster and how big a file you need to see it. I leave that to believers of Tokenize. ;-))

Code: QB64: [Select]

1. _Title "Split versus Tokenize" ' b+ 2022-02-01
2.  
3. ' Is Tokenize really worth the extra LOC compared to Split?
4. ' No! if you want to preserve blank lines.
5.  
6. '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
7.  
8. '            save this file as "Split versus Tokenize.bas"
9.  
10. '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
11.  
12. Open "Split versus Tokenize.bas" For Binary As #1
13. buf$ = Space$(LOF(1))
14.  
15. Get #1, , buf$
16.  
17. Close #1
18.  
19. deli$ = Chr$(13) + Chr$(10)
20.  
21. ReDim LoadMe$(1 To 1)
22.  
23. startSplit = Timer(.001)
24. Split buf$, deli$, LoadMe$()
25. splitTime = Timer(.001) - startSplit
26.  
27. For i = LBound(LoadMe$) To 10
28.     Print i, LoadMe$(i)
29. Next
30. 'For i = UBound(LoadMe$) - 5 To UBound(LoadMe$)
31. '    Print i, LoadMe$(i)
32. 'Next
33. Print "Time for Split was:"; splitTime
34.  
35. ' reset for tokenize
36. ReDim LoadMe$(1 To 1)
37. startTokenize = Timer(.001)
38. tokenize buf$, deli$, LoadMe$()
39. TokenizeTime = Timer(.001) - startTokenize
40. For i = LBound(LoadMe$) To 10
41.     Print i, LoadMe$(i)
42. Next
43. 'For i = UBound(LoadMe$) - 5 To UBound(LoadMe$)
44. '    Print i, LoadMe$(i)
45. 'Next
46. Print "Time for Tokenize was:"; TokenizeTime  ' << edit: had wrong variable in here
47.  
48.  
49.  
50. ' note: I buggered this twice now, FOR base 1 array REDIM MyArray (1 to 1) AS ... the (1 to 1) is not same as (1) which was the Blunder!!!
51. 'notes: REDIM the array(0) to be loaded before calling Split '<<<< IMPORTANT dynamic array and empty, can use any lbound though
52. 'This SUB will take a given N delimited string, and delimiter$ and create an array of N+1 strings using the LBOUND of the given dynamic array to load.
53. 'notes: the loadMeArray() needs to be dynamic string array and will not change the LBOUND of the array it is given.  rev 2019-08-27
54. Sub Split (SplitMeString As String, delim As String, loadMeArray() As String)
55.     Dim curpos As Long, arrpos As Long, LD As Long, dpos As Long 'fix use the Lbound the array already has
56.     curpos = 1: arrpos = LBound(loadMeArray): LD = Len(delim)
57.     dpos = InStr(curpos, SplitMeString, delim)
58.     Do Until dpos = 0
59.         loadMeArray(arrpos) = Mid$(SplitMeString, curpos, dpos - curpos)
60.         arrpos = arrpos + 1
61.         If arrpos > UBound(loadMeArray) Then ReDim _Preserve loadMeArray(LBound(loadMeArray) To UBound(loadMeArray) + 1000) As String
62.         curpos = dpos + LD
63.         dpos = InStr(curpos, SplitMeString, delim)
64.     Loop
65.     loadMeArray(arrpos) = Mid$(SplitMeString, curpos)
66.     ReDim _Preserve loadMeArray(LBound(loadMeArray) To arrpos) As String 'get the ubound correct
67. End Sub
68.  
69. Sub tokenize (toTokenize As String, delimiters As String, StorageArray() As String)
70.     Declare CustomType Library
71.         Function strtok%& (ByVal str As _Offset, delimiters As String)
72.     End Declare
73.     Dim As _Offset tokenized
74.     Dim As String tokCopy: tokCopy = toTokenize + Chr$(0)
75.     Dim As String delCopy: delCopy = delimiters + Chr$(0)
76.     Dim As _Unsigned Long lowerbound: lowerbound = LBound(StorageArray)
77.     Dim As _Unsigned Long i: i = lowerbound
78.     tokenized = strtok(_Offset(tokCopy), delCopy)
79.     While tokenized <> 0
80.         ReDim _Preserve StorageArray(lowerbound To UBound(StorageArray) + 1)
81.         StorageArray(i) = pointerToString(tokenized)
82.         tokenized = strtok(0, delCopy)
83.         i = i + 1
84.     Wend
85.     ReDim _Preserve StorageArray(UBound(StorageArray) - 1)
86. End Sub
87.  
88. Function pointerToString$ (pointer As _Offset)
89.     Declare CustomType Library
90.         Function strlen%& (ByVal ptr As _Unsigned _Offset)
91.     End Declare
92.     Dim As _Offset length: length = strlen(pointer)
93.     If length Then
94.         Dim As _MEM pString: pString = _Mem(pointer, length)
95.         Dim As String ret: ret = Space$(length)
96.         _MemGet pString, pString.OFFSET, ret
97.         _MemFree pString
98.     End If
99.     pointerToString = ret
100. End Function
101.  
102.  
103.  

Edit: had wrong Time variable in showing Tokenize Time. Ha! I was wondering how they were coming exactly the same each time tested.

[_vince]

nice, bplus.  Those tokenize and pointertostring functions should be banned.  Looks like a C compiler swallowed VB.NET -- not BASIC at all!

[bplus]

Looks like a C compiler swallowed VB.NET

LOL

I will bet if we really need a speed bump for some app, we might have to resort to this stuff?

[Cobalt]

LOL

I will bet if we really need a speed bump for some app, we might have to resort to this stuff?

At that point its time to go to a non-translated language. If speed is needed to that point better start learning machine code and using ASM.

[bplus]

Also weird is how some times are negative!?!

Does that mean if we run the program allot we will go back in time?

[SpriggsySpriggs]

The reason you weren't preserving blank lines is because you are using tokenize incorrectly. You should pass it either CHR$(13) or CHR$(10) (can't remember which). If you pass both, it'll split by both. It can take a list of delimiters.

[bplus]

The reason you weren't preserving blank lines is because you are using tokenize incorrectly. You should pass it either CHR$(13) or CHR$(10) (can't remember which). If you pass both, it'll split by both. It can take a list of delimiters.

Both are used as one delimiter for .bas files.

So you are saying I cannot use ", " as 1 delimiter for Tokenize either, I must only use only single char delimiters?

Advantage #3 for Split = variable length delimiters. :-))

[SpriggsySpriggs]

You're intentionally misinterpreting what I'm saying but that's ok. If you prefer to remain ignorant then by all means, please do.

[bplus]

You're intentionally misinterpreting what I'm saying but that's ok. If you prefer to remain ignorant then by all means, please do.

What did I misinterpret, .bas files are really split by Chr$(13) + Chr$(10) you need both in one delimiter.

[bplus]

@SpriggsySpriggs

If you want, I will say Tokenize has advantage of multiple single char delimiters, that must be how you "list" them in a single delimiter string...

Actually if that were true Tokenize should be splitting into even more blank lines?

I suspect Tokenize ignores blank lines? Maybe?

Easy enough to test....

[bplus]

4th advantage of Split = it preserves the Lbound of the storage array.

[SMcNeill]

What did I misinterpret, .bas files are really split by Chr$(13) + Chr$(10) you need both in one delimiter.

This isn't quite true.  :P

Files are delimited according to the CRLF line endings, which vary from OS to OS.

Old versions of Windows used CHR$(13) + CHR$(10) as the CRLF character.
Linux uses CHR$(10) for line endings.
Old Macs use CHR$(13) for line endings.
New Macs have swapped over to CHR$(10) line endings, if I remember correctly.
New versions of Windows have gotten with the program and generally default to the old CHR$(13) + CHR$(10) line endings, but they also now allow you to import and export your line endings however the heck you want!

It's your OS which determines what types of line endings your files have; not the BAS extension itself.  At the end of the day, all a BAS file is, is a TXT file with a little more descriptive extension on it.  ;)

[bplus]

Screw files then here is a test that proves tokenize ignores blank lines and changes the lbound of the load array to 0:

Code: QB64: [Select]

1. _Title "Tokenize blank line test" ' b+ 2022-02-01
2. b$ = "Line 1"
3. For i = 2 To 10
4.     If i = 4 Or i = 7 Or i = 10 Then
5.         b$ = b$ + Chr$(10) + "Line" + Str$(i)
6.     Else
7.         b$ = b$ + Chr$(10)
8.     End If
9. Next
10.  
11. deli$ = Chr$(10)
12.  
13. ReDim LoadMe$(1 To 1)
14. startSplit = Timer(.001)
15. Split b$, deli$, LoadMe$()
16. splitTime = Timer(.001) - startSplit
17. For i = LBound(LoadMe$) To UBound(LoadMe$)
18.     Print i, LoadMe$(i)
19. Next
20. Print "Time for Split was:"; splitTime
21. Sleep
22.  
23. ' reset for tokenize
24. ReDim LoadMe$(1 To 1)
25. startTokenize = Timer(.001)
26. tokenize b$, deli$, LoadMe$()
27. TokenizeTime = Timer(.001) - startTokenize
28. For i = LBound(LoadMe$) To UBound(LoadMe$)
29.     Print i, LoadMe$(i)
30. Next
31. Print "Time for Tokenize was:"; TokenizeTime
32.  
33.  
34.  
35. ' note: I buggered this twice now, FOR base 1 array REDIM MyArray (1 to 1) AS ... the (1 to 1) is not same as (1) which was the Blunder!!!
36. 'notes: REDIM the array(0) to be loaded before calling Split '<<<< IMPORTANT dynamic array and empty, can use any lbound though
37. 'This SUB will take a given N delimited string, and delimiter$ and create an array of N+1 strings using the LBOUND of the given dynamic array to load.
38. 'notes: the loadMeArray() needs to be dynamic string array and will not change the LBOUND of the array it is given.  rev 2019-08-27
39. Sub Split (SplitMeString As String, delim As String, loadMeArray() As String)
40.     Dim curpos As Long, arrpos As Long, LD As Long, dpos As Long 'fix use the Lbound the array already has
41.     curpos = 1: arrpos = LBound(loadMeArray): LD = Len(delim)
42.     dpos = InStr(curpos, SplitMeString, delim)
43.     Do Until dpos = 0
44.         loadMeArray(arrpos) = Mid$(SplitMeString, curpos, dpos - curpos)
45.         arrpos = arrpos + 1
46.         If arrpos > UBound(loadMeArray) Then ReDim _Preserve loadMeArray(LBound(loadMeArray) To UBound(loadMeArray) + 1000) As String
47.         curpos = dpos + LD
48.         dpos = InStr(curpos, SplitMeString, delim)
49.     Loop
50.     loadMeArray(arrpos) = Mid$(SplitMeString, curpos)
51.     ReDim _Preserve loadMeArray(LBound(loadMeArray) To arrpos) As String 'get the ubound correct
52. End Sub
53.  
54. Sub tokenize (toTokenize As String, delimiters As String, StorageArray() As String)
55.     Declare CustomType Library
56.         Function strtok%& (ByVal str As _Offset, delimiters As String)
57.     End Declare
58.     Dim As _Offset tokenized
59.     Dim As String tokCopy: tokCopy = toTokenize + Chr$(0)
60.     Dim As String delCopy: delCopy = delimiters + Chr$(0)
61.     Dim As _Unsigned Long lowerbound: lowerbound = LBound(StorageArray)
62.     Dim As _Unsigned Long i: i = lowerbound
63.     tokenized = strtok(_Offset(tokCopy), delCopy)
64.     While tokenized <> 0
65.         ReDim _Preserve StorageArray(lowerbound To UBound(StorageArray) + 1)
66.         StorageArray(i) = pointerToString(tokenized)
67.         tokenized = strtok(0, delCopy)
68.         i = i + 1
69.     Wend
70.     ReDim _Preserve StorageArray(UBound(StorageArray) - 1)
71. End Sub
72.  
73. Function pointerToString$ (pointer As _Offset)
74.     Declare CustomType Library
75.         Function strlen%& (ByVal ptr As _Unsigned _Offset)
76.     End Declare
77.     Dim As _Offset length: length = strlen(pointer)
78.     If length Then
79.         Dim As _MEM pString: pString = _Mem(pointer, length)
80.         Dim As String ret: ret = Space$(length)
81.         _MemGet pString, pString.OFFSET, ret
82.         _MemFree pString
83.     End If
84.     pointerToString = ret
85. End Function
86.  
87.  

[SMcNeill]

I'd say the redim issue is probably here:

        ReDim _Preserve StorageArray(lowerbound To UBound(StorageArray) + 1)
        StorageArray(i) = pointerToString(tokenized)
        tokenized = strtok(0, delCopy)
        i = i + 1
    Wend
    ReDim _Preserve StorageArray(UBound(StorageArray) - 1)


Notice the difference n the first and last lines?  The last line doesn't have the lowerbound set in the ReDim statement.

[bplus]

Yep! Fixed with lowerbound on line 70:

Code: QB64: [Select]

1. _Title "Tokenize blank line test" ' b+ 2022-02-01
2. b$ = "Line 1"
3. For i = 2 To 10
4.     If i = 4 Or i = 7 Or i = 10 Then
5.         b$ = b$ + Chr$(10) + "Line" + Str$(i)
6.     Else
7.         b$ = b$ + Chr$(10)
8.     End If
9. Next
10.  
11. deli$ = Chr$(10)
12.  
13. ReDim LoadMe$(1 To 1)
14. startSplit = Timer(.001)
15. Split b$, deli$, LoadMe$()
16. splitTime = Timer(.001) - startSplit
17. For i = LBound(LoadMe$) To UBound(LoadMe$)
18.     Print i, LoadMe$(i)
19. Next
20. Print "Time for Split was:"; splitTime
21. Sleep
22.  
23. ' reset for tokenize
24. ReDim LoadMe$(1 To 1)
25. startTokenize = Timer(.001)
26. tokenize b$, deli$, LoadMe$()
27. TokenizeTime = Timer(.001) - startTokenize
28. For i = LBound(LoadMe$) To UBound(LoadMe$)
29.     Print i, LoadMe$(i)
30. Next
31. Print "Time for Tokenize was:"; TokenizeTime
32.  
33.  
34.  
35. ' note: I buggered this twice now, FOR base 1 array REDIM MyArray (1 to 1) AS ... the (1 to 1) is not same as (1) which was the Blunder!!!
36. 'notes: REDIM the array(0) to be loaded before calling Split '<<<< IMPORTANT dynamic array and empty, can use any lbound though
37. 'This SUB will take a given N delimited string, and delimiter$ and create an array of N+1 strings using the LBOUND of the given dynamic array to load.
38. 'notes: the loadMeArray() needs to be dynamic string array and will not change the LBOUND of the array it is given.  rev 2019-08-27
39. Sub Split (SplitMeString As String, delim As String, loadMeArray() As String)
40.     Dim curpos As Long, arrpos As Long, LD As Long, dpos As Long 'fix use the Lbound the array already has
41.     curpos = 1: arrpos = LBound(loadMeArray): LD = Len(delim)
42.     dpos = InStr(curpos, SplitMeString, delim)
43.     Do Until dpos = 0
44.         loadMeArray(arrpos) = Mid$(SplitMeString, curpos, dpos - curpos)
45.         arrpos = arrpos + 1
46.         If arrpos > UBound(loadMeArray) Then ReDim _Preserve loadMeArray(LBound(loadMeArray) To UBound(loadMeArray) + 1000) As String
47.         curpos = dpos + LD
48.         dpos = InStr(curpos, SplitMeString, delim)
49.     Loop
50.     loadMeArray(arrpos) = Mid$(SplitMeString, curpos)
51.     ReDim _Preserve loadMeArray(LBound(loadMeArray) To arrpos) As String 'get the ubound correct
52. End Sub
53.  
54. Sub tokenize (toTokenize As String, delimiters As String, StorageArray() As String)
55.     Declare CustomType Library
56.         Function strtok%& (ByVal str As _Offset, delimiters As String)
57.     End Declare
58.     Dim As _Offset tokenized
59.     Dim As String tokCopy: tokCopy = toTokenize + Chr$(0)
60.     Dim As String delCopy: delCopy = delimiters + Chr$(0)
61.     Dim As _Unsigned Long lowerbound: lowerbound = LBound(StorageArray)
62.     Dim As _Unsigned Long i: i = lowerbound
63.     tokenized = strtok(_Offset(tokCopy), delCopy)
64.     While tokenized <> 0
65.         ReDim _Preserve StorageArray(lowerbound To UBound(StorageArray) + 1)
66.         StorageArray(i) = pointerToString(tokenized)
67.         tokenized = strtok(0, delCopy)
68.         i = i + 1
69.     Wend
70.     ReDim _Preserve StorageArray(lowerbound To UBound(StorageArray) - 1) ' <<< fix with lowerbound
71. End Sub
72.  
73. Function pointerToString$ (pointer As _Offset)
74.     Declare CustomType Library
75.         Function strlen%& (ByVal ptr As _Unsigned _Offset)
76.     End Declare
77.     Dim As _Offset length: length = strlen(pointer)
78.     If length Then
79.         Dim As _MEM pString: pString = _Mem(pointer, length)
80.         Dim As String ret: ret = Space$(length)
81.         _MemGet pString, pString.OFFSET, ret
82.         _MemFree pString
83.     End If
84.     pointerToString = ret
85. End Function
86.  
87.  
88.  

Score is now Split 3 and Tokenize 1  :)