QB64.org Forum

Active Forums => Programs => Topic started by: SMcNeill on January 27, 2021, 10:43:15 am

Title: File To Array
Post by: SMcNeill on January 27, 2021, 10:43:15 am

Constantly, folks are asking for the quickest way to load a large file into an array for their needs.  Usually, we start off simple by just telling them, "Use FOR BINARY, rather than FOR INPUT in your code.  It saves a ton of time!" 

Yet, there's always somebody who then comes along and says, "Is there a way faster than that?"

And the answer has always been: "Load it all at once and parse it."

BUT NOT ANYMORE!!  I NOW BRING YOU THE LATEST, THE GREATEST, THE BESTEST NEW.....

FileToArray!!

Code: QB64: [Select]

1. REDIM lines(0) AS STRING
2. DEFLNG A-Z
3.  
4. 'make a quick file for testing purposes
5. PRINT "Creating data file.  Please wait a moment....."
6. OPEN "temp.txt" FOR OUTPUT AS #1
7. FOR i = 1 TO 1000000
8.     PRINT #1, "This is line #"; i; ". And here's my number: "; INT(RND * 10000000)
9. NEXT
10. CLOSE
11. PRINT "Data file created.  Now doing time tests...."
12.  
13.  
14. 'Start the timer
15. t## = TIMER
16. FileToArray "temp.txt", lines()
17. t1## = TIMER
18. count = UBOUND(lines) - LBOUND(lines)
19. PRINT USING "Loaded and parsed #,###,### lines, in ###.#### seconds."; count, t1## - t##
20.  
21. FOR i = LBOUND(lines) TO LBOUND(lines) + 9
22.     PRINT i; lines(i)
23. NEXT
24.  
25. PRINT
26.  
27. 'and here's an example using an array which starts at index 1, instead of 0
28.  
29. REDIM lines(1 TO 1) AS STRING
30. t## = TIMER
31. FileToArray "temp.txt", lines()
32. t1## = TIMER
33. count = UBOUND(lines) - LBOUND(lines)
34. PRINT USING "Loaded and parsed #,###,### lines, in ###.#### seconds."; count, t1## - t##
35.  
36. FOR i = LBOUND(lines) TO LBOUND(lines) + 9
37.     PRINT i; lines(i)
38. NEXT
39. SLEEP
40.  
41.  
42.  
43. CLS
44. PRINT "And to compare times..."
45. count = 1000000
46. t## = TIMER(0.001)
47. REDIM lines(1000000) AS STRING 'I'm going to cheat here and correctly size the lines array from the start
48. '               to shave off a little time constantly resizing it, and to keep my test code down to just
49. '               the time it takes for the program to load.
50. OPEN "temp.txt" FOR INPUT AS #1
51. FOR i = 1 TO 1000000
52.     LINE INPUT #1, lines(i)
53. NEXT
54. CLOSE
55. t1## = TIMER(0.001)
56. PRINT USING "Using FOR INPUT, I loaded and parsed #,###,### lines, in ###.#### seconds."; count, t1## - t##
57.  
58. t## = TIMER(0.001)
59. REDIM lines(1000000) AS STRING 'I'm going to cheat here and correctly size the lines array from the start
60. 'to shave off a little time constantly resizing it, and to keep my test code down to just
61. 'the time it takes for the program to load.
62. OPEN "temp.txt" FOR BINARY AS #1
63. FOR i = 1 TO 1000000
64.     LINE INPUT #1, lines(i)
65. NEXT
66. CLOSE
67. t1## = TIMER(0.001)
68. PRINT USING "Using FOR BINARY, I loaded and parsed #,###,### lines, in ###.#### seconds."; count, t1## - t##
69.  
70. t## = TIMER(0.001)
71. FileToArray "temp.txt", lines()
72. t1## = TIMER(0.001)
73. PRINT USING "Using FileToArray, I loaded and parsed #,###,### lines, in ###.#### seconds."; count, t1## - t##
74.  
75.  
76. SUB FileToArray (file$, FileToArray_lines() AS STRING)
77.     DIM FileToArray_CRLF AS STRING
78.     FileToArray_Handle = FREEFILE
79.     OPEN file$ FOR BINARY AS #FileToArray_Handle
80.     temp$ = SPACE$(LOF(FileToArray_Handle))
81.     GET #FileToArray_Handle, 1, temp$
82.     CLOSE #FileToArray_Handle
83.     'find CRLF
84.     IF INSTR(temp$, CHR$(13)) THEN FileToArray_CRLF = CHR$(13)
85.     IF INSTR(temp$, CHR$(10)) THEN FileToArray_CRLF = CHR$(10)
86.     IF INSTR(temp$, CHR$(13) + CHR$(10)) THEN FileToArray_CRLF = CHR$(13) + CHR$(10)
87.     IF INSTR(temp$, CHR$(10) + CHR$(13)) THEN FileToArray_CRLF = CHR$(10) + CHR$(13)
88.     'if there's no line endings, then simply just send what we have as a single line.
89.     count = LBOUND(FileToArray_lines)
90.     IF FileToArray_CRLF = "" THEN REDIM FileToArray_lines(count TO count): FileToArray_lines(count) = temp$: EXIT SUB
91.     'parse into an array
92.  
93.     U = 1000000 + count
94.     REDIM FileToArray_lines(count TO U) AS STRING
95.  
96.     DO
97.         L = INSTR(l1, temp$, FileToArray_CRLF)
98.         IF L THEN
99.             IF count > U THEN U = U + 1000000: REDIM _PRESERVE FileToArray_lines(LBOUND(FileToArray_lines) TO U) AS STRING
100.             FileToArray_lines(count) = MID$(temp$, l1, L - l1)
101.             l1 = L + LEN(FileToArray_CRLF)
102.             count = count + 1
103.         ELSE
104.             FileToArray_lines(count) = MID$(temp$, l1)
105.         END IF
106.     LOOP UNTIL L = 0
107.     REDIM _PRESERVE FileToArray_lines(LBOUND(FileToArray_lines) TO count) AS STRING
108. END SUB

IT WORKS WITH ARRAYS DIMENSIONED FROM 0!  IT WORKS WITH ARRAYS DIMENSIONED FROM 1!  IT WORKS WITH ARRAYS WITH ANY SORT OF LBOUND THAT YOU CAN IMAGINE, AND THEN IT RESIZES ITSELF FROM THERE!!

Oh my Gawd!!

AND IT'S FAST!  IT'S BLAZING FAST!  AS THE SCREENSHOT BELOW SHOWS!!!

 

AND FOR A LIMITED TIME ONLY, YOU CAN GET IT FOR THE LOW, LOW PRICE OF NOTHING, OR FOR THREE EASY PAYMENTS OF NOTHING!!

TAKE ADVANTAGE OF THIS OFFER AND DON'T LET IT PASS YOU BY NOW!!

*Post in honor of Billy Mays Hayes -- one of the greatest salesmen ever to grace the realm of late night infomercials.

---

But, in all seriousness, guys, this is about as simple as it gets.   REDIM an array as a string to hold your data, then simply call the sub FileToArray with the name of the file, and the name of that array, and let it load and parse it for you.

If anyone finds any little glitches, or cases where it fails to work, feel free to report them and I'll update the routine as I make corrections.

AND YES, PETE, IT DOES WORK IN SCREEN 0!

Title: Re: File To Array
Post by: Pete on January 27, 2021, 11:18:17 am

Sorry Billy Steve, I've already used that method, and for quite some time, on various text projects. Oh well, I guess if you haven't seen it, it's new to you. (That's from Darrin Stevens, some TV ad guy who made that line famous when those Hollywood %^%$#s started shoving re-runs down our throats.) Now that's actually the fasted way to PARSE PETE OFF.

Pete

PS I would not have come up with the control character parsing method, if you hadn't come up with the load the whole  file at once method, many moons ago. I did thank you for that, a few times, but since it's part of this topic, I'm happy to say thanks again! That was a real gem, free too, and lucky for me, I waited to get it on the have one get one free sale.

Title: Re: File To Array
Post by: SMcNeill on January 27, 2021, 12:17:56 pm

Aye; nothing new here.  This is the same way I’ve been loading and parsing strings since the caveman ages of LET.  The only real difference is I’ve finally put the routine in a simple SUB to add to an $INCLUDE library.

Why I’ve never did that before, is beyond me... Honestly, I guess I’m just so used to typing it out, that I simply never considered turning it into a SUB before.  Now that I have though, I’m certain it’ll end up becoming part of my goto toolset.  ;)

Title: Re: File To Array
Post by: bplus on January 27, 2021, 01:27:57 pm

Yeah sorta been doing this already, already have the sweet little wonderful parser Split1000!
https://www.qb64.org/forum/index.php?topic=1607.0

 but this:

Code: QB64: [Select]

1.     'find CRLF
2.     IF INSTR(temp$, CHR$(13)) THEN FileToArray_CRLF = CHR$(13)
3.     IF INSTR(temp$, CHR$(10)) THEN FileToArray_CRLF = CHR$(10)
4.     IF INSTR(temp$, CHR$(13) + CHR$(10)) THEN FileToArray_CRLF = CHR$(13) + CHR$(10)
5.     IF INSTR(temp$, CHR$(10) + CHR$(13)) THEN FileToArray_CRLF = CHR$(10) + CHR$(13)
6.  

would have prevented me from loading files into arrays the slow way with FOR INPUT.

And if we ARE opening to read a txt or bas file or determine if readable and not binary, that would be useful to have test for too.

Update: Oh a clue might be having a line over a certain length.

Text Only | Text with Attachments