Random Access to a Sequential File

[Dimster]

Just wondering ..... A file created for "Output" or "Append" is naturally written sequentially. To access a record in it requires a sequential thumb thru and the larger the sequential file is the longer it takes to arrive at the record you may want. Random Access Files have the obvious advantage of going directly to the record of choice ... so I'm wondering if there maybe a way of using Random Access on a Sequential File. Here some code which give you an idea of what I mean but do not run this code as it will write to a "d" drive if you have one.

Code: QB64: [Select]

1. Dim TestArray(1 To 25)
2. For x = 1 To 25: TestArray(x) = x + 3: Next
3. Open "d:TestFile" For Output As #100
4. For x = 1 To 25: Write #100, x, TestArray(x): Next
5. Close 100
6. Open "d:TestFile" For Append As #100
7. Write #100, 27, 721
8. Close 100
9. Open "d:TestFile" For Input As #100
10. Do While Not EOF(100)
11.     Input #100, RecordNum, Value
12.     Print "Record # "; RecordNum; " Carries a value of "; Value
13. Loop
14. Close 100
15. Open "d:TestFile" For Random As #100 Len = Len(TestFile)
16. Seek #100, 22
17. 'Input #100, RecordNum, Value
18. Get #100, RecordNum, Value
19. Print "Record # "; RecordNum; "Carries a value of "; Value

Needless to say the code doesn't work. It is curious that using either INPUT #100, RecordNum, Value  or GET #100, RecordNum,Value do not show up as an error however running the code with INPUT generates "Bad File Mode", while running it with GET does not produce Record #22 which was being Sought but Record #27 and rather than that value being 721 it comes up as 1.002449E-08

I have tried this same Random opening without using Len = Len(TestFile) and the results are the same with the exception the value = 0 rather than 1.002449E-08.

Moral of the story is Random and Sequential do not mix. I think Sequential needs a SEEK

[Pete]

Sequential files have the advantage of no fixed record length. Without a fixed length, however, there is no way to do a "seek" function, other than to say find the 40th entry by reading all 40 entries, from the start. Thankful that QB64 has LINE INPUT work with open for binary. That's a huge speed increase for searching records in sequential files. Anyway, if there were an option to make sequential files fixed record lengths, then what you want would be possible. Oh wait, then they'd be random access files! :D

Pete

[bplus]

I recommend loading the sequential file into an array, process strings through array, store array back to file when done.
1.Use the binary one gulp method to load file into giant string.
2.Use Split to convert the string to array.
3.Do your business with the array = same as random access only much speedier.
4.Join array to giant string (if only for symmetry) CR + LF as joiner
5.Now just save file for output and drop the giant string into it.

Do you need to see the code?

Update:

Code: QB64: [Select]

1. Option _Explicit 'Load and Save Text File b+ 2021-04-10
2.  
3. ReDim Shared CRLF As String
4. CRLF = Chr$(13) + Chr$(10) ' delimiter for txt files
5.  
6. ReDim As Long i, fCnt, lNum
7. ReDim txt$(1 To 100) ' make an array of txt
8. For i = 1 To 100
9.     txt$(i) = "This is line" + Str$(i)
10. Next
11. SaveArr2File txt$(), "Test 100 strings.txt" 'save to file
12. ReDim test$(1 To 1) '                        create array to store file contents into
13. fCnt = LoadFile2ArrCnt&("Test 100 strings.txt", test$()) 'load file
14. For i = 1 To fCnt 'show loaded file notice, it is 100 lines
15.     Print test$(i)
16. Next
17. Do 'access line records randomly
18.     Print
19.     Input "Enter line / record to check (0 quits)"; lNum
20.     If lNum = 0 Or lNum > UBound(test$) Then System
21.     Print test$(lNum)
22. Loop
23.  
24. Sub SaveArr2File (Arr() As String, FileName$) 'assuming base 1 file loaded with code below
25.     ReDim As Long i
26.     ReDim b$
27.     For i = 1 To UBound(arr)
28.         If i = 1 Then b$ = Arr(1) Else b$ = b$ + CRLF + Arr(i)
29.     Next
30.     Open FileName$ For Output As #1
31.     Print #1, b$
32.     Close #1
33. End Sub
34.  
35. Function LoadFile2ArrCnt& (TxtFile$, Arr() As String)
36.     Dim b$
37.     If _FileExists(TxtFile$) Then
38.         Open TxtFile$ For Binary As #1
39.         b$ = Space$(LOF(1))
40.         Get #1, , b$
41.         Close #1
42.         ReDim _Preserve Arr(1 To 1) As String
43.         Split b$, CRLF, Arr()
44.         LoadFile2ArrCnt& = UBound(arr)
45.     End If
46. End Function
47.  
48. ' note: I buggered this twice now, FOR base 1 array REDIM MyArray (1 to 1) AS ... the (1 to 1) is not same as (1) which was the Blunder!!!
49. 'notes: REDIM the array(0) to be loaded before calling Split '<<<< IMPORTANT dynamic array and empty, can use any lbound though
50. 'This SUB will take a given N delimited string, and delimiter$ and create an array of N+1 strings using the LBOUND of the given dynamic array to load.
51. 'notes: the loadMeArray() needs to be dynamic string array and will not change the LBOUND of the array it is given.  rev 2019-08-27
52. Sub Split (SplitMeString As String, Delim As String, LoadMeArray() As String)
53.     Dim curpos As Long, arrpos As Long, LD As Long, dpos As Long 'fix use the Lbound the array already has
54.     curpos = 1: arrpos = LBound(loadMeArray): LD = Len(Delim)
55.     dpos = InStr(curpos, SplitMeString, Delim)
56.     Do Until dpos = 0
57.         LoadMeArray(arrpos) = Mid$(SplitMeString, curpos, dpos - curpos)
58.         arrpos = arrpos + 1
59.         If arrpos > UBound(loadMeArray) Then ReDim _Preserve LoadMeArray(LBound(loadMeArray) To UBound(loadMeArray) + 1000) As String
60.         curpos = dpos + LD
61.         dpos = InStr(curpos, SplitMeString, Delim)
62.     Loop
63.     LoadMeArray(arrpos) = Mid$(SplitMeString, curpos)
64.     ReDim _Preserve LoadMeArray(LBound(loadMeArray) To arrpos) As String 'get the ubound correct
65. End Sub
66.  
67.  

[NOVARSEG]

Ok I started out programming in 1983 using random access files etc.  That was largely due to the nature of FLOPPY drives = SLOW.    With Giga bit hard drives etc   a 100 GB file can be read in no time in binary mode, All you need is code to parse your data base file into logical fields that make sense. 

[Dimster]

Thanks for this guys, very helpful. Also, thanks for the code @bplus . It seems to be pretty fast at locating the exact record.

When you have 10 files and each one reaching 4000 sequential lines of data, getting to a record is getting time consuming.

[191Brian]

Another option for large data files to big to hold in memory is to use the random file type and build a record index in an array.

[Pete]

For me, I use this method for loading records into arrays and back into a file...

Code: QB64: [Select]

1. ' Make a temp file...
2. OPEN "$myfile.txt" FOR OUTPUT AS #1
3. FOR i = 1 TO 15
4.     PRINT #1, LTRIM$(STR$(i))
5. NEXT
6. CLOSE #1
7. ' Now load the file into arrays, and then put the arrays back into the file.
8. OPEN "$myfile.txt" FOR BINARY AS #1
9. a$ = SPACE$(LOF(1))
10. GET #1, , a$
11. CLOSE #1
12. i = 0: seed% = 1
13. DO WHILE INSTR(seed%, a$, CHR$(13) + CHR$(10))
14.     i = i + 1
15.     x$ = MID$(a$, seed%, INSTR(seed%, a$, CHR$(13) + CHR$(10)) - seed%)
16.     REM PRINT x$: SLEEP
17.     seed% = seed% + LEN(x$) + 2
18.     REDIM _PRESERVE myarray$(i)
19. LOOP
20. ' Now put it back into the file...
21. OPEN "$myfile.txt" FOR OUTPUT AS #1
22. FOR j = 1 TO i
23.     PRINT j, i, UBOUND(myarray$)
24.     PRINT #1, myarray$(j)
25. NEXT
26. CLOSE #1
27.  

You can unREM the PRINT line to see the result on the screen, but the point is, the file gets created, parsed, and put back, so in this case, unedited, it will be the same.

Pete

[Dimster]

Thanks Pete - Seems from the various approaches, the trick is to first convert the data to string values. In your code you have the lines :

Code: QB64: [Select]

1. Open "$myfile.txt" For Binary As #1
2. a$ = Space$(LOF(1))
3. Get #1, , a$
4. Close #1

That "a$ = Space$(LOF(1))" is it just filling $myfile.txt with blank spaces? and if so, would that not erase the 1 to 15 which was just printed into that file?

Also, the "Get #1, , a$" The double comma's would imply a "skip over the record number" but $myfile.txt was a sequentially created file with no record number or index number written to it. So by opening $myfile.txt for BINARY automatically adds a record number as if it was created as a Random access file????? Must admit I avoid Binary as some what confusing. I do realize that Opening for Binary is going to get the data one bite at a time but why Binary works with GET (a Random Access tool) and not INPUT ( a sequential access tool) in your code has me scratching my head.

[191Brian]

Hi
Files opened as binary don't have to be read a byte at a time. Get with no record size (double comma) reads the whole file in one go.

[SMcNeill]

Hi
Files opened as binary don't have to be read a byte at a time. Get with no record size (double comma) reads the whole file in one go.

Get with no middle parameter simple reads from the spot where you last read from.

From example:

OPEN “temp.txt” FOR BINARY AS #1
PUT #1, 1, “1234567890”
CLOSE #1

OPEN “temp.txt” FOR BINARY AS #1
a$ = SPACE$(1)
FOR I = 1 TO 10
    GET #1, , a$
    PRINT I, a$
NEXT

With the above, we get one byte at a time, starting at the beginning of the file and reading it sequentially one character at a time.

a$ = SPACE$(1) gives us a single character to delimitate how much information we read at a time.  In this case, we only read 1 character.

a$ = SPACE$(LOF(1)) sets a string to the whole length of the file, so the read statement reads the whole file in with one GET statement.

[SMcNeill]

Another example:

OPEN “temp.txt” FOR BINARY AS #1
PUT #1, 1, “1234567890”
CLOSE #1

OPEN “temp.txt” FOR BINARY AS #1
a$ = SPACE$(3)
b$ = SPACE$(2)
c$ = SPACE$(5)
GET #1, , a$
GET #1, , b$
GET #1, , c$
PRINT a$, b$, c$

We get 3-bytes, 2-bytes, 5-bytes, all sequentially, so we end up “123”,  “45”,  “67890”.

[191Brian]

Binary files are great for saving array's to disk as you can write and read the whole array in one go. I use a sequential file and binary files to save the game state in my campsite manager game.

I write the key game variables and the sizes of the arrays to a sequential file then write the array's to binary files see below.

Sub SaveGame
    Dim fH As Long

    fH = FreeFile
    gameName = TextInputBox(100, 250, "Enter your save name", gameName)
    Open saveFolder + gameName + ".save" For Output As fH

    If _FileExists(saveFolder + gameName + ".save") Then
        Print #fH, gameName; ","; Str$(UBound(gameassets)); ","; Str$(UBound(playerassets)); ","; Str$(UBound(bookRec)); ","; Str$(hoursRun); ","; Str$(currDay)
        Close #fH
        Open saveFolder + gameName + ".gasset" For Binary As fH
        Put #fH, , gameAssets()
        Close #fH
        Open saveFolder + gameName + ".passet" For Binary As fH
        Put #fH, , playerAssets()
        Close #fH
        Open saveFolder + gameName + ".booking" For Binary As fH
        Put #fH, , bookRec()
        Close #fH

    Else
        ShowMsgBox 100, 300, "Failed to create save file " + saveFolder + gameName + ".save"
    End If

End Sub

Sub LoadGame
    Dim fH As Long
    Dim xName As String
    Dim gaQty As Integer
    Dim paQty As Integer
    Dim bkQty As Integer
    Dim si As Integer

    fH = FreeFile
    Open saveFolder + gameName + ".save" For Input As fH
    Input #fH, xName, gaQty, paQty, bkQty, hoursRun, currDay
    Close #fH

    ReDim gameAssets(gaQty) As asset
    Open saveFolder + gameName + ".gasset" For Binary As fH
    Get #fH, , gameAssets()
    Close #fH

    ReDim playerAssets(paQty) As asset
    Open saveFolder + gameName + ".passet" For Binary As fH
    Get #fH, , playerAssets()
    Close #fH

    ReDim bookRec(bkQty) As asset
    Open saveFolder + gameName + ".booking" For Binary As fH
    Get #fH, , bookRec()
    Close #fH

End Sub

[Dimster]

Ok, so if I have an array Dimmed (1 to 25) ..Dim MyArray(1 to 25) ...For I= 1 to 25:MyArray(I) = I +2: Next... and the program coding accesses a File and writes the entire array, closes, reopens and writes the entire array 10 more times. I would have 11 sequential writes to the same file carrying 25 pieces of data on each line.

I believe, in the case of MyArray, each element of the array would be 4 bytes. If I wanted to access only record #5 in the sequential file, then that record would start (25*4)*5 bytes away, correct???

So using Binary as the mode, to just extract that sequential record #5 would be

a$= ((Space$(4)*25)*5
get #1,,a$

[191Brian]

Depending on the application I wouldn't bother mixing accessing the file for individual elements from the file I would just use the array and save and load to and from file between sessions. 

[SMcNeill]

Ok, so if I have an array Dimmed (1 to 25) ..Dim MyArray(1 to 25) ...For I= 1 to 25:MyArray(I) = I +2: Next... and the program coding accesses a File and writes the entire array, closes, reopens and writes the entire array 10 more times. I would have 11 sequential writes to the same file carrying 25 pieces of data on each line.

I believe, in the case of MyArray, each element of the array would be 4 bytes. If I wanted to access only record #5 in the sequential file, then that record would start (25*4)*5 bytes away, correct???

So using Binary as the mode, to just extract that sequential record #5 would be

a$= ((Space$(4)*25)*5
get #1,,a$

Not quite....

DIM MyArray (1 TO 25) AS *whatever*

Now, before we can GET or PUT specific records, we need to know how large each record is.  (RANDOM access files work the same way for us, btw.)  To get the length, we just take the LENgth of *whatever*.

For example:

DIM MyArray (1 TO 25) AS LONG
L = LEN(MYArray(1))  ‘L is 4 here, as our type is a LONG.  If we had an INTEGER, L would only be 2.

At this point, you can now open the file for binary access:

OPEN file$ FOR BINARY AS #1

And, to get the Nth record, it’s position is at (N - 1) * L +1.

So the first record:  FirstRecordPosition = (1 - 1) * 4 + 1.  (Or, at byte #1 for my LONG type array above.)
SecondRecordPosition = (2 -1) * 4 + 1 = 5
FifthRecordPosition = (5 -1) * 4 + 1 = 17

Now that you know how to calculate a record position, you just use it for the second parameter of your GET statement.

GET #FileHandle, (DesiredRecordNumber - 1) * RecordLength +1, RecordType

So, for the 5th record of the above, it’d be GET #1, 17, LongVariableType


.....

So if we DIM MyArray(1 TO 25) AS STRING * 11, we’d do the following to get the 5th record:

T$ = SPACE$(11)
GET #1, (5 - 1) * 11 + 1, T$


(Which breaks down to GET #1, 45, T$ to get the 11-byte string that starts at position #45 in the data.)