Looking for a speedy way search through the contents of text files

[hanness]

As a part of a program that I have written, I parse through a number of log files looking for errors. I know that the error messages that I am looking for will always contain the word "Error" starting at the 22nd character of the line.

Up until now, the way I handle this is that I open the log file, read a line of text, check to see if the word "Error" appears starting at the 22nd character of the line, then proceed to the next line of text. I repeat this process until I reach the end of the file.

The problem with this method is that it is slow. The log files I am searching can be 40MB+ each and I may have to parse 25+ log files.

My question: Rather than reading in one line at a time, is there any way to search the entire file at once in QB64? My thinking is that I would search the entire file, then if I find the word "Error" I would check to see if this is located starting at the 22nd character of the line. My hope is that if such a method exists, it might be faster than reading in one line at a time.

[FellippeHeitor]

Maybe not exactly what you're after, but there are some tips to it:
1- Are you opening that file FOR INPUT? Cause you can open files to LINE INPUT but use FOR BINARY in the OPEN line. That'll speed things up greatly.

-= OR =-

2- You can read the whole file at once like this:

Code: QB64: [Select]

1. OPEN "myBigFile.txt" FOR BINARY AS #1
2. a$ = SPACE$(LOF(1))
3. GET #1, 1, a$
4. CLOSE #1

Then you can parse it, like this:

Code: QB64: [Select]

1. foundError = INSTR(a$, "Error")
2. IF foundError THEN
3.     'At this point, the variable foundError contains the first occurrence, if any, of the word "Error" in the file.
4.     DO
5.         'Do something with the data, probably using LEFT$, RIGHT$, MID$
6.         'then find the next occurrence of "Error":
7.         foundError = INSTR(foundError + 1, a$, "Error")
8.     LOOP WHILE foundError > 0 'the loop instruction will only go back to the beginning of the block if foundError > 0, that is, if we still have "Error" in this file
9. END IF

[Pete]

I'd say Fell and I use the same system, so rather than post what I do, which would just be different variable names, I'll try to explain a bit what is happening here.

First, The entire file gets loaded into memory with a BINARY file read. To load the whole file at once, we find the length of the file LOF(1), as used in Fell's example, and fill a variable with that many spaces: a$ = SPACE$(LOF(1)). If you are not familiar with the GET statement, it is asking to start at the first record in the file, the "1" in the middle, and it grabs as many bytes as defined by the length of the variable, which in this case means get the entire length of the file, or all the records. So those SPACEs are no longer spaces, now they are your complete file records. To parse that mother, we need to step through it and find each instance of the search term, "Error". The INSTR() function has this cool seed feature. Although most of the time we see INSTR() with 2 parameters, the variable followed by the character or characters to search for, the seed option allows us to move the index through our file to the previous find. Fell adds a +1 to that, so the record is indexed one character past the "Error" so essentially the string (file) is now read starting as rror....( and all the rest of the stuff, until the next "Error" term is found in the loop. This continues until the seed, in Fell's example, foundError, is +1 past the "E" in the last "Error" find. That means in the next loop the INSTR() value is zero, and therefore the loop is exited.

Happy parsing!

Pete

[hanness]

Excellent! Thanks for the suggestions. I'm going to try these to see what kind of results I get.

[hanness]

Looking at the responses I received, I think I have a framework for figuring this out. The one monkey wrench in the works is that the word "Error" can appear in other places in the file, but I want only occurrences where that word appears at precisely the 22nd character of any line of text.

Looking at the file in detail I can see that each line of text ends with ASCII 13 10 (I guess that's basically carriage return / line feed). So that will will let me break up the huge string into individual lines which I can then parse.

Do you really think that opening it for BINARY and doing all this processing would still be faster than opening for INPUT and doing individual LINE INPUT statements?

[SMcNeill]

Looking at the responses I received, I think I have a framework for figuring this out. The one monkey wrench in the works is that the word "Error" can appear in other places in the file, but I want only occurrences where that word appears at precisely the 22nd character of any line of text.

Looking at the file in detail I can see that each line of text ends with ASCII 13 10 (I guess that's basically carriage return / line feed). So that will will let me break up the huge string into individual lines which I can then parse.

Do you really think that opening it for BINARY and doing all this processing would still be faster than opening for INPUT and doing individual LINE INPUT statements?

Open it for binary with Line Input.

OPEN “datafile.txt” FOR BINARY AS #1
DO UNTIL EOF(1)
    LINE INPUT #1, text$
    IF MID$(22, text$, 5) = “Error” THEN ‘Do your error stuff
LOOP


Files opened for INPUT read a single byte at a time to allow synchronized READ/WRITE access.  Files opened for BINARY read by disk sector size (usually 512 bytes at a time), so require MUCH fewer disk reads and performs several hundred times faster.

[RhoSigma]

If you wanna go some deeper, your problem sounds like a perfect application for using this: https://www.qb64.org/forum/index.php?topic=2101.msg113328#msg113328

[Pete]

QB64 has remarkable speed when opening a file for BINARY, even if you read it line by line, like in Steve's example. Now you could parse out the eol characters CHR$(13) + CHR$(10) and get the same results with something like this, too....

CAUTION This demo makes a file named tmpx123.tmp in the directory you run this program in. On the rare chance you already have a file named tmpx123.tmp in that folder DO NOT RUN THIS PROGRAM until you change the file name or your file will be overwritten with this test data.

Code: QB64: [Select]

1. a$(1) = "This is a test of a file parsing algorithm."
2. a$(2) = "If the word......... Error #1"
3. a$(3) = "This is the second.. Error #2"
4. a$(4) = "Here is some more text on a line."
5. a$(5) = "That concludes this simple presentation"
6. a$(6) = "To search the word.. Error #3 at position 22 of any line."
7. OPEN "tmpx123.tmp" FOR OUTPUT AS #1
8. FOR i = 1 TO 6
9.     PRINT #1, a$(i)
10. NEXT
11. CLOSE #1
12.  
13. ' Now we can see if the search term "Error" is at any line #22, even when loading the entire file at once."
14. ff = FREEFILE
15. OPEN "tmpx123.tmp" FOR BINARY AS #ff
16. x$ = SPACE$(LOF(ff))
17. GET #ff, 1, x$
18. CLOSE #ff
19.  
20. i = 0 ' Variable just to keep track of which line has the search term in it...
21. search$ = "Error"
22. DO
23.     i = i + 1
24.     a$ = MID$(x$, 1, INSTR(x$, CHR$(13) + CHR$(10)) - 1)
25.     x$ = MID$(x$, LEN(a$) + 3)
26.     IF LEFT$(x$, 1) = CHR$(10) THEN a$ = "": x$ = MID$(x$, 3)
27.     IF MID$(a$, 22, LEN(search$)) = search$ THEN PRINT i, a$ ' Looks for the word "Error" 22 characters in on any line.
28. LOOP

Pete