Text Only | Text with Attachments

QB64.org Forum

Active Forums => QB64 Discussion => Topic started by: hotlnc on October 25, 2018, 02:24:17 pm

Title: There is a sequential file line limit in Basic?
Post by: hotlnc on October 25, 2018, 02:24:17 pm
It looks like I've hit a boundary that I didn't know existed.  I'm doing sequential reads and writes on files that have ASCII text lines (assembler listing output files). My QB64 programs have been working great until I started to write and read files with 10,000+ lines in them. I can process no further than about 10K lines. Is there a line limit in Basic/QB64? I am not holding the lines in memory. I process a line or two and then output that line. So this is not an memory issue.

Is there a work around?
Title: Re: There is a sequential file line limit in Basic?
Post by: SMcNeill on October 25, 2018, 02:30:48 pm
No limit like that.  Can you share your code with us?  I'll be happy to take a look and see if I can help sort out what's going on for you.
Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 25, 2018, 02:48:50 pm
Sure. Here you go.
Title: Re: There is a sequential file line limit in Basic?
Post by: SMcNeill on October 25, 2018, 03:12:00 pm
And any chance for a test file to run it with?  I just downloaded the BAS and am looking for anything which might stand out even now.  :)
Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 25, 2018, 03:23:02 pm
Sure.  Thanks for your help.

The file labeled "GpsTx3G.LST" is small enough to work fine. Output file "GpsTx3G0p0.ASM
The file labeled "J13Gps4G.LST has the 10K limit on the output file: "J13Gps4G0p0.ASM"

Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 25, 2018, 03:26:18 pm
I guess I can only attach one file at a time.  Here is the smaller one.
Title: Re: There is a sequential file line limit in Basic?
Post by: SMcNeill on October 25, 2018, 03:34:12 pm
Oddly enough, I didn't seem to have any issues with it. 

Code: QB64: [Select]
  1. '160115 Added Better support for Multiple Temp Macros (Temp_Mack1, Temp_Mack2 etc)
  2. '160624 Removed black lines at top of certain macro's
  6.  
  7. _TITLE "Macro Flatten to Assembly File -- REV 160624"
  8. '
  9. aCommand = COMMAND$
  10. x1 = LEN(aCommand) 'we need to strip it out, so get SIZE of string
  11. IF x1 > 0 THEN 'IF called by multi-edit, a command will have the entire path and file
  12.     DO WHILE x1 <> 0 'This routine will back through the characters, looking for the first
  13.         IF MID$(aCommand, x1, 1) <> "\" THEN 'back slash. The characters preceeding the back slash is the file name
  14.             x1 = x1 - 1
  15.         ELSE
  16.             EXIT DO
  17.         END IF
  18.     LOOP
  19.     aFile1 = MID$(aCommand, x1 + 1)
  21. '-----------------------------------------------------------------
  22. aCommand = COMMAND$
  23. '   aCommand = "trolling"
  24. 'aCommand = "c:\mplabfls\active\" + aFile1 + "\" + aFile1
  25. aCommand = ".\J13Gps4G"
  26. x2 = LEN(aCommand)
  27. x1 = x2
  28.  
  29. DO UNTIL MID$(aCommand, x2, 1) = "\"
  30.     x2 = x2 - 1
  31.     IF x2 <= 1 THEN
  32.         EXIT DO
  33.     END IF
  35.  
  36. IF x2 <= 1 THEN
  37.     PRINT aCommand
  38.     END 'STOP
  40.  
  41. 'x2 and x1 ok
  42. aPath = LEFT$(aCommand, x2)
  43. aFile = MID$(aCommand, x2 + 1, x1 - x2)
  44.  
  45. aSource = aPath + aFile + ".LST"
  46. PRINT "Source--->"; aSource
  47. aDest = aPath + "combo.asm"
  48. aReverse = aPath + "Rev0p0.asm"
  49. PRINT "Dest----->"; aDest
  50. aFind = "                       list"
  51. '----------------------------------------
  52. OPEN aSource FOR INPUT AS 1
  53. OPEN aDest FOR OUTPUT AS 2
  54. OPEN aReverse FOR OUTPUT AS 3
  55. '----------------------------------------
  57.     IF aNext = "" THEN
  58.         LINE INPUT #1, aIn
  59.     ELSE
  60.         aIn = aNext
  61.         aNext = ""
  62.     END IF
  63.     '----------------------------------------
  64.     IF INSTR(aIn, ".asm") > 0 THEN
  65.         PRINT #2, ";" + aIn
  66.         aIn = LCASE$(aIn)
  67.         x1 = INSTR(aIn, "include")
  68.         x2 = INSTR(aIn, "syscon")
  69.         aIn = MID$(aIn, x1) 'get include line less line number
  70.         IF (x1 > 0) AND (x2 = 0) THEN
  71.             PRINT #3, "        " + aIn
  72.             PRINT aIn
  73.         ELSEIF x2 > 0 THEN
  74.             PRINT #3, ";        " + aIn
  75.             PRINT aIn
  76.         END IF
  77.         aIn = ""
  78.     END IF
  79.     '----------------------------------------
  80.     IF (INSTR(LCASE$(aIn), ".inc") > 0) AND (INSTR(aIn, "#") = 0) THEN
  81.         PRINT #2, ";" + aIn
  82.         PRINT #3, ";" + aIn
  83.         aIn = ""
  84.     END IF
  85.     '----------------------------------------
  86.     IF (INSTR(LCASE$(aIn), ";<delete") > 0) THEN
  87.         aIn = ""
  88.     END IF
  89.     '----------------------------------------
  90.     IF (INSTR(aIn, "Bank15On") > 0) THEN
  91.         aIn = ""
  92.     END IF
  93.     '----------------------------------------
  94.     IF NOT EOF(1) THEN
  95.         LINE INPUT #1, aNext
  96.         '        ELSE
  97.         '            CLOSE 1
  98.         '            print "Closed 1"
  99.         '            END
  100.     END IF
  101.     '----------------------------------------
  102.     IF (INSTR(aNext, "                            ") = 1) THEN
  103.         aIn = aIn + MID$(aNext, 29) 'fix the line
  104.         aNext = ""
  105.     END IF
  106.     '----------------------------------------
  107.     IF (LEN(aIn) > 28) AND (INSTR(aIn, aFind) = 0) THEN 'only work with real lines
  108.         xMacro = INSTR(MID$(aIn, 23, 5), "M")
  109.         IF (INSTR(MID$(aIn, 23, 5), "0") = 1) OR (xMacro = 5) THEN 'macro and valid assembly lines only pass this point
  110.             IF (xMacro = 5) THEN 'Macro Lines are processed here
  111.                 IF LEN(aSaved) > 0 THEN
  112.                     PRINT #2, ";" + aSaved
  113.                     PRINT #3, aSaved
  114.                     x1 = 1: aSpace = ""
  115.                     DO WHILE MID$(aSaved, x1, 1) = " "
  116.                         aSpace = aSpace + " "
  117.                         x1 = x1 + 1
  118.                     LOOP
  119.                     IF LEN(aSpace) > 0 THEN
  120.                         aSpace = aSpace + " " 'add 1 for comment character
  121.                     END IF
  122.                     IF MID$(aIn, 3, 1) <> " " THEN 'only display the macro if code generated in macro
  123.                         GOSUB MakeMacroString
  124.                         IF LEN(aMacro) > 0 THEN
  125.                             PRINT #2, aSpace + aMacro
  126.                         END IF
  127.                     END IF
  128.                     aSaved = ""
  129.                 ELSE
  130.                     IF (MID$(aIn, 3, 1) <> " ") THEN 'OR (INSTR(aIn, "local") > 0)
  131.                         GOSUB MakeMacroString
  132.                         IF LEN(aMacro) > 0 THEN
  133.                             PRINT #2, aSpace + aMacro
  134.                         END IF
  135.                     END IF
  136.                 END IF
  137.             ELSE 'assembly lines are processed
  138.                 IF INSTR(aIn, "]:") = 0 THEN
  139.                     IF (LEN(aSaved) = 0) THEN
  140.                         aSaved = MID$(aIn, 29)
  141.                     ELSE
  142.                         PRINT #2, aSaved
  143.                         PRINT #3, aSaved
  144.                         aSaved = MID$(aIn, 29)
  145.                     END IF
  146.                     IF INSTR(aIn, "This Is The End") > 0 THEN
  147.                         PRINT #2, aSaved
  148.                         PRINT #3, aSaved
  149.                         PRINT aSaved
  150.                         EXIT DO
  151.                     END IF
  152.                 END IF
  153.                 ';----------------------------------------------Process Local Macro's if any
  154.             END IF
  155.         END IF
  156.     END IF
  161. aFile1 = aPath + aFile1 + "0p0.ASM"
  162.  
  163. PRINT aSource
  164. PRINT aDest
  165. PRINT aFile1
  166. 'KILL aFile1
  167. 'NAME aDest AS aFile1
  168. '        system
  171. '----------------------------------------------Subroutines
  172. MakeMacroString:
  173. aMacro = MID$(aIn, 29)
  174. x1 = 1
  175. DO UNTIL MID$(aMacro, x1, 1) <> " "
  176.     x1 = x1 + 1
  178. aMacro = "  " + MID$(aMacro, x1)
  179. '----------------------------------------------Process temp labels
  180. IF INSTR(aMacro, "local   Mac_Temp") > 0 THEN
  181.     aMacro = "" 'delete local macros
  182.     xTempLabel = xTempLabel + 1
  183.     aTempLabel = "_" + MID$(STR$(xTempLabel), 2)
  184. ELSEIF INSTR(aMacro, "Mac_Temp") > 0 THEN
  185.     x1 = INSTR(aMacro, "Mac_Temp")
  186.     x2 = INSTR(aMacro, "Mac_Temp:")
  187.     x2a = INSTR(aMacro, ":")
  188.     x3 = INSTR(aMacro, "_")
  189.     x4 = INSTR(aMacro, "goto    Mac_Temp")
  190.     x5 = INSTR(aMacro, "bra     Mac_Temp")
  191.     x6 = INSTR(aMacro, "bnz     Mac_Temp")
  192.     x7 = INSTR(aMacro, "set     Mac_Temp")
  193.     IF x3 < x1 THEN 'strip any label present and add new one
  194.         aMacro = LEFT$(aMacro, x3 - 1) + aTempLabel + MID$(aMacro, x1)
  195.     ELSEIF x2 > 0 THEN 'install new label on numberless labels
  196.         aMacro = LEFT$(aMacro, x2 - 1) + aTempLabel + MID$(aMacro, x2)
  197.     ELSEIF (x4 > 0) OR (x5 > 0) OR (x6 > 0) OR (x7 > 0) THEN 'INstall lable on forward defined labels
  198.         aMacro = LEFT$(aMacro, x1 - 1) + aTempLabel + MID$(aMacro, x1)
  199.     ELSEIF (x2 = 0) AND (x2a > 0) THEN
  200.         aMacro = aTempLabel + LEFT$(aMacro, x2a)
  201.         x8 = INSTR(aMacro, "  ")
  202.         IF x8 > 0 THEN
  203.             aMacro = LEFT$(aMacro, x8 - 1) + MID$(aMacro, x8 + 2)
  204.         END IF
  205.     END IF
  207.  
  209. '-----------------------------------------------------------------------------------------------

The only thing I did here was change the aCommand to: aCommand = ".\J13Gps4G" as I'd copied the file into my QB64 directory and ran it. 

Attached are the output files, which clock in at almost 16,000 lines.


EDIT:
When it runs, it mentions an "include "c:\...\macros_j13_c.asm"   <--- is this an included file which it needs to translate as well?  If so, the issue may be in the code that handles processing it somehow, as I don't have such a file/directory on my PC for it to find and glitch out on.
Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 25, 2018, 03:36:49 pm
Noted on changing the path.

Look at the output file REV0p0.  It terminated at the same spot mine did -- about 10K lines.
Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 25, 2018, 03:38:57 pm
The program doesn't crash or anything -- it just stops inputting/outputting (can't tell which.)
Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 25, 2018, 03:42:59 pm
The file you pointed out "include "c:\...\macros_j13_c.asm"" is a file that was a macro file that was included and is no longer needed.  That line is commented out for the output file.
Title: Re: There is a sequential file line limit in Basic?
Post by: SMcNeill on October 25, 2018, 03:43:54 pm
It's processing to the end of the file...

Here's the end of combo.asm:
Code: QB64: [Select]
  1. ;----------------------------------------
  2.                 ;       11111111        Default
  3. CONFIG2H:       equ     11111111b       ;CONFIGURATION REGISTER 2 HIGH (BYTE ADDRESS 300003h)
  4.                 ;       UUUU....-Unimplemented: Program the corresponding Flash Configuration bit to ‘1’
  5.                 ;           1111-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:32,768
  6.                 ;           1110-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:16,384   65.6 seconds WDT
  7.                 ;           1101-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:8,192    Here causes 32.7 seconds WDT trigger
  8.                 ;           1100-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:4,096
  9.                 ;           1011-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:2,048
  10.                 ;           1010-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:1,024
  11.                 ;           1001-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:512
  12.                 ;           1000-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:256

And the end of Rev0p0.asm is exactly the same:

Code: QB64: [Select]
  1. ;----------------------------------------
  2.                 ;       11111111        Default
  3. CONFIG2H:       equ     11111111b       ;CONFIGURATION REGISTER 2 HIGH (BYTE ADDRESS 300003h)
  4.                 ;       UUUU....-Unimplemented: Program the corresponding Flash Configuration bit to ‘1’
  5.                 ;           1111-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:32,768
  6.                 ;           1110-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:16,384   65.6 seconds WDT
  7.                 ;           1101-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:8,192    Here causes 32.7 seconds WDT trigger
  8.                 ;           1100-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:4,096
  9.                 ;           1011-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:2,048
  10.                 ;           1010-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:1,024
  11.                 ;           1001-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:512
  12.                 ;           1000-WDTPS<3:0>: Watchdog Timer Postscale Select bits -- 1:256

There's something in the middle which isn't being processed 100%, it seems.
Title: Re: There is a sequential file line limit in Basic?
Post by: SMcNeill on October 25, 2018, 03:49:27 pm
Using WinMerge (which lets us compare the differences in 2 files), it's easy to spot some of the items that aren't the same:

combo.ASM:
Code: QB64: [Select]
  1. SevenDecade_Ascii2Binary:       ;this is stored in accumulatorH:L
  2. ;        Load    TablePointer,Table_7Decades     ;setup to point in decades table
  3.            Ac set 0        ;make ACCESS Bank since already bank 0
  4.            movlw   upper Table_7Decades
  5.            movwf   TBLPTRU,Ac
  6.            Ac set 0        ;make ACCESS Bank since already bank 0
  7.            movlw   high Table_7Decades
  8.            movwf   TBLPTRH,Ac
  9.            Ac set 0        ;make ACCESS Bank since already bank 0
  10.            movlw   low Table_7Decades
  11.            movwf   TBLPTRL,Ac
  12. ;        move    DivisorL,7              ;set up outer loop counter
  13.            Ac set 1
  14.            movlw   7
  15.            movwf   DivisorL,Ac
  16.         bra     Asc2Bin_0       ;and use common
  17. ;----------------------------------------

Rev0p0.ASM:
Code: QB64: [Select]
  1. SevenDecade_Ascii2Binary:       ;this is stored in accumulatorH:L
  2.         Load    TablePointer,Table_7Decades     ;setup to point in decades table
  3.         move    DivisorL,7              ;set up outer loop counter
  4.         bra     Asc2Bin_0       ;and use common
  5. ;----------------------------------------

There's several lines there missing, and the file is riddled with these deletions, which is what's dropping the length down from ~16000 lines to ~9000.   
Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 25, 2018, 04:02:39 pm
The purpose of this program is to flatten an listing file that was generated by a macro assembler.  The macros have been expanded and the program is very hard to read.  The Combo program removes the macro lines and just shows native code. This could can then be compiled with an assembler.

Using your example, after running COMBO.BAS, it would look like this:
Code: QB64: [Select]
  1. SevenDecade_Ascii2Binary:       ;this is stored in accumulatorH:L
  2. ;        Load    TablePointer,Table_7Decades     ;setup to point in decades table
  3.            Ac set 0        ;make ACCESS Bank since already bank 0
  4.            movlw   upper Table_7Decades
  5.            movwf   TBLPTRU,Ac
  6.            Ac set 0        ;make ACCESS Bank since already bank 0
  7.            movlw   high Table_7Decades
  8.            movwf   TBLPTRH,Ac
  9.            Ac set 0        ;make ACCESS Bank since already bank 0
  10.            movlw   low Table_7Decades
  11.            movwf   TBLPTRL,Ac
  12. ;        move    DivisorL,7              ;set up outer loop counter
  13.            Ac set 1
  14.            movlw   7
  15.            movwf   DivisorL,Ac
  16.         bra     Asc2Bin_0       ;and use common
  17.  

Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 25, 2018, 04:07:39 pm
The REV0p0 file uses the macro file and is not a flattened file. It simply reverses the listing process and regurgitates the pre-combo file.

Look at the file "J13Gps4G0p0.ASM"

It terminates too early. There is more in the LST file to process, but nothing comes out.
Title: Re: There is a sequential file line limit in Basic?
Post by: FellippeHeitor on October 25, 2018, 04:51:41 pm
Quote from: hotlnc on October 25, 2018, 04:02:39 pm
How do you make those "code" blocks?

There's a button with a # when you're writing a post. You can also manually enter [ code] and [/ code] (without the spaces) and whatever you type in between those tags will be put in a codebox.

Welcome to the forum, btw.
Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 25, 2018, 04:55:35 pm
Noted.  Thanks. I guess I should have placed my cursor on the # sign. It would have told me what it was...
Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 27, 2018, 05:29:10 pm
Opps.  I was using an integer to count.  Using a long integer I get better line counts:

As far as I can tell, I'm getting an early End Of File. And always at the same spot. I'm processing 53,313 lines and that is it.

Using a different input file, I can process 51,770 lines and have good results.  The entire file is processed.

I've backed up to qb64-1.1-20160902.48 and got the same results, although this version doesn't like tab characters in the Basic language.

Do you guys have any ideas for me to try?
Title: Re: There is a sequential file line limit in Basic?
Post by: Pete on October 27, 2018, 06:04:48 pm
If one of those lines being read has a CHR$(26) in it, that's what is causing the read to stop. Sequential files read CHR$(26) as end of file, so the file ends, regardless if there are other lines in the file past it.

Try using FOR BINARY instead of FOR INPUT. BINARY is immured to the end of file character reaction. So if the works, that was the problem.

This may not be your problem, but it is something I discovered working with sequential files many years ago, when I included CHR$(26) as an arrow symbol in a help database. As soon as the read encountered it, the file stopped being read.

Pete
Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 27, 2018, 08:13:12 pm
Thanks for the reply.

Changing "For Input" with "For Binary" made the program run much, much faster.  But same result.

I busted the input file into two parts -- one large; one small. the small one contains the chr$(26) spot.

Running the program on both files, I get good output from the larger, except it is missing the last line of the file.  The second file (about 850 lines) terminated at the same spot it did when it was all one file.

I've replace the listing area around where the there could be an embedded chr$(26), but got same result.

Do you remember how DOS marks the EOF?

 
Title: Re: There is a sequential file line limit in Basic?
Post by: Pete on October 27, 2018, 08:38:39 pm
Here is an example of how CHR$(26) acts differently in SEQUENTIAL vs BINARY file reading...

Code: QB64: [Select]
  1. OPEN "forumtest.txt" FOR OUTPUT AS #1
  2. PRINT #1, "This is an EOF character test."
  3. PRINT #1, "The next line contains a chr$(26)."
  4. PRINT #1, "Here is an " + CHR$(26) + " arrow character."
  5. PRINT #1, "This concludes the test."
  7.  
  8. PRINT "Test #1 - Sequential"
  9. OPEN "forumtest.txt" FOR INPUT AS #1
  11.     LINE INPUT #1, a$
  12.     PRINT a$
  16. PRINT "Test #2 - Binary"
  17.  
  18. OPEN "forumtest.txt" FOR BINARY AS #1
  20.     LINE INPUT #1, a$
  21.     PRINT a$
  24.  
  25.  
Title: Re: There is a sequential file line limit in Basic?
Post by: SMcNeill on October 27, 2018, 08:46:54 pm
If you think the issue is end of file termination, write a routine to check against LOF.

DO
  DO
     LINE INPUT #1, whatever$
     S = SEEK(1) 'know the byte position
  LOOP UNTIL EOF(1)
  IF S < LOF(1) THEN 'If position is less than the file length
     SEEK #1, S + 1 'advance past that termination character
  END IF
LOOP UNTIL S >= LOF(1)
Title: Re: There is a sequential file line limit in Basic?
Post by: hotlnc on October 28, 2018, 12:34:35 pm
Well, it passed your test. My problem is not an issue of an early EOF.

The passed result on your test forces me to look into the program for an error.

And I think I found the problem.
Code: QB64: [Select]
  1.             IF (INSTR (MID$ (aIn, 23, 5), "0") = 1) OR (xMacro = 5) THEN    'macro and valid assembly lines only pass this point
  2. Should read:
  3.             IF VAL (MID$ (aIn, 23, 5)) > 0 OR xMacro = 5 THEN    'macro and valid assembly lines only pass this point
  4.  
This is why the program was terminating early -- the line number went from 09999 to 10000 and failed the first conditional INSTR -- there was no ASCII "0" in position #1.
At first I fixed it by replacing "= 1" to "> 0". But that would limit the line number to less than 11111.

So I used the VAL statement and that allows  the number to be larger.

Thanks for the help guys.
Text Only | Text with Attachments