Text Only | Text with Attachments

QB64.org Forum

Active Forums => QB64 Discussion => Topic started by: OldMoses on September 23, 2018, 01:25:09 pm

Title: "Out of Memory" mystery
Post by: OldMoses on September 23, 2018, 01:25:09 pm
UPDATE: Mystery solved, thanks for your help guys.

Been running my updated grain harvest database program. The machine generally sits in the drying room waiting for a truck to come in and someone to enter data. This might occur a couple times an hour to a couple hours between. I updated the program to use four 32 bit display screens and colors, loading 10 button images that range from roughly 2 to 20 kB each in the main module. Once in the program's display/input loop only _PUTIMAGE is used, repeatedly, along with general print, locate and a couple line statements. The basic data handling algorithm has remained largely unchanged and always worked for hours on end before. Only the main display screen is called for updating while awaiting some form of input. All the _PUTIMAGE commands occur during this looping.

After running unattended for somewhere around an hour or two the following error consistently occurs:

A small dialog box with:

"Critical error..." < in the top bar

"Out of Memory" < in the main part of the box

followed by an OK button.

It's never actually happened when I was around to witness it. I ran it this morning and went to church, was gone for a good 2 hours + and true to its M.O., it was stopped when I returned. I don't relish the idea of sitting and staring at it until it occurs...

I opened Notepad to compose this post and it would open, but the dialog box remained on top and until I clicked the OK button; vanquishing both the dialog and my program that caused it, I could do nothing with Notepad.

I've poured over what I believe to be the salient code and can't find an issue so far. The main display/input loop was running _LIMIT 200 and I reasoned that if the program were loading up memory with something while looping unattended, changing to a slower limit value would alter the time it took to throw the error. Does that sound like a reasonable test?

Yesterday I did that. After changing to _LIMIT 30, the program still threw the error around an hour or two. Now I'm wondering if something in the operating environment is to blame. I'm running Windows 10 Home 1803 64bit on a Lenovo Ideapad 330S Core I5 w/ 8GB RAM. QB64 is version 1.2 20180202/85. Sleep modes are disabled while the machine is plugged in.

Does anyone have any experience or insight regarding this issue? I'm nearing the end of my rope...

My next move is to run the application on another system and see if the same thing happens.
Title: Re: "Out of Memory" mystery
Post by: TerryRitchie on September 23, 2018, 01:49:44 pm
Something in your main loop is continuously needing more RAM and taking it. You can verify this by opening Task Manager and watching your program's RAM usage. You'll more than likely see it climbing little by little over time.

Make sure there is nothing like a _LOADIMAGE stuck in your main loop that would cause this type of behavior. Would you be willing to share your code or the main loop of it for investigation?
Title: Re: "Out of Memory" mystery
Post by: SMcNeill on September 23, 2018, 01:50:28 pm
Are you using such commands as _LOADIMAGE, _NEWIMAGE' _LOADFONT, or anything else which loads data into the program?

You say to load 10 button images...  Are you positive that you only load them ONCE??  Without seeing the actual code, all I can do is guess at the issue, but it sounds as if the program has a gradual memory leak.

This will cause your issue (pseudocode only):

DO
    LOADIMAGE
    PUTIMAGE
    DISPLAY
LOOP

In the above, we're loading an image over and over and over, until we crash.   There's 2 basic cures for this:

DO
   LOADIMAGE
   PUTIMAGE
   DISPLAY
   FREEIMAGE
LOOP

Method One:  Free the image after using it, as above.

LOADIMAGE
DO
    PUTIMAGE
    DISPLAY
LOOP

Method Two:  Load the image only once and leave it in memory while the program runs.

********************

Just guessing, I'd guess it's a LOAD without FREE issue, somewhere in your code.

Title: Re: "Out of Memory" mystery
Post by: Pete on September 23, 2018, 01:52:33 pm
Did you update to a different version of QB64 since the time you had it running without a memory issue? I think Rob had to debug QB64 a couple of times for memory leaks. I had one several years ago.

QB45 had far less memory and would throw "Out of Stack Space" error message if a gosub routine was repeatedly looped but not exited properly with a return, as when encountering a GOTO statement within the gosub routine.

I've had a couple of things I've mis-coded in the past, and the exit never occurred in a loop routine, which somewhat quickly lead to the out of memory error. Unfortunately, I can't remember the code the routine at the moment proving that the worst offender for an out of memory error is aging. :(

You'll probably get a lot more feedback shortly. Sorry I' can't be of more help here, I don't do much with non-SCREEN 0 apps. I just wanted to let you know this is situation does occur in other people's programs.

Ah, I see Terry and Steve just responded while I was writing this.

Pete
Title: Re: "Out of Memory" mystery
Post by: OldMoses on September 23, 2018, 02:11:05 pm
Quote from: TerryRitchie on September 23, 2018, 01:49:44 pm
Something in your main loop is continuously needing more RAM and taking it. You can verify this by opening Task Manager and watching your program's RAM usage. You'll more than likely see it climbing little by little over time.

Make sure there is nothing like a _LOADIMAGE stuck in your main loop that would cause this type of behavior. Would you be willing to share your code or the main loop of it for investigation?

I'm fairly certain I've only loaded images in the main module. I hadn't thought of using TM, I'll do that now.

In the meantime, I'm zipping the whole thing along with a sample datafile and a brief "readme" to explain the startup sequence.
Title: Re: "Out of Memory" mystery
Post by: OldMoses on September 23, 2018, 02:14:41 pm
Quote from: SMcNeill on September 23, 2018, 01:50:28 pm

LOADIMAGE
DO
    PUTIMAGE
    DISPLAY
LOOP

Method Two:  Load the image only once and leave it in memory while the program runs.

********************

Just guessing, I'd guess it's a LOAD without FREE issue, somewhere in your code.

This is the method I am trying to use. I can't find anything but putimage in anything that functions from within the loop.
Title: Re: "Out of Memory" mystery
Post by: OldMoses on September 23, 2018, 02:18:44 pm
Running Task Manager on it, it definitely is ticking up in memory usage by about .2 to .3 MB per second or so.
Title: Re: "Out of Memory" mystery
Post by: OldMoses on September 23, 2018, 02:35:09 pm
Alright, Task Manager has yielded some interesting results.

The memory leak only occurs in the main display. It stops when doing data entry.

Also, If I change to a field that has no data in it, the leak slows markedly, suggesting that the leak is in the data displayed. Perhaps my crazy color scheme is doing something. I better look at the data handling subs Echo_Load and Entry_Print more closely.
Title: Re: "Out of Memory" mystery
Post by: TerryRitchie on September 23, 2018, 03:55:01 pm
Quote from: OldMoses on September 23, 2018, 02:18:44 pm
Running Task Manager on it, it definitely is ticking up in memory usage by about .2 to .3 MB per second or so.

Yeah, that's pretty significant. I'm glad Task Manager is giving you some insight. I see others have already downloaded your code. I will later on as well (working on something else right now) but I'm sure the others will probably have an answer for you by then.
Title: Re: "Out of Memory" mystery
Post by: OldMoses on September 23, 2018, 04:46:10 pm
Hi Pete,
I got this computer last month and downloaded the most recent version on the site right after I got it. I was probably running an older version last fall when I first discovered QB64 and used it on an older version (3.2.1) of my program. I can't honestly say whether there would have been memory issues back then as I don't know that I ever let the program just sit on its own long enough. Back then I was sitting down to it and doing data entry sessions that generally lasted under a half hour. It was a blast though, no more DosBox mounting headaches, folks for miles around were wondering what that "squeeee" noise was... :D

Quote from: Pete on September 23, 2018, 01:52:33 pm
Did you update to a different version of QB64 since the time you had it running without a memory issue? I think Rob had to debug QB64 a couple of times for memory leaks. I had one several years ago.
Title: Re: "Out of Memory" mystery
Post by: OldMoses on September 23, 2018, 05:55:07 pm
OK, by commenting out certain SUB calls I've been able to narrow the issue down to any call to SUB Entry_Print. In a loop it causes a continuous memory leak. One other SUB uses Entry_Print, only not in a loop and with no scrolling function. SUB View_File re-displays the data once through. It is a holdover from earlier versions which I kept as it is a way to display changes in dry bushels caused by changing the target moisture. When it runs there is a one time jump in memory usage only. So it seems consistent with any call to Entry_Print being the issue.

Am I just nuts in that I can't see a problem here???

Code: QB64: [Select]
  1. SUB Entry_Print (entry AS LoadRecord)
  2.  
  3.     '-------------------------DISPLAY----------------------------------------------------
  4.     ' SUB: Entry_Print
  5.     '
  6.     ' Purpose:
  7.     ' Displays tabbed data from record number passed by 'entry' when called.
  8.     '
  9.     ' Passed parameters:
  10.     ' entry sends the load record for display
  11.     '
  12.     ' Called from:
  13.     ' Add_Record, Edit_File, Refresh_Screen, Echo_Loads and View_File
  14.     '
  15.     '-----------------------------------------------------------------------------------
  16.  
  17.     PRINT TAB(1); USING "###"; entry.load;
  18.     PRINT TAB(6); entry.Date;
  19.     PRINT TAB(17); entry.Time;
  20.     PRINT TAB(28); RTRIM$(entry.truck);
  21.     PRINT TAB(37); USING "###,###"; entry.Gross;
  22.     PRINT TAB(45); USING "###,###"; entry.Net;
  23.     PRINT TAB(54); USING "#,###.##"; entry.Wet;
  24.     PRINT TAB(64); USING "##.##"; entry.Test;
  25.     PRINT TAB(70); USING "#,###.##"; entry.Dry
  26.  
  27. END SUB 'Entry_Print
  28.  

BTW, View_File is accessed by the "reView" menu choice.
Title: Re: "Out of Memory" mystery
Post by: Cobalt on September 23, 2018, 07:39:13 pm
I don't believe its with that particular sub, running a test program with just a DO:LOOP calling that SUB  and running a program with only a DO:LOOP i saw the same mem seep which topped out on my machine around 38-40 megs and then fluctuated there.
within your actual program there was a major memory jump with a more rapid climb after entering the farm information using CORN option where the memory jumped to ~58megs then climbed by 200-400k every update in task manager.
Title: Re: "Out of Memory" mystery
Post by: OldMoses on September 23, 2018, 10:44:08 pm
Quote from: Cobalt on September 23, 2018, 07:39:13 pm
I don't believe its with that particular sub, running a test program with just a DO:LOOP calling that SUB  and running a program with only a DO:LOOP i saw the same mem seep which topped out on my machine around 38-40 megs and then fluctuated there.
within your actual program there was a major memory jump with a more rapid climb after entering the farm information using CORN option where the memory jumped to ~58megs then climbed by 200-400k every update in task manager.

That's pretty much what I've been getting too. The thing is, I've been able to get the leak to stop by commenting out various instances of calling Entry_Print in those subs that call it. Of course I lose the associated displays when I do so, but most other functionality remains. For instance, SUB Echo_Loads calls Entry_Print in two instances controlled by an IF...THEN structure, one in normal display mode and the other in Edit mode. If I comment out the edit mode call, edit mode goes blank when chosen, the leak stops and I can still edit though I have no visual reference to know which field I'm clicking on. If I return to normal mode the leak continues. The opposite is true if I comment out the normal mode call.

I'm open to the possibility that the leak is caused elsewhere, but if so, the brain teaser is raised exponentially.

I considered the possibility that passing a user defined parameter to Entry_Print was the culprit and adjusted the SUB to accept a straight integer index number and referenced the record variable directly, using the index as an array pointer. I've even tried moving the Entry_Print code out to the subs that were calling it. I found out, as Newt said, "It won't make any difference..."

Now I'm left to wonder if QB64 doesn't like something about the use of TAB or USING in this application. Desperate times call for desperate measures, I suppose.
Title: Re: "Out of Memory" mystery
Post by: SMcNeill on September 23, 2018, 11:20:01 pm
I dub this one a complete mystery.   

I downloaded, extracted the zip into my QB64 folder, and tested the code....   Never once managed to get the memory usage to go above 50MB, no matter what options I tried to select or enter.

The only change I made was to comment out the line making the EXEICON (It couldn't find the icon for whatever reason), and then I let it run and go.  If there's a leak in there somewhere, I'm not seeing it here on my end.  Somebody will need to walk me through the steps to reproduce the issue, before I'll be much help digging into it, I'm afraid. 


EDIT:   Scratch all that.    Test this:

Code: QB64: [Select]
  2.     Entry_Print
  4.  
  5.  
  6. SUB Entry_Print '(entry AS LoadRecord)
  7.  
  8.     '-------------------------DISPLAY----------------------------------------------------
  9.     ' SUB: Entry_Print
  10.     '
  11.     ' Purpose:
  12.     ' Displays tabbed data from record number passed by 'entry' when called.
  13.     '
  14.     ' Passed parameters:
  15.     ' entry sends the load record for display
  16.     '
  17.     ' Called from:
  18.     ' Add_Record, Edit_File, Refresh_Screen, and View_File
  19.     '
  20.     '-----------------------------------------------------------------------------------
  21.  
  22.     PRINT TAB(1); USING "###"; entry.load;
  23.     PRINT TAB(6); entry.Date;
  24.     PRINT TAB(17); entry.Time;
  25.     PRINT TAB(28); RTRIM$(entry.truck$);
  26.     PRINT TAB(37); USING "###,###"; entry.Gross;
  27.     PRINT TAB(45); USING "###,###"; entry.Net;
  28.     PRINT TAB(54); USING "#,###.##"; entry.Wet;
  29.     PRINT TAB(64); USING "##.##"; entry.Test;
  30.     PRINT TAB(70); USING "#,###.##"; entry.Dry
  31.  
  32. END SUB 'Entry_Print
  33.  
Title: Re: "Out of Memory" mystery
Post by: SMcNeill on September 23, 2018, 11:27:23 pm
Removing the use of TAB corrects the issue.  Apparently TAB has a memory leak in it somewhere:

Code: QB64: [Select]
  2.     Entry_Print
  4.  
  5.  
  6. SUB Entry_Print '(entry AS LoadRecord)
  7.  
  8.     '-------------------------DISPLAY----------------------------------------------------
  9.     ' SUB: Entry_Print
  10.     '
  11.     ' Purpose:
  12.     ' Displays tabbed data from record number passed by 'entry' when called.
  13.     '
  14.     ' Passed parameters:
  15.     ' entry sends the load record for display
  16.     '
  17.     ' Called from:
  18.     ' Add_Record, Edit_File, Refresh_Screen, and View_File
  19.     '
  20.     '-----------------------------------------------------------------------------------
  21.  
  22.     LOCATE , 1: PRINT USING "###   "; entry.load;
  23.     LOCATE , 6: PRINT entry.Date;
  24.     LOCATE , 17: PRINT entry.Time;
  25.     LOCATE , 28: PRINT RTRIM$(entry.truck$);
  26.     LOCATE , 37: PRINT USING "###,###"; entry.Gross;
  27.     LOCATE , 45: PRINT USING "###,###"; entry.Net;
  28.     LOCATE , 54: PRINT USING "#,###.##"; entry.Wet;
  29.     LOCATE , 64: PRINT USING "##.##"; entry.Test;
  30.     LOCATE , 70: PRINT USING "#,###.##"; entry.Dry
  31.  
  32. END SUB 'Entry_Print
  33.  
Title: Re: "Out of Memory" mystery
Post by: OldMoses on September 24, 2018, 08:13:02 am
Thank you, Steve. That works perfectly in the application. It's behaving itself now. Would that be a QB64 issue then?

That also explains why my character handling program doesn't have this issue, it uses no TABs. Certainly not loop called ones.

Entry_Print, with it's TAB statements, has been around since the very early days of development. The application is at least 20 years old, and most likely older. It was only later that I got more into using LOCATE for output formatting. In fact, I wasn't aware that row references in LOCATE were optional, so I will be excising TAB from my coding efforts for a while.

I think what happened to the EXEICON was that I compiled the program when the ico file was in the root directory. I later moved all image files to the 'images' directory but forgot to change the path to the ico. I'm kinda like a dwarf playing in his treasure hoard, so intent upon playing with the shiny baubles that he forgets to take a potty break...poor coding hygiene....

I also want to shout out a big thank you to Terry again, I never thought of Task Manager as a debugging tool before he suggested it. I wouldn't have had a clue without it.
Title: Re: "Out of Memory" mystery
Post by: Pete on September 24, 2018, 01:19:51 pm
Quote from: SMcNeill on September 23, 2018, 11:27:23 pm
Removing the use of TAB corrects the issue.  Apparently TAB has a memory leak in it somewhere:

Code: QB64: [Select]
  2.     Entry_Print
  4.  
  5.  
  6. SUB Entry_Print '(entry AS LoadRecord)
  7.  
  8.     '-------------------------DISPLAY----------------------------------------------------
  9.     ' SUB: Entry_Print
  10.     '
  11.     ' Purpose:
  12.     ' Displays tabbed data from record number passed by 'entry' when called.
  13.     '
  14.     ' Passed parameters:
  15.     ' entry sends the load record for display
  16.     '
  17.     ' Called from:
  18.     ' Add_Record, Edit_File, Refresh_Screen, and View_File
  19.     '
  20.     '-----------------------------------------------------------------------------------
  21.  
  22.     LOCATE , 1: PRINT USING "###   "; entry.load;
  23.     LOCATE , 6: PRINT entry.Date;
  24.     LOCATE , 17: PRINT entry.Time;
  25.     LOCATE , 28: PRINT RTRIM$(entry.truck$);
  26.     LOCATE , 37: PRINT USING "###,###"; entry.Gross;
  27.     LOCATE , 45: PRINT USING "###,###"; entry.Net;
  28.     LOCATE , 54: PRINT USING "#,###.##"; entry.Wet;
  29.     LOCATE , 64: PRINT USING "##.##"; entry.Test;
  30.     LOCATE , 70: PRINT USING "#,###.##"; entry.Dry
  31.  
  32. END SUB 'Entry_Print
  33.  

I mentioned the possibility of a memory leak, but wow Steve, you hunted that down fast! I hope it can be fixed in an upcoming version of QB64. Tab isn't used much, I suspect, so it doesn't surprise me if this is a memory leak, it hasn't been found until now.

I remember when I used to debug programs while Rob was developing. If my program threw an error, and I knew it wasn't coming from my code, I had to piece by piece eliminate things in the code until I removed a line that removed the problem. Then I'd report it so it could be fixed. Ah, those were the days... and I don't miss them one friggin' bit!

Pete :D
Title: Re: "Out of Memory" mystery
Post by: OldMoses on September 24, 2018, 01:40:05 pm
I've had a chance to run the program in its working environment this morning, running Task Manager in the background. It was still ticking up slightly, although it would have taken a week of constant running to fill up. I went looking for more TABs and found a few in the loop associated with printing out totals. After removing them, it holds steady at under 40 MB when not actually processing input.

It would seem to confirm that TAB is a troublemaker.
Title: Re: "Out of Memory" mystery
Post by: SMcNeill on September 24, 2018, 02:42:09 pm
Quote
Thank you, Steve. That works perfectly in the application. It's behaving itself now. Would that be a QB64 issue then?

It is.   I took a look inside libqb last night and found the function for tab (func_tab, maybe?  I was half asleep and don't remember the exact name for it...).   From initial impressions, I'm going to assume it's similar to the memory leak PRINT used to have:  it looks as if the routine uses qbs_new to allocate memory for a few temporary QBStrings, but then it doesn't qbs_free them.

The fix may be as simple as freeing the strings before leaving the routine, and if so, somebody should be able to sort that glitch out rather quickly and push the needed change into the repo.   ;)
Text Only | Text with Attachments