MEMORY LEAK problem using _LOADIMAGE

[Richard]

Hopefully it is something simple I have overlooked …



For this topic I define "Memory leak" as relating to the RAM installed (NOT in solid state drive if present) and the "leak" monitored using


Task Manager > Processes > Memory %             (Windows 10 built in program)



The attached program, written as is, WILL FAIL and crash the system (WARNING) and the problem relates to _LOADIMAGE and the QB64 inbuilt help mentions "It is important to free unused or discarded images … to prevent CPU memory overflow errors".

  [ You are not allowed to view this attachment ]  

I have tried various things mentioned with the help screen but NOTHING works for me -  Task Manager always shows an ever increasing Memory useage  and eventually the program crashes.

My computer Win 10 x64  has 32 Gbytes RAM and the attached program when run uses :-

 <.0477 GB  before the 1st _LOADIMAGE
22.0661 GB  at the 1000th _LOADIMAGE
CRASHES      at the 2318th _LOADIMAGE

The peak memory useage is about 28 Gbytes   for a  .BMP  image  of size 16,876 KB

Of course, for simplicity for this topic, I have just "repeated" the image used and applied very simple text and graphic tasks.  It fails in my intended application for an automated "slide show" program with extra information.

[TerryRitchie]

You have your _LOADIMAGE inside your loop. _LOADIMAGE only needs to be done once.

Code: QB64: [Select]

1. version$ = "Memory_leak_FORUM'.bas"
2.  
3. image$ = "image.bmp" '       insert YOUR very large picture file here
4.  
5. count& = 1000000 '           number of _Loadimage(image$) to be tried
6.  
7. xxmax% = 1024
8. yymax% = 600
9. SCREEN _NEWIMAGE(xxmax%, yymax%, 32) 'use a part of the total display
10.  
11. image& = _LOADIMAGE(image$)
12.  
13. image% = 512 '                        displayed (square) picture size
14.  
15. lastline% = 25
16. VIEW PRINT 1 TO lastline%
17. LOCATE lastline%, 1: PRINT "abc" '                    simple text task
18. LINE (0, 0)-(xxmax%, yymax%), &HFF80FF80& '       simple graphics task
19.  
20. FOR i& = 1 TO count&
21.     _PUTIMAGE (xxmax% - image%, 0)-(xxmax%, image%), image&
22.     LOCATE lastline%, 50
23.     PRINT USING "#,###,###"; i&; '           simple update of text task
24. NEXT i&
25. END
26.  

[Richard]

Thank you for your reply.

My intended application would load many thousands of large images - I was trying to simulate the effect using only one image rather than actually supplying thousands of images (and this simulation DOES FAIL with just the same one image).

For illustration purposes - I only used the one image file

[TerryRitchie]

If you are going to load many images you need to put them into something like an array. Loading the same image over and over again within the same handle will cause issues as you see.

[Richard]

Thankyou for your reply.

Lets suppose that for illustrative purposes I had my one image (in reality all my many thousands images) in array A()$ where each element of the array  is the FILE/path name

Suppose in the loop I said

image$=A$(i&)
image&=_LOADIMAGE(image$)

Are you saying that only ONE handle is being used for the complete loop process (HOPEFULLY you are correct and solves my problem)?

Thanks

[SMcNeill]

Before you reuse an image handle, be certain to free the image from memory first.

IF handle < -1 THEN _FREEIMAGE handle: handle = 0 ‘check to free old image, if it exists
handle = _LOADIMAGE(“myimage.bmp”) ‘then load the new image

[EricE]

Here is your program with _FREEIMAGE called to release memory.

The image used is that of the IBM 704 computer.
https://upload.wikimedia.org/wikipedia/commons/2/20/IBM_Electronic_Data_Processing_Machine_-_GPN-2000-001881.jpg

JPG converted to BMP using PicPick.
https://picpick.app/en/

Code: QB64: [Select]

1. version$ = "Memory_leak_FORUM'.bas"
2.  
3. image$ = "IBM_Electronic_Data_Processing_Machine_-_GPN-2000-001881.bmp" '       insert YOUR very large picture file here
4.  
5. count& = 1000000 '           number of _Loadimage(image$) to be tried
6.  
7. xxmax% = 1024
8. yymax% = 600
9. SCREEN _NEWIMAGE(xxmax%, yymax%, 32) 'use a part of the total display
10. image% = 512 '                        displayed (square) picture size
11.  
12. lastline% = 25
13. VIEW PRINT 1 TO lastline%
14. LOCATE lastline%, 1: PRINT "abc" '                    simple text task
15. LINE (0, 0)-(xxmax%, yymax%), &HFF80FF80& '       simple graphics task
16.  
17. FOR i& = 1 TO count&
18.     image& = _LOADIMAGE(image$)
19.     _PUTIMAGE (xxmax% - image%, 0)-(xxmax%, image%), image&
20.     LOCATE lastline%, 50
21.     PRINT USING "#,###,###"; i&; '           simple update of text task
22.     _FREEIMAGE image&
23. NEXT i&
24. END
25.  

[luke]

Hopefully it is something simple I have overlooked …

Yes.

You're trying to load a million images simultaneously. By your own admission these images are "big", though you only give the file size. When you're loading it into memory it's 4 bytes per pixel regardless of the storage compression, so that's not an entirely helpful measurement.

Anyway, if your images are 1500 x 1500 that's 1500^2 * 4 = 8.5MB per image. 8.5MB * 1000000 = 8381GB, which has no hope of fitting in memory. You managed 2318 images, which totals just over 19GB. That's a reasonable effort, I'd say.

You're going to need to be a little smarter about how you manage resources.

[Richard]

Thanks Steve

Refer attached (modified) program - IT APPEARS TO WORK !!!

  [ You are not allowed to view this attachment ]  

I just did a trial run up to > 3000 repeats of my large image (before the limit was ~2000 with crash) and I stopped it there since it appears to be working OK.

Task Manager oscillated around     99 to 123 Mbytes      but no observable trend for 3000 x (16 Mbyte images).

Hopefully can you explain the mystery behind the handle <-1 (just curious) (what if it was = -1 or 0?)

In my code have I done anything incorrectly that could "catch me out" many images down the road?

Many thanks.

Now to see what Eric and Luke? have to say...

[SMcNeill]

Hopefully can you explain the mystery behind the handle <-1 (just curious) (what if it was = -1 or 0?)

http://www.qb64.org/wiki/LOADIMAGE

If you look at the wiki, you’ll see that a handle of -1 is what you get when the image fails to load.  (Bad file path, corrupted image, bad format... Lots of reasons for this, but -1 is the “image failed to load” result.)

0 is, of course, an unused handle.  It’s never been utilized at all and is the default value you start with.

Valid images are *always* assigned a value < -1 when they load properly, so a good practice is the simple one I posted above: check to see if the handle is less than -1, and if so, free it before reusing it.  (After all, you can’t free handles that don’t exist.)


If you want a visible breakdown of whats happening, grab a handful of change.  Toss a quarter in a cup and call it “Image&”.

Now, when we talk about Image&, you know I’m talking about that particular quarter.  Right?  When I ask, “Is Image& heads or tails,” you’d look in the cup and tell me which it is.

Now, throw another quarter in the cup, and we’ll now call it, “Image&”.  We’re finished with the first quarter, so we’ll just reuse the name so I don’t need to make up a new one.  Now, is Image& heads or tails?

Now, continue this process for 10,000,000 times....

WHAT??  You can’t?  Your cup overfilled at 2817 quarters and won’t hold any more???

.....

This is what your images are doing in memory — just filling up, until theres no place left to hold them.

The only way you can put 10,000,000 quarters in a cup is if you empty that cup before tossing quarters in it each time, and the same holds true with your images.  ;D

[Richard]

Thanks Eric for your reply.

I just ran your version of my program (with the only change being a path D:\ added to image).

The program ran for 3600 images (I stopped it at that point) and IT APPEARS TO WORK !!!

Task Manager   oscillated around     67 Mbytes (very rare this low)  to    138 Mbytes
with a typical range of 115 - 131 Mbytes. 

No observable trend was seen throughout.

Your .bmp ended up being  20.4 Mbytes (and that's only in B&W !).

Interesting that no handle value check (c/- Steve) was needed.

Many thanks

[Richard]

Thanks Luke for your reply.

Just for your information my image is  3200 x 1800 pixels (.bmp  24.bit) not that it matters for this discussion - so I did OK in getting that many loaded.


What I really was wanting to do with my million pictures was to VIEW ONE AT A TIME, like a slide projector with a one million capacity carosel. As soon as I viewed one image, it was to be "discarded" (overwritten), and a new image to take its place (with no 'knowledge' of any other image).

I DID NOT want the effect of one million slide projectors simultaneously (or like the old fashioned projector for planetarium star display).

To summarise, I only wanted ONE image loaded and VIEWED, at any one time, then discarded before the next image is loaded and viewed.

Sorry for any confusion that may have caused.

[Richard]

Thanks Steve for your explanation - I just did not have a "spring return" in the base of the "cup" to eject the "quarter" when I finished with it (before the next one inserted).

Thanks for explaining the "handle stuff" - its not like   OPEN … AS #n     where I choose the "handle".  The "error" aspect may be very important and now explain things regarding problems I have with my "Slide Projector (Viewer)" program.

[Richard]

Thank you everyone.

I now have two slightly different approaches for my program which work with very big images (3200x1800 pixels   or   bigger  16-24 Mbytes).

For my "Slide Projector (Viewer) Program" I will adopt these changes - I did try something similar then before, and it still did not work - but there may be "error" conditions (re handles) or simply code in the wrong place. At least I have reference programs now to compare against. 

Also, with my Slide Projector Program I was working with much smaller images, (10 to 200 KBytes) and if above still does not work, will investigate into adding delays in the loops.

Many thanks

[SMcNeill]

Interesting that no handle value check (c/- Steve) was needed.

2 things to consider here:

1) The method I showed checks to see IF a handle needs clearing, and then frees it if needed, before using that handle.  Eric’s method makes certain to clear the handle after you use it.  Both will end up clearing the images for you, so use whichever you prefer.

2) Eric’s use of just _FREEIMAGE may end up generating errors for you, in the case where the image failed to load properly.  You can’t free an image from memory that never went into memory properly.  (Think of it as trying to toss that quarter into the cup, but missing the cup.  Your “spring ejector”, as you called it, can’t eject a quarter that’s not in the cup to begin with, now can it?  Generally, you want to check, “IF there’s a quarter here, THEN eject it”.  Likewise, IF Image& < -1 THEN _FREEIMAGE Image&.)

Note on 2: Chances are, if the _FREEIMAGE line is going to toss an error for “Image not loaded properly”, then so is every other reference to that handle, such as the _PUTIMAGE line.  I’d do something like the following to prevent those errors:


    DO
        image& = _LOADIMAGE(image$(i&))
        i& = i& + 1
   LOOP UNTIL image& < -1
    _PUTIMAGE....

Now, the above assumes your list of images is stored in an array image$(), and it loads images one at a time from that array, skipping over any that fail to load properly in memory, so it prevents those errors from ever popping up in your program.