Print Page - MEMORY LEAK problem using

Active Forums => QB64 Discussion => Topic started by: Richard on February 25, 2020, 02:48:19 am

Title: MEMORY LEAK problem using _LOADIMAGE
Post by: Richard on February 25, 2020, 02:48:19 am

Hopefully it is something simple I have overlooked …

For this topic I define "Memory leak" as relating to the RAM installed (NOT in solid state drive if present) and the "leak" monitored using

Task Manager > Processes > Memory % (Windows 10 built in program)

The attached program, written as is, WILL FAIL and crash the system (WARNING) and the problem relates to _LOADIMAGE and the QB64 inbuilt help mentions "It is important to free unused or discarded images … to prevent CPU memory overflow errors".

[ This attachment cannot be displayed inline in 'Print Page' view ]

I have tried various things mentioned with the help screen but NOTHING works for me - Task Manager always shows an ever increasing Memory useage and eventually the program crashes.

My computer Win 10 x64 has 32 Gbytes RAM and the attached program when run uses :-

<.0477 GB before the 1st _LOADIMAGE
22.0661 GB at the 1000th _LOADIMAGE
CRASHES at the 2318th _LOADIMAGE

The peak memory useage is about 28 Gbytes for a .BMP image of size 16,876 KB

Of course, for simplicity for this topic, I have just "repeated" the image used and applied very simple text and graphic tasks. It fails in my intended application for an automated "slide show" program with extra information.

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: TerryRitchie on February 25, 2020, 02:55:28 am

You have your _LOADIMAGE inside your loop. _LOADIMAGE only needs to be done once.

Code: QB64: [Select]

version$ = "Memory_leak_FORUM'.bas"
 
image$ = "image.bmp" '       insert YOUR very large picture file here
 
count& = 1000000 '           number of _Loadimage(image$) to be tried
 
xxmax% = 1024
yymax% = 600
SCREEN _NEWIMAGE(xxmax%, yymax%, 32) 'use a part of the total display
 
image& = _LOADIMAGE(image$)
 
image% = 512 '                        displayed (square) picture size
 
lastline% = 25
VIEW PRINT 1 TO lastline%
LOCATE lastline%, 1: PRINT "abc" '                    simple text task
LINE (0, 0)-(xxmax%, yymax%), &HFF80FF80& '       simple graphics task
 
FOR i& = 1 TO count&
    _PUTIMAGE (xxmax% - image%, 0)-(xxmax%, image%), image&
    LOCATE lastline%, 50
    PRINT USING "#,###,###"; i&; '           simple update of text task
NEXT i&
END
 

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: Richard on February 25, 2020, 03:01:41 am

Thank you for your reply.

My intended application would load many thousands of large images - I was trying to simulate the effect using only one image rather than actually supplying thousands of images (and this simulation DOES FAIL with just the same one image).

For illustration purposes - I only used the one image file

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: TerryRitchie on February 25, 2020, 03:11:17 am

If you are going to load many images you need to put them into something like an array. Loading the same image over and over again within the same handle will cause issues as you see.

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: Richard on February 25, 2020, 03:24:03 am

Thankyou for your reply.

Lets suppose that for illustrative purposes I had my one image (in reality all my many thousands images) in array A()$ where each element of the array is the FILE/path name

Suppose in the loop I said

image$=A$(i&)
image&=_LOADIMAGE(image$)

Are you saying that only ONE handle is being used for the complete loop process (HOPEFULLY you are correct and solves my problem)?

Thanks

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: SMcNeill on February 25, 2020, 03:48:03 am

Before you reuse an image handle, be certain to free the image from memory first.

IF handle < -1 THEN _FREEIMAGE handle: handle = 0 ‘check to free old image, if it exists
handle = _LOADIMAGE(“myimage.bmp”) ‘then load the new image

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: EricE on February 25, 2020, 03:57:34 am

Here is your program with _FREEIMAGE called to release memory.

The image used is that of the IBM 704 computer.
https://upload.wikimedia.org/wikipedia/commons/2/20/IBM_Electronic_Data_Processing_Machine_-_GPN-2000-001881.jpg

JPG converted to BMP using PicPick.
https://picpick.app/en/ (https://picpick.app/en/)

Code: QB64: [Select]

version$ = "Memory_leak_FORUM'.bas"
 
image$ = "IBM_Electronic_Data_Processing_Machine_-_GPN-2000-001881.bmp" '       insert YOUR very large picture file here
 
count& = 1000000 '           number of _Loadimage(image$) to be tried
 
xxmax% = 1024
yymax% = 600
SCREEN _NEWIMAGE(xxmax%, yymax%, 32) 'use a part of the total display
image% = 512 '                        displayed (square) picture size
 
lastline% = 25
VIEW PRINT 1 TO lastline%
LOCATE lastline%, 1: PRINT "abc" '                    simple text task
LINE (0, 0)-(xxmax%, yymax%), &HFF80FF80& '       simple graphics task
 
FOR i& = 1 TO count&
    image& = _LOADIMAGE(image$)
    _PUTIMAGE (xxmax% - image%, 0)-(xxmax%, image%), image&
    LOCATE lastline%, 50
    PRINT USING "#,###,###"; i&; '           simple update of text task
    _FREEIMAGE image&
NEXT i&
END
 

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: luke on February 25, 2020, 04:22:10 am

Quote from: Richard on February 25, 2020, 02:48:19 am

Hopefully it is something simple I have overlooked …

Yes.

You're trying to load a million images simultaneously. By your own admission these images are "big", though you only give the file size. When you're loading it into memory it's 4 bytes per pixel regardless of the storage compression, so that's not an entirely helpful measurement.

Anyway, if your images are 1500 x 1500 that's 1500^2 * 4 = 8.5MB per image. 8.5MB * 1000000 = 8381GB, which has no hope of fitting in memory. You managed 2318 images, which totals just over 19GB. That's a reasonable effort, I'd say.

You're going to need to be a little smarter about how you manage resources.

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: Richard on February 25, 2020, 04:32:18 am

Thanks Steve

Refer attached (modified) program - IT APPEARS TO WORK !!!

[ This attachment cannot be displayed inline in 'Print Page' view ]

I just did a trial run up to > 3000 repeats of my large image (before the limit was ~2000 with crash) and I stopped it there since it appears to be working OK.

Task Manager oscillated around 99 to 123 Mbytes but no observable trend for 3000 x (16 Mbyte images).

Hopefully can you explain the mystery behind the handle <-1 (just curious) (what if it was = -1 or 0?)

In my code have I done anything incorrectly that could "catch me out" many images down the road?

Many thanks.

Now to see what Eric and Luke? have to say...

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: SMcNeill on February 25, 2020, 04:54:08 am

Quote from: Richard on February 25, 2020, 04:32:18 am

Hopefully can you explain the mystery behind the handle <-1 (just curious) (what if it was = -1 or 0?)

http://www.qb64.org/wiki/LOADIMAGE

If you look at the wiki, you’ll see that a handle of -1 is what you get when the image fails to load. (Bad file path, corrupted image, bad format... Lots of reasons for this, but -1 is the “image failed to load” result.)

0 is, of course, an unused handle. It’s never been utilized at all and is the default value you start with.

Valid images are *always* assigned a value < -1 when they load properly, so a good practice is the simple one I posted above: check to see if the handle is less than -1, and if so, free it before reusing it. (After all, you can’t free handles that don’t exist.)

If you want a visible breakdown of whats happening, grab a handful of change. Toss a quarter in a cup and call it “Image&”.

Now, when we talk about Image&, you know I’m talking about that particular quarter. Right? When I ask, “Is Image& heads or tails,” you’d look in the cup and tell me which it is.

Now, throw another quarter in the cup, and we’ll now call it, “Image&”. We’re finished with the first quarter, so we’ll just reuse the name so I don’t need to make up a new one. Now, is Image& heads or tails?

Now, continue this process for 10,000,000 times....

WHAT?? You can’t? Your cup overfilled at 2817 quarters and won’t hold any more???

.....

This is what your images are doing in memory — just filling up, until theres no place left to hold them.

The only way you can put 10,000,000 quarters in a cup is if you empty that cup before tossing quarters in it each time, and the same holds true with your images. ;D

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: Richard on February 25, 2020, 05:09:49 am

Thanks Eric for your reply.

I just ran your version of my program (with the only change being a path D:\ added to image).

The program ran for 3600 images (I stopped it at that point) and IT APPEARS TO WORK !!!

Task Manager oscillated around 67 Mbytes (very rare this low) to 138 Mbytes
with a typical range of 115 - 131 Mbytes.

No observable trend was seen throughout.

Your .bmp ended up being 20.4 Mbytes (and that's only in B&W !).

Interesting that no handle value check (c/- Steve) was needed.

Many thanks

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: Richard on February 25, 2020, 05:30:58 am

Thanks Luke for your reply.

Just for your information my image is 3200 x 1800 pixels (.bmp 24.bit) not that it matters for this discussion - so I did OK in getting that many loaded.

What I really was wanting to do with my million pictures was to VIEW ONE AT A TIME, like a slide projector with a one million capacity carosel. As soon as I viewed one image, it was to be "discarded" (overwritten), and a new image to take its place (with no 'knowledge' of any other image).

I DID NOT want the effect of one million slide projectors simultaneously (or like the old fashioned projector for planetarium star display).

To summarise, I only wanted ONE image loaded and VIEWED, at any one time, then discarded before the next image is loaded and viewed.

Sorry for any confusion that may have caused.

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: Richard on February 25, 2020, 06:00:15 am

Thanks Steve for your explanation - I just did not have a "spring return" in the base of the "cup" to eject the "quarter" when I finished with it (before the next one inserted).

Thanks for explaining the "handle stuff" - its not like OPEN … AS #n where I choose the "handle". The "error" aspect may be very important and now explain things regarding problems I have with my "Slide Projector (Viewer)" program.

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: Richard on February 25, 2020, 06:13:28 am

Thank you everyone.

I now have two slightly different approaches for my program which work with very big images (3200x1800 pixels or bigger 16-24 Mbytes).

For my "Slide Projector (Viewer) Program" I will adopt these changes - I did try something similar then before, and it still did not work - but there may be "error" conditions (re handles) or simply code in the wrong place. At least I have reference programs now to compare against.

Also, with my Slide Projector Program I was working with much smaller images, (10 to 200 KBytes) and if above still does not work, will investigate into adding delays in the loops.

Many thanks

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: SMcNeill on February 25, 2020, 08:12:15 am

Quote from: Richard on February 25, 2020, 05:09:49 am

Interesting that no handle value check (c/- Steve) was needed.

2 things to consider here:

1) The method I showed checks to see IF a handle needs clearing, and then frees it if needed, before using that handle. Eric’s method makes certain to clear the handle after you use it. Both will end up clearing the images for you, so use whichever you prefer.

2) Eric’s use of just _FREEIMAGE may end up generating errors for you, in the case where the image failed to load properly. You can’t free an image from memory that never went into memory properly. (Think of it as trying to toss that quarter into the cup, but missing the cup. Your “spring ejector”, as you called it, can’t eject a quarter that’s not in the cup to begin with, now can it? Generally, you want to check, “IF there’s a quarter here, THEN eject it”. Likewise, IF Image& < -1 THEN _FREEIMAGE Image&.)

Note on 2: Chances are, if the _FREEIMAGE line is going to toss an error for “Image not loaded properly”, then so is every other reference to that handle, such as the _PUTIMAGE line. I’d do something like the following to prevent those errors:

DO
image& = _LOADIMAGE(image$(i&))
i& = i& + 1
LOOP UNTIL image& < -1
_PUTIMAGE....

Now, the above assumes your list of images is stored in an array image$(), and it loads images one at a time from that array, skipping over any that fail to load properly in memory, so it prevents those errors from ever popping up in your program.

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: EricE on February 25, 2020, 01:11:11 pm

As SMcNeill points out it is always good practice to check the function return value.
Here is the second program with error checking and error message output added.

Image used is again that of the IBM 704. ("Now THAT was a computer!")

Code: QB64: [Select]

version$ = "Memory_leak_FORUM-2'.bas"
 
'       insert YOUR very large picture file here
image$ = "IBM_Electronic_Data_Processing_Machine_-_GPN-2000-001881.bmp"
'                       "D:\Dit.bmp" is my test .bmp file (~ 16 Mbyte)
 
count& = 1000000 '           number of _Loadimage(image$) to be tried
 
xxmax% = 1024
yymax% = 600
SCREEN _NEWIMAGE(xxmax%, yymax%, 32) 'use a part of the total display
image% = 512 '                        displayed (square) picture size
 
lastline% = 25
VIEW PRINT 1 TO lastline%
LOCATE lastline%, 1: PRINT "abc" '                    simple text task
LINE (0, 0)-(xxmax%, yymax%), &HFF80FF80& '       simple graphics task
 
FOR i& = 1 TO count&
    image& = _LOADIMAGE(image$) ' image to be loaded
    IF image& < -1 THEN
        _PUTIMAGE (xxmax% - image%, 0)-(xxmax%, image%), image&
        LOCATE lastline%, 50
        PRINT USING "#,###,###"; i&; '           simple update of text task
 
        _FREEIMAGE image&
        image& = 0 '         checked to free old image, if it existed
    ELSE
        ' Image load error handler or notification code here
        LOCATE lastline% - 2, 1
        PRINT "Error loading image "; i&;
    END IF
NEXT i&
 
END

Title: Re: MEMORY LEAK problem using _LOADIMAGE
Post by: Richard on February 25, 2020, 02:18:18 pm

Thanks Steve and Eric

I have been running my "Slide Projector Program" now for a number of hours - started at the end of my last reply. It has cycled through 16,010 images (without a hitch) and is still going! Task Manager is oscillating around 77 to 96 Mbyte and over all this time had extremely slowly "creeped" to the 86 to 96 Mbyte range. The images (.jpg) typically ranged in size from 10 KB to ~ 200KB.

I inserted Steve's "handle <-1 etc check".

Before I started this topic, my Slide Projector Program resulted in the need for the following code fragments :-

ON ERROR GOTO Errhandler ' ********************************************

~~~~~~~~~
eror_258% = 0 ' reset error # 258 flag
GOSUB PUT_image
~~~~~~~~~

Errhandler:
eror% = -1

SELECT CASE ERR
CASE 258: eror_258% = -1 ' _Putimage invalid handle
CASE 5::: eror_005% = -1: ' ' _FreeImage illegal function call
CASE ELSE: BEEP: BEEP: BEEP
END SELECT
RESUME NEXT ' moves program to code following the error
RETURN

PUT_image:
img& = _LOADIMAGE(img$)
_PUTIMAGE (x0%, y0%)-(x1%, y1%), img& ' 512x512 image TRHC
IF img& < -1 THEN _FREEIMAGE img&: img& = 0' Steve's suggestion inserted here

StopWatch& = TIMER
IF NOT (eror_258%) THEN
StopWatchSET& = TIMER
END IF
IF (StopWatch& - StopWatchSET&) > StopWatchMax& THEN GOTO BEEPsystemEND ' just BEEP : SYSTEM: END

RETURN

The SYSTEM command may not be needed.
The Stopwatch code was handy for automatic shutdown. In my case the timeout was preset to 3 minutes since last VALID image.
The handle (img& <-1) etc as suggested by Steve was inserted.

ERROR #258 frequently occurred with my very old images - say roughly 1% were well and truly corrupted which neither _PUTIMAGE or windows PAINT, etc could do anything with - and say roughly 1% _PUTIMAGE cannot (as the program was written) do anything BUT are viewable in Windows PAINT, PHOTOS, and just about any other commercial viewer app CAN display - the remaining 98% were well behaved.

Many thanks.

By the way Eric - where does one insert the USB memory stick on the IBM and more importantly where is the display?

For those curious to know - Slide projector program (1st run) STILL RUNNING - 37,038 images (1.40 GB) processed, time elapsed ~ 14.5 hours (if my time calculations are correct), Task Manager oscillating 85 - 88 MB (ocassionally up to 95 MB) and no trending now easily visible --- program did not timeout or anything --- PROGRAM STILL CONTINUOUSLY RUNNING without intervention.

For those curious to know - Slide Projector Program (1st run) - 58,241 images (2.15 GB) processed, time elapsed ~23 hours, Task Manager 84-89 MB oscillating (final value 84.2) no observable trend --- PROGRAM terminated by me.

No more updates!

Text Only | Text with Attachments

QB64.org Forum

Active Forums => QB64 Discussion => Topic started by: Richard on February 25, 2020, 02:48:19 am