Question for Steve on using _MEM

[Cobalt]

Hey Steve, Being the top MEM routine guy I know I wanted to ask you if it would be possible to load an image straight from a file into a memory handle bypassing _LOADIMAGE? I have a series of images as part of an animation I want to store in a single file and be able to load right into an image handle without having to make a temp file and coping the data from one file to another then load it and delete the temp file over and over. The animation is only 48 frames(640x480x32) but repeats in an endless loop until a menu action is selected.

So do you think I could have the animation file open, read the image data for the current frame into a STRING variable and use _MEM commands to push it into an image handle? And just what keywords should I be looking into to pull this stunt off?

I know how to pack the images and keep track of image size and offset, but have neglected my study of _MEM commands to know if it would even work.

[SMcNeill]

There’s no real way to bypass _LOADIMAGE, as it’s our library which decodes the various image formats and uncompresses them properly into a screen format forums.  (At least, no way to decode the info, if it’s in a standard image format, such as PNG, BMP, GIF, etc...)

640x480x4 (4 bytes for 32 bit images) is about 1 MB per frame, without any compression.  Would it be possible to simply save that information in a 48 MB file, or would that be too large for your requirements?  If you don’t mind saving it in an uncompressed format, then it’d be quite easy to store all the info in one file and restore it as needed with _MEM.  Let me know if that’d be an option for you, and I’ll write you a quick demo of the process fairly quickly.  ;)

[Cobalt]

Yeah size doesn't bother me, although I don't want to wind up creating animation files that would be too big for some members to be able to download. in the end it would be stored the same, a number of elements(frames), offsets and sizes, then image(frame) data.

not sure if it would be possible to use the zlib.dll and compress\uncompress on the fly too?

[SMcNeill]

There shouldn’t be any issues with using zlib, if you don’t mind packaging the dll with your code.  I’ll write you up a demo later tonight and post it for you.  ;)

[Cobalt]

Thanks Steve!

[SMcNeill]

See if this doesn't highlight all the relevant parts of what you're trying to do:

Code: QB64: [Select]

1. $IF REMARK THEN
2.     Library declarations for Windows, both 32 and 64-bit versions
3.     Note, there's more options defined than what we're actually making use of for this program.
4.     Ignore the unneeded functionality; I just copy/pasted the declare statements from the SAVEIMAGE library,
5.     as they contain the same needed compress and decompress functions, which we end up calling in the
6.     deflate and inflate routines.
7.  
8.     Also notice that I'm being lazy and using the precompiler to store a whole series of remarks in a block,
9.     with this IF REMARK THEN block.
10.  
11.     Since REMARK is never defined with a LET statement, this code will *never* be seen, or executed, by QB64,
12.     thus making it the perfect means to lay out a block statement of remarks.
13.  
14.     This demo was created on Friday, June 21st, 2019, by Steve McNeill (SMcNeill on most internet BASIC forums),
15.     and is free to use, modify, and exploit, in all code and projects, with no credit needed, nor expected.
16. $END IF
17.  
18. $IF 32BIT THEN
19.     DECLARE DYNAMIC LIBRARY ".\zlib1"
20.     FUNCTION zlibVersion$ ()
21.     FUNCTION zlibCompileFlags~& ()
22.     FUNCTION compressBound& (BYVAL len AS LONG)
23.     FUNCTION compress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
24.     FUNCTION compress2& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG, BYVAL level AS LONG)
25.     FUNCTION uncompress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
26.     FUNCTION crc32~& (BYVAL PNGCRC AS LONG, BYVAL buf AS _OFFSET, BYVAL len AS LONG)
27.     FUNCTION adler32~& (BYVAL adler AS LONG, BYVAL buf AS _OFFSET, BYVAL len AS LONG)
28.     FUNCTION zError$ (BYVAL codenum AS INTEGER)
29.     END DECLARE
30. $ELSE
31.     DECLARE DYNAMIC LIBRARY ".\zlib"
32.         FUNCTION zlibVersion$ ()
33.         FUNCTION zlibCompileFlags~& ()
34.         FUNCTION compressBound& (BYVAL len AS LONG)
35.         FUNCTION compress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
36.         FUNCTION compress2& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG, BYVAL level AS LONG)
37.         FUNCTION uncompress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
38.         FUNCTION crc32~& (BYVAL PNGCRC AS LONG, BYVAL buf AS _OFFSET, BYVAL len AS LONG)
39.         FUNCTION adler32~& (BYVAL adler AS LONG, BYVAL buf AS _OFFSET, BYVAL len AS LONG)
40.         FUNCTION zError$ (BYVAL codenum AS INTEGER)
41.     END DECLARE
42. $END IF
43.  
44. 'DEFLNG A-Z
45. SCREEN _NEWIMAGE(640, 480, 32) 'A default screen, the size specified.
46.  
47. P = _PIXELSIZE: IF P = 0 THEN P = 2 '0 is a designation for text screens, which use 2 bytes per character for memory storage
48. ScreenSize = _WIDTH * _HEIGHT * P
49. Frames = 48 'In our animation
50. UncompressedStringSize = Frames * ScreenSize 'the total size needed to store all the animation images, uncompressed (Should be about 48 MB)
51. UncompressedString$ = SPACE$(UncompressedStringSize) 'a placeholder for the whole animation, uncompressed.
52. SingleScreen$ = SPACE$(ScreenSize) 'a placeholder for a single screen worth of info
53.  
54. f = _LOADFONT("OLDENGL.TTF", 32)
55. IF f THEN _FONT f
56.  
57. DIM m AS _MEM
58. m = _MEMIMAGE(0)
59.  
60.  
61.  
62. FOR i = 1 TO 48 'let's draw a cheesy animation demo
63.     LOCATE 1, 400 - i * 8
64.     PRINT "Steve is Awesome."
65.     _MEMGET m, m.OFFSET, SingleScreen$ 'Get one screen of information
66.     MID$(UncompressedString$, (i - 1) * ScreenSize + 1, ScreenSize) = SingleScreen$ 'And put it in the proper place for storage
67.     _LIMIT 10
68. NEXT
69.  
70. CompressedString$ = Deflate(UncompressedString$)
71. _FONT 16
72. LOCATE 4, 1
73. PRINT "Original Size of Animation in memory:"; LEN(UncompressedString$)
74. PRINT "Size of Compressed Animation in memory:"; LEN(CompressedString$)
75.  
76. 'At this point, you'd normally save the CompressedString$ to disk to save the whole animation at once.
77.  
78. 'Then when needed, you'd load the compressedstring and then inflate it, like so:
79.  
80. UncompressedString$ = "" 'To erase the string, just to show that we're restoring it from the compressed version
81. UncompressedString$ = Inflate(CompressedString$)
82.  
83. PRINT
84. PRINT "Press <ANY KEY> to view the cheesy animation."
85.  
86. SLEEP 'and now a pause so we can restore our animation
87.  
88. CLS
89. FOR i = 0 TO 47
90.     temp$ = ""
91.     temp$ = MID$(UncompressedString$, i * ScreenSize + 1, ScreenSize)
92.     _MEMPUT m, m.OFFSET, temp$
93.     _LIMIT 10
94. NEXT
95.  
96.  
97.  
98.  
99. FUNCTION Deflate$ (text$)
100.     $CHECKING:OFF
101.     DIM FileSize AS LONG, CompSize AS LONG
102.     DIM m AS _MEM
103.     FileSize = LEN(text$): CompSize = compressBound(FileSize)
104.     m = _MEMNEW(FileSize)
105.     _MEMPUT m, m.OFFSET, text$ 'set the variable length text into a memblock so it won't move in memory as we access it.
106.     REDIM CompBuff(1 TO CompSize) AS _UNSIGNED _BYTE
107.     Result = compress2(_OFFSET(CompBuff(1)), CompSize, m.OFFSET, FileSize, 9)
108.     REDIM _PRESERVE CompBuff(1 TO CompSize) AS _UNSIGNED _BYTE
109.     _MEMFREE m
110.     m = _MEM(CompBuff())
111.     Deflate$ = SPACE$(CompSize)
112.     _MEMGET m, m.OFFSET, Deflate$
113.     _MEMFREE m
114.     Deflate$ = MKL$(FileSize) + Deflate$
115.     $CHECKING:ON
116. END FUNCTION
117.  
118. FUNCTION Inflate$ (text$)
119.     $CHECKING:OFF
120.     DIM m AS _MEM, m1 AS _MEM
121.     FileSize& = CVL(LEFT$(text$, 4))
122.     text1$ = MID$(text$, 5): temp$ = SPACE$(FileSize&)
123.     m = _MEMNEW(FileSize&): m1 = _MEMNEW(LEN(text1$))
124.     _MEMPUT m1, m1.OFFSET, text1$
125.     Result = uncompress(m.OFFSET, FileSize&, m1.OFFSET, LEN(text1$))
126.     _MEMGET m, m.OFFSET, temp$
127.     _MEMFREE m
128.     _MEMFREE m1
129.     Inflate$ = temp$
130.     $CHECKING:ON
131. END FUNCTION
132.  

Notice, this compresses our animation from 58MB down to only 230KB, but I doubt this would be the normal level of shrinkage you'd see, as 90%+ of this screen is probably just a plain black screen and compresses down to a simple statement of (45,000 pixels of BLACK), which only takes a few minimal bytes to represent.  Compression of your actual screen animation will vary, depending on how efficiently the zlib library can work with your data.

Note 2: This only works on windows.  It'll work on Linux/Mac, if you guys have a version of zlib/zlib1 on your system, and point the DECLARE LIBRARY to the proper place where it'll work for them.  I'm not a Linux/Mac expert, not by any means, and without a system to test things on, I haven't sorted out how exactly to make use of the zlib library for Linux users yet.

[Cobalt]

Thanks Steve, everything worked until I tried to set it up for how I wanted to handle it, so I could have multiple animations in one file. Now it crashes when it tries to play back from the file. a windows crash ie: "program name" has stopped working.

checking the animation file it looks packed correctly can you see anything off hand that I goofed on?

Code: [Select]

'*******$include: 'Zlib.bi'
$IF 32BIT THEN
 DECLARE DYNAMIC LIBRARY ".\zlib1"
  FUNCTION zlibVersion$ ()
  FUNCTION zlibCompileFlags~& ()
  FUNCTION compressBound& (BYVAL LEN AS LONG)
  FUNCTION compress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
  FUNCTION compress2& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG, BYVAL level AS LONG)
  FUNCTION uncompress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
  FUNCTION crc32~& (BYVAL PNGCRC AS LONG, BYVAL buf AS _OFFSET, BYVAL LEN AS LONG)
  FUNCTION adler32~& (BYVAL adler AS LONG, BYVAL buf AS _OFFSET, BYVAL LEN AS LONG)
  FUNCTION zError$ (BYVAL codenum AS INTEGER)
 END DECLARE
$ELSE
 DECLARE DYNAMIC LIBRARY ".\zlib"
 FUNCTION zlibVersion$ ()
 FUNCTION zlibCompileFlags~& ()
 FUNCTION compressBound& (BYVAL LEN AS LONG)
 FUNCTION compress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
 FUNCTION compress2& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG, BYVAL level AS LONG)
 FUNCTION uncompress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
 FUNCTION crc32~& (BYVAL PNGCRC AS LONG, BYVAL buf AS _OFFSET, BYVAL LEN AS LONG)
 FUNCTION adler32~& (BYVAL adler AS LONG, BYVAL buf AS _OFFSET, BYVAL LEN AS LONG)
 FUNCTION zError$ (BYVAL codenum AS INTEGER)
 END DECLARE
$END IF


SCREEN _NEWIMAGE(640, 400, 32)

P = _PIXELSIZE: IF P = 0 THEN P = 2 '0 is a designation for text screens, which use 2 bytes per character for memory storage
ScreenSize = _WIDTH * _HEIGHT * P
Frames = 48 'In our animation
UncompressedStringSize = Frames * ScreenSize 'the total size needed to store all the animation images, uncompressed (Should be about 48 MB)
UncompressedString$ = SPACE$(UncompressedStringSize) 'a placeholder for the whole animation, uncompressed.
SingleScreen$ = SPACE$(ScreenSize) 'a placeholder for a single screen worth of info

DIM Offs(48) AS LONG, Sizs(48) AS LONG

DIM m AS _MEM
m = _MEMIMAGE(0)

OPEN "Vortex.Ani" FOR BINARY AS #1

Elements% = 48 'number of frames
Offset& = 2 + 96 * 8 'begining offset of image data by 1 integer and 96 longs(48 for offsets, 48 for sizes)

FOR i = 0 TO 47 'Load images to store(vortex animation 0-47 images)
 IF Temp& THEN _FREEIMAGE Temp&
 f$ = "vortex  " + LTRIM$(STR$(i)) + ".bmp"
 Temp& = _LOADIMAGE(f$, 32)
 
 _PUTIMAGE , Temp&, _DISPLAY

 _MEMGET m, m.OFFSET, SingleScreen$ 'Get one screen of information
 
 CompressedString$ = Deflate(SingleScreen$)
 Image$ = Image$ + CompressedString$ 'place compressed image data in a temp string until we get them all done
 Offs(i) = Offset&
 Sizs(i) = LEN(CompressedString$)
 Offset& = Offset& + Sizs(i)
 _DELAY .05
NEXT

PUT #1, , Elements%
PUT #1, , Offs()
PUT #1, , Sizs()
PUT #1, , Image$
CLS
FOR i% = 0 TO 10
 PRINT Offs(i%), Sizs(i%)
NEXT i%

SLEEP


CLS
DO
 FOR i = 0 TO 47
  temp$ = SPACE$(Sizs(i))
  '  temp$ = MID$(UncompressedString$, i * ScreenSize + 1, ScreenSize)
  GET #1, Offs(i), temp$

  PRINT "about to decompress!"
  _DELAY 2.5
  UncompressedString$ = Inflate(temp$)
  PRINT "DECOMPRESSED!"
  _DELAY 2.5

  _MEMPUT m, m.OFFSET, UncompressedString$
  _DELAY .1
 NEXT
LOOP UNTIL INKEY$ = CHR$(27)

CLOSE #1

'*******$include:'Compress.bi'
FUNCTION Deflate$ (text$)
 $CHECKING:OFF
 DIM FileSize AS LONG, CompSize AS LONG
 DIM m AS _MEM
 FileSize = LEN(text$): CompSize = compressBound(FileSize)
 m = _MEMNEW(FileSize)
 _MEMPUT m, m.OFFSET, text$ 'set the variable length text into a memblock so it won't move in memory as we access it.
 REDIM CompBuff(1 TO CompSize) AS _UNSIGNED _BYTE
 Result = compress2(_OFFSET(CompBuff(1)), CompSize, m.OFFSET, FileSize, 9)
 REDIM _PRESERVE CompBuff(1 TO CompSize) AS _UNSIGNED _BYTE
 _MEMFREE m
 m = _MEM(CompBuff())
 Deflate$ = SPACE$(CompSize)
 _MEMGET m, m.OFFSET, Deflate$
 _MEMFREE m
 Deflate$ = MKL$(FileSize) + Deflate$
 $CHECKING:ON
END FUNCTION

FUNCTION Inflate$ (text$)
 $CHECKING:OFF
 DIM m AS _MEM, m1 AS _MEM
 FileSize& = CVL(LEFT$(text$, 4))
 text1$ = MID$(text$, 5): temp$ = SPACE$(FileSize&)
 m = _MEMNEW(FileSize&): m1 = _MEMNEW(LEN(text1$))
 _MEMPUT m1, m1.OFFSET, text1$
 Result = uncompress(m.OFFSET, FileSize&, m1.OFFSET, LEN(text1$))
 _MEMGET m, m.OFFSET, temp$
 _MEMFREE m
 _MEMFREE m1
 Inflate$ = temp$
 $CHECKING:ON
END FUNCTION

its crashing when it tries to INFLATE the data, that much I've isolated. you may have to put your "cheesy Animation" back in the code in place of the vortex images.

[Cobalt]

I found one problem, the program no longer "stopped working". That was math: the offset should have be 96*4 not *8(longs are 4 bytes opps) but now I get out of memory errors after 2 decodes, and the image does not display.


PICKY PICKY PICKY math!!!!

forgot its not 96 longs its 98 longs(0-48 = 49 elements *2=98)
and then +1 to the offset for SOME reason?!
now it works...

this will play back the results.

Code: [Select]

'*******$include: 'Zlib.bi'
$IF 32BIT THEN
 DECLARE DYNAMIC LIBRARY ".\zlib1"
  FUNCTION zlibVersion$ ()
  FUNCTION zlibCompileFlags~& ()
  FUNCTION compressBound& (BYVAL LEN AS LONG)
  FUNCTION compress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
  FUNCTION compress2& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG, BYVAL level AS LONG)
  FUNCTION uncompress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
  FUNCTION crc32~& (BYVAL PNGCRC AS LONG, BYVAL buf AS _OFFSET, BYVAL LEN AS LONG)
  FUNCTION adler32~& (BYVAL adler AS LONG, BYVAL buf AS _OFFSET, BYVAL LEN AS LONG)
  FUNCTION zError$ (BYVAL codenum AS INTEGER)
 END DECLARE
$ELSE
 DECLARE DYNAMIC LIBRARY ".\zlib"
 FUNCTION zlibVersion$ ()
 FUNCTION zlibCompileFlags~& ()
 FUNCTION compressBound& (BYVAL LEN AS LONG)
 FUNCTION compress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
 FUNCTION compress2& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG, BYVAL level AS LONG)
 FUNCTION uncompress& (BYVAL dest AS _OFFSET, destLen AS LONG, BYVAL source AS _OFFSET, BYVAL sourceLen AS LONG)
 FUNCTION crc32~& (BYVAL PNGCRC AS LONG, BYVAL buf AS _OFFSET, BYVAL LEN AS LONG)
 FUNCTION adler32~& (BYVAL adler AS LONG, BYVAL buf AS _OFFSET, BYVAL LEN AS LONG)
 FUNCTION zError$ (BYVAL codenum AS INTEGER)
 END DECLARE
$END IF


SCREEN _NEWIMAGE(640, 400, 32)

P = _PIXELSIZE: IF P = 0 THEN P = 2 '0 is a designation for text screens, which use 2 bytes per character for memory storage
ScreenSize = _WIDTH * _HEIGHT * P
Frames = 48 'In our animation
UncompressedStringSize = Frames * ScreenSize 'the total size needed to store all the animation images, uncompressed (Should be about 48 MB)
UncompressedString$ = SPACE$(UncompressedStringSize) 'a placeholder for the whole animation, uncompressed.
SingleScreen$ = SPACE$(ScreenSize) 'a placeholder for a single screen worth of info

DIM Offs(48) AS LONG, Sizs(48) AS LONG

DIM m AS _MEM
m = _MEMIMAGE(0)

OPEN "Vortex.Ani" FOR BINARY AS #1


GET #1, , Elements%
GET #1, , Offs()
GET #1, , Sizs()

SLEEP


'CLS
DO
 FOR i = 0 TO 47
  temp$ = SPACE$(Sizs(i))
  '  temp$ = MID$(UncompressedString$, i * ScreenSize + 1, ScreenSize)
  GET #1, Offs(i), temp$

  UncompressedString$ = Inflate(temp$)

  _MEMPUT m, m.OFFSET, UncompressedString$
  _DELAY .1
  temp$ = ""
 NEXT
LOOP UNTIL INKEY$ = CHR$(27)

CLOSE #1

'*******$include:'Compress.bi'
FUNCTION Deflate$ (text$)
 $CHECKING:OFF
 DIM FileSize AS LONG, CompSize AS LONG
 DIM m AS _MEM
 FileSize = LEN(text$): CompSize = compressBound(FileSize)
 m = _MEMNEW(FileSize)
 _MEMPUT m, m.OFFSET, text$ 'set the variable length text into a memblock so it won't move in memory as we access it.
 REDIM CompBuff(1 TO CompSize) AS _UNSIGNED _BYTE
 Result = compress2(_OFFSET(CompBuff(1)), CompSize, m.OFFSET, FileSize, 9)
 REDIM _PRESERVE CompBuff(1 TO CompSize) AS _UNSIGNED _BYTE
 _MEMFREE m
 m = _MEM(CompBuff())
 Deflate$ = SPACE$(CompSize)
 _MEMGET m, m.OFFSET, Deflate$
 _MEMFREE m
 Deflate$ = MKL$(FileSize) + Deflate$
 $CHECKING:ON
END FUNCTION

FUNCTION Inflate$ (text$)
 $CHECKING:OFF
 DIM m AS _MEM, m1 AS _MEM
 FileSize& = CVL(LEFT$(text$, 4))
 text1$ = MID$(text$, 5): temp$ = SPACE$(FileSize&)
 m = _MEMNEW(FileSize&): m1 = _MEMNEW(LEN(text1$))
 _MEMPUT m1, m1.OFFSET, text1$
 Result = uncompress(m.OFFSET, FileSize&, m1.OFFSET, LEN(text1$))
 _MEMGET m, m.OFFSET, temp$
 _MEMFREE m
 _MEMFREE m1
 Inflate$ = temp$
 $CHECKING:ON
END FUNCTION


[SMcNeill]

and then +1 to the offset for SOME reason?!

It’s from the way the offsets are indexed.

A string (and a file saved on your drive) starts with the first byte.  Let’s say ABCD is saved on your drive.  That “A” is in byte #1, “B” is in byte number 2, and so on.

In memory, for whatever reason the designers did it for, MEM blocks start at the ZERO position.  _MEMPUT m, m.OFFSET, “ABCD”, and that “A” is in byte #0, “B” is in byte #1, and so on.

Files count from 1, _MEM counts from 0, and that difference will always necessitate either a +1 or -1 to map things properly, depending on which you start with and which you end with.  ;)

[Cobalt]

Ah that makes since.
Thanks again Steve.

[Cobalt]

Taking this to the next level! not only compresses multiple animation sets, also changes screen size for each set! although I, as the end user, have to keep track at what size each set is saved as. might change that too!

618,916,800 bytes down to 15,679,655! reduction of ~97.5% not bad, but still a big download for some members.