Loading faster with _Loadimage?

[40wattstudio]

My game has a DO LOOP that loads 35 .png images for an opening animation. Each image measures 1280 x 1280 with sizes ranging from 1 to 1.5MB. So about 35 to 52 MB worth of images here.
It takes the game approximately 8-9 seconds to get them all loaded.

Are there any cool tricks with or in conjunction with _LOADIMAGE that might help in reducing load times?

I'm also a bit curious about the Wiki entry about the Mode 32 and Mode 33 options for _LOADIMAGE. For Mode 33, does hardware accelerated mean that it should load images faster? I did some experiments with Mode 33 but didn't notice any time difference.

[NOVARSEG]

File access takes some time. I think LOADIMAGE is fast. Try converting PNG to BMP and see if that makes a difference.

[40wattstudio]

I was a bit doubtful because converting everything to .bmp increased all the individual images to about 6MB . . . but I have to say it did cut the load time down to about 5-6 seconds instead of 9. I'm guessing it has to do with the fact that .png is compressed and the decompressing takes longer to process than to deal with a 6MB file (bmp) that's already decompressed.
Thanks for the suggestion!

[NOVARSEG]

very interesting 40wattstudio

I said before that LOADIMAGE was fast and it is but since you are using BMP files then there is some code that me and bplus steve and others worked on that might be even faster than LOADIMAGE

I will try to find it.

[NOVARSEG]

The SUB version .  returns H = image height in pixels which can be negative or positive and W = image width in pixels
returns image handle.

The code can handle negative height BMPs too.

Code: QB64: [Select]

1. Sub loadBMP24 (FILENAME As String, W As Long, H As Long, HANDLE As Long)
2.  
3.     Dim L As _Unsigned Long
4.     Dim I As _Unsigned Integer
5.     Dim C As _Unsigned Long
6.     Dim R As _Unsigned Long
7.     Dim OF As _Unsigned Long
8.     Dim N As _Unsigned Long
9.     Dim tt As String * 4
10.     Dim t As String * 3
11.     Dim pad As Integer
12.     Dim a As _MEM 'not defined
13.  
14.     N = 0
15.  
16.     Open FILENAME For Binary As #1
17.     Get #1, 11, OF 'Offset to picture dat
18.     Get #1, 15, L 'Header size
19.     Get #1, , W 'image width
20.     Get #1, , H 'image height
21.     Get #1, , I 'biPlanes. Specifies the number of color planes on the target device.
22.     Get #1, , I 'bits per pixel. Must = 24 for the program to work.
23.     If I <> 24 Then W = 0: GoTo LL1
24.     HANDLE = _NewImage(W, Abs(H), 32)
25.     a = _MemImage(HANDLE)
26.     pad = 4 - (W * 3) Mod 4
27.     If pad = 4 Then pad = 0 'pad = multiple of 4
28.     If H < 0 Then
29.         For R = 0 To Abs(H) - 1
30.             For C = 1 To W * 3 Step 3
31.                 Get #1, R * (W * 3 + pad) + C + OF, t
32.                 tt = t + Chr$(255)
33.                 _MemPut a, a.OFFSET + N, tt
34.                 N = N + 4
35.             Next C
36.         Next R
37.     End If
38.     If H > 0 Then
39.         For R = H - 1 To 0 Step -1
40.             For C = 1 To W * 3 Step 3
41.                 Get #1, R * (W * 3 + pad) + C + OF, t
42.                 tt = t + Chr$(255)
43.                 _MemPut a, a.OFFSET + N, tt
44.                 N = N + 4
45.             Next C
46.         Next R
47.     End If
48.     LL1:
49.     Close
50. End Sub

[NOVARSEG]

Well the above code was tested again and it is really slow.  I think it has to do with the file access. Trying to read more bytes of data from a file per file read.

[NOVARSEG]

Ok

    DIM fileBuffer AS STRING * 16383
    DIM m AS _MEM
    m = _MEM(fileBuffer)

I need these 3 lines of code to work with GET.  fileBuffer is the memory block with a max size of 16383 bytes.  MEMGET must   access fileBuffer and then do some calcs and MEMPUT to the handle given by NEWIMAGE etc.

In assembler I could make a DOS function call to READFILE and it will read a file by a specified number of bytes and put those bytes in, say, fileBuffer.    GET has no such number input as it gets it's length info from string length.  Right away we gotta do some serious gymnastics to get this to work.

The objective here is to read a file by chunks of a least 16383 bytes at a time instead of 3 bytes at a time. This should seriously speed up the code.

MEM works with fixed length strings (which is good). The last file read will be a chunk either 0 bytes or less than 16383 bytes, so that looks like a variable length string is involved but MEM must work with fileBuffer which has a fixed max length of 16383.

If the last chunk is , say, 501 bytes it must be stored to fileBuffer.  In that way MEM references the same memory block.

However GET wants something like string = space$(chunksize)  but now MEM is hooped I think.

Maybe our API wizard could give us a READFILE SUB

(write a specified number of bytes to fileBuffer)



***** correction

I said "read a file by chunks of a least 16383 bytes"

should read " read a file by chunks of 16383 bytes or less"

[NOVARSEG]

@SpriggsySpriggs

I'm trying to code a fast BMP viewer /slide show.  The code is essentially complete but it is very slow due to inefficient file access.  GET statement makes it a nightmare to code this properly.  GET is so prehistoric it just don't work for what I want to do.

Can you write a READFILE sub? (API) That will make GET obsolete.   GET can be still used to support existing code. 

READFILE SUB criteria

A file is read by a specified number of bytes which is stored in a memory block named fileBuffer

fileBuffer can be  Dimensioned outside of the sub maybe like

DIM SHARED fileBuffer as STRING * 16383   (example length)

Now API might have a different way to describe fileBuffer etc ?

So all we got to do is specify the file name and bytes to read. Of course the number of bytes to read can't exceed the string length.

GET is like trying to write an OS with batch file commands.



   

[SMcNeill]

There’s nothing wrong with GET.

If you want the fastest load times, just get the whole file at once and then parse it.

OPEN file$ FOR BINARY AS #1
temp$ = SPACE$(LOF(1))
GET #1, 1, temp$
CLOSE


Then parse it out in 12345 bytes at a time, until finished, if that’s what you need.


(And, since you’re one of those folks who likes to argu, “Noooo, my way is much better!!”, I just want to bring one thing to your attention in libqb.cpp:

Code: [Select]

                    if (ReadFile(f_w->file_handle,data,size2,(unsigned long*)&bytesread,NULL)){
                        data+=bytesread; f->pos+=bytesread; gfs_read_bytes_value+=bytesread;
                        if (bytesread!=size2){   
                            ZeroMemory(data,size+(size2-bytesread));//nullify remaining buffer
                            f->eof_passed=1; return -10;
                        }//eof passed
                        }else{
As you can see, QB64 is already using ReadFile internally for GET…

Which means it can’t be as obsolete as you apparently think it is.  :P

[NOVARSEG]

 @SMcNeill

I have learned that you are usually correct and yes I love a good argument as long as it is a constructive one.

So READFILE is inside some compiled C++  code.  The thing is GET statement gets it's bytesToRead value from string length.  GET should be like this

GET handle, bytesToRead,  fileBuffer

where handle is a file handle
bytesToRead is the number of bytes to read from the file
fileBuffer is where the file bytes are stored.

Only one fileBuffer.

However you can not enter a bytesToRead value directly into GET.   GET essentially second guesses what the coder would REALLY like.  That leads to complications.

GET figures out bytesToRead from (example)

string1 = space$(bytesToRead)
GET #1,string1   etc

so string1 is a variable length string - not fixed.

The thing is MEM  works with a single memory block (fixed length) but GET works with multiple (variable) memory blocks (strings). It does not work out too well.

If you want the fastest load times, just get the whole file at once and then parse it.

I thought about that and it does work but to me that is a band aid approach.

Trust me!   if I had a READFILE API sub the BMP viewer would be a 1000 times faster.

About 50 hours has gone into the BMP viewer why stop there. We have a resident API wizard and so much absolute talent,  what other forum has that  - none.

The other thing is that a READFILE sub is almost the same as a WRITEFILE sub


Think of all the high speed file access QB64 can do with some practical API subs.  Some might think it is UNBASIC.  Well look at modern day OSes has anyone noticed that they do not resemble the OSes when BASIC was invented. 

[NOVARSEG]

*******

example subs

ReadFile handle, bytesToRead, rFileBuffer

WriteFile handle, bytesToWrite, wFileBuffer

It might be advantage to have two buffers, one for read and one for write.

[NOVARSEG]

Lets look into the WIndows readfile etc


https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-readfile

Reads data from the specified file or input/output (I/O) device. Reads occur at the position specified by the file pointer if supported by the device.

That is OK we might as well include a file pointer sub too. here is how WIndows wants the sub  (er whatever it is) to look like

BOOL ReadFile(
  HANDLE       hFile,
  LPVOID       lpBuffer,
  DWORD        nNumberOfBytesToRead,
  LPDWORD      lpNumberOfBytesRead,
  LPOVERLAPPED lpOverlapped
);

I don't what the BOOL is . Those parameters look OK but there is LPOVERLAPPED lpOverlapped and that one i'm not familiar with.

There is only a 5 minute window to edit posts so this post and  other posts will have mistakes so keep that in mind.

lpOverlapped

A pointer to an OVERLAPPED structure is required if the hFile parameter was opened with FILE_FLAG_OVERLAPPED, otherwise it can be NULL.

If hFile is opened with FILE_FLAG_OVERLAPPED, the lpOverlapped parameter must point to a valid and unique OVERLAPPED structure, otherwise the function can incorrectly report that the read operation is complete.

For an hFile that supports byte offsets, if you use this parameter you must specify a byte offset at which to start reading from the file or device. This offset is specified by setting the Offset and OffsetHigh members of the OVERLAPPED structure. For an hFile that does not support byte offsets, Offset and OffsetHigh are ignored.

For more information about different combinations of lpOverlapped and FILE_FLAG_OVERLAPPED, see the Remarks section and the Synchronization and File Position section.

and that is why we pay $20,000 for an expert to explain this all to us.

[NOVARSEG]

The only way I can learn Windows API is with a tutor. I'm not gonna spend 1000s of hours trying to figure this out on my own.   

Until then I'm not gonna be able to finish the BMP viewer.  Hey BG your windows API is bullshit.

[NOVARSEG]

I've decided to leave the forum.  nothing more to do here. API is the last hurdle - it is not worth it.

[SMcNeill]

I've decided to leave the forum.  nothing more to do here. API is the last hurdle - it is not worth it.

Bye.