Working on a Audio+Video file format to use.

[Dav]

It's been a project of mine to come up with an audio/video file format for QB64 programs to use/play, so here's an idea I am working on for doing that.

For the audio, I noticed that _SNDPLAY can play .OGG files properly even with junk attached to the end of them.  For the compressed video, I used the new _DEFLATE command to compress screen grabs and attached those frames to the end of the OGG file.  The result is one file with audio and video that can be played back with a small amount of code.

Could a few of you test this code and see if the small video file I've attached plays OK for you?  If it works OK, then I can post a converter program to make your own video files to play with in QB64.  Thanks.

- Dav

Code: QB64: [Select]

1. SCREEN _NEWIMAGE(640, 480, 32)
2.  
3. REDIM m AS _MEM: m = _MEMIMAGE(0)
4.  
5. b& = _SNDOPEN("fish2.dav"): _SNDLOOP b&
6.  
7. OPEN "fish2.dav" FOR BINARY AS #1
8.  
9. DO
10.     'jump to where video data begins
11.     SEEK 1, 14470
12.  
13.     FOR vid = 0 TO 55
14.         framelen$ = INPUT$(4, 1)
15.         frame$ = INPUT$(CVL(framelen$), 1)
16.         frame$ = _INFLATE$(frame$)
17.  
18.         _MEMPUT m, m.OFFSET, frame$
19.  
20.         _LIMIT 30
21.  
22.         IF INKEY$ <> "" THEN EXIT DO
23.  
24.     NEXT
25.  
26. LOOP
27.  
28. CLOSE #1: _SNDSTOP b&
29.  
30. _MEMFREE m
31.  
32. END
33.  
34.  

Test file to use -->

fish2.dav

 

[SierraKen]

This took awhile to compile to .exe for some reason, but after a minute or 2, it ran and worked great! I like the fish and the sounds.
The problem you might run into combining sound and video is computer lag slowing one or the other making them not lined up perfectly and in sync.
But this might be the very first BASIC language video program anyone has ever made, I've never seen any other like it before. I wish you luck on this. :)

[bplus]

Plays OK for me, what are you using for bait?

[Dav]

Thanks for testing.  I doubt real syncing audio and video can be done this way, but I have some ideas.  Maybe this will only be useful for small intros.  It's the only method so far I can figure out to have audio&video in one file (without extracting a bunch of image files to HD) while using QB64 commands to play it. 

- Dav

[euklides]

Video works. No sound ?
Very interesting !!!

[Petr]

Hi Dav,
This is really a very interesting solution. Maybe it would be a little better to place the audio-video boundary information (in the example shown, it is byte 14470) at the end of the file, then it will be possible to load it with the player. 32-bit images were used in the video, right? Of course, I had to dig into it a bit.
To synchronize audio and video - you can control the position of the sound via _SNDGETPOS in your concept. And also in your draft you can track the position of the frames, just add a frame counter to the loop. I think that's a perfect concept.

I forgot! Video + audio plays right for me!

[TempodiBasic]

Hi Dav

Great!
It runs ok. I cannot understand if syncronization is ok or no if video is too short to hear/see a distortion between video and sound.

A combination of sound and video! It can be an internal video player for cutscene or little videohelp or moreover....

[Petr]

So. I think the code will be better instead of bullshit. First, use the first program to upgrade the existing .dav file

Then use the second program to play the upgraded DAV file. The second program shows a simple video / audio synchronization (no _LIMIT)

Code: QB64: [Select]

1. DIM ByteBorder AS LONG, FrameCounter AS LONG
2. ByteBorder& = 14470& 'value, which return your DAV video maker
3. FrameCounter = 55&
4. OPEN "fish2.dav" FOR BINARY AS #1
5. PUT #1, LOF(1) + 1, ByteBorder
6. PUT #1, , FrameCounter
7. CLOSE 1
8.  

Code: QB64: [Select]

1. SCREEN _NEWIMAGE(640, 480, 32)
2. REDIM m AS _MEM: m = _MEMIMAGE(0)
3. B& = _SNDOPEN("fish2.dav"): _SNDLOOP B&
4.  
5. OPEN "fish2.dav" FOR BINARY AS #1
6. DIM framelen AS LONG, border AS LONG, FrameCount AS LONG
7. SEEK #1, LOF(1) - 7
8. GET #1, , border
9. GET #1, , FrameCount
10. DO
11.     'jump to where video data begins
12.     SEEK 1, border
13.     FrameDelay = _SNDLEN(B&) / FrameCount
14.     T! = TIMER + FrameDelay
15.     vid = 0
16.  
17.     DO UNTIL vid = FrameCount
18.         IF TIMER > T! THEN
19.             GET #1, , framelen
20.             fram$ = SPACE$(framelen)
21.             GET #1, , fram$
22.             frame$ = _INFLATE$(fram$)
23.             _MEMPUT m, m.OFFSET, frame$
24.             vid = vid + 1
25.             T! = TIMER + FrameDelay
26.         END IF
27.         IF INKEY$ <> "" THEN EXIT DO
28.     LOOP
29. LOOP
30. CLOSE #1: _SNDSTOP B&
31. _MEMFREE m
32. END
33.  

Good news for us is, that ogg file size does not affect the value returned by the _sndlen function.

[Ashish]

This is awesome! Great job! :D

Here is my 3D twist to your program Dav :) (Need fish2.dav to run)

Code: QB64: [Select]

1. SCREEN _NEWIMAGE(640, 480, 32)
2.  
3. buffer& = _NEWIMAGE(_WIDTH, _HEIGHT, 32)
4. REDIM SHARED m AS _MEM: m = _MEMIMAGE(buffer&)
5.  
6. b& = _SNDOPEN("fish2.dav"): _SNDLOOP b&
7.  
8. OPEN "fish2.dav" FOR BINARY AS #1
9.  
10. DO
11.     'jump to where video data begins
12.     SEEK 1, 14470
13.  
14.     FOR vid = 0 TO 55
15.         framelen$ = INPUT$(4, 1)
16.         frame$ = INPUT$(CVL(framelen$), 1)
17.         frame$ = _INFLATE$(frame$)
18.  
19.         _MEMPUT m, m.OFFSET, frame$
20.  
21.         _LIMIT 30
22.  
23.         IF INKEY$ <> "" THEN EXIT DO
24.  
25.     NEXT
26.  
27. LOOP
28.  
29. CLOSE #1: _SNDSTOP b&
30.  
31. _MEMFREE m
32.  
33. SUB _GL ()
34.     STATIC texture AS LONG, init, t
35.     IF init = 0 THEN
36.         _glGenTextures 1, _OFFSET(texture)
37.         init = 1
38.     END IF
39.  
40.     _glEnable _GL_TEXTURE_2D
41.     _glClearColor 1, 1, 1, 1
42.     '_glClear _GL_COLOR_BUFFER_BIT
43.  
44.     _glBindTexture _GL_TEXTURE_2D, texture
45.     _glTexImage2D _GL_TEXTURE_2D, 0, _GL_RGBA, _WIDTH, _HEIGHT, 0, _GL_BGRA_EXT, _GL_UNSIGNED_BYTE, m.OFFSET
46.  
47.     'set out texture filtering
48.     _glTexParameteri _GL_TEXTURE_2D, _GL_TEXTURE_MAG_FILTER, _GL_LINEAR
49.     _glTexParameteri _GL_TEXTURE_2D, _GL_TEXTURE_MIN_FILTER, _GL_NEAREST
50.  
51.  
52.     t = t + 1
53.     _glRotatef t, 0.5, 1.5, 1
54.  
55.     _glBegin _GL_QUADS
56.     _glTexCoord2f 0, 1
57.     _glVertex2f -0.7, 0.7
58.     _glTexCoord2f 1, 1
59.     _glVertex2f 0.7, 0.7
60.     _glTexCoord2f 1, 0
61.     _glVertex2f 0.7, -0.7
62.     _glTexCoord2f 0, 0
63.     _glVertex2f -0.7, -0.7
64.     _glEnd
65.  
66.     _glBindTexture _GL_TEXTURE_2D, 0
67.     _glBegin _GL_LINE_LOOP
68.  
69.     _glVertex2f -0.7, 0.7
70.  
71.     _glVertex2f 0.7, 0.7
72.  
73.     _glVertex2f 0.7, -0.7
74.  
75.     _glVertex2f -0.7, -0.7
76.     _glEnd
77.  
78.  
79.  
80.  
81.  
82. END SUB
83.  
84.  
85.  

[Dav]

That's cool, Ashish!  I really got to start looking into GL!

Peter, Interesting -  Thanks!  I was hoping for something like that - and saving info to a header is something I was working on too (screen width,height, fps). Yes it is 32bit screen images because I used _LOADIMAGE to load a series of BMP images (though the BMPS's were 256 color ones).  Because of that the 256 color video (now 32 bit) is much larger than it needs to be.    I recently found a thread about loading 256 images and think I can implement it (here).  256 color videos is what I was aiming for because of file size - _DEFLATE doesn't seem to compress busy screen images too great.  A 3 second video can become several MB's in 32-bit, so a 256 color video will be more usable to me.

I made another short video of someone talking, and the video and audio syncs well in that.   I'll work on it some more and post something more here when I can.  Thanks for the encouragement all.

- Dav

[FellippeHeitor]

Great idea and concept, Dav. Thanks for sharing. I'll be watching this thread closely ❤️

[Dav]

Thanks, Fellippe.  I was able to make a 640x480, 256 color video this morning.  The file size was reduced from 3.7MB to 2.6MB.  Not as great reduction as I hoped, but it something at least.  I had to go old school to load a 256 BMP onto the screen.  Glad I saved my old Qbasic code!  Here that is for your amusement...

Code: QB64: [Select]

1.  
2. SCREEN _NEWIMAGE(640, 480, 256)
3.  
4. Load256BMP("image.bmp")
5.  
6. SUB Load256BMP (file$)
7.     'Loads uncompressed 256 color 640x480 BMP into QB64
8.     'NOTE: DOES NOT VALIDATE OR ERROR CHECK BMP file.
9.     '      THE CORRECT BMP IS ASSUMED.
10.  
11.     OPEN file$ FOR BINARY AS #11
12.  
13.     SEEK #11, 55 'skip header to pallette data
14.  
15.     OUT 968, 0
16.     FOR c% = 1 TO 256 'load & out pallette to screen
17.         A$ = INPUT$(4, 11)
18.         OUT 969, ASC(MID$(A$, 3, 1)) \ 4
19.         OUT 969, ASC(MID$(A$, 2, 1)) \ 4
20.         OUT 969, ASC(MID$(A$, 1, 1)) \ 4
21.     NEXT
22.  
23.     FOR y% = 479 TO 0 STEP -1 'do 480 rows
24.         A$ = INPUT$(640, 11) 'load a row of 640 pixels.
25.         FOR x% = 0 TO 639
26.             PSET (x%, y%), ASC(MID$(A$, x% + 1, 1)) 'plot pixel
27.         NEXT
28.     NEXT
29.  
30.     CLOSE 11
31. END SUB

[Dav]

Ok, here's a demo that shows where it's at right now, and also fully shows how you can make your own audio/video files.  I like how Petr did what he did, but I haven't implemented that yet.  I have saved the needed info to the end of the video file.   I decided to abandon 256 mode for now, couldn't get the palette to be consistent in different screen sizes for some reason. I'll go back to that later down the road.

The demo attachment  here creates a GIANT.DAV animation out of a sequence of images and .OGG file and then plays it back.  Btw, I called them .DAV files not because of my name, but for _DEFLATE+AUDIO+VIDEO files is what they are.

Lots of room for improvement here, like option to NOT have audio, just video.  I'm posting this now as is because I won't have much time to work on it this week.

 Deflate/Audio/Video GIANT demo -->

giant-dav.zip

(NOTE: Run GIANTMAKE.BAS first to create the GIANT.DAV video file)

- Dav

[FellippeHeitor]

Btw, I called them .DAV files not because of my name, but for _DEFLATE+AUDIO+VIDEO files is what they are.

... not JUST because of your name, you mean ☺️

Looking forward to compiling this one. I'll get back to you shortly.

[bplus]

Ok, here's a demo that shows where it's at right now, and also fully shows how you can make your own audio/video files.  I like how Petr did what he did, but I haven't implemented that yet.  I have saved the needed info to the end of the video file.   I decided to abandon 256 mode for now, couldn't get the palette to be consistent in different screen sizes for some reason. I'll go back to that later down the road.

The demo attachment  here creates a GIANT.DAV animation out of a sequence of images and .OGG file and then plays it back.  Btw, I called them .DAV files not because of my name, but for _DEFLATE+AUDIO+VIDEO files is what they are.

Lots of room for improvement here, like option to NOT have audio, just video.  I'm posting this now as is because I won't have much time to work on it this week.

 Deflate/Audio/Video GIANT demo -->

giant-dav.zip

(NOTE: Run GIANTMAKE.BAS first to create the GIANT.DAV video file)

- Dav

Works for me with some .\ added. Cool!