Steve64 Repo

[SMcNeill]

So I finally decided to sit down and make my own repo for QB64 stuff.  Before anyone asks, "Are you trying to break off from the QB64 team and do your own stuff?", the answer is, "Nope!"

All I'm out to do is set me up a nice little repo to play around in and test things out to see if I can make them work properly.  Github Desktop recently did an update to their GUI, and I'm wanting to test it out and learn it better.  I've also been wanting to test out adding the capability to have SUB/FUNCTIONs go anywhere in our code and still compile without issues -- like so:

Code: [Select]

PRINT foo

FUNCTION foo
STATIC x
x = x + 1
foo = x
END FUNCTION

PRINT foo
QB45 used to allow us to write code that looked like that, (the above would print 1 and then print 2 to the screen), and it's something I've long wanted to get working for us so that library files could be merged and kept together and $INCLUDEd at the top of our code.

So, this seemed like a good time for me to just work on it all together...  A repo for experimental changes and testing purposes, which I can use to archive things which I want to keep, as well...

You'll find the repo here: https://github.com/SteveMcNeill/Steve64

Currently there's 22 archives in it (zip, 7z, rar) and over 50 *.BAS files all saved there.

Inside the /source folder, you'll see my latest attempt to work out the kinks with the SUB/FUNCTION anywhere placement, in the qb64SFA.bas file, for those of you who might want to play around with it and test it out.  It compiles most small programs which I test  it on, but it dies when trying to compile QB64.bas itself, and I don't know yet exactly what the glitch might be.  I'll keep playing around with it as time and effort allows...

Anywho...   Enjoy!  Feel free to take, steal, borrow, or modify anything you want from the repo.  It's all 100% available for the public to make use of. 

[SMcNeill]

Current differences in my repo and standard QB64:

Zlib has been added into the source and is now one of our “parts”.  This gives us two new commands, which can be used to compress and uncompress strings.

Function _DEFLATE$ (Text$)
Function _INFLATE$ (Text$, {Uncompressed_Size})


Custom IDE enhancements:

A clock in bottom of the screen.
$AUTOSAVE:minutes — a new command to time stamp and AUTOSAVE our work.
$REPLACE — a new command to set quick typing codes for ease of coding.


Differences between the repo and QB64:

OPTION _EXPLICT won’t throw errors for variables declared via SHARED in a sub/function.

When OPTION _EXPLICIT does throw an error, it overwrites the clipboard with a default DIM statement so the user can paste it in the appropriate place.  (If desired, of course.)

_INSTRREV now terminates with a return of 0, rather than going into an endless loop.

_PUTIMAGE can now work with the same _DEST and _SOURCE.

Corrected IDE error message for "Unidentified open control block" when a SUB/FUNCTION is open without an END SUB/FUNCTION.


Window's CONSOLE support has been extended so that:

CSRLIN support added.
POS(0) support added.
LOCATE support added. (Works with optional parameters.)
COLOR support added.
SCREEN support added to get both character and color information of any point on the console.
tab() glitch fixed. (Which could cause an endless loop when printing with comma spaced text/numbers.)
_WIDTH support added.
_HEIGHT support added.
WIDTH support added, with 2 new parameters added so we can set the buffer width and buffer height also.
CLS support semi-added.  (Doesn't accept colored backgrounds; it clears the screen black.  I'm getting tired of working up console stuff which I probably won't ever use myself...)
SLEEP support added.
END support added so we now end with any keypress and not just enter.

_CINP(toggle) support added, which allows us to get single character keystrokes from the console.
_CONSOLEFONT FontName, FontSize support added, which allows us to change the font and font size we use for the console.
_CONSOLECURSOR _SHOW|_HIDE, cursorsize support added, which allows us to show or hide the cursor in the console, and to change its size (from 0 to 100), as desired.


New keyboard commands added:

_CAPSLOCK -- returns 1 if caps lock is on, 0 if it isn't.
_NUMLOCK -- as above, but for num lock
_SCROLLOCK -- as above, but for scroll lock

_TOGGLE_CAPSLOCK -- toggles the caps lock state for us.
_TOGGLE_NUMLOCK -- same, but for num lock
_TOGGLE_SCROLLOCK -- same, but for scroll lock

Further details can be read in the posts which follow.

[RhoSigma]

'Note:  _INFLATE$ requires that we know the size of the original text to get it back properly.
'           When saving your compressed data, it's probably a wise idea to save it with the legnth
'           before the compressed information, like most archives, images, ect does.

Why not doing this automatically within _DEFLATE? I do it the same way in my LZW packer lib, just after you copied the compressed string over into the qbs string simply add MKL$(originalSize) to it.
Now _INFLATE$ can just look at the last 4 bytes of the given compressed string, CVL() it and use it.

[SMcNeill]

Why not doing this automatically within _DEFLATE? I do it the same way in my LZW packer lib, just after you copied the compressed string over into the qbs string simply add MKL$(originalSize) to it.
Now _INFLATE$ can just look at the last 4 bytes of the given compressed string, CVL() it and use it.

I’d considered doing that, but if somebody uses the routine to open other files (like a PNG image), you’re not going to get those bytes tacked on to it.  We’d end up with corrupted/incomplete data.

I also considered making it an auto-sizing routine, but worry it could become very CPU expensive and time consuming...

DO
    BuffSize = BuffSize + 3000000
    ‘Try the extraction
LOOP UNTIL result <> -3 ‘zlib tosses a -3 result for buffer not large enough

Basically resize and keep trying until the buffer is large enough to hold the uncompressed data — but how much delay would that generate, with a 600MB compressed file?

Best solution might be to make that parameter optional.  If it’s there, go directly to uncompress that size data; otherwise resize until we get a valid result.

[RhoSigma]

To reduce the amount of loops until the buffer is large enough, you could make a good guess on the initial size.

What's the average compression ratio zlib does reach (needs some testing with different filetypes of course), for simplicity lets assume 75%, then you could start with a buffer size which is at least 4 times as large as the passed in compressed data. In best case it's already large enough, in worst case you have to expand the buffer. Expanding coud than be done by adding another eg. 25% of the current buffer size bytes (that's IMHO better than adding a fixed size). All what you must do in the end is to return only the finally used size of the buffer, because the buffer becomes probably larger than finally required for unpacking, but i guess zlib inflate will return the unpacked size on success.

As this buffer enlarging probably takes place in your created C-function for _INFLATE$ you could simply use realloc() for this purpose, which would do the task as efficient as possible.

Hence:
bufSize = inSize*4;
mem = malloc(bufSize);
If required {
    bufSize += (bufSize \ 100 * 25);
    realloc(mem, bufSize);
}

[SMcNeill]

ZLIB has now been successfully integrated into QB64's parts system.

Now, I'm certain a lot of folks have no idea what that statement means, so I'll explain it, briefly:


Basically, QB64 calls on several various libraries to make their options available for us.  If we need those libraries, we include them in our EXE which we generate, else we leave them out.  Galleon called this QB64's "Part's System" where we can include and exclude only the parts which are needed for our code.  It was his attempt to try and decrease the size of EXE files created by QB64, and helped considerably (though it could help more, if QB64 made more use of it than it currently does).

The ZLIB library, which I've added to my repo, has now been integrated into that parts system.  If you use _DEFLATE$ or _INFLATE$, then QB64 will automatically link to those libraries and add the necessary functionality into the language for you.  If you don't need them, you go on creating EXE files the same as before, with ZLIB nowhere in your code.

As long as you're a Window's user, you should be able to grab a copy of my repo and run code such as the following, with no problems:

Code: QB64: [Select]

1. text$ = "I like cheese."
2. FOR i = 1 TO 10: text$ = text$ + text$: NEXT
3. de$ = _DEFLATE$(text$)
4. PRINT LEN(de$), LEN(text$)
5. PRINT de$
6. SLEEP
7. CLS
8. o$ = _INFLATE$(de$, LEN(text$)) 'Note:  _INFLATE$ requires that we know the size of the original text to get it back properly.
9. '                                When saving your compressed data, it's probably a wise idea to save it with the legnth
10. '                                before the compressed information, like most archives, images, ect does.
11. '                                Just:
12. '                                PUT #filehandle, ,OriginalSize
13. '                                PUT #filehandle, ,CompressedString
14. PRINT o$
15. PRINT LEN(o$), LEN(text$)

Now all I need to do is sort out how the heck to colorize the keywords, so they'll be the proper color and not white in the IDE.

[RhoSigma]

Now all I need to do is sort out how the heck to colorize the keywords, so they'll be the proper color and not white in the IDE.

The keywords list is here qb64\source\ide\ide_global.bas around line 120. As far as i found out it's the only list, which holds regular as well as meta keywords. I guess the different coloring in the IDE is then done on the fly by checking if the keyword starts with a $ sign, then meta-keyword color is used, otherwise regular keyword color.
I've added _INFLATE/_DEFLATE as well as your $COLOR meta keyword and it seems to work properly.

BTW - Congrats and thank you for this valueable addition.

[SMcNeill]

Change pushed into repo; we now have the keywords properly colorized inside the IDE. 

Many thanks, Rho, for pointing out the proper place for those.  I was searching all over the dang place and couldn't find them!  ;)

[SMcNeill]

Thought I'd also point out, for those who are cloning my repo and might not know, Steve64 repo also has a few other features which I use/abuse, which the standard QB64 doesn't have.

* There's a clock down at the lower part of the screen, as I always lose track of time when coding.  Without it, I'd never know what time of day it is, usually.

* There's also the inclusion of the $AUTOSAVE command, which will save timestamped copies of your work, for when you want that extra sense of security for large projects.  Sure, we have CTRL-Z for undoing things, but it's a real PITA to back up 10,000 times and try and undo back to where you were an hour ago.  $AUTOSAVE lets us specify a time and then QB64 will autosave our work every X amount of time (as long as our work has changed).  Usage is:

$AUTOSAVE:minutes

So, a working example would be the following to autosave our program every 15 minutes, with a fresh timestamp:

$AUTOSAVE:15

There's a few other little tweaks and changes to things in the repo, but nothing too major.  OPTION _EXPLICIT doesn't complain when we declare a variable via SHARED inside a sub.  When we get an error with option explicit, it gives us a DIM statement all set up in our clipboard, so all we have to do is paste it in the  proper place where we want it.  Maybe a few other things, but I can't think of what they are off the top of my head.

Just be aware; Steve64 isn't 100% QB64, as it's my own personal version of QB64 which I tend to tweak and alter shamelessly so that it'll behave the way that's most efficient for my programming needs/style.  ;)

[SMcNeill]

As talked about above, with RhoSigma (Is anyone else even looking at this topic?  All I hear is crickets from the user base here...  :P), the second parameter on _INFLATE$ has now became optional.

Personally, I wouldn't recommend using it as an optional parameter, but ultimately, that's now up to you guys.

The way it works is exactly as I described above -- set an initial size for the decompression buffer, try and decompress and see if we manage to decompress in the buffer allocated, and if not, try again.  And again... And again... And again...  ...Until we have enough of a buffer to finally hold the uncompressed data.

What could be one pass of the decompression routine (with you passing it the correct size in advance), may instead end up being dozens (or potentially hundreds) of passes as we keep trying and allocating more and more memory until we know how much we need.  It's not efficient.  It's not something I personally think folks should do very often (if at all).  BUT, I'll be the first to admit -- I'll probably be using the option myself.  (The initial buffer is set at 10mb in size, so I imagine that it'll be large enough to hold most things I'd need decompressed -- a 1600x1200x32-bit image is less than 8mb, for instance -- so it'd just be a shortcut for usage with relatively smallish files for me.)

Initial buffer size is 10mb, and it resizes by 10mb per attempt, so a 500mb compressed file would take 50 tries before it automatically resized to allocate enough memory to decompress it.  (For heaven's sake though, if you're working with compressing 500mb files, use an extra 4 dang bytes to store the size at the front or the end of the file, and save the world the hassle!)

A very quick example of the changes in action would be a program like this one:

Code: QB64: [Select]

1. text$ = "I like cheese."
2. FOR i = 1 TO 20: text$ = text$ + text$: NEXT
3. de$ = _DEFLATE$(text$)
4. PRINT LEN(de$), LEN(text$)
5. PRINT de$
6. SLEEP
7. CLS
8. o$ = _INFLATE$(de$) 'Note:  use _INFLATE$ optional's parameter if we know the size of the original text to get it back properly.
9. '                                When saving your compressed data, it's probably a wise idea to save it with the legnth
10. '                                before the compressed information, like most archives, images, ect does.
11. '                                Just:
12. '                                PUT #filehandle, ,OriginalSize
13. '                                PUT #filehandle, ,CompressedString
14. PRINT o$
15. PRINT LEN(o$), LEN(text$)

[SW]

thanks for adding default compression in QB64.
But it bit outdated. What about something better... like LZ4 ?

[SMcNeill]

thanks for adding default compression in QB64.
But it bit outdated. What about something better... like LZ4 ?

Zlib is used with/in libpng, which is what I actually need it for — PNG file creation.  All that’s left for me, at this point, is adding the CRC and Adler functions into the language for us, and then I’ll be done and moving on to other things in the near future.  ;)

Feel free to use the history of my repo as a means to help import whichever library you require for your own needs.  The process wasn’t as difficult as I’d first imagined it would be.  ;)

[RhoSigma]

Steve,
thanks for considering and finally make the "original size" argument optional, guess 10mb will be sufficient for most things people will do with QB64.

Are you willing to merge all these great enhancements in your personal QB64 copy into the official repo anytime?

After you posted the basic instructions for adding new commands into the language in the other thread, I was thinking about adding my HSB/HSV colorspace stuff into QB64, which is now only available in my Library collection. However, I will probably contribute to the official repo. For sure it's easily possible to have more QB64 copies on the drive, and all will coexist each in its own folder, but after already having a couple of them (eg. the original v0.954 SDL, another SDL which I've fixed a little bit by backporting enhancements/fixes done in v1.1-v1.3, your SDL v0.954e?, the official v1.3 and the actual dev build), it becomes hard to remember all the specialties in each version, so it would be reasonable to bring the changes into the official repo once upon time.

[SMcNeill]

10mb is the starting size I chose for the buffer, but it’ll resize as needed to uncompress larger things.  It just won’t be very efficient at it; so it’s probably best to store the size yourself, for larger files/blocks of memory.

I’ll be happy to push changes into QB64 repo so everyone can use these commands, but it’d be nice to get them working for Linux/Mac as well, first.  I can run Linux in a virtual machine, and test things for it, but I don’t have any means to test anything for Mac.  When adding to the official version of QB64, we try and make certain everything works cross-platform, as much as possible, so it might take a while before the changes are actually merged into the development build (and later become part of the next official build). 

Eventually, I imagine we’ll see the changes merged in “official” QB64, but I wouldn’t hold my breath expecting to see them incorporated overnight.

[SMcNeill]

Pushed a few more things into my repo, mainly to customize how it behaves for me. 

Added $REPLACE into the language, as a quick typing replacement function.  Only affects the IDE itself; has no actual effect on any programs.  It can safely be removed from any program and will change functionality ZILCH...

So what's it do?

First, set a quick code for typing something:

Code: [Select]

$REPLACE ?? = _PRINTSTRING(
Then type something using that code:

Code: [Select]

??10,10), "Hello World"
The moment you hit enter, that code will magically change to become:

Code: [Select]

_PRINTSTRING(10,10), "Hello World"
With excessively long commands, or type names, it can be a complete lifesafer in terms of knuckle stress and typing needed to work on a program.

Just one cavet:  Be careful what you set as a replace code.  With the above, typing in, PRINT "What's your name??" would instantly turn into PRINT "What's your name_PRINTSTRING(".   

Usually I'll just make a quick replacement with q(number).   

$REPLACE q1 = MyVariableWithALongArseName.
$REPLACE q2 = AnotherLongArseNamedVariable.

Then I can type in:

q1x = q2x + 1

And it'll become:

MyVariableWithALongArseName.x = AnotherLongArseNamedVariable.x + 1

That's a ton of typing to save!


Also added _BORDERWIDTH and _TITLEBARHEIGHT, so we can get the value of these elements, if we ever want them
_BORDERWIDTH is, oddly enough, the width of the border of our screen, while _TITLEBARHEIGHT gives us the height of our titlebar.  Works in both Linux and Windows, though I don't know about Mac.


Altered _INSTRREV so it terminates at 0, instead of going into an endless repetitive loop.


I don't know if anyone else will ever take advantage of these enhanced (at least I think they are) features, but they're in there, so I thought I'd share here and let folks be aware of them so they can decide for themselves if they want to make use of them or not.