Author Topic: Embedding files in programs (FileToDATA convertor)  (Read 44517 times)

0 Members and 1 Guest are viewing this topic.

Offline RhoSigma

  • QB64 Developer
  • Forum Resident
  • Posts: 565
    • View Profile
Embedding files in programs (FileToDATA convertor)
« on: November 17, 2018, 06:13:23 pm »
The following two small programs are meant to convert any file (eg. Images, Sounds, Databases etc.) into an easy embedable format. Embedding is then as simple as putting a $INCLUDE line at the end of your program. The converted files will also have a "Ready to use" writeback function, which will rebuild the embedded file on disk when needed.

The first converter tool is named MakeDATA, as you may imagine by its name, it will convert the given file into a block of DATA lines. This is easy to use and absolutly BASIC only. It's best to embed small files like icons or sprites. However, for large files like fullsize digital photos or MP3 music it has a big backdraw at the compiling speed of your program and the final EXE size on one side and also to the writeback speed on the other side.

So for bigger files the second tool MakeCARR should be your choice, which its advantages are detailed below the MakeDATA codebox.

Quote
Note for Windows users:
The codeboxes below contain simple SCREEN 0 (text) based versions of the converter tools. You may have to go in to change your default paths, but they should work reliable on all OS's supported by QB64.
If you're on Windows, then you may rather wish to use my GuiTools based versions of it, just move on to
The GuiTools Framework: https://qb64forum.alephc.xyz/index.php?topic=809
Both converter tools are available as part of the QB64GuiTools.7z source archive.




And now for the simple (SCREEN 0) people:



Both of the following tools require the 'lzwpacker.bm' file available from my Libraries Collection here: https://qb64forum.alephc.xyz/index.php?topic=809

If you're using at least QB64 v1.4 and don't wanna use the extra Lzw packer libarary, but the QB64 built-in zlib compression instead, then simply substitute the LzwPack$ call (line 84) with the respective _DEFLATE$ call and the LzwUnpack$ call (line 187) with an _INFLATE$ call. Also delete the $INCLUDE line at the end.

MakeDATA.bas
Code: QB64: [Select]
  1. '+---------------+---------------------------------------------------+
  2. '| ###### ###### |     .--. .         .-.                            |
  3. '| ##  ## ##   # |     |   )|        (   ) o                         |
  4. '| ##  ##  ##    |     |--' |--. .-.  `-.  .  .-...--.--. .-.        |
  5. '| ######   ##   |     |  \ |  |(   )(   ) | (   ||  |  |(   )       |
  6. '| ##      ##    |     '   `'  `-`-'  `-'-' `-`-`|'  '  `-`-'`-      |
  7. '| ##     ##   # |                            ._.'                   |
  8. '| ##     ###### |  Sources & Documents placed in the Public Domain. |
  9. '+---------------+---------------------------------------------------+
  10. '|                                                                   |
  11. '| === MakeDATA.bas ===                                              |
  12. '|                                                                   |
  13. '| == Create a DATA block out of the given file, so you can embed it |
  14. '| == in your program and write it back when needed.                 |
  15. '|                                                                   |
  16. '| == The DATAs are written into a .bm file together with a ready to |
  17. '| == use write back FUNCTION. You just $INCLUDE this .bm file into  |
  18. '| == your program and call the write back FUNCTION somewhere.       |
  19. '|                                                                   |
  20. '| == This program needs the 'lzwpacker.bm' file available from the  |
  21. '| == Libraries Collection here:                                     |
  22. '| ==      https://www.qb64.org/forum/index.php?topic=809            |
  23. '| == as it will try to pack the given file to keep the DATA block   |
  24. '| == as small as possible. If compression is successful, then your  |
  25. '| == program also must $INCLUDE 'lzwpacker.bm' to be able to unpack |
  26. '| == the file data again for write back. MakeDATA.bas is printing   |
  27. '| == a reminder message in such a case.                             |
  28. '|                                                                   |
  29. '+-------------------------------------------------------------------+
  30. '| Done by RhoSigma, R.Heyder, provided AS IS, use at your own risk. |
  31. '| Find me in the QB64 Forum or mail to support&rhosigma-cw.net for  |
  32. '| any questions or suggestions. Thanx for your interest in my work. |
  33. '+-------------------------------------------------------------------+
  34.  
  35. '--- if you wish, set any default paths, end with a backslash ---
  36. srcPath$ = "" 'source path
  37. tarPath$ = "" 'target path
  38. '-----
  39. IF srcPath$ <> "" THEN
  40.     COLOR 15: PRINT "Default source path: ": COLOR 7: PRINT srcPath$: PRINT
  41. IF tarPath$ <> "" THEN
  42.     COLOR 15: PRINT "Default target path: ": COLOR 7: PRINT tarPath$: PRINT
  43.  
  44. '--- collect inputs (relative paths allowed, based on default paths) ---
  45. source:
  46. LINE INPUT "Source Filename: "; src$ 'any file you want to put into DATAs
  47. IF src$ = "" GOTO source
  48. target:
  49. LINE INPUT "Target Basename: "; tar$ 'write stuff into this file (.bm is added)
  50. IF tar$ = "" GOTO target
  51. '-----
  52. ON ERROR GOTO abort
  53. OPEN "I", #1, srcPath$ + src$: CLOSE #1 'file exist check
  54. OPEN "O", #2, tarPath$ + tar$ + ".bm": CLOSE #2 'path exist check
  55.  
  56. '--- separate source filename part ---
  57. FOR po% = LEN(src$) TO 1 STEP -1
  58.     IF MID$(src$, po%, 1) = "\" OR MID$(src$, po%, 1) = "/" THEN
  59.         srcName$ = MID$(src$, po% + 1)
  60.         EXIT FOR
  61.     ELSEIF po% = 1 THEN
  62.         srcName$ = src$
  63.     END IF
  64. NEXT po%
  65. '--- separate target filename part ---
  66. FOR po% = LEN(tar$) TO 1 STEP -1
  67.     IF MID$(tar$, po%, 1) = "\" OR MID$(tar$, po%, 1) = "/" THEN
  68.         tarName$ = MID$(tar$, po% + 1)
  69.         EXIT FOR
  70.     ELSEIF po% = 1 THEN
  71.         tarName$ = tar$
  72.     END IF
  73. NEXT po%
  74. MID$(tarName$, 1, 1) = UCASE$(MID$(tarName$, 1, 1)) 'capitalize 1st letter
  75.  
  76. '--- init ---
  77. OPEN "B", #1, srcPath$ + src$
  78. filedata$ = SPACE$(LOF(1))
  79. GET #1, , filedata$
  80. rawdata$ = LzwPack$(filedata$, 20)
  81. IF rawdata$ <> "" THEN
  82.     OPEN "O", #1, tarPath$ + tar$ + ".lzw"
  83.     CLOSE #1
  84.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  85.     PUT #1, , rawdata$
  86.     CLOSE #1
  87.     packed% = -1
  88.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  89.     packed% = 0
  90.     OPEN "B", #1, srcPath$ + src$
  91. fl& = LOF(1)
  92. cntL& = INT(fl& / 32)
  93. cntB& = (fl& - (cntL& * 32))
  94.  
  95. '--- .bm include file ---
  96. OPEN "O", #2, tarPath$ + tar$ + ".bm"
  97. PRINT #2, "'============================================================"
  98. PRINT #2, "'=== This file was created with MakeDATA.bas by RhoSigma, ==="
  99. PRINT #2, "'=== you must $INCLUDE this at the end of your program.   ==="
  100. IF packed% THEN
  101.     PRINT #2, "'=== ---------------------------------------------------- ==="
  102.     PRINT #2, "'=== If your program is NOT a GuiTools based application, ==="
  103.     PRINT #2, "'=== then it must also $INCLUDE: 'lzwpacker.bm' available ==="
  104.     PRINT #2, "'=== from the Libraries Collection here:                  ==="
  105.     PRINT #2, "'===    https://www.qb64.org/forum/index.php?topic=809    ==="
  106. PRINT #2, "'============================================================"
  107. PRINT #2, ""
  108. '--- writeback function ---
  109. PRINT #2, "'"; STRING$(LEN(tarName$) + 18, "-")
  110. PRINT #2, "'--- Write"; tarName$; "Data$ ---"
  111. PRINT #2, "'"; STRING$(LEN(tarName$) + 18, "-")
  112. PRINT #2, "' This function will write the DATAs you've created with MakeDATA.bas"
  113. PRINT #2, "' back to disk and so it rebuilds the original file."
  114. PRINT #2, "'"
  115. PRINT #2, "' After the writeback call, only use the returned realFile$ to access the"
  116. PRINT #2, "' written file. It's your given path, but with an maybe altered filename"
  117. PRINT #2, "' (number added) in order to avoid the overwriting of an already existing"
  118. PRINT #2, "' file with the same name in the given location."
  119. PRINT #2, "'----------"
  120. PRINT #2, "' SYNTAX:"
  121. PRINT #2, "'   realFile$ = Write"; tarName$; "Data$ (wantFile$)"
  122. PRINT #2, "'----------"
  123. PRINT #2, "' INPUTS:"
  124. PRINT #2, "'   --- wantFile$ ---"
  125. PRINT #2, "'    The filename you would like to write the DATAs to, can contain"
  126. PRINT #2, "'    a full or relative path."
  127. PRINT #2, "'----------"
  128. PRINT #2, "' RESULT:"
  129. PRINT #2, "'   --- realFile$ ---"
  130. PRINT #2, "'    - On success this is the path and filename finally used after all"
  131. PRINT #2, "'      applied checks, use only this returned filename to access the"
  132. PRINT #2, "'      written file."
  133. PRINT #2, "'    - On failure this function will panic with the appropriate runtime"
  134. PRINT #2, "'      error code which you may trap and handle as needed with your own"
  135. PRINT #2, "'      ON ERROR GOTO... handler."
  136. PRINT #2, "'---------------------------------------------------------------------"
  137. PRINT #2, "FUNCTION Write"; tarName$; "Data$ (file$)"
  138. PRINT #2, "'--- option _explicit requirements ---"
  139. PRINT #2, "DIM po%, body$, ext$, num%, numL&, numB&, rawdata$, stroffs&, i&, dat&, ff%";
  140. IF packed% THEN PRINT #2, ", filedata$": ELSE PRINT #2, ""
  141. PRINT #2, "'--- separate filename body & extension ---"
  142. PRINT #2, "FOR po% = LEN(file$) TO 1 STEP -1"
  143. PRINT #2, "    IF MID$(file$, po%, 1) = "; CHR$(34); "."; CHR$(34); " THEN"
  144. PRINT #2, "        body$ = LEFT$(file$, po% - 1)"
  145. PRINT #2, "        ext$ = MID$(file$, po%)"
  146. PRINT #2, "        EXIT FOR"
  147. PRINT #2, "    ELSEIF MID$(file$, po%, 1) = "; CHR$(34); "\"; CHR$(34); " OR MID$(file$, po%, 1) = "; CHR$(34); "/"; CHR$(34); " OR po% = 1 THEN"
  148. PRINT #2, "        body$ = file$"
  149. PRINT #2, "        ext$ = "; CHR$(34); CHR$(34)
  150. PRINT #2, "        EXIT FOR"
  151. PRINT #2, "    END IF"
  152. PRINT #2, "NEXT po%"
  153. PRINT #2, "'--- avoid overwriting of existing files ---"
  154. PRINT #2, "num% = 1"
  155. PRINT #2, "WHILE _FILEEXISTS(file$)"
  156. PRINT #2, "    file$ = body$ + "; CHR$(34); "("; CHR$(34); " + LTRIM$(STR$(num%)) + "; CHR$(34); ")"; CHR$(34); " + ext$"
  157. PRINT #2, "    num% = num% + 1"
  158. PRINT #2, "WEND"
  159. PRINT #2, "'--- write DATAs ---"
  160. PRINT #2, "RESTORE "; tarName$
  161. PRINT #2, "READ numL&, numB&"
  162. PRINT #2, "rawdata$ = SPACE$((numL& * 4) + numB&)"
  163. PRINT #2, "stroffs& = 1"
  164. PRINT #2, "FOR i& = 1 TO numL&"
  165. PRINT #2, "    READ dat&"
  166. PRINT #2, "    MID$(rawdata$, stroffs&, 4) = MKL$(dat&)"
  167. PRINT #2, "    stroffs& = stroffs& + 4"
  168. PRINT #2, "NEXT i&"
  169. PRINT #2, "IF numB& > 0 THEN"
  170. PRINT #2, "    FOR i& = 1 TO numB&"
  171. PRINT #2, "        READ dat&"
  172. PRINT #2, "        MID$(rawdata$, stroffs&, 1) = CHR$(dat&)"
  173. PRINT #2, "        stroffs& = stroffs& + 1"
  174. PRINT #2, "    NEXT i&"
  175. PRINT #2, "END IF"
  176. PRINT #2, "ff% = FREEFILE"
  177. PRINT #2, "OPEN file$ FOR OUTPUT AS ff%"
  178. IF packed% THEN
  179.     PRINT #2, "CLOSE ff%"
  180.     PRINT #2, "filedata$ = LzwUnpack$(rawdata$)"
  181.     PRINT #2, "OPEN file$ FOR BINARY AS ff%"
  182.     PRINT #2, "PUT #ff%, , filedata$"
  183.     PRINT #2, "PRINT #ff%, rawdata$;"
  184. PRINT #2, "CLOSE ff%"
  185. PRINT #2, "'--- set result ---"
  186. PRINT #2, "Write"; tarName$; "Data$ = file$"
  187. PRINT #2, "EXIT FUNCTION"
  188. PRINT #2, ""
  189. PRINT #2, "'--- DATAs representing the contents of file "; srcName$
  190. PRINT #2, "'---------------------------------------------------------------------"
  191. PRINT #2, tarName$; ":"
  192. '--- read LONGs ---
  193. PRINT #2, "DATA "; LTRIM$(STR$(cntL& * 8)); ","; LTRIM$(STR$(cntB&))
  194. tmpI$ = SPACE$(32)
  195. FOR z& = 1 TO cntL&
  196.     GET #1, , tmpI$: offI% = 1
  197.     tmpO$ = "DATA " + STRING$(87, ","): offO% = 6
  198.     DO
  199.         tmpL& = CVL(MID$(tmpI$, offI%, 4)): offI% = offI% + 4
  200.         MID$(tmpO$, offO%, 10) = "&H" + RIGHT$("00000000" + HEX$(tmpL&), 8)
  201.         offO% = offO% + 11
  202.     LOOP UNTIL offO% > 92
  203.     PRINT #2, tmpO$
  204. NEXT z&
  205. '--- read remaining BYTEs ---
  206. IF cntB& > 0 THEN
  207.     PRINT #2, "DATA ";
  208.     FOR x% = 1 TO cntB&
  209.         GET #1, , tmpB%%
  210.         PRINT #2, "&H" + RIGHT$("00" + HEX$(tmpB%%), 2);
  211.         IF x% <> 16 THEN
  212.             IF x% <> cntB& THEN PRINT #2, ",";
  213.         ELSE
  214.             IF x% <> cntB& THEN
  215.                 PRINT #2, ""
  216.                 PRINT #2, "DATA ";
  217.             END IF
  218.         END IF
  219.     NEXT x%
  220.     PRINT #2, ""
  221. PRINT #2, "END FUNCTION"
  222. PRINT #2, ""
  223. '--- ending ---
  224.  
  225. '--- finish message ---
  226. COLOR 10: PRINT: PRINT "file successfully processed..."
  227. COLOR 9: PRINT: PRINT "You must $INCLUDE the created file (target name + .bm extension) at"
  228. PRINT "the end of your program and call the function 'Write"; tarName$; "Data$(...)'"
  229. PRINT "in an appropriate place to write the file back to disk."
  230. IF packed% THEN
  231.     COLOR 12: PRINT: PRINT "Your program must also $INCLUDE 'lzwpacker.bm' available from"
  232.     PRINT "the Libraries Collection here:"
  233.     PRINT "     https://www.qb64.org/forum/index.php?topic=809"
  234.     PRINT "to be able to write back the just processed file."
  235.     KILL tarPath$ + tar$ + ".lzw"
  236. done:
  237. '--- error handler ---
  238. abort:
  239. COLOR 12: PRINT: PRINT "something is wrong with path/file access, check your inputs and try again..."
  240. RESUME done
  241.  
  242. '$INCLUDE: 'QB64Library\LZW-Compress\lzwpacker.bm'
  243.  
  244.  

And here comes the second tool MakeCARR. It will do the whole thing in an array on C/C++ level, rather then in DATAs on the BASIC level. Although it's handling is a bit more tricky, as you get not only a .bm file, but also a .h file, and both must match (ie. the DECLARE LIBRARY path in the .bm must point to the .h), this approch has several advantages especially for big files:
  • Unless DATAs, which are included in the final EXE as written (ie. as ASCII chars), a C-Array containing numbers is embedded as (you guess) array of binary numbers, hence even uncompressed it would not take more space than the original file. This makes the compression even more valuable, as it really reduces the final EXE size, instead of just compensating the Number-to-Ascii bloat only as for the DATAs.
  • As the array is stored as successive numbers in memory, it's possible to write back the entire array with just one disk access, which is much faster than reading all single DATAs and concatenate them into one big string, which is then written out.
  • As the converted data is not in the included .bm file anymore (but in the .h file now), the syntax checking/compiling in the IDE will need less, depending on filesize much less time to finish, as it doesn't need to check 100s (or even 1000s) of DATA lines.
  • On C/C++ level it's easy to expand the given writeback path/filename into a full qualified absolut path using a standard library call. This path/filename is returned through the writeback function and can be used in your program to always safely access the written file, doesn't matter how often you change the current working directory using the CHDIR statement.
  • For any files, which are only needed temporarily during program runtime you can specify an auto-cleanup, which automatically deletes the written file again, as soon as your program terminates. This feature is also easily accessible on C/C++ level through an 'atexit()' function.


Again, if you're using at least QB64 v1.4 and don't wanna use the extra Lzw packer libarary, but the QB64 built-in zlib compression instead, then simply substitute the LzwPack$ call (line 98) with the respective _DEFLATE$ call and the LzwUnpack$ call (line 312) with an _INFLATE$ call. Also delete the $INCLUDE line at the end.

MakeCARR.bas
Code: QB64: [Select]
  1. '+---------------+---------------------------------------------------+
  2. '| ###### ###### |     .--. .         .-.                            |
  3. '| ##  ## ##   # |     |   )|        (   ) o                         |
  4. '| ##  ##  ##    |     |--' |--. .-.  `-.  .  .-...--.--. .-.        |
  5. '| ######   ##   |     |  \ |  |(   )(   ) | (   ||  |  |(   )       |
  6. '| ##      ##    |     '   `'  `-`-'  `-'-' `-`-`|'  '  `-`-'`-      |
  7. '| ##     ##   # |                            ._.'                   |
  8. '| ##     ###### |  Sources & Documents placed in the Public Domain. |
  9. '+---------------+---------------------------------------------------+
  10. '|                                                                   |
  11. '| === MakeCARR.bas ===                                              |
  12. '|                                                                   |
  13. '| == Create a C/C++ array out of the given file, so you can embed   |
  14. '| == it in your program and write it back when needed.              |
  15. '|                                                                   |
  16. '| == Two files are created, the .h file, which contains the array(s)|
  17. '| == and some functions, and a respective .bm file which needs to   |
  18. '| == be $INCLUDEd with your program and does provide the FUNCTION   |
  19. '| == to write back the array(s) into any file. All used functions   |
  20. '| == are standard library calls, no API calls are involved, so the  |
  21. '| == writeback should work on all QB64 supported platforms.         |
  22. '|                                                                   |
  23. '| == Make sure to adjust the path for the .h file for your personal |
  24. '| == needs in the created .bm files (DECLARE LIBRARY), if required. |
  25. '| == You may specify default paths right below this header.         |
  26. '|                                                                   |
  27. '| == This program needs the 'lzwpacker.bm' file available from the  |
  28. '| == Libraries Collection here:                                     |
  29. '| ==      https://www.qb64.org/forum/index.php?topic=809            |
  30. '| == as it will try to pack the given file to keep the array(s) as  |
  31. '| == small as possible. If compression is successful, then your     |
  32. '| == program also must $INCLUDE 'lzwpacker.bm' to be able to unpack |
  33. '| == the file data again for write back. MakeCARR.bas is printing   |
  34. '| == a reminder message in such a case.                             |
  35. '|                                                                   |
  36. '+-------------------------------------------------------------------+
  37. '| Done by RhoSigma, R.Heyder, provided AS IS, use at your own risk. |
  38. '| Find me in the QB64 Forum or mail to support&rhosigma-cw.net for  |
  39. '| any questions or suggestions. Thanx for your interest in my work. |
  40. '+-------------------------------------------------------------------+
  41.  
  42. '--- if you wish, set any default paths, end with a backslash ---
  43. srcPath$ = "" 'source path
  44. tarPath$ = "" 'target path
  45. '-----
  46. IF srcPath$ <> "" THEN
  47.     COLOR 15: PRINT "Default source path: ": COLOR 7: PRINT srcPath$: PRINT
  48. IF tarPath$ <> "" THEN
  49.     COLOR 15: PRINT "Default target path: ": COLOR 7: PRINT tarPath$: PRINT
  50.  
  51. '--- collect inputs (relative paths allowed, based on default paths) ---
  52. source:
  53. LINE INPUT "Source Filename: "; src$ 'any file you want to put into a C/C++ array
  54. IF src$ = "" GOTO source
  55. target:
  56. LINE INPUT "Target Basename: "; tar$ 'write stuff into this file(s) (.h/.bm is added)
  57. IF tar$ = "" GOTO target
  58. '-----
  59. ON ERROR GOTO abort
  60. OPEN "I", #1, srcPath$ + src$: CLOSE #1 'file exist check
  61. OPEN "O", #2, tarPath$ + tar$ + ".bm": CLOSE #2 'path exist check
  62.  
  63. '--- separate source filename part ---
  64. FOR po% = LEN(src$) TO 1 STEP -1
  65.     IF MID$(src$, po%, 1) = "\" OR MID$(src$, po%, 1) = "/" THEN
  66.         srcName$ = MID$(src$, po% + 1)
  67.         EXIT FOR
  68.     ELSEIF po% = 1 THEN
  69.         srcName$ = src$
  70.     END IF
  71. NEXT po%
  72. '--- separate target filename part ---
  73. FOR po% = LEN(tar$) TO 1 STEP -1
  74.     IF MID$(tar$, po%, 1) = "\" OR MID$(tar$, po%, 1) = "/" THEN
  75.         tarName$ = MID$(tar$, po% + 1)
  76.         EXIT FOR
  77.     ELSEIF po% = 1 THEN
  78.         tarName$ = tar$
  79.     END IF
  80. NEXT po%
  81. MID$(tarName$, 1, 1) = UCASE$(MID$(tarName$, 1, 1)) 'capitalize 1st letter
  82.  
  83. '---------------------------------------------------------------------
  84. ' Depending on the source file's size, one or more array(s) are
  85. ' created. This is because some C/C++ compilers seem to have problems
  86. ' with arrays with more than 65535 elements. This does not affect the
  87. ' write back, as the write function will take this behavior into account.
  88. '---------------------------------------------------------------------
  89.  
  90. '--- init ---
  91. OPEN "B", #1, srcPath$ + src$
  92. filedata$ = SPACE$(LOF(1))
  93. GET #1, , filedata$
  94. rawdata$ = LzwPack$(filedata$, 20)
  95. IF rawdata$ <> "" THEN
  96.     OPEN "O", #1, tarPath$ + tar$ + ".lzw"
  97.     CLOSE #1
  98.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  99.     PUT #1, , rawdata$
  100.     CLOSE #1
  101.     packed% = -1
  102.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  103.     packed% = 0
  104.     OPEN "B", #1, srcPath$ + src$
  105. fl& = LOF(1)
  106. cntL& = INT(fl& / 32)
  107. cntV& = INT(cntL& / 8180)
  108. cntB& = (fl& - (cntL& * 32))
  109.  
  110. '--- .h include file ---
  111. OPEN "O", #2, tarPath$ + tar$ + ".h"
  112. PRINT #2, "// ============================================================"
  113. PRINT #2, "// === This file was created with MakeCARR.bas by RhoSigma, ==="
  114. PRINT #2, "// === use it in conjunction with its respective .bm file.  ==="
  115. PRINT #2, "// ============================================================"
  116. PRINT #2, ""
  117. PRINT #2, "// --- Array(s) representing the contents of file "; srcName$
  118. PRINT #2, "// ---------------------------------------------------------------------"
  119. '--- read LONGs ---
  120. tmpI$ = SPACE$(32)
  121. FOR vc& = 0 TO cntV&
  122.     IF vc& = cntV& THEN numL& = (cntL& MOD 8180): ELSE numL& = 8180
  123.     PRINT #2, "static const unsigned int32 "; tarName$; "L"; LTRIM$(STR$(vc&)); "[] = {"
  124.     PRINT #2, "    "; LTRIM$(STR$(numL& * 8)); ","
  125.     FOR z& = 1 TO numL&
  126.         GET #1, , tmpI$: offI% = 1
  127.         tmpO$ = "    " + STRING$(88, ","): offO% = 5
  128.         DO
  129.             tmpL& = CVL(MID$(tmpI$, offI%, 4)): offI% = offI% + 4
  130.             MID$(tmpO$, offO%, 10) = "0x" + RIGHT$("00000000" + HEX$(tmpL&), 8)
  131.             offO% = offO% + 11
  132.         LOOP UNTIL offO% > 92
  133.         IF z& < numL& THEN PRINT #2, tmpO$: ELSE PRINT #2, LEFT$(tmpO$, 91)
  134.     NEXT z&
  135.     PRINT #2, "};"
  136.     PRINT #2, ""
  137. NEXT vc&
  138. '--- read remaining BYTEs ---
  139. IF cntB& > 0 THEN
  140.     PRINT #2, "static const unsigned int8 "; tarName$; "B[] = {"
  141.     PRINT #2, "    "; LTRIM$(STR$(cntB&)); ","
  142.     PRINT #2, "    ";
  143.     FOR x% = 1 TO cntB&
  144.         GET #1, , tmpB%%
  145.         PRINT #2, "0x" + RIGHT$("00" + HEX$(tmpB%%), 2);
  146.         IF x% <> 16 THEN
  147.             IF x% <> cntB& THEN PRINT #2, ",";
  148.         ELSE
  149.             IF x% <> cntB& THEN
  150.                 PRINT #2, ","
  151.                 PRINT #2, "    ";
  152.             END IF
  153.         END IF
  154.     NEXT x%
  155.     PRINT #2, ""
  156.     PRINT #2, "};"
  157.     PRINT #2, ""
  158. '--- some functions ---
  159. PRINT #2, "// --- Saved full qualified output path and filename, so we've no troubles"
  160. PRINT #2, "// --- when cleaning up, even if the current working folder was changed"
  161. PRINT #2, "// --- during program runtime."
  162. PRINT #2, "// ---------------------------------------------------------------------"
  163. PRINT #2, "char "; tarName$; "Name[8192]; // it's a safe size for any current OS"
  164. PRINT #2, ""
  165. PRINT #2, "// --- Cleanup function to delete the written file, called by the atexit()"
  166. PRINT #2, "// --- handler at program termination time, if requested by user."
  167. PRINT #2, "// ---------------------------------------------------------------------"
  168. PRINT #2, "void Kill"; tarName$; "Data(void)"
  169. PRINT #2, "{"
  170. PRINT #2, "    remove("; tarName$; "Name);"
  171. PRINT #2, "}"
  172. PRINT #2, ""
  173. PRINT #2, "// --- Function to write the array(s) back into a file, will return the"
  174. PRINT #2, "// --- full qualified output path and filename on success, otherwise an"
  175. PRINT #2, "// --- empty string is returned (access/write errors, file truncated)."
  176. PRINT #2, "// ---------------------------------------------------------------------"
  177. PRINT #2, "const char *Write"; tarName$; "Data(const char *FileName, int16 AutoClean)"
  178. PRINT #2, "{"
  179. PRINT #2, "    FILE *han = NULL; // file handle"
  180. PRINT #2, "    int32 num = NULL; // written elements"
  181. PRINT #2, ""
  182. PRINT #2, "    #ifdef QB64_WINDOWS"
  183. PRINT #2, "    if (!_fullpath("; tarName$; "Name, FileName, 8192)) return "; CHR$(34); CHR$(34); ";"
  184. PRINT #2, "    #else"
  185. PRINT #2, "    if (!realpath(FileName, "; tarName$; "Name)) return "; CHR$(34); CHR$(34); ";"
  186. PRINT #2, "    #endif"
  187. PRINT #2, ""
  188. PRINT #2, "    if (!(han = fopen("; tarName$; "Name, "; CHR$(34); "wb"; CHR$(34); "))) return "; CHR$(34); CHR$(34); ";"
  189. PRINT #2, "    if (AutoClean) atexit(Kill"; tarName$; "Data);"
  190. PRINT #2, ""
  191. FOR vc& = 0 TO cntV&
  192.     PRINT #2, "    num = fwrite(&"; tarName$; "L"; LTRIM$(STR$(vc&)); "[1], 4, "; tarName$; "L"; LTRIM$(STR$(vc&)); "[0], han);"
  193.     PRINT #2, "    if (num != "; tarName$; "L"; LTRIM$(STR$(vc&)); "[0]) {fclose(han); return "; CHR$(34); CHR$(34); ";}"
  194.     PRINT #2, ""
  195. NEXT vc&
  196. IF cntB& > 0 THEN
  197.     PRINT #2, "    num = fwrite(&"; tarName$; "B[1], 1, "; tarName$; "B[0], han);"
  198.     PRINT #2, "    if (num != "; tarName$; "B[0]) {fclose(han); return "; CHR$(34); CHR$(34); ";}"
  199.     PRINT #2, ""
  200. PRINT #2, "    fclose(han);"
  201. PRINT #2, "    return "; tarName$; "Name;"
  202. PRINT #2, "}"
  203. PRINT #2, ""
  204. '--- ending ---
  205.  
  206. '--- .bm include file ---
  207. OPEN "O", #2, tarPath$ + tar$ + ".bm"
  208. PRINT #2, "'============================================================"
  209. PRINT #2, "'=== This file was created with MakeCARR.bas by RhoSigma, ==="
  210. PRINT #2, "'=== you must $INCLUDE this at the end of your program.   ==="
  211. IF packed% THEN
  212.     PRINT #2, "'=== ---------------------------------------------------- ==="
  213.     PRINT #2, "'=== If your program is NOT a GuiTools based application, ==="
  214.     PRINT #2, "'=== then it must also $INCLUDE: 'lzwpacker.bm' available ==="
  215.     PRINT #2, "'=== from the Libraries Collection here:                  ==="
  216.     PRINT #2, "'===    https://www.qb64.org/forum/index.php?topic=809    ==="
  217. PRINT #2, "'============================================================"
  218. PRINT #2, ""
  219. PRINT #2, "'-----------------"
  220. PRINT #2, "'--- Important ---"
  221. PRINT #2, "'-----------------"
  222. PRINT #2, "' If you need to move around this .bm file and its respective .h file"
  223. PRINT #2, "' to fit in your project, then make sure the path in the DECLARE LIBRARY"
  224. PRINT #2, "' statement below does match the actual .h file location. It's best to"
  225. PRINT #2, "' specify a relative path assuming your QB64 installation folder as root."
  226. PRINT #2, "'---------------------------------------------------------------------"
  227. PRINT #2, ""
  228. '--- writeback function ---
  229. PRINT #2, "'"; STRING$(LEN(tarName$) + 19, "-")
  230. PRINT #2, "'--- Write"; tarName$; "Array$ ---"
  231. PRINT #2, "'"; STRING$(LEN(tarName$) + 19, "-")
  232. PRINT #2, "' This function will write the array(s) you've created with MakeCARR.bas"
  233. PRINT #2, "' back to disk and so it rebuilds the original file."
  234. PRINT #2, "'"
  235. PRINT #2, "' After the writeback call, only use the returned realFile$ to access the"
  236. PRINT #2, "' written file. It's the full qualified absolute path and filename, which"
  237. PRINT #2, "' is made by expanding your maybe given relative path and an maybe altered"
  238. PRINT #2, "' filename (number added) in order to avoid the overwriting of an already"
  239. PRINT #2, "' existing file with the same name in the given location. By this means"
  240. PRINT #2, "' you'll always have safe access to the file, no matter how your current"
  241. PRINT #2, "' working folder changes during runtime."
  242. PRINT #2, "'"
  243. PRINT #2, "' If you wish, the written file can automatically be deleted for you when"
  244. PRINT #2, "' your program will end, so you don't need to do the cleanup yourself."
  245. PRINT #2, "'----------"
  246. PRINT #2, "' SYNTAX:"
  247. PRINT #2, "'   realFile$ = Write"; tarName$; "Array$ (wantFile$, autoDel%)"
  248. PRINT #2, "'----------"
  249. PRINT #2, "' INPUTS:"
  250. PRINT #2, "'   --- wantFile$ ---"
  251. PRINT #2, "'    The filename you would like to write the array(s) to, can contain"
  252. PRINT #2, "'    a full or relative path."
  253. PRINT #2, "'   --- autoDel% ---"
  254. PRINT #2, "'    Shows whether you want the auto cleanup (see description above) at"
  255. PRINT #2, "'    the program end or not (-1 = delete file, 0 = don't delete file)."
  256. PRINT #2, "'----------"
  257. PRINT #2, "' RESULT:"
  258. PRINT #2, "'   --- realFile$ ---"
  259. PRINT #2, "'    - On success this is the full qualified path and filename finally"
  260. PRINT #2, "'      used after all applied checks, use only this returned filename"
  261. PRINT #2, "'      to access the written file."
  262. PRINT #2, "'    - On failure (write/access) this will be an empty string, so you"
  263. PRINT #2, "'      should check for this before trying to access/open the file."
  264. PRINT #2, "'---------------------------------------------------------------------"
  265. PRINT #2, "FUNCTION Write"; tarName$; "Array$ (file$, clean%)"
  266. PRINT #2, "'--- declare C/C++ function ---"
  267. PRINT #2, "DECLARE LIBRARY "; CHR$(34); tarPath$; tar$; CHR$(34); " 'Do not add .h here !!"
  268. PRINT #2, "    FUNCTION Write"; tarName$; "Data$ (FileName$, BYVAL AutoClean%)"
  269. PRINT #2, "END DECLARE"
  270. PRINT #2, "'--- option _explicit requirements ---"
  271. PRINT #2, "DIM po%, body$, ext$, num%";
  272. IF packed% THEN PRINT #2, ", real$, ff%, rawdata$, filedata$": ELSE PRINT #2, ""
  273. PRINT #2, "'--- separate filename body & extension ---"
  274. PRINT #2, "FOR po% = LEN(file$) TO 1 STEP -1"
  275. PRINT #2, "    IF MID$(file$, po%, 1) = "; CHR$(34); "."; CHR$(34); " THEN"
  276. PRINT #2, "        body$ = LEFT$(file$, po% - 1)"
  277. PRINT #2, "        ext$ = MID$(file$, po%)"
  278. PRINT #2, "        EXIT FOR"
  279. PRINT #2, "    ELSEIF MID$(file$, po%, 1) = "; CHR$(34); "\"; CHR$(34); " OR MID$(file$, po%, 1) = "; CHR$(34); "/"; CHR$(34); " OR po% = 1 THEN"
  280. PRINT #2, "        body$ = file$"
  281. PRINT #2, "        ext$ = "; CHR$(34); CHR$(34)
  282. PRINT #2, "        EXIT FOR"
  283. PRINT #2, "    END IF"
  284. PRINT #2, "NEXT po%"
  285. PRINT #2, "'--- avoid overwriting of existing files ---"
  286. PRINT #2, "num% = 1"
  287. PRINT #2, "WHILE _FILEEXISTS(file$)"
  288. PRINT #2, "    file$ = body$ + "; CHR$(34); "("; CHR$(34); " + LTRIM$(STR$(num%)) + "; CHR$(34); ")"; CHR$(34); " + ext$"
  289. PRINT #2, "    num% = num% + 1"
  290. PRINT #2, "WEND"
  291. PRINT #2, "'--- write array & set result ---"
  292. IF NOT packed% THEN
  293.     PRINT #2, "Write"; tarName$; "Array$ = Write"; tarName$; "Data$(file$ + CHR$(0), clean%)"
  294.     PRINT #2, "real$ = Write"; tarName$; "Data$(file$ + CHR$(0), clean%)"
  295.     PRINT #2, "IF real$ <> "; CHR$(34); CHR$(34); " THEN"
  296.     PRINT #2, "    ff% = FREEFILE"
  297.     PRINT #2, "    OPEN real$ FOR BINARY AS ff%"
  298.     PRINT #2, "    rawdata$ = SPACE$(LOF(ff%))"
  299.     PRINT #2, "    GET #ff%, , rawdata$"
  300.     PRINT #2, "    filedata$ = LzwUnpack$(rawdata$)"
  301.     PRINT #2, "    PUT #ff%, 1, filedata$"
  302.     PRINT #2, "    CLOSE ff%"
  303.     PRINT #2, "END IF"
  304.     PRINT #2, "Write"; tarName$; "Array$ = real$"
  305. PRINT #2, "END FUNCTION"
  306. PRINT #2, ""
  307. '--- ending ---
  308.  
  309. '--- finish message ---
  310. COLOR 10: PRINT: PRINT "file successfully processed..."
  311. COLOR 9: PRINT: PRINT "You must $INCLUDE the created file (target name + .bm extension) at"
  312. PRINT "the end of your program and call the function 'Write"; tarName$; "Array$(...)'"
  313. PRINT "in an appropriate place to write the file back to disk."
  314. IF packed% THEN
  315.     COLOR 12: PRINT: PRINT "Your program must also $INCLUDE 'lzwpacker.bm' available from"
  316.     PRINT "the Libraries Collection here:"
  317.     PRINT "     https://www.qb64.org/forum/index.php?topic=809"
  318.     PRINT "to be able to write back the just processed file."
  319.     KILL tarPath$ + tar$ + ".lzw"
  320. done:
  321. '--- error handler ---
  322. abort:
  323. COLOR 12: PRINT: PRINT "something is wrong with path/file access, check your inputs and try again..."
  324. RESUME done
  325.  
  326. '$INCLUDE: 'QB64Library\LZW-Compress\lzwpacker.bm'
  327.  
  328.  
« Last Edit: April 13, 2022, 04:50:23 pm by RhoSigma »
My Projects:   https://qb64forum.alephc.xyz/index.php?topic=809
GuiTools - A graphic UI framework (can do multiple UI forms/windows in one program)
Libraries - ImageProcess, StringBuffers (virt. files), MD5/SHA2-Hash, LZW etc.
Bonus - Blankers, QB64/Notepad++ setup pack

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Embedding files in programs (FileToDATA convertor)
« Reply #1 on: November 17, 2018, 06:32:03 pm »
I've did something similar to this by converting the file to hex values and then storing those as DATA statements in QB64 and then back again.  It doubles file size (1 ASCII byte is stored as 2 hex-values), so I never use it to embed anything except small resource files. 

I'm not at the PC right now, so I'm just curious:  How does the embedded file size here compare to the original usually?  Larger?  Smaller?  The same?

I'll personally dig into this better when I have a little more free time.  Embedded files can make project distributation a much simpler process in many cases.  :)
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

FellippeHeitor

  • Guest
Re: Embedding files in programs (FileToDATA convertor)
« Reply #2 on: November 17, 2018, 06:48:56 pm »
Sounds like a great idea for an installer. Thanks for sharing, Rho.

Offline RhoSigma

  • QB64 Developer
  • Forum Resident
  • Posts: 565
    • View Profile
Re: Embedding files in programs (FileToDATA convertor)
« Reply #3 on: November 17, 2018, 06:52:40 pm »
I've did something similar to this by converting the file to hex values and then storing those as DATA statements in QB64 and then back again.  It doubles file size (1 ASCII byte is stored as 2 hex-values), so I never use it to embed anything except small resource files. 

I'm not at the PC right now, so I'm just curious:  How does the embedded file size here compare to the original usually?  Larger?  Smaller?  The same?

I'll personally dig into this better when I have a little more free time.  Embedded files can make project distributation a much simpler process in many cases.  :)

Same process, I use hex values too, as it simply looks better in formatted DATA lines :)
As to the size now it uses LZW compression, I'd say it depends on the reached ratio. I've some config files here (approx. 5KB each) which compress pretty good between 65-80% and they come out pretty much the same size as DATAs as they are in the original size.

Sounds like a great idea for an installer. Thanks for sharing, Rho.

Well, that's a thing I wasn't thinking about at all, but you're absolutly right, nice idea.
My Projects:   https://qb64forum.alephc.xyz/index.php?topic=809
GuiTools - A graphic UI framework (can do multiple UI forms/windows in one program)
Libraries - ImageProcess, StringBuffers (virt. files), MD5/SHA2-Hash, LZW etc.
Bonus - Blankers, QB64/Notepad++ setup pack

Offline RhoSigma

  • QB64 Developer
  • Forum Resident
  • Posts: 565
    • View Profile
Re: Embedding files in programs (FileToDATA convertor)
« Reply #4 on: November 17, 2018, 09:42:24 pm »
So folks, one more on top,
MakeDATA.bas got a Big Brother in the initial post above called MakeCARR.bas.
My Projects:   https://qb64forum.alephc.xyz/index.php?topic=809
GuiTools - A graphic UI framework (can do multiple UI forms/windows in one program)
Libraries - ImageProcess, StringBuffers (virt. files), MD5/SHA2-Hash, LZW etc.
Bonus - Blankers, QB64/Notepad++ setup pack

Offline Pete

  • Forum Resident
  • Posts: 2361
  • Cuz I sez so, varmint!
    • View Profile
Re: Embedding files in programs (FileToDATA convertor)
« Reply #5 on: November 17, 2018, 11:10:30 pm »
This reminds me of some work I did in early 2000's to set up my office programs on several computers. That installer was about 2300 lines of code. I can't find the prototype I designed that created a single exe that would run and extract itself into 20+ exe programs. At least that's how I recall it working. A nice advancement to QB64 was the vast increase in memory. All of those programs were condensed into one. That made set up a snap.

Pete
Want to learn how to write code on cave walls? https://www.tapatalk.com/groups/qbasic/qbasic-f1/

Offline Petr

  • Forum Resident
  • Posts: 1720
  • The best code is the DNA of the hops.
    • View Profile
Re: Embedding files in programs (FileToDATA convertor)
« Reply #6 on: November 18, 2018, 03:03:21 am »
Thank you very much for sharing your program, RhoSigma. LZW compression is also used in GIF format if I do not mistake. It is definitely a very useful thing. Thanks a lot.

Offline Pete

  • Forum Resident
  • Posts: 2361
  • Cuz I sez so, varmint!
    • View Profile
Re: Embedding files in programs (FileToDATA convertor)
« Reply #7 on: November 18, 2018, 02:42:43 pm »
This would have been great when I needed a lot of data on a 1.44 MB floppy disk. I'm glad those days are gone. When I made my winzip clone, I just linked up many exe files but, of course, that isn't data compression.

With storage so vastly expanded, well, I'm just going to flat out ask: How important is data compression now?

Curious. Did you come up with this algorithm by thinking your way to a solution, trial and error method, or was this something you studied and implemented?

Pete
Want to learn how to write code on cave walls? https://www.tapatalk.com/groups/qbasic/qbasic-f1/

Offline RhoSigma

  • QB64 Developer
  • Forum Resident
  • Posts: 565
    • View Profile
Re: Embedding files in programs (FileToDATA convertor)
« Reply #8 on: November 18, 2018, 02:50:17 pm »
How important? - Very important for embedded files, less size = less compile time = less turnaround times when testing and debugging.

Well, first I was searching the internet for something ready to use in QB64, but as mentioned, most examples were very rudimentary or hard to follow, so it's a rewrite especially with QB64 in mind.
My Projects:   https://qb64forum.alephc.xyz/index.php?topic=809
GuiTools - A graphic UI framework (can do multiple UI forms/windows in one program)
Libraries - ImageProcess, StringBuffers (virt. files), MD5/SHA2-Hash, LZW etc.
Bonus - Blankers, QB64/Notepad++ setup pack

Offline Pete

  • Forum Resident
  • Posts: 2361
  • Cuz I sez so, varmint!
    • View Profile
Re: Embedding files in programs (FileToDATA convertor)
« Reply #9 on: November 18, 2018, 03:47:06 pm »
I didn't think in terms of debugging. I guess that's important to coders, unlike me, who make mistakes. :D

Rob got QB64 compilation times consistently better from days of old. I think he tweaked the process so minimal changes in the code could be handled faster, too. Anyway, I used to wait for 30 - 40 minutes 10 ears ago to compile an 80,000 line program. That's probably cut to about 3 minutes now in the latest GL version. Still, 90-seconds would beat that over time. I recall needing 8 or more tries to get a particularly hard bug situation solved. Back then, that would take most of the day (with time needed to recode, included.) Today if compressing data could save 15 - 20 minutes for every hour of debugging, yes, I agree, that would be huge!

Pete
Want to learn how to write code on cave walls? https://www.tapatalk.com/groups/qbasic/qbasic-f1/

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Embedding files in programs (FileToDATA convertor)
« Reply #10 on: November 18, 2018, 04:03:55 pm »
How important? - Very important for embedded files, less size = less compile time = less turnaround times when testing and debugging.

Well, first I was searching the internet for something ready to use in QB64, but as mentioned, most examples were very rudimentary or hard to follow, so it's a rewrite especially with QB64 in mind.

What I'd love to be able to do is something like:

OPEN "Data:LabelX" FOR INPUT AS #1
DO UNTIL EOF(1)
    INPUT #1, whatever$
    PRINT whatever$
NEXT

Much like we could open either a filename$ or "SCRN" to swap between file and screen printing, it'd be nice to swap between internal and external data files.  Then you could easily test the code with external files, and save embedding the DATA until last, reducing compile times and IDE responsiveness/ease of navigation.
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline Pete

  • Forum Resident
  • Posts: 2361
  • Cuz I sez so, varmint!
    • View Profile
Re: Embedding files in programs (FileToDATA convertor)
« Reply #11 on: November 18, 2018, 04:19:57 pm »
@Steve: I did a lot of that when QB64 was new. Not so much with the data, as I almost always used external data due to space issues in QB45, but with subs, etc. that were not part of the bug I was tracing. I made a program that would unload selected subs, rem out the calls, and then run the program. That reduced the code form say 80,000 lines to maybe 10,000 lines, and those 45 minute waits were drastically reduced. Still, it was a PITA, just a less time consuming PITA.

Pete
Want to learn how to write code on cave walls? https://www.tapatalk.com/groups/qbasic/qbasic-f1/

Offline RhoSigma

  • QB64 Developer
  • Forum Resident
  • Posts: 565
    • View Profile
Re: Embedding files in programs (FileToDATA convertor)
« Reply #12 on: November 18, 2018, 06:49:04 pm »
What I'd love to be able to do is something like:

OPEN "Data:LabelX" FOR INPUT AS #1
DO UNTIL EOF(1)
    INPUT #1, whatever$
    PRINT whatever$
NEXT

Much like we could open either a filename$ or "SCRN" to swap between file and screen printing, it'd be nice to swap between internal and external data files.  Then you could easily test the code with external files, and save embedding the DATA until last, reducing compile times and IDE responsiveness/ease of navigation.

Cool idea, can we add that as feature request for version 1.3 or 1.4 please? - Would eliminate the need to write back embedded files in the first place.
« Last Edit: November 18, 2018, 06:50:57 pm by RhoSigma »
My Projects:   https://qb64forum.alephc.xyz/index.php?topic=809
GuiTools - A graphic UI framework (can do multiple UI forms/windows in one program)
Libraries - ImageProcess, StringBuffers (virt. files), MD5/SHA2-Hash, LZW etc.
Bonus - Blankers, QB64/Notepad++ setup pack

Offline Pete

  • Forum Resident
  • Posts: 2361
  • Cuz I sez so, varmint!
    • View Profile
Re: Embedding files in programs (FileToDATA convertor)
« Reply #13 on: November 18, 2018, 07:19:50 pm »
As long as we're brainstorming a bit here, the QB45 IDE has a LOAD / UNLOAD Module feature. Although QB64 did not reproduce QB45's multi-modular abilities, mostly because the memory improvements made that feature almost unnecessary, another IDE feature could be loading and unloading subs and functions. That not only helps with compiling time when debugging, it also cuts out a lot of clutter when you have to add to really large programs.

Pete
Want to learn how to write code on cave walls? https://www.tapatalk.com/groups/qbasic/qbasic-f1/

Offline RhoSigma

  • QB64 Developer
  • Forum Resident
  • Posts: 565
    • View Profile
Re: Embedding files in programs (FileToDATA convertor)
« Reply #14 on: November 19, 2018, 01:45:39 am »
... another IDE feature could be loading and unloading subs and functions ...

Well that's a thing, what I'd expect the used C/C++ compiler is doing for us automatically when linking all compiled stuff into the final EXE. At least in my old Commodore Amiga days, the linking process did leave out all functions, which never were called (or referenced in general) somewere in the code (the used IDE/Compiler there was  MaxonC++ 4.0). Doing so makes sense, as otherwise eg. static link libraries would be included in whole into the EXE, even if you only need 1 or 2 functions out of it. However, I don't know the exact behavior of g++ here, maybe an extra optimization switch may be required. Well all this is of course only affecting the EXE size, it won't change the C/C++ compile times, as first all sourcefiles are compilled and the function sort out will then happen while linking.
My Projects:   https://qb64forum.alephc.xyz/index.php?topic=809
GuiTools - A graphic UI framework (can do multiple UI forms/windows in one program)
Libraries - ImageProcess, StringBuffers (virt. files), MD5/SHA2-Hash, LZW etc.
Bonus - Blankers, QB64/Notepad++ setup pack