QB64.org Forum

Active Forums => Programs => Topic started by: RhoSigma on November 17, 2018, 06:13:23 pm

Title: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on November 17, 2018, 06:13:23 pm
The following two small programs are meant to convert any file (eg. Images, Sounds, Databases etc.) into an easy embedable format. Embedding is then as simple as putting a $INCLUDE line at the end of your program. The converted files will also have a "Ready to use" writeback function, which will rebuild the embedded file on disk when needed.

The first converter tool is named MakeDATA, as you may imagine by its name, it will convert the given file into a block of DATA lines. This is easy to use and absolutly BASIC only. It's best to embed small files like icons or sprites. However, for large files like fullsize digital photos or MP3 music it has a big backdraw at the compiling speed of your program and the final EXE size on one side and also to the writeback speed on the other side.

So for bigger files the second tool MakeCARR should be your choice, which its advantages are detailed below the MakeDATA codebox.

Quote
Note for Windows users:
The codeboxes below contain simple SCREEN 0 (text) based versions of the converter tools. You may have to go in to change your default paths, but they should work reliable on all OS's supported by QB64.
If you're on Windows, then you may rather wish to use my GuiTools based versions of it, just move on to
The GuiTools Framework: https://qb64forum.alephc.xyz/index.php?topic=809
Both converter tools are available as part of the QB64GuiTools.7z source archive.




And now for the simple (SCREEN 0) people:



Both of the following tools require the 'lzwpacker.bm' file available from my Libraries Collection here: https://qb64forum.alephc.xyz/index.php?topic=809

If you're using at least QB64 v1.4 and don't wanna use the extra Lzw packer libarary, but the QB64 built-in zlib compression instead, then simply substitute the LzwPack$ call (line 84) with the respective _DEFLATE$ call and the LzwUnpack$ call (line 187) with an _INFLATE$ call. Also delete the $INCLUDE line at the end.

MakeDATA.bas
Code: QB64: [Select]
  1. '+---------------+---------------------------------------------------+
  2. '| ###### ###### |     .--. .         .-.                            |
  3. '| ##  ## ##   # |     |   )|        (   ) o                         |
  4. '| ##  ##  ##    |     |--' |--. .-.  `-.  .  .-...--.--. .-.        |
  5. '| ######   ##   |     |  \ |  |(   )(   ) | (   ||  |  |(   )       |
  6. '| ##      ##    |     '   `'  `-`-'  `-'-' `-`-`|'  '  `-`-'`-      |
  7. '| ##     ##   # |                            ._.'                   |
  8. '| ##     ###### |  Sources & Documents placed in the Public Domain. |
  9. '+---------------+---------------------------------------------------+
  10. '|                                                                   |
  11. '| === MakeDATA.bas ===                                              |
  12. '|                                                                   |
  13. '| == Create a DATA block out of the given file, so you can embed it |
  14. '| == in your program and write it back when needed.                 |
  15. '|                                                                   |
  16. '| == The DATAs are written into a .bm file together with a ready to |
  17. '| == use write back FUNCTION. You just $INCLUDE this .bm file into  |
  18. '| == your program and call the write back FUNCTION somewhere.       |
  19. '|                                                                   |
  20. '| == This program needs the 'lzwpacker.bm' file available from the  |
  21. '| == Libraries Collection here:                                     |
  22. '| ==      https://www.qb64.org/forum/index.php?topic=809            |
  23. '| == as it will try to pack the given file to keep the DATA block   |
  24. '| == as small as possible. If compression is successful, then your  |
  25. '| == program also must $INCLUDE 'lzwpacker.bm' to be able to unpack |
  26. '| == the file data again for write back. MakeDATA.bas is printing   |
  27. '| == a reminder message in such a case.                             |
  28. '|                                                                   |
  29. '+-------------------------------------------------------------------+
  30. '| Done by RhoSigma, R.Heyder, provided AS IS, use at your own risk. |
  31. '| Find me in the QB64 Forum or mail to support&rhosigma-cw.net for  |
  32. '| any questions or suggestions. Thanx for your interest in my work. |
  33. '+-------------------------------------------------------------------+
  34.  
  35. '--- if you wish, set any default paths, end with a backslash ---
  36. srcPath$ = "" 'source path
  37. tarPath$ = "" 'target path
  38. '-----
  39. IF srcPath$ <> "" THEN
  40.     COLOR 15: PRINT "Default source path: ": COLOR 7: PRINT srcPath$: PRINT
  41. IF tarPath$ <> "" THEN
  42.     COLOR 15: PRINT "Default target path: ": COLOR 7: PRINT tarPath$: PRINT
  43.  
  44. '--- collect inputs (relative paths allowed, based on default paths) ---
  45. source:
  46. LINE INPUT "Source Filename: "; src$ 'any file you want to put into DATAs
  47. IF src$ = "" GOTO source
  48. target:
  49. LINE INPUT "Target Basename: "; tar$ 'write stuff into this file (.bm is added)
  50. IF tar$ = "" GOTO target
  51. '-----
  52. ON ERROR GOTO abort
  53. OPEN "I", #1, srcPath$ + src$: CLOSE #1 'file exist check
  54. OPEN "O", #2, tarPath$ + tar$ + ".bm": CLOSE #2 'path exist check
  55.  
  56. '--- separate source filename part ---
  57. FOR po% = LEN(src$) TO 1 STEP -1
  58.     IF MID$(src$, po%, 1) = "\" OR MID$(src$, po%, 1) = "/" THEN
  59.         srcName$ = MID$(src$, po% + 1)
  60.         EXIT FOR
  61.     ELSEIF po% = 1 THEN
  62.         srcName$ = src$
  63.     END IF
  64. NEXT po%
  65. '--- separate target filename part ---
  66. FOR po% = LEN(tar$) TO 1 STEP -1
  67.     IF MID$(tar$, po%, 1) = "\" OR MID$(tar$, po%, 1) = "/" THEN
  68.         tarName$ = MID$(tar$, po% + 1)
  69.         EXIT FOR
  70.     ELSEIF po% = 1 THEN
  71.         tarName$ = tar$
  72.     END IF
  73. NEXT po%
  74. MID$(tarName$, 1, 1) = UCASE$(MID$(tarName$, 1, 1)) 'capitalize 1st letter
  75.  
  76. '--- init ---
  77. OPEN "B", #1, srcPath$ + src$
  78. filedata$ = SPACE$(LOF(1))
  79. GET #1, , filedata$
  80. rawdata$ = LzwPack$(filedata$, 20)
  81. IF rawdata$ <> "" THEN
  82.     OPEN "O", #1, tarPath$ + tar$ + ".lzw"
  83.     CLOSE #1
  84.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  85.     PUT #1, , rawdata$
  86.     CLOSE #1
  87.     packed% = -1
  88.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  89.     packed% = 0
  90.     OPEN "B", #1, srcPath$ + src$
  91. fl& = LOF(1)
  92. cntL& = INT(fl& / 32)
  93. cntB& = (fl& - (cntL& * 32))
  94.  
  95. '--- .bm include file ---
  96. OPEN "O", #2, tarPath$ + tar$ + ".bm"
  97. PRINT #2, "'============================================================"
  98. PRINT #2, "'=== This file was created with MakeDATA.bas by RhoSigma, ==="
  99. PRINT #2, "'=== you must $INCLUDE this at the end of your program.   ==="
  100. IF packed% THEN
  101.     PRINT #2, "'=== ---------------------------------------------------- ==="
  102.     PRINT #2, "'=== If your program is NOT a GuiTools based application, ==="
  103.     PRINT #2, "'=== then it must also $INCLUDE: 'lzwpacker.bm' available ==="
  104.     PRINT #2, "'=== from the Libraries Collection here:                  ==="
  105.     PRINT #2, "'===    https://www.qb64.org/forum/index.php?topic=809    ==="
  106. PRINT #2, "'============================================================"
  107. PRINT #2, ""
  108. '--- writeback function ---
  109. PRINT #2, "'"; STRING$(LEN(tarName$) + 18, "-")
  110. PRINT #2, "'--- Write"; tarName$; "Data$ ---"
  111. PRINT #2, "'"; STRING$(LEN(tarName$) + 18, "-")
  112. PRINT #2, "' This function will write the DATAs you've created with MakeDATA.bas"
  113. PRINT #2, "' back to disk and so it rebuilds the original file."
  114. PRINT #2, "'"
  115. PRINT #2, "' After the writeback call, only use the returned realFile$ to access the"
  116. PRINT #2, "' written file. It's your given path, but with an maybe altered filename"
  117. PRINT #2, "' (number added) in order to avoid the overwriting of an already existing"
  118. PRINT #2, "' file with the same name in the given location."
  119. PRINT #2, "'----------"
  120. PRINT #2, "' SYNTAX:"
  121. PRINT #2, "'   realFile$ = Write"; tarName$; "Data$ (wantFile$)"
  122. PRINT #2, "'----------"
  123. PRINT #2, "' INPUTS:"
  124. PRINT #2, "'   --- wantFile$ ---"
  125. PRINT #2, "'    The filename you would like to write the DATAs to, can contain"
  126. PRINT #2, "'    a full or relative path."
  127. PRINT #2, "'----------"
  128. PRINT #2, "' RESULT:"
  129. PRINT #2, "'   --- realFile$ ---"
  130. PRINT #2, "'    - On success this is the path and filename finally used after all"
  131. PRINT #2, "'      applied checks, use only this returned filename to access the"
  132. PRINT #2, "'      written file."
  133. PRINT #2, "'    - On failure this function will panic with the appropriate runtime"
  134. PRINT #2, "'      error code which you may trap and handle as needed with your own"
  135. PRINT #2, "'      ON ERROR GOTO... handler."
  136. PRINT #2, "'---------------------------------------------------------------------"
  137. PRINT #2, "FUNCTION Write"; tarName$; "Data$ (file$)"
  138. PRINT #2, "'--- option _explicit requirements ---"
  139. PRINT #2, "DIM po%, body$, ext$, num%, numL&, numB&, rawdata$, stroffs&, i&, dat&, ff%";
  140. IF packed% THEN PRINT #2, ", filedata$": ELSE PRINT #2, ""
  141. PRINT #2, "'--- separate filename body & extension ---"
  142. PRINT #2, "FOR po% = LEN(file$) TO 1 STEP -1"
  143. PRINT #2, "    IF MID$(file$, po%, 1) = "; CHR$(34); "."; CHR$(34); " THEN"
  144. PRINT #2, "        body$ = LEFT$(file$, po% - 1)"
  145. PRINT #2, "        ext$ = MID$(file$, po%)"
  146. PRINT #2, "        EXIT FOR"
  147. PRINT #2, "    ELSEIF MID$(file$, po%, 1) = "; CHR$(34); "\"; CHR$(34); " OR MID$(file$, po%, 1) = "; CHR$(34); "/"; CHR$(34); " OR po% = 1 THEN"
  148. PRINT #2, "        body$ = file$"
  149. PRINT #2, "        ext$ = "; CHR$(34); CHR$(34)
  150. PRINT #2, "        EXIT FOR"
  151. PRINT #2, "    END IF"
  152. PRINT #2, "NEXT po%"
  153. PRINT #2, "'--- avoid overwriting of existing files ---"
  154. PRINT #2, "num% = 1"
  155. PRINT #2, "WHILE _FILEEXISTS(file$)"
  156. PRINT #2, "    file$ = body$ + "; CHR$(34); "("; CHR$(34); " + LTRIM$(STR$(num%)) + "; CHR$(34); ")"; CHR$(34); " + ext$"
  157. PRINT #2, "    num% = num% + 1"
  158. PRINT #2, "WEND"
  159. PRINT #2, "'--- write DATAs ---"
  160. PRINT #2, "RESTORE "; tarName$
  161. PRINT #2, "READ numL&, numB&"
  162. PRINT #2, "rawdata$ = SPACE$((numL& * 4) + numB&)"
  163. PRINT #2, "stroffs& = 1"
  164. PRINT #2, "FOR i& = 1 TO numL&"
  165. PRINT #2, "    READ dat&"
  166. PRINT #2, "    MID$(rawdata$, stroffs&, 4) = MKL$(dat&)"
  167. PRINT #2, "    stroffs& = stroffs& + 4"
  168. PRINT #2, "NEXT i&"
  169. PRINT #2, "IF numB& > 0 THEN"
  170. PRINT #2, "    FOR i& = 1 TO numB&"
  171. PRINT #2, "        READ dat&"
  172. PRINT #2, "        MID$(rawdata$, stroffs&, 1) = CHR$(dat&)"
  173. PRINT #2, "        stroffs& = stroffs& + 1"
  174. PRINT #2, "    NEXT i&"
  175. PRINT #2, "END IF"
  176. PRINT #2, "ff% = FREEFILE"
  177. PRINT #2, "OPEN file$ FOR OUTPUT AS ff%"
  178. IF packed% THEN
  179.     PRINT #2, "CLOSE ff%"
  180.     PRINT #2, "filedata$ = LzwUnpack$(rawdata$)"
  181.     PRINT #2, "OPEN file$ FOR BINARY AS ff%"
  182.     PRINT #2, "PUT #ff%, , filedata$"
  183.     PRINT #2, "PRINT #ff%, rawdata$;"
  184. PRINT #2, "CLOSE ff%"
  185. PRINT #2, "'--- set result ---"
  186. PRINT #2, "Write"; tarName$; "Data$ = file$"
  187. PRINT #2, "EXIT FUNCTION"
  188. PRINT #2, ""
  189. PRINT #2, "'--- DATAs representing the contents of file "; srcName$
  190. PRINT #2, "'---------------------------------------------------------------------"
  191. PRINT #2, tarName$; ":"
  192. '--- read LONGs ---
  193. PRINT #2, "DATA "; LTRIM$(STR$(cntL& * 8)); ","; LTRIM$(STR$(cntB&))
  194. tmpI$ = SPACE$(32)
  195. FOR z& = 1 TO cntL&
  196.     GET #1, , tmpI$: offI% = 1
  197.     tmpO$ = "DATA " + STRING$(87, ","): offO% = 6
  198.     DO
  199.         tmpL& = CVL(MID$(tmpI$, offI%, 4)): offI% = offI% + 4
  200.         MID$(tmpO$, offO%, 10) = "&H" + RIGHT$("00000000" + HEX$(tmpL&), 8)
  201.         offO% = offO% + 11
  202.     LOOP UNTIL offO% > 92
  203.     PRINT #2, tmpO$
  204. NEXT z&
  205. '--- read remaining BYTEs ---
  206. IF cntB& > 0 THEN
  207.     PRINT #2, "DATA ";
  208.     FOR x% = 1 TO cntB&
  209.         GET #1, , tmpB%%
  210.         PRINT #2, "&H" + RIGHT$("00" + HEX$(tmpB%%), 2);
  211.         IF x% <> 16 THEN
  212.             IF x% <> cntB& THEN PRINT #2, ",";
  213.         ELSE
  214.             IF x% <> cntB& THEN
  215.                 PRINT #2, ""
  216.                 PRINT #2, "DATA ";
  217.             END IF
  218.         END IF
  219.     NEXT x%
  220.     PRINT #2, ""
  221. PRINT #2, "END FUNCTION"
  222. PRINT #2, ""
  223. '--- ending ---
  224.  
  225. '--- finish message ---
  226. COLOR 10: PRINT: PRINT "file successfully processed..."
  227. COLOR 9: PRINT: PRINT "You must $INCLUDE the created file (target name + .bm extension) at"
  228. PRINT "the end of your program and call the function 'Write"; tarName$; "Data$(...)'"
  229. PRINT "in an appropriate place to write the file back to disk."
  230. IF packed% THEN
  231.     COLOR 12: PRINT: PRINT "Your program must also $INCLUDE 'lzwpacker.bm' available from"
  232.     PRINT "the Libraries Collection here:"
  233.     PRINT "     https://www.qb64.org/forum/index.php?topic=809"
  234.     PRINT "to be able to write back the just processed file."
  235.     KILL tarPath$ + tar$ + ".lzw"
  236. done:
  237. '--- error handler ---
  238. abort:
  239. COLOR 12: PRINT: PRINT "something is wrong with path/file access, check your inputs and try again..."
  240. RESUME done
  241.  
  242. '$INCLUDE: 'QB64Library\LZW-Compress\lzwpacker.bm'
  243.  
  244.  

And here comes the second tool MakeCARR. It will do the whole thing in an array on C/C++ level, rather then in DATAs on the BASIC level. Although it's handling is a bit more tricky, as you get not only a .bm file, but also a .h file, and both must match (ie. the DECLARE LIBRARY path in the .bm must point to the .h), this approch has several advantages especially for big files:


Again, if you're using at least QB64 v1.4 and don't wanna use the extra Lzw packer libarary, but the QB64 built-in zlib compression instead, then simply substitute the LzwPack$ call (line 98) with the respective _DEFLATE$ call and the LzwUnpack$ call (line 312) with an _INFLATE$ call. Also delete the $INCLUDE line at the end.

MakeCARR.bas
Code: QB64: [Select]
  1. '+---------------+---------------------------------------------------+
  2. '| ###### ###### |     .--. .         .-.                            |
  3. '| ##  ## ##   # |     |   )|        (   ) o                         |
  4. '| ##  ##  ##    |     |--' |--. .-.  `-.  .  .-...--.--. .-.        |
  5. '| ######   ##   |     |  \ |  |(   )(   ) | (   ||  |  |(   )       |
  6. '| ##      ##    |     '   `'  `-`-'  `-'-' `-`-`|'  '  `-`-'`-      |
  7. '| ##     ##   # |                            ._.'                   |
  8. '| ##     ###### |  Sources & Documents placed in the Public Domain. |
  9. '+---------------+---------------------------------------------------+
  10. '|                                                                   |
  11. '| === MakeCARR.bas ===                                              |
  12. '|                                                                   |
  13. '| == Create a C/C++ array out of the given file, so you can embed   |
  14. '| == it in your program and write it back when needed.              |
  15. '|                                                                   |
  16. '| == Two files are created, the .h file, which contains the array(s)|
  17. '| == and some functions, and a respective .bm file which needs to   |
  18. '| == be $INCLUDEd with your program and does provide the FUNCTION   |
  19. '| == to write back the array(s) into any file. All used functions   |
  20. '| == are standard library calls, no API calls are involved, so the  |
  21. '| == writeback should work on all QB64 supported platforms.         |
  22. '|                                                                   |
  23. '| == Make sure to adjust the path for the .h file for your personal |
  24. '| == needs in the created .bm files (DECLARE LIBRARY), if required. |
  25. '| == You may specify default paths right below this header.         |
  26. '|                                                                   |
  27. '| == This program needs the 'lzwpacker.bm' file available from the  |
  28. '| == Libraries Collection here:                                     |
  29. '| ==      https://www.qb64.org/forum/index.php?topic=809            |
  30. '| == as it will try to pack the given file to keep the array(s) as  |
  31. '| == small as possible. If compression is successful, then your     |
  32. '| == program also must $INCLUDE 'lzwpacker.bm' to be able to unpack |
  33. '| == the file data again for write back. MakeCARR.bas is printing   |
  34. '| == a reminder message in such a case.                             |
  35. '|                                                                   |
  36. '+-------------------------------------------------------------------+
  37. '| Done by RhoSigma, R.Heyder, provided AS IS, use at your own risk. |
  38. '| Find me in the QB64 Forum or mail to support&rhosigma-cw.net for  |
  39. '| any questions or suggestions. Thanx for your interest in my work. |
  40. '+-------------------------------------------------------------------+
  41.  
  42. '--- if you wish, set any default paths, end with a backslash ---
  43. srcPath$ = "" 'source path
  44. tarPath$ = "" 'target path
  45. '-----
  46. IF srcPath$ <> "" THEN
  47.     COLOR 15: PRINT "Default source path: ": COLOR 7: PRINT srcPath$: PRINT
  48. IF tarPath$ <> "" THEN
  49.     COLOR 15: PRINT "Default target path: ": COLOR 7: PRINT tarPath$: PRINT
  50.  
  51. '--- collect inputs (relative paths allowed, based on default paths) ---
  52. source:
  53. LINE INPUT "Source Filename: "; src$ 'any file you want to put into a C/C++ array
  54. IF src$ = "" GOTO source
  55. target:
  56. LINE INPUT "Target Basename: "; tar$ 'write stuff into this file(s) (.h/.bm is added)
  57. IF tar$ = "" GOTO target
  58. '-----
  59. ON ERROR GOTO abort
  60. OPEN "I", #1, srcPath$ + src$: CLOSE #1 'file exist check
  61. OPEN "O", #2, tarPath$ + tar$ + ".bm": CLOSE #2 'path exist check
  62.  
  63. '--- separate source filename part ---
  64. FOR po% = LEN(src$) TO 1 STEP -1
  65.     IF MID$(src$, po%, 1) = "\" OR MID$(src$, po%, 1) = "/" THEN
  66.         srcName$ = MID$(src$, po% + 1)
  67.         EXIT FOR
  68.     ELSEIF po% = 1 THEN
  69.         srcName$ = src$
  70.     END IF
  71. NEXT po%
  72. '--- separate target filename part ---
  73. FOR po% = LEN(tar$) TO 1 STEP -1
  74.     IF MID$(tar$, po%, 1) = "\" OR MID$(tar$, po%, 1) = "/" THEN
  75.         tarName$ = MID$(tar$, po% + 1)
  76.         EXIT FOR
  77.     ELSEIF po% = 1 THEN
  78.         tarName$ = tar$
  79.     END IF
  80. NEXT po%
  81. MID$(tarName$, 1, 1) = UCASE$(MID$(tarName$, 1, 1)) 'capitalize 1st letter
  82.  
  83. '---------------------------------------------------------------------
  84. ' Depending on the source file's size, one or more array(s) are
  85. ' created. This is because some C/C++ compilers seem to have problems
  86. ' with arrays with more than 65535 elements. This does not affect the
  87. ' write back, as the write function will take this behavior into account.
  88. '---------------------------------------------------------------------
  89.  
  90. '--- init ---
  91. OPEN "B", #1, srcPath$ + src$
  92. filedata$ = SPACE$(LOF(1))
  93. GET #1, , filedata$
  94. rawdata$ = LzwPack$(filedata$, 20)
  95. IF rawdata$ <> "" THEN
  96.     OPEN "O", #1, tarPath$ + tar$ + ".lzw"
  97.     CLOSE #1
  98.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  99.     PUT #1, , rawdata$
  100.     CLOSE #1
  101.     packed% = -1
  102.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  103.     packed% = 0
  104.     OPEN "B", #1, srcPath$ + src$
  105. fl& = LOF(1)
  106. cntL& = INT(fl& / 32)
  107. cntV& = INT(cntL& / 8180)
  108. cntB& = (fl& - (cntL& * 32))
  109.  
  110. '--- .h include file ---
  111. OPEN "O", #2, tarPath$ + tar$ + ".h"
  112. PRINT #2, "// ============================================================"
  113. PRINT #2, "// === This file was created with MakeCARR.bas by RhoSigma, ==="
  114. PRINT #2, "// === use it in conjunction with its respective .bm file.  ==="
  115. PRINT #2, "// ============================================================"
  116. PRINT #2, ""
  117. PRINT #2, "// --- Array(s) representing the contents of file "; srcName$
  118. PRINT #2, "// ---------------------------------------------------------------------"
  119. '--- read LONGs ---
  120. tmpI$ = SPACE$(32)
  121. FOR vc& = 0 TO cntV&
  122.     IF vc& = cntV& THEN numL& = (cntL& MOD 8180): ELSE numL& = 8180
  123.     PRINT #2, "static const unsigned int32 "; tarName$; "L"; LTRIM$(STR$(vc&)); "[] = {"
  124.     PRINT #2, "    "; LTRIM$(STR$(numL& * 8)); ","
  125.     FOR z& = 1 TO numL&
  126.         GET #1, , tmpI$: offI% = 1
  127.         tmpO$ = "    " + STRING$(88, ","): offO% = 5
  128.         DO
  129.             tmpL& = CVL(MID$(tmpI$, offI%, 4)): offI% = offI% + 4
  130.             MID$(tmpO$, offO%, 10) = "0x" + RIGHT$("00000000" + HEX$(tmpL&), 8)
  131.             offO% = offO% + 11
  132.         LOOP UNTIL offO% > 92
  133.         IF z& < numL& THEN PRINT #2, tmpO$: ELSE PRINT #2, LEFT$(tmpO$, 91)
  134.     NEXT z&
  135.     PRINT #2, "};"
  136.     PRINT #2, ""
  137. NEXT vc&
  138. '--- read remaining BYTEs ---
  139. IF cntB& > 0 THEN
  140.     PRINT #2, "static const unsigned int8 "; tarName$; "B[] = {"
  141.     PRINT #2, "    "; LTRIM$(STR$(cntB&)); ","
  142.     PRINT #2, "    ";
  143.     FOR x% = 1 TO cntB&
  144.         GET #1, , tmpB%%
  145.         PRINT #2, "0x" + RIGHT$("00" + HEX$(tmpB%%), 2);
  146.         IF x% <> 16 THEN
  147.             IF x% <> cntB& THEN PRINT #2, ",";
  148.         ELSE
  149.             IF x% <> cntB& THEN
  150.                 PRINT #2, ","
  151.                 PRINT #2, "    ";
  152.             END IF
  153.         END IF
  154.     NEXT x%
  155.     PRINT #2, ""
  156.     PRINT #2, "};"
  157.     PRINT #2, ""
  158. '--- some functions ---
  159. PRINT #2, "// --- Saved full qualified output path and filename, so we've no troubles"
  160. PRINT #2, "// --- when cleaning up, even if the current working folder was changed"
  161. PRINT #2, "// --- during program runtime."
  162. PRINT #2, "// ---------------------------------------------------------------------"
  163. PRINT #2, "char "; tarName$; "Name[8192]; // it's a safe size for any current OS"
  164. PRINT #2, ""
  165. PRINT #2, "// --- Cleanup function to delete the written file, called by the atexit()"
  166. PRINT #2, "// --- handler at program termination time, if requested by user."
  167. PRINT #2, "// ---------------------------------------------------------------------"
  168. PRINT #2, "void Kill"; tarName$; "Data(void)"
  169. PRINT #2, "{"
  170. PRINT #2, "    remove("; tarName$; "Name);"
  171. PRINT #2, "}"
  172. PRINT #2, ""
  173. PRINT #2, "// --- Function to write the array(s) back into a file, will return the"
  174. PRINT #2, "// --- full qualified output path and filename on success, otherwise an"
  175. PRINT #2, "// --- empty string is returned (access/write errors, file truncated)."
  176. PRINT #2, "// ---------------------------------------------------------------------"
  177. PRINT #2, "const char *Write"; tarName$; "Data(const char *FileName, int16 AutoClean)"
  178. PRINT #2, "{"
  179. PRINT #2, "    FILE *han = NULL; // file handle"
  180. PRINT #2, "    int32 num = NULL; // written elements"
  181. PRINT #2, ""
  182. PRINT #2, "    #ifdef QB64_WINDOWS"
  183. PRINT #2, "    if (!_fullpath("; tarName$; "Name, FileName, 8192)) return "; CHR$(34); CHR$(34); ";"
  184. PRINT #2, "    #else"
  185. PRINT #2, "    if (!realpath(FileName, "; tarName$; "Name)) return "; CHR$(34); CHR$(34); ";"
  186. PRINT #2, "    #endif"
  187. PRINT #2, ""
  188. PRINT #2, "    if (!(han = fopen("; tarName$; "Name, "; CHR$(34); "wb"; CHR$(34); "))) return "; CHR$(34); CHR$(34); ";"
  189. PRINT #2, "    if (AutoClean) atexit(Kill"; tarName$; "Data);"
  190. PRINT #2, ""
  191. FOR vc& = 0 TO cntV&
  192.     PRINT #2, "    num = fwrite(&"; tarName$; "L"; LTRIM$(STR$(vc&)); "[1], 4, "; tarName$; "L"; LTRIM$(STR$(vc&)); "[0], han);"
  193.     PRINT #2, "    if (num != "; tarName$; "L"; LTRIM$(STR$(vc&)); "[0]) {fclose(han); return "; CHR$(34); CHR$(34); ";}"
  194.     PRINT #2, ""
  195. NEXT vc&
  196. IF cntB& > 0 THEN
  197.     PRINT #2, "    num = fwrite(&"; tarName$; "B[1], 1, "; tarName$; "B[0], han);"
  198.     PRINT #2, "    if (num != "; tarName$; "B[0]) {fclose(han); return "; CHR$(34); CHR$(34); ";}"
  199.     PRINT #2, ""
  200. PRINT #2, "    fclose(han);"
  201. PRINT #2, "    return "; tarName$; "Name;"
  202. PRINT #2, "}"
  203. PRINT #2, ""
  204. '--- ending ---
  205.  
  206. '--- .bm include file ---
  207. OPEN "O", #2, tarPath$ + tar$ + ".bm"
  208. PRINT #2, "'============================================================"
  209. PRINT #2, "'=== This file was created with MakeCARR.bas by RhoSigma, ==="
  210. PRINT #2, "'=== you must $INCLUDE this at the end of your program.   ==="
  211. IF packed% THEN
  212.     PRINT #2, "'=== ---------------------------------------------------- ==="
  213.     PRINT #2, "'=== If your program is NOT a GuiTools based application, ==="
  214.     PRINT #2, "'=== then it must also $INCLUDE: 'lzwpacker.bm' available ==="
  215.     PRINT #2, "'=== from the Libraries Collection here:                  ==="
  216.     PRINT #2, "'===    https://www.qb64.org/forum/index.php?topic=809    ==="
  217. PRINT #2, "'============================================================"
  218. PRINT #2, ""
  219. PRINT #2, "'-----------------"
  220. PRINT #2, "'--- Important ---"
  221. PRINT #2, "'-----------------"
  222. PRINT #2, "' If you need to move around this .bm file and its respective .h file"
  223. PRINT #2, "' to fit in your project, then make sure the path in the DECLARE LIBRARY"
  224. PRINT #2, "' statement below does match the actual .h file location. It's best to"
  225. PRINT #2, "' specify a relative path assuming your QB64 installation folder as root."
  226. PRINT #2, "'---------------------------------------------------------------------"
  227. PRINT #2, ""
  228. '--- writeback function ---
  229. PRINT #2, "'"; STRING$(LEN(tarName$) + 19, "-")
  230. PRINT #2, "'--- Write"; tarName$; "Array$ ---"
  231. PRINT #2, "'"; STRING$(LEN(tarName$) + 19, "-")
  232. PRINT #2, "' This function will write the array(s) you've created with MakeCARR.bas"
  233. PRINT #2, "' back to disk and so it rebuilds the original file."
  234. PRINT #2, "'"
  235. PRINT #2, "' After the writeback call, only use the returned realFile$ to access the"
  236. PRINT #2, "' written file. It's the full qualified absolute path and filename, which"
  237. PRINT #2, "' is made by expanding your maybe given relative path and an maybe altered"
  238. PRINT #2, "' filename (number added) in order to avoid the overwriting of an already"
  239. PRINT #2, "' existing file with the same name in the given location. By this means"
  240. PRINT #2, "' you'll always have safe access to the file, no matter how your current"
  241. PRINT #2, "' working folder changes during runtime."
  242. PRINT #2, "'"
  243. PRINT #2, "' If you wish, the written file can automatically be deleted for you when"
  244. PRINT #2, "' your program will end, so you don't need to do the cleanup yourself."
  245. PRINT #2, "'----------"
  246. PRINT #2, "' SYNTAX:"
  247. PRINT #2, "'   realFile$ = Write"; tarName$; "Array$ (wantFile$, autoDel%)"
  248. PRINT #2, "'----------"
  249. PRINT #2, "' INPUTS:"
  250. PRINT #2, "'   --- wantFile$ ---"
  251. PRINT #2, "'    The filename you would like to write the array(s) to, can contain"
  252. PRINT #2, "'    a full or relative path."
  253. PRINT #2, "'   --- autoDel% ---"
  254. PRINT #2, "'    Shows whether you want the auto cleanup (see description above) at"
  255. PRINT #2, "'    the program end or not (-1 = delete file, 0 = don't delete file)."
  256. PRINT #2, "'----------"
  257. PRINT #2, "' RESULT:"
  258. PRINT #2, "'   --- realFile$ ---"
  259. PRINT #2, "'    - On success this is the full qualified path and filename finally"
  260. PRINT #2, "'      used after all applied checks, use only this returned filename"
  261. PRINT #2, "'      to access the written file."
  262. PRINT #2, "'    - On failure (write/access) this will be an empty string, so you"
  263. PRINT #2, "'      should check for this before trying to access/open the file."
  264. PRINT #2, "'---------------------------------------------------------------------"
  265. PRINT #2, "FUNCTION Write"; tarName$; "Array$ (file$, clean%)"
  266. PRINT #2, "'--- declare C/C++ function ---"
  267. PRINT #2, "DECLARE LIBRARY "; CHR$(34); tarPath$; tar$; CHR$(34); " 'Do not add .h here !!"
  268. PRINT #2, "    FUNCTION Write"; tarName$; "Data$ (FileName$, BYVAL AutoClean%)"
  269. PRINT #2, "END DECLARE"
  270. PRINT #2, "'--- option _explicit requirements ---"
  271. PRINT #2, "DIM po%, body$, ext$, num%";
  272. IF packed% THEN PRINT #2, ", real$, ff%, rawdata$, filedata$": ELSE PRINT #2, ""
  273. PRINT #2, "'--- separate filename body & extension ---"
  274. PRINT #2, "FOR po% = LEN(file$) TO 1 STEP -1"
  275. PRINT #2, "    IF MID$(file$, po%, 1) = "; CHR$(34); "."; CHR$(34); " THEN"
  276. PRINT #2, "        body$ = LEFT$(file$, po% - 1)"
  277. PRINT #2, "        ext$ = MID$(file$, po%)"
  278. PRINT #2, "        EXIT FOR"
  279. PRINT #2, "    ELSEIF MID$(file$, po%, 1) = "; CHR$(34); "\"; CHR$(34); " OR MID$(file$, po%, 1) = "; CHR$(34); "/"; CHR$(34); " OR po% = 1 THEN"
  280. PRINT #2, "        body$ = file$"
  281. PRINT #2, "        ext$ = "; CHR$(34); CHR$(34)
  282. PRINT #2, "        EXIT FOR"
  283. PRINT #2, "    END IF"
  284. PRINT #2, "NEXT po%"
  285. PRINT #2, "'--- avoid overwriting of existing files ---"
  286. PRINT #2, "num% = 1"
  287. PRINT #2, "WHILE _FILEEXISTS(file$)"
  288. PRINT #2, "    file$ = body$ + "; CHR$(34); "("; CHR$(34); " + LTRIM$(STR$(num%)) + "; CHR$(34); ")"; CHR$(34); " + ext$"
  289. PRINT #2, "    num% = num% + 1"
  290. PRINT #2, "WEND"
  291. PRINT #2, "'--- write array & set result ---"
  292. IF NOT packed% THEN
  293.     PRINT #2, "Write"; tarName$; "Array$ = Write"; tarName$; "Data$(file$ + CHR$(0), clean%)"
  294.     PRINT #2, "real$ = Write"; tarName$; "Data$(file$ + CHR$(0), clean%)"
  295.     PRINT #2, "IF real$ <> "; CHR$(34); CHR$(34); " THEN"
  296.     PRINT #2, "    ff% = FREEFILE"
  297.     PRINT #2, "    OPEN real$ FOR BINARY AS ff%"
  298.     PRINT #2, "    rawdata$ = SPACE$(LOF(ff%))"
  299.     PRINT #2, "    GET #ff%, , rawdata$"
  300.     PRINT #2, "    filedata$ = LzwUnpack$(rawdata$)"
  301.     PRINT #2, "    PUT #ff%, 1, filedata$"
  302.     PRINT #2, "    CLOSE ff%"
  303.     PRINT #2, "END IF"
  304.     PRINT #2, "Write"; tarName$; "Array$ = real$"
  305. PRINT #2, "END FUNCTION"
  306. PRINT #2, ""
  307. '--- ending ---
  308.  
  309. '--- finish message ---
  310. COLOR 10: PRINT: PRINT "file successfully processed..."
  311. COLOR 9: PRINT: PRINT "You must $INCLUDE the created file (target name + .bm extension) at"
  312. PRINT "the end of your program and call the function 'Write"; tarName$; "Array$(...)'"
  313. PRINT "in an appropriate place to write the file back to disk."
  314. IF packed% THEN
  315.     COLOR 12: PRINT: PRINT "Your program must also $INCLUDE 'lzwpacker.bm' available from"
  316.     PRINT "the Libraries Collection here:"
  317.     PRINT "     https://www.qb64.org/forum/index.php?topic=809"
  318.     PRINT "to be able to write back the just processed file."
  319.     KILL tarPath$ + tar$ + ".lzw"
  320. done:
  321. '--- error handler ---
  322. abort:
  323. COLOR 12: PRINT: PRINT "something is wrong with path/file access, check your inputs and try again..."
  324. RESUME done
  325.  
  326. '$INCLUDE: 'QB64Library\LZW-Compress\lzwpacker.bm'
  327.  
  328.  
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: SMcNeill on November 17, 2018, 06:32:03 pm
I've did something similar to this by converting the file to hex values and then storing those as DATA statements in QB64 and then back again.  It doubles file size (1 ASCII byte is stored as 2 hex-values), so I never use it to embed anything except small resource files. 

I'm not at the PC right now, so I'm just curious:  How does the embedded file size here compare to the original usually?  Larger?  Smaller?  The same?

I'll personally dig into this better when I have a little more free time.  Embedded files can make project distributation a much simpler process in many cases.  :)
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: FellippeHeitor on November 17, 2018, 06:48:56 pm
Sounds like a great idea for an installer. Thanks for sharing, Rho.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on November 17, 2018, 06:52:40 pm
I've did something similar to this by converting the file to hex values and then storing those as DATA statements in QB64 and then back again.  It doubles file size (1 ASCII byte is stored as 2 hex-values), so I never use it to embed anything except small resource files. 

I'm not at the PC right now, so I'm just curious:  How does the embedded file size here compare to the original usually?  Larger?  Smaller?  The same?

I'll personally dig into this better when I have a little more free time.  Embedded files can make project distributation a much simpler process in many cases.  :)

Same process, I use hex values too, as it simply looks better in formatted DATA lines :)
As to the size now it uses LZW compression, I'd say it depends on the reached ratio. I've some config files here (approx. 5KB each) which compress pretty good between 65-80% and they come out pretty much the same size as DATAs as they are in the original size.

Sounds like a great idea for an installer. Thanks for sharing, Rho.

Well, that's a thing I wasn't thinking about at all, but you're absolutly right, nice idea.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on November 17, 2018, 09:42:24 pm
So folks, one more on top,
MakeDATA.bas got a Big Brother in the initial post above called MakeCARR.bas.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: Pete on November 17, 2018, 11:10:30 pm
This reminds me of some work I did in early 2000's to set up my office programs on several computers. That installer was about 2300 lines of code. I can't find the prototype I designed that created a single exe that would run and extract itself into 20+ exe programs. At least that's how I recall it working. A nice advancement to QB64 was the vast increase in memory. All of those programs were condensed into one. That made set up a snap.

Pete
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: Petr on November 18, 2018, 03:03:21 am
Thank you very much for sharing your program, RhoSigma. LZW compression is also used in GIF format if I do not mistake. It is definitely a very useful thing. Thanks a lot.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: Pete on November 18, 2018, 02:42:43 pm
This would have been great when I needed a lot of data on a 1.44 MB floppy disk. I'm glad those days are gone. When I made my winzip clone, I just linked up many exe files but, of course, that isn't data compression.

With storage so vastly expanded, well, I'm just going to flat out ask: How important is data compression now?

Curious. Did you come up with this algorithm by thinking your way to a solution, trial and error method, or was this something you studied and implemented?

Pete
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on November 18, 2018, 02:50:17 pm
How important? - Very important for embedded files, less size = less compile time = less turnaround times when testing and debugging.

Well, first I was searching the internet for something ready to use in QB64, but as mentioned, most examples were very rudimentary or hard to follow, so it's a rewrite especially with QB64 in mind.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: Pete on November 18, 2018, 03:47:06 pm
I didn't think in terms of debugging. I guess that's important to coders, unlike me, who make mistakes. :D

Rob got QB64 compilation times consistently better from days of old. I think he tweaked the process so minimal changes in the code could be handled faster, too. Anyway, I used to wait for 30 - 40 minutes 10 ears ago to compile an 80,000 line program. That's probably cut to about 3 minutes now in the latest GL version. Still, 90-seconds would beat that over time. I recall needing 8 or more tries to get a particularly hard bug situation solved. Back then, that would take most of the day (with time needed to recode, included.) Today if compressing data could save 15 - 20 minutes for every hour of debugging, yes, I agree, that would be huge!

Pete
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: SMcNeill on November 18, 2018, 04:03:55 pm
How important? - Very important for embedded files, less size = less compile time = less turnaround times when testing and debugging.

Well, first I was searching the internet for something ready to use in QB64, but as mentioned, most examples were very rudimentary or hard to follow, so it's a rewrite especially with QB64 in mind.

What I'd love to be able to do is something like:

OPEN "Data:LabelX" FOR INPUT AS #1
DO UNTIL EOF(1)
    INPUT #1, whatever$
    PRINT whatever$
NEXT

Much like we could open either a filename$ or "SCRN" to swap between file and screen printing, it'd be nice to swap between internal and external data files.  Then you could easily test the code with external files, and save embedding the DATA until last, reducing compile times and IDE responsiveness/ease of navigation.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: Pete on November 18, 2018, 04:19:57 pm
@Steve: I did a lot of that when QB64 was new. Not so much with the data, as I almost always used external data due to space issues in QB45, but with subs, etc. that were not part of the bug I was tracing. I made a program that would unload selected subs, rem out the calls, and then run the program. That reduced the code form say 80,000 lines to maybe 10,000 lines, and those 45 minute waits were drastically reduced. Still, it was a PITA, just a less time consuming PITA.

Pete
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on November 18, 2018, 06:49:04 pm
What I'd love to be able to do is something like:

OPEN "Data:LabelX" FOR INPUT AS #1
DO UNTIL EOF(1)
    INPUT #1, whatever$
    PRINT whatever$
NEXT

Much like we could open either a filename$ or "SCRN" to swap between file and screen printing, it'd be nice to swap between internal and external data files.  Then you could easily test the code with external files, and save embedding the DATA until last, reducing compile times and IDE responsiveness/ease of navigation.

Cool idea, can we add that as feature request for version 1.3 or 1.4 please? - Would eliminate the need to write back embedded files in the first place.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: Pete on November 18, 2018, 07:19:50 pm
As long as we're brainstorming a bit here, the QB45 IDE has a LOAD / UNLOAD Module feature. Although QB64 did not reproduce QB45's multi-modular abilities, mostly because the memory improvements made that feature almost unnecessary, another IDE feature could be loading and unloading subs and functions. That not only helps with compiling time when debugging, it also cuts out a lot of clutter when you have to add to really large programs.

Pete
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on November 19, 2018, 01:45:39 am
... another IDE feature could be loading and unloading subs and functions ...

Well that's a thing, what I'd expect the used C/C++ compiler is doing for us automatically when linking all compiled stuff into the final EXE. At least in my old Commodore Amiga days, the linking process did leave out all functions, which never were called (or referenced in general) somewere in the code (the used IDE/Compiler there was  MaxonC++ 4.0). Doing so makes sense, as otherwise eg. static link libraries would be included in whole into the EXE, even if you only need 1 or 2 functions out of it. However, I don't know the exact behavior of g++ here, maybe an extra optimization switch may be required. Well all this is of course only affecting the EXE size, it won't change the C/C++ compile times, as first all sourcefiles are compilled and the function sort out will then happen while linking.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: luke on November 19, 2018, 02:22:41 am
In general the linker will leave out object files that are not providing any required symbols. This basically means that libqb needs to be split into separately-compilable files, while trying to keep the inter-file dependencies down. I believe this is actually somewhat close to the structure of the original BCOM45.LIB, with its B$ functions.

I have tried to separate out the pieces of libqb and it proves to be non-trivial. I suppose the best way to go about it is as a gradual process.

I'm bad with naming things, but how about this:
Code: QB64: [Select]
  1. $attach: 'image.png'
  2. img& = _LOADIMAGE("ATTACH:image.png")
With the same syntax working for OPEN and the other file functions. This fits the trend we have with SCRN: and COM:
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on November 19, 2018, 04:58:08 am
Code: QB64: [Select]
  1. $attach: 'image.png'
  2. img& = _LOADIMAGE("ATTACH:image.png")
With the same syntax working for OPEN and the other file functions. This fits the trend we have with SCRN: and COM:

Well, this appoch seems similar to resource table extraction, as many professional programs use to embed its toolbar icons. Maybe its possible (instead of attaching/emedding files via DATAs or C-arrays) to simply get better/easier support to access the resouce table in QB64 with some new commands:

eg:
Code: QB64: [Select]
  1. '$INCRESOURCE: 'imageA.png'
  2. '$INCRESOURCE: 'imageB.png'
  3. '$INCRESOURCE: 'imageC.png'
  4.  
  5. 'open resource by name
  6. OPEN "RSRCN:imageB.png" AS #1 'shortcut open syntax, as readonly is implied
  7.  
  8. 'or open resource by index
  9. OPEN "RSRCI:3" AS #1 'should open imageC.png then
  10.  
  11. ....
  12. ....
  13.  

Same usage in _LOADIMAGE/_LOADFONT etc., however, I'm not so familiar with resource table specs, is it possible to place font files or simple text files there too (such as INI/CFG or even TXT)?
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: SMcNeill on November 19, 2018, 06:54:18 am
I don't know if a number reference is necessarily a good idea.  Imagine your code used in a set of $INCLUDE files... Is ImageB.png the 2nd file, or the 22nd?

Personally, I like an idea similar to this:

$RESOURCE:ImageB.PNG, 123456
      First value is the resource name which we want to reference it by.
      The number value sets a hard limit for the max size of the resource, which we can then manually
      copy/paste here, or get the IDE to load from file with a simple Load Resource option in the menu.
      NO ERROR CHECKING/SYNTAX CORRECTION OCCURS HERE.
      It'd be nice if this was a collapsible mid section, for ease of code navigation...
$END RESOURCES

Then you can use "$RESOURCE:name" for any file operation.  (Even OUTPUT/PUT could be supported as long as space was reserved first.)l

handle = _LOADIMAGE("$RESOURCE:ImageB.PNG",32)
font = _LOADFONT("$RESOURCE:InternalFont2.TTF", 16, "MONOSPACE")
OPEN "$RESOURCE:OwnerData.txt" FOR OUTPUT AS #1

And such...
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: Pete on November 19, 2018, 12:05:31 pm
Why not just make an auto-detect function to assign space? If the size varies because the embedded element is changed, the exe would auto-detect it and assign space accordingly. Of course this means the $RESOURCE meta commands would have to be read first, wherever they occurred in the code, to establish the space values, prior to executing the statements statements. Possible? I don't work with C/C++ so memory allocation, pointers, etc. are foreign to me.

Pete
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on November 19, 2018, 12:25:30 pm
Whatever the developers do, assuming they take the discussed ideas, should be easy to handle and fit into the existing concepts of the QB64 language.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: Dav on November 19, 2018, 12:52:05 pm
Great that resource files are being considered!  I always thought the way rapidq handles resource files is simple but useful.

'Define the resource
 $RESOURCE BMP1 AS "FUNNY.BMP"

'Use the resource like this
OPEN BMP1 FOR BINARY AS #1
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: Pete on November 19, 2018, 06:29:14 pm
@ Dav: I'm not seeing the usefulness in the rename portion here. I believe I tried RapidQ once, but it was object oriented so I dropped it like a rock. Anyway, do you see any advantage in BMP1 representing "FUNNY.BMP" because I don't. I do see that using $RESOURCE to import a resource into a program is valuable.

On a side note, how is everything after the storms? I hope you only had some minor problems, if any, but that was a lot of water over a long period of time.

Also, please have a look at this thread: https://www.qb64.org/forum/index.php?topic=799.msg100082#new I know you have written some pretty large projects, like your IDE.

Pete

Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: Dav on November 20, 2018, 10:19:26 am
Hi Pete!  Well, that's a better idea there, bypass making a new variable and just use the original filename after declaring it a resource.  I mostly wanted to point out how small and easy it was in rapidq to add/use resource files.  As for that language, I hear ya, I didn't warm up to it that much either.  But it had some ok networking capabilities which I used to make a few projects.

We got through the storms ok.  Lost power about a week, had some tree debris to saw up. Nothing big fell on the house. The creek flooded and came up to about 6 feet from the house (witnesses said, I didn't see it), but water never got into the house.  We got through it a lot better than others did.  Taught me some lessons on being better prepared.  Thank you for asking. 

I was reading that "big one" thread.  80k lines is a big program!  I think my IDE is about 20k.  I remember having some larger programs over the years, due to embedding data in the code probably.  Will have to dig into some old HD's and take a peek.

- Dav
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on December 11, 2018, 06:56:59 pm
Hi all,
just updated the MakeDATA.bas codebox in the initial post with a new version, which gives a really nice speedup in the created writeback function compared to the old version. This is especially good for somewhat larger files. Tested it with the cyberbit.ttf font file, the writeback call finishes more than 300 times faster now (before approx. half an hour, now 5 seconds).

It's done just by taking out the temporary string operations, which QB64 does under the hood when adding more chars to a variable length string. If you do so in a loop (eg. reading DATAs and concatenate them into one string), then the complexity raises very quickly to an unthinkable amount, causing this immense slowdown.

Now i define the required length of the string before entering the loop with SPACE$(n), and then just replace chars in it using MID$ with an ever increasing char offset.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on December 28, 2018, 06:22:39 am
EDIT: Obsolete informations deleted.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on February 25, 2019, 05:33:33 pm
EDIT: Obsolete informations deleted.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on March 01, 2019, 08:11:07 pm
EDIT: Obsolete informations deleted.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: RhoSigma on October 05, 2021, 04:32:09 am
Major update regarding the recursion issue (https://qb64forum.alephc.xyz/index.php?topic=4209), also the code files generated by these tools are OPTION _EXPLICIT friendly now and the code generated by MakeCARR should now work under Linux and MacOS.

Please take the new updated code from the inital post here (https://qb64forum.alephc.xyz/index.php?topic=790.msg6907#msg6907) and make sure to carfully read the entire post.



@Qwerkey, the code must be updated in the "Utilities" board too. When doing so, please also remove the lzwpacker.bm attachement there, cause it's also outdated since a long time, rather mention my Libraries Collection (https://qb64forum.alephc.xyz/index.php?topic=809) as official place to get the latest version of the required Lzw library.
Title: Re: Embedding files in programs (FileToDATA convertor)
Post by: bplus on October 05, 2021, 12:48:02 pm
I'm glad this got bumped up again, I need to do a real study of this, run some tests. 2 years ago I guess I missed it or something, or maybe now the time is ripe for me to review.