How is DATA stored in memory?

[SMcNeill]

So here’s something I was wondering about today: How is our DATA statements stored in memory?

Let’s say I have the following: DATA 123, 456, 789, Foo, EOD

Is that stored and read as a single string?  “123, 456, 789, Foo, EOD”

Individual strings?  “123”, “456”, “789”, “Foo”, “EOD”

As a series of integers and strings?

Just how the heck is that data actually stored in memory, and is there ever anyway to free it from a program’s memory if it’s never needed anymore to reduce memory usage?

If I assign values to an array manually like so:
x(1) = “123”
x(2) = “456”
x(3) = “789”
x(4) = “foo”

Then I can always free that memory with a REDIM x(0) AS STRING, an ERASE, or a CLEAR statement.

But is there anyway to remove DATA from memory?  Or is it just stuck there always using resources even when it doesn’t need to?

[RhoSigma]

Hi Steve,

QB64 will write the DATA content into the file internal\temp\data.bin almost as is, it just substitutes the comma used for separating the individual values with a dot (for whatever reason).

E.g. DATA 123,456,&DEADBEEF,&HBADC0DE,text,"quoted text",&B10010011

becomes 123.456.&DEADBEEF.&HBADC0DE.text."quoted text".&B10010011

to use your analogy, it writes it as one string concatenated by dots.

The C++ compiler will translate data.bin into data.o without changing the data from data.bin, but adding reference symbols and some tables. Not sure, but my best guess would be, that the tables contain the offsets of RESTORE lables into the data block.

The data.o is then simply linked into the final EXE, hence your question about freeing the DATA memory is answered with a clear NO. There's no way to remove/free the data from a running program.

READing the data is practically almost identical to reading values from a file, as far I've learned it when implementing the &B stuff for files and DATA. See here: https://www.qb64.org/forum/index.php?topic=3627.msg129626#msg129626 (especially the 3rd paragraph)

You can easily check all this with a HEX editor. If you don't have one yet, I can recommend "HEX Editor MX" from NEXT-Soft, It's free to use without limitations. They would only expect you to register for extended support, which 1st is not needed for such an easy editor and 2nd the company obviously is no longer a operational unit. It is/was a small german company and the link below brings you directly to the HEX-Editor download page.

However, the editor itself can be switched to english too, using the Menu>Optionen>Einstellungen and then on the tab "Sprache" select english from the dropdown list, then click OK and restart the editor and it will work in english. https://www.nextsoft.de/index.php?view=products/tools/hexedit/download.htm&right=products/tools/hexedit/rightmenu.htm

[RhoSigma]

Oh no, guess my eyes getting worse,

forget what I said about substituting commas to dots. No, it all remain commas (should have a look on the ASCII code instead of the ASCII dump in the HEX Editor), hence the data goes just as written in the DATA lines into the data.bin file.

[SMcNeill]

Many thanks Rho.  I was going to dig into it, but thought I’d simply ask first as I knew you and a few others have delved into the internals of our DATA routines more than I have.  My biggest concern was basically about freeing a large data block for lower memory machines (like a Raspberry Pi), or locating their position in memory and overwriting it at a later date — neither of which seems easily possible to do.

The issue I’m currently looking at is helping a friend with a BAS program they’re trying to compile for a Pi.  The big issue is embedded fonts, images, and sounds stored in massive DATA statements with no way to free them to free up the resources needed for their actual customer data files. 

I suppose at this point it’s either direct data values (x(1) = “foo”: x(2) = “bar”: ect), or else its moving from DATA to CSV external files.  (Or add more memory/pagefile to the Pi!)

Embedding data inside a program can be nice, but not if your target machine doesn’t have the memory to load it!  I feel like we’ve regressed back to the days of the old Apple IIc machines…. LOL

[RhoSigma]

Embedding data inside a program can be nice, but not if your target machine doesn’t have the memory to load it!  I feel like we’ve regressed back to the days of the old Apple IIc machines…. LOL

That's true, embedding small things like icons for a toolbar etc. is pretty ok, but fullsize images,sounds etc. would bloat the final EXE a lot, not to mention the time the syntax parser needs at each keypress to run through 1000s of DATA lines.
The simple problem here is, that DATA stores everything in string form, even numbers. E.g. &HDEADBEEF would eat up 10 bytes although 4 bytes would be enough if stored as binary number.

You probably know my MakeDATA converter https://www.qb64.org/forum/index.php?topic=2288.0

The second tool is called MakeCARR, which converts a file into a C-Array rather than DATAs. This tool will provide these two advantages, numbers are stored binary, no 1000s of DATA lines which slow down the syntax parser.
Also follow the link from the Library entry to the my original post about it to read more.

Maybe this would be an alternative to try before you going the long way to convert everything into CSV files.

[Petr]

In the completed EXE file, however, it is possible to read and overwrite the contents of the DATA block (it is readable there as in the source code), the only condition is that the original and new strings have the same length - and good editor for it.

[Petr]

@SMcNeill
There is another possibility. It depends on what the DATA blocks contain. If it was only for local use, if you could force a US ascii table, you can write a BASE 128 decoder, or use BASE 64 for DATA block compatibility to ensure that DATA blocks don't grow too big.