Author Topic: Steve's Video Tutorials  (Read 20933 times)

0 Members and 1 Guest are viewing this topic.

Offline Qwerkey

  • Forum Resident
  • Posts: 755
    • View Profile
Re: Steve's Video Tutorials
« Reply #45 on: September 29, 2019, 10:27:40 am »
Technical Query No. 2

Steve, you have been teaching us dim folk what's happening with _MEM, and I now begin to see that _PUTting and _GETting to/from memory involves addressing individual bytes in a block of memory, each byte being 256 bits (0-255).  And so we're only ever talking about numbers: the hardware never uses letters, of course, even if we're storing a string.  So, when we store, say, a 4-letter string in _MEM this is four bytes next to each other, each byte storing the ASCII value of each letter.

I hope that I haven't got even this simple idea wrong!

Using your first video and adapting your code, I am now able to directly read a hard disk stored dictionary file into _MEM, and I have used the following code:

Code: QB64: [Select]
  1. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
  2. FIELD #1, 4 AS ULoc$
  3. NoWords& = LOF(1) / 4
  4.  
  5. M = _MEMNEW(LOF(1))
  6.  
  7.  
  8. FOR N& = 1 TO LOF(1) / 4
  9.     GET #1, N&
  10.     _MEMPUT M, M.OFFSET + ((N& - 1) * 4), ULoc$
  11. NEXT N&
  12.  
  13. temp$ = SPACE$(4)
  14. _MEMGET M, M.OFFSET, temp$
  15. PRINT temp$
  16.  
  17. _MEMGET M, M.OFFSET + 4, temp$
  18. PRINT temp$
  19.  
  20. _MEMGET M, M.OFFSET + 8, temp$
  21. PRINT temp$
  22.  
  23. _MEMGET M, M.OFFSET + (NoWords& - 1) * 4, temp$
  24. PRINT temp$
  25.  
  26. _MEMGET M, M.OFFSET + M.SIZE - 4, temp$
  27. PRINT temp$
  28.  
  29. _MEMGET M, M.OFFSET + M.SIZE - 8, temp$
  30. PRINT temp$
  31.  
  32. _MEMGET M, M.OFFSET + LOF(1) - 8, temp$
  33. PRINT temp$
  34.  
  35. temp$ = SPACE$(5)
  36. _MEMGET M, M.OFFSET + M.SIZE - 5, temp$
  37. PRINT temp$
  38.  

NB This code is not usable: I have not supplied the file from which the data are read.  It is illustrative only.

What surprises me is that nowhere in the code have I had to tell the computer to convert individual letters into their ASCII code.

In the statement DIM M AS _MEM, there is no indication that strings are going to be used (as opposed to numbers next to each other).

And in the statement _MEMPUT M, M.OFFSET + ((N& - 1) * 4), ULoc$, we are able to _PUT a string without converting to numbers.  And in the statement _MEMGET M, M.OFFSET, temp$, the block of numbers is converted back to a string without having to code such an action.

Is it therefore the case that when you _MEMPUT or _MEMGET and your code has a string, QB64 just does the string to numeric byte conversions?  I feel that this must be so, and if using "normal" string assignment, storing and retrieving, QB64 is always doing these conversions without the coder knowing anything about it.

I think that what I'm saying is that I'd have expected the _MEM dimensioning statement to be something like DIM M AS _MEM AS STRING, just to let the computer know that this will deal with strings, not numbers.
« Last Edit: September 29, 2019, 10:28:48 am by Qwerkey »

Offline Qwerkey

  • Forum Resident
  • Posts: 755
    • View Profile
Re: Steve's Video Tutorials
« Reply #46 on: September 29, 2019, 11:08:53 am »
Steve, I think that I've managed to answer my own question.  I tried adding 1 to one of the _MEM bytes of previous reply, and indeed the character (was "B") changed to "C".

Code:
_MEMPUT M, M.OFFSET + 1, _MEMGET(M, M.OFFSET + 1, _UNSIGNED _BYTE) + 1 AS _UNSIGNED _BYTE

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Steve's Video Tutorials
« Reply #47 on: September 29, 2019, 12:19:33 pm »
Quote
I think that what I'm saying is that I'd have expected the _MEM dimensioning statement to be something like DIM M AS _MEM AS STRING, just to let the computer know that this will deal with strings, not numbers.

You're making your computer out to be smarter than it really is.  ;)

When teaching my daughter how to code, the very first lesson I tried to stress with her is that computers aren't very smart -- no matter what people commonly think, nowadays.  Computers aren't complicated because they're HARD; they're complicated because they're TOO SIMPLE.  No matter what you think the computer is doing, all it's actually doing is adding or comparing 0s and 1s.  EVERYTHING, in the end, breaks down to bits which are either ON or OFF.  (-1 OR 0, as we represent them in QB64.)

The simple truth is: A computer doesn't use any numbers, nor any characters, nor strings.  All your computer knows is ON and OFF state of bits.  It's up to the PROGRAMMER to define what a sequence of bits represent -- either an integer (in various sizes), a real number (again, in various precision sizes), or a character/string.

As I was trying to stress in the first tutorial, _MEM is inherently nothing more than opening a file for binary access.  Tell me, would you ever think of doing the following:

OPEN "temp.txt" FOR BINARY AS STRING , FILENUMBER 1

That OPEN statement simply says, "We want to access a storage area on the hard drive, and read and write data to it."  We don't tell it that we want to work with strings with it, or numbers, or a strange combination of the two.  We just issue a command that says, "We want to use a segment of the drive, call it "temp.txt", and get and put data to that segment."

_MEM, is more-or-less, a direct mirror to those BINARY access commands.  Just as OPEN is the command to set the stage to interact with DATA on the hard drive, DIM m AS _MEM is doing the same job and setting the stage for us to interact with DATA somewhere in memory.

Let me give you a quick example to run and study, which helps highlight the issue:

Code: QB64: [Select]
  1. OPEN "temp.txt" FOR OUTPUT AS #1 'This just opens a new file, erasing any old file with the same name.
  2. OPEN "temp.txt" FOR BINARY AS #1 'And this now opens that blank file so I can get/put information to it.
  3. text$ = "1234567890" 'a simple 10 character text string.
  4. PUT #1, 1, text$ 'which I'm now writting to the disk.
  5.  
  6. 'All very common practice, and nothing strange or mysterious going on so far.
  7. 'I opened a file, I put a string into it.
  8. 'But now....
  9.  
  10. 'Let's start by making certain that the file contains the data I want:
  11. in$ = SPACE$(10) ' I want a string 10 bytes long, so I can get all 10 bytes at once
  12. GET #1, 1, in$ 'which I get from the drive
  13. PRINT in$ 'and then print to see my string and make certain that I did, indeed, place string information to the drive.
  14.  
  15. SLEEP 'at this point, run the program and verify what the result is for yourself.
  16. 'Exit the program after the first pause (Where the SLEEP statement would appear), and then come back and follow the code
  17. 'as I comment it below.
  18.  
  19. 'READY TO CONTINUE??
  20.  
  21.  
  22.  
  23. 'Good!
  24.  
  25. 'At this point, you can now see that we wrote a string to the drive...
  26. 'OR...
  27. 'Did we really???
  28.  
  29. 'Let's test a few things:
  30.  
  31. DIM b AS _BYTE, i AS INTEGER, l AS LONG, i64 AS _INTEGER64 'let's first define a few variables as various types
  32. DIM s AS SINGLE, d AS DOUBLE, f AS _FLOAT, c AS STRING * 1 'c for character, otherwise all our variable names are...
  33. '        just the first letter of what they represent (with the exception of i64 for integer64, as i stands for integer)
  34.  
  35. GET #1, 1, b 'get a byte from the start of that file
  36. GET #1, 1, i 'get an integer from the start of that file
  37. GET #1, 1, l 'a long
  38. GET #1, 1, i64 'an integer64
  39. GET #1, 1, s 'a single
  40. GET #1, 1, d 'a double
  41. GET #1, 1, f ' a float
  42. GET #1, 1, c 'and finish with a character
  43.  
  44. 'Now surely, since we wrote the data as a string, these commands are going to fail and toss us an error.  Right??
  45.  
  46. 'Let's try and print them, then run it and let's see!!
  47. PRINT b, i, l, i64
  48. PRINT s, d, f
  49. PRINT c 'the character will print to the 3rd line, as it's the letter "1" and I don't want it lost in the numbers above.
  50.  
  51. SLEEP 'another pause, like before, so we can stop running the code at the proper spot as we study this example code.
  52. 'NOW, RUN THE PROGRAM AGAIN, THIS TIME STOPPING IT AT THE SECOND PAUSE (SLEEP) SEGMENT.
  53.  
  54. 'Executed the code, yet?
  55.  
  56. 'If so, were you shocked by the results?
  57.  
  58. 'We wrote string data to the drive, but then somehow we managed to get it all back as numeric values???
  59. 'WTH is up with that??
  60.  
  61.  
  62. 'It's simply because it's as I was saying in my post over at the QB64 forums -- computers are much stupider than folks believe.
  63. 'All the computer does is set bits on the hard drive to be either ON or OFF, and then we GET and PUT those bits into specific
  64. 'areas of the hardrive with our OPEN and GET/PUT statement.
  65.  
  66. 'OPEN defines a block of the drive that we're going to use...In this case from where "temp.txt" starts, to where "temp.txt"
  67. 'ends at.  I have no real idea where the heck that segment is; I trust the open statement to find the right spot on the
  68. 'drive and then make that access available for me.
  69.  
  70. 'Then, once that segment of the drive is made available, I futher specify which area of that segment that I want to interact
  71. 'with, with the GET and PUT statement.
  72. 'GET FILEHANDLE, POSITION, INFORMATION....
  73. 'Or, as we commonly see it: GET #1, 1, variable  (for the first byte of that file)
  74.  
  75. 'It's by defining our variable types that our computer knows how to interpet those 0 and 1 bits that we want to read/write
  76. 'to the drive.
  77.  
  78. 'text$ = "1234567890"  <--this tells the computer to convert those 10 characters into the 0 and 1 representations of string values.
  79. 'DIM b as _BYTE <-- this tells the computer that those 0s and 1s are going to represent an integer value from -128 to 127
  80.  
  81. 'The hard drive is ONLY storing bits which are either ON or OFF (0 or 1).
  82. 'It doesn't care what we put in our files -- it doesn't know the difference between a string or a number at all.
  83. 'All our hard drive knows is "This bit here is a 0.  This bit after it is a 1.  The next bit is a 0.  Then a 1.  Another 1...."
  84.  
  85. 'MEM is the *EXACT* same thing, as I'll illustrate below:
  86.  
  87. DIM M AS _MEM 'just like that OPEN command, we're now preparing to open a segment of memory
  88. M = _MEMNEW(10) 'in this case, I'm deciding it only needs to be 10 bytes of memory.
  89.  
  90. _MEMPUT M, M.OFFSET, text$ 'And, if you still remember our original text$, I just put it in memory, in that memblock.
  91. in2$ = SPACE$(10) 'and here I'm setting a second in$ so I can get and print that information to the screen
  92. _MEMGET M, M.OFFSET, in2$ 'and I just got in2$ from that spot in memory.
  93.  
  94. 'Now, I don't think there's any real reason for a third SLEEP statement in this demo, so I'm just going to do as I did
  95. 'above and _MEMGET and _MEMPUT that same information into a series of variables:
  96.  
  97. DIM b2 AS _BYTE, i2 AS INTEGER, l2 AS LONG, i642 AS _INTEGER64 'the 2 here says they're all new variables, unrelated
  98. DIM s2 AS SINGLE, d2 AS DOUBLE, f2 AS _FLOAT, c2 AS STRING * 1 'to the ones which I used before.
  99.  
  100. _MEMGET M, M.OFFSET, b2 'get a byte from the start of that mem block
  101. _MEMGET M, M.OFFSET, i2 'an integer
  102. _MEMGET M, M.OFFSET, l2 'long
  103. _MEMGET M, M.OFFSET, i642 'integer64
  104. _MEMGET M, M.OFFSET, s2 'single
  105. _MEMGET M, M.OFFSET, d2 'double
  106. 'I'm turning error checking off here, as the next line would technically toss an error for us.
  107. '"WHY," you ask?
  108. 'Because we only have 10 bytes reserved for memory and a _FLOAT requires 32 of them.  (They're a HUGE variable type.)
  109. 'You can't make a blanket from a tablecloth, and you can't make a float from 10 bytes.
  110. '(Well, at least you can't, without tossing an error -- which I'm just blindly ignoring for this demo and which
  111. 'COULD cause some serious corruption in my program if I wasn't careful. After all, who knows what variable/data is stored
  112. 'in the bytes immediately after the 10 where my memblock is mysteriously located??)
  113. _MEMGET M, M.OFFSET, f2 'float
  114. 'Lesson to take from this:  ONLY USE CHECKING:OFF when you either: A) Know your code is 100% correct
  115. 'OR                                                                B) You seriously don't give a damn if you corrupt your data.
  116. _MEMGET M, M.OFFSET, c2 'and finally, let's get that last character
  117.  
  118. PRINT b2, i2, l2, i642 'and now print the results to the screen
  119. PRINT s2, d2, f2
  120.  
  121. 'RUN AND COMPARE WHAT YOU SEE......
  122.  
  123. '.
  124. '..
  125. '...
  126.  
  127. 'No difference in the outputs?
  128.  
  129.  
  130. 'Shocked??
  131.  
  132. 'If so, then don't be.  As I was stressing in my first video: _MEM is nothing more complex than working with a file opened
  133. 'FOR BINARY access.  GET/PUT gets and puts sequences of 0s and 1s onto our hard drive.
  134. '                _MEMGET/_MEMPUT does the exact same thing, except it does it somewhere in memory.
  135.  
  136. 'Now, as to whether that sequence of 0s and 1s is going to be integers, real numbers, strings, or some user-defined type
  137. 'which is a combination of all three...
  138. 'Our memory doesn't care about any of that!  All it does is faithfully hold a sequence of 0s and 1s in a specific order.
  139.  
  140. 'It's up to us to define how we want to interpet that sequence of 0s and 1s.
  141. 'And we do that either via how we define our variable types:  DIM b AS _BYTE, i AS INTEGER
  142. 'Or we specify it specifically: _MEMPUT m, m.offset, 123 AS _BYTE
  143.  
  144. 'Make sense now?
  145. 'Hopefully, the above little demo will clear up all your confusion for you.
  146. 'If not, just ask once again and let me know what/where you're lost, and I'll try and help you understand what's going on.  ;)
  147.  

Study the code above and see if it helps answer your query for you.  ;)


https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Steve's Video Tutorials
« Reply #48 on: September 29, 2019, 12:28:44 pm »
Steve, I think that I've managed to answer my own question.  I tried adding 1 to one of the _MEM bytes of previous reply, and indeed the character (was "B") changed to "C".

Code:
_MEMPUT M, M.OFFSET + 1, _MEMGET(M, M.OFFSET + 1, _UNSIGNED _BYTE) + 1 AS _UNSIGNED _BYTE

And this is often the very best way to learn these commands:  Simply test them out and try various things with them and see how they behave.  Trust that you can't actually break anything -- QB64 and your OS protects you from any serious issues such as destroying your computer.  About the worst you can do is make your program crash, and can you name ANY programmer who hasn't already done that a couple of thousand times in their career?? 

When we crash, we just have to go in and debug whatever the hell went wrong.  ;D

The best way to learn any new command in a programming language is simply to use it and learn from trial and error.  If it doesn't behave as you expect it should, ask about it and then you'll either:
A) Have someone explain to you what's happening and why it happened
OR...
B) You find out that you discovered a bug in the command (and Lord knows, we have plenty of those), which can be addressed and hopefully corrected in a later version of the software. 
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline Qwerkey

  • Forum Resident
  • Posts: 755
    • View Profile
Re: Steve's Video Tutorials
« Reply #49 on: September 29, 2019, 01:10:24 pm »
Thanks Steve.  I went through your extra program and with my trial-and-error am begining to to get to grips with it.  I'm still used to coding BASIC where you never have to think how to handle individual bytes.  No wonder they made the acronym BASIC: for the people who don't actually know what they're doing!  The story of my life, but, hey, I'm happy with that.

Offline Qwerkey

  • Forum Resident
  • Posts: 755
    • View Profile
Re: Steve's Video Tutorials
« Reply #50 on: September 30, 2019, 04:46:18 am »
Next dim question (I've got a million of them!).  As I mentioned above, I have the intention to produce a _MEM version of my Crossword Generator program which uses heaps of processing to produce a correctly filled crossword grid.  Am I right in thinking that when I want to have fastest processing, all things that were "normal" variables get replaced by _MEM objects?  So, for example
Code: QB64: [Select]
  1. FOR N& = 1 to 1000000000
  2.         Lots of _MEM processing
  3. NEXT N&
even the N& variable would (somehow) be replaced by a _MEM object?

Or, using Steve's code in reply #21:
Code: QB64: [Select]
  1.     IF _MEMGET(m, m.OFFSET + o, _UNSIGNED LONG) = _RGBA32(255, 255, 255, 255) THEN
  2.         _MEMPUT m, m.OFFSET + o, 256 AS _UNSIGNED _BYTE
  3.     END IF
  4.     o = o + 4
  5. LOOP UNTIL o >= m.SIZE - 4
should the variable o be replaced by a _MEM object to speed things up?  I'm confused by the mixing of "normal" variables (slow) and _MEM objects (fast).

Maybe things such as this will become clearly when Steve has completed the library of _MEM tutorials.
« Last Edit: September 30, 2019, 05:49:35 am by Qwerkey »

Offline Qwerkey

  • Forum Resident
  • Posts: 755
    • View Profile
Re: Steve's Video Tutorials
« Reply #51 on: September 30, 2019, 07:33:55 am »
Default values of memory blocks.

With the code here, I was surprised to find that without line 11, the subsequent WHILE/WEND loop did not execute.

What I was trying to do was to index the memory block byte position of object M with a one-byte object MCount whose value is increased from the starting value by one at each loop cycle.

I originally found that the MCount starting value was 23 and on other runs it was 21.  So line 11 sets the starting value of MCount to be zero.  Thinking that it would behave like a variable, I assumed that its starting value be zero.  But as the program allocates a block of memory, the starting value is just what is in that part of the (unused) memory at that time???

Code: QB64: [Select]
  1. M = _MEMNEW(20)
  2.  
  3. Junk$ = "ABCDEFGHIJKLMNOPQRST"
  4.  
  5. _MEMPUT M, M.OFFSET, Junk$
  6.  
  7. DIM MCount AS _MEM
  8. MCount = _MEMNEW(1)
  9.  
  10. _MEMPUT MCount, MCount.OFFSET, 0 AS _UNSIGNED _BYTE
  11.  
  12. PRINT _MEMGET(MCount, MCount.OFFSET, _UNSIGNED _BYTE)
  13.  
  14. WHILE _MEMGET(MCount, MCount.OFFSET, _UNSIGNED _BYTE) < 20
  15.  
  16.     PRINT _MEMGET(MCount, MCount.OFFSET, _UNSIGNED _BYTE); _MEMGET(M, M.OFFSET + _MEMGET(MCount, MCount.OFFSET, _UNSIGNED _BYTE), STRING * 1)
  17.  
  18.     _MEMPUT MCount, MCount.OFFSET, _MEMGET(MCount, MCount.OFFSET, _UNSIGNED _BYTE) + 1 AS _UNSIGNED _BYTE
  19.  
  20.  
  21. _MEMFREE MCount
  22.  

Oh, by the way, mucking around with actual blocks of memory is real fun!

A further aside:  Steve, you have trouble remembering that the command is _MEMFREE rather than _FREEMEM (cf. _FREEIMAGE etc.).  Here's a little aide memoire for you, but you'll need to be familiar with the British TV comedy programme "Are You Being Served?".  I don't suppose that this ever reached the US, but if it did and you ever saw it, just remember the Mr Humphries character and his catchphrase - I'm Free! (to be said a very camp fashion).  And then keep saying "Mem Free!".  I cannot ever get this command wrong.
« Last Edit: September 30, 2019, 10:18:09 am by Qwerkey »

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Steve's Video Tutorials
« Reply #52 on: September 30, 2019, 10:24:32 am »
Next dim question (I've got a million of them!).  As I mentioned above, I have the intention to produce a _MEM version of my Crossword Generator program which uses heaps of processing to produce a correctly filled crossword grid.  Am I right in thinking that when I want to have fastest processing, all things that were "normal" variables get replaced by _MEM objects?  So, for example
Code: QB64: [Select]
  1. FOR N& = 1 to 1000000000
  2.         Lots of _MEM processing
  3. NEXT N&
even the N& variable would (somehow) be replaced by a _MEM object?

Or, using Steve's code in reply #21:
Code: QB64: [Select]
  1.     IF _MEMGET(m, m.OFFSET + o, _UNSIGNED LONG) = _RGBA32(255, 255, 255, 255) THEN
  2.         _MEMPUT m, m.OFFSET + o, 256 AS _UNSIGNED _BYTE
  3.     END IF
  4.     o = o + 4
  5. LOOP UNTIL o >= m.SIZE - 4
should the variable o be replaced by a _MEM object to speed things up?  I'm confused by the mixing of "normal" variables (slow) and _MEM objects (fast).

Maybe things such as this will become clearly when Steve has completed the library of _MEM tutorials.

Directly accessing a variable like o isn’t that much of a bottleneck in performance; it’s a fairly straightforward process.  Using a command like POINT is often quite slow, as it has a lot of internal error checking, and value translating going on.  You’ll see different amounts of improvement for various commands/functions which you can swap out to _MEM usage, so it’s hard to say how much you can improve performance with any single change.

_MEMGET (m, m.OFFSET + o, _UNSIGNED LONG) <— This is the equivalent of a POINT(x, y) command, so even by looking up the value of o and adding it to m.OFFSET, it’s still much faster than looking up x, looking up y, looking at _SOURCE, checking to make certain everything is valid, looking at the point value, reading it as a _FLOAT, and then converting it to an _UNSIGNED LONG when returning it...

Now, would it have been even faster to use _MEMPUT/_MEMGET instead of o?

Probably.  I doubt it’d be much faster, but it probably would end up slightly faster.  The question is, would it be worth the loss to readability and increase in coding time and complexity to replace it?

Instead of the above, we’d see things like: (where we DIM m2 AS _MEM: m2 = _MEMNEW(4), rather than using o)

_MEMGET (m, m.OFFSET + _MEMGET(m2, m2.OFFSET, LONG), LONG)

Then, instead of o = o + 4, we’d have:

_MEMPUT m2, m2.OFFSET, (_MEMGET(m2, m2.OFFSET, LONG) +4) AS LONG

It might be a nanosecond faster executing, but is it worth it for the increase in coding time and complexity of understanding the code?

Only the programmer can answer that question, and the answer might vary for different programs.  (Or even different processes inside a program.)  You’d want a recursive SORT process to run as fast as possible, as it can definitely become a program bottleneck, but a process which only adds 1 to each element of an Array isn’t going to be much of a concern.

It’s a trade off: Speed vs readability and complexity, with increased coding time.  Onlyyou can determine when it’s worth the trouble for your program’s needs.  ;)
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline Cobalt

  • QB64 Developer
  • Forum Resident
  • Posts: 878
  • At 60 I become highly radioactive!
    • View Profile
Re: Steve's Video Tutorials
« Reply #53 on: September 30, 2019, 10:38:00 am »
memory involves addressing individual bytes in a block of memory, each byte being 256 bits (0-255).  And so we're only ever talking about

Just wanted to say, each BYTE is 8bits, capable of representing a VALUE of 0-255.

You're making your computer out to be smarter than it really is.  ;)

Yep, Computers can only do what they are told and at the fundamental level they only know ON and OFF (1 an 0)[I think you can say they only know voltage hi(+5v~) and voltage lo(~0.5v].

That is why we are supposed to be the USER and not the other way around. And Knowledge is Power!
Granted after becoming radioactive I only have a half-life!

Offline Qwerkey

  • Forum Resident
  • Posts: 755
    • View Profile
Re: Steve's Video Tutorials
« Reply #54 on: October 01, 2019, 09:29:44 am »
"_MEM processing is 100's of times faster than with standard variable processing".  This is what the gurus said.

So, before plummeting into adapting my two programs (Crossword Generator and Gravitation Simulator) which use a lot of processing, I thought that I'd check out this statement.

Code Trial 1A loads a file from disk into a _MEM object and then intensely manipulates that _MEM object (swaps bytes around).  In my core-I5 processor this uses a whole processor's-worth of CPU and the process takes 69.1s.  With $CHECKING:OFF it takes 25.4s.

Code Trial 1B loads the same file into a string, and the same manipulation as in Trial 1A is performed using standard techniques (no _MEM).  The process takes 253s.  So the _MEM processing is faster, but not by a large factor.

So I thought that maybe the process in Trial 1A was slowed down by the presence of standard variables to do the indexing.  Code Trial 1C is as 1A but with everything put into _MEM.  I was shocked to find that this took 157s.

An invariant maxim is "never believe gurus unless you know what they say is true" (particularly pertinent to mythologies, of course).

So, either the gurus were wrong or I don't know what I'm doing, which as all members will readily attest is totally without foundation.

Tria 1A
Code: QB64: [Select]
  1. 'Trial 1A _MEM processing, Variable Indexing
  2.  
  3. 'Load from file into _MEM object
  4. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
  5. FIELD #1, 4 AS ULoc$
  6.  
  7. NoWords& = LOF(1) / 4
  8.  
  9. M = _MEMNEW(LOF(1))
  10. PRINT M.SIZE '/ 2147483647
  11.  
  12. DIM MDum AS _MEM
  13. MDum = _MEMNEW(1)
  14.  
  15.  
  16. FOR N& = 1 TO LOF(1) / 4
  17.     GET #1, N&
  18.     _MEMPUT M, M.OFFSET + ((N& - 1) * 4), ULoc$
  19. NEXT N&
  20.  
  21.  
  22. 'Now do some manipulations in _MEM
  23.  
  24. Start! = TIMER
  25.  
  26. FOR N& = 1 TO 100000
  27.     'for K& = 0 to  M.SIZE -1  !!! Not allowed
  28.     FOR K& = 0 TO 4 * NoWords& - 1
  29.         _MEMPUT MDum, MDum.OFFSET, _MEMGET(M, M.OFFSET + (4 * NoWords& - 1 - K&), _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
  30.         _MEMPUT M, M.OFFSET + (4 * NoWords& - 1 - K&), _MEMGET(M, M.OFFSET + K&, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
  31.         _MEMPUT M, M.OFFSET + K&, _MEMGET(MDum, MDum.OFFSET, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
  32.     NEXT K&
  33. NEXT N&
  34.  
  35. PRINT TIMER - Start!
  36.  
  37.  

Trial 1B
Code: QB64: [Select]
  1. 'Trial 1B Variable processing, Variable Indexing
  2.  
  3. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
  4. FIELD #1, 4 AS ULoc$
  5.  
  6. NoWords& = LOF(1) / 4
  7.  
  8. S$ = ""
  9. FOR N& = 1 TO LOF(1) / 4
  10.     GET #1, N&
  11.     S$ = S$ + ULoc$
  12. NEXT N&
  13.  
  14.  
  15.  
  16. 'Now do some manipulations with the array - same as Trial 1A
  17.  
  18. Start! = TIMER
  19.  
  20. FOR N& = 1 TO 100000
  21.     FOR K& = 1 TO LEN(S$)
  22.         Dum$ = MID$(S$, K&, 1)
  23.         MID$(S$, K&, 1) = MID$(S$, LEN(S$) - K&, 1)
  24.         MID$(S$, LEN(S$) - K&, 1) = Dum$
  25.     NEXT K&
  26. NEXT N&
  27.  
  28. PRINT TIMER - Start!
  29.  
  30.  

Trial 1C
Code: QB64: [Select]
  1. 'Trial 1C _MEM processing, _MEM Indexing
  2.  
  3. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
  4. FIELD #1, 4 AS ULoc$
  5.  
  6. NoWords& = LOF(1) / 4
  7.  
  8. 'M Holds the data
  9. M = _MEMNEW(LOF(1))
  10. PRINT M.SIZE
  11.  
  12. 'MSize 4-byte _MEM holds size of M
  13. DIM MSize AS _MEM
  14. MSize = _MEMNEW(4)
  15. _MEMPUT MSize, MSize.OFFSET, LOF(1) AS _UNSIGNED LONG
  16. PRINT _MEMGET(MSize, MSize.OFFSET, _UNSIGNED LONG)
  17.  
  18. 'MDum 1-byte _MEM used to temporarily hold swapped byte
  19. DIM MDum AS _MEM
  20. MDum = _MEMNEW(1)
  21.  
  22. '$CHECKING:OFF
  23.  
  24. FOR N& = 1 TO LOF(1) / 4
  25.     GET #1, N&
  26.     _MEMPUT M, M.OFFSET + ((N& - 1) * 4), ULoc$
  27. NEXT N&
  28.  
  29.  
  30. 'N 4-byte _MEM to index count
  31. N = _MEMNEW(4)
  32. _MEMPUT N, N.OFFSET, 0 AS _UNSIGNED LONG
  33.  
  34. PRINT _MEMGET(N, N.OFFSET, _UNSIGNED LONG)
  35.  
  36. 'K 4-byte _MEM used to index swap
  37. K = _MEMNEW(4)
  38. _MEMPUT K, K.OFFSET, 0 AS _UNSIGNED LONG
  39.  
  40. PRINT _MEMGET(K, K.OFFSET, _UNSIGNED LONG)
  41.  
  42. 'Now do some manipulations in _MEM
  43.  
  44. Start! = TIMER
  45.  
  46. WHILE _MEMGET(N, N.OFFSET, _UNSIGNED LONG) < 100000 'Don't know how to do FOR/NEXT with _MEM objects
  47.  
  48.     _MEMPUT K, K.OFFSET, 0 AS _UNSIGNED LONG
  49.     WHILE _MEMGET(K, K.OFFSET, _UNSIGNED LONG) < _MEMGET(MSize, MSize.OFFSET, _UNSIGNED LONG)
  50.  
  51.         _MEMPUT MDum, MDum.OFFSET, _MEMGET(M, M.OFFSET + (_MEMGET(MSize, MSize.OFFSET, _UNSIGNED LONG) - _MEMGET(K, K.OFFSET, _UNSIGNED LONG) - 1), _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
  52.         _MEMPUT M, M.OFFSET + (_MEMGET(MSize, MSize.OFFSET, _UNSIGNED LONG) - _MEMGET(K, K.OFFSET, _UNSIGNED LONG) - 1), _MEMGET(M, M.OFFSET + _MEMGET(K, K.OFFSET, _UNSIGNED LONG), _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
  53.         _MEMPUT M, M.OFFSET + _MEMGET(K, K.OFFSET, _UNSIGNED LONG), _MEMGET(MDum, MDum.OFFSET, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
  54.  
  55.         _MEMPUT K, K.OFFSET, _MEMGET(K, K.OFFSET, _UNSIGNED LONG) + 1 AS _UNSIGNED LONG
  56.  
  57.     WEND
  58.  
  59.     _MEMPUT N, N.OFFSET, _MEMGET(N, N.OFFSET, _UNSIGNED LONG) + 1 AS _UNSIGNED LONG
  60.  
  61.  
  62. PRINT TIMER - Start!
  63.  
  64. _MEMFREE MSize
  65.  
  66.  
* 4UKEng.rnd (Filesize: 11.6 KB, Downloads: 177)
« Last Edit: October 01, 2019, 10:32:35 am by Qwerkey »

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Steve's Video Tutorials
« Reply #55 on: October 01, 2019, 10:14:41 am »
Quote
With $CHECKING:OFF it takes 25.4s....

...The process takes 253s.  So the _MEM processing is faster, but not by a large factor.

From 253 seconds to 25.4 seconds is 10x faster already, and that’s not a “large factor”?

Also, don’t forget, this guru also told you:
Quote
You’ll see different amounts of improvement for various commands/functions which you can swap out to _MEM usage, so it’s hard to say how much you can improve performance with any single change.

And, I’m thinking that you can tweek that code to speed it up more, with just a little work.

M.OFFSET + (4 * NoWords& - 1 - K&) <— there’s a good bit of calculating going on in this statement, and it’s used multiple times.  Assigning it once to a different offset would probably be faster....

And, unless I’m misreading what the code is supposed to do, you’re just swapping contents of your strings?

  FOR K& = 1 TO LEN(S$)
        Dum$ = MID$(S$, K&, 1)  <— a temp$
        MID$(S$, K&, 1) = MID$(S$, LEN(S$) - K&, 1) <— move info from one spot to another
        MID$(S$, LEN(S$) - K&, 1) = Dum$ <— place info back into that original spot from the temp$
    NEXT K&

So far, you’re seeing an improvement of 10x to the time, and you still haven’t learned or used _MEMCOPY, Which is the optimized way to do this sort of process.  http://www.qb64.org/wiki/MEMCOPY

https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline Qwerkey

  • Forum Resident
  • Posts: 755
    • View Profile
Re: Steve's Video Tutorials
« Reply #56 on: October 01, 2019, 10:35:00 am »
O Great Master, wow!  That was quick!  I only just popped out for drink.

No, you're not misreading what I was doing in the trial and yes I haven't managed to come across _MEMCOPY (will look into it).  But I just wanted to compare exactly the same process without / with _MEM in the trial.

M.OFFSET + (4 * NoWords& - 1 - K&) <— there’s a good bit of calculating going on in this statement, and it’s used multiple times.  Assigning it once to a different offset would probably be faster....

In Trial 1C, I replaced that calculated stuff with a constant _MEM object and this trial was very much slower.  Maybe I just did something wrong there.

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Steve's Video Tutorials
« Reply #57 on: October 01, 2019, 01:58:46 pm »
Just curious; How's this little version perform for you?

Code: QB64: [Select]
  1. 'Trial 1A _MEM processing, Variable Indexing
  2.  
  3. 'Load from file into _MEM object
  4. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
  5. FIELD #1, 4 AS ULoc$
  6.  
  7. NoWords& = LOF(1) / 4
  8.  
  9. M = _MEMNEW(LOF(1))
  10. PRINT M.SIZE '/ 2147483647
  11.  
  12. DIM MDum AS _MEM
  13. MDum = _MEMNEW(1)
  14.  
  15.  
  16. FOR N& = 1 TO LOF(1) / 4
  17.     GET #1, N&
  18.     _MEMPUT M, M.OFFSET + ((N& - 1) * 4), ULoc$
  19. NEXT N&
  20.  
  21.  
  22. 'Now do some manipulations in _MEM
  23.  
  24. DIM Precalc AS _OFFSET
  25.  
  26. Start! = TIMER
  27. Precalc = M.OFFSET + 4 * NoWords& - 1 'Move as much of your math outside the loop as possible
  28.  
  29. FOR N& = 1 TO 100000
  30.     'for K& = 0 to  M.SIZE -1  !!! Not allowed
  31.     'You can't mix OFFSETs with other values, as above with that FOR statement.
  32.     'Instead, convert it into a simple DO sequence, such as below.
  33.     K& = 0
  34.     DO
  35.         _MEMPUT MDum, MDum.OFFSET, _MEMGET(M, Precalc - K&, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
  36.         _MEMPUT M, Precalc - K&, _MEMGET(M, M.OFFSET + K&, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
  37.         _MEMPUT M, M.OFFSET + K&, _MEMGET(MDum, MDum.OFFSET, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
  38.         K& = K& + 1
  39.     LOOP UNTIL K& >= M.SIZE
  40. NEXT N&
  41. PRINT TIMER - Start!
  42.  

You mentioned that it was taking you 25.4s to run the code.  All I've done here is precalculate the math and simplify the calculations inside the loop a bit, in an effort to showcase how much of a difference they can make sometimes.  On my PC, the runtime was about 13 seconds, and after simplifying the math by calculating the values that won't change outside the loop, it dropped runtime down to about 11 seconds.  A 15% performance increase, with a very minor alteration completely unrelated to our mem commands at all.  ;)

If it affects your PC with a similar change, it should reduce times down from 25 seconds to what?  About 21 seconds?
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Steve's Video Tutorials
« Reply #58 on: October 01, 2019, 02:18:33 pm »
And, completely unrelated to the _MEM commands, I thought I'd take a moment and also showcase something else to help highlight just exactly how SLOOOOOOOOW the QB64 string commands are.

First, run this piece of your original code and time it.  (Take a nap, it takes a while, coming in at almost 3 minutes to execute on my PC.)

This is the exact same as TRIAL 1B above:

Code: QB64: [Select]
  1. 'Trial 1B Variable processing, Variable Indexing
  2.  
  3. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
  4. FIELD #1, 4 AS ULoc$
  5.  
  6. NoWords& = LOF(1) / 4
  7.  
  8. S$ = ""
  9. FOR N& = 1 TO LOF(1) / 4
  10.     GET #1, N&
  11.     S$ = S$ + ULoc$
  12. NEXT N&
  13.  
  14.  
  15.  
  16. 'Now do some manipulations with the array - same as Trial 1A
  17.  
  18. Start! = TIMER
  19.  
  20. FOR N& = 1 TO 100000
  21.     FOR K& = 1 TO LEN(S$)
  22.         Dum$ = MID$(S$, K&, 1)
  23.         MID$(S$, K&, 1) = MID$(S$, LEN(S$) - K&, 1)
  24.         MID$(S$, LEN(S$) - K&, 1) = Dum$
  25.     NEXT K&
  26. NEXT N&
  27.  
  28. PRINT TIMER - Start!
  29.  

Now, I'm going to rewrite the routine to work with numeric values and ASCII values, rather than with strings.  Compile and test the following for its runtime:

Code: QB64: [Select]
  1. 'Trial 1B Variable processing, Variable Indexing
  2.  
  3. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
  4. FIELD #1, 4 AS ULoc$
  5.  
  6. NoWords& = LOF(1) / 4
  7.  
  8. S$ = ""
  9. FOR N& = 1 TO LOF(1) / 4
  10.     GET #1, N&
  11.     S$ = S$ + ULoc$
  12. NEXT N&
  13.  
  14.  
  15.  
  16. 'Now do some manipulations with the array - same as Trial 1A
  17.  
  18. L = LEN(S$) + 1
  19.  
  20. Start! = TIMER
  21.  
  22. FOR N& = 1 TO 100000
  23.     FOR K& = 1 TO LEN(S$)
  24.         'Dum$ = MID$(S$, K&, 1)
  25.         B = ASC(S$, K&)
  26.         'MID$(S$, K&, 1) = MID$(S$, LEN(S$) - K&, 1)
  27.         ASC(S$, K&) = ASC(S$, L - K&)
  28.         'MID$(S$, LEN(S$) - K&, 1) = Dum$
  29.         ASC(S$, L - K&) = B
  30.     NEXT K&
  31. NEXT N&
  32.  
  33. PRINT TIMER - Start!
  34.  

Now, just by swapping over to ASC, instead of MID$, the runtime on my PC has now dropped down to less than 1 minute. (56 seconds.)

From 187 seconds (MID$) to 56 seconds (ASC), to 11 seconds (_MEM)...

Which is why I said it's hard to tell folks, "You're going to always see a performance boost of XX percent with _MEM."  Various commands have different levels of overhead to them, so it's impossible to say how much of a performance boost you'll see, until you can actually compare routines side-by-side. 

In this case, the original code took 187 seconds, the _MEM version took 11 seconds, so that's a performance increase by 17x...  Which, I think really helps to highlight exactly why it's nice for QB64 programmers to have the ability to use _MEM inside their code.  ;)
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline Dimster

  • Forum Resident
  • Posts: 500
    • View Profile
Re: Steve's Video Tutorials
« Reply #59 on: October 01, 2019, 04:02:24 pm »
Steve, in your _MEMPUT examples you have been using whole numbers - ie _MEMPUT m, m.OFFSET, 986543.
Would _MEMPUT also calculate the value? for example _MEMPUT m, m.OFFSET, 98000 + 6543 

or is it better coding practice to have the value calculated before the PUT such as  V1 = 98000 + 6543 : _MEMPUT m, m.OFFSET, V1 ?
 
Really liking your videos : finished #3 with popcorn to spare.