Steve's Video Tutorials

[Qwerkey]

Technical Query No. 2

Steve, you have been teaching us dim folk what's happening with _MEM, and I now begin to see that _PUTting and _GETting to/from memory involves addressing individual bytes in a block of memory, each byte being 256 bits (0-255).  And so we're only ever talking about numbers: the hardware never uses letters, of course, even if we're storing a string.  So, when we store, say, a 4-letter string in _MEM this is four bytes next to each other, each byte storing the ASCII value of each letter.

I hope that I haven't got even this simple idea wrong!

Using your first video and adapting your code, I am now able to directly read a hard disk stored dictionary file into _MEM, and I have used the following code:

Code: QB64: [Select]

1. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
2. FIELD #1, 4 AS ULoc$
3. NoWords& = LOF(1) / 4
4.  
5. DIM M AS _MEM
6. M = _MEMNEW(LOF(1))
7.  
8. $CHECKING:OFF
9.  
10. FOR N& = 1 TO LOF(1) / 4
11.     GET #1, N&
12.     _MEMPUT M, M.OFFSET + ((N& - 1) * 4), ULoc$
13. NEXT N&
14.  
15. temp$ = SPACE$(4)
16. _MEMGET M, M.OFFSET, temp$
17. PRINT temp$
18.  
19. _MEMGET M, M.OFFSET + 4, temp$
20. PRINT temp$
21.  
22. _MEMGET M, M.OFFSET + 8, temp$
23. PRINT temp$
24.  
25. _MEMGET M, M.OFFSET + (NoWords& - 1) * 4, temp$
26. PRINT temp$
27.  
28. _MEMGET M, M.OFFSET + M.SIZE - 4, temp$
29. PRINT temp$
30.  
31. _MEMGET M, M.OFFSET + M.SIZE - 8, temp$
32. PRINT temp$
33.  
34. _MEMGET M, M.OFFSET + LOF(1) - 8, temp$
35. PRINT temp$
36.  
37. temp$ = SPACE$(5)
38. _MEMGET M, M.OFFSET + M.SIZE - 5, temp$
39. PRINT temp$
40.  
41. _MEMFREE M

NB This code is not usable: I have not supplied the file from which the data are read.  It is illustrative only.

What surprises me is that nowhere in the code have I had to tell the computer to convert individual letters into their ASCII code.

In the statement DIM M AS _MEM, there is no indication that strings are going to be used (as opposed to numbers next to each other).

And in the statement _MEMPUT M, M.OFFSET + ((N& - 1) * 4), ULoc$, we are able to _PUT a string without converting to numbers.  And in the statement _MEMGET M, M.OFFSET, temp$, the block of numbers is converted back to a string without having to code such an action.

Is it therefore the case that when you _MEMPUT or _MEMGET and your code has a string, QB64 just does the string to numeric byte conversions?  I feel that this must be so, and if using "normal" string assignment, storing and retrieving, QB64 is always doing these conversions without the coder knowing anything about it.

I think that what I'm saying is that I'd have expected the _MEM dimensioning statement to be something like DIM M AS _MEM AS STRING, just to let the computer know that this will deal with strings, not numbers.

[Qwerkey]

Steve, I think that I've managed to answer my own question.  I tried adding 1 to one of the _MEM bytes of previous reply, and indeed the character (was "B") changed to "C".

Code:
_MEMPUT M, M.OFFSET + 1, _MEMGET(M, M.OFFSET + 1, _UNSIGNED _BYTE) + 1 AS _UNSIGNED _BYTE

[SMcNeill]

I think that what I'm saying is that I'd have expected the _MEM dimensioning statement to be something like DIM M AS _MEM AS STRING, just to let the computer know that this will deal with strings, not numbers.

You're making your computer out to be smarter than it really is.  ;)

When teaching my daughter how to code, the very first lesson I tried to stress with her is that computers aren't very smart -- no matter what people commonly think, nowadays.  Computers aren't complicated because they're HARD; they're complicated because they're TOO SIMPLE.  No matter what you think the computer is doing, all it's actually doing is adding or comparing 0s and 1s.  EVERYTHING, in the end, breaks down to bits which are either ON or OFF.  (-1 OR 0, as we represent them in QB64.)

The simple truth is: A computer doesn't use any numbers, nor any characters, nor strings.  All your computer knows is ON and OFF state of bits.  It's up to the PROGRAMMER to define what a sequence of bits represent -- either an integer (in various sizes), a real number (again, in various precision sizes), or a character/string.

As I was trying to stress in the first tutorial, _MEM is inherently nothing more than opening a file for binary access.  Tell me, would you ever think of doing the following:

OPEN "temp.txt" FOR BINARY AS STRING , FILENUMBER 1

That OPEN statement simply says, "We want to access a storage area on the hard drive, and read and write data to it."  We don't tell it that we want to work with strings with it, or numbers, or a strange combination of the two.  We just issue a command that says, "We want to use a segment of the drive, call it "temp.txt", and get and put data to that segment."

_MEM, is more-or-less, a direct mirror to those BINARY access commands.  Just as OPEN is the command to set the stage to interact with DATA on the hard drive, DIM m AS _MEM is doing the same job and setting the stage for us to interact with DATA somewhere in memory.

Let me give you a quick example to run and study, which helps highlight the issue:

Code: QB64: [Select]

1. OPEN "temp.txt" FOR OUTPUT AS #1 'This just opens a new file, erasing any old file with the same name.
2. CLOSE #1
3. OPEN "temp.txt" FOR BINARY AS #1 'And this now opens that blank file so I can get/put information to it.
4. text$ = "1234567890" 'a simple 10 character text string.
5. PUT #1, 1, text$ 'which I'm now writting to the disk.
6.  
7. 'All very common practice, and nothing strange or mysterious going on so far.
8. 'I opened a file, I put a string into it.
9. 'But now....
10.  
11. 'Let's start by making certain that the file contains the data I want:
12. in$ = SPACE$(10) ' I want a string 10 bytes long, so I can get all 10 bytes at once
13. GET #1, 1, in$ 'which I get from the drive
14. PRINT in$ 'and then print to see my string and make certain that I did, indeed, place string information to the drive.
15.  
16. SLEEP 'at this point, run the program and verify what the result is for yourself.
17. 'Exit the program after the first pause (Where the SLEEP statement would appear), and then come back and follow the code
18. 'as I comment it below.
19.  
20. 'READY TO CONTINUE??
21.  
22.  
23.  
24. 'Good!
25.  
26. 'At this point, you can now see that we wrote a string to the drive...
27. 'OR...
28. 'Did we really???
29.  
30. 'Let's test a few things:
31.  
32. DIM b AS _BYTE, i AS INTEGER, l AS LONG, i64 AS _INTEGER64 'let's first define a few variables as various types
33. DIM s AS SINGLE, d AS DOUBLE, f AS _FLOAT, c AS STRING * 1 'c for character, otherwise all our variable names are...
34. '        just the first letter of what they represent (with the exception of i64 for integer64, as i stands for integer)
35.  
36. GET #1, 1, b 'get a byte from the start of that file
37. GET #1, 1, i 'get an integer from the start of that file
38. GET #1, 1, l 'a long
39. GET #1, 1, i64 'an integer64
40. GET #1, 1, s 'a single
41. GET #1, 1, d 'a double
42. GET #1, 1, f ' a float
43. GET #1, 1, c 'and finish with a character
44.  
45. 'Now surely, since we wrote the data as a string, these commands are going to fail and toss us an error.  Right??
46.  
47. 'Let's try and print them, then run it and let's see!!
48. PRINT b, i, l, i64
49. PRINT s, d, f
50. PRINT c 'the character will print to the 3rd line, as it's the letter "1" and I don't want it lost in the numbers above.
51.  
52. SLEEP 'another pause, like before, so we can stop running the code at the proper spot as we study this example code.
53. 'NOW, RUN THE PROGRAM AGAIN, THIS TIME STOPPING IT AT THE SECOND PAUSE (SLEEP) SEGMENT.
54.  
55. 'Executed the code, yet?
56.  
57. 'If so, were you shocked by the results?
58.  
59. 'We wrote string data to the drive, but then somehow we managed to get it all back as numeric values???
60. 'WTH is up with that??
61.  
62.  
63. 'It's simply because it's as I was saying in my post over at the QB64 forums -- computers are much stupider than folks believe.
64. 'All the computer does is set bits on the hard drive to be either ON or OFF, and then we GET and PUT those bits into specific
65. 'areas of the hardrive with our OPEN and GET/PUT statement.
66.  
67. 'OPEN defines a block of the drive that we're going to use...In this case from where "temp.txt" starts, to where "temp.txt"
68. 'ends at.  I have no real idea where the heck that segment is; I trust the open statement to find the right spot on the
69. 'drive and then make that access available for me.
70.  
71. 'Then, once that segment of the drive is made available, I futher specify which area of that segment that I want to interact
72. 'with, with the GET and PUT statement.
73. 'GET FILEHANDLE, POSITION, INFORMATION....
74. 'Or, as we commonly see it: GET #1, 1, variable  (for the first byte of that file)
75.  
76. 'It's by defining our variable types that our computer knows how to interpet those 0 and 1 bits that we want to read/write
77. 'to the drive.
78.  
79. 'text$ = "1234567890"  <--this tells the computer to convert those 10 characters into the 0 and 1 representations of string values.
80. 'DIM b as _BYTE <-- this tells the computer that those 0s and 1s are going to represent an integer value from -128 to 127
81.  
82. 'The hard drive is ONLY storing bits which are either ON or OFF (0 or 1).
83. 'It doesn't care what we put in our files -- it doesn't know the difference between a string or a number at all.
84. 'All our hard drive knows is "This bit here is a 0.  This bit after it is a 1.  The next bit is a 0.  Then a 1.  Another 1...."
85.  
86. 'MEM is the *EXACT* same thing, as I'll illustrate below:
87.  
88. DIM M AS _MEM 'just like that OPEN command, we're now preparing to open a segment of memory
89. M = _MEMNEW(10) 'in this case, I'm deciding it only needs to be 10 bytes of memory.
90.  
91. _MEMPUT M, M.OFFSET, text$ 'And, if you still remember our original text$, I just put it in memory, in that memblock.
92. in2$ = SPACE$(10) 'and here I'm setting a second in$ so I can get and print that information to the screen
93. _MEMGET M, M.OFFSET, in2$ 'and I just got in2$ from that spot in memory.
94.  
95. 'Now, I don't think there's any real reason for a third SLEEP statement in this demo, so I'm just going to do as I did
96. 'above and _MEMGET and _MEMPUT that same information into a series of variables:
97.  
98. DIM b2 AS _BYTE, i2 AS INTEGER, l2 AS LONG, i642 AS _INTEGER64 'the 2 here says they're all new variables, unrelated
99. DIM s2 AS SINGLE, d2 AS DOUBLE, f2 AS _FLOAT, c2 AS STRING * 1 'to the ones which I used before.
100.  
101. _MEMGET M, M.OFFSET, b2 'get a byte from the start of that mem block
102. _MEMGET M, M.OFFSET, i2 'an integer
103. _MEMGET M, M.OFFSET, l2 'long
104. _MEMGET M, M.OFFSET, i642 'integer64
105. _MEMGET M, M.OFFSET, s2 'single
106. _MEMGET M, M.OFFSET, d2 'double
107. $CHECKING:OFF
108. 'I'm turning error checking off here, as the next line would technically toss an error for us.
109. '"WHY," you ask?
110. 'Because we only have 10 bytes reserved for memory and a _FLOAT requires 32 of them.  (They're a HUGE variable type.)
111. 'You can't make a blanket from a tablecloth, and you can't make a float from 10 bytes.
112. '(Well, at least you can't, without tossing an error -- which I'm just blindly ignoring for this demo and which
113. 'COULD cause some serious corruption in my program if I wasn't careful. After all, who knows what variable/data is stored
114. 'in the bytes immediately after the 10 where my memblock is mysteriously located??)
115. _MEMGET M, M.OFFSET, f2 'float
116. $CHECKING:ON
117. 'Lesson to take from this:  ONLY USE CHECKING:OFF when you either: A) Know your code is 100% correct
118. 'OR                                                                B) You seriously don't give a damn if you corrupt your data.
119. _MEMGET M, M.OFFSET, c2 'and finally, let's get that last character
120.  
121. PRINT b2, i2, l2, i642 'and now print the results to the screen
122. PRINT s2, d2, f2
123. PRINT c2
124.  
125. 'RUN AND COMPARE WHAT YOU SEE......
126.  
127. '.
128. '..
129. '...
130.  
131. 'No difference in the outputs?
132.  
133.  
134. 'Shocked??
135.  
136. 'If so, then don't be.  As I was stressing in my first video: _MEM is nothing more complex than working with a file opened
137. 'FOR BINARY access.  GET/PUT gets and puts sequences of 0s and 1s onto our hard drive.
138. '                _MEMGET/_MEMPUT does the exact same thing, except it does it somewhere in memory.
139.  
140. 'Now, as to whether that sequence of 0s and 1s is going to be integers, real numbers, strings, or some user-defined type
141. 'which is a combination of all three...
142. 'Our memory doesn't care about any of that!  All it does is faithfully hold a sequence of 0s and 1s in a specific order.
143.  
144. 'It's up to us to define how we want to interpet that sequence of 0s and 1s.
145. 'And we do that either via how we define our variable types:  DIM b AS _BYTE, i AS INTEGER
146. 'Or we specify it specifically: _MEMPUT m, m.offset, 123 AS _BYTE
147.  
148. 'Make sense now?
149. 'Hopefully, the above little demo will clear up all your confusion for you.
150. 'If not, just ask once again and let me know what/where you're lost, and I'll try and help you understand what's going on.  ;)
151.  

Study the code above and see if it helps answer your query for you.  ;)

[SMcNeill]

Steve, I think that I've managed to answer my own question.  I tried adding 1 to one of the _MEM bytes of previous reply, and indeed the character (was "B") changed to "C".

Code:
_MEMPUT M, M.OFFSET + 1, _MEMGET(M, M.OFFSET + 1, _UNSIGNED _BYTE) + 1 AS _UNSIGNED _BYTE

And this is often the very best way to learn these commands:  Simply test them out and try various things with them and see how they behave.  Trust that you can't actually break anything -- QB64 and your OS protects you from any serious issues such as destroying your computer.  About the worst you can do is make your program crash, and can you name ANY programmer who hasn't already done that a couple of thousand times in their career?? 

When we crash, we just have to go in and debug whatever the hell went wrong.  ;D

The best way to learn any new command in a programming language is simply to use it and learn from trial and error.  If it doesn't behave as you expect it should, ask about it and then you'll either:
A) Have someone explain to you what's happening and why it happened
OR...
B) You find out that you discovered a bug in the command (and Lord knows, we have plenty of those), which can be addressed and hopefully corrected in a later version of the software. 

[Qwerkey]

Thanks Steve.  I went through your extra program and with my trial-and-error am begining to to get to grips with it.  I'm still used to coding BASIC where you never have to think how to handle individual bytes.  No wonder they made the acronym BASIC: for the people who don't actually know what they're doing!  The story of my life, but, hey, I'm happy with that.

[Qwerkey]

Next dim question (I've got a million of them!).  As I mentioned above, I have the intention to produce a _MEM version of my Crossword Generator program which uses heaps of processing to produce a correctly filled crossword grid.  Am I right in thinking that when I want to have fastest processing, all things that were "normal" variables get replaced by _MEM objects?  So, for example

Code: QB64: [Select]

1. FOR N& = 1 to 1000000000
2.         Lots of _MEM processing
3. NEXT N&

even the N& variable would (somehow) be replaced by a _MEM object?

Or, using Steve's code in reply #21:

Code: QB64: [Select]

1. DO
2.     IF _MEMGET(m, m.OFFSET + o, _UNSIGNED LONG) = _RGBA32(255, 255, 255, 255) THEN
3.         _MEMPUT m, m.OFFSET + o, 256 AS _UNSIGNED _BYTE
4.     END IF
5.     o = o + 4
6. LOOP UNTIL o >= m.SIZE - 4

should the variable o be replaced by a _MEM object to speed things up?  I'm confused by the mixing of "normal" variables (slow) and _MEM objects (fast).

Maybe things such as this will become clearly when Steve has completed the library of _MEM tutorials.

[Qwerkey]

Default values of memory blocks.

With the code here, I was surprised to find that without line 11, the subsequent WHILE/WEND loop did not execute.

What I was trying to do was to index the memory block byte position of object M with a one-byte object MCount whose value is increased from the starting value by one at each loop cycle.

I originally found that the MCount starting value was 23 and on other runs it was 21.  So line 11 sets the starting value of MCount to be zero.  Thinking that it would behave like a variable, I assumed that its starting value be zero.  But as the program allocates a block of memory, the starting value is just what is in that part of the (unused) memory at that time???

Code: QB64: [Select]

1. DIM M AS _MEM
2. M = _MEMNEW(20)
3.  
4. Junk$ = "ABCDEFGHIJKLMNOPQRST"
5.  
6. _MEMPUT M, M.OFFSET, Junk$
7.  
8. DIM MCount AS _MEM
9. MCount = _MEMNEW(1)
10.  
11. _MEMPUT MCount, MCount.OFFSET, 0 AS _UNSIGNED _BYTE
12.  
13. PRINT _MEMGET(MCount, MCount.OFFSET, _UNSIGNED _BYTE)
14.  
15. WHILE _MEMGET(MCount, MCount.OFFSET, _UNSIGNED _BYTE) < 20
16.  
17.     PRINT _MEMGET(MCount, MCount.OFFSET, _UNSIGNED _BYTE); _MEMGET(M, M.OFFSET + _MEMGET(MCount, MCount.OFFSET, _UNSIGNED _BYTE), STRING * 1)
18.  
19.     _MEMPUT MCount, MCount.OFFSET, _MEMGET(MCount, MCount.OFFSET, _UNSIGNED _BYTE) + 1 AS _UNSIGNED _BYTE
20.  
21. WEND
22.  
23. _MEMFREE M
24. _MEMFREE MCount
25.  

Oh, by the way, mucking around with actual blocks of memory is real fun!

A further aside:  Steve, you have trouble remembering that the command is _MEMFREE rather than _FREEMEM (cf. _FREEIMAGE etc.).  Here's a little aide memoire for you, but you'll need to be familiar with the British TV comedy programme "Are You Being Served?".  I don't suppose that this ever reached the US, but if it did and you ever saw it, just remember the Mr Humphries character and his catchphrase - I'm Free! (to be said a very camp fashion).  And then keep saying "Mem Free!".  I cannot ever get this command wrong.

[SMcNeill]

Next dim question (I've got a million of them!).  As I mentioned above, I have the intention to produce a _MEM version of my Crossword Generator program which uses heaps of processing to produce a correctly filled crossword grid.  Am I right in thinking that when I want to have fastest processing, all things that were "normal" variables get replaced by _MEM objects?  So, for example

Code: QB64: [Select]

1. FOR N& = 1 to 1000000000
2.         Lots of _MEM processing
3. NEXT N&

even the N& variable would (somehow) be replaced by a _MEM object?

Or, using Steve's code in reply #21:

Code: QB64: [Select]

1. DO
2.     IF _MEMGET(m, m.OFFSET + o, _UNSIGNED LONG) = _RGBA32(255, 255, 255, 255) THEN
3.         _MEMPUT m, m.OFFSET + o, 256 AS _UNSIGNED _BYTE
4.     END IF
5.     o = o + 4
6. LOOP UNTIL o >= m.SIZE - 4

should the variable o be replaced by a _MEM object to speed things up?  I'm confused by the mixing of "normal" variables (slow) and _MEM objects (fast).

Maybe things such as this will become clearly when Steve has completed the library of _MEM tutorials.

Directly accessing a variable like o isn’t that much of a bottleneck in performance; it’s a fairly straightforward process.  Using a command like POINT is often quite slow, as it has a lot of internal error checking, and value translating going on.  You’ll see different amounts of improvement for various commands/functions which you can swap out to _MEM usage, so it’s hard to say how much you can improve performance with any single change.

_MEMGET (m, m.OFFSET + o, _UNSIGNED LONG) <— This is the equivalent of a POINT(x, y) command, so even by looking up the value of o and adding it to m.OFFSET, it’s still much faster than looking up x, looking up y, looking at _SOURCE, checking to make certain everything is valid, looking at the point value, reading it as a _FLOAT, and then converting it to an _UNSIGNED LONG when returning it...

Now, would it have been even faster to use _MEMPUT/_MEMGET instead of o?

Probably.  I doubt it’d be much faster, but it probably would end up slightly faster.  The question is, would it be worth the loss to readability and increase in coding time and complexity to replace it?

Instead of the above, we’d see things like: (where we DIM m2 AS _MEM: m2 = _MEMNEW(4), rather than using o)

_MEMGET (m, m.OFFSET + _MEMGET(m2, m2.OFFSET, LONG), LONG)

Then, instead of o = o + 4, we’d have:

_MEMPUT m2, m2.OFFSET, (_MEMGET(m2, m2.OFFSET, LONG) +4) AS LONG

It might be a nanosecond faster executing, but is it worth it for the increase in coding time and complexity of understanding the code?

Only the programmer can answer that question, and the answer might vary for different programs.  (Or even different processes inside a program.)  You’d want a recursive SORT process to run as fast as possible, as it can definitely become a program bottleneck, but a process which only adds 1 to each element of an Array isn’t going to be much of a concern.

It’s a trade off: Speed vs readability and complexity, with increased coding time.  Onlyyou can determine when it’s worth the trouble for your program’s needs.  ;)

[Cobalt]

memory involves addressing individual bytes in a block of memory, each byte being 256 bits (0-255).  And so we're only ever talking about

Just wanted to say, each BYTE is 8bits, capable of representing a VALUE of 0-255.

You're making your computer out to be smarter than it really is.  ;)

Yep, Computers can only do what they are told and at the fundamental level they only know ON and OFF (1 an 0)[I think you can say they only know voltage hi(+5v~) and voltage lo(~0.5v].

That is why we are supposed to be the USER and not the other way around. And Knowledge is Power!

[Qwerkey]

"_MEM processing is 100's of times faster than with standard variable processing".  This is what the gurus said.

So, before plummeting into adapting my two programs (Crossword Generator and Gravitation Simulator) which use a lot of processing, I thought that I'd check out this statement.

Code Trial 1A loads a file from disk into a _MEM object and then intensely manipulates that _MEM object (swaps bytes around).  In my core-I5 processor this uses a whole processor's-worth of CPU and the process takes 69.1s.  With $CHECKING:OFF it takes 25.4s.

Code Trial 1B loads the same file into a string, and the same manipulation as in Trial 1A is performed using standard techniques (no _MEM).  The process takes 253s.  So the _MEM processing is faster, but not by a large factor.

So I thought that maybe the process in Trial 1A was slowed down by the presence of standard variables to do the indexing.  Code Trial 1C is as 1A but with everything put into _MEM.  I was shocked to find that this took 157s.

An invariant maxim is "never believe gurus unless you know what they say is true" (particularly pertinent to mythologies, of course).

So, either the gurus were wrong or I don't know what I'm doing, which as all members will readily attest is totally without foundation.

Tria 1A

Code: QB64: [Select]

1. 'Trial 1A _MEM processing, Variable Indexing
2.  
3. 'Load from file into _MEM object
4. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
5. FIELD #1, 4 AS ULoc$
6.  
7. PRINT LOF(1)
8. NoWords& = LOF(1) / 4
9.  
10. DIM M AS _MEM
11. M = _MEMNEW(LOF(1))
12. PRINT M.SIZE '/ 2147483647
13.  
14. DIM MDum AS _MEM
15. MDum = _MEMNEW(1)
16.  
17. $CHECKING:OFF
18.  
19. FOR N& = 1 TO LOF(1) / 4
20.     GET #1, N&
21.     _MEMPUT M, M.OFFSET + ((N& - 1) * 4), ULoc$
22. NEXT N&
23.  
24. CLOSE #1
25.  
26. 'Now do some manipulations in _MEM
27.  
28. Start! = TIMER
29.  
30. FOR N& = 1 TO 100000
31.     'for K& = 0 to  M.SIZE -1  !!! Not allowed
32.     FOR K& = 0 TO 4 * NoWords& - 1
33.         _MEMPUT MDum, MDum.OFFSET, _MEMGET(M, M.OFFSET + (4 * NoWords& - 1 - K&), _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
34.         _MEMPUT M, M.OFFSET + (4 * NoWords& - 1 - K&), _MEMGET(M, M.OFFSET + K&, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
35.         _MEMPUT M, M.OFFSET + K&, _MEMGET(MDum, MDum.OFFSET, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
36.     NEXT K&
37. NEXT N&
38.  
39. PRINT TIMER - Start!
40.  
41. END
42.  

Trial 1B

Code: QB64: [Select]

1. 'Trial 1B Variable processing, Variable Indexing
2.  
3. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
4. FIELD #1, 4 AS ULoc$
5.  
6. PRINT LOF(1)
7. NoWords& = LOF(1) / 4
8.  
9. S$ = ""
10. FOR N& = 1 TO LOF(1) / 4
11.     GET #1, N&
12.     S$ = S$ + ULoc$
13. NEXT N&
14.  
15. PRINT LEN(S$)
16.  
17. CLOSE #1
18.  
19. 'Now do some manipulations with the array - same as Trial 1A
20.  
21. Start! = TIMER
22.  
23. FOR N& = 1 TO 100000
24.     FOR K& = 1 TO LEN(S$)
25.         Dum$ = MID$(S$, K&, 1)
26.         MID$(S$, K&, 1) = MID$(S$, LEN(S$) - K&, 1)
27.         MID$(S$, LEN(S$) - K&, 1) = Dum$
28.     NEXT K&
29. NEXT N&
30.  
31. PRINT TIMER - Start!
32.  
33. END
34.  

Trial 1C

Code: QB64: [Select]

1. 'Trial 1C _MEM processing, _MEM Indexing
2.  
3. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
4. FIELD #1, 4 AS ULoc$
5.  
6. PRINT LOF(1)
7. NoWords& = LOF(1) / 4
8.  
9. 'M Holds the data
10. DIM M AS _MEM
11. M = _MEMNEW(LOF(1))
12. PRINT M.SIZE
13.  
14. 'MSize 4-byte _MEM holds size of M
15. DIM MSize AS _MEM
16. MSize = _MEMNEW(4)
17. _MEMPUT MSize, MSize.OFFSET, LOF(1) AS _UNSIGNED LONG
18. PRINT _MEMGET(MSize, MSize.OFFSET, _UNSIGNED LONG)
19.  
20. 'MDum 1-byte _MEM used to temporarily hold swapped byte
21. DIM MDum AS _MEM
22. MDum = _MEMNEW(1)
23.  
24. '$CHECKING:OFF
25.  
26. FOR N& = 1 TO LOF(1) / 4
27.     GET #1, N&
28.     _MEMPUT M, M.OFFSET + ((N& - 1) * 4), ULoc$
29. NEXT N&
30.  
31. CLOSE #1
32.  
33. 'N 4-byte _MEM to index count
34. DIM N AS _MEM
35. N = _MEMNEW(4)
36. _MEMPUT N, N.OFFSET, 0 AS _UNSIGNED LONG
37.  
38. PRINT _MEMGET(N, N.OFFSET, _UNSIGNED LONG)
39.  
40. 'K 4-byte _MEM used to index swap
41. DIM K AS _MEM
42. K = _MEMNEW(4)
43. _MEMPUT K, K.OFFSET, 0 AS _UNSIGNED LONG
44.  
45. PRINT _MEMGET(K, K.OFFSET, _UNSIGNED LONG)
46.  
47. 'Now do some manipulations in _MEM
48.  
49. Start! = TIMER
50.  
51. WHILE _MEMGET(N, N.OFFSET, _UNSIGNED LONG) < 100000 'Don't know how to do FOR/NEXT with _MEM objects
52.  
53.     _MEMPUT K, K.OFFSET, 0 AS _UNSIGNED LONG
54.     WHILE _MEMGET(K, K.OFFSET, _UNSIGNED LONG) < _MEMGET(MSize, MSize.OFFSET, _UNSIGNED LONG)
55.  
56.         _MEMPUT MDum, MDum.OFFSET, _MEMGET(M, M.OFFSET + (_MEMGET(MSize, MSize.OFFSET, _UNSIGNED LONG) - _MEMGET(K, K.OFFSET, _UNSIGNED LONG) - 1), _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
57.         _MEMPUT M, M.OFFSET + (_MEMGET(MSize, MSize.OFFSET, _UNSIGNED LONG) - _MEMGET(K, K.OFFSET, _UNSIGNED LONG) - 1), _MEMGET(M, M.OFFSET + _MEMGET(K, K.OFFSET, _UNSIGNED LONG), _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
58.         _MEMPUT M, M.OFFSET + _MEMGET(K, K.OFFSET, _UNSIGNED LONG), _MEMGET(MDum, MDum.OFFSET, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
59.  
60.         _MEMPUT K, K.OFFSET, _MEMGET(K, K.OFFSET, _UNSIGNED LONG) + 1 AS _UNSIGNED LONG
61.  
62.     WEND
63.  
64.     _MEMPUT N, N.OFFSET, _MEMGET(N, N.OFFSET, _UNSIGNED LONG) + 1 AS _UNSIGNED LONG
65.  
66. WEND
67.  
68. PRINT TIMER - Start!
69.  
70. _MEMFREE M
71. _MEMFREE N
72. _MEMFREE K
73. _MEMFREE MSize
74. _MEMFREE MDum
75.  
76. END
77.  

[SMcNeill]

With $CHECKING:OFF it takes 25.4s....

...The process takes 253s.  So the _MEM processing is faster, but not by a large factor.

From 253 seconds to 25.4 seconds is 10x faster already, and that’s not a “large factor”?

Also, don’t forget, this guru also told you:

You’ll see different amounts of improvement for various commands/functions which you can swap out to _MEM usage, so it’s hard to say how much you can improve performance with any single change.

And, I’m thinking that you can tweek that code to speed it up more, with just a little work.

M.OFFSET + (4 * NoWords& - 1 - K&) <— there’s a good bit of calculating going on in this statement, and it’s used multiple times.  Assigning it once to a different offset would probably be faster....

And, unless I’m misreading what the code is supposed to do, you’re just swapping contents of your strings?

  FOR K& = 1 TO LEN(S$)
        Dum$ = MID$(S$, K&, 1)  <— a temp$
        MID$(S$, K&, 1) = MID$(S$, LEN(S$) - K&, 1) <— move info from one spot to another
        MID$(S$, LEN(S$) - K&, 1) = Dum$ <— place info back into that original spot from the temp$
    NEXT K&

So far, you’re seeing an improvement of 10x to the time, and you still haven’t learned or used _MEMCOPY, Which is the optimized way to do this sort of process.  http://www.qb64.org/wiki/MEMCOPY

[Qwerkey]

O Great Master, wow!  That was quick!  I only just popped out for drink.

No, you're not misreading what I was doing in the trial and yes I haven't managed to come across _MEMCOPY (will look into it).  But I just wanted to compare exactly the same process without / with _MEM in the trial.

M.OFFSET + (4 * NoWords& - 1 - K&) <— there’s a good bit of calculating going on in this statement, and it’s used multiple times.  Assigning it once to a different offset would probably be faster....

In Trial 1C, I replaced that calculated stuff with a constant _MEM object and this trial was very much slower.  Maybe I just did something wrong there.

[SMcNeill]

Just curious; How's this little version perform for you?

Code: QB64: [Select]

1. 'Trial 1A _MEM processing, Variable Indexing
2.  
3. 'Load from file into _MEM object
4. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
5. FIELD #1, 4 AS ULoc$
6.  
7. PRINT LOF(1)
8. NoWords& = LOF(1) / 4
9.  
10. DIM M AS _MEM
11. M = _MEMNEW(LOF(1))
12. PRINT M.SIZE '/ 2147483647
13.  
14. DIM MDum AS _MEM
15. MDum = _MEMNEW(1)
16.  
17. $CHECKING:OFF
18.  
19. FOR N& = 1 TO LOF(1) / 4
20.     GET #1, N&
21.     _MEMPUT M, M.OFFSET + ((N& - 1) * 4), ULoc$
22. NEXT N&
23.  
24. CLOSE #1
25.  
26. 'Now do some manipulations in _MEM
27.  
28. DIM Precalc AS _OFFSET
29.  
30. Start! = TIMER
31. Precalc = M.OFFSET + 4 * NoWords& - 1 'Move as much of your math outside the loop as possible
32.  
33. FOR N& = 1 TO 100000
34.     'for K& = 0 to  M.SIZE -1  !!! Not allowed
35.     'You can't mix OFFSETs with other values, as above with that FOR statement.
36.     'Instead, convert it into a simple DO sequence, such as below.
37.     K& = 0
38.     DO
39.         _MEMPUT MDum, MDum.OFFSET, _MEMGET(M, Precalc - K&, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
40.         _MEMPUT M, Precalc - K&, _MEMGET(M, M.OFFSET + K&, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
41.         _MEMPUT M, M.OFFSET + K&, _MEMGET(MDum, MDum.OFFSET, _UNSIGNED _BYTE) AS _UNSIGNED _BYTE
42.         K& = K& + 1
43.     LOOP UNTIL K& >= M.SIZE
44. NEXT N&
45. $CHECKING:ON
46. PRINT TIMER - Start!
47.  
48. END

You mentioned that it was taking you 25.4s to run the code.  All I've done here is precalculate the math and simplify the calculations inside the loop a bit, in an effort to showcase how much of a difference they can make sometimes.  On my PC, the runtime was about 13 seconds, and after simplifying the math by calculating the values that won't change outside the loop, it dropped runtime down to about 11 seconds.  A 15% performance increase, with a very minor alteration completely unrelated to our mem commands at all.  ;)

If it affects your PC with a similar change, it should reduce times down from 25 seconds to what?  About 21 seconds?

[SMcNeill]

And, completely unrelated to the _MEM commands, I thought I'd take a moment and also showcase something else to help highlight just exactly how SLOOOOOOOOW the QB64 string commands are.

First, run this piece of your original code and time it.  (Take a nap, it takes a while, coming in at almost 3 minutes to execute on my PC.)

This is the exact same as TRIAL 1B above:

Code: QB64: [Select]

1. 'Trial 1B Variable processing, Variable Indexing
2.  
3. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
4. FIELD #1, 4 AS ULoc$
5.  
6. PRINT LOF(1)
7. NoWords& = LOF(1) / 4
8.  
9. S$ = ""
10. FOR N& = 1 TO LOF(1) / 4
11.     GET #1, N&
12.     S$ = S$ + ULoc$
13. NEXT N&
14.  
15. PRINT LEN(S$)
16.  
17. CLOSE #1
18.  
19. 'Now do some manipulations with the array - same as Trial 1A
20.  
21. Start! = TIMER
22.  
23. FOR N& = 1 TO 100000
24.     FOR K& = 1 TO LEN(S$)
25.         Dum$ = MID$(S$, K&, 1)
26.         MID$(S$, K&, 1) = MID$(S$, LEN(S$) - K&, 1)
27.         MID$(S$, LEN(S$) - K&, 1) = Dum$
28.     NEXT K&
29. NEXT N&
30.  
31. PRINT TIMER - Start!
32.  
33. END

Now, I'm going to rewrite the routine to work with numeric values and ASCII values, rather than with strings.  Compile and test the following for its runtime:

Code: QB64: [Select]

1. 'Trial 1B Variable processing, Variable Indexing
2.  
3. OPEN "4UKEng.rnd" FOR RANDOM AS #1 LEN = 4
4. FIELD #1, 4 AS ULoc$
5.  
6. PRINT LOF(1)
7. NoWords& = LOF(1) / 4
8.  
9. S$ = ""
10. FOR N& = 1 TO LOF(1) / 4
11.     GET #1, N&
12.     S$ = S$ + ULoc$
13. NEXT N&
14.  
15. PRINT LEN(S$)
16.  
17. CLOSE #1
18.  
19. 'Now do some manipulations with the array - same as Trial 1A
20.  
21. DIM B AS _BYTE
22. L = LEN(S$) + 1
23.  
24. Start! = TIMER
25.  
26. FOR N& = 1 TO 100000
27.     FOR K& = 1 TO LEN(S$)
28.         'Dum$ = MID$(S$, K&, 1)
29.         B = ASC(S$, K&)
30.         'MID$(S$, K&, 1) = MID$(S$, LEN(S$) - K&, 1)
31.         ASC(S$, K&) = ASC(S$, L - K&)
32.         'MID$(S$, LEN(S$) - K&, 1) = Dum$
33.         ASC(S$, L - K&) = B
34.     NEXT K&
35. NEXT N&
36.  
37. PRINT TIMER - Start!
38.  
39. END

Now, just by swapping over to ASC, instead of MID$, the runtime on my PC has now dropped down to less than 1 minute. (56 seconds.)

From 187 seconds (MID$) to 56 seconds (ASC), to 11 seconds (_MEM)...

Which is why I said it's hard to tell folks, "You're going to always see a performance boost of XX percent with _MEM."  Various commands have different levels of overhead to them, so it's impossible to say how much of a performance boost you'll see, until you can actually compare routines side-by-side. 

In this case, the original code took 187 seconds, the _MEM version took 11 seconds, so that's a performance increase by 17x...  Which, I think really helps to highlight exactly why it's nice for QB64 programmers to have the ability to use _MEM inside their code.  ;)

[Dimster]

Steve, in your _MEMPUT examples you have been using whole numbers - ie _MEMPUT m, m.OFFSET, 986543.
Would _MEMPUT also calculate the value? for example _MEMPUT m, m.OFFSET, 98000 + 6543 

or is it better coding practice to have the value calculated before the PUT such as  V1 = 98000 + 6543 : _MEMPUT m, m.OFFSET, V1 ?
 
Really liking your videos : finished #3 with popcorn to spare.