Post by: Sanmayce on January 20, 2021, 05:18:19 am
Hi,
an oldboy being a big fan of QB 2.0, QB 4.0, QB 4.5 and QBX 7.1 here.
So glad that few days ago found QB64, a dream relived, as I see it.
Many thanks to QB64 team, really you gladdened my eyes, the IDE feels as it should, the stand alone executable is seconds away, so convenient, bravo!
25 years ago I wrote some original tools in QB45 + MASM functions, loved that pair, thought that experience was past long gone, now I see I haven't killed the quickening within.
Last night wrote my first QB64 program, gladly sharing it here , my wish is to continue to enrich its functionality...
To community, feel free to point out the stupid parts you see in it.
My idea is to invoke my [searching] console tools (written in C) via this "GUI SHELL" and showing their output in this QB64 window...
Code: QB64: [Select]
- ' DragDropWheel.bas
- ' written in QB64 v1.4 by Kaze, 2021-Jan-20
- _TITLE "DragDropWheel"
- XdimCOL = 160
- YdimROW = 44 ' ensure old laptops with 768pixels vertical will hold the whole window
- wide& = _DESKTOPWIDTH
- high& = _DESKTOPHEIGHT
- SCREEN handle&
- ' Either one must be uncommented:
- '_AUTODISPLAY 'no need of refreshing
- _ACCEPTFILEDROP 'enables drag/drop functionality
- 'Y = CSRLIN 'save the row
- 'X = POS(0) 'save the column
- filecount% = 0
- REDIM FileArray$(1000000) 'create dynamic array: 3880000 alocates 532MB - bigger values need 570+MB and give "Out Of Memory"
- FileArray$(i) = ""
- PRINT "Drag files from a folder and drop them in this window..."
- 'REDIM FileArray$(filecount%)
- 'pressakey$ = INPUT$(1)
- a$ = ""
- 'FOR i = 1 TO _TOTALDROPPEDFILES
- 'a$ = _DROPPEDFILE(i)
- 'NEXT'
- _FINISHDROP 'If _FINISHDROP isn't called here then _TOTALDROPPEDFILES never gets reset.
- ' ELSE
- ' a$ = "Scroller.$$$"
- ' SHELL _HIDE "DIR /B *.* > Scroller.$$$"
- filecount% = filecount% + 1
- FileArray$(filecount%) = filename$
- CLOSE #1
- _LIMIT 30
- 'PRINT "Printing filenames in current directory... ";: PRINT LTRIM$(STR$(filecount%))
- cry = CSRLIN
- crxOLD = crx
- cryOLD = cry
- 'LOCATE cry, crx, 1, 30, 31
- 'DO: a$ = INKEY$: LOOP UNTIL a$ <> "" ' prevent ASC empty string read error
- 'LOCATE cryOLD, crx, 1, 30, 31: COLOR 3, 0: PRINT FileArray$(cryOLD);
- cryOLD = cry
- ELSE 'it 'cry' could be changed by Mouse Wheel too, check it
- 'LOCATE cry, crx, 1, 30, 31: COLOR 0, 3: PRINT FileArray$(cry);
- 'DO WHILE INKEY$ <> "": LOOP ' have to clear the keyboard buffer
- '_LIMIT 30 'commented because the wheel up/down was not working?!
Post by: SMcNeill on January 20, 2021, 06:15:46 am
Code: QB64: [Select]
In the above, my mousewheel will scroll from 0 to 100, and wrap from high to low, without any lag or drag on performance. For your needs, just substitute your (max line count - number of lines on screen) instead of 100, and scroll should be the line you’d start printing to the screen. (Assuming no word wrap occurs, of course.)
Post by: FellippeHeitor on January 20, 2021, 11:11:03 am
Post by: Pete on January 20, 2021, 11:28:27 am
Don't forget to tell him about NOT adding other mouse commands, like _MOUSEBUTTON(1), etc. inside that loop, but rather to add them outside and after it terminates.
Pete
Post by: Pete on January 20, 2021, 11:42:22 am
Hey Fell, looks like you've got another old-TIMER in the fold. We're getting to be a pretty big group around here. If this keeps up, you'll have to offer us a senior discount. Since this is an open source project, that means you'll owe us money! :D
@Sanmayce
Welcome. I like GUI the stuff, too. I've got a project I'm working on currently, posted over at the QBasic forum. Since you're an old-timer, do you remember that forum? Network54 used to host it. Now it's on Tapatalk. Mallord was the first admin, followed by Mac, and Mac turned it over to me, when he passed away in 2008. Anyway, I'm just curious if you recall it, or Pete's QBasic forum, or any of the others back in the day.
Pete
Post by: bplus on January 20, 2021, 12:04:18 pm
Post by: FellippeHeitor on January 20, 2021, 12:06:45 pm
Code: QB64: [Select]
Post by: Pete on January 20, 2021, 12:09:58 pm
This is a neat addition to the language. I have it in V. 1.3. Back in the day, I had to make my own routine. I dragged a copy of FreeBASIC in it once, by mistake, which created a memory incident. It returned an "Are you out of your mind," error.
Pete
Post by: bplus on January 20, 2021, 12:14:18 pm
@bplus
This is a neat addition to the language. I have it in V. 1.3. Back in the day, I had to make my own routine. I dragged a copy of FreeBASIC in it once, by mistake, which created a memory incident. It returned an "Are you out of your mind," error.
Pete
LOL
Yeah I am seeing this as nice substitute for expensive File Dialog! Would like to see demo of handling multiple files, something to play around with for sure!
Post by: FellippeHeitor on January 20, 2021, 12:14:56 pm
Post by: NOVARSEG on January 20, 2021, 10:50:02 pm
I tried uncommenting the _LIMIT 30 in your code and it does work but it takes forever (just over 2 minutes) for the wheel to update. That is because there is usually tons of mouse x, y position data in the mouse buffer and LIMIT 30 is too slow to process that data. The wheel data is usually at the end of the x, y position data.
Actually the wheel and position data can be intermixed but when the wheel data is at the end of a large amount of position data, there is this apparent delay in wheel update.
Even with no mouse position data in the buffer, the wheel response is pretty slow with _LIMIT 30
So try _LIMIT 500
Post by: Sanmayce on January 20, 2021, 11:38:27 pm
My passion is text processing of huge English text, searching, decompressing, indexing them in order to ease quick n-gram requests i.e. to have spell-checkers, and ... phrase-checkers of English texts under the fingers.
C and QB64 are so instrumental in that regard, eager to pursue this dream of mine...
@SMcNeill
Thank you Steve, not only for you suggestion (which prompted me to revise my first pseudo-buggy attempt) but for useful sharings of yours. Showing different scenarios being handled is the best way to help others, such snippets/etudes are instrumental for sure.
The idea is to write a simplistic viewer working with the "heap" by using the QB64's memaloc()-like new functions instead of array of NON-FIXED length strings. Wanna try how the scroller behaves with 1GB file text loaded in RAM.
Your MEMUSAGE tool is cool, MUCH NEEDED, accessing Windows APIs has to be explored as you did (I have no experience in using APIs), some functionality has to be "stolen" from DLLs, wanna write first that exact viewer...
Welcome aboard, @Sanmayce! Cool to see you're already getting it going with several modern keywords as well! Looking forward to your future contributions.
Thank you man! Surely in future will share all my code along with my feedback.
Fellippe, I bend a knee before you. resurrecting QB is just awesome! Many beers I owe you.
As far as I recall, only said this once back, to another developer - a true master of coding compression in C.
Really enliked your presentation about 'InForm', please consider doing such reviews more often, at least not stopping them:
https://www.youtube.com/watch?v=S3D4zEUQ0sQ&list=PLD7-OthpyuXNbrB12Cl5_3vBpRtvN7CA8&index=5 (https://www.youtube.com/watch?v=S3D4zEUQ0sQ&list=PLD7-OthpyuXNbrB12Cl5_3vBpRtvN7CA8&index=5)
Nice work with this Form Designer, wanna try it after some weeks, my tempo is slow but ongoing, a real fan here, that is.
My expertise is amateurish, I have been playing with search algorithms, and Windows (as far as Form Designer goes) not wanting to go into C++ and APIs, just played for a month or so with VisualStudio (ATL project) in order to make my GUI shell for invoking my console executables written in C. Just assigned some SHELL _HIDE like instance behind each button. The result: www.sanmayce.com/Downloads/_GW_(Bare-Minimum_2018-Mar-16).zip (http://www.sanmayce.com/Downloads/_GW_(Bare-Minimum_2018-Mar-16).zip)
It looks very nice on Windows XP, and so ugly with the new Windows 7+ schemes, therefore for more than 3 years I abandoned it, shame.
[ Invalid Attachment ]
[ Invalid Attachment ]
[ Invalid Attachment ]
Similar thing is what I want to write with QB64 - a simplistic text editor sensitive to all my textual processing tools - my textual madness is deep and constant, my wish is in the near future to have phrase sensitive functionality (giving statistics about English language n-grams generated by external tools) while hovering over this text window, both with cursor and mouse.
Simply, I am hell bent on making and sharing one 100% FREE lightweight (and thanks to QB64 portability) text assistant/sidekick/application in the near future.
Want to ask for possible way to load our own bitmap (from DOS times) fonts, saw that '_FONT 16' is default for SCREEN 0, if there is a way to load those fonts will bring additional retro flavor as they were very nice, sharing them, to take a look if interested:
@Sanmayce
Welcome. I like GUI the stuff, too. I've got a project I'm working on currently, posted over at the QBasic forum. Since you're an old-timer, do you remember that forum? Network54 used to host it. Now it's on Tapatalk. Mallord was the first admin, followed by Mac, and Mac turned it over to me, when he passed away in 2008. Anyway, I'm just curious if you recall it, or Pete's QBasic forum, or any of the others back in the day.
Pete
Thanks, will share some GUI of mine in the future, my Gallowwalker GUI, in the meantime you are welcome to see some of its functionality at:
https://github.com/Sanmayce/1gram/issues/1 (https://github.com/Sanmayce/1gram/issues/1)
As for the old forum, no, was not active in Internet for a long time, haven't seen it. Otherwise, I am a chatterbox :P
The idea is to have the first English Phrase-Checker - checking all n-grams (phrases of n words) of your text (the loaded into this QB64 assistant) against mega corpora of English n-grams, hundreds of millions strong!
@Sanmayce welcome! I did not know or see demo of _ACCEPTFILEDROP, thanks! Hopefully we can drop in more than one file eventually.
Thank you, @bplus, I myself am uncovering new things on a daily basis, so many gaps ate there in my know-how.
Did a minor tweak (pseudo-bug fixage), rather add-on, now one can use the left button and "jump" on the desired line, along with scrolling with keys and the wheel:
Code: QB64: [Select]
- ' DragDropWheelScroller.bas
- ' written in QB64 v1.4 by Kaze, 2021-Jan-21
- ' Thanks go to the www.qb64.org/forum members for sharing useful excerpts/etudes
- _TITLE "DragDropWheel"
- XdimCOL = 100
- YdimROW = 44 ' ensure old laptops with 768pixels vertical will hold the whole window
- wide& = _DESKTOPWIDTH
- high& = _DESKTOPHEIGHT
- SCREEN handle&
- '_FONT 16 'wish we could use the old 8x16 bitmap/raster fonts from DOS times...
- ' Either one must be uncommented:
- '_AUTODISPLAY 'no need of refreshing
- _ACCEPTFILEDROP 'enables drag/drop functionality
- 'Y = CSRLIN 'save the row
- 'X = POS(0) 'save the column
- filecount% = 0
- REDIM FileArray$(1000000) 'create dynamic array: 3880000 alocates 532MB - bigger values need 570+MB and give "Out Of Memory"
- FileArray$(i) = ""
- PRINT "Drag files from a folder and drop them in this window..."
- 'REDIM FileArray$(filecount%)
- 'pressakey$ = INPUT$(1)
- a$ = ""
- 'FOR i = 1 TO _TOTALDROPPEDFILES
- 'a$ = _DROPPEDFILE(i)
- 'NEXT'
- _FINISHDROP 'If _FINISHDROP isn't called here then _TOTALDROPPEDFILES never gets reset.
- 'ELSE
- 'a$ = "Scroller.$$$"
- 'SHELL _HIDE "DIR /B *.* > Scroller.$$$"
- filecount% = filecount% + 1
- FileArray$(filecount%) = filename$
- CLOSE #1
- _LIMIT 30
- 'PRINT "Printing filenames in current directory... ";: PRINT LTRIM$(STR$(filecount%))
- cry = CSRLIN
- crxOLD = crx
- cryOLD = cry
- 'LOCATE cry, crx, 1, 30, 31
- 'DO: a$ = INKEY$: LOOP UNTIL a$ <> "" ' prevent ASC empty string read error
- 'LOCATE cryOLD, crx, 1, 30, 31: COLOR 3, 0: PRINT FileArray$(cryOLD);
- cryOLD = cry
- ELSE 'it 'cry' could be changed by Mouse Wheel too, check it
- 'Kinda trump the arrows and wheel with MOUSEmovement i.e. us having both the inverse(selected) line plus darklighting the current(under the mouse pointer) one
- cry = _MOUSEY
- 'dummy me, the 2021-Jan-20 was not smoot friendly, should have updated here (not in the loop afterwards) i.e. _DISPLY shows two inversed lines, ugh [
- 'LOCATE cryOLD, crx, 1, 30, 31: COLOR 3, 0: PRINT FileArray$(cryOLD);
- cryOLD = cry
- 'dummy me, the 2021-Jan-20 was not smoot friendly, should have updated here (not in the loop afterwards) i.e. _DISPLY shows two inversed lines, ugh ]
- 'LOCATE cry, crx, 1, 30, 31: COLOR 0, 3: PRINT FileArray$(cry);
- 'DO WHILE INKEY$ <> "": LOOP ' have to clear the keyboard buffer
- '_LIMIT 30 'commented because the wheel up/down was not working?!
Kinda want to revive the DOS functionality of my old QB project:
Post by: FellippeHeitor on January 20, 2021, 11:43:42 pm
Quote
Fellippe, I bend a knee before you. resurrecting QB is just awesome! Many beers I owe you.
As far as I recall, only said this once back, to another developer - a true master of coding compression in C.
Really enliked your presentation about 'InForm', please consider doing such reviews more often, at least not stopping them:
https ://www.youtube.com/watch?v=S3D4zEUQ0sQ&list=PLD7-OthpyuXNbrB12Cl5_3vBpRtvN7CA8&index=5
Nice work with this Form Designer, wanna try it after some weeks, my tempo is slow but ongoing, a real fan here, that is.
We all bend our knees before @Galleon, the guy who created QB64 in the first place. I'm barely part of the team. He does come around from time to time.
I'm so glad you enjoyed InForm! That's truly my code baby. Let me know if I can be of any assistance with it.
Post by: Sanmayce on January 20, 2021, 11:53:39 pm
We all bend our knees before @Galleon, the guy who created QB64 in the first place. I'm barely part of the team. He does come around from time to time.
I'm so glad you enjoyed InForm! That's truly my code baby. Let me know if I can be of any assistance with it.
Oh, I bend a knee the third time, then. VIVA @Galleon.
I can tell from your video, you are bona-fide QB fellow, indeed!
Excuse me for the previous messy post of mine, couldn't fit all the illustrative screenshots at once, that is what happens when many fellow members address a new fellow at once, likey-likey :P
Post by: NOVARSEG on January 21, 2021, 12:17:33 am
Quote
My passion is text processing of huge English text, searching, decompressing, indexing them in order to ease quick n-gram requests i.e. to have spell-checkers, and ... phrase-checkers of English texts under the fingers.
C and QB64 are so instrumental in that regard, eager to pursue this dream of mine...
You will like the fact that QB64 string size is over 2GB
Quote
25 years ago I wrote some original tools in QB45 + MASM functions
Did you ever use DOS interrupts . Ralf Brown interrupt list etc?
Post by: Sanmayce on January 21, 2021, 01:00:35 am
Thank you for the hint, will try it, currently I am using an old laptop with Windows XP 32bit paired with a buggy ATI videocard (causing text to be blurred if more than ~130 columns are used), next week I will get access to my main laptop with Windows 10 64bit with i5-7200u and 36GB, at the moment my brother is repairing it, it is 4 years old, almost.
Thanks Sanmayce
You will like the fact that QB64 string size is over 2GB
Did you ever use DOS interrupts . Ralf Brown interrupt list etc?
Oh, good to know, I love strings, the thing is, over the years just switched to C and changed the approach to pointers. Will share inhere how the scrolling of 1+GB texts is handled...
As for interrupts, yeah, I wrote some resident keyboard tools, switching fonts and languages, KAZUMA.COM it was called, it featured AFAIR 11 fonts and using the 28 lines. But mostly I used MASM to write all my functions and link them to QB, the main thing was to use XMS memory from QB, also writing 3D windows waiting for the refresh beam to scan again, the time between refreshing was pretty enough to achieve smooth scrolling and drawing. Also, wrote OTANE.BAS - the fastest VESA 1.2 graphical PCX viewer/scroller, it rotated images, AFAIR reaching 44 fps on some 486DX100MHz in 1024x768, forgot already, did attach them anyway:
Post by: Sanmayce on January 22, 2021, 02:53:26 am
Just added few things:
- ICON;
- increased the LIMIT to 500 iterations, as suggested by @NOVARSEG (glad to see the scroll is smoothly going, now);
- sensitive to writing to CONSOLE;
- added the must-have system stats by @SMcNeill from his MEMUSAGE.BAS;
Wanted to have the very first revision attached as a starting point.
My brother finished repairing my main laptop 'Compressionette', sharing three screenshots:
And the source attached:
Post by: Sanmayce on January 24, 2021, 01:13:33 pm
- the loaded file now is viewable i.e. scrollable;
- loading the file alternatively via command prompt;
- TAB2SPC expander;
- Line Wrapper, commented out since currently is not needed;
- now there is a status line;
- should be compilable on Linux;
- Mouse Button 3 - works as if PgDn is pressed;
- LSHIFT+RSHIFT prints in the lowest/status line the time needed to load;
- LCtrl+Home, LCtrl+End - going to the top/bottom left position;
In the future, I intend to write a streaming revision able to show only the window (dynamically loading only the needed for visualization content) thus saving RAM and TIME, for example current English Wikipedia XML dump is 81+ billion bytes having longest line around 1 million chars, it means that the RAM for browsing this Wikipedia will be less than 60x1MB i.e. 60 lines padded with spaces... For now, let us see how one paragon 500MB file stresses the current ALL-TO-RAM approach:
On my fastest laptop with i5-7200U did try to load the whole Oxford English Dictionary, the performance of LINE INPUT is awful, in next revisions will bypass it altogether by replacing it with my own pointer based function, similar to the 'LineWordreporter.c' (included in the attached package) - wrote it to serve as a quick file reporter.
The benchmark shows, OED took 23,187 seconds or 6+ hours to be loaded via LINE INPUT, unacceptable! Any idea why so?
Holding the Mouse Button 3 for:
19:19:00
19:22:38
or 22*60+38-19*60+00=218 seconds scrolled all the 4 million lines, or 4,071,706/60=67861 pages, which is 67861/218=311 Pages-Per-Second, nice.
There is a Keyboard SUB called 'ReturnCombo' returning in shared variables many to-be-used shortcuts as:
- LSHIFT_RSHIFT
LCTRL_RCTRL
LALT_RALT
LALT_HOME
LALT_END
LALT_INS
LALT_DEL
LALT_PGUP
LALT_PGDN
LALT_Left
LALT_Right
LALT_Up
LALT_Down
RALT_HOME
RALT_END
RALT_INS
RALT_DEL
RALT_PGUP
RALT_PGDN
RALT_Left
RALT_Right
RALT_Up
RALT_Down
LSHIFT_HOME
LSHIFT_END
LSHIFT_INS
LSHIFT_DEL
LSHIFT_PGUP
LSHIFT_PGDN
LSHIFT_Left
LSHIFT_Right
LSHIFT_Up
LSHIFT_Down
RSHIFT_HOME
RSHIFT_END
RSHIFT_INS
RSHIFT_DEL
RSHIFT_PGUP
RSHIFT_PGDN
RSHIFT_Left
RSHIFT_Right
RSHIFT_Up
RSHIFT_Down
LCTRL_HOME
LCTRL_END
LCTRL_INS
LCTRL_DEL
LCTRL_PGUP
LCTRL_PGDN
LCTRL_Left
LCTRL_Right
LCTRL_Up
LCTRL_Down
RCTRL_HOME
RCTRL_END
RCTRL_INS
RCTRL_DEL
RCTRL_PGUP
RCTRL_PGDN
RCTRL_Left
RCTRL_Right
RCTRL_Up
RCTRL_Down
LSHIFT_LCTRL_HOME
LSHIFT_LCTRL_END
LSHIFT_LCTRL_INS
LSHIFT_LCTRL_DEL
LSHIFT_LCTRL_PGUP
LSHIFT_LCTRL_PGDN
LSHIFT_LCTRL_Left
LSHIFT_LCTRL_Right
LSHIFT_LCTRL_Up
LSHIFT_LCTRL_Down
LSHIFT_BackSpace
LSHIFT_TAB
LSHIFT_SPACE
LSHIFT_ESC
LSHIFT_ENTER
RSHIFT_BackSpace
RSHIFT_TAB
RSHIFT_SPACE
RSHIFT_ESC
RSHIFT_ENTER
LCTRL_SPACE
LCTRL_ENTER
RCTRL_SPACE
RCTRL_ENTER
Code: QB64: [Select]
- ' Masakari.bas (DragDropWheelScroller.bas)
- ' written in QB64 v1.4 by Kaze, 2021-Jan-24
- ' Thanks go to the www.qb64.org/forum members for sharing useful excerpts/etudes
- ' Wow, this is nice indeed: http://www.qb64.org/wiki/Windows_Libraries
- ' Currently are implemented only:
- ' Mouse:
- ' Button 1 - sets the cursor and the inverse line to the chosen position
- ' Button 3 - PgDn
- ' Wheel Up - Up
- ' Wheel Dn - Dn
- ' Keyboard:
- ' Up
- ' Down
- ' Left - still no sideways scroll
- ' Right - still no sideways scroll
- ' LCtrl+Home - going to the top left position
- ' LCtrl+End - going to the bottom left position
- ' Alt+x or Alt+q - quit to the system, without demanding keypress.
- _TITLE "MASAKARI 128x60, The 'Holy Axe' English Text Sidekick"
- $EXEICON:'ColumnChart.ico'
- _CONSOLETITLE "Masakari console window"
- PRINT "Masakari, revision 2, written in QB64 by Kaze"
- PRINT "Usage: Masakari filename|/help"
- '_CONSOLE OFF
- '_SCREENSHOW
- 'DEFLNG A-Z
- '_DEFINE A-Z AS _UNSIGNED _INTEGER64
- FUNCTION MemInUsePercent~&&
- FUNCTION TotalPhysicalMem~&&
- FUNCTION FreePhysicalMem~&&
- FUNCTION TotalPagingFile~&&
- FUNCTION FreePagingFile~&&
- FUNCTION TotalVirtualMem~&&
- FUNCTION FreeVirtualMem~&&
- FUNCTION FreeExtendedMem~&&
- FUNCTION GetCPULoad#
- XdimCOL = 128
- YdimROW = 44 ' ensure old laptops with 768pixels vertical will hold the whole window
- wide& = _DESKTOPWIDTH
- high& = _DESKTOPHEIGHT
- SCREEN handle&
- _DEST handle&
- ' _DEST 0 refers to the present program SCREEN. You can use 0 to refer to the present program
- ' SCREEN.
- '_FONT 16 'wish we could use the old 8x16 bitmap/raster fonts from DOS times...
- ' Either one must be uncommented:
- '_AUTODISPLAY 'no need of refreshing
- TimeA = TIMER
- PurpleFlag = 0
- 'PurpleFlag = 1
- NormalFRGr = 3
- NormalBCKGr = 0
- InverseFRGr = 0
- InverseBCKGr = 3
- COLOR NormalFRGr, NormalBCKGr
- 'Y = CSRLIN 'save the row
- 'X = POS(0) 'save the column
- REDIM FileArray$(1) 'create dynamic array: 3880000 alocates 532MB - bigger values need 570+MB and give "Out Of Memory"
- _ACCEPTFILEDROP 'enables drag/drop functionality
- 'pressakey$ = INPUT$(1)
- a$ = ""
- PostfixedToHeader = CSRLIN
- ' MEMUSAGE by Steve [
- ' GetCPULoad = 0 is idle, 1 is fully used.
- ' Multiply by 100 for a percentage
- PRINT "Total Physical Memory: ";
- PRINT "Free Physical Memory: ";
- PRINT "Total Paging File: ";
- PRINT "Free Paging File: ";
- PRINT "Total Virtual Memory: ";
- PRINT "Free Virtual Memory: ";
- 'PRINT "Free Extended Virtual Memory:"
- 'PRINT " "; FreeExtendedMeml; " bytes"
- ' MEMUSAGE by Steve ]
- ReturnCOMBO
- 'FOR i = 1 TO _TOTALDROPPEDFILES
- 'a$ = _DROPPEDFILE(i)
- 'NEXT'
- _FINISHDROP 'If _FINISHDROP isn't called here then _TOTALDROPPEDFILES never gets reset.
- 'ELSE
- 'a$ = "Scroller.$$$"
- 'SHELL _HIDE "DIR /B *.* > Scroller.$$$"
- 'WrapFlag = 0
- 'Wwidth% = XdimCOL
- 'ul& = 0
- 'LongestLine = 0
- 'DO WHILE NOT EOF(1)
- ' LINE INPUT #1, l$: ul& = ul& + 1
- ' ExpandTabs l$
- ' IF LEN(l$) > LongestLine THEN LongestLine = LEN(l$)
- ' IF WrapFlag THEN
- ' lX$ = l$
- ' DO WHILE LEN(lX$) > Wwidth%
- ' Glupak% = Wwidth%
- ' DO UNTIL MID$(lX$, Glupak% + 1, 1) = " " AND MID$(lX$, Glupak%, 1) <> " "
- ' Glupak% = Glupak% - 1
- ' IF Glupak% = 0 THEN
- ' PRINT "Rejecting line(#"; LTRIM$(STR$(ul&)); ", "; LTRIM$(STR$(LEN(l$))); "chars) that cannot be wrapped!"
- ' _DISPLAY: SYSTEM
- ' GOTO B4Txpanar
- ' END IF
- ' LOOP
- ' PRINT #3, LEFT$(lX$, Glupak%)
- ' lX$ = STRING$(1, " ") + LTRIM$(MID$(lX$, Glupak% + 1, LEN(lX$) - (Glupak%)))
- ' LOOP
- ' PRINT #3, lX$
- ' B4Txpanar:
- ' END IF
- 'LOOP
- LongestLine = 0
- filecount = 0
- filecount = filecount + 1
- CLOSE #1
- filecount = 0
- ExpandTabs l$
- NEXT j
- filecount = filecount + 1
- FileArray$(filecount) = l$
- CLOSE #1
- _LIMIT 5
- 'IF INKEY$ = CHR$(27) THEN SYSTEM
- 'IF keycode& = 27 THEN SYSTEM
- ' QB64 _KEYHIT and _KEYDOWN Values
- '
- 'Esc F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 Sys ScL Pause
- ' 27 15104 15360 15616 15872 16128 16384 16640 16896 17152 17408 34048 34304 +316 +302 +019
- '`~ 1! 2@ 3# 4$ 5% 6^ 7& 8* 9( 0) -_ =+ BkSp Ins Hme PUp NumL / * -
- '126 33 64 35 36 37 94 38 42 40 41 95 43 8 20992 18176 18688 +300 47 42 45
- ' 96 49 50 51 52 53 54 55 56 57 48 45 61
- 'Tab Q W E R T Y U I O P [{ ]} \| Del End PDn 7Hme 8/? 9PU +
- ' 9 81 87 69 82 84 89 85 73 79 80 123 125 124 21248 20224 20736 18176 18432 18688 43
- ' 113 119 101 114 116 121 117 105 111 112 91 93 92 55 56 57
- 'CapL A S D F G H J K L ;: '" Enter 4/?- 5 6/-?
- '+301 65 83 68 70 71 72 74 75 76 58 34 13 19200 19456 19712 E
- ' 97 115 100 102 103 104 106 107 108 59 39 52 53 54 n
- 'Shift Z X C V B N M ,< .> /? Shift ? 1End 2/? 3PD t
- '+304 90 88 67 86 66 78 77 60 62 63 +303 18432 20224 20480 20736 e
- ' 122 120 99 118 98 110 109 44 46 47 49 50 51 r
- 'Ctrl Win Alt Spacebar Alt Win Menu Ctrl ?- ? -? 0Ins .Del
- '+306 +311 +308 32 +307 +312 +319 +305 19200 20480 19712 20992 21248 13
- ' 48 46
- ' Lower value = LCase/NumLock On __________________ + = add 100000
- TimeB = TIMER
- PLAY "L8V2a-c-"
- CurrentLine = 1
- ' Here is the layout:
- ' The window-frame is 1..crx or 1..128 | The file-frame is 1..LongestLine
- ' . . | .
- ' . . | .
- ' cry 60 | filecount
- ' FileArrayWINDOW$(60) | FileArray(filecount)
- ' if File_Frame_x < 128 then PADDING to 128 else File_Frame_x = 1..LongestLine-(128-1)
- ' if File_Frame_y < 60 then PADDING to 60 else File_Frame_y = 1..filecount-(60-1)
- UpdateCLine 1, 1, InverseFRGr, InverseBCKGr, 1
- Dumbo$ = "File Size: " + AddCommas$(FileSize) + "; Longest Line: " + AddCommas$(LongestLine) + "; Total Lines: " + AddCommas$(filecount)
- UpdateNextToCCL_DONE MostRightField
- cry = CSRLIN
- crxOLD = crx
- cryOLD = cry
- File_Frame_x = 1
- File_Frame_y = 1
- ReturnCOMBO
- ReportTimeToLoad MostRightField
- 'DO: a$ = INKEY$: LOOP UNTIL a$ <> "" ' prevent ASC empty string read error
- 'CASE 27: COLOR 7, 0: SYSTEM 'END
- CASE 72:
- cry = cry - 1 'up
- ELSE 'scrolling is needed
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- CASE 80:
- ELSE 'scrolling is needed
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- UpdateCLine cryOLD, 1, NormalFRGr, NormalBCKGr, cryOLD
- cryOLD = cry
- ELSE 'it 'cry' could be changed by Mouse Wheel too, check it
- ELSE 'scrolling is needed
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- cry = cry - 1 'up
- ELSE 'scrolling is needed
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- crxOLD = crx
- cryOLD = cry
- cry = _MOUSEY
- crx = _MOUSEX
- UpdateCLine cryOLD, 1, NormalFRGr, NormalBCKGr, cryOLD
- cryOLD = cryOLD - 1
- cryOLD = cryOLD + 1
- ' Don't execute PgDn (advancing the 'CurrentLine') if 'File_Frame_y' is not "eligible":
- 'IF File_Frame_y < filecount - (YdimROW - 1) THEN File_Frame_y = File_Frame_y + 1
- 'IF File_Frame_y + 1 <= filecount - (YdimROW - 1) THEN File_Frame_y = File_Frame_y + 1
- 'CAUTION [
- ' Next three line beep NOT
- 'DEFLNG A-Z
- 'aaa = 61
- 'bbb = -48
- 'IF aaa <= bbb THEN BEEP: END
- 'Next three line beep
- 'aaa~& = 61
- 'bbb~& = -48
- 'IF aaa~& <= bbb~& THEN BEEP: END
- 'Next three line beep NOT
- 'aaa& = 61
- 'bbb& = -48
- 'IF aaa& <= bbb& THEN BEEP: END
- 'Next line works even when unsigned!
- 'IF File_Frame_y + YdimROW - (filecount - (YdimROW - 1)) <= 0 THEN
- 'CAUTION ]
- 'Next line doesn't work when unsigned!
- File_Frame_y = File_Frame_y + YdimROW
- 'IF CurrentLine < filecount THEN CurrentLine = CurrentLine + 1
- 'IF CurrentLine + 1 <= filecount THEN CurrentLine = CurrentLine + 1
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- UpdateCLine cryOLD, 1, NormalFRGr, NormalBCKGr, cryOLD
- cryOLD = cry
- cry = CSRLIN
- crxOLD = crx
- cryOLD = cry
- File_Frame_x = 1
- File_Frame_y = 1
- CurrentLine = 1
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- cry = CSRLIN
- crxOLD = crx
- cryOLD = cry
- File_Frame_x = 1
- File_Frame_y = filecount - (YdimROW - 1)
- CurrentLine = filecount
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- cry = CSRLIN
- crxOLD = crx
- cryOLD = cry
- File_Frame_x = 1
- File_Frame_y = 1
- CurrentLine = filecount
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- UpdateCLine cry, 1, InverseFRGr, InverseBCKGr, cry
- 'DO WHILE INKEY$ <> "": LOOP ' have to clear the keyboard buffer
- SUB ReturnCOMBO
- SHARED LSHIFT_RSHIFT
- SHARED LCTRL_RCTRL
- SHARED LALT_RALT
- SHARED LALT_HOME
- SHARED LALT_END
- SHARED LALT_INS
- SHARED LALT_DEL
- SHARED LALT_PGUP
- SHARED LALT_PGDN
- SHARED LALT_Left
- SHARED LALT_Right
- SHARED LALT_Up
- SHARED LALT_Down
- SHARED RALT_HOME
- SHARED RALT_END
- SHARED RALT_INS
- SHARED RALT_DEL
- SHARED RALT_PGUP
- SHARED RALT_PGDN
- SHARED RALT_Left
- SHARED RALT_Right
- SHARED RALT_Up
- SHARED RALT_Down
- SHARED LSHIFT_HOME
- SHARED LSHIFT_END
- SHARED LSHIFT_INS
- SHARED LSHIFT_DEL
- SHARED LSHIFT_PGUP
- SHARED LSHIFT_PGDN
- SHARED LSHIFT_Left
- SHARED LSHIFT_Right
- SHARED LSHIFT_Up
- SHARED LSHIFT_Down
- SHARED RSHIFT_HOME
- SHARED RSHIFT_END
- SHARED RSHIFT_INS
- SHARED RSHIFT_DEL
- SHARED RSHIFT_PGUP
- SHARED RSHIFT_PGDN
- SHARED RSHIFT_Left
- SHARED RSHIFT_Right
- SHARED RSHIFT_Up
- SHARED RSHIFT_Down
- SHARED LCTRL_HOME
- SHARED LCTRL_END
- SHARED LCTRL_INS
- SHARED LCTRL_DEL
- SHARED LCTRL_PGUP
- SHARED LCTRL_PGDN
- SHARED LCTRL_Left
- SHARED LCTRL_Right
- SHARED LCTRL_Up
- SHARED LCTRL_Down
- SHARED RCTRL_HOME
- SHARED RCTRL_END
- SHARED RCTRL_INS
- SHARED RCTRL_DEL
- SHARED RCTRL_PGUP
- SHARED RCTRL_PGDN
- SHARED RCTRL_Left
- SHARED RCTRL_Right
- SHARED RCTRL_Up
- SHARED RCTRL_Down
- SHARED LSHIFT_LCTRL_HOME
- SHARED LSHIFT_LCTRL_END
- SHARED LSHIFT_LCTRL_INS
- SHARED LSHIFT_LCTRL_DEL
- SHARED LSHIFT_LCTRL_PGUP
- SHARED LSHIFT_LCTRL_PGDN
- SHARED LSHIFT_LCTRL_Left
- SHARED LSHIFT_LCTRL_Right
- SHARED LSHIFT_LCTRL_Up
- SHARED LSHIFT_LCTRL_Down
- SHARED LSHIFT_BackSpace
- SHARED LSHIFT_TAB
- SHARED LSHIFT_SPACE
- SHARED LSHIFT_ESC
- SHARED LSHIFT_ENTER
- SHARED RSHIFT_BackSpace
- SHARED RSHIFT_TAB
- SHARED RSHIFT_SPACE
- SHARED RSHIFT_ESC
- SHARED RSHIFT_ENTER
- SHARED LCTRL_SPACE
- SHARED LCTRL_ENTER
- SHARED RCTRL_SPACE
- SHARED RCTRL_ENTER
- 'LALT:
- 'RALT:
- 'LSHIFT:
- 'RSHIFT:
- 'LCTRL:
- 'RCTRL:
- 'LSHIFT+LCTRL: NOTICE: LSHIFT+LCTRL+Left triggers 3 variables on - 1] LSHIFT_LCTRL_Left 2] LSHIFT_Left 3] LCTRL_Left
- 'LSHIFT:
- 'RSHIFT:
- 'LCTRL:
- 'RCTRL:
- '_KEYCLEAR
- TabV% = 8
- b$ = "": f& = 0
- ' |
- ' |
- ' \|/
- 'TabV% - (f% - (f% \ TabV%) * TabV%) =
- 'TabV% - (f% MOD TabV%)
- f& = (f& \ TabV%) * TabV% + TabV%
- f& = f& + 1
- l$ = b$
- s$ = ""
- SHARED CLL_Field
- cry = CSRLIN
- SHARED YdimROW
- cry = CSRLIN
- SHARED YdimROW
- cry = CSRLIN
- cry = CSRLIN
- COLOR FRGR, BACKGR
- COLOR FRGR, BACKGR
- SHARED YdimROW
- SHARED TimeA, TimeB
- cry = CSRLIN
Post by: SMcNeill on January 24, 2021, 01:24:36 pm
OPEN file$ FOR INPUT AS #1
try:
OPEN file$ FOR BINARY AS #1
Post by: Pete on January 24, 2021, 01:35:51 pm
On a side note, I think I need to move to Bulgaria. Sure, I've only known five, well six people now from that country, but on an intelligence level they are all 10 for 10. Oh don't get me wrong, when it comes to brilliant people, the U.S. is a five-star country too, and one of those stars is half shaded in. Oh, and no comments from the peanut galley that if Pete moves to Bulgaria, then there goes the neighborhood!
Nice coding there Tex,
Pete
Post by: Sanmayce on January 24, 2021, 04:36:38 pm
Thank you @SMcNeill and @Pete , didn't know that LINE INPUT works in BINARY mode.
Now we know that for OED, "tons faster" is roughly 4.1x or 23,187 seconds / 5,571 seconds, see the screenshot.
The attached revision 2+ now differs only in these two lines:
Code: QB64: [Select]
- ' OPEN a$ FOR INPUT AS #1
- ...
- ' OPEN a$ FOR INPUT AS #1
Steve is spot on when it comes to using BINARY for speed. Just be sure to keep using your IF _FILEEXISTS condition statement. Unlike OPEN for INPUT, which errors out if the file doesn't exist, Open for BINARY just creates the file. BINARY can also be used to write to the file, but you have to use PUT statements. QB64 added the ability to use LINE INPUT instead of the standard GET statements, so all you have to do is swap out the FOR INPUT to FOR BINARY. Y0u already have the _FILEEXISTS, so you should be good to go.
Pete, your wording is as it should, much appreciated, the flow, the explanation, just, Mutsi!
Please both with Steve stick around as I intend in next weeks to add slowly more features helping the user with text stats/windows...
Dummy me, looking in my old archives I saved only files as 3D.ASM drawing the windows with shadows, but lost the .BAS files, have to rewrite from .ASM to .BAS, ugh.
[ This attachment cannot be displayed inline in 'Print Page' view ]
Also, the idea is this tool to work on Linux, remotely my brother did run for me QB64 for Linux, the video shows r.2+ scrolling the 15MB long 'Arabian Nights' file on one old laptop with 1366x768 resolution, notice how the window fits in:
YouTube butchered the scroll after rendering it, so the original .MKV is on my Internet Drive:
https://drive.google.com/file/d/1RvMejR3cb_IWBSr3Jw94c4cIBd8oKa8Y/view?usp=sharing (https://drive.google.com/file/d/1RvMejR3cb_IWBSr3Jw94c4cIBd8oKa8Y/view?usp=sharing)
In the attached r.2+ package, there is MASAKARI_64bit.elf (468,660 bytes) - the *nix executable (static, stripped, compressed with upx).
Latest Fedora 33 was used, very glad that QB64 is *nix friendly.
My cameraman (my brother) did fine job filming it, I am impressed with the quality of capturing, no tearing, no frame dropping, so glad that my Text Sidekick will work on Linux as well.
As for Bulgarians, I beg to differ, most of my countrymen are disgrace to the Bulgarian spirit, and that is to put it mildly. To me, "intelligence" means little, many smartasses are out there, the kind-and-appreciative mindset is what I seek and love in humans.
To me, Bulgaria is a wonderful part of the Earth, however soullessness is rampant here, the soulful people are minority, as everywhere I guess... nah, except Nihon.
Post by: Pete on January 24, 2021, 04:56:33 pm
Glad you made use of BINARY, but ouch on having to go from ASM to BASIC. I coded a C/C++ WP routine I wrote back into BASIC once. It was just about like rewriting the whole thing, too.
We'll stay in touch,
Pete
Post by: SMcNeill on January 24, 2021, 05:55:12 pm
OPEN file$ FOR BINARY AS #1
text$ = SPACE$(LOF(1))
GET #1, 1, text$ ‘read the whole file at once
CLOSE
DO
parse text$ for CRLF characters (usually CHR$(10) and CHR$(13), in some combination)*
LOOP until all of text$ is parsed
I’ve posted countless examples of this process on the forums before, so a search should find one with a little digging.
Overall time speeds will be quite a bit faster than reading over and over from the drive, but exact improvements depend on drive speed vs memory speed. (For example, a SSD would see less improvement than a 5200RPM drive)
With extremely large files, I’d definitely suggest the read-and-parse method to load the whole file.
*Windows usually has CHR$(13) + CHR$(10) line endings. Linux/Newer Mac (above version 9) use CHR$(10). Legacy Mac (before version 9) use CHR$(13).
I’d check the file for which of those the file contains, and then parse for it as your line delimiter.
Post by: SMcNeill on January 24, 2021, 06:09:35 pm
Code: [Select]
REDIM FileArray$(1 TO 100000)
LongestLine = 0
IF _FILEEXISTS(a$) THEN
PRINT: PRINT "Loading..."
_DISPLAY
filecount = 0
OPEN a$ FOR BINARY AS #1
DO UNTIL EOF(1)
LINE INPUT #1, l$
filecount = filecount + 1
IF UBOUND(FileArray$) < filecount THEN REDIM _PRESERVE FileArray$(1 TO UBOUND(FileArray$) + 100000)
LINE INPUT #1, FileArray$(filecount)
IF LEN(FileArray$(filecount))) > LongestLine THEN LongestLine = LEN(FileArray$(filecount))
LOOP
CLOSE #1
REDIM FileArray$(1 TO filecount)You are now reading and assigning your file to your array in a single pass, rather than having to read it once (to count lines), and then read it a second time (to assign it to your array).
Half the disk reads, twice the speed.
Post by: Sanmayce on January 26, 2021, 12:46:36 am
In this revision 3+, some changes were made, as:
- removing the flashing window when command line option as "/help" or "-h" are used;
- removing the PLAY statement, it has some dependencies on Linux;
- fixing the path during loading the file, I guess _CWD$ and _STARTDIR$ have to be swapped;
@SMcNeill Thanks, your suggestions got me thinking how to solve the whole "load crisis" in one sweep, didn't have enough time to play with it, in next days I hope will end the crisis by removing usage of strings altogether, I intend to use you malloc-like approach
Code: QB64: [Select]
and manually find lines' endings. Also your MACOS comment about CR endings prompted for parsing in a way to handle all the CRLF, LF, CR texts, maybe will replace every LF with CR unless it is paired in CRLF, in that case just skipping it. Thus we would have a modified text file having only CR as a ending.
In order to have tons faster loading, loading in 5000+ seconds should be dropped to 5 seconds, a ton being a thousand, you know. Will deal with this in next days, for now another thing interests me, making the simplistic-and-universal "File Load Menu List", universal, thanks to a tool I wrote (forked from the awesome WATCOM C) many years ago which works exactly the same with Windows and Linux, it is called Yoshi.exe or Youshi.elf, the .C source is also included in the attached to this post package. Basically, it lists all the files in current folder recursively, that is, giving us (optionally with full paths) a list ready to be used in Masakari, thus in next revisions I will make the loading of a file (both on Windows and Linux) the same and a CAKEWALK - just running some script (here m.bat) and scrolling within the filelist, pressing Enter and then loading this file in the very same window. This is my conception for dealing with problems - writing a dedicated module in .C and running it from a GUI, or outside before entering the GUI. In the incoming revisions many problems were solved already in superfast fashion, so Masakari will only invoke them.
[ This attachment cannot be displayed inline in 'Print Page' view ]
Code: QB64: [Select]
- ' Masakari.bas (DragDropWheelScroller.bas)
- ' written in QB64 v1.4 by Kaze, 2021-Jan-25
- ' Thanks go to the www.qb64.org/forum members for sharing useful excerpts/etudes
- ' Wow, this is nice indeed: http://www.qb64.org/wiki/Windows_Libraries
- ' Currently are implemented only:
- ' Mouse:
- ' Button 1 - sets the cursor and the inverse line to the chosen position
- ' Button 3 - PgDn
- ' Wheel Up - Up
- ' Wheel Dn - Dn
- ' Keyboard:
- ' Up
- ' Down
- ' Left - still no sideways scroll
- ' Right - still no sideways scroll
- ' LCtrl+Home - going to the top left position
- ' LCtrl+End - going to the bottom left position
- ' Alt+x or Alt+q - quit to the system, without demanding keypress.
- _TITLE "MASAKARI 128x60, The 'Holy Axe' English Text Sidekick"
- $EXEICON:'ColumnChart.ico'
- _CONSOLETITLE "Masakari console window"
- PRINT "Masakari, revision 3+, written in QB64 by Kaze"
- PRINT "Usage: Masakari filename|/help"
- PRINT "Usage: Masakari filename|-h"
- '_CONSOLE OFF
- '_SCREENSHOW
- _SCREENSHOW ' Why so?!
- 'DEFLNG A-Z
- '_DEFINE A-Z AS _UNSIGNED _INTEGER64
- FUNCTION MemInUsePercent~&&
- FUNCTION TotalPhysicalMem~&&
- FUNCTION FreePhysicalMem~&&
- FUNCTION TotalPagingFile~&&
- FUNCTION FreePagingFile~&&
- FUNCTION TotalVirtualMem~&&
- FUNCTION FreeVirtualMem~&&
- FUNCTION FreeExtendedMem~&&
- FUNCTION GetCPULoad#
- XdimCOL = 128
- YdimROW = 40 ' ensure old laptops with 768pixels vertical will hold the whole window
- wide& = _DESKTOPWIDTH
- high& = _DESKTOPHEIGHT
- SCREEN handle&
- _DEST handle&
- ' _DEST 0 refers to the present program SCREEN. You can use 0 to refer to the present program
- ' SCREEN.
- '_FONT 16 'wish we could use the old 8x16 bitmap/raster fonts from DOS times...
- ' Either one must be uncommented:
- '_AUTODISPLAY 'no need of refreshing
- PurpleFlag = 0
- 'PurpleFlag = 1
- NormalFRGr = 3
- NormalBCKGr = 0
- InverseFRGr = 0
- InverseBCKGr = 3
- COLOR NormalFRGr, NormalBCKGr
- 'Ugh, they are reversed?!
- 'PRINT "Current working directory path: "; CHR$(34); _CWD$; CHR$(34)
- 'PRINT "User's program calling path: "; CHR$(34); _STARTDIR$; CHR$(34)
- 'Y = CSRLIN 'save the row
- 'X = POS(0) 'save the column
- REDIM FileArray$(1) 'create dynamic array: 3880000 alocates 532MB - bigger values need 570+MB and give "Out Of Memory"
- _ACCEPTFILEDROP 'enables drag/drop functionality
- 'pressakey$ = INPUT$(1)
- a$ = ""
- PostfixedToHeader = CSRLIN
- ' MEMUSAGE by Steve [
- ' GetCPULoad = 0 is idle, 1 is fully used.
- ' Multiply by 100 for a percentage
- PRINT "Total Physical Memory: ";
- PRINT "Free Physical Memory: ";
- PRINT "Total Paging File: ";
- PRINT "Free Paging File: ";
- PRINT "Total Virtual Memory: ";
- PRINT "Free Virtual Memory: ";
- 'PRINT "Free Extended Virtual Memory:"
- 'PRINT " "; FreeExtendedMeml; " bytes"
- ' MEMUSAGE by Steve ]
- ReturnCOMBO
- 'FOR i = 1 TO _TOTALDROPPEDFILES
- 'a$ = _DROPPEDFILE(i)
- 'NEXT'
- _FINISHDROP 'If _FINISHDROP isn't called here then _TOTALDROPPEDFILES never gets reset.
- 'ELSE
- 'a$ = "Scroller.$$$"
- 'SHELL _HIDE "DIR /B *.* > Scroller.$$$"
- 'WrapFlag = 0
- 'Wwidth% = XdimCOL
- 'ul& = 0
- 'LongestLine = 0
- 'DO WHILE NOT EOF(1)
- ' LINE INPUT #1, l$: ul& = ul& + 1
- ' ExpandTabs l$
- ' IF LEN(l$) > LongestLine THEN LongestLine = LEN(l$)
- ' IF WrapFlag THEN
- ' lX$ = l$
- ' DO WHILE LEN(lX$) > Wwidth%
- ' Glupak% = Wwidth%
- ' DO UNTIL MID$(lX$, Glupak% + 1, 1) = " " AND MID$(lX$, Glupak%, 1) <> " "
- ' Glupak% = Glupak% - 1
- ' IF Glupak% = 0 THEN
- ' PRINT "Rejecting line(#"; LTRIM$(STR$(ul&)); ", "; LTRIM$(STR$(LEN(l$))); "chars) that cannot be wrapped!"
- ' _DISPLAY: SYSTEM
- ' GOTO B4Txpanar
- ' END IF
- ' LOOP
- ' PRINT #3, LEFT$(lX$, Glupak%)
- ' lX$ = STRING$(1, " ") + LTRIM$(MID$(lX$, Glupak% + 1, LEN(lX$) - (Glupak%)))
- ' LOOP
- ' PRINT #3, lX$
- ' B4Txpanar:
- ' END IF
- 'LOOP
- TimeA = TIMER
- LongestLine = 0
- filecount = 0
- ' OPEN a$ FOR INPUT AS #1
- filecount = filecount + 1
- CLOSE #1
- filecount = 0
- ' OPEN a$ FOR INPUT AS #1
- ExpandTabs l$
- NEXT j
- filecount = filecount + 1
- FileArray$(filecount) = l$
- CLOSE #1
- _LIMIT 5
- 'IF INKEY$ = CHR$(27) THEN SYSTEM
- 'IF keycode& = 27 THEN SYSTEM
- ' QB64 _KEYHIT and _KEYDOWN Values
- '
- 'Esc F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 Sys ScL Pause
- ' 27 15104 15360 15616 15872 16128 16384 16640 16896 17152 17408 34048 34304 +316 +302 +019
- '`~ 1! 2@ 3# 4$ 5% 6^ 7& 8* 9( 0) -_ =+ BkSp Ins Hme PUp NumL / * -
- '126 33 64 35 36 37 94 38 42 40 41 95 43 8 20992 18176 18688 +300 47 42 45
- ' 96 49 50 51 52 53 54 55 56 57 48 45 61
- 'Tab Q W E R T Y U I O P [{ ]} \| Del End PDn 7Hme 8/? 9PU +
- ' 9 81 87 69 82 84 89 85 73 79 80 123 125 124 21248 20224 20736 18176 18432 18688 43
- ' 113 119 101 114 116 121 117 105 111 112 91 93 92 55 56 57
- 'CapL A S D F G H J K L ;: '" Enter 4/?- 5 6/-?
- '+301 65 83 68 70 71 72 74 75 76 58 34 13 19200 19456 19712 E
- ' 97 115 100 102 103 104 106 107 108 59 39 52 53 54 n
- 'Shift Z X C V B N M ,< .> /? Shift ? 1End 2/? 3PD t
- '+304 90 88 67 86 66 78 77 60 62 63 +303 18432 20224 20480 20736 e
- ' 122 120 99 118 98 110 109 44 46 47 49 50 51 r
- 'Ctrl Win Alt Spacebar Alt Win Menu Ctrl ?- ? -? 0Ins .Del
- '+306 +311 +308 32 +307 +312 +319 +305 19200 20480 19712 20992 21248 13
- ' 48 46
- ' Lower value = LCase/NumLock On __________________ + = add 100000
- TimeB = TIMER
- 'PLAY "L8V2a-c-" 'removed in order to get rid of audio dependency
- CurrentLine = 1
- ' Here is the layout:
- ' The window-frame is 1..crx or 1..128 | The file-frame is 1..LongestLine
- ' . . | .
- ' . . | .
- ' cry 60 | filecount
- ' FileArrayWINDOW$(60) | FileArray(filecount)
- ' if File_Frame_x < 128 then PADDING to 128 else File_Frame_x = 1..LongestLine-(128-1)
- ' if File_Frame_y < 60 then PADDING to 60 else File_Frame_y = 1..filecount-(60-1)
- UpdateCLine 1, 1, InverseFRGr, InverseBCKGr, 1
- Dumbo$ = "File Size: " + AddCommas$(FileSize) + "; Longest Line: " + AddCommas$(LongestLine) + "; Total Lines: " + AddCommas$(filecount)
- UpdateNextToCCL_DONE MostRightField
- cry = CSRLIN
- crxOLD = crx
- cryOLD = cry
- File_Frame_x = 1
- File_Frame_y = 1
- ReturnCOMBO
- 'PLAY "L4V2f" 'removed in order to get rid of audio dependency
- ReportTimeToLoad MostRightField
- 'DO: a$ = INKEY$: LOOP UNTIL a$ <> "" ' prevent ASC empty string read error
- 'CASE 27: COLOR 7, 0: SYSTEM 'END
- CASE 72:
- cry = cry - 1 'up
- ELSE 'scrolling is needed
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- CASE 80:
- ELSE 'scrolling is needed
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- UpdateCLine cryOLD, 1, NormalFRGr, NormalBCKGr, cryOLD
- cryOLD = cry
- ELSE 'it 'cry' could be changed by Mouse Wheel too, check it
- ELSE 'scrolling is needed
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- cry = cry - 1 'up
- ELSE 'scrolling is needed
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- crxOLD = crx
- cryOLD = cry
- cry = _MOUSEY
- crx = _MOUSEX
- UpdateCLine cryOLD, 1, NormalFRGr, NormalBCKGr, cryOLD
- cryOLD = cryOLD - 1
- cryOLD = cryOLD + 1
- ' Don't execute PgDn (advancing the 'CurrentLine') if 'File_Frame_y' is not "eligible":
- 'IF File_Frame_y < filecount - (YdimROW - 1) THEN File_Frame_y = File_Frame_y + 1
- 'IF File_Frame_y + 1 <= filecount - (YdimROW - 1) THEN File_Frame_y = File_Frame_y + 1
- 'CAUTION [
- ' Next three line beep NOT
- 'DEFLNG A-Z
- 'aaa = 61
- 'bbb = -48
- 'IF aaa <= bbb THEN BEEP: END
- 'Next three line beep
- 'aaa~& = 61
- 'bbb~& = -48
- 'IF aaa~& <= bbb~& THEN BEEP: END
- 'Next three line beep NOT
- 'aaa& = 61
- 'bbb& = -48
- 'IF aaa& <= bbb& THEN BEEP: END
- 'Next line works even when unsigned!
- 'IF File_Frame_y + YdimROW - (filecount - (YdimROW - 1)) <= 0 THEN
- 'CAUTION ]
- 'Next line doesn't work when unsigned!
- File_Frame_y = File_Frame_y + YdimROW
- 'IF CurrentLine < filecount THEN CurrentLine = CurrentLine + 1
- 'IF CurrentLine + 1 <= filecount THEN CurrentLine = CurrentLine + 1
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- UpdateCLine cryOLD, 1, NormalFRGr, NormalBCKGr, cryOLD
- cryOLD = cry
- cry = CSRLIN
- crxOLD = crx
- cryOLD = cry
- File_Frame_x = 1
- File_Frame_y = 1
- CurrentLine = 1
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- cry = CSRLIN
- crxOLD = crx
- cryOLD = cry
- File_Frame_x = 1
- File_Frame_y = filecount - (YdimROW - 1)
- CurrentLine = filecount
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- cry = CSRLIN
- crxOLD = crx
- cryOLD = cry
- File_Frame_x = 1
- File_Frame_y = 1
- CurrentLine = filecount
- UpdateWindowFrame NormalFRGr, NormalBCKGr
- UpdateCLine cry, 1, InverseFRGr, InverseBCKGr, cry
- 'DO WHILE INKEY$ <> "": LOOP ' have to clear the keyboard buffer
- '_TITLE "Mouse Feedback"
- 'DO
- ' DO WHILE _MOUSEINPUT ' Check the mouse status
- ' PRINT _MOUSEX, _MOUSEY, _MOUSEBUTTON(1); _MOUSEBUTTON(2); _MOUSEBUTTON(3), _MOUSEWHEEL
- ' LOOP
- 'LOOP UNTIL INKEY$ <> ""
- 'Syntax:
- 'infoExists%% = _MOUSEINPUT
- 'Description:
- '- Returns -1 if new mouse information is available, otherwise it returns 0.
- '- Must be called before reading any of the other mouse functions. The function will not miss any
- ' mouse input even during an INPUT entry.
- '- Use in a loop to monitor the mouse buttons, scroll wheel and coordinate positions.
- '- To clear all previous mouse data, use _MOUSEINPUT in a loop until it returns 0.
- ' _FONT (function) creates a new alphablended font handle from a designated image handle
- ' _FONT (statement) sets the current _LOADFONT function font handle to be used by PRINT or
- ' _PRINTSTRING.
- ' _FONTHEIGHT (function) returns the font height of a font handle created by _LOADFONT.
- ' _FONTWIDTH (function) returns the font width of a MONOSPACE font handle created by _LOADFONT.
- ' _FREEFONT (statement) frees a font handle value from memory
- ' _LOADFONT (function) loads a TrueType font (.TTF) file of a specific size and style and returns
- ' a font handle value.
- ' _DISPLAY (statement) turns off automatic display while only displaying the screen changes when
- ' called.
- ' SHELL (QB64 function) executes a DOS command or calls another program. Returns codes sent by END
- ' or SYSTEM.
- ' _SHELLHIDE (function) hides a DOS command or call to another program. Returns codes sent by END
- ' or SYSTEM.
- ' _CWD$ (function) returns the current working directory path as a STRING.
- ' _DONTWAIT (SHELL action) allows the program to continue without waiting for the other program to
- ' close.
- ' _FILEEXISTS (function) returns -1 if the file name string parameter exists. Zero if it does not.
- ' _HIDE (SHELL action) hides the DOS screen output during a shell.
- ' _SHELLHIDE (function) executes a DOS command or calls another program. Returns codes sent by END
- ' or SYSTEM.
- ' _STARTDIR$ (function) returns the user's program calling path as a STRING.
- 'The _MEMNEW function allocates new memory and returns a _MEM memory block referring to it.
- 'Syntax:
- ' memoryBlock = _MEMNEW(byteSize)
- '- The byteSize parameter is the desired byte size of the memory block based on the variable type
- ' it will hold.
- 'Description:
- '- The memoryBlock value created holds the elements .OFFSET, .SIZE, .TYPE and .ELEMENTSIZE.
- '- _MEMNEW does not clear the data previously in the memory block it allocates, for speed purposes.
- '- To clear previous data from a new memory block, use _MEMFILL with a byte value of 0.
- '- When a new memory block is created the memory .TYPE value will be 0.
- '- If the read only memory block .SIZE is 0, the memory block was not created.
- '- All values created by memory functions must be freed using _MEMFREE with a valid _MEM variable.
- 'Code Examples:
- 'Example: Shows how SINGLE numerical values can be passed, but non-fixed STRING lengths cannot get
- 'the value.
- 'DIM m AS _MEM
- 'DIM f AS STRING * 5
- 'm = _MEMNEW(5) 'create new memory block of 5 bytes
- 'a = 12345.6
- '_MEMPUT m, m.OFFSET, a 'put single value
- '_MEMGET m, m.OFFSET, b 'get single value
- 'PRINT "b = "; b
- 'c$ = "Doggy"
- '_MEMPUT m, m.OFFSET, c$ 'put 5 byte string value
- '_MEMGET m, m.OFFSET, d$ 'get unfixed length string value
- '_MEMGET m, m.OFFSET, f 'get 5 byte string value
- 'e$ = _MEMGET(m, m.OFFSET, STRING * 5) 'get 5 byte string value
- 'PRINT "d$ = "; d$; LEN(d$) 'prints empty string
- 'PRINT "e$ = "; e$; LEN(e$)
- 'PRINT "f = "; f; LEN(f)
- ''OPENICON.BAS
- ''============
- ''Maximizes a minimized program using WinAPI.
- ''By Dav, DEC/2020
- 'DECLARE DYNAMIC LIBRARY "user32"
- ' FUNCTION OpenIcon& (BYVAL hwnd AS LONG)
- 'END DECLARE
- 'PRINT
- 'PRINT "Minimize this program at it will pop back up in 2 secs."
- 'DO
- ' IF _SCREENICON THEN
- ' SLEEP 2: x& = OpenIcon&(_WINDOWHANDLE)
- ' END IF
- 'LOOP
- ' http://www.qb64.org/wiki/CONSOLETITLE
- 'Example: Hiding the main program window while displaying the console window with a title.
- SUB ReturnCOMBO
- SHARED LSHIFT_RSHIFT
- SHARED LCTRL_RCTRL
- SHARED LALT_RALT
- SHARED LALT_HOME
- SHARED LALT_END
- SHARED LALT_INS
- SHARED LALT_DEL
- SHARED LALT_PGUP
- SHARED LALT_PGDN
- SHARED LALT_Left
- SHARED LALT_Right
- SHARED LALT_Up
- SHARED LALT_Down
- SHARED RALT_HOME
- SHARED RALT_END
- SHARED RALT_INS
- SHARED RALT_DEL
- SHARED RALT_PGUP
- SHARED RALT_PGDN
- SHARED RALT_Left
- SHARED RALT_Right
- SHARED RALT_Up
- SHARED RALT_Down
- SHARED LSHIFT_HOME
- SHARED LSHIFT_END
- SHARED LSHIFT_INS
- SHARED LSHIFT_DEL
- SHARED LSHIFT_PGUP
- SHARED LSHIFT_PGDN
- SHARED LSHIFT_Left
- SHARED LSHIFT_Right
- SHARED LSHIFT_Up
- SHARED LSHIFT_Down
- SHARED RSHIFT_HOME
- SHARED RSHIFT_END
- SHARED RSHIFT_INS
- SHARED RSHIFT_DEL
- SHARED RSHIFT_PGUP
- SHARED RSHIFT_PGDN
- SHARED RSHIFT_Left
- SHARED RSHIFT_Right
- SHARED RSHIFT_Up
- SHARED RSHIFT_Down
- SHARED LCTRL_HOME
- SHARED LCTRL_END
- SHARED LCTRL_INS
- SHARED LCTRL_DEL
- SHARED LCTRL_PGUP
- SHARED LCTRL_PGDN
- SHARED LCTRL_Left
- SHARED LCTRL_Right
- SHARED LCTRL_Up
- SHARED LCTRL_Down
- SHARED RCTRL_HOME
- SHARED RCTRL_END
- SHARED RCTRL_INS
- SHARED RCTRL_DEL
- SHARED RCTRL_PGUP
- SHARED RCTRL_PGDN
- SHARED RCTRL_Left
- SHARED RCTRL_Right
- SHARED RCTRL_Up
- SHARED RCTRL_Down
- SHARED LSHIFT_LCTRL_HOME
- SHARED LSHIFT_LCTRL_END
- SHARED LSHIFT_LCTRL_INS
- SHARED LSHIFT_LCTRL_DEL
- SHARED LSHIFT_LCTRL_PGUP
- SHARED LSHIFT_LCTRL_PGDN
- SHARED LSHIFT_LCTRL_Left
- SHARED LSHIFT_LCTRL_Right
- SHARED LSHIFT_LCTRL_Up
- SHARED LSHIFT_LCTRL_Down
- SHARED LSHIFT_BackSpace
- SHARED LSHIFT_TAB
- SHARED LSHIFT_SPACE
- SHARED LSHIFT_ESC
- SHARED LSHIFT_ENTER
- SHARED RSHIFT_BackSpace
- SHARED RSHIFT_TAB
- SHARED RSHIFT_SPACE
- SHARED RSHIFT_ESC
- SHARED RSHIFT_ENTER
- SHARED LCTRL_SPACE
- SHARED LCTRL_ENTER
- SHARED RCTRL_SPACE
- SHARED RCTRL_ENTER
- 'LALT:
- 'RALT:
- 'LSHIFT:
- 'RSHIFT:
- 'LCTRL:
- 'RCTRL:
- 'LSHIFT+LCTRL: NOTICE: LSHIFT+LCTRL+Left triggers 3 variables on - 1] LSHIFT_LCTRL_Left 2] LSHIFT_Left 3] LCTRL_Left
- 'LSHIFT:
- 'RSHIFT:
- 'LCTRL:
- 'RCTRL:
- '_KEYCLEAR
- TabV% = 8
- b$ = "": f& = 0
- ' |
- ' |
- ' \|/
- 'TabV% - (f% - (f% \ TabV%) * TabV%) =
- 'TabV% - (f% MOD TabV%)
- f& = (f& \ TabV%) * TabV% + TabV%
- f& = f& + 1
- l$ = b$
- s$ = ""
- SHARED CLL_Field
- cry = CSRLIN
- SHARED YdimROW
- cry = CSRLIN
- SHARED YdimROW
- cry = CSRLIN
- cry = CSRLIN
- COLOR FRGR, BACKGR
- COLOR FRGR, BACKGR
- SHARED YdimROW
- SHARED TimeA, TimeB
- cry = CSRLIN
- '@rem This is 'MyCompile.bat', should be placed and run from qb64\:
- '@rem 'recompile_win.bat' found at E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64\internal\temp:
- '@rem @echo off
- '@rem cd %0\..\
- '@rem echo Recompiling...
- '@rem cd ../c
- '@rem c_compiler\bin\g++ -mconsole -s -Wfatal-errors -w -Wall qbx.cpp libqb\os\win\libqb_1_4_0000000001000.o ..\..\.\internal\temp\icon.o -D DEPENDENCY_NO_SOCKETS -D DEPENDENCY_NO_PRINTER -D DEPENDENCY_ICON -D DEPENDENCY_NO_SCREENIMAGE parts\core\os\win\src.a -lopengl32 -lglu32 -mwindows -static-libgcc -static-libstdc++ -D GLEW_STATIC -D FREEGLUT_STATIC -lwinmm -lgdi32 -o "..\..\MASAKARI.exe"
- '@rem pause
- 'rem We need to copy 3 files to qb64\internal\c
- 'rem dynamic name-and-content (name changes should get the one generated after QB64 -c MASAKARI.BAS from qb64\internal\c\libqb\os\win): libqb_1_4_0000000001000.o
- 'rem static name-and-content (take it from qb64\internal\temp): icon.o
- 'rem static name-and-content (take it from qb64\internal\c\parts\core\os\win): src.a
- 'rem Should go to E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64\internal\c
- 'rem Assuming we are in the "home folder" i.e. qb64:
- 'cd internal\c
- 'g++ -mconsole -s -Wfatal-errors -w -Wall qbx.cpp libqb_1_4_0000000001000.o icon.o -D DEPENDENCY_NO_SOCKETS -D DEPENDENCY_NO_PRINTER -D DEPENDENCY_ICON -D DEPENDENCY_NO_SCREENIMAGE src.a -lopengl32 -lglu32 -mwindows -static-libgcc -static-libstdc++ -D GLEW_STATIC -D FREEGLUT_STATIC -lwinmm -lgdi32 -o "MASAKARI_mycompile.exe"
- '@rem The output of above:
- '@rem
- '@rem E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64>MyCompile.bat
- '@rem
- '@rem E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64>cd internal\c
- '@rem
- '@rem E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64\internal\c>g++ -mconsole -s -Wfatal-errors -w -Wall qbx.cpp libqb_1_4_0000000001000.o icon.o -D DEPENDENCY_NO_SOCKETS -D DEPENDENCY_NO_PRINTER -D DEPENDENCY_ICON -D DEPENDENCY_NO_SCREENIMAGE src.a -lopengl32 -lglu32 -mwindows -static-libgcc -static-libstdc++ -D GLEW_STATIC -D FREEGLUT_STATIC -lwinmm -lgdi32 -o "MASAKARI_mycompile.exe"
- '@rem
- '@rem E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64\internal\c>dir masa*
- '@rem Volume in drive E is Sanmayce_111GB
- '@rem Volume Serial Number is 1410-10F9
- '@rem
- '@rem Directory of E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64\internal\c
- '@rem
- '@rem 01/26/2021 06:11 AM 3,248,640 MASAKARI_mycompile.exe
- '@rem 1 File(s) 3,248,640 bytes
- '@rem 0 Dir(s) 36,627,079,168 bytes free
- '@rem
- '@rem
- '@rem E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64\internal\c>c:\WINDOWS\system32\fc MASAKARI_mycompile.exe ..\..\MASAKARI_original.exe /b
- '@rem Comparing files MASAKARI_mycompile.exe and ..\..\MASAKARI_ORIGINAL.EXE
- '@rem 00000088: 53 18
- '@rem 00000089: 96 93
- '@rem 000000D8: F9 BE
- '@rem 000000D9: A3 A0
- '@rem
- '@rem E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64\internal\c>
Post by: Dav on January 26, 2021, 09:07:03 am
Nice work.
- Dav
Post by: Sanmayce on January 26, 2021, 10:44:07 am
gradually will refine it, many things are to be tested, my goal is Linux and Windows variants to be identical.
Hope, after some weeks to share one functional tool, at least covering some must-have features, in the near future I envision merging all my C tools under the Masakari hood.
Post by: Pete on January 26, 2021, 10:48:23 am
Very thoughtfully written. Even takes into account displaying well on old laptops like mine.
Nice work.
- Dav
That, and I like his coding, too. I hope he sticks around after this project is completed. He was a good choice in this year's draft pick. And to think, we only had to trade Clippy for him!
Pete
Post by: SMcNeill on January 26, 2021, 06:05:12 pm
In order to have tons faster loading, loading in 5000+ seconds should be dropped to 5 seconds, a ton being a thousand, you know.
Here's some code for you to play around with:
Code: QB64: [Select]
And this is using a version of the OED which I found tucked away on a corner of my drive, though my copy is only about 350MB in size.
As you can see, I'm loading and parsing 6.7M lines in less than 2 seconds, and that's with a file of 374 MB (392,676,176 bytes) in size. Even if you double the size, you'd still only be looking at about 4 seconds to load and parse.
That's a ton faster, by your requirements as listed above. ;D
Post by: NOVARSEG on January 27, 2021, 01:18:19 am
Quote
(forked from the awesome WATCOM C) many years ago which works exactly the same with Windows and Linux, it is called Yoshi.exe or Youshi.elf, the .C source is also included in the attached to this post package. Basically, it lists all the files in current folder recursively, that is, giving us (optionally with full paths)
I did the same thing but wrote in assembler. It saves all the file names with full paths in a file for later processing. Walks the files on any USB stick or hard drive and it is fast.
Post by: Sanmayce on January 27, 2021, 05:08:41 am
Steve, a ton indeed, but what took you so long? Heh-heh.
Without strstr()-like, inhere instr(), exploitation, building these strings is causa perduta, I knew that, yet I wanted to know how QB64 behaves.
Last night have had few hours and have come up with new parser, which I enliked instantly, mainly because it is written with minimal (in a lazy manner I converted the string array to ... a string function) efforts/changes.
[ This attachment cannot be displayed inline in 'Print Page' view ]
It is maybe 30x (150 seconds) slower than Steve's, however it is written having Wikipedia in mind, it has sub-variant not needing the file to be in RAM, commented for now.
The benefits:
- now the parsing is in one pass only;
- the simulated "string" now has a field housing its length, so LEN() is unnecessary;
- for the sub-variant not loading the file into physical memory the RAM footprint is MUTSUNKA - only (8bytes offset + 8bytes length) x lines, or 16x4MB=64+MB (in fact 133MB) for OED.DSL with 4,071,706 lines.
Since my wish is to browse files as big as 'enwiki-20210101-pages-articles.xml' 75.6 GB (81,193,612,108 bytes) on machines with 32GB the current parser fits in, since this XML dump requires 18GB (1,218,205,075 lines x 16bytes).
I chose 8bytes to house the maximal line length, since there are DNA sequences (not wrapped) exceeding 4 billion chars.
Code: QB64: [Select]
- F:\_KAZE_CPU_Benchmark_Fuzzy-Search-Wikipedia_OCN>dir
- 01/17/2021 09:12 81,193,612,108 enwiki-20210101-pages-articles.xml
- 02/24/2018 12:23 69,120 LineWordreporter.exe
- 01/17/2021 05:16 52,761 LineWordreporter_r1stats.zip
- F:\_KAZE_CPU_Benchmark_Fuzzy-Search-Wikipedia_OCN>dir enwiki-20210101-pages-articles.xml/b 1>q
- F:\_KAZE_CPU_Benchmark_Fuzzy-Search-Wikipedia_OCN>LineWordreporter.exe q
- LineWordreporter, revision 1_stats, written by Kaze.
- Example:
- D:\>LineWordreporter.exe LQ2048.lst
- Note1: Files can exceed 4GB limit.
- Buffered counting ...
- Reading enwiki-20210101-pages-articles.xml ...
- LineWordreporter: Longest word: 104,895
- F:\_KAZE_CPU_Benchmark_Fuzzy-Search-Wikipedia_OCN>
Will play more when have time...
I did the same thing but wrote in assembler. It saves all the file names with full paths in a file for later processing. Walks the files on any USB stick or hard drive and it is fast.
Salute you, "trivial" task done well is what I enjoy.
Post by: bplus on January 27, 2021, 12:58:49 pm
https://www.qb64.org/forum/index.php?topic=1876.msg111064#msg111064
and InForm version:
https://www.qb64.org/forum/index.php?topic=1874.msg111060#msg111060
No drag and drop, you just navigate your hard drive files and folders, load files into reader and select text for clipboard copy/paste.
Ha! looks like I used slow file loading method and I could update it now for Linux users, but not much interest when posted.
So far with Masakari, I've only managed to drag and drop 1 file into it. Have I a need of more instruction? It seems to be done after displaying 1 file but I did figure a trick for drag and drop from Windows Explorer drag to icon on toolbar which pops Masakari and then can drag/drop in.
What is Masakari anyway?
Quote
Dictionary source: Medieval Glossary. More: English to English translation of masakari. (masa-kari) is the Japanese word for an "axe" or a "hatchet", and is used to describe various tools of similar structure. As with axes in other cultures, ono are sometimes employed as weapons.
Ah! :)
Post by: Sanmayce on January 28, 2021, 05:13:15 am
What is Masakari anyway?
Literally, a broadaxe, metaphorically, a powerful tool/weapon to cut evil/ignorance, I have made this booklet: http://www.sanmayce.com/MSKR/masakari_3.pdf (http://www.sanmayce.com/MSKR/masakari_3.pdf)
Also, as a big fan of Mech Commander I loved those battle mechs, the most awesome one was Masakari - the brutal destroyer in the field.
Here comes Masakari r.5 ...
Glad to share the first complete revision, featuring:
- Ability to load a filelist (a file containing filenames, with paths or not) pressing SPACE and the selected one will be loaded in the same window;
- Ability to browse files bigger than available RAM;
- The parser is 64bit i.e. 4+GB files can be uploaded in fast mode;
- Two modes, fast (memory used will be filesize+lines*16) and efficient (smaller memory footprint, just lines*16 bytes);
- Built-in benchmark #1: LShift+RShift - Reporting (in the status line in red color) the time for load;
- Built-in benchmark #2: LCtrl+RCtrl - Reporting (in the status line in red color) the time for PgDn-ing the entire file;
On my laptop (i5-7200u @2.5GHz, 36GB DDR4 2133MHz, Windows 10, Samsung 860 PRO 256GB SATA 3), current r.5 loaded the Oxford English Dictionary in 96 seconds, in its fast mode (i.e. ToLoadOrNotFlag = 1).
In its efficient mode (i.e. ToLoadOrNotFlag = 0) the load time was 3,342 seconds.
However, Task Manager shows Memory Usage: 1248MB for fast and 141MB for the efficient one.
The benchmark #2 simulates holding PgDn key until the end is reached, in fast mode it gave 1162 seconds, or 4,071,706 lines / 60 lines / 1162 seconds = 58 Pages-Per-Second.
The benchmark #2 simulates holding PgDn key until the end is reached, in efficient mode it gave 1241 seconds, or 4,071,706 lines / 60 lines / 1241 seconds = 54 Pages-Per-Second.
In the incoming days will run Wikipedia benchmark i.e. loading the 76GB in fast mode, curious to see how much time will take.
Wanna use this revision for a while to see what else could be refined and simplified, I need a quick browser in Linux and Windows command prompts, two keys only in order to browse all files in current folder and its branches, and by hitting space to load for browsing desired file - 3 keys in total:
1] m
2] enter
[mouse wheel, or arrows to choose from a filelist]
3] space
The first being a .sh file executing yoshi.elf and masakari.elf - the three placed in bin/
Edit: Had to fix one bug, namely:
Code: QB64: [Select]
- 'Ugh, buggy, in r.5 below two "malloc" lines should be using 'NumberOfLFs' not 'FileSize'...
- MhandleOFF = _MEMNEW(8&& * (NumberOfLFs + 1)) 'create new memory block of 8*NumberOfLFs bytes - each line has its own 64bit Offset
- MhandleLEN = _MEMNEW(8&& * (NumberOfLFs + 1)) 'create new memory block of 8*NumberOfLFs bytes - each line has its own 64bit Length, kinda overkill, could be 32bit
Now loading Wikipedia dump...
Loaded successfully in ... few hours.
Two things to be updated, pressing the two Shifts has to be revised (status line is not wide enough for GB in size files):
- expand the window from 128 to at least 198, in 1920x1080, see the current line - this line will be in next revisions - I have plans those 60 columns to be used as a side window (with purple/indigo foreground and black background), sidekicking/assisting the left window - while they both being in one physical window.
In next days, I intend to make a single add-on, showing in this (already calling it 'Indigo') window the richest wordlist of English language - the Schizandrafield revision C - being 788,068,084 distinct-words strong! Crazy is good, isn't it, thus when loading some text in Masakari, the user would see whether the word (the phrase, in the future, also) where cursor is appear in corpora listed below, automatically the Indigo "window" will be positioned at the proper line (of course the Schizandrafield will not be loaded in RAM but searched as a fixed length field)...
- the reported seconds were negative, still don't have adequate function returning seconds when midnight is passed...
Edit, 2021-Jan-29:
Okay, did fix the above two issues, the load time in fast mode was 14048/3600= 4- hours (when have enough time will write faster parser), a side issue is that after the Windows 10 PRO update 2020 H2 OS Build 19042.746 my CPU was "downgraded" (CPU-Z and AIDA64 both report 13x multiplier instead of 31x) somehow from having 26GB/s Memory Read (3.1GHz max turbo) down to 16GB/s Memory Read (1.3GHz with no turbo), therefore results on other machines should be better. My first guess is that the hacking (installing 32GB stick which was said was impossible) triggered this behavior, updating BIOS could resolve it. Anyway, by chance my value of 65536MB for swap file in Windows was by chance enough in this Wikipedia load, 36GB + 64GB Virtual = 99GB, after loading the 76GB Wikipedia only 400MB were free, after some minutes Windows did automatic compression and freed ~8GB, so my advice is one to have 128GB virtual RAM since Wikipedia is steadily increased in size, AFAIR 6 years ago it was half that size.
Code: QB64: [Select]
- WKE_0,000,001_kaoudi
Three fields constitute the whole line (to be padded with spaces at the right):
3 bytes for TAG
2 bytes for two delimiters - the underscore
9 bytes for Number Of Occurrences
32 bytes for the word (the longest word in Heritage Dictionary was 31)
So, 3+2+9+32=46, even 14 more characters left.
The booklet is in .PDF here:
Code: [Select]
[Schizandrafield 1-gram Corpus, revision C, derives from next corpora:]
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Corpus Tag, Name | Corpus size (in bytes) | Total Words | Unique Words | Needed memory to rip in a single pass |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| AHD, American_Heritage_Dictionary_4_(En-En)_WHOLEWORDS.dsl | 41,742,099 | 7,083,439 | 176,377 | 16,512KB |
| BNC, Machine-Learning_British-National-Corpus_XML-edition.tar | 4,680,140,800 | 980,238,337 | 367,921 | 34,389KB |
| BRE, Britannica_Encyclopedia_2010_1.563_miled_(En-En)_ANSI.dsl | 297,981,779 | 46,958,317 | 356,417 | 33,322KB |
| CAL, Cambridge_Advanced_Learner's_Dictionary_4th_Ed_(En-En).dsl | 76,257,922 | 13,994,351 | 57,787 | 5,417KB |
| CCA, Collins_COBUILD_Advanced_Learner's_English_Dictionary_(5th_ed)_(En-En)_WHOLEWORDS.dsl | 22,960,485 | 4,553,432 | 61,575 | 5,772KB |
| DMC, DeepMind_Q_and_A_Dataset_cnn_downloads_(92579_files).tar | 7,270,204,928 | 988,686,300 | 499,244 | 46,623KB |
| DMD, DeepMind_Q_and_A_Dataset_dailymail_downloads_(219506_files).tar | 59,019,643,392 | 7,533,039,068 | 886,887 | 82,582KB |
| EDR, encyclopediadramaticase-20150628-current.tar | 302,959,104 | 41,642,066 | 456,251 | 42,635KB |
| EJD, Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar | 107,784,192 | 16,158,364 | 195,273 | 18,281KB |
| EJN, ENAMDICT_Japanese_names | 26,392,511 | 1,645,705 | 343,460 | 32,106KB |
| FDU, For_Dummies_978-ebooks_Collection.tar | 811,308,544 | 122,351,592 | 302,353 | 28,282KB |
| GGB, Google_Books_corpus_version_20130501_English_All_Nodes.txt | 10,624,363,237 | 178,439,407 | 7,477,257 | 664,635KB |
| HCN, Hacker_News_2006_to_2017-jul.json | 7,046,506,518 | 1,075,832,079 | 2,188,058 | 201,838KB |
| IST, INTERNET_SACRED_TEXT_ARCHIVE_DVD-ROM_9_(English_140479_htm_files).tar | 2,037,880,832 | 304,410,076 | 1,333,036 | 123,688KB |
| LDC, Longman_Dictionary_of_Contemporary_English_5th_Ed_(En-En)_WHOLEWORDS.dsl | 52,870,741 | 9,722,686 | 85,217 | 7,987KB |
| MCD, Macmillan_English_Dictionary_(En-En)_.dsl | 79,686,074 | 11,750,813 | 67,340 | 6,312KB |
| MCT, Macmillan_English_Thesaurus_(En-En).dsl | 29,580,755 | 4,650,089 | 39,528 | 3,708KB |
| NSO, New_Shorter_Oxford_English_Dictionary_fifth_edition.tar | 132,728,832 | 25,920,769 | 259,990 | 24,321KB |
| OED, Oxford_English_Dictionary_2nd_Edition_Version_4_(En-En)_WHOLEWORDS.dsl.txt | 564,235,251 | 101,798,550 | 1,089,240 | 101,214KB |
| OSH, OSHO.TXT | 206,908,949 | 31,957,006 | 58,893 | 5,522KB |
| PGT, Project_Gutenberg_DVD-2010_(29180_files).tar | 11,110,769,152 | 1,870,216,915 | 3,847,963 | 350,566KB |
| RDD, Reddit_Comments_(JSON_objects)_from_(2005-12_to_2018-01).json | 2,277,975,364,152 | 333,829,940,270 | 689,949,388 | 54,703,959KB |
| RHW, Random_House_Webster's_Unabridged_Dictionary_(En-En)_.dsl | 53,483,152 | 9,367,457 | 282,580 | 26,428KB |
| SNT, Machine-Learning_WestburyLab.NonRedundant.UsenetCorpus_(47860_English_language_non-binary-file_news_groups).tar | 39,513,013,248 | 6,316,689,948 | 4,835,188 | 437,662KB |
| STX, archive.org_stackexchange_(346_corpora_2017-Oct-12).tar | 274,935,801,856 | 38,077,068,727 | 29,194,792 | 2,344,214KB |
| TAL, the-anarchist-library-2016-01-18-en_html.tar | 153,703,936 | 24,339,935 | 136,000 | 12,738KB |
| TXF, TEXTFILES.COM_(58096_files).tar | 1,382,122,496 | 192,893,874 | 1,008,780 | 93,840KB |
| URB, Machine-Learning_Urban_Dictionary_Definitions_Corpus_(1999_-_May-2016).words.json | 1,917,822,288 | 263,253,093 | 2,631,962 | 241,852KB |
| WKD, dumps.wikimedia.org_Germany_dewiki-20180220-pages-articles.xml | 18,954,897,343 | 2,362,729,484 | 17,415,343 | 1,467,593KB |
| WKE, dumps.wikimedia.org_English_enwiki-20180220-pages-articles.xml | 65,865,333,874 | 8,739,196,084 | 39,440,894 | 3,071,920KB |
| WKF, dumps.wikimedia.org_France_frwiki-20180220-pages-articles.xml | 17,802,386,071 | 2,429,769,009 | 12,192,025 | 1,055,599KB |
| WKI, dumps.wikimedia.org_Italy_itwiki-20180220-pages-articles.xml | 10,887,321,918 | 1,372,430,005 | 8,960,466 | 790,544KB |
| WKN, dumps.wikimedia.org_Netherlands_nlwiki-20180220-pages-articles.xml | 6,808,875,477 | 800,283,886 | 8,596,100 | 760,225KB |
| WKP, dumps.wikimedia.org_Portugal_ptwiki-20180220-pages-articles.xml | 6,891,588,341 | 940,571,722 | 6,736,349 | 602,633KB |
| WKS, dumps.wikimedia.org_Spain_eswiki-20180220-pages-articles.xml | 12,200,295,384 | 1,682,091,803 | 9,780,910 | 858,723KB |
| WMB, dumps.wikimedia.org_English_enwikibooks-20180220-pages-articles.xml | 641,413,774 | 94,801,223 | 971,592 | 90,438KB |
| WMN, dumps.wikimedia.org_English_enwikinews-20180220-pages-articles.xml | 201,872,863 | 27,328,349 | 404,890 | 37,839KB |
| WMP, dumps.wikimedia.org_English_specieswiki-20180220-pages-articles.xml | 1,009,303,358 | 107,856,282 | 2,765,079 | 254,016KB |
| WMQ, dumps.wikimedia.org_English_enwikiquote-20180220-pages-articles.xml | 410,396,147 | 64,809,361 | 561,894 | 52,455KB |
| WMS, dumps.wikimedia.org_English_enwikisource-20180220-pages-articles.xml | 8,352,677,820 | 1,282,920,260 | 8,547,509 | 755,716KB |
| WMU, dumps.wikimedia.org_English_enwikiversity-20180220-pages-articles.xml | 369,987,893 | 52,322,592 | 640,606 | 59,767KB |
| WMV, dumps.wikimedia.org_English_enwikivoyage-20180220-pages-articles.xml | 354,919,743 | 49,701,293 | 734,403 | 68,474KB |
| WMW, dumps.wikimedia.org_English_enwiktionary-20180220-pages-articles.xml | 5,379,842,566 | 597,315,425 | 14,743,543 | 1,259,179KB |
| WUD, Webster's_Unabridged_3_(En-En)_WHOLEWORDS_ANSI.dsl | 134,706,719 | 24,014,478 | 364,352 | 34,052KB |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[Schizandrafield 1-gram Corpus, revision C, holds within next tagged-counted-wordlists:]
237,859,509 dumps.wikimedia.org_Spain_eswiki-20180220-pages-articles.1gram
161,745,186 dumps.wikimedia.org_Portugal_ptwiki-20180220-pages-articles.1gram
211,751,899 dumps.wikimedia.org_Netherlands_nlwiki-20180220-pages-articles.1gram
216,667,608 dumps.wikimedia.org_Italy_itwiki-20180220-pages-articles.1gram
969,819,544 dumps.wikimedia.org_English_enwiki-20180220-pages-articles.1gram
452,471,230 dumps.wikimedia.org_Germany_dewiki-20180220-pages-articles.1gram
294,822,021 dumps.wikimedia.org_France_frwiki-20180220-pages-articles.1gram
23,100,108 dumps.wikimedia.org_English_enwikibooks-20180220-pages-articles.1gram
204,113,124 dumps.wikimedia.org_English_enwikisource-20180220-pages-articles.1gram
9,347,743 dumps.wikimedia.org_English_enwikinews-20180220-pages-articles.1gram
13,259,587 dumps.wikimedia.org_English_enwikiquote-20180220-pages-articles.1gram
15,131,207 dumps.wikimedia.org_English_enwikiversity-20180220-pages-articles.1gram
18,031,161 dumps.wikimedia.org_English_enwikivoyage-20180220-pages-articles.1gram
355,881,851 dumps.wikimedia.org_English_enwiktionary-20180220-pages-articles.1gram
65,846,403 dumps.wikimedia.org_English_specieswiki-20180220-pages-articles.1gram
4,653,581 Encyclopaedia_Judaica_(in_22_volumes)_TXT.1gram
8,889,960 Webster's_Unabridged_3_(En-En)_WHOLEWORDS_ANSI.dsl.1gram
3,288,498 the-anarchist-library-2016-01-18-en_html.tar.1gram
6,973,687 Random_House_Webster's_Unabridged_Dictionary_(En-En)_.dsl.1gram
26,501,920 Oxford_English_Dictionary_2nd_Edition_Version_4_(En-En)_WHOLEWORDS.dsl.txt.1gram
1,404,700 OSHO.TXT.1gram
6,280,919 New_Shorter_Oxford_English_Dictionary_fifth_edition.tar.1gram
941,039 Macmillan_English_Thesaurus_(En-En).dsl.1gram
1,609,423 Macmillan_English_Dictionary_(En-En)_.dsl.1gram
2,044,933 Longman_Dictionary_of_Contemporary_English_5th_Ed_(En-En)_WHOLEWORDS.dsl.1gram
7,354,174 For_Dummies_978-ebooks_Collection.tar.1gram
11,464,911 encyclopediadramaticase-20150628-current.tar.1gram
8,840,362 ENAMDICT_Japanese_names.1gram
1,466,506 Collins_COBUILD_Advanced_Learner's_English_Dictionary_(5th_ed)_(En-En)_WHOLEWORDS.dsl.1gram
1,380,759 Cambridge_Advanced_Learner's_Dictionary_4th_Ed_(En-En).dsl.1gram
8,621,181 Britannica_Encyclopedia_2010_1.563_miled_(En-En)_ANSI.dsl.1gram
4,268,314 American_Heritage_Dictionary_4_(En-En)_WHOLEWORDS.dsl.1gram
186,006,261 Google_Books_corpus_version_20130501_English_All_Nodes.1gram
32,262,044 INTERNET_SACRED_TEXT_ARCHIVE_DVD-ROM_9_(English_140479_htm_files).1gram
96,997,413 Project_Gutenberg_DVD-2010_(29180_files).1gram
24,179,167 TEXTFILES.COM_(58096_files).1gram
797,764,729 archive.org_stackexchange_(346_corpora_2017-Oct-12).1gram
8,823,193 Machine-Learning_British-National-Corpus_XML-edition.1gram
65,616,610 Machine-Learning_Urban_Dictionary_Definitions_Corpus_(1999_-_May-2016).words.1gram
120,939,472 Machine-Learning_WestburyLab.NonRedundant.UsenetCorpus_(47860_English_language_non-binary-file_news_groups).1gram
12,375,594 DeepMind_Q_and_A_Dataset_cnn_downloads_(92579_files).1gram
20,878,468 DeepMind_Q_and_A_Dataset_dailymail_downloads_(219506_files).1gram
16,261,229,827 reddit.1gram
54,744,506 Hacker_News_2006_to_2017-jul.json.1gram
[The way they were ripped:]
E:\Schizandrafield_workshop>dir dumps.wikimedia.org_English_enwiki-20180220-pages-articles.xml/b >dumps.wikimedia.org_English_enwiki-20180220-pages-articles.lst
E:\Schizandrafield_workshop>echo WKE>Leprechaun.tag
E:\Schizandrafield_workshop>Leprechaun_x-leton_32bit_Intel_01_008p.exe dumps.wikimedia.org_English_enwiki-20180220-pages-articles.lst dumps.wikimedia.org_English_enwiki-20180220-pages-articles.1gram 1399888 Y
...
E:\Schizandrafield_workshop>type dumps.wikimedia.org_English_enwiki-20180220-pages-articles.1gram|more
WKE_0,000,003_byjnf
WKE_0,000,001_richardmullaney
WKE_0,000,001_vfycdm
WKE_0,000,001_kaoudi
WKE_0,000,001_bzolqoptifgvgptya
WKE_0,000,016_bristolcombination
WKE_0,000,001_hermanaustinamstate
WKE_0,000,004_meweyqhwn
WKE_0,000,002_habsfrontcricket
WKE_0,000,001_hobergsclub
WKE_0,000,001_kwagedle
WKE_0,000,001_buvvhel
WKE_0,000,001_wnjyat
WKE_0,000,001_dilmenpeds
WKE_0,000,001_ovldtxrjaaqiy
...
E:\Schizandrafield_workshop>
[The way they were concatenated/sorted:]
C:\>copy/b *.1gram unsorted
C:\>sort.exe /+15 /M 1048576 /T d: "unsorted" /O "Schizandrafield_Corpus_revision_C_(44-corpora_-unique-words).sorted"
Thanks to my brother, here the video and the Linux trio (m,yoshi,masakari - just place them in folder with path to it) come:
Post by: Sanmayce on February 01, 2021, 12:46:21 pm
Gunlight
Daylight
Sunset
Loss
My word is for Zennish sensitivity... to be activated and the artistism within to reveal, by itself, nifty practical etudes.
In r.6 were added:
- Four different sub-variants: r6_Fast_Wrapper, r6_Slow_Wrapper, r6_Fast_Vanilla, r6_Slow_Vanilla:
-- Fast/Slow mode - decided by the variable 'ToLoadOrNotFlag', 1/0 respectively
-- Wrapper/Vanilla mode - decided by the variable 'WrapFlag', 1/0 respectively
- When loading, there is progress bar in the 'Status Line'
- In the 'Status Line' now reporting Current Line Number (was reluctant to add it since my plan for streaming was in conflict, meh)
- Title is now containing the actual size of the text window, plus the filename of loaded file
- The command prompt now offers "help" i.e. all the keyboard and mouse combos.
- In order to browse via mouse only, I have come up with two quite ergonomic (the left hand hitting 'm' and 'Enter' and 'Space', while the right is on the mouse non-stop) mouse combos, mimicking 'LCtrl+Home' and 'LCtrl+End':
-- Button 2 + Wheel Up - going to the top left position
-- Button 2 + Wheel Dn - going to the bottom left position
In r.6 were fixed:
- Now, Left Mouse click cannot be placed past last line (when loaded lines are less than Window Height i.e. YdimROW)
The screenshot shows Oxford English Dictionary loaded by r6_Fast_Wrapper_64bit on my crippled laptop with i5-7200U 1.3GHz (32+4)GB:
The screenshot shows English Wikipedia XML dump from 2021-Jan-01 loaded by r6_Fast_Vanilla_64bit on laptop with Ryzen 7 4800H 4300MHz, 2x32GB 3200MHz:
Also, a batch file compiling the four sub-variants is included:
Code: [Select]
E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64>_Make_EXEs.bat
E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64>qb64 -c MASAKARI_r6_Fast_Vanilla.BAS
E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64>qb64 -c MASAKARI_r6_Fast_Wrapper.BAS
E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64>qb64 -c MASAKARI_r6_Slow_Vanilla.BAS
E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64>qb64 -c MASAKARI_r6_Slow_Wrapper.BAS
E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64>fc MASAKARI_r6_Fast_Vanilla.BAS MASAKARI_r6_Fast_Wrapper.BAS
Comparing files MASAKARI_r6_Fast_Vanilla.BAS and MASAKARI_R6_FAST_WRAPPER.BAS
***** MASAKARI_r6_Fast_Vanilla.BAS
WrapFlag = 0 ' 1 means wrapping
ToLoadOrNotFlag = 1 ' 1 means fast load but memory greedy; 0 means slow load but memory efficient
***** MASAKARI_R6_FAST_WRAPPER.BAS
WrapFlag = 1 ' 1 means wrapping
ToLoadOrNotFlag = 1 ' 1 means fast load but memory greedy; 0 means slow load but memory efficient
*****
E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64>fc MASAKARI_r6_Fast_Vanilla.BAS MASAKARI_r6_Slow_Vanilla.BAS
Comparing files MASAKARI_r6_Fast_Vanilla.BAS and MASAKARI_R6_SLOW_VANILLA.BAS
***** MASAKARI_r6_Fast_Vanilla.BAS
WrapFlag = 0 ' 1 means wrapping
ToLoadOrNotFlag = 1 ' 1 means fast load but memory greedy; 0 means slow load but memory efficient
***** MASAKARI_R6_SLOW_VANILLA.BAS
WrapFlag = 0 ' 1 means wrapping
ToLoadOrNotFlag = 0 ' 1 means fast load but memory greedy; 0 means slow load but memory efficient
*****
E:\_KAZE_Smxrt_Benchmarks\QB64_kit_v1.4_2.48 GB\qb64>fc MASAKARI_r6_Fast_Vanilla.BAS MASAKARI_r6_Slow_Wrapper.BAS
Comparing files MASAKARI_r6_Fast_Vanilla.BAS and MASAKARI_R6_SLOW_WRAPPER.BAS
***** MASAKARI_r6_Fast_Vanilla.BAS
WrapFlag = 0 ' 1 means wrapping
ToLoadOrNotFlag = 1 ' 1 means fast load but memory greedy; 0 means slow load but memory efficient
***** MASAKARI_R6_SLOW_WRAPPER.BAS
WrapFlag = 1 ' 1 means wrapping
ToLoadOrNotFlag = 0 ' 1 means fast load but memory greedy; 0 means slow load but memory efficient
*****
In the attached .ZIP package, there is README.TXT:
Code: [Select]
README.TXT
A quick .DIZ for Masakari, revision 6
Masakari is a free and open source tool with simplistic GUI aiming at sidekicking browsing text files in Linux/Windows command prompts.
This .ZIP package contains:
```
02/01/2021 03:16 PM 4,446 README.TXT ! This file
01/29/2021 03:05 PM 171 m ! Linux .sh file invokes yoshi and masakari
01/29/2021 03:05 PM 336,644 yoshi ! Linux .ELF 64bit
02/01/2021 06:43 PM 331,852 masakari ! Linux .ELF 64bit same as 'MASAKARI_r6_Fast_Vanilla_64bit'
02/01/2021 03:16 PM 1,453,926 ColumnChart.ico ! Needed during compilation
02/01/2021 03:16 PM 2,991 MEM.H ! Needed during compilation
02/01/2021 03:16 PM 492 m.bat ! Accepts wildcards, invokes Yoshi.exe and MASAKARI_r6_Fast_Vanilla_64bit.exe
02/01/2021 03:16 PM 52,633 MASAKARI_r6_Fast_Vanilla.BAS
02/01/2021 03:16 PM 3,288,576 MASAKARI_r6_Fast_Vanilla_32bit.exe
02/01/2021 03:16 PM 3,647,488 MASAKARI_r6_Fast_Vanilla_64bit.exe
02/01/2021 06:43 PM 331,852 MASAKARI_r6_Fast_Vanilla_64bit.elf
02/01/2021 03:16 PM 52,633 MASAKARI_r6_Fast_Wrapper.BAS
02/01/2021 03:16 PM 3,288,576 MASAKARI_r6_Fast_Wrapper_32bit.exe
02/01/2021 03:16 PM 3,647,488 MASAKARI_r6_Fast_Wrapper_64bit.exe
02/01/2021 06:43 PM 331,852 MASAKARI_r6_Fast_Wrapper_64bit.elf
02/01/2021 03:16 PM 52,633 MASAKARI_r6_Slow_Vanilla.BAS
02/01/2021 03:16 PM 3,288,576 MASAKARI_r6_Slow_Vanilla_32bit.exe
02/01/2021 03:16 PM 3,647,488 MASAKARI_r6_Slow_Vanilla_64bit.exe
02/01/2021 06:44 PM 331,852 MASAKARI_r6_Slow_Vanilla_64bit.elf
02/01/2021 03:16 PM 52,633 MASAKARI_r6_Slow_Wrapper.BAS
02/01/2021 03:16 PM 3,288,576 MASAKARI_r6_Slow_Wrapper_32bit.exe
02/01/2021 03:16 PM 3,647,488 MASAKARI_r6_Slow_Wrapper_64bit.exe
02/01/2021 06:44 PM 331,860 MASAKARI_r6_Slow_Wrapper_64bit.elf
02/01/2021 03:16 PM 660,672 M_r6_OED.png ! Screenshot of Oxford English Dictionary loaded by Fast_Wrapper
02/01/2021 03:16 PM 416,939 warwalk_180hue-39sat_x3_NearestNeighbor.gif ! The logo
02/01/2021 03:16 PM 240 z.bat
02/01/2021 03:16 PM 340 _Make_EXEs.bat
02/01/2021 03:16 PM 41,814 Yoshi.exe ! tool for generating 'dir/b' and 'ls' like output
02/01/2021 03:16 PM 972,575 Yoshi7-.zip
02/01/2021 03:16 PM 39 _MAKE_HEX_dump.bat ! give it a filename to generate HEX output in text format
02/01/2021 03:16 PM 111,328 DUMP_HEX_header.c
02/01/2021 03:16 PM 77,312 DUMP_HEX_header.exe
02/01/2021 03:16 PM 69,120 LineWordreporter.exe
02/01/2021 03:16 PM 52,761 LineWordreporter_r1stats.zip
```
New releases and announcements at https://twitter.com/Sanmayce
This is how the it looks like in the prompt:
```
E:\qb64>MASAKARI_r6_Fast_Vanilla.exe -h
Masakari, revision 6_Fast_Vanilla, written in QB64 by Kaze, source code downloadable at https://www.qb64.org/forum
Usage: Masakari filename|/help
Currently are implemented only:
Mouse:
Button 1 - sets the cursor and the inverse line to the chosen position
Button 3 - PgDn
Wheel Up - Up
Wheel Dn - Dn
Button 2 + Wheel Up - going to the top left position
Button 2 + Wheel Dn - going to the bottom left position
Keyboard:
Up
Down
Left - still no sideways scroll
Right - still no sideways scroll
LCtrl+Home - going to the top left position
LCtrl+End - going to the bottom left position
Alt+X or Alt+Q - quit to the system, without demanding keypress.
Space - loads the highlighted line (if it is an actual filename)
Benchmarking:
LShift+RShift - Reporting (in the status line in red color) the time for load
LCtrl+RCtrl - Reporting (in the status line in red color) the time for PgDn-ing the entire file
E:\qb64>
```
Enfun!
Kaze,
2021-Jan-31
In the incoming revision 7, there will be an additional parser (like the 'Slow' sub-variant, now) NOT loading the entire file, in RAM will be loaded only the 16bytes (Offset+Size) descriptors ... in a buffered way. I fully expect those nasty 4672 seconds to be shrunk to few minutes.
Post by: Sanmayce on February 03, 2021, 04:58:44 am
Will upload Masakari revision 7 tomorrow, busy with other things, want to add the old 'LCtrl+S' combo spell-checking the current window...
Yet, want to share the first known to me 64bit parser of CRLF or LF (or both, since Linux and Windows text files could be part of a .tar file), 87 lines in length:
Code: QB64: [Select]
- ReadBytes = 0 ' Have to load the 2+GB "malloc" in chunks...
- 'IF INSTR(1, chunk128KB$, CHR$(0)) THEN PRINT "Termination! 'Null' character encountered.": _DISPLAY: END
- I = 1
- NumberOfLFs = NumberOfLFs + 1
- I = I + 1
- ReadBytes = ReadBytes + (128 * 1024 * 1024)
- 'IF INSTR(1, RemainingChunk$, CHR$(0)) THEN PRINT "Termination! 'Null' character encountered.": _DISPLAY: END
- I = 1
- NumberOfLFs = NumberOfLFs + 1
- I = I + 1
- ' Parser for r.7 [[[[[[
- ReadBytes = 0 ' Have to load the 2+GB "malloc" in chunks...
- ChunkLen = 128 * 1024
- j = 1 ' is the current offset where new GET reads
- QWORDlast = 1
- PrevByte = ""
- I = 1
- QWORD = (j - 1) + FoundAt
- LineLen13 = QWORD - QWORDlast
- QWORDlast = QWORD + 1
- filecount = filecount + 1
- I = FoundAt + 1
- j = j + (ChunkLen)
- ReadBytes = ReadBytes + (ChunkLen)
- 'Beware, yes be aware that above line should have been applied for above/first fragment because the filesize could be multiple of the chunk length i.e. no remaining chunk, however it was feinted by '<'
- I = 1
- QWORD = (j - 1) + FoundAt
- LineLen13 = QWORD - QWORDlast
- QWORDlast = QWORD + 1
- filecount = filecount + 1
- I = FoundAt + 1
- ' Parser for r.7 ]]]]]]
The speed is MUTSI! Oxford English Dictionary was loaded in 5 seconds on my 1.3GHz i5-7200U, the memory usage is roughly 400MB:
Also tried the vanilla (non-wrapped) English Wikipedia, the load time is 1188 seconds, the memory usage is around 19GB:
Also, the new revision, already, handles unwrappable files (such as Wikipedia) by dumping all unwrappable lines into filename+".unwrappable" while wrapping all the rest, and reports in the status line how many lines are unwrappable, excited to see in next days for the first time Wikipedia loaded in readable format, expecting some
The actual wrapped Wikipedia loaded in 375 seconds:
Post by: Sanmayce on February 05, 2021, 09:55:33 am
Code: [Select]
README.TXT
A quick .DIZ for Masakari, revision 7
Masakari is a free and open source tool with simplistic GUI aiming at sidekicking browsing text files in Linux/Windows command prompts.
This .ZIP package contains:
```
README.TXT ! This file
catalog_2021-02-05_16-20-22.png ! Screenshot of how 'masakari.tgz' was created
masakari.tgz: ! Contains the trio, with attributes (executable flag) set:
m ! Linux .sh file invokes yoshi and masakari
yoshi ! Linux .ELF 64bit
masakari ! Linux .ELF 64bit same as 'MASAKARI_r7_Vanilla_64bit'
ColumnChart.ico ! Needed during compilation
MEM.H ! Needed during compilation
m.bat ! Accepts wildcards, invokes Yoshi.exe and MASAKARI_r7_Vanilla_64bit.exe
MASAKARI_r7_Vanilla.BAS
MASAKARI_r7_Vanilla_32bit.exe
MASAKARI_r7_Vanilla_64bit.elf
MASAKARI_r7_Vanilla_64bit.exe
MASAKARI_r7_Wrapper.BAS
MASAKARI_r7_Wrapper_32bit.exe
MASAKARI_r7_Wrapper_64bit.elf
MASAKARI_r7_Wrapper_64bit.exe
MASAKARI_r7_Vanilla_OED.png ! Screenshot of Oxford English Dictionary loaded by Vanilla
MASAKARI_r7_Vanilla_Wikipedia.png ! Screenshot of English Wikipedia loaded by Vanilla
warwalk_180hue-39sat_x3_NearestNeighbor.gif ! The logo
z.bat
_Make_EXEs.bat
Yoshi.exe ! tool for generating 'dir/b' and 'ls' like output
Yoshi7-.zip
_MAKE_HEX_dump.bat ! give it a filename to generate HEX output in text format
DUMP_HEX_header.c
DUMP_HEX_header.exe
LineWordreporter.exe
LineWordreporter_r1stats.zip
```
New releases and announcements at https://twitter.com/Sanmayce
This is how it looks like in the prompt:
```
E:\qb64>MASAKARI_r7_Vanilla_32bit.exe -h
Masakari, revision 7_Vanilla, written in QB64 by Kaze, source code downloadable at https://www.qb64.org/forum
Usage: Masakari filename|/help
Currently are implemented only:
Mouse:
Button 1 - sets the cursor and the inverse line to the chosen position
Button 3 - PgDn
Wheel Up - Up
Wheel Dn - Dn
Button 2 + Wheel Up - going to the top left position
Button 2 + Wheel Dn - going to the bottom left position
Button 2 + dragging (from left to right, or from right to left) for at least 90 columns/cells (within 2 seconds) - same as Alt+X, Alt+Q
Button 2 + dragging (from top to bottom) for at least 5 lines/cells (within 2 seconds) - same as PgUp
Button 2 + dragging (from bottom to top) for at least 5 lines/cells (within 2 seconds) - same as PgDn
Keyboard:
Up
Down
PgUp
PgDn
Left - still no sideways scroll
Right - still no sideways scroll
LCtrl+Home - going to the top left position
LCtrl+End - going to the bottom left position
Alt+X or Alt+Q - quit to the system, without demanding keypress.
Space - loads the highlighted line (if it is an actual filename)
Benchmarking:
LShift+RShift - Reporting (in the status line in red color) the time for load
LCtrl+RCtrl - Reporting (in the status line in red color) the time for PgDn-ing the entire file
Note1: The 'Vanilla' sub-variant loads textual files without wrapping the lines.
Note2: The 'Wrapper' sub-variant makes the text file viewable without side/lateral scroll,
If unwrappable lines exist then those lines are dumped to filename+".unwrappable",
otherwise, the wrapped lines are dumped to filename+".wrapped", and auto-loaded.
If wrapped file exists during start then it is used, not re-created.
E:\qb64>
```
Enfun!
Kaze,
2021-Feb-05
Post by: Sanmayce on February 19, 2021, 08:09:58 am
No time at the moment, but here is how it looks like on Windows XP 32bit:
The drag-and-drop, waiting for a file:
[ This attachment cannot be displayed inline in 'Print Page' view ]
The Search Panel:
[ This attachment cannot be displayed inline in 'Print Page' view ]
My brother's fast testmachine (AMD 4800H, 16 threads, 64GB RAM, m2 SSD with 2400MB/s) will be used, but it cannot keep up with the awesomeness of Kazahana, it devours the read data with its 16-threads thus reaching 3000MB/s search rates, to be seen...
Post by: Sanmayce on February 22, 2021, 02:46:11 pm
Step #1:
Step #2:
The search rate was 1500+MB/s or 48 seconds for fulltext searching in Wikipedia, not that good because the nvme SSD could do 2400MB/s, for some reason Windows didn't allow it, in next weeks will retest the same benchmark on Linux.
I did the video, however not satisfied with the result, too many blurred scenes, yet, it shows:
- decompressing the .zip package;
- sending the shortcut of Masakari.exe to the Desktop;
- running the shortcut;
- drag-and-drop the Wikipedia;
- searching with Kazahana;
- auto-scrolling with LAlt+RAlt combo, and pressing PgUp key in order to advance ... backwards while the scroll is on (line-by-line).
Was curious how one of the best editors (UltraEdit v.28) deals with the same benchmark, here is what I saw:
UltraEdit v.28:
Load Time: 0s (it has streaming approach)
Fulltext Search Time: 17min 10s
Masakari r7+:
Load Time: 309s (it has indexing approach)
Fulltext Search Time: 49s
Wikipedia can be downloaded at:
https://dumps.wikimedia.org/enwiki/ (https://dumps.wikimedia.org/enwiki/)
Post by: NOVARSEG on February 23, 2021, 12:41:37 am
chunk128KB$ = SPACE$(128 * 1024 * 1024)
should be 128MB$ ?
****
Quote
I = 1
DO WHILE I <= (128 * 1024 * 1024)
I = INSTR(I, chunk128KB$, CHR$(10))
IF I = 0 THEN EXIT DO
NumberOfLFs = NumberOfLFs + 1
I = I + 1
LOOP
I think that INSTR returns the position of CHR$(10) in the string
from wiki
Quote
The function returns the position% in the baseString$ where the searchString$ was found.so
I = 0
DO
I = INSTR(1 + I, chunk128KB$, CHR$(10))
IF I = 0 THEN EXIT DO
NumberOfLFs = NumberOfLFs + 1
LOOP
When the code exits the LOOP there is likely some bytes that are left over at the end of the string that might be data for the next line.
not tested
Post by: SMcNeill on February 23, 2021, 02:32:28 am
@Sanmayce
chunk128KB$ = SPACE$(128 * 1024 * 1024)
should be 128MB$ ?
****
I think that INSTR returns the position of CHR$(10) in the string
from wikiso
I = 0
DO
I = INSTR(1 + I, chunk128KB$, CHR$(10))
IF I = 0 THEN EXIT DO
NumberOfLFs = NumberOfLFs + 1
LOOP
When the code exits the LOOP there is likely some bytes that are left over at the end of the string that might be data for the next line.
not tested
CHR$(10) is the EOL character (End of Line). Anything past it goes on the next line.
Post by: NOVARSEG on February 23, 2021, 03:38:14 am
Code: QB64: [Select]
Post by: Sanmayce on February 23, 2021, 09:32:46 am
Thanks, you are right, but the possible line left behind unhandled is "COVERED" by adding one extra line in the "malloc" lines, which are right behind the loopS searching for number of LFs:
Code: QB64: [Select]
The idea is simply to know the number of LF in the incoming file, if the file doesn't end in LF, the above +1 ensures we don't miss the last line.
The best way to count is not INSTR() nor memchr(), it is a manual/dedicated vectorized function counting all the LFs in the vector (not as vectorized memchr() which stops at first hit). Of course for best speeds, it has to be multi-threaded, I found that 4 threads are enough to saturate the memory read of MAIN RAM. Imagine the scenario you have 24GB + 77GB or just machine with 128GB RAM, then the whole Wikipedia will be cached, in such scenario search speed goes up 2x or 3x, as a minimum, I tested a text corpus 400+ million lines (subtitles from movies) 13GB strong, and Kazahana reported 4+GB/s:
If you are interested in what I say, see NyoTengu's source code, in there the fastest search, known to me, is done achieving 30+GB/s search rates.
Always glad to talk to coders appreciating the details... ask me what you see as NG (no good), will try in next months to address it.
Add-on, Mar-05:
Always, it is a good idea to have one's sources in hardcopy i.e. on paper, in .PDF, here they are:
MASAKARI_r8+_Vanilla.BAS:
TetraNyoTengu.c:
Upload Denied, size limitation...
Post by: Sanmayce on March 05, 2021, 06:09:09 pm
Now I have my master laptop fixed and running both latest Windows 10 and latest Linux Fedora 33, with QB64 v1.5:
Soon will write a console tool converting UTF-8 to Gesch codepage, thus allowing Russian texts to be browsed as well with the superb Toshiba TxL2 font (spotted after watching @FellippeHeitor videos on YouTube, thanks a lot), for the moment I wrote codepage 1251 to Gesch codepage - the one used by Masakari to show Cyrillic:
[ This attachment cannot be displayed inline in 'Print Page' view ]
In the attached package, there is a quick description file:
Code: [Select]
README.DIZ.TXT
A quick .DIZ for Masakari, revision 8+
Masakari is a free and open-source tool with simplistic GUI, aiming at sidekicking browsing textual/.tar files in Linux/Windows command prompts.
New releases and announcements at twitter.com/Sanmayce
New features:
- Automatically switches to the excellent Toshiba unicode 16x32px font, in 4K, otherwise using its 8x16px variant in 1366x768, 1680x1050 and FHD modes;
- Allows browsing Bulgarian/Russian texts;
- Scrollwise, refined bigtime, ability to scroll with mouse-keyboard combos or just with mouse alone;
- Source code in form of a .PDF booklet - specially done for QB64 community and coders wanting to help/learn/improve.
This is how it looks like in the prompt:
```
E:\Masakari_source-ELF-EXE_rev8+_Kazahana_NyoTengu_Leprechaun>MASAKARI_Vanilla.exe -h
___ ___ _ _ _____
| \/ | | | (_) | _ |
| . . | __ _ ___ __ _ | | __ __ _ _ __ _ _ __ ___ __ __ \ V / _
| |\/| | / _` |/ __| / _` || |/ // _` || '__|| | | '__|/ _ \\ \ / / / _ \ _| |_
| | | || (_| |\__ \| (_| || <| (_| || | | | | | | __/ \ V /_ | |_| | |_ _|
\_| |_/ \__,_||___/ \__,_||_|\_\\__,_||_| |_| |_| \___| \_/(_)\_____/ |_|
Masakari, revision 8+_Vanilla, written in QB64 by Kaze, source code downloadable at https://www.qb64.org/forum
Usage: Masakari [filename]|[/help]
Note: The 'filename' could be a filelist, i.e. a list of filenames (see Space and Double-Left-Click).
Mouse:
Button 1 - sets the cursor and the inverse line to the chosen position
Button 3 - Burst PgDn
Wheel Up/Dn - Up/Dn
Button 2 + Wheel Up - going to the top left position
Button 2 + Wheel Dn - going to the bottom left position
Button 2 + dragging (sideways) for at least 100 columns/cells (within 2 seconds) - same as Alt+X, Alt+Q
Button 2 + dragging (from top to bottom) for at least 5 lines/cells (within 2 seconds) - same as PgUp
Button 2 + dragging (from bottom to top) for at least 5 lines/cells (within 2 seconds) - same as PgDn
Button 1 + Button 1 (within 0.33 seconds) - Double-Left-Click - PgUp or loads the inverse line (if it is a valid file)
Button 2 + Button 2 (within 0.33 seconds) - Double-Right-Click - PgDn
LShift + Button 1/2 - Burst Up/Dn
LCtrl + Button 1/2 - Burst PgUp/PgDn
LAlt + Button 1 = LCtrl+Home - going to the top left position
LAlt + Button 2 = LCtrl+End - going to the bottom left position
Keyboard:
F1 - this help panel
Up/Dn/PgUp/PgDn
Left/Right - still no sideways scroll
LCtrl+Home - going to the top left position
LCtrl+End - going to the bottom left position
Alt+X or Alt+Q - quit to the system, without demanding keypress
Space - loads the highlighted line (if it is an actual filename)
RCtrl|RShift - (can hold it) highlighting all unfamiliar words to masakari.wrd file/wordlist
LCtrl+F3 - Search Panel, Kazahana (with 16 threads) searches the file, currently displayed
Benchmarking:
LAlt+RAlt - Reporting (in the status line in red color) the time for load
LCtrl+RCtrl - Reporting (in the status line in red color) the time for PgDn-ing (page-by-page) the entire file
LShift+RShift - Reporting (in the status line in red color) the time for Down-ing (line-by-line) the entire file
Esc - Stops the auto-scrolling started with LCtrl+RCtrl or LShift+RShift
Note1: During the LShift+RShift auto-scroll, you can still press/hold RCtrl, during the LCtrl+RCtrl, you cannot.
Note2: During the auto-scroll, you can still press PgUp, PgDn, LCtrl+Home, LCtrl+End, Esc.
Note3: The 'Vanilla' sub-variant loads textual files without wrapping the lines.
Note4: The 'Wrapper' sub-variant makes the text file viewable without side/lateral scroll.
If unwrappable lines exist then those lines are dumped to filename+".unwrappable",
otherwise, the wrapped lines are dumped to filename+".wrapped", and auto-loaded.
If wrapped file exists during start then it is used, not re-created.
E:\Masakari_source-ELF-EXE_rev8+_Kazahana_NyoTengu_Leprechaun>
```
Some explanations:
1,472,368 1251toGesch ! 64bit ELF !
5,784 1251toGesch.bas ! Convertor from 1251 codepage to Gesch codepage ! Allowing reading Bulgarian/Russian !
1,724,416 1251toGesch.exe ! 32bit EXE !
<DIR> Kazahana_sources_binaries ! The binaries and source of fastest scalar fulltext searcheress !
484,124 kazahana ! 64bit ELF !
111,118 Kazahana.exe ! 32bit EXE !
<DIR> NyoTengu ! The binaries and source of fastest vector fulltext searcheress !
96,256 TetraNyotengu.exe ! 32bit EXE !
1,042,360 libiomp5md.dll ! TetraNyotengu needs this Intel OpenMP .DLL!
<DIR> x-gram_corpora ! The binaries and source of fastest x-gram ripper !
10,229,600 masakari.ind ! Needed by RCtrl or RShift in Masakari - highlights the unfamiliar to masakari.wrd words !
3,903,143 masakari.wrd ! Kaze's unigram English corpus i.e. spell-check wordlist !
1,453,926 ColumnChart.ico ! The ICON of Masakari !
2,991 MEM.H ! Needed for Windows compilation of Masakari !
2,537,104 MASAKARI_Vanilla ! 64bit ELF !
135,085 MASAKARI_Vanilla.BAS
4,319,232 MASAKARI_Vanilla.exe ! 32bit EXE !
2,537,104 MASAKARI_Wrapper ! 64bit ELF !
135,085 MASAKARI_Wrapper.BAS
4,319,232 MASAKARI_Wrapper.exe ! 32bit EXE !
98 ml.sh ! this script gives 'ls -l' in a scrollable list !
98 m1.sh ! Allows to choose-n-load a file from the the shown filelist !
51 mGesch.sh ! Converts codepage 1251 to Gesch and loads wrapped file in Masakari !
418 m.bat
1,633 MokujIN JADE 217 prompt.lnk ! Kaze's Windows command prompt !
303,296 cour.ttf
115,068 lucon.ttf
346,524 MxPlus_Cordata_PPC-400.ttf
146,744 MxPlus_ToshibaTxL2_8x16.ttf ! The Kaze's font-of-choice, many thanks go to VileR, https://int10h.org/oldschool-pc-fonts/ !
570,901 Gulyakovskiyi_E._Dolgiyi_Voshod_Na_Yenne.html ! The Line-By-Line Scroll Rate Benchmark, sh mGesch.sh Gulyakovskiyi_E._Dolgiyi_Voshod_Na_Yenne.html !
Enfun!
Kaze,
2021-Mar-05
Post by: Sanmayce on May 24, 2021, 10:25:33 pm
here comes MASAKARI revision 8+++, enjoy!
Three new features were added:
- Thanks to VileR (https://int10h.org), the superb Japanese 6x12 font was added - it allows beautiful 300x80 textual matrix in FHD;
- Creating the unseen x-gram (better defined than n-gram) tree - called PAGODA - by mere clicking Left Mouse Button + Right Mouse button, over a word;
- Added three searching (via keyboard) modes - F3, Lshift+F3, LAlt+F3.
Look up the attached package for README.DIZ.TXT for a bit more detailed description.
Just to get a visual sense of what the power of PAGODA structure is:
Step #1, just clicking Left Mouse Button + Right Mouse Button, over a word, it highlights it:
Step #2, browsing the PAGODA tree, it saves a ton of time for searching/browsing by giving the "environment of a word" - built by all unigrams, bigrams, trigrams, tetragrams and pentagrams adjusted in such a manner that the "root/pillar" word stays centered, whereas the branches come in top-bottom and right-left order:
More about PAGODA structure can be found in this 8 pages booklet:
Also, the font sharings of VileR (an Israeli guy) have huge impact on me, see below the ultrab 300 columns in 1920x1080 - the font matrix of 6x12 is the smallest one allowing undistorted shape to be retained, as far as I know. Sometimes, a file with long lines (200+) has to be browsed and searched, then the new MASAKARI_Vanilla_NEC[.exe] comes in handy, it is to be used, instead of Toshiba 'Vanilla':
Sadly, didn't have enough time to create the Linux counterparts of files (the ELFs and .sh batches) needed to create PAGODA, to be done...
Also, the PDF booklet with the source code didn't fit in the 20MB limit, so here are the 68 pages, it turns out that the Toshiba font is quite good when printing in hardcopy as well, it gives the Matrix Printer feel of it:
http://www.sanmayce.com/Masakari.zip
Hope, fellow members, interested in textual processing/visualization, to benefit from this package, as I benefited down the years from ... other sharers.
Post by: Sanmayce on July 21, 2021, 02:40:11 am
First, I wrote this standalone testbed, and later have included it into the latest Masakari - to be shared in next hours...
Code: QB64: [Select]
- 'mouse mumbo-jumbo [
- ShutDownDuration = 4 ' 4 seconds
- DclickTime = 0.33 ' 1/3 of a second, usually it is 0.27s, so 0.33 suits even the slow clickers
- PrevClick# = 0
- PRINT "You may exit with Esc or by holding left mouse button for 4 seconds..."
- mwheel = _MOUSEWHEEL
- 'PrevClick# = TIMER(0.001)
- Button1LOG_firstDetection#(MumboJumbo) = Button1LOG_firstDetection#(MumboJumbo + 1)
- Button1LOG_ForHowLongHolded#(MumboJumbo) = Button1LOG_ForHowLongHolded#(MumboJumbo + 1)
- Button1LOG_firstDetection#(4) = PrevClick#
- Button1LOG_ForHowLongHolded#(4) = ForHowLongWasPressed#
- PrevClick# = 0
- _LIMIT 500
- 'Note1: From above experiments I see Double-Click fits usually in DclickTime s window.
- 'Note2: Double-Click is detected when the difference between two initial clicks is below DclickTime s i.e. Button1LOG_firstDetection#(4)-Button1LOG_firstDetection#(3) < DclickTime
- ' ForHowLongWasPressed# = B1 - A1 i.e. duration of holding down:
- ' where
- ' A1=first detection of the click B1=released A2=first detection of the click B2=released
- ' [ ][ ][ ][ ]
- 'mouse mumbo-jumbo ]
Soon will write a console tool converting UTF-8 to Gesch codepage, thus allowing Russian texts to be browsed as well with the superb Toshiba TxL2 font...
Did it, now browsing 800,000+ lines of Russian (either in UTF-8 or CP-1251) mini-articles is possible, when a line is of interest, just release Enter over it, and when done reading, again release Enter in order to return in non-wrapped view:
As always, I attached the Linux binaries, speaking of releasing keys, for some reason there is a discrepancy when compared with Windows binaries - on Linux it doesn't wait for releasing the key but acts as soon as the key is hit?! Any ideas why?
Post by: Sanmayce on July 25, 2021, 05:50:05 am
It's time to break away, no more limitations on browsing Italian/French/German/Spanish/Bulgarian/Russian ... without using any UNICODE!
In next days will write the search panel in which all those symbols (from my Gesch codepage), for now I wrote only the converter from UTF-8 to Gesch, attached at the end.
Here it is the "legacy" codepage, which in my opinion was made by a total schmuck:
[ This attachment cannot be displayed inline in 'Print Page' view ]
Here is the newwave in codepage encodings :P
[ This attachment cannot be displayed inline in 'Print Page' view ]
As always, I checked all the converted symbols against the https://utf8-chartable.de, and there were no mismatches:
[ This attachment cannot be displayed inline in 'Print Page' view ]
[ This attachment cannot be displayed inline in 'Print Page' view ]
One quick example is how Don Quixote is readable in Gesch encoding:
[ This attachment cannot be displayed inline in 'Print Page' view ]
Simply, when I need to browse some UTF-8 file, I won't be bothered whether there are Russian or/and French or/and German texts within, just will run UTF8toGesch.exe converter and Drag-n-Drop the .Gesch file.
In next releases will add the ability to search for those symbols.
Halving the file size comes as a "side effect", as a bonus. For example I have one Russian corpus 15,000+ ebooks in UTF-8 strong and 11GB in size. Guess what, instead of fulltext searching 11GB in UTF-8 the traverse distance becomes ~6GB.
Post by: Sanmayce on July 28, 2021, 12:30:02 pm
Holy hell, why no one introduced the analogues/counterparts of "double-clicks" - the first that comes to mind: the "double-hits".
Both, being just taps.
Currently, in Masakari there are:
- shortcuts, purely keyboardish;
- shortcuts, mix of keys and mouse buttons;
- shortcuts, purely mouseish, just mouse stuff - buttons and wheel.
From revision 8.1+ onward, following double-hits were implemented:
Double LShift; Double LCtrl; Double LAlt; Double RAlt.
The thing is that "legacy" shortcuts as Ctrl+F3 (even the F3) force the user to "locate" their position and to overreach - meaning hovering pass the SPACEBAR area - which is most easy to hit since the wrists/palms have support, and are simply closer.
To me, the traditional key sequences are to be replaced/enriched with more convenient, more close to the first row of the keyboard, ones.
Currently, my main keyboard is a nasty one (laptopish, with low profile, non-tactile, half-sized F-keys, and tightly packed), when e.g. a LCtrl+F3 search is needed, spotting this halved F3 is a drag, therefore the double-hit of e.g. LCtrl saves the situation - just double-tap it - the timings between the two pressings are handled by the same logic as the mouse double/triple/quadruple-clicks.
The following stand-alone etude is the same as the previously shared mouse_mumbo-jumbo.bas, except:
Code: QB64: [Select]
Code: QB64: [Select]
The attached package contains the most refined revision so far, with all accompanying tools, 32bit and 64bit executables for Windows, and 64bit ELF for Linux.
There is one file, called Word-list_584879_words_Russian_Spell-Check.txt.Gesch.oneline.198, it serves as a scroll line-by-line testfile benchmark, on my laptop running both Windows 10 and Fedora 33, it gave 45 Lines-Per-Second scroll speed:
[ This attachment cannot be displayed inline in 'Print Page' view ]
And the PDF booklet with the QB64 sourcecode at:
www.sanmayce.com/Masakari.zip (http://www.sanmayce.com/Masakari.zip)
[ This attachment cannot be displayed inline in 'Print Page' view ]
Post by: Sanmayce on October 12, 2021, 07:45:33 am
Didn't want to initiate a thread about hashing, especially for having the so needed fastest 128bit hash function.
As the need for speed is constantly on my mind, recently I wrote Gumbotron:
https://github.com/google/highwayhash/issues/102# (https://github.com/google/highwayhash/issues/102#)
The idea is to solve problematic search/index scenarios with reliable/optimized tools/etudes and feeding the QB64 application with resultant data.
Too often 1,000,000,000 records e.g. 64,128 or 256 bytes long are to be searched, how to deal with it?!
The discussions in other sub-forums focus on languages and their advance features, but the problem stays as long as one sees within the box and forgets to look outwith, that is, beyond the languages limitations.
Gumbotron is such a case, it makes the hashing in inside-out not in outside-in manner, meaning all the permutations of the key for hashing are dealt with outside the hasher itself, the "traditional" hashers make all kind of rotations/multiplications/shifts within the body serving as a LINEAR MIXER, but the really smart coders already did such a LINEAR MIXER hardwarely embedding all the stages of AES into a single instruction with latency 3 clocks and throughput 1 clock!
So, the moral is to exploit all the available resources (either languages, CPU extensions, algorithms), not to burden oneself with giving tasks as forcing a dog to pull a plug, yes the ox is better, but dog has other functionality - it is a matter of proper assignment.
Hm, don't feel I am getting my point across, back in the day Assembler/Assembly was popularized by the "bad guy" Phil Katz, the documentary says a lot. The favorite part of mine is where the screen says "... the damage was done" referring to speeding up the ARC 10x, rewriting C parts into Assembly. Very funny and sad at the same time, the bad guy gives PKARC/PKZIP and teaches in a practical way how things are to be done, and the community slams him as being the devil. If the devil offers superspeeds, let us reject them since they are ... born in sin, hee-hee.
Also, reading, in the forum, the discussion about highlighting/popularizing the usefulness of QB64, didn't want to write obvious things, yet to me the most obvious way is by ... example.
Yes, sharing etudes/tools written well with real practical value would showcase the power of one approach/language/method in a macho (Clint Eastwood's last film taught me this word) way.
Imagine, a well-written tool, it would serve as an ad for the language used, quite naturally. Having such a good forum prompts for sharing code that catches the attention of users and makes them enthusiasts, daring to try themselves, but they have to have a starting point/code/tool to play with.
Will share this thread/tool on the biggest English language forum:
https://forum.thefreedictionary.com/postst31183p4_MASAKARI--The-people-s-choice--General-Purpose-Grade--English-wordlist.aspx (https://forum.thefreedictionary.com/postst31183p4_MASAKARI--The-people-s-choice--General-Purpose-Grade--English-wordlist.aspx)
And just wanted to announce the incoming new Masakari revision allowing dictionaries lookups in a better way than most software out there, including the original Oxford, Webster, Wiktionary...
My vision is to present all the .DSL files into an unified/standardized manner, simplistic and ergonomic, having in mind cross-searches as well.
The draft looks like this (yes, targeting the old and super durable laptops, as Thinkpad, having low resolutions as 1366x768):
Writing such a practical tool (allowing quick lookups into 16+1+1+1 major dictionaries) could serve as a starting point for contemplating new ideas of refining and boosting the search capabilities even more...
I intend gradually to add all these:
Code: QB64: [Select]
- 02/28/2021 08:34 PM 42,257,313 Dictionary_Specification_Language_(ABBYY_Software_House)_American_Heritage_Dictionary_4_(En-En)_UTF-8.dsl
- 02/28/2021 08:36 PM 299,225,953 Dictionary_Specification_Language_(ABBYY_Software_House)_Britannica_Encyclopedia_2010_1.563_miled_(En-En)_ANSI.dsl
- 02/28/2021 08:58 PM 75,122,799 Dictionary_Specification_Language_(ABBYY_Software_House)_Cambridge_Advanced_Learner's_Dictionary_4th_Ed_(En-En).dsl
- 02/28/2021 09:00 PM 23,413,283 Dictionary_Specification_Language_(ABBYY_Software_House)_Collins_COBUILD_Advanced_Learner's_English_Dictionary_(5th_ed)_(En-En).dsl
- 02/28/2021 09:05 PM 37,870,978 Dictionary_Specification_Language_(ABBYY_Software_House)_Longman_Activator_2nd_Ed_(En-En).dsl
- 02/28/2021 09:10 PM 52,743,002 Dictionary_Specification_Language_(ABBYY_Software_House)_Longman_Dictionary_of_Contemporary_English_5th_Ed_(En-En).dsl
- 02/28/2021 09:19 PM 83,412,925 Dictionary_Specification_Language_(ABBYY_Software_House)_Macmillan_English_Dictionary_(En-En)_UTF-8.dsl
- 02/28/2021 09:19 PM 29,772,872 Dictionary_Specification_Language_(ABBYY_Software_House)_Macmillan_English_Thesaurus_(En-En)_UTF-8.dsl
- 02/28/2021 09:15 PM 101,528,058 Dictionary_Specification_Language_(ABBYY_Software_House)_Merriam-Webster's_Collegiate_Dictionary_11th_Edition_(En-En).dsl
- 02/28/2021 09:29 PM 57,854,694 Dictionary_Specification_Language_(ABBYY_Software_House)_Oxford_Advanced_Learner's_Dictionary_8th_Edition_(En-En).dsl
- 02/28/2021 09:32 PM 44,362,425 Dictionary_Specification_Language_(ABBYY_Software_House)_Oxford_American_Dictionary_2nd_Edition.dsl
- 02/28/2021 09:33 PM 11,747,238 Dictionary_Specification_Language_(ABBYY_Software_House)_Oxford_American_Thesaurus_(En-En).dsl
- 02/28/2021 09:38 PM 559,408,265 Dictionary_Specification_Language_(ABBYY_Software_House)_Oxford_English_Dictionary_2nd_Edition_Version_4_(En-En)_ANSI.dsl
- 02/28/2021 09:33 PM 53,483,030 Dictionary_Specification_Language_(ABBYY_Software_House)_Random_House_Webster's_Unabridged_Dictionary_(En-En)_UTF-8.dsl
- 02/28/2021 09:53 PM 12,202,934 Dictionary_Specification_Language_(ABBYY_Software_House)_The_Collins_Cobuild_School_Dictionary_of_American_English_(En-En).dsl
- 02/28/2021 08:26 PM 137,754,899 Dictionary_Specification_Language_(ABBYY_Software_House)_Webster's_Unabridged_3_(En-En)_UTF-8.dsl
- 03/28/2019 02:31 PM 5,340,386 dict2txt_Online_Etymology_Dictionary1.1.txt
- 10/08/2021 08:42 AM 7,015,793,795 enwiktionary-20210920-pages-articles.xml
- 07/10/2021 09:57 PM 1,917,822,288 Machine-Learning_Urban_Dictionary_Definitions_Corpus_(1999_-_May-2016).words.json
For a long time, the need for English language speedy lookups into constellation of dictionaries (on a netbook) drives me toward writing a tool sidekicking macholy, once and for all.
Post by: Sanmayce on October 23, 2021, 11:22:27 pm
https://www.qb64.org/forum/index.php?topic=1598.0
This function is a must-have, for example it can be used for controlling _LIMIT in a "stepping" fashion, as the CPUs are downclocked.
Consider a main loop with _LIMIT 500 and the user is using the application touch-and-go-ly, it is a good idea to reduce the 500 by 10% each minute (down to 90%) of inactiveness i.e. not doing keyboard input.
Here is a bit modified variant, after 13 minutes it enters maximum ... minimum:
Already incorporated this etude in Masakari r.8.1++, also finished the Schizandrafield rev.D lookup functionality:
Simple functionalities such as automatic highlighting of identical entries (above and below) to the current/inverse line are nice since the eyes stop constantly seeking whether neighbors are identical.
Also, finally, did what every GUI tool has to offer - the Color Scheme defined in a file (allowing the user to play with it), in this case the file is called Masakari.RGB - it is 48 bytes long, all the 16 colors given from 0 to 15 in three bytes sequence, RGB:
Sometimes, when working outdoors (on a shiny day), more brightness is needed, laptops lack such a regulator so just pressing 'I' will increase all channels by 1 point:
The feeling, of seeing the actual window colorized, is different from seeing some small box, moreover the combinations of foreground and background are tricky to please the user, therefore one can play with channels and intensity and once seeing a pleasing combination then can save the RGB values on paper and remap whichever color in the Masakari.RGB is wanted to be replaced.
Post by: Sanmayce on October 30, 2021, 08:42:14 am
Back in the day, I wrote an XMS QuickSort, now dusted off and shared here.
From time to time, I visit the very informative site 'rosettacode' where multi languages etudes are given side by side, it is a shame QB64 is not presented there as it should, especially unacceptable is the situation with the QS64, standing for QuickSort 64bit, not having its QB64 presence.
Today, I have added my Quicksort variant, thus putting QB64 in this must-be "Quick" category:
https://rosettacode.org/wiki/Sorting_algorithms/Quicksort#QB64 (https://rosettacode.org/wiki/Sorting_algorithms/Quicksort#QB64)
Soon will include this very subroutine into Masakari sorting the QWORD long indexes to lines...
Also, did make a benchmark (source code and binary attached) with QB64 executable which can be used as a general speed benchmark, stressing caches and somewhat uncached (or rather not well cached) RAM accesses (despite QS being with good locality) - very interesting it would be when AMD 3D cache comes along, and mostly with M1 Apple's unified memory (with RAM latency ~35ns being the half of PCs, AFAIK).
Whoever owns M1 Max, please compile it and share with us how well Apple performs.
On my Kaby Lake laptop (i5-7200U 3.1GHz max turbo, 36GB DDR4 2133MHz) it took 11 minutes to sort 1.7 billion QWORDS (i.e. 8 bytes elements). They are in range 0..9, which makes sorting harder due to the many repetitions.
Here we go:
[ This attachment cannot be displayed inline in 'Print Page' view ]
The panel at the end checks for sanity:
[ This attachment cannot be displayed inline in 'Print Page' view ]
Actually, the digits come from the 22,338,618 bytes long file containing the dump of 2^74207281-1, which was the biggest known prime number in 2016.
Here is the dumper of my testdataset, thanks to Fabrice Bellard:
Code: C++: [Select]
- // https://bellard.org/mersenne.html
- // A small C program to print the biggest prime number
- // Here it is (448 bytes):
- /*
- int m=1811939329,N=1,t[1<<26]={2},a,*p,i,e=73421233,s,c,U=1;g(d,h){for(i=s;i<1<<
- 25;i*=2)d=d*1LL*d%m;for(p=t;p<t+N;p+=s)for(i=s,c=1;i;i--)a=p[s]*(h?c:1LL)%m,p[s]
- =(m*1U+*p-a)*(h?1LL:c)%m,*p=(a*1U+*p)%m,p++,c=c*1LL*d%m;}main(){while(e/=2){N*=2
- ;U=U*1LL*(m+1)/2%m;for(s=N;s/=2;)g(136,0);for(p=t;p<t+N;p++)*p=*p*1LL**p%m*U%m;
- for(s=1;s<N;s*=2)g(839354248,1);for(a=0,p=t;p<t+N;)a+=*p<<(e&1),*p++=a%10,a/=10;
- }while(!*--p);for(t[0]--;p>=t;)putchar(48+*p--);}
- */
- // This program computes 2^74207281-1, which was the biggest known prime number in 2016 (about 23 million digits !). For more information about how it was // found and who found it, look at the GIMPS Project .
- //
- // I compiled it successfully with gcc with Linux. In order to compile it, your C compiler must support the 64 bit long long type.
- //
- // In order to have a small and yet asymptotically efficient code, I decided to do the computation of 2N-1 directly in base 10. The power involves repeated // squarings and multiplications by two. The squarings are implemented by doing fast convolutions using a Number Theoretic Transform.
- //
- // A previous version of this program to compute 2^6972593-1 won the International Obfuscated C Code Contest of Year 2000.
- //
- // Thanks to Marco Cecchi who suggested some syntactic changes to save a few characters.
- //
- // This program is Freeware.
- //
- // Fabrice Bellard - http://bellard.org/
- // last update: Sep 26, 2016
Post by: jack on October 30, 2021, 09:58:06 am
From time to time, I visit the very informative site 'rosettacode' where multi languages etudes are given side by side, it is a shame QB64 is not presented there as it should, especially unacceptable is the situation with the QS64, standing for QuickSort 64bit, not having its QB64 presence.yes I agree, here's one for the graphics experts https://rosettacode.org/wiki/Honeycombs
Post by: bplus on October 30, 2021, 11:46:22 am
Quote
What is the quicksort routine that QB64 community uses?!
I've been using a recursive QuickSort, eg for Single Type:
Code: QB64: [Select]
- Hi = finish: Lo = start
- Middle = array((Lo + Hi) / 2) 'find middle of array
- Lo = Lo + 1: Hi = Hi - 1
Post by: bplus on October 30, 2021, 11:48:00 am
yes I agree, here's one for the graphics experts https://rosettacode.org/wiki/Honeycombs
I did that one a couple years ago in JB.
Post by: Sanmayce on November 01, 2021, 04:59:07 am
Glad that you also see the miss, will try to share rev.3 which will feature an unseen partitioning, hopefully helping in scenarios with duplicative keys almost dominating the array...
@bplus
Thanks, that's what I need, variants of QB64 community helping in refining the current fastest ones.
Since my focus is on 64bit file/string manipulations (as in the example with Wikipedia XML ~74GB having 1+billion lines), all LONG variables are obsolete i.e. 2 billion (signed, you know) are limiting dealing with big data. So, I have included your tiny (modified to be 64bit) variant as performer #3, thus we can see how my (rev.1) published on ROSETTACODE fares against yours, also as performer #2, I have put rev.2 boosting speed by just finishing with Insertionsort (nothing new), my dummy logic was to choose 7999 elements long chunks unsorted, because 7999*8<64KB, hoping the L1 data cache can house it. I have an idea to write entirely new partitioning in rev.3, we will see in practice its behavior.
Run/Performer #3 proves slower than the ROSETTACODE QB64 etude:
Code: QB64: [Select]
- Lo = start: Hi = finish
- Middle = array((Lo + Hi) \ 2)
- Lo = Lo + 1: Hi = Hi - 1
It would be nice fellow members to share their ideas and benchmarks, so we can pair the QuickBasic64 with QuickSort64.
Here comes the 32bit code generated by QB64 v.2.01, Core 2 Duo 2.9GHz, DDR2 266MHz:
Here comes the 64bit code generated by QB64 v.2.01, i5-7200U 3.1GHz, DDR4 1066MHz:
Wanna see modern machines running this benchmark in order to get the picture more clearly...
Post by: Sanmayce on November 07, 2021, 08:24:10 am
Here comes the new Quicksort rev.3 with the new 'Magnetica' partitioning, compared side-by-side with the mainstream recursive Quicksort with 'Classica' partitioning.
Generally, we have three major scenarios:
- FEW distinct keys;
- MANY distinct keys;
- ALL distinct keys.
In first two cases, the new implementation is faster or equal to the "Standard" one:
Under Windows:
Under Linux:
Sadly, in the third case, it is somewhat slower.
Anyway, benchmarking will show the practical value of 'Magnetica', personally, I started to use it already.
Post by: Sanmayce on November 16, 2021, 03:40:05 pm
[ This attachment cannot be displayed inline in 'Print Page' view ]
Enjoy!
Post by: Sanmayce on November 21, 2021, 07:47:54 pm
Couldn't resist not to make a video showing the beautiful Magnetica train - the actual visualized sort process.
The sorting pauses for 3 seconds each time when the orange train collides with the Jndx - the right "sentinel" colored in red.
I chose for the testfile an old interview in English, ~27KB long, the window/matrix for sorting is 300 columns x 80 lines <27KB.
This type of sorts are quite useful in compression/indexing since each position demands statistics for order 1..8 bytes long Building-Blocks, inhere we sort order 1 i.e. letters.
For more fun, added Quicksort 'Classica' recursive dealing with the same array, the interesting thing here is one to compare Classica's Swaps and Comparisons versus Magnetica's ones:
Magnetica: 66,008 and 115,721
Classica: 132,663 and 355,221
Total Smackdown.
Post by: Sanmayce on November 25, 2021, 11:32:16 pm
Thanks to PCBoy Studios, here comes a short animated demo of Quicksort 'Magnetica' (initial release):
Now, the new function is more robust against quadratic behavior, since it was strengthened with median of 5 pivot - branchlessly on top of that!
And for all fellow members who need maximum speed in their sorting, here comes the usage of the "header" written in C, ready to go in QB64:
Code: QB64: [Select]
- Sub Quicksort_QB64_v5 (ByVal QWORDS As _Offset, Byval Left As _Integer64, Byval Right As _Integer64)
Enfun!
Oh, and @bplus your recursive quicksort crashes (with mere 100,000 elements) without even giving an error on exit, it occurs with 'Organ Pipe' data. Your test program modified is given below in order to reproduce the crash. My advice, start using the attached QB64 "intrinsic", updated on 2021-Nov-28:
Code: QB64: [Select]
- _Title "Recursive Vrs NonRecursive test with Quicksort" 'b+ 2021-11-01, Sanmayce added some stuff 2021-Nov-26
- ArrayHigh = 2 * 50000
- 'ArrayHigh = 2 * 5000000
- Declare CustomType Library "qsm" ' Notice that 'CustomType' makes things work, using 'STATIC' gives errors during compilation
- Sub Quicksort_QB64_v5 (ByVal QWORDS As _Offset, Byval Left As _Integer64, Byval Right As _Integer64)
- 'For i = 1 To ArrayHigh
- ' n(i) = i '13 'i
- 'Next
- 'For i = ArrayHigh To 2 Step -1 ' Fisher Yates Shuffle
- ' Swap n(i), n(Int(Rnd * i) + 1)
- ' m(i) = n(i) ' copy m off n
- ' z(i) = n(i) ' copy m off n
- 'Next
- ' Generate 'OrganPipe' data:
- n(i) = i
- m(i) = n(i) ' copy m off n
- z(i) = n(i) 'z(i - 1) = n(i) ' copy m off n
- Else 'ArrayHigh \ 2 + 1 .. ArrayHigh
- n(i) = ArrayHigh - i + 1
- m(i) = n(i) ' copy m off n
- z(i) = n(i) 'z(i - 1) = n(i) ' copy m off n
- 'Print z(1), z(2)
- 'Print z(ArrayHigh - 1), z(ArrayHigh)
- a$ = "1234567890"
- b$ = "ABCDEFGHIJ"
- Print a$
- Print a$
- Print "Test arrays ready. Testing Recursive QuickSort first."
- 'n(1) = ArrayHigh
- 'm(1) = n(1)
- 'z(1) = n(1)
- 'crashes with OrganPipe
- GoTo 1
- QuickSort 1, ArrayHigh
- Print "QuickSort checked, QuickSort time was"; QuickSortT#
- 1
- Quicksort_QB64
- Print "Quicksort_QB64 checked, Quicksort_QB64 time was"; QuicksortQB64T#
- 'T# = Timer(.001)
- 'Quicksort_QB64_v2
- 'QuicksortQB64T# = Timer(.001) - T#
- 'For i = 1 To ArrayHigh - 1
- ' If z(i) > z(i + 1) Then Print "NOT OK!": End
- 'Next
- 'Print "Quicksort_QB64_v2 checked, Quicksort_QB64_v2 time was"; QuicksortQB64T#
- 'Print _Offset(z(0))
- 'Print _Offset(z(2))
- 'If z(i - 1) > z(i - 1 + 1) Then Print "NOT OK!": End
- Print "Quicksort_QB64_v5 checked, Quicksort_QB64_v5 time was"; QuicksortQB64Tv5#
- ' recursive
- Lo = start: Hi = finish
- Middle = n((Lo + Hi) \ 2)
- Lo = Lo + 1: Hi = Hi - 1
- ' non recursive
- ' ref: https://rosettacode.org/wiki/Sorting_algorithms/Quicksort#QB64
- Sub Quicksort_QB64
- LeftMargin = Left
- StackPtr = 0
- StackPtr = StackPtr + 1
- Stack&&(StackPtr + LeftMargin) = Left
- StackPtr = StackPtr + 1
- Stack&&(StackPtr + LeftMargin) = Right
- Do 'Until StackPtr = 0
- Right = Stack&&(StackPtr + LeftMargin)
- StackPtr = StackPtr - 1
- Left = Stack&&(StackPtr + LeftMargin)
- StackPtr = StackPtr - 1
- Do 'Until Left >= Right
- Pivot~&& = m((Left + Right) \ 2)
- Indx = Left
- Jndx = Right
- Indx = Indx + 1
- Jndx = Jndx - 1
- Indx = Indx + 1
- Jndx = Jndx - 1
- StackPtr = StackPtr + 1
- Stack&&(StackPtr + LeftMargin) = Indx
- StackPtr = StackPtr + 1
- Stack&&(StackPtr + LeftMargin) = Right
- Right = Jndx
- Sub Quicksort_QB64_v2
- InsertionsortTHRESHOLD = 0 '19
- StackPtr = 0
- StackPtr = StackPtr + 1
- Stack&&(StackPtr + LeftMargin) = Left
- StackPtr = StackPtr + 1
- Stack&&(StackPtr + LeftMargin) = Right
- Do 'Until StackPtr = 0
- Right = Stack&&(StackPtr)
- Left = Stack&&(StackPtr - 1)
- StackPtr = StackPtr - 2
- Do 'Until Left >= Right
- ''Classica' partitioning [
- 'Pivot~&& = z((Left + Right) \ 2)
- 'Indx = Left
- 'Jndx = Right
- 'Do
- ' Do While (z(Indx) < Pivot~&&)
- ' Indx = Indx + 1
- ' Loop
- ' Do While (z(Jndx) > Pivot~&&)
- ' Jndx = Jndx - 1
- ' Loop
- ' If Indx <= Jndx Then
- ' If Indx < Jndx Then Swap z(Indx), z(Jndx)
- ' Indx = Indx + 1
- ' Jndx = Jndx - 1
- ' End If
- 'Loop While Indx <= Jndx
- ''Classica' partitioning ]
- ' 'Magnetica' partitioning [
- Jndx = Right
- PL = Left
- PR = Left
- Pivot~&& = z(PR)
- PL = PL + 1
- PR = PR + 1
- PR = PR + 1
- Else ' < i.e. Pivot~&& < z(PR + 1)
- Jndx = Jndx - 1
- Jndx = Jndx - 1
- Jndx = PL - 1
- Indx = PR + 1
- ' 'Magnetica' partitioning ]
- StackPtr = StackPtr + 2
- Stack&&(StackPtr - 1) = Indx
- Stack&&(StackPtr) = Right
- Right = Jndx
- 'Left = LBound(z)
- 'Right = UBound(z)
- 'For i = Left + 1 To Right ' i.e. 1 to size-1 omitting the first (which is 0)
- ' j = i
- ' Do While j >= 1 'Nah, start using a sentinel
- ' If z(j - 1) > z(j) Then Swap z(j - 1), z(j) Else Exit Do
- ' j = j - 1
- ' Loop
- 'Next i
Post by: bplus on November 26, 2021, 12:21:35 pm
Quote
Oh, and @bplus your recursive quicksort crashes (with mere 100,000 elements) without even giving an error on exit,
Did 100,000,000 items for me without mods:
[ This attachment cannot be displayed inline in 'Print Page' view ]
Post by: Sanmayce on November 28, 2021, 05:44:30 pm
Did 100,000,000 items for me without mods:
You didn't notice what the QB64 snippet holds, in above post. It is your test, but modified to sort 'Organ Pipe' instead of 'Random' dataset. It crashes with 100,000 using QB64 32bit. It is interesting to see how much QB64 64bit can handle before crashing. Beside this, qsm.h is from 11x to 2x faster, that's why I recommend it. In the screenshot below, Run #1 is your QB64 code, whereas Run #2 is qsm.h:
Code: QB64: [Select]
- Laptop Intel i5-7200U 3.1GHz max turbo, 36GB DDR4 2133MHz:
- +--------------------+---------------------------+---------------------------+---------------------------+
- | Performer/Keys | FEW distinct | MANY distinct | ~ALL distinct |
- +--------------------+-------------+-------------+-------------+-------------+-------------+-------------+
- | Operating System, | Windows 10, | Fedora 33, | Windows 10, | Fedora 33, | Windows 10, | Fedora 33, |
- | Compiler, -O3 | Intel v15.0 | GCC 10.2.1 | Intel v15.0 | GCC 10.2.1 | Intel v15.0 | GCC 10.2.1 |
- +--------------------+-------------+-------------+-------------+-------------+-------------+-------------+
- | qsort | 58 seconds | 360 seconds | 343 seconds | 535 seconds | 444 seconds | 519 seconds |
- | Magnetica r.2 | 35 seconds | 35 seconds | 214 seconds | 222 seconds | 271 seconds | 286 seconds |
- | Bentley-McIlroy | 39 seconds | 41 seconds | 203 seconds | 230 seconds | 275 seconds | 313 seconds |
- +--------------------+-------------+-------------+-------------+-------------+-------------+-------------+
- FEW = 2,233,861,800 keys, of them distinct = 10;
- MANY = 2,482,300,900 keys, of them distinct = 2,847,531;
- ~ALL = 2,009,333,753 keys, of them distinct = 1,912,608,132
Quick highlights:
- GLIBC's qsort is just trash;
- Intel's qsort is trashy;
- Bentley-McIlroy is good, however the disassembly (-O3) shows many weird vector instructions used, not good for a scalar code;
- Magnetica r.2 is nice, the disassembly (-O3) gladdens the eyes.
Finally, wanted all fellow members, interested in writing their own "intrinsics" (in its original sense - "essential"), to have one booklet showing how much C code can reinforce the QB64 muscle flexing:
With this my Quicksort detour is completed.
Post by: Sanmayce on December 07, 2021, 10:13:37 pm
Sometimes, the need to view broken/long lines arises, then Masakari comes sorting them, FAST!
Looking what the most powerful editor UltraEdit v28.20 (latest version) can do with loaded file:
https://www.ultraedit.com/wiki/images/2/25/Ultraedit_sort.png (https://www.ultraedit.com/wiki/images/2/25/Ultraedit_sort.png)
I see only the basic functionality, but when comes to spotting/browsing longest lines in huge files UE is not ultra enough, to fill the gap, here comes the fastest sorter combined with the fastest scheme of representing the lines - a pair of QWORDS: Offset:Length, naturally the sorting is simplified down to sorting all those pairs without accessing external RAM nor computing length of lines.
Again the longest XML file (English Wikipedia) is our testdataset:
[ This attachment cannot be displayed inline in 'Print Page' view ]
The shortcut 'Shift+1' sorts by Length, the longest line is at the bottom:
[ This attachment cannot be displayed inline in 'Print Page' view ]
The shortcut 'Shift+2' sorts by Offset, in fact, it returns the original file (instead of reloading it) i.e. REVERT.
As far as I know, there is still no tool/editor capable to sort 1+ billion lines in (5:01:02 - 4:59:15) = 107 seconds and allow to browse all the longest lines.
Post by: Sanmayce on December 09, 2021, 09:37:31 am
Glad to share another superfast function - UCASE_XMM, it replaces the superslow UCASE$() built-in.
Vectorization using XMM registers trumps bigtime the scalar variant.
[ This attachment cannot be displayed inline in 'Print Page' view ]
My intent is to include more superfast functions in next edition of my "manatarka" custom library, QB64 deserves some boost in textual department, as:
- a replacement for INSTR() built-in, memmem(), the FASTEST variant (to my knowledge) - Railgun();
- 128bit hasher, the FASTEST variant (to my knowledge) - Gumbotron();
- UCASE_YMM, the AVX variant.
Post by: Sanmayce on December 10, 2021, 02:26:35 am
Okay, the testmachine is my laptop with i5-7200U, 36GB DDR4, the testdatafile is the ~4MB King James Bible.
The speedup is awesome, keep in mind the longer the Needle/Pattern the bigger the speedup becomes, short needles in range 2..6 are not that indicative of how superfast Railgun() is:
My SCALAR memmem(), called Railgun_Trolldom, is 2543/1413= 1.79x faster than INSTR() when the Needle is "gods".
My VECTOR memmem(), called Railgun_Nyotengu, is 6359/1413= 4.50x faster than INSTR() when the Needle is "gods".
As for the UCASE_XMM, the speedup is 30x to 60x:
In the future, UCASE_XMM is to be incorporated into NyoTengu_XMM - then we will have the FASTEST Case-Insensitive INSTR() counterpart!
The manatarka.h housing the UCASE_XMM, Railgun_Trolldom and Railgun_Nyotengu is attached along with the QB64 test program.
Enjoy!
Post by: SMcNeill on December 10, 2021, 02:52:27 am
Post by: Sanmayce on December 10, 2021, 03:06:46 am
Try https://www.qb64.org/wiki/STRICMP when you don't care about case while comparing strings.
This is good to have as well, but I am not aware of INSTR() that is Case-Insensitive, let alone superfast.
If we are forced to search for "gods" in the Project Gutenberg KJV Bible, case-insensitively then we have 2 more hits i.e. "Gods".
The idea is QB64 community to have even faster functions than GLIBC.
Post by: Sanmayce on December 11, 2021, 01:10:13 am
My Gumbotron hash function (more in Reply #47 in this thread) is the fastest 128bit hasher, it comes in XMM and YMM variants i.e. for SSE4.1 and AVX2, while generating the same hash. The screenshot shows the Gumbotron_YMM giving the correct hash (as it should be, given in the orange console by Nakamichi):
[ This attachment cannot be displayed inline in 'Print Page' view ]
It returns a pair of QWORDS, so if the usecase demands 64bit hash you can use either loPart or HiPart.
It is nearly as fast as the fastest 64bit hashers, but much stronger. In practice, 64bit hashers generate a lot of collisions even for casual tasks dealing with few million (try 7 billion - e.g. the names of all people) keys, nah, Gumbotron's 2x64bit scheme proves supereffective, my latest compressor Nakamichi uses this very hash without any problems - heavily tested hasher - battle-tested it is.
Imagine a search scenario where you have many keys/elements (different in length, or not), then by hashing them you can tighten them down to 16 bytes, after that you can use Quicksort 'Magnetica' to sort those hashes, and "finalize" with goodold Binary-Search.
In Nakamichi, I exploit Gumbotron even better, due to its superior 128bit-ness, it is used as a lossy compressor - shrinking all kind of keys 16+ in size (18, ... 256) down to 16 bytes. If there was a single collision then big problems would ensue, but the linear mixer provided by AES is doing great job, so, no problema.
The full source (and Windows binary) is attached.
Post by: Sanmayce on December 26, 2021, 11:48:14 pm
That is, the previous stupid checking of each word against the whole 1,000,000+ Binary-Search pool now was limited only to THE fraction of the pool where the corresponding length is located - which lowers the number of seeks/comparisons.
Additionally, all the UNFAMILIAR words, now, are dumped as an index, but placed in the very beginning, in order of appearance.
Thanks to that feature, one is allowed quickly to spot misspelled words, as I did only with a few glances, 2 buggy "daughters" found:
daughers
daughers
Project Gutenberg failed to deliver quality, obviously the correctors still rely on non-automatic methods.
Finally, we have a decent benchmark utilizing the Binary-Search in a speedy manner, it is a good start for all kind of experiments since it is creating as a result the .HTM counterpart (with all unfamiliar words in BOLD, all OED distinct words are used as a spell-check wordlist) of our .TXT file:
[ This attachment cannot be displayed inline in 'Print Page' view ]
It throws all the words of KJV Bible at OED 2nd edition, which is 700,000+ words against the 1,000,000+ OED words:
[ This attachment cannot be displayed inline in 'Print Page' view ]
The spotted errors are to go along with the entire .HTM into .DOC|.PDF file for hardcopy i.e. paper reviewing, with good old pencils:
[ This attachment cannot be displayed inline in 'Print Page' view ]
Spell-checking KJV Bible in 1.677 seconds, can we do better?
[ This attachment cannot be displayed inline in 'Print Page' view ]
On Linux, the newer GCC (v10.2.1) generates 100ms faster code, lowering the conversion down to:
[ This attachment cannot be displayed inline in 'Print Page' view ]
[ This attachment cannot be displayed inline in 'Print Page' view ]
Anyway, this TXT2HTM code will be part of incoming Masakari r.9, enabling having spell-checked .HTMs only a key and few seconds away...
The source code and binaries for Linux and Windows (also the OED wordlist) are in the attached package.
Post by: Sanmayce on February 22, 2022, 03:52:53 am
I spotted a conditional swap within the mainloop, now, in r.5 this branch is removed and put outside the hottest loop.
So, r.5 is algorithmically superior compared to r.4, the performance varies between AMD and Intel, between ICL and GCC, overall it pushes the limits once more.
Behold the simplicity itself:
Code: QB64: [Select]
- // Magnetica r.5BB:
- /*
- ; mark_description "Intel(R) C++ Compiler XE for applications running on Intel(R) 64, Version 15.0.0.108 Build 20140726";
- ; mark_description "-S -O3 -D_N_HIGH_PRIORITY";
- Quicksort_QB64_v9 PROC
- ; parameter 1: rcx
- ; parameter 2: rdx
- ; parameter 3: r8
- .B12.1:: ; Preds .B12.0
- push r13 ;1172.72
- push r14 ;1172.72
- push r15 ;1172.72
- push rbp ;1172.72
- mov eax, 800024 ;1172.72
- call __chkstk ;1172.72
- mov eax, 2 ;1191.5
- mov QWORD PTR [40+rsp], rdx ;1190.17
- mov QWORD PTR [48+rsp], r8 ;1191.17
- .B12.2:: ; Preds .B12.20 .B12.1
- mov rdx, QWORD PTR [32+rsp+rax*8] ;1193.17
- mov r13, QWORD PTR [24+rsp+rax*8] ;1194.16
- add rax, -2 ;1195.31
- cmp r13, rdx ;1197.47
- jge .B12.20 ; Prob 10% ;1197.47
- .B12.3:: ; Preds .B12.2
- mov r9, r13 ;1199.13
- .B12.4:: ; Preds .B12.18 .B12.3
- mov r10, QWORD PTR [rcx+r13*8] ;1229.13
- lea r15, QWORD PTR [r13+rdx] ;1229.48
- sar r15, 1 ;1229.56
- mov r14, r13 ;1200.13
- mov r11, rdx ;1198.13
- mov rbp, r9 ;1199.13
- mov r9, r14 ;1200.13
- mov r8, QWORD PTR [rcx+r15*8] ;1229.13
- mov QWORD PTR [rcx+r15*8], r10 ;1229.13
- mov QWORD PTR [rcx+r13*8], r8 ;1229.13
- .B12.5:: ; Preds .B12.4 .B12.13
- inc r14 ;1282.27
- mov r10, QWORD PTR [rcx+r14*8] ;1283.29
- cmp r8, r10 ;1283.29
- ja .B12.12 ; Prob 22% ;1283.29
- .B12.6:: ; Preds .B12.5
- jae .B12.13 ; Prob 50% ;1287.36
- .B12.7:: ; Preds .B12.6
- mov r15, QWORD PTR [rcx+r11*8] ;1288.35
- cmp r8, r15 ;1288.35
- jae .B12.11 ; Prob 10% ;1288.35
- .B12.9:: ; Preds .B12.7 .B12.9
- dec r11 ;1289.39
- mov r15, QWORD PTR [rcx+r11*8] ;1288.35
- cmp r8, r15 ;1288.35
- jb .B12.9 ; Prob 82% ;1288.35
- .B12.11:: ; Preds .B12.9 .B12.7
- mov QWORD PTR [rcx+r14*8], r15 ;1291.21
- dec r14 ;1296.31
- mov QWORD PTR [rcx+r11*8], r10 ;1291.21
- dec r11 ;1295.35
- jmp .B12.13 ; Prob 100% ;1295.35
- .B12.12:: ; Preds .B12.5
- mov r15, QWORD PTR [rcx+rbp*8] ;1284.21
- mov QWORD PTR [rcx+rbp*8], r10 ;1284.21
- inc rbp ;1285.31
- mov QWORD PTR [rcx+r14*8], r15 ;1284.21
- .B12.13:: ; Preds .B12.12 .B12.11 .B12.6
- cmp r14, r11 ;1281.24
- jl .B12.5 ; Prob 82% ;1281.24
- .B12.14:: ; Preds .B12.13
- jle .B12.16 ; Prob 78% ;1299.22
- .B12.15:: ; Preds .B12.14
- mov r8, QWORD PTR [8+rcx+r11*8] ;1300.21
- mov r10, QWORD PTR [8+rcx+r14*8] ;1300.21
- mov QWORD PTR [8+rcx+r14*8], r8 ;1300.21
- mov QWORD PTR [8+rcx+r11*8], r10 ;1300.21
- .B12.16:: ; Preds .B12.14 .B12.15
- inc r14 ;1655.25
- cmp rdx, r14 ;1730.49
- jle .B12.18 ; Prob 62% ;1730.49
- .B12.17:: ; Preds .B12.16
- add rax, 2 ;1731.39
- mov QWORD PTR [24+rsp+rax*8], r14 ;1732.17
- mov QWORD PTR [32+rsp+rax*8], rdx ;1733.17
- .B12.18:: ; Preds .B12.17 .B12.16
- lea rdx, QWORD PTR [-1+rbp] ;1654.25
- cmp r13, rdx ;1197.47
- jl .B12.4 ; Prob 82% ;1197.47
- .B12.20:: ; Preds .B12.18 .B12.2
- DB 15 ;1743.28
- DB 31 ;1743.28
- DB 64 ;1743.28
- DB 0 ;1743.28
- DB 15 ;1743.28
- DB 31 ;1743.28
- DB 128 ;1743.28
- DB 0 ;1743.28
- DB 0 ;1743.28
- DB 0 ;1743.28
- DB 0 ;1743.28
- test rax, rax ;1743.28
- jne .B12.2 ; Prob 99% ;1743.28
- .B12.21:: ; Preds .B12.20
- add rsp, 800024 ;1753.1
- pop rbp ;1753.1
- pop r15 ;1753.1
- pop r14 ;1753.1
- pop r13 ;1753.1
- ret ;1753.1
- ALIGN 16
- .B12.22::
- ; mark_end;
- Quicksort_QB64_v9 ENDP
- */
In order gladly to say QB64 stands for "Quickness" once again, my wish is QB64 community to have the qsm.h header (built-in function to QB64 library) outperforming both qsort() C implementations of Intel and GNU, enjoy.
Post by: Sanmayce on March 08, 2022, 05:10:05 pm
As I see it, he did dive deep in sorting, achieving outstanding results.
Yet, intuitively, I see different approach, which potentially can reach and even surpass his awesome speeds, to be seen...
Currently, the picture of hi-speed sorting is this (in order to reproduce the results, the benchmark package is attached):
Code: [Select]
// Test run: 2022-Mar-08:
// Laptop "Compressionette", Intel 'Kaby Lake' i5-7200U 3.1GHz max turbo, 36GB DDR4 2133MHz:
// +--------------------+---------------------------+---------------------------+---------------------------+---------------------------+----------------------------+-----------------------------+
// | Performer/Keys | #1, FEW distinct | #2, MANY distinct | #3, MANYmore distinct | #4, ALL distinct | #5, ALLmore distinct | #6, ALLmax distinct |
// +--------------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+--------------+--------------+--------------+
// | Operating System, | Windows 10, | Fedora 35, | Windows 10, | Fedora 35, | Windows 10, | Fedora 35, | Windows 10, | Fedora 35, | Windows 10, | Fedora 35, | Windows 10, | Fedora 35, |
// | Compiler, -O3 | Intel v15.0 | GCC 11.2.1 | Intel v15.0 | GCC 11.2.1 | Intel v15.0 | GCC 11.2.1 | Intel v15.0 | GCC 11.2.1 | Intel v15.0 | GCC 11.2.1 | Intel v15.0 | GCC 11.2.1 |
// +--------------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+--------------+--------------+--------------+
// | qsort | 59 seconds | 377 seconds | 336 seconds | 541 seconds | 157 seconds | 195 seconds | 435 seconds | 534 seconds | 851 seconds | 1036 seconds | N.A. | N.A. |
// | Magnetica v.13 | 31 seconds | 30 seconds | 202 seconds | 196 seconds | 88 seconds | 85 seconds | 259 seconds | 250 seconds | 506 seconds | 493 seconds | N.A. | N.A. |
// | Bentley-McIlroy | 38 seconds | 36 seconds | 205 seconds | 208 seconds | 92 seconds | 94 seconds | 279 seconds | 281 seconds | 544 seconds | 553 seconds | N.A. | N.A. |
// | Crumsort | 30 seconds | 32 seconds | 132 seconds | 150 seconds | 64 seconds | 70 seconds | 184 seconds | 192 seconds | 357 seconds | 376 seconds | N.A. | N.A. |
// +--------------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+--------------+--------------+--------------+
// | Best Time (bare | | | | | | |
// | bone in-place QS): | 30s PARITY | 132s for Crumsort | 64s for Crumsort | 184s for Crumsort | 357s for Crumsort | N.A. |
// +--------------------+---------------------------+---------------------------+---------------------------+---------------------------+----------------------------+-----------------------------+
Code: [Select]
// +--------------------+---------------------------+---------------------------+---------------------------+---------------------------+----------------------------+-----------------------------+
// | Performer/Keys | #1, FEW distinct | #2, MANY distinct | #3, MANYmore distinct | #4, ALL distinct | #5, ALLmore distinct | #6, ALLmax distinct |
// +--------------------+---------------------------+---------------------------+---------------------------+---------------------------+----------------------------+-----------------------------+
// | Operating System, | Fedora 35, GCC 11.2.1 | Fedora 35, GCC 11.2.1 | Fedora 35, GCC 11.2.1 | Fedora 35, GCC 11.2.1 | Fedora 35, GCC 11.2.1 | Fedora 35, GCC 11.2.1 |
// | Compiler, -O3 | instructions; IPC | instructions; IPC | instructions; IPC | instructions; IPC | instructions; IPC | instructions; IPC |
// +--------------------+---------------------------+---------------------------+---------------------------+---------------------------+----------------------------+-----------------------------+
// | qsort | 3,302,993,934,921; 2.75 | 2,983,579,082,155; 1.75 | 886,263,153,476; 1.41 | 2,352,769,705,563; 1.38 | 4,527,367,288,814; 1.39 | N.A. |
// | Magnetica v.13 | 131,873,917,282; 1.00 | 658,478,895,600; 1.02 | 309,149,594,748; 1.08 | 884,297,729,161; 1.06 | 1,726,931,029,634; 1.08 | N.A. |
// | Bentley-McIlroy | 164,915,835,204; 1.09 | 681,956,155,364; 1.00 | 322,584,009,352; 1.04 | 944,038,959,690; 1.02 | 1,719,825,062,847; 0.97 | N.A. |
// | Crumsort | 312,328,497,447; 2.29 | 1,295,551,817,497; 2.57 | 603,276,911,007; 2.53 | 1,597,685,291,532; 2.46 | 3,091,001,982,856; 2.51 | N.A. |
// +--------------------+---------------------------+---------------------------+---------------------------+---------------------------+----------------------------+-----------------------------+
Code: [Select]
// Speed Roster, (the base speed 1.00x is GLIBC's qsort):
// Rank #1: 2683/820= 3.27x = 32+150+ 70+192+ 376= 820 seconds for Crumsort
// Rank #2: 2683/1054= 2.54x = 30+196+ 85+250+ 493= 1054 seconds for Magnetica v.13
// Rank #3: 2683/1172= 2.28x = 36+208+ 94+281+ 553= 1172 seconds for Bentley-McIlroy
// Rank #4: 2683/2683= 1.00x = 377+541+195+534+1036= 2683 seconds for GLIBC's qsort
//
// Legend (The time is exactly the Sort process time):
// #1,FEW = 2,233,861,800 keys, of them distinct = 10; 178,708,944 bytes 22338618_QWORDS.bin; elements = 178,708,944/8 *100; // Keys are 100 times duplicated
// #2,MANY = 2,482,300,900 keys, of them distinct = 2,847,531; 24,823,016 bytes mobythesaurus.txt; elements = 24823016 -8+1; // BuildingBlocks are size-order+1, they are 100 times duplicated
// #3,MANYmore = 1,137,582,073 keys, of them distinct = 77,275,994; 1,137,582,080 bytes linux-5.15.25.tar; elements = 1137582080 -8+1; // BuildingBlocks are size-order+1
// #4,ALL = 2,009,333,753 keys, of them distinct = 1,912,608,132; 2,009,333,760 bytes Fedora-Workstation-Live-x86_64-35-1.2.iso; elements = 2009333760 -8+1; // BuildingBlocks are size-order+1
// #5,ALLmore = 3,803,483,825 keys, of them distinct = 3,346,259,533; 3,803,483,832 bytes Fedora-Workstation-35-1.2.aarch64.raw.xz; elements = 3803483832 -8+1; // BuildingBlocks are size-order+1
// #6,ALLmax = 7,798,235,435 keys, of them distinct = 6,770,144,405; 7,798,235,442 bytes math.stackexchange.com_en_all_2019-02.zim; elements = 7798235442 -8+1; // BuildingBlocks are size-order+1
Code: [Select]
// Notes:
// - Scandum's Crumsort is the FASTEST in-place sorter, known to me, hail Scandum!
// - All the runs were in "Current priority class is REALTIME_PRIORITY_CLASS" for Windows and "Current priority is -20." for Linux;
// To see more stats (the tables were deriving from) see 'log_i5-7200U_MAR08.txt';
// - Benchmark needs 32GB RAM, and 64GB for the 6th testset;
// - The whole package (except the 3rd, 4th, 5th and 6th datasets) is downloadable at:
// www.sanmayce.com/QS_showdown_r13.zip
// - To reproduce the roster, run on Windows or Linux:
// - BENCH_ICL32GB.BAT
// - BENCH_ICL64GB.BAT
// - sh bench_gcc32GB.sh
// - sh bench_gcc64GB.sh
// - 3rd dataset is downloadable at:
// https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.25.tar.xz
// - 4th dataset is downloadable at:
// https://download.fedoraproject.org/pub/fedora/linux/releases/35/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-35-1.2.iso
// - 5th dataset is downloadable at:
// https://download.fedoraproject.org/pub/fedora/linux/releases/35/Workstation/aarch64/images/Fedora-Workstation-35-1.2.aarch64.raw.xz
// - 6th dataset is downloadable at:
// https://download.kiwix.org/zim/stack_exchange/math.stackexchange.com_en_all_2019-02.zim
// - Managed to reduce the mainloop down to 68 bytes (from 71 in r.12), the gain comes from switching to pointers, had to do that long time ago, wanted to keep the etude ARRAY-syntax friendly. Now, r.12 uses arrays, whereas r.13 uses pointers. Another turning point, GCC now beats ICL in all tests.
The current best is at:
https://github.com/scandum/crumsort
Here comes Magnetica v13, a view from the kitchen, this time around, GCC v11.2.1 proves superior to ICL v15.0, generating faster code in all tests:
Code: [Select]
// And the Assembly code generated by GCC 11.2.1 for Magnetica_v13:
// 001 Quicksort_QB64_v13:
// 002 .LFB214:
// 003 .cfi_startproc
// 004 pushq %r15
// 005 .cfi_def_cfa_offset 16
// 006 .cfi_offset 15, -16
// 007 leaq -8(%rdi,%rsi,8), %r10
// 008 movq %rdi, %xmm0
// 009 movq %rdi, %rax
// 010 pushq %r14
// 011 .cfi_def_cfa_offset 24
// 012 .cfi_offset 14, -24
// 013 movq %r10, %xmm2
// 014 movl $2, %r15d
// 015 pushq %r13
// 016 .cfi_def_cfa_offset 32
// 017 .cfi_offset 13, -32
// 018 punpcklqdq %xmm2, %xmm0
// 019 pushq %r12
// 020 .cfi_def_cfa_offset 40
// 021 .cfi_offset 12, -40
// 022 pushq %rbp
// 023 .cfi_def_cfa_offset 48
// 024 .cfi_offset 6, -48
// 025 pushq %rbx
// 026 .cfi_def_cfa_offset 56
// 027 .cfi_offset 3, -56
// 028 subq $143920, %rsp
// 029 .cfi_def_cfa_offset 143976
// 030 movups %xmm0, -96(%rsp)
// 031 .p2align 4,,10
// 032 .p2align 3
// 033 .L5115:
// 034 movq %r10, %rcx
// 035 subq $2, %r15
// 036 subq %rax, %rcx
// 037 testq %rcx, %rcx
// 038 jle .L5096
// 039 leaq 8(%rax), %rbx
// 040 .L5113:
// 041 movq %rcx, %rsi
// 042 movq (%rax), %rdx
// 043 sarq $3, %rsi
// 044 cmpq $55, %rcx
// 045 ja .L5097
// 046 jmp *.L5099(,%rsi,8)
// 047 .section .rodata
// 048 .align 8
// 049 .align 4
// 050 .L5099:
// 051 .quad .L5097
// 052 .quad .L5104
// 053 .quad .L5103
// 054 .quad .L5102
// 055 .quad .L5101
// 056 .quad .L5100
// 057 .quad .L5098
// 058 .text
// 059 .p2align 4,,10
// 060 .p2align 3
// 061 .L5098:
// 062 movq 8(%rax), %rbp
// 063 movq 16(%rax), %rbx
// 064 xorl %ecx, %ecx
// 065 movq 24(%rax), %r11
// 066 movq 32(%rax), %r10
// 067 cmpq %rdx, %rbp
// 068 movq 40(%rax), %r9
// 069 movq 48(%rax), %r12
// 070 setb %cl
// 071 cmpq %rdx, %rbx
// 072 adcl $0, %ecx
// 073 cmpq %rdx, %r11
// 074 adcl $0, %ecx
// 075 cmpq %rdx, %r10
// 076 adcl $0, %ecx
// 077 cmpq %rdx, %r9
// 078 adcl $0, %ecx
// 079 cmpq %rdx, %r12
// 080 adcl $0, %ecx
// 081 xorl %r8d, %r8d
// 082 cmpq %rdx, %rbp
// 083 movl %ecx, -108(%rsp)
// 084 setnb %r8b
// 085 xorl %ecx, %ecx
// 086 cmpq %rbx, %rbp
// 087 seta %cl
// 088 addl %ecx, %r8d
// 089 xorl %ecx, %ecx
// 090 cmpq %r11, %rbp
// 091 seta %cl
// 092 addl %ecx, %r8d
// 093 xorl %ecx, %ecx
// 094 cmpq %r10, %rbp
// 095 seta %cl
// 096 addl %ecx, %r8d
// 097 xorl %ecx, %ecx
// 098 cmpq %r9, %rbp
// 099 seta %cl
// 100 addl %ecx, %r8d
// 101 xorl %ecx, %ecx
// 102 cmpq %r12, %rbp
// 103 seta %cl
// 104 xorl %edi, %edi
// 105 addl %ecx, %r8d
// 106 cmpq %rdx, %rbx
// 107 setnb %dil
// 108 xorl %ecx, %ecx
// 109 cmpq %rbx, %rbp
// 110 setbe %cl
// 111 addl %ecx, %edi
// 112 xorl %ecx, %ecx
// 113 cmpq %r11, %rbx
// 114 seta %cl
// 115 addl %ecx, %edi
// 116 xorl %ecx, %ecx
// 117 cmpq %r10, %rbx
// 118 seta %cl
// 119 addl %ecx, %edi
// 120 xorl %ecx, %ecx
// 121 cmpq %r9, %rbx
// 122 seta %cl
// 123 addl %ecx, %edi
// 124 xorl %ecx, %ecx
// 125 cmpq %r12, %rbx
// 126 seta %cl
// 127 xorl %esi, %esi
// 128 addl %ecx, %edi
// 129 cmpq %rdx, %r11
// 130 setnb %sil
// 131 xorl %ecx, %ecx
// 132 cmpq %r11, %rbp
// 133 setbe %cl
// 134 addl %ecx, %esi
// 135 xorl %ecx, %ecx
// 136 cmpq %r11, %rbx
// 137 setbe %cl
// 138 addl %ecx, %esi
// 139 xorl %ecx, %ecx
// 140 cmpq %r10, %r11
// 141 seta %cl
// 142 addl %ecx, %esi
// 143 xorl %ecx, %ecx
// 144 cmpq %r9, %r11
// 145 seta %cl
// 146 addl %ecx, %esi
// 147 xorl %ecx, %ecx
// 148 cmpq %r12, %r11
// 149 seta %cl
// 150 addl %ecx, %esi
// 151 xorl %ecx, %ecx
// 152 cmpq %rdx, %r10
// 153 setnb %cl
// 154 xorl %r13d, %r13d
// 155 cmpq %r10, %rbp
// 156 setbe %r13b
// 157 addl %r13d, %ecx
// 158 xorl %r13d, %r13d
// 159 cmpq %r10, %rbx
// 160 setbe %r13b
// 161 addl %r13d, %ecx
// 162 xorl %r13d, %r13d
// 163 cmpq %r10, %r11
// 164 setbe %r13b
// 165 addl %r13d, %ecx
// 166 xorl %r13d, %r13d
// 167 cmpq %r9, %r10
// 168 seta %r13b
// 169 addl %r13d, %ecx
// 170 xorl %r13d, %r13d
// 171 cmpq %r12, %r10
// 172 seta %r13b
// 173 xorl %r14d, %r14d
// 174 addl %r13d, %ecx
// 175 cmpq %rdx, %r9
// 176 setnb %r14b
// 177 xorl %r13d, %r13d
// 178 cmpq %r9, %rbp
// 179 setbe %r13b
// 180 addl %r14d, %r13d
// 181 xorl %r14d, %r14d
// 182 cmpq %r9, %rbx
// 183 setbe %r14b
// 184 addl %r13d, %r14d
// 185 xorl %r13d, %r13d
// 186 cmpq %r9, %r11
// 187 setbe %r13b
// 188 addl %r14d, %r13d
// 189 xorl %r14d, %r14d
// 190 cmpq %r9, %r10
// 191 setbe %r14b
// 192 addl %r13d, %r14d
// 193 xorl %r13d, %r13d
// 194 cmpq %r12, %r9
// 195 seta %r13b
// 196 addl %r14d, %r13d
// 197 movslq -108(%rsp), %r14
// 198 movq %rdx, (%rax,%r14,8)
// 199 movslq %r8d, %rdx
// 200 movl -108(%rsp), %r14d
// 201 movq %rbp, (%rax,%rdx,8)
// 202 movslq %edi, %rdx
// 203 movq %rbx, (%rax,%rdx,8)
// 204 movslq %esi, %rdx
// 205 movq %r11, (%rax,%rdx,8)
// 206 movslq %ecx, %rdx
// 207 movq %r10, (%rax,%rdx,8)
// 208 movslq %r13d, %rdx
// 209 movq %r9, (%rax,%rdx,8)
// 210 leal (%r14,%r8), %edx
// 211 addl %edi, %edx
// 212 addl %esi, %edx
// 213 addl %ecx, %edx
// 214 movl $21, %ecx
// 215 addl %r13d, %edx
// 216 subl %edx, %ecx
// 217 movslq %ecx, %rdx
// 218 movq %r12, (%rax,%rdx,8)
// 219 .p2align 4,,10
// 220 .p2align 3
// 221 .L5096:
// 222 testq %r15, %r15
// 223 je .L5095
// 224 .L5121:
// 225 movq -104(%rsp,%r15,8), %r10
// 226 movq -112(%rsp,%r15,8), %rax
// 227 jmp .L5115
// 228 .p2align 4,,10
// 229 .p2align 3
// 230 .L5103:
// 231 movq 8(%rax), %rdi
// 232 movq 16(%rax), %r8
// 233 xorl %ecx, %ecx
// 234 cmpq %rdx, %rdi
// 235 setb %cl
// 236 cmpq %rdx, %r8
// 237 adcl $0, %ecx
// 238 xorl %esi, %esi
// 239 cmpq %rdx, %rdi
// 240 setnb %sil
// 241 xorl %r9d, %r9d
// 242 cmpq %r8, %rdi
// 243 seta %r9b
// 244 addl %r9d, %esi
// 245 movslq %ecx, %r9
// 246 movq %rdx, (%rax,%r9,8)
// 247 movslq %esi, %rdx
// 248 addl %esi, %ecx
// 249 movq %rdi, (%rax,%rdx,8)
// 250 movl $3, %edx
// 251 subl %ecx, %edx
// 252 movslq %edx, %rdx
// 253 movq %r8, (%rax,%rdx,8)
// 254 testq %r15, %r15
// 255 jne .L5121
// 256 .L5095:
// 257 addq $143920, %rsp
// 258 .cfi_remember_state
// 259 .cfi_def_cfa_offset 56
// 260 popq %rbx
// 261 .cfi_def_cfa_offset 48
// 262 popq %rbp
// 263 .cfi_def_cfa_offset 40
// 264 popq %r12
// 265 .cfi_def_cfa_offset 32
// 266 popq %r13
// 267 .cfi_def_cfa_offset 24
// 268 popq %r14
// 269 .cfi_def_cfa_offset 16
// 270 popq %r15
// 271 .cfi_def_cfa_offset 8
// 272 ret
// 273 .p2align 4,,10
// 274 .p2align 3
// 275 .L5104:
// 276 .cfi_restore_state
// 277 movq 8(%rax), %rcx
// 278 cmpq %rdx, %rcx
// 279 sbbq %rsi, %rsi
// 280 andl $8, %esi
// 281 cmpq %rdx, %rcx
// 282 movq %rdx, (%rax,%rsi)
// 283 movl $1, %edx
// 284 sbbl $0, %edx
// 285 movslq %edx, %rdx
// 286 movq %rcx, (%rax,%rdx,8)
// 287 jmp .L5096
// 288 .p2align 4,,10
// 289 .p2align 3
// 290 .L5101:
// 291 movq 8(%rax), %r10
// 292 movq 16(%rax), %r9
// 293 xorl %ecx, %ecx
// 294 movq 24(%rax), %r8
// 295 movq 32(%rax), %rdi
// 296 cmpq %rdx, %r10
// 297 setb %cl
// 298 cmpq %rdx, %r9
// 299 adcl $0, %ecx
// 300 cmpq %rdx, %r8
// 301 adcl $0, %ecx
// 302 cmpq %rdx, %rdi
// 303 adcl $0, %ecx
// 304 xorl %esi, %esi
// 305 cmpq %rdx, %r10
// 306 setnb %sil
// 307 xorl %r11d, %r11d
// 308 cmpq %r9, %r10
// 309 seta %r11b
// 310 addl %r11d, %esi
// 311 xorl %r11d, %r11d
// 312 cmpq %r8, %r10
// 313 seta %r11b
// 314 addl %r11d, %esi
// 315 xorl %r11d, %r11d
// 316 cmpq %rdi, %r10
// 317 seta %r11b
// 318 addl %esi, %r11d
// 319 xorl %esi, %esi
// 320 cmpq %rdx, %r9
// 321 setnb %sil
// 322 xorl %ebx, %ebx
// 323 cmpq %r9, %r10
// 324 setbe %bl
// 325 addl %ebx, %esi
// 326 xorl %ebx, %ebx
// 327 cmpq %r8, %r9
// 328 seta %bl
// 329 addl %ebx, %esi
// 330 xorl %ebx, %ebx
// 331 cmpq %rdi, %r9
// 332 seta %bl
// 333 addl %esi, %ebx
// 334 xorl %esi, %esi
// 335 cmpq %rdx, %r8
// 336 setnb %sil
// 337 xorl %ebp, %ebp
// 338 cmpq %r8, %r10
// 339 setbe %bpl
// 340 addl %ebp, %esi
// 341 xorl %ebp, %ebp
// 342 cmpq %r8, %r9
// 343 setbe %bpl
// 344 addl %ebp, %esi
// 345 xorl %ebp, %ebp
// 346 cmpq %rdi, %r8
// 347 seta %bpl
// 348 addl %ebp, %esi
// 349 movslq %ecx, %rbp
// 350 addl %r11d, %ecx
// 351 movq %rdx, (%rax,%rbp,8)
// 352 movslq %r11d, %rdx
// 353 addl %ebx, %ecx
// 354 movq %r10, (%rax,%rdx,8)
// 355 movslq %ebx, %rdx
// 356 addl %esi, %ecx
// 357 movq %r9, (%rax,%rdx,8)
// 358 movslq %esi, %rdx
// 359 movq %r8, (%rax,%rdx,8)
// 360 movl $10, %edx
// 361 subl %ecx, %edx
// 362 movslq %edx, %rdx
// 363 movq %rdi, (%rax,%rdx,8)
// 364 jmp .L5096
// 365 .p2align 4,,10
// 366 .p2align 3
// 367 .L5102:
// 368 movq 8(%rax), %r10
// 369 movq 16(%rax), %r9
// 370 xorl %ecx, %ecx
// 371 movq 24(%rax), %r8
// 372 cmpq %rdx, %r10
// 373 setb %cl
// 374 cmpq %rdx, %r9
// 375 adcl $0, %ecx
// 376 cmpq %rdx, %r8
// 377 adcl $0, %ecx
// 378 xorl %edi, %edi
// 379 cmpq %rdx, %r10
// 380 setnb %dil
// 381 xorl %esi, %esi
// 382 cmpq %r9, %r10
// 383 seta %sil
// 384 addl %esi, %edi
// 385 xorl %esi, %esi
// 386 cmpq %r8, %r10
// 387 seta %sil
// 388 addl %esi, %edi
// 389 xorl %esi, %esi
// 390 cmpq %rdx, %r9
// 391 setnb %sil
// 392 xorl %r11d, %r11d
// 393 cmpq %r9, %r10
// 394 setbe %r11b
// 395 addl %r11d, %esi
// 396 xorl %r11d, %r11d
// 397 cmpq %r8, %r9
// 398 seta %r11b
// 399 addl %r11d, %esi
// 400 movslq %ecx, %r11
// 401 addl %edi, %ecx
// 402 movq %rdx, (%rax,%r11,8)
// 403 movslq %edi, %rdx
// 404 addl %esi, %ecx
// 405 movq %r10, (%rax,%rdx,8)
// 406 movslq %esi, %rdx
// 407 movq %r9, (%rax,%rdx,8)
// 408 movl $6, %edx
// 409 subl %ecx, %edx
// 410 movslq %edx, %rdx
// 411 movq %r8, (%rax,%rdx,8)
// 412 jmp .L5096
// 413 .p2align 4,,10
// 414 .p2align 3
// 415 .L5100:
// 416 movq 8(%rax), %r12
// 417 movq 16(%rax), %rbp
// 418 xorl %ecx, %ecx
// 419 movq 24(%rax), %r11
// 420 movq 32(%rax), %r10
// 421 cmpq %rdx, %r12
// 422 movq 40(%rax), %rbx
// 423 setb %cl
// 424 cmpq %rdx, %rbp
// 425 adcl $0, %ecx
// 426 cmpq %rdx, %r11
// 427 adcl $0, %ecx
// 428 cmpq %rdx, %r10
// 429 adcl $0, %ecx
// 430 cmpq %rdx, %rbx
// 431 adcl $0, %ecx
// 432 xorl %r9d, %r9d
// 433 cmpq %rdx, %r12
// 434 setnb %r9b
// 435 xorl %esi, %esi
// 436 cmpq %rbp, %r12
// 437 seta %sil
// 438 addl %esi, %r9d
// 439 xorl %esi, %esi
// 440 cmpq %r11, %r12
// 441 seta %sil
// 442 addl %esi, %r9d
// 443 xorl %esi, %esi
// 444 cmpq %r10, %r12
// 445 seta %sil
// 446 addl %esi, %r9d
// 447 xorl %esi, %esi
// 448 cmpq %rbx, %r12
// 449 seta %sil
// 450 xorl %r8d, %r8d
// 451 addl %esi, %r9d
// 452 cmpq %rdx, %rbp
// 453 setnb %r8b
// 454 xorl %esi, %esi
// 455 cmpq %rbp, %r12
// 456 setbe %sil
// 457 addl %esi, %r8d
// 458 xorl %esi, %esi
// 459 cmpq %r11, %rbp
// 460 seta %sil
// 461 addl %esi, %r8d
// 462 xorl %esi, %esi
// 463 cmpq %r10, %rbp
// 464 seta %sil
// 465 addl %esi, %r8d
// 466 xorl %esi, %esi
// 467 cmpq %rbx, %rbp
// 468 seta %sil
// 469 xorl %edi, %edi
// 470 addl %esi, %r8d
// 471 cmpq %rdx, %r11
// 472 setnb %dil
// 473 xorl %esi, %esi
// 474 cmpq %r11, %r12
// 475 setbe %sil
// 476 addl %esi, %edi
// 477 xorl %esi, %esi
// 478 cmpq %r11, %rbp
// 479 setbe %sil
// 480 addl %esi, %edi
// 481 xorl %esi, %esi
// 482 cmpq %r10, %r11
// 483 seta %sil
// 484 addl %esi, %edi
// 485 xorl %esi, %esi
// 486 cmpq %rbx, %r11
// 487 seta %sil
// 488 addl %esi, %edi
// 489 xorl %esi, %esi
// 490 cmpq %rdx, %r10
// 491 setnb %sil
// 492 xorl %r13d, %r13d
// 493 cmpq %r10, %r12
// 494 setbe %r13b
// 495 addl %r13d, %esi
// 496 xorl %r13d, %r13d
// 497 cmpq %r10, %rbp
// 498 setbe %r13b
// 499 addl %r13d, %esi
// 500 xorl %r13d, %r13d
// 501 cmpq %r10, %r11
// 502 setbe %r13b
// 503 addl %r13d, %esi
// 504 xorl %r13d, %r13d
// 505 cmpq %rbx, %r10
// 506 seta %r13b
// 507 addl %r13d, %esi
// 508 movslq %ecx, %r13
// 509 movq %rdx, (%rax,%r13,8)
// 510 movslq %r9d, %rdx
// 511 movq %r12, (%rax,%rdx,8)
// 512 movslq %r8d, %rdx
// 513 movq %rbp, (%rax,%rdx,8)
// 514 movslq %edi, %rdx
// 515 movq %r11, (%rax,%rdx,8)
// 516 movslq %esi, %rdx
// 517 movq %r10, (%rax,%rdx,8)
// 518 leal (%rcx,%r9), %edx
// 519 movl $15, %ecx
// 520 addl %r8d, %edx
// 521 addl %edi, %edx
// 522 addl %esi, %edx
// 523 subl %edx, %ecx
// 524 movslq %ecx, %rdx
// 525 movq %rbx, (%rax,%rdx,8)
// 526 jmp .L5096
// 527 .L5097:
// 528 sarq $2, %rsi
// 529 leaq (%rax,%rsi,8), %rcx
// 530 movq (%rcx), %r8
// 531 movq %rdx, (%rcx)
// 532 movq %r8, (%rax)
// 533 cmpq %rax, %r10
// 534 jbe .L5105
// 535 movq %rax, %rdx
// 536 movq %rax, %r9
// 537 movq %r10, %rcx
// 538 jmp .L5111
// 539 .p2align 4,,10
// 540 .p2align 3
// 541 .L5122:
// 542 movq (%r9), %rdi
// 543 movq %rsi, (%r9)
// 544 leaq 16(%rdx), %rsi
// 545 addq $8, %r9
// 546 movq %rdi, 8(%rdx)
// 547 movq %r11, %rdx
// 548 .L5107:
// 549 cmpq %rdx, %rcx
// 550 jbe .L5112
// 551 .L5111:
// 552 movq 8(%rdx), %rsi
// 553 leaq 8(%rdx), %r11
// 554 cmpq %r8, %rsi
// 555 jb .L5122
// 556 ja .L5120
// 557 leaq 16(%rdx), %rsi
// 558 movq %r11, %rdx
// 559 cmpq %rdx, %rcx
// 560 ja .L5111
// 561 .L5112:
// 562 movq 8(%rdx), %rdi
// 563 movq %rsi, %xmm0
// 564 movq %rcx, %rsi
// 565 movq %r10, %xmm1
// 566 subq %rdx, %rsi
// 567 leaq -8(%r9), %r10
// 568 punpcklqdq %xmm1, %xmm0
// 569 addq $2, %r15
// 570 movq %rdi, %r8
// 571 sarq $63, %rsi
// 572 subq 8(%rcx), %r8
// 573 movups %xmm0, -112(%rsp,%r15,8)
// 574 andq %r8, %rsi
// 575 subq %rsi, %rdi
// 576 movq %rdi, 8(%rdx)
// 577 addq %rsi, 8(%rcx)
// 578 movq %r10, %rcx
// 579 subq %rax, %rcx
// 580 testq %rcx, %rcx
// 581 jg .L5113
// 582 jmp .L5096
// 583 .p2align 4,,10
// 584 .p2align 3
// 585 .L5110:
// 586 subq $8, %rcx
// 587 .L5120:
// 588 movq (%rcx), %rdi
// 589 cmpq %r8, %rdi
// 590 ja .L5110
// 591 movq %rdi, 8(%rdx)
// 592 subq $8, %rcx
// 593 movq %rsi, 8(%rcx)
// 594 movq %r11, %rsi
// 595 jmp .L5107
// 596 .p2align 4,,10
// 597 .p2align 3
// 598 .L5105:
// 599 movq %rbx, %rsi
// 600 movq %rax, %rdx
// 601 movq %rax, %r9
// 602 movq %r10, %rcx
// 603 jmp .L5112
// 604 .cfi_endproc
Tomorrow night, will run the GCC and ICL executables on Windows 10, the testmachine will be 'Brutalitto' - the mini-monster with Zen 2 4800H and 64GB DDR4 - running the heaviest sort benchmark on Internet - 7+ billion QWORDS...
Add-on:
Code: [Select]
Test run: 2022-Mar-09:
Laptop "Brutalitto", AMD 'Renoir' 4800H 4.3GHz max turbo, 64GB DDR4 3200MHz:
+--------------------+---------------------------+---------------------------+---------------------------+---------------------------+----------------------------+-----------------------------+
| Performer/Keys | #1, FEW distinct | #2, MANY distinct | #3, MANYmore distinct | #4, ALL distinct | #5, ALLmore distinct | #6, ALLmax distinct |
+--------------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+--------------+--------------+--------------+
| Operating System, | Windows 10, | Windows 10, | Windows 10, | Windows 10, | Windows 10, | Windows 10, | Windows 10, | Windows 10, | Windows 10, | Windows 10, | Windows 10, | Windows 10, |
| Compiler, -O3 | Intel v15.0 | GCC 11.2.1 | Intel v15.0 | GCC 11.2.1 | Intel v15.0 | GCC 11.2.1 | Intel v15.0 | GCC 11.2.1 | Intel v15.0 | GCC 11.2.1 | Intel v15.0 | GCC 11.2.1 |
+--------------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+--------------+--------------+--------------+
| qsort | 42 seconds | 45 seconds | 242 seconds | 280 seconds | 113 seconds | 131 seconds | 319 seconds | 354 seconds | 620 seconds | 695 seconds | 1282 seconds | 1438 seconds |
| Magnetica v.13 | 22 seconds | 21 seconds | 135 seconds | 134 seconds | 60 seconds | 59 seconds | 177 seconds | 177 seconds | 350 seconds | 349 seconds | 724 seconds | 722 seconds |
| Bentley-McIlroy | 24 seconds | 24 seconds | 146 seconds | 142 seconds | 66 seconds | 64 seconds | 200 seconds | 193 seconds | 391 seconds | 376 seconds | 850 seconds | 803 seconds |
| Crumsort | 20 seconds | 19 seconds | 91 seconds | 81 seconds | 44 seconds | 38 seconds | 126 seconds | 109 seconds | 246 seconds | 211 seconds | 567 seconds | 479 seconds |
+--------------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+--------------+--------------+--------------+
| Best Time (bare | | | | | | |
| bone in-place QS): | 19s for Crumsort | 81s for Crumsort | 38s for Crumsort | 109s for Crumsort | 211s for Crumsort | 479s for Crumsort |
+--------------------+---------------------------+---------------------------+---------------------------+---------------------------+----------------------------+-----------------------------+
Code: [Select]
Speed Roster, (the base speed 1.00x is GLIBC's qsort):
Rank #1: 2943/937= 3.14x = 19+ 81+ 38+109+211+ 479= 937 seconds for Crumsort
Rank #2: 2943/1462= 2.01x = 21+134+ 59+177+349+ 722= 1462 seconds for Magnetica v.13
Rank #3: 2943/1602= 1.83x = 24+142+ 64+193+376+ 803= 1602 seconds for Bentley-McIlroy
Rank #4: 2943/2943= 1.00x = 45+280+131+354+695+1438= 2943 seconds for GLIBC's qsort
Post by: Sanmayce on March 13, 2022, 08:54:21 pm
Test run: 2022-Mar-14:
Laptop "Compressionette", Intel 'Kaby Lake' i5-7200U 3.1GHz max turbo, 36GB DDR4 2133MHz:
Code: [Select]
+--------------------+-------------------------+--------------------------+--------------------------+---------------------------+---------------------------+-----------------------------+
| Performer/Keys | #1, FEW distinct | #2, MANY distinct | #3, MANYmore distinct | #4, ALL distinct | #5, ALLmore distinct | #6, ALLmax distinct |
+--------------------+-------------------------+--------------------------+--------------------------+-------------+-------------+---------------------------+--------------+--------------+
| Operating System, | Fedora 35, GCC 11.2.1 | Fedora 35, GCC 11.2.1 | Fedora 35, GCC 11.2.1 | Fedora 35, GCC 11.2.1 | Fedora 35, GCC 11.2.1 | Fedora 35, GCC 11.2.1 |
| Compiler, -O3 | instructions; IPC | instructions; IPC | instructions; IPC | instructions; IPC | instructions; IPC | instructions; IPC |
+--------------------+-------------------------+--------------------------+--------------------------+---------------------------+---------------------------+-----------------------------+
| qsort | 385/342 seconds | 527/234 seconds | 165/55 seconds | 516/128 seconds | 999/252 seconds | N.A. |
| | 6,448,450,744,497; 2.82 | 4,921,980,445,033; 2.06 | 1,369,822,972,984; 1.94 | 3,250,596,357,540; 1.58 | 6,250,299,799,611; 1.59 | N.A. |
+--------------------+-------------------------+--------------------------+--------------------------+---------------------------+---------------------------+-----------------------------+
| Magnetica v.14 | 29/6 seconds | 198/47 seconds | 86/22 seconds | 241/66 seconds | 472/134 seconds | N.A. |
| | 171,452,642,615; 1.14 | 936,751,248,598; 1.17 | 443,844,553,192; 1.25 | 1,291,624,832,269; 1.28 | 2,503,942,225,085; 1.28 | N.A. |
+--------------------+-------------------------+--------------------------+--------------------------+---------------------------+---------------------------+-----------------------------+
| Bentley-McIlroy | 38/13 seconds | 223/47 seconds | 100/26 seconds | 304/65 seconds | 597/139 seconds | N.A. |
| | 246,587,334,436; 1.23 | 1,105,253,285,645; 1.25 | 507,140,963,617; 1.23 | 1,460,132,564,676; 1.22 | 2,848,210,143,410; 1.22 | N.A. |
+--------------------+-------------------------+--------------------------+--------------------------+---------------------------+---------------------------+-----------------------------+
| Crumsort 1.1.5.3 | 29/4 seconds | 129/4 seconds | 61/2 seconds | 173/3 seconds | 339/7 seconds | N.A. |
| | 351,332,884,902; 2.43 | 1,284,147,329,900; 2.80 | 603,065,127,518; 2.77 | 1,605,371,408,284; 2.64 | 3,102,283,088,926; 2.71 | N.A. |
+--------------------+-------------------------+--------------------------+--------------------------+---------------------------+---------------------------+-----------------------------+
Legend (The time is exactly the Sort process time, first value is for unsorted, second one is for sorted).
Post by: mdijkens on March 14, 2022, 04:45:19 am
I am regularly working with really big files (some over to 100GB) and have been optimizing 2 functions to read/process big files very fast.
I thought I'd add them here for information:
Code: QB64: [Select]
and for csv's:
Code: QB64: [Select]
- 'Print "Reading lines from "; fileName$; " ";: cpos% = Pos(0) '@@ progress info
- ff% = FreeFile
- sep& = 0
- lines& = -1
- buf$ = block 'Mid$(block, InStr(block, Chr$(10)) + 1)
- r0& = 1
- ' Process lin$
- lines& = lines& + 1
- 'Locate , cpos%, 0: Print lines&; '@@ progress info
- Next curblock&
- Close #ff%
- buf$ = ""
- 'Locate , cpos%, 0 '@@ progress info
- CSV.read& = lines&
Post by: Sanmayce on March 14, 2022, 06:12:56 am
Very interesting thread! Thank you!
I am regularly working with really big files (some over to 100GB) and have been optimizing 2 functions to read/process big files very fast.
I thought I'd add them here for information:
...
You are welcome, at the moment have no time to make some dedicated example of sorting (fixed-length or/and variable-length i.e. string (LF instead of NULL) type) tool.
Let me know what interests you most, what is the main bottleneck you encounter in your processings - the parsing, the sorting, the searching...
This thread is all about these BASIC things, but optimized, my wish is to share and thus show the power of QB64 (as in the old days when pairing QB and ASM was, resembles QB64 and C duo, now).
All these micro-projects contribute to the Masakari project, the idea being, we all to have powerful and fast text-oriented routines. If you haven't looked up the Masakari text loading (somewhat similar to yours), it allows sorting lines per size or by their offset in memory (i.e. returning to the original state), the quick usage being:
Code: QB64: [Select]
- Declare CustomType Library "qsm_linesize" ' Notice that 'CustomType' makes things work, using 'STATIC' gives errors during compilation
- 'Quicksort_QB64_v7_linesize MhandleOFF.OFFSET, MhandleLEN.OFFSET, 0, ElementsMinusOne
For example, do you have a ... nifty (for Linux, Windows) filename parser? A month ago I was tired of not having such a function, and wrote DIRWALKER (the skeleton at the moment, listing AS IT SHOULD the current folder), wanted to write a tiny tool, but the most important work is done, the rest is just gluing into your code. See the BAS+C source code in the attached archive. Wanna have the HEAVY-DUTY functionality of sorting FILESIZE, NAME, MODIFIED TIME fields/columns... for millions of files, FAST!
Post by: mdijkens on March 14, 2022, 06:45:10 am
If I need sorting I add a custom QuickSort for the specific column
Code: QB64: [Select]
- $VersionInfo:FileDescription=mdDir
- $VersionInfo:ProductName=mdDir
- $VersionInfo:InternalName=mdDir
- $VersionInfo:OriginalFilename=mdDir
- '$Let DEVTEST = 1
- Type parmType
- init
- f& = Dir(parm.fspec)
- '$include: 'DIR.BI'
- Sub init
- Print "mdDir.exe v"; getVersion$
- getParams
- ChDir "E:\_todo\qb64dev"
- Sub getParams
- 'd:\dev\qb64\*.bas /t console:only /c
- cmd$ = getCommand$(ccount%)
- Case "S"
- parm.subs = TRUE
- Case "T"
- stext(0) = "sinit"
- Case "C"
- parm.icase = TRUE
- Print " mdDIR [fspec] [/s] [/t search1 [...] [/c]]"
- Print " fspec = [d:][path\][filespec] e.g. 'd:\data\*.txt'"
- Print " /s = Include sub-directories"
- Print " /t search1 [...] = Search content for 'search1' (and '...')"
- Print " /c = Ignore case in search"
- parm.fspec = cmd$
- parm.search = parm.search + 1
- stext$(parm.search) = cmd$
- Next ccount%
- parm.fspec = "*.*"
- b% = DIR.CheckPath(p$)
- ddcount& = 0
- f& = DIR.GetFiles(p$)
- DIR.SortFiles
- ddcount& = DIR.Dcount
- subdir$ = "|"
- subdir$ = subdir$ + DIR.Dname(i&) + "|"
- Next i&
- DIR.SortFiles
- ddcount& = DIR.Dcount
- subdir$ = "|"
- subdir$ = subdir$ + DIR.Dname(i&) + "|"
- Next i&
- f& = DIR.GetFiles(p$)
- DIR.SortFiles
- sp& = 0
- Next s&
- Next i&
- f& = 0
- Dir& = f&
- a = DIR.Fattr(i&)
- Next i&
- cpos = cpos + bsize
- Next curblock
- Close #ff%
- Next s%
- content = ""
- textSearch = TRUE
- textSearch = FALSE
- Sub PrintDirs
- a = DIR.Dattr(i&)
- DIR.Dshort(i&); Attr$; DIR.Dtime(i&);
- Next i&
- _MemFree m
- getCommand$ = cmd$(n%)
- getCommand$ = ""
- Function getVersion$
DIR.BI:
Code: QB64: [Select]
- ' ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
- ' º DIR.BI v1.22 (20200815) maurits@dijkens.com º
- ' ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ
- ' º º
- ' º ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿ º
- ' º ³ ³ º
- ' º ³ FUNCTION DIR.CheckPath (dfspec$) ³ º
- ' º ³ FUNCTION DIR.GetFiles& (dfspec$) ³ º
- ' º ³ SUB DIR.SortFiles () ³ º
- ' º ³ SUB DIR.QSortFiles (leftN AS LONG, rightN AS LONG) ³ º
- ' º ³ SUB DIR.QSortDirs (leftN AS INTEGER, rightN AS INTEGER) ³ º
- ' º ³ ³ º
- ' º ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ º
- ' º º
- ' ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
- Type __DIR_FILETIME_TYPE
- Type __DIR_SYSTEMTIME_TYPE
- Type WIN32_FIND_DATAA
- ftCreationTime As __DIR_FILETIME_TYPE
- ftLastAccessTime As __DIR_FILETIME_TYPE
- ftLastWriteTime As __DIR_FILETIME_TYPE
- DIR.err:
- dfspec$ = dfspec$ + ".\*.*"
- Next pend%
- dfspec$ = dfspec$ + "*.*"
- dfspec$ = dfspec$ + "\*.*"
- pend% = dp% + 2
- dspec$ = _CWD$
- dspec$ = dspec$ + "\"
- DIR.CheckPath% = 0
- DIR.CheckPath% = 0
- Var$ = dfspec$
- DIR.Dcount = 0
- DIR.Fcount = 0
- Attribute = finddata.dwFileAttributes
- Filename$ = finddata.cFileName
- ' store date/time
- x& = FileTimeToLocalFileTime&(finddata.ftLastWriteTime, LocalTime)
- x& = FileTimeToSystemTime&(LocalTime, SysTime)
- DIR.Dcount = DIR.Dcount + 1
- DIR.Dname(DIR.Dcount) = Filename$
- DIR.Dtime(DIR.Dcount) = Var$
- DIR.Dattr(DIR.Dcount) = Attribute
- Filename$ = finddata.cAlternateFileName
- DIR.Dshort(DIR.Dcount) = Filename$
- DIR.Fcount = DIR.Fcount + 1
- DIR.Fname(DIR.Fcount) = Filename$
- DIR.Fsize(DIR.Fcount) = F&&
- DIR.Ftime(DIR.Fcount) = Var$
- DIR.Fattr(DIR.Fcount) = Attribute
- Filename$ = finddata.cAlternateFileName
- DIR.Fshort(DIR.Fcount) = Filename$
- x& = FindClose(Wfile.Handle)
- DIR.GetFiles& = DIR.Fcount
- DIR.GetFiles& = -1
- DIR.QSortDirs 1, DIR.Dcount
- DIR.QSortFiles 1, DIR.Fcount
- leftNIdx = leftN
- rightNIdx = rightN
- pivot = (leftN + rightN) / 2
- leftNIdx = leftNIdx + 1
- rightNIdx = rightNIdx - 1
- leftNIdx = leftNIdx + 1
- rightNIdx = rightNIdx - 1
- rightNIdx = rightNIdx + 1
- pivot = rightNIdx
- leftNIdx = leftNIdx - 1
- pivot = leftNIdx
- DIR.QSortFiles leftN, pivot - 1
- DIR.QSortFiles pivot + 1, rightN
- leftNIdx = leftN
- rightNIdx = rightN
- pivot = (leftN + rightN) / 2
- leftNIdx = leftNIdx + 1
- rightNIdx = rightNIdx - 1
- leftNIdx = leftNIdx + 1
- rightNIdx = rightNIdx - 1
- rightNIdx = rightNIdx + 1
- pivot = rightNIdx
- leftNIdx = leftNIdx - 1
- pivot = leftNIdx
- DIR.QSortDirs leftN, pivot - 1
- DIR.QSortDirs pivot + 1, rightN