QB64.org Forum
Active Forums => QB64 Discussion => Topic started by: MLambert on March 10, 2019, 10:36:33 am
-
Hi,
I do a lot of string manipulation and process volumes of data using QB64 and MySQL.
Can anyone suggest a faster non-interpretive language to use than QB64? I need to reduce the processing times.
I am processing on an I9 CPU 4Ghz using 20 cores and SSD drives with threading.
There is basically just one program that performs massive data analysis and calculations.
I believe that I have normalised my data and optimised the table and database structures.
I just want to see if there is a faster programming language that will execute faster than QB64.
Regards,
Mike
-
Speed? for this, you must choose a programming language which is more close to hardware interference. C and C++ can be chosen (though assembly is much closer to hardware than these). C/C++ can provide you better result.
Have you tried using $CHECKING:OFF to your code?
PS : It is amazing to know that you have i9 with 20 cores! Are you also a gamer? Can I know your graphic card manufacturer and VRAM?
-
Just take the time to learn machine language. I've been working on that bit by bit over the last 40 years. I'm up to the "o" in Hello World. No, the first "o" but thanks for your optimism. Hey, I had a lot going on in 1983...
Assembly is doable, but man it's a headache. QB64 already makes C/C++ routines. My bet is they are poorly optimized, so they will probably not run as fast as an optimized C/C++ program. I've heard FreeBASIC people claim FB is faster, but I'd rather shoot myself in the foot and hop off a cliff than switch to FB... or listen to FB people :P Anyway, IF FB or any other language translated directly to a well optimized sequence in machine code, that would be as fast as you could get, at least to my understanding.
Ashish has good advice about adding the meta command $CHECKING:OFF to the top of your code. Just make sure it is completely debugged. Why? Because that meta command disables the error checking function in QB64, which means less time wasted on checking for errors, which will not occur.
People who use Python and want to speed it up, it's a higher level language and prone to be a bit sluggish as such, use C-extensions in to optimize hot spots for improved performance. Think of it as their little blue pill...
Also, if you can, define all your variables as integers. Integers get processed faster, because they are allocated less memory. DEFINT h-j at the top of your code would make any variables that start with h, i, or j be assigned as integers. No need for % sign after the variables in the code, either.
Well, try the $CHECKING:OFF recommendation and I hope it's fast enough for whatever you are doing, and if it isn't about a project, and just a curiosity, that's fine, too.
Pete
-
Since no one else mentioned it, if you're going to be using QB64 and you're running it on Windows, get the latest 64-bit development build. It's quite a lot faster than the previous default 32-bit versions, it turns out.
-
Hey Bert. What have you used the 64-bit version on that you know for a fact it is faster? I downloaded it and I've used it for the past couple of months, but my hunch is the stuff I'm into isn't very speed sensitive.
Pete
-
Hi All,
Many many thks for your helpful replies.
I do speak assembler .. but I don't have the time to re-write in assembler .. may have to.
I also speak machine language .. but would need to learn the new CPUs. Used to punch this by hand into cards. Had to learn Holerith code.
$CHECKING:OFF - will give this a go as well as integers.
Ashish .. no I am not a gamer .. never have been. I have been paid by the hour to work on computers since 1969 so whenever I sit at a computer I prefer to earn $s. By the way it's a Radeon HD 7620 2GB .. all it displays is text .
I forgot to mention , I am running windows 7 64bit. Is windows 10 faster ??
I did consider python but it is interpretive and needs to run within its own environment.
Will change variable to integers .. although I need to track accesses and this will blow the limits on integers... so not all variables to integers.
Again,
Thank you all very much.
Regards,
Mike
P.S. My shout.
-
Hey Bert. What have you used the 64-bit version on that you know for a fact it is faster? I downloaded it and I've used it for the past couple of months, but my hunch is the stuff I'm into isn't very speed sensitive.
Hey Pete. I tested the speed using a program [banned user] posted, where the speed advantage of the 64-bit QB64 was very significant (something on the order of 10+ seconds execution time on the 32-bit reduced to 0.3 seconds with the 64-bit QB64), and I used a number-crunching test I wrote, where the speed advantage was not quite 2X. I ran these tests on a Core i7 and a Core i5. The Core i7 was, predictably, always faster than the Core i5, but the improvement of execution times was significant with either processor.
On the other hand, the speed test Steve wrote some time ago, the one with the graphics, did not seem to improve with the 64-bit QB64. I think that test had more to do with memory speed. The execution times were virtually identical, 32-bit or 64-bit QB64.
-
Thanks Bert. Well I'm going to stick with the 64-bit version. It has, as the 32-bit one, been working like a charm. There are times I think it compiles a little slower though. The speed difference with QB64, in general, over QuickBASIC is astounding. Considering QuickBASIC made stand alone executable programs, which did run faster than the interrupted QBASIC files, doesn't make a lot of sense to me. I know all of these exes are linked into machine code, so I guess it's the baggage from the linking libraries and how the code is or is not optimized, or something like that, that makes one run faster than the other.
Pete
-
I am processing on an I9 CPU 4Ghz using 20 cores and SSD drives with threading.
Since QB64 doesn't support multithreading (and probably will not anytime soon), you're missing out on a theoretical 20x performance gain. But this would involve you to write efficient multithreaded code. I have written multithreaded code in C using pthreads, openCL, and openMP -- that and C++, if you prefer, could be your options.
If you want a QB-like option, freebasic is your primary option. Since FB supports function pointers, you should be able to use pthreads and write multithreaded code but I have never done it myself personally. In any other circumstance, freebasic will outperform QB64 binaries most of the time but this will depend on the code you write.
-
If the tasks are separable, you could try launching process instances that work on separate segments and once all segments complete, merge the results. Maybe using Shell _don'twait to launch each instance. I'm considering using this approach for multiple gigabyte fixed-length records to divide and conquer file processing. WinDoze has a 2 GB limit on array memory per application while Linux does not., so I have to find workarounds to avoid lots of waiting. I'm using a core i7, with 16 GB Optane drive buffer. Sure is faster than my old HP. The performance is hit was minimal and kept 2:/4 cores busy in this instance. I launched 2 of the same executable, one with a slightly different name. Using _integer64 data types, 1GB/process. Instead of 2 consecutive waits of 16s, you get one wait of 22s, saving 31%in time. Maybe this approach will save enough time to compensate for the slower parts. This of course was the result of FlashSort. My approach uses a very primitive file-based semaphore to signal completion of the second executable running the very same code. The second program creates a file to signal it's done. The first checks for its presence after it has completed and introduces a small delay to kill the semaphore file. Not too tricky for two instances and it saves you about (30-40)% versus sequential execution. Not bad for a simple change.
-
Since QB64 doesn't support multithreading.....
[/quote]
Well awhile ago we had a bug that I found, if you used "end" instead of "system" or let the program self terminate.
All processors would go 100% waiting for you to hit any key to exit.
Got the cpu to go full blast doing nothing. Multi-threaded. In it's natural state QB64 does not multi-thread.
-
Since QB64 doesn't support multithreading.....
Well awhile ago we had a bug that I found, if you used "end" instead of "system" or let the program self terminate.
All processors would go 100% waiting for you to hit any key to exit.
Got the cpu to go full blast doing nothing. Multi-threaded. In it's natural state QB64 does not multi-thread.
I just ran a test and do not notice any difference with 11-22-2018 v1.2 development build. Had this been fixed?
-
@MLambert
in the folder qb64\internal\c in the file makeline_win.txt, add -O2 for optimization
-
Others might laugh at this, but modern versions of Fortran (believe it or not!) are actually quite close to QB45 code. To those of us brought up on fixed-format Fortran (Fortran IV, Fortran 66, or pure Fortran 77), this may seem strange.
But take a look at Fortran 90, Fortran 95, or even Fortran 2003, and you can see a distinct evolution towards the sort of syntax that we are used to as Basic programmers.
Malcolm
-
That probably explains why Dean, QBguy, likes modern FORTRAN. He's a pretty amazing coder, so I always wondered why he'd like that language, as the only intro I had to it was punch cards from the 1970's. If I only had that language experience, it would have turned me off on coding forever.
Pete
-
That probably explains why Dean, QBguy, likes modern FORTRAN. He's a pretty amazing coder, so I always wondered why he'd like that language, as the only intro I had to it was punch cards from the 1970's. If I only had that language experience, it would have turned me off on coding forever.
Pete
Take a look at this attachment, sourced off the Internet, of what I believe to be Microsoft Fortran from the mid-1990s. You can see the evolution in progress -- it's nothing like the punched cards I used, either!
Obviously, more modern versions have extended this similarity even further. But with minimal re-work, porting of simple programs is very possible.
Malcolm
-
It didn't work. I printed out your PNG, punched out the "0" s in the code, and ran it through the trash compactor. Sadly, nothing printed. :(
Pete
-
can someone post a link for the newer fortrans 90, 95 or 2003?
-
can someone post a link for the newer fortrans 90, 95 or 2003?
Gee, just Google them. Thousands and thousands and thousands of pages. Search terms "Silverfrost Fortran", "Intel Fortran", "The Fortran Company", and "Lahey Fortran" are where I'd start.
Plenty of good books on Amazon/eBay at about the $5 mark.
And very a good *free* compiler at gfortran -- the GNU Fortran project. My advice: avoid the gcc Fortran compiler, which is no longer maintained.
Malcolm
-
there's g95 http://www.g95.org/downloads.shtml#Win but it's rather old, I would find a mingw distribution which includes fortran, perhaps you might find something useful here https://chocolatey.org/packages?q=development
-
Thank you all for some great ideas.
Jack ... If I run the command in the Txt file with the optimise parameter .. will this produce a new QB64.exe ?
_Vince - Could you pls give me a couple of verbal lines .. not code .. of an example of multithreading ?
Again ,
Thank you everyone
Mike
-
Thank you all for some great ideas.
Jack ... If I run the command in the Txt file with the optimise parameter .. will this produce a new QB64.exe ?
Again ,
Thank you everyone
Mike
as far as I know, it only affects the programs that you compile with QB64
-
Hi Jack,
What I meant was will it build a new QB64.exe file ...
Thks,
Mike
-
not as far as I know.
-
Hi Jack,
What will it do then ?
Mike
-
adding -O2 will cause your program compiled with QB64 to be optimized, sometimes the resulting exe will run significantly faster.
-
Sorry to go around in circles ... where do I use the -O2 parameter ??
-
If you compile QB64.bas with -O2, you’ll create an optimized version of QB64(2).EXE. (Or some such variant upon the name.)
-
@MLambert
in the folder qb64\internal\c in the file makeline_win.txt, add -O2 for optimization
QB64 translates your basic program to C++ and then invokes the C++ compiler to compile that code with the command options in makeline_win.txt
-
Thks Jack.
c_compiler\bin\g++ -s -Wfatal-errors -w -Wall qbx.cpp -lws2_32 -lwinspool parts\core\os\win\src.a -lopengl32 -lglu32 -lwinmm -lgdi32 -mwindows -static-libgcc -static-libstdc++ -D GLEW_STATIC -D FREEGLUT_STATIC -lksguid -lole32 -lwinmm -ldxguid -o ..\..\
Pls show me where I place the -O2 parameter ..
Thks,
Mike
-
c_compiler\bin\g++ -O2 -s -Wfatal-errors -w -Wall qbx.cpp -lws2_32 -lwinspool parts\core\os\win\src.a -lopengl32 -lglu32 -lwinmm -lgdi32 -mwindows -static-libgcc -static-libstdc++ -D GLEW_STATIC -D FREEGLUT_STATIC -lksguid -lole32 -lwinmm -ldxguid -o ..\..\
-
Thank you Jack
-
Thank you ... I will investigate this.
Mike
-
If you have 20 cores to work with, your bottleneck will be using your CPU.
If you already have single-core code worked out in QB64, I'd suggest you try to find a way to parallellise it, so it can be run with a command line parameter, then run 30-40 instances at once to max out your CPU load to full. If you're processing directories, that's a simple way to pass the parameter to your QB64 app. You can use the command line tool xargs to specify how many of the directories to pass to your QB64 app at once, it's handy for quick parallellisation tasks.
What is your code actually doing? Something that can be broken down into a bunch of pieces, or does later code rely on earlier code?
I'm not sure if QB64 supports calling an app in the background, but it should be doable by adding a "&" after the command to place it as a background job on GNU+Linux, possibly doing it as ( mycommand & ) to force it into a subshell. If you're on Windows, I think the "START" command can launch an app in the background, so it detaches from your app. Then you just need your app to monitor for the sub-processes to finish. An easy way to do this is to use filenames like SUB_00001.TMP while it is processing a data set, and when it completes rename it to SUB_00001.TXT or similar. Then your app just waits for .TXT files to appear, then does something with the results.
-
You can use SHELL _DON'TWAIT to launch multiple simultaneous instances. Been there.
-
You can use SHELL _DON'TWAIT to launch multiple simultaneous instances. Been there.
Oh, nice, QB64 added that by itself. Hopefully that command is fully cross-platform compatible.
-
Thank you for the input. I have started to run parallel processes and this definitely does work faster.
Mike
-
What sort of time decrease/performance gain are you seeing?
Just to be clear, to max out a 20 core system you will need significantly more than 20 concurrent tasks. I recommend starting at twice as many tasks as cores. This is because each task will be doing things that make it wait (such as reading disk, waiting for database, reading directory entries, loading DLLs, etc.), which will reduce CPU usage below 100% of 1 core per task. When paralellising tasks on my 4 core system, I usually needed 6-12 simultaneous tasks to maintain 100% usage of all cores. Don't be afraid to try 100 or more concurrent tasks, just track the total processing time to see if it improves or slows down when adding tasks.
-
Hi Raven,
Thks for the input. I have broken the program into 41 different processes. Initially it took 16 hours to process 100,000,000 + records.
Introducing and extra 20 processes reduced it to 7 hours and this is bearable for the time being. I am converting the code to use MySQL and re-writing parts to change the methodology from a 'reload' every time I add transactions to an 'update' situation.
On preliminary testing this has reduced it down to less than 60 seconds. It is not the introduction of MySQL but the concept of update and not reload or rebuild the database.
I will have to reduce this even further to be realtime.
This thread in the forum did help me.
Thks to all.
Mike