RAM Disk for temp/work files

[blametroi]

Has any thought been given to allowing the various temp files for editing and compiling? I normally have one up and available. That may be a bit retro, but this is QB :)

[FellippeHeitor]

Hi. I don't think I understand your question.

[doppler]

A true Ramdisk, usable as a disk drive. https://sourceforge.net/projects/imdisk-toolkit/
Because it uses drivers that Microsoft not vetted it will complain.  On install.  That is normal.
Use the correct version 32/64, this is very important.

Since QB64 does not designate a temp location, there is no improvement in compile.  If you want a work location for temp files in your program have at it.  Want to use the Ramdisk for trashing files in and out, not trashing a hard drive.  Best use!

BUT BE VERY WARNED.  Any Ramdisk created will reduce available system ram.  Got to get it from somewhere.
I don't recommend on systems less than 16GB.

[blametroi]

Right now it looks to me that QB64 creates work files under internal/temp. I know that some of those files need to be persistent (bookmarks.bin) but others probably don't need to reside on permanent storage.

I don't know if moving the various C++ fragment files to ramdisk would improve performance, but figured I'd bring up the idea.

[FellippeHeitor]

The files in internal/temp (and internal/temp2, internal/temp3 etc) are generated by the IDE while you're editing your program. If the IDE is closed, they can all be deleted, but they will be generated again, at the same locations, with every program you work with. There's no way to instruct QB64 to save the temp files somewhere else right now, as it relies on the folder structure to work.

[Richard]

Adding further to what has already been said...


"Temp" files, like the program one is editing - I gather every key stroke while in the IDE generates an autosave of the temp file of the whole program - please advise if I am wrong saying this, as this fact is extremely important...

So editing a program constantly "trashes the hard drive" only when one is "touching the keyboard/mouse/etc". Suppose you were editing a small program, for example 1 KB in size, I do not know the effect on mechanical hard drives BUT on Solid State Drives (NVME NAND) the technology here is affected by the design phenomenon "BLOCK WRITE ERASE" - whereby one simply cannot rewrite a single byte - the saving of a file requires a whole block to be erased first and the whole file then written to that freshly erased block. Here's the catch - on SSD NVME NAND - no manufacturer states the block erase size - some reports by users on the internet mention, from their own test programs, that this value ranges from 256 KBytes to 16 MBytes. After, as a side note, correspondence with INTEL confirmed that the block erase  write is in the MByte range.

Summary - if I write a file of say only one byte to my SSD NVME NAND - up to 16 MByte (depends on manufacturer) is erased to accomondate that one byte file - this is greatly affecting the ENDURANCE (write endurance is finite and often stated as TBW (tera bytes written) by the manufacturer). Imagine what 1000 edits (i.e. by 1000 keystrokes/mouse movements/etc while in IDE for that session) would affect the endurance on SSD.

On the other hand, RAM memory whether it is dynamic or static, uses different technology and it is generally accepted that RAM has infinite write endurance - so it is immune to "trashes" - however RAM is not persistent (switch off the power and you lose the data).

Conclusion - to extend life (c/f endurance write) use RAM for temp files not SSD. Some RAM can easily be allocated to a RAMdrive (IMdisk etc) and the RAMdrive itself can be configured as persistent (which relies on SSD/HDD for the back up) or non-persistent.

I am considering loading the whole QB64 folder onto non-persistent RAMdrive for maximum write endurance (but I must manually save the edited file because if the computer restarts or crashes - i lose latest work).

[blametroi]

I had forgotten @Richard's concern. I only have my OS and a few games on my system drive, an SSD, and keep everything else on an actual spinning hard drive.

As a quick test, I removed all the TEMP directories under INTERNAL and created links (via the mklink command) to my RAM drive.

No sign of problems but no obvious performance benefit at compile time. So, fwiw, it looks as this might be a possible workaround. I'm going to test with some larger programs but based on @FellippeHeitor's replies I don't expect any problems.

[doppler]

I had forgotten @Richard's concern

The boot drive of windows gets a lot of trashing about.  Mainly the temp directory, pagefile and swapfile.  All three of these high activity locations can be created or symbolically pointed to another drive like a hard disk spinning drive.  It's not something the average person can do.  Let alone think about it.  But for the health of the SSD must be done.
 

[Richard]

@blametroi

It gets interesting to analyze performance increases.

As a general "rule of thumb" - going from spinning HDD to SSD (if the SSD is a SATA NVME NAND) - one can often see a 10x speed improvement (the computer company HP in their promotions say their engineers get a 17x improvement).

As I see it, your operating system (together with QB64 folder) is on a SSD (I guess it is SATA NVME NAND).

If one has a PCIe NVME SSD (which uses a much much faster "BUS" than SATA which is by design to maintain total compatibility to old spinning hard drives AND why bother having a SATAv123 -just go for something totally different i.e. PCIe (now version 5 very expensive is out even much faster) - one could expect speed improvements over a SSD (NVME NAND).

Now for the catch...

Suppose you were just wanting to copy a 4 GigaByte file (I choose such a large (sort-of) file purely for timing accuracy concerns.  Using the high precision timer routine mentioned a while back on this forum - where timing resolution (note I DID NOT SAY accuracy) approaches microseconds - the following are timings on my computer all done on the same day etc


' 4gb file -> T:\     (unit = seconds, rounded up to milliseconds)
'
'1  PCI256g      10.730 11.035 10.300 10.071  9.494 9.642 9.726
'2  RamDrive     11.974  9.323  9.468  9.391                   
'3  SATAhd       74.408 74.467  9.671 10.348                   
'4  sdhc         51.884                                         
'5  usbNTFS     135.160  9.900                                 
'6  usbEXFAT     42.833 53.888                                 
'7  SATAred      13.640  9.554                                 
'8  PCI2tb      ~ 1.75
'9  eHDD        125.407                                         
'10 PCI2tb-usb3  11.643     
'11 PCI2tb-usb2 124.882           

where T:\ was a special RAMdrive (configured from my 32GByte RAM)

You will notice I said "not accuracy" above, the mutiple runs above show a timing spread for the same task.

My OS etc is on a PCIe NVME 256 GByte .M.2 drive
The RAMdrive (NOT T:\) is also configured from my 32GByte RAM
The SATAhd was a old spinning HDD about 240 Gbyte (used many many times)
The sdhc was a Samsung 512 Gbyte SDHC card
The usb sticks (different capacities) formatted as indicated
The SATAred is a SSD NVME NAND (at the time essentially unused)
THE PCI2tb was the 2 TByte   M.2 PCIe NVME NAND but was temporarily installed (OS now on HDD)
The eHDD was the exact same SATAhd above but installed into a USB adapter -  and using a usb3 port
The PCI2tb was a PCIe NVME 2 TByte installed into a usb adapter - and using the respective usb ports




The above shows some instances where I "forgot" to clear the Windows memory cache before reruns - Windows, it seems, remembers the latest 4 GByte file until I wipe any knowledge of windows accessing that file.

A side note, is that for the above trials I needed to have my Windows 10 x64 OS on a mechanical HDD - simply because I have ONLY one PCIe slot (the proper location for my OS). Windows 10, in the meantime, has "yet again" changed the rules and so now I CANNOT BOOT OFF a HDD - my computer now can only work with the OS installed on the one and only PCIe slot.


I "cheated" by using a fair sized file (4 GByte) - in reality (say QB64 programs) files are much smaller and many different things occur. For example, if you broke up the 4 Gbyte file into a memory equivalent number of smaller files - you would anticipate that because of "overheads" to process each file that it would take longer.

Just for fun, I tried running the trials but using 1 Gbytes into exactly 4 KBytes (4096 bytes) files - i.e. 262,144 of them. I gave up when doing SATAred to T:\   -  it was taking about a whole day to copy. Effectively now the "random" file effects (and other things) now kick in.

So there comes a point when "smallish" files do not notice the difference between various memory media.





I am interested in the mklink commands (never heard of it before) - how to use, where to get etc would be appreciated.

[Richard]

@doppler

Any information on how to configure  pagefile and swapfile to use RAMdrive and how big these can get?

I am only just starting to try to tell windows that the TEMP folder is going to be on the RAM drive (A:\) but when trying a simple QB64 program just to "create and copy Windows Temp folder  on A:\" - for some reason it will not. My A:\ drive is purposely run in non-persistent mode (hence the create folder etc).

At present (subject to alteration) my 32 GByte RAM is split to have a 10 GByte RAMdrive (A:\)

[blametroi]

@Richard, MKLINK is a little known bit of Windows. For a command reference I usually go to  ss64 for information. https://ss64.com/nt/mklink.html

MD H:\QB64\TEMP
MD H:\QB64\TEMP2
...
MKLINK /D E:\QB64\internal\temp H:\QB64\TEMP
MKLINK /D E:\QB64\internal\temp2 H:\QB64\TEMP2
...

I discovered MKLINK when I started storing all of my config files for linux and windows in a directory and put the repo up on github. So, one .vimrc to rule them all :)

For temp, I've changed the environment variables (I find two in my system), for TEMP and TMP. I point both at my physical drive. I had each of these in both the system and user level environment variables, but I don't know off hand if that's normal or an artifact of my messing around in the past.

I've left the swap/page stuff alone. I've got a 16gb system and I only see paging if I've done something really stupid :)

[SMcNeill]

One thing to be careful of with linked directories is use of ..\ directories.

For example, lets look at: A:/foo/foo1/foo2

Normally, if we start at foo2, a CD ..\ will move us up to foo1, then up to foo, then up to A:

BUT, what if we link foo1 to: B:/whatever/we/want/to

We’d start at foo2, move up to foo1, then move up to B:/whatever/we/want


Which leads to a simple warning: Be careful with using linked paths, especially when using ../ to travel up file paths.  You may find yourself ending up in rather unexpected places (especially on a school or work network), where you won’t have any read or write permissions.

[Richard]

@SMcNeill

Just checking - are the slashes wrong?

A:\foo\foo1\foo2  more correct?

[SMcNeill]

@SMcNeill

Just checking - are the slashes wrong?

A:\foo\foo1\foo2  more correct?

Probably, as it’s past midnight and I haven’t been to bed yet, even though my day started at 2AM this morning.  ;)

The concept is valid, even ifthe paths got a little tired and slanted in the wrong direction. 

[zaadstra]

I see several ways in the topic to look at the question ;-)

I am using the same ImDisk ramdisk, for speed and to reduce SSD wear.  In programs that write tons of workfiles I check if the ramdisk is there, and otherwise I pollute the program dir:

Code: QB64: [Select]

1. IF _DIREXISTS("R:\TEMP") THEN
2.    IF NOT _DIREXISTS("R:\Double") THEN MKDIR "R:\Double"
3.    tempdir$ = "R:\Double\"
4. ELSE
5.    tempdir$ = _STARTDIR$ + "\" ' where we are running
6. END IF

I always have ImDisk creating a \TEMP folder when starting.