QB64.org Forum

Active Forums => Programs => Topic started by: NOVARSEG on January 30, 2021, 12:36:13 am

Title: Create large files FAST!
Post by: NOVARSEG on January 30, 2021, 12:36:13 am
Create a 4GB file in under a minute.


Code: QB64: [Select]
  1. a = "A"
  2.  
  3. OPEN "test111" FOR BINARY AS #1
  4.  
  5. SEEK #1, 4000000000
  6.  
  7. PUT #1, , a
  8.  
  9.  
  10.  
Title: Re: Create large files FAST!
Post by: bplus on January 30, 2021, 09:46:00 am
LOL yeah but why? ;-))
Title: Re: Create large files FAST!
Post by: SpriggsySpriggs on January 30, 2021, 10:15:16 am
What is the purpose of this
Title: Re: Create large files FAST!
Post by: Pete on January 30, 2021, 03:29:35 pm
I guess some people just hate unused disk space; but I'm not one of them.

Pete
Title: Re: Create large files FAST!
Post by: bplus on January 30, 2021, 03:57:38 pm
Hey do you think that may stop MS updates?

Sorry, disk full.  ;-))
Title: Re: Create large files FAST!
Post by: Pete on January 30, 2021, 05:36:05 pm
Been there, tried that. Nice concept, but no. Apparently MS just searches out you non-G rated photos, and overwrites them with updates. Oh, unless you download those photos from an approved MS Store source. I don't know about you, but I cant' force myself to look at even one pic of Bill Gates in a Speedo, and that guy Ballmer, I don't make enough money anymore to afford that number of therapy sessions.

Actually, that's a great question. If it weren't for needed non-MS antivirus updates, it might be worth messing with. God knows all the usual methods to disable updates eventually fail to work anymore. Other than that, you just have to play Task Manager monitor, and kill those pesky Update and Update Orchestrator processes.

Pete

Title: Re: Create large files FAST!
Post by: NOVARSEG on January 30, 2021, 07:06:40 pm
Why?

Well ahem . . . let me go back to the old DOS interrupts

http://www.ctyme.com/intr/rb-2799.htm

You set the "file pointer"  (using assembler)  beyond the end of a file and then WRITE zero bytes.

Yep it is possible to write zero bytes.   Then close the file.

After the write, the file is updated with the new file size.  So what was written ? actually nothing was written.   Writing zero bytes still updates the FAT (FAT NTFS etc)

That is why it takes very little time to create a large file that way

Under FAT32 the largest file size is 2 ^32 -1 bytes

However,   QB64 (and whatever QB45 had) can not write zero bytes.

Using PUT you gotta write  1 byte to make this work.

Quote
DIM a AS STRING
a = "A"
OPEN "test111" FOR BINARY AS #1
SEEK #1, 4000000000
PUT #1, , a
CLOSE

The "A" will be the last byte of the file. Why?  because the SEEK statement moves the file pointer to the 4,000,000,000 th byte position.

I'm not sure of the limits of the SEEK statement in QB64 but I imagine that really large files could be created this way.




Title: Re: Create large files FAST!
Post by: SpriggsySpriggs on January 30, 2021, 08:21:56 pm
Yeah I guess if I want a 4 gig empty file then cool.
Title: Re: Create large files FAST!
Post by: SMcNeill on January 30, 2021, 08:24:27 pm
Note: You don’t even need a SEEK to do this.

DIM a AS STRING
a = "A"
OPEN "test111" FOR BINARY AS #1
PUT #1, 4000000000, a
CLOSE
Title: Re: Create large files FAST!
Post by: NOVARSEG on January 30, 2021, 08:52:16 pm

That's right, the second comma is for position

PUT #1, 4000000000, a

It must do the same thing as SEEK.
Title: Re: Create large files FAST!
Post by: NOVARSEG on January 30, 2021, 10:21:53 pm
File size  =  68,719,476,736

Took about 16 minutes

Quote
DIM a AS STRING
a = "A"
OPEN "test111" FOR BINARY AS #1
PUT #1, 2 ^ 36, a
CLOSE


Title: Re: Create large files FAST!
Post by: SpriggsySpriggs on January 30, 2021, 10:24:20 pm
Let's do this! Feel free to test this:

Code: QB64: [Select]
  1. a = "A"
  2. OPEN "test111" FOR BINARY AS #1
  3. SEEK #1, 1024^4
  4. PUT #1, , a

OK but don't actually run it
Title: Re: Create large files FAST!
Post by: NOVARSEG on January 30, 2021, 10:55:16 pm
1024  ^ 4 = 2 ^ 40 = about 1024GB  just over a Terabyte.    There is no telling what would happen.

Even if it worked, what would happen if an app tried to open the file?

Title: Re: Create large files FAST!
Post by: SpriggsySpriggs on January 30, 2021, 10:57:58 pm
1024 ^ 4 is a Terabyte, with Microsoft data structure.
Title: Re: Create large files FAST!
Post by: NOVARSEG on January 30, 2021, 11:04:03 pm
to do it safely need bigger than a terabyte hard drive

 my computer would take about 4 hours

A huge file with a single "A" at the end!!!
Title: Re: Create large files FAST!
Post by: SpriggsySpriggs on January 30, 2021, 11:04:34 pm
Let's find out
Title: Re: Create large files FAST!
Post by: NOVARSEG on January 30, 2021, 11:12:07 pm
Set a timer to see how long it takes. My drive is only 500GB
Title: Re: Create large files FAST!
Post by: SpriggsySpriggs on January 30, 2021, 11:12:44 pm
Then do it as (1024 ^ 4)/2
Title: Re: Create large files FAST!
Post by: NOVARSEG on January 30, 2021, 11:14:43 pm
I'm actually less than 500GB because of other files.

To make it worthwhile, the test should be for the terabyte file. If  wait 2 hours, might as well wait 4 hours
Title: Re: Create large files FAST!
Post by: SpriggsySpriggs on January 30, 2021, 11:17:47 pm
Let me know how it goes
Title: Re: Create large files FAST!
Post by: NOVARSEG on January 30, 2021, 11:20:54 pm
My comp can't handle it. If anyone wants to try this, then you might be the first to do so.

Be careful the code might crash your computer.
Title: Re: Create large files FAST!
Post by: Pete on January 30, 2021, 11:21:57 pm
Or, if you want to create a whole lot of empty space in an operating system, consider building a politician, instead.

Pete
Title: Re: Create large files FAST!
Post by: NOVARSEG on January 30, 2021, 11:25:43 pm
Maybe, but a single  "A" at the end of a terabyte file is an amazing thing.
Title: Re: Create large files FAST!
Post by: Pete on January 30, 2021, 11:35:51 pm
So why not just go the extra steps? Add a dash and a hole, and viola, ya got yourself a politician.
Title: Re: Create large files FAST!
Post by: NOVARSEG on January 30, 2021, 11:59:40 pm
Steve has successfully created the world's first terabyte file in 22 minutes.

https://www.qb64.org/forum/index.php?topic=3579.0
Title: Re: Create large files FAST!
Post by: luke on January 31, 2021, 04:17:53 am
Meanwhile on Linux:
Code: [Select]
# Create file with size 1TiB
~$ dd if=/dev/zero of=testfile bs=1TiB seek=1 count=0 status=none
# Append 'A' to the end of the file
~$ echo A >> testfile
# Confirm size: 1.1TiB
~$ ls -lh testfile
-rw-r--r-- 1 luke luke 1.1T Jan 31 20:13 testfile
# Print last 2 bytes of file
~$ tail -c 2 testfile
A
And it all ran in less time than it takes to blink.

https://en.wikipedia.org/wiki/Sparse_file is a little bit of magic.
Title: Re: Create large files FAST!
Post by: Petr on January 31, 2021, 04:41:58 am
The space on harddrive was removed immediately, but the program remained running and the hard drive was running at full speed :), wrote this new file at an average speed of 130 megabits per second.

Title: Re: Create large files FAST!
Post by: SMcNeill on January 31, 2021, 05:09:52 am
Steve has successfully created the world's first terabyte file in 22 minutes.

https://www.qb64.org/forum/index.php?topic=3579.0

First terabyte file?  What?  Are you living in the 90s??

I've got databases of stuff archived now that are sitting around 4.2TB of space on my drive, and I'm just a home storage junkie.  I imagine that there's businesses and such out there somewhere that have petabytes of stuff archived.  We passed the realm of 20MB hard drives over three decades ago.  The world has moved on since QBASIC -- it's a QB64 existence now!

If all you have is 500MB of storage, you might want to go out and get something like: https://www.amazon.com/Seagate-Portable-External-Hard-Drive/dp/B088S5HDTV

10TB drive for between $150 and $200, and if you catch them on sale (or refurbished), you can get them for about half that price.  I used to have 5 hard drives with between 2 and 4TB of storage on them, but I was running out of room and upgraded recently to running 5 external 16TB drives.  I figure 80TB should handle my needs for the next several years, and by then, they'll probably have much larger systems that I can invest in for a reasonable set of prices. 

Give the world another 20 years, and TBs of storage is going to be laughed at, much like GBs of it is now.  :P
Title: Re: Create large files FAST!
Post by: Pete on January 31, 2021, 11:32:46 am
One real life example, that guy Jared, from Subway. He had a petabyte hard drive, completely filled by a peta-file.

So here's a non-funny take on it. What do you guys suppose apps like CCleaner use when they perform a disk cleaning? In other words, does anyone know if the system actually overwrites the data, still on the hard drive, with a simple program like this one it it were to also calculate the disc free space, and use that as the file size of the empty file. I would think that might be possible, so one would not have to write characters to the entire disk. I would be assuming the blank spaces would have to somehow overwrite the existing non-indexed data. Does it rally work that way? No biggie, not something I need, but this just got me curious as to how people who make these cleaning apps approach the situation.

Pete
Title: Re: Create large files FAST!
Post by: NOVARSEG on February 01, 2021, 03:08:33 am
@Pete

Only a single byte is written to the end of the file, all the other bytes remain the same.