DIM a (Max Index)

[soniaa]

Hi, sorry for my english, ...I'm ITA,
I have a problem,
I have 8.000.000.000 of items,
the value of any ia is max 100
I want create:
Dim q(8000000000) as _byte
BUT THES Produce error
the max index accept is 4292870137
Please help me to solve thes problem,
ciao da Soniaa.

[soniaa]

I use win 7 64 bit
qb64  (64bit version)
my ram is 6GB

[bplus]

Hi soniaa,

Welcome to the forum!

I doubt this will be very satisfactory, but in the old days we extended memory by using the hard disk for storing stuff until ready to process that part... divide and conquer.

OK, OK, hopefully someone will show up and know more about memory management than I. :)

[TerryRitchie]

That number equals 4GB so it's more than likely the upper limit for arrays in QB64. I would suggest breaking the array into two pieces if possible, two each at 4,000,000,000 indexes each.

Edit: I just saw you have 6GB in your computer for RAM, so this task will require the hard drive to come into play as Bplus suggested.

[bplus]

I have 8GB but no error and no answer and no hang for this:

Code: QB64: [Select]

1. DIM a(4000000000) AS _BYTE
2. DIM b(4000000000) AS _BYTE
3. a(4000000000) = 1
4. b(4000000000) = 1
5. PRINT a(4000000000) + b(4000000000)
6.  

[codeguy]

I think all QB64 programs have a max heap size of 2 GB or WinDoze enforces this limit. It is possible to use an _INTEGER64 to create subscripts that extend well beyond the range of 32-bit LongInt. Once past 4 billion or so with _UNSIGNED LONG, it wraps back around 0, while the standard LONG becomes negative and has a limit of about 2 billion before doing so. If you have large indexes, I suggest _INTEGER64 as the index data type.

'* && is _INTEGER64 suffix
SuperBigIndex&& = 2 ^ 37
DIM ac(SuperBigIndex&& TO SuperBigIndex&& + 1048575) AS _INTEGER64

I tried

SuperBigIndex&& = 2 ^ 37
DIM ac(0 TO SuperBigIndex&& + 1048575) AS _INTEGER64

and was not surprised to see an out of memory error.

[RhoSigma]

I want create:
Dim q(8000000000) as _byte
.
.
.
my ram is 6GB

Should be obvious that it is impossible, even if QB64 would not impose any limits on the array index. You want 8 billion bytes but you only have 6 billion bytes installed in your system.

[codeguy]

You'll most definitely need enough disk space to handle what RAM and QB64 cannot. I would make it a random access file and either segment access to 2 GB using long or use _INTEGER64 variables to specify positions for random access reads. Rho is absolutely correct. There is no way to get 8GB to fit into 6GB and still be usable.

[soniaa]

Thanks for all menbers for yours response,

I think the real motive of my problem is writed in the response of TerryRitchie,
the limit is of qb64 compiler,
not is the my RAM...
...I have do a same test whith only 2GB of RAM and the result is identical:
DIM a(1 TO 4292870136) AS _BYTE  ---- go fine
DIM a(1 TO 4292870140) AS _BYTE  ---- not  go (error chrash when start the .exe)

[soniaa]

I have a goog idea;
Many of my data is value 0
i can no create array for this,
in thes mode the array I want to DIM are approximately 3.000.000.000

Now BIG Problem;
I want put my data into the arrays one by one,
example:
visitornumber1000000000 = 5
visitornumber2000000000 = 7
I do;
DIM a(1000000000 to 1000000000) as _BYTE
a(1000000000) = 5

when I do;
DIM a(2000000000 to 2000000000) as _byte
a(2000000000) = 7
I receive error !!!

I have test this;
REM $DYNAMIC 
DIM a(1000000000 to 1000000000) as _BYTE
a(1000000000) = 5
REDIM _PRESERVE a(2000000000 to 2000000000)
But receive error

HELLPP MEE...
Thanks

[FellippeHeitor]

Your best bet is to keep your data in a file on disk instead of loading it all into memory at once. That'll solve your problem in an instant.

[bplus]

It works but is very slow to setup, after setup speedier access:

Code: QB64: [Select]

1. DIM b1 AS _BYTE
2. DIM b2 AS _BYTE
3.  
4. OPEN "c.dat" FOR BINARY AS #1
5.  
6. 'store some numbers in a really big dat file
7. 'FOR i&& = 5000000000 TO 5000000010
8. '    b1 = INT(RND * 120)
9. '    PUT #1, i&&, b1
10. '    PRINT b1; " ";
11. 'NEXT
12.  
13. GET #1, 5000000000, b1
14. GET #1, 5000000005, b2
15. PRINT b1, b2, b1 + b2
16. CLOSE #1
17.  
18.  

[Qwerkey]

I agree with bplus (as ever) that writing to disk is the only way and it is, unfortunately, very slow:

Code: QB64: [Select]

1. Start! = TIMER
2. RANDOMIZE (TIMER)
3. OPEN "temp.rnd" FOR RANDOM AS #1 LEN = 1
4. FIELD 1, 1 AS a$
5. FOR b~& = 1 TO 30000
6.     c%% = INT(RND * 100)
7.     LSET a$ = CHR$(c%%)
8.     PUT #1, b~&
9. NEXT b~&
10. CLOSE #1
11. PRINT TIMER - Start!

With just 30000 entries, my Core i5 computer takes 1s to write that many entries.  We can only wish you luck, Sonia, with 8,000,000,000 entries.  You must be wanting an entry for every person on the planet.  [Here in the UK we use commas instead of periods for the thousands separator].

[SMcNeill]

If the issue is just with the index, and not an actual memory issue, can you read and work with the data as a different variable type?

Instead of:

Dim q(8000000000) as _byte

Try:

Dim q(2000000000) as LONG

Once in memory, it's simple enough to break those longs down to individual bytes, if necessary. 

[SMcNeill]

I agree with bplus (as ever) that writing to disk is the only way and it is, unfortunately, very slow:

Code: QB64: [Select]

1. Start! = TIMER
2. RANDOMIZE (TIMER)
3. OPEN "temp.rnd" FOR RANDOM AS #1 LEN = 1
4. FIELD 1, 1 AS a$
5. FOR b~& = 1 TO 30000
6.     c%% = INT(RND * 100)
7.     LSET a$ = CHR$(c%%)
8.     PUT #1, b~&
9. NEXT b~&
10. CLOSE #1
11. PRINT TIMER - Start!

With just 30000 entries, my Core i5 computer takes 1s to write that many entries.  We can only wish you luck, Sonia, with 8,000,000,000 entries.  You must be wanting an entry for every person on the planet.  [Here in the UK we use commas instead of periods for the thousands separator].

If your disk access speeds are slow, and are a bottleneck in your program, reduce the number of times you have to access the disk as much as possible. 

Try this:

Code: QB64: [Select]

1. Start! = TIMER
2. RANDOMIZE (TIMER)
3. OPEN "temp.rnd" FOR BINARY AS #1
4.  
5. t$ = ""
6. FOR b~& = 1 TO 30000
7.     c%% = INT(RND * 100)
8.     LSET a$ = CHR$(c%%)
9.     t$ = t$ + LTRIM$(RTRIM$(STR$(b~&)))
10.     IF LEN(t$) > 1000 THEN PUT #1, , t$: t$ = ""
11. NEXT b~&
12. PUT #1, , t$
13. CLOSE #1
14. PRINT TIMER - Start!
15.  

Instead of writing every value to the disk, we save the results in a temp string (t$) and dump them to the file every time the string is over 1000 bytes in size...  Disk access is reduced considerably, and our access speeds are no longer the bottleneck for performance that they were.