Text Only | Text with Attachments

QB64.org Forum

Active Forums => QB64 Discussion => Topic started by: Cobalt on September 14, 2019, 03:35:15 am

Title: Reading Directories and Files without SHELLing out
Post by: Cobalt on September 14, 2019, 03:35:15 am
So I'm trying to add the ability to save (or load) game files from different directories for some odds and ends I am working on. Last time I did this I used DIRECTQBs routines. And I'm finding QB64 lacks any inbuilt way of loading Directories. and _DIR$ is sadly all but useless.(BTW why is that?)  I did however find a little gem by none other than Steve.

But I found a problem Steve,
it only recognizes  '.' and '..' as directories and treats everything else as files. it comes from a post on your forum which I found searching this forum for ways of obtaining a directory list.
Code: QB64: [Select]
  2.  FUNCTION load_dir& (s AS STRING)
  3.  FUNCTION has_next_entry& ()
  4.  SUB close_dir ()
  5.  SUB get_next_entry (s AS STRING, flags AS LONG, file_size AS LONG)
  7.  
  8. REDIM Dir(0) AS STRING, File(0) AS STRING
  9.  
  10. 'D$ = _CWD$
  11. D$ = "E:\SMB64\"
  12. GetFileList D$, Dir(), File()
  13.  
  14. PRINT "SUBDIRECTORIES"; UBOUND(dir)
  15. FOR i = 1 TO UBOUND(dir)
  16.  PRINT Dir(i),
  20.  
  21. PRINT "FILES"; UBOUND(file): PRINT: PRINT
  23. FOR i = 1 TO UBOUND(file)
  24.  PRINT File(i),
  27.  
  28. SUB GetFileList (SearchDirectory AS STRING, DirList() AS STRING, FileList() AS STRING)
  29.  CONST IS_DIR = 1
  30.  CONST IS_FILE = 2
  31.  DIM flags AS LONG, file_size AS LONG
  32.  
  33.  REDIM _PRESERVE DirList(100), FileList(100)
  34.  DirCount = 0: FileCount = 0
  35.  
  36.  IF load_dir(SearchDirectory) THEN
  37.   DO
  38.    length = has_next_entry
  39.    IF length > -1 THEN
  40.     nam$ = SPACE$(length)
  41.     get_next_entry nam$, flags, file_size
  42.     PRINT flags;
  43.     IF flags AND IS_DIR THEN
  44.      DirCount = DirCount + 1
  45.      IF DirCount > UBOUND(DirList) THEN REDIM _PRESERVE DirList(UBOUND(DirList) + 100)
  46.      DirList(DirCount) = nam$
  47.     ELSEIF flags AND IS_FILE THEN
  48.      FileCount = FileCount + 1
  49.      IF FileCount > UBOUND(filelist) THEN REDIM _PRESERVE FileList(UBOUND(filelist) + 100)
  50.      FileList(FileCount) = nam$
  51.     END IF
  52.    END IF
  53.   LOOP UNTIL length = -1
  54.   close_dir
  57.  REDIM _PRESERVE DirList(DirCount)
  58.  REDIM _PRESERVE FileList(FileCount)
  60.  

I added a couple of lines to see what was going on, printing the UBOUNDs for DIR() and FILE() as well as the FLAGS variable in the sub.

I've attached the output I get from my Super Mario directory, as you can see it only says there are 2 directories, yet anything listed in the file section without an extension is a directory too. This is true only cause I always make sure to have an extension on my files and never in my directories. Not everybody does so I couldn't use that as a sorting method.

I've also attached the header file needed to run the program in case anybody else wanted to take a look. as I can't remember the link to the topic its under on your forum right off this late. If nobody beats me to it I'll find it and post the link when I wake up. as for now I'm going to sleep.
Title: Re: Reading Directories and Files without SHELLing out
Post by: doppler on September 14, 2019, 09:06:24 am
Usable now, but when fully fleshed out.  Will become another tool bag trick for me.
Title: Re: Reading Directories and Files without SHELLing out
Post by: SMcNeill on September 14, 2019, 09:40:23 am
What is drive D for you?  I've never ran into any trouble with the routine sorting the files and directories for me, so I'm curious why it's failing  for you.  These are just simple little POSIX routines, so they're generally very reliable, and, so far, I don't think I've  ever gotten the results you're having.

Is D a USB drive?  An old drive, perhaps in a format it might not recognize?  (FAT16 or some such?)  Are you using an old OS which might not have the fully functional POSIX functions?  There's a lot of little things which it could be, but I don't have a clue as to why it's not sorting properly for you.  Test it out for me, if you can, on a few different drives/machines and let me know which ones it does and doesn't work for.  Once I know exactly what architecture it fails to sort on, then I might be able to figure out a way to patch the issue so we can work around it without any issues.

When it does fail to autosort, the easy solution is to just place them all into a single list and then use _FILEEXISTS and _DIREXISTS to sort them for file/directory usage.

EDIT:  In fact, since it's a simple enough change, I'll just add it directly into the routine itself as a secondary means of sorting/testing for us.  Try this and see if it sorts on your system for you:

Code: QB64: [Select]
  2.     FUNCTION load_dir& (s AS STRING)
  3.     FUNCTION has_next_entry& ()
  4.     SUB close_dir ()
  5.     SUB get_next_entry (s AS STRING, flags AS LONG, file_size AS LONG)
  7.  
  8. REDIM Dir(0) AS STRING, File(0) AS STRING
  9.  
  10. 'D$ = _CWD$
  11. D$ = _CWD$
  12. GetFileList D$, Dir(), File()
  13.  
  14. PRINT "SUBDIRECTORIES"; UBOUND(dir)
  15. FOR i = 1 TO UBOUND(dir)
  16.     PRINT Dir(i),
  20.  
  21. PRINT "FILES"; UBOUND(file): PRINT: PRINT
  23. FOR i = 1 TO UBOUND(file)
  24.     PRINT File(i),
  27.  
  28. SUB GetFileList (SearchDirectory AS STRING, DirList() AS STRING, FileList() AS STRING)
  29.     CONST IS_DIR = 1
  30.     CONST IS_FILE = 2
  31.     DIM flags AS LONG, file_size AS LONG
  32.  
  33.     REDIM _PRESERVE DirList(100), FileList(100)
  34.     DirCount = 0: FileCount = 0
  35.  
  36.     IF load_dir(SearchDirectory) THEN
  37.         DO
  38.             length = has_next_entry
  39.             IF length > -1 THEN
  40.                 nam$ = SPACE$(length)
  41.                 get_next_entry nam$, flags, file_size
  42.                 PRINT flags;
  43.                 IF flags AND IS_DIR OR _DIREXISTS(nam$) THEN
  44.                     DirCount = DirCount + 1
  45.                     IF DirCount > UBOUND(DirList) THEN REDIM _PRESERVE DirList(UBOUND(DirList) + 100)
  46.                     DirList(DirCount) = nam$
  47.                 ELSEIF flags AND IS_FILE OR _FILEEXISTS(nam$) THEN
  48.                     FileCount = FileCount + 1
  49.                     IF FileCount > UBOUND(filelist) THEN REDIM _PRESERVE FileList(UBOUND(filelist) + 100)
  50.                     FileList(FileCount) = nam$
  51.                 END IF
  52.             END IF
  53.         LOOP UNTIL length = -1
  54.         close_dir
  55.     ELSE
  56.     END IF
  57.     REDIM _PRESERVE DirList(DirCount)
  58.     REDIM _PRESERVE FileList(FileCount)
Title: Re: Reading Directories and Files without SHELLing out
Post by: Cobalt on September 14, 2019, 07:20:49 pm
If you mean drive E, its a partition on my laptop, on the main hard disk. Does the same thing on my desktop when trying to read the structure of a different specified location.

However, it does seem to work when using D$ = _CWD$ and reading the execution directory. Though that is not very helpful if a user wants to change directories when saving or loading.
Title: Re: Reading Directories and Files without SHELLing out
Post by: SMcNeill on September 14, 2019, 07:27:17 pm
Does it work with the change above?  If not, I’ll need to do a lot of digging to see what the difference is. 
Title: Re: Reading Directories and Files without SHELLing out
Post by: Cobalt on September 14, 2019, 09:00:05 pm
Only if your reading the Execution directory, as soon as you pick a new directory to read it fails to separate the directories from the files again.

what does yours do if you have it try to read D$="C:\" instead of D$= _CWD$ ?
Title: Re: Reading Directories and Files without SHELLing out
Post by: SMcNeill on September 14, 2019, 09:16:43 pm
Quote from: Cobalt on September 14, 2019, 09:00:05 pm
Only if your reading the Execution directory, as soon as you pick a new directory to read it fails to separate the directories from the files again.

what does yours do if you have it try to read D$="C:\" instead of D$= _CWD$ ?

You're right.  The issue is relative paths for us.   _DIREXISTS("my folder\") isn't the same thing as _DIREXISTS("C:\my folder\").  We strip those paths out so all we get back is the file structure for the specified folder, and not the WHOLE path + file/directory listing each time.

Try this patch and see how it performs for you:
Code: QB64: [Select]
  2.     FUNCTION load_dir& (s AS STRING)
  3.     FUNCTION has_next_entry& ()
  4.     SUB close_dir ()
  5.     SUB get_next_entry (s AS STRING, flags AS LONG, file_size AS LONG)
  7.  
  8. REDIM Dir(0) AS STRING, File(0) AS STRING
  9.  
  10. 'D$ = _CWD$
  11. D$ = "C:\"
  12. GetFileList D$, Dir(), File()
  13.  
  14. PRINT "SUBDIRECTORIES"; UBOUND(dir)
  15. FOR i = 1 TO UBOUND(dir)
  16.     PRINT Dir(i),
  21.  
  22. PRINT "FILES"; UBOUND(file): PRINT: PRINT
  24. FOR i = 1 TO UBOUND(file)
  25.     PRINT File(i),
  28.  
  29. SUB GetFileList (SearchDirectory AS STRING, DirList() AS STRING, FileList() AS STRING)
  30.     CONST IS_DIR = 1
  31.     CONST IS_FILE = 2
  32.     DIM flags AS LONG, file_size AS LONG
  33.  
  34.     REDIM _PRESERVE DirList(100), FileList(100)
  35.     DirCount = 0: FileCount = 0
  36.  
  37.     IF load_dir(SearchDirectory) THEN
  38.         DO
  39.             length = has_next_entry
  40.             IF length > -1 THEN
  41.                 nam$ = SPACE$(length)
  42.                 get_next_entry nam$, flags, file_size
  43.                 IF (flags AND IS_DIR) OR _DIREXISTS(SearchDirectory + nam$) THEN
  44.                     DirCount = DirCount + 1
  45.                     IF DirCount > UBOUND(DirList) THEN REDIM _PRESERVE DirList(UBOUND(DirList) + 100)
  46.                     DirList(DirCount) = nam$
  47.                 ELSEIF (flags AND IS_FILE) OR _FILEEXISTS(SearchDirectory + nam$) THEN
  48.                     FileCount = FileCount + 1
  49.                     IF FileCount > UBOUND(filelist) THEN REDIM _PRESERVE FileList(UBOUND(filelist) + 100)
  50.                     FileList(FileCount) = nam$
  51.                 END IF
  52.             END IF
  53.         LOOP UNTIL length = -1
  54.         close_dir
  55.     ELSE
  56.     END IF
  57.     REDIM _PRESERVE DirList(DirCount)
  58.     REDIM _PRESERVE FileList(FileCount)
Title: Re: Reading Directories and Files without SHELLing out
Post by: Cobalt on September 14, 2019, 10:17:49 pm
That seems to do the trick. Thanks Steve.

Odd that something that should be really simple takes so much effort anymore.
Title: Re: Reading Directories and Files without SHELLing out
Post by: bplus on September 15, 2019, 02:03:23 am
I am back to . and .. with all the sub-directories as files when I change D$ = "C:\" to my users folder "C:\users\marka"

Am I misunderstanding how to use this? Should SearchDirectory really mean search drive?
Title: Re: Reading Directories and Files without SHELLing out
Post by: Petr on September 15, 2019, 03:46:45 am
Hi. I think, this very small SUB solved all...
PLEASE TEST IT, if it works for you.


Code: QB64: [Select]
  2.     FUNCTION load_dir& (s AS STRING)
  3.     FUNCTION has_next_entry& ()
  4.     SUB close_dir ()
  5.     SUB get_next_entry (s AS STRING, flags AS LONG, file_size AS LONG)
  7.  
  8. REDIM Dir(0) AS STRING, File(0) AS STRING
  9.  
  10. D$ = _CWD$
  11. 'D$ = "" 'INSERT PATH HERE
  12.  
  13.  
  14.  
  15. SetPath D$
  16. GetFileList D$, Dir(), File()
  17.  
  18. PRINT "SUBDIRECTORIES"; UBOUND(dir)
  19. FOR i = 1 TO UBOUND(dir)
  20.     PRINT Dir(i),
  25.  
  26. PRINT "FILES"; UBOUND(file): PRINT: PRINT
  28. FOR i = 1 TO UBOUND(file)
  29.     PRINT File(i),
  32.  
  33.  
  34. SUB SetPath (p$)
  35.     L$ = RIGHT$(p$, 1)
  36.     $IF WIN THEN
  37.         IF L$ <> "\" THEN p$ = p$ + "\"
  38.     $ELSEIF LIN THEN
  39.         if L$ <> "/" then p$ = p$ + "/"
  40.     $END IF
  42.  
  43.  
  44.  
  45. SUB GetFileList (SearchDirectory AS STRING, DirList() AS STRING, FileList() AS STRING)
  46.     CONST IS_DIR = 1
  47.     CONST IS_FILE = 2
  48.     DIM flags AS LONG, file_size AS LONG
  49.  
  50.     REDIM _PRESERVE DirList(100), FileList(100)
  51.     DirCount = 0: FileCount = 0
  52.  
  53.     IF load_dir(SearchDirectory) THEN
  54.         DO
  55.             length = has_next_entry
  56.             IF length > -1 THEN
  57.                 nam$ = SPACE$(length)
  58.                 get_next_entry nam$, flags, file_size
  59.                 IF (flags AND IS_DIR) OR _DIREXISTS(SearchDirectory + nam$) THEN
  60.                     DirCount = DirCount + 1
  61.                     IF DirCount > UBOUND(DirList) THEN REDIM _PRESERVE DirList(UBOUND(DirList) + 100)
  62.                     DirList(DirCount) = nam$
  63.                 ELSEIF (flags AND IS_FILE) OR _FILEEXISTS(SearchDirectory + nam$) THEN
  64.                     FileCount = FileCount + 1
  65.                     IF FileCount > UBOUND(filelist) THEN REDIM _PRESERVE FileList(UBOUND(filelist) + 100)
  66.                     FileList(FileCount) = nam$
  67.                 END IF
  68.             END IF
  69.         LOOP UNTIL length = -1
  70.         close_dir
  71.     ELSE
  72.     END IF
  73.     REDIM _PRESERVE DirList(DirCount)
  74.     REDIM _PRESERVE FileList(FileCount)
  76.  
  77.  
Title: Re: Reading Directories and Files without SHELLing out
Post by: bplus on September 15, 2019, 09:42:31 am
Thanks Petr,

I guess I was using it incorrectly, I needed a slash at the end for the SearchDirectory variable.

Well this should work nicely in my Tiny Navigator which could then work in Linux?

For Windows, either slash direction works, you can even mix and match :)
Title: Re: Reading Directories and Files without SHELLing out
Post by: doppler on September 15, 2019, 09:54:52 am
Thank-you, Thank-you, Thank-you

I have been using shell and capture output to mange files and directories for so long.

Using less shells and this code is so much more efficient.

Why anyone would want to use the "FILES" command is beyond me at this point now.
Why look at files listed, when you can't do a damn thing with them.
Title: Re: Reading Directories and Files without SHELLing out
Post by: Petr on September 15, 2019, 10:06:26 am
Hi BPlus, yeah, direntry.h is compatible with Linux, so your file browser will also work in Linux. You just have to be careful of the characters size in the names of folders and files, that's what Linux is sensitive to.
But there is also an option in Linux such as DIR to a file in windows, so you do this in Linux -   with command "ls > file.txt".

Hi Doppler. FILES is a prehistory maintained for compatibility with older Stone Age ...  :)
Title: Re: Reading Directories and Files without SHELLing out
Post by: Cobalt on September 15, 2019, 11:59:18 am
Glad so many people are happy I brought this up, As I stated in my first post it is a little Gem of a routine.

Won't actually return file size, but you could always add a simple OPEN:LOF:CLOSE to get file sizes as it reads in the files(I mean put the code in the ELSEIF area). I do not need that functionality in what I'm working on, but something like Bplus is talking about you might what to show file size as well.

Wonder why they made the Linux OS so picky? Maybe its just me but I can't see the need for case sensitivity in file names. Be like having a variable named ABCD and having 15 variations on it simply by using upper and lower case, that would be horrible to keep track of I would think! The forward-backward slash thing I don't know, I tend to use either one at random which may be bad practice in reality, but I still feel its rather nit picky to be arbitrarily enforcing one or the other.
Title: Re: Reading Directories and Files without SHELLing out
Post by: SMcNeill on September 15, 2019, 12:41:30 pm
Quote from: Cobalt on September 15, 2019, 11:59:18 am
Glad so many people are happy I brought this up, As I stated in my first post it is a little Gem of a routine.

Won't actually return file size, but you could always add a simple OPEN:LOF:CLOSE to get file sizes as it reads in the files(I mean put the code in the ELSEIF area). I do not need that functionality in what I'm working on, but something like Bplus is talking about you might what to show file size as well.

It *does* get the file_size for us; we just never return it anywhere to make use of it:
Code: [Select]
get_next_entry nam$, flags, file_size
If you need file size, change the routine just a bit and make your file listing an array to hold that information, like so:
Code: QB64: [Select]
  2.     FUNCTION load_dir& (s AS STRING)
  3.     FUNCTION has_next_entry& ()
  4.     SUB close_dir ()
  5.     SUB get_next_entry (s AS STRING, flags AS LONG, file_size AS LONG)
  7.  
  8. TYPE File_Type
  9.     Size AS _INTEGER64
  10.     Name AS STRING
  12.  
  13.  
  14. REDIM Dir(0) AS STRING, File(0) AS File_Type
  15.  
  16. D$ = _CWD$
  17. D$ = "C:\"
  18. GetFileList D$, Dir(), File()
  19.  
  20. PRINT "SUBDIRECTORIES"; UBOUND(dir)
  21. FOR i = 1 TO UBOUND(dir)
  22.     PRINT Dir(i)
  27.  
  28. PRINT "FILES"; UBOUND(file): PRINT: PRINT
  30. FOR i = 1 TO UBOUND(file)
  31.     PRINT File(i).Name, File(i).Size
  34.  
  35. SUB GetFileList (SearchDirectory AS STRING, DirList() AS STRING, FileList() AS File_Type)
  36.     CurrDir$ = _CWD$
  37.     CHDIR SearchDirectory
  38.     CONST IS_DIR = 1
  39.     CONST IS_FILE = 2
  40.     DIM flags AS LONG, file_size AS LONG
  41.  
  42.     REDIM _PRESERVE DirList(100), FileList(100) AS File_Type
  43.     DirCount = 0: FileCount = 0
  44.  
  45.     IF load_dir(_CWD$) THEN
  46.         DO
  47.             length = has_next_entry
  48.             IF length > -1 THEN
  49.                 nam$ = SPACE$(length)
  50.                 get_next_entry nam$, flags, file_size
  51.                 IF flags AND IS_DIR THEN
  52.                     DirCount = DirCount + 1
  53.                     IF DirCount > UBOUND(DirList) THEN REDIM _PRESERVE DirList(UBOUND(DirList) + 100)
  54.                     DirList(DirCount) = nam$
  55.                 ELSEIF flags AND IS_FILE THEN
  56.                     FileCount = FileCount + 1
  57.                     IF FileCount > UBOUND(filelist) THEN REDIM _PRESERVE FileList(UBOUND(filelist) + 100) AS File_Type
  58.                     FileList(FileCount).Name = nam$
  59.                     FileList(FileCount).Size = file_size
  60.                 END IF
  61.             END IF
  62.         LOOP UNTIL length = -1
  63.         close_dir
  64.     ELSE
  65.     END IF
  66.     REDIM _PRESERVE DirList(DirCount)
  67.     REDIM _PRESERVE FileList(FileCount) AS File_Type
  68.     CHDIR CurrDir$
  70.  

(And, since this also temporarily changes the path for us, it corrects the issue you were experiencing before, where relative path vs absolute path caused issues.  No worries though; it changes it back once it's finished, so your program won't have any pathing problems when you plug it in.)
Title: Re: Reading Directories and Files without SHELLing out
Post by: doppler on September 16, 2019, 09:12:46 am
One last question for this neat new tool bag trick.

When directories and files are found.  Will they always be in sorted order ?

I created a test directory with files and directories created out of order.  Running the program
everything became sorted properly.

I expected the list to be as found (out of order in directory) not sorted.

I can take advantage of this.  If this will always be the case.
Title: Re: Reading Directories and Files without SHELLing out
Post by: SMcNeill on September 16, 2019, 10:04:19 am
To be honest, I don’t know.  Usually you can change sorting order by parameter (sort by name, or size, or type), and I don’t remember if I specified for this to sort by name, or if it’s just doing it by default.  (In which case, the default sort method might not be the same all the time.). I’ll have to dig back into my own code later, and sort out how/if it’s handling sorting for us.

It’s a great question; I just don’t have a great answer for it, at the moment.  😳
Title: Re: Reading Directories and Files without SHELLing out
Post by: Petr on September 16, 2019, 10:18:58 am
I think (but I'm not sure) that under Linux I got output that was not alphabetized. Some Linux users would have to try that. I shut down the Linux computer because I was using an old snail with Linux disguised as a computer ...
Title: Re: Reading Directories and Files without SHELLing out
Post by: Cobalt on September 16, 2019, 10:55:05 am
Quote from: SMcNeill on September 15, 2019, 12:41:30 pm
It *does* get the file_size for us; we just never return it anywhere to make use of it:
thats odd, when I tried storing and printing the file size earlier(when I made that post) it only showed '0' for all file sizes.. thats why I thought it wasn't working. In fact it does show file size now.. very odd. I sit corrected.


*some how a /code was stuck in my post... must have missed it when I made the quote. all fixed.
Title: Re: Reading Directories and Files without SHELLing out
Post by: SMcNeill on September 16, 2019, 11:36:13 am
Quote from: Cobalt on September 16, 2019, 10:55:05 am
thats odd, when I tried storing and printing the file size earlier(when I made that post) it only showed '0' for all file sizes.. thats why I thought it wasn't working. In fact it does show file size now.. very odd. I sit corrected.

I'd say that was the same issue with it not sorting directories and folders for you.  The version I posted last fixed that issue as well, so you shouldn't have a problem with either from now on.  ;)
Title: Re: Reading Directories and Files without SHELLing out
Post by: luke on September 17, 2019, 09:43:27 am
The 'order' of the returned files is determined by the file system driver, and tends to be a reflection of how things are organised in the file system data structures.

In short, they're in absolutely no order at all.

You'll need to read all the items then sort them yourself by whatever parameter you want.
Text Only | Text with Attachments