Text Only | Text with Attachments

QB64.org Forum

Active Forums => Programs => Topic started by: SpriggsySpriggs on October 26, 2020, 01:16:52 pm

Title: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on October 26, 2020, 01:16:52 pm
So I have been keeping this code solely on the Discord server but since a new user needed help with something that my code deals directly with I have decided to go ahead and make a dedicated post. I have made a small header file called "pipecom.h" which allows you to run a command and pipe back the console output to your program. Credit for help goes to @luke for making a version that doesn't have memory issues. Here is an example of some code you can run:

Code: QB64: [Select]
  1. DECLARE LIBRARY "pipecom"
  2.     FUNCTION pipecom$ (cmd AS STRING)
  4.  
  6.  
  7. F = pipecom("dir /b *.BAS")
  8.  

 


It can be used just like SHELL and the library works in Mac, Windows, and Linux. No more having to write to a file and then read it back! Of course, with Windows, you might see a console window that appears for the duration of the SHELLed program. To avoid this, use "PowerShell -windowstyle hidden" before the rest of your command and escape any CHR$(34)s with a backslash. Windows now operates just as well as Linux and Mac with no console window.
Title: Re: Pipe Console Output to QB64 program!
Post by: bplus on October 26, 2020, 04:43:11 pm
Yes! this looks to be a good replacement for DIR$(). I like that it comes in as string and you can get all the details or just file names.

@SpriggsySpriggs
Are there other switches? Probably have to leave it to Linux users to look up the ones they might need and I will ask you about the Windows ones ;-))

Also this doesn't seem to work if I add path to spec but that's OK, I guess. :)
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on October 26, 2020, 05:01:34 pm
Quote from: bplus on October 26, 2020, 04:43:11 pm
Yes! this looks to be a good replacement for DIR$(). I like that it comes in as string and you can get all the details or just file names.

@SpriggsySpriggs
Are there other switches? Probably have to leave it to Linux users to look up the ones they might need and I will ask you about the Windows ones ;-))

Also this doesn't seem to work if I add path to spec but that's OK, I guess. :)

@bplus This simply runs a command/program. Any command/program. However you would do it for SHELL is how it would be done for this. I'm not doing anything with the command except passing it to the C++ library and getting the output. Let me see the code you say isn't working with a path spec.
Title: Re: Pipe Console Output to QB64 program!
Post by: bplus on October 26, 2020, 06:04:40 pm
I tried "C:\\*.*" and "..\*.*" the root and the directory just above current.
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on October 26, 2020, 06:41:07 pm
Quote from: bplus on October 26, 2020, 06:04:40 pm
I tried "C:\\*.*" and "..\*.*" the root and the directory just above current.
Did you put "dir"?
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on October 26, 2020, 07:28:50 pm
Quote from: bplus on October 26, 2020, 06:04:40 pm
I tried "C:\\*.*" and "..\*.*" the root and the directory just above current.

@bplus I'm confused, bro. I did this and it worked fine:

Code: QB64: [Select]
  1. DECLARE LIBRARY "pipecom"
  2.     FUNCTION pipecom$ (cmd AS STRING)
  4.  
  6.  
  7. F = pipecom("dir C:\")
  8. 'F = pipecom("dir /b C:\*.*") 'if you want just filenames
  9.  
  11.  
  13.  
  14. F = pipecom("dir ..\")
  15. 'F = pipecom("dir /b ..\") 'if you want just filenames
  16.  

 
 

 
 


P.S. Again, it isn't meant to just replace DIR$(). This is just a library that pipes ANY console command/program output back to the QB64 program. You just have to know the usage of the commands you are calling.
Title: Re: Pipe Console Output to QB64 program!
Post by: bplus on October 26, 2020, 09:12:45 pm
OK thanks Spriggsy, I thought in Windows it didn't matter which way the slant leaned but in this case it does!

It does work fine leaning the integer division way \ but not regular division /

And it doesn't matter 1 or 2 slants to root drive.

 ;-))
Title: Re: Pipe Console Output to QB64 program!
Post by: Jesterdev on October 27, 2020, 08:13:30 pm
New to QB64, and this is exactly what I came here looking for. I'm attempting to create an interactive shell as a way of learning QB64 and teaching myself the the Linux shell. I've attempted a few things, but QB64 says it cannot find the library. I even tried pointing directly to it. Where exactly does the pipecom.h file need to be located?

Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on October 27, 2020, 08:17:00 pm
@Jesterdev First off, welcome to the family! Second, glad you found something you find useful! Third, the pipecom.h file needs to be kept in the same folder as your source OR if you move it to a different folder, just change your DECLARE LIBRARY block to show the path where it is. Fourth, be sure to join our Discord server! Many of us are active there and since we all live in various timezones you are bound to receive an answer to any questions you have! https://discord.gg/7E4W9Pv (https://discord.gg/7E4W9Pv)
Title: Re: Pipe Console Output to QB64 program!
Post by: Jesterdev on October 27, 2020, 08:37:38 pm
Thank you, appreciate the response. That's where I have it, in the same folder as my source. Still says it cannot find it. I'll try the discord also. Even copy and pasting the code from this post results in the same thing.
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on October 27, 2020, 08:41:24 pm
@Jesterdev Ah, I'm now wondering if your QB64 executable is in the same folder as your source! If not, then you do need to change the DECLARE LIBRARY to show the path (without the extension)
Title: Re: Pipe Console Output to QB64 program!
Post by: FellippeHeitor on October 27, 2020, 08:42:51 pm
Either keep the .h file in QB64’s folder or with your source. If you choose to keep it with the source, declare it as "./pipecom".

Welcome aboard.
Title: Re: Pipe Console Output to QB64 program!
Post by: luke on October 27, 2020, 09:24:59 pm
It's worth mentioning that this will leak memory on each call, because the string copy isn't freed.

Freeing that data turns out to be an absolute pain though because of how QB64 calls external functions. I tried to come up with a version that doesn't leak:
Code: [Select]
string pipecom_buffer;

const char* pipecom (char* cmd){
    FILE* stream;
    const int max_buffer = 256;
    char buffer[max_buffer];
    pipecom_buffer.clear();

    stream = popen(cmd, "r");
    if (stream) {
        while (!feof(stream)) {
            if (fgets(buffer, max_buffer, stream) != NULL) {
                pipecom_buffer.append(buffer);
            }
        }
        pclose(stream);
    }
return pipecom_buffer.c_str();
}It appears to work, but it relies on QB64's behaviour of (usually) making a copy of string data. There are cases where it doesn't copy (like if you do `LEN(pipecom("foo"))`) because the string is read-only and that's okay, but I can't verify that it'll be fine in all cases.

Anyway, if you noticed your memory usage was steadily increasing over time, try this version instead.
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on October 27, 2020, 09:26:47 pm
@luke I noticed that if I only return a c_str then I lose data in Windows. Have you tried this in Windows? This is a common problem mentioned in Stack Overflow.
Never mind, your way works. People on Stack Overflow didn't have a good solution.
Title: Re: Pipe Console Output to QB64 program!
Post by: Jesterdev on October 27, 2020, 09:29:00 pm
That was the trick! "./" worked. I also tried "/home/name/QB64/source/program/" and that didn't take either. The wiki was not exactly clear either, but I tried.

Thank so much, I really appreciate it. I'm jumping ahead a bit here. I'm working through Terry Ritchie's QB64 Game Programming tutorial, but I've been excited about this project for some time and can't help myself.

Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on October 27, 2020, 09:42:43 pm
Quote from: Jesterdev on October 27, 2020, 09:29:00 pm
That was the trick! "./" worked. I also tried "/home/name/QB64/source/program/" and that didn't take either. The wiki was not exactly clear either, but I tried.

@Jesterdev Sorry! I'm so used to keeping my source and everything in the same folder as the executable so I just naturally assume others will as well! Hope it helps you do what you need! Please tag me if you end up posting some code that uses the library so I can see what you come up with!
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on November 01, 2020, 05:43:08 pm
Updated pipecom library with new code specifically for Windows using the WinAPI that makes the output the same as what you would get on Linux and Mac with no console window.
Title: Re: Pipe Console Output to QB64 program!
Post by: badger on November 02, 2020, 03:10:19 pm
Hello

Does not look like you can use /P in there I have been trying that but it just goes to the end of the data and ends. Is that what it is supposed to do

Badger
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on November 02, 2020, 04:20:10 pm
@badger Can you be more specific? What is it you are trying to do? Can you post some code?
Title: Re: Pipe Console Output to QB64 program!
Post by: mpgcan on November 03, 2020, 03:44:01 am
You use the /P switch to have the Command Prompt pause results after it displays each screen. You have to press a key to continue viewing the next page of results. The command prompt window is hidden because of this it cannot receive input hence you cannot use /P.
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on November 03, 2020, 08:46:42 am
Quote from: mpgcan on November 03, 2020, 03:44:01 am
You use the /P switch to have the Command Prompt pause results after it displays each screen. You have to press a key to continue viewing the next page of results. The command prompt window is hidden because of this it cannot receive input hence you cannot use /P.
@mpgcan @badger Yes, you cannot do a command that will require user input. The same is true for a regular SHELL. This is meant to run a command and grab immediate output. You cannot interact with the program you are calling.
Title: Re: Pipe Console Output to QB64 program!
Post by: badger on November 03, 2020, 03:54:05 pm
Hello

Sorry just saw your post. there is what i mean. It does not give me the paging option it just goes to the end of the task and exits.

Badger

Code: QB64: [Select]
  1. DECLARE LIBRARY "pipecom"
  2.     FUNCTION pipecom$ (cmd AS STRING)
  4.  
  6.  
  7. 'F = pipecom("dir *.BAS")
  8. F = pipecom("dir /b /s /p*.BAS") 'if you need just the filenames and not the rest of the output
  9.  
  11.  
  12.  
  13.  
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on November 03, 2020, 03:55:59 pm
@badger Yes, you won't be able to interact with the console. This behaves just like SHELL except that it returns the text from the screen. You cannot provide input after the command is ran. This is expected. If you want the entire directory then you would have to ask for the whole list whether you used pipecom or if you used SHELL.
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on November 03, 2020, 04:15:26 pm
Updated 04:14 PM on 11/03/2020

Fixed bug in pipecom where the buffer wasn't cleared between calls which could lead to duplication of data.
Title: Re: Pipe Console Output to QB64 program!
Post by: mpgcan on November 05, 2020, 05:47:00 am
Hi Spriggsy
What an excellent solution using pipes. I have tried using these in the pass but always failed, never understood what bits to twiddle.

I downloaded the latest (05/11/2020 08:05) version of your pipecom library and it failed to compile on Windows 8.1.

No big deal, change file pipecom.bas to:
Code: QB64: [Select]
  2.  
  3. DECLARE LIBRARY "pipecom"
  4.     FUNCTION pipecom$ (cmd AS STRING)
  6.  
  8.  
  9. comm = pipecom$("dir /b *.BAS")
  10.  
  11. PRINT comm
  12.  
Compiled and run with no issues.

Looking at pipecom.h
Lines "int first;" and "int last;" are leftover code? delete these not required.

The small buffer size does not look right, proposal:

Replace this code:
Code: QB64: [Select]
  1.     char buf[2] = { }; //I know, the buffer is small but I had to in order to work with things like PING
  2.     DWORD dwRead = 0;
  3.     while (ReadFile(hStdOutPipeRead, &buf, 2, &dwRead, NULL) && dwRead)
  4.     {
  5.         buf[dwRead] = '\0';
  6.                 pipecom_buffer.append(buf);
  7.     }
  8.  
With the following code:
Code: QB64: [Select]
  1. //Note: ReadFile will return when buffer is full, or pipe is closed
  2. #define BUFFSZ 4096       //Seems to work here. Better to place at top of code
  3.     char buf[BUFFSZ + 1]; //+1 Space for null terminator
  4.     DWORD dwRead = 0;     //Number of bytes read. Automatically reset to 0 on each call to ReadFile
  5.     while (ReadFile(hStdOutPipeRead, buf, BUFFSZ, &dwRead, NULL))
  6.     {
  7.         buf[dwRead] = '\0';
  8.         pipecom_buffer.append(buf);
  9.     }
  10.  
Not sure what the issue you had with PING (buf[2]) however with the above changes I tried this code:
comm = pipecom$("ping google.com")
 it works.

All the best.
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on November 05, 2020, 10:29:13 am
Yes, you are correct about int first and int last. Those were pieces I didn't see when I uploaded it last. As for the buffer. I know when I had large buffer sizes that a ping would only return part of the console output. I had not tested it again after making the last changes to see if making a larger buffer would still work. Yes, it works. I had the buffer at 4096 at the beginning and it never worked correctly with ping unless I changed it to 2. So really all I'm going to do is undo some changes. I don't know why you would have had compilation errors. Did you look at the compilation log?
Title: Re: Pipe Console Output to QB64 program!
Post by: mpgcan on November 06, 2020, 03:49:44 am
Hi Spriggsy
   The term I used was incorrect instead of "failed to compile" I should have said failed to run!
The code you currently have for pipecom.bas is:
Code: QB64: [Select]
  2.  
  3. DECLARE LIBRARY "pipecom"
  4.     FUNCTION pipecom (cmd AS STRING)
  6.  
  8.  
  9. comm = "dir /b *.BAS"
  10.  
  11. PRINT comm
  12.  

It should be changed to:

Code: QB64: [Select]
  2.  
  3. DECLARE LIBRARY "pipecom"
  4.     FUNCTION pipecom$ (cmd AS STRING)
  6.  
  8.  
  9. comm = pipecom$("dir /b *.BAS")
  10.  
  11. PRINT comm
  12.  

All the best,
Again thanks for the working code.
Title: Re: Pipe Console Output to QB64 program!
Post by: SpriggsySpriggs on November 06, 2020, 11:36:15 am
@mpgcan Yes, you are correct. The file in my upload was wrong. I think I actually accidentally uploaded that one specifically. Even so, the very first post in this thread is valid code. Did you give it a look? It shows the proper usage so that there is no guesswork.
Text Only | Text with Attachments