Text Only | Text with Attachments

QB64.org Forum

Active Forums => QB64 Discussion => Topic started by: loudar on July 27, 2019, 07:06:53 am

Title: Wrote code that will extract the text of Wikipedia articles into an array.
Post by: loudar on July 27, 2019, 07:06:53 am
Wrote this little block of code that will download any Wikipedia article and extract the raw text from it. This can be used for e.g. own dictionaries, encyclopedias or processing of information. Feel free to use and toy around with and update me on it :>

Figuring out how to shell out curl was a bit of a hassle, but now it works! :D

Code: QB64: [Select]
  1. word$(c) = "QB64" 'replace with the word you want to have extracted, could be used in a loop to extract more than one word
  2. shellcmd$ = "start powershell -Command " + CHR$(34) + "curl https://en.wikipedia.org/wiki/" + word$(c) + " -o temp.txt" + CHR$(34)
  3. SHELL _HIDE shellcmd$ 'starts powershell from the CMD directly with the command because curl doesn't work from CMD
  4.  
  5. SLEEP 5 'waits for the file to be created, there is probably a better way to determine when the file is created
  6.  
  7. OPEN "temp.txt" FOR INPUT AS #1
  8. x = 0
  10.     LINE INPUT #1, line$
  11.     IF MID$(line$, 1, 3) = "<p>" THEN 'only reads paragraphs, as this is is the safest method for getting the raw text, feel free to toy around with other things as well
  12.         x = x + 1
  13.         p = 0
  14.         DO
  15.             p = p + 1
  16.             IF MID$(line$, p, 1) = "<" THEN 'for now just excludes every html command to get a clean text
  17.                 DO
  18.                     p = p + 1
  19.                 LOOP UNTIL MID$(line$, p, 1) = ">"
  20.             ELSE
  21.                 rawline$(x) = rawline$(x) + MID$(line$, p, 1) 'saves the respective paragraph as a raw line, accessible with parameter x as count for paragraphs in an article
  22.             END IF
  23.         LOOP UNTIL p = LEN(line$)
  24.         'PRINT rawline$(x) 'uncomment this line to print the raw data on the screen and check for eventual bugs in the raw lines
  25.         'here is the space for processing of each line
  26.     END IF
  27. LOOP UNTIL EOF(1) = -1
Title: Re: Wrote code that will extract the text of Wikipedia articles into an array.
Post by: bplus on July 27, 2019, 09:43:17 am
This is interesting and nice simple start, thanks for sharing.
Title: Re: Wrote code that will extract the text of Wikipedia articles into an array.
Post by: SMcNeill on July 27, 2019, 10:46:42 am
This is very similar to how I extracted the wiki keywords from our QB64 wiki here: https://www.qb64.org/forum/index.php?topic=756.msg6457#msg6457

If you’re not downloading secure pages (https instead of http), you shouldn’t need to powershell and use curl to get the page to parse it.  An example of using QB64 for the whole process can be viewed via the link above.
Title: Re: Wrote code that will extract the text of Wikipedia articles into an array.
Post by: Pete on July 27, 2019, 12:05:32 pm
I download html pages into a single variable. In fact, I read the file with the binary method you posted so years back. It's a bit better than the one the IMan posted 15+ years ago, at QBF. Easy to parse it after that. I haven't done Wiki pages, so I don't know if I'd handle them the same way or not.

Wget has been improved upon, so I believe https is now supported. It sucks I can't be sure, because it was well over a year ago I did any of this stuff, but I think I'm correct, because I downloaded the latest version, wheres before, I stuck to using cURL, because you could associate it with Firefox and easily download pages from https sites; however, I like the use of Powershell here. There is a whole new world of possibilities thanks to Powershell, so it's nice to see another one of its uses demoed.

On behalf of all the QB sheeple, thanks for shearing.

Pete

 
Title: Re: Wrote code that will extract the text of Wikipedia articles into an array.
Post by: loudar on July 28, 2019, 08:25:30 am
Quote from: SMcNeill on July 27, 2019, 10:46:42 am
If you’re not downloading secure pages (https instead of http), you shouldn’t need to powershell and use curl to get the page to parse it.

I tried it without powershell and it didn't work, neither via QB or manually in the CMD. That's why I chose to go with it, seemed like the only and the simplest option! :D
Title: Re: Wrote code that will extract the text of Wikipedia articles into an array.
Post by: MLambert on August 12, 2019, 11:52:37 pm
Hi,
I do a lot of screen scraping and I use curl.

I just shell to it .... shell "curl.exe etc..."

I don't understand why powershell is needed.

Mike
Text Only | Text with Attachments