Text Only | Text with Attachments

QB64.org Forum

Active Forums => QB64 Discussion => Topic started by: smokingwheels on February 18, 2021, 07:43:59 am

Title: I have a blocklist for my pihole and need to remove some lines
Post by: smokingwheels on February 18, 2021, 07:43:59 am
I open my list then compare each line with a array of urls from my whitelist.
The problem is the longer my whitelist is the longer it takes to complete the task.

Should I try 1 url at a time then replace the list for each url?
This means I would need to read and write 200 mb of data for each url.

Currently I have to compare each line to 80 urls that I would like whitelisted or removed from my blocklist
The list is over 6 million lines of text.

Blocklist link
 https://t.co/vRyCbbpcth?amp=1
Title: Re: I have a blocklist for my pihole and need to remove some lines
Post by: FellippeHeitor on February 18, 2021, 08:03:09 am
Hope the idea helps. Adapt as required:

Code: QB64: [Select]
  1. open "crazy long file.txt" for binary as #1
  2. wholefile$ = space$(lof(1))
  3. get #1, 1, wholefile$
  5.  
  6. for i=1 to 80
  7.     if instr(wholefile$, arrayOfUrls$(i)) > 0 then
  8.         'this url is in the file, do something with it
  9.     end if
  11.  
Title: Re: I have a blocklist for my pihole and need to remove some lines
Post by: smokingwheels on February 18, 2021, 09:56:29 am
ok will give it a go soon. Thank you
Title: Re: I have a blocklist for my pihole and need to remove some lines
Post by: smokingwheels on February 18, 2021, 10:48:26 am
I tried to run an old program but it seems to only work with 2 files open.
I have not converted it yet

Code: QB64: [Select]
  1. Rem Updated 7-1-2018
  2. Rem runs at 20 kHz
  3. Rem 30-1-2018
  4. Rem runs at 18.8 kHz
  5. Rem About 7 meg after crunchmaster.sh
  6. Rem found a space #...AND null string in the same day.
  7. Rem fixed some errors 4 NOV 17
  9. On Timer(1) GoSub health
  10.  
  11. Open "/home/john/Downloads/qb64/pi-hole/Master/w10" For Input As #4
  12.  
  13. Print "ghjkl"
  14. Rem array
  15. Dim whitelist$(2000)
  16.  
  18.     i = i + 1
  19.  
  20.     Print "1234"
  21.  
  22.     Line Input #4, a$
  23.     Print a$
  24.     whitelist$(i) = a$
  25.     Print Len(whitelist$(i))
  26.  
  29.  
  30. Open "/home/john/Downloads/qb64/pi-hole/Master/list5" For Input As #1
  31. Print "file 1 open"
  32. Open "/home/john/Downloads/qb64/pi-hole/Master/piholelist" For Output As #2
  33.  
  34.  
  35.  
  36. Rem chomp 2 try to remove null from gravity.
  37.  
  38. one = 1
  39. stack = 578
  41.     Line Input #1, a$
  42.     a = a + 1
  43.  
  44.     For i = 1 To stack
  45.  
  46.         wline = Len(whitelist$(i))
  47.         Print whitelist$(i)
  48.         'PRINT RIGHT$(a$, 10), whitelistline
  49.  
  50.         If Right$(a$, wline) = Right$(whitelist$(i), wline) Then GoTo skipoutput
  51.  
  52.     Next
  53.     lline = Len(a$)
  54.     'IF lline > 71 THEN GOTO skipoutput
  55.     Print #2, a$
  56.     skipoutput:
  57.     Hz = Hz + one
  59. Close #1: Close #2: Close #3: Close #4
  60. Print a, b
  61. Print "Thankyou Happy Hunting..."
  63.  
  64. health:
  65. Print Hz, a, b
  66. Hz = 0
  68.  
  69.  
Text Only | Text with Attachments