Author Topic: Word ladder - Rosetta Code  (Read 21195 times)

0 Members and 1 Guest are viewing this topic.

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Word ladder - Rosetta Code
« Reply #60 on: September 14, 2021, 08:22:18 pm »
@SMcNeill 
All your changes are fine including the t substitution for jk + 1, except using other than default screen that adds about a .1 sec to time on my system. Average is 2.27 secs on my system.

@david_uwi 
Yours are running best times on my system right around 2 secs on my system when I add
Code: QB64: [Select]
as Steve has done in his mods.

The path coming out is not the first alphabetically, it might be the last. Are you still doing multiple paths with that mod of yours?

I am also having problems incorporating your mod with Steve's.

Update: oh reverse the for loop! I think I have it. I will add Steve's changes to yours.
« Last Edit: September 14, 2021, 08:26:20 pm by bplus »

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Word ladder - Rosetta Code
« Reply #61 on: September 14, 2021, 09:44:31 pm »
Most odd! I could not get Steve's mod with ASC to work with david_uwi's latest code??? I left the code block in commented out. With david_uwi's latest mod mixed with Steve's changes I get 2.01 secs average on my system.
I expect it will be much lower on Steve's and johnno56.

Code: QB64: [Select]
  1. _Title "Word ladder v david_uwi 2021-09-14" '2021-09-10 david has already improved his first version!
  2. ' ref:  https://www.qb64.org/forum/index.php?topic=4157.msg135293#msg135293
  3. ' get original version above, I am modifying below to study
  4. ' 2021-09-11 in v3 b+ mod david_uwi I modify to compare whole set to my orignal whole set
  5. ' 2021-09-11 in v4 I think I can load the words faster
  6. ' 2021-09-12 in v5 lets just call the external file once, use david's idea instead of _continue,
  7. ' I think it looks cleaner without _continue which tool out skipping a For loop by a THEN (GOTO) line #.
  8. ' Ave of 5 runs before this (mod 4) was 11.27.. see if we improve still more, also DefLng A-Z  oh wow.
  9. ' Ave of 5 runs now 8.74 another 2.5 secs shaved off
  10. ' 2021-09-12 in v6 though using asc instead of mid$ might go faster.
  11. ' Ave of 5 runs now is 4.45 secs another 4.29 sec off!
  12.  
  13. ' 2021-09-13 v7 b+ mod david_uwi = v6 b+ mod but I will attempt to translate letter variables into
  14. ' more meaningful in attempts to understand what is being done in his code.
  15. ' Shaved significant time by making store$ (used to be q4) not fixed!
  16. ' Yes run a quick check that target word connects to something does save us so much time that
  17. ' we get a better overall time for the whole set even though we added a tiny bit of time to
  18. ' the ones that do connect.
  19. ' Ave of 5 runs is now 2.57 sec another 1.88 secs shaved off.
  20.  
  21. ' Steve's mods, mainly with $checking:off saves about .3 sec for me with ave 2.27
  22.  
  23. ' v david_uwi 2021-09-14 tried to work in Steves mods with david_uwi's latest improvement
  24. ' plus I insist the shortest path is the one that sorts out first alphabetically :)
  25. ' Running an average of 2.01 secs for my (bplus) system.
  26.  
  27. DefLng A-Z ' 2 secs off from old Single default!
  28. ReDim Shared Fwords$(1 To 1), UBF ' ubound of Fwords$
  29. start! = Timer(.001)
  30.  
  31. ' do just once for all ladder calls
  32. Open "unixdict.txt" For Binary As #1 ' take the file in a gulp
  33. buf$ = Space$(LOF(1))
  34. Get #1, , buf$
  35. Split buf$, Chr$(10), Fwords$()
  36. UBF = UBound(fwords$)
  37.  
  38. ' test set of ladder calls
  39. ladder "boy", "man" '     quick
  40. ladder "girl", "lady" '   this takes awhile
  41. ladder "john", "jane" '   quick enough
  42. ladder "alien", "drool" ' cool but takes a long long time!
  43. ladder "child", "adult" ' and this takes awhile
  44. ladder "play", "ball" '   goes quick
  45. ladder "fun", "job" '     ditto
  46. Print: Print "Total time including one disk file access:"; Timer(.001) - start!
  47.  
  48. Sub ladder (q1$, q2$)
  49.     tt! = Timer(.001) 'include time taken to load to RAM bplus mod to accuracy (.001)
  50.     Dim w(10000) As String * 5 ' words from file not String * 5 'make sure we have enough storage!!
  51.     Dim store$(10000, 100) ' wow  went from fixed string! storing connect words  to not fixed string and shaved a sec off time!!!
  52.     Dim storeIndexs(100)
  53.     Dim z4(10000, 100) As String
  54.     FI = 1
  55.     wordLength = Len(q1$)
  56.     If wordLength < 5 Then q1$ = q1$ + Space$(5 - wordLength): q2$ = q2$ + Space$(5 - wordLength)
  57.     While FI <= UBF
  58.         If Len(Fwords$(FI)) = wordLength Then
  59.             maxWordIndex = maxWordIndex + 1
  60.             If Fwords$(FI) = q1$ Then w(maxWordIndex) = "*****" Else w(maxWordIndex) = Fwords$(FI)
  61.         End If
  62.         FI = FI + 1
  63.     Wend
  64.  
  65.     'q2$ needs to have at least one connect or skip to end
  66.     '(this block will add a little more time to each ladder but save over a sec on adult or any target word with no connects)
  67.     For i = 1 To maxWordIndex
  68.         If w(i) <> q2$ Then ' just check before entering loop
  69.             cnt = 0
  70.             For j = 1 To wordLength
  71.                 If Asc(w(i), j) <> Asc(q2$, j) Then cnt = cnt + 1
  72.             Next j
  73.             If cnt = 1 Then ' q2$ has a connect good to go
  74.                 targetOK = -1: Exit For
  75.             End If
  76.         End If
  77.     Next i
  78.     If targetOK = 0 Then Print "No path found! from "; q1$; " to "; q2$;: GoTo skip
  79.  
  80.     ' carry on
  81.     jk = 1
  82.     storeIndexs(jk) = 1
  83.     store$(1, 1) = q1$
  84.     Do
  85.         For i = 1 To maxWordIndex
  86.             If w(i) <> "*****" Then '_Continue '  500  just check before entering loop
  87.                 cnt = 0
  88.                 For kk = 1 To storeIndexs(jk)
  89.                     cnt = 0
  90.                     For j = 1 To wordLength
  91.                         If Asc(w(i), j) = Asc(store$(kk, jk), j) Then cnt = cnt + 1 Else zz = j
  92.                     Next j
  93.                     If cnt = wordLength - 1 Then
  94.                         t = jk + 1
  95.                         storeIndexs(t) = storeIndexs(t) + 1
  96.                         store$(storeIndexs(t), t) = w(i)
  97.                         z4(storeIndexs(t), t) = z4(kk, jk) + Mid$(w(i), zz, 1) + Chr$(zz + 48) + " " 'stores a letter and change position
  98.                         w(i) = "*****"
  99.                     End If
  100.                 Next kk
  101.             End If
  102.         Next i
  103.         kflag = 0
  104.         ''*****new routine!!!
  105.         cnu = 0
  106.         For i = storeIndexs(t) To 1 Step -1 ' b+ reversed this for shortest path alphabetically
  107.             cnu = 0
  108.             For iq = 1 To wordLength
  109.                 If Asc(store$(i, t), iq) = Asc(q2$, iq) Then cnu = cnu + 1
  110.             Next iq
  111.             If cnu = wordLength - 1 Then kflag = 99: final$ = z4(i, t)
  112.         Next i
  113.  
  114.         If storeIndexs(t) = 0 Then kflag = 99
  115.         jk = jk + 1
  116.         If kflag = 99 Then Exit Do
  117.     Loop
  118.     If storeIndexs(jk) = 0 Then Print "No path found! from "; q1$; " to "; q2$; ' b+ removed a print blank line
  119.  
  120.     If storeIndexs(jk) > 0 Then 'this block works the next (commented out) wont
  121.         xlen = Len(final$)
  122.         'Print:
  123.         t$ = q1$
  124.         For i = 1 To xlen Step 3
  125.             c1$ = Mid$(final$, i, 1)
  126.             c2$ = Mid$(final$, i + 1, 1)
  127.             Mid$(q1$, Val(c2$), 1) = c1$
  128.             t$ = t$ + " " + q1$
  129.         Next i
  130.         Print t$ + " " + q2$
  131.     End If
  132.  
  133.  
  134.     'If storeIndexs(jk) > 0 Then ' this is Steve's substitution but wont work here  ???
  135.     '    xlen = Len(final$)
  136.     '    t$ = q1$ + " "
  137.     '    For i = 1 To xlen Step 3
  138.     '        c1 = Asc(final$, i)
  139.     '        c2 = Asc(final$, i + 1)
  140.     '        Asc(q1$, c2) = c1 ' >>>>>>>>>>>>>>> errors on this line    ???
  141.     '        t$ = t$ + q1$ + " "
  142.     '    Next i
  143.     '    Print t$
  144.     'End If
  145.  
  146.  
  147.     skip:
  148.     Print "time taken = "; Timer(.001) - tt!; " seconds"
  149.  
  150. Sub Split (SplitMeString As String, delim As String, loadMeArray() As String)
  151.     Dim curpos As Long, arrpos As Long, LD As Long, dpos As Long 'fix use the Lbound the array already has
  152.     curpos = 1: arrpos = LBound(loadMeArray): LD = Len(delim)
  153.     dpos = InStr(curpos, SplitMeString, delim)
  154.     Do Until dpos = 0
  155.         loadMeArray(arrpos) = Mid$(SplitMeString, curpos, dpos - curpos)
  156.         arrpos = arrpos + 1
  157.         If arrpos > UBound(loadMeArray) Then ReDim _Preserve loadMeArray(LBound(loadMeArray) To UBound(loadMeArray) + 1000) As String
  158.         curpos = dpos + LD
  159.         dpos = InStr(curpos, SplitMeString, delim)
  160.     Loop
  161.     loadMeArray(arrpos) = Mid$(SplitMeString, curpos)
  162.     ReDim _Preserve loadMeArray(LBound(loadMeArray) To arrpos) As String 'get the ubound correct
  163.  

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Word ladder - Rosetta Code
« Reply #62 on: September 14, 2021, 10:16:30 pm »
You’re getting the error you commented on from this line:

                        z4(storeIndexs(t), t) = z4(kk, jk) + MID$(w(i), zz, 1) + CHR$(zz + 48) + " " 'stores a letter and change position

Change CHR$(zz + 48) to just CHR$(zz) as I mentioned previously.with:

“I also swapped out the CHR$(zz + 48) to simply store the CHR$(zz), and then used ASC(final$, i + 1) to get that value back directly, rather than having to store it as a string and then take the VAL of it...”

By just storing zz, we can turn these 2 lines :

           c2$ = MID$(final$, i + 1, 1)
            MID$(q1$, VAL(c2$), 1) = c1$

Into:
     c2 = Asc(final$, i + 1)
    Asc(q1$, c2) = c1 ' >>>>>>>>>>>>>>> errors on this line    ???

We remove the use of VAL completely, and swap 2 MID$ for 2 ASC commands, as well as making c2 an integer instead of a string.



Basically iclly you’re storing numeric values as string characters “1”, “2”, “3”, while I’m storing them as ASCII characters CHR$(1), CHR$(2), CHR$(3).

You get your return values back via VAL(MID$…) while I get mine back via ASC.
« Last Edit: September 14, 2021, 10:21:58 pm by SMcNeill »
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Word ladder - Rosetta Code
« Reply #63 on: September 14, 2021, 10:33:26 pm »
Ah goot getting rid of 48 works!

Looks like string concatenation takes a tiny bit longer than just printing words? Maybe?
Code: QB64: [Select]
  1. _Title "Word ladder v david_uwi 2021-09-14" '2021-09-10 david has already improved his first version!
  2. ' ref:  https://www.qb64.org/forum/index.php?topic=4157.msg135293#msg135293
  3. ' get original version above, I am modifying below to study
  4. ' 2021-09-11 in v3 b+ mod david_uwi I modify to compare whole set to my orignal whole set
  5. ' 2021-09-11 in v4 I think I can load the words faster
  6. ' 2021-09-12 in v5 lets just call the external file once, use david's idea instead of _continue,
  7. ' I think it looks cleaner without _continue which tool out skipping a For loop by a THEN (GOTO) line #.
  8. ' Ave of 5 runs before this (mod 4) was 11.27.. see if we improve still more, also DefLng A-Z  oh wow.
  9. ' Ave of 5 runs now 8.74 another 2.5 secs shaved off
  10. ' 2021-09-12 in v6 though using asc instead of mid$ might go faster.
  11. ' Ave of 5 runs now is 4.45 secs another 4.29 sec off!
  12.  
  13. ' 2021-09-13 v7 b+ mod david_uwi = v6 b+ mod but I will attempt to translate letter variables into
  14. ' more meaningful in attempts to understand what is being done in his code.
  15. ' Shaved significant time by making store$ (used to be q4) not fixed!
  16. ' Yes run a quick check that target word connects to something does save us so much time that
  17. ' we get a better overall time for the whole set even though we added a tiny bit of time to
  18. ' the ones that do connect.
  19. ' Ave of 5 runs is now 2.57 sec another 1.88 secs shaved off.
  20.  
  21. ' Steve's mods, maily with $checking:off saves about .3 sec for me with ave 2.27
  22.  
  23. ' v david_uwi 2021-09-14 tried to work in Steves mods with david_uwi's latest improvement
  24. ' plus I insist the shortest path is the one that sorts out first alphabetically :)
  25. ' Running an average of 2.01 secs for my (bplus) system.
  26.  
  27. DefLng A-Z ' 2 secs off from old Single default!
  28. ReDim Shared Fwords$(1 To 1), UBF ' ubound of Fwords$
  29. start! = Timer(.001)
  30.  
  31. ' do just once for all ladder calls
  32. Open "unixdict.txt" For Binary As #1 ' take the file in a gulp
  33. buf$ = Space$(LOF(1))
  34. Get #1, , buf$
  35. Split buf$, Chr$(10), Fwords$()
  36. UBF = UBound(fwords$)
  37.  
  38. ' test set of ladder calls
  39. ladder "boy", "man" '     quick
  40. ladder "girl", "lady" '   this takes awhile
  41. ladder "john", "jane" '   quick enough
  42. ladder "alien", "drool" ' cool but takes a long long time!
  43. ladder "child", "adult" ' and this takes awhile
  44. ladder "play", "ball" '   goes quick
  45. ladder "fun", "job" '     ditto
  46. Print: Print "Total time including one disk file access:"; Timer(.001) - start!
  47.  
  48. Sub ladder (q1$, q2$)
  49.     tt! = Timer(.001) 'include time taken to load to RAM bplus mod to accuracy (.001)
  50.     Dim w(10000) As String * 5 ' words from file not String * 5 'make sure we have enough storage!!
  51.     Dim store$(10000, 100) ' wow  went from fixed string! storing connect words  to not fixed string and shaved a sec off time!!!
  52.     Dim storeIndexs(100)
  53.     Dim z4(10000, 100) As String
  54.     FI = 1
  55.     wordLength = Len(q1$)
  56.     If wordLength < 5 Then q1$ = q1$ + Space$(5 - wordLength): q2$ = q2$ + Space$(5 - wordLength)
  57.     While FI <= UBF
  58.         If Len(Fwords$(FI)) = wordLength Then
  59.             maxWordIndex = maxWordIndex + 1
  60.             If Fwords$(FI) = q1$ Then w(maxWordIndex) = "*****" Else w(maxWordIndex) = Fwords$(FI)
  61.         End If
  62.         FI = FI + 1
  63.     Wend
  64.  
  65.     'q2$ needs to have at least one connect or skip to end
  66.     '(this block will add a little more time to each ladder but save over a sec on adult or any target word with no connects)
  67.     For i = 1 To maxWordIndex
  68.         If w(i) <> q2$ Then ' just check before entering loop
  69.             cnt = 0
  70.             For j = 1 To wordLength
  71.                 If Asc(w(i), j) <> Asc(q2$, j) Then cnt = cnt + 1
  72.             Next j
  73.             If cnt = 1 Then ' q2$ has a connect good to go
  74.                 targetOK = -1: Exit For
  75.             End If
  76.         End If
  77.     Next i
  78.     If targetOK = 0 Then Print "No path found! from "; q1$; " to "; q2$;: GoTo skip
  79.  
  80.     ' carry on
  81.     jk = 1
  82.     storeIndexs(jk) = 1
  83.     store$(1, 1) = q1$
  84.     Do
  85.         For i = 1 To maxWordIndex
  86.             If w(i) <> "*****" Then '_Continue '  500  just check before entering loop
  87.                 cnt = 0
  88.                 For kk = 1 To storeIndexs(jk)
  89.                     cnt = 0
  90.                     For j = 1 To wordLength
  91.                         If Asc(w(i), j) = Asc(store$(kk, jk), j) Then cnt = cnt + 1 Else zz = j
  92.                     Next j
  93.                     If cnt = wordLength - 1 Then
  94.                         t = jk + 1
  95.                         storeIndexs(t) = storeIndexs(t) + 1
  96.                         store$(storeIndexs(t), t) = w(i)
  97.                         z4(storeIndexs(t), t) = z4(kk, jk) + Mid$(w(i), zz, 1) + Chr$(zz) + " " 'stores a letter and change position
  98.                         w(i) = "*****"
  99.                     End If
  100.                 Next kk
  101.             End If
  102.         Next i
  103.         kflag = 0
  104.         ''*****new routine!!!
  105.         cnu = 0
  106.         For i = storeIndexs(t) To 1 Step -1 ' b+ reversed this for shortest path alphabetically
  107.             cnu = 0
  108.             For iq = 1 To wordLength
  109.                 If Asc(store$(i, t), iq) = Asc(q2$, iq) Then cnu = cnu + 1
  110.             Next iq
  111.             If cnu = wordLength - 1 Then kflag = 99: final$ = z4(i, t)
  112.         Next i
  113.  
  114.         If storeIndexs(t) = 0 Then kflag = 99
  115.         jk = jk + 1
  116.         If kflag = 99 Then Exit Do
  117.     Loop
  118.     If storeIndexs(jk) = 0 Then Print "No path found! from "; q1$; " to "; q2$; ' b+ removed a print blank line
  119.  
  120.     'If storeIndexs(jk) > 0 Then 'this block works the next (commented out) wont
  121.     '    xlen = Len(final$)
  122.     '    'Print:
  123.     '    t$ = q1$
  124.     '    For i = 1 To xlen Step 3
  125.     '        c1$ = Mid$(final$, i, 1)
  126.     '        c2$ = Mid$(final$, i + 1, 1)
  127.     '        Mid$(q1$, Val(c2$), 1) = c1$
  128.     '        t$ = t$ + " " + q1$
  129.     '    Next i
  130.     '    Print t$ + " " + q2$
  131.     'End If
  132.  
  133.  
  134.     If storeIndexs(jk) > 0 Then ' this is Steve's substitution
  135.         xlen = Len(final$)
  136.         t$ = q1$ + " "
  137.         For i = 1 To xlen Step 3
  138.             c1 = Asc(final$, i)
  139.             c2 = Asc(final$, i + 1)
  140.             Asc(q1$, c2) = c1
  141.             t$ = t$ + q1$ + " "
  142.         Next i
  143.         Print t$ + " " + q2$
  144.     End If
  145.  
  146.  
  147.     skip:
  148.     Print "time taken = "; Timer(.001) - tt!; " seconds"
  149.  
  150. Sub Split (SplitMeString As String, delim As String, loadMeArray() As String)
  151.     Dim curpos As Long, arrpos As Long, LD As Long, dpos As Long 'fix use the Lbound the array already has
  152.     curpos = 1: arrpos = LBound(loadMeArray): LD = Len(delim)
  153.     dpos = InStr(curpos, SplitMeString, delim)
  154.     Do Until dpos = 0
  155.         loadMeArray(arrpos) = Mid$(SplitMeString, curpos, dpos - curpos)
  156.         arrpos = arrpos + 1
  157.         If arrpos > UBound(loadMeArray) Then ReDim _Preserve loadMeArray(LBound(loadMeArray) To UBound(loadMeArray) + 1000) As String
  158.         curpos = dpos + LD
  159.         dpos = InStr(curpos, SplitMeString, delim)
  160.     Loop
  161.     loadMeArray(arrpos) = Mid$(SplitMeString, curpos)
  162.     ReDim _Preserve loadMeArray(LBound(loadMeArray) To arrpos) As String 'get the ubound correct
  163.  
  164.  

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Word ladder - Rosetta Code
« Reply #64 on: September 14, 2021, 10:47:00 pm »
Quote
Looks like string concatenation takes a tiny bit longer than just printing words? Maybe?

Eh, not? The runs are too close in times.

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Word ladder - Rosetta Code
« Reply #65 on: September 14, 2021, 11:15:25 pm »
Eh, not? The runs are too close in times.

Maybe a difference in SCREEN 0 vs SCREEN 32?  A single time printing is about .1 to .2 seconds faster on my machine, when I tested it.

Or a difference in graphics cards?
« Last Edit: September 14, 2021, 11:16:47 pm by SMcNeill »
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline david_uwi

  • Newbie
  • Posts: 71
    • View Profile
Re: Word ladder - Rosetta Code
« Reply #66 on: September 15, 2021, 02:58:07 am »
I'm not sure that checking the last word for a path is a good idea. It works in this case, but I think that ADULT is something of an anomaly as there is no one letter substitution that will form a word - how often is that going to happen?

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Word ladder - Rosetta Code
« Reply #67 on: September 15, 2021, 04:44:22 am »
I'm not sure that checking the last word for a path is a good idea. It works in this case, but I think that ADULT is something of an anomaly as there is no one letter substitution that will form a word - how often is that going to happen?

A lot.  Check the list Iposted on the other page.

acts, ally, ohm, ugh….  There’s a ton of them!

Just about any word with more than 15 letters can be automatically eliminated, as they only connect to one partner word directly.  (Example: underclassmAn ONLY connects to underclassmEn.) It’s hard to claim there’s any “chain” between them when they connect directly.



One thing I’m curious about is why there’s a need to sort and copy the list over and over each time.

    WHILE FI <= UBF
        IF LEN(Fwords$(FI)) = wordLength THEN
            maxWordIndex = maxWordIndex + 1
            IF Fwords$(FI) = q1$ THEN w(maxWordIndex) = "*****" ELSE w(maxWordIndex) = Fwords$(FI)
        END IF
        FI = FI + 1
    WEND

Why not sort the wordlist by length ONCE at the time when you load the data in from the disk, rather than repeatedly each time it’s called?

(Pseudocode follows)
OPEN file$
GET word
wordnum(LEN(word)) = wordnum(LEN(word)) + 1 ‘counter for words of same length
wordlist(LEN(word),wordnum) = word ‘2d array to store words by length, wordnum
CLOSE

Words are then sorted in one pass and don’t have to be ever again.
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3972
    • View Profile
    • Steve’s QB64 Archive Forum
Re: Word ladder - Rosetta Code
« Reply #68 on: September 15, 2021, 05:02:09 am »
A second thing I’m curious about is with regards to why various data isn’t removed completely from the word lists at load time?  The routine that you guys are optimizing is obviously only intended to work with a very specific subset of the data, so why keep all the unnecessary data?

For example, the solved list you were generating stored values in a letter + position combo.  (b2 a1 y3, for example when going from “boy” to “bay”) 

At this point, you’re only storing a single digit for letter position, so you’ve decided to automatically reduce the dataset to words shorter than 10 letters.

Why not filter those out automatically to save processing them repeatedly?



      IF Fwords$(FI) = q1$ THEN w(maxWordIndex) = "*****" ELSE w(maxWordIndex) = Fwords$(FI)

With the way the above is configured, is it even possible to process a 6 letter word?  “*” is used when a letter matches the desired position, correct?  Or am I reading what’s going on here wrongly?

Seems to me that 6 or seven letter words wouldn’t work just because they have too many letters.  Will this work to see if “cheese” can turn into “butter”?

If not, then the dataset could be permanently reduced down to < 6, and not just < 10.
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Word ladder - Rosetta Code
« Reply #69 on: September 15, 2021, 11:28:58 am »
Quote
One thing I’m curious about is why there’s a need to sort and copy the list over and over each time.

    WHILE FI <= UBF
        IF LEN(Fwords$(FI)) = wordLength THEN
            maxWordIndex = maxWordIndex + 1
            IF Fwords$(FI) = q1$ THEN w(maxWordIndex) = "*****" ELSE w(maxWordIndex) = Fwords$(FI)
        END IF
        FI = FI + 1
    WEND

Why not sort the wordlist by length ONCE at the time when you load the data in from the disk, rather than repeatedly each time it’s called?

(Pseudocode follows)
OPEN file$
GET word
wordnum(LEN(word)) = wordnum(LEN(word)) + 1 ‘counter for words of same length
wordlist(LEN(word),wordnum) = word ‘2d array to store words by length, wordnum
CLOSE

Words are then sorted in one pass and don’t have to be ever again.

OK I did have the file unloaded into an array of word length strings but building those strings is concatenation and that takes time. I can't use w3$(), w4$(), w5$()... arrays because that would make it a nightmare using david_uwi's algo for all those different arrays.

Plus david_uwi's algo seems to need to mark the start word with "*****" (which BTW can be generalized to string$(wordLength, "*") for trying words > 5 letters to answer a question in next reply) so when we are copying  words into an array, we catch that starring of start word in the process, other wise we'd have to go through the list of wN$() find and star that word and then unstar it for next puzzle with same amount of letters.


Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Word ladder - Rosetta Code
« Reply #70 on: September 15, 2021, 11:46:36 am »
It is right not to build the solution exclusively to solve the particular test set of words to ladder. If a set of ladder words had more than a couple repeat word lengths it may save time running such a set by doing the word length strings and splitting that string in the ladder sub. I think it would take allot more than 2 puzzles of same word length to realize savings and in meantime the times for 1 or 2 puzzles of same letter length times will suffer.

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Word ladder - Rosetta Code
« Reply #71 on: September 16, 2021, 01:41:03 pm »
@david_uwi, @SMcNeill, @johnno56 and whoever else following this thread:

Are you ready to have your socks knocked off?

I now have Word Ladder code with david_uwi's algorithm generalized to any N letter words.
Here I have added
"cheese", "butter"  (doesn't connect)
and
"seaman", "skater" (does connect)
to our test of set and average .67 secs for the entire set!!!

Code: QB64: [Select]
  1. _Title "Word ladder v 2021-09-16" '2021-09-10 david has already improved his first version!
  2. ' ref:  https://www.qb64.org/forum/index.php?topic=4157.msg135293#msg135293
  3. ' get original version above, I am modifying below to study
  4. ' 2021-09-11 in v3 b+ mod david_uwi I modify to compare whole set to my orignal whole set
  5. ' 2021-09-11 in v4 I think I can load the words faster
  6. ' 2021-09-12 in v5 lets just call the external file once, use david's idea instead of _continue,
  7. ' I think it looks cleaner without _continue which tool out skipping a For loop by a THEN (GOTO) line #.
  8. ' Ave of 5 runs before this (mod 4) was 11.27.. see if we improve still more, also DefLng A-Z  oh wow.
  9. ' Ave of 5 runs now 8.74 another 2.5 secs shaved off
  10. ' 2021-09-12 in v6 though using asc instead of mid$ might go faster.
  11. ' Ave of 5 runs now is 4.45 secs another 4.29 sec off!
  12.  
  13. ' 2021-09-13 v7 b+ mod david_uwi = v6 b+ mod but I will attempt to translate letter variables into
  14. ' more meaningful in attempts to understand what is being done in his code.
  15. ' Shaved significant time by making store$ (used to be q4) not fixed!
  16. ' Yes run a quick check that target word connects to something does save us so much time that
  17. ' we get a better overall time for the whole set even though we added a tiny bit of time to
  18. ' the ones that do connect.
  19. ' Ave of 5 runs is now 2.57 sec another 1.88 secs shaved off.
  20.  
  21. ' Steve's mods, maily with $checking:off saves about .3 sec for me with ave 2.27
  22.  
  23. ' v david_uwi 2021-09-14 tried to work in Steves mods with david_uwi's latest improvement
  24. ' plus I insist the shortest path is the one that sorts out first alphabetically :)
  25. ' Running an average of 2.01 secs for my (bplus) system.
  26.  
  27. ' v 2021-09-16 can we get 6 letters going?  I can't seem to get more that a 3 word connect by eye
  28. ' ah goot! seaman to skater works  Hurray! with the added 6 letter words to ladder it is taking even
  29. ' less time than before I generalized to do more than 5 letter words. This is because w() array no
  30. ' longer is fixed String * 5 but now variable length. Also reason #3 why we have to load the n letter
  31. ' words every time is because we change w(i) with starUsed$ signal as we go through it to
  32. ' indicate we've processed this word already. Average 1.49 secs per run!!!
  33. ' Holy moley! another 2 huge cuts in time!!! both storage and changes arrays don't have to be 10000 long!
  34. ' Now .67 secs average on my (b+) older laptop
  35.  
  36. DefLng A-Z ' 2 secs off from old Single default!
  37. ReDim Shared Fwords$(1 To 1), UBF ' ubound of Fwords$
  38. start! = Timer(.001)
  39.  
  40. ' do just once for all ladder calls, Fwords$() contain the entire list of dictionary
  41. Open "unixdict.txt" For Binary As #1 ' take the file in a gulp
  42. buf$ = Space$(LOF(1))
  43. Get #1, , buf$
  44. Split buf$, Chr$(10), Fwords$()
  45. UBF = UBound(fwords$) ' track the ubound of Fwords$
  46.  
  47. ' test set of ladder calls
  48. ladder "boy", "man" '     quick
  49. ladder "girl", "lady" '   this takes awhile
  50. ladder "john", "jane" '   quick enough
  51. ladder "alien", "drool" ' cool but takes a long long time!
  52. ladder "child", "adult" ' and this takes awhile
  53. ladder "play", "ball" '   goes quick
  54. ladder "fun", "job" '     ditto
  55. ' These 6 letter words added to our test set to show it has been generalized past 5 letter words.
  56. ladder "cheese", "butter" ' Steve challnges to do more than 5 letter words  not going to connect
  57. ladder "seaman", "skater" ' I think this will connect
  58. Print: Print "Total time including one disk file access:"; Timer(.001) - start!
  59.  
  60. Sub ladder (q1$, q2$)
  61.     tt! = Timer(.001) ' time each ladder call, doesn't include one time download of words to Fwords$()
  62.     Dim w(10000) As String '* 5   <<< no fixed string huge time savings!!!! w() contains all words of Len(q1$)
  63.     Dim store$(100, 100) ' wow  went from fixed string storing connect words to not fixed string and shaved a sec off time!!!
  64.     Dim storeIndexs(100) ' tracking indexes to changes
  65.     Dim changes(100, 100) As String ' tracking the change letter and position going from one word to next
  66.     ' does changes have to be 10000? no! and another huge time cut!!!
  67.     ' does store$ have to be 10000? no! and another huge time cut!!!
  68.     FI = 1
  69.     wordLength = Len(q1$)
  70.     'If wordLength < 5 Then q1$ = q1$ + Space$(5 - wordLength): q2$ = q2$ + Space$(5 - wordLength)
  71.     starUsed$ = String$(wordLength, "*") ' this is signal that word is used
  72.     While FI <= UBF
  73.         If Len(Fwords$(FI)) = wordLength Then
  74.             maxWordIndex = maxWordIndex + 1
  75.             If Fwords$(FI) = q1$ Then w(maxWordIndex) = starUsed$ Else w(maxWordIndex) = Fwords$(FI)
  76.         End If
  77.         FI = FI + 1
  78.     Wend
  79.  
  80.     'q2$ needs to have at least one connect or skip to end
  81.     '(this block will add a little more time to each ladder but save over a sec on adult or any target word with no connects)
  82.     For i = 1 To maxWordIndex
  83.         If w(i) <> q2$ Then ' just check before entering loop
  84.             cnt = 0
  85.             For j = 1 To wordLength
  86.                 If Asc(w(i), j) <> Asc(q2$, j) Then cnt = cnt + 1
  87.             Next j
  88.             If cnt = 1 Then ' q2$ has a connect good to go
  89.                 targetOK = -1: Exit For
  90.             End If
  91.         End If
  92.     Next i
  93.     If targetOK = 0 Then Print "No path found! from "; q1$; " to "; q2$: GoTo skip
  94.  
  95.     ' carry on with daid_uwi's original algo modified by b+ for more general use and speed help from SMcNeill
  96.     jk = 1
  97.     storeIndexs(jk) = 1
  98.     store$(1, 1) = q1$
  99.     Do
  100.         For i = 1 To maxWordIndex
  101.             If w(i) <> starUsed$ Then
  102.                 cnt = 0
  103.                 For kk = 1 To storeIndexs(jk)
  104.                     cnt = 0
  105.                     For j = 1 To wordLength
  106.                         If Asc(w(i), j) = Asc(store$(kk, jk), j) Then cnt = cnt + 1 Else zz = j
  107.                     Next j
  108.                     If cnt = wordLength - 1 Then
  109.                         t = jk + 1
  110.                         storeIndexs(t) = storeIndexs(t) + 1
  111.                         store$(storeIndexs(t), t) = w(i)
  112.                         changes(storeIndexs(t), t) = changes(kk, jk) + Mid$(w(i), zz, 1) + Chr$(zz) + " " ' try Steve's T substitution version
  113.                         w(i) = starUsed$
  114.                     End If
  115.                 Next kk
  116.             End If
  117.         Next i
  118.         kflag = 0
  119.         ''*****new routine!!! by david_uwi
  120.         cnu = 0
  121.         For i = storeIndexs(t) To 1 Step -1 ' b+ reversed this for shortest path alphabetically
  122.             cnu = 0
  123.             For iq = 1 To wordLength
  124.                 If Asc(store$(i, t), iq) = Asc(q2$, iq) Then cnu = cnu + 1
  125.             Next iq
  126.             If cnu = wordLength - 1 Then kflag = 99: final$ = changes(i, t)
  127.         Next i
  128.  
  129.         If storeIndexs(t) = 0 Then kflag = 99
  130.         jk = jk + 1
  131.         If jk > 100 Then Print "No path found! from "; q1$; " to "; q2$: GoTo skip 'b+ added this for words that wont connect else error's out
  132.         If kflag = 99 Then Exit Do
  133.     Loop
  134.     If storeIndexs(jk) = 0 Then Print "No path found! from "; q1$; " to "; q2$ ' b+ removed a print blank line
  135.     If storeIndexs(jk) > 0 Then ' this is Steve's t substitution for david_uwi's jk + 1 using Asc instead of Mid$
  136.         xlen = Len(final$)
  137.         t$ = q1$
  138.         For i = 1 To xlen Step 3
  139.             c1 = Asc(final$, i)
  140.             c2 = Asc(final$, i + 1)
  141.             Asc(q1$, c2) = c1
  142.             t$ = t$ + " " + q1$
  143.         Next i
  144.         Print t$; " "; q2$
  145.     End If
  146.     skip:
  147.     Print "time taken = "; Timer(.001) - tt!; " seconds"
  148.  
  149. Sub Split (SplitMeString As String, delim As String, loadMeArray() As String)
  150.     Dim curpos As Long, arrpos As Long, LD As Long, dpos As Long 'fix use the Lbound the array already has
  151.     curpos = 1: arrpos = LBound(loadMeArray): LD = Len(delim)
  152.     dpos = InStr(curpos, SplitMeString, delim)
  153.     Do Until dpos = 0
  154.         loadMeArray(arrpos) = Mid$(SplitMeString, curpos, dpos - curpos)
  155.         arrpos = arrpos + 1
  156.         If arrpos > UBound(loadMeArray) Then ReDim _Preserve loadMeArray(LBound(loadMeArray) To UBound(loadMeArray) + 1000) As String
  157.         curpos = dpos + LD
  158.         dpos = InStr(curpos, SplitMeString, delim)
  159.     Loop
  160.     loadMeArray(arrpos) = Mid$(SplitMeString, curpos)
  161.     ReDim _Preserve loadMeArray(LBound(loadMeArray) To arrpos) As String 'get the ubound correct
  162.  
  163.  

 
Word ladder v 2021-09-16.PNG

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Word ladder - Rosetta Code
« Reply #72 on: September 16, 2021, 01:45:33 pm »
Ha! if I do child to adult 10,000 more times maybe it wont take any time. ;-))

BTW thanks to Steve's post of all the word connects I was able to guess that seaman to skater might be a good 6 letter test.

Is it the best? (best = longest path you can get for 6 letter words)
« Last Edit: September 16, 2021, 01:50:25 pm by bplus »

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Word ladder - Rosetta Code
« Reply #73 on: September 16, 2021, 04:45:50 pm »
Nutz, girl to lady picked up 2 words and alien to drool picked up one.

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Word ladder - Rosetta Code
« Reply #74 on: September 16, 2021, 05:08:09 pm »
Aha! If I run it without error checking it skips over places where the subscript goes over the array size, with error checking the change and store arrays are clearly seen as too small.