Author Topic: Text Corrector (Read 2127 times)

PoliMi · « **on:** October 12, 2021, 04:24:29 pm »

Text Corrector corrects writing errors while the user is composing a text. It doesn't require a vocabulary file, instead it learns any language from a given sample text. The algorithm not only extracts the words from that sample, but it also extracts the links between words and analyzes the probabilistic features of the language. Basing on the collected data, typing errors are then modeled as gaussian distributions which mean is the right value.

While the user is typing a new word, in order to predict which word the user intends to write, the algorithm evaluates vocabulary words affinity, which is a weighted mean of three parameters:
- LENGTH(i): the probabilistic distance between the length of the typed word and the length of the i-th vocabulary word
- CONTEXT(i): the estimated frequency that, given one last typed word, the i-th vocabulary word follows
- STRUCTURE(i, j): the probabilistic distance of the j-th letter of the typed word from the j-th position in the i-th vocabulary word

To enhance prediction performance, the algorithm recalibrates these parameters with two further correction mechanisms:
- BIAS: the weight of the LENGTH parameter is reduced while few letters have been typed, since with few letters typed it's probable that length is not definitive (i.e., that more letters will be typed, and length will soon increase)
- BONUS: if the exact typed word exists in the vocabulary, its affinity is brought to the maximum, so that it prevails on all other vocabulary words.

Even though in order to fully work the algorithm needs a long and complete text sample that is representative of the language, the default sample hard-coded in the program can already show a good example of its predictive behavior.

Code: QB64: [Select]

_Title "Text Corrector"
Common Shared L: L = 100000 + 1 ' max number of links + 1
Dim Shared numWords As Integer
Dim Shared numLinks As Integer
Dim Shared numCountedLinks As Integer
Dim Shared numSearchedCountedLinks As Integer
Dim Shared totSearchedCountedLinks As Integer
Dim Shared links(L, 2) As String
Dim Shared countedLinks(L, 3) As String
Dim Shared searchedCountedLinks(L, 3) As String
Dim Shared words(L) As String
Dim Shared wordsProbability(L, 4) As Double ' 1) LENGTH   2) CONTEXT   3) STRUCTURE   4) GLOBAL
Dim Shared GaussTable(0 To 400) As Double: fillGaussTable
 
If _FileExists("sample.txt") Then
    Open "sample.txt" For Input As #1
    Do Until EOF(1)
        Line Input #1, temp$
        sample$ = sample$ + temp$ + " "
    Loop
    Close #1
Else
    sample$ = "Today is a beautiful day, tomorrow I'm going to the park. The park is far from home, so today I will stay at home or maybe I will go to the supermarket. By the way, your house is beautiful and big, I really like it!"
End If
 
lastArchivedWord$ = "."
isFirstTime = -1
lastInputWasPunctuation = 0
learnFromSample (sample$)
Do
    If Not key$ = "" Or isFirstTime Then
        isFirstTime = 0
        bias1 = 0.10 / (2 ^ (1 * Len(newWord$))) ' with few letters context >> length
        bias2 = 0.05 / (2 ^ (1 * Len(newWord$))) ' with few letters structure >> length
        computeWordAffinity lastArchivedWord$, newWord$, 0.15 - bias1 - bias2, 0.30 + bias1, 0.55 + bias2, 1
        refreshTextEditor archivedText$, lastArchivedWord$, newWord$
    End If
 
    key$ = LCase$(InKey$): _Limit 20
    If Not key$ = "" Then
        If isPunctuation(key$) Then
            archivedText$ = archivedText$ + " " + lastArchivedWord$ + " "
            If newWord$ = "" Then
                archivedText$ = archivedText$ + " "
            Else
                archivedText$ = archivedText$ + " " + highestAffinityWord$ + " "
                newWord$ = ""
            End If
            lastArchivedWord$ = key$
            lastInputWasPunctuation = -1
        Else
            Select Case key$
                Case " ":
                    If Not lastInputWasPunctuation Then
                        archivedText$ = archivedText$ + " " + lastArchivedWord$ + " "
                        If isPunctuation(newWord$) Then
                            lastArchivedWord$ = newWord$
                        Else
                            lastArchivedWord$ = highestAffinityWord$
                        End If
                        newWord$ = ""
                    Else
                        lastInputWasPunctuation = 0
                        key$ = ""
                    End If
                Case Chr$(8):
                    If Not Len(newWord$) = 0 Then
                        newWord$ = Left$(newWord$, Len(newWord$) - 1)
                    Else
                        key$ = ""
                    End If
                Case Chr$(13): key$ = ""
                Case Chr$(27): System
                Case Else:
                    newWord$ = newWord$ + key$
                    lastInputWasPunctuation = 0
            End Select
        End If
    End If
Loop
 
 
 
 
Sub refreshTextEditor (archivedText$, lastArchivedWord$, newWord$)
    showComputedWordAffinity
    toprint$ = archivedText$
    If Not isPunctuation(lastArchivedWord$) Then toprint$ = toprint$ + " "
    toprint$ = toprint$ + lastArchivedWord$
    If Not isPunctuation(newWord$) Then toprint$ = toprint$ + " "
    toprint$ = toprint$ + newWord$ + "_"
    showText destandardize$(Right$(toprint$, Len(toprint$) - 2), -1), Len(newWord$), -1
    _Display
End Sub
 
Sub showText (text$, numOfFinalCharsToColor, colorFinalChar)
    Do While Len(text$) > 80 * 10
        text$ = Right$(text$, Len(text$) - 80)
    Loop
    Color 7: Locate 1, 1
    Print "Write something, I'll try to correct you:"
    Color 15: Locate 2, 1
    line$ = ""
    For I = 1 To Len(text$)
        line$ = line$ + Mid$(text$, I, 1)
        If I = Len(text$) - numOfFinalCharsToColor Then Color 13
        If colorFinalChar And I = Len(text$) Then Color 7
        Print Mid$(text$, I, 1);
        If Len(line$) >= 80 Or I = Len(text$) Then line$ = "": Print
    Next I
End Sub
 
Sub showComputedWordAffinity
    best$ = highestAffinityWord$
    Cls: Locate 13, 1: Color 8
    Print "WORD", "LENGTH", "CONTEXT", "STRUCTURE", "GLOBAL"
    For i = 1 To min(numWords, 10)
        If words(i) = best$ Then Color 15
        If Not isSpecial(words(i)) Then
            If Len(words(i)) > 12 Then
                Print Left$(words(i), 11); " ",
            Else
                Print words(i); " ",
            End If
            Print Int(wordsProbability(i, 1) * 100); "%   ", Int(wordsProbability(i, 2) * 100); "%   ", Int(wordsProbability(i, 3) * 100); "%   ", max(Int(wordsProbability(i, 4) * 100), 0); "%"
        End If
        If words(i) = best$ Then Color 8
    Next i
End Sub
 
Function highestAffinityWord$
    For I = 1 To numWords
        If Not isSpecial(words(I)) Then
            highestAffinityWord$ = words(I)
            Exit For
        End If
    Next I
End Function
 
Sub learnFromSample (sample$)
    decomposeIntoLinks (sample$)
    orderLinks
    countLinks
    collectWords
End Sub
 
Sub computeWordAffinity (oldword$, newWord$, A, B, C, BONUS) ' A = length weight, B = context weight, C = structure weight, BONUS = value added to global probability (constraint: A + B + C = 1)
    For I = 1 To numWords
        wordsProbability(I, 1) = 0
        wordsProbability(I, 2) = 0
        wordsProbability(I, 3) = 0
        wordsProbability(I, 1) = probability(Len(newWord$), Len(words(I))) / probability(0, 0)
        wordsProbability(I, 3) = structureLikeness(newWord$, words(I))
    Next I
    searchLinks (oldword$)
    For I = 1 To numSearchedCountedLinks
        position = wordPosition(searchedCountedLinks(I, 2))
        wordsProbability(position, 2) = Val(searchedCountedLinks(I, 3)) / totSearchedCountedLinks
    Next I
    For I = 1 To numWords
        wordsProbability(I, 4) = Int((A * wordsProbability(I, 1) + B * wordsProbability(I, 2) + C * wordsProbability(I, 3)) * 10 ^ 4) / 10 ^ 4
        If wordsProbability(I, 1) = 1 And wordsProbability(I, 3) = 1 Then wordsProbability(I, 4) = min(1, wordsProbability(I, 4) + BONUS)
    Next I
    orderComputedWords
    shuffleWords
End Sub
 
Sub orderComputedWords
    Do
        changed = 0
        For I = 2 To numWords
            If wordsProbability(I, 4) > wordsProbability(I - 1, 4) Then
                buffer$ = words(I - 1)
                buffer1 = wordsProbability(I - 1, 1)
                buffer2 = wordsProbability(I - 1, 2)
                buffer3 = wordsProbability(I - 1, 3)
                buffer4 = wordsProbability(I - 1, 4)
                words(I - 1) = words(I)
                wordsProbability(I - 1, 1) = wordsProbability(I, 1)
                wordsProbability(I - 1, 2) = wordsProbability(I, 2)
                wordsProbability(I - 1, 3) = wordsProbability(I, 3)
                wordsProbability(I - 1, 4) = wordsProbability(I, 4)
                words(I) = buffer$
                wordsProbability(I, 1) = buffer1
                wordsProbability(I, 2) = buffer2
                wordsProbability(I, 3) = buffer3
                wordsProbability(I, 4) = buffer4
                changed = 1
            End If
        Next I
    Loop Until changed = 0
End Sub
 
Sub shuffleWords
    For I = 2 To numWords
        If wordsProbability(I, 4) = wordsProbability(I - 1, 4) Then
            buffer$ = words(I - 1)
            buffer1 = wordsProbability(I - 1, 1)
            buffer2 = wordsProbability(I - 1, 2)
            buffer3 = wordsProbability(I - 1, 3)
            buffer4 = wordsProbability(I - 1, 4)
            words(I - 1) = words(I)
            wordsProbability(I - 1, 1) = wordsProbability(I, 1)
            wordsProbability(I - 1, 2) = wordsProbability(I, 2)
            wordsProbability(I - 1, 3) = wordsProbability(I, 3)
            wordsProbability(I - 1, 4) = wordsProbability(I, 4)
            words(I) = buffer$
            wordsProbability(I, 1) = buffer1
            wordsProbability(I, 2) = buffer2
            wordsProbability(I, 3) = buffer3
            wordsProbability(I, 4) = buffer4
        End If
    Next I
End Sub
 
Function wordPosition (searchedWord$)
    For I = 1 To numWords
        If words(I) = searchedWord$ Then
            wordPosition = I
            Exit For
        End If
    Next I
End Function
 
Sub collectWords
    oldstr$ = countedLinks(1, 1)
    j = 1
    For i = 2 To numCountedLinks
        If Not countedLinks(i, 1) = oldstr$ Then
            If Not isSpecial(oldstr$) Then
                words(j) = oldstr$
                j = j + 1
            End If
            oldstr$ = countedLinks(i, 1)
        End If
    Next i
    If Not isSpecial(oldstr$) Then
        words(j) = oldstr$
        numWords = j
    Else
        numWords = j - 1
    End If
End Sub
 
Function structureLikeness (myWord$, vocabularyWord$)
    Dim totDistance As Double: totDistance = 0
    For I = 1 To Len(myWord$)
        distance = charDistance(Mid$(myWord$, I, 1), vocabularyWord$, I)
        totDistance = totDistance + probability(distance, 0)
    Next I
    If Len(myWord$) = 0 Then
        structureLikeness = 0
    Else
        structureLikeness = totDistance / (Len(myWord$) * probability(0, 0))
    End If
End Function
 
Sub searchLinks (keyword$)
    For I = 1 To numCountedLinks
        searchedCountedLinks(I, 1) = ""
        searchedCountedLinks(I, 2) = ""
        searchedCountedLinks(I, 3) = ""
    Next I
    j = 0
    totSearchedCountedLinks = 0
    For I = 1 To numCountedLinks
        If countedLinks(I, 1) = keyword$ Then
            j = j + 1
            searchedCountedLinks(j, 1) = countedLinks(I, 1)
            searchedCountedLinks(j, 2) = countedLinks(I, 2)
            searchedCountedLinks(j, 3) = countedLinks(I, 3)
            totSearchedCountedLinks = totSearchedCountedLinks + Val(countedLinks(I, 3))
        End If
    Next I
    numSearchedCountedLinks = j
End Sub
 
Sub countLinks
    oldstr1$ = links(1, 1)
    oldstr2$ = links(1, 2)
    counter = 1
    j = 1
    For I = 2 To numLinks
        If links(I, 1) = oldstr1$ And links(I, 2) = oldstr2$ Then
            counter = counter + 1
        Else
            countedLinks(j, 1) = oldstr1$
            countedLinks(j, 2) = oldstr2$
            countedLinks(j, 3) = Str$(counter)
            oldstr1$ = links(I, 1)
            oldstr2$ = links(I, 2)
            j = j + 1
            counter = 1
        End If
    Next I
    countedLinks(j, 1) = oldstr1$
    countedLinks(j, 2) = oldstr2$
    countedLinks(j, 3) = Str$(counter)
    numCountedLinks = j
End Sub
 
Sub decomposeIntoLinks (sample$)
    sample$ = standardize$(sample$)
    Dim oldword As String
    Dim newword As String
    Dim extractedword As String
    For I = 1 To Len(sample$)
        j = InStr(I, sample$, " ")
        If j = 0 Then Exit For
        extractedword = Mid$(sample$, I - 1, j - I + 1)
        If Len(extracted) > 0 Then
            oldword = newword
            newword = extractedword
            If Not I = 1 Then
                row = row + 1
                links(row, 1) = _Trim$(oldword)
                links(row, 2) = _Trim$(newword)
            End If
        End If
        I = j + 1
    Next I
    numLinks = row
End Sub
 
Sub orderLinks
    Do
        changed = 0
        For I = 2 To numLinks
            result = compare(links(I - 1, 1) + Chr$(255) + links(I - 1, 2), links(I, 1) + Chr$(255) + links(I, 2))
            If result = 2 Then
                buffer1$ = links(I - 1, 1)
                buffer2$ = links(I - 1, 2)
                links(I - 1, 1) = links(I, 1)
                links(I - 1, 2) = links(I, 2)
                links(I, 1) = buffer1$
                links(I, 2) = buffer2$
                changed = 1
            End If
        Next I
    Loop Until changed = 0
End Sub
 
Function standardize$ (sample$)
    sample$ = "." + _Trim$(LCase$(sample$)) + ". "
    For i = 1 To Len(sample$)
        char$ = Mid$(sample$, i, 1)
        If isUnsupported(char$) Then char$ = " "
        If isPunctuation(char$) Then char$ = " " + char$ + " "
        standardize$ = standardize$ + char$
    Next i
End Function
 
Function destandardize$ (sample$, uppercase)
    If uppercase Then
        sample$ = UCase$(Left$(_Trim$(sample$), 1)) + Right$(_Trim$(sample$), Len(_Trim$(sample$)) - 1)
    Else
        sample$ = _Trim$(sample$)
    End If
    For i = 1 To Len(sample$)
        char$ = Mid$(sample$, i, 1)
        nextchar$ = Mid$(sample$, i + 1, 1)
        If char$ = " " And isPunctuation(nextchar$) Then
        ElseIf char$ = " " And nextchar$ = " " Then
        Else
            destandardize$ = destandardize$ + char$
        End If
        punctuation = isPunctuation(char$)
        If punctuation = -1 Then
            destandardize$ = destandardize$ + " " + destandardize$(Right$(sample$, Len(sample$) - i - 1), -1)
            Exit For
        ElseIf punctuation = -2 Then
            destandardize$ = destandardize$ + " " + destandardize$(Right$(sample$, Len(sample$) - i - 1), 0)
            Exit For
        End If
    Next i
End Function
 
Function probability (value, mean) ' with standard deviation = 2
    probability = cumulativeProbability(value + 0.5, mean, 2) - cumulativeProbability(value - 0.5, mean, 2)
End Function
 
Function cumulativeProbability (value, mean, deviation)
    value = (value - mean) / deviation
    adaptedvalue = Abs(Int(value * 100))
    If adaptedvalue <= 400 Then
        cumulativeProbability = GaussTable(adaptedvalue)
    Else
        cumulativeProbability = 1
    End If
    If value < 0 Then cumulativeProbability = 1 - cumulativeProbability
End Function
 
Sub fillGaussTable
    GaussTable(0) = 0.5000
    GaussTable(5) = 0.5199
    GaussTable(10) = 0.5398
    GaussTable(15) = 0.5596
    GaussTable(20) = 0.5793
    GaussTable(25) = 0.5987
    GaussTable(30) = 0.6179
    GaussTable(35) = 0.6368
    GaussTable(40) = 0.6554
    GaussTable(45) = 0.6736
    GaussTable(50) = 0.6915
    GaussTable(55) = 0.7088
    GaussTable(60) = 0.7257
    GaussTable(65) = 0.7421
    GaussTable(70) = 0.7580
    GaussTable(75) = 0.7734
    GaussTable(80) = 0.7881
    GaussTable(85) = 0.8023
    GaussTable(90) = 0.8159
    GaussTable(95) = 0.8289
    GaussTable(100) = 0.8413
    GaussTable(105) = 0.8531
    GaussTable(110) = 0.8643
    GaussTable(115) = 0.8749
    GaussTable(120) = 0.8849
    GaussTable(125) = 0.8944
    GaussTable(130) = 0.9032
    GaussTable(135) = 0.9115
    GaussTable(140) = 0.9192
    GaussTable(145) = 0.9265
    GaussTable(150) = 0.9332
    GaussTable(155) = 0.9394
    GaussTable(160) = 0.9452
    GaussTable(165) = 0.9505
    GaussTable(170) = 0.9554
    GaussTable(175) = 0.9599
    GaussTable(180) = 0.9641
    GaussTable(185) = 0.9678
    GaussTable(190) = 0.9713
    GaussTable(195) = 0.9744
    GaussTable(200) = 0.9772
    GaussTable(210) = 0.9821
    GaussTable(220) = 0.9861
    GaussTable(230) = 0.9893
    GaussTable(240) = 0.9918
    GaussTable(250) = 0.9938
    GaussTable(260) = 0.9953
    GaussTable(270) = 0.9965
    GaussTable(280) = 0.9974
    GaussTable(290) = 0.9981
    GaussTable(310) = 0.9990
    GaussTable(390) = 1
    GaussTable(400) = 1
    Dim lastValidValue As Double
    Dim lastValidValuePosition As Integer
    Dim nextValidValue As Double
    Dim nextValidValuePosition As Integer
    Dim interpolation As Double
    For i = 0 To 399
        If GaussTable(i) = 0 Then
            nextValidValue = 0
            j = i + 1
            While nextValidValue = 0
                If Not GaussTable(j) = 0 Then
                    nextValidValue = GaussTable(j)
                    nextValidValuePosition = j
                Else
                    j = j + 1
                End If
            Wend
            interpolation = Int((nextValidValue - lastValidValue) * (i - lastValidValuePosition) / (nextValidValuePosition - lastValidValuePosition) * 10 ^ 4) / 10 ^ 4
            GaussTable(i) = lastValidValue + interpolation
        Else
            lastValidValue = GaussTable(i)
            lastValidValuePosition = i
        End If
    Next i
End Sub
 
Function compare (str1$, str2$) ' returns 0 if str1$ = str2$, 1 if str1$ < str2$, 2 if str1$ > str2$
    If Len(str1$) < Len(str2$) Then compare = 1
    If Len(str1$) > Len(str2$) Then compare = 2
    For I = 1 To min(Len(str1$), Len(str2$))
        If Asc(Mid$(str1$, I, 1)) > Asc(Mid$(str2$, I, 1)) Then
            compare = 2
            Exit For
        End If
        If Asc(Mid$(str1$, I, 1)) < Asc(Mid$(str2$, I, 1)) Then
            compare = 1
            Exit For
        End If
    Next I
End Function
 
Function charDistance (char$, word$, from)
    charDistance = 10
    For I = 0 To min(max(from - 1, Len(word$) - from), 10)
        If from - I > 0 Then
            If Mid$(word$, from - I, 1) = char$ Then
                charDistance = I
                Exit For
            End If
        End If
        If from + I <= Len(word$) Then
            If Mid$(word$, from + I, 1) = char$ Then
                charDistance = I
                Exit For
            End If
        End If
    Next I
End Function
 
Function isSpecial (word$)
    isSpecial = 0
    If isPunctuation(word$) Or word$ = " " Or word$ = "" Then isSpecial = -1
End Function
 
Function isPunctuation (char$) ' returns -1 if next word is uppercase, otherwise returns -2
    isPunctuation = 0
    If char$ = "." Or char$ = "!" Or char$ = "?" Then isPunctuation = -1
    If char$ = "," Or char$ = ":" Or char$ = ";" Then isPunctuation = -2
End Function
 
Function isUnsupported (char$)
    isUnsupported = 0
    If char$ = "“" Or char$ = "”" Or char$ = Chr$(13) Or char$ = Chr$(0) Or char$ = "-" Or char$ = Chr$(34) Or char$ = "/" Or char$ = "(" Or char$ = ")" Or char$ = "^" Or char$ = "[" Or char$ = "]" Or char$ = "{" Or char$ = "}" Or char$ = "_" Or char$ = "<" Or char$ = ">" Then isUnsupported = -1
End Function
 
Function min (int1, int2)
    If int1 < int2 Then
        min = int1
    Else
        min = int2
    End If
End Function
 
Function max (int1, int2)
    If int1 > int2 Then
        max = int1
    Else
        max = int2
    End If
End Function

Pete · « **Reply #1 on:** October 12, 2021, 06:03:45 pm »

I'm not sure what to think of this. I mean given a long enough vocabulary list, maybe; but the sample gets very little correct and when you try to load large text files, it either cannot handle strings in excess of 10,000 characters or it just takes too long to process. With a text file with a string length of 5308 it took 10-seconds to process and got about 60% correct. For instance...

The days are getting shorter and the nights are getting longer.

was "corrected" to...

The days are getting more and the lights are getting more.

8 for 11. The words nights, longer, and shorter were not in the vocab list. So, not bad, but how did it miss nights? It was in the vocab list.

Next...

i'm a Yankee Doodle Dandy got "corrected" to

i'm a leaker people hands.

2 for 5 and it did not capitalize the i in i'm. You guessed it, Yankee, Doodle, and Dandy were not in the vocab list.

Anyway...

The post reminded me of working with the Levenshtein distance string metric in my word processor to add spell check. The thing is, I had to include an extensive hash file to get reasonable results. To actually use this to get your computer to learn any language is not going to happen without a much faster way to load and analyze a word list.

Pete

Sanmayce · « **Reply #2 on:** October 12, 2021, 08:47:05 pm »

Hi PoliMi,
it's nice to see another text auto-completion fan here.
Please consider making something like a benchmark of yours in order to show what are the speeds, memory requirements and so on.

As far as I see, the ultimate task is to make a phrase-checker (being not a mere spell-checker dealing with 1-grams i.e. single words) analyzing 2-grams, 3-grams and 4-grams. In next year, will reupdate my old phrase-checking package and consolidate the code with my Masakari tool.

Few months back, I played with a small English article (80 lines long, 28KB), the package is attached for all who want to play with it. It could serve as a staring point for examples/tests. My textual madness knows no limits, the end game is to have corpora of already ripped unigrams, bigrams, trigrams, tetragrams, pentagrams and to use them as a reference phrase-check file) against a given file, not (as in the example below) the file itself to be used as a self-checking reference.

The console dump below shows that in less than 1 minute (not that bad)
3:35:40.04
3:34:44.20
we obtain the occurrences of all 1-grams up to 5-grams housing 'take':

Code: QB64: [Select]

E:\z\JRJ>dir
 Volume in drive E is Sanmayce_111GB
 Volume Serial Number is 1410-10F9
 
 Directory of E:\z\JRJ
 
10/13/2021  03:34 AM    <DIR>          .
10/13/2021  03:34 AM    <DIR>          ..
04/22/2021  05:36 AM    <DIR>          JRJ_essay
04/22/2021  05:37 AM           109,582 Kazahana_Hexadecad_GCC_102_32bit.exe
04/22/2021  05:37 AM         2,494,334 Kazahana_sources_binaries.zip
04/22/2021  05:37 AM           135,680 Leprechaun_x-leton_32bit_Intel_01_001p.exe
04/22/2021  05:37 AM           134,144 Leprechaun_x-leton_32bit_Intel_01_008p.exe
04/22/2021  05:37 AM           134,144 Leprechaun_x-leton_32bit_Intel_01_512p.exe
04/22/2021  05:37 AM           139,264 Leprechaun_x-leton_32bit_Intel_02_001p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_02_008p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_02_512p.exe
04/22/2021  05:37 AM           139,264 Leprechaun_x-leton_32bit_Intel_03_001p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_03_008p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_03_512p.exe
04/22/2021  05:37 AM           138,240 Leprechaun_x-leton_32bit_Intel_04_001p.exe
04/22/2021  05:37 AM           136,704 Leprechaun_x-leton_32bit_Intel_04_008p.exe
04/22/2021  05:37 AM           136,704 Leprechaun_x-leton_32bit_Intel_04_512p.exe
04/22/2021  05:37 AM           139,776 Leprechaun_x-leton_32bit_Intel_05_001p.exe
04/22/2021  05:37 AM           138,240 Leprechaun_x-leton_32bit_Intel_05_008p.exe
04/22/2021  05:37 AM           138,240 Leprechaun_x-leton_32bit_Intel_05_512p.exe
04/22/2021  05:37 AM           695,925 Leprechaun_x-leton_r17tag.7z
04/22/2021  05:37 AM            96,993 LineJustify_PAGODAo5.c
04/22/2021  05:37 AM            73,728 LineJustify_PAGODAo5.exe
04/22/2021  05:37 AM            94,162 LineWordreporter.c
04/22/2021  05:37 AM            77,312 LineWordreporter.exe
04/22/2021  05:37 AM             1,633 MokujIN JADE 217 prompt.lnk
04/22/2021  05:37 AM            28,393 The_Task_of_a_Philosopher.txt
04/22/2021  05:37 AM             3,335 XGRAM_PAGODA5.bat
04/22/2021  05:37 AM             2,181 XGRAM_RIP_file.bat
04/22/2021  05:37 AM            35,015 Yoshi.exe
04/22/2021  05:37 AM           969,892 Yoshi7-.zip
              28 File(s)      6,743,797 bytes
               3 Dir(s)   3,712,290,816 bytes free
 
E:\z\JRJ>time
The current time is:  3:34:44.20
Enter the new time:
 
E:\z\JRJ>XGRAM_RIP_file.bat The_Task_of_a_Philosopher.txt
 
E:\z\JRJ>set TMP=d:\
 
E:\z\JRJ>set TEMP=d:\
 
E:\z\JRJ>Leprechaun_x-leton_32bit_Intel_01_001p.exe The_Task_of_a_Philosopher.txt.lst The_Task_of_a_Philosopher.txt.01 1123123 Y
Leprechaun_singleton (Fast-In-Future Greedy n-gram-Ripper), rev. 17, written by Svalqyatchx.
Purpose: Rips all distinct 1-grams (1-word phrases) with length 1..31 chars from incoming texts.
Feature1: All words within x-lets/n-grams are in range 1..31 chars inclusive.
Feature2: In this revision 512MB 1-way hash is used which results in 67,108,864 external B-Trees of order 3.
Feature3: In this revision, 1 pass is to be made.
Feature4: If the external memory has latency 99+microseconds then !(look no further), IOPS(seek-time) rules.
Pass #1 of 1:
Size of input file with files for Leprechauning: 31
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1097MB ... OK
Size of Input TEXTual file: 28,393
/; 00,004,391P/s; Phrase count: 4,391 of them 1,313 distinct; Done: 64/64
Bytes per second performance: 28,393B/s
Phrases per second performance: 4,391P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,002,626P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
 
Total memory needed for one pass: 124KB
Total distinct phrases: 1,313
Total time: 1 second(s)
Total performance: 4,391P/s i.e. phrases per second
Leprechaun: Done.
 
E:\z\JRJ>Leprechaun_x-leton_32bit_Intel_02_001p.exe The_Task_of_a_Philosopher.txt.lst The_Task_of_a_Philosopher.txt.02 1123123 Y
Leprechaun_doubleton (Fast-In-Future Greedy n-gram-Ripper), rev. 17, written by Svalqyatchx.
Purpose: Rips all distinct 2-grams (2-word phrases) with length 5..41 chars from incoming texts.
Feature1: All words within x-lets/n-grams are in range 1..31 chars inclusive.
Feature2: In this revision 512MB 1-way hash is used which results in 67,108,864 external B-Trees of order 3.
Feature3: In this revision, 1 pass is to be made.
Feature4: If the external memory has latency 99+microseconds then !(look no further), IOPS(seek-time) rules.
Pass #1 of 1:
Size of input file with files for Leprechauning: 31
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1097MB ... OK
Size of Input TEXTual file: 28,393
-; 00,003,929P/s; Phrase count: 3,929 of them 3,133 distinct; Done: 64/64
Bytes per second performance: 28,393B/s
Phrases per second performance: 3,929P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,006,266P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
 
Total memory needed for one pass: 355KB
Total distinct phrases: 3,133
Total time: 1 second(s)
Total performance: 3,929P/s i.e. phrases per second
Leprechaun: Done.
 
E:\z\JRJ>Leprechaun_x-leton_32bit_Intel_03_001p.exe The_Task_of_a_Philosopher.txt.lst The_Task_of_a_Philosopher.txt.03 1123123 Y
Leprechaun_tripleton (Fast-In-Future Greedy n-gram-Ripper), rev. 17, written by Svalqyatchx.
Purpose: Rips all distinct 3-grams (3-word phrases) with length 9..41 chars from incoming texts.
Feature1: All words within x-lets/n-grams are in range 1..31 chars inclusive.
Feature2: In this revision 512MB 1-way hash is used which results in 67,108,864 external B-Trees of order 3.
Feature3: In this revision, 1 pass is to be made.
Feature4: If the external memory has latency 99+microseconds then !(look no further), IOPS(seek-time) rules.
Pass #1 of 1:
Size of input file with files for Leprechauning: 31
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1097MB ... OK
Size of Input TEXTual file: 28,393
\; 00,003,574P/s; Phrase count: 3,574 of them 3,430 distinct; Done: 64/64
Bytes per second performance: 28,393B/s
Phrases per second performance: 3,574P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,006,860P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
 
Total memory needed for one pass: 389KB
Total distinct phrases: 3,430
Total time: 1 second(s)
Total performance: 3,574P/s i.e. phrases per second
Leprechaun: Done.
 
E:\z\JRJ>Leprechaun_x-leton_32bit_Intel_04_001p.exe The_Task_of_a_Philosopher.txt.lst The_Task_of_a_Philosopher.txt.04 1123123 Y
Leprechaun_quadrupleton (Fast-In-Future Greedy n-gram-Ripper), rev. 17, written by Svalqyatchx.
Purpose: Rips all distinct 4-grams (4-word phrases) with length 13..51 chars from incoming texts.
Feature1: All words within x-lets/n-grams are in range 1..31 chars inclusive.
Feature2: In this revision 512MB 1-way hash is used which results in 67,108,864 external B-Trees of order 3.
Feature3: In this revision, 1 pass is to be made.
Feature4: If the external memory has latency 99+microseconds then !(look no further), IOPS(seek-time) rules.
Pass #1 of 1:
Size of input file with files for Leprechauning: 31
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1097MB ... OK
Size of Input TEXTual file: 28,393
|; 00,003,238P/s; Phrase count: 3,238 of them 3,205 distinct; Done: 64/64
Bytes per second performance: 28,393B/s
Phrases per second performance: 3,238P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,006,410P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
 
Total memory needed for one pass: 426KB
Total distinct phrases: 3,205
Total time: 1 second(s)
Total performance: 3,238P/s i.e. phrases per second
Leprechaun: Done.
 
E:\z\JRJ>Leprechaun_x-leton_32bit_Intel_05_001p.exe The_Task_of_a_Philosopher.txt.lst The_Task_of_a_Philosopher.txt.05 1123123 Y
Leprechaun_quintupleton (Fast-In-Future Greedy n-gram-Ripper), rev. 17, written by Svalqyatchx.
Purpose: Rips all distinct 5-grams (5-word phrases) with length 17..61 chars from incoming texts.
Feature1: All words within x-lets/n-grams are in range 1..31 chars inclusive.
Feature2: In this revision 512MB 1-way hash is used which results in 67,108,864 external B-Trees of order 3.
Feature3: In this revision, 1 pass is to be made.
Feature4: If the external memory has latency 99+microseconds then !(look no further), IOPS(seek-time) rules.
Pass #1 of 1:
Size of input file with files for Leprechauning: 31
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1097MB ... OK
Size of Input TEXTual file: 28,393
/; 00,002,913P/s; Phrase count: 2,913 of them 2,905 distinct; Done: 64/64
Bytes per second performance: 28,393B/s
Phrases per second performance: 2,913P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,005,810P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
 
Total memory needed for one pass: 443KB
Total distinct phrases: 2,905
Total time: 1 second(s)
Total performance: 2,913P/s i.e. phrases per second
Leprechaun: Done.
 
E:\z\JRJ>sort /+10 /M 1012012 The_Task_of_a_Philosopher.txt.01 /O The_Task_of_a_Philosopher.txt.01.txt
 
E:\z\JRJ>sort /+10 /M 1012012 The_Task_of_a_Philosopher.txt.02 /O The_Task_of_a_Philosopher.txt.02.txt
 
E:\z\JRJ>sort /+10 /M 1012012 The_Task_of_a_Philosopher.txt.03 /O The_Task_of_a_Philosopher.txt.03.txt
 
E:\z\JRJ>sort /+10 /M 1012012 The_Task_of_a_Philosopher.txt.04 /O The_Task_of_a_Philosopher.txt.04.txt
 
E:\z\JRJ>sort /+10 /M 1012012 The_Task_of_a_Philosopher.txt.05 /O The_Task_of_a_Philosopher.txt.05.txt
 
 Volume in drive E is Sanmayce_111GB
 Volume Serial Number is 1410-10F9
 
 Directory of E:\z\JRJ
 
10/13/2021  03:34 AM            25,290 The_Task_of_a_Philosopher.txt.01.txt
10/13/2021  03:34 AM            74,892 The_Task_of_a_Philosopher.txt.02.txt
10/13/2021  03:34 AM           101,334 The_Task_of_a_Philosopher.txt.03.txt
10/13/2021  03:34 AM           113,997 The_Task_of_a_Philosopher.txt.04.txt
10/13/2021  03:34 AM           120,996 The_Task_of_a_Philosopher.txt.05.txt
               5 File(s)        436,509 bytes
               0 Dir(s)   3,711,840,256 bytes free
 
E:\z\JRJ>copy The_Task_of_a_Philosopher.txt.01.txt Gallowwalker.1.txt.sorted
        1 file(s) copied.
 
E:\z\JRJ>copy The_Task_of_a_Philosopher.txt.02.txt Gallowwalker.2.txt.sorted
        1 file(s) copied.
 
E:\z\JRJ>copy The_Task_of_a_Philosopher.txt.03.txt Gallowwalker.3.txt.sorted
        1 file(s) copied.
 
E:\z\JRJ>copy The_Task_of_a_Philosopher.txt.04.txt Gallowwalker.4.txt.sorted
        1 file(s) copied.
 
E:\z\JRJ>copy The_Task_of_a_Philosopher.txt.05.txt Gallowwalker.5.txt.sorted
        1 file(s) copied.
 
E:\z\JRJ>dir
 Volume in drive E is Sanmayce_111GB
 Volume Serial Number is 1410-10F9
 
 Directory of E:\z\JRJ
 
10/13/2021  03:34 AM    <DIR>          .
10/13/2021  03:34 AM    <DIR>          ..
10/13/2021  03:34 AM            25,290 Gallowwalker.1.txt.sorted
10/13/2021  03:34 AM            74,892 Gallowwalker.2.txt.sorted
10/13/2021  03:34 AM           101,334 Gallowwalker.3.txt.sorted
10/13/2021  03:34 AM           113,997 Gallowwalker.4.txt.sorted
10/13/2021  03:34 AM           120,996 Gallowwalker.5.txt.sorted
04/22/2021  05:36 AM    <DIR>          JRJ_essay
04/22/2021  05:37 AM           109,582 Kazahana_Hexadecad_GCC_102_32bit.exe
04/22/2021  05:37 AM         2,494,334 Kazahana_sources_binaries.zip
04/22/2021  05:37 AM           135,680 Leprechaun_x-leton_32bit_Intel_01_001p.exe
04/22/2021  05:37 AM           134,144 Leprechaun_x-leton_32bit_Intel_01_008p.exe
04/22/2021  05:37 AM           134,144 Leprechaun_x-leton_32bit_Intel_01_512p.exe
04/22/2021  05:37 AM           139,264 Leprechaun_x-leton_32bit_Intel_02_001p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_02_008p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_02_512p.exe
04/22/2021  05:37 AM           139,264 Leprechaun_x-leton_32bit_Intel_03_001p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_03_008p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_03_512p.exe
04/22/2021  05:37 AM           138,240 Leprechaun_x-leton_32bit_Intel_04_001p.exe
04/22/2021  05:37 AM           136,704 Leprechaun_x-leton_32bit_Intel_04_008p.exe
04/22/2021  05:37 AM           136,704 Leprechaun_x-leton_32bit_Intel_04_512p.exe
04/22/2021  05:37 AM           139,776 Leprechaun_x-leton_32bit_Intel_05_001p.exe
04/22/2021  05:37 AM           138,240 Leprechaun_x-leton_32bit_Intel_05_008p.exe
04/22/2021  05:37 AM           138,240 Leprechaun_x-leton_32bit_Intel_05_512p.exe
04/22/2021  05:37 AM           695,925 Leprechaun_x-leton_r17tag.7z
04/22/2021  05:37 AM            96,993 LineJustify_PAGODAo5.c
04/22/2021  05:37 AM            73,728 LineJustify_PAGODAo5.exe
04/22/2021  05:37 AM            94,162 LineWordreporter.c
04/22/2021  05:37 AM            77,312 LineWordreporter.exe
04/22/2021  05:37 AM             1,633 MokujIN JADE 217 prompt.lnk
04/22/2021  05:37 AM            28,393 The_Task_of_a_Philosopher.txt
10/13/2021  03:34 AM            25,290 The_Task_of_a_Philosopher.txt.01.txt
10/13/2021  03:34 AM            74,892 The_Task_of_a_Philosopher.txt.02.txt
10/13/2021  03:34 AM           101,334 The_Task_of_a_Philosopher.txt.03.txt
10/13/2021  03:34 AM           113,997 The_Task_of_a_Philosopher.txt.04.txt
10/13/2021  03:34 AM           120,996 The_Task_of_a_Philosopher.txt.05.txt
04/22/2021  05:37 AM             3,335 XGRAM_PAGODA5.bat
04/22/2021  05:37 AM             2,181 XGRAM_RIP_file.bat
04/22/2021  05:37 AM            35,015 Yoshi.exe
04/22/2021  05:37 AM           969,892 Yoshi7-.zip
              38 File(s)      7,616,815 bytes
               3 Dir(s)   3,711,393,792 bytes free
 
E:\z\JRJ>time
The current time is:  3:34:58.06
Enter the new time:
 
E:\z\JRJ>XGRAM_PAGODA5.bat take
 
E:\z\JRJ>if exist Gallowwalker.1.txt.sorted goto OK1
 
E:\z\JRJ>if exist Gallowwalker.2.txt.sorted goto OK2
 
E:\z\JRJ>if exist Gallowwalker.3.txt.sorted goto OK3
 
E:\z\JRJ>if exist Gallowwalker.4.txt.sorted goto OK4
 
E:\z\JRJ>if exist Gallowwalker.5.txt.sorted goto OK5
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`take" Gallowwalker.1.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `take
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 25,290 bytes
Kazahana: Total/Checked/Dumped xgrams: 1,313/1,313/1
Kazahana: Performance: 0 KB/clock
Kazahana: Performance: 16 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.1-1.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`take_." Gallowwalker.2.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `take_.
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 74,892 bytes
Kazahana: Total/Checked/Dumped xgrams: 3,133/3,133/4
Kazahana: Performance: 0 KB/clock
Kazahana: Performance: 32 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 95/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.2-1.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`._take" Gallowwalker.2.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `._take
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 74,892 bytes
Kazahana: Total/Checked/Dumped xgrams: 3,133/3,133/4
Kazahana: Performance: 0 KB/clock
Kazahana: Performance: 39 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.2-2.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`take_._." Gallowwalker.3.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `take_._.
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 101,334 bytes
Kazahana: Total/Checked/Dumped xgrams: 3,430/3,430/3
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 43 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.3-1.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`._take_." Gallowwalker.3.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `._take_.
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 101,334 bytes
Kazahana: Total/Checked/Dumped xgrams: 3,430/3,430/4
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 43 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.3-2.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`._._take" Gallowwalker.3.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `._._take
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 101,334 bytes
Kazahana: Total/Checked/Dumped xgrams: 3,430/3,430/4
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 43 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.3-3.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`take_._._." Gallowwalker.4.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `take_._._.
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 113,997 bytes
Kazahana: Total/Checked/Dumped xgrams: 3,205/3,205/2
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 40 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.4-1.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`._take_._." Gallowwalker.4.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `._take_._.
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 113,997 bytes
Kazahana: Total/Checked/Dumped xgrams: 3,205/3,205/3
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 40 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.4-2.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`._._take_." Gallowwalker.4.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `._._take_.
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 113,997 bytes
Kazahana: Total/Checked/Dumped xgrams: 3,205/3,205/4
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 40 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.4-3.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`._._._take" Gallowwalker.4.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `._._._take
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 113,997 bytes
Kazahana: Total/Checked/Dumped xgrams: 3,205/3,205/4
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 40 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.4-4.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`take_._._._." Gallowwalker.5.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `take_._._._.
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 120,996 bytes
Kazahana: Total/Checked/Dumped xgrams: 2,905/2,905/2
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 36 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.5-1.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`._take_._._." Gallowwalker.5.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `._take_._._.
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 120,996 bytes
Kazahana: Total/Checked/Dumped xgrams: 2,905/2,905/2
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 36 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.5-2.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`._._take_._." Gallowwalker.5.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `._._take_._.
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 120,996 bytes
Kazahana: Total/Checked/Dumped xgrams: 2,905/2,905/3
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 36 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.5-3.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`._._._take_." Gallowwalker.5.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `._._._take_.
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 120,996 bytes
Kazahana: Total/Checked/Dumped xgrams: 2,905/2,905/4
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 36 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.5-4.txt
 
E:\z\JRJ>Kazahana_Hexadecad_GCC_102_32bit.exe "`._._._._take" Gallowwalker.5.txt.sorted 1023
Kazahana, a typhoon-class exact & wildcards & Levenshtein Distance (Wagner-Fischer) searcher, r. 1-++fix+nowait_critical_nixFIX_Wolfram+fixITER+EX+CS_fix_DEFINE_Trolldom, copyleft Kaze 2019-May-21.
Enforcing Case Sensitive wildcard mode ...
Enforcing SLOW wildcard mode ...
Current priority class is REALTIME_PRIORITY_CLASS.
Pattern: `._._._._take
omp_get_num_procs( ) = 2
omp_get_max_threads( ) = 2
Enforcing HEXADECAD i.e. hexadecuple-threads ...
Allocating Master-Buffer 1023KB ... OK
|; Speed: 00,000,000,000 bytes/clock; Traversed: 120,996 bytes
Kazahana: Total/Checked/Dumped xgrams: 2,905/2,905/3
Kazahana: Performance: 1 KB/clock
Kazahana: Performance: 36 xgrams/clock
Kazahana: Performance: Total/fread() clocks: 79/0
Kazahana: Performance: I/O time, i.e. fread() time, is 0 percents
Kazahana: Done.
 
E:\z\JRJ>sort /R Kazahana.txt /O Kazahana_take.5-5.txt
 
E:\z\JRJ>dir Kazahana_take.*.txt/b/on 1>q
 
E:\z\JRJ>LineJustify_PAGODAo5.exe q
LineJustify_PAGODAo5, revision 1, written by Kaze.
Purpose: Padding the left side of x-grams with SPACEs in order to form the main pillar.
Example:
D:\>LineWordreporter.exe LineJustify_PAGODAo5.lst
Note: Files can exceed 4GB limit.
Buffered PADDing ...
 
E:\z\JRJ>copy Kazahana_take.*.txt.PAD Kazahana_take.PAGODA-order-5.txt/b/y
Kazahana_take.1-1.txt.PAD
Kazahana_take.2-1.txt.PAD
Kazahana_take.2-2.txt.PAD
Kazahana_take.3-1.txt.PAD
Kazahana_take.3-2.txt.PAD
Kazahana_take.3-3.txt.PAD
Kazahana_take.4-1.txt.PAD
Kazahana_take.4-2.txt.PAD
Kazahana_take.4-3.txt.PAD
Kazahana_take.4-4.txt.PAD
Kazahana_take.5-1.txt.PAD
Kazahana_take.5-2.txt.PAD
Kazahana_take.5-3.txt.PAD
Kazahana_take.5-4.txt.PAD
Kazahana_take.5-5.txt.PAD
        1 file(s) copied.
 
E:\z\JRJ>LineWordreporter.exe q 1>>schisch.log
 
E:\z\JRJ>echo.1>>schisch.log
 
E:\z\JRJ>Leprechaun_x-leton_32bit_Intel_01_008p.exe q q.wrd 1234567 Y
Leprechaun_singleton (Fast-In-Future Greedy n-gram-Ripper), rev. 17, written by Svalqyatchx.
Purpose: Rips all distinct 1-grams (1-word phrases) with length 1..31 chars from incoming texts.
Feature1: All words within x-lets/n-grams are in range 1..31 chars inclusive.
Feature2: In this revision 512MB 1-way hash is used which results in 67,108,864 external B-Trees of order 3.
Feature3: In this revision, 8 passes are to be made.
Feature4: If the external memory has latency 99+microseconds then !(look no further), IOPS(seek-time) rules.
Pass #1 of 8:
Size of input file with files for Leprechauning: 345
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1206MB ... OK
Size of Input TEXTual file: 16
/; 00,000,001P/s; Phrase count: 1 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 81
-; 00,000,009P/s; Phrase count: 9 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 87
\; 00,000,017P/s; Phrase count: 17 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 75
|; 00,000,026P/s; Phrase count: 26 of them 3 distinct; Done: 64/64
Size of Input TEXTual file: 104
/; 00,000,038P/s; Phrase count: 38 of them 3 distinct; Done: 64/64
Size of Input TEXTual file: 121
-; 00,000,050P/s; Phrase count: 50 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 59
\; 00,000,058P/s; Phrase count: 58 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 89
|; 00,000,070P/s; Phrase count: 70 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 138
/; 00,000,086P/s; Phrase count: 86 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 148
-; 00,000,102P/s; Phrase count: 102 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 65
\; 00,000,112P/s; Phrase count: 112 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 69
|; 00,000,122P/s; Phrase count: 122 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 112
/; 00,000,137P/s; Phrase count: 137 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 165
-; 00,000,157P/s; Phrase count: 157 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 133
\; 00,000,172P/s; Phrase count: 172 of them 5 distinct; Done: 64/64
Bytes per second performance: 1,462B/s
Phrases per second performance: 172P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,000,010P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
Pass #2 of 8:
Size of input file with files for Leprechauning: 345
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1206MB ... OK
Size of Input TEXTual file: 16
/; 00,000,001P/s; Phrase count: 1 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 81
-; 00,000,009P/s; Phrase count: 9 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 87
\; 00,000,017P/s; Phrase count: 17 of them 3 distinct; Done: 64/64
Size of Input TEXTual file: 75
|; 00,000,026P/s; Phrase count: 26 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 104
/; 00,000,038P/s; Phrase count: 38 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 121
-; 00,000,050P/s; Phrase count: 50 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 59
\; 00,000,058P/s; Phrase count: 58 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 89
|; 00,000,070P/s; Phrase count: 70 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 138
/; 00,000,086P/s; Phrase count: 86 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 148
-; 00,000,102P/s; Phrase count: 102 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 65
\; 00,000,112P/s; Phrase count: 112 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 69
|; 00,000,122P/s; Phrase count: 122 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 112
/; 00,000,137P/s; Phrase count: 137 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 165
-; 00,000,157P/s; Phrase count: 157 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 133
\; 00,000,172P/s; Phrase count: 172 of them 4 distinct; Done: 64/64
Bytes per second performance: 1,462B/s
Phrases per second performance: 172P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,000,008P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
Pass #3 of 8:
Size of input file with files for Leprechauning: 345
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1206MB ... OK
Size of Input TEXTual file: 16
/; 00,000,001P/s; Phrase count: 1 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 81
-; 00,000,009P/s; Phrase count: 9 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 87
\; 00,000,017P/s; Phrase count: 17 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 75
|; 00,000,026P/s; Phrase count: 26 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 104
/; 00,000,038P/s; Phrase count: 38 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 121
-; 00,000,050P/s; Phrase count: 50 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 59
\; 00,000,058P/s; Phrase count: 58 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 89
|; 00,000,070P/s; Phrase count: 70 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 138
/; 00,000,086P/s; Phrase count: 86 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 148
-; 00,000,102P/s; Phrase count: 102 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 65
\; 00,000,112P/s; Phrase count: 112 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 69
|; 00,000,122P/s; Phrase count: 122 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 112
/; 00,000,137P/s; Phrase count: 137 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 165
-; 00,000,157P/s; Phrase count: 157 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 133
\; 00,000,172P/s; Phrase count: 172 of them 2 distinct; Done: 64/64
Bytes per second performance: 1,462B/s
Phrases per second performance: 172P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,000,004P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
Pass #4 of 8:
Size of input file with files for Leprechauning: 345
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1206MB ... OK
Size of Input TEXTual file: 16
/; 00,000,001P/s; Phrase count: 1 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 81
-; 00,000,009P/s; Phrase count: 9 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 87
\; 00,000,017P/s; Phrase count: 17 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 75
|; 00,000,026P/s; Phrase count: 26 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 104
/; 00,000,038P/s; Phrase count: 38 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 121
-; 00,000,050P/s; Phrase count: 50 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 59
\; 00,000,058P/s; Phrase count: 58 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 89
|; 00,000,070P/s; Phrase count: 70 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 138
/; 00,000,086P/s; Phrase count: 86 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 148
-; 00,000,102P/s; Phrase count: 102 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 65
\; 00,000,112P/s; Phrase count: 112 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 69
|; 00,000,122P/s; Phrase count: 122 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 112
/; 00,000,137P/s; Phrase count: 137 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 165
-; 00,000,157P/s; Phrase count: 157 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 133
\; 00,000,172P/s; Phrase count: 172 of them 1 distinct; Done: 64/64
Bytes per second performance: 1,462B/s
Phrases per second performance: 172P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,000,002P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
Pass #5 of 8:
Size of input file with files for Leprechauning: 345
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1206MB ... OK
Size of Input TEXTual file: 16
/; 00,000,001P/s; Phrase count: 1 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 81
-; 00,000,009P/s; Phrase count: 9 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 87
\; 00,000,017P/s; Phrase count: 17 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 75
|; 00,000,026P/s; Phrase count: 26 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 104
/; 00,000,038P/s; Phrase count: 38 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 121
-; 00,000,050P/s; Phrase count: 50 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 59
\; 00,000,058P/s; Phrase count: 58 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 89
|; 00,000,070P/s; Phrase count: 70 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 138
/; 00,000,086P/s; Phrase count: 86 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 148
-; 00,000,102P/s; Phrase count: 102 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 65
\; 00,000,112P/s; Phrase count: 112 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 69
|; 00,000,122P/s; Phrase count: 122 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 112
/; 00,000,137P/s; Phrase count: 137 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 165
-; 00,000,157P/s; Phrase count: 157 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 133
\; 00,000,172P/s; Phrase count: 172 of them 3 distinct; Done: 64/64
Bytes per second performance: 1,462B/s
Phrases per second performance: 172P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,000,006P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
Pass #6 of 8:
Size of input file with files for Leprechauning: 345
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1206MB ... OK
Size of Input TEXTual file: 16
/; 00,000,001P/s; Phrase count: 1 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 81
-; 00,000,009P/s; Phrase count: 9 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 87
\; 00,000,017P/s; Phrase count: 17 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 75
|; 00,000,026P/s; Phrase count: 26 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 104
/; 00,000,038P/s; Phrase count: 38 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 121
-; 00,000,050P/s; Phrase count: 50 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 59
\; 00,000,058P/s; Phrase count: 58 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 89
|; 00,000,070P/s; Phrase count: 70 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 138
/; 00,000,086P/s; Phrase count: 86 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 148
-; 00,000,102P/s; Phrase count: 102 of them 3 distinct; Done: 64/64
Size of Input TEXTual file: 65
\; 00,000,112P/s; Phrase count: 112 of them 3 distinct; Done: 64/64
Size of Input TEXTual file: 69
|; 00,000,122P/s; Phrase count: 122 of them 3 distinct; Done: 64/64
Size of Input TEXTual file: 112
/; 00,000,137P/s; Phrase count: 137 of them 3 distinct; Done: 64/64
Size of Input TEXTual file: 165
-; 00,000,157P/s; Phrase count: 157 of them 3 distinct; Done: 64/64
Size of Input TEXTual file: 133
\; 00,000,172P/s; Phrase count: 172 of them 4 distinct; Done: 64/64
Bytes per second performance: 1,462B/s
Phrases per second performance: 172P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,000,008P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
Pass #7 of 8:
Size of input file with files for Leprechauning: 345
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1206MB ... OK
Size of Input TEXTual file: 16
/; 00,000,001P/s; Phrase count: 1 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 81
-; 00,000,009P/s; Phrase count: 9 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 87
\; 00,000,017P/s; Phrase count: 17 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 75
|; 00,000,026P/s; Phrase count: 26 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 104
/; 00,000,038P/s; Phrase count: 38 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 121
-; 00,000,050P/s; Phrase count: 50 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 59
\; 00,000,058P/s; Phrase count: 58 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 89
|; 00,000,070P/s; Phrase count: 70 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 138
/; 00,000,086P/s; Phrase count: 86 of them 2 distinct; Done: 64/64
Size of Input TEXTual file: 148
-; 00,000,102P/s; Phrase count: 102 of them 4 distinct; Done: 64/64
Size of Input TEXTual file: 65
\; 00,000,112P/s; Phrase count: 112 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 69
|; 00,000,122P/s; Phrase count: 122 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 112
/; 00,000,137P/s; Phrase count: 137 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 165
-; 00,000,157P/s; Phrase count: 157 of them 5 distinct; Done: 64/64
Size of Input TEXTual file: 133
\; 00,000,172P/s; Phrase count: 172 of them 5 distinct; Done: 64/64
Bytes per second performance: 1,462B/s
Phrases per second performance: 172P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,000,010P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
Pass #8 of 8:
Size of input file with files for Leprechauning: 345
Allocating HASH memory 536,870,977 bytes ... OK
Allocating memory 1206MB ... OK
Size of Input TEXTual file: 16
/; 00,000,001P/s; Phrase count: 1 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 81
-; 00,000,009P/s; Phrase count: 9 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 87
\; 00,000,017P/s; Phrase count: 17 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 75
|; 00,000,026P/s; Phrase count: 26 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 104
/; 00,000,038P/s; Phrase count: 38 of them 0 distinct; Done: 64/64
Size of Input TEXTual file: 121
-; 00,000,050P/s; Phrase count: 50 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 59
\; 00,000,058P/s; Phrase count: 58 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 89
|; 00,000,070P/s; Phrase count: 70 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 138
/; 00,000,086P/s; Phrase count: 86 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 148
-; 00,000,102P/s; Phrase count: 102 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 65
\; 00,000,112P/s; Phrase count: 112 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 69
|; 00,000,122P/s; Phrase count: 122 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 112
/; 00,000,137P/s; Phrase count: 137 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 165
-; 00,000,157P/s; Phrase count: 157 of them 1 distinct; Done: 64/64
Size of Input TEXTual file: 133
\; 00,000,172P/s; Phrase count: 172 of them 1 distinct; Done: 64/64
Bytes per second performance: 1,462B/s
Phrases per second performance: 172P/s
Time for putting phrases into trees: 1 second(s)
Flushing UNsorted phrases: 100%; Shaking trees performance: 00,000,002P/s
Time for shaking phrases from trees: 1 second(s)
Leprechaun: Current pass done.
 
Total memory needed for one pass: 8KB
Total distinct phrases: 25
Total time: 9 second(s)
Total performance: 19P/s i.e. phrases per second
Leprechaun: Done.
 
E:\z\JRJ>sort /R q.wrd /O Kazahana_take.PAGODA-order-5.wrd
 
E:\z\JRJ>dir Kazahana_take.* 1>>schisch.log
 
E:\z\JRJ>echo.1>>schisch.log
E:\z\JRJ>dir
 Volume in drive E is Sanmayce_111GB
 Volume Serial Number is 1410-10F9
 
 Directory of E:\z\JRJ
 
10/13/2021  03:35 AM    <DIR>          .
10/13/2021  03:35 AM    <DIR>          ..
10/13/2021  03:34 AM            25,290 Gallowwalker.1.txt.sorted
10/13/2021  03:34 AM            74,892 Gallowwalker.2.txt.sorted
10/13/2021  03:34 AM           101,334 Gallowwalker.3.txt.sorted
10/13/2021  03:34 AM           113,997 Gallowwalker.4.txt.sorted
10/13/2021  03:34 AM           120,996 Gallowwalker.5.txt.sorted
04/22/2021  05:36 AM    <DIR>          JRJ_essay
04/22/2021  05:37 AM           109,582 Kazahana_Hexadecad_GCC_102_32bit.exe
04/22/2021  05:37 AM         2,494,334 Kazahana_sources_binaries.zip
10/13/2021  03:35 AM                16 Kazahana_take.1-1.txt
10/13/2021  03:35 AM                81 Kazahana_take.2-1.txt
10/13/2021  03:35 AM                87 Kazahana_take.2-2.txt
10/13/2021  03:35 AM                75 Kazahana_take.3-1.txt
10/13/2021  03:35 AM               104 Kazahana_take.3-2.txt
10/13/2021  03:35 AM               121 Kazahana_take.3-3.txt
10/13/2021  03:35 AM                59 Kazahana_take.4-1.txt
10/13/2021  03:35 AM                89 Kazahana_take.4-2.txt
10/13/2021  03:35 AM               138 Kazahana_take.4-3.txt
10/13/2021  03:35 AM               148 Kazahana_take.4-4.txt
10/13/2021  03:35 AM                65 Kazahana_take.5-1.txt
10/13/2021  03:35 AM                69 Kazahana_take.5-2.txt
10/13/2021  03:35 AM               112 Kazahana_take.5-3.txt
10/13/2021  03:35 AM               165 Kazahana_take.5-4.txt
10/13/2021  03:35 AM               133 Kazahana_take.5-5.txt
10/13/2021  03:35 AM             3,761 Kazahana_take.PAGODA-order-5.txt
10/13/2021  03:35 AM               423 Kazahana_take.PAGODA-order-5.wrd
04/22/2021  05:37 AM           135,680 Leprechaun_x-leton_32bit_Intel_01_001p.exe
04/22/2021  05:37 AM           134,144 Leprechaun_x-leton_32bit_Intel_01_008p.exe
04/22/2021  05:37 AM           134,144 Leprechaun_x-leton_32bit_Intel_01_512p.exe
04/22/2021  05:37 AM           139,264 Leprechaun_x-leton_32bit_Intel_02_001p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_02_008p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_02_512p.exe
04/22/2021  05:37 AM           139,264 Leprechaun_x-leton_32bit_Intel_03_001p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_03_008p.exe
04/22/2021  05:37 AM           137,728 Leprechaun_x-leton_32bit_Intel_03_512p.exe
04/22/2021  05:37 AM           138,240 Leprechaun_x-leton_32bit_Intel_04_001p.exe
04/22/2021  05:37 AM           136,704 Leprechaun_x-leton_32bit_Intel_04_008p.exe
04/22/2021  05:37 AM           136,704 Leprechaun_x-leton_32bit_Intel_04_512p.exe
04/22/2021  05:37 AM           139,776 Leprechaun_x-leton_32bit_Intel_05_001p.exe
04/22/2021  05:37 AM           138,240 Leprechaun_x-leton_32bit_Intel_05_008p.exe
04/22/2021  05:37 AM           138,240 Leprechaun_x-leton_32bit_Intel_05_512p.exe
04/22/2021  05:37 AM           695,925 Leprechaun_x-leton_r17tag.7z
04/22/2021  05:37 AM            96,993 LineJustify_PAGODAo5.c
04/22/2021  05:37 AM            73,728 LineJustify_PAGODAo5.exe
04/22/2021  05:37 AM            94,162 LineWordreporter.c
04/22/2021  05:37 AM            77,312 LineWordreporter.exe
04/22/2021  05:37 AM             1,633 MokujIN JADE 217 prompt.lnk
10/13/2021  03:35 AM             1,763 schisch.log
04/22/2021  05:37 AM            28,393 The_Task_of_a_Philosopher.txt
10/13/2021  03:34 AM            25,290 The_Task_of_a_Philosopher.txt.01.txt
10/13/2021  03:34 AM            74,892 The_Task_of_a_Philosopher.txt.02.txt
10/13/2021  03:34 AM           101,334 The_Task_of_a_Philosopher.txt.03.txt
10/13/2021  03:34 AM           113,997 The_Task_of_a_Philosopher.txt.04.txt
10/13/2021  03:34 AM           120,996 The_Task_of_a_Philosopher.txt.05.txt
04/22/2021  05:37 AM             3,335 XGRAM_PAGODA5.bat
04/22/2021  05:37 AM             2,181 XGRAM_RIP_file.bat
04/22/2021  05:37 AM            35,015 Yoshi.exe
04/22/2021  05:37 AM           969,892 Yoshi7-.zip
              56 File(s)      7,624,224 bytes
               3 Dir(s)   3,711,369,216 bytes free
 
E:\z\JRJ>time
The current time is:  3:35:40.04
Enter the new time:
 
E:\z\JRJ>notepad Kazahana_take.PAGODA-order-5.txt
 
E:\z\JRJ>
 

Also, just this night my 1-gram corpus (being the biggest in Internet) has been updated:

In those 800+ million unique words a lot of noise is there, but the important thing is that it serves as the "has it been used before" reference. If using the tagged .1gram corpus (23GB in size) it can tell how many times and in which sub-corpus a word appeared.

Code: QB64: [Select]

10/12/2021  06:08 AM    24,332,870,289 Schizandrafield_Corpus_revision_D_(45-corpora_-unique-words).sorted
10/12/2021  01:05 PM     9,036,707,422 Schizandrafield_Corpus_revision_D_(45-corpora_-unique-words).sorted.wrd
 
G:\Schiz_revision-D>type "Schizandrafield_Corpus_revision_D_(45-corpora_-unique-words).sorted.wrd"|more
a
aa
aaa
aaaa
aaaaa
aaaaaa
aaaaaaa
aaaaaaaa
aaaaaaaaa
aaaaaaaaaa
aaaaaaaaaaa
aaaaaaaaaaaa
aaaaaaaaaaaaa
aaaaaaaaaaaaaa
aaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaab
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaac
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaad
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaah
...
G:\Schiz_revision-D>
 

Sadly, no enough storage capacity available at the moment, even compressed with bzip2 the mere wordlist is ~2GB. Of course, in the near future will share the 23GB as well, somehow...

Petr · « **Reply #3 on:** October 13, 2021, 11:56:09 am »

Quote

Sadly, no enough storage capacity available at the moment, even compressed with bzip2 the mere wordlist is ~2GB. Of course, in the near future will share the 23GB as well, somehow...

Wouldn't it be possible to use the public repository and just add a link here?

bplus · « **Reply #4 on:** October 13, 2021, 12:01:31 pm »

Yeah, How big are Clouds?

Petr · « **Reply #5 on:** October 13, 2021, 12:06:15 pm »

Quote

Posted by: bplus

Yeah, How big are Clouds?

Google Drive about 15 Gigabytes

News:

Author Topic: Text Corrector (Read 2127 times)

PoliMi

Text Corrector

Pete

Re: Text Corrector

Sanmayce

Re: Text Corrector

Petr

Re: Text Corrector

bplus

Re: Text Corrector

Petr

Re: Text Corrector