STxAxTIC

I think I have reached the end and although the overall results are better than I expected
they fall short of my needs.  The best result for a single 40 string run have been 38 of 40
correct for a hit rate of 95% but the overall average for 100 back-test runs has been .77%

I think I have exhausted my options for the predictors I am working on and don't see anyway
to improve it beyond it"s current rate.  Predictors 1 and 2 closely mimic the one you built
with minor differences so unless you find something magical to add then I think we can call
it finished.

Anyway, thanks for everything.

R1 
 

 
   

P.S.

In the test above it's set to add the boost weight to the predictors (0) output values.
which is part of blending method 1, blending method 2 adds the boost to the 1's output.
Just wanted to clear that up.  There are 5 different blending methods of which #5 is the
random test option.  The blending methods can also be combined but I have never tested
that option yet.

R1

STxAxTIC

It's my fault because I have been jumping around all over the place with respect to
just about everything.  I am working with several different data-sets and with two
different predictors.

Ok, for clarity I will stick to one data set attached file below named Temp.txt.  This file
contains 40 strings each with 750 entries or data-points.

What I did for the random test was to use my prng function that's old as the hills and
needs updated.  It uses a time based seed with a few extra things thrown in to try and
make it more random.

For the test mentioned in that post I simply added the prng at the end of the prediction
process.  What it does is ignore the predicted value and generates a random value.  The
Prng  generates a value from 1 to 100 which I will call (r).  After the value is generated
I use r=r mod 2.  If r > 0 then the prediction is (1), if r=(0) then the prediction is 0.
 
Because the prng is time based I allow the predictors to run I just swap out the prediction
with the randomly generated value.  I could have used Rand(0,1) to get the value but
by allowing it to first generate a value between 1 and 100 then using mod to get a odd
even value it seems to be more random.  The plan is to swap out the old prng with the
one you posted, just haven't got to it yet.

In the random test the prng managed one 70% correct prediction across the entire 40 string
run.  Most however were in the high 40's but some were as low as 30% correct.  With the
prng I am using the test results were expected.  The overall average gotten from storing the
individual results and then dividing to get a average came out to be something like 50%.  Since
working with 0 or 1 as the prediction I would think the results "were as" expected.

I have been playing around with the 3 in 1 predictor I mentioned early in the topic, specifically
with blending outputs into a single prediction via playing with the weights.  I have two weights
for each predictor, one of which is a sort of static boost that is used when processing strings where
zeros outnumber the ones by a large margin.  Without these boost-weights the zero biased strings
will always be predicted (0).  Probably not a good method, just another rabbit hole grasping a straws
to try and improve the overall.

There is a forth value that is generated for each string which is the population of 0's and 1's within
the entire string being analyzed.  I haven't figured how to use it but think it might be processed into
the main predictions to help with the zero biased strings.

The best test score for the 3 in 1 predictor was one 38 out of 40 or a 95% correct.  Being the
stupid monkey that I am, I did not save the configuration settings for that run.  My best score
since then has been been 92.5%, 37 out of 40 correct.  These 90% plus hits are few and far
between, sad to say.   In general, running 100 test I end up with an average overall score of
77% and a best of 90%. Very few show better than 90%

It seems that I have hit a wall and can't fine tune beyond my current level.  I do however have
hope that I can somehow include the 4th value, ie, the overall counts for both 0's and 1's into
the prediction and push it a little higher.   

If I compare the random test vs the 3 in 1 results then it does seem that I have made some
progress but it turns out it's not enough to meet my needs. 

A couple post back I posted a file that contains the raw outputs for each of the 3 in 1 predictors
plus the 4th overall 0 and 1 counts.  This was generated to allow me to try and fine tune the
predictor without running it.  The first four values are the 0's values, the second 4 are the 1's.
The last entry is the correct value, ie, the value that should be predicted.

The file is comma delimited which makes it easy to load the 3 in 1 predictors outputs which I then
play around with trying to configure the weights in such a way to increase the hits.  Doing a 40
string 100 event test takes several minutes but doing it this way, it takes less than a second when
just working with the predictors raw outputs.

I did find your thoughts on dissipation of patterns interesting.  Patterns emerge, run for some
extended length and then dissolve for some unknown length of time only to reappear.  Will they
start up again and if so when; that's the million dollar question.  Lets say we are working with a
6 digit binary pattern.  The pattern has finite arrangements of 000000 to 111111 6^2=64.  If
dealing with random strings one would expect each to appear 1 in 64 on average +/-  some small
standard deviation. 

Whenever we see patterns repeating at a greater than expected average then it seems that some
thing else may be going on that's not easily derived.  Also the lack of some patterns adds to the
dilemma.   

Anyway this is getting long so I will close. 

R1



   
 

STxAxTIC

LOL, I had to drag out another portable table and make a u-shaped configuration.
It was too time consuming closing the lids so I could get to the back row to make
adjustments.  Overall the test yielded nothing, after 5 hours the best was around
90% and the overall average was in the high 70's.  Going to have to try something
else.

I hate to bother you with another request but today I came up with another idea
for tuning.  If interested the attached file contains comma delimited entries of
the raw outputs of the 3 predictors plus a 4th that shows the overall hit shows for
both 0's and 1,s and the last entry in each line is the correct prediction value.

This data is compiled by taking the 3 predictors output probabilities and then using
P^2, and nothing else.  The data contains 100 test with a blank line at the end of
each run to separate them.  I used 40 strings in the test so each run contains 40
results.   

The first 4 columns are the (0) output probabilities, the second 4 are for the (1's).
The last entry is the actual value that showed in the next update.  The lines that
have all 1'a are the strings that are made up of all zeros, ie, no ones in the string
being processed. 

The idea is to try to find a small range that could be added at the end of each
prediction represented by each row.  As it is now I add the 3 predictors outputs
and then divide by 3 for both 0's and 1's then take whichever is higher and use
that value as the prediction.

Here is one of the best results from yesterday's test.  I was hoping for high 90's or maybe
even 100% No such luck
 
Test # 64
Total Strings = 40
[P1 =18]  [Hits =14]  [Misses = 4]
[P0 =22]  [Hits =22]  [Misses = 0]
Actual 1's =14   H-Ratio =1
Actual 0's =26   H-Ratio =.8461
Overall hits=36 of 40
Predictors score =.9
1's Missed on string/s ->12,27,31,38
0's Missed on string/s ->
A=0001-0111-0000-0011-0001-0011-0001-0001-0001-0011
P=0001-0111-0001-0011-0001-0011-0011-0011-0001-0111
End of prediction.............

The goal of the file below would be to use it as a training set in order to calculate  lower and
upper threshold range that can be applied in the final stage to tip the output in the direction
so to increase the overall hits.  This data was calculated without my so called boost weights.

Anyway, if you have the time to mess with it, otherwise please don't feel any obligation.  Just
throwing it out there.  I am going to calculate a running average then calculate a tipping value
which would be the value needed to off set the predicted enough to swing the prediction in
favor of the correct value.  Sounds simple enough but may need a new training set after each
update.       
 

 File image, explains the data structure.
 
https://i.postimg.cc/zDdmYrTx/help.png

     

Best score today

Test # 76
Total Strings = 40
[P1 =10]  [Hits = 8]  [Misses = 2]
[P0 =30]  [Hits =29]  [Misses = 1]
Actual 1's = 9   H-Ratio =.8888
Actual 0's =31   H-Ratio =.9354
Overall hits=37 of 40
Predictors score =.925
1's Missed on string =24,32
0's Missed on string =23
A=0000-0001-0001-0011-0001-0010-0001-0000-0000-0011
P=0000-0001-0001-0011-0001-0001-0001-0001-0000-0011

Still tinkering

R1

STxAxTIC

I'm not sure how exactly you're optimizing your solution, but imagine a search in parameter space. Lemme freestyle this... Suppose your models are M1, M2, and M3. Then the nth digit in your sequence is predicted by each model, denoted as M1(n), M2(n), M3(n). Okay, so you talked about switching these on or off. Did you try blending them? Like this:

P(n) = w(1)*M1(n) + w(2)*M2(n) + w(3)*M3(n)

This means the total prediction of the nth bit is the weighted average of all three models, and the question becomes what the hell do we do with w(1), w(2), and w(3)? The most prudent thing possible is to imagine a sphere... actually ya know what? Let me do this in two dimensions first. Suppose for clarity you have only two models. Then we can skip the third everything:

P(n) = w(1)*M1(n) + w(2)*M2(n)

I'm still working on several methods, I added a setting to the configuration so that I and have more
that one option.  Right now it allows up to 4 different methods to be used.  what I did was add the
raw data output, ie, probabilities from each predictor to an array which can be passed directly to the
method set in the config utility.  This way I can test them against each other.  I also automated the
test option so that it's possible to run 100's of test unsupervised.  The results are written to a text
file which can then be analyzed by hand.

If your still tinkering around with this and have some suggestions then I will try them out.  I am in
the process of adding the methods mentioned above.

R1

Picture of the config setting for selecting the blending method.  I suppose it's only limited to the
imagination, but right now it's set to allow 4 different methods.  Who knows, maybe some kind
of crazy strange code will take it over the top.  I am open to anything.   


https://i.postimg.cc/DZ8Yst1N/BLEND.png

 STxAxTIC

Thanks, it will take me a while to digest everything so as to decide what will best fit
what I am doing right now.  You mentioned my combining the tools but last night I
got to thinking and had the though, what if I added a negative value as the weight.
I set M1 positive but M2 and M3 both to a negative value and guess what?  I saw a
improvement right off.  After toying around I was able to get above 70% and the
big thing was the overall predicted (1's) dropped but the hit ratio improved.

Most days after a data update I check to see how many values repeat from the previous
update.  I have several sets of data but the one set I use most often normally has around
35 to 40 repeats out of 105 strings I track.  Setting the M2 and M3 weights to a negative
value gave the thing a big push.  I am seeing around 35 (1's) in the output with a hit rate
close to 70%.  In one test the program came up with 38 predictions and 32 were correct,
Wow, never seen that coming.   Anyway, still tinkering with the settings, must be millions
of ways to adjust things, each run takes a few minutes so it's a slow slow process and already
thinking of how to automate it.  I have six or seven old Dell 6000 that were configured for
max performance just setting in my closet collecting dust.  I may have to dig them out and
put them to work.  Come to think of it, a couple of them are in like new condition which were
never used.       

Anyway, mucho gracias and will let you know how it all works out.

R1

Hi all

I need some help with a scoring method for the above mentioned prediction tool.
Below is a picture of the scoreboard so to say but what I want to do is combine
all the data into a overall score so that I can track the overall effects of my fine
tuning.


https://i.postimg.cc/L6sKrkMs/SCORE.png


This results shown in the picture are based on a real data back-test.  This allows
me to analyze the predictors results to the actual event.  The first item in the pic
shows the back-test number, the second shows the number of strings processed
by the predictor.   The third is the number of predictions made by the predictor.
The forth value shows the actual number of strings that did repeat. The fifth value
shows the number of correct predictions from the possible 42 that did repeat.
The sixth line is the number of incorrect predictions made.  The seventh is a
simple probability gotten form dividing the total correct guesses by the number
of predictions made.  36/67= .5373. 

So out of 105 strings processed, there were a possible 42 correct choices for a (1)
to repeat.  Of these the predictor managed to predict 36 for 36/42=0.857 correct.

I need to combine everything in such a way that I can score the predictors overall
performance.  Also remember the predictor only predicts (1's) ie repeats.
         
overall predictions = 67/105=.638
correct predictions = 36/67=.537 but we need to factor in the 105 strings processed
at least I think this needs to be included.  etc..

Thanks in advance.

PS.  The predictor guessed correctly 36 times from a pool of the 42 possible in 67 predictions
made from analyzing 105 strings.  Calculate a score given all the variables.

R1