Evaluators module

Prequential([metrics])

Prequential data stream evaluator.

TestThenTrain([metrics, verbose])

Test Than Train data stream evaluator.
class strlearn.evaluators.Prequential(metrics=(<function accuracy_score>, <function balanced_accuracy_score>))

Bases: object

Prequential data stream evaluator.

Implementation of prequential evaluation procedure, based on sliding windows instead of separate data chunks. Window moves by a fixed number of instances in order to preserve some of the already processed ones. After each step, samples that are currently in the window are used to test the classifier and then for training.

Parameters:

metrics (tuple or function) – Tuple of metric functions or single metric function.

Variables:

  • classes (array-like, shape (n_classes, )) – The class labels.

  • scores (array-like, shape (stream.n_chunks, len(metrics))) – Values of metrics for each processed data chunk.

Example:

>>> import strlearn as sl
>>> stream = sl.streams.StreamGenerator()
>>> clf = sl.classifiers.AccumulatedSamplesClassifier()
>>> evaluator = sl.evaluators.PrequentialEvaluator()
>>> evaluator.process(clf, stream, interval=50)
>>> print(evaluator.scores_)
...
[[0.95       0.9483469  0.94805282 0.9483469  0.95412844]
[0.96       0.95728313 0.95696445 0.95728313 0.96460177]
[0.96       0.95858586 0.95848154 0.95858586 0.96396396]
...
[0.92       0.91987179 0.91986621 0.91987179 0.91666667]
[0.91       0.91065705 0.91050889 0.91065705 0.90816327]
[0.925      0.92567027 0.9250634  0.92567027 0.92610837]]
process(stream, clfs, interval=100)

Perform learning procedure on data stream.

Parameters:

  • stream (object) – Data stream as an object

  • clfs (tuple or function) – scikit-learn estimator of list of scikit-learn estimators.

  • interval (integer, optional (default=100)) – The number of instances by which the sliding window moves before the next evaluation and training steps.

class strlearn.evaluators.SparseTrainDenseTest(n_repeats=5, metrics=(<function accuracy_score>, <function balanced_accuracy_score>), verbose=False)

Bases: object

Sparse Train Dense Test data stream evaluator.

Implementation of sparse-train-dense-test evaluation procedure, where each individual data chunk is first used to test the classifier and then it is used for given number of chunks of training.

Parameters:

  • metrics (tuple or function) – Tuple of metric functions or single metric function.

  • verbose (boolean) – Flag to turn on verbose mode.

Variables:

  • classes (array-like, shape (n_classes, )) – The class labels.

  • scores (array-like, shape (stream.n_chunks, len(metrics))) – Values of metrics for each processed data chunk.

process(stream, clfs)

Perform learning procedure on data stream.

Parameters:

  • stream (object) – Data stream as an object

  • clfs (tuple or function) – scikit-learn estimator of list of scikit-learn estimators.

class strlearn.evaluators.TestThenTrain(metrics=(<function accuracy_score>, <function balanced_accuracy_score>), verbose=False)

Bases: object

Test Than Train data stream evaluator.

Implementation of test-then-train evaluation procedure, where each individual data chunk is first used to test the classifier and then it is used for training.

Parameters:

  • metrics (tuple or function) – Tuple of metric functions or single metric function.

  • verbose (boolean) – Flag to turn on verbose mode.

Variables:

  • classes (array-like, shape (n_classes, )) – The class labels.

  • scores (array-like, shape (stream.n_chunks, len(metrics))) – Values of metrics for each processed data chunk.

Example:

>>> import strlearn as sl
>>> stream = sl.streams.StreamGenerator()
>>> clf = sl.classifiers.AccumulatedSamplesClassifier()
>>> evaluator = sl.evaluators.TestThenTrainEvaluator()
>>> evaluator.process(clf, stream)
>>> print(evaluator.scores_)
...
[[0.92       0.91879699 0.91848191 0.91879699 0.92523364]
[0.945      0.94648779 0.94624912 0.94648779 0.94240838]
[0.92       0.91936979 0.91936231 0.91936979 0.9047619 ]
...
[0.92       0.91907051 0.91877671 0.91907051 0.9245283 ]
[0.885      0.8854889  0.88546135 0.8854889  0.87830688]
[0.935      0.93569212 0.93540766 0.93569212 0.93467337]]
process(stream, clfs)

Perform learning procedure on data stream.

Parameters:

  • stream (object) – Data stream as an object

  • clfs (tuple or function) – scikit-learn estimator of list of scikit-learn estimators.