$$ \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\x}{\mathbf{x}} \newcommand{\y}{\mathbf{y}} \newcommand{\wv}{\mathbf{w}} \newcommand{\av}{\mathbf{\alpha}} \newcommand{\bv}{\mathbf{b}} \newcommand{\N}{\mathbb{N}} \newcommand{\id}{\mathbf{I}} \newcommand{\ind}{\mathbf{1}} \newcommand{\0}{\mathbf{0}} \newcommand{\unit}{\mathbf{e}} \newcommand{\one}{\mathbf{1}} \newcommand{\zero}{\mathbf{0}} \newcommand\rfrac[2]{^{#1}\!/_{#2}} \newcommand{\norm}[1]{\left\lVert#1\right\rVert} $$ # Cross Validation ## Description A prevalent problem when utilizing machine learning algorithms is _overfitting_, or when an algorithm “memorizes” the training data but does a poor job extrapolating to out of sample cases. A common method for dealing with the overfitting problem is to hold back some subset of data from the original training algorithm and then measure the fit algorithm’s performance on this hold-out set. This is commonly known as _cross validation_. A model is trained on one subset of data and then _validated_ on another set of data. ## Cross Validation Strategies There are several strategies for holding out data. FlinkML has convenience methods for * Train-Test Splits * Train-Test-Holdout Splits * K-Fold Splits * Multi-Random Splits ### Train-Test Splits The simplest method of splitting is the `trainTestSplit`. This split takes a DataSet and a parameter _fraction_. The _fraction_ indicates the portion of the DataSet that should be allocated to the training set. This split also takes two additional optional parameters, _precise_ and _seed_. By default, the Split is done by randomly deciding whether or not an observation is assigned to the training DataSet with probability = _fraction_. When _precise_ is `true` however, additional steps are taken to ensure the training set is as close as possible to the length of the DataSet $\cdot$ _fraction_. The method returns a new `TrainTestDataSet` object which has a `.training` attribute containing the training DataSet and a `.testing` attribute containing the testing DataSet. ### Train-Test-Holdout Splits In some cases, algorithms have been known to ‘learn’ the testing set. To combat this issue, a train-test-hold out strategy introduces a secondary holdout set, aptly called the _holdout_ set. Traditionally, training and testing would be done to train an algorithms as normal and then a final test of the algorithm on the holdout set would be done. Ideally, prediction errors/model scores in the holdout set would not be significantly different than those observed in the testing set. In a train-test-holdout strategy we sacrifice the sample size of the initial fitting algorithm for increased confidence that our model is not over-fit. When using `trainTestHoldout` splitter, the _fraction_ `Double` is replaced by a _fraction_ array of length three. The first element corresponds to the portion to be used for training, second for testing, and third for holdout. The weights of this array are _relative_, e.g. an array `Array(3.0, 2.0, 1.0)` would results in approximately 50% of the observations being in the training set, 33% of the observations in the testing set, and 17% of the observations in holdout set. ### K-Fold Splits In a _k-fold_ strategy, the DataSet is split into _k_ equal subsets. Then for each of the _k_ subsets, a `TrainTestDataSet` is created where the subset is the `.training` DataSet, and the remaining subsets are the `.testing` set. For each training set, an algorithm is trained and then is evaluated based on the predictions based on the associated testing set. When an algorithm that has consistent grades (e.g. prediction errors) across held out datasets we can have some confidence that our approach (e.g. choice of algorithm / algorithm parameters / number of iterations) is robust against overfitting. [K-Fold Cross Validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation) ### Multi-Random Splits The _multi-random_ strategy can be thought of as a more general form of the _train-test-holdout_ strategy. In fact, `.trainTestHoldoutSplit` is a simple wrapper for `multiRandomSplit` which also packages the datasets into a `trainTestHoldoutDataSet` object. The first major difference, is that `multiRandomSplit` takes an array of fractions of any length. E.g. one can create multiple holdout sets. Alternatively, one could think of `kFoldSplit` as a wrapper for `multiRandomSplit` (which it is), the difference being `kFoldSplit` creates subsets of approximately equal size, where `multiRandomSplit` will create subsets of any size. The second major difference is that `multiRandomSplit` returns an array of DataSets, equal in size and proportion to the _fraction array_ that it was passed as an argument. ## Parameters The various `Splitter` methods share many parameters. | Parameter | Type | Description | Used by Method | | --- | --- | --- | --- | | `input` | `DataSet[Any]` | DataSet to be split. | `randomSplit` `multiRandomSplit` `kFoldSplit` `trainTestSplit` `trainTestHoldoutSplit` | | `seed` | `Long` | Used for seeding the random number generator which sorts DataSets into other DataSets. | `randomSplit` `multiRandomSplit` `kFoldSplit` `trainTestSplit` `trainTestHoldoutSplit` | | `precise` | `Boolean` | When true, make additional effort to make DataSets as close to the prescribed proportions as possible. | `randomSplit` `trainTestSplit` | | `fraction` | `Double` | The portion of the `input` to assign to the first or `.training` DataSet. Must be in the range (0,1) | `randomSplit` `trainTestSplit` | | `fracArray` | `Array[Double]` | An array that prescribes the proportions of the output datasets (proportions need not sum to 1 or be within the range (0,1)) | `multiRandomSplit` `trainTestHoldoutSplit` | | `kFolds` | `Int` | The number of subsets to break the `input` DataSet into. | `kFoldSplit` | ## Examples
``` // An input dataset- does not have to be of type LabeledVector val data: DataSet[LabeledVector] = ... // A Simple Train-Test-Split val dataTrainTest: TrainTestDataSet = Splitter.trainTestSplit(data, 0.6, true) // Create a train test holdout DataSet val dataTrainTestHO: trainTestHoldoutDataSet = Splitter.trainTestHoldoutSplit(data, Array(6.0, 3.0, 1.0)) // Create an Array of K TrainTestDataSets val dataKFolded: Array[TrainTestDataSet] = Splitter.kFoldSplit(data, 10) // create an array of 5 datasets val dataMultiRandom: Array[DataSet[T]] = Splitter.multiRandomSplit(data, Array(0.5, 0.1, 0.1, 0.1, 0.1)) ```