How do I take the correct classification predictions of an ml algo (i.e. random forest/neural net) and sort the instances in each category?

Question

I am trying to sort the instances within each of 5 classification categories in a dataset that has been put through both a random forest classifier and a neural network with 99% accuracy on each.

Essentially what I am trying to do is stack a sorting algorithm on top of a random forest or a neural net (depending on which will boast a much more efficient and swift process in being integrated with a sorting algorithm post-output) so that the correctly classified instances within each category can be sorted into separate organized and easily comprehensible lists.

I have tried researching this but all I have found are examples of ensemble learning and traditional stacking in order to achieve higher overall accuracy on an algorithm.

I am trying to see how I can take the correct predictions from either of these algorithms and sort them by some arbitrary features that I will engineer, but all I am wondering at the moment is how to integrate a sorting algorithm in the first place into the output of a classification algorithm.

Answer 1

For the ANN, it should be the average of the error per instance from testing (prediction) when each instance is left out of training. ANNs can unfortunately learn based on the order of instances used for training, so it helps to train/test and then shuffle (permute, or randomly re-order) and then assign to k-folds, then train/test again in order to prevent the ANN from learning based on instance order. The RF won't do this, since the in-bag objects in all the bootstraps (trees) that were sampled with replacement will vary appreciably.

(FYI - there's commonly an option to train using "batch" or "online" error updates during back-propagation. Batch sums the error over the training objects, then updates coeffs. Online updates coeffs as each training instance is processed. I tend to like online, since online seems to reduce error faster. Batch can get off the gradient path easier, since the summed error can come in large "chunks" and throw the ANN off course)

As far as sorting instances within each class label after training/testing, you can do anything you wish. Certainly tracking each instance as it is tested/predicted and building an array/vector of errors will give you a distribution of error per instance.

Quite often, test objects can widely very in their feature values, so e.g. k-means cluster analysis can be used during ANN training. See my blog on this.

In short, it's not just sorting when done. Ensembles are on the back end of an attack plan, but the checklist for best practices would look like:

1. Randomly shuffle the order of all instances.
2. Assign instances to $k$-folds (partitions), e.g. 10-fold.
3. Train with instances in folds 1-9, test (predict class) of instances in fold 10. Increment confusion matrix. Then train with instances in folds 1-8,10 and test instances in fold 9, update confusion matrix, ...., cycle through all test folds.
4. Repeat steps 1-3 nine more times (this is called doing another "re-partition").
5. Calculate average error for each predicted instance, then sort.

The above steps are only for the ANN. The RF performs its own bookkeeping.