Knowledge Pruning MNIST: How I Hit 99% Accuracy Utilizing Half the Knowledge | by Chris Lettieri | Jan, 2025

Discover how the chosen samples seize extra diverse writing kinds and edge instances.

In some examples like cluster 1, 3, and eight the furthest level does simply appear like a extra diverse instance of the prototypical middle.

Cluster 6 is an attention-grabbing level, showcasing how some photographs are tough even for a human to guess what it’s. However you’ll be able to nonetheless make out how this could possibly be in a cluster with the centroid as an 8.

Current analysis on neural scaling legal guidelines helps to clarify why knowledge pruning utilizing a “furthest-from-centroid” method works, particularly on the MNIST dataset.

Knowledge Redundancy

Many coaching examples in giant datasets are extremely redundant.

Take into consideration MNIST: what number of almost equivalent ‘7’s do we actually want? The important thing to knowledge pruning isn’t having extra examples — it’s having the precise examples.

Choice Technique vs Dataset Dimension

One of the crucial attention-grabbing findings from the above paper is how the optimum knowledge choice technique adjustments primarily based in your dataset dimension:

• With “quite a bit” of knowledge : Choose more durable, extra various examples (furthest from cluster facilities).
• With scarce knowledge: Choose simpler, extra typical examples (closest to cluster facilities).

This explains why our “furthest-from-centroid” technique labored so effectively.

With MNIST’s 60,000 coaching examples, we had been within the “ample knowledge” regime the place choosing various, difficult examples proved most helpful.

Inspiration and Targets

I used to be impressed by these two latest papers (and the truth that I’m a knowledge engineer):

Each discover varied methods we will use knowledge choice methods to coach performant fashions on much less knowledge.

Methodology

I used LeNet-5 as my mannequin structure.

Then utilizing one of many methods under I pruned the coaching dataset of MNIST and educated a mannequin. Testing was completed towards the complete take a look at set.

Resulting from time constraints, I solely ran 5 assessments per experiment.

Full code and outcomes out there right here on GitHub.

Technique #1: Baseline, Full Dataset

• Customary LeNet-5 structure
• Skilled utilizing 100% of coaching knowledge

Technique #2: Random Sampling

• Randomly pattern particular person photographs from the coaching dataset

Technique #3: Okay-means Clustering with Completely different Choice Methods

Right here’s how this labored:

1. Preprocess the pictures with PCA to scale back the dimensionality. This simply means every picture was decreased from 784 values (28×28 pixels) into solely 50 values. PCA does this whereas retaining a very powerful patterns and eradicating redundant data.
2. Cluster utilizing k-means. The variety of clusters was mounted at 50 and 500 in several assessments. My poor CPU couldn’t deal with a lot past 500 given all of the experiments.
3. I then examined totally different choice strategies as soon as the information was cluster:

• Closest-to-centroid — these characterize a “typical” instance of the cluster.
• Furthest-from-centroid — extra consultant of edge instances.
• Random from every cluster — randomly choose inside every cluster.

• PCA decreased noise and computation time. At first I used to be simply flattening the pictures. The outcomes and compute each improved utilizing PCA so I stored it for the complete experiment.
• I switched from normal Okay-means to MiniBatchKMeans clustering for higher pace. The usual algorithm was too gradual for my CPU given all of the assessments.
• Organising a correct take a look at harness was key. Shifting experiment configs to a YAML, robotically saving outcomes to a file, and having o1 write my visualization code made life a lot simpler.

Median Accuracy & Run Time

Listed below are the median outcomes, evaluating our baseline LeNet-5 educated on the complete dataset with two totally different methods that used 50% of the dataset.

Accuracy vs Run Time Full Outcomes

The under charts present the outcomes of my 4 pruning methods in comparison with the baseline in purple.

Key findings throughout a number of runs:

• Furthest-from-centroid constantly outperformed different strategies
• There undoubtedly is a candy spot between compute time and and mannequin accuracy if you wish to discover it to your use case. Extra work must be completed right here.

I’m nonetheless shocked that simply randomly decreasing the dataset offers acceptable outcomes if effectivity is what you’re after.

Future Plans

1. Take a look at this on my second mind. I need to positive tune a LLM on my full Obsidian and take a look at knowledge pruning together with hierarchical summarization.
2. Discover different embedding strategies for clustering. I can attempt coaching an auto-encoder to embed the pictures fairly than use PCA.
3. Take a look at this on extra complicated and bigger datasets (CIFAR-10, ImageNet).
4. Experiment with how mannequin structure impacts the efficiency of knowledge pruning methods.

These findings recommend we have to rethink our method to dataset curation:

1. Extra knowledge isn’t all the time higher — there appears to be diminishing returns to greater knowledge/ greater fashions.
2. Strategic pruning can really enhance outcomes.
3. The optimum technique is dependent upon your beginning dataset dimension.

As folks begin sounding the alarm that we’re operating out of knowledge, I can’t assist however surprise if much less knowledge is definitely the important thing to helpful, cost-effective fashions.

I intend to proceed exploring the house, please attain out if you happen to discover this attention-grabbing — comfortable to attach and speak extra 🙂