CodeUtility
Making your developer's life easier!
AI Concepts in Python

Classification Basics: Softmax and k-Nearest Neighbours in Python

Classification means choosing a label, such as cat or dog. A model can output one score per class and let softmax turn the scores into probabilities, or a simple method like k-nearest neighbours can let the closest known examples vote.

The concept in plain words

Classification means choosing a label, such as cat or dog. A model can output one score per class and let softmax turn the scores into probabilities, or a simple method like k-nearest neighbours can let the closest known examples vote.

Turn scores into class probabilities with softmax, score them with cross-entropy and classify points with k-nearest neighbours.

Key ideas

  • Softmax keeps the order of the scores and makes them add up to 1.
  • Categorical cross-entropy punishes confident wrong answers the most.
  • k-NN has no training step: it compares a new point with the stored examples.
  • Naive Bayes and decision trees (Gini impurity, information gain) are other classic classifiers.

When should you use it?

  • Understand what happens inside neural networks and AI libraries.
  • Learn the idea before you implement it in the matching challenges.
  • Prepare for machine learning courses, interviews and projects.

See it in action

Click anywhere on the left chart to classify a new point by a vote of its k nearest neighbours. Change the scores on the right to see softmax react.

Example code

Run code →
main.py
import math
from collections import Counter

# Softmax turns raw scores into probabilities that add up to 1.
def softmax(scores):
    exps = [math.exp(s - max(scores)) for s in scores]
    total = sum(exps)
    return [e / total for e in exps]

probs = softmax([2.0, 1.0, 0.1])
print("softmax:", [round(p, 3) for p in probs])
# Cross-entropy is small when the true class gets a high probability.
print("cross-entropy if class 0 is right:", round(-math.log(probs[0]), 3))

# k-nearest neighbours: label a point by a vote of the closest examples.
examples = [((1, 1), "cat"), ((1, 2), "cat"), ((5, 5), "dog"), ((6, 5), "dog"), ((5, 6), "dog")]
def knn(point, k=3):
    nearest = sorted(examples, key=lambda e: math.dist(point, e[0]))[:k]
    return Counter(label for _, label in nearest).most_common(1)[0][0]

print("(2, 1) ->", knn((2, 1)))
print("(5, 4) ->", knn((5, 4)))

Expected output

softmax: [0.659, 0.242, 0.099]
cross-entropy if class 0 is right: 0.417
(2, 1) -> cat
(5, 4) -> dog

How it works

softmax() exponentiates the scores and divides by their sum, so the three probabilities add up to 1; subtracting the maximum first avoids overflow. Cross-entropy is −log of the probability of the correct class. knn() sorts the examples by distance and lets the 3 nearest vote.

Change the values and run the program in the CodeUtility online Python compiler without installing Python locally.

Practice exercises

Modify the runnable example with the exercises below to build understanding beyond copying the result.

  1. Change the inputs or weights and predict the output before running.
  2. Print the intermediate values to follow each step of the calculation.
  3. Solve the linked challenges below to implement the idea yourself.
Run in Python IDE →

Practice with a challenge

Use what this lesson covers to solve a real problem. Each challenge runs in your browser with instant tests and an AI review.

All challenges