Machine learning basics

01The idea: learning rules from examples

In normal programming you write the rules: if the email contains 'free money', mark it as spam. In machine learning you provide examples — thousands of emails, each labelled spam or not — and an algorithm finds the rules itself. Think of how a child learns what a dog is: nobody gives them a definition; they see many dogs and non-dogs and build an intuition. A model is a mathematical function with adjustable numbers called parameters (or weights). Training means adjusting those parameters so the model's outputs match the known answers as closely as possible. Each input is described by features — measurable properties such as an order's amount, time of day or the customer's history — and the answer we want to predict is the label or target.

• Feature: an input property (price, word count, pixel values)
• Label / target: the answer to learn (spam, price, category)
• Model: a function with parameters adjusted during training

02Types of learning

Supervised learning uses labelled examples. If the label is a category (spam / not spam, which product category), it is classification; if it is a number (house price, delivery time), it is regression. Unsupervised learning has no labels and finds structure on its own: clustering customers into segments, detecting unusual transactions, or compressing data. Reinforcement learning learns by trial and error from rewards, as in game-playing agents and robot control. Self-supervised learning creates labels from the data itself — for example, hiding the next word in a sentence and predicting it. That is how large language models are pre-trained on huge amounts of text, before being refined with human feedback. For most business problems, supervised learning on tabular data is where you start.

03Training vs inference

Training is the learning phase: the model sees examples, makes predictions, measures how wrong it is with a loss function, and nudges its parameters to reduce that loss — usually with gradient descent, which repeatedly takes small steps 'downhill' on the error surface. It is computationally heavy and happens occasionally. Inference is the using phase: the trained model, now frozen, receives new inputs and returns predictions. It happens constantly, in production, and must be fast and cheap. This split matters for engineering: training needs data pipelines, experiments and often GPUs; inference needs an API, monitoring and latency budgets. When you call an LLM API, you are doing inference on a model someone else trained.

1from sklearn.model_selection import train_test_split
2from sklearn.ensemble import RandomForestClassifier
3from sklearn.metrics import classification_report
4 
5X, y = load_orders_features()  # features and labels (1 = fraud, 0 = ok)
6X_train, X_test, y_train, y_test = train_test_split(
7    X, y, test_size=0.2, stratify=y, random_state=42
8)
9 
10model = RandomForestClassifier(n_estimators=300, random_state=42)
11model.fit(X_train, y_train)            # training
12 
13pred = model.predict(X_test)           # inference on unseen data
14print(classification_report(y_test, pred))

04Train, validation and test sets

A model that is tested on the same data it learned from will look brilliant and fail in real life — like a student who memorised last year's exam answers. So you split data into three parts. The training set is for learning. The validation set is for choices you make: which algorithm, which settings (hyperparameters), which features. The test set is locked away and used once at the end to estimate real-world performance honestly. Typical splits are 70/15/15 or 80/10/10. Cross-validation rotates the validation part across several folds to get a more stable estimate on smaller datasets. For time-based data — sales, prices, churn — split by time: train on the past, test on the future, because that is how the model will be used.

05Overfitting and underfitting

Overfitting means the model learned the noise and quirks of the training data instead of the general pattern: excellent training scores, poor scores on new data. Underfitting is the opposite: the model is too simple to capture the pattern and does badly everywhere. Picture fitting a curve through points: a straight line may underfit, a wildly wiggly line through every point overfits, and a smooth curve generalises. Signs of overfitting: a big gap between training and validation scores. Remedies: more and more varied data, a simpler model, regularisation (penalising extreme parameters), early stopping, dropout in neural networks, and removing features that leak the answer. The goal is always generalisation — doing well on data the model has never seen.

• High train, low validation score → overfitting
• Low train and low validation score → underfitting
• Fix with data, simpler models, regularisation, early stopping

06Metrics: choosing what 'good' means

Accuracy — the share of correct predictions — is misleading when classes are imbalanced. If 1% of transactions are fraud, a model that always says 'not fraud' is 99% accurate and useless. So you use better metrics. Precision: of everything flagged as fraud, how much really was fraud? Recall: of all real fraud, how much did we catch? F1 combines both. ROC-AUC and PR-AUC measure how well the model ranks positives above negatives across all thresholds. For regression, MAE (mean absolute error) is easy to explain in real units, and RMSE punishes big errors more. Always tie the metric to business cost: in fraud, a missed fraud may cost far more than a false alarm, so you tune the decision threshold accordingly, often favouring recall.

1from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score
2 
3proba = model.predict_proba(X_test)[:, 1]
4pred = (proba >= 0.3).astype(int)   # threshold chosen from business costs
5 
6print("precision", precision_score(y_test, pred))
7print("recall   ", recall_score(y_test, pred))
8print("f1       ", f1_score(y_test, pred))
9print("roc_auc  ", roc_auc_score(y_test, proba))

07Gradient descent and how neural networks learn

Training is an optimisation problem. The loss function turns 'how wrong is the model' into a single number — cross-entropy for classification, mean squared error for regression. The gradient is the direction in which the loss increases fastest with respect to each parameter; gradient descent moves parameters a small step the other way. The step size is the learning rate: too high and training diverges or bounces around, too low and it crawls. In practice we use mini-batches (stochastic gradient descent) and adaptive optimisers such as Adam, which adjust the step size per parameter. Neural networks compute gradients through all layers with backpropagation, an efficient application of the chain rule. Large models add tricks — learning-rate warm-up and decay, normalisation layers, gradient clipping — but the core loop is the same: predict, measure loss, compute gradients, update.

1import numpy as np
2 
3# Linear regression y ≈ w*x + b with plain gradient descent
4x = np.array([1, 2, 3, 4, 5], dtype=float)
5y = np.array([3, 5, 7, 9, 11], dtype=float)   # true: w=2, b=1
6w, b, lr = 0.0, 0.0, 0.05
7 
8for epoch in range(500):
9    pred = w * x + b
10    err = pred - y
11    loss = (err ** 2).mean()
12    w -= lr * 2 * (err * x).mean()   # gradient of loss w.r.t. w
13    b -= lr * 2 * err.mean()         # gradient of loss w.r.t. b
14 
15print(round(w, 2), round(b, 2), round(loss, 4))

08Bias–variance trade-off and regularisation

Prediction error has three parts: bias (error from wrong assumptions — a model too simple to capture the pattern), variance (error from sensitivity to the particular training sample — a model so flexible it changes a lot with small data changes) and irreducible noise. Simple models have high bias and low variance; complex models the reverse. Regularisation deliberately adds a little bias to reduce variance: L2 (ridge) shrinks all weights, L1 (lasso) pushes some to exactly zero and selects features, tree models limit depth or leaf size, and neural networks use weight decay, dropout and early stopping. Ensembles reduce variance too: random forests average many decorrelated trees, while gradient boosting adds small trees that correct previous errors. Modern deep learning complicates the classic picture — very large networks can generalise well despite enormous capacity — but on everyday tabular problems the trade-off is a reliable guide.

09Data leakage and evaluation traps

Leakage is the most common reason a model looks great offline and fails in production. Target leakage: a feature encodes the outcome, like 'account_closed_date' when predicting churn. Temporal leakage: aggregates computed over the whole dataset, including the future, or random splits on time-ordered data. Group leakage: the same customer or patient appears in both train and test sets, so the model recognises individuals rather than patterns — use group-aware splits. Pre-processing leakage: scaling, imputing or selecting features on the full dataset before splitting; fit these steps inside a pipeline on training data only. A quick test: if performance seems too good to be true, inspect feature importance and ask, for each top feature, 'would I really know this value at prediction time?'

1from sklearn.pipeline import make_pipeline
2from sklearn.preprocessing import StandardScaler
3from sklearn.linear_model import LogisticRegression
4from sklearn.model_selection import GroupKFold, cross_val_score
5 
6# Scaling is fitted inside each fold -> no pre-processing leakage
7pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
8 
9# Same customer never in both train and validation folds
10scores = cross_val_score(pipe, X, y, groups=customer_ids,
11                         cv=GroupKFold(n_splits=5), scoring="roc_auc")
12print(scores.mean(), scores.std())

10From notebook to production (MLOps essentials)

A model in a notebook is an experiment; a model in production is a service with dependencies. Essentials: reproducible training (versioned data snapshot, code commit, random seeds, environment), a model registry storing artefacts with their metrics, and the exact same feature logic in training and serving — mismatches here (training–serving skew) are a classic silent bug, which feature stores aim to prevent. Serve models behind an API or batch job with input validation, latency budgets and a fallback rule. Monitor three layers: system (latency, errors), data (feature distributions, missing values, drift) and outcome (real metrics once labels arrive). Roll out with shadow mode or canaries, compare against the current model, and keep rollback simple. Retrain on a schedule or trigger, and re-check fairness and subgroup performance each time.