Intermediate30 min

Feature Engineering

Transformations, interactions, binning, domain features, and why this still outperforms model tinkering.

Assumes you know

Encoding Categorical Features

Feature Engineering

Intuition first

A model can only use the relationships its inputs make visible. If churn depends on the ratio of complaints to purchases, a model given both counts separately has to discover division on its own — which linear models cannot do at all, and trees can only approximate with a staircase of splits.

Feature engineering is supplying the representation in which the pattern is simple. The same data, reshaped, can turn a problem a linear model fails at into one it solves exactly.

The counter-pressure is the curse of dimensionality: every feature you add enlarges the hypothesis space and dilutes distance metrics. So this is not "generate as many features as possible". It is finding the few that make the structure visible, which is a domain question more than a technical one.

Transformations of single features

Log transform — for right-skewed positive quantities (income, counts, prices, durations). Compresses the tail and makes multiplicative relationships additive:

\log(a \times b) = \log a + \log b

Use $\log(1 + x)$ when zeros are present.

Power and Box–Cox — a parametrised family that includes log and square root, with $\lambda$ chosen to maximise normality.

Binning — converting a continuous feature into ordered categories. Lets a linear model capture non-monotone effects, at the cost of discarding within-bin information. Often harmful with trees, which can already split anywhere.

Standardisation and normalisation — required for anything distance-based or regularised, irrelevant for trees.

Interactions and ratios

The highest-value features are usually combinations, because these express domain knowledge a model cannot infer cheaply.

\text{ratio} = \frac{x_1}{x_2 + \epsilon}, \qquad \text{product} = x_1 x_2, \qquad \text{difference} = x_1 - x_2

Solved problem 1 · One ratio replacing an impossible fit

A model predicts whether a customer will complain. Available: complaints (count) and purchases (count). The true rule is that customers complain when more than 20% of their purchases result in a complaint.

Show why a linear model on the raw counts cannot express this, and what feature fixes it.

Step 1 — write the true boundary

\frac{\text{complaints}}{\text{purchases}} > 0.2

Multiply through by purchases, which is positive:

\text{complaints} > 0.2 \times \text{purchases}

Step 2 — check whether a linear model can represent it

Rearranged, the boundary is

\text{complaints} - 0.2\,\text{purchases} > 0

This is linear in the two raw features. So a linear model with weights $w = (1, -0.2)$ and zero intercept represents it exactly.

Step 3 — revise the claim, and find where it actually fails

The initial framing was wrong, and it is worth being precise about why. A threshold on a ratio with a fixed cutoff is a linear boundary through the origin, so linear models handle it.

Where they fail is when the ratio enters non-linearly. Suppose instead that complaint probability is proportional to the ratio itself:

\Prob(\text{complain}) = \sigma\!\left(\beta \cdot \frac{\text{complaints}}{\text{purchases}}\right)

Now the log-odds depend on complaints divided by purchases, and no linear function $w_1 x_1 + w_2 x_2$ equals $x_1 / x_2$ . The model cannot express it at any weights.

Step 4 — demonstrate with numbers

Three customers:

	complaints	purchases	ratio
A	2	10	0.20
B	20	100	0.20
C	10	20	0.50

A and B have identical ratios and should receive identical predictions. Any linear function gives

f(A) = 2w_1 + 10w_2, \qquad f(B) = 20w_1 + 100w_2 = 10\,f(A)

So $f(B) = f(A)$ forces $f(A) = 0$ , and then the model outputs zero for every customer on the 0.2 line regardless of scale — it cannot assign them a common non-zero score while distinguishing C.

Step 5 — the engineered feature

Add

x_{\text{ratio}} = \frac{\text{complaints}}{\text{purchases} + 1}

Now A and B both take value $\approx 0.18$ and $0.198$ , C takes $0.476$ , and a linear model on this single feature separates them. One division does what no amount of tuning on the raw counts achieves.

Answer

A fixed-cutoff ratio rule is linear in the raw counts, but a model whose response depends on the ratio magnitude is not representable linearly. The engineered feature complaints / (purchases + 1) makes it a one-dimensional linear problem. The +1 prevents division by zero and shrinks estimates for customers with few purchases.

Categorical features

Encoding	Output width	Good for	Risk
One-hot	One column per level	Low cardinality, linear models	Explodes on high cardinality
Ordinal	One column	Genuinely ordered categories	Implies false ordering if unordered
Target / mean	One column	High cardinality, trees	Leakage unless cross-fitted
Frequency	One column	When rarity is informative	Collides distinct levels
Hashing	Fixed width	Very high cardinality, streaming	Collisions, no interpretability

Dates and cyclical features

A timestamp is not a number — it is a bundle of features. Extract hour, day of week, month, is-weekend, is-holiday, days-since-last-event.

Cyclical values need care. Encoding hour as $0 \dots 23$ tells the model that 23:00 and 00:00 are maximally distant when they are adjacent. Encode the angle instead:

x_{\sin} = \sin\!\left(\frac{2\pi h}{24}\right), \qquad x_{\cos} = \cos\!\left(\frac{2\pi h}{24}\right)

Two features preserving the circle, so hour 23 and hour 0 are neighbours.

python

import numpy as np
import pandas as pd

df = pd.DataFrame({"ts": pd.to_datetime([
    "2026-01-05 23:30", "2026-01-06 00:30", "2026-01-06 12:00",
])})

h = df["ts"].dt.hour + df["ts"].dt.minute / 60
df["hour_sin"] = np.sin(2 * np.pi * h / 24)
df["hour_cos"] = np.cos(2 * np.pi * h / 24)

# Euclidean distance in the (sin, cos) plane respects the wrap-around.
p = df[["hour_sin", "hour_cos"]].to_numpy()
print(f"23:30 to 00:30  distance {np.linalg.norm(p[0] - p[1]):.4f}")  # small
print(f"23:30 to 12:00  distance {np.linalg.norm(p[0] - p[2]):.4f}")  # large

Aggregations over groups

For entity-level prediction, most signal lives in aggregates of that entity's history: count, mean, max, standard deviation, trend, recency, time between events.

How much does this matter?

For tabular problems, feature engineering is usually the highest-leverage activity — typically more valuable than swapping model families or tuning hyperparameters.

For images, audio and text, largely the opposite. Deep networks learn representations from raw input, and hand-crafted features (SIFT, MFCC, n-gram counts) have been displaced by learned ones. The remaining work is augmentation and tokenisation rather than feature construction.

Exercise 1

A model predicts delivery delay. Features: distance_km, driver_id, order_time, n_items, restaurant_id. Propose five engineered features and state what each captures.

Show solution

hour_sin / hour_cos from order_time — captures rush-hour effects while keeping 23:00 adjacent to midnight. A raw hour integer would place them 23 apart.
is_peak (boolean) — a binary flag for 12:00–14:00 and 18:00–21:00. Lets a linear model express a step change that the smooth cyclical encoding cannot.
driver_historical_mean_delay, target-encoded out-of-fold — driver_id is high cardinality, so one-hot is impractical. Must be computed from past deliveries only, with smoothing towards the global mean for new drivers.
restaurant_mean_prep_time, same construction — separates restaurant-side delay from travel delay, which distance_km alone conflates.
items_per_km = n_items / (distance_km + 1) — distinguishes a large order nearby from a small order far away. These have similar totals but different delay mechanics, and a linear model cannot form the ratio itself.

Also worth adding if the data supports it: driver_active_orders at order time (congestion), and day_of_week. Both must respect the snapshot cutoff.

The two target encodings are the highest-value and the highest-risk items here. Both go inside the cross-validation pipeline.

Exercise 2

Why does log-transforming a feature help logistic regression but do nothing for a random forest?

Show solution

A logistic regression's log-odds are linear in the inputs, so the shape of a feature matters. If the true relationship is logarithmic — each doubling of income adds a fixed amount to risk — then feeding raw income forces a linear fit onto a curve, leaving systematic bias. Feeding $\log(\text{income})$ makes the relationship exactly linear and the model can fit it precisely.

A random forest splits on thresholds: income <= 50000. A monotone transformation relabels the thresholds but cannot change which rows fall on each side, because $x \leq t \iff \log x \leq \log t$ for positive $x$ . Every achievable partition of the data before the transform is achievable after it, so the fitted tree is identical (up to tie-breaking).

The general principle: tree ensembles are invariant to monotone transformations of individual features; linear and distance-based models are not. This is why tabular pipelines built around gradient boosting spend their effort on interactions and aggregations — which change what is representable — rather than on scaling and normalising, which change nothing.

Next: Feature Selection — the counterweight, removing what does not earn its dimension.