Estimation

Fitting a model

The scatter plot below has a blue line which tries to estimate the relationship between x and y in the plot. This line allows you to quickly summarize the plot, and it allows you to estimate the y-value for new x-values.

Unfortunately, the line doesn't fit the data very well yet. The yellow squares represent error: the distance from each point to the line forms one side of a square. The total squared error, or loss, is shown at the bottom of the plot. Click on the plot, then use the arrow keys to improve the line's fit by reducing the loss as much as possible.

Searching systematically

So far you've been adjusting the slope and intercept by feel. There's a more systematic way: at any point, you can ask "if I nudge the intercept up a little, does the total squared error go up or down? What about the slope?" The answer to those two questions is the gradient, and repeatedly moving in the direction that decreases the loss is called gradient descent.

Turn on "Show gradient hints." Two arrows appear:

• A vertical arrow near the left edge of the plot shows which way to nudge the intercept.
• A vertical arrow near the right edge of the plot shows which way to nudge the slope—since changing the slope moves the right end of the line the most.

Each arrow points in the direction that decreases total squared error, and its length shows by how much—a long arrow means there's a lot of room for improvement; a short arrow means you're close to the bottom.

Two datasets, revisited

This lab uses Jupyter notebooks—an interactive environment for writing and running code. The notebook works through four short sections. Each one demos a technique on the Pokémon dataset from the Pokémon lab, then asks you to repeat it (and push further) on BRFSS, the health survey dataset from that same lab.

Your written answers go in questions.md, not the notebook—each checkpoint below tells you exactly what to answer and where.

💻 Start the notebook:

$ jupyter lab

Open lab_estimation.ipynb. A column reference for both datasets is near the top.

💻 Run the first few cells to load both datasets.

The BRFSS dataset has one row per survey respondent and includes:

Estimating income

In the toy, you adjusted two parameters by hand (slope and intercept), to fit a line to 24 points. BRFSS has over 160,000 respondents and many possible predictors, so we let scikit-learn do the fitting. The pattern is the same one you'll use for every model in this course:

 1from sklearn.linear_model import LinearRegression
 2from sklearn.model_selection import train_test_split
 3
 4df = pd.read_csv("brfss_2020.csv")
 5X = df[["education"]]
 6y = df["income"]
 7
 8X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
 9
10model = LinearRegression()
11model.fit(X_train, y_train)
12predictions = model.predict(X_test)

Lines 5–6 set up X and y. Notice the type of each: X is a DataFrame—a table, even though it only has one column here. X could just as easily hold five columns, one per predictor, and the rest of this code wouldn't change. y is a Series: a single value per example, which is why it's lowercase—unlike X, it's never a table with more than one column.

Line 8 divides the data into a training set and a test set, using train_test_split. This is the same idea you'll meet in every later lab: a model evaluated on the data it was trained on can look better than it really is, simply by memorizing quirks of that particular data. A test set the model never saw during training gives an honest measurement of how well it generalizes to new examples.

Lines 10–12 are the fit/predict pattern itself. fit is the gradient-descent-or-closed-form step: it searches for the parameters (slope and intercept, or one coefficient per predictor) that minimize the loss on the training data from line 8. predict is what you did by hovering over the toy—given new input, it returns the model's estimate; here it's applied to the test data, the half of line 8's split the model never trained on.

How well does the model fit?

The toy measured loss as total squared error. RMSE (root mean squared error) is the same idea, reported in a way you can actually interpret: it ends up in the same units as y, instead of squared units. Here's a function that computes it:

1def root_mean_squared_error(y_true, y_pred):
2    errors = y_true - y_pred
3    squared_errors = errors ** 2
4    mean_squared_error = squared_errors.mean()
5    return mean_squared_error ** 0.5

Line 2 finds every error at once: y_true and y_pred are the same length (one actual value and one prediction per example), so errors ends up holding one actual-minus-predicted difference per example. Line 3 squares each of those, elementwise—same as the yellow squares in the toy. That squaring is also why total squared error isn't directly interpretable on its own: squaring a value in, say, dollars gives you a number in squared dollars, which isn't a unit anyone can picture. Line 4 averages all the squared errors into a single number (and, as a side effect, makes RMSE comparable across datasets of different sizes, unlike a running total). Line 5 takes the square root, which undoes the squaring from line 3 and lands the result back in the original, interpretable units of y.

A model predicting income (banded 1–8) with RMSE = 2 is, roughly, "typically off by about 2 income bands." Always ask whether that's good or bad relative to the range and spread of $y$ itself—an RMSE of 2 is unremarkable if $y$ ranges from 1 to 8, but would be a near-perfect fit if $y$ ranges from 1 to 1000.

💻 Work through sections 1–3 of the notebook: a single continuous-ish predictor, a binary predictor, and an ordinal predictor.

Multiple regression and overfitting

You can include more than one predictor:

from sklearn.linear_model import LinearRegression

X = df[["education", "income", "exercise", "age", "no_doctor"]]
y = df["health"]
model = LinearRegression()
model.fit(X, y)

💻 Work through section 4 of the notebook: multiple regression.

Closing discussion