Data Science from Scratch: First Principles with Python

What if I told you that adding your most highly educated, highly active users to a data set could mathematically make your entire user base, look, you know, substantially less sociable.

Yeah, I mean it completely sounds like a broken algorithm, right, but it's actually this fundamental statistical trap that catches data teams off guard literally every single.

Day, right, and today we are dismantling those traps. We're doing a deep dive into the core concepts of data science, pulling directly from Joel Gruse's Data Science from scratch, and the mission here is straightforward. We are stepping entirely away from those prepackaged software libraries.

Exactly because relying purely on high level abstractions, you know, just typing import pandas and calling a dot mean function, it creates this really dangerous blind spot.

You're just trusting a black box.

Right. When you treat your analytical tools as black boxes, you lose the ability to actually interrogate the underlying assumptions. I mean, you end up optimizing for algorithms you don't fully understand.

Which inevitably leads to very confident, mathematically sound, and completely incorrect conclusions.

Yes, the worst kind of incorrect conclusions.

So to ground this deep dive for you, the listener, we're placing you in a very specific hypothetical scenario today. You have just been brought in as the founding data scientist at a startup called Data Sciencestor, which is, you know, a social network tailored entirely for data professionals.

Sounds like a very niche market.

Very niche, But the point is there is no legacy infrastructure. You are tasked with building the analytical pipeline completely from scratch.

Which means before you can analyze a single user's behavior, you have to actually choose your architecture. And the source material strongly advocates for Python.

But not just because it's popular, right, The rationale goes way beyond its ecosystem of data tools.

Oh. Absolutely, the argument is rooted entirely in Python's core design philosophy. It goes back to the zen of Python, which basically dictates that you know, explicit is better than implicit.

Right, And we see this manifested most clearly in how Python enforces structural readability through white space.

Yes, the white space rule. I mean, if you look at C plus plus or Java, the scope of a function is defined by curly braces.

Which the compiler basically just ignores. Yeah, right, like it ignores the indentation completely exactly.

You can have this incredibly complex, deeply nested logic crammed onto a single line of text and the machine will parse it perfectly, even if it is completely illegible to the next engineer who has to inherit your code base.

Right.

But Python removes that option entirely. The visual structure of the code must match the logical structure, or the interpreter will just throw an indentation error and refuse to run.

It's honestly like a Marie Condo approach to coding.

A Mariecondo approach.

Yeah, like with a curly brace language, you can take this absolute disaster of a messy room, shove all the tangled logic into a closet, slam the compiler door shut, and it just runs.

That's yeah, that's a great way to put it.

But Python. Python forces you to organize the closet so the structure is visible the second you open the file. Everyone can see exactly what sparks joy or what causes fatal crash.

That's exactly it. And this emphasis on explicit structure has really only become more critical with the shift to Python three, especially when we start talking about the adoption of type annotations in these big data pipelines.

Right, because Python natively is dynamically typed yes.

Meaning a variable can hold an integer and then literally in the next line of code it can be reassigned to a string, which.

Is great if you're just writing a quick script.

Sure, in a localized script, that flexibility speeds up development, But in a massive data ingestion pipeline, it is a massive liability because you.

Don't know what data is actually flowing.

Through the pipe exactly. Let's say your pipeline is pulling user engagement metrics for data science stor and some localized anomaly introduces a string, maybe it's like a text based nan or just a null character into a feature set that is mathematically acting floating point numbers.

Oh right.

In a purely dynamic setup, the pipeline might not even crash immediately. It might just perform a silent type coercion or propagate that null value all the way through your downstream transformations.

Which ultimately just corrupts the training data for whatever a machine learning model you're building.

Exactly, and you wouldn't even realize the error until the model's accuracy mysteriously degraded in production weeks later.

Wow. So by using type annotations, you're basically establishing a strict contract for your functions.

Precisely you declare explicitly upfront that a specific ingestion module expects an integer and returns a float period, and.

Then static type checkers can actually analyze the codebase before it even runs, flagging any potential violations.

It completely shifts the paradigm. You go from basically crossing your fingers and hoping the data conforms your expectations to architecting a system that mathematically guarantees the data types before the processing even begins.

So you're building transparency in resilience right into the foundation of data sciencestor.

You have to, because once you have a type safe pipeline aggregating all this clean user data, the immediate next step in the analytical life cycle is exploratory data analysis.

Right. You want to visualize the distributions to spot the broader trends.

Which introduces a completely different class of vulnerability into your workflow.

Ah. Yes, because you've moved from the strict, unforgiving logic of the compiler to the highly subjective translation of data into pixels.

Yes, human visual processing hardware has all these built in heuristics, and those heuristics are remarkably easy to exploit.

The source material actually highlights this using that plotlib, specifically looking at how you can manipulate the axis in bar charts.

It's such a classic trap.

Right. Let's say you were presenting platform growth to the Data Science Hastor Board of Directors, and in twenty seventeen the platform was mentioned five hundred times. Then in twenty eighteen it was mentioned five hundred and five times.

Which is, let's be honest, actional increase. It's barely a blip in the actual volume.

Right, But if you construct a bar chart for the board and you set the axis to start at four ninety nine and end at five oh six, you dramatically alter the visual narrative. You really do, because the bar for twenty seventeen sits at a value of one unit above the baseline, but the bar for twenty eighteen rises to six units above the baseline. Yep.

Visually, that twenty eighteen bar is taking up six times the physical space on the screen exactly.

It looks like this towering, exponential six hundred percent increase, even though the underlying data barely even moved.

And this is exactly where understanding cognitive psychology intersects with data science. Our visual cortex processes different geometric shapes using completely different underlying rules.

Wait, let me push back on this rule for a second. Sure isn't zooming in on the axis? Just I don't know a helpful way to highlight the relevant detail. If I'm tracking a metric from five hundred to five h five, why wouldn't I want to zoom in to show that's specific variance.

Well, it depends on the chart you're using. If we look at a line chart, we are evaluating the angle of the slope. The cognitive focus is on the trajectory and the rate of change over time. Okay, So zooming in on a line chart to expose localized volatility, say tracking minute by minute stock fluctuations between one hundred, one hundred and five dollars, is completely analytically valid. The slope still remains a true representation of the localized variance.

Right because I'm just looking at the angle of the line going.

Up and down exactly. But the visual processing mechanism for a bar chart is fundamentally different. With a bar chart, the human brain instinctively equates the value of the data point with the total two dimensional area.

Of the bar, like the actual amount of ink printed on the page.

Yes, the literal amount of ink. So by truncating the axis and starting at four ninety nine, you are divorcing the area of the bar from its mathematical value.

Oh wow, I see what you mean.

Yeah, you are asking the viewer's brain to process a physical shape that is six times larger, while expecting them to override their own visual instincts by reading the tiny numbers printed on the axis.

It's a total cognitive mismatch.

Exactly, a non zero axis on a bar chart basically mathematically lies to the viewer's visual cortex, and.

The book points out that the same kind of distortion applies to variants in scato plots too.

Oh absolutely.

Like if we map out user test scores with test one on the x axis and test two on the axis, the scaling of those axes defines the perceived standard deviation. If you're plotting, library automatically scales the x axis to cover a twenty point spread, but then it stretches the axis to cover a forty point spread just to fill up the screen.

Then the visual density of your clusters is completely compromised.

Right, The data along the axis will appear to have significantly higher variants simply because the pixels are stretched further apart. You have to force comparable axes to maintain the integrity of the distribution.

You do, but you know, visualization is really just a tool for spotting aggregate trends, and to truly understand the mechanics of a social platform like data sign ancestor aggregate trends aren't enough.

No, the executive team wants to know who the key influencers are. They want to find the nodes with the highest degree centrality, right.

Which means we have to analyze the topology of the network.

Itself and calculating degree centrality basically just means counting who has the most friends. But doing that requires analyzing the edges, the connections between the users, and in a.

Raw data format, this usually exists as an edge list. You know, user zero is friends with user one, User zero's friends with user two, user one is friends with user three, and so on, which.

Is fine for a tiny data set. Iterating through a short list to count a specific users connections is trivial, sure, but as.

The platform scales to say, millions of users, the computational complexity of that search becomes a massive bottleneck. We are talking about big O notation.

Here, right, the dreaded big oh.

Exactly. Searching an unstructured edge list requires an O of n operation where n is the number.

Of edges, Meaning to find all connections for you US one, the algorithm literally has to traverse the entire list, evaluating every single pair to see if User one is present.

It is computationally expensive, and it scales terribly as the network grows, which is why data scientists transition into linear algebra. They translate the network structure by representing the connections as an adjacency matrix.

Okay, I like to use an analogy for this efficiency jump. It's here. The edge list is like an old school rolodex. If you want to know who User one knows, you have to flip through every single card in the entire box to check right, very slow. But the matrix is like a giant wall size pegboard. The rows represent every user and the columns represent those exact same users. If User A is friends with the user B. You stick a PEG in the intersecting cell basically a one. If

I love that pegboard analogy, and the performance implications of that transformation are profound. When we restructure the data into a matrix, we change the algorithmic complexity of finding a user's connections from an O of n sequential search to an OH of one constant time look up.

OH of one, So it's instantaneous.

Exactly if you need to know user fives connections, the system doesn't search at all. It just jumps directly to the memory address of row five and retrieves the background, which also aligns perfectly with modern hardware architecture. Yes it does. Traversing a linked list or an edge list often means jumping around to different non contiguous blocks of memory.

Which causes cache misses and slows down.

The processing exactly. But a matrix stores these values in contiguous memory blocks, and that contiguous memory layout allows you to leverage semity operations single instruction, multiple data.

Because modern CPUs and particularly GPUs are explicitly designed to perform parallel math operations on contiguous arrays of numbers.

Right, So, by representing the social network as a matrix, you can and utilize parallel processing to calculate the Egen values of the matrix, which.

Gives you eigenvector centrality.

Yes, a far more sophisticated metric that doesn't just measure how many friends a user has, but how influential those friends actually are. The way you structure your data fundamentally dictates the analytical power you can bring to bear.

Okay, so let's take stock. You have built a type safe pipeline, You are forcing comparable axis on your charts to avoid those visual distortions, and you are utilizing GPU accelerated matrix operations to map network topology in constant time.

The architecture is rack solid.

It is. But then the VP of Growth knocks on your door. They want you to build a statistical profile of the typical user's behavior.

Of course they do.

They want to correlate the number of friends a user has with the number of daily minutes they spend on the platform, and this introduces us to the fragility of standard statistical metrics.

Oh absolutely, when you're summarizing distributions the traditional mean, the average is notoriously brittle.

Yeah. OK has this great classic example about university graduate.

Salaries the UNC geography major. Yes, in the mid nineteen eighties, the major at the University of North Carolina with the highest means starting salary was geography.

And it wasn't because the market suddenly deeply valued cartography.

No, it was solely because a single graduate named Michael Jordan entered the NBA.

Right. Because the mean is calculated by summing all the values and dividing by the count, it distributes the weight of every value equally across the data set.

Which means a massive multi million dollar outlier pulls the entire mathematical center of gravity toward itself. It completely obscures the typical distribution of the data.

Whereas the median, by contrast, just relies on positional rank. It isolates the middle value and renders those extreme taiales irrelevant.

Exactly, and this vulnerability to outliers it extends directly into how we measure correlation.

Too, Right, like Pearson's correlation coefficient, which evaluates the linear relationship between two variables.

Yes, but the underlying us for Pearson's relies on calculating covariance, and covariance involves multiplying the deviations of each data point from the mean.

Okay, And because you are multiplying those deviations, a massive outlier doesn't just like slightly skew the result. It mathematically dominates the entire calculation.

It completely takes over. Let's return to the VP's hypothesis, more friends equals more time spent on data sciencestor. You run the correlation and the coefficient comes back incredibly weak.

The data basically suggests there is no relationship, right.

But then you actually plot the data on a scatterplot and you see this massive dense cluster showing a very clear positive trend, and then way off in the corner, one single data point sitting completely isolated on the far edges of the plot.

And upon investigation, that single point is an internal test account exactly.

A developer just gave it one hundred friends, but it only logs one minute of activity a day.

So that single test account has such an extreme deviation from the mean on both axes that when those deviations are multiplied together in the covariance formula, it just violently yanks the line of best fit away from the actual user cluster.

Yes, and the moment you drop that single test account from the matrix, the correlation coefficient jumps up. The underlying truth was there all along. It was just masked by the mathematical weight of a single anomaly.

Which brings us to the absolute most insidious statistical trap in data science. Simpson's paradox.

Oh, my favorite.

Outliers are easy to spot if you just visualize the distribution, right, But Simpsons paradox hies entirely within the aggregate structure of the data itself.

It does. It occurs when a clear trend appears in multiple distinct groups of data, but then completely disappears or even reverses when those groups are combined.

So let's apply this to data science estor, say you are analyzing regional engagement to see which coast is friendlier. You calculate the overall average connections.

Okay, let's look at the numbers.

The West Coast user base averages eight point two friends per user. The East Coast user base averages six point five friends. So the aggregate data heavily favors the West coast.

Really, but then.

You introduce a confounding variable. You stratify the data based on educational background users with a PhD and users without a PhD.

So you isolate the PhD subgroup and suddenly the East Coast data scientists average significantly more friends than the West Coast PhDs.

Okay, so the East Coast wins the PhD demographic right.

Then you isolate the non PhD subgroup, and once again, the East Coast data scientists average more friends than the West Coast non PhDs.

We stop right there.

What's wrong?

How is it mathematically possible for the East Coast to win in both individual subcategories but losing the total overall? I mean, that feels like it violates basic arithmetic.

It really does feel like magic. But it comes down to unequal weighting in the denominators of those subsets.

Okay, break that down for me.

The paradox is driven by the distribution of the confounding variable, in this case, the Phdso look at the underlying topology of the users across the entire platform. Users with PhDs simply have fewer connections. They average around three friends.

Which are they're busy doing research right?

While users without PhDs are highly active, averaging around ten to thirteen frames.

So a PhD basically acts as a massive downward weight on a group's average.

Precisely, now, look at the regional distribution. The East Coast user base is heavily saturated with PhDs.

Because of all the universities and research hopes.

Exactly, they have a massive concentration of these low connection users, pulling their overall denominator down. The West Coast user base, however, is overwhelmingly composed of non.

PhDs, part of culture, right right, So.

When you aggregate the data, the sheer volume of highly connected non PhDs on the West Coast mathematically drowns out the East Coast higher performance within the individual tiers.

Wow, so the regional bucketing completely masks the educational weighting completely.

If you hadn't joined the network table with the edguitational background table, you would have delivered a presentation to the board concluding that West Coast users are inherently more sociable.

And you would have optimized millions of dollars in marketing campaigns around that assumption, fully backed by mathematically flawless yet factually entirely backwards data.

And this, right here is the core lesson of doing data science from scratch. It forces you to recognize that statistical tools are not objective arbiters of truth.

No, they are mathematical lenses.

Exactly when we calculate a correlation or an aggregate mean, the foundational, unspoken assumption is always ceteris parabus, all else being equal, we assume the underlying distributions or uniform. Simpson's paradox proves how lethal that assumption can be to a business model.

The infrastructure of data science really requires rigor at every single layer of the stack. I mean, it demands type safe ingestion to prevent silent pipeline corruption. It demands a physiological understanding of how end users process the geometry of a visualization. It requires structuring memory into matrices to unlock computational scale. And it requires a deep, almost paranoid skepticism of aggregated metrics.

It does, and I want to leave you with a final thought to apply outside the boundaries of data sciencestor.

Let's hear it.

Every single day you are bombarded with viral statistics, algorithmic recommendations, and definitive correlations in the news. Every single one of those metrics was aggregated by someone making an assumption about uniformity. Knowing what you know now about Simpson's paradox, ask yourself, in the infinitely complex overlapping matrices of human behavior, is all else ever truly equal? When you see a definitive

Wow? It definitely changes how you interpret the next dashboard you look at it, the next headline, your read well, we'll leave it there for today's deep die into the underlying mechanics of data science. Remember to always verify your access, check your distributions for outliers, and we will catch you on the next deep dive.