Comparing samples can feel like judging cupcakes from different bakeries. Some cupcakes are bigger. Some are sweeter. Some were baked on a rainy day. In statistics, these differences are called sources of variation. They matter a lot.
TLDR: When we compare samples, we must ask where the differences come from. Some differences come from real effects, while others come from noise, group differences, or measurement error. Simple tests like the t test work in some cases, but more complex situations need tools like ANOVA, mixed models, or nonparametric tests. The best method depends on how your data was collected.
Why variation is the main character
Variation means things are not all the same. That is normal. People have different heights. Plants grow at different speeds. Machines produce slightly different parts. Even the same person can give different test results on different days.
When we compare samples, we ask a simple question:
Are these samples really different, or are we just seeing random wiggles?
That is the whole game. Statistics helps us avoid being fooled by wiggles.
Common sources of variation
Before choosing a method, we need to name the possible sources of variation. Think of them as suspects in a mystery.
- Random variation: Natural chance. It happens even when nothing special is going on.
- Group variation: Real differences between groups, such as men and women, farms and factories, or treatment and control groups.
- Measurement error: Mistakes or limits in tools. A scale may be slightly off.
- Within sample variation: Differences inside one group.
- Between sample variation: Differences from one group to another.
- Batch variation: Data collected at different times, places, or conditions.
- Observer variation: Different people may measure things in slightly different ways.
If you ignore these sources, your result may look more exciting than it is. That is how statistics gets spicy in the wrong way.
The basic case: two independent samples
Let us start simple. You have two groups. Group A drinks coffee. Group B drinks tea. You measure reaction time. Each person is in only one group.
This is called an independent samples setup. The usual tool is the independent samples t test.
It checks whether the two group means are different enough to seem real. It also considers spread. Big spread makes differences harder to trust. Small spread makes differences easier to trust.
Use a t test when:
- You compare two groups.
- The outcome is a number.
- The groups are separate.
- The data is roughly bell shaped, or the sample size is decent.
But wait. There is a twist.
When variation is unequal: Welch’s t test
Sometimes one group is much more spread out than the other. Maybe the coffee group has reaction times all over the place. The tea group is calm and consistent.
A regular t test assumes the two groups have similar variation. If that is not true, use Welch’s t test.
Welch’s t test is like a regular t test with better manners. It does not force both groups to have the same variance. It is often safer. Many statisticians prefer it as the default for two independent groups.
Simple rule: If you are unsure whether the variation is equal, Welch is a friendly choice.
Paired samples: when data has a buddy
Now imagine you measure the same people before and after exercise. Each “before” value belongs to an “after” value. These are not independent. They are paired.
For this, use a paired t test.
The paired t test compares the differences within each pair. It asks, “Did people change?” It does not treat the before group and after group like strangers.
Use it when:
- The same person is measured twice.
- Twins or matched subjects are compared.
- Each value has a natural partner.
This method removes some person to person variation. That can make the test more powerful. It is like comparing each cupcake to its own earlier version.
More than two groups: meet ANOVA
What if you compare three diets? Or four teaching methods? Or five fertilizers?
You could run many t tests. But that creates a problem. The more tests you run, the more likely you are to find a “difference” by accident.
This is where ANOVA helps. ANOVA means analysis of variance.
That name sounds boring. But the idea is neat.
ANOVA compares:
- Variation between groups
- Variation inside groups
If between group variation is large compared with within group variation, ANOVA says, “Something may be happening here.”
A basic ANOVA tells you whether at least one group differs. It does not always tell you which one. For that, you use post hoc tests, such as Tukey’s test.
Two sources at once: two way ANOVA
Life rarely gives us only one factor. Suppose you study plant growth. You test fertilizer type and sunlight level. That gives you two sources of variation.
A two way ANOVA can test:
- The effect of fertilizer.
- The effect of sunlight.
- The interaction between fertilizer and sunlight.
An interaction means one factor changes the effect of another. Maybe Fertilizer A works great in full sun but poorly in shade. That is useful to know.
Without testing interaction, you might miss the best part of the story.
Nested variation: data inside data
Sometimes samples are grouped inside larger groups. Students are inside classrooms. Classrooms are inside schools. Patients are inside hospitals.
This is called nested data. It has layers.
If you ignore the layers, you may pretend you have more independent information than you really do. That makes results look stronger than they are.
For nested data, use mixed effects models, also called multilevel models.
These models can handle:
- Students within classes.
- Patients within clinics.
- Measurements within people.
- Products within batches.
Mixed models are powerful because they separate different sources of variation. They can say, “Some variation is from students. Some is from classrooms. Some is just random noise.”
Repeated measurements: same subject, many times
What if you measure blood pressure every week for six months? Now each person has many values. These values are related.
A simple t test will not do. A basic ANOVA may also struggle.
You may need:
- Repeated measures ANOVA
- Mixed effects models
- Time series methods
These methods understand that today’s measurement may be related to last week’s measurement. Data has memory. Statistics should respect that.
When data is not normal
Some data is messy. Very messy. It may be skewed. It may have outliers. It may be counts, ranks, or categories.
If the usual assumptions fail, consider nonparametric tests. These tests make fewer assumptions about the shape of the data.
Common options include:
- Mann Whitney U test: For two independent groups.
- Wilcoxon signed rank test: For paired samples.
- Kruskal Wallis test: For more than two independent groups.
- Friedman test: For repeated or matched groups.
These methods often compare ranks instead of raw values. They are useful when data is weird but still meaningful.
Different sample sizes
Different sample sizes are common. Maybe one group has 20 people and another has 200. That is not automatically bad. But it can affect the analysis.
Unequal sample sizes become more serious when variation is also unequal. This can make standard methods less reliable.
Good choices include:
- Welch’s t test for two groups.
- Welch’s ANOVA for many groups.
- Robust methods when outliers are present.
- Bootstrapping when assumptions are unclear.
Bootstrapping is like remixing your data many times. The computer resamples the data and estimates uncertainty. It is practical and flexible.
A simple decision guide
Here is a quick way to choose a method:
- Two separate groups? Use a t test, or Welch’s t test if variances differ.
- Two paired groups? Use a paired t test.
- More than two groups? Use ANOVA.
- More than one factor? Use two way ANOVA or a related model.
- Nested or repeated data? Use mixed effects models.
- Messy non normal data? Use nonparametric or robust methods.
Final thoughts
Comparing samples is not just about asking if numbers are different. It is about asking why they are different. The source of variation decides the method.
If your data is simple, simple tools work well. If your data has layers, pairs, batches, or repeated measures, use a method that understands that structure.
Statistics is not magic. It is more like detective work with calculators. Find the suspects. Follow the variation. Then choose the right test.
