01 · Boundary only
Why is P(X=2) not the actual significance here?
Hint
The rule rejects for more than one count.
Worked solution
It omits rejection at 0 and 1. Use P(X≤2), the probability of the whole critical region.
Understand · explore · practise
Actual significance level
Calculate the null probability of a critical region, distinguish actual and nominal significance and handle independent repetitions without treating the level as a probability that a hypothesis is true.
Before you startCritical regions, complements and independent events.
01 / The actual level is a probability of rejection
Calculate it using the null distribution and the full rule.
Actual significance = P(X is in the critical region | H₀).
It is the chance of rejecting a true null hypothesis, assuming the null sampling model is correct.
The nominal level is the target used to construct the test. The actual level is what its discrete rejection rule achieves. The two need not be equal.
Measure the probability of the ruleExplore
All probabilities use X~B(12,0.5) under H₀. Choose a proposed rejection region; some are deliberately unsuitable for a 5% test.
02 / Measure a one-tailed region
Use the whole region, not just its boundary.
Watch: the region determines the actual level
Pause, replay or seek freely. The notes explain the same idea and stay in view.
For X~B(12,0.5) under H₀, a 5% lower-tailed test rejects when X≤2.Worked example
P(X≤2)=79/4096 ≈ 0.01928711
Sum the masses at 0,1 and 2.
Actual significance ≈ 1.928711%
Multiply the probability by 100 to express it as a percentage.
Nominal significance is 5%
The next possible lower region, X≤3, has probability about 7.300%, so it is too large.
02 · Percentage
A critical region has null probability 0.034. Give the actual significance as a percentage.
Hint
Multiply by 100.
Worked solution
3.4%, not 0.034%.
03 / Add the probabilities of disjoint tails
A two-tailed region has contributions from both ends.
Under the same null model, reject for X≤2 or X≥10.Worked example
P(X≤2)=79/4096
The lower region contains 0,1,2.
P(X≥10)=79/4096 by symmetry
The upper region contains 10,11,12.
Total = 158/4096≈0.03857422
The tails are disjoint, so actual significance is about 3.857422%.
03 · Two unequal tails
A two-tailed region has null tail probabilities 0.018 and 0.021. Find its actual significance.
Hint
The tail regions do not overlap.
Worked solution
0.039, or 3.9%. Do not double just one tail unless the probabilities are equal.
04 · Wrong operation
Should the two tail probabilities be multiplied?
Hint
Can one count lie in both disjoint tails?
Worked solution
No. The rule rejects in the lower OR upper tail, so add their probabilities. Multiplication would not represent this union.
04 / Discreteness limits the available levels
Check how the region was specified.
With an at-most construction, the actual level does not exceed the target. Some exercises explicitly ask for each tail to be as close as possible to a target instead. That different convention can allow a tail to exceed its target; calculate and state the actual total rather than assuming it. The next lesson compares these constructions carefully.
05 · Infer from a label
A question calls a test “5%”. May you always substitute 0.05 for its actual false-rejection probability?
Hint
Was the exact region probability calculated?
Worked solution
No. A discrete test can achieve a smaller level, and an explicit closest-tail convention needs separate checking. Use the actual probability of the stated rule.
06 · Empty region
What actual significance does an empty critical region have?
Hint
Can the test ever reject?
Worked solution
Zero. This does not prove H₀; it means the rule never rejects.
05 / Nonrejection is a complementary event
Keep the conditioning under H₀.
For the one-tailed region X≤2, the probability of not rejecting under H₀ is 1 − 79/4096= 4017/4096 ≈ 0.98071289. This is a probability about sample outcomes when H₀ holds. It is not a 98.071% probability that H₀ is true after a nonrejection.
07 · Interpret carefully
Explain what actual significance0.02 means.
Hint
State the condition as well as the outcome.
Worked solution
If H₀ and the sampling assumptions hold, this test rule rejects with probability 0.02. It does not mean a 2% probability that H₀ is true.
08 · Wrong certainty
A test does not reject. Is its null therefore correct?
Hint
A test may fail to detect a real change.
Worked solution
No. Report insufficient evidence against H₀ at the stated level. Nonrejection is not proof.
06 / For independent repeats, use the actual level
Different repetition rules have different probabilities.
Two independent samples each use a test whose actual null rejection probability is a=79/4096. Assume H₀ holds for both.Worked example
Both tests reject: a²≈0.00037199
Multiply because the test outcomes are independent.
At least one rejects: 1−(1−a)²≈0.03820223
Complement the event that neither rejects.
These are different combined decision rules
Neither probability is obtained by blindly squaring the nominal 0.05.
09 · Both with a simpler level
Two independent tests each have actual null rejection probability 0.03. Find the probability both reject when both nulls hold.
Hint
Multiply 0.03 by itself.
Worked solution
0.0009, or 0.09%.
10 · At least one
For the same two tests, find the probability at least one rejects.
Hint
Use the complement of neither.
Worked solution
1 − 0.97² = 0.0591, or 5.91%.
11 · Same data
May you multiply the rejection probabilities when both tests use overlapping observations?
Hint
Independence has not been established.
Worked solution
Not without justification. Shared data can make the test outcomes dependent. The product calculation needs independence.
07 / Avoid three common probability confusions
A level, a p-value and a hypothesis are different objects.
12 · Level versus p-value
How does actual significance differ from the p-value for one observed sample?
Hint
One concerns the complete fixed rejection rule.
Worked solution
Actual significance is the null probability of the full rejection region. A p-value measures the observed-or-more-extreme evidence for the particular sample, under the specified test convention.
13 · Repeated searching
Why can repeatedly collecting new samples until one rejects raise the chance of a false alarm?
Hint
More opportunities to reject alter the overall rule.
Worked solution
The combined procedure rejects if any attempt rejects. Even independent tests can then have an overall false-rejection probability greater than the level of one test.
14 · Invalid 5% claim
A proposed rejection rule includes every supported count. What is its actual significance, and can it be an at-most 5% test?
Hint
It always rejects.
Worked solution
Actual significance is 1, or 100%. It cannot satisfy an at-most 5% requirement.
08 / Calculate the probability of the stated rule
Use the null model and distinguish union from repetition.
Sum every rejecting outcome under H₀. Add disjoint tail probabilities, convert to a percentage carefully and distinguish the result from the nominal target. For repeated independent tests, apply the stated combined rule using actual levels.
Section 1 of 8 · The actual level is a probability of rejection