Understand · explore · practise
Check who a sampling frame includes, spot coverage and response problems, and explain why a larger sample does not automatically remove bias. Original scenarios and worked practice.
Before you startPopulations, samples, means and percentages.
01 / Population versus frame
Target population → sampling frame → selected sample → usable responses
Each step can lose or distort information.
A library wants to know how many of its current members use the online catalogue. A current membership register can include people who never enter the building. A list of building visitors cannot cover those remote-only members.
Use the model to compare these frames. The known percentages make the consequences visible; in a real investigation the excluded group’s behaviour may be unknown.
Bars show group sizes, not sample sizes. Grey marks an excluded group.
Frame covers 600 of 600 members (100%).
No group is excluded by this frame.
Known catalogue use in this invented population: 264/600 = 44%.
Of the 360 visitors, 144 use the online catalogue. Of the 240 remote-only members, 120 use it. The figures are invented. Within-group percentages describe the complete groups, not guaranteed results of smaller samples.
02 / What makes a useful frame?
A sampling frame is the list or other record from which units can be selected: for example, an up-to-date enrolment register for current students, or serial numbers for items in a specified batch. A useful frame covers the target population, identifies units clearly and avoids duplicate entries.
A university wants to sample students currently enrolled on its evening courses. Suggest a frame and one check before use.
Match enrolment status, course type and survey date.
Use the current evening-course enrolment register. Remove withdrawn students, add recent enrolments and check for duplicate students if some attend more than one course.
Would a public telephone directory be a complete frame for all adults in a town?
Not every adult has a listed personal entry.
No. Unlisted residents and people without a listed number are missing; an entry may represent a household rather than one adult. The frame does not match the target units.
03 / Undercoverage
Frame: 360 of the 600 members
The remaining 240 cannot be selected.
Coverage = 360/600 = 60%
Forty per cent of the target population is absent from the frame.
Catalogue use among visitors = 144/360 = 40%
The full population rate is (144 + 120)/600 = 44%.
Using the whole visitor group still gives 40%
Collecting more within this restricted frame does not recover the missing group.
Pause, replay or seek freely. The notes explain the same idea and stay in view.
In this invented example, using 40% for the population rate underestimates 44% by how much?
State the difference in percentage points.
Four percentage points. This differs from a 4% relative change; no relative-change calculation is needed here.
04 / Duplicates and ineligible entries
There are five equally selectable entries
Member A appears twice.
A has probability 2/5 of selection on one draw
Each other member has probability 1/5.
The four people do not have equal selection chances
Clean the frame before applying a simple random method.
A current staff list still contains last year’s leavers and omits new starters. Identify two frame problems.
Consider both extra and missing units.
Leavers are ineligible entries; new starters are undercovered. Update the list to match current staff before selection.
A sports centre samples membership records, but people with two memberships appear twice. Why might this distort a survey of people?
The target unit is a person, not a membership record.
Those people can have more chances of selection. Deduplicate by person or use a design which explicitly accounts for unequal selection probabilities.
05 / Nonresponse
Suppose 100 members are randomly selected from a complete frame but only 42 reply. The chosen sample has size 100; the respondent set has size 42. If the chance of replying is related to the answer, the responses may be biased.
A response rate is a useful description, but its size alone does not reveal the direction or size of nonresponse bias. Consider who is missing and why.
Of 160 invited customers, 104 reply. Find the response rate and explain why this does not establish that the replies are representative.
Divide replies by invitations.
The response rate is 104/160 = 65%. Respondents may differ systematically from nonrespondents; the percentage alone cannot settle representativeness.
Suggest one improvement when an email-only questionnaire misses members who rarely use email.
Provide another way to participate without changing the target population.
Offer a suitable alternative such as a paper or telephone response, with consistent questions, and follow up selected nonrespondents where appropriate. This can improve access but does not guarantee all bias disappears.
06 / Measurement and question wording
A scale with a constant offset, an unclear time period, or a leading survey question can distort measurements. “How much do you agree that our excellent service deserves an award?” pushes a favourable view. A neutral question offers balanced response options and a clear reference period.
Improve: “Don’t you agree our new timetable is much better?”
Allow improvement, no change and deterioration.
For example: “Compared with last term, how well does the current timetable meet your needs?” Offer better, about the same, worse and not sure/not applicable, with a clearly stated comparison period.
A scale adds 0.2 kg to every recorded mass. Does taking a larger sample remove this offset from the sample mean?
The same error affects each observation.
No. The sample mean is also increased by 0.2 kg. Calibrate or correct the measurement process.
07 / Bias versus variation
Sampling variation arises because different samples include different units. Bias is a systematic tendency for the method to produce distorted results. One sample being high does not prove the method is biased; a large sample also does not prove the method is sound.
In the library model, we know the excluded group’s rate, so we can demonstrate the effect. Without that information we may identify a risk of bias without knowing its direction.
A survey excludes people who work nights. No information about their opinions is available. Can you confidently say the survey overestimates satisfaction?
Exclusion identifies a possible problem, not its direction.
No. Their satisfaction might be higher or lower. Explain the undercoverage risk without asserting an unsupported direction.
08 / Match the remedy to the problem
Missing groups: improve coverage. Duplicates: clean the frame. Nonresponse: improve access and follow-up. Measurement: improve the instrument or question.
A useful criticism names the problem and links it to a practical repair.
Random selection within a defective frame does not bring excluded units back. Increasing sample size under a sound design can reduce sampling variability, but does not automatically repair the other problems.
A council asks only people leaving its leisure centre whether a new swimming pool should be funded. It wants the views of all adult residents. Give a limitation and an improvement.
Who is absent from the selection location?
Residents who do not use the leisure centre are excluded and may have different priorities. Use a suitable frame covering adult residents, select across that population, and provide accessible ways to respond.
09 / Audit the route from population to answer
Survey A randomly selects 80 people from a current register of all eligible members. Survey B asks 400 volunteers from a social-media page. Is B necessarily more reliable for all members?
Compare coverage and self-selection as well as size.
No. Page users may differ from other members, and volunteering may depend on strength of opinion. A’s random selection from a suitable frame is a stronger starting point, although its response and measurement processes still need checking.
Section 1 of 9 · Population versus frame