9.7 - Evaluating a Method - Errors, Anomalies & Reliability
What an evaluation question looks like
- After the results are in, the exam asks you to judge how well the method worked.
- There's no single fixed question - a paper picks from a small set of evaluation tasks, each following a fixed rule, so the marks are there once you know the rule.
The commonest question - Error and improvement
During the experiment the temperature of the water bath fell.
(i) Identify a source of error in this investigation.
(ii) Suggest an improvement that would reduce this error.
The two parts are marked together: the error only scores if the improvement you give actually fixes that error.
The four evaluation tasks
Alongside the error-and-improvement pair, papers ask about anomalies and reliability:
- Identify a source of error, and pair it with an improvement that reduces it.
- Spot an anomalous result and say what to do with it.
- Explain why the investigation is repeated, or why a mean is taken.
- Explain why a large or representative sample is used.
Sources of error and improvements
- A source of error is something in the method that makes a reading less accurate.
- Every error must be paired with an improvement that actually reduces it - an error on its own scores nothing (just as a safety hazard scores nothing without a precaution).
- Always name the error, then say what you'd change to fix that particular error.
Common errors and their improvements
The same handful of errors come up across different experiments. Each improvement targets its own error:
| Source of error | Why it affects the result | Improvement that fixes it |
|---|---|---|
| Adding a liquid drop by drop from a dropping pipette | Drop size varies, so the volume added isn't consistent | Measure the volume with a syringe or measuring cylinder |
| Judging the end-point (a colour change) by eye | The exact moment of change is a matter of opinion, so different people time it differently | Compare against a colour standard, a prepared colour chart, or a colorimeter |
| The water bath cooling during the run | The temperature is meant to be held constant, so a drift changes the rate | Use a thermostatically controlled water bath |
| Shaking or stirring the tubes by hand | The amount of mixing differs from one tube to the next | Stir each tube at a fixed speed with a mechanical stirrer |
- The improvement must match the error: a syringe fixes an inconsistent volume, a thermostatic bath fixes a drifting temperature.
- Reaching for the wrong pair - such as "use a syringe" to fix a temperature that fell - does not score.
Anomalies and reliability
- An anomaly is a result that doesn't fit the pattern of the others - an odd one out, well away from the trend the rest of the readings make.
- When you find one, identify it and then leave it out when you calculate the mean, so one stray reading doesn't drag the average off.
- You can't tell a reading is anomalous until you have others to compare it against - which is one reason for repeating readings.
Repeating and taking a mean makes a result reliable
- Reliability means getting consistent values in repeats of the experiment (SQA's wording).
- To achieve it, repeat the investigation and take a mean of the readings at each value of the variable you changed.
- Two reasons sit behind repeating and taking a mean:
- A single reading might be an anomaly you'd never spot on its own.
- Consistent values across the repeats show the result is not a one-off.
Reliability vs validity
The difference between reliability and validity is mark-critical - read which one the question is after:
- Improving the results - Repeats and a mean earn the reliability mark.
- Why other variables were kept the same - The answer is about validity (a fair test); "reliable" is wrong here.
A representative sample
- A representative sample is a separate idea. It comes up whenever the experiment measures a number of living things rather than repeating one measurement.
- Measuring just one tells you little, because that one might be unusual - e.g. finding the average length of a leaf, or how much water a raisin takes up.
- Instead, measure ten, take a mean so the figure represents the group, and choose them without bias.
Common exam questions
Question 1
A student times a colour change by watching for it by eye. Give one source of error and an improvement that would reduce it.
The end-point is judged by eye, so the exact moment is subjective; compare the colour against a prepared colour standard or use a colorimeter so the end-point is decided the same way each time.
Question 2
Why does "the water bath is a source of error" on its own not score?
Because an error only scores when it is paired with a matching improvement; you also need to say what you would change, such as using a thermostatically controlled water bath to hold the temperature steady.
Question 3
One of five readings is far higher than the other four. What is it called, and what should you do with it when finding the mean?
It is an anomaly, and you leave it out of the mean, calculating the average from the four consistent readings.
Question 4
Describe how the reliability of a set of results could be improved.
Repeat the investigation and take a mean at each value of the variable you changed, so that consistent values across the repeats show the result is reliable.
Question 5
A question asks why the other variables were kept the same. Why is "to make the results reliable" the wrong answer?
Because keeping the other variables the same is what makes the test fair, so the answer is about validity, not reliability; reliability comes from repeating the investigation and averaging.
Question 6
A student measured the mass gained by one raisin left in water. Why is measuring several raisins better?
Because one raisin might be unusual and not represent the rest; measuring several and taking a mean gives a representative result and reduces the effect of any single odd one.