image: What Works for Children
home :: about the project :: resources :: for practitioners :: events :: latest news :: contacts


useful terms for understanding and assessing research

Introduction: Why Assess Research?
When reading a research report it is essential to think about its quality and any possible bias in the study. This may result from systematic errors in the way the study was designed or in the analysis of data. Some typical factors that may result in bias are:
  • The wording of a question asked (which may encourage a particular response)
  • The type of interviewer (e.g. a male or female interviewer may lead to different responses)
  • The selection of people to be studied - is the sample truly representative of the population about whom claims are being made? (e.g. does a study purporting to be about young people in general only include young people recruited from a popular school in an afluent area?)

Critical appraisal is a technique for reading research and working out how valid and relevant the research is. Critical appraisal helps us to work out how likely it is that the results of research are biased because of the way the research was carried out.

In studies of interventions (services or activities) a study may conclude that an intervention was effective in dealing with a problem. Critical appraisal can help us work out if the intervention only appeared to be effective because of bias in the research methods. This may happen, for example, if:

  • No comparison group was used and those receiving the intervention would have got better anyway
  • All the people for whom the intervention didn't work left the study and their results were not included
  • The comparison group, if used, was not really comparable to the group given the intervention

This Glossary...
explains different types of study design and some key concepts in quantitative and qualitative research.
More information on how to read research is available in our Evidence Guide. The glossary is arranged alphabetically. Words underlined are explained in other sections of the glossary.

 

Glossary

 

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
A
Action Research
The practitioners within a service take on a researching role and systematically observe and record whilst also carrying out their work. A researcher will often guide and supervise the process, but the practitioner carries out the data collection and interprets the findings. An action research project will often have service change as a central aim, where the change comes as a result of the research findings. As opposed to an in-house evaluation, the findings from action research are often publicised to enable the findings to be scrutinised and tested out elsewhere.
B
 

C

Case study
A case study is used when the researcher wants to investigate the complexities of a single case and its interaction with the surroundings. A case study needs to be described in detail so that the reader may relate the findings to a similar case.

 
Case-control studies
Individuals with a particular problem are 'matched' with people (the control group) without the problem. The exposure of the two groups to possible causes is then compared. This can be used to investigate risk factors.

Example: what are the risk factors for suicide in adolescence?

A group of forty young people aged 15-18 with one suicide attempt (or more) are matched with another similar sized group of 15-18 year olds who do not have a record of attempted suicide. The matching ensures that the social and economic environment of the groups is similar (eg urban, school drop-outs, single parent families). The researchers will have various theories about risk factors (what is causing the suicide attempts). For example, one risk factor could be a poor relationship with the parents. The researchers would look at whether there was a difference in this between the two groups. Similarly, they could look at other factors such as relationships outside the family, involvement in work, peer relationships, hobbies etc.

 
Cohort studies
These collect information from or about children at regular intervals, often from shortly after birth until later in adulthood. Cohort studies can be used to investigate associations between early development and experiences, and later outcomes

e.g. what distinguishes those people who are able to move out of poverty?

A limitation of both case-control and cohort studies is that there may be other factors not measured which are responsible for the differences in outcomes between the groups in the study. For example, if we compare high accident families with low accident families to identify risk factors of home injury, we will be in danger of overlooking things. We might not realise that in one area the health visitors are running an accident awareness campaign, or local stores do not stock a certain type of safety equipment.

 

Confidence interval (CI)
A confidence interval associated with a result tells us the likelihood that the same result would be found if the whole population were studied rather than just a sample. For example a newspaper might report that the average IQ of researchers is 99. If the 95% confidence interval is 80-120 this means that we can be 95% sure that the average IQ of all researchers of the type sampled, will be between 80 and 120.

A measure of effect tells us something about what the intervention does for a particular sample. For example, one research study found that family and parenting programmes decreased the time spent by delinquent young people in institutions by an average of 51.34 days.[1] The 95% confidence interval was 30.16 to 72.52 days. This means that we can be 95% certain that, when delivered to other similar samples in a similar way, these types of family and parenting programmes will reduce the time spent in institutions by between 30.16 and 72.52 days.

If we are examining the confidence interval around a mean difference (i.e. the difference between average results for the intervention group and the control group), and the interval includes the value zero, the relationship between the intervention and the outcome is not statistically significant as it includes the possibility that there is zero effect.

We should examine confidence intervals carefully, because this lack of statistical significance may be because the sample is small, rather than because the treatment is not effective (in which case there will usually be a large confidence interval). Equally in a very large sample a very small and possibly unimportant effect may be statistically significant.

 

Control group
A control group is used in order to try to establish whether any effect found in the intervention group was due to the intervention or would have occurred anyway. The control group is the comparison group that gets a different service/intervention (or no service/intervention) from the intervention group.

 

Critical Appraisal
A systematic way of assessing a research study, and considering it in terms of validity, bias, results and relevance to your own work.

D

Document analysis
The researcher reads systematically through documents to look for answers to a research question. In social research this can be all sorts of documents; meeting minutes, regulations, letters, media coverage… Some researchers will ask their respondents (children or adults) to record a diary related to certain activities (e.g. medication, home work, diet, leisure activities).

 
E

Effectiveness
Describes the extent to which an intervention improves the outcome(s) for those receiving it and the extent to which these benefits outweigh the harm (if any) caused by the intervention.

 
F

Focus groups
The researcher facilitates and leads a group of individuals through a discussion around a specific topic. Focus groups can be structured to varying degrees and the researcher may chose to be directive or take on a more observing role, depending on the objective of the research.

 
G

Grounded theory
A grounded theory approach involves the systematic gathering and analysis of data (gathered, for example, from observation, interviews or focus groups). The approach involves the development of theory alongside the analysis of data (rather than as a precursor to analysis) as the researcherl looks for issues that repeatedly emerge from the data.

 
H
 
I

Intervention group
The group that receives an intervention (service, medicine, treatment). See also case-control studies, and randomised controlled trials.

 

Intervention
A type of service, programme or policy (e.g. health promotion campaigns) or (in medicine) a drug.

 
J
 
K
 
L
 
M

Meta-analysis
A statistical technique that pools the results from several studies into one overall estimate of the effect of an intervention. See also systematic review.

 
N
 
O

Observation
In qualitative research, observation may be used as a method to record behaviour and interaction within groups or individuals. The observations may be audio or video taped or put down in words. The researcher may actively take part in the interaction, depending on the research objective.

 

Odds
Odds give a ratio of occurrence to non-occurrence of an event. Odds are a way of expressing the likelihood of an event such as reconviction after an intervention. The odds of reconviction would be the expected number of young offenders reconvicted divided by the expected number of young offenders not reconvicted. If three out of every ten young offenders receiving the intervention is reconvicted the odds would be 3/7 = 0.4 (see further explanation below under odds ratio).

 

Odds ratio (OR)
The odds ratio (OR) looks at the relationship between the effect in the control versus the intervention group. It is the ratio of the odds of the event occurring in the experimental group relative to the odds of the event occurring in the control group. This is sometimes used as a measure of the effectiveness of an intervention. The OR is calculated by dividing the odds of the event occurring in the intervention group by the odds of it occurring in the control group.

Example: What effect do parenting programmes have on reconviction?
(NB: this is a fictional example)


In the intervention group parents of 32 young people received a parenting programme. In the control group parents of 30 young people did not.

  Parenting programme Control
Reconvicted
2
20
Not reconvicted
30
10

Odds that those whose parents receive parenting programmes are reconvicted: 2/30 = 0.07
Odds that those whose parents do not receive parenting programmes are reconvicted: 20/10 = 2

Odds ratio: 0.07/2 = 0.035

If the event is a negative event, such as reconviction and the OR < 1, then the treatment may be effective. In the example above OR 0.035, which means that parenting programmes could have an effect on reconvictions.

If OR = 1 the intervention has no effect (i.e. no difference between the intervention and the control group). An OR > 1 would suggest that the treatment of interest was actually less effective than no treatment (or an alternative).

On its own, an OR is not very informative – a confidence interval is also needed (see above).

 

Outcome
Changes or effects that happen as a result of the intervention. Outcomes may be for: individuals, families, communities or organisations.

e.g. a reduction in offending behaviour may be an outcome of an (effective) offending prevention programme.

 
P

P-value
A p-value expresses the likelihood thata result was due to chance. E.g. p = .03 means that there is a 3% chance that the population value lies outside the confidence interval.

 
Population
In a statistical sense, the population is the complete set of whatever is the object of study (individuals, objects or scores), from which a sample may be taken in order to make inferences about the whole population.
 
Population surveys
A sample of a chosen population, or the whole population (e.g. in the case of the UK census) is asked to provide responses to questions on the subject of interest. Can be used to measure the prevalence of problems.

e.g. how common is depression?
 
Power
The probability that an experiment will be able to detect an effect of a variable (e.g. an intervention) if the variable has a true effect.
 
Q

Qualitative research
Concerned with the meanings people give to their experiences and how they make sense of the world. Often studies people in their natural settings.  A range of methods can be used including participant and non-participant observation, talking with people (interviews, focus groups) and reading what they have written. Can be used to find out about social processes and what matters to people, how these vary in different circumstances, and why.

e.g. what do young people and volunteers value in mentoring relationships?

 

Quasi-experimental studies
Used to examine the effects of an intervention. One or more control groups are used but participants are not randomly allocated to one group or the other. 'Naturally-occurring' control groups are often used. Commonly, one group will receive a particular service while the other does not, or receives another type of service. In a quasi-experimental study the two groups are sometimes matched on key characteristics. However, it is not possible to match on all relevant factors including unknown ones. It may be that the service is delivered to the group that needs it the most enhancing the risk of bias as the two groups are not truly comparable.

Example: does mentoring reduce offending behaviour?
One group receives mentoring the other does not. There is a risk that the young people's involvement in offending will influence the allocation of service. If those who offend less are given a mentor because they are perceived to be easier to work with and compared with a group already offending more, our findings will be biased.

Practical reasons will sometimes require a quasi-experimental design. When evaluating the impact from changes to the environment such as traffic calming or playground improvement an area-wide implementation is necessary. In these cases you can only compare one area with another because it is impossible to randomly decide who will receive the intervention.

 
R

Random sample
In a random sample each case (person/subject) in the population of interest has an equal chance of being included in the sample.

 
Randomised controlled trials (RCT)
An experiment in which individuals are randomly allocated to either receive or not receive an intervention (or to receive a different intervention). The two groups are then followed up to determine the effect of the intervention, by identifying differences between those who did and those who did not receive it.

Example: What is more effective in reducing offending behaviour in young people - parenting programmes or mentoring?
The amount of offending behaviour is measured at baseline (police records, parent- and self-report, school records) for all young people. All the families agreeing to participate in the study are then randomly allocated to receive mentoring or attend parenting groups or be on a waiting list (
control group). 12 months later the offending behaviour is again measured and the groups compared. Preferably, further measures are taken one, two or more years later to measure long-term effects.

 

Reliability
Refers to the likelihood that the same results would be found if the study was repeated in the same way.

 
S

Sample
A subset of cases (individuals, objects or scores) selected from the population to be studied.

 

Sample size and power
A 'good' study involves some consideration of sample size  - i.e.  the number of participants to recruit to the study. This is a crucial determinant of whether a difference will be detected if it really exists. Sometimes the number of participants in a study is chosen because the number 'seems appropriate', or because that is how many participants the study can afford to test or interview. However, the appropriate size for a particular study depends on the likely size of the effect you are trying to detect - e.g. the likely size of the Odds Ratio (OR), or the magnitude of the difference between two means. Where the effect is likely to be small, then larger study numbers are required in order to detect the effect.

 

Semi-structured interview
The researcher has a set of themes they want to discuss with a respondent, but they are not bound by these themes, and can investigate emerging issues arising during the course of the interview.

 

Statistical significance (see P-value)
Significance levels show you how likely it is that a result is due to chance. The level at which a result is said to be 'significant' is arbitrary but the most common level used is .05. If a result is said to be significant at the .05 level (p>.05) this means that the finding has a five percent (.05) chance of not being true (see also p-value). A statistically significant result does not necessarily mean that a result is significant in a practical sense. E.g. a very small and unimportant effect may be found to be statistically significant if a very large sample were studied.

 

Structured interview
The researcher has a set number of questions which are asked to all respondents.

 

Systematic Review
A systematic review (SR) is a critical assessment and evaluation of existing research that addresses a specific question by following a fixed approach for locating, appraising and analysing all studies addressing the question of interest. SRs can be used to look at the effectiveness of interventions e.g. does mentoring for young offenders reduce their likelihood of re-offending? When a systematic review pools data across studies to provide an estimate of the overall treatment effect, we call it a meta-analysis.

 
T

Triangulation
The use of more than one theory, methods, data sources or researchers to enhance the rigour of the research.

Example: A researcher has interviewed 26 young people about their first year in secondary school.
S/he may choose to give the data to a second researcher for analysis, and in addition interview teachers and parents of the young people to hear their stories. School records of achievement and attendance may be of relevance, as well as looking at school rules and timetables. If some of the young people's stories illustrate a more general point, a case study may be carried out.
 
U
 
V

Validity
If the internal validity is high, the study has been designed and carried out in such a way as to avoid systematic bias - which means that it will give you a good estimate of the effectiveness of the intervention. External validity is also sometimes called transferability or generalisability, and refers to the extent that you can generalise the findings from one study and apply them to other populations, settings and arrangements.

 
W
 
X
 
Y
 
Z

   1.   Woolfenden, S. R., Williams, K., and Peat, J. Family and parenting interventions in children and adolescents with conduct  disorder and delinquency aged 10-17. The Cochrane Library (Oxford) 1 (CD003015), (1) 2001;(Oxford)1-26.
page last updated 23/08/2006
copyright : feedback