Qualitative impact assessment protocol (2024)

Synonyms:

QUIP

Contributing author

Fiona RemnantRebekah Avard

Qualitative Impact Assessment Protocol (QuIP) is an impact evaluation approach, that draws on contribution analysis,without a control group that uses narrative causal statements elicited directly from intended project beneficiaries.

QuIP studies serve to provide an independentreality check of a predeterminedtheory of changewhich helps stakeholders to assess, learn from, and demonstrate the social impact of their work. QuIP’s approach places project beneficiaries’ voices at the centre of the evaluation, enabling them to share and feedback their experiences in an open, credible, and respectful way.

The QuIP gathers evidence of a project’s impact throughnarrative causal statementscollected directly from intended project beneficiaries. Respondents are asked to talk about the main changes in their lives over a pre-defined recall period and prompted to share what they perceive to be the maindriversof these changes, and to whom or what theyattributeany change - which may well be from multiple sources.

A control group is not required as evidence of attribution is sought through respondents’ own accounts ofcausal mechanismslinking X to Y alongside Z(key informant attribution) rather than by relying on statistical inference based on variable exposure to X. This narrative data is intended to be triangulated with other data and to complement quantitative evidence on changes in X, Y and Z obtained through routine project monitoring or other methods.

Typically, a QuIP study involves 24 semi-structuredinterviewsand fourfocus groups, conducted in the native language by highly-skilled, local researchers. However, this number is not fixed and will depend on the sampling approach used. The research team conducting interviews are independent andblindfoldedwhere appropriate; they are not aware who has commissioned the research or which project is being assessed. This helps to mitigate and reduce pro-project and confirmation bias, as well as enable a broader and more open discussion with respondents about all outcomes and drivers of change.

The QuIP approach to sampling is to select cases through rigorouspurposive sampling,rather than seeking a large representative sample. This means attempting to sample to ‘saturation’, where each new story adds little to the existing set of information. Where good monitoring data is available it can be used to decide the number, location and variation of respondents selected, based on differences in context, geography, treatment and/or positive and negative results from existing monitoring data.

A QuIP questionnaire includesopen ended questions, with supplementary prompts, as well asclosed questions. The questionnaire covers various areas of respondents’ lives, including domains such as health and sanitation, food consumption, intrahousehold relationships and wellbeing. QuIP questionnaires are designed to reflect the areas of people’s lives assumed to be affected from the project’s theory of change, but the questions are framed around outcomes rather than inputs in order to collect information more broadly about what has changed and to captureunintendedandunexpected outcomes.

Once the data has been collected, the QuIP uses a robustthematic codingmethod, systematically coding for drivers, outcomes and attribution. A common critique of qualitative research is the large quantity of data to deal with; QuIP addresses this partly by only coding stories ofchangerather than all data, but also by taking a very systematic and replicable approach to coding which speeds the process up. The results allow analysis ofkey stories of change- looking for emerging trends and patterns between different respondent types. Findings are presented in interactive dashboards and summary reports using visualisations and coded extracts. The QuIP aims to gather evidence about the causal processes of change, not toquantifyimpact, so it cannot provide average treatment effects or statistically representative counts. It does present rich, detailed stories of change in a digestible way - allowing every individual story to have a voice in the results.

Example

The QuIP was developed by researchers at the University of Bath’s Centre for Development Studies (CDS). After developing the method through a three-year DFID/ESRC funded research project, staff from CDS set up Bath Social Development Research (BSDR) to continue the development and dissemination of the QuIP. Since then BSDR has used QuIP in over 30 studies across 16 countries (April, 2019). Projects evaluated range from rural livelihoods interventions to microfinance initiatives.

For example, Tearfund have conducted three separate QuIP studies between 2016-18 to evaluate their Church and Community Mobilisation (CCM) project in Uganda, Sierra Leone and Bolivia. This is a project where there is no real baseline, beneficiaries are fluid and impacts cannot easily be measured quantitatively. The QuIP offered an opportunity to understand more about how people’s lives, beliefs and attitudes had changed, and what had influenced any changes.

In individual and focus group interviews in Uganda respondents reported positive changes, such as increased empowerment and improved community relationships, as well as negative changes, such as decreased material assets and reduced productivity. Analysis of the data provided a broad picture of drivers of change, including the impact of the CCM project amongst others. Over half the respondents cited CCM, unprompted, as a driver of positive change in their lives. Tearfund then validated the analysis with a wider group of respondents to understand what this meant for future community work. This helped to close the feedback loop and engage respondents with the data collected through interviews.

Find out about other project partners BSDR have worked with. You can also read more in-depth case studies in the QuIP bookAttributing Development Impact: QuIP Casebook(2019 - available for free online) - which also contains detailed theoretical background and practical guidelines.

Steps:

1. Co-design

Questionnaire and sampling strategy are developed with the commissioner of the research.

2. Case-selection

A mix of purposive and random sampling is used to select cases from the list of intended beneficiaries.

3. Data collection

Semi-structured individual interviews and focus group discussions are conducted by a local field team who are blindfolded to the nature of the project being evaluated.

4. Analysis

Analysis can be both exploratory and confirmatory. Thematic coding is used to analyse drivers, outcomes, and attribution.

5. Report

QuIP reports present and discuss the results from coded qualitative data (narrative casual statements). The data is also presented in useful graphs and visualisations in an interactive dashboard platform designed to engage project teams in analysis of reported perceptions of change.

6. Flexible feedback

The commissioning organisation are encouraged to share and discuss the results with the project team and beneficiaries.

Issues and considerations

Why blindfold?

Through double blindfolding (keeping both researchers and respondents unaware of who has commissioned the research or which project is being assessed), the QuIP methodreducesconfirmation bias and enables participants to be more honest and open about their experiences. Blindfolding is considered ethically acceptable under the utilitarian philosophy of working for the greater good. The rationale for blindfolding is explained to all parties before obtaining consent and everyone involved in the study is fully debriefed after the research project has finished. The term ‘blindfolded’ is used to reflect that this state is not permanent and the blindfold is removed at the end of the study. In feedback/debriefing sessions researchers and respondents have acknowledged the necessity and usefulness of blindfolding where possible, but it is not necessary or practical for all QuIP studies.