Assessing Rural Transformations: Piloting a Qualitative Impact Protocol in Malawi and Ethiopia

This paper contributes to the literature on qualitative approaches to impact evaluation, particularly in complex contexts. It reports on substantive and methodological findings from four pilot studies of a protocol for qualitative impact evaluation of NGO sponsored rural development projects in Malawi and Ethiopia. Two of the projects aimed to build resilience to climate change through support for a spectrum of livelihood diversification activities, while two focused on smallholder involvement in the value chains of specific cash crops. The protocol was designed and tested through action research with the aim of generating evidence in a credible, timely and cost-effective way to confirm the causal theories underpinning project actions, as well as to explore incidental sources of change and unanticipated effects. The paper describes the methodology, provides an overview of findings and reflects on lessons learnt in addressing problems of attribution, confirmation bias and generalizability. It suggests scope for further development of responses to these issues based on self-reported attribution, partial blinding of respondents and nesting qualitative evaluation in quantitative monitoring.

Key words: Impact evaluation, qualitative methods, food security, climate change adaptation, rural livelihoods, Malawi, Ethiopia, NGOs, confirmation bias, mixed methods, attribution