SUPPORTING IMPACTSupporting impact across a multi-dimensional research programme
across a multi-dimensional research programme*
The Evidence and Policy Group of the DFID-ESRC Growth Research Programme helps academic researchers improve the impact of their work on policy and practice; influencing their understanding of how impact happens and providing opportunities for them to engage with policymakers and practitioners. The research programme is multidimensional, and the chapter outlines the implications for how a facilitating organisation such as the Evidence and Policy Group can act most effectively. The key is ‘strategic opportunism’: the group works by setting a general direction rather than specific objectives and responding to opportunities for impact as they arise. This means a flexible approach to planning and budgeting that encourages innovation and building relationships to create opportunities.
The DFID-ESRC Growth Research Programme (DEGRP) contributes to evidence-informed policymaking for inclusive and sustainable growth in low-income countries. A jointly funded initiative launched in 2011 by the Department for International Development (DFID) and the Economic and Social Research Council (ESRC), it has funds of £20.9m spread over four themes: agriculture, finance, innovation and the relationships between China and the African continent. The programme funds 44 different projects: 19 focusing on agriculture, 11 on innovation, nine on finance and five on China–Africa relationships.
The DEGRP Evidence and Policy Group (EPG) was set up in 2012 to support researchers in achieving impact. An interdisciplinary team from the Overseas Development Institute (ODI) was contracted to help the programme and its individual projects maximise their profile and the uptake and likely impact of the research. With a budget of £1.4m over four years, this is a substantial investment by its two funders, and a significant attempt to enhance the impact of their social science research. This chapter reflects on some of the lessons the EPG has learned. It focuses on how the EPG has influenced research projects’ thinking about impact, not the impacts they have subsequently had on policy and practice. As such, we hope it offers ideas for organisations such as donors or universities who want to increase the likelihood of impact from the research they fund.
Understanding the role of the EPG and what it is able to achieve means understanding how the programme was set up. DFID and the ESRC jointly oversee the DEGRP programme: the ESRC is responsible for ensuring the research is academically robust, and DFID for ensuring that it focuses on the needs of the poor in low-income countries. The programme thus has a dual remit. It needs to promote world-class, cutting-edge research, pushing the boundaries of knowledge and creating public goods in the form of new datasets, models or approaches. But it also needs to ensure that this research influences processes and policies for inclusive and sustainable economic growth. Like other similar programmes, it strives to deliver ‘engaged excellence’, as James Georgalakis so neatly puts it (Georgalakis 2016).
Different types of impact
Early in the life of the EPG we challenged ourselves to consider how to frame our own impacts; how we would assess our influence on the research process. Drawing on the definitions of impact outlined by the ESRC, we describe the EPG’s impacts as:
• Instrumental: impacts on the policies and practices of researchers, on how they go about the process of achieving impact;
• Conceptual: contributing to understanding, influencing knowledge about and attitudes towards impact;
• Capacity building: strengthening the ability of researchers to work towards impact.
However, research does not happen in isolation: building networks of people and organisations able to understand the research and make use of it is part and parcel of achieving broad-based impact, which takes on a life of its own after the project has ended. We added a fourth category:
• Connectivity: improving relationships between researchers, policymakers and practitioners so that they can develop their own networks in future.
Within each type of impact we can also consider what role we played. We identify three possible causal roles, examples of which are given throughout the chapter in boxes 1–3 (and see, for example, the work of the International Food Policy Research Institute [IFPRI] on the impact of policy research in Place and Hazell 2015; see also Pasanen and Shaxson 2016: 24).
• Direct, attributable impacts: where it is possible to claim that without the EPG’s intervention the researchers would not have changed their approach to impact (see Box 1);
• Plausible and distinct contributions to sustainable change: we can plausibly claim to have contributed to how the research was able to achieve impact, and can distinguish what we did from contributions made by others or from external factors (see Box 2);
• Influencing context: where what we did was part of a wider push for impact. It may be difficult to identify our specific contribution, but what we did helped shape the context within which the project worked (see Box 3).
The EPG as a knowledge intermediary
So, what do we actually do? The EPG’s role has evolved over time, partly in response to changing governance arrangements and partly as the full complexity of the programme has emerged. Our overall approach can be described using the K* framework for knowledge interaction, set out in Figure 1, which distinguishes four broad types of knowledge function (the K* framework is described in Harvey, Lewin and Fisher 2012; Shaxson, Bielak et al. 2012).
Figure 1: The K* framework, setting out the different functions of a knowledge intermediary
Source: Shaxson, Bielak et al. (2012)
As an information intermediary we host information provided by the research projects, making their research available via the DEGRP website. As a knowledge translator we produce policy briefs and policy-focused research syntheses; as a knowledge broker we engage in current policy debates; and as an innovation broker we provide the wherewithal for researchers to engage with policymakers and practitioners by improving their opportunities to meet. The balance between the four roles is described and assessed in section 4.4, but what is important to know is that it changes constantly, depending on the demand from researchers and on what we jointly believe could make the most difference to their work.
Translating, brokering and facilitating in multiple dimensions
Over time, the EPG’s approach to being a knowledge intermediary has become more reactive than proactive; less planned and more responsive to current events. Understanding why means considering the different dimensions of the programme.
The ESRC procurement process is an open call, though its boundaries are set by the call specification document that was developed by the two funding agencies with technical support from international advisers. This process encourages a broad response from researchers, but means that links between individual projects are serendipitous rather than planned. This gives rise to a programme that varies across multiple dimensions.
4.1 First dimension: thematic
While there are four research themes—agriculture, finance, innovation and China-Africa relationships—these are broadly defined. Within each it has (just about) been possible to identify technical sub-themes such as irrigation, financial regulation, or where innovation happens. However, the openness of the commissioning process means that these sub-themes are baskets of reasonably similar projects rather than strands of work that can be synthesised to draw lessons. There is no regional theme – most projects are in Africa but the programme covers 20 countries. Nor is it possible to distinguish a thematic approach to end users: some projects work directly with smallholder farmers or small businesses, some with medium-sized or international businesses/producers and some with representatives of global organisations. Many work across two of those categories.
DIRECT AND ATTRIBUTABLE IMPACTS: USING THE EPG IMPACT GUIDANCE
Following the DEGRP impact guidance (Shaxson 2016) prompted one project to radically change its approach to impact. The researchers originally described a somewhat passive methodology that relied on uptake of their findings by transnational development agencies, bilateral donors and philanthropists. Policymakers would be reached via a research organisation with greater field presence.
The DEGRP impact guidance sets out four steps: mapping stakeholders, developing a theory of change, understanding the team’s role as knowledge intermediaries (using the K* spectrum outlined in Fig. 1) and developing their communication and engagement strategy. Using the DEGRP guidance encouraged the team to map their stakeholders, consider what changes would be likely and what knowledge intermediary role they could play. In doing this they realised that they had insufficient knowledge of Ugandan policy processes and that this would be key to developing actionable recommendations.
The revised impact pathway set out an innovative approach to turning complex research findings on farmers’ attitudes to risk into policy recommendations. Initial findings were discussed with the project’s stakeholders and turned into a locally informed policy brief. This was updated via a series of interviews with a wide range of national stakeholders, before being discussed at a final workshop in Kampala that involved senior policy officials as well as representatives from non-governmental organisations (NGOs), farmer organisations, donors and the private sector.
The EPG supported the team as they implemented their plan, offering advice on what issues to prioritise for different audiences and how to write for policymakers and practitioners. Two EPG staff helped facilitate the final workshop. The process is described in a report and infographic (Verschoor 2015).
The EPG can claim to have had played a direct, attributable role in achieving conceptual, instrumental and capacity building impacts on the research team; and to have made a plausible and distinct contribution to change in the project’s connectivity – the project made the connections, but the EPG impact guidance provided the impetus.
4.2 Second dimension: researchers’ individual characteristics
The individual characteristics of the principal investigators (PIs) and their research teams vary widely. They differ according to:
• The range and depth of existing personal connections to policymakers and practitioners. Some had already built close relationships with the people and organisations they were hoping to influence, such as local manufacturers, central bank governors or senior policy officials. Others had good connections to local research organisations but limited connections to policymakers or practitioners.
• The different appetites for public engagement. Some are happy to work closely with policy officials around preliminary findings, but others have little interest in engaging until after peer-reviewed reports and articles have been published.
• The different appetites for the impact agenda. Projects based at UK institutions were already familiar with the requirements of the Research Excellence Framework for an impact focus; projects at non-UK universities were less so. However, the a priori appetite for the impact agenda seems to be more personal than institutional.
• Their existing communication skills. Crafting policy- and practice-relevant messages came more naturally to some than to others. This is not necessarily a question of writing skills or style, more of being able to identify what issues from their research would be most likely to interest their audiences.
• The institutional support available from their home institutions. Some projects crafted their own detailed websites, which were hosted by their universities and linked to wider programmes of work. Others operated on a more individual basis, with less institutional backup.
4.3 Third dimension: the nature of the research approach
Some projects aim to inform specific policy questions; some are less targeted and more conceptual in nature. Most work with large quantitative datasets, but some have an explicit focus on mixed methods, and a few involve social anthropologists as well as economists. Most are cross-sectional analyses, though some have a longitudinal component as well (one is primarily a longitudinal analysis). The focus on cross-cutting issues (such as gender) varies, as do the methods for data collection and analysis; both qualitative and quantitative.
A PLAUSIBLE AND DISTINCT CONTRIBUTION TO CHANGE: CATALYSING CONNECTIVITY
Three projects in the DEGRP portfolio are working on irrigation issues in East Africa (see Harrison 2015; Meinzen-Dick 2015; Woodhouse 2015): one that had already ended and two that had recently begun. They had been commissioned entirely separately and were unaware of each other’s existence. The EPG brought the projects together at a workshop that specifically sought to encourage interaction between all projects within each theme. The two ongoing projects developed joint plans for outreach, but when one built good relationships with an irrigation authority in one East African country this developed into a joint policy workshop that also involved the earlier project that had ended. The EPG topped up project budgets to facilitate the workshop (funding flights for two researchers who had not already planned to be in the region). The longer-term impacts of this have yet to be realised, but a longer series of collaborative events have been planned.
The projects might have learned about each other through ongoing programme-level communications, and the first policy workshop was the direct result of connections made by one of the PIs. However, the EPG made a plausible and distinct contribution to connectivity: catalysing relationships that will influence how the projects relate to each other and to policymakers in the region.
4.4 The EPG’s role in a multidimensional programme
This multidimensionality means that there are no blueprints for how to achieve impact. We have had to think carefully about our role as a knowledge intermediary.
It has been straightforward to operate as an information intermediary, ensuring that information from each of the projects is easily accessible via the website. Our work as a knowledge translator has also been relatively uncomplicated; synthesising the research messages, crafting and carefully targeting short briefing notes. The audiences for EPG outputs are mainly national policymakers and practitioners, though for some projects it has been important to engage at a global level. The EPG has supported this translation function where the project’s host university has limited skills or experience, or where specific opportunities have been identified – for example in regional or national media – and the PI’s contacts are limited. A recent addition to our translation function is the Research In Context series, where the EPG lead sets a piece of DEGRP research in the wider policy context (see, for example, DEGRP 2016).
However, our anticipated role as a knowledge broker has been more limited. Knowledge brokers actively engage in policy debates, but although the EPG’s technical leads work as knowledge brokers in their ODI jobs, it became apparent early on that policy officials did not want to hear what the EPG thought the research was saying: they wanted to hear it directly from the researchers themselves. Instead, we play a stronger role as an innovation broker, improving opportunities for engagement and uptake by ensuring that the PIs can present their research in person. We have done this in three ways.
First, we have provided light-touch guidance to improve policy engagement and influence of research. While it is important to think about research impact early on, ideas about how to achieve impact change with the context and as emerging findings provide nuance to messages that were envisaged earlier (see ODI 2014). The EPG provided a half-day workshop for all researchers to discuss how to consider impact, then left them alone for a year to develop their understanding of the context of their research before asking them to update their pathways to impact. We provided guidance but no template for their revised plan. Because we could provide tailored support to each project, we wanted to encourage as much innovation as possible in approaches to achieving impact. This flexible approach paid off: some projects produced relatively simple plans, which they then followed closely. Some provided new ideas for conceptualising and planning for impact, which were shared more widely. And as Box 1 notes, one project completely revised its approach.
Second, our three technical leads have extensive networks of policymakers and practitioners; nationally, regionally and globally. As a result, DEGRP researchers have shared panels and co-presented with senior policy officials and people from international organisations, raising their own profile and that of their research. Collaborating with well-respected local research organisations such as the African Economic Research Consortium (Nairobi), the Science and Technology Policy Research Institute (Accra), the African Center for Economic Transformation (Accra), the South African Institute for International Affairs (Johannesburg) and the Centre for Policy Dialogue (Dhaka) further enhanced those networks, particularly with national policymakers and national media. These national events have been balanced with smaller panel-type events that have taken advantage of key international researchers and policymakers who happened to pass through London, ODI’s live-streaming facilities and its broad international audience.
Third, we have a large budget for events. Along with maintaining the website, events have become the EPG’s major focus. Much of our work involves planning, facilitating and wrapping up engagements of one form or another – from small four-person panels to large conferences of over 100 people – and finalising the publications that result. The events are relatively simple to put on and host, and instead of lengthy event reports we ask presenters for two-page policy-relevant essays that are collated into a single document and prefaced with an editorial by the EPG research lead (see, for example, the report from the event co-hosted with the South African Institute of International Affairs, DEGRP 2016).
CONTEXTUAL IMPACTS: FOSTERING DEBATE
Most DEGRP projects focus on Africa, but the EPG and funders were keen to hold an event in Asia to raise the programme’s profile there as much as possible. Early findings from a project on training female supervisors in the Bangladeshi garment industry (Woodruff 2015) provided the kernel of an EPG-funded workshop that brought together senior policymakers and researchers from across the region to address issues around innovation policy. The workshop (Centre for Policy Dialogue 2014) was co-hosted with a local thinktank, the Centre for Policy Dialogue, whose connections ensured attendance by high-level policy officials from both Bangladesh and Pakistan. The EPG funded the attendance of a senior United Nations Industrial Development Organization (UNIDO) official from Mauritius to speak about the lessons that could be drawn for Bangladesh from the history of transformation in the Mauritian garment sector. The DEGRP project was then still at a relatively early stage so had no findings to communicate, though the PI was well known and had worked on similar issues previously. However, the presence of very senior policymakers at the event meant that there was intense interest from the region: over one thousand people watched the event online.
It was culturally inappropriate to ask for feedback from those who attended the event, so it is impossible to analyse what effects it might have had on the project’s impact. We are not sure whether the main impacts of the workshop were conceptual, instrumental, capacity building or connectivity. However, we would claim that the EPG helped shape the context within which future project results would be disseminated.
There is an increasing amount of experience from the sharp end of work to improve research impact (see DFID 2016; Reed 2016), but to our knowledge the EPG is still an innovative attempt to manage this at a programme level. Four lessons stand out from the past four years.
5.1 Be strategically opportunistic
In such a multidimensional programme a prescriptive approach to supporting impact will not work. We describe our approach as ‘strategic opportunism’: setting a general direction and then responding to specific opportunities (see Isenberg 1987). This means we need to be flexible: helping facilitate workshops, advising on writing op-eds or opinion pieces, producing short films, or funding a workshop that was not in the initial project budget but will make an important contribution to impact. The price PIs pay for this tailored support is an output for every input such as a blog piece, a short essay, a case study or a set of short films we can publicise to raise the profile of their research and of the programme.
There are no firm criteria for selecting which projects are supported in what ways: it is an ongoing discussion within the EPG team that draws on their knowledge of upcoming events, requests from PIs for specific types of support, emerging findings, and innovative ideas about how we could present projects and their work.
5.2 Plan for the short term, but fund for the long term
This need for flexibility means that we can only construct detailed work plans three to four months in advance. Opportunities arise at short notice – even regular events organised by international organisations might be cancelled or shift their focus. Our planning and reporting cycle has evolved over time: a detailed annual work plan became a six-monthly work plan, which ultimately became a quarterly work plan with a six-month forward look. This is only possible because the EPG’s funding is not tightly prescribed. Outputs are reported annually against the logical framework, but within broad budget lines (events, tailored support to projects, programme communications, reimbursable) there is considerable flexibility. We have an agreed annual budget envelope and an agreed number of deliverables of different types, but within those limits we work with the researchers to decide what is appropriate.
5.3 Relationships, relationships, relationships
Flexible funding is important, but the excellent relationships between DFID, ESRC and the EPG are the foundation of our strategically opportunistic approach. Early, lengthy discussions about the purpose of the programme (via the wording of the outcome statement in the logical framework) was time well spent. As with any programme there have been glitches, but maintaining a focus on what being a ‘centre of excellence’ means for DEGRP has helped overcome them.
The main task of the EPG’s technical leads is to build relationships with each research project, so they can offer advice on how to maximise the likelihood of impact. A good deal of effort goes into maintaining these individual contacts, and the time is well funded. The technical leads’ professional networks are very valuable, particularly for early-career researchers who have not yet developed a good range or depth of contacts.
5.4 Facilitate but do not interpolate
Defining ‘connectivity’ as an impact the EPG can have has encouraged us to concentrate on our knowledge translation and innovation brokering functions, rather than setting the EPG up as a knowledge broker. Instead of interpolating in the debates, the focus has instead been on building researchers’ own knowledge brokering capabilities, helping them become more comfortable with the concept and building their networks. Some were already experienced brokers; others have needed support. This has been an effective strategy, as the early achievements set out in section 2 demonstrate. Improving connectivity and supporting mutual learning between grantees has become increasingly important (see Box 2), through dedicated grant-holder workshops and in-country events.
6.1 We’ve made our own luck
The EPG has been a successful experiment: a great deal has been learned on all sides about how to improve the likelihood of impact. Chief among these is that strategic opportunism is only really possible if you are well networked and able to act quickly and flexibly (as the Roman philosopher Seneca is reported to have said, luck is what happens when preparation meets opportunity). This means taking time to build and maintain strong relationships, but using those to build researchers’ connectivity and supporting their engagement in the debate as soon as opportunities arise. The EPG uses its technical expertise to facilitate knowledge brokering for project impact but our technical leads are not themselves active brokers in policy debates.
6.2 We haven’t always got it right
Most projects wish they could have done some things better and the EPG is no exception. There have been the usual project management challenges (no matter how far in advance we plan it is never far enough), but there is a wider issue about how we have balanced local and global impacts. While our focus on providing tailored support to projects may have helped them achieve good local impact, we could perhaps have done more to help them embed their messages within wider global debates.
6.3 But it’s still a supply-side approach
As projects engage with the impact agenda, they want to run workshops to connect with policymakers and practitioners. Although the EPG has wide networks, the same people keep turning up on our invitation lists. If all projects continue to hold early project engagement workshops at the beginning and end of their work, in much-studied countries like Tanzania or Bangladesh, a back-of-the-envelope calculation is that some diligent officials would need to spend an average of two days a week in such meetings, all year round. Is there an alternative? Could government departments be encouraged to be more proactive in setting the questions they really want answered and inviting researchers to engage with them? The DFID-funded Building Capacity to Use Research Evidence (BCURE) programme is beginning to address this issue, but something more systematic needs to happen to prevent policymakers being overloaded. Could funders commit to supporting intermittent research afternoons for clusters of public agencies at which all PIs from new and reporting studies come together to present their work? The next era of work on uptake must be about strengthening and systematising the demand for research.
Centre for Policy Dialogue (2014) ‘Challenges of Economic Transformation and Growth in Bangladesh: Agriculture and Manufacturing’, https://degrp.odi.org/event/challenges-of-economic-transformation-and-growth-in-bangladesh-agriculture-and-manufacturing/
DEGRP (2016) ‘How Farming Stimulates Rural Economies: Insights from Southern Africa’, https://degrp.odi.org/publication/how-farming-stimulates-rural-economies-insights-from-southern-africa/
DEGRP (2015a) ‘China–Africa: A Maturing Relationship? Growth, Change and Resilience’, https://degrp.odi.org/event/china-africa-a-maturing-relationship-growth-change-and-resilience/
DEGRP (2015b) ‘Shaping Economic Transformation in Tanzania’, https://degrp.odi.org/event/shaping-economic-transformation-in-tanzania/
DFID (2016) Research Uptake: A Guide for DFID-funded Research Programmes, London: Department for International Development
Georgalakis, J. (2016) ‘New DFID Guide on Research Uptake: 7 Things to Love and 7 to Worry About’, IDS blog, 26 April 2016, http://www.ids.ac.uk/opinion/new-dfid-guide-on-research-uptake-7-things-to-love-and-7-to-worry-about (accessed 3 December 2016)
Harrison, E. (2015) ‘Innovations to Promote Growth Among Small-Scale Irrigators in Africa’, https://degrp.odi.org/project/innovations-to-promote-growth-among-small-scale-irrigators-in-africa/
Harvey, B.; Lewin, T. and Fisher, C. (2012) ‘Is Development Research Communication Coming of Age?’ IDS Bulletin 43.5: 1–8, doi: 10.1111/j.1759-5436.2012.00356.x
Isenberg, D. (1987) ‘The Tactics of Strategic Opportunism’, Harvard Business Review, March, https://hbr.org/1987/03/the-tactics-of-strategic-opportunism (accessed 3 December 2016)
Meinzen-Dick, R. (2015) ‘Assessing Models of Public–Private Partnerships for Irrigation Development in Africa (AMPPPIDA)’, https://degrp.odi.org/project/assessing-models-of-public-private-partnerships-for-irrigation-development-in-africa-ampppida/
ODI (2014) ROMA: A Guide to Policy Engagement and Policy Influence, www.roma.odi.org/ (accessed 3 December 2016)
Pasanen, T. and Shaxson, L. (2016) How to Design a Monitoring and Evaluation Framework for a Policy Research Project, London: Overseas Development Institute, www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10259.pdf (accessed 3 December 2016)
Place, F. and Hazell, P.B.R. (2015) Workshop on Best Practice Methods for Assessing the Impact of Policy-oriented Research: Summary and Recommendations for the CGIAR, Independent Impact Assessment Report 41, Washington DC: International Food Policy Research Institute
Reed M.S. (2016) The Research Impact Handbook, St John’s Well, UK: Fast Track Impact
Shaxson, L. (2016) Achieving Policy Impact: Guidance Note, London: DEGRP, https://degrp.odi.org/impact/
Shaxson, L.; Bielak, A.; Ahmed, I.; Brien, D.; Conant, B.; Fisher, F.; Gwyn, E.; Klerkx, L.; Middleton, A.; Morton, S.; Pant, L. and David Phipps. (2012) Expanding Our Understanding of K*: A Concept Paper Emerging from the K* Conference held in Hamilton, Ontario, Canada, April 2012, Hamilton, Ontario: United Nations University Institute for Water, Environment and Health
Verschoor, A. (2015) ‘A Behavioural Economic Analysis of Agricultural Investment Decisions in Uganda, Growth Research Programme’, https://degrp.odi.org/project/a-behavioural-economic-analysis-of-agricultural-investment-decisions/
Woodhouse, P. (2015) ‘Studying African Farmer-led Irrigation (SAFI)’, https://degrp.odi.org/project/studying-african-farmer-led-irrigation-safi/
Woodruff, C. (2015) ‘Training, Productivity, and Upgrading in the Bangladesh Apparel Sector’, https://degrp.odi.org/project/training-productivity-and-upgrading-in-the-bangladesh-apparel-sector/
*The EPG is a team effort and this chapter is the product of four years of discussion with my colleagues Louise Ball, Natalie Brighty, Caroline Cassidy, Sarah Turner, Dirk Willem te Velde and Steve Wiggins. I am also grateful for helpful comments on previous drafts from James Georgalakis, Tiina Pasanen, Paul Sanderson and an anonymous reviewer.
 See www.esrc.ac.uk/research/impact-toolkit/what-is-impact/ (accessed 3 December 2016).
 See the call specification and policy relevant questions at www.esrc.ac.uk/funding/funding-opportunities/dfid-esrc-growth-research-programme-degrp-call-3/ (accessed 3 December 2016).