EVIDENCE AND INNOVATION:
Lessons learned from the MSF Scientific Days*
We present learning from our experience of organising the MSF Scientific Days, a conference that includes both medical research and innovation. We discuss the learning opportunities between both worlds, differences in their cultures, and the challenges in agreeing what constitutes evidence of impact of innovation projects. We draw on the experiences of organising these events and in particular the juxtaposition of the medical research and innovation days to describe key lessons we have learned about what helps or hinders ideas turn into evidence and impact within MSF: the need for rigorous evaluation and communication of findings whether positive or negative; the need for ethics oversight; developing solid processes for uptake; and accountability and learning mechanisms.
Since 2004, Médecins sans Frontières/Doctors Without Borders (MSF) has hosted the MSF Scientific Days, an open and free-to-attend annual conference at which medical and programmatic research from across MSF is presented to an audience largely composed of humanitarians, academics, medical, and non-governmental organisation (NGO) staff (Box 1). In 2015, to create a home for analyses of projects that did not fit into conventional medical research paradigms but that were undoubtedly relevant to medical programming, we added an ‘Innovation Day’.
AIMS OF THE MSF SCIENTIFIC DAYS
The MSF Scientific Days has multiple aims. Initially it was set up to help raise the quality of research conducted by MSF, by creating a platform for academics and other NGO staff to critique the work presented, in addition to enabling knowledge sharing, promotion of networks and exchange of ideas. This expanded over time to include the aim of improving the quality of medical programming, through challenging MSF and other humanitarian organisations with evidence. In addition, two of the most important aims of the event are transparency and accountability. MSF has an ethical imperative to strive to ensure that the research we conduct and innovations we undertake deliver impact to our beneficiaries. Key to this is being transparent about our projects and where they have succeeded and failed. We use the MSF Scientific Days as a way to disseminate this information, both within MSF and importantly to viewers in the countries in which the research was conducted. In 2016, the event took place in London, New Delhi and Johannesburg, and through livestreaming technology more than 11,000 people participated from 125 countries. We also use surveys to follow up what happened next to the research and projects presented – whether they met their aims or had any impact. This information is presented back to the audience the following year, thus closing the accountability loop.
The addition of the Innovation Day raised multiple challenges in organisation. The dialogue between the research and innovation worlds was subject to miscommunication and misunderstanding. Determining what was research and what was innovation was not always straightforward. The cultures were often quite different, and structures and networks in place in the research world were not present in the innovation world and vice versa. A particular challenge was agreement on what constituted evidence of effect or outcome. The focus by the research world on evidence was initially a hard sell to the innovation world. How this shift by innovators occurred from perceiving evidence as irrelevant to rather being a useful innovation tool forms part of this chapter. In addition, we discuss the limits of evidence alone and what the research world could learn from innovators. We draw on the experiences of organising these events and in particular the juxtaposition of the medical research and innovation days to describe key lessons we have learned about what helps or hinders ideas turn into evidence and impact within MSF.
Innovation in MSF
Innovation is a word that is susceptible to hype, derived from the business world, and risks being so overused as to become meaningless. There is, of course, an artificial line between research and innovation, when in reality the two overlap. We focus in this article on disruptive innovation rather than quality improvement, and use innovation here to simply mean a new or a different way of doing things that creates value or has impact. Or, put more simply, it is something new that is useful. This could be a novel way of transporting biological samples (Chikwanha and Pujo 2015), or implementing a new model of patient care (Gunnarsson, Zughui, Tarwaneh and Altas 2015). Innovation is integral to the work of humanitarian organisations such as MSF since the emergency nature of our work means we must adapt rapidly to find new solutions to problems we see in the field. Innovators in MSF take many forms – doctors, logisticians, project coordinators – many of whom do not identify as innovators, nor recognise that they are doing something particularly novel. Additionally, MSF has over 400 projects in over 60 countries (MSF 2015a), which can make transferring knowledge between projects challenging. The MSF Innovation Day has showcased exciting innovations that potentially could have a big impact on MSF field programmes (MSF 2015b, 2016), but it has also highlighted many of the barriers that prevent an idea becoming something that leads to change.
Cultural challenges and opportunities
The cultural exchange that happens between the innovation and research events at the MSF Scientific Days has been beneficial for each side. The lively presentation styles and ‘innovation pitch’ sessions (Vincent-Smith 2016) adopted by innovators have shown researchers more engaging ways to communicate their findings, and innovators have seen stronger means of evaluation from researchers. Surveys to determine the perceived impact of research and innovation presented at the MSF Scientific Days allowed insights into the inherent culture difference between the research and innovation communities around when and how to communicate about a project (Annex). Innovators tend to favour mainstream media communication whereas researchers usually wait for peer review before there is excitement and wider dissemination of findings. Innovation has a ‘sell’ culture, rooted in the need to sell the dream to get funding. An example of the difference in media approach occurred at the height of the West Africa Ebola outbreak, when MSF collaborated with software developers from the Google Social Impact Team to create a product that would help to overcome some of the challenges of patient data management in our Ebola Management Centres. The solution was the ‘Ebola tablet’, which was to facilitate a real-time electronic medical record in an extreme biohazard environment. The tablet garnered much media attention (Metz 2015; Mudasad 2015). However, the impact on patients was minimal because delays in deployment meant that patient numbers had declined before it could be implemented (Jobanputra et al. 2016).
Key lessons in evidence in innovation
4.1 Rigorous evaluation and communication of findings – positive and negative
The Humanitarian Innovation Fund (ELRHA 2014) acknowledges that ‘the effectiveness of humanitarian interventions is limited by the inability of the humanitarian system to create and harness successful innovations and learn from unsuccessful innovation trials’. It has also been suggested that the humanitarian sector values action over analysis and lacks the infrastructure to design effective evaluations (Proudlock and Ramalingam 2008). The excitement of innovative programmes can lack the rigour associated with biomedical research. In innovation abstracts submitted to the MSF Scientific Days, we have often seen a lack of evaluation and difficulty in understanding what constitutes evidence of impact. Given the time and resources invested in any innovation project, we have an obligation to rigorously evaluate innovations and to share both positive and negative outcomes, so we can learn from them and avoid repeating mistakes. Indeed, being prepared to ‘fail forward and learn’ was a key message delivered at MSF Scientific Days 2016 (Vincent-Smith 2016). To address the challenges faced by lack of evaluation of innovation projects, we are instigating a pairing of innovators with researchers in the Scientific Days editorial committee. The idea is that the researchers can offer mentorship and support to ensure innovations are properly assessed. This approach has been followed by the International Rescue Committee, which has described how it brings researchers together with innovators to strengthen both the design and the evaluation of interventions (Ramalingam and Bound 2016).
The abstracts submitted to MSF Scientific Days have also highlighted the lack of sound project management practice in many innovative initiatives. This practice should include building a solid business case including extensive stakeholder mapping and SWOT (strengths, weaknesses, opportunities, threats) analysis, launching proof of concepts before developing a complete project charter and plan and moving to execution, and the advantages and disadvantages of various project management methodologies, such as 'agile' approaches. The appearance of Project Management Office structures across MSF is an attempt to respond to this need.
Betts and Bloom define the process of humanitarian innovation in four stages: (1) defining a problem or identifying an opportunity; (2) finding potential solutions; (3) testing, adapting and implementing a solution; and (4) appropriately scaling the solution (Bloom and Betts 2013). The abstracts submitted to the MSF Scientific Days have shown challenges and solutions in how each of these stages have been managed, assessed or evaluated as we highlight in the examples below.
Defining a problem; finding potential solutions
Issues with the stage of problem identification were exemplified by a presentation at the 2016 MSF Scientific Days on the evaluation of the use of an electronic medical record for use in emergencies (EMR-E) (Marr 2016). The EMR-E was piloted in a nutrition emergency in Chad with the plan to move away from paper-based patient data management towards a tablet-based solution. However, the evaluation revealed that the EMR-E was not ready for deployment, as the software was not stable and the user interface needed more work to function safely. The key problem with this project was the lack of a rigorous situation analysis that preceded finding the potential solution. Paper turned out to be a complex technology to replace, and the proposed solution did not fit the identified problem.
Testing, adapting and implementing a solution
However, the EMR-E evaluation was rigorous and the presentation of the negative outcomes of the evaluation conveyed a strong message that was highly rated by the conference audience in the feedback forms. The importance was shown of being able to learn from a negative outcome and take action on the basis of robust evidence. The presentation was distinct from others because it was the only one submitted that showed negative findings. It was also the only one that was assessed by an independent evaluator. In view of the potential conflict of interests arising from evaluating one’s own innovation, we would suggest innovation projects are evaluated by an external person where possible.
Appropriately scaling the solution
Innovations must ideally have strategies for wider implementation so they can be applied at scale. If they only benefit one small community they are much less useful than if they can be adapted for a range of settings. This issue of scalability has also been recognised by Oxfam (Ramalingam and Bound 2016) who, when analysing their own innovation culture, realised that the question should not be ‘Is it innovative?’ and instead should be ‘Does it have potential to bring change at scale and what are the ways that impact can be increased?’ For instance, at the 2016 Scientific Days, an innovation pitch was made for a dashboard application to aid disease surveillance. The dashboard aggregates epidemiological data from multiple sources (e.g. MSF, ministries of health, World Health Organization) in a visual display that ‘turns data into information’ (Ait Bouziad et al. 2016). The dashboard was developed with scalability in mind because the geography and dataset will be different for each setting in which it is implemented. It also used open source code so other institutions can benefit from this technology.
4.2 Ethics oversight
Once the problem has been analysed and a solution identified, the ethical implications of that intervention need careful consideration. MSF projects must align with core medical and humanitarian values to ensure we do not cause undue risk or exploitation of our patients. Our medical research is governed by an independent ethics review board and adheres to the principles of medical ethics, but innovation projects that do not involve human subjects do not fall under this system.
In 2015, the lack of ethical oversight of some of the projects presented at the Innovation Day was questioned by our audience. In addition, there was an artificial divide that we struggled with as organisers around material that potentially was suitable for presentation in either the research or the Innovation Day. All material in the research day required a statement about ethics oversight. It was clearly unsatisfactory to have a system where material required ethics oversight if accepted in one event but not the other. However, decreeing that the ethical procedures in place for research should be followed by innovators risks stifling innovation with heavy processes.
This tension led us to develop a light, parallel structure for the ethical guidance of MSF innovation projects – an ethics framework for innovation that innovators can self-apply to reflect on the potential harms or benefits to our patients (Box 2) (Sheather et al. 2016). It addresses many of the points on evaluation raised above. We plan to evaluate its utility through auditing the abstracts submitted to MSF Scientific Days and surveying authors.
A MÉDECINS SANS FRONTIÈRES ETHICS FRAMEWORK FOR HUMANITARIAN INNOVATION
This framework is intended to be used to guide work that does not directly involve human participants and does not lie within the purview of formal research ethics oversight.
1. Clearly identify the problem you are seeking to address, and what benefit you expect the innovation to have. This step may seem obvious, so what is its ethical significance? When identifying the problem, there should be consideration of up-stream solutions that may address the problem in a holistic and sustainable way. For instance, rather than focusing on technocratic fixes, what are the sociopolitical determinants of the problem and the wider possibilities for solutions? Who has stakes in finding a solution and who may have interests in perpetuating the problem? Is the problem a moving target? Collaboration and cross-fertilisation with other disciplines should be considered in order to help to see the problem from various perspectives. In short, do not underestimate the importance of fully identifying the problem.
2. Ensure that the innovation shows respect for human dignity. While this is a broad concept, it has practical implications. The focus of concern is respect for human beings, reminding us that the simplest or most direct solutions may not be ethically appropriate. Innovators must show due respect for the multiple and overlapping interests of those affected by the innovation. It extends beyond a concern for physical wellbeing to include psychological and cultural integrity. It also incorporates a concern for individual privacy and a respect for the confidentiality of individual, family and community-based data.
3. Clarify how you will involve the end user from the start of the process. Innovation should be driven by the requirements of the user. The innovation cycle should be participatory, using methods to involve relevant individuals and communities. Innovators must be sensitive to power dynamics between and within cultures and power imbalances between aid workers and beneficiaries.
4. Identify and weigh harms and benefits. When considering innovations, a critical first step is the identification, as far as is reasonably possible, of potential harms along with the anticipated benefits. The next step involves weighing these harms and benefits.
• Where reasonably foreseeable harms outweigh the likely benefits, implementation will not be ethical. Potential harms include, but are not limited to, physical and psychological harms to individuals. There is also need to consider potential harm to communities.
• Where innovation involves a favourable balance of benefits and harms, all reasonable steps must be taken to minimise (mitigate) the harms as far as possible. Unnecessary harms must be eliminated. Where harms are unavoidable, those affected should be informed of the nature and severity of the risks involved.
• Conflicted partnerships or conflicts of interest may result in reputational harm to the organisation. If these are identified then oversight by an existing Ethics Review Board is recommended.
5. Describe the distribution of harms and benefits, and ensure that the risk of harm is not borne by those who do not stand to benefit. Innovators need to give careful consideration to the distribution of benefits and harms associated with their projects. Do the risks or benefits fall unequally across groups? If so, is it appropriate to proceed, and how can these inequalities of distribution be addressed or mitigated? Equally, it is important that the innovation takes into account vulnerable groups; it may be ethically warranted to give particular attention to those who have particular needs. Just as we tend to give more health care to the unwell, so particular attention may need to be given to those who are vulnerable or who may not be able to protect their own interests. This is expressed in the humanitarian principle of impartiality. In addition, consider whether anyone is ‘wronged’ by the innovation. A ‘wrong’ is an infringement that is distinct from harm. For example, selecting one group for an innovation project over another may wrong the other group (as opposed to harming them).
6. Plan (and carry out) an evaluation that delivers the information needed for subsequent decisions to implement or scale up the innovation; and then ensure that the beneficiaries have access to the innovation. Innovation requires an acceptance of the risk of failure – not all innovation projects will achieve their desired outcome. But in all cases, we can learn and apply these lessons in the future. Given the time, energy and resources that these projects require, rigorous evaluation and sharing of lessons is itself a moral obligation. Therefore, consideration should be given to dissemination of findings, since it may be important to avoid further exposure to potential harm by sharing findings, whether these are positive or negative. Likewise, there should be a willingness and strategy for wider implementation of the innovation if found to be successful, and a commitment to ensure beneficiaries – at least in the communities where it was tested and ideally in similar communities affected by humanitarian crises – have access to the innovation subsequently.
4.3 Developing solid processes for uptake
Even if innovations are of a high quality, robustness doesn’t always lead to an impact on patients. Implementing change is arguably the most important stage of any research or innovation project and often the most neglected.
At MSF Scientific Days 2015, a study demonstrated that instead of having to be kept refrigerated to a temperature below 8 0C, insulin can stay thermostable for up to four weeks at temperatures seen in tropical settings (Kaufmann 2015). The potential impact on our diabetic patients could be huge – for people without refrigerators, instead of needing to visit a clinic twice per day to receive insulin, patients could visit once per month and store their medication at home. However, the presentation and dissemination of the findings was not sufficient to enact change. A year later, the only projects in MSF that had changed policy were those where a staff member involved with the study had actively lobbied for change. The reasons for this could be many, but often who is responsible for translating results into impact is not clear. In the innovation world there are lots of labs or incubator concepts that nurture the earlier stages of innovation, but for the later stages there is a lack of definition of who makes sure something happens.
4.4 Accountability and learning
We have recognised the need to develop processes and systems that will embed impact into our research and innovation culture. First, at the point a research or innovation project is conceived we ask project staff to be explicit about their plans for dissemination and impact. This increases the amount of thinking at the early stages of a project about how work will be translated and to identify where more support can be given to ensuring this happens. Second, we have new initiatives such as the MSF Sapling Nursery and the Transformational Investment Capacity (TIC) Fund, where MSF staff or members of our association can submit proposals for funding to develop their innovative projects. The Sapling Nursery is an incubator that is for disruptive and field-driven innovations. It encourages innovation by offering funding of up to €50,000 to ‘plant seeds’ and providing a safe space to pilot and evaluate projects. If they are successful they can apply for further funding to scale up. The TIC is an initiative that invests funds, intellectual capital and human resources in larger-scale projects that can transform MSF’s abilities to meet the needs of our patients. Third, we are starting to critically analyse the factors that determine the uptake of innovations. With this aim, we have developed ReMIT (Research Management & Impact Tool), an open source web application that tracks the research process, and captures where findings have been disseminated and any impacts on patients, policies, or programmes. There are plans to adapt this for application to our innovative projects so we can further understand what approaches do or do not work for the knowledge translation or impact.
The joint venture of the MSF Scientific Days between the innovation and research worlds has been fruitful for both. The MSF Scientific Days in itself acts as an integral part of the system to help in sharing and scalability across MSF as well as to external organisations, and this type of joint venture between disparate fields of work is highly recommended. The juxtaposition of these worlds revealed gaps in the innovation structure that were needed to ensure delivery of robust, useful and ethical projects. Although the focus of this chapter has been on evidence in innovation projects, the problems of scale up and dissemination of use are also highly relevant to research (Annex). The experience has also revealed the need for better communication and presentation of research material, as well as the need to avoid making the mistakes of the research world through introducing burdensome bureaucracy to the innovation process.
Annex: Impact of research and innovation projects presented at the MSF Scientific Days 2015
Questionnaires were sent to all authors of oral presentations 9 months after their presentation. The questions differed slightly between the research and innovation days. The response rate for the research day was 94 per cent (15/16) and for the innovation day it was 86 per cent (12/14).
MSF Scientific Day – Research results
|Did your study have an effect on:|
|Was your study:|
|Published in a scientific journal?||6||5||4|
|Presented at another conference?||13||2||-|
|Discussed in a blog or news-story?||2||13||-|
|Disseminated another way?||4||11||-|
Measuring a direct effect on patients is hard, and answers to this question were variable and degree of direct effect uncertain: 47 per cent of studies had either had or expected shortly to have an effect on patients; of the remainder some did not know as they had no data and some had presented project descriptions so the research itself did not have a direct effect. Effect on programmes was clearer, with 60 per cent able to describe an effect on operations. Effects on policies were described by 47 per cent of authors, with some others noting that policies were already in place but poorly implemented or that they would not know if this had happened. Of the 53 per cent that had had other effects, many of these were further research studies: 73 per cent had either been published in a scientific journal or publication was planned or in progress; most (87 per cent) had been presented at other conferences or meetings. However, 87 per cent had not been discussed in news media or blogs and only 27 per cent disseminated in other ways (internal documents, reports).
MSF Scientific Day – Innovation results
|Has your project:|
|Achieved its original aims?||5||2||5|
|Been replicated in MSF or externally?||5||7||-|
|Led to another project being launched?||6||6||-|
|Did your project have an effect on:|
|Was your project:|
|Helped by being presented at the MSF Scientific Day?||9||3||-|
|Published in a scientific journal?||0||7||5|
|Presented at another confernence?||9||3||-|
|Discussed in a blog or news-story?||7||5||-|
|Disseminated another way?||6||6||-|
Forty per cent of presenters said their projects had reached their original aims, but the same amount described their work as ongoing or were unsure. Both projects that had not fully reached their aims had taken place in the Ebola outbreak and piloting had been cut short by the end of the outbreak. Most (60 per cent) of projects had not been replicated, but classification was difficult for some, including cases where projects were services rather than products. Half the projects had led directly to the launch of another project.
Two-thirds of the projects had had effects on programmes, and 40 per cent had other effects such as creation of collaborations, inclusion in training or creation of working groups. Only two had had effect on policies, and two had had no impacts on programmes or policies, or other effects.
Seventy-five per cent said that presenting at the MSF Scientific Day had helped their project, many citing increased MSF buy-in to their work, with two of the three who said it hadn’t specifically helped noting, however, that the increased visibility was useful. Sixty per cent had not been published in a journal; 40 per cent had ongoing or planned publication; one project noted that journal publication would not be relevant. Seventy-five per cent had also been presented at other MSF and external conferences. Sixty per cent had received media coverage, with 50 per cent disseminated at other meetings, training, or via YouTube; only two had not been disseminated in any other fora.
Ait Bouziad, I.; Gayton, I.; Masters, P.; Johnson, S. and Raimbault, B. (2016) ‘A Dashboard to Support (Almost) Effortless Spatial Epidemiology’, F1000Research, doi:10.7490/F1000RESEARCH.1112094.1
Bloom, L. and Betts, A. (2013) The Two Worlds of Humanitarian Innovation, Working Paper 94, Oxford: Refugee Studies Centre, University of Oxford
Chikwanha, Isaac and Pujo, Eric (2015) ‘New Technology for an Old Disease: UAVs for Tuberculosis Sample Transport in Papua New Guinea’, F1000Research, doi:10.7490/F1000RESEARCH.1000122.1
Gunnarsson. J. (2015) ‘Meeting the Health Needs of Syrian Refugees in Jordan: A Novel Model for Non-communicable Disease Care in a Refugee Setting’, F1000Research, doi:10.7490/F1000R ESEARCH.1000116.1
ELRHA (2014) Impact Framework: Humanitarian Innovation Fund, ELRHA, http://www.elrha.org/wp-content/uploads/2015/01/HIF-Impact-Framework.pdf (accessed 15 December 2016)
Kaufmann, Beatrice (2015) ‘A Potential Revolution in Diabetes Type 1 Care: Heat Stability of Insulin in Tropical Conditions’, F1000Research, doi:10.7490/F1000RESEARCH.1000113.1
Marr, Rob (2016) ‘Evaluation of the Electronic Medical Record for Emergencies (EMR-E) platform’, F1000Research, doi:10.7490/F1000RESEARCH.1112074.1
Metz, C. (2015) ‘Google Builds a New Tablet for the Fight Against Ebola’, WIRED, www.wired.com/2015/03/google-builds-new-tablet-fight-ebola (accessed 15 December 2016)
MSF (2015a) International Activity Report 2015, Geneva: Médecins Sans Frontières, www.msf.org.uk/sites/uk/files/international_activity_report_2015_en_0.pdf (accessed 16 December 2016)
MSF (2015b) ‘Médecins Sans Frontières UK Scientific Day: Innovation 2015’, F1000Research, https://f1000research.com/channels/msf/sciday-uk-innovation (accessed 16 December 2016)
MSF (2016) ‘Médecins Sans Frontières UK Scientific Day: Innovation 2016’, F1000Research, https://f1000research.com/channels/msf/uk-innovation-2016 (accessed 16 December 2016)
Proudlock, K. and Ramalingam B. (2009) ‘Improving Humanitarian Impact Assessment: Bridging Theory and Practice’ in ALNAP’s 8th Review of Humanitarian Action, www.alnap.org/material/81.aspx (accessed 16 December 2016)
Ramalingam, B. and Bound K. (2016) Innovation for International Development: Navigating the Paths and Pitfalls, London: Nesta, www.nesta.org.uk/sites/default/files/innovation_in_international_development_v7.pdf (accessed 16 December 2016)
Sheather, J.; Jobanputra, K.; Schopper, D.; Pringle, J.; Venis, S.; Wong, S. and Vincent-Smith, R. (2016) ‘A Médecins Sans Frontières Ethics Framework for Humanitarian Innovation’, PLoS Med. 13: e1002111. doi:10.1371/journal.pmed.1002111
Vincent-Smith, Robin (2016) ‘Introduction to the Innovation Pitch: MSF Scientific Days 2015’, F1000Research, doi:10.7490/F1000RESEARCH.1112077.1
*We thank all members of the MSF Scientific Day editorial committees as well as the presenters and field staff involved in preparing and presenting their research and projects at the events.
 MSF Scientific Days, MSF UK, www.msf.org.uk/content/msf-scientific-days.
 MSF OCA Sapling Nursery Innovation Process, MSF UK, www.msf.org.uk/msf-oca-sapling-nursery-innovation-process.
 MSF Ethics Review Board, MSF International, www.msf.org/en/msf-ethics-review-board
 MSF OCA Sapling Nursery Innovation Process | MSF UK, www.msf.org.uk/msf-oca-sapling-nursery-innovation-process.
 MSF, Transformational Investment Capacity, http://msf-transformation.org.
 MSF, Research Management and Impact Tool (REMIT), http://remit.oca.msf.org.