A beginner’s guide to inclusive monitoring and evaluation: from talking to doing

The way in which we think about the purpose and scope of monitoring and evaluation (M&E) in development and humanitarian contexts is changing. Rather than a process that primarily aims to satisfy donor requirements for reporting and compliance, M&E is increasingly seen as an integral part of the project life cycle that should also contribute to project improvement, organizational learning and accountability to multiple stakeholders—in particular to beneficiaries.

This change can be seen in the work of many M&E specialists, who now also address accountability, learning and planning in their work (the ‘A’, ‘L’, and ‘P’ in MEAL, MEL, PME or PMEL). It is also noticeable in the expectation that project managers have a solid understanding of M&E and how it can be used to enhance the effectiveness of programming. Finally, this expanded scope is evident in the frequent adoption of results-based or rights-based approaches to M&E by organizations.

The ultimate aim of development and humanitarian action is of course to improve people’s lives, whether by working to reduce poverty, inequality, or injustice over the long term or through the immediate alleviation of suffering and vulnerability. It follows that the extent to which we include all affected persons in our programming—also reaching those who are most marginalised, subject to discrimination or at risk of exclusion—has become a measure of the effectiveness of our interventions. The participation of affected populations in project management processes is also beneficial to effectiveness: it can make a project more responsive to actual (rather than assumed) needs and promote the sustainability of results through stronger community ownership.

Since the purpose of M&E is also to ensure the effectiveness of our interventions, mainstreaming inclusion is therefore a primary concern of M&E practitioners.

Prefer a video instead? Watch the Webinar recording!

View the presentation slides of the webinar

If you like this article don't forget to register to the ActivityInfo newsletter to receive new guides, articles and webinars on various M&E topics!

View the French version of this article

View the Spanish version of this article

What is inclusion?

Although inclusion has become a frequent buzzword and an ideal to which many organizations express their commitment, it is a concept that is still poorly defined and understood throughout the project life cycle. This is especially true when it comes to the design of M&E systems and implementation of M&E processes.

When we hear the term ‘inclusion’, we most often think of approaches to programming that actively consider disability or the impact of gender roles (‘disability-inclusive’ and ‘gender-inclusive’ approaches), usually in projects that specifically target persons with disabilities or women. Considering these groups is certainly an important part of being inclusive: disability-specific or women-focused interventions are essential to respond to the particular needs of these groups and address the systemic barriers they may face to benefitting from our interventions.

Focusing on just one status or identity (‘having a disability’ or ‘being a woman’), however, is not enough if we want to mainstream inclusion. It can lead us to overlook other groups who may also be marginalised or at risk of exclusion in our project’s environment. We could also miss other aspects of a person’s identity that may further complicate their ability to fully benefit from our intervention.

When we mainstream inclusion in our programming we recognise that people have multiple and intersecting identities that impact whether or not they experience discrimination or exclusion, and to what extent. Someone is not ‘just’ a woman or a person with a disability: one can be a forcibly displaced woman who is also the head of a household or a boy with an intellectual disability from a marginalised ethnic group living in a rural area.

Mainstreaming inclusion means all groups, regardless of identity, are visible and meaningfully represented in our programming. It also means that we take steps to identify and reduce the barriers to their representation. When we mainstream inclusion and have a targeted focus on a particular group at the same time, we follow what is called a ‘twin-track approach’.

Characteristics of inclusive M&E

What does ‘mainstreaming inclusion’ mean for M&E practitioners? Sometimes, we may be working on interventions that have considered inclusion from the project identification phase onwards and for which there is an obvious roadmap to follow, such as a theory of change and a logframe. Other times, we might find ourselves in situations where inclusion has not been considered as consistently and there is no established framework or baseline to work with, as may be the case in an acute emergency response.

The good news is that it is never too late to apply an inclusive approach to an intervention using our M&E processes and tools, and there are many ways to do it. Here are a few:

Characteristics of inclusive monitoring and evaluation
Characteristics of inclusive monitoring and evaluation

Capture inclusion concerns in your baseline

Before designing an inclusive M&E system, we need to make sure we have all the information we need. The initial situation analysis, needs assessment or baseline study on which we base our methodological choices needs to tell us which groups or forms of exclusion in the project’s environment we need to pay particular attention to. Therefore, this preliminary analysis needs to carefully consider power relationships and the causes of discrimination and inequalities in a given context. Only when we have a complete picture can we make an informed decision about what indicators to use, what level of disaggregation is required in our data, what data collection methods are most appropriate, and so on.

Use inclusive indicators, both quantitative and qualitative

Indicators tell us whether an intervention is making progress or has achieved intended results and alert us when something is not going according to plan. Making sure different groups are reflected in our indicators lets us measure and evaluate results accurately and make a course correction during the project if our monitoring data shows us this is necessary.

Formulating relevant indicators is therefore a precondition for mainstreaming inclusion in M&E. There are several ways to do this, depending on the aims of our intervention. Let’s take a look at some different kinds of indicators and how we can approach them:

  • Person-related and non-person-related indicators

Result indicators that measure intended changes among people (individuals, households, communities, or societies) can be differentiated according to different groups of beneficiaries to monitor whether this change is the same. An example of this would be an indicator that differentiates between ‘refugees living in a camp setting’ as compared to ‘refugees living in urban housing’. For a project or an activity that targets a particular group, a specific indicator can be used. An example of this would be measuring change for women only, rather than women and men.

Neutral indicators, which are neither differentiated nor specific, are appropriate in cases where having a certain status or identity is not relevant to the changes being observed. For example, if the indicator is ‘all health advisors acquire knowledge about ethical standards of care for persons with disabilities’, it does not matter whether the health advisors themselves have disabilities or not.

Indicators for intended changes that are not measured in relation to people, but which have an impact on people’s inclusion in the project’s environment, can also reflect the degree to which an intervention is inclusive in its aims. These could include changes in institutional performance (outcome level) or the availability of a new product or service (output level). Imagine a project that supports a ministry in designing and implementing a policy. Even if the policy is not primarily focused on gender, you could have a gender-inclusive outcome indicator such as ‘every year, the ministry publishes gender-differentiated statements on the results of the policy’.

  • Combining quantitative and qualitative indicators

It is important to consider the use of both quantitative and qualitative indicators, as some changes are countable while others are not. Quantitative measures alone cannot capture inherently qualitative concepts, such as ‘empowerment’ or ‘gender equality’. Imagine you use ‘female-to-male ratio’ as an indicator for a capacity-building course. Even if you find that the course had a high rate of women participating, this does not tell you whether women and men equally benefited from it. To better understand the actual impact of the training, you would have to ask questions such as ‘Was the training content relevant to both men and women?’ or ‘Did participating in the training produce equally positive results for both?’. In other words, equal participation may be necessary for gender equality, but it is not sufficient by itself. Useful indicators for inclusion will focus on the substance and the quality of results, and define specific criteria for what constitutes each in the context of the project.

  • Using indicators to set inclusion targets

We can use indicators proactively to keep ourselves accountable if we have made a commitment to inclusion. Output indicators are often used to monitor quality and participation, which are both important aspects of inclusion within a project. We can set targets or quotas for inclusion at the output level accordingly. An example would be setting a target for a certain level of participation of a particular group in our activities, or of satisfaction among beneficiaries. In terms of accountability, input-level indicators can also give us important clues about whether an organization backs up its commitment to inclusion by actually investing human and financial resources.

Use a mixed-methods design for data collection and analysis

Just as quantitative and qualitative indicators can tell us different things about the performance and quality of our intervention, quantitative and qualitative methods for data collection and analysis also have different uses and benefits. The former are especially useful for describing what is happening to a large population and for generalising results, while the latter are especially useful for answering ‘why’ questions that allow us to better explain our findings in their particular context.

Having some combination of both methods helps us understand the effects of an intervention on a population in general and on different groups within it. Using triangulation, we can test the validity of our results for diverse groups. Having both perspectives also gives us the flexibility to explain changes in complex contexts and to understand what works and for whom, what does not work, and why. It also allows us to extract lessons that can improve a project’s on-going delivery or inform future interventions.

Collect disaggregated monitoring data

Disaggregating data according to relevant characteristics will allow you to assess the impact of your intervention on different beneficiaries, reveal any patterns concerning their participation in activities or use of products and services, and keep track of your project’s inclusion goals. By not disaggregating, some groups risk becoming ‘invisible’ in our data, which could lead to an intervention not being adequately targeted and therefore leaving people behind.

Data can be disaggregated according to many categories, for example by sex and gender, age, disability (and type of impairment, if applicable), geographic location (urban, rural, remote, etc.), race, ethnicity, class, income, language group, migratory status, marital status, sexual orientation, level of education, or religion. The choice depends on what information is most relevant and useful to your intervention and your indicators, and to the particular context you are working in. The initial situation analysis should inform this choice.

Ideally, disaggregated data should be captured in the baseline prepared for an intervention, and should thereafter be consistently collected, analysed, and reported on. Having good quality disaggregated monitoring data will also have a direct impact on the quality and usefulness of later evaluations in terms of inclusion.

Encourage the participation of stakeholders in M&E, in particular beneficiaries

The concept of inclusion is closely linked to that of participation. Involving stakeholders in the monitoring and evaluation of your intervention can make it more effective, accountable and sustainable. The beneficiaries of your intervention are ‘subject experts’ on their own situation and possess personal, localised and contextual knowledge. This knowledge can inform the design of an M&E system, help us gain a deeper understanding of findings, and ensure that results can be used by and for affected communities themselves.

There are many ways in which participation can be integrated in our M&E system, such as:

  • Deciding how progress should be defined and measured: collaboratively choosing indicators and metrics can make an intervention more realistic, culturally relevant and responsive in its aims.

  • Using participatory data collection methods: methods that actively involve primary stakeholders give us insights into the opinions and experiences of beneficiaries and can facilitate the data collection process itself. Many methods have been designed specifically for participation, such as community mapping or participatory ranking.

    Even if an M&E framework does not use such methods, however, we can still ensure more commonly used methods are as inclusive as possible by identifying and reducing the barriers to participation and representation for all groups. For example, you may hold separate focus group discussions for different groups (according to their sex, ethnicity, being a member of a host or displaced community, etc.) to enable people to fully express themselves if they would feel inhibited or unsafe in a mixed group.

  • Involving beneficiaries in interpreting data and acting on findings: a collaborative approach to interpretation is a useful way to jointly arrive at conclusions with beneficiaries or to review and validate conclusions drawn by M&E practitioners.

Take a look at my next article on Participatory approaches to M&E in which we look more closely at the benefits and challenges of using these approaches and the participatory tools available to M&E practitioners.

Establish a beneficiary feedback mechanism

Systematically collecting beneficiaries’ views can provide useful insights into the impact of an intervention and the quality of its processes. A beneficiary feedback mechanism can gather information about the extent to which an intervention reaches and benefits (or is perceived to reach and benefit) different groups equitably. For it to be an inclusive and representative monitoring tool—and not one that is dominated only by powerful groups—we need to identify and remove barriers to its use by the most marginalised groups. It is also important to put in place a process to make changes based on feedback.

When appropriately designed, a beneficiary feedback mechanism can encourage our interventions to be more demand-led, people-centred and inclusive. Tools for such a mechanism include suggestion boxes, a dedicated phone number or email address to receive feedback, a staffed help desk at the project site and community scorecards.

Uphold ethical and safety considerations

An M&E system that is not inclusive and therefore does not consider the differential impact it may have on people can have negative consequences. It may at best lead to some groups being left behind by an intervention, and may at worst have the unintended consequence of reinforcing discrimination or putting people’s safety in jeopardy. The methodological and practical choices we make about our M&E processes are also ethical choices that should be conflict-sensitive and be guided by the ‘do no harm’ principle.

When deciding what data to collect and how to do so, we should always weigh the benefits of possessing that data against the potential risks for informants. For example, since certain participatory data collection methods can be time-consuming, you might choose not to use them in an emergency response setting. Even though the data you could collect through these methods would be beneficial to the intervention or your organization, you determine that the process would place an undue burden on beneficiaries, who would otherwise be using their time and energy on survival activities.

Doing no harm also applies to how we go about data collection. For example, when interviewing survivors of gender-based and sexual violence, you should be able to provide them with referral information to support services in case they experience distress. You would carefully safeguard their anonymity in case they would be stigmatised in their community if others knew about their experience. This principle applies to those collecting data as well. The interviewer in this scenario would be at risk of secondary trauma when they hear their interviewee’s firsthand account of a traumatic experience. When recruiting interviewers, you would therefore need to consider their capacity to collect this kind of data and make sure that adequate support and/or supervision is in place for them.

Explicitly refer to inclusion in reporting and evaluations

Even if an intervention does not have any specific inclusion-related objectives, it is still possible to apply an inclusion lens by highlighting relevant findings from the monitoring data in our reporting. For example, it is worth noting if one group of people uses a product or service much less than another. As M&E practitioners, we draw insights from our data for the benefit of project decision-makers. By consistently highlighting inclusion gaps in our reports, we can make the issue more visible to them.

It is likewise good practice to build specific questions into an evaluation to assess how an intervention has performed with regard to inclusion. Consulting with beneficiaries to formulate questions is a good way to know what to ask in order to appropriately assess impact and change for different groups. If an intervention did not consider inclusion in its programming, this in itself is an interesting finding and the consequences of that decision are worth probing in an evaluation.

There is clearly a lot that we can do to make M&E more inclusive. Exactly how and the extent to which we decide to do it depends on our aims: we can simply collect data about the inclusiveness of an intervention itself; we can go a step further by making sure that primary stakeholders, particularly those who are normally marginalised or excluded, participate in our M&E processes; and we can actively use these processes and our findings to stimulate social change in our project’s environment.

Our choices will ultimately also be shaped and possibly limited by our circumstances. When designing an M&E system, we need to bear in mind organizational capacity, the human and financial resources available to us, and the context in which we are operating. In the end, however, even small changes that we make within these constraints can have a big impact on the visibility and meaningful representation of otherwise marginalised and excluded groups in our programming.

Using ActivityInfo for inclusive M&E

The tools we use to manage information have an important role to play in every M&E system and they can help us mainstream inclusion in our M&E processes. ActivityInfo was specifically designed to be used in challenging humanitarian and development contexts, and can therefore lower barriers to inclusive practices in several ways.

Quick setup and support: ActivityInfo allows organizations to very quickly and independently set up a database and data collection forms. Not having to build a database system from scratch or rely on spreadsheets being correctly exchanged and updated saves precious time in those early moments of an emergency response or intervention when it is especially critical to capture baseline data. The ActivityInfo team is available to troubleshoot and answer any support and technical questions, meaning organizations can focus their energy on the design of their intervention instead of system configurations and technical difficulties.

Accessibility: The platform makes it easier to ensure stakeholders are meaningfully represented in the data collected. ActivityInfo can be accessed offline and through both desktop and mobile devices. This facilitates data collection in remote locations and areas with low Internet connectivity. It therefore allows organizations to include populations in their monitoring and evaluation activities, who would otherwise be difficult to access. Forms are easy to navigate and can be designed in any language, which is a useful feature when deploying self-administered surveys in a local language, for example. The interface itself is also available in multiple languages, making it easy to use throughout an organization’s operations even if these span different countries, and facilitating third-party monitoring.

Flexibility in survey design: In addition, the flexibility of the form design feature allows the collection of quantitative and qualitative data for aggregated and disaggregated process and results indicators at any level of the intervention. Questionnaires can be constructed with validation and relevance rules to ensure the quality of the answers and can include built-in calculations to help organizations draw and refine conclusions as data is added to the form.

Data security: ActivityInfo is built on a secure foundation. Data is stored on servers in multiple locations across Europe to ensure there are enough backups in case of a natural disaster or failure of a server in a region. There is also a wide variety of user permissions that can be set in the platform to completely control who can access what data, thereby making it easy to protect beneficiaries’ personal information. With specific form fields, such as serial numbers, you can also create anonymous records and even build a complete case management or referral system. The ActivityInfo team also offers webinars and guides on data security to inform M&E practitioners about best practices and security risks.

Reporting tools to encourage participation: Finally, the platform facilitates accurate monitoring and collaborative data interpretation. Reports on ActivityInfo are always updated in real-time as data comes in, so it becomes easier to track and assess the quality and impact of an intervention, including for disaggregated population groups. Organizations can involve beneficiaries and other stakeholders by easily publishing and sharing visual or tabular reports based on the data collected in the platform. Published dashboards, maps and other reports are easily accessible and can be used to facilitate discussions about findings among project stakeholders.

Enabling conditions for inclusion

As M&E practitioners, we have tremendous power and responsibility: we can decide what counts as ‘progress’ and ‘impact’. We determine what gets measured and evaluated, and whose voice is taken into account. We are often in a uniquely privileged position to listen directly to affected populations and to steer the direction of an intervention through the evidence we present. We can therefore do a lot to advocate for inclusive practices in M&E and in development and humanitarian programming more generally.

And we should: while I’ve made the case for inclusive M&E today from the perspective of programming effectiveness, there are equally important and related ethical arguments too. Equality and non-discrimination are core human rights principles that are important to the way many of us approach our work (for example, they are embodied in the pledge to leave no one behind that is central to the Sustainable Development Goals). ‘Nothing about us without us’, a slogan closely associated with the disability rights movement, conveys the related principle that all people have the right to participate in the decision-making processes that affect their lives and well-being, and M&E is closely linked to those processes. Affected populations in this view are not just stakeholders in our interventions, but rights-holders. As development and humanitarian professionals, we strive to apply and adhere to professional standards, norms and codes of conduct, many of which are grounded in these fundamental principles of human rights.

Considering inclusion therefore naturally lends itself to our work, but we cannot do it alone. Inclusive M&E requires a set of enabling conditions that are underpinned by an organizational commitment to inclusion, such as an organizational culture that encourages the consideration of inclusion; the mainstreaming of inclusion into different aspects of the organization’s operations (SOPs, policies) and approach to programming (in the theory of change); dedicated human and financial resources (the size of the M&E team, having a separate budget line for inclusive M&E); and capacity for inclusive M&E (do we have access to subject experts and inclusion experts for input and guidance?).

We therefore need to be champions for inclusion not just in our own work, but also within our organizations and to our clients. To make programming more effective and responsive to the people we ultimately serve, we need to advocate for the conditions that will help us translate commitment to inclusion into action.

The team of ActivityInfo would like to warmly thank Ms Naomi Falkenburg for this insightful and detailed guide on inclusive Monitoring and Evaluation. Ms Falkenburg is a certified ActivityInfo Partner since July 2020.

Naomi Falkenburg is an independent consultant who works with humanitarian and development actors to design, manage and learn from their interventions and to conduct insightful research and analyses. Naomi believes the most impactful interventions are evidence-driven and inclusive, and she is passionate about empowering others to use M&E to this end. Prior to becoming a consultant, Naomi worked at several UN agencies and international NGOs in West Africa, Europe, and East Asia, on themes such as gender equality, disability inclusion, migration and forced displacement, decent work and skills development for youth, and digital inclusion. She is certified in project management for development (Project DPro) and member of the International Association for Professionals in Humanitarian Assistance and Protection (PHAP). Naomi holds an MSc from the University of Oxford’s Department of International Development and a BA in Politics and International Studies from the University of Warwick.

Sources and further reading