Monitoring and evaluation, data management and the role of ActivityInfo in handling quantitative and quantifiable qualitative data in IBP
Executive Summary
The International Budget Partnership (IBP) was founded in 1997 and has the objective to make public finance systems worldwide more transparent and accountable through its work, running programmes that introduce social change in very complex contexts. This calls for handling a multitude of types of data coming from various sources, in a holistic way.
Managing all the different sets of data, and finding the links between them is an ongoing challenge due to the constantly changing landscape. IBP uses ActivityInfo for Monitoring and Evaluation. They track and analyse quantitative and quantifiable qualitative data on change processes that accounts for around 40% of all the tracked data.
ActivityInfo is used to systematically put together quantifiable information and it was introduced to the organisation as the solution to the need for a more systematic and structured data collection process.
The potential for offline data collection and creating databases that can be comparable to each other as well as the fact that no technical consultant was needed to get started or to apply changes to the databases were some of the most important factors that led IBP to choose ActivityInfo as a Monitoring and Evaluation software.
Rose Nierras, Director for Country Strategies in IBP assisted the creation of this Case Study with a detailed overview of the role of information management in IBP highlighting the vast range of information that needs to be collected via a variety of tools and processes.
Suvarna Hulawale, Monitoring and Evaluation Officer in IBP, contributed to the creation of this Case Study, offering invaluable insights regarding the challenges that a Monitoring and Evaluation Officer faces in implementing a new framework, the way IBP works towards a more systematic process for quantifiable qualitative data collection and the reasons why IBP chose ActivityInfo as the Monitoring and Evaluation software and data collection tool to work with.
Additionally, Vivian Ntinyari, Operations Officer for IBP in the Kenya country office enriched our understanding by offering the view from the country level perspective.
The International Budget Partnership
The International Budget Partnership (IBP) was formed in 1997. IBP collaborates with civil society organizations (CSOs) around the world to analyze, monitor, and influence government budget processes, institutions, and outcomes and ensure that scarce public resources are used effectively to improve service delivery and fight poverty. IBP works to achieve these goals by promoting public finance systems and processes that are transparent, participatory, and accountable and by building the capacity of CSOs to participate effectively in these systems.
In 2016, IBP created the Strategy and Learning Team (SALT) to bring together the learning efforts and connect them directly to the strategic decision making at the organization, programme and project levels. SALT shapes overall strategy and learning frameworks, undertakes evaluations, facilitates strategic reflection, and brings in the latest thinking from the field. IBP has always been known as a learning organization. Recently, the organization has given greater emphasis on systematic and rigorous review and reflections, evaluation and planning.
Learning is central to all of these processes and it is IBP’s intention to gain a better understanding of complex challenges and generate meaningful evidence and insights to inform and shape all the organization’s efforts.
This starts with articulating clear theories of change with outcomes, assumptions and learning questions, and setting up an integrated evaluation and learning approach that regularly makes available insights about the work on the ground.
In Kenya, IBP has been working on a wide range of budget issues engaging with civil society, media and government at both national and county level.
The focus of the country office is on increasing public understanding of and engagement with budgets and budget transparency and their work consists of research, technical assistance and training, and advocacy.
The role of information management within IBP
IBP runs programmes that introduce social change in very complex settings. Trying to realise social change in such complex contexts requires different types of information, at different points in time and from different sources, as there is a multitude of constantly changing factors involved that affect the landscape.
This calls for handling a multitude of types of data coming from various sources, in a comprehensive, holistic way. To be able to make decisions based on insights from data gathered, to monitor the work and progress made and to report to donors, a whole range of information stored in different bins of systems or tools has to be properly managed to facilitate knowledge sharing.
Rose Nierras, Director for Country Strategies is responsible for managing IBP’s Country programmes, while commenting on the complexity of the data management needs, she states,
“The kind of programmes that IBP implements attempts to introduce social change in very complex contexts. There are so many factors involved, there are so many moving pieces. There are different data sets that we need and actually work with, to help us monitor our work, track any progress we are making, decide how best to move with our programmes when we are actually faced with choices, learn about what it takes to enable and support change, and report to our donors. Trying to realize social change in very complex contexts requires different types of information from different sources.”
Collecting information does not equal generating meaningful information and insights. Actual knowledge comes from the way information is used and the way this becomes relevant to the mission of the organization as well as each programme.
An effective data management system must be able to connect the right pieces of information with each other. Users of the system must be able to make these connections. Also, staying relevant is important. The whole system in place for collecting information needs to be constantly evolving and adapting to the external factors that makes it possible for the information to be useful.
“A good system does not need to have all the information, all the time. The system must make it possible for us to pull up the information we need to make informed decisions, to plan more effective support activities for the partner organizations we work with, and report our progress to our donors. The situations on the ground are constantly changing , so we should be able to calibrate the system, whenever necessary,” adds Nierras.
The Kenya Case
An example to highlight how fragile an inflexible data collection system can be in an ever-evolving landscape and how hard it is to capture this concept within a tool is found in the case of Kenya.
Right now IBP collects gender disaggregated data on training participants with the type of skills/ capacity areas. Changes in the political landscape might make it necessary for IBP to adjust the strategies employed, to build different capacities and skill sets for partner CSOs. If the information collection systems are not flexible, they become meaningless.
“In the case of Kenya, we still collect gender-disaggregated training information, even if we change what kind of training we give. The newly elected public officials at both the national and the county levels may result in shifts in the political contexts; and these, in turn, may make our work easier or more difficult.
This may mean that we may need to modify how we do things. So we may deliver capacity building initiatives in slightly different ways, but we will still be collecting gender-disaggregated data,” states Nierras.
“The challenge here is to be able to determine what information we need to track and monitor. Together, the information we set out to track and monitor must be able to signal to us that there may be changes in how we are implementing our programmes because we are not making the kind of progress that we have set out to achieve.
Data systems need to stay relevant to not only the contexts where programmes are implemented, but also to the goal and objectives of the programme and the mission of the organization,” she concludes.
Meanwhile, at the country level the Operations Officer, needs to have a comprehensive, up-to-date overview of all the activities that take place in the field so that information can flow seamlessly between the different levels and offices of the organization. The actual progress of every activity needs to be efficiently transferred and translated to the project management level. This calls for team coordination to systematically collect and organize otherwise scattered information.
Vivian Ntinyari, Operations Officer for IBP in the Kenya office, comments:
“Connecting the organization’s strategy to the implementation of our interventions is a challenge that I have to address continuously.”
ActivityInfo as a tool for managing quantitative information in IBP
In IBP, the data gathered are both quantitative and qualitative and they come from various settings and contexts. IBP uses ActivityInfo to track, and analyse quantitative and quantifiable qualitative data that accounts for around 40% of all the data that the organization tracks.
ActivityInfo is used to ensure that the data gathered in it are consolidated in a way that is meaningful. It helps to quantify qualitative data, aggregate, disaggregate, and compare data in a more systematic way. Then, this data is combined with other data from other sources to produce an overview of the entire progress of programmes and to assist decision making and report writing for donors.
“I estimate that about 40% of data that the organization tracks is captured on ActivityInfo.There are huge information sets that are not in ActivityInfo, that also feed into the organization’s planning and decision making processes.
ActivityInfo has actually helped to organize information so that we are able to meet very specific needs of different units within the organization.
For example, Kenya needed to disaggregate between gender participation in activities but this wasn’t needed by South Africa, nor by India. ActivityInfo made it possible for Vivian and our other Kenya colleagues to track and pull up this information whenever they needed to,” states Nierras.
Also, another critical set of data that IBP handles is the data related to grants given to civil society partners. These partners provide reports to IBP regarding their work. In some cases, ActivityInfo is used to crosstabulate, cross-validate or triangulate data in relation to these reports.
Nierras explains,
“The grants that IBP gives to civil society partners in countries account for about 40% of our budgets in country. Partners that receive grant funding support from us have to write regular reports. For a whole host of reasons, it does not make sense to include grant information (such as funds transfers, etc.) in ActivityInfo. This is because we need to be able to carefully track this information to comply with US federal rules and regulations. So ActivityInfo does not have the information on the actual funds that are transferred to partners.
But there are other grant-related support activities that country offices provide to our partners that are tracked. This would include activities such as training and the provision of technical assistance. In ActivityInfo, this information can be used to cross-check and validate information in the narrative reports that we regularly receive from partners”.
In addition, in IBP Kenya, ActivityInfo is used to provide activity level information for the quarterly reviews and reflections carried out in the country office. These are sessions where the team, sometimes with the participation of other partners, reflects on the progress of programme implementation.
“There are regular review and reflection sessions in each of the country offices. Typically, this happens on a quarterly basis and essentially they consider what they planned to do, what they accomplished, why they got there or why they didn’t, and what worked and what didn’t.
Sometimes the reviews and reflections are done with partners and sometimes it‘s just the team.
In Kenya, Vivian has taken the initiative to actually process information from ActivityInfo in time for these quarterly reviews and reflections,” explains Nierras.
Vivian Ntinyari, Operations Officer for IBP in the Kenya office comments,
“The reports I export from ActivityInfo feed into our internal quarterly reflection sessions to help us collectively track our efficiency and evaluate the effectiveness of our implementation strategies.”
Transitioning from Excel to ActivityInfo
As core funding support for IBP’s work grew, this also came with the need for capturing IBP’s contributions to realizing social change in terms of both a form of quantitative and quantifiable qualitative information. Most of the donors that extended core funding support to IBP asked IBP to complete logical frameworks and/or result frameworks, that also set specific quantitative targets that would measure progress. Nierras explains:
“When this was happening, our data collection systems were more canted towards qualitative information, and only minimally to quantitative data. Because we needed to track progress in terms of the targets set in our logical framework, we needed to further develop our data collection systems.
That was when we brought in Suvarna as an M&E Officer to help set this up for IBP with particular emphasis on certain programmes including those that were being implemented in countries.”
The systematic approach called for a significant amount of work to map out all the relevant stakeholders internal to IBP (at country, programme and IBP level) and externally (donors), their information needs, current sources and the creation of a pilot database in Excel. The process of identifying information need was participatory in order to create buy-in for more systematic data collection efforts, and to promote use of data internally for decision making and learning.
Suvarna Hulawale, the Monitoring and Evaluation (M&E) Officer who introduced ActivityInfo to IBP at a later stage, offered her insights on this initial phase of IBP’s data collection systems:
“What I did is map out all the logical frameworks we had submitted to all our donors, the IBP and programme specific theories of change, and the information needs for decision making and learning at both the country level and programme levels. The programme and country managers were involved in this process. Based on this, I created a database in Excel which we used for a year.”
Only when this database was successfully piloted and they gained confidence that they are collecting right information, did IBP decide to invest on a database platform.
The decision to move to different platform was due to the fact that the staff complained about the user interface and risk of losing data when multiple people had access to it. ActivityInfo was introduced by Suvarna Hulawale and then Learning Manager, Albert van Zyl to provide a more robust structure to the collection of data that can be quantified. This way the data was gathered systematically and could be used in conjunction with other data sets to support the realization of programme objectives.
“So what happened here was, we looked at the results framework, we set targets and then people had to start filling in information. Originally Excel was fine but the size of the data sets caused for Excel to slow down the collection and use of the data. It was then that we migrated to ActivityInfo,” notes Nierras.
As for the actual transition process within the organization, capacity building and turning systematized data collection into a standardized practice played an important role.
“Because most of the users were not aware of the importance of M&E, and how data can help them implement their strategy better there was a lot of resistance to puting certain things in place,” states Hulawale.
Capacity building is crucial to help people within the organization understand the importance of data and how to use it, in addition to how to use the database system.
Setting up a database system and understanding the importance and the relevance of the data to the actual work are two different things but have huge impact on each other. In addition to the capacity building, turning data collection into a standardized practice that is expected within people’s working duties, helps create a buy-in for a new system.
Suvarna Hulawale explained how she handled the transition process:
“There has to be a holistic approach. You cannot simply put a system in place; you also need to focus on capacity building . For example, we can create a really great database but if we don’t work to build people’s capacity... and capacity doesn’t mean necessarily how to input data but understanding the importance of data and the relevance to their work, and ability to use the data.
So the capacity is one thing and setting-up the system is another thing. There also have to be incentives attached to using the database.
Such incentives, include making the database work as part of the job profile of people who are expected to maintain and use the system.”
Leveraging the capabilities of ActivityInfo for systematic quantitative data collection
People working in the field need an easy and quick way to input data, especially where Internet connectivity is a challenge. They also don’t have time to waste waiting for the DC office to introduce changes in the database to better reflect the shifts in the country strategies.
Additionally, people in the DC office of IBP were concerned with the security of the data collected, as well as with the time and expertise needed for the configuration of a database or for the implementation of changes in existing forms.
Driven by these challenges and after some research, IBP started using ActivityInfo in early 2016 as a data collection platform for systematic data collection for their programmes.
“The criteria we used were: the user interface - how easy it is for a user to input data in the form, configuration - how long would it take? Can somebody non-technical do it or do we have to depend on a consultant?
Other considerations were the security of the data, the cost, and the time it was going to take to build the database platform and the offline functionality.”, states Hulawale.
At the country office level, the need for an easy and straightforward way to do data entry was met by ActivityInfo as well. Vivian Ntinyari, Operations Officer for IBP in Kenya is responsible for the evaluation and reporting of the programme activities in Kenya and to do so, apart from being in the field regularly, she has to keep track of her team’s work as well.
She has to summarize the wide collection of the data offered by her colleagues and share it with the M&E Officer at IBP through Activityinfo.
“I collect information throughout the project cycle. I also collect data from every staff member, every Friday, on the project activities and research that they were specifically involved in through the week to get a full and complete picture and an in-person review of the progress on each activity.
My role is to summarize the data, and input these into ActivityInfo. In that way, Suvarna and I are able to simultaneously monitor IBP Kenya’s project activities on a regular basis and generate reports to meet the donors’ information demands.”
As for the process of data entry and the creation of forms and reports, ActivityInfo is used after a short training, even when the users don’t have a M&E background. Ntinyari notes,
“I don’t have background on M&E. Data entry in ActivityInfo is pretty easy and straightforward, even for novices. The reports require a little bit of guidance and that is where Suvarna graciously comes in to guide me with the configuration. In comparison to the use of spreadsheets as a monitoring and evaluation tool, data entry, managing data quality, data validation and report generation is easier with ActivityInfo.”
Field officers could collect data easily, even when offline. IBP wanted to be able to make changes on their forms and databases quickly and easily, without depending on a technical consultant and without delaying the country staff.
“Being an M&E person, I know that if we depended on someone else to configure our database or introduce every single change that we wanted to make in the Forms we use, this would take too much time.
People in the field would need to unnecessarily wait for these to happen before they were able to enter or use information.These delays could cause for people to disengage from the system altogether”, adds Hulawale.
At the country level, the ability to work offline as well as the fact that ActivityInfo is cloud-based facilitates the overall work of the Operations Officer.
“The ability to use ActivityInfo offline and the fact that it is cloud- -based helps me in two ways. I can enter data when I have limited access to internet, as it often is the situation in the field, and then the data I collect and analyse is accessible to all my colleagues as and when they need it”, explains Ntinyari, Operations Officer in IBP Kenya.
Another issue that IBP was facing regarding their data collection system was dealing with specific information needs of specific country offices. While most of the data being collected needed to be consolidated for IBP-wide totals, some data needed to be reported on, only at the country level. This could have been solved by having multiple data collection systems, such as different kinds of platforms in place for each country, but consolidating the data across platforms for IBP-wide totals would be both difficult and costly.
Hulawale explained how IBP leveraged the potential of ActivityInfo for the creation of very customized databases.
“There were numbers of Form fields or numbers of data points common across 3 countries but there were also some data points which were specific to particular countries. The best example for this has already been discussed – only our Kenya office needed to collect gender-disaggregated data to fulfill donor requirements.”
In ActivityInfo, IBP was able to create different databases for each country programme and individual programmes.
The IBP system has a total of seven atabases. To keep them comparable, thus allowing information to be consolidated across each database, the Form fields that were common across the countries and the programmes, were carefully aligned. For the specific information needs at country and programme level, specific Form fields were included for the relevant country, such as for example, the addition of a field on gender-disaggregated data to the Kenya Forms.
Another example could be the names of the campaign and engagement as a dropdown list. Each country has its own campaign and engagement, so country-specific dropdown lists were created. This way, the database captures the data needed at country and programme level as well as at IBP level.
“By creating separate databases and having a separate Form for each of them , we were able to include the gender disaggregated numbers field for the Kenya team while the other countries did not have to respond to that.
By aligning Form fields, we kept databases comparable. Form fields for publications would contain the same information across the databases - who is the author, what is the title of publication, what is the purpose of the publication. This would make it possible to consolidate information on publications easily enough across all databases.” adds Hulawale.
Conclusion
Trying to realize social change in very complex contexts requires managing different types of information from different sources in a way that is flexible and constantly adapting to the changing world.
For the Internation Budget Partnership, an effective system needed to handle qualitative and quantitative data that could be connected to other data sets in other systems (e.g., grants management system), and can allow users to pull out information that is needed for decision making about programmes.
Organizing and structuring a system for quantifiable qualitative data about IBP’s contribution and contribution data collection related to targets of results frameworks called for a different approach. Following a year of running a pilot database in Excel to address this challenge, IBP was introduced to ActivityInfo. Activityinfo has been used since 2016 in IBP to help structure a specific set of data sets that accounts for about the 40% of the total data handled in the organization.
Some of the reasons ActivityInfo was selected as a tool for this process were the flexibility potential of the Forms, the fact that there was no need for a technical consultant to apply changes or configurations to the databases as well as the offline availability and the security of the data collected.
Hulawale, M&E Officer in IBP, concludes:
“When we decided to go for ActivityInfo, one of the reasons was how quickly we can configure the database . As soon as somebody says we want to add this campaign to this Form we can do that in a minute.
So we managed to have a database in place in the first quarter of 2016 and people were able to input the data for that first quarter database. The amount of time and ease it takes to configure ActivityInfo has greatly helped in the quick transition to the new database.”
We would like to warmly thank Rose Nierras, Director for Country Strategies in IBP, Suvarna Hulawale, Monitoring and Evaluation Officer in IBP and Vivian Ntinyari, Operations Officer for IBP in the Kenya country office for providing their valuable insights for the creation of this Case Study.