ACDI/VOCA manages M&E at the global level and supports the success of over 25 project teams with a common MIS platform

Download printable version

We would like to thank Dr. Emmanuel Dormon, Managing Director and Head of the M&E, Reporting and Learning department, Dr. Jennifer Himmelstein, Director of Corporate Analysis, Mr. Robert Sackey, Database and Technology Systems Manager, and Mr. Ryo Nakagawara, Data Scientist at ACDI/VOCA for their valuable insights for the creation of this Case Study.

You can also watch a presentation by ACDI/VOCA where representatives of the organization discuss how they approach Monitoring and Evaluation and Information Management at global level and share insights, lessons learned, and best practices as well as their experience in introducing ActivityInfo into their processes.

Information management and M&E objectives in ACDI/VOCA

ACDI/VOCA is a nonprofit international development organization that offers technical and management assistance in agribusiness, financial services, enterprise development, community development and food security-promoting broad-based economic growth. With a diverse portfolio of projects, the organization is contributing to the UN SDGs and supports the vision of a world in which people are empowered to succeed in the global economy.

Working with over 25 projects in at least 20 countries at a time, the organization has demanding information management needs including donor indicator reporting, monitoring the strategic impact of the organization as a whole and supporting cross-project learning. To do the former, the Monitoring & Evaluation, Reporting and Learning team at the Home Office, aggregates and analyzes data from databases of projects with similar themes, approaches and/or objectives to learn the success factors and challenges and shares recommendations for adaptive management.

ACDI/VOCA also has a set of corporate indicators that the MERL team tracks using data from the individual project databases. The corporate indicators are developed in line with the organization’s strategic plan and are linked to six of the UN’s sustainable development goals (SDGs). These indicators derive from indicators related to the progress of each individual project. To be able to have an overview, the M&E team needs to aggregate indicators from all these projects. However, this is not a simple matter of adding indicators together. Instead, the M&E team needs to be able to pull specific, relevant indicators from various projects and combine them together so as to gain insights that can inform each corporate indicator.

In addition, through its projects, ACDI/VOCA is applying cross-project learning to detect concepts that work or don’t work well in certain contexts or countries so as to adjust and improve. In this setting too, the goal is to be able to look at a variety of projects which are applying a particular concept or approach, e.g. a market system concept, and pull and compare data across some or all of them.

To align the work of a number of diverse projects which are spread out globally, the M&E team maintains information management systems that go beyond a ‘simple database’. Initially built on Microsoft’s Access and SQL server based databases and later migrated to ActivityInfo, ACDI/VOCA’s information management system is a sophisticated tool that supports their work from the project level to a corporate level. Today, ACDI/VOCA is using ActivityInfo to host this comprehensive information management system that serves and supports the data collection activities of over 25 projects and facilitates the synthesis of data for the reporting of high-quality donor indicator data, to inform the corporate indicators and support cross-project learning activities.

Diverse projects, diverse challenges

ACDI/VOCA’s projects are very diverse in terms of location (Asia, Africa, Latin America, Eastern Europe), funding (15-40 million, USD), team size (15-60 people) and skills (M&E Directors, MIS specialists, etc.). Also, in some projects, external partner organizations are involved. Every project is designed as a standalone project with its own M&E plan, goals and objectives, budget, indicators and targets. Project teams need to report to the project donor which is often the US government (USAID and USDA) and there are very specific regulations that need to be followed. Reporting needs to be done on a quarterly basis, bi-annually and annually. Most projects would have a 5 year duration and have annual targets as well as overall life of project targets.

To create comprehensive reports out of the project indicators, aggregate data for the corporate indicators that are linked to the UN SDGs to contribute to the UN impact report, the organization had initially built a system using Microsoft SQL Server, Access Web Databases, Power BI, multiple mobile data collection tools, and a custom integration and analysis layer written in R. Despite its advanced structure, the system still didn’t provide all of the features ACDI/VOCA M&E staff desired for optimal and streamlined functionality.

The system did not have an optimal solution for a data collection platform that could be used via a mobile interface, function on and off-line, or could feed directly into the Access Web Databases. The previous system used various data collection tools such as Google sheets, Kobo, ODK and others. The R/development team developed an internal API to connect ODK and Kobo to Google Sheets and then to Access Web databases. While this worked for some projects, for others, it caused delays and more workload for the development team. Finally, Microsoft announced the end of support for Microsoft Access Web Databases, adding new urgency. As a result, the organization decided to upgrade their platform and chose ActivityInfo to replace the previous database system.

“Since we were using Access databases and there is no actual mobile collection interface for Access databases, we had to connect so many different platforms such as Google sheets, Kobo, ODK and that was complicated. For some projects, it worked quite well but for other projects, a lot of time was needed to keep a lid on all the different problems of the 5 things we're trying to integrate at once. We got so busy with that so we couldn’t do other things that we wanted.” Ryo Nakagawara, Data Scientist, ACDI/VOCA.

“In terms of the value of the relational database schema, we wanted to maintain this so we could continue to track and analyze the multiple ways we assist each individual and organization (i.e. if one individual received multiple trainings, financing, and what that individuals' outcomes were in terms of say yield and sales as a result of these forms of assistance). ActivityInfo was one of the few solutions that maintained the relational database schema that we are very much attached to and that is why we supported it from the beginning.” Dr. Jennifer Himmelstein, Director of Corporate Analysis, ACDI/VOCA.

Steps towards a solution

Piloting the solution

Following a five month pilot of ActivityInfo with a selection of very diverse and complex projects, ACDI/VOCA moved on with the implementation of ActivityInfo to over 20 projects. Today, the organization centralizes project data and monitors the projects’ progress in the ActivityInfo platform and collects from project reports the information needed to update corporate indicators and for cross-project learning using R.

Thanks to a very carefully designed change management plan and an in-depth analysis of the situation, ACDI/VOCA derived valuable lessons during the pilot phase and managed to successfully engage the project staff in the ActivityInfo implementation. Before the migration, there was always a pre-migration discussion, and the actual migration had the form of a training, during which the project staff would learn specific things about the platform.

“Starting small was important. Starting with a few projects, learning from these pilots and getting people to say what challenges they’ve faced and how they were or weren’t able to overcome them, helped us in moving from the pilot to the complete implementation. Before we even did the migration the project staff got trained on what was going to happen, why, and what the new platform was going to look like. So they could identify what they didn’t know. As we were developing the pilot and migrating the projects we were all learning and that helped a lot when we came to the actual implementation phase.” Dr. Emmanuel Dormon, Managing Director and Head of the M&E, Reporting and Learning department, ACDI/VOCA.

Planning, involving those who will be actually using the databases, hands-on training alongside the migration and move of ownership to the project teams were key elements of the change management process. Now, when new projects move to ActivityInfo, the same process is followed. As for the database contents, forms and databases are designed in ActivityInfo based on the Activity Monitoring Evaluation & Learning Plan (AMELP) and a database schema designed collaboratively by HQ M&E support staff and project M&E staff.

Complementary skills inside teams and ongoing support

Another important factor was the combination of skills and knowledge in the project teams along with close support from key people from the Headquarters. All projects have an M&E Director. Depending on their size, a project team would also consist of 3 to 5 Monitoring and Evaluation officers and often, a Management Information Systems (MIS) specialist or someone who would combine knowledge of M&E and MIS. The Headquarters M&E project support staff would collaboratively work with the project M&E staff in the development of database forms and fields, utilizing the indicators performance indicator reference sheets in the AMELP, activities in the workplan and relevant technical staff to inform the process. The project MIS specialist is primarily responsible for managing the database and ensuring data security, building performance dashboards and generating accurate reports.

Then, the project team would take the lead in updating and managing the database, the data collection process and in the provision of reports and the Headquarters team would take on a more supportive role.

Setting common standards across projects and partners

To achieve consistency, the organization has set standard expectations for the way data collection should be done across all projects. All project staff are expected to use the mobile data collection feature in ActivityInfo for data collection and storage. This improves data quality because project staff using their phones or tablets to collect the data can directly link results to beneficiaries, firms, or other elements in the database. There is no longer a need to export from a separate mobile data collection tool, match the results to previously collected data, and then upload into the database-field devices are synchronized automatically and project indicators update in real time.

“Everybody is expected to use the mobile data collection feature. This is directly linked to the forms in the database and that makes the process really smooth. One of the reasons we like this kind of system -where the mobile data collection is part of the package- is that this way we avoid all the additional work outside the system.” Dr. Emmanuel Dormon, Managing Director and Head of the M&E, Reporting and Learning department, ACDI/VOCA.

“It's changed data quality and that’s the other benefit of the data collection interface aligning with the data storage interface. In terms of data quality, we can monitor data as it comes in whether that’s one of our partners, contractors, etc. We can make sure data aligns with different requirements and that basically everything is on the same platform. So it streamlines everything which is great and saves time as well.” Dr. Jennifer Himmelstein, Director of Corporate Analysis, ACDI/VOCA.

In some cases, ACDI/VOCA is also working with external partner organizations who implement specific aspects of a project. The partner organizations also need to collect data using the same database fields found in ActivityInfo forms so data upload and aggregation is smooth.

“In the past, partner organizations would collect data and send it to us and we would have to clean it and then upload it. That is one of the reasons we like the ActivityInfo platform because we want our partners -who are collecting data- to be able to use exactly the same database. And we provide them only with the access they need to collect the data they need. It saves us a lot of time.” Dr. Emmanuel Dormon, Managing Director and Head of the M&E, Reporting and Learning department, ACDI/VOCA.

Finally, all teams are expected to analyze data for their projects and create dashboards using the built-in analysis tools or Power BI.

Migrating from Access and Microsoft SQL Server databases

During the pilot phase, ACDI/VOCA started migrating the first projects from the previous system which was hosted using Microsoft SQL Server connected with an Access Web Database in Sharepoint. The decision on which projects to include in the migration was based on the stage of the project; for projects closer to their end, it didn’t make sense to put a migration plan in place. For the projects included in the migration, timing was very important as the organization didn’t want to disrupt the project, especially around the donor reporting period. The migration would always take place in communication with the project managers via a collaborative, hands-on approach.

For the migration, the ActivityInfo team worked with Ryo Nakagawara, Data Scientist in ACDI/VOCA, to develop an R script to migrate the table schemas and data from SQL Server directly to ActivityInfo. Maintaining the integrity of the data and keeping the relationships between forms intact was crucial in that stage. Following the pilot, 20 databases of ongoing and new projects were added to ActivityInfo. Tens of millions of rows of data were migrated smoothly and with the support of the ActivityInfo team.

“For me the most important thing was making sure that the relationships between all the different forms would be intact. Especially as soon as I knew that the way ActivityInfo makes the relationships, the links between forms and different data, was a bit different. But Alex helped a lot and we worked together so that the data integrity stayed intact. I can just click a few buttons, type in the new ActivityInfo database id and then I can go do other things, depending on how much data is in the Access database, it takes an hour or a couple of hours for some of our bigger databases and then it’s all there.” Ryo Nakagawara, Data Scientist at ACDI/VOCA.

As for the process Ryo advises:

“First things first, just get all the data in ActivityInfo and then from there start developing and changing and configuring it to the kind of architecture that ActivityInfo is better suited for.”

Result

Following the successful implementation of the ActivityInfo platform to these first 20 projects, ACDI/VOCA continually adds new projects to the platform and collaborates with the ActivityInfo team on further developments on data analysis and other platform features.

The organization maintained the relational database schema which was focal to the design of the original information system thanks to the flexibility of the form and database design in the platform. The Headquarters team saves time and improves data quality and integrity as the data collection mechanism is aligned with the data storage and ActivityInfo provides crucial features such as a real-time overview of the data added or changed on the platform regardless of who submitted it. In addition, it is possible to pull and aggregate data from project to corporate level creating dashboards using the API and Power BI.

“The major highlights of ActivityInfo were data integrity, data quality, data audit, user friendliness, data collection with mobile tools, and the possibility to aggregate from different projects to corporate level.” Dr. Emmanuel Dormon, Managing Director and Head of the M&E, Reporting and Learning department, ACDI/VOCA.

“When we did our initial assessment of the different data collection and storage platforms available ActivityInfo was one of the few platforms that provided the relational database schema, easy traceability and navigation of raw data for internal and external data quality audits, and allowed for customization of database forms and fields as well as a range of user permissions. It really is unique in this sense. It also assists in mitigating double counting of individuals/organizations that we assist. Then, it was important being able to utilize PowerBI and of course the data collection interface and there is the opportunity to just evolve with the team in developing the platform. It was honestly the platform that was most aligned with what we were already doing with Access so I thought it made the most sense to go that way.” Dr. Jennifer Himmelstein Director of Corporate Analysis, ACDI/VOCA.