Understanding the Numbers: A Q&A with EngenderHealth’s Deputy Director of Organizational Metrics

Kristen Kelly is the Deputy Director of Organizational Metrics in EngenderHealth’s Impact, Research and Evaluation department. This department plays a critical role in supporting the organization in two of our core value areas: (1) evidence and innovation and (2) leadership and learning. In an interview conducted by Amy Agarwal, EngenderHealth’s Principal Writer and Editor, Kristen shares some of her thoughts and experiences related to gathering, understanding, and using data at EngenderHealth.

Your department is responsible for overseeing all of EngenderHealth’s data to ensure accuracy and quality. What are some of the key data that we collect?

Since much of EngenderHealth’s work focuses on sexual and reproductive health and rights, many of the data we track focus on contraception. For example, in our project-supported facilities, we track delivery of contraceptive methods, including short-acting, long-acting reversible, and permanent methods. Short-acting methods include condoms, oral contraceptives, and injectables; long-acting reversible contraceptives include implants and intrauterine devices (IUD); permanent methods include female and male sterilization.

How do we collect data and how do we use the data we collect? Why is data collection and use so important?

How we collect data depends on the type of data and the local context. In some cases, we can access most of the data from national health management information systems. In other cases, we may need to access data from registers maintained at health facilities, or even create new methods for gathering that data at the facility or community level.

Then, by using standardized tools, such as the Marie Stopes International Impact 2 calculator, we can use these data to estimate key impact indicators, such as the number of unintended pregnancies averted. Also, by monitoring these data over time, we can track trends and evaluate the effectiveness of our interventions by measuring change. We can also use these data and related analyses to inform future programming in order to maximize our impact.

In addition to using data to help us internally ensure we deliver effective programs, these data are also important to our external audiences and stakeholders. Just as we are accountable for proper financial stewardship of our projects, we are also required to routinely share data as part of technical reports we submit to our donors. We also seek opportunities to publish our data in order to advance the global evidence base, for example, through conference participation and peer-reviewed journals.

We recently changed our approach for calculating impact. What led to this change and what key challenges and lessons did you learn from supporting this transition?

We shifted our approach in order to help remedy various challenges associated with our assumptions. We are now using contraceptive commodity data, whereas we were previously focusing on aggregate client data. We understand that clients may switch to different methods, visit different facilities, opt for short-acting methods (which require multiple visits per year), and request multiple methods for dual protection—and any of these actions could affect the accuracy of our impact estimates.

Our new approach helps prevent us from overestimating our success by using conservative formulas to convert commodities delivered into clients served on an annual basis. For example, for every long-acting or permanent method provided in a year, we estimate one client received contraceptive care; however, for short-acting methods, we use the number of individual method commodities that a client needs to be protected from pregnancy for an entire year for this estimate—such as four three-month injectables or 120 condoms.

While this transition will initially require extra effort to shift internal practices and ensure awareness among our stakeholders, it is an effort worth making to ensure we are presenting the most accurate information possible, both to our internal decision-makers as well as our external stakeholders.