Disaggregating Data to Create Theories of Equity

Disaggregating data — analyzing subgroups of a population — is a critical step to understanding how, if at all, the outcomes of an evaluation differ across beneficiaries. All too often, this crucial step is skipped or not presented in the final report to internal or external audiences. Perhaps the disaggregated data are considered “too complex” to work with and report, as disaggregated data may threaten the status quo through uncovering inequities within the evaluand. This is unfortunate as disaggregating the data leads to an improved understanding of the value of a program, project, or initiative being evaluated, and whether that looks the same for all involved. The contexts in which we work are complex, with various stakeholders and power dynamics, which must also be considered. Disaggregating data allows us to identify the inequities present and create a new theory of equity to complement an intervention’s theory of change.

According to Patton’s theory of Utilization-Focused Evaluation, “evaluations should be judged by their utility and actual use; therefore, evaluators should facilitate the evaluation process and design any evaluation with careful consideration of how everything that is done, from beginning to end, will affect use” (Patton, 2003). If that is extended to data disaggregation, then evaluators should consider how disaggregating the data will facilitate the utility and actual use of the evaluation throughout every step of the evaluation process. It is within this framework that I suggest considering how to approach disaggregating data in evaluation.

Disaggregating Data in a COVID-19 Needs Assessment

I was recently conducting a needs assessment for rural communities in Central America measuring the effect of COVID-19 on various program outcomes aligned with the program’s logical framework. All the data was collected electronically via mobile data collection using a purposive sample of community leaders that each held various roles. Ninety responses were received across 60 communities. The overall results for a couple of the indicators looked like the following:

If the analysis ended here, I would have missed an important finding. To analyze the disaggregated data, ANOVA tests of significance were used to measure differences across groups. I found that the volunteer health workers had statistically significant different responses to the health-related questions than the rest of the respondents (p=.002), indicating that there is a significant difference in the responses of the volunteer health workers as compared to the rest of the respondents. The volunteer health workers (n=27) were more likely than non-health workers to report “major effects,” both positive and negative, when compared.

This presents possible new theories:

1. The volunteer health workers are more aware of the health situation than the general population because of the additional training they received.

2. The volunteer health workers are showing a form of stereotype boost, (Armenta, 2010) and felt that they needed to show a stronger response to the health-related questions than the general population because of their position in a leadership role.

3. The volunteer health workers understood the questions better and felt more confident in their responses.

There were differences in other leadership groups as well. Bank leaders reported major negative effects for income and employment as did bank members, while non-members of both groups reported a median response of minor negative effect. To contrast this difference, teachers and non-teachers all reported a median response of major negative effects of COVID-19 on school attendance with no statistical significance between groups.

In this case, disaggregated raised questions about the reliability and validity of the data that otherwise would not have been considered. Would more responses have significantly changed the results? This seemed plausible, though unlikely, as the pattern of statistical significance appeared in multiple areas or sectors of the evaluation results.

More importantly, what if a subgroup is uniquely able to provide insight to answer a question because of their identities and experiences? Without disaggregating the data, we lose the ability to identify the significance of these responses which could become critical in informing future decisions. This has implications for future evaluation designs as it could be appropriate to select a different sampling method such as stratified random sampling if we expect to disaggregate the data by specific subgroups. Constantly reexamining our processes allows us to improve our work as evaluators.

Approach to Disaggregating Data

Disaggregated data often refers to analyzing data based on gender, race, ethnicity, age, etc. to reflect the diversity of stakeholders (Bamberger & Segone, 2011). While I believe that these standard categories are crucial and need to be used to facilitate more equitable evaluations, I further suggest examining what is appropriate to the specific evaluation context to inform how data should be disaggregated to provide the most value in that context. In the example I shared, I also disaggregated the data by gender and age. This was important to see if different indicators were affecting different subgroups of the population differently, as that also would not have been seen in the original analysis, but leadership roles were the key component to understanding the evaluation as designed.

There is no one-size-fits-all approach to disaggregating data in evaluation. As always, the evaluand and context should drive how the data are analyzed and used. The primary intended users of the evaluation should further guide how to disaggregate the data. Taking a participatory approach to designing the data collection and analysis process is essential to this being possible.

Creating Theories of Equity

Once the data are disaggregated and the findings are presented, the question must be asked about what comes next. The Annie. E. Casey Foundation recommends targeting the right data and using disaggregated data to spur results and action. “The point of collecting and analyzing disaggregated data is to use data as a mirror and tool to uncover the drivers of disparity and inequity and increase opportunities and outcomes” (Annie E. Casey Foundation, 2016). Once we understand the differences between subgroups of a population based on the outcomes of an evaluation, we, as evaluators, are empowered to make recommendations about how to increase equity moving forward. Disaggregated data allows us to understand the disparities between groups and make recommendations about how to increase equity in the program in the future.

Rhonda Vonshay Sharpe suggests, “It’s simply not enough to look at the numbers based on age or race or gender, we must look at the interactions of all of those factors at once” (Vonshay Sharpe, 2020). Examining these interactions and intersections when disaggregating data is crucial as in the real world, we each hold many identities at the same time. Without this added context, the results are not facilitating the actual use of the evaluation as intended.

As evaluators, we must also consider what the lessons learned from the disaggregated data mean for the evaluand. Two of the four main components of equitable evaluation are the “ability of the design to reveal structural and systems-level drivers of inequity (present-day and historically)” and the “degree to which communities have the power to shape and own how evaluation happens” (Equitable Evaluation Initiative). We must work with communities to shape the data definitions and analyze and report on disaggregated data to reveal these drivers of inequity. New “theories of equity” can be added to traditional “theories of change” based on these findings. Further, disaggregated results may lead to entirely new approaches to the evaluation design or dissemination to ensure the utility of sub-group results.

Without theories of equity, theories of change alone may perpetuate inequalities within the evaluand by not explicitly addressing equity and inequity in the program. Creating a theory of equity is an opportunity for reflection for all involved to consider who the program is serving and whether that aligns with its goals. The theory of equity should drive the program and guide what data to collect and how to evaluate the program, including what is the most appropriate way to disaggregate data.

Through evaluative thinking with disaggregated data, findings are uncovered that could have otherwise been missed. To understand the real-world complexities in evaluations, we must determine what level they need to be understood. Evaluations measure value, and it is crucial that evaluations disaggregate data to continue measuring and adding value. Creating theories of equity and equitable change is a step we can take towards a more equitable evaluation practice.

References:

The Annie E. Casey Foundation. (2016, March 20). By The Numbers: A Race for Results Case Study. Retrieved July 19, 2020, from http://staging.aecf.org/resources/a-race-for-results-case-study-2/

Bamberger, M and Segone, M (2011) How to design and manage Equity-focused evaluations, UNICEF Evaluation Office. Retrieved from http://mymande.org/sites/default/files/EWP5_Equity_focused_evaluations.pdf

Better Evaluation. (2016, September 17). Describe the theory of change. Retrieved July 25, 2020, from https://www.betterevaluation.org/en/node/5280

Equitable Evaluation Initiative. (n.d.). Equitable Evaluation Framework™. Retrieved July 19, 2020, from https://www.equitableeval.org/ee-framework

Patton M.Q. (2003) Utilization-Focused Evaluation. In: Kellaghan T., Stufflebeam D.L. (eds) International Handbook of Educational Evaluation. Kluwer International Handbooks of Education, vol 9. Springer, Dordrecht

Vonshay Sharpe, R. (2020, May 13). Show Me The Data, But Disaggregate It First. Retrieved July 19, 2020, from https://wiserpolicy.org/show-me-the-data-but-disaggregate-it-first/