Initial Trend Analysis Report, Slides of Social change

Download Initial Trend Analysis Report and more Slides Social change in PDF only on Docsity!

Initial Trend

Analysis Report

BUILD Developmental

Evaluation

JULY 15, 2019

Contents

Introduction ...................................................................................................................................................... 1 Objectives of the Initial Trend Analysis................................................................................................ 1 ITA approach ............................................................................................................................................. 1 Limitations of ITA analyses .................................................................................................................... 2

Lessons from the ITA for the evaluation moving forward ........................................................................ 3 Overall lessons ......................................................................................................................................... 3 Case study methods ................................................................................................................................ Case study processes .............................................................................................................................. Full cohort survey ....................................................................................................................................

Data analysis and drawing conclusions .....................................................................................................

Next steps ....................................................................................................................................................... Finalizing the case study samples and starting the case studies ................................................. Key activities during the evaluative phase .......................................................................................

Annex One: Standard Data Collection Guide ............................................................................................. Annex Two: Case Study Profiles .................................................................................................................. Case study: Distance travelled Title: Emerging organizations ............................................................................................................................. 15 Case study: Distance travelled Title: Organizations engaging at the grassroots .............................................................................................. 22 Case study template: Distance travelled Title: Established organizations .......................................................................................................................... 30 Case study: Distance travelled Title: Organizations with leadership transitions .............................................................................................. 38 Case study: Distance travelled Title: Networks ....................................................................................................................................................... 47 Case study: Deep dives Title: Challenging environments ......................................................................................................................... 55 Case study: Deep dives Title: Organizations historically led by People of Color ................................................................................... 60

Introduction 2

early August (see Table 1). The convenings have provided an opportunity for the evaluation team to observe and absorb the grantees' experiences of the BUILD convenings and to obtain a clear- er understanding of the conversations underway between Ford POs and grantees about issues of strategic importance.

In addition, the convenings are an important opportunity to get feedback on proposed case studies that would benefit from additional expertise, for example challenges affecting People of Color-led institutions, one of the Deep Dive case studies. This iterative process to case study development is leading to a rich set of profiles and a strong set of proposed grantees for the evaluative phase. Giv- en the time involved in gathering feedback and communicating with grantees about their interest, some still need to be confirmed. During the summer, the team will also hold initial conversations with grantees that have confirmed their participation in order to introduce them to the purpose and process of the case studies and get initial feedback on the case studies' approach and questions.

Table 1: Activities undertaken during the ITA

Activity Visit to Indonesia office and grantees (November 4-9, 2018) Attendance at JustFilms convening (November 13-14, 2018) Attendance at OSA/MXCA convening (December 3-4 in Johannesburg, 2018) Visit to States Working Group grantees in Texas, Louisiana and Florida (February 1-10, 2019) Attendance at Women’s Leadership convening (March 6-7, 2019) Attendance at Puerto Rico Housing and Land Use convening (April 30-May 2, 2019) Attendance at BUILD-TTI Funders Meeting on Organizational Strengthening (May 3, 2019) Meetings with members of the evaluation team and BUILD Team (October 9-10, 2018 in Stockholm, other meetings in New York, January 31, April 1-2, May 3, 2019)

Limitations of ITA analyses

The BUILD and evaluation teams recognized from the outset that the evaluation would not be able to measure change against a ‘baseline’ given the heterogeneity of the BUILD grantees, absence of baseline assessments at the start of BUILD grants, and changes in and diversity of ways that initial proposals and reporting have been handled. For this reason, the case studies have been framed in such a way as to encourage grantees to self-benchmark , comparing their state of institutional and programmatic health when they first received a BUILD grant to their current state, and reflecting on the changes that have brought them to their current status. With this objective in mind, the eval- uation will primarily use the Most Significant Change method and the data gathered will be largely qualitative since it will be difficult to quantify changes in terms of ‘hard’, generalizable data from across the sample (see below).

Lessons from the ITA for the evaluation moving forward 3

Lessons from the ITA for the evaluation moving forward

Overall lessons

The ITA largely confirmed the appropriateness of the developmental evaluation (DE) ap- proach designed in the familiarization phase, given that DE embraces complexity and diversity of grantees, facilitates learning, and creates shared ownership among all stakeholders.

The proposed topics for the case studies have also been largely confirmed. Only two major changes have been made to the original plans from the familiarization phase. The first relates to the ‘Deep Dive’ case study originally planned for both People of Color-Led Organizations in the US, and Indigenous/ Afrodescendent Peoples' Organizations in the Global South. The evaluation and BUILD teams recognized that organizations in the Global South are likely to be confronting different issues than those led by Peo- ple of Color in the US. As such, the groups should not be merged into one case study and for this first Deep Dive case study will focus on grantees that have been historically led by People of Color in the US.

Plans for a possible Deep Dive on organizations of Indigenous/Afrodescendent Peoples have also been somewhat stymied by security and logistical factors that limit engagement with most of these grantees. The evaluation team will continue to actively monitor opportunities to address what is recognized to be a potential gap in the analysis. At the least, the evaluation team will meet grantees of Indigenous/Afrodescendent Peoples at a planned convening (date to be confirmed).

The second major change agreed during the ITA phase relates to the idea for a Deep Dive comparing earlier and later cohorts of BUILD grantees. This case study was expected to provide insight into BUILD's outcomes as a clearer and more intentional design emerged within the BUILD program. Giv- en the large range of changes underway in the broader context of the Ford Foundation, the evalua- tion and BUILD teams recognized that attribution of changes to BUILD per se could be problematic, so this Deep Dive will not be pursued. Nonetheless, the influence of changes in BUILD over time remains a topic that interests the evaluation team, as noted below.

The evaluation team has also proposed exploring several topics that may not warrant a full case study, or for which it may not be feasible to collect a significant quantity of data. In these situa- tions, we are considering undertaking a series of briefer ‘Vignettes’. The evaluation will continue to pursue discussions around additional Deep Dives. We see a value in remaining flexible at this point regarding the continuum between Vignettes and full-fledged Deep Dive case studies, and propose that we continuously calibrate the intensity, quantity and depth of the cases to reflect emerging issues and the extent to which the grantees are comfortable engaging. Currently, the following topics are being considered for either a Vignette or a Deep Dive:

• What is unique about institutional strengthening among organizations led by Indigenous/ Afrodescendent Peoples? Likely to develop into a Deep Dive

Lessons from the ITA for the evaluation moving forward 5

We recognize that the Distance Travelled grantee site visits, described below, are closely spaced over time for finding measurable changes. This is unfortunate, but we feel that the period between visits will be sufficient for assessing changes in thinking and early applications of the lessons that the grantees are learning when they (many for the first time) have the resources and time to reflect on how to strengthen their institutions and find ways to become more impactful.

The case study topics have been selected in order to explore what we expect will be these cate- gories of change. In analyzing the overall cohort of grantees across the case studies we will look for other patterns of change, but the Developmental Evaluation approach demands reserving judgement and allowing the interviews to reveal which factors will prove significant or measurable. Detailed sets of questions have been developed for each case study (included in the case study profiles in Annex Two), which reflect the evaluation team’s current ‘hunches’ regarding the most salient issues for the four overall learning questions in relation to each case study topic.

The case studies will include two sets of questions (see below for details):

• ‘Standard’ issues that will be assessed, compared and analyzed across the entire cohort of grantees participating in the case studies (i.e., 56, plus others that may be included in the sam- ple as part of deep dives to be conducted in the future). See Annex One for a summary of these ‘standard’ questions.
• Case study-specific questions that have been developed in order to drill down further into the key evaluation learning questions developed during the familiarization phase. The questions listed in the case study profiles in Annex Two have been tailored to focus on the issues that emerged in the familiarization phase of the evaluation, the co-creation workshop and subse- quent discussions.
• Both data sets will be synthesized to uncover patterns that may be unique to a given category of grantee, or which may appear across the overall cohort of grantees.

All the case studies will be conducted primarily using semi-structured interviews and will be triangu- lated with a variety of quantitative data salient to the issues being analyzed (largely drawing on avail- able grantee reporting). Other methods will vary across the case studies. As noted, most will employ a Most Significant Change methodology as a way to provide an open-ended approach to analyzing the heterogeneity of the ‘distances travelled’ by the grantees.^2 Some case studies will use After Action Reviews and Critical Incident Analyses to look at the ways that BUILD support may have contributed to more effective ways to manage specific events or challenges that can shed light on broader processes.

The proposed case study methods are tentative and may be adapted further once grantees provide feedback. The principle of not over-burdening the grantees is central to our planning and we want to ensure that grantees know that we are attentive to this. Lastly, the evaluation team is aware of

2 Related methods including Outcome Harvesting and Outcome Mapping have been considered, but ultimately rejected due to (a) the risk of placing too onerous a burden on grantees for monitoring processes over time, and (b) risks that an emphasis on outcomes related to a specific intervention might detract from attention to broader processes around institutional change and mission impact.

Lessons from the ITA for the evaluation moving forward 6

a growing number of studies and parallel evaluation initiatives that are being undertaken by the Foundation. We hope to be able to draw on the findings of these activities as complementary data and analyses. It will be important, however, to ensure that grantees are not confused by parallel initiatives that are underway.

Case study processes

Grantees will be asked about a set of general issues related to their mission and work and their experience with BUILD (see Annex One, Standard Data Collection Guide ). This information will help shape the analysis in the BUILD developmental evaluation as a whole. Following this general information, grantees will be asked questions related to the focus theme of their case study (see Annex Two, Case Study Profiles ).

The following summary of process, background data and reporting describes how the case studies will generally be conducted. There may be some variation from case to case as described in the case study profiles.

Steps and processes

1. Initial consultation – We will begin by talking with grantees about the objectives of the case study in which they are participating, and validating the lines of inquiry, questions, sub-ques- tions, indicators and methods with grantees. These conversations will be done via e-mail and phone calls in June-August 2019, prior to visiting grantees. During these conversations, we will ask grantees to provide us with relevant existing documentation so we can begin to get a sense of how the BUILD grant fits in with the grantees’ work and institutional trajectory. We will not ask for additional reports or analyses.
2. Grantee site visits – With the Distance Travelled cases, there will be two visits during the evaluation where we seek to learn about changes grantees have experienced over the time period of the BUILD grant. Each of these two visits will be two- to three-days long, depending on the nature of the specific case study. The first round of visits will take place during the latter half of 2019, and the second visits in mid to late 2020. With the Deep Dive case studies, we expect there will be just one visit of two- to three-days that will take place during the latter half of 2019. With both types of case studies, if there are significant developments in the life of the grantee during the course of the evaluation that could inform the case study, it is possible that an additional visit will be added. This optional visit would be to learn if, and how, the BUILD grant played a role during those developments. The Distance Travelled cases on Networks and Grassroots Organizing (and possibly others) may involve additional time and travel to visit net- work members and grassroots partners. For all visits, the team will strive to adapt the timing to respond to grantee needs and minimize strains during critical work periods.
3. Who we anticipate meeting – For both types of case studies we expect to meet with man- agement and staff of grantee institutions, as well as board members. To the extent possible,

Lessons from the ITA for the evaluation moving forward 8

Background data sources

Documentation review – The evaluation team will review all relevant documentation related to participating grantees including proposals, recommendations for grant approval, narrative reports and other analyses. We will review additional information provided by the grantees such as strategic plans, information about organizational structure and governance (including agreements with part- ners, charters, agreements with members, etc.) and assessment of progress outside of BUILD itself (such as annual reports, monitoring reports, results of stakeholder surveys, etc.). The evaluators may review OMT analyses if grantees wish to share them.

Financial resilience analysis – The evaluation team will review available financially related data from the grantees participating in the case studies to assess changes in (a) operating reserves, (b) ability to withstand financial shocks, (c) diversity/stability of other sources of funding (recognizing that diversifi- cation carries with it increased transaction costs and may not necessarily have positive implications), (d) proportions of restricted and unrestricted funding, (e) ability to maintain stable staffing and offer longer (more attractive) contracts, and (f) other notable changes in financial resilience during the course of the BUILD support. We recognize that the data available will not be strictly comparable and direct attribution of changes to BUILD support may not always be possible. US grantees are expected to have more uni- form (and thus comparable) financial data. We judge that the overall trends across this large cohort of grantees will reveal strongly indicative trends. We predict that the financial data will consist of:

• Available grantee financial information
• Analysis conducted by Ford Foundation staff for specific sets of BUILD grantees
• Data collected by BUILD grantees as part of their monitoring and evaluation routines
• Guidestar Financial SCAN data (primarily for US grantees)

Additional literature – In addition to grantee-specific documentation, the evaluation team will re- view other related literature including relevant academic studies, debates within the philanthropic and broader community, evaluations of similar programming, reports of additional Ford Foundation evaluation efforts, etc. in order to widen the perspectives on the issues being analyzed.

This additional information will also ensure that the case study reports inform the discourse in the social justice and philanthropic communities around the issues analyzed in the case studies and the overall BUILD developmental evaluation.

Full cohort survey

In addition to the case studies, we plan to conduct surveys of all BUILD grantees to learn about their experiences of BUILD and to compare and contrast the information with what we observe in the case study work. The results will help gauge the degree to which case study findings are likely to be true across the BUILD cohort. We plan to survey the cohort twice over the course of the evaluation.

Lessons from the ITA for the evaluation moving forward 9

We expect there to be three possible outcomes from the survey:

1. We do not learn anything new, meaning that the experiences of our case study grantees are validated by the survey of the whole cohort.
2. We learn about trends in the first round of the survey that we did not detect or appropriately prioritize from Familiarization and ITA phases, so we can adjust and follow those lines of inqui- ry while we have time. This is a developmental outcome of the survey activity.
3. We detect patterns of experience with BUILD, or even results, that we could not detect based on the small sample sets in the case studies. This could be relevant for various subsets: indigenous peoples’ rights related grantees, arts related grantees, small grantees, large grantees, etc.

The first survey will ask grantees:

• What areas of institutional strengthening, per the BUILD pyramid, did the grantee focus on with BUILD support
• How, and the degree to which, the BUILD grant and support was beneficial to the grantee
• Factors of BUILD that positively and negatively influenced institutional strengthening for grantees
• Which drivers of inequality the grantee focused on
• How the organization defines social change and how they intend to contribute to these outcomes
• Which factors of institutional strengthening have been most instrumental in enhancing the grantee’s impact on inequality
• The impact of BUILD on new or existing funding and partnerships
• Which specific aspects of BUILD support they find important, and
• What were the direct or indirect impacts on the field as a result of organizational changes made with BUILD support.

In addition, grantees will be asked if they are interested to have additional contact with the evalua- tion team in order to further explore the questions from the survey.

The survey was first tested in June 2019 with the NIRAS team, the Ford Foundation BUILD team and POs. Based on the results, we will update the survey and translate it into Spanish in July 2019, and administer the survey with grantees in September 2019.

The survey will be administered again in the beginning of 2021. The second round will have two sets of responses. The first set will be grantees that had a BUILD grant and responded during the first round in 2019. We may be able to detect aggregate patterns of change over time in this set. The second set will be grantees who did not have a grant during the first round of the survey. This set will be used to add to the early survey looking at the strategic focus, priorities and ‘early’ effects.

The analysis will focus on change trends and the degree to which the full cohort responses, as an aggregate, reflect what we observe in the case studies.

Data analysis and drawing conclusions 11

Anonymity – Full anonymity is nearly impossible in the evaluation, but confidentiality will be a pri- mary concern for the grantees. The evaluation team will anonymize data unless there is a reason to refer to the identity of a respondent. Where this occurs, the evaluation team will request permis- sion from the grantee.

Next steps 12

Next steps

Finalizing the case study samples and starting the case studies

As noted above, the intended samples for the case studies have now all been identified and grantees notified of our interest in having them participate. As the evaluation team receives an indication of interest from a grantee, an initial phone call is being arranged to hear the grantee’s initial reactions to the case study approach and overarching questions, and to request further background infor- mation. For some grantees, ongoing reassessment of security and other factors will be necessary.

The first round of actual fieldwork with the grantees is expected to begin at the end of August 2019 and be largely completed by December 31, 2019. This will be followed, in early 2020, by preparation of brief initial reports that will be shared with the grantees for verification and validation. The evaluation team will experiment with ways to enable exchange and dialogue among the grantees in a given case.

Key activities during the evaluative phase

Activity When With whom

BUILD/evaluation team meetings Approximately in October and Aprileach year BUILD and evaluation teams

Participation in convenings To be determined on a case by casebasis Evaluation team

5 case studies – distance travelled July – December 2019 Grantees, FF POs and BUILD team

5 case studies – deep dives July – December 2019 Grantees, FF POs and BUILD team Survey(s) September – December 2019 Grantees Evaluative report April 2020 BUILD team – in-person meeting Communication and learning discussions with grantees (either as side events of BUILD con- venings or as separate gatherings)

April – June 2020 US and regional; grantees, FF POsand BUILD team

5 continued case studies – distance travelled August – December 2020^ Grantees, FF POs and BUILD team 5-8 case studies – deep dives (possibly divided into a larger number of vignettes)

August – December 2020 Grantees, FF POs and BUILD team

Survey(s) August – December 2020 Grantees Evaluation report II February 2021 BUILD team – in-person meeting Final learning discussions with grantees, FF and philanthropy funders

March 2021 US and regional grantees, FF POsand BUILD team

Annex One: Standard Data Collection Guide 14

Influence on Net- works/ Fields – Did BUILD impact grantees’ roles in leading or taking part in networks in a way that strengthened those networks?

Data sources : Assessment of influence and outcome contribution, including interviews with constituents, stakeholders, observers Standard questions :

• Increased influence among other organizations and on mission goals
• Improved network leadership and contribution · Network member perception · Funder, stakeholder perception · Public/constituency perception
• Grantee perception of having achieved an appropriate and viable level of engagement with actors in the network

Design and imple- mentation – Has BUILD been organized and implemented optimally to achieve the desired impacts?

Data sources : Grantee interviews Standard questions :

• Review each of the BUILD ‘strategic levers’
• Flexible funding – evidence that funding enabled, encouraged, allowed and/or prodded intentionality around internal strengthening, external influence and strategic impact
• Long-term funding commitment – evidence that funding enabled, encouraged, allowed and/or prodded intentionality around internal strengthening, external influence and stra- tegic impact
• Grantee/Program Officer partnership – is the relationship collaborative? (grantee feels like a respected partner, making decisions, authoring the plan, regular supportive con- tact, grantee-PO trust and transparency, more than money, balanced); if there were changes in Ford strategies, how did this affect the relationship?
• CCTA impacts – increased and/or improved peer connections (new IS/operational part- ners, strategic and technical sharing/support, implemented practices, ‘sense of team’ i.e. mutual moral support)
• OMT influence – useful to organization, provided strategic clarity/coherence, was adaptable to broad and diverse contexts
• Effects of other processes underway at Ford Foundation – notably strategy shifts
• Grantee impressions of dialogue on Life After BUILD, BUILD 2.0, tie-off management (where relevant) and related processes, and how these engagements are influenced by having a BUILD grant
• Where is BUILD most/least important for institutional strengthening/mission impact
• Reflections on impact of being designated as ‘privileged’ in relation to others in the field due to access to a BUILD grant
• Unintended consequences and outcomes
• Other design questions: e.g., Who is left behind in the BUILD model? Are networks ade- quately supported? What might be better designed to support greater inclusion?

• Annex Two: Case Study Proles Distance travelled • Emerging Organizations 15

Annex Two: Case Study Profiles

Case study: Distance travelled Title: Emerging organizations Responsible team member: Raphaëlle

Synopsis

Even if BUILD has primarily been focused on well-established organizations, there is a significant cohort of organizations that have used the grant to formalize and stabilize themselves after having worked with less formalized processes and systems and/or less complex or specialized structures or staff.

These include:

• organizations with a relatively new or informal organizational structure;
• organizations with less formalized, undocumented or lower degree of strategic clarity and in- stitutional strength compared to larger, established actors in the field;
• organizations that may still be led and shaped by a founder director;
• organizations that have grown in size or staffing rapidly and now require a more formal gover- nance structure and organizational systems.

This is a fundamentally different process than that of other grantees that were already well estab- lished, and it is hypothesized that the contours of these theories of change will differ.

How learning questions will be addressed

Learning questions Relevant sub-questions Notable features related to this case Does strengthening key institutions and networks advance (or consolidate past advances in) social justice? If so, how? In what context?

To what extent can emerging organizations weigh in/ contribute to achieving broader social justice outcomes and has this changed during the period of BUILD support? What contextual factors are conducive or obstruct this?

• Annex Two: Case Study Proles Distance travelled • Emerging Organizations 17

Has BUILD supported grantees to develop their strategic clarity and coherence in the effort to dismantle inequality?

Are emerging organizations more likely to develop their first strategic plan with their BUILD grant? Do they plan longer-term than they have before, and what are the implications on their strategic direction? Does having a BUILD grant allow for more clarity and refinement on mission, vision, comparative advantage, place in the ecosystem?

Did BUILD help the organization “scale” or decentralize?

Do emerging organizations typically aim to grow/ scale their work, and if so, is BUILD enabling a more ambi- tious approach? If they want to stay small, what does this imply for the BUILD process? What is the most common scaling approach? Geo- graphical, sectorial, intersectionality, etc. Has scaling been too fast/ too ambitious for emerging organizations who received a BUILD grant? What are the unintended consequences? Does the formalization (that may accompany BUILD support to emerging organizations) lead to greater centralization or enable them to look further afield and decentralize? How is the BUILD impact dif- ferent from other donors or other types of Ford Foundation grant-making?

Explore this question for the perspective of organi- zations who are surviving off of small project based funding versus organizations which have lost a major (unique?) funder and need to be ‘saved by BUILD’ and/ or diversify their funding. Is diversification a problem for small, young organiza- tions that may become fragmented when responding to a range of donor demands? Is BUILD helping them to identify an appropriate business model to avoid this? How have organizations strengthened essential orga- nizational culture while un- dertaking major institutional changes?

Is BUILD enabling such thinking for emerging organi- zations which have a less established organizational culture? An already strong organizational culture? Are changes in the organizational culture happening in conjunction with major institutional changes, or do emerging organizations ‘keep the two separate’ to some extent? Is this limiting or not? (e.g. operating as an established organization but keeping the ear- ly-stage culture)

• Annex Two: Case Study Proles Distance travelled • Emerging Organizations 18

Are grantees becoming more resilient in relation to financial or contextual risks?

Explore this question for the perspective of organi- zations who are surviving off of small project based funding versus organizations which have lost a major (unique?) funder and need to be ‘saved by BUILD’ and/ or diversify their funding. Is diversification a problem for small, young organiza- tions that may become fragmented when responding to a range of donor demands? Is BUILD helping them to identify an appropriate business model to avoid this?

Did BUILD strengthen grantees’ roles in leading or taking part in networks and what have been their broader impacts?

If so, how? Do emerging organizations see leading or taking part in networks within their sector as a priority (perhaps given limited capacities and desire not to overstretch)? If so, how is BUILD enabling them to do so, and how is this different from what they were already doing ‘be- fore BUILD’? Is BUILD a catalyst for establishing their first formal partnership or collaborations with others? Are BUILD grantees developing stronger capacities to catalyze, lead and/or support collabo- rative mobilization within the fields they engage in?

Does becoming a BUILD grantee enable emerging or- ganizations to ‘get a seat at the table’ in their fields? To what extent is this beneficial to them (versus other types of BUILD grantee organizations)? Are there other ways in which BUILD helps emerging organizations to lead/ support/ participate in collabo- rative mobilization within their fields of interest? Have BUILD grantees elevated the work of their partners to respond to the strategic chal- lenges in the field?

Maybe not so relevant for emerging organizations, which are being elevated themselves presumably. We will however explore if emerging organizations are growing from a role as a service provider to becoming a larger/older organization in the network (e.g. Upturn). Do BUILD grantees influence the ways that power is distrib- uted among members of a net- work and if so, how?

Does becoming a BUILD grantee enable emerging or- ganizations to ‘get a seat at the table’ in their fields? Is this leading to stronger networks/ filling a gap in the field/ providing an alternative/ ‘shaking the grounds’ for more established organizations and their ways of operating? Do these networks generate broader influence in their re- spective fields?

Do emerging organizations generate influence within their groups of peers and partners who are similar in size and scope but are not BUILD grantees?

Has BUILD been organized and implemented optimally so as to achieve desired impacts? How has the General Operating Support (GOS) component of the BUILD grant contributed to the institutional strengthening of BUILD grantees?

Could be interesting to look into where GOS encour- aged emerging organizations to continue in full scale operational mode when it would be healthier to step back more and reflect on what might be a more appro- priate balance and ambition (similar to the grassroots organizations case study).