The Impact Indicators Tips Booklet: Practical and Credible Methods For Using the “But For” Rule to Document Extension Community Development Impacts

George Morse, Special Advisor, North East Regional Center for Rural Development;
Charles French, University of New Hampshire Cooperative Extension; and,
Scott Chazdon, University of Minnesota Extension

This document explores practical and credible methods for using the “but for” rule to document Extension Community Development Impacts. It was published in 2016 as a PDF, but has been updated here in HTML format. To receive a copy of the archived PDF, please email nercrd@psu.edu.

---

Contents

• Executive Summary
• Impact Indicators: What and Why?
• Benefits of Impact Indicators
• Defining Impacts Conceptually and Empirically
• General Principles for Collecting Indicators
• “Participation” Indicator Tips
• The “But For” Attribution Principle
• “Action” Indicators Tips
• “Impact” Indicators Tips
• Reporting and Communicating Impacts
• References and Notes

Executive Summary

Extension organizations nationally are constantly asked by stakeholders—including county, state and Federal funders—to document the impacts of their public engagement efforts in order to merit continued support. Until recently, community development (CD) Extension programs did not have a coordinated response to these questions. In 2010, twelve North Central Region States were the first to successfully aggregate their CD impact data around a set of common indicators, with affirmation from program participants that the impacts were indeed accurately reflected. This process was developed and refined and now serves as a model for the other three regions who are drawing from the tools developed and the lessons learned by the North Central Region. Given that context, this paper is written for Extension’s Community Development educators, especially field educators and state specialists, who work with major programs to strengthen methods in reporting program impacts, particularly changes in actions and long-term results.

Impact Indicators: What and Why?

“Is Extension making a difference?” “If so, how much?” These are the questions that Extension CD programs are asking themselves and being asked by funders. Impact indicators provide a clear concise means of addressing these questions. As such, it is critical that organizations clearly define, capture, and communicate their impacts (Starr & Hattendorf, 2013). Indicators also serve to document change over time—including changing demographic trends or audience needs—which can be invaluable resources for strategic planning. Lastly, indicators serve as great data snippets for communicating impacts back to funders and stakeholders, in combination with narrative data such as testimonials. All of these benefits lead back to enhancing Extension’s common challenge: sustaining its ability to secure public funding.

This paper provides in-depth tips on how to go back and check on impacts using the “but for” rule: the process in which a non-Extension person states that “but for” the program, the impacts would not have happened. While this paper discusses CD programs, Extension Directors and program leaders in all program areas will also find the principles helpful in identifying the type of staff development needed for collecting and documenting the impacts of programs, initiatives or other efforts. The specific tips are not mandates but suggested approaches.

The CD impact indicators effort began in 2006, when North Central Extension Directors asked program leaders in all program areas to find common quantitative indicators of the impacts of Extension programming. While the other program areas sought to measure the impact of a single program delivered in all North Central states, the CD program leaders elected to produce common desired CD outcomes to measure, rather than measuring impacts of a single common program. The North Central CD program leaders created four distinct program logic models for different types of community development programming that were being delivered in the region: community economic development, community leadership development, organizational development, and participatory community planning (NCRCRD, 2010; Nichols, Blake, Chazdon, Radhakrishna, 2015).While few states conducted programming in all four of these areas, the logic models were instrumental in supporting state efforts to develop strategies for measuring program outcomes and impacts.

With these logic models in hand, and in recognition of the need to share program impacts and sustain effective programming, state CD program leaders began collecting, reporting, and aggregating data on several community development impact indicators. As of 2014, CD Impact Indicators were being collected in the North Central and Southern regions with pilot efforts underway in the Northeast (NCRCRD, 2015 & SRDC, 2015).

The indicators vary depending on the nature of Extension community development programming in each region. Table 1 illustrates the data collected for the North Central region for 2014. This region includes twelve states (IL, IN, IA, KS, MI, MN, MO, NE, ND, OH, SD, and WI).

North Central States 2014 Impact Indicators	Average	Total
Educational Contacts	41,999	461,991
Number of racial minority contacts	4,403	44,026
Number of Hispanic contacts	1,821	16,390
Number of participants reporting new leadership roles and opportunities undertaken	364	3,635
Number of business plans developed	225	1,798
Number of community and organizational plans developed	54	595
Number of community and organization plans implemented.	61	672
No. of businesses created	87	697
No. of jobs created	859	6,872
No of jobs retained	753	6,025
Dollar value of volunteer hours leveraged to deliver programs (Independent Sector value)	$289,813	$2,318,500
Dollar value of organization and/or community-generated volunteer hours (based on independent sector value)	$297,887	$2,680,980
Number of volunteer hours for community generated work	17,767	159,906
Dollar value of efficiencies and savings	$250,272	$1,501,630
Dollar value of grants and resources generated by communities	$1,734,512	$19,079,630
Dollar value of resources leveraged by firms	$28,890,479	$260,014,311

Table 2 reports similar data for the Southern region which includes thirteen states (AL, AR, FL, GA, KY, LA, MS, NC, OK, SC, TN, TX, VA) and Puerto Rico and Virgin Islands. While there is some variation between states and regions, the majority of indicators are similar to that shown in Table 1. While state practices for collecting the data vary, some states have linked their reporting of indicators to formal program evaluation methods.

Regional Set Impact Indicators	Total	Universities Reporting
Dollar value of grants generated by organization or community	$120,670,027	9
Dollar value of other in-kind resources contributed by organization or community	$590,250	7
Number of plans (new or revised) adopted and that have begun to be implemented by community, agency, local government, business or disaster.	317	9
Number of participants who report new leadership roles and opportunities undertaken	3,688	8
Number of businesses created, retained, or expanded	57	7
Number of jobs created/retained	9,619	7
Number of new alliances or networks formed through some type of formal agreement or MOU	48	6
Estimated effort (Number of FTE’s)	62.41	9
Regional Set Impact Indicators	Total	Universities Reporting
Dollar value of grants generated by organization or community	$120,670,027	9

The Northeast region, which is just starting to collect data, follows the work on the North Central and Southern regions. An example of the NE indicators are shown in Table 7.

Benefits of Impact Indicators

Adoption of impact indicators by Extension state and field educators depends on the benefits of working these exceeding the costs. This section looks at the benefits primarily from the state and field educator perspective.

Benefit 1: Your program is likely to have greater funding, making it possible to reach more people or more communities.

Both public and private funders like to back programs that have a proven track record of making a difference. So investing time in these can help you expand resources to reach more people.
Eighty six percent of CD educators in Iowa and Missouri named this a benefit of indicators (Morse, 2015). Extension leaders want to convince state and federal funders that Extension programs are a good investment. Regardless of whether capacity-building funds or competitive funds are sought, having evidence-based programming impacts will be essential to ongoing public funding, especially in light of continuing pressures on public budgets. According to a 2010 study, since 1975, federal funding (in inflation-adjusted terms) for Cooperative Extension had declined by 40% (Wang, 2014) and even more since then. On a per capita basis, this translates to 58% cut in federal funds for Extension. Likewise, state and local funding cuts have resulted in major restructuring in a number of states (Morse, Markell, O’Brien, Ahmed, Klein, & Coyle, 2009). Pressures on public budgets are expected to continue in the foreseeable future. So it is more important than ever that Extension programs collect and aggregate impact data at the regional, and ultimately, the national levels.

One way to convince entities to support Extension programs is to show value to the public. In the past, Extension evaluation efforts have focused on private value, in other words, the value to program participants (McDowell, 1985). Over the past decade, a public value movement explored ways to identify how Extension programs make a difference to taxpayers who do not participate in Extension programs (Kalambokidis, 2003, 2011; Morse, Markell, O’Brien, Ahmed, Klein, & Coyle, 2009; Franz, 2011 & 2013; Chazdon & Paine, 2014; Kalambokidis, Hinz, & Chazdon, 2015). Whether the educational program is to increase the profits of farmers, to improve the health of children, to reduce environmental dangers of nitrates leeching into wells, or to provide better schools for low-income families, there are both private and public value to each of these programs. Yet, telling the story on either one requires documenting the effectiveness of the programs in achieving their educational goals.

Since the New Deal, the federal government has been interested in documenting whether programs work (Rossi, Lipsey & Freeman, 2004, Economic Report of the President, 2014). In particular, the 1998 AREERA act required the Secretary of Agriculture to evaluate research, evaluation and education programs. An in-depth history of evaluation trends within the Cooperative Extension system shows the continuing push to measure more impacts as well as participation and learning (Nichols, Blake, Chazdon, & Radhakrishna (2015).

Benefit 2: As a spinoff of #1, you are more likely to get raises if your programs are seen as successful. Indicators can help you demonstrate this.

Seventy-five percent of the CD educators in Iowa and Missouri reported the indicators were valuable in their personal promotion materials (Table 3). The indicators can be used directly to demonstrate a strong program. In addition, they can be used for journal articles or other types of scholarships which are becoming more important in Extension promotion criteria (Olson, Skuza, & Blinn, 2007; Morse, Markell, O’Brien, Ahmed, Klein, & Coyle, 2009). Informally, success in securing grants can be important to salary increases and the indicators are helpful in demonstrating success.

Benefit 3: The indicators help you improve your Extension programming, via feedback on what works and what doesn’t.

Eighty-six percent of the CD educators surveyed used the indicators to help them improve their programs (Table 3). The feedback from collecting the impact indicators provides insights on the program that allow continual improvement.

How Used	Percent
Improving the program	86%
Marketing my program – building stakeholder support	79%
My promotion materials	75%
Grant applications	54%
Justification for user fees or financial sponsors	54%
None of the above	7%

Benefit 4: If you make small adjustments in your program to facilitate impact indicator collection later, it engages the community more fully, builds more stakeholder support and leads to greater impacts.

Almost four in five (79%) of the educators reported that the indicators helped them build stakeholder support (Table 3).

One of the cornerstones for collecting impact data, as is discussed in more detail below, is the need to engage with key informants in the communities served by Extension programs, in order to gain third party accounts of the impacts of Extension programs. This process of checking in with former program participants is extremely beneficial, not only because it often yields solid evidence of impact, but also because it often leads to continued engagement between Extension and the community or organization being consulted. In Minnesota, where this process of checking in with former participants has grown, a whole new line of Extension programming targeting program “alumni” has emerged. This has been particularly important in the area of community leadership development, as these program “alumni” have the potential to be a real force in rural community development in all corners of the state. As Extension educators continue to engage, community members are also far more willing to respond to evaluative questions about indicators and the numerous ways that Extension education has been an impetus for their community development activities.

Benefit 5: The indicators help to justify participation fees as well as time costs of the Extension program.

Over half of the educators reported that they used the indicators to justify program fees. As fees and grants become more important sources of the funding mix, this is important (Table 3).

Benefit 6: The indicator results are helpful when applying for grants.

Likewise, more than half of the educators used indicator results in their grant applications (Table 3).

The new “effective altruism movement” describes its mission as: “to foster projects which use evidence and analysis to help others as much as possible.” (See the Center for Effective Altruism.)
This means that private philanthropy also wants evidence that programs that they support have measurable impacts (MacAskill, 2015).

One way of sustaining Extension is to seek new partners, and philanthropists are more likely to engage with partners with tangible outcomes to show.

As shown in Table 4, field educators see that impact indicators are important for funding, followed by usefulness in improving their programs. More than one in five felt the results were beneficial to their annual review.

Benefit	Percent
Helps justify funding	86%
Useful in improving our programs	45%
Evaluation and annual review	21%

Challenge	Percent
Time to collect and report results	48%
Inconsistent approaches used in data collection	21%
Problems understanding the “but for” attribution principle	21%
Extension field educators losing contact with community leaders	21%
Securing responses from community leaders	17%

Benefit 7: As you participate with others from other states striving to report indicators, this will enrich everyone’s ability to show that Extension makes a difference.

This benefit happens as educators trade ideas via journal articles, webinars, and conferences on their successes and challenges in using indicators. There is a National CRED Indicators Work Group that meets monthly to exchange ideas and develop educational materials.

There has always been somewhat of a “putting the cart before the horse” aspect to collecting impact indicator data. State program leaders believed that if they created a platform, and an appetite, for impact information, then the states would need to respond by creating more systematic ways of collecting impact data. To some extent, this has been the way things have unfolded in the states and regions that are currently engaged in reporting. According to Nichols, Blake, Chazdon, & Radhakrishna (2015), community leadership development has also been a rich arena for multistate research and evaluation activity. Several research efforts have studied the relationship between individual outcomes and community impacts including a large multistate study using the Community Capitals Framework, funded by the USDA’s National Research Initiative. Another example is the impact evaluation tool known as Ripple Effect Mapping (REM) that has been used by the Horizons program, a nine-state community leadership program (Nichols, Blake, Chazdon, & Radhakrishna, 2015).

Benefit 8: The indicators help you create a positive impression.

Scott Loveridge (Loveridge and Elrod, 2016) suggests that one way to think about the benefits of collecting and reporting the indicators is to ask yourself: Whom do you want to impress? In an informative and fun webinar, he lists at least eight people we are accountable to (yourself, your life partner, your supervisor, your dean, your university president, your state government, NIFA/USDA, and Congress). Then he suggests how the indicators can help you with each of these, starting with one’s own satisfaction in knowing what works and what does not.

Challenges to Extension Field Specialists

Given the benefits that field specialists see in reporting program impact indicators, why do program leaders in these states feel the impacts are still underreported? While “time required” was expected to be a perceived obstacle, we were surprised that only about one in five mentioned one or more of six other challenges. Yet, each can be an important obstacle to Educators in the Northeast who consider documenting impact indicators. To support field staff, Mary Leuci, Chair of the National Impact Indicators Group, suggested educational materials are needed on how to collect impact indicators. The next section explores ways to reduce challenges and obstacles for field educators.

Defining “Impacts” Conceptually and Empirically

In an analysis of evaluation studies published in the Journal of Extension, Workman and Scheer (2012) found that only 5.6% of all evaluations published since 1965 measured long-term impacts while two-thirds dealt with measuring short-term or mid-term outcomes. Despite thirty years of calls for more findings on long-term impacts (Smith & Straughn, 1983; Stup, 2003; Kelsey, 2008, Morse, Markell, O’Brien, Ahmed, Klein, & Coyle, 2009), there has been almost no increase in the percentage of studies focused on long-term impacts (Workman & Scheer, 2012). In order to address this need for more emphasis on longer-term impacts, the indicators focus on the actions taken and the outcomes rather than on learning and change in attitudes.

The term “impacts” is used in multiple ways. First, we will define how we use the term. Then we compare the three major means of determining impacts.

Logic models are used to identify the likely impacts of an educational program and help identify the types of data that are needed. The Logic Model below is adapted from the most widely used logic model used in Extension (Taylor-Powell, Steele, & Douglah, 1996) as adapted to economic development by the North Central Regional Center for Rural Development (2010). Although Workman and Scheer (2012) reported that nearly one-third of the Journal of Extension reported changes in learning, less than six percent of these articles report long-term impacts. In the indicator project, special attention is, therefore, given to these under-reported indicators. There is no question that the other stages of the logic model are very important, especially for improving programs. But ultimately, even program improvement depends on knowing if we are achieving the program’s ultimate goals.

Logic Model

Inputs

• What we invest: Faculty/staff, volunteer & research time & expertise, travel and materials, etc.
• Indicators involved: All

Outputs

• Participants/who we reach: Communities and their formal & informal leaders, businesses, governments and non-profits.
• Activities/ what we do: Research, teaching, extension, coaching, facilitating, technical assistance
• Indicators involved: participation indicators

Outcomes/Impacts:

• Short-Term Results – Learning: Increased knowledge & understanding for economic trends and conditions and community strategies for achieving their desired future
• Medium-Term Results – Actions:
  • Increased resources leveraged
  • Increased networks and partnerships
  • More informed decision-making and leadership
  • Indicators involved: action indicators
• Long-Term Results – Conditions:
  • Creation, retention and expansion of sustainable economic opportunities
  • Increased wealth and income
  • Reduced poverty
  • Indicators involved: impact indicators

This paper focuses on only three aspects of the logic model: participation, action and impacts. The first two are easily defined but “impacts” needs further discussion.

Defining “Impacts”

“Impacts” are the changes in conditions that can be attributed to the Extension program. For example, indicators might be “the number of community plans” or the “number of jobs created.” The difference in an indicator number before and after an Extension program is not necessarily a measure of the impact. This before and after difference would be the impact only if the Extension program was the only influence on the size of the indicator.

For example, using the “before and after” method for Extension programs related to community policies implemented for the year 2010, Figure 1 would suggest that Extension had an impact on 37 policies. However, if 15 of these would have happened anyway, then Extension can claim impacts on only 22 (the difference between 37 and 15). Using the same method, the impacts in 2015 are only 39 rather than the total of 57.

Three Alternative Ways to Measure Impacts

Since it is impossible to observe the “without program” result, it might be that the total change comes from factors other than the Extension program. There are three approaches for estimating the impacts with and without the program, accounting for the changes that might have happened even without the program.

These alternatives are:

1. Find comparable potential participants (individuals, firms or communities) that are not in the program and track how the potential impacts for this group compare with the changes for participants. The difference is the impact;
2. Randomly select which potential participants can be in the program and compare the results between participants and non-participants, with the difference being the impact; or
3. Use the “but for attribution principle,” asking informed community members questions to consider the multiple factors influencing the change and whether the Extension program was a critical part.

Comparable participant model

While the first two methods are the gold standard in measuring impacts ((Bartik and Bingham, 1997; Economic Report of the President, 2014), they are not practical for most Extension programs. The comparable participant approach is politically doable but requires a lot of time, skill, and financing to implement. To implement this, data must be collected on both participants and non-participants. Given the difficulty in finding time to even interview participants, this is infeasible without major grant funding.

Very few educational programs have these resources, with a few exceptions in the 4-H program (Lerner, et.al. 2005; Ratkos, & Knollenberg, 2015), which is more attractive to researchers due to the large number of individual participants. Illustrating the complexity and resource density of the key study of 4-H impacts (Lerner, et. al., 2005), the massive effort required the participation of 1,700 fifth graders and 1,117 parents in 40 cities and towns in 13 states with the research done by 16 co-authors. To control for other influences, the study used regression analysis rather than a strict comparable participant approach.

Random selection of participant model

Likewise, the random selection of participants is not feasible either because it is politically necessary to accept communities or participants on a first-come, first-serve basis or because there are not enough participants. In addition, many programs lack the sample size to do valid comparisons.
For example, in major community-level programs that include strategic planning (e.g. business retention and expansion), most states only have the capacity to handle one to three programs at a time. Few states would have ten or twelve per year.

“But for” attribution principle model

The “but for” attribution principle provides a means for considering the factors in addition to the Extension program that might influence the indicators. “But for” is a practical means of finding reasonable estimates for programs which are not large enough to use either the comparable participant or the random selection method. When done correctly, the “but for” attribution approach is a credible means of identifying changes that would not have happened without the Extension program.

General Principles for Collecting Indicators

This section covers general principles that apply to all indicators, regardless of the method of collecting the data (“but for” attribution principle, random selection, or paired participants). The principles were developed after a team of Northeast Extension professionals studied the work from the North Central and Southern regions and did pilot efforts.

How Tips Were Developed

The Northeast Learning Circle on CD Impact Indicators started in March 2015 by studying tips provided in a webinar by leaders of the North Central impact indicator initiative (Borich, Chazdon, Leuci, & Loveridge, 2013), plus several follow-up webinars with leaders from both the North Central and the Southern (Welborn & Jakes, 2015) regions. Then the NE Learning Circle tested these ideas with individual pilot projects to collect indicator data from current or recent Community Development (CD) programs. Six members of the NE Learning Circle (Brown, French, Haskell, McConnon, Morse, and Nichols) explored alternative ways to collect indicators. Other members of the NE Learning Circle (see sidebar on previous page) provided feedback on the pilot program efforts.

McDowell (1985) described how economic principles suggest program design will increase the public’s willingness to provide political support; the same work has relevance to selecting the programs where the “but for” attribution principle can be used most easily and credibility. Key points from McDowell’s article influence many of the tips that follow.

McDowell (1985) suggests the following conditions necessary for clients to provide political support for any Extension program:

• Benefits to the participants must exceed all of their costs, including time, travel and fees.
• Participants identify Extension as critical to achieving these benefits, i.e. they know that “but for” this Extension program they would not have these benefits.
• Net benefits to each participant must be recent enough and large enough that they are willing to speak up on behalf of Extension.
• Cost of going back and checking with participants about whether Extension had an impact is reasonable.

McDowell’s article shows how these four conditions suggest the need for Extension programs with particularized information (i.e., information that is specific to individuals, situations, or times). Likewise, it is easier and more productive to collect data on impact indicators for programs with particularized information. In fact, the indicators and the political support, although different, are closely related. As we provide tips for collecting indicator data, we will sometimes refer back to the role of particularized information and these four conditions.

The following principles evolved from our NE Learning Circle:

1. Focus on programs that provide particularized information;
2. Focus only on major programs;
3. Limit number of key indicators;
4. Learn how to use the “but for” attribution principle; and,
5. Document methods of collecting impacts.

Focus on Programs That Provide Particularized Information

“Particularized information” refers to information that is specific to individuals, situations, or times. McDowell (1985) argues “information that is particularized to individuals or specific settings is “more meaningful and more often employed in actual decision making than is generalized information.” Essentially “particularized information,” refers to written or verbal reports that address the unique needs of a particular individual or community.

Particularized information also provides more opportunities to build ongoing relationships with the audiences. Strong relationships with an audience is essential both for creating greater impacts and for the audience being willing to be quoted on attribution statements (McDowell, 1985). Written materials can also help brand the program as one that Extension made a critical contribution towards—one that would not have happened otherwise (McDowell, 1985). The community-level reports in community visioning and business retention and expansion programs are good examples of using written reports as a base for the impact indicators.

Focus Only on Major Programs

Focusing only on major programs is advantageous for five reasons:

1. Economies of Scale: The primary advantages of indicators (improving and promoting the programs) are more relevant to larger, state-wide programs and programs that continue over multiple years.
2. Economies of Scale over Time: There are economies of scale in collecting indicator data over time from programs with unique, data collection aspects. This does not mean, however, that the program needs to be static. Rather, it means that the ultimate goals and the audience are similar over time, while the pedagogical and andragogical methods might vary.
3. Higher Quality Data: Focus results in higher quality data, improving the value of the indicators.
4. Increasing Returns to Time Invested: There are diminishing returns to the time invested in reporting impacts if staff report about every program and educational activity. Conversely, there are increasing returns when educators focus on major programs.
5. Better Response Rates: Just as the longer a client or a group of clients interacts with Extension, the better the political support (McDowell, 1985), and the more likely they are to be willing to work with you on the indicators.

For these reasons, we recommend focusing on major programs. At the beginning of a new program, it is often difficult to tell whether or not it will evolve into a major program. Yet, each specialist or educator knows whether they are planning and hoping that it will continue or whether the event is a “one-time” request that has to be filled. If the latter, we recommend using only standard end-of-the-event evaluations. Naturally, some first-time offers for a local educator are part of a major statewide effort and should be included in the impact indicator efforts.

Focus on Key Indicators

You do not need to collect data on every indicator shown in Tables 1 and 2 or discussed later in this report. None of our pilot programs included all of the indicators for a single community. The North Central and Southern regions report the same experience for individual programs.

Think about the logic model of your program and identify those indicators that are Not Applicable (NA) because you are almost certain to not find impacts while spending precious time looking for them. Focus, instead, on the major goals of the program and look for impacts there. However, be continually curious and open to be surprised by impacts you may not have expected and have a mechanism to include new areas of impact. In several of our pilot projects, we found impacts that were unanticipated.

You want the effort to both take as little time as possible and impose as little risk of embarrassment on the community members answering your questions about impacts. Hence, it is best to focus on the key impacts that are likely in the program for which you are collecting indicators.

Learn How to Use “But For” Attribution Principle

All higher-level impact indicators rely on the “but for” attribution principle to identify the changes and then reveal whether or not Extension was an essential part of the ‘team’ that led to change. Our feedback from educators and specialists showed a high level of misunderstanding of what “but for” statements claimed to do and how to do these well. Since there are many ways to use the “but for” attribution, it is discussed in depth later in the article and leads into our next principle.

Document Methods of Collecting Impacts

Careful documentation of the methods of calculating the indicators and/or of which local leader gave a “but for” statement that validates Extension was a critical part of the team adds credibility to the results. Furthermore, it helps Extension colleagues learn alternative ways to collect data.

Careful documentation will sometimes lead to disputes between Extension colleagues about what methods are the best for a given type of program or impact, yet this is one of the benefits of doing the documentation. Our goal is to learn about how our programs are doing so we can “make the best better,” as the 4-H motto urges.

Next we look at specific tips for collecting data on the six “participation” indicators.

“Participation” Indicator Tips

Tips related to the six participation indicators are covered in this section. The Northeast Learning Circle added three new participation indicators: the number of business contacts, number of public agencies contacted, and the number of employees working in the firms or agencies contacted. The reasons for these additions are outlined below. While “participation” indicators are easy to collect, these indicators are also important in communicating the breadth and depth of Extension programming.

Number of Individual Educational Contacts

Definition: Persons who received educational services via face-to-face or live distance-education sessions. Persons participating more than once or in different types of sessions should be counted for each session.

Tips: Counting works if that is all you need. If you want to survey later about other indicators, you will need contact information.

• Have participants in face-to-face sessions either pre-register with names and emails or sign-up at the educational event with names and emails.
• In on-line events, have people pre-register and give their name and email.
• Carefully guard these names and emails. They are gold for both your indicator efforts and for future programming.
• In cohort programs where groups of 20 to 30 persons meet multiple times, there is a debate on whether to count each person for each contact or just count them once. Some universities and extension services have specific guidelines. Whichever option you chose, note this in your documentation.

Number of Racial Minority Contacts

Definition: Contacts who self-report as non-white racial status.

Tips: Follow your institution’s approach on this or use the following Tips:

• On event feedback forms, ask an optional question about racial status and check with your human resources office to ensure you use the same categories as USDA requires for diversity reporting,
• Distribute the feedback forms as the next to last significant item in your session in order to capture high return rates.
• With online events, use the contact emails to link to an optional, online feedback survey with same questions.

Number of Hispanic Contacts

Definition: Contacts who self-report their ethnicity as Hispanic or Latina/o.

Tips: Same as for racial minorities.

Number of Firm Contacts

Definition: Firms that participate in programs which are aimed at helping the firms address their problems, e.g. small business management educational programs, business retention and expansion programs, and other educational programs or action research. Firms participating more than once or in different types of sessions should be counted for each session.

Why we added this new indicator: When the owners or leaders of firms participate in an educational program, it often has an impact that goes beyond the firm. Hence, this indicator gives a perspective different than the “number of individual educational contacts” indicator. When working with small business development programs or others aimed at this group, funders and legislators often ask how many firms were involved.

Tips:

• Count the number of firms even if each firm has several individuals participating. For example, if 40 individuals from 25 firms participate in the business retention and expansion program, the value for this indicator would be 25. The 40 individual participants are reported in the first indicator, “Number of Individual Educational Contacts.” Is this double counting? No. Each indicator describes a different aspect of the program impact.
• Regardless of initial method of contact, secure the email of the firm representatives the program worked with most closely. Having several is useful in the event of employee turnover.
• For programs already completed, explore whether a report exists on the program that describes the number and types of firms contacted. For example, in the Minnesota and New Hampshire business retention and expansion programs, their community reports list the participating firms (for example, see Tuck, Darger, & Ahmed, 2012)

Number of Public Agencies Contacted

Definition: Public agencies and/or non-profit organizations that participate in programs which are aimed at helping their agencies or organizations address their problems. Examples are:

• Local government agencies or organizations that are participating in leadership development programs in order to be more effective;
• Business retention and expansion programs learning how to support their local firms in productive ways, or;
• “First Impressions” programs aimed to help the communities recognize problems, or other educational programs or action research (Nix, Eades, & Frost, 2013).

Agencies or non-profit organizations participating more than once or in different types of sessions should be counted for each session.

Why we added this new indicator: This indicator is a parallel to the number of firms and public sector and non-profit firms are major employers in many communities.

Tips: Use the same tips as for private firms. Some state extension reporting systems make it difficult to include these public entities and track them. This makes it important for individual educators to keep these records.

Number of Employees Working In the Firms or Agencies Contacted

Definition: This indicator reflects the total employees working at the private firm or the public sector agency that participated in the Extension program and not the number of individuals who participated.

Why we added this new indicator. It is included because many of these individuals benefit when the leaders of a firm or an agency participate. Further, this is information that legislative leaders and private funders are often interested in. These data reflect the number of employees at the time of the educational event and not the impact of the program.

Tips:

• Ask this either in registration materials, in end-of-event evaluations or immediately after the educational event by an online email. Data on this are available in public sources for larger firms but often are unavailable for small firms.
• It is important to get these data at the time of the educational event so you can compare them with data after the firms might have expanded later on as a result of the program. This seems like extra work until several years later when you cannot reconstruct it.
• Data on employment six months or a year after the program can be gathered by a very short survey, using email to contact the firm and an online survey platform.
• Report both the number of employees from private firms and from public agencies rather than individually. Keep track of the levels for each, however.
• If working with regional agencies in several counties or with small non-profits with few employees and many volunteers, you might need to explore this further to give the full picture. If so, simply document how you did this.

The “But for” Attribution Principle

First, we explicitly define the “but for” attribution principle and then provide tips to use it effectively and efficiently.

What is the “But For” Attribution Principle?

The “But For” Attribution Principle has its origins in tort law. It has historically been used to determine if there is proximate cause that a defendant’s actions caused injury (West’s Encyclopedia of American Law, 2008) In the Extension evaluation context, we are interested in positive community impact, not injury, but we are still concerned with cause. Our concern is if Extension’s efforts had a causal effect on the community development impact.

The NCRCRD defines the “but for” attribution principle as “someone from outside Extension must be willing to state the program produced the result” (NCRCRD, 2013). While we endorse this innovative approach, we modify it slightly to read as shown here:

“But For” Attribution Principle — Northeast definition: The “but for” attribution principle can identify an impact and its size when “someone from outside Extension is willing to state that the Extension program was a critical factor in producing the program’s impact.

This slight modification from the NCRCRD’s definition explicitly communicates that there are other critical partners and factors in producing the impact. For example, if Extension teaches a small business a new practice that results in its expansion, the business itself is a key partner in making this happen. The educational program by itself does not result in implementation, and yet, if the small business owner says that “but for” the Extension program she would not have made this change, then Extension had an impact.

There are other program efforts which involve both Extension and one or more other agencies that assist the same firm. Say Extension introduces the new technology but a state agency helps with the financing necessary to implement it. Then Extension, the state agency, and the firm are all critical partners in the end result. Again, if someone from outside Extension is willing to state that without the initial Extension program the change would not have happened, this counts as an impact.

The “But For” Attribution Principle is a Legitimate Method

Since the use of the “but for” clause has a checkered history in local development, including the manner in which it is used, it is sometimes viewed with skepticism in Extension. But the differences between these two uses are significant and Extension’s use is legitimate.

“But for” in Local Development is Questionable.

“But for” statements have been a popular means of acknowledging the role of economic development programs for many years. Nationally, federal, state and local governments gave over $45 billion in tax incentives for economic development from 2008 to 2012 (Tax Policy Center, 2015). Most of these programs require the granting unit to find that “but for” the tax incentive, the development would not have happened without the incentives. In most cases, these “but for” statements are simply assertions without any economic analysis to justify them (Persky, Felsenstein, & Wiewel, 1997). While most local officials are aware that there is strong evidence that tax incentives seldom are effective, they also know that tax incentives matter in a few cases (Morse and Farmer, 1986; Hanson & Rohlin, 2011). Hence, denying the tax incentive makes the officials vulnerable to claims that the firm would have located in the city if only it had been given a tax incentive (Persky, Felsenstein, & Wiewel, 1997). While this uncritical use of the “but for” statements in local economic development casts a shadow on its legitimacy, it does not carry over to the way it is used in Extension.

“But for” in Extension is Legitimate.

Extension uses impact indicators very differently than in local economic development. These differences are important to it being legitimate.

1. In Extension, the “but for” statement is collected after the project rather than before; and
2. Those being asked about the impact have little incentive to claim that the Extension program made a difference if it did not.

First, the timeline makes a difference. Those providing the “but for” statements for Extension programs do so after the impacts have occurred rather than before. Hence, the persons who are asked for statements know intimately whether or not there are any impacts. Also, they are aware of the context before the program and what factors in addition to the program might have influenced impact changes. In contrast, those providing “but for” for tax incentives are doing it before the changes and are aware of neither the impacts nor changes in the context which might happen in the future during the period the impacts occur. Therefore, it is easier to be more accurate “ex poste” than “ex ante,” even without incentives to inflate the impacts.

Second, there are fewer incentives to inflate the impacts. If the names, communities, and contact information of those providing “but for” statements are recorded, there is less danger that individuals will have reasons to provide misleading “but for” statements. If the individual claims the program made no difference or if they say there was an change but that it would have happened anyway, the community members providing “but for” statements for Extension do not lose any funding. Compare this to the risk of local officials who refuse to say that “but for” granting a tax abatement, a firm will locate elsewhere. If the firm does locate elsewhere, the local officials will be blamed, even if they were absolutely correct that a tax abatement would not have mattered.

General Tips For Using the “But For” Attribution Principle in Extension

We now focus on the “but for” statements since most of the commonly accepted indicators can use the “but for” statements. Note that some tips for using the “but for” attribution principle are part of program design and implementation and set the stage for effectively collecting impact data
post program.

Ask about Future Plans during the Program.

Unlike technical transfer programs, in CD programs the participants set their own goals. To measure effectiveness of a program it is essential to know what these goals are.

Written Plans Are Important.

Written plans are important both for effective program implementation and for collecting data on impacts. The process of developing a written plan forces groups to reach a consensus on what they plan to do (Haskell and Cyr, 2011). Without written plans, some groups lose track of what they agreed to try to do, especially for the second tier priorities. Naturally, few plans are implemented exactly as written but they are still an essential document of the consensus reached by the group or the goals of individuals. President Eisenhower (1962) summed up the importance of plans stating: “I have always found that plans are useless, but planning is indispensable.” The way to develop a written plan varies by the type of program:

• Programs aimed at individual decisions: If a program helps individual decision-makers, it is sufficient to simply ask an open question, possibly after some group discussion. An example is: “What is one thing you have done differently after attending the workshop?” (Haskell & Morse, 2015)
• Programs aimed at group decisions: If the program requires a group decision based on the information provided, as in Business Retention and Expansion Programs, First Impression Programs, or Community Visioning Programs, provide time in the training program for the group to develop a draft a plan. Don’t just assume that your outreach teaching is so effective that the group will self-organize and write a plan.

Whenever possible, have strategic plans for community groups posted on your Extension website, such as this example from the University of Minnesota).

These written plans, either from individuals or groups, provide data for “going back and checking” on whether or not the plans have been implemented (Tuck, Darger, & Ahmed, 2012). This makes it much easier to contact local leaders to ask about the “but for” statements.

Examples of prior written reports helps in the branding and marketing the program. Most importantly the availability of the written plan improves the odds that groups will implement their ideas (Morse and Ha, 1997).

Include names of local champions.

It is easier to use the “but for” approach when you have the names of local champions who are responsible for implementation. This is one way to implement McDowell’s (1985, p. 718) solicitation condition. Particularly in programs done primarily by state or field educators not located in the community, having this contact information makes it much easier to “go back and check.” With the names in the Menahga BR&E report (Tuck, Darger & Ahmed, 2012), it was feasible for George Morse to “go back and check” on this Minnesota program using distance-education methods.

Build an Ongoing Relationship with Participants.

Even major program efforts need to build an ongoing relationship with Extension participants in order to effectively solicit either support or responses for impact indicators.

Of the many ways of doing this, we encourage developing a coaching model during the program implementation phase and building expectations that Extension staff will “go back and check.”

Coaching during the implementation phase:

Some Community Development programs build in periodic follow-up meetings with community groups after strategic plans are developed. For example, the Minnesota and New Hampshire business retention and expansion programs meet quarterly for one year after the plan is developed, even though there is a written plan and each priority project has two or three local champions (Morse & Ha, 1997; French & Gagne, 2010; French & Morse, 2015). In the Minnesota business retention program, we found that often the local champions for each project hustled to work on their project in the weeks just prior to the quarterly meetings. Also, when some of the small groups reported progress, this often sparked the other groups to greater action.

Obviously, it is much easier to collect the data when there are more impacts. Helping groups effectively implement their plans is a major step toward more impacts.

Develop a “Go Back and Check” Expectation:

During the program, or even in the program recruitment phase, let the participants know that you plan to “go back and check” on what is happening as a result of the program so that you can learn from them what approaches are most effective. One strategy is to build into the program an opportunity for participants to develop personal action plans for the steps they plan to take to apply what they’ve learned (Chazdon, Horntvedt & Templin, 2016). This process of developing an expectation for follow-up is complementary to the prior recommendation to provide ongoing support for implementation after the program.

Return visits:

In West Virginia’s Community Design Team program, the campus faculty leaders visited the community about six months after the program was completed. “This follow-up visit allows the community to ask questions about the report, update the team on their progress in implementing recommendations, and brainstorm about how to overcome obstacles they may have encountered” (Loveridge, 2002).

Local “reporters”:

Another approach for “going back and checking” is to cultivate several local community members as “reporters” on what is happening with the implementation and whether there are any impacts. These people, assuming they are not Extension employees, can later be the ones you ask for “but for” statements. Their reporting can simply be forwarding relevant local news stories or a quick email update.

Google alerts:

A new-media method using “Google alerts” on the local program or specific projects mentioned in the reports is another means of “going back and checking.” To build credibility, these methods need to incorporate “but for” questions which allow respondents to indicate that a change occurred and an opportunity to indicate whether or not Extension was a critical part of the team creating the change. Specific tips for doing this follow in the next section.

Ask One or Two Local People Not Working For Extension.

To avoid the perception of explicit bias, the recommended best practice is to ask one or two local people who do not work for Extension but are familiar with the program about the impact. If these individuals indicate that “but for” the Extension program the impact would not have happened, this is much more credible than Extension staff claiming their program created one or more specific impacts.

The “but for” attribution principle is used for all of the indicators in the “action” and “impact” indicators. We turn to these next.

“Action” Indicators Tips

The following five indicators (numbers 7 through 12) move along the logic model to medium-term outcomes and reflect actions that individuals or groups take to move toward their ultimate objectives.

Number of Business Plans Developed

Definition: This indicator includes formal business plans and informal strategic changes in either private or public sector firms which are developed as a result of an Extension program.

Tips:

1. Ask participants at the end of small business programs whether they plan to develop or change a formal business plans or to make other informal strategic changes. If possible, design the questions to be specific to the training program. The “Action Items” method described by Chazdon, Horntvedt & Templin (2016) may be useful in this context.
2. After the program, maintain contact by email or other social media formats, send participants a limited number of free pieces related to developing business plans, and ultimately build a relationship with participants. This not only leads to greater possibility of implementation and knowledge of impacts but improves the odds that you get a response when you survey them about doing their plans.
3. After four to six months, send an email asking: “What is one thing you have done differently related to informal strategic changes in your firm or in developing a formal business plan, as a result of participating in the XYZ Extension program in East Overshoe last October?” This is one example of how to use the “but for” attribution principle in data collecting.
4. Give program participants the option of sharing their names, since that allows you to follow-up with them in those cases where they have outstanding results and you wish to develop a short case study.

Number of Business Making Changes in Marketing or Business Management

Definition: This reflects the number of participants in Extension educational events who made changes in their marketing plans and strategies, networking, using social media, etc. and/or the number of firms establishing a new pricing policy, customer service policy, developing a new recordkeeping system, etc.

Why we added this: This indicator is similar to indicator #10 on “Number of Community and Organizational Policies & Plans Implemented.”

Tips:

• Use the same procedure as outlined in Indicator #7 and then add the next question in tip 2.
• “If you decided to use new marketing approaches or new business management processes, why was that?” And “Was there any topic covered in Extension’s small business development program that caused you to decide that starting a new business was not a good fit for you?”
• For examples of using this data see Bassano & McConnon, 2011.

Number of Community and Organizational Policies and Plans Developed

Definition: This includes both formally adopted plans for official agencies as well as strategies which were developed but not adopted as a result of the Extension program. For example, planning or zoning plans might be developed and either adopted or not.

Tips:

• Find a written document that outlines the plan. Without a written plan, skip this indicator. The document might be the minutes of a city council or a local land use foundation, a comprehensive plan, or simply the plan for one of the priority projects within a larger project.
• If the plan is developed after the educational activities, ask participants the following “but for” questions:
  • “Last year, you participated in the XYZ Extension program and were looking at ways to develop a plan to do XYZ. Did your group do that? Could I get a copy of the plan?”
  • If they have a plan: “What role, if any, did the XYZ Extension program that you participated in last [insert month] have in helping your group develop your plan?”
  • “Would your group have developed the same plan if you and your group had not participated in the XYZ Extension program?” If the answer to this is “YES,” there is no impact. With a “YES” answer, Extension was at the table but was not a critical part of the team in the view of the respondent. However, if the answer is “YES,” if we had been able to receive help from another agency, this means that they viewed the extension role as critical and that they could not have done it without help.
• These “but for” questions can be asked in a focus group, Ripple Effects Mapping session (Hansen Kollock, Flage, Chazdon, Paine & Higgins, 2012), an email survey or face to face interview. Increasingly, you can ask these questions in a focus group done by video conferencing from your desk or through a local public library.
• Be sure that the focus group includes only non-Extension employees. It may be most credible if someone outside the Extension program delivery team, especially someone skilled in facilitation, runs the focus group. Naturally, this often is not possible due to cost. Explore whether other extension colleagues who are not part of the program or a graduate student can facilitate the group.
• Whenever possible, make the developing of a written plan an integral part of the programming (Tuck, Darger & Ahmed, 2012). This provides a tangible particularized output that helps to brand Extension’s role and makes it easier to collect both political support (McDowell, 1985) and indicators. When Extension is facilitating the use of survey data in a group process with community leaders, there is no need to ask if the plan would have been developed without Extension. This is particularly true if Extension is invited in to help in the development of the plan. The primary benefit, however, is that this improves the odds of implementation and creates more impacts.
• It is okay to list the overall plan and each of its major action plans as separate items. For example, in the Menahga BR&E program (Tuck, Darger & Ahmed, 2012), there was one overall plan and five priority projects, so six plans were reported. However, the key point is to document exactly what was done in the event that someone wants to know about this.

Number of Community and Organizational Policies and Plans Implemented

Definition: This includes community and organizational plans which have been wholly or partially adopted or implemented for which an Extension educational program was critical to their development or to their implementation.

Tips:

• Focus on programs which have a written plan and where Extension’s role was either:
  • Helping participants develop the plan, and/or
  • Helping participants develop leadership skills that enabled them to be more effective in developing or implementing their plan.
• Determine if there are written documents that describe the role that Extension played in the implementation. If so, this makes the documentation much easier.
• Ask local community members familiar with the program the following “but for” questions:
  • “As you know, Extension’s community development programs help community groups explore new ways to solve problems. So we would like to learn about your experience on your XYZ project. Here is a list of action steps your group developed in (insert month & year) for the XYZ project.”
    • “Which of these projects have you been able to implement or are in the process of implementing?”
    • “What role, if any, did the Extension staff help you organize the implementation steps for doing this program?” If the answer is “NO ROLE,” then there is no impact on this. However, we suggest you ask: “In what ways do you think Extension might have helped your group do more on implementation?” This question might provide insights on how Extension can improve the program in the future.
  • If Extension had a role in the implementation, ask:“Would your group have been able to be at the same stage of implementation if Extension had not helped you during the implementation?” If the answer is “YES,” there is no impact. With a “YES” answer, Extension was at the table but was not a critical part of the team in the view of the respondent.
  • These “but for” questions can be asked in in a focus group method, Ripple Effects Mapping session, an email survey, or with face-to-face interviews.

Number of Volunteer Hours

Definition: This indicator counts the number of hours devoted by non-Extension staff to the program spent in training, action research, and implementation of projects spun off from program.

Why we included this under action indicators: The reason we include hours volunteered under action outcomes is because programs which stimulate more volunteer time have demonstrated to community members that there are benefits to either themselves or their communities which are worth investing in.

Tips:

• Include the time of those who participated in the Extension educational activities and of those who were recruited by participants to help on follow-up projects.
• If the project is action research or engaged scholarship, there probably are written reports that describe the leadership roles of volunteers. Check these to see if names are provided. If so, this is an easy way to document the number of volunteers.
• In some programs it is easy to calculate the number of hours volunteered without asking volunteers. For example, in the business retention program, each volunteer visitor spends about two hours per visit (one hour at the firm and one hour in preparation, reporting, and travel). Depending on whether the local program used one or two visitors per firm, you can calculate a reasonable estimate by knowing the number of firms. Make a note to document the estimation method you use.
• In programs where the time required for different roles is not known, it is advantageous for future programming to find out. It makes it easier to recruit people for programs where they have some understanding of the time required, or even of the variability involved.
• Should paid employees of an agency that is benefiting from the program be counted as volunteers? Yes. There is clearly an opportunity cost to their participation, with replacements needed, they have to work overtime to compensate for the time lost at work, or possibly lose paying customers during the hours they are absent from the business. Hence, including these paid-to-participate employees provides a clearer picture of the value of the time spent on the program. If you include paid employees, note this in your documentation
• In some regions, volunteer hours to implement community projects developed via the program are different than volunteer hours to deliver the initial educational program.

Dollar Value of Volunteer Hours Leveraged to Deliver Programs

Definition: The hours of volunteer time multiplied by the average dollar value per hour.

Tips:

• Use the hours estimated in the prior indicator.
• Multiply the hours by the value for your state found on the “Value of Volunteer Time” website
  • Note that the independent sector value varies annually and by state. Document this in notes. State and local funders consider estimates using state data more legitimate than national average rates so look up the state rates.

“Impact” Indicators Tips

The eleven “impact” indicators (Indicators 13 through 23) address changes in leadership capacity, employment, efficiency, and resources. These eleven reflect indicators that both private and public funding sources often ask about as they evaluate whether or not to continue or expand funding for Extension programs.

Number of New Leadership Roles

Definition: The number of participants reporting new leadership roles and opportunities undertaken as a result of having participated in an Extension educational program.

Tips:

• Include both formal leadership roles (e.g., board member, chair of committee, town councilor, etc.) and informal roles (e.g., advocate, group leader).
• It may be possible to integrate measurement of this indicator into program evaluation surveys. University of Minnesota Extension uses an organizational roles survey at the beginning and end of each leadership and civic engagement cohort program to identify the organizational roles held by cohort members and to identify if members increase their level of leadership in any of these roles, or add new roles, during the program (Lott and Chazdon, 2009)
• In action research programs or engaged scholarship where the roles are specified by the program, leadership roles by participants can be counted without asking them a “but for” question. Clearly, without the program, they would not have played these roles.
• In leadership training programs or educational programs that teach new skills, a follow-up survey that asks the following “but for” questions can be used:
  • “Since participating in the Extension program on XZY in (insert month, year), have you participated in any of the following: (insert a list of potentially related community groups)?”
  • “What was your role in these groups?”
  • “Did your participation in the XYZ Extension program provide you with any ideas, experiences, tools, or skills that were helpful in your new role?” If “NO,” the program had no impact. Then, we suggest you ask: “Do you have any suggestions of how Extension might change the program in the future to make it more useful to a person trying to do what you have done?”

Number of Businesses Created

Definition: This includes both new business startups and businesses that moved into the area as a result of actions taken by community leaders due to what they learned at an Extension educational program.

Tips:

• If the program works directly with businesses, it is easier to track impacts if they actually started up or moved into the area by simply surveying them in six-month to one-year intervals.
• Include public sector entities if they move into the community as a result of local actions. For example, if a new office building for the state department of highways is built in the community, this counts. But as always, document the name and nature of the business.
• If the Extension program works with community leaders who then facilitate new startups or attraction of firms, then it is necessary to survey a knowledgeable person or use a focus group to ask “but for” questions.
  • a“Since (insert month and year) of the XZY Extension program—have you had any new business startups in your community? Any new businesses move into the community?” If “NO” there is no impact. If “YES,” go on to the next question.
  • “What groups or agencies provided some help to your group or the firms in these startups? In the newly attracted firms?”
  • “Did the Extension program help you identify any of these groups or agencies?”
  • “Was the Extension program on XYZ at all helpful to you in facilitating these startups? For attracting these firms?” If “NO,” then ask: “Do you have any suggestions on what might be changed in the Extension program to make it helpful to people in your role?”
  • If the local leaders found the program helpful, ask, “How did it help?” Then ask: “Would you have been just as successful as you were without the Extension program?” If “YES,” there is no impact.

Number of Jobs Created

Definition: The number of new jobs from business startups or ones that moved into town as a result of efforts in which Extension was a part of the team that facilitated this, with Extension’s role being providing new ideas and skills which helped local leaders.

Tips:

• Only use the number of jobs from firms that were identified in the prior indicator as having been influenced by community action that was guided in part by ideas that came from Extension programs or by information that was directly provided to firms. Bottom line is that if the firm is not identified in the prior indicator, the new jobs in that firm do not count here.
• Otherwise, count the number of new jobs here.
• Include both full-time jobs and part-time jobs. Document numbers of part-time independently of full-time in the event it is requested. Don’t include “temp” positions that are very short term.
• Count the owner as one job.

Number of Business Expansions

Definition: The number of businesses which expanded their employment as a result of efforts in which Extension was a part of the team that facilitated this, with Extension’s role being providing new ideas and skills which helped the business expand.

Tips:

• Use this indicator to report only for Extension programs that have this as an explicit goal. Otherwise, you will need to demonstrate the logic behind the claim that it influences the change. Also, the odds of having contact information is very low.
• Send a very short survey to the firms that were helped after one year and ask what their employment level is. The survey might also include two questions about their satisfaction with local governmental services and/or concerns with these as a motivation to respond.
• For the firms which expanded, here are a series of “but for” questions, which should be asked directly to a key decision maker in the firm:
  • “In –Month, Year—you participated in the XYZ Extension program. Did this program give you any ideas or skills that were useful in the expansion? If so, could you give me an example? If not, are there any suggestions you would give Extension to help them do a better job in the future?”
  • If Extension gave some ideas, ask: “What other factors led to your expansion?”
  • If there were other factors, ask: “Would your expansion have occurred anyway if you had not participated in the Extension program?” If answer is “No,” then it is okay to claim credit for an impact. That is, Extension was part of the team that led to the expansion. If the answer is “Yes” or “Probably,” then Extension was not critical to the expansion and no impact should be claimed.

Number of Jobs Expanded in Existing Businesses

Definition: These are new jobs from existing businesses that expanded their total employment as a result of a project in which Extension was an essential team member or as a result of ideas or skills which the business leaders learned from the Extension program.

Tips: For firms that pass the “but for” test in Indicator #16, ask: “For the increase in the number of employees, do you have a rough estimate of how many more your business hired as a result of participating in the Extension program compared to the other factors that encouraged Expansion?”

Number of Firms Retained

Definition: These are firms which are at risk due to changes in product demand, energy costs, costs of raw materials, new technology, or global competition, which are helped to survive by becoming more efficient or shifting to new product lines.

Tips:

• Focus on firms which are in industries with rapid changes due to new technologies, changing prices, and global competition, e.g. the paper mills.
• Use this indicator to report only for Extension programs that have this as an explicit goal. Otherwise, you will have difficulty demonstrating the logic behind the claim that it influences the change or having relevant contact information.
• Generally, the large firm retention cases are ones that happen late in the firm’s decision cycle and there is generally little chance to make a permanent change that has a long lasting impact. However, these are also cases in which Extension professionals are likely to be called in to work on a specific aspect. The non-Extension professionals will be able to answer “but for” questions similar to those asked for business expansions.
• In business retention and expansion programs, the focus on retention is at a much earlier stage, where local and state professionals are trying to identify ways to assist firms to become more efficient or to adopt new product lines early enough to adjust to global trends. For the firms which are retained, here are a series of “but for” questions, which should be asked of firm representatives:
  • “In –Month, Year—you participated in the XYZ Extension program. Did you pick up any ideas from either your peers in that program, state agencies or Extension personnel that you have used in your business? If so, could you give me an example and who helped guide you to that idea?”
  • If Extension provided some ideas, ask: “Do you think these ideas have helped make your firm more competitive in the global market?”
  • If there were some other factors, ask: “Would the changes you made have occurred anyway if you had not participated in the Extension program?” If answer is “No,” then it is okay to claim credit for an impact. That is, Extension was part of the team that led to the expansion. If the answer is “Yes” or “Probably,” then Extension was not critical to the expansion and no impact should be claimed.

Number of Jobs Retained

Definition: The number of existing jobs in firms which were at risk of being moved or eliminated that were protected by actions taken with critical input from Extension programs.

Tips:

• It is difficult to estimate what number of jobs are saved by these changes and for how long. On the one hand, all the jobs may have been saved, unless the change was for a new labor-saving technology. Then, it saves only the jobs left while the cost of the other jobs fits under indicator #20, dollar value of efficiencies.
• Do the best you can on this one and document how you did it.

Dollar Value of Efficiencies and Savings

Definition: These are savings which come from improved processes, better workforce training, the adoption of new technology, scale economies, and better business practices. Some common examples of this is the reduction in wage costs when a new technology makes it possible to produce the same output with fewer workers, and when government units can provide more/better services by joining forces. While the technology loss of jobs seems like a negative outcome, it can be an important factor in the survival of the firm and the remainder of the jobs in that business, non-profit or public agency

Tips:

• This is the type of change you can capture by asking participants to indicate what changes they plan to take at the end of an Extension educational program or event and then going back and seeing if they did adopt it. Presumably they would not say they are planning to adopt the new method unless it was new to them.
• After asking if they have adopted a specific change, you can ask them, “Roughly how much was saved on an annual basis as a result of adopting this change?”

Dollar Value of Grants and Resources Leveraged by Community

Definition: Grants received as a result of the Extension program.

Tips:

• The “as a result of an Extension program” generally means that the community or participants learned about a strategic need in their community and could use this data to support a grant, learned how to do better grant applications, or learned new leadership skills which helped their group be more successful in grant applications.
• If working with a community group, ask the leadership of that group the “but for” question.
  • “Did you receive any grants is past year?”
  • If yes, “Was any aspect of the XYZ Extension program helpful to you in preparing the grant proposal?” If so, ask: “How did Extension help you, by providing information, by coaching leaders on grant writing, or helping the team do strategic planning? For those grants which Extension had a small but critical part in helping you secure them, what was their value?”
    • If the Extension audience included individuals rather than a community group, ask the same questions as above.

Dollar Value of Grants and Resources Leveraged by Businesses

Definition: Grants or contracts received as a result of the Extension program.

Tips: Same as for the community in Indicator #21.

Value of the Program to Participants

Definition: This is the value to the participants of the Extension program rather than the fee or the cost of participation.

Tips:

• If using a six month survey, ask a question similar to the following question used by James McConnon and Louis Bassano used the following question:
  • “Please place a value on the knowledge and skills you acquired as a result of participating in the Hancock County Business Conference and Trade Show.
    • $1- $500, $501-$1,000 to $4,501 -$5,000 and __other amount.”
  • Another approach is to use contingent valuation and ask a questions similar to one suggested by Ohio State economists (Roe, Haab, & Sohngen, 2004) used it to estimate Extension participants’ willingness to pay (WTP) for an event in the end of event evaluation:
    • “In these times of tightening budgets, support for programs such as this might be decreased such that an increase in registration fees is necessary. Would you have attended this event if the registration fee would had been $__X higher?”
  • For details on using the contingent valuation approach outlined above, see Morse, 2012.

Reporting and Communicating Impact Data Tips

Reporting Tips:

The manner in which the data is reported and recorded influences both its credibility with private and public funding sources, its value for program improvement and marketing, and its future value for additional research. Here are the tips related to reporting and recording the impact data.

1. Report the name of the local person providing the “but for” statement;
2. Require data submitted be documented;
3. Report the data in the year learned;
4. Ensure data submitted by teams; and
5. Use a simple commercial survey platform to minimize the cost of development.

Report Name of Local Person Providing “But For.”

The “but for” statement is much more credible when a name, community, and contact information are available. Having this information can be important if an Extension administrator is asked to give examples to illustrate the impacts or wants to use anecdotes to promote Extension.

Likewise it can be important if skeptics claim that the impacts are being inflated.

Ask permission to use their name before putting them in public or media outlets.

Require Data Submitted Be Documented.

Data is viewed as more credible when documentation of how it is collected is provided.

• If a legislature or funder asks how Extension helped to create 12,897 jobs in 2014, it will be possible to provide specific examples or even the full databases.
• Documentation provides an eventual means for CD specialists and educators to determine which methods are most practical and most accurate. In time certain protocols might become accepted as the preferred ones.
• Impacts can be traced over time.
• Documentation provides opportunities for follow-up contacts to develop more detailed anecdotes on successful programs or to do in-depth evaluation.

As shown in the examples, each data set names the communities involved. When the “but for” attribution principle is used to identify and document an impact, the name of the individual providing this statement and contact information should be recorded. We recommend these data not be made public unless the individuals making the statements provide a written release.

Report Impacts in the Year Learned.

Community development is a process that takes place over years, so it is unreasonable to expect that all impacts will take place in the year of the program. Due to the lag time in implementation and the fact that field educators often learn of major impacts a few months or even a year or more after they happen, how should impacts be handled? For example, if an impact happened in 2014 but is discovered in 2016, should you ask for the 2014 data to be adjusted or simply report it as 2016 data? We recommend you report the impacts in the year in which you learn about them. It is impractical to go back and revise earlier reports. Further the long-term picture is the same.

Ensure Data Submitted By Teams.

One concern some educators express is there could be double-counting if educators and specialists submitted individual results on projects that involve an Extension team. To avoid this possibility, the data should list the team members working on the results and the person responsible for submitting the data. In states that consider impact indicator data in annual personnel reviews, this type of data will indicate how the person contributed to the program’s team effort.

It is very useful if at least two members of the team independently develop individual impact estimates and then exchange notes on both the final results and how they were estimated. When Haskell and Morse calculated impacts of the pilot program, “Communicating the Public Value of Public Libraries,” the initial estimates for three items were different, leading to major discussions. The final estimates were improved and future estimates will not require this type of independent and collective work (Haskell, 2016).

Use a Simple Commercial Survey Platform.

The Northeast (NE) region is developing a simple online survey, using SurveyMonkey®, to collect impact indicator data. This technique will minimize the cost and simplify the process of aggregating the results. The results can be exported to Excel for additional analysis.

Tips on Communicating the Impacts:

If benefit/cost analysis is being done for a program, we need to know not only the impact of a program but also what percentage of the impact can be attributed to each of the partners working on the program. The goal associated with collecting impact indicators is more modest. The process simply documents (1) when Extension is a critical part of the team and (2) that the impact would not have occurred without Extension being part of the team. The major tips for communicating the impacts are:

1. Give credit to partners and report full impacts;
2. Include explicit acknowledgment of all partners in public reports;
3. Report the results on a per capita basis; and
4. Document impact calculations.

Give Credit to Partners and Report Full Impacts.

Many Extension programs include multiple partners, with Extension helping a community identify other relevant sources of help that are more proximate to the final result. Many other projects require active participation by firms or agencies participating in the program. For example, when a small business program teaches a new management skill, which the firm adopts, causing expansion opportunities supported by a loan from USDA Rural Development, and this leads to growth in the firm, what percentage of the credit goes to each?

The recommended Extension impact indicator approach does not attempt to divide the credit between the various actors. Rather, ask the question of whether each entity was a critical part of the team. If Extension was critical to the project moving ahead, then the recommended practice is to count the full impact and to note that this was a team effort. On the source notes, the others on the team should be listed.

Include Explicit Acknowledgment of All Partners in Public Reports.

In the final report, we encourage explicit acknowledgment that the impacts were created by a team effort if this is the case. We offer three guidelines for clarity:

• Be clear that without Extension’s involvement the impacts would not have happened;
• Do not report any impacts which would have occurred without Extension; and
• State the results in a clear, concise manner.

Hence, we recommend reporting using a format similar to that used in the North Central Region that adds the following language:

“The results shown here reflect impacts which would not have happened without Extension’s programs. In many cases, Extension recruited additional private and public sector entities to assist in delivering the program, without which, the impacts would not have been realized.”

In the table listing the impact indicators, we recommend not listing all the partners. However, in the database that documents the impacts, we recommend listing the partners. This will be helpful when used in media stories, award applications, or NIFA impact reports.

Report the Results in a Variety of Formats.

While the North Central format for reporting results (NCRCRD, 2015) is extremely useful for state, regional, and federal decision makers, county funders will want data relevant to the local area. We recommend that data collection be easily, yet authentically translatable for local, state, regional and national efforts where applicable. Just as methods for collecting impact indicators are evolving, so are the methods for reporting results.

Current popular and effective methods include:

• Provide succinct reports as electronic visuals with user- and print-friendly downloads (one page or less); this assures electronic and print versions for varied preferences;
• Provide data individualized to the request, e.g., annual county report, chancellor’s State of the University, NERCRD annual report.
• Use relevant infographics to illustrate the impact of only a few indicators rather than stressing all aspects of the program; (see additional examples of how these were used watch the “Shoestring Evaluation” webinar – French and Hancock, 2016).
• Include short case study narratives to strengthen the impact message. (See examples at Ahmed, 2014).
• Use photographs to illustrate real people in real programs in real communities, being sure to get the necessary photo releases at the time the picture is taken.

References

• ‘‘Agricultural Research, Extension, and Education Reform Act of 1998’’. (AREERA)
• Ahmed, Adeel. “Extension helps revitalize the City of Monticello through its Business Retention and Expansion Program.” Most Significant Change Story. University of Minnesota Extension. 2014.
• Bartik, Timothy J., and Richard D. Bingham. “Can economic development programs be evaluated?” Bingham, Richard D. and Mier, Robert “Dilemmas of urban economic development: issues in theory and practice.” Pp. 246-277. 1997.
• Bassano, Louis V. & McConnon, James C. “Strategic partnerships that strengthen extension’s community-based entrepreneurship programs: an example from Maine.” Journal of Extension. 49 (50) Article 5 FEA3. 2011.
• Borich, T., Chazdon, S., Leuci, M., & Loveridge, S. “Common Impact Indicators in Extension Community Development.” 2013.
• Public Value and Program Evaluation. Journal of Human Sciences and Extension 2(2).
• Chazdon, S, Horntvedt, J & Templin, E. (2016). From Knowledge to Action: Tips for Encouraging and Measuring Program-Related Behavior Change. Journal of Extension 54(2).
• Economic Report of the President. (2014). “Chapter 7. Evaluation as a tool for improving federal programs.” March 2014.
• Franz, Nancy K. “Advancing the public value movement: Sustaining Extension during tough times.” Journal of Extension 2011. [Online], 49(2) Article 2COM2.
• Franz, Nancy K. “Improving Extension programs: Putting public value stories and statements to work.” Journal of Extension, 2013. [Online], 51(3) Article 3TOT1.
• French, Charlie and Hancock, Casey. “Shoestring Evaluation: New Hampshire’s Experience Measuring CRED Impacts with Limited Resources and Expertise.” National CRD Indicators Webinar Series, February 2, 2016.
• French, Charles A., and Gagne, Michele. Ten years of community visioning in New Hampshire: the meaning of “success”. Community Development, 2010.  41, no.2 (2010): 223-239.
• French, Charles and Morse, George. Extension Stakeholder Engagement: An Exploration of Two Cases Exemplifying 21st Century Adaptions. Journal of Human Sciences and Extension 2015. [Online], 3(2).
• Hansen Kollock, D., Flage, L., Chazdon, S., Paine, N. & Higgins, L. (2012). Ripple Effect Mapping: A “Radiant” Way to Capture Program Impacts. Journal of Extension 50(5).
• Hanson, Andrew and Rohlin, Shawn. “The effect of location-based tax incentives on establishment location and employment across industry sectors.” Public Finance Review. 39. pp.195-225. 2010.
• Haskell, Jane E., and Louise Franck Cyr. “Community facilitator education: How training can lead to positive impacts at the community level.” Group Facilitation 11 (2011): 5.
• Haskell, J. E. & Morse, G. W. What is your library worth? Extension use public value workshops in communities. Journal of Extension 2015. [Online], 53(2) Article 2FEA1.
• Haskell, Jane E. (2016). Indicator Impact discussion, email and telephone. February.
• Headd, Brian. Redefining business success: distinguishing between closure and failure. Small Business Economics 21: 51-61. 2003.
• Kalambokidis, Laura. Identifying the public value in Extension programs. Department of Applied Economics, College of Agricultural, Food, and Environmental Sciences, University of Minnesota, 2003.
• Kalambokidis, Laura. “Spreading the word about Extension’s public value. Journal of Extension, 2011. [Online] 49(2). Article 2FEA1.
• Kalambokidis, Laura, Hinz, L., and Chazdon, S. “Using Economic Principles to Show How Extension Programs Create Public Value.” In Bryson, John M., Barbara C. Crosby, and Laura Bloomberg, eds. Creating Public Value in Practice: Advancing the Common Good in a Multi-sector, Shared-power, No-one-wholly-in-charge World. Vol. 194. CRC Press, 2015.
• Kelsey, Kathleen D. “Do workshops work for building evaluation capacity among cooperative extension service faculty.” Journal of Extension, 2008.  46, no. 6 6RIB4.
• Lerner, Richard M., Jacqueline V. Lerner, Jason B. Almerigi, Christina Theokas, Erin Phelps, Steinunn Gestsdottir, Sophie Naudeau et al. “Positive Youth Development, Participation in community youth development programs, and community contributions of fifth-grade adolescents findings from the first wave Of the 4-H study of Positive Youth Development.” The Journal of Early Adolescence 25, no. 1 (2005): 17-71.
• Lott, S. & Chazdon, S. (2009). The Effects of Leadership Development: Individual Outcomes and Community Impacts. University of Minnesota Extension.
• Loveridge, Scott. Keys to engaging faculty in service: Lessons from West Virginia’s community design team. Journal of Planning Education and Research. 21: 331-339. 2002.
• Loveridge, Scott & Elrod, Brent. “The Importance of Collecting Impact Numbers to Promote Community Development Work.” National CRD Indicators Webinar Series, April 25, 2016.
• MacAskill, William. Doing Good Better: Effective Altruism and a Radical New Way to Make a Difference. Faber & Faber, 2015.
• McDowell, George R. “The political economy of extension program design: Institutional maintenance issues in the organization and delivery of extension programs.” American Journal of Agricultural Economics 67, no. 4 (1985): 717-725.
• Morse, George. Cost recovery plan for extension programs extension. Economics Notes #2013-2. Applied Economics, University of Minnesota, 2013.
• Morse, George W. “Agreeing on practical extension user fees” Extension Economics Notes # 2012-4
• Morse, George W. “Survey Results from Two States Using the “But For Attribution Principle.” Unpublished paper. 2015.
• Morse, George W., and Farmer, Michael C. “Location and investment effects of a tax abatement program.” National Tax Journal (1986): 229-236.
• Morse, George W. and Ha, Inhyuck. How Successful Are Business Retention and Expansion Implementation Efforts. Economic Development Review. 1997. 15(1). pp. 8-13.
• Morse, George W., Markell, Jeanne, O’Brien, Philip, Ahmed, Adeel, Klein, Thomas, and Coyle, Larry. The Minnesota response: Cooperative Extension’s money and mission crisis. iUniverse, 2009.
• Nichols, A., Blake, S. M., Chazdon, S., Radhakrishna, R., (2015). From farm results demonstrations to Multistate impact designs: Cooperative extension navigates its way through evaluation pathways. Journal of Human Sciences and Extension, 3(2), 83-107.
• Nix, Kelly, Eades, Daniel, & Frost, Robin, “First Impressions: an effective approach for community improvement.” Journal of Extension, Volume 51, #3, Article #3RIB8.
• North Central Region CRED Program Leaders. “Community Resource and Economic Development (CRED) Programming Impacts Report” North Central Regional Center for Rural Development (NCRCRD), September 2010.
• North Central Regional Center for Rural Development (NCRCRD) 2015 “Impact Reports, 2014: Full Report for North Central Extension Community Development Programs.” 13 pages.
• Olson, Kent D., Skuza, Jennifer A., & Blinn, Charles R. “Extension educators view of scholarship and performance evaluation criteria.’ Journal of Extension. 45(4), Article 4RIB1. 2007
• Persky, Joseph, Daniel Felsenstein, and Wim Wiewel. “How do we know that” but for the incentives” the development would not have occurred.” Dilemmas of urban economic development (1997): 28-45.
• Ratkos, Judy, and Knollenberg, Lauren. “College transition study shows 4-H helps youth prepare for and succeed in college.” Journal of Extension, 2015. [Online] 53(4), Article 4FEA7.
• Roe, Brian, Haab, Timothy, & Sohngen, Brent. “The value of agricultural economics extension programming” Review of Agricultural Economics. 2004.
• Rossi, P.H., Lipsey, M.W. and Freeman, H.E., Evaluation: A systematic approach. 7th edition. 2004. .Thousand Oaks, CA: Sage. .
• SRDC (Southern Rural Development Center). Strengthening Rural Places, Southern Region Successes July 2014 – June 2015.
• Smith, M. F., and A. A. Straughn. “Impact evaluation: A challenge for Extension.” Journal of Extension 21, no. 9/10 (1983): 55-63.
• Starr, K. & L. Hattendorf. (2013). Rigorous Yet Realistic Measurement. Mulago Foundation, San Francisco, CA.
• Stup, Richard. “Program evaluation: use it to demonstrate value to potential clients.” Journal of Extension 41, no. 4 (2003).
• Tax Policy Center. “Tax Incentives for Economic Development: What are tax incentives for economic development”. Accessed September 10, 2015.
• Tuck, Brigid, Darger, Michael, & Ahmed, Adeel. Menahga Business Retention and Expansion Strategies Program SUMMARY REPORT. Extension Center for Community Vitality, University of Minnesota. August 2012.
• Taylor-Powell, E., Steele, S., & Douglah, M. (1996). Planning a program evaluation. University of Wisconsin-Extension-Cooperative Extension, Program Development and Evaluation Unit Web site.
• Welborn, Rachel and Jakes, Susan On-line webinar, with the NE Learning Circle on Impact Indicators, March 26, 2015.
• West’s Encyclopedia of American Law, edition 2. “But for” Rule. (n.d.)  Retrieved May 4 2016.
• Wang, S. L. “Cooperative Extension System: Trends and Economic Impacts on U.S. Agriculture.” Choices. Quarter 1. 2014.
• Workman, Jeffrey D., and Scott D. Scheer. “Evidence of impact: Examination of evaluation studies published in the Journal of Extension.” Journal of Extension50, no. 2 (2012).

Authors: George Morse, Charlie French, Scott Chazdon

Publication: Published by NERCRD   Date Published: October 17, 2016