n the last edition of Public Policy and Practice, readers were introducted to the South Carolina Municipal Benchmarking Project (SCMBP)--a collaborative benchmarking effort in which municipalities in South Carolina are sharing performance measurement data in key services.¹ Since the project's inception in 1996, the purpose has been to provide a forum for South Carolina municipalities to compare easily performance statistics in order to ultimately make service delivery improvements. The project participants are completing their fifth year of data collection. Project staff has spent the last year analyzing the data and the impact that the data and project participation are having on the participating municipalities. This article, the second in a two-part series, presents the summary findings from the data analysis. It also provides examples of how the participating organizations have incorporated the performance data into their decision-making processes and made operational improvements. Finally, the authors explore other potential uses of the performance measurement data.  

DATA ANALYSIS

During the 2000 project year, project staff asked the service managers in each of the three initial service areas to predict cause-and-effect relationships among the data being collected. In the initial analysis of the data, there were a few interesting correlations.² However, given the small sample size and the fact that there were data that some cities were not able to provide, project staff wanted to be more certain that the results were not spurious or due to chance. Since a substantial amount of data have been collected over the past five years, project staff looked at the correlations between data for each year. The average of the correlations over the five-year period can serve as an indicator of the strength of the relationship between certain variables. Again, due to the small sample size and the fact that some data were not collected over the full five-year period, any claims of significant findings and assumptions regarding cities in general cannot be made. Yet some interesting findings occurred in each of the three service areas for cities participating in the project and are presented below.

Solid Waste

About a third of the cities participating in the project use what is known as a "task system" in their residential garbage collection. In a task system, workers are allowed to go home after they finish their routes even though they may not have worked a full eight-hour day. The workers, however, are paid for eight hours. Some of the solid waste managers believe that the use of the task system would result in greater productivity. During the five-year period in which data were collected, no significant differences were found between the productivity (defined as tons collected per FTE) of employees under the task system compared to the productivity of employees working a traditional eight-hour day. Although it does not seem to result in greater productivity, the task system most likely has a positive effect on employee satisfaction.

One variable that has consistently had an effect on productivity is the type of equipment used to collect residential garbage. Three cities use an automated system that allows one employee to collect residential garbage on a route. Other cities use semi-automated or manual equipment that require two or three–person crews. Over the five-year period, the cities using the automated systems collected an average of 3,463 tons of garbage per FTE, while the cities using other systems collected an average of 816 tons per FTE. Another way to define productivity is “cost per ton” of residential garbage collected. The cities using the automated systems spent an average of $29.40 to collect a ton of garbage over the five-year period. The other cities spent an average of $63.70 to collect the same amount during that time period.

Police

A common desire among police departments participating in this benchmarking project is to determine their departments' effectiveness. One difficulty that arises in making this determination is defining effectiveness. Certainly a lower crime rate is an effectiveness measure. Success also could be defined as a high crime clearance rate. The data on these two outcome measures yield some interesting findings.

The only variable project staff analyzed that had an effect on the crime clearance rate was population. It was found that the smaller the city’s population, the higher the clearance rate.  Police officials are not surprised at this result. They surmise that in smaller communities, citizens (and the police officers) tend to know a larger percentage of their fellow citizens and thus are more aware of activities, both legal and illegal, in their communities. Variables such as number of sworn officers, the number of cases assigned to investigators, and the number of neighborhood watch programs and crime prevention programs were not related to crime clearance rate in this analysis.

When looking at crime rate (crimes reported per 1,000 citizens), it was found that a strong positive relationship between the number of sworn officers and the crime rate. In other words, the more officers a community employed, the more crimes were reported. Explaining this result leads into the "chicken/egg quandary.” One possible explanation of this finding could simply be that an increased number of officers increase the probability of crime detection. Or, hiring more officers could be a community’s reaction to an increase in the crime rate.

Project staff also found a modest positive relationship between the number of crime prevention programs conducted by the police department and the crime rate. Again, is this relationship due to citizens reporting more crime as a result of attending a crime prevention program, or is the increase in crime prevention programs in response to an increased crime rate?

Fire  

Fire departments invest a considerable amount of time and money in developing and conducting fire prevention programs. According to SCMBP data, this kind of investment seems to pay off. Further, there are modest inverse relationships between both the number of education programs and the number of program attendees when compared with estimated fire dollar loss. In other words, when the number of programs and attendees increases, the estimated amount of fire dollar loss decreases.

As was discovered in the analysis of the police data, it is sometimes difficult to tell which variable is the cause and which is the effect. In the analysis of the fire data, it was found that a strong positive relationship existed between the number of commercial fires and the number of fire code inspections. One might surmise that fire code inspections should lead to a reduction in the number of commercial fires. Since this is not the case, according to SCMBP data, a more likely explanation is that an increase in commercial fires has caused city fire departments to step up their code inspection efforts. After reviewing this data with the fire service committee, fire chiefs agree that as a result of a particular type of commercial fire, their departments may increase inspections in an effort to uncover certain unsafe practices or conditions. 

OVERALL EFFECTIVENESS MODEL

During the course of this project, several city managers/administrators have approached project staff about developing a measure of overall effectiveness. This overall measure likely would include the following critical outcome measures:

However, the service area managers have balked at developing an overall measure. They believe that some outcome measures are more important than others, too many of the outcome measures are beyond their control and that the evaluation of a department’s performance cannot be reduced to looking at one number.

Project staff has experimented with a number of different algorithms in attempting to develop this measure of overall effectiveness. The difficulty in developing such a measure is akin to the dilemma faced by the committee that decides which factors are considered and the weight they are assigned in deciding the NCAA Bowl Championship football rankings. SCMBP project staff is currently working on a model that would include standard critical outcome measures, but would allow for each city to determine how much each of the outcome measures is weighted in determining the overall effectiveness measure.

PROJECT IMPACT AND RESULTS

By benchmarking, local governments are better equipped with the necessary information to answer the questions of how efficiently and effectively they are providing services to the community. Through sharing performance measurement statistics, organizations are able to learn the best practices for service delivery. The crucial element of performance benchmarking is applying these best practices in an effort to make improvements. Project staff has focused the last year on measuring the results of the SCMBP. Is the project accomplishing its purpose of allowing jurisdictions to easily compare performance statistics to make service delivery improvements? In an effort to answer this key question, project staff has polled the participants to gauge what impact the SCMBP is having on department service delivery and the organizations at-large.³ Specifically, those surveyed were asked what changes have resulted from the performance data and their involvement in the project. Responses are summarized below by type or category of utilization. The objective of this section is to share the stories of the progressive organizations that have incorporated their experiences in the SCMBP and analysis of the performance data into their decision-making processes.   

Many local governments use retreats as an opportunity for elected officials and executive staff to set the strategic direction for the organization, to identify priorities, and to develop action plans. Retreat participants use data typically from sources such as the budget and citizen surveys to help them make strategic decisions. SCMBP data also are valuable tools in the retreat setting.   

One of the participating municipalities was interested in incorporating the performance results from the SCMBP in its annual planning retreat. The purpose of the retreat was for the city to determine its priorities or critical services and to identify council expectations of these services. As part of the structured review process, department managers were asked to explain to city council how well their departments were performing. The police, fire, and sanitation departments were asked to rely on data from the SCMBP to answer these questions.

After reviewing the data presented in Figure 1 (below), one issue that surfaced as a concern for this council was first responder calls that the fire department provided to the community.⁴ The city's number of first responder calls (City B in Figure 1) was nearly three times the average of all participants. The council expressed concern about the costs associated with responding to all medical calls and if the responses were warranted and necessary. For several years the city's firefighters responded as a first responder unit to every medical call within the city limits that was dispatched by the county dispatch center. There was some concern regarding the county's call screening methodology (i.e., how they defined medical calls needing first responder).  

FIGURE 1

CITY	Population Category*	Number of First Responder calls per 1,000 service population
City A	1	67.80
City B	1	115.99
City C	2	8.42
City D	2	0.78
City E	2	25.22
City F	2	41.08
City G	3	26.57
City H	3	50.34
City I	3	41.06
City J	3	38.40
City K	3	49.08
City L	3	8.24
City M	3	26.51
City N	3	12.56
AVERAGE**		36.57

*The following population ranges are used as the population categories for the SCMBP: 1 = 0-15,000; 2 = 15,001-30,000; and 3 = 30,001 and above.

**The average data represents the average for all participating cities.

Firefighters provide medical assistance during a fire response. Photo by Robert Busbee.

As a result of the discussion that transpired during the annual planning retreat, city staff worked with the 19 surrounding volunteer fire departments and the county EMS to develop a standard protocol for the entire county which consists of a list of 12 medical calls that generate a first responder response in the county. The city implemented this new protocol in July 2002. The city administrator expects the city to realize a cost savings over the next year by not responding to as many medical calls. When asked what role the data from the SCMBP had played in this change, the city administrator responded:

Another municipality included benchmarking data in a review of municipal departments during a council retreat. Project staff prepared a comparative summary from the 2000 SCMBP data for the town and the eight other cities in the same population category. The data showed that the city's efforts in preventing crime and fire prevention programs seemed to result in a lower crime rate and a lower estimated fire dollar loss.⁵     

As has been mentioned previously in this article, the productivity of the fully automated garbage truck is noteworthy. Council members were interested in why the city's public works department had not looked into purchasing one of these vehicles. The city's public works director explained that, while the truck would work fine in some neighborhoods, it would not be feasible to use in many neighborhoods due to the design of the streets and on-street parking practices. Council members agreed that residential developers should consider how the design of their neighborhoods affects the delivery of services. Council also encouraged the city's planning department to involve other town departments in the review of residential development plans to see how the design of a new community may affect the ability of a department to deliver services.  

Local governments, similar to state agencies, are facing increasingly tight budgets while experiencing greater service demands. Given the push for property tax relief over the last decade, user fees and charges have become the fastest growing revenue sources for local governments in South Carolina. These revenue sources place the financing responsibility on the users who create the service need and who may not reside within the jurisdiction limits (The South Carolina Local Government Steering Committee, 2000, pp. 39-40). In order to establish or increase existing user fees and charges fairly, local governments must first be able to determine the cost for providing the service. The second obstacle for passing user fee/charge ordinances is the political process (Rivenbark, 2000, p. 3). Elected officials will typically accept the need for fees more readily, if they have data illustrating the costs incurred by the organization for service delivery.            

The costing methodology of the SCMBP provides participants with the ability to determine the full cost for providing services. One jurisdiction, for example, has used the costing methodology of the project and analysis of the fees charged by the other participants to justify a fee increase for residential and commercial garbage fees. The municipal staff and elected officials of this jurisdiction realized the disparity between the city's fee structure and the cost for providing the collection services, and as a result were able to make more informed decisions about user fees. The city's sanitation manager explained that charging appropriate fees for these collection services takes a portion of the financing burden away from the general fund.

Further, several participating cities have purchased a costing software program to assist them in gathering the direct and indirect costs required for each of the services included in the SCMBP. The finance departments of these cities have the ability to determine the cost for providing any municipal service. One of the participating cities has used the costing software to determine the total cost for its park and recreation services.⁶ As a result, the city manager was able to present accurate financial data to the city council on the costs for providing parks and recreation services and programs to the community. In analyzing this information and the fee policy, the council voted to increase the non-city resident fees for these programs and services. Though this city's analysis transpired before this service area was added as a new service to the SCMBP, the ability to determine the cost for these services and programs is an indirect result and hence a benefit of the city's participation in the project.    

3. Identifying and implementing the best practices

Having a desire to learn from other organizations is at the heart of benchmarking. If public managers believe that their departments are performing at the most efficient level and lack the need for improvement, benchmarking will be a wasteful and somewhat painful endeavor. Listed below (Figure 2) is the SCMBP's benchmarking model. Project staff facilitates the process steps involved in the model through committee meeting discussions and the analysis of the benchmarking data.

FIGURE 2
Benchmarking Model

Understand the data that you submit and ask questions when you need clarification. Be committed to learn from other departments before dismissing the differences. Select the key indicators or sub-service areas that you will analyze. Determine the factors that you will use in selecting benchmarking partners, realizing that we can also learn from the processes of those that we do not select. Analyze your department's performance. Review the performance results of your benchmarking partners. Conduct a gap analysis: a. Are there significant differences?  b. If so, what are the differences?  c. What are the process steps or other factors that could be causing these differences? If these departments are performing at a higher level, are there operational changes that you could borrow, adapt and/or adopt for your department? Review the results of the other participants that are performing well and repeat steps 7 and 8.  Implement these changes and monitor the results.

City managers/administrators and department managers who are committed to analyzing their own performance welcome the opportunity to share ideas and lessons learned with their colleagues. Examples are presented below illustrating how the participants have utilized the performance results and made improvements to their service delivery.            

As previously described, one of the findings of the sanitation data of the SCMBP is the efficiency that a fully automated collection methodology for residential garbage has on an operation. This has also proven to be true in a similar project called the North Carolina Local Government Performance Measurement Project (Rivenbark, 2000, p. 3). Unlike the non-automated and semi-automated garbage trucks, the fully automated "one-arm bandit" trucks are ideally operated by one employee who, without leaving the cab of the truck, drives the truck and operates the hydraulic arm to pick up the roll cart. The cost per ton, cost per collection point, and tonnage collected per FTE are the three measures that illustrate the efficiency of this collection methodology and are presented in the figure below.

FIGURE 3
Residential Garbage Collection

CITY	Population Category*	Cost per Ton	Cost per Collection Point	Number of Tons Collected per FTE
City A	1	$74	$83	1,120
City B	1	$42	$117	1,454
City C	1	$132	$98	816
City D	1	$71	$77	1,117
City E	1	$100	$89	889
City F	1	$57	$40	2,142
City G	2	$224	$120	346
City H	2	$91	$74	910
City I	2	$66	$72	794
City J	2	$114	$84	642
City K	3	$39	$41	2,659
City L	3	$66	$49	1,114
City M	3	$39	$36	1,147
City N	3	$50	$45	831
City O	3	$23	$50	1,269
City P	3	$31	$27	1,327
City Q	3	$115	$83	729
City R	3	$60	$70	867
City S	3	$34	$24	967
AVERAGE**		$75	$67	1,113

*The following population ranges are used as the population categories for the SCMBP: 1 = 0-15,000; 2 = 15,001-30,000; and 3 = 30,001 and above.

**The average data represents the average for all participating cities.

The three cities noted in bold text in Figure 3 use fully automated garbage trucks with a one-man crew for residential garbage collection. City G's collection methodology consists of backyard pick-up with three- to four-man crews. In analyzing these results, one can conclude that the fully automated, one-man crew methodology is a more efficient collection method and has been identified as one of the best practices for residential garbage collection. Additionally, a benefit of this best practice that is not highlighted in the SCMBP is the potential reduction in Workers' Compensation claims (Rivenbark, 2000, p. 3).

Fully-automated one-arm bandit truck collects residential garbage. Photo by Marcie Leaphart.

Have the project's findings facilitated any change in the participants' service delivery methods? One city, for instance, began phasing in the fully automated collection system in late 2001. The first phase of implementation was established on a residential route that had the newest roll carts. For complete implementation to occur, the city will address the age of the roll carts in the rest of the community and change its ordinance to require that all garbage be containerized. A crucial element identified by the sanitation director is that the city's management understands the capital investment that is involved in changing to the fully automated system and the long-term gains for the organization.⁷ When asked to comment on the change, the sanitation director stated, "After realizing the lack of overhead in cities that use the fully automated trucks, why would anyone not consider automation?"

This director poses an interesting question. City G, as previously described, collects residential garbage using the least automated collection methodology. Given the efficiencies that are realized by using the fully automated equipment, why would this city not change? One compelling response is that the political environment in which local governments operate must not be forgotten. Citizens that reside in City G may have expectations that the city provide the level of service they receive with backyard collection, regardless of the cost. One of the lessons that project staff has learned is, although a best practice is identified, it must be analyzed to determine if it is the "best fit" for a community. Three of the participating cities' departments that do not use the fully automated methodology are analyzing the performance results and operating details of the system to determine if it is best for their communities. 

4. Additional project uses and results 

Another change being considered by participants in their police and fire departments is the adoption of false alarm ordinances. With the increased use of home security systems, police and fire departments are responding to more false alarms than ever before. Each time a unit responds to a false alarm, the municipality incurs an expense associated with the response. Based on the false alarm data of the SCMBP, a police department and a fire department are actively pursuing false alarm ordinances for their jurisdictions. These ordinances will require businesses and residents to pay a fee for any false alarm after a certain number of occurrences. The purpose of these ordinances is to deter businesses and residents from having a large number of false alarms and to recover a percentage, albeit small, of the costs associated with responding to the alarms.             

As a result of attempting to collect the necessary data for the SCMBP, several departments realized that they did not have an adequate data tracking system. In order to provide these data, these departments have since implemented data tracking systems. In addition to ensuring the accuracy and comparability of the data, these tracking information systems also benefit the municipalities by providing managers with a means of internal reporting of department performance results to elected officials and citizens.            

Another benefit of the SCMBP is the use of the standard set of performance measures as a management tool. Several participating organizations are using the historical reports from the SCMBP to track their own performance over time in order to identify trends. If a trend analysis indicates a problem in a particular service area, the manager can take steps to address the issue. This allows managers to detect service delivery needs and to see if improvement attempts have been successful and have had a positive impact on service delivery.  

POTENTIAL USES

In addition to the uses of the data that have been previously described, our research indicates that there are more potential ways in which organizations can use performance measurement data.

Citizens deserve to know if their tax dollars are being used wisely in providing local government services. As representatives of the citizenry, elected officials have a responsibility to ensure that services are operating in a cost efficient manner as well as meeting stated objectives. One method for answering this need for accountability is program performance measures (Tigue and Strachota, 1995, p. v). Including performance measures in budget documents is a common practice of local governments. These measures should strive to illustrate the degree to which departments are delivering services and programs in an effective and efficient manner (i.e., focus more on program outcomes than outputs).

As municipalities review their operations and look for cost-efficiency opportunities, the managers and elected officials should consider the level of service that they are providing by asking the following questions: Is the current service level necessary? Is it cost efficient? Is it meeting the needs of our citizens? Through the collective benchmarking process of the SCMBP, answers to these questions can be found. When asked how the quality data has been incorporated into their decision-making process, two of the city managers responded that they have included results from the SCMBP's citizen satisfaction surveys to justify budget increases to maintain the current level of service.⁸

Several participating cities have reported using performance data from the SCMBP in their budgeting process. The uses range from simply including the department's performance results compared to the average of the participating cities in the budget document to using the performance data as justification for budget requests. In analyzing their performance results in comparison with other city departments, the department managers are better prepared to show departmental effectiveness or a gap between service level and needs. If the comparative benchmarking statistics illustrate that the department is providing services and programs at an efficient level, the department managers have used the data to justify maintaining the current level of funding. If the same data indicate that the service is not being provided in the most efficient manner, it can be used as justification for more resources.

Additionally, performance measurement can be used as part of an organization’s formal strategic planning process. Performance measurement data may be employed to evaluate the effectiveness of different strategies, to gauge how well city departments implement programs, and to evaluate whether goals and objectives are being met. In Austin, Texas, for example, the fire department uses a benchmark of a 3 1/2-minute response time for pumper trucks to plan the location of future fire stations (Epstein and Campbell, 2000, p. 14). In Winston-Salem, North Carolina, data from a citizen survey also are being used to identify gaps in city services and service improvement opportunities (Bernstein, 2000, p. 14). Although project staff is not aware of any participants in the SCMBP that have incorporated the data into a formal strategic planning process, examples of several cities using the data for making strategic decisions in retreat settings have been cited earlier in this article.

3. Performance evaluations

The benchmarking data also can be used as a part of the performance appraisal process of public managers. In Sunnyvale, California, for instance, performance measures are used as part of the management contract goals, which depending on goal accomplishment, can lead to a 10% increase or 5% reduction in pay (Chan, 1994, pp. M3-6). In Winston-Salem, North Carolina, reports of annual performance measures also are part of the department head performance reviews. The Winston-Salem Board of Aldermen use performance measure data in its annual review of the City Manager (Bernstein, 2000, p. 17). At this time, project staff is not aware of the SCMBP data being used in performance evaluations of department heads or city managers/administrators. Project staff encourages those city managers/administrators and department managers who have developed targets or goals as a result of the SCMBP performance data to incorporate these goals into employee performance appraisals. What better way to help employees see how their work fits into the mission of the department and overall organization?

CONCLUSION

One question project participants ask is, “How can an organization objectively evaluate its performance unless it has a reference point?” As affirmed by some of the participating managers, the SCMBP is a valuable exercise that can uncover opportunities for improvement and illustrate effective service delivery methods. However, not all participating cities have taken advantage of the wealth of data and analysis that has resulted from the project. This article has attempted to share examples of the ways that participating cities have used the data in hopes that other jurisdictions will see the value of incorporating comparative performance measures in planning and decision making.

So, has the South Carolina Municipal Benchmarking Project accomplished its purpose? Project managers believe there are still lessons to be learned and best practices to be shared from this benchmarking initiative. Once the participants have an opportunity to review the 2001 benchmarking findings and analysis, the SCMBP Steering Committee will meet to answer this question and determine the strategic direction for the project. 

The authors would like to acknowledge the research assistance that Eunice Low, a graduate assistant with the Institute for Public Service and Policy Research, provided for this article. We would like to thank those participating city managers/administrators and department managers of the SCMBP who responded to Benchmarking Results Questionnaire and provided insightful feedback.

NOTES

¹For a copy of The South Carolina Municipal Benchmarking Project – Part 1: An Endeavor Toward Excellence at the Local Level, please visit the previous edition of Public Policy & Practice: http: www.iopa.sc.edu/eJournal/Benchmarking.htm.

²Two statistical techniques were used, depending on the type of variable being analyzed.  A Chi-square test was used where one of the variables was nominal data (e.g., type of equipment used) and a Pearson Product Moment Correlation was used for other data.

³A questionnaire asking about their city’s participation and the benefits of the project data was sent to all of the city managers/administrators. Department managers received a questionnaire that focused on the benefits and changes that these participants have realized at the department level.

⁴First responder service is defined as sending trained personnel to medical emergencies who can arrive before an ambulance. The purpose is to stabilize patients' medical conditions by quickly controlling bleeding, restoring breathing and circulation, and taking other actions to keep patients alive and to make them as comfortable as possible until the arrival of an ambulance. Firefighters can provide an effective first responder service, because fire stations are strategically located throughout most communities and are therefore often closer to medical incidents than ambulances, and firefighters are trained in medical procedures as a part of their job.

⁵There also is a relationship between these variables over the five-year project period.  The first year this city participated in the project was 2000.

⁶At the time of the city's cost analysis, parks and recreation was not a service included in the SCMBP. This service was added a year later to the project in 2001.

⁷The public works director from one of the cities that currently uses the fully automated system has identified the following issues as ones that must be taken into account when considering how to implement the system: type of roll cart (i.e. durability and capacity), location of roll cart (must be placed at least 5 feet from any obstruction) and higher level of equipment maintenance.

⁸Each of the participating jurisdictions has a citizen satisfaction survey conducted on a biennial basis. This random telephone survey, conducted by the Institute for Public Service and Policy Research's Survey Research Lab, polls city residents about their level of satisfaction with the city services that are included in the SCMBP.  

REFERENCES

Bernstein, D. (2000). GASB SEA research case study: City of Winston-Salem, North Carolina. Focusing on government efficiency and public confidence. Winston-Salem, NC: Performance Measurement for Government.

Chan, A. (1994). Managing a government like a business: The Sunnyvale system. Government Finance Review, 7-11.  

Epstein, P. & Campbell, W. (2000). GASB SEA research case study: City of Austin. Austin, TX: Performance Measurement for Government.

Rivenbark, B. (2000, October). The art of using performance and cost data. Institute of Government Bulletin.  (Publication No. 31). Chapel Hill, NC: The University of North Carolina at Chapel Hill.

The Local Government Funding System Steering Committee (2000). The South Carolina local government funding system: Findings and recommendations. Columbia, SC: Center for Governance.

Tigue, P. & Strachota, D. (1995). The use of performance measures in city and county budgets. Chicago, IL: Government Finance Officers Association.

ABOUT THE AUTHORS

Anna Berger, B.A., M.P.A., is the project director for the South Carolina Municipal Benchmarking Project. Her primary responsibilities with the Center for Governmental Services include assisting government organizations with performance benchmarking, reengineering processes, and strategic planning and performance measurement. Anna Berger can be contacted at Aberger@sc.edu.

Bill Tomes, B.S., MPER, is a senior research associate with the Center for Governmental Services. His research interests include employee recruitment and selection, measuring human performance, and personality and work performance.  He directs technical assistance projects for the Center and serves as a faculty member for the Center's various professional development programs. Bill Tomes can be contacted at Bill-Tomes@sc.edu.