Evaluation Challenges and Tips

NOTE: This toolkit was published by Building Changes in 2013 to help counties meet a 2014 state mandate that all counties have a coordinated entry system for clients entering the homeless system. It has not been updated since then and does not necessarily reflect current or best practice.

The challenges and tips detailed in this section are actual experiences and results of other counties’ evaluation processes. The purpose of this information is to lessen miscalculations and oversights that can delay the success of a coordinated entry system.

Challenges

• Evaluation is only as good as the data that goes into the data-collection systems; maintaining data entry and collection standards becomes more difficult in relation to the number of users entering data.
• Misplaced understandings and expectations of coordinated entry can negatively affect the evaluation process, which can interfere with recognizing successes and areas for improvement.
• Financial costs of resources to perform good evaluations, either through existing staff, additional staff, or through contracted evaluators; costs include developing effective surveys and questionnaires
• Creating institutional energy to incorporate performance-based contracting
• Collecting survey information from clients
• Maintaining partner-agency enthusiasm to continue to help evaluate and problem solve
• HMIS lacks the capacity to quantify services (outputs) and outcomes consistently
• Making sure outcomes are not outputs by ensuring that what is measured reflects a change in the access, assessment, and referral system, not just in performing services
• Designing survey questions that result in answers that provide a valid and reliable evaluation of coordinated entry and services

Tips

• Remember that the evaluation of a coordinated entry system must be based on the purpose of the system: to streamline access, assessment, and referral processes for housing and other services. Having unrealistic expectations of the system will result in ineffective evaluations. Stay focused on and evaluate the system’s purpose. Keep in mind that coordinated entry is one part of a broader group of systems strategies that together aim to reduce and end homelessness.
• That said, coordinated entry evaluation typically examines other systems, such as prevention, tailored services, rapid re-housing, or linkages to economic opportunity. It’s important to recognize that other changes to the overall system take place with coordinated entry—even on an informal level—and should be included in an evaluation process.
• Don’t wait until all evaluation data is available to begin evaluating the system. Within a few months, HMIS and parallel systems data can provide valuable insights to the intake, assessment, and referral process.
• Recognize that success will look different for the short term versus the long term. Short-term successes can be measured by vacancy rates in shelter/transitional programs, number of successful client-to-agency matches, fewer first-time homeless households due to prevention efforts, alignment of services, etc. Short-term goals can indicate whether the coordinated system is on track to work well with other systems (e.g., rapid re-housing) to reach the long-term goal of reducing and ending homelessness.
• Consistent provider network meetings can serve as an informal survey to understand providers’ experience with the system to address concerns regarding intake, assessment, and referrals.
• Communicate evaluation results to partners and the community. Make sure results are shared with direct service staff as well as agency leads/management. It’s important that partners are invested in the evaluation process.
• To improve the evaluation process, ask agency partners to identify a staff person for data-entry quality control; this will improve data accuracy and completeness.
• To meet the cost of performing system evaluations, consider reaching out to local universities to access students and university staff to develop surveys and questionnaires as well as collect and analyze data.

Next: Evaluation Resources