0 Developing An Evaluation Plan For TB Control Programs Division.ppt

0Developing An Developing An Evaluation PlanEvaluation PlanFor TB Control ProgramsFor TB Control ProgramsDivision of Tuberculosis EliminationDivision of Tuberculosis EliminationNational Center for HIV,STD,and TB PreventionNational Center for HIV,STD,and TB PreventionCenters for Disease Control and PreventionCenters for Disease Control and Prevention1Developing An Developing An Evaluation PlanEvaluation PlanFor TB Control ProgramsFor TB Control ProgramsReference:A Guide to Developing an Evaluation Plan2Why Develop an Evaluation Plan?n nProvides a cohesive approach to conducting Provides a cohesive approach to conducting evaluation and using the resultsevaluation and using the resultsn nGuides evaluation activitiesGuides evaluation activitiesvvExplains what,when,how,why,whoExplains what,when,how,why,whon nDocuments the evaluation process for all Documents the evaluation process for all stakeholdersstakeholdersn nEnsures implementation fidelityEnsures implementation fidelity3Guide to Developing An Evaluation Plann nDocument referenced throughout presentationn nProvides a template and instructions to help TB program staff develop an evaluation plann nSteps to evaluation are explained in detail n nCompleting sections and tables will result in an evaluation plan4The CDC Program Evaluation Framework5The CDC Program Evaluation Frameworkn nSystematic method for evaluationvvBased on research and experienceBased on research and experiencevvFlexible and adaptableFlexible and adaptablen nPromotes a participatory approachn nFocuses on using evaluation findings6Sections of an Evaluation Plann nIntroductionIntroductionn nStakeholder Assessment Stakeholder Assessment vvStep 1:Engage StakeholdersStep 1:Engage Stakeholdersn nBackground and Description of the TB Background and Description of the TB Program and Program Logic ModelProgram and Program Logic ModelvvStep 2:Describe the ProgramStep 2:Describe the Programn nFocus of the EvaluationFocus of the EvaluationvvStep 3:Focus the Evaluation DesignStep 3:Focus the Evaluation Design7Sections of an Evaluation Plann nGathering Credible Evidence:Data Collection vvStep 4:Gather Credible EvidenceStep 4:Gather Credible Evidencen nJustifying Conclusions:Analysis and InterpretationvvStep 5:Justify ConclusionsStep 5:Justify Conclusionsn nEnsuring Use and Sharing Lessons Learned:Reporting and DisseminationvvStep 6:Ensure Use and Share Lessons LearnedStep 6:Ensure Use and Share Lessons Learned8Introduction An introduction provides background information,identifies the purpose of the evaluation,and provides a roadmap of the plan.n nEvaluation GoalEvaluation GoalvvWhat is the purpose of the evaluation?What is the purpose of the evaluation?n nEvaluation TeamEvaluation TeamvvWho is your evaluation coordinator?Who is your evaluation coordinator?vvWho are the members of your evaluation team?Who are the members of your evaluation team?Reference:Table 1 in the Evaluation Plan GuideReference:Table 1 in the Evaluation Plan Guide9Stakeholder AssessmentStakeholders are individuals with vested interests in Stakeholders are individuals with vested interests in the success of the TB program.Involving stakeholders the success of the TB program.Involving stakeholders increases the credibility of the evaluation and ensures increases the credibility of the evaluation and ensures that findings are used as intended.that findings are used as intended.n nWho are the stakeholders in your TB program?Who are the stakeholders in your TB program?n nWhat are their interests in the evaluation?What are their interests in the evaluation?n nWhat role do they play in the evaluation?What role do they play in the evaluation?n nHow do you plan to engage the stakeholders?How do you plan to engage the stakeholders?Reference:Table 2 in the Evaluation Plan GuideReference:Table 2 in the Evaluation Plan Guide10Background and Description of the TB ProgramThe program description ensures that stakeholders have a shared understanding of the program and identifies any unfounded assumptions and gaps.n nNeedvvWhat problem does your program address?What problem does your program address?vvWhat are the causes and consequences of the problem?What are the causes and consequences of the problem?vvWhat is the magnitude of the problem?What is the magnitude of the problem?vvWhat changes or trends impact the problem?What changes or trends impact the problem?11Background and Descriptionn nContextContextvvWhat are environmental factors that affect your program?What are environmental factors that affect your program?n nTarget Population Target Population vvDoes your program target the TB concerns of one population?Does your program target the TB concerns of one population?n nProgram ObjectivesProgram ObjectivesvvWhat objectives have been set for your program?What objectives have been set for your program?n nStage of DevelopmentStage of DevelopmentvvIs this a new initiative or is it well established?Is this a new initiative or is it well established?12Background and Descriptionn nResourcesResourcesvvWhat resources are available to conduct the program activities?What resources are available to conduct the program activities?n nActivitiesActivitiesvvWhat are program staff doing to accomplish program objectives?What are program staff doing to accomplish program objectives?n nOutputs Outputs vvWhat are the direct and immediate results of program activities What are the direct and immediate results of program activities(materials produced,services delivered,etc.)?(materials produced,services delivered,etc.)?n nOutcomesOutcomesvvWhat are the intended effects of the program activities?What are the intended effects of the program activities?Reference:Table 3 in the Evaluation Plan GuideReference:Table 3 in the Evaluation Plan Guide13Program Logic ModelA logic model is a graphic depiction of the program description.n nArrows describe the links between resources,activities,outputs and outcomes n nA logic modelvvProvides a sense of scope of your programProvides a sense of scope of your programvvEnsures that systematic decisions are made about Ensures that systematic decisions are made about what is to be measuredwhat is to be measuredvvHelps to identify and organize indicatorsHelps to identify and organize indicators14Program Logic Model15Contact InvestigationContact InvestigationGoal:Prevent TB among contacts to cases(by finding and testing contacts for Goal:Prevent TB among contacts to cases(by finding and testing contacts for TB and LTBI,and then treating infected contacts to completion).TB and LTBI,and then treating infected contacts to completion).A AInputsInputsB BActivitiesActivitiesC CShort-termShort-termOutcomesOutcomesD DIntermediateIntermediateOutcomesOutcomesE ELong-term Long-term OutcomesOutcomes1 1Adequate infrastructureAdequate infrastructurei i Qualified,trained and Qualified,trained and and motivated staff and motivated staffii Community and congregateii Community and congregate setting partnerships setting partnershipsiiiiii Policies,procedures,and Policies,procedures,and guidelines guidelines iv Ongoing data collection,iv Ongoing data collection,monitoring,and reporting monitoring,and reporting systems systemsv Adequate physical,v Adequate physical,diagnostic,and treatment diagnostic,and treatment resources resourcesvi Linkages between vi Linkages between jurisdictionsjurisdictionsviivii Adequate data collection Adequate data collection tools tools viii Partnership with privateviii Partnership with private providers providers 1 1a aa aInterview/reinterview casesInterview/reinterview casesi Build rapporti Build rapportii Provide educationii Provide educationiii Obtain informationiii Obtain information about source case and about source case and contacts contacts b bLocate and evaluate Locate and evaluate contact:contact:i Follow-upi Follow-upii Educationii Educationiii Examination&testing*iii Examination&testing*c cOfferOffer treatmenttreatmentd dTreat contact case Treat contact case managementmanagement(DOT/DOPT/incentives(DOT/DOPT/incentives 2 2a aComprehensiveComprehensiveinterview toolinterview toolb bStaff trained in interviewStaff trained in interviewtechniquestechniques3 3a aLegal mandate to collectLegal mandate to collectcontact information fromcontact information fromcongregate settingscongregate settings2 2a aReportingReporting3 3a aMonitor:Monitor:i Data collectioni Data collectionii Data managementii Data managementii Data analysisii Data analysisiv Data disseminationiv Data dissemination a a4 4Conduct periodic reviewConduct periodic reviewof cases/contacts and progressof cases/contacts and progresstoward contact treatment goalstoward contact treatment goalsa aCases identify contactsCases identify contacts1 12 2a aContactsContacts educatededucated3 3a aContacts evaluatedContacts evaluated4 4a aContacts followed upContacts followed up5 5a aContacts start treatmentContacts start treatment6 6a aEvidence-based decisionsEvidence-based decisionsabout continuation or about continuation or termination of contact termination of contact investigation investigation 2 2a a1 1a aContacts completeContacts completeappropriate treatment for appropriate treatment for active TB or LTBIactive TB or LTBI 1 1a aActive TB cured in Active TB cured in contactscontactsb b TB(prevented)in TB(prevented)in contacts with LTBI contacts with LTBI 2 2a aReduced incidence Reduced incidence and prevalence of TBand prevalence of TB3 3a aTB eliminatedTB eliminatedImproved approaches forImproved approaches forcontact investigationcontact investigation16Sample Logic ModelSample Logic ModelProject Description of TB Support ProgramProject Description of TB Support ProgramResourcesResourcesActivitiesActivitiesOutputsOutputsOutcomesOutcomesInitialInitialSubsequentSubsequentShort-termShort-termIntermediateIntermediateLong-termLong-termLHAs ofSalvadoran communityTB StaffFunding Community based organizations serving Salvadoran communityHiring LHAsLHAs trainingLHAs hired&trained Provide Spanish speaking and culturally competent services for Salvadoran communityIncrease utilization of TB services by Salvadoran CommunityEliminate TB in Salvadoran communityCommunity outreachEducationTB outreach and education conductedIncrease TB knowledge in Salvadoran communityCounseling&support providedTrust built between health care providers and the Salvadoran communityTB screening/testingTesting done and referrals madeEarly TB and LTBI detection and interventions Reduce TB transmissionPatients offered treatmentDevelop treatment plansDOT visitsTreatment plansDOT administeredPatients accept treatment for TB and LTBI;Patients adhere to treatmentComplete treatmentIncrease completion of therapy rate(reduce hospital admissions for TB among Salvadorans)17Focus of the EvaluationSince you cannot feasibly evaluate everything,you must focus the evaluation by prioritizing and selecting evaluation questions.n nStakeholder NeedsvvWho will use the evaluation findings?Who will use the evaluation findings?vvHow will the findings be used?How will the findings be used?vvWhat do stakeholders need to learn/know from the What do stakeholders need to learn/know from the evaluation?evaluation?18Focus of the Evaluationn nProcess EvaluationProcess EvaluationvvWhat resources were required?What resources were required?vvWhat program activities were accomplished?What program activities were accomplished?vvWere they implemented as planned?Were they implemented as planned?n nOutcome EvaluationOutcome EvaluationvvIs the program producing the intended outcomes?Is the program producing the intended outcomes?vvIs there progress toward program objectives and goals?Is there progress toward program objectives and goals?19Focus of the Evaluationn nEvaluation QuestionsvvBased on the needs of your stakeholdersvvAddress process and outcome n nAssess Your QuestionsvvFeasible to collectvvProvide accurate results20Focus of the Evaluationn nKey Issues in Evaluation DesignvvWill you have a comparison or control group?Will you have a comparison or control group?vvWhen will you collect data?When will you collect data?vvWill the data be collected retrospectively or Will the data be collected retrospectively or prospectively?prospectively?vvWhat type of data do you need?What type of data do you need?vvWhat data do you have already?21Focus of the Evaluationn nOther Design ConsiderationsvvStandards for“good”evaluationStandards for“good”evaluationvvTimelinessTimelinessvvStage of developmentStage of developmentvvData neededData neededn nStrengthen Your DesignvvMix methods whenever possibleMix methods whenever possiblevvUse repeated measuresUse repeated measuresvvTriangulateTriangulate22Gathering Credible Evidence:Data CollectionIdentify indicators,standards,and data sources Identify indicators,standards,and data sources to address evaluation questions.to address evaluation questions.n nIndicatorsIndicatorsvvVisible,measurable signs of program performanceVisible,measurable signs of program performancevvReflect program objectives,logic model and evaluation Reflect program objectives,logic model and evaluation questionsquestionsn nProgram Benchmarks and TargetsProgram Benchmarks and TargetsvvReasonable expectations of program performanceReasonable expectations of program performancevvBenchmarks against which to measure performanceBenchmarks against which to measure performanceReference:Table 4 in your Evaluation Plan GuideReference:Table 4 in your Evaluation Plan Guide23EvaluationEvaluationQuestionQuestionProcess and Outcome IndicatorsProcess and Outcome IndicatorsProgram BenchmarksProgram BenchmarksHave Spanish-speaking persons been treated appropriately for LTBI or TB?Number of Spanish-speaking persons treated by clinic for TB&LTBI between Jan June.Increase in the number of Spanish-speaking patientsNumber of times clinical treatment standards are met for Spanish-speaking patientsClinical standards are met 100%of timePercent of time that signs and forms are available in Spanish and written for persons with low-literacy skillsPatient education signs and forms in Spanish are available 100%of time;literacy level of materials is at a 3rd grade reading level Linking evaluation questions,indicators and program benchmarks.Linking evaluation questions,indicators and program benchmarks.Example from the Guide Table 4.Example from the Guide Table 4.Gathering Credible Evidence:Data Collection24Gathering Credible Evidence:Data Collectionn nData CollectionvvWhere are the data?vvWhat methods will be used to collect data?vvHow often will the data be collected?vvWho will collect the data?n nTools for Data CollectionvvCollect only the information you need vvEasy to administer and useReference:Table 5 in your Evaluation Plan GuideReference:Table 5 in your Evaluation Plan Guide25Table 5.Data Collection PlanTable 5.Data Collection PlanIndicatorIndicatorData SourcesData SourcesCollectionCollectionWhoWhoWhenWhenHowHowGathering Credible Evidence:Data CollectionLinking indica