Consider the big picture to provide some context for your HealthPathways programme and evaluation?

• What is the current state of your health system? 
• What do you want your health system to look like? 
• Is there a plan? How will you achieve your ideal health system? 
• What do you expect from HealthPathways?
• How does HealthPathways fit in to your plan to achieve your ideal health system?

Planning an evaluation can be daunting as there are many aspects that you could investigate and a variety of possible methods.

The Evaluation Framework aims to provide a comprehensive resource, but this can also lead to information overload. Feel free to contact the R&E Hub if you require assistance.

Not all evaluations need to be large and comprehensive. Succinct, targeted evaluations can create value and are often easier to manage.

Consider your scope carefully:

• Refine the scope of your evaluation to ensure it is manageable.
• Start small and then expand your evaluation as needed.
• Consider whether you have enough resourcing for your evaluation.

Key questions help focus your evaluation. Answering these questions should be the main purpose of your evaluation.

When formulating your key questions, remember to:

• include stakeholders in your evaluation planning.
• review governance committee documentation to identify key performance indicators (KPIs), deliverables, and programme objectives.
• clearly state the purpose of the evaluation and the outputs required to determine success.
• consider your available resourcing:
  • Are the key questions and required outputs achievable?
  • Does your team have the capacity and capability to carry out the evaluation?
  • Can you obtain additional resourcing, e.g. local university or internal expertise?
  • Are the data available? Could any other data be used as a proxy?

Another important consideration is that HealthPathways offers different benefits to different users:

• For general practice, it is a manual for day-to-day use.
• For allied health and the third sector, it increases their visibility in the wider health system and ensures their services are used appropriately.
• For specialist services, it is a tool for managing patient flow and resources.
• For change managers, it facilitates negotiation of new ways of working and embeds changes in the health system.
• For funders, it is a tool for improving efficiency and sustainability.
• For peak bodies and governments, it is a means of getting evidence-informed policy and guidelines into practice at scale.

You will likely identify a range of goals, key questions, and data sources (see graphic below) for your evaluation. It is important that you clearly understand your goals. Once finalised, the key questions should inform which evaluation elements and data sources you select for your evaluation. Other evaluation elements may depend on local drivers such as needs, expertise, time frames, and resourcing.

The figure below shows nine possible research questions. Other questions might be proposed, though they are likely to be categorised as sub-questions of these ones.

The questions can be divided into three general research categories: subjective, objective, and controlled:

• Subjective research: Soliciting user opinions or experiences, e.g. through surveys, focus groups, interviews. Users may include HealthPathways clinicians, managers, and health care consumers.
• Objective research: Measuring data points, e.g. times, rates, outcomes, page views, pathway go-lives. These may also be compared over time or across different systems.
• Controlled research: Comparing measurements over time or across different systems and attributing changes to specific interventions or changes, having controlled for other factors.

Some of this research is easy to carry out, such as creating user surveys or reviewing Google Analytics. While subjective information, such as how well users like HealthPathways, can be essential to your evaluation, it doesn't provide definitive answers about the success of your local implementation or whether it's having an impact on health outcomes.

Controlled research into clinical or process outcomes can answer these questions, but is harder to carry out, as shown below.

Your evaluation team should include both subject-matter expertise and evaluation expertise. Consider your available resourcing and whether your team has the capability to conduct an evaluation. Effective evaluations ideally require a mix of expertise, and independence.

• What evaluation expertise does your team have?
• Are there external personnel who could assist, e.g. statisticians, university researchers or hospital analytics teams?

The team requires a range of skill-sets, including people who can assist with:

• planning and facilitating the evaluation.
• statistics, which are important for determining sample sizes, significance, attribution, and clinical impact studies.
• communicating progress and results to governance, stakeholders, and the wider community of users.

Programme logic is a useful tool for illustrating the evaluation process and determining activities and outcomes during your evaluation. It can help:

• clarify the purpose of the evaluation.
• aid the communication by building a common understanding of how the evaluation will work
• identify key questions, activities, and outcomes.

The Evaluation Framework tables may help you develop your programme logic.

For further guidance, see New South Wales Ministry of Health – Developing and Using Program Logic: A Guide section 6.1.

The evaluation design step refines and details your key questions.

• Use your programme logic to expand and detail the questions.
• Determine how you will answer the questions.
• Establish success criteria. For guidance, criteria, and standards, see BetterEvaluation – Determine What Success Looks Like.

The SMART characteristics can be useful when refining key questions and methodologies:  

• Specific 
• Measurable
• Attainable
• Relevant
• Timely

Having finalised your key questions, select appropriate methodologies and data sources. Input from a statistician can be valuable at this point.

Always consider any ethics implications of your methodologies and data.

Consult with stakeholders regularly during the design stage. This will help keep them engaged and account for their views.

Before you start collecting data, consider testing your methodologies, For example,

• trial survey questions with a small group and refine them before sending them out to the entire survey population.
• conduct some exploratory data analysis to check data quality and make sure that data can answer your questions.
• check if the clinical coding of data is consistent between healthcare providers.

Consider your data carefully. Will the data enable you to answer your key questions?

• Veracity – The degree to which the data is correct?
• Validity – The degree to which the data measure what they are intended to measure?
• Precision – Are the data sufficiently detailed?
• Uncertainty – Are the data collected consistently? Are the definitions and methodologies the same when repeating measurements over time?

Record any difficulties encountered during data collection that may influence the quality of the data. Even if the data is flawed, it still might be useful. However, it is important to understand the data's limitations and include any uncertainties in your evaluation report.

Interrogate the data to reveal themes and actionable information. This is an important process and, depending on the data, can be time-consuming and labour-intensive. Appropriate resourcing is required. The results of the data analysis should provide answers to your key questions.

When analysing the data:

• Be sceptical of your results. Can they be confirmed by using independent data or methodologies?
• Do the results make logical sense? Can you explain them? 
• Does the data tell a story? Can you use it to build a narrative? 
• Are the results statistically significant? 

When writing your evaluation report:

• Remember the purpose of the evaluation and consider your audience.
• Check that you have answered all the key questions. If any have not been answered, explain why not.
• Consider any implications of the outcomes of the evaluation?
• Include enough information in your methodology section to enable your results to be replicated.
• Use the data and outcomes to build the story of your evaluation and support its conclusions.
• Discuss any uncertainties or problems with the data or methodologies. If the data is flawed, explain its limitations and any impact this may have on the results or conclusions.
• Create a narrative around any figures and explain any trends or points of interest. If a figure is important enough to be presented in the report, it should also be discussed.
• Review the outcomes and consider if they support the Quadruple Aim of Healthcare of improved:
  • patient experience.
  • population health outcomes.
  • efficiency and sustainability (reduced costs).
  • health care provider experience.
• Consider distributing a draft version of the report to stakeholders, executive sponsors, and other interested parties to gather and incorporate any feedback.
  • Arrange peer review of the report both within and outside your HealthPathways programme.
  • Solicit critical reflection by asking individual stakeholders for independent feedback.
  • Arrange external review to gather feedback from external experts or anonymous reviewers.

For more guidance on writing an evaluation report, see Centers for Disease Control and Prevention – Developing an Effective Evaluation Report.

Follow your communications plan, that was determined at the planning stage, to distribute the evaluation report.

Consider sharing and publishing your results on the R&E Hub Publication Database or in a peer-reviewed journal.

Review the entire evaluation process to determine if your team has learnt anything it could improve on in future evaluations.

• If you needed to redo the evaluation, would you do it the same way?
• What parts of the evaluation worked or did not work?
• Can your outcomes and/or process be shared with the HealthPathways community?
• Could the results be written up as a research article?

Document any lessons learnt for future reference, so that they can be shared and not lost.