Canterbury used the Canterbury Outcomes Framework, an outcomes matrix featuring multiple cascading measures, to measure the accumulative impact of many smaller gains such as reduced treatment-related errors and the reduced length of inpatient stays.

• Ensure a baseline is used.
• Key items to include:
  • Acute (urgent) admission rates.
  • Readmission rates.
  • Length of stay.
  • Emergency Department attendance rates.
  • Frail elderly rates.
  • Outpatient DNA and wait time.
  • Elective surgery metrics such as specialist assessment to surgery times and rates.

For example, measure the perception of whether (and how) whole of system thinking is changing by conducting repeat surveys with staff and patients in all sectors and at all levels. Questions could include:

• Staff
  • Do you feel part of the Health System?
  • Do you feel a responsibility for patients whether in or out of hospital?
  • What do you think of relationships between hospital, general practice, and the third sector?
  • What key values or phrases have you adopted? (For example, Canterbury uses patient waiting times as a proxy for efficiency and quality.)
• Patients
  • Do you think we need to work in a different way or do you think we are doing fine?

For example, measure:

• The quality of referrals balanced by the quality of hospital discharge and advice letters.
• Referral processes such as use of e-Referral.
• The quality of triage against capacity and acuity.
• The use of alternatives to hospital care such as written or phone advice, allied health, or third sector responses.
• First specialist assessment to follow-up ratios.

At a department level, look at:

• The above metrics at a service level.
• Whole system participation, attitudes, and leadership.
• Movement of new and follow-up work to assess whether hospitals are providing the services that only they can provide or are providing primary care too.
• The quality of community follow-up, plus dropout rates

For example, you could measure desirable outcomes like the movement of traditional hospital tasks into the community, traditional general practitioner tasks to nursing or pharmacy, and traditional pharmacy and nursing tasks to patients or the third sector.

For example:

• Increased use of the third sector.
• Increased proportion of patients provided with patient information.
• Increased use of joint decision making or advance care planning.

Measure agreement with common sense arguments such as:

• Good things
  • Having a manual for general practice describing the local health system, the recommended care for conditions, and the services available.
  • Sitting down with clinicians, managers, and funders to agree how care will be delivered for groups of patients and reveal agreements in a website.
  • Reducing frustration levels and unnecessary variation.
  • Quick reflection of clinical evidence to GP’s desktops.
• Bad things
  • General practice flying blind in a fog of anecdote and experience

Consider obtaining department metrics before, and after, launch of a pathway or suite of pathways. For example:

• Referral decline rates.
• Do not attend rates.
• FSA/surgery and follow-up rates.
• Quality of referral information.

Case by case audits of patient cohorts assess if their outcomes are as expected based on the new pathway. This is meticulous work and ideally metrics are collected automatically as part of usual business. In practice, the data is often in multiple systems or on paper, and needs manual extraction. 

By focusing an evaluation on one specific disorder (instead of one specific pathway) you can track the long-term impact of pathways working in conjunction with other improvement efforts in the health system. Using the same generic tools, like Google analytics and staff surveys, you could look at all the stages of a multifaceted intervention that includes pathways, but isn’t limited to them.

This is a useful tool for observing usage trends for your HealthPathways site. However, analysis needs to be high level and not over-interpreted. See the HealthPathways Online Help for further information.

For example, survey general practitioners, practice nurses, and hospital clinicians. Evaluation studies often employ a mixed methods aproach that includes surveys.

Examples:

• Is HealthPathways effective? An online survey of hospital clinicians, general practitioners and practice nurses
• Publication Database – HealthPathways Evaluation

For example, when Canterbury conducted its urinary incontinence surveys, they surveyed general practice staff before and after a localised pathway on the topic appeared on its site. The survey covered difficulty of making diagnosis, confidence in providing initial treatment, and other measurable factors that can be improved with the publication of an effective clinical pathway. This is now being repeated in a number of districts.
This type of survey:

• Gives good information on GP knowledge of the subject.
• Highlights differences in knowledge (suggesting if your HealthPathways site is making a difference).
• Can be used to track data from many different data points in pathways, if local data is being routinely collected.