X
GO
Programme Evaluation – Large Scope

This sample plan contains a large number of key questions. You can reduce the number of questions to make your evaluation more targeted and succinct.

Depending on the age of your HealthPathways programme, your evaluation may focus on different aspects of HealthPathways:

  • New regions might focus on initial growth, e.g. number of pathways localised, page views, and users.
  • Established regions may be more interested in sustained growth, expansion throughout the wider health system, and clinical redesigns.

Key questions or programme aims

  1. Is the HealthPathways programme well supported by the executive stakeholders, funders, and project partners?
  2. How does the wider health system view and use HealthPathways?
  3. Are the HealthPathways programme objectives being met?
  4. How is the HealthPathways programme utilising its resourcing (time)?
  5. Has the implementation of HealthPathways been successful for the programme and clinicians?
  6. Are there any service redesign, clinical improvement, or localisation opportunities?
  7. How do GPs perceive HealthPathways?
  8. Is health professionals' knowledge and use of, and engagement with HealthPathways increasing?
  9. Are the engagement activities effective?

Publication database examples

Programme Evaluation – Large Scope

Key questions/Programme aims

What questions will identify if the programme is achieving its aims?

Outcomes/Programme logic

What difference do we aim to make? 

What do we expect to achieve? 

Who will benefit?

Indicators

How will we know if progress is tracking well? 

What changes will we look for? 

What indicators will help answer our key questions?

Methods/Data sources

What data will we collect and how?

Is the HP programme well supported by the executive stakeholders, funders, and project partners?

  • Is HP considered and used as the single source of truth?
  • Do project partners consider HP to be an enabler of system change?
  • Does the HP programme have executive champions?
  • What do the stakeholders and funders perceive the value of HP to be?
  • Is HP being used as a tool to support broader system change?
  • Is HP an integral part of the service planning process?
  • Support commitment from project partners for:
    • Successful engagement and integrations
    • Continued funding
    • Contribution to and endorsement of HP as an integral part of service planning and process
    • Feedback on and endorsement of the long-term goals, including evaluation
  • Review feedback
  • Continued executive and project partner support for:
    • HP programme
    • Engagement
    • HP integration with the wider health system
  • Governance committee meeting minutes
  • Interviews with governance committee members
  • Feedback:
    • HP programme team
    • Clinical and non-clinical leaders
    • Key personnel

How does the wider health system view and use HP?

  • Is HP considered and used as the single source of truth?
  • Is HP accessible to, and supported and used by the wider health system?
  • Is HP viewed as a tool for communication?
  • Have relationships improved between primary, secondary, tertiary care?
  • Identify opportunities for increased collaboration between primary, secondary, and tertiary care
  • Identify any programme barriers
  • Determine the usefulness of HP as a vehicle for health system communication and relationship building
  • Establish the level of health system support from:
    • Key leaders
    • SMEs
    • NGOs
    • Allied health
    • Clinicians
  • Identify participation in and engagement with the HP programme
  • Review feedback on HP's influence on the wider health system
  • Engagement:
    • SMEs with CWGs
    • SMEs with drafting process
    • Allied health
  • Interviews with governance committee members, and key clinical and non-clinical leaders
  • Feedback:
    • CWGs
    • SMEs/CEs
    • Allied health
    • Clinicians
    • Dot
  • Surveys

Are the HP programme objectives being met?

  • Track programme indicators over time:
    • Localisations
    • Reviews
    • Development/drafts
    • Partial updates
  • Number of pathways or pages in development or draft, localisations, reviews (12 categories)
  • Dot

How is the HP programme utilising its resourcing (time)?

  • Is the programme efficient?
  • Are some specialties more resource intensive than others?
  • Monitor time usage 
  • Identify time-consuming specialties
  • Review feedback
  • Time per specialty per month
  • Time per task per month:
    • Development
    • Reviews
    • Localisations
    • Partial updates
    • Meetings and correspondence
    • Site administration
  • Dot
  • Feedback:
    • SMEs/CEs
    • CWGs
  • HP programme team records
  • Identify any barriers in specialty development:
    • Time taken for specialty development, localisations, and reviews
    • Page views per specialty
    • High-resource (time) low-use specialties
  • Review feedback
  • Time taken per specialty for development, localisation, and review
  • Page views per specialty
  • Dot
  • Google Analytics
  • Feedback:
    • SMEs/CEs
    • CWGs
    • HP programme team

Has the implementation of HP been successful for:

  • the programme
  • clinicians
  • Are clinicians satisfied with the HP programme?
  • Identify any programme barriers or constraints?
  • Is the programme working as intended?
  • Review feedback
  • Focus groups
  • Interviews
  • Surveys
  • Feedback:
    • Governance group
    • SMEs/CEs
    • CWGs
    • HP programme team

Are there any service redesign, clinical improvement, or localisation opportunities?

  • Identify and adjust work development priorities
  • Update priority plans based on user feedback 
  • Identify service redesign opportunities
  • Reduce unlocalised page usage
  • Page views:
    • Localised
    • Unlocalised
  • Search terms
  • Review feedback
  • Google Analytics
  • Feedback:
    • CWGs
    • SMEs/CEs
    • Clinicians
  • Enlist SME and primary care involvement in pathway localisation
  • Estimated number of contributors by pathway specialty
  • CWG feedback
  • Dot
  • HP programme team records
  • CWG feedback

How do GPs perceive HP?

  • Is the HP programme creating value for end users?
  • What improvements could be made from a GP perspective?
  • Establish the level of support from GP users
  • Identify areas for targeted improvements
  • Review feedback
  • Clinician feedback:
    • Interviews
    • Focus groups
    • Surveys
  • Outreach activities:
    • Education events
    • Practice visits

Is health professionals's knowledge and use of, and engagement with HP increasing?

  • Is site activity increasing?
  • Are your engagement activities effective?
  • Increased participation in and engagement with HP
  • Increased user knowledge of and confidence in appropriate care and referral services available locally
  • Positive trends in number of:
    • Users (new and existing)
    • Sessions
    • Page views
  • Google Analytics
  • Improved user experience
  • User feedback on HPs impact on the quality of consultations
  • Focus groups
  • Interviews
  • Surveys
  • Engagement, education, and outreach activities
  • Number of communications
  • Estimated reach of communications
  • Google Analytics and campaign tracking
  • Feedback
  • Maintain continued usefulness of HP and use it as a tool for communication and relationship building
  • Review feedback
  • Number of CWGs
  • Feedback
  • CWGs
  • SMEs/CEs
  • Review and log all feedback
  • Feedback:
    • Dot
    • Engagement activities
    • News stories
    • Events

Are the engagement activities effective?

  • Is the engagement strategy working as intended?
  • How has the engagement strategy achieved this?
  • Which activities are most effective?
  • Determine the extent and impact of your engagement, education, and outreach activities
  • Number of communications and events
  • Estimated reach of communications
  • Review feedback
  • Google Analytics and campaign tracking
  • Feedback:
    • Clinicians
    • HP programme team
    • Stakeholders