Archived - Evaluation Plan 2008-2009 to 2012-2013

Date: June 27, 2008

PDF Version (271 Kb, 21 Pages)

Table of Contents

• Introduction
  • Purpose
  • Context for Evaluation
• Evaluation Planning Approach
  • Planning Assumptions
  • Planning Methodology
• The Five-Year Evaluation Plan
  • Coverage
  • Risk Considerations
• Evaluation Management and Costs
• Consultations
• Updating The Plan
  • INAC Evaluation Plan 2008-2009 – 2012-2013
  • People Strategic Outcome: Socio-Economic Policy and Regional Operations
  • Government Strategic Outcome: Treaties and Aboriginal Government
  • Land Strategic Outcome: Lands and Trust Services
  • Internal Services Strategic Outcome: Policy and Strategic Direction
  • Economy Strategic Outcome: Aboriginal Economic Development
  • North Strategic Outcome: Northern Affairs
  • Office of the Federal Interlocutor Strategic Outcome
  • Internal Services Strategic Outcome: Chief Financial Officer
  • People Strategic Outcome: Resolution Sector
  • Special Studies – Chief Audit and Evaluation Executive
• Appendix A
• Evaluative Work by Year by Sector
  • 2008 - 2009
  • 2009 - 2010
  • 2010 - 2011
  • 2011 - 2012
  • 2012 - 2013

Introduction

Purpose

This document presents Indian and Northern Affairs Canada's (INAC) Evaluation Plan to assess the value for money of its programs and policies, and covers the five year period from 2008-2009 to 2012-2013.

The plan proposes increased coverage from recent years to address the requirements of the Treasury Board Secretariat's (TBS) policy on evaluation, Transfer Payment Policy and the Federal Accountability Act, which all require that there be a "review"/evaluation, of all direct spending, including transfer payment programs, every five years. As such, this plan provides for full evaluation coverage of all the grants and contributions administered by INAC.

Context for Evaluation

Evaluation provides Canadians, Parliamentarians, Ministers, central agencies and deputy heads with an evidence-based assessment of the relevance and performance of policies and programs. In the Government of Canada, evaluation is the systematic collection and analysis of evidence on the outcomes of policies and programs. Evaluation is the primary source of neutral and systematic information on the ongoing relevance and performance of policies and programs, alternative ways of achieving expected results, and program design improvements. Evaluation helps the Government of Canada to design and deliver programs and services that are accountable, focused on results and which serves Canadian citizens.

Thus it is a key function of government which informs management and resource allocation within departments and government-wide. Evaluation is essential for responsible, disciplined and results based spending as it informs departments whether their programs are delivering the immediate results and longer-term outcomes for which they were designed at an affordable cost. Evaluation ties strategic reviews of program spending to ongoing relevance and performance, and informs resource reallocation decisions.

Evaluation reports can also help Ministers understand how new spending proposals fit with existing programs, as well as identifying synergies, and avoiding wasteful duplication. As such, evaluation:

1. supports accountability to Parliament and Canadians;
2. informs government decisions on policies, new program proposals, and resource allocation and reallocation;
3. helps deputy heads manage for results; and
4. supports policy and program improvements by providing a basis for identifying lessons learned and best practices.

The framework for the systematic collection and analysis of performance information is the department's Management, Resources and Results Structure (MRRS). Evaluations are organized along the strategic outcomes of the Program Activity Architecture (PAA).

Evaluation Planning Approach

The 2008-2009 to 2012-2013 Evaluation Plan proposes a strategic, risk-based approach to the assessment of the department's programs and activities.

It covers all grant and contribution programs which is in keeping with the Federal Accountability Act, the Treasury Board Secretariat (TBS) evaluation policy as well as the Policy on Transfer Payments. Where possible programs are clustered to allow for the evaluation function to study and conclude on entire policy areas, using an optimal number of studies focused on broader horizontal issues. This approach also supports annual reporting along the structure of the department's Program Activity Architecture.

The plan also proposes a series of special studies to address horizontal risks and factors that affect, either directly or indirectly, achievement of the department's intended results by assessing some of the complexities of INAC's operating environment and strengths of current practices, instruments and approaches.

A special annual assessment (The Health of INAC's Performance Information System) which is a requirement of TBS evaluation policy will study information being collected by the Department and its usefulness for departmental results reporting purposes, emphasising performance information for programs due to be evaluated the following year.

Planning Assumptions

A number of assumptions have been made in developing the plan:

• While introduction of the new Policy on Evaluation is anticipated in the Fall of 2008/09, departments have been advised by TBS officials to design their evaluation functions and plans in compliance with the proposed policy;
• TBS supports cluster evaluations based on the department's PAA and policy frameworks; and
• Evaluations supporting accountability requirements will be led by the Evaluation Branch.

Planning Methodology

Documentation of INAC's "evaluation universe" was a critical first step in developing the plan to ensure its comprehensiveness. The department's PAA, its grant and contribution programs, their authorities and expiry dates were all analysed. The plan reflects INAC PAA elements / strategic outcomes and its grant and contribution programs.

A second step was to identify past commitments to, and central agency expectations for, evaluations, which required a review of departmental Treasury Board submissions' program authorities, management responses to Auditor General audits, and previous evaluations.

In addition, an analysis was done of issues and risks related to achievement of expected outcomes based on management risk discussions, evaluation and audit findings, including those of the Office of the Auditor General, reports, and other documentation of program design and delivery issues facing the department.

The Five-Year Evaluation Plan

The INAC Evaluation Plan, 2008-2009 to 2012-2013, proposes comprehensive and sustainable coverage of the department's programs and activities.

Coverage

The plan includes a total of 46 evaluations and three special studies (49 in total) over a five-year period, focussed on INAC's key policy and program areas. The 46 evaluations provide coverage of the department's approximately 50 grant and contribution authorities. The special studies assess factors that cut across programs and policies that affect, either directly or indirectly, achievement of the department's objectives.

Tables in the next section present evaluation projects and special studies. Appendix A presents evaluation projects by sector and year.

Risk Considerations

The Treasury Board policy framework for determining risks and the risk assessment approach in the Policy on Transfer Payments were used to assess risks for evaluation planning. These risk assessments have been considered in the identification of evaluation projects and will be further considered in the identification of evaluation issues for individual projects. The risk assessments also factor significantly in the timing and resourcing for evaluations.

The first risk consideration by Treasury Board is referred to as Mission-Risk and includes the following elements:

• Size of population affected or targeted by the program
• Consequences if the program or infrastructure risk materializes
• Probability of the program or infrastructure risk materializing
• Degree of control or influence by the department
• Time available to prevent, mitigate, avoid risk

The second risk consideration is that of materiality, which includes the following elements:

• Funding profile
• Public confidence
• Political sensitivity

The third risk consideration is that of performance information availability; which is a key concern in determining the scoping and sequencing of evaluations. Elements included in this risk consideration are:

• Sound underlying program theory
• Robust performance management framework
• Quality evaluation
• Continued relevance of existing performance information
• Time available to generate required performance information

These risk considerations were assessed by the evaluation team in consultation with program managers.

Most evaluations conducted by the Evaluation, Performance Measurement and Review Branch (EPMRB) will be summative in nature, focussing on evaluating the relevance and performance of the programs, as opposed to formative evaluations which typically focus on implementation and delivery issues.

Evaluation Management and Costs

Evaluation projects are led by Senior Evaluation Managers, with support from other staff of the department's Audit and Evaluation Sector and, in most cases, consulting resources. The project evaluation teams will seek input from other departmental officials, especially those with management responsibilities related to the program under evaluation.

The Senior Evaluation Managers will generally call on consulting companies to do fieldwork related to evaluations, e.g., literature reviews, interviews, case studies and expert focus groups.

Costs of consulting resources will vary, depending on project's scope and complexity. A preliminary per project estimate could range from $175,000 to $500,000. All costs will be borne by the responsible sector. A cost estimate will be developed by the Senior Evaluation Manager prior to its commencement, as part of the project Terms of Reference. Once Terms of Reference are approved by departmental senior management, sectors will be expected to arrange for the transfer of funds to cover project costs.

Costs of special projects that are not sector or authority-specific will be borne by the Audit and Evaluation Sector.

Consultations

The plan was discussed with senior management in all sectors of INAC to ensure that it provides for full coverage and that the approach is sound.

Updating the Plan

The plan is updated and tabled annually at the Audit and Evaluation Committee, to take account of evolving needs for coverage and strategic information. Because it is a rolling plan, annual updates will also identify evaluation activities for the fifth year of the renewed plan. The updated plan will reflect work not completed as planned and carried over to another fiscal year.

INAC Evaluation Plan 2008-2009 – 2012-2013

APPENDIX A

Evaluative Work by Year by Sector

2008 - 2009

2009 - 2010

2010 - 2011

2011 - 2012

2012 - 2013