Five-Year Plan for Evaluation and Performance Measurement Strategies 2016-17 to 2020-21

1.1 Purpose of the Five-Year Plan

The primary purpose of the Plan is to help the Deputy Head ensure that credible, timely and neutral information on the ongoing relevance and performance of direct program spending and ongoing programs of grants and contributions is available to support evidence-based decision making and speaks to results for Canadians.^{Footnote 2} The Plan also:

• Provides an opportunity to align evaluations to ensure that the information needs of the Department and other evaluation users (e.g., other government departments where we are delivering horizontal programming) are being met;
• Helps ensure that evaluations supporting program redesign are planned and completed in advance of program renewal;
• Provides an annual platform for program managers and the Head of Evaluation to discuss the development and implementation of performance measurement strategies that effectively support evaluations;
• Allows departmental units responsible for the development of the Report on Plans and Priorities and the Departmental Performance Reports, as well as other groups engaged in strategic planning and reporting activities, to identify when evaluations will be available to inform their work;
• Initiates regular communication and consensus building on evaluation needs and priorities across the Department; and
• Provides central agencies with advanced notice of when evaluations will be available to inform their work (e.g., in support of Memoranda to Cabinet, Treasury Board submissions, etc.).

Moreover, the Plan serves as an important tool for the Head of Evaluation and the Branch in managing project workflows and planning the activities of its human resources. Status reports on the Plan are typically presented at all regular meetings of the EPMRC to provide assurance of timely implementation, and alert the Committee to any required changes or additions. The EPMRC typically meets four times during the fiscal year.

1.2 Background and Departmental Context

The Plan has been developed using the Treasury Board's Guide to Developing a Departmental Evaluation Plan but also with an eye towards forthcoming guidance in the Treasury Board Policy on Results and Resource Alignment. The impending update of the 2009 Policy on Evaluation was based on the conclusion that the government-wide Program Alignment Architectures and Program Measurement Frameworks had reached a level of maturity and required repurposing. According to the Treasury Board Secretariat, existing reporting tools were felt to be overburdening departments with reporting compliance and were focusing on low level indicators rather than the achievement of results. In addition, evaluations are viewed as not being sufficiently timely, and are often limited by insufficient performance information. As a result, in some instances, the ability to demonstrate that it was achieving its results was constrained.

The new policy is expected to allow departments to establish a results framework that identifies core responsibilities, associated results, and appropriate indicators. It will also limit requirements so that departments will collect performance information while remitting only some of this to Treasury Board. It will allow departments to prioritize evaluation coverage and issues, while professionalizing the evaluation function. It will continue to ask departments to report on both direct program spending and ongoing program of grant and contribution evaluation coverage, but will grant greater flexibility by allowing for risk-based planning for direct program spending coverage, and the option to defer evaluations beyond five years where average yearly expenditures are less than $5 million.

1.3 Departmental Mandate

INAC is the federal department primarily responsible for meeting the Government of Canada's obligations and commitments to First Nations, Inuit and Métis, and for fulfilling the federal government's constitutional responsibilities in the North.^{Footnote 3}

The Department's overall mandate derives from: the Constitution Act (1982), the Indian Act; the Department of Indian Affairs and Northern Development Act; territorial acts; treaties; comprehensive claims and self-government agreements, as well as various other statutes affecting Indigenous people and the North.

A significant amount of the Department's mandate also derives from policy decisions and program practices that have evolved over time. Its mandate is also framed by judicial decisions where there are direct policy implications for the Department.^{Footnote 4}

1.4 Program Structures and Performance

Departmental activities are largely structured by funding arrangements or formal agreements with First Nations and/or provincial or territorial governments. Most departmental programs, representing a majority of INAC spending are, in fact, delivered through partnerships with Indigenous communities and federal-provincial or federal-territorial agreements. INAC also works with urban Indigenous peoples, Métis and non–status Indians, many of whom live in rural areas.

1.5 Planned Expenditures and Transfer Payments

According to the 2015 Public Accounts of Canada, INAC is the fifth largest federal department in terms of total ministerial net expenditures, and the third largest in terms of total transfer payments and grants and contributions, after Finance Canada and Employment and Social Development Canada.

According to the 2016–2017 Main Estimates^{Footnote 5}, INAC spends over $7.5 billion annually, with approximately 88 percent ($6.7 billion) going directly to recipients through transfer payments. Total forecasted spending for 2016-17 is outlined in the table below according to the Department's strategic outcomes.

2.1 Drivers for Evaluation Planning

Program evaluations are required as per the current Treasury Board Policy on Evaluation (2009), Section 6.1.7. There is also a requirement under the Financial Administration Act (Section 42.1) to evaluate all ongoing grants and contribution programs every five years. Evaluation planning is conducted with those requirements in mind, and also in the spirit of robust management practices, which aid to ensure that results are being delivered for Canadians.

Each year, the Audit and Assurance Services Branch develops priority rankings for auditable units in the Department in a manner consistent with guidance from the Office of the Comptroller General. Risk considerations include: the impact of identified risks on the achievement of departmental outcomes; materiality; scope; potential for public scrutiny; legal risk; prevalence of risk; and the severity of consequences. This Plan is developed with alignment to the Risk-Based Audit Plan (2016-2017 to 2018-2019) in mind. Risk rankings are shown in Appendix B and are colour-coded to reflect the level of risk identified in the Risk-Based Audit Plan.^{Footnote 6}

2.2 What's New in the Plan

Note that the Government's Results Agenda, driven by publicly articulated ministerial mandates, the impending Policy on Results and Resource Alignment, and key government priorities being communicated to INAC, are collectively poised to spur change in program structures and available performance information in the Department. This could result in the creation of new program entities and the articulation of new results expectations and associated performance indicators. Notwithstanding this, Treasury Board Secretariat guidance clarifies that 2016-17 planning is to be done according to existing policies and coverage requirements.

The first step in preparing the 2016-17 to 2020-21 Plan included ensuring that all ongoing program spending is being evaluated every five years. Credit for evaluation coverage is granted upon deputy head approval of a report, under the premise that approval allows for the sharing of information in support of management action and the sharing of results with the public. This review was based on the Department's 2016-17 Program Alignment Architecture (Appendix A) and, as in previous years, is aligned with INAC's Management, Resources and Results Structure.

It also included the review of program areas within the Department to ensure that each is evaluated prior to program renewal or according to timing commitments made to the Treasury Board, and that Federal Accountability Act requirements are being met.

As a result, this document now includes two 'coverage tables', which all departments must complete and submit to the Treasury Board Secretariat on an annual basis. Appendix B – Indigenous and Northern Affairs Canada – Five-year Evaluation Schedule 2016-17 to 2020-21, outlines the sequencing of all program evaluations over the next five years with planned start and end dates.^{Footnote 7} As a result, the Plan shows the expected conduct of 44 evaluations covering 100 percent of direct program spending between 2016-17 and 2020-21.

The Five-Year Evaluation Schedule is attached in Appendix B. Appendix C – Indigenous and Northern Affairs Canada – Coverage of Ongoing Programs of Grants and Contributions – 2016-17 to 2020-21, includes an overview of the timing of evaluations as required by the Financial Administration Act.

2.3 Planned Evaluation Coverage of Direct Program Spending, Internal Services and Reviews

Looking back on the previous year, ten evaluation reports addressing direct program spending were approved in 2015-16, including two that began in the previous fiscal year. These evaluations represent $1.2 billion (15.02 percent) of the Department's direct program spending and include:

• Evaluation of Access to Capital and Business Services Program;
• Evaluation of Consultation and Accommodation;
• Evaluation of Contaminated Sites South of 60 Program (formerly identified as Contaminated Sites (On Reserve));
• Evaluation of Eco-Energy for Aboriginal and Northern Communities;
• Evaluation of Education Facilities and Community Infrastructure;
• Evaluation of the Administration of Reserve Land;
• Evaluation of the Impact of Self-Government Agreements;
• Evaluation of the Implementation of Income Assistance Reform On-Reserve (joint evaluation with Employment and Social Development Canada);
• Evaluation of Lands and Economic Development Services; and
• Evaluation of Petroleum and Minerals.

One additional evaluation was completed and approved in 2015-16, namely the Evaluation of INAC Recruitment, Development and Retention Activities. This Internal Services evaluation was conducted to address INAC management needs and spoke to important administrative activities at INAC that the Department wished to examine.

Looking forward to 2016-17, the following evaluations, which began in 2015-16 and address direct program spending, are planned to be completed and approved:

• Evaluation of Governance and Institutions of Government;
• Evaluation of On-Reserve Housing;
• Evaluation of Northern Governance — Support for Territorial Relations and Northern Organizations; and
• Horizontal Evaluation of Climate Change Adaptation (to be completed jointly with Environment Canada).

In addition to the above evaluations that address direct program spending, additional work begun in 2015-16 will be completed in 2016-17. This includes the delivery of five Internal Services evaluations and two reviews:

• Evaluation of Business Management Units (Internal Services Evaluation);
• Evaluation of the Annual Agreement Management Cycle (formerly Accelerated Annual Funding Cycle) (Internal Services Evaluation);
• Evaluation of the Implementation of Workplace 2.0 (formerly Accommodation 2.0) (Internal Services Evaluation);
• Review of the Implementation of the Tripartite Education Agreement (formerly identified as the First Nations Education Steering Committee) (Review); and
• Lessons Learned Study of Personal Credits (Review/Lessons Learned Study).

Therefore, nine evaluations that began in 2015-16 will be completed and submitted to EPMRC for approval in 2016-17.

Appendix D – Evaluation and Performance Measurement Review Committee – 2016-2017 Planned Schedule of Items, provides a list of all planning commitments and milestones/deliverables for the Branch. In addition to the work being completed above, the table shows that 18 new evaluations will be launched at some point in 2016-17. Of these, 16 will focus on programs with direct program spending and two will focus on Internal Services, namely the Evaluation of the Aboriginal Leadership Development Initiative and the Evaluation of Contracting and Procurement Hubs. Of that 18, five will be completed and submitted to EPMRB for approval in 2016-17.

The graph on the following page shows the level of activity that EPMRB will need to engage in over the course of 2016-17 and into the following year in order to meet the plan and deliver against policy requirements. The red line shows the volume of active projects when Terms of Reference development and project close-out (process for web posting) is taken into consideration. Note the peak of activity at the close of 2016-17 when a series of new Terms of Reference for future evaluations is expected to be launched.

*TOR – Terms of Reference
*DM – Deputy Minister

Text alternative for Graph 1: Volume of Planned Active Evaluation Projects (2016-17)

Graph 1 shows the volume of planed active evaluation projects to take place between January 2016 and November 2017. The vertical axis of the graph is labelled "Number of active projects". The axis is numbered from zero to twenty-five, in increments of five. The horizontal axis is unlabelled and shows the time span by month, beginning with January 2016 and ending with November 2017.

The graph shows three lines. One is a straight horizontal line across the entire graph, starting and ending at 17.5 on the vertical axis. Text within the graph labels the line as the "Average number of active projects".

A second fluctuating horizontal line is labelled "Volume of evaluations including TORs (3m) and closeout (4m). The points on the line are as follows: January 2016, 16; February 2016, 15; March 2016, 15; April 2016, 15; May 2016, 20; June 2016, 16; July 20l6, 16; August 2016, 17; September 2016, 19; October 2016, 18; November 2016, 18; December 2016, 19; January 2017, 16; February 2017, 20; March 2017, 23; April 2017, 20; May 2017, 20; June 2017, 20; July 2017, 18; August 2017, 18; September 2016, 18; October 2017, 15; November 2017, 15.

A third fluctuating horizontal line is labelled "Volume of evaluations based on TORs approval date to DM approval date only". The points on the line are as follows: January 2016, 9; February 2016, 8; March 2016, 8; April 2016, 8; May 2016, 10; June 2016, 13; July 20l6, 9; August 2016, 9; September 2016, 12; October 2016, 11; November 2016, 11; December 2016, 12; January 2017, 11; February 2017, 10; March 2017, 16; April 2017, 13; May 2017, 13; June 2017, 13; July 2017, 11; August 2017, 11; September 2016, 11; October 2017, 8; November 2017, 8.

2.4 Planned Evaluation Coverage of Ongoing Programs of Grants and Contributions

INAC currently delivers 36 authorities (voted grants and contributions). A one-to-one relationship between the authorities listed in the Main Estimates and the inventory of ongoing programs of grants and contributions deemed a 'program' by the Department exists in only seven instances. A number of programs draw on multiple grants and contributions. Conversely, there are grants and contributions that contribute to multiple departmental programs.

The Branch takes the above into account when planning in an effort to ensure that evaluations examine the relevance and performance of all voted grants and contributions over a five-year cycle. Evaluation activities are organized to capture necessary information either through discrete evaluations of individual authorities/grants and contributions, or through evaluations that speak to the relevance and performance of authorities imbedded in larger programs. Each evaluation report will clearly identify the authorities that have been addressed through the evaluation activities.

On this basis, the Plan outlines the planned timing of evaluations in Appendix C – Indigenous and Northern Affairs Canada – Coverage of Ongoing Programs of Grants and Contributions – 2016-17 to 2020-21. This table links all authorities to all departmental programs. This helps clarify the links that exist between and amongst authorities and programs in order to support the Department's efforts to meet Federal Accountability Act requirements and to provide the necessary performance information.

In 2016-17, it is anticipated that 22 of 36 authorities (61 percent) will be addressed in full or in part as part of the Branch's planned evaluations and reports that are expected to be approved.

2.5 Evaluation, Performance Measurement and Review Branch Performance Metrics

EPMRB has established performance measures to ensure progress towards coverage requirements. The Audit and Evaluation Sector Business Plan 2016-17 has set targets for the Branch regarding the initiation, progression and completion of scheduled evaluations. The Branch has targeted 50 percent of scheduled evaluations being initiated in the first quarter of the fiscal year; an additional 25 percent in the second quarter and the final 25 percent in the third quarter. Thus, for Year 1 of the Plan, five evaluations will be started in Q1, four to start in Q2 and the remaining two in Q3. In addition, the Branch has established targets for its deliverables, as follows:

• Eighty percent of evaluations will deliver preliminary findings as scheduled; and
• Sixty-five percent of evaluations projects will see final reports delivered as scheduled.

2.6 Implementing the Plan

Evaluations conducted by EPMRB are used to inform decision making and reporting (such as the Reports on Plans and Priorities, Departmental Performance Reports, Annual Report on the State of Performance Measurement); and to assess progress made by programs in achieving expected results and to incorporate lessons learned in order to improve program design and delivery.

Increasingly, as a means of managing workload, improving the timeliness of evaluations and ensuring the greatest relevance of the evaluation function, the Branch will work to focus on priority issues in its evaluations, and will seek to scope them to respond to the most pressing relevance, performance and efficiency theme as determined in its planning phase. Impending policy changes provide for more flexibility in the identification of issues, allowing EPMRB more flexibility in scoping its work. Once defined, the Terms of Reference for each evaluation will then be approved by the EPMRC.

Further, in 2015-16 EPMRB put in place new standing offer agreements with a range of firms providing access to evaluation, survey and statistical and quantitative analysis services. In support of achieving its plan, the Branch will appropriately complement its in-house led activities with the skills of additional evaluation resources where appropriate.

The table below provides an overview of evaluation expenditures and human resources, both planned and actual over a three year horizon.

Note that while 22 full time equivalent positions are listed, the Branch, must work to address position vacancies on an ongoing basis as it has seldom been at full capacity given the general movement of staff to other opportunities and the ensuing timelines needed to fill positions. The Branch also leverages the skills and talents of undergraduate and graduate students throughout the year.

2.7 Other Branch Activities

In addition to conducting evaluations, the Branch undertakes a wide range of activities to support and strengthen the Department's evaluation and performance measurement work. These include providing advice and support on performance measurement, corporate planning and reporting at the departmental level (e.g., Program Alignment Architecture and Performance Management Framework), supporting capacity building and outreach activities, and developing research and special studies on evaluation and performance measurement issues.