Neutral Assessment of the Evaluation Function
Date : January 2014
PDF Version (325 Kb, 29 Pages)
Table of contents
This report sets out the results from the neutral assessment of the evaluation function at Aboriginal Affairs and Northern Development Canada (AANDC or the Department).
1.0 Background
The evaluation function is one of several senior governance functions at AANDC. Its role is to provide an evidence-based, neutral assessment of the relevance and performance of its programs. Treasury Board of Canada Policy sets out the expectations for a federal department’s or agency’s evaluation function in the Policy on Evaluation, April 1, 2009. One of the Policy’s requirements is that deputy heads ensure that a neutral assessment of their departmental evaluation function is conducted at a minimum of once every five years. This is the first neutral assessment of AANDC’s evaluation function.
Treasury Board (TB) is not prescriptive in terms of the conduct or content of a neutral assessment. To conduct this neutral assessment, AANDC’s Head of EvaluationFootnote 1 (HoE) engaged a senior evaluation consultant who had not performed any evaluation work for AANDC in recent years. The HoE and the consultant agreed that the objectives of the neutral assessment would be to address the following questions:
- Does the evaluation function conform to Treasury Board’s expectations?
- Is the evaluation organization well-managed?
- Is the function adding value to AANDC?
2.0 Scope and Approach
This neutral assessment exercise was conducted over a three month period from November 2013 to January 2014. It provides findings and conclusions about the state of the function at the time of the assessment. Findings and conclusions related to the evaluation function relate to the practices at the department level; findings and conclusions that relate to the evaluation organization relate to the practices within the HoE’s organization – the Evaluation, Performance Measurement and Review Branch (EPMRB).
To conclude on project-level matters, the exercise reviewed documentation from six evaluation projects completed between April 1, 2009 and March 31, 2013. To conclude on organization- and function-level matters, the exercise examined practices in place at the time of the assessment.
The framework used for this neutral assessment includes three distinct components, focusing on:
- Governance of the evaluation function;
- The evaluation organization’s professional practice at the organization level and at the project level; and
- Monitoring and performance reporting associated with the evaluation function.
The framework includes criteria that were derived from policy, standards, and directives associated with the practice of evaluation. International standards on evaluation were also considered in developing the framework. Additional criteria were included that reflected sound management practices that were applicable to an evaluation function. The framework criteria were reviewed and agreed to by the Head of Evaluation. They are included as an appendix.
The neutral assessment was based on an analysis of documentation and interviews with members of AANDC senior management and the departmental evaluation committee.
As requested by the HoE, the analysis was conducted to an audit level of assurance. In addition to his experience in evaluation in the Canadian federal government, the consultant conducting the exercise is a Certified Internal Auditor.
3.0 Conclusions from the Neutral Assessment
The conclusions address the three assessment areas of governance; professional practice; and monitoring and performance reporting. Findings that support these conclusions are provided in Section 4.0 of this report.
The neutral assessment determined that AANDC’s evaluation function:
- Conforms to Treasury Board’s expectations with regard to governance
- Conforms to professional practice expectations at the function level
- Conforms to professional practice expectations at the project level
- Conforms to monitoring and performance reporting expectations
The findings indicate that AANDC’s evaluation function is meeting expectations. The findings also indicate that the evaluation function is adding value to department, and that the evaluation organization is well-managed.
The neutral assessment identified process improvements related to evaluation planning. There are no other recommended actions required to improve conformance.
The findings that support these conclusions are presented below.
_____________________________
Phil Carr, Certified Internal Auditor
4.0 Findings from the Neutral Assessment
The findings presented in this section inform the conclusions on whether the evaluation function either conforms to expectations, or its conformance to expectations is at risk.
Conformance indicates that processes and practices in place are judged to be in accordance with expectations set out in policy, standards, and directives. Where conformance is at risk, the processes and practices in place have been judged to fall short of expectations, and they negatively impact the evaluation function's capability to perform its responsibilities in a cost-effective manner.
The summary findings are presented below according to the structure of the neutral assessment framework agreed to at the outset of the project.
4.1 Governance of the Evaluation Function
The neutral assessment's governance criteria address values and ethics, as well as the responsibilities of the Deputy Minister with respect to the evaluation function, the departmental evaluation committee, and the HoE.
4.1.1 Deputy Head Responsibilities: Conforms to Expectations
The deputy head is fulfilling his responsibilities related to governance. The Deputy Minister has established an evaluation function, appointed a departmental evaluation committee and an HoE at the EX-03 level, and has approved a five-year Evaluation Plan. The Deputy has approved in a timely manner the reports of completed evaluations, including associated management action plans.
4.1.2 Evaluation Committee Responsibilities: Conforms to Expectations
The departmental evaluation committee is fulfilling the responsibilities set out in the Policy on Evaluation. It reviews and approves evaluation-related matters; it meets policy expectations in terms of its composition; and its stated roles and responsibilities meet TB expectations.
4.1.3 Head of Evaluation Responsibilities: Conforms to Expectations
The Head of Evaluation is fulfilling the HoE responsibilities related to governance. EPMRB conducts its work in a neutral, cost-effective manner; it supports the departmental evaluation committee; and evaluators adhere to the TB evaluation policy suite.
4.1.4 Values and Ethics: Conforms to Expectations
Evaluators are involved in a wide range of activities that require them to act in a professional and ethical manner; for example, they interact with groups all across the Department, they interact with AANDC's outside stakeholders, they initiate and manage contracts, and they have access to privileged information.
Interviews and reviewed documents indicate that EPMRB behaves professionally and in a manner consistent with federal government values and ethics.
4.2 The Evaluation Function's Professional Practice
The neutral assessment's professional practice criteria address: planning and conducting work at the function level; planning, conducting and reporting at the project level; and managing the evaluation function. The summary findings are presented below.
4.2.1 Planning and Conducting at the Function Level
4.2.1.1 Evaluation Planning at the Function Level: Conforms to Expectations
Measures are in place related to the evaluation plan. Documentation indicates that:
- Adequate measures are in place for the development of the Department's five-year evaluation plan.
- The five-year evaluation plan is approved and communicated.
- The most recent evaluation plan presents a five-year strategy for 88 percent coverage over the next five years. The coverage decisions are suitably explained.
- It would be useful for the plan to identify how evaluations completed in recent years have contributed to coverage. This perspective would provide the Deputy Minister with an understanding of the likelihood of achieving full coverage over a five year period.
4.2.1.2 Roles Related to Evaluation and Performance Measurement: Conforms to Expectations
The assessment analysis determined that the evaluation function is meeting expectations related to conducting evaluations. The evaluation projects address core issues and they count towards coverage requirements. Information on coverage is provided to Treasury Board Secretariat (TBS) as part of the annual Management Accountability Framework (MAF) exercise. The TBS Centre of Excellence for Evaluation (CEE) reviews evaluation project reports and other products, and has indicated to EPMRB that the reports reviewed meet expectations. CEE also monitors the extent to which completed evaluations contribute to coverage. This provides assurance on the quality of evaluations and the extent to which coverage is being achieved.
The assessment analysis determined that the evaluation organization is meeting expectations related to departmental performance measurement. EPMRB is participating actively in a department-wide initiative to improve AANDC's practices and processes related to performance measurement and reporting. As set out in TB expectations, the HoE comments on performance measurement strategies, reviews Cabinet documents, and provides advice on departmental performance measurement strategies and frameworks. A process is in place for the HoE to receive all TB submissions and Cabinet documents for review.
With respect to performance measurement at the function level, deputy heads are to ensure that that ongoing performance measurement is implemented throughout the Department so that sufficient performance information is available to effectively support the evaluation of programs. Reviewed evaluation projects indicated that performance information was available to support the majority of those projects. Also, EPMRB is leading a department-wide Performance Measurement Strategy Action Plan that is intended to, among other objectives, identify and address gaps in the performance information needed to support evaluations.
4.2.2 Planning, Conduct, and Reporting at the Project Level
To address this sub-section of the neutral assessment, the evaluation organization was asked to identify what they considered to be their best completed evaluation projects since the implementation of the 2009 Policy on Evaluation. The following assessment is based on the analysis of documentation provided on six projects: four selected from the best projects identified by EPMRB, and two additional projects chosen at random. The projects were:
- Impact Evaluation of the ecoENERGY for Aboriginal and Northern Communities
- Impact Evaluation of Treaty Related Measures in British Columbia
- Evaluation of the Family Violence Prevention Program
- Summative Evaluation of the Elementary/Secondary Education Program on Reserve
- Impact Evaluation of the Urban Aboriginal Strategy
- Evaluation of the Government of Canada Program for International Polar Year
The neutral assessment analysis at the project level includes: consultation, project planning, conduct, reporting, and use.
4.2.2.1 Consultation on Evaluation Projects: Conforms to Expectations
Consultations took place during the design and implementation of evaluation projects. These consultations involved program personnel in the planning and scoping of evaluations, and in developing methodologies that would be used in the evaluation research.
4.2.2.2 Evaluation Project Planning : Generally Conforms to Expectations
According to theTB Standard on Evaluation for the Government of Canada, plans should discuss approaches, methodologies, roles and responsibilities, and project costing.
The analysis of project information determined that formal project plans (e.g., evaluation frameworks) were done for every project, and that important information on approaches and methodologies is included in these plans. However, information on project budgeting and team roles and responsibilities could be provided more fully and consistently in project plans.
4.2.2.3 Conduct of Evaluation Projects: Conforms to Expectations
The conduct of evaluation projects includes project leadership, stakeholder consultation, project management and control, use of evidence, and information control.
Evaluation projects are led by AANDC's HoE, and stakeholders are consulted as part of the research methodologies. Evaluation reports indicate that conclusions are based on evidence.
Practices are in place related to project management and control, and information control.
4.2.2.4 Evaluation Project Reporting: Conforms to Expectations
Project reporting includes the completeness and communication of evaluation reports, and their accessibility to the public.
The reports reviewed are clear, meet the requirements of the Standard on Evaluation, include management responses and action plans, and have been submitted to TBS.
Completed evaluation reports are posted on AANDC's website.
4.2.2.5 Use of Evaluations: Conforms to Expectations
This requirement includes use of evaluation findings by the deputy head and the Department, and availability of evaluation information to key stakeholders such as the Minister, Parliament, and Canadians.
Completed evaluation studies are posted on AANDC's website and, therefore, available to all.
Interviews indicated that the Deputy Minister and many Assistant Deputy Ministers (ADMs) have used evaluation information in program decision-making. Some ADMs could cite specific examples of their use of evaluation findings. Further, ADMs who were less familiar with evaluations also indicated that they believe the function adds significant value to the Department. Some senior managers indicated that the evaluation analyses "could have gone further", suggesting that they value evaluation, but still had unmet analytical needs, which they were pursuing within their own organizations.
The ADMs who had worked with the HoE emphasized the considerable value added that she brought to the evaluation function. They highlighted that their positive relationship with the HoE contributed significantly to the usefulness of evaluation studies.
Overall, interviewees indicated that they value the evaluation function (including those who are still learning about it) and anticipate that evaluation studies will be useful to them.
4.2.3 Evaluation Competencies: Conforms to Expectations
The evaluation projects examined were conducted by EPMRB evaluation personnel and consulting firms that had demonstrated their competencies in the area of evaluation. Therefore, projects were undertaken by competent personnel.
4.2.4 Improvement to Products and Services: Conforms to Expectations
The reviewed projects indicate that the EPMRB has well-established management practices. Interviews with ADMs indicated that EPMRB representatives have met with them to consult about the evaluation plan, to discuss the role of evaluation, and discuss areas where they could add value to departmental senior managers and their organizations.
4.3 Monitoring and Performance Reporting of the Evaluation Function
The neutral assessment's Monitoring and Performance Reporting criteria address annual and ongoing reporting obligations.
4.3.1 Annual Report on the State of Performance Measurement: Conforms to Expectations
These reports have been produced for the 2009/10, 2010/11 and 2012/13 fiscal years.
4.3.2 Monitoring and Performance Reporting by the Deputy Heads: Conforms to Expectations
The Deputy Minister has indicated that he has been well served by the evaluation function, and that it has considerable influence in the Department. This neutral assessment is a component of the performance reporting activity. Also, the HoE provides TBS with information annually on behalf of the deputy head.
5.0 Recommended Actions
The neutral assessment identified process improvements related to evaluation planning. There are no other recommended actions required to improve conformance.
Interviewees indicated that their most significant current challenge is for their organizations to find the time necessary to support an evaluation study. They indicated that, with recent resource reductions, it has become even more important for evaluation studies to focus on addressing the highest value-added topics and issues, to obtain the highest return on the time and effort invested in an evaluation.
Further, interviews indicated that evaluation analyses would be even more comprehensive and useful if they relied on a broader diversity of information sources. As an example, AANDC’s longstanding evaluation practice of relying heavily on case studies could be complemented by in-depth literature reviews. These and other methods would help to enhance the discussion of the technical and conceptual issues that affect evaluation findings and conclusions.
Appendix
Aboriginal Affairs and Northern Development Canada
Framework for the Neutral Assessment of the Evaluation Function
Objectives of the Neutral Assessment
To address the following questions:
- Does the evaluation function conform to Treasury Board’s expectations?
- Is the evaluation organization well-managed?
- Is the function adding value to AANDC?
| A | Governance There is conformance with the TB Policy on Evaluation requirements regarding Governance |
|
| A.1 | Values and Ethics There is conformance with the TB Standard on Evaluation for the Government of Canada requirements regarding Values and Ethics |
|
| A.1.a | The head of evaluation ensures that evaluations are conducted in a professional and ethical manner | |
| 1 | Evaluators behave in a manner that is consistent with the "Values and Ethics Code for the Public Service." | S 6.3.1.1 |
| 2 | Any contract by which an external evaluator (i.e. consultant) is engaged to provide evaluation services is to include a requirement that the "Values and Ethics Code for the Public Service" be complied with. | S 6.3.1.1 |
| 3 | Individuals performing evaluation work act in a neutral manner and with integrity in their relationships with stakeholders. | S 6.3.1.2 |
| 4 | Any conflict of interest is dealt with openly, honestly, and promptly so that it does not compromise evaluation processes or results. | S 6.3.1.3 |
| A.2 | Deputy Head There is conformance with the TB Policy on Evaluation requirements regarding the responsibilities of the Deputy Head |
|
| A.2.a | The deputy head has fulfilled his/her responsibilities in regard to the evaluation function | |
| 1 | Deputy heads are responsible for establishing a robust, neutral evaluation function in their department and for ensuring that their department adheres to this policy and to its supporting directive and standard. | P 6.1 |
| A.2.b | The deputy head has fulfilled his/her responsibilities in regard to the Departmental Evaluation Committee | |
| 1 | Deputy heads ensure that a committee of senior departmental officials (referred to as the Departmental Evaluation Committee) is assigned the responsibility for advising the deputy head on all evaluation and evaluation-related activities of the Department. | P 6.1.3 |
| 2 | Deputy heads ensure that the departmental evaluation committee has full access to information and documentation needed or requested to fulfill its responsibilities. | P 6.1.9 |
| A.2.c | The deputy head has fulfilled his/her responsibilities in regard to the head of evaluation | |
| 1 | Deputy heads designate a head of evaluation at an appropriate level as the lead for the evaluation function in the Department. | P 6.1.1 |
| 2 | Heads of evaluation are required to meet competency requirements, as specified by the Treasury Board of Canada Secretariat. | D 6.1.1.1 |
| 3 | Deputy heads ensure that the head of evaluation has direct and unencumbered access to the deputy head, as required. | P 6.1.2 |
| 4 | Deputy heads ensure that (the departmental evaluation committee and – addressed in A2b2) the head of evaluation have full access to information and documentation needed or requested to fulfill their responsibilities. | P 6.1.9 |
| A.2.d | The deputy head has fulfilled his/her responsibilities in regard to the operations of the evaluation function | |
| 1 | Deputy heads approve for annual submission to the Treasury Board of Canada Secretariat, a rolling five-year departmental evaluation plan. | P 6.1.7 |
| 2 | Deputy heads confirm that the departmental evaluation plan:
|
P 6.1.7 |
| 3 | Deputy heads approve evaluation reports, management responses and action plans in a timely manner. | P 6.1.4 |
| A.3 | Departmental Evaluation Committee There is conformance with the TB Policy on Evaluation requirements regarding the departmental evaluation committee |
|
| A.3.a | The departmental evaluation committee has fulfilled its structural responsibilities | |
| 1 | Deputy head ensures that the departmental evaluation committee is:
|
P 6.1.3 |
| A.3.b | The departmental evaluation committee has fulfilled its role and responsibilities | |
| 1 | Role of the Departmental Evaluation Committee This committee serves as an advisory body to the deputy head related to the departmental evaluation plan, resourcing, and final evaluation reports and may also serve as the decision-making body on other evaluation and evaluation-related activities of the Department |
P Annex B |
| 2 | Responsibilities of the Departmental Evaluation Committee
|
P Annex B |
| A.4 | Head of Evaluation There is conformance with the TB Policy on Evaluation requirements regarding the head of evaluation |
|
| A.4.a | The head of evaluation has fulfilled his/her responsibilities in regard to governance | |
| 1 | The head of evaluation is responsible for managing an efficient and effective evaluation unit. | D 6.1.2.1 |
| 2 | The head of evaluation ensures that the evaluation unit conducts evaluations in a neutral, cost-effective manner. | D 6.1.2.1 |
| 3 | The head of evaluation is responsible for supporting the Departmental Evaluation Committee, that is assigned the responsibility for guiding and overseeing the evaluation function. | D 6.1.2.4 |
| 4 | The head of evaluation is responsible for ensuring that evaluation work and persons involved in designing, conducting and managing evaluations are informed of and adhere to the Policy on Evaluation, the Directive on the Evaluation Function, and the Standard on Evaluation for the Government of Canada. | D 6.1.2.2 |
| B | Professional Practice There is conformance with the TB Policy on evaluation requirements regarding Professional Practice |
|
| B.1 | Departmental Evaluation Plan There is conformance with the TB Policy on Evaluation requirements regarding the Departmental Evaluation Plan |
|
| B.1.1 | Evaluation Plan There is conformance with the TB Policy on evaluation requirements regarding the evaluation plan |
|
| B.1.1.a | Adequate measures are in place for the development of the evaluation plan pending the Department’s full implementation of the Policy on Evaluation | |
| 1 | Pending the full implementation of Section 6.1.8 (a)Footnote 2, which can be no later than March 31, 2013, deputy heads will ensure that approved departmental evaluation plans demonstrate progress towards achieving coverage of all departmental direct program spending (excluding ongoing programs of grants and contributions) over five years | P 1.3.1.a |
| 2 | Pending the full implementation of Section 6.1.8 (a), which can be no later than March 31, 2013, deputy heads will ensure that departmental evaluation plans that they approve for submission to the Treasury Board of Canada Secretariat as per Section 6.1.7Footnote 3 of this policy, and that do not demonstrate evaluation coverage of all direct program spending over the ensuing five-year period, use a risk-based approach to planning coverage of direct program spending (excluding ongoing programs of grants and contributions.) | P 1.3.1.b |
| 3 | Pending the full implementation of Section 6.1.8 (a), which can be no later than March 31, 2013, deputy heads will ensure that … Departmental evaluation plans that use a risk-based approach to planning coverage of direct program spending will include a written, risk-based rationale to explain the Department's evaluation coverage and non-coverage choices. | P 1.3.1.b |
| B.1.1.b | Adequate measures are in place for the development of the evaluation plan | |
| 1 | The Deputy head ensures that the following evaluation coverage requirements are met and reflected in the departmental evaluation plan:
|
P 6.1.8 |
| 2 | The head of evaluation is responsible for developing a rolling five-year departmental evaluation plan (hereafter referred to as the departmental evaluation plan), and updating the plan annually. | D 6.1.3.a |
| 3 | The head of evaluation is responsible for ensuring that their departmental evaluation plans:
|
D 6.1.3.b |
| 4 | The head of evaluation is responsible for identifying and recommending to the deputy head and the Departmental Evaluation Committee a risk-based approach for determining the evaluation approach and level of effort to be applied to individual evaluations comprised in the five-year departmental evaluation plan, and the appropriate level of resources required to conduct individual evaluations included in the plan. | D 6.1.3.c |
| B.1.1.c | The evaluation plan is approved and communicated | |
| 1 | The head of evaluation is responsible for: submitting the departmental evaluation plan, as approved by the deputy head, to the Treasury Board of Canada Secretariat at the beginning of each fiscal year. | D 6.1.3.d |
| 2 | The head of evaluation is responsible for submitting the … confirmation from the deputy head as required by Section 6.1.7 of the Policy on Evaluation. | D 6.1.3.d |
| B.2 | Nature of Evaluation There is conformance with the TB Policy on Evaluation requirements regarding the Nature of Evaluation |
|
| B.2.1 | Evaluation Roles and Responsibilities There is conformance with the TB Policy on Evaluation requirements regarding the evaluation function’s role and responsibilities |
|
| B.2.1.a | The evaluation function has fulfilled its responsibilities in regard to conducting evaluations | |
| 1 | The head of evaluation is responsible for: implementing the approved departmental evaluation plan while ensuring the timely completion of individual evaluations, in accordance with the Standard on Evaluation for the Government of Canada | D 6.1.3.e |
| 2 | To address value for money, evaluations will be required to assess all core issues identified below:
|
D Annex A |
| 3 | The head of evaluation is responsible for ensuring that all evaluations that are intended to count toward the coverage requirements of subsections "a", "b", or "c" of Section 6.1.8 of the Policy on Evaluation, include clear and valid conclusions about the relevance and performance of programs by addressing all core issues outlined in Annex A. | D 6.1.3.f |
| B.2.1.b | The evaluation function has fulfilled its responsibilities in regard to departmental performance measurement | |
| 1 | The deputy head ensures that the head of evaluation is consulted on the performance measurement strategies embedded in the Department's Management, Resources and Results Structure. | P 6.1.11 |
| 2 | The head of evaluation is responsible for reviewing and providing advice on the performance measurement strategies for all new and ongoing direct program spending, including all ongoing programs of grants and contributions, to ensure that they effectively support an evaluation of relevance and performance. | D 6.1.4 |
| 3 | The head of evaluation is responsible for reviewing and providing advice on the accountability and performance provisions to be included in Cabinet documents (Memoranda to Cabinet, Treasury Board submissions). | D 6.1.4 |
| 4 | The head of evaluation is responsible for reviewing and providing advice on the performance measurement framework embedded in the organization's Management, Resources and Results Structure. | D 6.1.4 |
| B.3 | Evaluation Project Process There is conformance with the TB Policy on Evaluation requirements regarding the Evaluation project process |
|
| B.3.1 | Managing the Evaluation Function There is conformance with the TB Policy on Evaluation requirements regarding managing the evaluation function |
|
| B.3.1.a | Appropriate consultation is taking place | |
| 1 | In managing the evaluation function in departments, the head of evaluation consults appropriately with program managers, stakeholders, and peer review or advisory committees during evaluation project design and implementation. | D 6.2.1 |
| 2 | Peer review, advisory, or steering committee groups are used where appropriate to input to evaluation planning and processes and to the review of evaluation products to improve their quality. The head of evaluation or the evaluation manager directs these committees. | S 6.1.2.2 |
| B.3.1.b | Adherence to the Policy, Directive and Standard | |
| 1 | The head of evaluation is responsible for ensuring that evaluation work and persons involved in designing, conducting and managing evaluations are informed of and adhere to the Policy on Evaluation, the Directive on the Evaluation Function, and the Standard on Evaluation for the Government of Canada. | D 6.1.2.2 |
| B.3.2 | Evaluation Project Planning Phase There is conformance with the TB Policy on Evaluation requirements regarding the Planning Phase |
|
| B.3.2.a | Evaluation project plans are developed and recorded | |
| 1 | Evaluation planning and design requirements ensure that the basic elements of a good evaluation are in place at the outset of an evaluation. | S 6.2 |
| Evaluation procedures should be practical and responsive to the way the program operates. | JCSEE F2Footnote 4 | |
| 2 | Evaluations are cost-effective and risk-based:
|
S 6.2.1 |
| 3 | Standard on Measurement and Analysis
|
S 6.2.2 |
| Evaluations should employ technically adequate designs and analyses that are appropriate for the evaluation purposes. | JCSEE A6 | |
| 4 | Roles and responsibilities of project team members involved in specific evaluations are articulated in writing and agreed upon at the outset of the evaluation. | S 6.1.1.2 |
| 5 | In planning and conducting the evaluation, those involved in the evaluation comply with government policies related to information collection, use, preservation and dissemination. | S 6.3.4.1 |
| Evaluations should employ systematic information collection, review, verification, and storage methods. | JCSEE A5 | |
| B.3.2.b | Evaluation projects conform with the Policy on Government of Canada Communications | |
| 1 | The head of evaluation is responsible for consulting with the departmental head of communications on evaluation work that could potentially be considered "public opinion research" as defined in the Policy on Government of Canada Communications. | D 6.1.2.3 |
| B.3.3 | Conduct of Evaluation Project Phase There is conformance with the TB Policy on Evaluation requirements regarding the conduct of evaluations |
|
| B.3.3.a | Leading evaluation projects | |
| 1 | The head of evaluation is responsible for directing evaluation projects. | S 6.1.1.1 |
| B.3.3.b | Stakeholder involvement in evaluation projects | |
| 1 | Evaluations incorporate sufficient and appropriate consultation, including with major stakeholders, and, where appropriate, apply the advice and guidance of specialists and other knowledgeable persons. | S 6.1.2 |
| B.3.3.c | Evaluation projects are managed and controlled | |
| Sound fiscal decision-making is employed in the management of evaluation projects so that expenditures are accounted for, and the Government of Canada receives good value for its money. | S 6.3.3.1 | |
| Evaluations should use effective project management strategies. | JCSEE F1 | |
| Evaluations should use resources effectively and efficiently. | JCSEE F4 | |
| Evaluations should account for all expended resources and comply with sound fiscal procedures and processes. | JCSEE P7 | |
| When engaging external resources, evaluation projects comply with Treasury Board contracting policies. | S 6.3.3.2 | |
| When engaging external resources, contracts by which an external evaluator is engaged to provide evaluation services include a requirement to comply with this standard. | S 6.3.3.3 | |
| B.3.3.d | There is sufficient and appropriate evidence to achieve the evaluation project objectives and to support conclusions and engagement results | |
| 1 | The deputy head ensures that ongoing performance measurement is implemented throughout the Department so that sufficient performance information is available to effectively support the evaluation of programs. | P 6.1.10 |
| 2 | Evaluators must identify, analyze, evaluate, and document sufficient information to achieve the project’s objectives. | Based on IIAFootnote 5 2300 |
| 3 | Evaluators must identify sufficient, reliable, relevant, and useful information to achieve the engagement’s objectives. | Based on IIA 2310 |
| 4 | Evaluators must base conclusions and project results on appropriate analyses. | Based on IIA 2320 |
| B.3.3.e | Relevant information is documented, maintained and controlled | |
| 1 | Stakeholders and participants in an evaluation are informed of the level of confidentiality and privacy that is afforded them under the Privacy Act. | S 6.3.4.2 |
| 2 | Evaluators must document relevant information to support the conclusions and engagement results. | Based on IIA 2330 |
| 3 | The Head of Evaluation must develop retention requirements for engagement records, regardless of the medium in which each record is stored. These retention requirements must be consistent with the organization’s guidelines and any pertinent regulatory or other requirements. | Based on IIA 2330.A2 |
| B.3.4 | Reporting Phase There is conformance with the TB Policy on Evaluation requirements regarding the Reporting Phase |
|
| B.3.4.a | Evaluation reports are complete and appropriately communicated | |
| 1 | The head of evaluation is responsible for issuing evaluation reports (and other evaluation products, as appropriate) directly to the deputy head and the Departmental Evaluation Committee in a timely manner. | D 1.2.5.6 |
| 2 | Evaluation reporting requirements ensure that reports present the findings, conclusions and recommendations in a clear and neutral manner. | S 6.4 |
| Evaluations should document programs and their contexts with appropriate detail and scope for the evaluation purposes. | JCSEE A4 | |
| 3 | Evaluation reports are written so that senior managers and external readers can readily focus on and understand the important issues being reported. Evaluation reports:
|
S 6.4.1 |
| 4 | The head of evaluation is required to submit complete evaluation reports, management responses and action plans (in electronic format) to the Treasury Board of Canada Secretariat immediately upon approval of these reports by the deputy head. | D 6.1.5.1 |
| B.3.4.b | Evaluation reports are accessible to the public | |
| 1 | The deputy head ensures that complete, approved evaluation reports along with management responses and action plans are made easily available to Canadians in a timely manner while ensuring that the sharing of reports respects the Access to Information Act, Privacy Act, and the Government Security Policy. | P 6.1.6D |
| The head of evaluation is required to: make approved evaluation reports, along with the management responses and action plans, available to the public with minimal formality and post them on departmental websites in both official languages in a timely manner following their approval by the deputy head. | 6.1.5.2 | |
| The deputy head ensures that evaluation reports posted on departmental websites respect the Access to Information Act, Privacy Act, and the Government Security Policy. | D 6.1.5.3 | |
| B.3.5 | Use of Evaluations There is conformance with the TB Policy on Evaluation requirements regarding the use of evaluation information |
|
| B.3.5.a | Deputy heads use evaluation results | |
| 1 | The deputy head uses evaluation findings to inform program, policy, resource allocation and reallocation decisions. | D 6.1.1.7 |
| 2 | Impacts of evaluations, including savings or improvements produced, are reported to the deputy head. | S 6.1.3.1 |
| B.3.5.b | Evaluation information is available for use by others | |
| 1 | Evaluation information on the ongoing relevance and performance of direct program spending is available to ministers, departments and central agencies and used to support evidence-based decision-making on policy, expenditure management and program improvements. | S 6.4.2.1 |
| 2 | Evaluation information on the ongoing relevance and performance of direct program spending is available to Parliament and Canadians to support government accountability for results achieved by policies and programs. | S 6.4.2.2 |
| B.4 | Evaluation Competencies There is conformance with the TB Policy on Evaluation requirements regarding the proficiency and professional care |
|
| B.4.a | The head of evaluation ensures that evaluation resources have the training, competencies and experience to perform their responsibilities | |
| 1 | The head of evaluation is required to ensure that the person or persons involved in designing, conducting or managing evaluations has, or collectively have the appropriate training, required competencies and experience in evaluation. | D 6.1.1.2 |
| The person or persons carrying out the evaluation or evaluation-related work possesses or collectively possess the knowledge and competence to carry out the evaluation. | S 6.3.2.1 | |
| Evaluations should be conducted by qualified people who establish and maintain credibility in the evaluation context. | JCSEE U1 | |
| B.5 | Quality Assurance and Improvement There is conformance with the TB Policy on Evaluation requirements regarding Quality Assurance and Improvement |
|
| B.5.a | The head of evaluation ensures the development and maintenance of the quality assurance and improvement program | |
| 1 | Ongoing, systematic inquiry is employed to identify future improvements to evaluation products and services. | D 6.1.3.2 |
| HoEs are responsible for developing and maintaining a quality assurance and improvement program that covers all aspects of the Evaluation function and continuously monitors its effectiveness. | Based on IIA 1300 |
|
| Periodic reviews performed through self-assessment or by other persons within the organization with sufficient knowledge of evaluation practices. | Based on IIA 1311 |
|
| Evaluators should use these and other applicable standards to examine the accountability of the evaluation design, procedures employed, information collected, and outcomes. | JCSEE E2 | |
| C | Monitoring and Performance Reporting There is conformance with the TB Policy on Evaluation requirements regarding Monitoring and Performance Reporting |
|
| C.1 | Head of Evaluation’s Annual Report | |
| C.1.a | The head of evaluation has submitted his/her annual reports to the departmental evaluation committee | |
| 1 | The head of evaluation is responsible for submitting to the Departmental Evaluation Committee an annual report on the state of performance measurement of programs in support of evaluation. | D 6.1.4.4 |
| C2 | Deputy Head There is conformance with the TB Policy on Evaluation requirements regarding Monitoring and Performance Reporting |
|
| C.2.a | The deputy head has fulfilled his/her responsibilities in regard to monitoring and performance reporting | |
| 1 | The deputy head is responsible for monitoring compliance with this policy in their departments to ensure its effective implementation. | P 7.1 |
| 2 | The deputy head is responsible for ensuring that a neutral assessment of their departmental evaluation function is conducted at a minimum of once every five years. | P 7.1 |
| 3 | The deputy head is responsible for addressing issues that arise regarding compliance with this policy, and with its associated directive and standard, and ensuring that appropriate remedial actions are taken to address these issues. | P 7.2 |
| C.2.b | The deputy head has the information to respond to the Secretary of the Treasury Board | |
| 1 | The Secretary of the Treasury Board … may require the deputy head to provide information on the following:
|
P 7.3 |