Lessons Learned Study of the Common Experience Payment Process

Introduction

Throughout the duration of a project, lessons are learned and opportunities for improvement are brought to light. As part of a process that is designed or seeks to ensure continuous improvement, documenting lessons learned helps project administrators discover the sources of problems that arose and avoid those problems in later project stages or future projects. Best practices are also determined for the benefit of similar projects in the future.

This report is the final output of the Lessons Learned Study of the Common Experience Payment (CEP), which examined the response of the Office of Indian Residential Schools Resolution Canada (OIRSRC) to the March 2007 court approval of the Indian Residential Schools Settlement Agreement (Settlement Agreement). The Study objective was to determine, to the extent possible, what went right and what went wrong in the design and implementation of the response to the Settlement Agreement and to present its findings in this report. It was also to present information relevant to better planning of future projects and the prevention or minimization of risks as well as improvement of new project implementation.

The Study, undertaken by the Evaluation, Performance Measurement and Review Branch (EPMRB) of Indigenous and Northern Affairs Canada (INAC), analyzed information from multiple sources. First, key internal documents and files were examined, such as assessments conducted by operational areas within the OIRSRC to determine the eligibility of applicants. Second, interviews were conducted with available staff involved in the CEP delivery. Third, reviews of literature relative to projects similar to the CEP (in Canada and internationally) were undertaken. Finally, administrative and financial documents were examined.

CEP Background

The Royal Commission on Aboriginal Peoples in its 1996 report brought the experiences of former Aboriginal students at Indian Residential Schools (IRS) to national attention. In 2001, the Office of Indian Residential Schools Resolution Canada (OIRSRC) was created as a separate department with a mandate to address the legacy of the IRS and to manage and resolve claims. The OIRSRC was thus responsible for administering claims made by former students of the IRS system. According to a 2005 report by Siggner & Associates Inc. entitled "Estimating the Residential School Attendee Population for the Years 2001, 2005 and 2006," there were an estimated 78,994 former residential school students alive as of 2006.

In May 2006, the Government of Canada announced the approval by all parties of the Settlement Agreement and in March 2007, the Settlement Agreement received approval from nine courts across Canada. Under the Agreement, a Trust Fund of $1.9 billion was established for the Common Experience Payment, which provided eligible former students $10,000 for the first year (or part thereof) of their residence at an eligible IRS plus $3,000 for each additional year (or part thereof). As per the Settlement Agreement, if more than $40 million remained in the Trust Fund after all CEP had been paid, funds would be made available to CEP recipients, upon application, in the form of non-cash Personal Credits of up to $3,000 each for educational purposes.

In August 2007, the parties to the Settlement Agreement endorsed the CEP Validation Protocol.

In reading this report, it is necessary to understand the environment in which the OIRSRC had to implement the CEP and the challenges staff faced as a result. For example, once the court decision came down in March 2007, there was a period of consultations with the National Certification Committee to formalize a framework of processes that conformed to the Settlement Agreement. It was in late August 2007 that this was finalized, a month before the launch of the CEP. Thus, the CEP process had to be developed and implemented within a very short timeframe and deal with various challenges. Regional OIRSRC departmental personnel also encountered numerous challenges in administering the initiative. Nevertheless, the OIRSRC dealt with these issues as necessary within the timeframe of the CEP.

Out-of-scope issues

The Study was based on the OIRSRC's role as defined in its Memorandum of Understanding with Service Canada, as both organizations were responsible for administering the CEP. However, the study did not focus on questions relating to the relevance or effectiveness of the CEP for a number of reasons, including the fact that, as a court ordered settlement, the Government had a legal obligation to implement all components of the Settlement Agreement. A conscious effort was also made to avoid subjecting CEP recipients to multiple data collection processes, which could be burdensome to them and be misconstrued as re-victimizing former students. Moreover, a study completed by the Aboriginal Healing Foundation had considered the perspective of former students and an evaluation by Service Canada had addressed the relevance and effectiveness of the CEP.

Overall Lessons Learned Study Findings

The Settlement Agreement was the first and largest class action settlement of its kind in Canada's history. It tested the OIRSRC's abilities, and it is important to acknowledge at the outset that staff worked extremely hard in the delivery of the CEP. They demonstrated excellent professionalism and commitment and their dedication to responding to the needs of former students is commendable. As the Study found, the overall message directing the CEP response was to "get it done" in order to ensure that Canada lives up to its obligations and commitments to former students and their families.

What worked well?

Overall, implementation of the CEP can be viewed positively, recognizing that there were many challenges to be overcome. It is from these challenges that lessons can be learned to improve future initiatives.

Financial controls / forms and procedures

Based on documents reviewed, within the OIRSRC Oracle Application Software Implementation Strategy financial system were two key financial control points in the validation and financial payment process: the Financial Administration Act Section 34 and Section 33 approvals, where a s.34 sign-off confirmed that the payee was eligible for or entitled to the payment and a s.33 sign-off authorized the payment of the $8,000 provided to claimants as part of the Advance Payment process, which was to be deducted from any future CEP payment. A pre-audit was also performed by Service Canada to ensure appropriate control points were part of the Section 34 approvals.

Operational management

Key structures, resources, relationships, plans and tools were in place. However, when the OIRSRC resources were overwhelmed and unable to meet the huge demands, for instance, reconsideration requests, OIRSRC officials initiated a ramp-up of resource levels, and increased its use of private sector contractors for assistance in an effort to meet pressing needs.

Staff also showed remarkable dedication and endurance and an ability to respond rapidly to changing circumstances. The CEP staff's capacity to respond to pressing events was a crucial factor in maintaining momentum and reaching key milestones (e.g., the delivery of cheques to former students). This is perhaps best seen in light of two unanticipated challenges: the sudden avalanche of applications that were submitted and the Minister's instruction in the month of November 2007 that CEP officials complete 53,000 files by December 22nd. The ability of CEP staff to respond demonstrates the OIRSRC's ability to maintain a strong field operation and shift strategies as circumstances changed.

Outreach

Involving the community early in the implementation of the response to the Settlement Agreement proved effective. Local community members were helpful in contacting or assisting applicants and this was widely seen by study key informants as beneficial in terms of outreach. For example, CEP outreach efforts helped to ensure that the Indigenous community was well informed of the benefits available to former IRS students through the Settlement Agreement.

Awareness Initiative

Another example of involving the community early was the awareness initiative aimed at eligible former students at the beginning of the CEP. This informed them of the importance of having bank accounts, to exercise caution and to be aware of predatory behaviour by those who might try to mislead or take advantage of them. This was a key initiative and as a result, the chances of recipients being misled were controlled or at least limited to some extent.

According to study interviewees, the awareness initiative was a success. This was demonstrated when a leading member of the Indigenous community came to the OIRSRC to, symbolically, make the first call to the first recipient when the first cheque was to be delivered. Anecdotal evidence suggests that when this first call was made, the recipient on the other end of the line responded "You're a scam artist," which shows the effectiveness of the awareness initiative. It was also noted that the Royal Canadian Mounted Police and others were involved in protecting former IRS students from harm.

Communication

Communications with staff began early and was extensive. Nonetheless, the geographical separation of departmental offices created difficulties and challenges and there were times that those in the Vancouver office did not receive information that was meant for all CEP administrators. For example, during the late summer and early fall of 2007, documents and interviews show that personnel and organizational changes led to a communication breakdown such that information was not reaching all the National Research and Analysis Unit staff. However, over time these types of difficulties were addressed and communication improved.

Improvement was apparent in the documents reviewed, which showed that best practice sessions were used by the OIRSRC as one of the controls to improve communications. For example, CEP Research "Best Practices" sessions with all CEP researchers and staff were held to clarify process questions and identify approaches to improving overall productivity.

Another key accomplishment of the CEP was the well-organized administrative and governance processes. The departmental administration brought together the key decision makers to ensure accountability, tracking, and ultimate success. For example, senior management maintained regular communication with staff, monitoring progress and management as part of meeting the public and fiscal responsibility for the CEP. Because it was primarily a manager who reported to, and was accountable to, senior management, developing a positive rapport greatly facilitated the process. For example, the study found that the Vancouver office director frequently traveled between Vancouver and the National Capital Region and, to the extent possible, kept the Vancouver regional staff apprised of CEP developments in the National Capital Region and vice versa.

Primary Challenges

The CEP implementation challenges involved a combination of factors, including, for example, an Information Technology (IT) system that required significant work and a changing business environment. In fact, multiple factors operating in parallel created multiple challenges for the CEP. Overall, and to varying degrees, the Study found that though the CEP process met key implementation objectives, some areas in management and administration, as described below, needed attention but were not adequately addressed during the implementation phase of the CEP.

Information Technology

A critical element of the CEP response, IT, was executed late, ineffectively or not at all in the initial stages, which affected timeliness in the CEP delivery. There were two departments involved, Service Canada, which was much bigger with multiple critical service delivery requirements and experience and one, the OIRSRC, which was small with a singular focus. Implementing IT changes at Service Canada meant following rigid protocols, especially when making changes to template response letters.

In the CEP preparation phase, adequate time was needed for such an especially large, complex and national initiative. Recognizing the well-known consequences of any tardiness in delivering cheques to former students, major efforts at establishing key structures should have been made earlier, prior to the Settlement Agreement being approved, i.e. no later than August 2007, a month before the CEP implementation. To compound the situation, the OIRSRC around the same time was in the preparation stage of a machinery of government change process whereby it was to be incorporated into the much bigger Department of Indian and Northern Affairs Canada, now Indigenous and Northern Development Canada (INAC).

The IT challenges also affected coordination between the geographically separated CEP staff. IT tools are useful for integrating multiple factors and entities in order to ensure communication, uniformity and ease in the execution of tasks and other responsibilities. The study found, however, that due to IT problems, coordination of efforts between the two OIRSRC offices, as an example, suffered in the sense that the standards being used by the different offices were not the same.

The National Administration Committee

The National Administration Committee, an all party committee overseeing the administration of the Settlement Agreement, was the result of a requirement in the Settlement Agreement. The Committee strove to make decisions by consensus but when consensus was not achieved, decisions required a majority vote of five members.  In expanding the research report requirements, the National Administration Committee added an additional process following the first decision (made exclusively by the OIRSRC, reconsideration. To add to this situation, the National Administration Committee heard appeals by unsuccessful applicants, which led to the creation of its own process, and which caused significant delays. The National Administration Committee itself did not anticipate the high number of appeals. The OIRSRC (i.e. the CEP implementers) had to follow up to ensure that no former IRS student "fell through the cracks" as a result of this "alternate process".

Contingency Plan

At the pre-Settlement Agreement approval stage, there was a lack of a contingency plan. The Study did not find any document which demonstrated that a contingency plan was put in place between May 2006, when the Government of Canada announced the approval by all parties of the Settlement Agreement, and March 2007, when the Settlement Agreement received approval from nine courts across Canada. Moreover, most of the OIRSRC responses to the Settlement Agreement were simultaneous and undertaken in real time to ensure meeting the Settlement Agreement's requirements. A contingency plan prior to the Settlement Agreement, even if not finalized, could have proven invaluable to the CEP response effort.

The Settlement Agreement resulted in a massive undertaking to ensure its implementation. The overall conclusion is that the OIRSRC (CEP staff) maintained an efficient internal control in the majority of the key activities involved in delivering the CEP. However, certain areas required corrective action. In order to address these, decisions were made and revisions in practice undertaken that strengthened overall accountability and improved the management of the OIRSRC processes. The CEP is currently in a wind-down phase, expected to be completed when the requests for the addition of new schools to the Settlement Agreement are exhausted before the courts.

1.1 Overview

A Lessons Learned Study of the Common Experience Payment (CEP) was undertaken by the Evaluation, Performance Measurement and Review Branch (EPMRB) of the Audit and Evaluation Sector, Indigenous and Northern Affairs Canada (INAC). The objective of the study was to obtain an independent assessment with regard to how the Office of Indian Residential Schools Resolution Canada (OIRSRC) fulfilled its responsibilities in relation to the CEP, one of the components of the Indian Residential Schools Settlement Agreement. The lessons learned from the CEP experience are meant to help guide and inform any future government initiatives that provide for similar compensation payments.

The Aboriginal Healing Foundation completed studies on the impact of lump-sum compensation payments in 2007^{Footnote 2} and the impact of the CEP on recipients in 2010^{Footnote 3}. In addition, Service Canada (part of Employment and Social Development Canada) completed an evaluation of its CEP responsibilities in May 2013^{Footnote 4}. This INAC Lessons Learned Study will complement these studies by focusing on the OIRSRC response to the Indian Residential Schools Settlement Agreement (hereon referred to as the Settlement Agreement) through examining the design and delivery of its CEP responsibilities. The Study covers the period between 2007-08 and 2012-13.

1.2 Lessons Learned Study Objectives

The overall objective of the Study was to provide the Government of Canada with lessons learned and best practices to inform the design and delivery of possible future compensation payments. A "lessons learned" study is the process of gathering, documenting and analyzing responses to key events that happened during the design and implementation of a program and that may be beneficial to similar programs in the future.

Throughout each project or program's life cycle, lessons are learned and opportunities for improvement discovered by documenting best practices that were developed and implemented. Capturing lessons learned is an integral part of every project or program as part of a continuous improvement process and in helping program administrators discover any root causes of problems that occurred during the delivery of the program. This helps avoid any identical problems in programs that are similar in nature.

1.3 CEP Program Profile/Background

Chart 1 (Memorandum of Understanding)

Text alternative for Chart 1

The chart is composed of 2 overlapping circles creating three distinct sections. The first section identifies and lists the responsibilities of Service Canada which are:

• Invested with sole signing for FAA (Financial Administration Act) Sections 32 and 33
• Receive CEP applications, ensure they were complete and follow-up on missing information, as needed
• document and validate identification based on original birth certificate, or documents such as: driver's license, Indian Status Card, health card, Canadian Passport (with one picture)
• Process applications through specialize processing centres
• Send relevant CEP applications data to AANDC for decision
• When AANDC decision was received, check against Advance Payments reports
• Notify the applicant of the decision
• Notify Public Works and Government Services Canada to issue payment, and
• Provide general information using in person, outreach, call centre, web and mail channels.

The second section (where the circles intersect) represents and lists the 'Joint Responsibilities' of Service Canada and Aboriginal Affairs and Northern Affairs Canada (now Indigenous and Northern Affairs Canada) which are:

• Develop foundational documents
• Provide briefing material related to processing CEP applications and administering the Joint Trust Fund
• Develop systems and procedures to share information as needed to administer the $1.9 Joint Trust Fund
• Response to enquiries, Access to information and Privacy requests and correspondence
• Meet CEP services standards and timelines
• Ensure the privacy of personal information is protected
• Develop and establish the forms and process for CEP applications, and
• Manage and mitigate risks.

The third section represents and lists the responsibilities of Aboriginal Affairs and Northern Development Canada (now Indigenous and Northern Affairs Canada) which are:

• Invested with sole signing authority for FAA Section 34 ( Service Canada sub-delegated its responsibilities for Section 34 to AANDC)
• Provide support and information to inform and report to departmental officials and central agencies
• Respond to acquires and correspondence; examine all documentary evidence available to corroborate attendance at an Indian Residential School
• Validate schools to be part of the CEP
• Inform Service Canada of decisions to provide compensation, and

Inform Service Canada monthly of all settled and outstanding appeals.

 

2.1 Lessons Learned Issues and Questions

A key component of a successful program evaluation is the program lessons learned framework. As per the 2009 Treasury Board Policy on Evaluation, an evaluation is expected to explore five core issues related to relevance (i.e. continuing need for the program, alignment with government priorities, consistency with federal roles and responsibilities) and performance (i.e. achievement of expected outcomes and demonstration of efficiency and economy).

However, this is a special study in that it focuses only on lessons learned and best practices in order to inform any possible future initiatives. Hence, the primary review issues that were considered within the framework were: design and delivery; efficiency (use of resources); lessons learned; and best practices.

Design and Delivery - a good program design guides the implementation process, facilitates monitoring of implementation and provides a solid base for performance evaluation. Issues of program design are mainly assessed by using the program logic model. For example, looking at how the CEP design clearly established linkages among the initiative's components.

Program Efficiency - Efficiency focuses on the conversion of resources (inputs) into outputs that, in turn, support the achievement of expected outcomes.  For example, looking at the outputs produced in the realization of expected results and in relation to the quantity, quality, timeliness and appropriateness of the outputs produced. Also, looking to understand the rationale used by management in their selection of the design and delivery approaches in light of CEP's context, risks and assumptions and analyzing the processes used in creating the outputs. As such, this dimension of program efficiency is closely linked to, and can be supported by, the clear delineation of the program logic and the program theory.

Lessons Learned - This was expected to highlight strengths or weaknesses in preparation, design, and implementation, which often affects performance, outcome and impact.

Best Practices - a proven method, technique, or process for achieving a specific outcome under a specific circumstance and in an effective way (Implementation of the Settlement Agreement contributed to the strategic outcome).

3.1 Objective

The main objective of this Study was to determine lessons learned and best practices based on the Office of Indian Residential Schools Resolution Canada's (OIRSRC) role in the implementation (design and delivery and efficiency) of the CEP.

3.2 Scope

The study covered the period from fiscal year 2007-08 to 2012-13. The scope included the OIRSRC's activities pertaining to the CEP, excluding those undertaken by other departments or agencies who were partners or stakeholders. However, the study recognized the horizontal, multi-department/agency nature of the CEP, as set out in the Settlement Agreement.

The issues were addressed through a series of specific evaluation questions answered in Section 4.0.

3.3 Methodologies

An initial Terms of Reference developed in consultation with the Resolution and Individual Affairs Sector formed the basis for the methodology of the study. The key findings and recommendations presented in this report were drawn from multiple lines of evidence collected and analyzed using the following three methods:

1. Document, file, and data review: documents, files and data provided by CEP officials were reviewed. These included: CEP-related Memorandum of Understanding, Departmental Performance Report, Reports on Plans and Priorities, Speeches from the Throne, quarterly and annual reports. These helped address questions relating to processing times and administrative costs. Data of interest included the number of applications received, processed or rejected; the time it took to verify eligibility, the level of compensation and length of time it took to obtain approvals.
2. Key Informant interviews: telephone and in-person interviews with internal stakeholders and external stakeholders. Nine interviews were conducted with representatives of INAC and Service Canada. While this is a relatively small sample, given the focus on lessons learned and best practices, it is sufficient to draw together ideas and perspectives in relation to informing future initiatives.
3. Literature review and international comparative analysis: a review was undertaken of literature on both national and international practices relating to redress and Settlement Agreements with a focus on such areas as initiating, planning, executing, monitoring and control, and closing. The review examined the strengths and weaknesses of each of these phases. The six comparative analyses undertaken were:
   • Jericho Individual Compensation Program, (British Columbia)
   • Shelburne Youth Training Centre Compensation Program, (Nova Scotia)
   • Grandview Training School for Girls Compensation, (Ontario)
   • Kingsclear Training School Compensation, (New Brunswick)
   • Sir James Whitney School for the Deaf Compensation, (Ontario)
   • Tasmania's Stolen Generations Fund, (Australia)

3.4 Challenges/Limitations and Mitigation Strategies

The study was specifically designed not to elicit experiences of populations like former IRS students. As with any study that relies on the use of methodologies, the expected limitations included:

• Complexity and nature of the CEP and the requirement not to interview CEP applicants for reasons explained earlier (e.g. sensitivity of the subject matter). Moreover, in general, the results of interviews had limitations, for instance, information was based on interviewees' perceptions. The evaluation team also encountered issues regarding the availability and the desire of key informant interviewees to participate in the study.
• Complex and multi-faceted "horizontal" nature of CEP. The CEP was horizontal in that it represented a complex national initiative with significant variability across provinces and involved a wide range of key actors and stakeholder groups. Hence, excluding these and focusing solely on the OIRSRC role was a challenge.
• Inability of the study to verify whether complaints lodged against the CEP call centres were specifically directed at the OIRSRC, Service Canada or CEP lines operated by any other governmental or non-governmental organization.
• Difficulty in isolating the OIRSRC role, given the complexity of implementation and involvement of various partners.

The Court(s)

An Agreement-in-Principle, legally binding on all parties, was approved by the Court in November 2005. The Court continued to supervise the Settlement Agreement. This is important since the approach to delivering the CEP, the result of consultations between the OIRSRC and stakeholders, was subject to court approval before implementation.

The administrative concerns raised by the Court were addressed through the appointment of an amicus curiae, or 'friend of the Court'. This person, funded by the Government, had authority to monitor Canada's compliance with the Settlement Agreement. As such, the amicus had full access to all aspects of the Settlement Agreement, including the administration of the CEP and the Independent Assessment Process, for the sole purpose of reporting to the Court. Thus, the Court, upon receipt of advice and information from the amicus, could require parties to the Settlement Agreement to comply with the terms of the Settlement Agreement if it appeared there was non-compliance.

On the other hand, the Government of Canada managed the funding and implementation of the Settlement Agreement, providing corporate support for all Settlement Agreement obligations, including the Independent Assessment Process, implementation of the Truth and Reconciliation Commission, requests to add new institutions as Indian Residential Schools through Article 12, funding to the Aboriginal Healing Foundation, implementing mental health supports, establishing a Commemoration funding, and the CEP (the OIRSRC and Service Canada being jointly responsible for the administration of the CEP). This meant that where the Government of Canada did not meet the prescribed service requirements and timetables, and/or failed to provide the requisite resources, the Court could take action. The Court therefore possessed a broad range of compliance orders that it could use, from issuing court orders aimed at correcting a specific deficiency to exercising its powers to remove Canada as the Administrator where it was dissatisfied with the provision of corporate services, although such a move was considered unlikely. Moreover, the court was able to issue additional orders if it deemed it necessary to ensure a full and timely implementation of the Settlement Agreement. Thus, though the lead department, the OIRSRC did not have total control over its response to the Settlement Agreement.

Human Resource Management Challenges

The Human Resources or personnel function is key to an organization because it is the people in an organization that carry out most work activities. The various managers of the organization, including its Human Resources professionals, have the key responsibility for recruiting, retaining and organizing people so that they can effectively perform these activities. This requires not only recruiting, hiring, and managing employees, but, among other things, coordinating employee benefits and suggesting employee training and development strategies.

Internally, the OIRSRC faced Human Resources challenges in the wake of the court's approval of the Settlement Agreement. This meant that the OIRSRC encountered continuous change and challenges in its immediate future as it tried to implement the Settlement Agreement. For example, the OIRSRC did not have the organizational capacity (Human Resources and establishment) to respond to the high number of applications. With the approval of the Settlement Agreement, the OIRSRC workload increased from only two work streams, both of which related to resolving claims of IRS abuse (the Alternative Dispute Resolution process and Litigation), to the five components of the Settlement Agreement and which included the CEP. Both work streams continued for some time after the Settlement Agreement was implemented and, as a consequence, the OIRSRC could not start the design and implementation of the CEP from a blank slate but instead had to add the CEP on to the existing claims resolution infrastructure and business culture.

Among the business and operational environment issues in 2006 that impacted employees and staffing priorities were evolving federal priorities such as the Government's "Restoring Fiscal Balance in Canada" designed to address fiscal imbalance with the provinces and the potential reallocation of resources to address such priorities. These whole-of-government priorities also stretched the limits of the OIRSRC human resource capacity as Human Resources were also required to respond to this new demand while preparing for the Settlement Agreement, the impact of which remained unclear at the time to all parties involved.

Another example of an issue that required, government-wide, federal departments' attention was the Government of Canada's planned modernization of the classification system in a bid to strengthen and modernize the public service with the goal of making it one of the top ten employers in Canada. The federal government's Economics and Social Science Services Conversion process meant federal employees in certain categories faced the prospects of having their positions reclassified into other occupational groups. Issues arising out of this change included the need for employees to know its impact on classifications and jobs. The OIRSRC had to respond to volumes of questions relating to the conversion process and its impact. Documents reviewed show that while this was not unique to the OIRSRC, as a whole, the Department experienced significant challenges which needed to be addressed strategically.

For example, a critical challenge was the processing of the CEP and what was evident from documents examined was that the OIRSRC's obligations concerning CEP payments required a workforce that was not only skilled but also very highly motivated. Challenges then included hiring new research staff and/or consultants/contractors to deal with the expected or forecasted significant volume of applications within a short period of time, from September to mid-October 2007. These applications had to be processed individually, efficiently and also accurately in order to ensure due diligence in compensation of beneficiaries.

Related to this situation was the need to create an independent structure for the implementation of the Independent Assessment Process as well as establishing a Dispute Resolution Operations Branch to ensure that the OIRSRC was well positioned to meet the prevailing dispute resolution obligations and prepare for those at the Independent Assessment Process. Thus this, in itself, required not only rethinking but also retooling and reinvestment of the Department's human assets.

A further issue that surfaced during the Study, and which was linked to Human Resources, was the need to provide sensitivity training to those CEP staff on the frontlines (call centres, etc).

Aside from the Human Resources and staffing challenges, the OIRSRC faced a procurement challenge, relying on nearly 300 research consultants nationally to support the CEP. In mid-2007, the OIRSRC had inherited responsibilities for implementing its own research services' Standing Offer from Public Works and Government Services Canada. It then had to put out a Request for Proposals just as CEP applications started arriving. While contractors are technically not treated like staff, the OIRSRC still had to supervise, train, monitor quality control and deal with the numerous companies that represented the consultants.

The OIRSRC invested in people, later providing hands-on coaching of middle management, training on operational tools and techniques, ensuring challenges of a high stress environment and continuous change were managed. With the high turnover facing departments, there was a need for the OIRSRC to ensure succession planning and successful knowledge exchange for ongoing or improved performance.

Term positions and casual employment were some of the staffing options used to mitigate the impact of the vacant positions on the CEP design and delivery and yet this was, as the Lessons Learned Study found, not enough. The high priority nature of the court decision and the fact that one-third of the OIRSRC's employees were spread over the National Capital Region, Vancouver and Regina^{Footnote 10} required more human resources. In addition, employees, for example, those in the National Capital Region, were scattered over several buildings within their geographical locations.

Change Management Challenges

Communications Management Challenges

As a result of the spread of employees over several geographical locations and also the distribution within the National Capital Region, it was imperative that strong, regular and consistent lines of communications were put in place and maintained, not only with CEP administrators but also between the CEP and all departmental staff. Communication was needed in three key areas: (1) Directorate level – within the National Research and Analysis Unit (between the National Research Analysis Unit, Vancouver and the National Research and Analysis Unit, Ottawa); (2) Sector level – between the National Research Analysis Unit and the CEP Coordination Unit, and with the rest of the OIRSRC or Resolution and Individual Affairs Sector; and (3) External: between the OIRSRC/INAC and other federal departments and stakeholders.

Workforce Situation

The significant OIRSRC departmental changes and high staff turnover during the 2006-07 fiscal year also underlined the need to address workforce destabilization in order to ensure that the CEP was designed and delivered efficiently and economically. In this sense, the National Research Analysis Unit research and the CEP Coordination Unit managers had to engage in and undertake constant and ongoing staffing actions instead of devoting a full time focus on CEP needs. The CEP itself was a large staffing initiative and it required immediate attention. Moreover, the OIRSRC clearly faced significant hiring needs in order to meet its obligations under the Settlement Agreement. And, in addition to staffing, the National Research Analysis Unit had to also constantly juggle responsibilities linked to a sizable contract workforce of research consultants.

In the above context, the Study found that addressing the urgent and pressing requirements of the Settlement Agreement was still going on in addition to the OIRSRC's other normal, day-to-day activities of a full-fledged department such as: ensuring workplace well-being (e.g., supporting flexible work arrangements), learning and career development, providing orientation to new staff or providing competency profiles and learning opportunities, as well as training research consultants. These were all needed to produce employees who would enable a spirit of healing and reconciliation as it pertained to the Settlement Agreement.

However, in comparison with other departments that faced similar pressures, what made the CEP and the OIRSRC situation unique was the nature of the CEP process and the work attached to it. Most importantly, staffs were dealing with a situation where they faced a significant lack of trust, which was also a commonly identified barrier to communication between the CEP and external stakeholders, former students included. Moreover, these were former students of institutional abuse suffered at the hands of a different but dominant culture seen as a colonizing power. A great deal of sensitivity was needed and this added to the stress on the OIRSRC staff and the recognition of the urgent need to staff up and implement the Settlement Agreement.

Indian Residential Schools Settlement Agreement

On November 20, 2005, the Agreement-in-Principle was reached. It represented the consensus reached between legal counsel for former students, legal counsel for the Churches, the Assembly of First Nations, other Indigenous organizations and the Government of Canada. On May 8, 2006, the Settlement Agreement was approved by all parties and then considered by courts in nine jurisdictions across Canada, with the last court giving its approval on March 20, 2007. Implementation of the Settlement Agreement began on September 19, 2007, following a five month opt-out period. Article 5 (Common Experience Payment) of the Settlement Agreement sets out the legal obligations of the Government of Canada in the administration of the CEP.

CEP Policies and Procedures documents

In terms of the CEP Process and Validation Protocol, which was approved by the parties in late August 2007 and received formal court approval in February 2008, the CEP process began with the collection of applicant information, confirming its completeness and performing a preliminary assessment by verifying the applicant's identity against the required identity documents. The validation process itself was guided by eight validation principles, most noteworthy of which being the principle of erring on the side of caution in favour of the applicant. At the heart of the CEP was the validation process, whereby residency at an eligible IRS was confirmed. Created just prior to the launch of the CEP, the validation document was aligned with the guiding principles of the Settlement Agreement and provided a detailed step by step procedure to CEP delivery requirements as defined in the Settlement Agreement. It guided both departments (Service Canada and the OIRSRC) on how to manage claims from former students. Recognizing, however, that the validation of CEP applications posed many complex challenges (namely, the sheer volume of applications and the meeting of service standard requirements) for the Trustee of the CEP funds, which was the Government of Canada, the OIRSRC was meticulous in assembling this document so that it adhered to the Settlement Agreement.

The CEP Process Assessment Protocol dated September 6, 2007, had a significant influence on the final design of internal procedures, on training materials, on conducting the training, on finalizing system workflows, etc. CEP applications went live only 13 days later, on implementation day, September 19, 2007. This date was endorsed by the OIRSRC and the Department of Justice in August 2007 and then subsequently approved by the all-party National Certification Committee, which was the predecessor to the National Administration Committee.

All CEP applications were first automatically assessed by the Computer Assisted Research System^{Footnote 11} (CARS). CARS was an algorithm that searched over one million coded student records stored in the National Research Analysis Unit's CT Summation research document management database^{Footnote 12}. It processed over 600 calculations when completing each applicant search, including, for example, noting common phonetic spellings and typical manual typewriter mistypes. It also searched records 10 years on either side of the applicant's stated date range of attendance. CARS made automatic eligibility decisions in about 44 percent of the over 110,000 CEP applications received (the original CARS prototype projected a much higher validation rate).

Where CARS could not make a determination, the CEP application was manually reviewed by the National Research Analysis Unit research staff or a consultant. The basis of the conclusive result of the review was recorded in the Single Access Dispute Resolution Enterprise^{Footnote 13} (SADRE) system, including citation of all summation research, if applicable. As set out in the Validation Protocol, an inference and interpolation policy had been adopted to allow confirmation of duration where there were gaps in the IRS records held by the OIRSRC. As well, the Protocol allowed the OIRSRC to make a preliminary assessment by verifying the applicant's identity against the required identity documents.

For many of the former students, the CEP was their entry point into the services provided by the broader Settlement Agreement. To ensure that the spirit of reconciliation and healing, which was the ultimate aim of the Settlement Agreement, was reflected in the delivery of the CEP, the Court approved assessment principles to ensure that every eligible applicant received the correct amount of compensation and that this compensation reached the intended recipients. The assessment was to be fair, objective, timely and practical and, while minimizing the onus placed on applicants, it was to be efficient and executed with a minimum of errors.

When designing the implementation of the Settlement Agreement, the challenge was to ensure a fair and transparent process with solutions for every stakeholder, especially in the light of "missing records" and the extensive CEP-related media coverage. This created a variety of competing requirements and the need to balance these. Moreover, central agencies were also interested in the CEP process and the ultimate outcome of the implementation of the Settlement Agreement. Based on interviews and documents reviewed, CEP officials did their due diligence in order to satisfy the legal requirements to uphold Section 34 accountabilities while balancing the expectations flowing from the CEP assessment principles.

To address the "missing records" question, the OIRSRC hired PricewaterhouseCoopers to conduct an audit of records readiness in spring 2007 and which showed that gaps existed in the National Research Analysis Unit's historic record collection. A work plan to locate IRS student records had been implemented throughout late 2006 and much of 2007. This undertaking was concurrent with the CEP design and the work involved churches, provincial and territorial archives, various Indigenous organizations and former students. The audit also prompted the Department to implement inference and interpolation policies that would allow CEP eligibility decisions to be made when there were gaps in records.

The documents reviewed for the Study also clearly showed that the policies and procedures created were aligned with the Settlement Agreement's principles. For example, the Memorandum of Understanding between the OIRSRC and Service Canada conformed to the Settlement Agreement and defined the roles and responsibilities of the two organizations. Two levels of appeals were implemented, the first being Reconsideration, which dealt with applicants who were not satisfied with their claim decision. Where they were dissatisfied with the Reconsideration decision, applicants could appeal to the National Administration Committee. Importantly, the National Administration Committee was composed of members representing the parties identified in the Settlement Agreement and, therefore, acted as a check on the work being undertaken by the OIRSRC. Moreover, the Settlement Agreement clearly defined the guidelines for the CEP's implementation. The content of files and documents reviewed by the Study adhered to the principles. Finally, the Settlement Agreement required the National Administration Committee to produce and implement policy protocol and standard operating procedures documents with respect to the implementation of the Approval Orders.

Interviewees highlighted the enormous effort made by the OIRSRC to ensure that policies and procedures adhered to the principles set out in the Settlement Agreement. Every decision was made in consultation with departmental legal services and there was significant consultation with the National Certification Committee (pre-National Administration Committee), the parties and former students before the framework was finalized. The National Certification Committee, which was the transitional group, had a representative on the National Administration Committee and worked with other lawyers. Several meetings were held to ensure that every process or procedure conformed to the Settlement Agreement.

Finally, Terms and Conditions for Contributions for eligible Aboriginal Organizations for the purpose of providing advocacy and public education on a diverse range of issues related to the Settlement Agreement were also well aligned with the Settlement Agreement. The purpose of the Advocacy and Public Information Program was to inform the public and encourage the sharing of diverse viewpoints on a range of IRS Settlement Agreement issues, policies or programs and thereby allow the Government of Canada to respond to the widest range of information possible This would improve the quality of departmental decisions, identify where changes might be needed to enhance service delivery, and, ultimately, ensure the Indigenous community was well informed of the benefits available through the IRS Settlement Agreement. It was, moreover, essential for all Canadians to understand the Settlement Agreement and the impact that the legacy of the IRS had had on Indigenous communities. The Terms and Conditions also identified all eligible recipients as those defined in the Settlement Agreement who could apply for the CEP.

Based on analyses of the documents and the interviews conducted, it was clear that the Memorandum of Understanding with Service Canada together with the CEP Process and Assessment Protocol and the Advocacy and Public Information Program Terms and Conditions, ensured that the policies and procedures were aligned and respected the key principles defined in the Settlement Agreement and answered the CEP requirements as also defined in the Settlement Agreement.

All key informants agreed that the CEP documents supporting the Settlement Agreement were clear and designed to satisfy the requirements of the Settlement Agreement. An audit of the CEP trust fund (Designated Amount Fund), completed by Employment and Social Development Canada and presented to the Court in March 2013, acknowledged the strength of the checks and balances validation process.

Finding 2: Both within and between the Office of Indian Residential Schools Resolution Canada and Service Canada roles and responsibilities were well understood, including clearly defined relationships. However, challenges such as information not being communicated to others were encountered.

According to interviewees, at the outset, roles and responsibilities were somewhat clear at the ''ground level'' within the Department, but as the process evolved and more components were added, defining roles and responsibilities became harder. Key informants mentioned that relationships and the nature of the processes became less clear and "it really took some time to adjust" in that the CEP implementation gradually differed from the original conception. Moreover, stakeholders had a different understanding and inquired, for example, "Who paid for this?" and "Who authorized it/this?"

Senior management and staff. First, senior management had to concentrate on staffing director general-level positions. Interview evidence indicates that the relationship between staff and senior management was good because it was a small department"with a good feel and the communication was good; the employees were very dedicated, went above and beyond and in the best way; and it was a positive culture." There appeared to be a direct line to the Assistant Deputy Minister, again because it was a small group in a small department. Anecdotal evidence suggests that several employees were of the view that though this was a stressful period, they would recommend it as a place to work.

Geographical split. Due to the geographical division of the National Research Analysis Unit staff, it was difficult for some National Capital Region staff to report to a director who worked in the Vancouver National Research Analysis Unit office. Some individuals found it positive as they received a lot of direction directly "from my Director General and Assistant Deputy Minister." However, this on occasion constituted a management challenge for the National Director as it could mean a communication breakdown (e.g. exclusion from critical information being passed on). Overall, however, interviewees saw the CEP as a very horizontal, collaborative initiative "where everyone did their best to keep everyone in the loop, and senior managers were very supportive when issues arose."

At the CEP operational level, although efforts were made to be inclusive, the geographic distance, different time zones and competing mindsets (perspective and approach) were obstacles. For example, the Vancouver and National Capital Region staff did not always work the same way and further, the urgent nature of the CEP response was not dealt with in the same way so that internal procedures, discussed below, were challenges.

Another problem that came to light (due to the geographical separation) was one region's "more relaxed approach" to the CEP response "and so the urgency was not there." For example, the study learned that while senior management had informed the National Capital Region that they had room to "make mistakes, to err on the side of caution" due to the urgent nature of the CEP, this information, it appears, was not passed on to those in Vancouver. If there was far more comfort and confidence with the policies being developed in the National Capital Region than in Vancouver, then it is clear that there was a communication breakdown and this did disservice to the entire National Research Analysis Unit staff, whether they were based in the National Capital Region or in Vancouver.

In terms of the two National Research Analysis Units, prior to the arrival of a new director in February 2007, the research teams had been separate with minimal interaction and when the two were merged, there were differences in experiences and approaches. At different times, consultants were hired to address the National Research Analysis Unit's operational realities, including a change management consultant (early 2007), a focus on relationships (2008) and then an "operational sustainability assessment" undertaken in early 2009 led to a yearlong National Research Analysis Unit business transformation project.

Problems with the Information Technology (IT) system featured prominently here as well in terms of the relationship between the two National Research Analysis Unit regions. For example, while one region standardized its processes, the other did not and this, according to an interviewee, can be attributed to the fact that one office had very strong processes in place while the other did not necessarily have the same scope of standards. The centralizing of files was to make matters work better by making the work interchangeable. At the beginning, the National Capital Region was dealing with Saskatchewan, Manitoba, Ontario and the eastern provinces while British Columbia dealt with Alberta, the western provinces and the North. However, negotiations led to the National Capital Region covering Alberta. The resistance to standardization of the process could be attributed to a feeling of mistrust because one region considered the other was moving too quickly, even though both the National Capital Region and Vancouver had distinct roles and responsibilities.

For example, while interviewees agreed that there were significant attempts made to standardize research practices between the National Capital Region and Vancouver prior to implementation of the CEP, there continued to be challenges after the launch. One interviewee was of the view that one region's lack of confidence in the CARS translated into different outputs and productivity between the two offices, which compounded the situation because "one region started doing the other's files." This can largely be attributed to the geographical expanse between the two regions and where one location was able to keep its staff more informed so that there was a greater appreciation of the urgency involved. In addition, there was greater acceptance of senior management decisions and a willingness to trust the process and follow through.

There was also the transition from a small independent organization (OIRSRC) to the much larger INAC in June 2008, which led to issues related to organizational change. For instance, at the operational level, the OIRSRC's Research Unit (National Research Analysis Unit) responsible for validating CEP applications was reporting to one director general while the CEP Coordination Unit reported to another. As a result, there was a communication issue between the two CEP units. An interviewee noted that, "You could tell there was a clear distinction between the two groups. There were guidelines to be followed so the researchers followed them to a tee, whereas the workers on the frontlines may have a story that made sense but which may not fall within the guidelines. In this sense, there was less flexibility and there were differences of opinion, which posed challenges" (See "Change Management Challenges" Section 4 - Lessons Learned Findings: Key Observations).

The Office of Indian Residential Schools Resolution Canada Department and Service Canada. As stated above, the Memorandum of Understanding defined the roles and responsibilities between the OIRSRC and Service Canada for the design and delivery of the CEP. The Memorandum of Understanding put in place was well defined and it was noted that the relationship was very good between the OIRSRC and Service Canada, especially at the senior management level where they conferred weekly.

An issue identified during the interviews was the IT connectivity problem at the functioning, operational level between the two departments. Service Canada received all CEP applications, entered the data into its data system, which then connected with the OIRSRC’s claim database, the Single Access Dispute Resolution Enterprise (SADRE). Then, once an eligibility decision was made by the National Research Analysis Unit, the SADRE would be updated and data transmitted back to Service Canada. The IT systems were synchronized every 24 hours (e.g. fields had to properly align, etc.). But, as the study discovered, there was no time to test process workflow assumptions until CEP applications surged into Service Canada in September 2007. Consequently, several system issues had to be addressed within and between the two departments in real time. However, as the Study found, it seemed that by the time the issue got to senior management level, everything had been addressed because at that level, senior management received the information it needed. According to the key informant interviews, the OIRSRC Department commended Service Canada for its outreach and application intake role because when applications started arriving Service Canada was very supportive of CEP applicants. For example, some Service Canada call centres ''set up chairs for the elderly and provided food (tea, etc.) for former students who came to these centres’’. The major IT problem was due to the lack of time to implement the process and do some testing of the system within each department and between the two departments. Overall, most of the interviewees agreed that roles and responsibilities were well understood between the OIRSRC and Service Canada.

The Office of Indian Residential Schools Resolution Canada Department and the Department of Justice. The Department of Justice was involved in many aspects of the CEP since the Settlement Agreement was an obligation in an out-of-court Settlement Agreement. This meant that senior Department of Justice litigators played a key role in providing advice, interpretation and guidance in determining how the CEP should be processed, the scope of CEP eligibility and the management of all aspects of CEP appeals (and court appeals). As CEP processing began, the Department of Justice provided ongoing legal advice on a case-by-case basis. An interviewee noted that at the senior management level,relationships were strong and well understood, though at certain times there were differences, which led to additional challenges for the OIRSRC.

Overall, it must be reiterated that there were multiple players involved in the design and delivery of the CEP. As mentioned earlier, while the Government of Canada under the Settlement Agreement was responsible for the implementation of CEP, it was, on the other hand, one of many different parties signatories to the Settlement Agreement, which included representatives of IRS former students, churches, the Assembly of First Nations and Inuit organizations. Complicating the roles and responsibilities was the fact that the Settlement Agreement was also overseen by the court via the court-appointed monitor who ensured that the Government of Canada respected its legal obligations. As a result, the CEP was never fully under the OIRSRC's (the federal government) control and the federal level could not dictate how the CEP should be delivered with each service delivery process and protocol having to be vetted and approved prior to implementation. For example, the CEP Validation Protocol was the policy and procedural basis for how Canada was to assess CEP eligibility. Though this protocol was written by the Government of Canada it was subject to input, comments and edits from all parties before it was finally endorsed in August 2007, just one month prior to implementation.

Section 34

Pursuant to s.34 of the Financial Administration Act and the Treasury Board Policy on Delegation of Authorities, signing authorities are delegated to various levels of management of a department to enable them to administer programs and manage expenditures under their jurisdiction.

The Minister of the Department formally delegates and communicates spending and payment authorities through the financial delegation matrix. The incumbents with delegated financial authorities throughout the Department communicate these authorities through specimen signature documents.

Primary responsibility for verifying individual expenditures and payments rests with these incumbents who have been delegated the authority to confirm and certify entitlement, pursuant to s.34 of the Financial Administration Act. These incumbents are responsible for certifying that work, goods or services have been received for the payment requested, that the charge is reasonable and correct, and that the payee is entitled to payment. Each transaction that undergoes s.34 certification must be coded to indicate the type of transaction and the responsibility or cost centre identifies the area where the expenditure was incurred.

Once the s.34 certification has been properly executed and the information is put into the Corporate Administrative System by the appropriate office, all documentation is forwarded to officers having delegated payment authority under s.33 of the Financial Administration Act. Prior to exercising delegated authority under s.34, new managers must undergo a five-day mandatory classroom course. Before December 2006, managers had the option to take the five-day course or an online training course.

Tools

While the CEP staff worked towards meeting growing and changing demands, the Study found that the tools made available to staff in the CEP implementation presented many challenges. For example, the Single Access Dispute Resolution Enterprise (SADRE), the OIRSRC's case management database, required a high level of management.

Information obtained from documents highlights the constantly changing requirements related to IT. For example, the manual data contained in the SADRE database were not always consistent when compared to the source documentation. Inaccuracies identified within the SADRE database included, but were not limited to, date of birth and contact information (such as mailing address and phone number). An audit completed on December 4, 2008, found that in 20 percent of files tested in a first sample, applicant information did not match the information captured in the SADRE. In eight percent of files that were tested in a second sample, applicant information did not match the information captured in SADRE. However, the improvement in results that came about was attributed to the introduction of key process controls within the Advance Payment program.

During the course of the same audit, it was discovered that the mailing addresses for 1,514 files had been lost as a result of a system conversion within the SADRE. Consequently, eight percent of the sample files reviewed did not have mailing addresses in the SADRE. The system conversion had resulted in the loss of claimant information.

There were other technical issues: the linkage between the CARS and SADRE; and connectivity issues when transferring information to Service Canada. The SADRE required many IT updates and changes over a short time, which led to unavoidable work-around, version control issues and affected interoperability.

A review of the OIRSRC's Departmental Performance Report, 2007-08 provided insight into issues related to the SADRE. For example, capital expenditures planned for 2007-2008 were to be used to support the full implementation of the SADRE. The expenditures were also to cover necessary changes to this tool's capacity and functionality as a result of the transition from the then still ongoing Alternative Dispute Resolution Process to what was to become the new Independent Assessment Process and the delivery of the CEP. At the same time, the OIRSRC Canada was still developing its organizational structure in anticipation of the approval of the Settlement Agreement.

The CARS was developed in-house for the express purpose of capturing the expertise of a researcher and proved to be an invaluable tool for the CEP process. According to interviewees, and confirmed by documents reviewed, the CARS, when initiated, consistently deployed this expertise at a fraction of the time and cost it took to undertake manual research, while matching or even exceeding the accuracy of a researcher. This research step was necessary if the anticipated volumes of applications were to be met and addressed appropriately. In addition, interviewees mentioned that the CARS deployed advanced techniques to enable initial processing of applications and was supplemented by a team of expert researchers who manually validated issues like inconclusive or incomplete findings by the CARS.

Although the CARS was well designed, it had its limitations. For example, it was unable to identify gaps in student claims, and the number of years covering a period of time was the biggest gap that the system could not address.

Key informants indicated, therefore, that diligent processes put into place to identify errors and ensure deficiencies in the system were improved on a continual basis. An audit completed in December 2008 helped to isolate these key challenges and enabled staff to address the problems.

Finding 4: Former students were provided with good and ample information on eligibility criteria and how to apply.

According to interviews and news clippings reviewed, former students were made aware of the process through different mechanisms. First was the court-approved Notice Plan, implemented in 2006, 2007, 2011, and 2012. In 2006, the Government of Canada funded a court-approved Notice Plan, which aimed at reaching and informing former students and their family members of the Agreement. Hilsoft Notifications, which was engaged to assist with the process, designed the Notice Plan; the implementation process was undertaken in four phases, which extended from 2006 to 2012.

The Notice Plan encompassed advertising in various media outlets: Indigenous and mainstream print, television, and radio media; poster campaign; internet advertising; and direct mail to known former students. The communications materials were produced in English, French, and a number of Indigenous languages. According to documents reviewed, "the four phases of the Notice Plan were determined to have reached 98 percent of the target population, an average of 14 times".

There was also the media as an audience and public affairs television broadcasts were developed by media firms like Hilsoft. Further, pamphlets and posters were developed and distributed in band offices. In addition, because the importance of maintaining published service standards was stressed and also the Prime Minister's formal apology to former students in June 2008 for the legacy of the Residential Schools, awareness of the CEP increased. Such awareness information contributed to the CEP experiencing a number of unexpected challenges in terms of communication to applicants concerning delays in processing applications.

The OIRSRC kept reaching out, providing diverse service options for the highly complex target population. Thus, the Advocacy and Public Information Program was managed by the OIRSRC to encourage the sharing of information and to ensure that the Indigenous community, particularly former students and their families, were aware of all aspects of the CEP. The program, contribution funded, was launched in 2007-2008 to support all efforts to reach IRS former students and inform them of the benefits available to them under the Settlement Agreement.

The OIRSRC also undertook training sessions that included local Elders and Indigenous traditions and interviewees considered that these sessions were most appreciated by former students, their families and their communities. In this sense, the OIRSRC used local community members to recruit or assist applicants, something that, according to interviewees, was widely seen as beneficial.

Delivery- CEP Outreach (weaknesses)

The Study, on the other hand, found some weaknesses. For example, the lack of operator (call centres) awareness of emotional triggers during the CEP process came up as an issue. Interviewees volunteered that in reference to implementation, there was a lack of planning (i.e. training) for staff in terms of preparing them for the possibility of triggering the emotions of former students that could result as a result of an IRS-related outreach.

There were also challenges with respect to managing the helplines. Key informants noted that it was unclear to beneficiaries which number to call and there were high call volumes, in addition to initial language barriers. There was also the inability of service operators to update applicant files due to access and privacy restrictions. Documents show that several elderly applicants spoke an Indigenous language, which initially complicated both communication and outreach efforts.

Another challenge was having only a 28 day turnaround to issue cheques to applicants. During the planning phase in the months leading up to the launch, it was confirmed that it was difficult to obtain approval for funds from the Treasury Board in preparation for the launch of the CEP. This created delays in the short period available for planning, which impacted the data system that was being put in place. This in turn led to a delay in response to applications and eventually a delay in issuing cheques.

Administrative files reviewed did not provide any evidence that a gender-based analysis of the CEP was undertaken and the study cannot say if such an analysis could have made a difference by facilitating the development of a service delivery strategy for women in these circumstances. Neither could the Study ascertain if a gender-based service delivery strategy could have impacted processing delays by, as an example, advancing files or dealing with the labour intensity that would be required for issues such as names identified and/or name changes. The Study is aware that all of the key informant background data that were of import to the CEP included not only age, residence, years of school attendance, date of CEP receipt, etc., but also gender. File documents like application forms required applicants to provide basic background information, including gender.

Findings 5: There were service standards in place, designed to assist the Office of Indian Residential Schools Resolution Canada to deliver the CEP with quality and care and these were met.

Economy

Finding 8: Taking into account the urgent, sensitive nature and the scope of the Settlement Agreement, the CEP was delivered in an efficient manner.

The cost for the design and delivery of CEP between 2007-2008 and 2012-2013 was approximately $102,000,000 in salary and in B-base money (Operations and Maintenance). Considering the uniqueness of the CEP and the nature of the process, key informants were of the view that the cost for the design and delivery of this initiative could not have been reduced. The document review indicated that the CEP was logically delivered with a fair amount of effectiveness, efficiency and economy. Moreover, this is most accurate considering the nature of the CEP, the environment and the context within which the CEP had to be delivered and by whom. Moreover, interviews and documents related to the CEP, as well as the literature on redress matters demonstrate that in such situations, efficiency and economy are not viewed as central compared to ensuring the accurate and timely compensation of beneficiaries and the desire to address wrongs. The CEP goal, as set out in the Settlement Agreement, was to ensure that eligible former students were reached and served in a sensitive way respectful of their culture.

Mechanisms were developed to efficiently monitor the CEP process

With respect to economy, the CEP delivery was dependent on existing staff and, in some cases, on consultants who were engaged as part of the OIRSRC temporary staff. Further, a module for the SADRE had to be created urgently and in-house to manage CEP data. The administrative cost data also showed that overall costs appeared to be aligned with budget requests.

Financial controls were also in place to monitor all aspects of the CEP. For example, throughout the CEP process, rigorous checks and balances were used not only to make sure that former students received their allotted funds but also that payout to eligible former students was carried out properly. In this case, there were auditors (an audit and evaluation function) and third-party review by Crawford Class Action who were monitoring all aspects of CEP work done by the OIRSRC. Also, both the Court and the National Administration Committee approved all changes in policies and procedures, and monitored the progress of the CEP. The process was therefore considered to be very transparent.

Based on interviews, however, the number of applications that required research during the first few weeks after the launch of the CEP was very costly and further, the mechanism in place at the time was not cost-effective in that additional funds had to be injected into the CEP to meet its needs. The costs do not include the toll CEP took on the OIRSRC employees and CEP staff, in particular, in their efforts to ensure an appropriate and timely response to the Settlement Agreement.

Another control in place was an overpayment policy, which covered any possibility of an overpayment being made. The goal of this policy was to monitor possible overpayment to former students and it was handled by the National Research Analysis Unit decision makers on all CEP files. They also identified overpayments and the reasons why an overpayment was in fact considered to be an overpayment. Through documents reviewed, the Study noted that only 59 cases were reported compared to the 80,000 applications that were submitted.

Fraud prevention was also established by the OIRSRC and ensured that employees (staff and research consultants) who worked on the SADRE interface and submitted recommendations were not the same as those who were approving the Section 34 delegation activity; the goal was to avoid duplication. So, for example, a researcher could not be involved in the research as well as in the final approval process of applications.

Finally, there was the quarterly reporting to Treasury Board, which was also a means of monitoring costs. It should be noted that in implementing the Settlement Agreement "the OIRSRC was starting everything from scratch, building a new thing." According to an interviewee, the pre-implementation and the pre-launch of the response "was scary with so much coming at you at once and being a small department with the Settlement Agreement unlocking billions of dollars for thousands of recipients, topped by the fact that it was a very sensitive situation as well as it being the largest Settlement Agreement in Canada's history."

Finding 9: A number of unintended outcomes emerged from the design and delivery of the CEP. For example, the difficulties with Computer Assisted Research System and an increase in Access to Information and Privacy (ATIP) requests.

Lessons Learned

Draft Logic Model for 2.4.1 Common Experience Payments

Text alternative for Draft Logic Model for 2.4.1 Common Experience Payments

The CEP logic model (Appendix A) was a draft developed in consultation with CEP program officials. It presents the main activities, outputs, and expected immediate and long-term outcomes. Over the period covered by the study, the activities undertaken by the CEP were primarily focused on a key short-term outcome: payments to eligible former IRS residents.

Appendix A illustrates a Basic Results Logic model for the Common Experience Payments. The logic model flows from top to bottom. The top level represents Activities, the box contains the text Validate and review applications submitted for reconsideration. An arrow continues down from this box and splits into two arrows. The next level represents Outputs, and the forked arrow from the Activities lead to two Output boxes. The box contains the text Application Processed in accordance with the Settlement Agreement. The second box contains the text Payments delivered to former residents in Indian Residential Schools. From these boxes arrows continue down, but also crossing to the next level in model which represents Immediate Outcomes. The first box contains the text A fair resolution to the legacy if Indian Residential Schools is supported as set out in the Indian Residential Schools Settlement Agreement. The second box contains the text Every eligible applicant receives the correct amount of compensation. From there the descending arrows merge and point to one final box in the last level, which represents Longer Term Outcomes/Sub-Program Expected Results. The box contains the text Eligible CEP applications are processed in accordance with the Settlement Agreement.

Financial Administration Act (Section 34)

Marginal note: Payment for work, goods or services

34. (1) No payment shall be made in respect of any part of the federal public administration unless, in addition to any other voucher or certificate that is required, the deputy of the appropriate Minister, or another person authorized by that Minister, certifies

• (a) in the case of a payment for the performance of work, the supply of goods or the rendering of services,
  • (i) that the work has been performed, the goods supplied or the service rendered, as the case may be, and that the price charged is according to the contract, or if not specified by the contract, is reasonable,
  • (ii) where, pursuant to the contract, a payment is to be made before the completion of the work, delivery of the goods or rendering of the service, as the case may be, that the payment is according to the contract, or
  • (iii) where, in accordance with the policies and procedures prescribed under subsection (2), payment is to be made in advance of verification, that the claim for payment is reasonable; or
• (b) in the case of any other payment, that the payee is eligible for or entitled to the payment.

Marginal note: Policies and procedures

(2) The Treasury Board may prescribe policies and procedures to be followed to give effect to the certification and verification required under subsection (1).

Working Group Terms of Reference

 1. MANDATE

A Working Group will be established between INAC's Evaluation, Performance Measurement and Review Branch (EPMRB), and program officials at Headquarters. The mandate of the Working Group will be to assist the Evaluation team at different stages (Methodology, preliminary findings and draft final report) of the Lessons Learned Study.

Ensuring the quality of evaluations is one of the most important aspects of the Evaluation Branches' work. Quality is ensured by including an appropriate mix of decision-making, internal and external review and the application of quality control tools. The objective is to produce Lessons Learned Study products which are reliable, useful and defendable to both internal and external stakeholders.

Working Groups are an integral part of the external review process for evaluations.

 2. PRINCIPLES

The Working Group's guiding principles are to ensure that:

• comments are presented in a timely and professional manner;
• issues are consistent with Treasury Board Secretariat^{Footnote 19} policies and guidelines related to evaluation and program accountability; and that,
• Lessons Learned Study results are based on reliable and defendable evidence, and anchored in a methodology that is the most appropriate in the context of the Study.

 3. OBJECTIVES

The objectives of the Working Group are to present the interests and concerns of the parties involved in the process regarding the planning and implementation of the Lessons Learned Study through the provision of technical and strategic expertise that focus on ensuring:

• relevant Lessons Learned Study issues and questions are covered;
• The evaluators are aware of all pertinent factors and documents surrounding the Lessons Learned Study process;
• The evaluators have proper access to the relevant administrative data and information;
• Factual errors are avoided and data challenges are recognized and responded to, if possible;
• Reporting is clear, concise and balanced and set out in such a manner to facilitate management response

 4. MEMBERSHIP

The Working Group is composed of representatives from INAC Headquarters (EPMRB and program officials).

 5. ROLES AND RESPONSIBILITIES

EPMRB is responsible for:

• hiring and managing a consultant(s), if applicable, to assist the Lessons Learned Study team to conduct the Study
• collecting and analyzing information, producing technical reports if warranted, producing a final report
• setting the date and time of meetings (with the agreement of the other members) and setting the agenda;
• ensuring that meetings are run in as efficient a manner as possible and that every member has an opportunity to contribute;
• ensuring that a record of decisions is kept for each meeting, and that it is distributed to Working Group members for comments and revised as appropriate;
• ensuring that tasks that arise out of meetings are assigned to the appropriate member(s); and,
• setting deadlines for the completion of tasks.

Working Group members are responsible for:

• attending meetings in person or on the phone and contributing to the best of their knowledge and abilities;
• obtaining input from others in their respective areas and reporting back on the outcome to the Working Group, as needed;
• briefing the relevant people in their respective areas on the progress of the Study;
• identifying an alternate and briefing this alternate fully in case they cannot attend meetings; and
• completing tasks assigned to them within the agreed time frame.

 6. MEETINGS AND DURATION

The Working Group will operate from the time of its composition and initiation of the Lessons Learned Study to its completion. Working Group representatives will meet on an as is required basis to provide input, advice and feedback on:

1. Terms of Reference
2. Other documents that require their input, such as:
   1. Detailed Methodology Report
   2. Summary of key findings DECKs
   3. Lessons Learned Study Preliminary Findings
   4. Draft Final Lessons Learned Study Report

In reviewing progress documents, it may be sufficient for the EWG to exchange information via email.

 7. CONFIDENTIALITY

It is important for all members to ensure that information related to the Lessons Learned Study remains confidential until the final report is approved by the Evaluation, Performance Measurement and Review Committee. This is important in order to facilitate open discussion among members and in order to review the deliverables. It is understood that members may report internally within their sectors with the understanding that they will respect this confidentiality. Members will also respect the confidentiality of protected documents.