Archived - Impact Evaluation of the Income Assistance, National Child Benefit Reinvestment and Assisted Living Programs - Final Report

Introduction

The objective of this evaluation is to provide evidence to Treasury Board of the degree to which INAC's Income Assistance (IA), Assisted Living (AL), and National Child Benefit Reinvestment (NCBR) programs are achieving their intended outcomes. As evaluations of IA and NCBR had been completed as recently as last year (2007), and a thorough program review of AL was completed in the same period, this evaluation was intended to have a different focus.

As not enough information about impacts was provided in previous evaluations, the intended focus of this evaluation was to be on the impacts of these programs on First Nations communities, service providers and clients. Accordingly, a great deal of the effort put into the evaluation was directed at carrying out case studies in ten First Nation communities across the country that represent regional variations in service delivery, varying degrees of size and remoteness, and differing funding agreements guiding program and reporting requirements. Most of the case study communities delivered all three of the programs under review. Key informant interviews and literature and document reviews provided the evaluation team with the overall policy and program context in which the ten communities are operating, and in most cases, struggling to meet the often unique and challenging needs of community members.

Methodology

The evaluation developed multiple lines of evidence gathered through the following activities:

• A review of 331 Documents;
• A review of 25 Literature Sources;
• Review of 129 Administrative and Financial Data Documents;
• 17 Key Informant Interviews;
• 10 Community Case Studies, that included:
  • 77 interviews with community service providers;
  • 181 community surveys, principally with IA recipients;
  • 8 focus groups involving 78 IA/NCBR end-users;
  • 32 surveys of AL end-users;
  • Review of community documents/data sources.
• An Expert Panel on Assisted Living (5 participants);
• An Expert Panel on Income Assistance/NCBR (5 participants).
  
  

Overall Evaluation Findings

Overall, but to varying degrees, the three programs continue to meet vital socioeconomic and health needs in First Nations. One of the principal observations the evaluation would like to convey is the complexity of the issues under review, and that the deep and complex situation of on-reserve social needs and the programs designed to meet those needs, requires complex and long-term solutions. Adding to the complexity of the situation is INAC's policy of matching provincial programs; this results in significant variability in Income Assistance rates and measures across the country, and in Assisted Living program policy and delivery. Consequently, although one of the evaluation recommendations is to review IA funding in order to ensure basic needs are met, there are companion recommendations for integrated and long-term solutions that address the root causes of poverty and unemployment on reserve. Long-term solutions, are, however, just that; while they are being designed and implemented, ideally as part of a national strategy, basic needs funding may need to increase in the short-term.

Before outlining specific program findings, the report begins by presenting the challenges to achieving an impact evaluation. Despite these challenges, the case study method has provided a depth of analysis that offsets many of those challenges.

There are a number of significant challenges to achieving a meaningful assessment of program impacts, some of which were outlined in the 2007 evaluations. A number of the most salient are:

• Lack of Meaningful Outcome Indicators: INAC is in a challenging position: to meet Treasury Board accountability requirements, the department is asked to provide evidence of program impacts, while historically the frameworks for such evaluation and performance measurement have not been in place. In order to have meaningful evaluation data available for a good assessment of program impacts, evaluation frameworks outlining meaningful outcome indicators are required to direct data collection. INAC, in line with the overall culture of evaluation evolving across GOC departments, has developed a draft RMAF, including a logic model, but is in the process of developing a more robust performance measurement strategy aimed at providing better evidence of program impacts, to reflect the Treasury Board policy on transfer payments. That have such indicators for future evaluations, but at present the data collected can say little about impacts on end-users, service providers and communities as a whole. As these are targets of desired social development outcomes for Aboriginal people at the horizontal GOC level, the evaluation goal needs to be the collection of outcome data that shows program impacts at all of these levels.
  
  
• Policy and Program Variability Nationally: Evaluations are challenged by the wide range of variability of the programs from one region to the next, and in the case of NCBR, from one First Nation to the next. As the driving policy principle for both IA and AL is comparability with analogous provincial programs, the result has been a diverse mix of policies and program elements all under the INAC umbrella. In some regions, INAC is principally a funder, and in others, provides service directly to clients. Painting a coherent national picture of any one of these programs is an extremely complex task.
  
  
• Consistency and comparability of data: In part as a result of the policy and program variability just discussed, and in part a result of data system inconsistencies from region to region, evaluations are challenged to obtain a consistent and comparable set of data. Data frequency and type also varies within regions, according to the reporting requirements of the funding agreements particular First Nations have with INAC.
  
  
• Capacity issues at all levels: There are staff shortages at the national, regional and community level. Existing staff at the community level need training in the purposes and methods of good data collection; but there are also capacity shortages at the regional and national levels in data management and interpretation, and in reporting back to regions and communities for purposes of outcome-based program planning.

Overall Evaluation Recommendations

Outlined below are recommendations applicable to all three programs:

1. Create an Evaluation working group of INAC Audit and Evaluation Sector and Program Staff, and First Nations representatives to develop outcomes indicators for all three programs that will be meaningful and acceptable at the community level;
   
   
2. Develop a standard data system and standardization of indicators for all regions to facilitate comparability;
   
   
3. The Working Group Created should have a discussion of OCAP[Note 1] principles regarding program data.

Income Assistance Program Findings

The Program Continues to be Relevant: Community profiles prepared for each of the case study communities, as well as regional and national data, show that high levels of poverty, low educational attainment levels, high unemployment levels; poor housing conditions and overcrowded housing persist in First Nations communities, highlighting a continued need for income assistance until long term alternatives and solutions are found.

Basic Needs Are Reportedly Not Met: Despite the fact that almost 90% of the IA budget is directed to basic needs, there is virtually universal agreement by staff and end-users that basic needs are not being met. Explanations for this may include the following:

• INAC follows provincial rates, which are based on urban, rather than remote rural needs. Amounts payable under the income assistance program are based on the eligibility criteria and rate schedules of the reference province or territory.
  
  
• More than price differentials for goods in most cases, the costs of transportation are prohibitive for IA recipients, who reported having to pay taxi fares to nearby towns to buy clothes and/or food, in the absence of public transportation.
  
  
• A complex shelter regime means that shelter costs are not paid in all regions, with the result that many recipients are left with not enough revenue for food, clothing, transportation, and personal items after rent has been deducted from their benefits.
  
  
• Utility costs, particularly home heating, are reportedly higher in many First Nations communities than national averages. Expensive electric heat is often the only option, and housing stock in poor condition exacerbates heating costs.
  
  
• Transportation is a basic need in the communities surveyed. Lack of transportation is a barrier to meeting other basic food and supplies needs, but also to accessing employment.

Number of Single Recipients Rising: The proportion of single recipients to the entire caseload has grown over the past few years. Community perceptions reported in this evaluation are that the majority of these are youth who choose to access income assistance when they turn eighteen. The evaluation was not able to access the detailed client information that would support or negate this perception, although both community members and service providers alike made the observation. More than 50% of those who participated in the end-user survey were under the age of 35.

Supports for Effective Long term Solutions Widely Lacking: It is acknowledged that active measures and integrated approaches, including client case management; tailored approaches to employability and job readiness barriers; addressing education and training needs; and providing supports to parents such as training funds and child care, are effective at finding long-term solutions to low employment and high welfare dependency levels. While such approaches are being implemented by some provinces,, only a very small percentage (less than 2%) of INAC's IA expenditures go to supporting such measures. If the desired outcome is to alleviate hardship by supporting employable First Nations members currently on IA into long term employment, such measures will be part of an integrated and complex social development strategy by INAC. In addition to ensuring that basic needs are met in the short term, the long term solutions will require the funding and support of such measures in partnership with other relevant agencies such as HRSDC and provincial education and training bodies, as part of an integrated national strategy.

Long term Outcomes of Employment Support Measures Unknown: For the employment support measures that INAC has funded, there is insufficient evidence to be able to comment on their effectiveness. The measure currently used, "Person Months of Employment" is a rough measure, and does not provide information about the long term employment outcomes for an individual, or other critical information regarding employability and long term "alleviation of hardship." The survey of end-users in this evaluation was not able to capture this information, as those who would have successfully transitioned to work from welfare were not part of the sample.

Staff Capacity to Engage and Support Community Service Providers is low: National and Regional INAC staff capacity is not sufficient to provide the supports and engagement needed by community service providers to build their own capacity and enhance their service provision. Key informants at all levels reported this issue.

INAC's role and mandate for IA is questioned within the department: Most key informants within INAC see their role as "funder," and some question whether INAC should continue providing income assistance, or whether this is a provincial role. As one of the chief policy drivers is currently the requirement for INAC regions to match provincial rates and eligibility requirements, it is difficult for INAC to set meaningful IA policy; to do so would require collaborative discussions with the respective provincial ministries.

Income Assistance Program Recommendations

1. Develop, in partnership with relevant bodies such as HRSDC, AFN, and provincial ministries, an integrated strategy to address on-reserve labour and employment needs. The strategy would recognize the complex and unique needs of the on-reserve unemployed, such as restricted access to labour markets; multiple employability barriers; transportation needs; and the need for child care and other necessary supports while in training or educational upgrading programs.
   
   
2. In the near term, until a strategy to address the causes of welfare dependency is in place, and achieving the desired outcomes; and to provide better support for basic needs: review the 2% funding increase policy to assess whether it is meeting First Nations IA costs;
   
   
3. In the near term, address INAC staffing shortages and training needs at the national and regional levels;
   
   
4. In the near term, fund a representative sample of community needs assessments that will provide meaningful cost measures for items such as shelter, utilities and transportation;
   
   
5. In the longer term, create a working group of INAC, First Nations and Provincial representatives to develop a strategy for addressing IA jurisdictional and funding issues, including a discussion of the costs of needs in rural/remote communities;
   
   
6. Take the lead in initiating an integrated education and training strategy with HRSDC, Aboriginal organizations, and relevant provincial ministries, to address the education needs of First Nations youth in particular, as a way of reducing the number of youths who choose welfare over further education and/or employment;
   
   
7. Strengthen links with other relevant departments such as HRSDC to enhance information sharing so that long term employment outcomes can be measured, and develop more refined outcome indicators for future evaluation activities.

National Child Benefit Reinvestment Program Findings

The Program is Relevant: Case study community profiles as well as national and regional statistics show that continuing levels of poverty, high levels of unemployment, low educational attainment levels, scarcity of jobs, and the high percentage of children in First Nations populations show a continuing need for supports to low-income parents and children.

The Program is Meeting Community-Defined Needs: The evaluation found that NCBR programs are valued and responding to community-defined needs. The flexibility of the program, while posing challenges for reporting, appears to be a strength from this perspective. In particular, programs that provide hot breakfasts and/or lunches to children are highly valued by parents and educators, as are cultural teaching programs.

Scarcity of Meaningful Outcomes Data: Reporting requirements for the program result in very little meaningful outcomes data. Reporting is done annually and normally reports are on outputs only, such as numbers of participants and activities undertaken. Reporting is subject to over-counting of participating families: the same family or child can be counted numerous times if participating in several programs.

One Case Study community has educational outcomes data: One case study community, Walpole Island First Nation (Bkejwanong) applied much of its NCBR budget to school-based initiatives at the local elementary school, (based on assessed needs of students) such as individualized speech and art therapy for trauma and behavioural problems; a hot breakfast and lunch program; tutoring programs; purchase of sports equipment; and field trips for students. The school has improved EQAO scores as outcomes data.[Note 2]

Attribution of Outcomes is Extremely Difficult: Overall, attribution of outcomes for NCBR projects is extremely difficult, as the projects are often integrated with other programs, and "alleviation of poverty" is a complex, long-term and multifaceted outcome that would be attributed to many interacting factors.

National Child Benefit Reinvestment Program Recommendations

1. Initiate a formal discussion with First Nations organizations and INAC regional staff on the most effective way to address reporting issues so that meaningful outcomes can be measured;
   
   
2. Recommend to regions that they adopt a management regime similar to Saskatchewan region, which does the following:
   • Outlines clear expectations;
   • Sets targets in collaboration with First Nations;
   • Communicates the intent of NCBR;
   • Provide project proposal support.
   
   
   
3. Revise reporting mechanisms to avoid multiple counting of program participants.

Assisted Living Program Findings

The Program is of Vital Relevance in Communities: The evaluation found that AL services being delivered in communities are meeting a real need and are highly valued by community members and AL end-users alike, as a means of assisting the elderly and disabled to remain in their homes and have an improved quality of life. Aboriginal health and demographic trends indicate that need for the program will rapidly increase in the near future.

Significant Gaps in Services: Service gaps noted by the evaluation include:

• Children's AL needs are not being met by the program. While the program authority for children's special needs for Assisted Living has been in place since 2003, no funding has accompanied this authority. Evaluators were told that in some cases, parents are giving Child and Family Services custody of their children so that assisted living services can be accessed off reserve.
  
  
• In most jurisdictions, needs of the developmentally disabled or brain-injured are not covered on-reserve.
  
  
• None of the communities visited had foster care or group home facilities. Only about 1% of the AL service profile is foster care.
  
  
• Supportive housing that would allow the frail elderly to "age in place" is not provided under the AL program
  
  
• Respite care, although covered under program authorities, is seldom provided. None of the case study communities reported providing respite care for family caregivers.
  
  
• After-hours and week end needs of clients are generally not being met.

Integration with FNIHCC at Community Level: Community visits showed that, at the community level, the in-home AL services, in their regional variations, are de facto integrated with Health Canada's First Nations and Inuit Home and Community Care Program (FNIHCC) at the service delivery, if not the funding, level. This may be due in part, to lack of a service delivery funding component, but is most likely the practical and common sense solution devised by First Nations service providers to efficiently meet client needs. The requirements of double reporting and keeping funding separate are reported as onerous in some cases.

Desire for Elderly to "Age in Place": Community visits showed a desire in most communities for institutional care or other higher levels of care alternatives that will allow the frail elderly to "age in place." Families and communities, as well as the clients themselves, reported the impacts of losing elders to off-reserve locations when they require higher levels of care than can be provided by the present AL in-home component. This finding was supported by opinions of the expert panel on assisted living, who noted this as a general trend in provincial programming.

Funding Insufficient to Meet Needs: Funding is capped at 2% growth, and is not sufficient to meet the needs of either the in-home or institutional component of the program. Institutions need more funds for wages in particular. The funding formula relies too heavily on population rates and not on defined community needs.

Human Resource Challenges: One of the key findings of the evaluation is that the program is critically short of staff at the community level. Existing staff are noted as dedicated, but in danger of stress and burnout. Wages are not at par with off-reserve professional counterparts, making recruitment and retention of staff difficult.

Assessment of Outcomes Challenging: The program does not have an evaluation framework with defined outcome indicators. While supporting clients to "functional independence" is a program objective, the term is neither defined nor does it have supporting indicators. Even with such a framework, attribution of outcomes would require assessments according to a standard assessment tool, and would still be difficult in light of the complexity of interacting factors related to functional independence. More refined indicators would make attribution more achievable.

Assisted Living Program Recommendations

1. Continue the initiative to devolve the funding and authority for the in-home component to the FNIHCC program;
   
   
2. Secure Treasury Board funding for children's AL services, to resource the program authority in place since 2003;
   
   
3. Coordinate discussions at the Federal / Provincial / Territorial and First Nations level to address other AL service gaps, resolve jurisdictional issues and develop an integrated approach to a full continuum of care model;
   
   
4. Fund community-based AL needs assessments and use the information as a basis for reviewing current funding levels.

1.1 Purpose and Structure of the Report

This report provides a synthesis and analysis of all the data collected from all the lines of evidence used in this evaluation project.

This report is structured as follows:

• Section 1.0: Introduction and Background to the Evaluation;
• Section 2.0: Evaluation Methodology;
• Section 3.0: Income Assistance Program Evaluation Findings;
• Section 4.0: NCBR Program Evaluation Findings; and,
• Section 5.0: Assisted Living Program Evaluation Findings.

Each section that discusses findings of a particular program (i.e., Sections 3 through 5) provide a description of performance measurement, program effectiveness, program impacts, and conclusions and recommendations specific to that program.

1.2 Objectives of the Current Evaluation

INAC's Evaluation, Performance Measurement and Review Branch of Audit and Evaluation Sector (AES) contracted evaluations of the IA and NCBR programs in December 2007 while the AL program completed its own program-led review in November 2007. Upon review of the evaluation reports, the Treasury Board noted that it requires additional evaluation work be conducted on these programs in order to provide information about the "continued relevance, effectiveness and impact of the three programs" (Statement of Work, p. 1).

Accordingly, Audit and Evaluation Sector has committed to conducting follow-up impact evaluations of the IA, NCBR and AL programs, focused on an assessment of the impact of these programs on their end-users and assessing the programs' effectiveness with respect to their current program objectives. For the sake of cost effectiveness, a single evaluation of the three programs was conducted simultaneously.

The objectives of the current evaluation of the IA, NCBR and AL programs are:

• To report on the integrity / reliability of program data by conducting a review of the data collection capacity within the three programs;
  
  
• To assess program effectiveness with respect to the achievement of current program objectives; and,
  
  
• To assess the impact the programs are having on end-users.

1.3 Background to the Social Programs

This section of the report contains a brief background of the Income Assistance, National Child Benefit Reinvestment, and Assisted Living programs.

1.3.1 Income Assistance

Program Authorities and Objectives

In terms of total expenditures, Income Assistance (IA) program is the largest of INAC's Social Development Programs and is the fourth largest welfare program in Canada. In 2006-2007, IA provided basic services to approximately 150,000 individuals in 630 First Nation (FN) communities.

The program authority for the IA program comes from a decision of Treasury Board. INAC is required to provide social assistance benefits comparable to those provided to other Canadian citizens in the respective provinces. Following Cabinet's approval of the general parameters of the program, a submission is made to Treasury Board outlining how funding is to be spent.[Note 3]

The program's objectives are to: 'provide support for the basic and special assistance needs of indigent residents of First Nation reserves and their dependants'[Note 4] (in the case of contributions), 'provide financial assistance to meet basic daily living requirements as per program terms and conditions'[Note 5] (in the case of grants), and to 'provide support for eligible FN individuals to receive pre-employment training, support and / or other active measures.'[Note 6]

The program attempts to meet these objectives through providing funds for basic needs, and increasingly in recent years, assisting recipients to address barriers to employment. Jurisdictions such as Ontario and Alberta have implemented Active Measures programs that are integrated with social assistance. Specifically, the income assistance program in Ontario ("Ontario Works") is geared to active measures (the "employment assistance" component); Alberta's program also has a strong active measures component, as does British Columbia, through it's "Training Employment Support Initiative" (TESI) which allows communities to set up programs for members on social assistance in order to develop the skills to enter vocational training, educational programs, or employment.[Note 7] The First Nations programs in these regions are delivering active measures programming to varying degrees, as determined by their capacity and supports available to do so.

The four main funding components of the IA program are:

1. 'Basic needs - financial assistance to cover food, clothing and shelter;
   
   
2. Special needs - financial assistance for special needs allowances for goods and services that are essential to the physical and social well-being of an IA client but not included as items of basic needs, such as special diets, etc.;
   
   
3. Pre-employment supports - assistance may be provided to support activities that may include counselling and life skills, training in essential skills, transfers of income assistance entitlements to training and work experience projects; and
   
   
4. Service delivery - funding provided to First Nations administrators such as Tribal Councils, Chief and Council or the host province / territory to cover service delivery.'[Note 8]

The IA program's expected results are:

• 'The alleviation of hardship;
  
  
• The maintenance of functional independence on reserve to standards of the reference province or territory; and,
  
  
• Greater self-sufficiency for First Nation individuals and communities.'[Note 9]
  
  

Generally speaking, there are two types of generic funding arrangements used with First Nations that have not entered into their own self-government agreements, Comprehensive Funding Agreements (CFAs), which are 1-year in duration, contain programs funded through contributions, and have a component of flexible transfer payments and grants; and Canada / First Nations Funding Agreements (CFNFAs), which are block-funding agreements for up to 5 years and can include funding from other federal departments.[Note 10]

Currently 81,000 recipients and 159,000 beneficiaries are supported through the Income Assistance program across the country. In 2008-2009, IA had projected expenditures / operating budget of $696.6 million, with an anticipated annual increase of approximately 2% each year until 2012 – 2013.[Note 11]

The program's current policy is to provide IA services comparable to those in the provinces; accordingly, the various regions attempt to mirror rate changes in their jurisdictions, but are not always able to mirror types and levels of service. Aside from funding the program, INAC provides policy guidance and compliance monitoring.

In total, 18% of the existing IA program budget is dedicated to meeting basic needs of shelter (shelter allowance).[Note 12] Shelter Allowance (SA) is made available by each provincial government in Canada to individuals of low-income, experiencing poverty, or on social assistance. Available funds to address shelter needs vary across Canada and, in some communities, may only be accessed by those holding a rental agreement.

Regional Differences

In most regions, INAC provides funds directly to First Nations or Tribal Councils, who in turn deliver the program to eligible recipients; although there are regional variations to this model. In Ontario, the program is the responsibility of the province but is delivered by First Nations delivery agents due to the Memorandum of Agreement Respecting Welfare Programs for Indians (often referred to as the "1965 agreement") between the Government of Canada and Ontario. Under the 1965 agreement, the province assumed responsibility for providing the income assistance services to the 110 First Nations in Ontario, while INAC funds the provincial programs on-reserve. In Ontario, First Nations are the effective delivery agents for IA. INAC pays the "municipal" share (20% benefits, 50% administration costs), direct to the First nations. The province funds the provincial share (80% benefits, 50% of the administration costs) to deliver IA to recipients "normally resident on reserve"[Note 13] and the province then charges these costs back to INAC. The result is that INAC reimburses the province approximately 93% of the IA costs, as per the 1965 agreement.

A "second level" of service is also provided to First Nations in Quebec, Manitoba, British Columbia, and Ontario, which helps provide tools and support to First Nation communities. In Quebec, it is coordinated by First Nations and is not funded by INAC. In British Columbia, second level service is provided by the First Nations Social Development Society, a First-Nations run group that works closely with INAC regional staff. In Manitoba, INAC funds Tribal Councils as second level service delivery agents: their role is to provide "on the ground" support for administration. Ontario is currently funding a pilot project at 19 sites in which higher levels of administration funding are provided to "second level" entities, either tribal councils or groups of First Nations; the final evaluation of this pilot is forthcoming within the year.

1.3.2 National Child Benefit Reinvestment

Program Authorities and Objectives

The National Child Benefit Reinvestment (NCBR) is a component of Human Resources and Social Development Canada's (HRSDC) National Child Benefit (NCB) initiative. The NCB is a federal/provincial/territorial initiative an initiative under the Social Union Framework Agreement that is aimed at seeking solutions to child poverty; promoting attachment to the workforce; and achieving a greater degree of harmonization of programs. INAC's National Child Benefit Reinvestment (NCBR) program is an on-reserve counterpart to HRSDC's off-reserve NCBR administered by INAC through an Interdepartmental Letter of Agreement with HRSDC which was implemented in 1998 as a federal / provincial / territorial initiative, uses funds derived through a complex process of offsets from IA/NCBS savings.

The overall NCB initiative is coordinated through HRSDC; INAC's NCBR component is funded through Treasury Board (Vote 15 – Grants and Contributions).The NCBR program may operate under agreements made with a province, a territory, or a First Nation. The agreements may take several different forms including: MOUs between INAC and the province / territory / First Nation; bilateral agreements between INAC and the province / territory / First Nation; and ad hoc joint working relationships between INAC and First Nation authorities.[Note 14]

The program's objectives, in terms of its indigent residents on-reserve are: 'to help prevent and reduce the depth of child poverty, to provide incentives to work by ensuring that low income families with children will always be better off as a result of working, and to reduce overlap and duplication through the simplified administration of benefits for children.'[Note 15]

The program attempts to meet these objectives through the provision of funds to address interests in the five following areas:

• 'Childcare - Programs that enhance child care facilities to enable more low-income families to access space for their children;
  
  
• Child Nutrition - Programs to improve the health and well-being of children by giving them nutritious meals in school and nutritional education for their parents. This activity includes the delivery of food hampers for low-income families;
  
  
• Support to Parents - Programs to help parents give their children a sound start in life, including training in parenting skills and drop-in centres;
  
  
• Home-to-Work Transition - Programs intended to improve employment prospects, such as skills development and summer work projects for youth; and
  
  
• Cultural Enrichment - A broad category to teach traditional culture, provide peer and family support groups and bring together community elders, children and youth.'[Note 16]

The NCBR program's expected result is, 'by increasing income and employment support to all low-income families on reserve and streamlining government programs, low income families will recognize increases in children's health and development, increase in school readiness and ability to learn, and parents will fare better in the labour market, achieve a greater degree of financial independence for themselves and their children and participate more fully in their communities and Canadian society.'[Note 17]

INAC's NCBR program is project-based and proposal-driven. First Nations communities apply for funding for programs which fall under one or more of the following five activity areas: Childcare, child nutrition, supports to parents, home-to-work transition and cultural enrichment (described above). NCBR projects vary broadly in size and scope ranging from diapers for families in crisis to job counselling and training programs.

The flow of funding moves from Indian and Northern Affairs Canada (INAC) headquarters through to the regions, who in turn fund First Nations community projects based on submitted project proposals, although there are regional variations to the model. In general, First Nations have a great deal of flexibility in the spending of NCBR funds, within the parameters defined by their regionally-approved proposal. In 2008-2009 the NCBR program received $51.4 million in grants and contributions.

Regional Differences

There are differences in how the program is delivered across Canada; for example, the program is not delivered in the provinces of Manitoba, New Brunswick or Newfoundland and Labrador. Some provinces elect to have the NCB supplement go directly to families on reserve in addition to their IA benefits, rather than recover and reinvest the savings through NCBR programming.

1.3.3 Assisted Living

Program Authorities and Objectives

INAC's AL program (formerly Adult Care) provides support for the tasks of daily living to persons on-reserve with chronic illnesses and disabilities that restrict their functional independence.[Note 18]

The Assisted Living (AL) program is implemented under a separate policy authority and funding authority. These authorities are derived from Cabinet and the Treasury Board respectively.[Note 19]

The program's objective is 'provide social support programs which meet the special needs of infirm, chronically ill and disabled persons at standards reasonably comparable to the relevant province / territory of residence.'[Note 20]

The major components of the AL program are:

• In-home Care: Non-medical personal care (e.g. washing hair, preparing meals, housekeeping); at present this represents 58%[Note 21] of the service profile.
  
  
• Foster Care, which is comprised of supervision and care in a family setting; at present this represents less than 1%[Note 22] of the service profile; and
  
  
• Institutional Care: which provides Type I and Type II care in institutions. There are currently 32[Note 23] personal care homes (PCHs) on reserve across the country. If clients must go off reserve to access this type of care, INAC funds the care through reimbursement to the province. At present, this represents approximately 38%[Note 24] of the AL service profile. In the case of British Columbia, when a person is eligible and assessed at up to Intermediate Care level III, the Region funds the First Nations who have clients in institutions. The First Nations' administrating authority, in British Columbia, funds the comfort allowances and, in some cases, the user fees for clients in institutions through Basic, Social Assistance. Shelter is paid while a person is in temporary residential palliative care.

The AL program's expected results are:

• 'The alleviation of hardship;
  
  
• The maintenance of functional independence on reserve to standards of the reference province or territory; and,
  
  
• Greater self-sufficiency for First Nation individuals and communities.'[Note 25]
  
  

The AL budget also funds the institutional care of those individuals who must go off-reserve for this service. In 2008-2009, the AL expenditures / operating budget of $82.8 million, with an anticipated annual increase of approximately 2% each year until 2012-2013.[Note 26]

INAC's current involvement in AL primarily involves providing funding to First Nations, who in turn deliver non-medical assisted living programs and services to eligible community members. Health Canada, First Nations and Inuit Health Branch's First Nations and Inuit Home and Community Care Program (FNIHCC) provide medical and nursing care. The INAC and Health Canada programs are a part of a continuum of care and attempt to avoid duplication of services.

Regional Differences

There are variations across the country in the way that the AL program is established, implemented, and the programs / initiatives they fund (e.g. some regions fund Community Living[Note 27] while other do not). In general, the program is funded through contribution agreements and delivered by First Nations.

In Saskatchewan, there are instances where First Nations deliver AL services and other occasions where INAC delivers services. For example, INAC has a "paylist" of clients living off-reserve who are funded. Also, in Saskatchewan, residents on-reserve who need institutional care are paid by INAC to live in institutions off-reserve.

In Ontario, the Home Care component of the AL program (called "Homemakers") is a provincial responsibility (based on the 1965 Agreement). The Ontario Ministry of Health and Long-Term Care is responsible for program administration, and INAC reimburses the province through a funding formula; for institutional care, the institutions bill INAC directly each month. In Ontario, INAC subsidizes the gap between what low income persons can pay and the cost of the care for non-medical in-home care. This is an income-tested program (i.e. means test). In general, the provinces are further ahead in service provision than INAC. Ontario, however, is a unique case in terms of comparability, following the 1965 agreement; other regions are having difficulty keeping up with the level of provincial services.

In general, the provinces fund construction (e.g., for Personal Care Home facilities) for licensed institutions on-reserve, although there was a twenty year moratorium on such building that has only recently been lifted. Alberta and New Brunswick, however, do not provide funding for institution construction, or the Institutional Care component (i.e., Personal Care Homes) of the AL program.

This evaluation project was conducted through a series of activities:

• Development of the evaluation framework and methodology;
• Preliminary consultations;
• Document and literature review;
• Administrative and financial data review;
• Expert panels and key informant interviews; and,
• Case studies.

Each particular set of activities is briefly described in the following subsections.

2.1 Development of the Evaluation Framework and Methodology

This evaluation focused on the following five primary review issues:

• Data Collection & Performance Data System Adequacy - what type and quantity of program data is collected; are current indicators relevant; and is the current data collection system capacity adequate to measure program performance against intended outcomes?
  
  
• Relevance - do the programs continue to be consistent with departmental and government-wide priorities, and do they realistically address actual needs?
  
  
• Effectiveness - are the programs being administered and delivered in the most efficient and effective manner possible?
  
  
• Impact on End-Users- What impacts are the programs having on program recipients, and are these consistent with intended program outcomes?
  
  
• Cost-Effectiveness - are appropriate and efficient means being used to achieve outcomes, relative to alternative design and delivery approaches?[Note 28]

These issues formed the foundation for the framework for the evaluation.

For each review issue, one or more questions were developed that could be applied to several or all of the programs being evaluated. For each question, performance indicators and evaluation activities were identified. This series of questions, performance indicators and evaluation activities formed the framework for the evaluation. The framework formed the basis of the detailed methodology for the evaluation.

2.2 Preliminary Consultations

At the outset of the evaluation of the three projects, the evaluation team consulted with INAC headquarters and regional representatives (identified as preliminary consultation participants). A total of 13 preliminary interviews were conducted by the evaluation team, including 6 from INAC headquarters, one from each INAC Regional office (Atlantic, Quebec, Ontario, Manitoba, Saskatchewan, Alberta and British Columbia), as well as a consultant. The individuals consulted included those belonging to INAC's Evaluation branch, social program policy analysts, senior statistical officers, and managers of program funding and program operations.

The purpose of the preliminary consultations was to:

• Identify key documents, literature, and data sources;
• Identify key Informants;
• Identify Case Study Communities;
• Determine adequacy of data collection;
• Identify local / community contact to facilitate with applying end-user surveys; and
• Obtain high-level insight into program(s) successes and limitations.

The preliminary consultation activity began with the evaluation team identifying key INAC staff, in consultation with the client, along with contact information with each identified participant. The evaluation team developed an invitation letter, questionnaire, interview guide and interview compilation template prior to conducting the preliminary consultations. The evaluation team then contacted each participant to schedule and interview, and conducted the interviews at the agreed upon date. All interview data were entered into the client-approved interview compilation template.

2.3 Document and Literature Review

2.3.1 Document Review

The document review activity was designed to review a range of documents to gain an understanding of program objectives, performance measurement frameworks, performance of programs as assessed in program reports, previous audits and evaluations; and comparative programs delivered at the provincial level.

In the course of this activity, the evaluation team collected documents (either hard or electronic copies) either provided by – or identified by – the client and preliminary consultation participants, as well as relevant documents identified on the Internet. The evaluation team entered the document review findings into the client-approved document review template and upon further analysis, produced a document review report.

Documents consulted included program manuals, reports and reporting guides, program terms and conditions and other program-related documents; RMAFS/RBAFS; Annual Reports, previous evaluations and studies, and AG reports. Primary attention was paid to those documents that were provided by or identified by the preliminary contacts interviewed. The document review process provided another line of evidence for the evaluation and assisted the evaluation team in understanding the specifics about each of the three INAC social programs and how they are implemented in different provinces.

2.3.2 Literature Review

The evaluation team conducted a review of limited domestic and international literature to gain an understanding of the state of knowledge and key issues related to the three programs, particularly in terms of relevance. The evaluation team members, based on their knowledge of the field and in consultation with INAC's Evaluation Manager, developed a list of literature sources for review, with a focus on those sources which helped to frame the team's investigation of the three projects. The literature review provided another line of evidence for the evaluation.

The literature review process involved Internet searches for relevant literature, obtaining literature identified during the preliminary consultations, and entering the literature review findings into the client-approved literature review template. Based on the findings, a literature review report was produced.

2.4 Administrative and Financial Data Review

2.4.1 Administrative Data Review

The administrative data review was conducted at the national, regional and community levels (the latter was conducted as part of the case study methodology, described below). Administrative data was reviewed to provide information on program outputs and outcomes, budgets, the type of data being collected and what it is able to say about impacts; the extent to which program objectives are being met; and the level of consistency of data across regions.

The Administrative Review for the three programs was carried out with the assistance of our primary contact in this area at headquarters level, and with the regional INAC staff responsible for data reporting, coordinated via the Senior Evaluation Manager and the Audit and Evaluation Coordinators in the Regions. The evaluation team members worked directly with these staff members to gather and review the relevant social program data.

The types of data reviewed included:

• National roll-ups of regional level data;
• First Nations and Inuit Transfer Payments (FNITP) and Recipient Reporting Guide;
• Annual Reports;
• Monthly Reports;
• Quarterly Strategic Outcomes;
• Annual Audits of Program Expenditures;

The administrative data findings were entered in the client-approved data collection template.

2.4.2 Financial Data Review

The financial data review included an examination of program budgets, variance reports, contribution agreements, and year-end reports from partner organizations: these are all valuable sources of information on the financial management of programs.

The administrative and financial data review was conducted at two different stages in this assignment: initially, as a review of the initial information available, followed by a review of other documents / databases available at INAC Headquarters, Regional Offices and in the Case Study communities visited.

The financial data findings were entered in the client-approved data collection template, and synthesized, in combination with the administrative data, into a report.

2.5 Expert Panels and Key Informant Interviews

2.5.1 Expert Panels

Two expert panels were conducted in December 2008: one panel focused on the Assisted Living Program, and included experts in home care and assisted living from national, provincial and Aboriginal organization levels. The Income Assistance/NCBR panel was comprised of academic and service provider experts from the national, provincial and Aboriginal organizational levels.

Groups represented included:

• Representatives from Aboriginal organizations (national and provincial);
  
  
• Provincial government staff from linked/comparable program areas and Ministries;
  
  
• Representatives from linked social development programs such as First Nations and Inuit Home and Community Care, HRSDC / AHRDA;
  
  
• Specialists in Disability / Assisted Living issues, Income Assistance, and the National Child Benefit Reinvestment programs;
  
  
• Specialists on Child Poverty; and
  
  
• Academic specialists on social assistance best practices.

The Expert Panels were intended to provide expertise and insight related to the current state of knowledge in the social assistance and back-to-work fields, with respect to the IA, NCBR, and AL programs specifically. The goal of the expert panels was to address questions pertaining to program relevance, effectiveness and cost-effectiveness.

The evaluation team identified a preliminary list or relevant experts, in consultation with the client, and collected contact information on the experts.

The evaluation team conducted 2 Expert Panels, each comprised of approximately 3-5 experts. One panel addressed the topic of Assisted Living services and the other the topic of social assistance and active measures, and the NCBR program. The rationale for the approach of grouping the NCBR and IA programs has to do with the notion that the NCBR and IA are closely linked in terms of objectives and funding source. The expert panel sessions averaged approximately 2.5 hours in length: some members participated by teleconference and some in-person. A total of 9 experts participated in the expert panels (5 AL experts and 4 social assistance / active measures and NCBR experts).

The findings of the Expert Panel discussions were synthesized into an Expert Panel and Key Informant Report.

2.5.2 Key Informant Interviews

Key informant interviews were conducted during the period of October – December 2008 and included both IA, NCBR, and AL program stakeholders and non-stakeholders; examples of non-stakeholders include program managers in comparable provincial program areas. Key Informants included representatives of the following roles:

• INAC national and regional program managers and staff;
  
  
• Provincial government staff from linked / comparable program areas;
  
  
• Tribal Council/ Nation or First Nations Health Directors; and
  
  
• Directors of Personal Care Homes.

The objective of the key informant interviews was to insight on data collection and performance data system adequacy; program relevance; program effectiveness; and the programs' impact on end-users.

Key informants were identified by the preliminary consultation participants. The evaluation team submitted the list of key informants to the client for approval.

Wherever possible, interviews were conducted in person; those who were either unavailable at the time that the evaluation team was in the region or was located outside the Ottawa area or Case Study community were interviewed by phone. A total of seventeen key informant interviews were conducted.

Interview findings were summarized and synthesized into a separate report.

2.6 Case Studies

The rationale for the case study approach was to spend a concentrated amount of time (2-3 days) in each of ten communities as a way of maximizing data collection. During this time a number of data collection methods were employed to gather as much information about program impacts as possible in the allotted time.

A Case Study methodology allows for "depth" of data analysis at the expense of breadth. Rather than administer a large national survey that would likely yield slim results, a Case Study methodology increased the possibilities of retrieving meaningful data, allowing the team to interact with program administrators, program users, service providers, and community members at large; provided for access to program files on site; and allowed the evaluators to make direct observations on the delivery of the program.

Preliminary consultations resulted in the identification of a possible list of communities; final selection of the sample was achieved through the assistance of regional INAC staff. The list revised as several selected communities decline to participate. For each region, "first choice" communities and "back-up" communities were identified.

The identification of these prospective communities was guided by the following selection criteria:

• The communities, where possible have all three programs being evaluated;
  
  
• Accessibility (how open the community is to outsiders);
  
  
• The communities are appropriately represented in terms of location, remoteness, size, economic circumstances, success and hardships;
  
  
• Both CFNA and CFA funding arrangements are represented; and.
  
  
• The program data will be comparable, where possible.
  
  
• Two case studies done in each of B.C. and Manitoba because of the number of First Nations and the number of residents in First Nations in each of those regions.

When conducting case studies in Aboriginal communities, it is essential that the purpose of the evaluation, evaluation activities being conducted, and types of community stakeholders to be involved in the evaluation are clearly communicated to both the community leadership and – assuming the leadership (Chief and Council) are supportive – to the community members as a whole. Initial contact with each selected community was in the form of an introductory letter sent by INAC to the Chief and Council of the community. This was followed by phone calls to key IA contact persons in the community by the evaluation team.

A total of ten case study communities were finally selected and conducted (see Table 1).

2.6.1 Methodology for Carrying Out Case Study Community Visits

The evaluation team visited the 10 case study communities between the weeks of November 10 and December 15, 2008. A team of two evaluators participated in each community visit: each team had a minimum of one (and in most cases, two) senior evaluators with extensive experience in conducting consultations in First Nation communities. The evaluation team spent a total of approximately 50 person days conducting community visits.

During the community visits, the following data collection methods were used:

• Service Provider Interviews (Staff and Administrators);
  
  
• Program file review;
  
  
• Focus Groups with end-users of IA and NCBR;
  
  
• Survey of AL end-users; and
  
  
• Community survey of a targeted sample of residents / IA recipients.

In the course of the ten community case study visits, the evaluation team conducted a total of:

• 77 interviews with administration/community service providers;
  
  
• 181 surveys with community members;
  
  
• 8 focus groups involving a total of 78 participants;
  
  
• 32 surveys with AL end-users;
  
  
• 1 survey with an NCBR end-user; and
  
  
• Collected and reviewed 129 documents / data sources.

When travelling to regional sites for case studies, evaluators conducted in-person interviews with regional INAC staff in most regions; Saskatchewan and Quebec interviews were conducted by telephone, and a site visit was not completed in Ontario due to restrictions of time and staff availability.

A key aspect of the case study methodology was the hiring of a local research coordinator/assistant to assist in administration of community door-to-door survey; coordination of the end-user focus group; end-user surveys of AL program users; and translation (when necessary). Local research coordinator/assistants were successfully employed in 5 of the case study communities; in another community, social development staff provided the supporting role that the research coordinator/assistant would normally provide. The services provided by the local research coordinators/assistants (where available) greatly assisted in ensuring that the maximum number of community members participated in the surveys and focus groups.

Another key aspect of the project methodology was flexibility and adapting to the particular circumstances of each community during the case study visit. In communities where no local research coordinator/assistant was available, the evaluation team conducted the surveys and focus groups. Further, the evaluation team often relied on the advice of the First Nations office staff regarding methods for maximizing community participation.

In some communities, the IA/NCBR end users surveys were conducted by approaching recipients at the welfare office as they collected their cheques; in other communities, the approach used was going door-to-door with the local research assistant; and in some communities, a combination of both approaches was successfully employed. In one community, additional surveys were conducted by the local research coordinator/assistant following the community visit by the evaluation team – these additional surveys were emailed to the evaluation team. For the AL end user survey, in most communities the evaluator accompanied the home care coordinator, home care nurse, or home support worker to client homes; in some communities, AL end users were also surveyed in public meeting places (e.g., local shopping mall).

Given the fact that several of the community visits were shortly prior to Christmas (a time of year that is very demanding on First Nations office staff) and/or shortly following a death in the community (which usually results in low community willingness to participate in consultation activities), circumstances were not always favourable for maximizing community participation; nevertheless a robust community sample was achieved.

For each case study community, a separate case study report was produced, which synthesized the findings from all data collection methods used in the community visit.

3.1 Performance Measurement

Evaluators requested national, regional, and community level IA administrative and financial program data for a ten-year period, from 1997/1998 to 2007/2008. Requested data included the number of clients accessing the program, clients' characteristics, number of staff delivering the program, program expenditures and services by component, funding arrangement, employment creation, and welfare dependency rate.

3.1.1 Performance Measurement Framework

Table 2 below outlines the data reporting system for each of the three programs based on the Recipient Reporting Guide (RRG)[Note 29] and feedback from preliminary consultations. The table identifies and links each of the programs to its corresponding reports, funding agreements, data collection and report preparation body, report audience, reporting period, and the data being reported (by output and outcome).

3.1.2 IA Data Gaps

The data seen in Table 3 below represents, according to both the RRG and what was heard in the preliminary consultations, the IA output and outcome data INAC ideally intends to collect annually.

In order to comment on the adequacy of performance data it is necessary to determine what data is actually collected by the programs. In order to do this, administrative and financial data was gathered at the national, regional, and community levels for a ten-year period covering the years 1997/1998 through to 2007/2008. The data sought included the number of clients accessing the program, clients' characteristics, number of staff delivering the program, program expenditures and services by component, funding arrangement, employment creation, and welfare dependency rate.

National Level

A review of the national level IA data gaps showed that significant outcome indicators, which could speak to program success and overall performance, were missing. Salient outcome data gaps at the national level include:

• Total IA Expenditures data (missing for the years 1997-2005);
  
  
• Number of Person Months of Employment Created (missing for years 1997-2000-); and
  
  
• Service Delivery expenditures (missing for the year 2007 – 2008).

Regional Level

Similar to the national level, a review of regional level data gaps showed that outcome indicators which could provide insight into program performance and success were missing. Salient data gaps at the regional level include:

• Total IA Funding Expenditures data (missing for the year 1997-2005);
  
  
• Number of Person Months of Employment Created and program expenditures data (missing the years of 1997-2005 for the regions British Columbia and Alberta, and1997-2006 for all other regions); and
  
  
• Service Delivery Expenditures data (missing for the year 2007-2008).

Community Level

The case study communities were asked to provide data reports that they generate at the community level. Those reports were reviewed and data (where it existed) was entered into the community level administrative and financial data review templates. Data gaps were then identified for each community. Given the difference in the range and level of data provided by each community, the data gaps are different for each community.[Note 32]

Salient data gaps at the community level include:

• Total IA Expenditures (missing for the years of 1997-2008 for all communities); and
  
  
• Number of Person Months of Employment Created (missing for the years of 1997-2003 for Opaskwayak, for the years 1997-2008 for Eskasoni and Gitanyow, and for the years of 1997-2007 for all other communities).

3.1.3 Ability of Data to Measure Outcomes

Meaningful Indicators not Used

The program is unable to adequately and consistently measure meaningful outcomes. The chief reason for this is that indicators that would measure meaningful outcomes have not been developed and systematically used to direct data collection. Data that is collected is almost exclusively on outputs, rather than outcomes that measure program effectiveness. One of the exceptions to this is dependency rates; however, they are a rough measure. The information behind why dependency rates are rising or falling is not captured. If, for example, welfare dependency rates have fallen in a region, the program has no way of knowing the elements of this – whether former recipients have become permanently employed, or no longer meet eligibility requirements, have moved, or any other determining factors. In this regard, one regional staff member remarked that "We've lost track of what we need to measure and why. We don't know enough about the clients."

Data Gaps

Data gaps pose a challenge to assessing program performance and success at all levels (national, regional, and community) and hinder the ability to determine whether program outcomes are being achieved. INAC has data about income assistance expenditures and dependency rates; however, missing data on expenditures, welfare dependency, and employment created makes it difficult to determine whether the investment of income assistance dollars facilitates the movement off of social assistance programs.

3.1.4 Consistency and Comparability of Data

• Depending on the type of funding agreement, First Nations report either monthly or annually;
  
  
• Eligibility criteria and rates differ between INAC and provinces / territories;
  
  
• There is no common data platform. In some regions data is system-generated; for example in Quebec and Saskatchewan. Other regions may use other forms of reporting such as a reporting template;
  
  
• There is no consistent mechanism used for transmitting reports to the regions (i.e. some communities fax handwritten reports into the regional offices); and
  
  
• Employment creation is computed on a cumulative basis (i.e. spells of employment are added together across many individuals). Consequently, it is not possible to assess the number of days of full-time versus part-time employment created; nor is it possible to associate employment status with a particular recipient over time.

Frequency of reporting is linked to the type of funding agreement in place. 'The Comprehensive Funding Arrangement (CFA) is a program-budgeted funding agreement that INAC enters into with recipients for a one year duration.'[Note 33] This type of agreement requires regions to report monthly; whereas, the Canada / First Nations Funding Agreement (CFNFA), which is a block-budgeted funding agreement that INAC enters into with First Nations and Tribal Councils for a five year duration, requires much less frequent reporting.[Note 34] Accordingly, the data varies within regions according to funding agreement type. Almost the entire Atlantic region is funded under a CFNFA agreement which may account for the scarcity of data from that region.

3.1.5 Reporting Capacity

This section outlines evaluation findings with respect to the capacity of staff and systems to adequately report on the program.

National

There are a number of capacity issues at Headquarters that challenge the ability to measure outcomes:

• High staff turnover leads to a loss of corporate/institutional knowledge, and leads to quality control issues;
  
  
• Insufficient and inappropriately trained staff can compromise things such as follow up on public-private partnerships;
  
  
• There is a need for an appropriate data collection and inputting system. At present data needs to be entered manually (transposed) which increases the probability of errors; and
  
  
• System platforms change frequently.

Regional

Regions are lacking in staff and expertise in data management and analysis. In addition, regions are currently understaffed which compromises the level of detail applied to the work (i.e. carrying out comprehensive reviews reports), and hinders the region's ability to participate in national program meetings and other such activities.

Regional staff indicated frustration by the low level of feedback from Headquarters on reports they submit; this impedes their ability to plan programs based on evidence. Staff did not indicate whether this applied to specific reports, or all reports they submit.

Community

During the case study visits, program administrators were asked to provide program reports to the evaluation team. Of the 10 communities visited, 70% did not provide the evaluation team with program reports. The reasons for not providing these reports were as follows:

• Though communities had completed data reports (which they had sent to their regional INAC office) these reports were filed away and were not easily accessible at the time of the community case study visits (mentioned by 28.6% of communities who did not provide the team with program reports);
  
  
• INAC regional offices had already received the requested reports from the communities; therefore, it was recommended that the evaluation team contact the regional offices directly for the documents (mentioned by 57.1% of communities who did not provide the team with program reports).

3.2 Program Effectiveness

The evaluation aimed to determine how effective the program is in achieving its stated objectives to "alleviate hardship; maintain functional independence; and achieve levels of well-being reasonably comparable to the standards of the province or territory of residence."[Note 35] In doing so, the evaluation framework focused on questions of access; on the alignment of programs with program objectives; and on comparability of the programs with best practice.

3.2.1 Access to Services

INAC's Income Assistance program at the regional level is required, according to the program principles outlined in the National Manual, to provide income assistance to those "ordinarily resident on a reserve," "at standards reasonably comparable to the reference province or territory of residence."[Note 36] Exceptions to this are Ontario[Note 37] and Alberta, where the province, not INAC, dictates the terms and conditions of the program and INAC simply reimburses the province for these expenditures.

The program is not explicitly directed to enforce the same eligibility requirements as those employed by the province of reference, and the application of these is therefore inconsistent from one First Nation to another and one region to another. The IA Program Evaluation completed in 2007 noted that many First Nations are not enforcing the same eligibility requirements as their provincial counterpart,[Note 38] and some key informants in this evaluation concur on that point. It was emphasized that the region does not have the capacity to ensure that eligibility requirements are being followed by First Nations.

Overall, IA is accessible to eligible recipients on reserve, and there is a high level of awareness of the program and benefits available. Indeed, in most of the communities surveyed, residents expressed a concern that young people are too readily choosing welfare as an option as soon as they are able to qualify at age eighteen, rather than entering the work force or pursuing higher education.

The current evaluation did not find that community members were expressing concerns with being able to access Income Assistance. Evaluators found that the perception of inequitable access was expressed by some IA recipients in focus groups; not in terms of basic needs support, but discretionary funds for "special needs" such as replacement household appliances or one-time grants for children's sports equipment; or, gaining access to scarce employment opportunities through the First Nations. In these cases, recipients felt that IA staff members or First Nations Councils were showing favouritism to some residents over others, or that their needs were simply not being considered a priority.

3.2.2 Linkages / Integration with Complementary Programs

There are two main linkages at the federal level between INAC and Health Canada (providing support for addictions) and Human Resources and Social Development Canada (which provide training for IA recipients).

Many communities have developed liaisons with employment officers to benefit IA recipients. For example, Chehalis, B.C. has Employment Services that operate various programs including:

• Aboriginal Alternative Learning Program (funded by AHRDA), which receives Social Assistance referrals, identifies learning deficiencies and remediation tools;
  
  
• Structure of Intellect Program, which stimulates clients to use both halves of their brain;
  
  
• A computer-based program that develops numeracy and computer skills;
  
  
• Adult literacy skills;
  
  
• Reading program that works on improving memory skills; and,
  
  
• On-call jobs in the community (e.g., fishing, clearing brush).

There are linkages with AHRDA holders for employment and training programming (schools, tuition, employability skills training, and links to employment). Manitoba is operating in partnership with "Fire Spirit", a Winnipeg-based business that is helping First Nations assess job readiness and skill levels of residents, and then directing employment seekers to appropriate resources.

First Nation communities in BC have access to three programs to assist Social Assistance recipients make a transition to work: the Work Opportunity Program (WOP); the Training and Employment Support Initiative (TESI); and Aboriginal SA Recipient Employment Training (ASARET). However, the ASARET program was stopped because it contravened the Canadian Training Act, while the WOP was the main transition to work instrument for only one year. The main program accessed by SA recipients on-reserve is TESI: however, less than 5% of the unemployed, but employable, population on reserve accesses the program.

3.2.3 Service Gaps or Overlap

Basic needs assistance is being provided in all the communities surveyed, and the majority of respondents felt that there were not community members who needed IA but were not receiving it. In some communities there are programs that provide for basic needs of elders, but this does not constitute an overlap in service, as IA is not provided to those over 65.

Apparent service gaps exist in supports to finding and preparing for employment; 70.5% of community end-user IA respondents stated that they had never been assisted by a community program to find a job. Only a quarter of the community members who filled out the survey have been assisted; however, most jobs were short term.

INAC has had the program authority to implement active measures since 2003, but no increase in funding accompanied this change.[Note 39]

3.2.4 Comparability with Provincial Programs

There is a problem of definition: how is "reasonably comparable" defined? INAC has a long way to go to reach comparability; they don't have the same aggregated budgets as the provinces. The result is that INAC regions are re-allocating funds from capital budget dollars in order to match provincial assistance rate increases. Likewise, when a province reduces the rates, INAC does likewise, whether or not that is the best response to needs on reserves.

Provincial social assistance regimes normally have a much more robust system of supports for service delivery. "Second level" service delivery funding and administrative bodies such as the First Nations Social Development Society (FNSDS) in B.C. and the First Nations of Quebec Social Development Office (FNQSDO) are means by which some regions have replicated in part, the kinds of administrative supports widely available off-reserve. One of the case study sites, the Sto:lo Nation in B.C. region, for example, delivers Social Assistance (SA) using an aggregate model for nine member First Nations. The Sto:lo Nation, which has the authority to act on behalf of the First Nations, serves as liaison between INAC and the individual communities.

3.3 Program Impacts

The following section provides a discussion of funding and caseload composition trends over the ten-year period 1997-98 – 2007-08, based on departmental data provided. IA funding expenditures have increased each year 2005 to 2008. Program funding increased from 2005-2006 ($682,610,238) to 2006-2007 ($703,998,181) by 3.1% and from 2006-2007 to 2007-2008 ($742,000,696) by 5.4% (see Figure 1).

Figure 1: IA Funding Expenditures (Current $)

Figure 2: Total IA Basic Needs Funding Expenditures

The average monthly number of IA recipients on-reserve has fluctuated over the ten year period; however, the number of recipients increased significantly in fiscal year 2004-2005 and 2005-06. The number has dropped in 2007-2008 and may be attributed to the rate decrease of recipients in the Atlantic and British Columbia regions (-53.7% and -13.0% respectively) in 2007-2008 compared to 2006-2007.

Figure 3: Average Monthly Number of IA Recipients On-Reserve

The average number of monthly IA beneficiaries on-reserve has also fluctuated in the ten-year period. The number of beneficiaries steadily decreased from 1997 to a low in 2001-2002. Since that time, the number of beneficiaries increased each year until fiscal 2007-2008, when it decreased by approximately 1.8% from the previous year (see Figure 4).

Figure 4: Average Monthly Number of IA Beneficiaries On-Reserve

The ratio of reported IA beneficiaries and reported program recipients has decreased over the period. In 1997-1998, there was an average of 2.21 beneficiaries per recipient (see Figure 5). In 2007-2008, the ratio had decreased to an average of 1.99 beneficiaries to one recipient. In practical terms, this means that the total caseload in 2007-2008 has a higher ratio of single, rather than family recipients.

Figure 5: Ratio of Average Monthly IA Beneficiaries to IA Recipients

Figure 6, below, shows that employment created decreased between 2000-2001 and 2006-2007, but rebounded in 2007-2008. The rate of change from 2000-2001 to 2007-2008 was -35.1% while the rate of change from 2006-2007 to 2007-2008 was 16.6%. However, it is important to note that national level data is not able to appropriately comment on / address regional differences, such as the emphasis on active measures in British Columbia.

Figure 6: Person-Months of Employment Created

Income assistance dependency rates on-reserve decreased steadily until 2003-2004, but then began to rise. Over the course of the 10-year period, IA dependency rates have decreased by 5.1% overall (see Figure 7). 'In 2005-06, the average rate of dependency on-reserve was 36% compared to a national dependency rate of 5.5%.'[Note 40] There are significant regional variations in the dependency rate, however.

Figure 7: IA Dependency Rates (On-Reserve)

3.3.1 Extent to which Needs are Being Met

INAC's Income Assistance program is essentially a passive service with the goal of providing for the basic needs of recipients. Key informants within INAC emphasized that their role is that of a funder, not service provider.

Although the IA program matches provincial rates in all regions, recipients and service providers agree that needs are not being met. At the regional program level, IA funding increases have been capped at 2% since 1997, and this level of funding has not been sufficient to support program costs. Some provinces have increased rates and INAC regional offices have been in the position of having to use capital budgets to cover the cost of matching provincial increases.

The higher costs of goods in many remote communities have acted as a barrier to meeting the needs of IA recipients. Residents of remote reserve communities have little access to transportation, and therefore high costs for taxis (where available) to purchase groceries and other goods (such as clothing).This creates an increase in cost for basic needs that is often not met by an increase in IA rates. Clothing costs, particularly for children, have been identified as a need that needs to be addressed. Housing is often is short supply, and existing housing stock in need of repair. Many recipients are living in overcrowded houses. A number of interviewees reported mould in their homes.

3.3.2 Impacts on end-users

Some recipients in the end-user surveys stated that they are going into debt in order to meet basic needs, and that "IA is not enough to keep your house warm for the winter." Almost 83% of respondents in the end-user survey believed that they are not receiving enough income for basic needs, and one respondent stated that "people have to steal just to get by for the month."

One focus group participant who had just quit his job that day because he could no longer afford the transportation costs to get to his off-reserve job, commented that the system is so bad that "it makes good people do bad things." Similar impacts of high transportation costs were also noted in other case study communities.

People reported missing medical appointments because they don't have transportation to off-reserve sites and can't afford taxi fares; there is no public transportation available. One exception is Opaskwayak Cree Nation, who have a multipurpose van that provides transportation to Elders and anyone who needs it to go off-reserve for shopping or appointments for a two dollar fee.

Special Needs benefits are inadequate. Items such as children's sports equipment is out of reach for recipients, but important for child's development.

During one focus group discussion it was noted that some community members who qualify for IA are not receiving the full amount they should as a result of lack of education/awareness about the program, timidity, and fear of authority figures.

Impacts of staffing or staff capacity issues: In more than one case study community, recipients expressed their frustration with the way utility bills are handled. The amount of the bill is deducted from the cheque in advance, but on a number of occasions the bill was not paid by the IA program and their hydro was cut off as a result. Recipients expressed deep frustration and feelings of helplessness when this happened.

3.3.3 Impacts on Community Program Personnel

There is a strong desire from community program personnel to have more training, in order to effectively meet the potential negative impacts of the IA program; such as stress-related issues and invasion of personal boundaries, as well as safety issues that may arise from disgruntled recipients who feel they are not getting enough money.

3.3.4 Community-Level Impacts

The community level impacts are very difficult to assess. A main concern seen in communities is in regard to the number of 18 year olds going on IA as a matter of course as soon as they turn 18. Respondents felt that it creates a dependency on the program by residents as the program is seen as a way of life, rather than a means to sustain oneself while searching for employment.

3.4 Program Cost Drivers

Outlined below are the economic, health, regulatory, and demographic trends likely to affect the IA program on-reserve, along with examples of best practices in place likely to facilitate in addressing cost drivers.

• In 2005-2006, transfers totalled $682 million, close to 90% of which was spent on basic services for close to 150,000 individuals in 630 First Nation communities (INAC, 2007b). The costs of program delivery are separate from the benefits provided to recipients. Noting a shortage of service delivery funds, an AFN report stated that 89% of survey respondents noted a lack of funds to provide needed services in First Nations (Assembly of First Nations, 2005). More recently, the 2007 evaluation notes that "Funding formulae allocations for service delivery are made largely based on on-reserve population size as opposed to income assistance dependency rates" (INAC, 2007b, p. 6).
  
  
• Rate increases in provision and delivery of service have impacted the financial sustainability of the IA program. The requirement that INAC social assistance rates must be comparable to provincial rates is exerting pressure on IA budgets, which are increased by only 2% per year. Some of the examples stated in INAC's IA program evaluation (2007), which impact basic needs expenditures, include: Manitoba's increase to its northern food allowance; Saskatchewan's 19% increase in basic rates; and Alberta's higher rates for income assistance recipients who are classified as "Learners" (defined as "people who need upgrading or training so they can get a job") (Alberta Human Resource and Employment, n.d.a). Annual rent expenditures and fuel, utilities and shelter expenditures have risen since 2004-2005 and continue to rise (INAC, 2007b).
  
  
• The current global economic situation has significant implications for First Nations communities particularly in terms of the predictions that the credit market will tighten and activity in industrial sectors such as forestry (which is usually located close to First Nations communities) over the next 1 to 5 years will decrease. The way in which the federal government is looking to address the Canadian economy will impact the situation in First Nations communities. The current annual increase of 2 -5% on-reserve is not sufficient to address the current economic situation and will (if unchanged) continue to squeeze First Nations residents who already have limited resources. Crisis intervention and economic stimulation activities are needed but it is not known whether the current economic crisis will allow for the creation of sustainable forms of employment;
  
  
• More than half the Aboriginal population is under the age of 25 years. If these individuals do not pursue employment there will be a significant impact on social assistance programs. Flexibility is needed to facilitate the partnerships between AHRDAs, Head Start, and other programs similar in nature to ensure the needs of this group are proactively met;
  
  
• As more Aboriginal young people get better education, they may be seeking better jobs and may migrate off-reserve to find these jobs;

Best Practice: in Nova Scotia Income Assistance clients are able to remain on social assistance while in training programs to improve their access to employment. This policy has been successful, and may facilitate labour force attachment and reduced dependency rates;

Best Practice: INAC wants to facilitate the transition of income assistance clients from the program to employment. British Columbia has a best practice program in place which encourages clients to move off the program by providing them with a host of services including: motivation, training, and job searches through a network of partnerships with different service providers. This kind of network which promotes sharing of information is very helpful and successful because through the partnerships, the client is offered more assistance;

Best Practice: in Ontario, a study was completed in the Hamilton / Halton area which investigated the components that would facilitate a decrease in social dependency for single mothers. It was found that providing a portfolio of services including: child care, recreation, ECE, visits by health care professionals, and job training at a minimal cost was the best way to decrease social dependency for this group. The overall cost of providing the portfolio of services was about the same as the cost of regular social assistance however, when looking more globally, providing the portfolio of services resulted in a savings of approximately $3000 for the social assistance program;

Best Practice: in terms of service delivery, the 2007 IA program evaluation notes that INAC Ontario is the most advanced in assessing service delivery costs and is attempting to administer the full Ontario Works program. INAC and Ontario's Ministry of Children and Social Services also examined the cost per case and aggregation of services among First Nations to achieve economies of scale and to allow for specialization of services.

3.5 Conclusions and Recommendations

3.5.1 Conclusion

The Income Assistance program continues to be relevant, in light of the continued disparities in income, education, employment, standard of housing, and general well being between First Nations and the rest of Canada. Until effective and long-term alternatives and solutions are found that enhance individual and community well being, Income Assistance plays a role in alleviating hardship.

INAC's costs for provision of basic needs benefits are rising; almost 90% of the IA budget goes to basic needs; yet end-users and key informants in this evaluation reported that basic needs are not being met. End-users report going more deeply into debt just to cover the basic necessities of food, shelter, clothing, utilities and personal needs.

According to the terms and conditions of the program, INAC regions follow provincial welfare rates, which are determined in an urban, rather than a remote rural context. The rates do not take into account the unique needs of First Nations, and the reality that reserve residents don't have access to public transportation. The burden of high transportation costs were one of the most frequently reported impacts by IA recipients on their ability to meet basic needs with existing benefits.

The performance measurement framework has not supported the collection and analysis of meaningful outcome data for the IA program, and the attribution of such outcomes is a long-term and complex process. "Alleviation of hardship" as a desired outcome is too rough a measure to be useful. In terms of employment support measures, little is known about the complex employability needs or long-term outcomes of employment support measures for recipients, to be able to say whether interventions have been successful.

At current staffing levels, INAC does not have the capacity to make effective use of the data it collects, or to provide the support that regional offices and communities would like to see from headquarters.

Best practice in social assistance recognizes that active measures and integrated approaches, including client case management; tailored approaches to employability and job readiness barriers; addressing education and training needs; and providing supports to parents such as training funds and child care, are effective at finding long-term solutions to low employment and high welfare dependency levels. While such approaches are being implemented by some provinces, only a very small percentage (less than 2%) of INAC's IA expenditures go to supporting such measures.

The following section provides recommendations for addressing the challenges highlighted by the evaluation of the IA program.

3.5.2 Recommendations

The following seven recommendations are suggested for improving the effectiveness of the IA program.

1. Develop, in partnership with relevant bodies such as HRSDC, AFN, and provincial ministries, an integrated strategy to address on-reserve labour and employment needs. The strategy would recognize the complex and unique needs of the on-reserve unemployed, such as restricted access to labour markets; multiple employability barriers; transportation needs; and the need for child care and other necessary supports while in training or educational upgrading programs.
   
   
2. In the near term, until a strategy to address the causes of welfare dependency is in place, and achieving the desired outcomes; and to provide better support for basic needs: review the 2% funding increase policy to assess whether it is meeting First Nations IA costs;
   
   
3. In the near term, address INAC staffing shortages and training needs at the national and regional levels;
   
   
4. In the near term, fund a representative sample of community needs assessments that will provide meaningful cost measures for items such as shelter, utilities and transportation;
   
   
5. In the longer term, create a working group of INAC, First Nations and Provincial representatives to develop a strategy for addressing IA jurisdictional and funding issues, including a discussion of the costs of needs in rural/remote communities;
   
   
6. Take the lead in initiating an integrated education and training strategy with HRSDC, Aboriginal organizations, and relevant provincial ministries, to address the education needs of First Nations youth in particular, as a way of reducing the number of youths who choose welfare over further education and/or employment;
   
   
7. Strengthen links with other relevant departments such as HRSDC to enhance information sharing so that long term employment outcomes can be measured, and develop more refined outcome indicators for future evaluation activities.

4.1 Performance Measurement

Evaluators requested national, regional, and community level NCBR administrative and financial program data for a ten-year period between 1997/1998 and 2007/2008. The data sought included the number of clients accessing the program, clients' characteristics, number of staff delivering the program, program expenditures and services by component, funding arrangement, employment creation, and welfare dependency rate.

4.1.1 Performance Measurement Framework

Table 4 below outlines the data reporting system for each of the three programs based on the Recipient Reporting Guide (RRG)[Note 41] and feedback from preliminary consultations. The table identifies and links each of the programs to its corresponding reports, funding agreements, data collection and report preparation body, report audience, reporting period, and the data being reported (by output and outcome).

4.1.2 NCBR Data Gaps

The data seen in Table 5 below represents, according to both the RRG and what was heard in the preliminary consultations, the NCBR output and outcome data INAC ideally intends to collect annually.

In order to comment on the adequacy of performance data it is necessary to determine what data is actually collected by the programs. In order to do this, administrative and financial data was gathered at the national, regional, and community levels for a ten-year period covering the years 1997/1998 through to 2007/2008. The data sought included the number of clients accessing programs, number of program beneficiaries, and program reinvestment expenditures.

National Level

A review of the national level NCBR data gaps showed that significant outcome indicators, which could speak to program success and overall performance are not yet collected. Significant amounts of data were unavailable for the evaluation, as system incompatibilities would have required lengthy data manipulation that could not be performed by INAC HQ within the tight time frame of the evaluation. As a result, salient data gaps at the national level included:

• NCBR Funding Expenditures data (for the years 1997-1998 and 2007-2008)
  
  
• Total Reinvestment Funds Reported data (years of 1997-2001and 2007-2008);
  
  
• Number of Children Beneficiaries data (for the years 1997-1998 and 2007-2008);
  
  
• Number of Family Beneficiaries data (for the years 1997-1998 and 2007-2008); and
  
  
• NCBR Service Delivery Expenditures data (for the years of 1997-2008).

Regional Level

Similar to the national level, a review of regional level data gaps showed that outcome indicators which could provide insight into program performance and success were missing. Salient data gaps at the regional level include:

• NCBR Funding Expenditures data (not provided for the years 1997-1998 for Manitoba, and for the years 1997-1998 and 2006-2008 for all regions);
  
  
• Total Reinvestment Funds Reported data (not provided for the years of 1997-2001 for Alberta and Manitoba and for the years of 1997-2001 and 2007-2008 for all other regions); and
  
  
• NCBR Service Delivery Expenditures data (not provided for the years of 1997-2008).

Community Level

The communities visited as part of the community case studies were asked to provide any data reports that they may have which can be shared with the evaluation team. Those reports were reviewed and data (where it existed) was entered into the community level administrative and financial data review templates. Data gaps were then identified for each community. Given the difference in the range and level of data provided by each community, the data gaps are different for each community.[Note 43]

Salient data gaps at the community level include:

• NCBR Funding Expenditures data (missing for the years of 1997-2004 for Opaskwayak and Skownan, and missing for the years of 1997-2008 for all other communities); and
  
  
• NCBR Total Annual Reinvestment data (missing from the years of 1997-2001, 2002-2004, and 2007-2008 for Thunderchild; missing for 1997-1999 for Walpole Island; missing for the years of 1997-2000 and 2002-2003 for Opaskwayak; and missing for the years of 1997-2005 and 2007-2008 in Eskasoni).

4.1.3 Ability of Data to Measure Outcomes

Data gaps pose a challenge to assessing program performance and success at all levels (national, regional, and community). Missing data preclude the ability to compute cost per month of service for children and families, identify program expenditure trends, and trends related to delivery costs at the national level. First Nation communities that use NCBR funds are required to report annually on each reinvestment project, using a standard 1-page report form. The lack of data broken down per month makes it difficult to assess the intensity of services provided per beneficiary and the administrative cost at the national level.

In terms of measuring outcomes, the current objectives of the NCBR program, as identified in the National Manual, are to:

• Prevent and reduce the depth of child poverty;
  
  
• Provide incentives to work by ensuring that low-income families with children will always be better off as a result of working; and
  
  
• Reduce overlap and duplication through simplifying the administration of benefits for children.

The desired outcomes, or "anticipated results" as named in the National Manual, are improvements in:

• Children's health and development;
• School readiness and ability to learn;
• Parents' participation in the labour market;
• Financial independence; and
• Greater participation in their communities and Canadian society.

The current reporting requirement for NCBR makes the collection of measurable outcomes particularly challenging. Being an annual report, it is not possible to assess impacts on the short-term (e.g., monthly). Also, the number of clients is reported as if all clients have participated in the program for the same duration. The data system therefore treats equally a client served for one month as one served for a year. Also, the qualitative nature of many of the project benefits make it is difficult to synthesize the data in order to comment on the success and effectiveness of the program. The net result of these challenges is that it is difficult to obtain concrete or meaningful results regarding the impacts of NCBR-funded projects other than the number of beneficiaries.

It is also important to note that some of the outcomes of the NCBR funded projects may be so long-term that they are not possible to measure. For instance, education is a key component in obtaining employment: NCBR programs that encourage children to attend school regularly (via in-school meals and snack programs) and improve academic performance (via language therapy and other supports) at the primary school level may result in lower drop-out rates and increased employment of the same students years later.

Given the lack of measurable outcomes, anecdotal evidence must do (i.e., reports from professionals who work with children in a community over the course of many years) and indirect indicators of program impacts. In the case of Walpole Island First Nation, the Vice Principal of the Walpole Island Elementary School stated that the school relied heavily on NCBR funding for many of its programs[Note 44] and had noted several important health, social and educational benefits, the most important – and quantifiable -- of which was vast improvements in student performance in Ontario's standard educational tests (See Section 4.3.4 for more details on these impacts). In the absence of more frequent or quantitative data, observations by professionals in the community are the only source of information on the impacts of NCBR-funded programs in First Nation communities.

4.1.4 Consistency and Comparability of Data

There are a number of problems and inconsistencies with NCBR reporting requirements and methods that make data comparisons difficult. These include the following:

• First Nations data is gathered using templates which are produced by each region and are not necessarily consistent between regions;
  
  
• Financial data appears to be system generated;
  
  
• Data on numbers of families / children accessing services can be subject to "multiple counting" (e.g. one child may access 4 different projects, counts as 4 children);
  
  
• Variation in policy implementation across regions (e.g. in Manitoba and New Brunswick, IA programs do not reduce payments by the amount of the NCBR. Therefore, as funds are not recovered for reinvestment in NCBR projects, First Nations in these provinces do not deliver NCBR programming); and,
  
  
• Some regions require Monitoring Reports and others do not; however all regions are required to complete some level of reporting.

Another important challenge to measuring and reporting on outcomes is the great diversity and community-specificity of many NCBR funded projects: while this practice is good for the communities in the sense that they can tailor and create programs that address their specific needs, this makes comparability of data across communities – or for the entire program – particularly difficult.

4.1.5 Reporting Capacity

This section outlines evaluation findings with respect to the capacity of staff and systems to adequately report on the program.

National

The primary capacity issue regarding NCBR reporting, as identified during the preliminary consultations, is the high staff turnover rate, lack of staff orientation and high number of vacancies at INAC HQ. The lack of human resources capacity results in difficulties in accessing information.

The absence of outcome indicators for the NCBR program, combined with lack of procedures regarding why data is collected and how it should be collected also impacts the reporting capacity at the national as well as regional levels.

Regional

According to Key Informants from the regional offices, staffing is an important capacity issue, with offices being under-staffed and existing staff members being over-burdened. HQ participants in the preliminary consultations, however, praised the support provided by INAC regional staff to HQ.

Community

During the case study visits, program administrators were asked to provide program reports to the evaluation team. Of the 10 communities visited, 60% did not provide the evaluation team with program reports. The reasons for not providing these reports were as follows:

• INAC regional offices had already received the requested reports from the communities; therefore, it was recommended that the evaluation team contact the regional offices directly for the documents (mentioned by 16.7% of communities who did not provide the team with program reports);
  
  
• The community in question did not have an active NCBR program and therefore could not provide the evaluation team with program reports (applies to 50.0% of communities who did not provide the team with program reports).

4.2 Program Effectiveness

4.2.1 Access to Services

A key component of facilitating attachment to the labour force is the provision of employment training. Funding allocations for home-to-work transition projects increased each year between 2000 and 2005 and almost doubled between the 2003-2004 and 2004-2005 fiscal years ($8,886,000 to $17,508,700 respectively). However, since 2004-2005 funding allocations have been steadily decreasing. Allocations in 2005-2006 decreased by a rate of approximately 2.3% from the previous year and decreased by a rate of approximately 34.3% in 2006-2007 when compared to 2004-2005 levels (see Figure 8).

Figure 8: Funding Allocation for Home-to-Work Transition Projects[Note 45]

Of the eight regions investigated in this evaluation, 75% did not have data available, at the time of this evaluation, on home-to-work transition funding for fiscal year 2007-2008. The two regions, Alberta and British Columbia, which had 2007-2008 data available provided less funding for transition projects in 2007-2008 than they did in 2006-2007. The rate of change in Alberta from 2006-2007 to 2007-2008 was -21.1%, while British Columbia had a rate decrease of approximately 36.1%.

When addressing funding and the NCBR program, there are three important points to consider:

• The provincial method of IA/NCBS offsets, due to the INAC principle of comparability with the provinces;
  
  
• Funding is linked to the number of project proposals received (i.e. if the communities are not submitting proposals then funding expenditures will decrease); and
  
  
• Funding is linked to IA caseload rates, specifically, IA recipients with children under the age of 18.

Also important for facilitating labour force attachment is the availability of child care / daycare support services. These services provide parents with the opportunity to leave their children in appropriate care while either participating in employment training or pursuing employment. Funding allocations for child care / daycare projects over the course of the years 2000 to 2007 appears to be following a staggered pattern of increasing for three fiscal years and then decreasing for one year (see Figure 9).

Funding allocations for child care / daycare projects increased between the period of 2000 to 2003, decreased in the 2003-2004 fiscal year, increased significantly over the course of 2004-2006, and then decreased again in the 2006-2007 fiscal year. Between the period of 2003-2004 and 2005-2006 funding allocations increased by a rate of 276.9% (from $865,800 in 2003-2004 to $3,246,300 in 2005-2006). However, following that period, funding dollars have decreased. In 2006-2007 child care / daycare project dollars totalled $3,136,200 (a rate decrease of approximately 3.4%).

Figure 9: Funding Allocation for Child Care / Daycare Projects

In terms of supporting attachment to the labour force, INAC's NCBR program began increasing its total funding of supportive projects beginning in 2004-2005. Support for projects likely to facilitate employment continued, in the latter portion of the first decade of the 2000s, to exceed levels seen between the years of 2000-2001 through to 2003-2004.

4.2.2 Range and Quality of Services

The NCB Initiative has 2 main elements: (1) monthly payments (income supplement) to low-income families with children; and (2) benefits and services designed and delivered by the provinces, territories and First Nations to meet the needs of low-income families with children – this is the reinvestment of money (NCBR) saved from the supplement into projects to support low-income families with children. The intent of having 2 NCB components (supplements and reinvestments) was to help ensure that families would always be better off as a result of working - they would not receive added financial assistance by remaining on income assistance, nor would they be penalized by moving off income assistance. Thus, the adjustments were designed to counteract disincentive effects to entering the workforce.

Low income families with children benefit from the First Nations NCBR by having greater access to services that promote the well-being of families and children. Evaluations and studies of the NCBR show that the reach of the NCBR goes beyond low-income families with children on-reserve to include income assistance recipients, individuals in need, school children and youth.

In 2001-2002, First Nations NCBR projects across Canada totalled 1,118. The numbers of Reinvestment projects by region at that time are provided in Table 6.

Source: (INAC, 2004b)

First Nations reinvestments and investments cover five main areas:

1. Child/day care;
2. Child nutrition;
3. Support to Parents;
4. Home-to-Work Transition; and
5. Cultural Enrichment[Note 46] (details of each program area is provided in Section 1.3.2).

First Nations have a high degree of autonomy in the allocation of NCBR funds. Increasingly, they are funding projects that help individuals make the transition from welfare to the workforce or to gain work experience and skills for career advancement. The NCBR also enhances existing programs through the use of NCBR funds to "top up" these programs. In any particular community, a suite of NCBR-funded projects may be in operation, with several projects in each main area. In some of the larger case study communities, up to 30 NCBR programs are currently in operation.

No direct information could be obtained regarding the quality of the services provided through NCBR-funded programs on-reserve. However, Key Informants, regional contacts, service providers in the case study communities and community members in the case study communities were unanimous in stating the value and importance of the NCBR-funded projects, especially in the program areas of child nutrition, cultural enrichment, and supports to parents.

4.2.3 Linkages / Integration with Complementary Services

First Nation communities use NCBR funds to top up existing on-reserve programs and services, in fields such as health, employment and training, education, youth programming, basic needs (e.g., food, clothing and shelter) and cultural events. Examples of HRSDC programs topped up by NCBR funds include First Nations and Inuit Child Care Initiative and Aboriginal Human Resource Development Agreements. Health Canada programs topped up by NCBR funds include Aboriginal Head Start On Reserve. INAC programs topped up by NCBR funds include Income Assistance Pre-employment Supports, Work Opportunity Program, and Family Violence Funding.

The NCBR funds assist other programs to do things needed in the community like housing construction while providing productive jobs. Home to work projects are another area where NCBR funds supplement existing programs on reserve. Such projects focus primarily on job creation and target single mothers, youth, and summer employment for students to help them develop employability skills.

First Nations also link NCBR funds with other resources, such as day-care funding, general First Nations revenue, or employment funding events. Funding partnerships have been established with agencies such as Health Canada's Head Start and Brighter Futures, as well as the Aboriginal Healing Foundation, Canada Manpower, Pathways, and Training Employment Skills Initiative events. In some cases, NCBR programs are used by First Nations to supplement Health Canada and HRSDC programs to reduce poverty and promote integration into the labour force.

4.2.4 Service Gaps or Overlap

Several programs operating in First Nations have been identified that provide support to children similar to those provided by NCBR, as provided in Table 7.

NCBR project funds are often used to supplement existing programs on-reserve. This situation makes the issue of determining the extent of "overlap" particularly difficult and complicated.

4.2.5 Comparability with Provincial Program

Data on provincial child benefit program requirements varies, with different criteria per province, and often sliding scales of rate payments.

4.3 Program Impacts

4.3.1 Extent to which Needs are Being Met

Since 1999-2000, funding levels have remained within the 50 million dollar range; however, figures for 2006/07 have decreased by about 35% (see Figure 10). The reason for this decrease is not known, although it may be due to reporting issues.

Figure 10: NCBR Funding Expenditures (Current $)

The number of families benefiting from reinvestment projects was at its lowest (16,503) in 1998-1999. That number increased in 1999-2000 to approximately 26,164 families and then doubled in 2000-2001. Between the period of 2001 to 2004 the total number of families benefiting remained within the range of 48,000 to 59,500.

In 2004-2005 and 2005-2006 there were significant increases in the amount of families receiving some form of benefit from reinvestment projects; a 168.6% rate increase in 2004-2005 and a 280.9% increase the following year (see Figure 11). The number appears to have decreased dramatically in 2006-2007 (81.6%); however, this may be a reporting issue.

Figure 11: Number of Families Benefiting from Reinvestment Projects

In 1999-2000 NCBR funding expenditures increased by $20,125,300 (rate increase of 66.8% from the previous year) and the total number of families benefiting increased by 9,664 (rate increase of 58.6% from the previous year). The ratio of families to dollars expended in 1998-1999 was 1:$1,825.38 compared to a ratio of 1:$1,920.34 in 1999-2000. However, by 2005-2006, the ratio of funding expenditure to families benefiting had decreased to 1:$256.21.

The number of children benefiting from reinvestment projects (Figure 12) follows much the same trend as number of families benefiting (Figure 11). Fiscal year 2005-2006 marked the year with the greatest number of children benefiting (501,170) whereas fiscal year 1998-1999 had the least number of child beneficiaries (37,468).

Figure 12: Number of Children Benefiting from Reinvestment Projects

Funding for child nutrition projects has been fluctuating across the period of 2000 to 2007. Fiscal year 2000-2001 had the lowest level of investment ($5,002,700) for the period of 2000 to 2007. Investments exceeded $10,000,000 in 2004-2005 and continued to do so through to 2006-2007. The highest level of investment occurred in 2005-2006 when child nutrition project funding totalled $12,761,800 (see Figure 13).

Child nutrition projects funded by the NCBR are one of the key initiatives identified as important by program end users in the case study communities. In their statements of insufficient IA benefits, IA recipients often mentioned that there was not enough money to buy food. In those communities where food banks do exist, use is often restricted to two meals or so per month. Community members tend to be well aware of the child nutrition projects in their communities and parents frequently stated that the program has the two-fold benefit of feeding children in lower income families and ensuring that the children attend school.

Figure 13: Funding for Child Nutrition Projects

During the period of 2000 to 2007, projects providing supports to parents had the greatest level of funding in 2005-2006 ($12,584,100). The funding level in 2005-2006 reflected a rate increase of approximately 151.7% and a dollar increase of $7,584,300 when compared to dollar amounts in 2004-2005. In 2006-2007 there was a funding decrease of $5,978,000 (-47.5% rate change) from levels in 2005-2006 (see Figure 14).

Figure 14: Funding Levels for Projects Providing Support to Parents[Note 47]

Investments into cultural enhancement projects were at their lowest, within the 2000 to 2007 period, in 2006-2007 ($6,589,000) and at their highest in 2001-2002 ($19,219,400). Over the course of 2001-2002 to 2006-2007, investments into cultural enrichment projects decreased at a rate of 65.7% (see Figure 15).

Figure 15: Cultural Enrichment Projects Funding

4.3.2 Impacts on End-Users

Given the fact that the NCBR program run by INAC is designed to be flexible and allow for innovative projects to be designed by community service providers, the only "service gaps" that exist in the program are due to either funding constraints or project approval criteria. In some communities (e.g., Chehalis), the number of NCBR projects in operation are relatively few on account of scarce funding. Under these circumstances, service providers are compelled to identify the highest priority projects (in terms of maximum benefit with available funds) rather than the full spectrum of potential projects to address the various needs that the NCBR is designed to address. In terms of approval criteria, for those projects that are not approved for NCBR funds (e.g., clothing, sports teams, summer camps, powwows and guitar lessons), there are usually no other services or funds available to fulfil the community's perceived needs in these areas.

4.3.3 Impacts on Community Program Personnel

The administrative burden of the NCBR on program personnel is relatively minor, as the NCBR provides funding for programs that are delivered by others in the community. Nevertheless, IA staff are involved in the process of identifying projects to fund, producing a proposal for annual funding, and producing the annual report for each program funded by the NCBR. Given the fact that the IA program has numerous negative impacts on IA staff (as described in Section 3.4.3), the NCBR's burden on the staff makes matters worse.

4.3.4 Community-Level Impacts

All sources of evidence used in this evaluation (documents, preliminary interviews, key informants, expert panels, regional interviews, community-level surveys and interviews) stated that despite the lack of clear outcome indicators and quantitative data, there is a firm belief that the NCBR projects are having positive impacts at the community level.

According to INAC's NCB Progress Report (2002), in general, NCBR programs have resulted in the following positive impacts to First Nation families on reserve:

• First Nation families with children are less reliant on income assistance since the introduction of the NCB Initiative;
  
  
• Some IA caseloads of families with children have declined;
  
  
• The length of time some single-parent families spend on IA has declined;
  
  
• The number of low-income families has been reduced and the financial situation of the families has improved;
  
  
• For most families employment has become more financially attractive than IA dependency; and
  
  
• Funding is provided for services that make it possible for low-income families to contribute more fully to community economic, social and cultural activities.
  
  

One of the best described impacts in this evaluation were those provided by the Vice Principal of the Walpole Island Elementary School, who had been working in the community for many years and had been using NCBR funds for several years. The kinds of positive community-level impacts observed by this Vice Principal are described below.

• Those children who have been involved in the various programs (e.g., speech therapy, therapy for trauma, special learning needs) for several years have shown considerable improvement. These observations are supported by the results of Ontario's province wide testing for Grades 3 and 6 (referred to as "EQAO"). In these tests, levels "1" and "2" are below the province's standard, while level "3" meets the standard and level "4" exceeds the standard. In Walpole Island First Nation, the Grade 3 and 6 mean on all tests have shown a consistent increase of combined "3" and "4" scores from 11% (4 out of 38 students) in 2003/04 to 54% (38 out of 70 students ) in 2007/08.[Note 48] The Speech and Language Pathologist (supported by the Walpole Island Elementary School at $40,000 in 2007-2008) and Art and Play Therapist (supported by the Walpole Island Elementary School at $8,000 in 2007-2008) were paid through NCBR funds: both professionals reported marked improvement in the school children's performance and behaviour.[Note 49]
  
  
• NCBR funding has also contributed to the cultural enrichment of the community through traditional healing and cultural camps. NCBR brings healers into the community, who provide spiritual counselling. By reconnecting with traditional practices, children's self-esteem has improved. A positive cultural identity helps to produce resiliency in the community and an ability to "bounce back" from adversity. Most IA end users surveyed in the community feel that cultural programs for children are very important. NCBR support of language and culture for children is considered an important need, as many parents have lost much of their language and culture.
  
  
• In the years since NCBR funding started, children in the community have become better nourished. As a whole, both IA program users and delivery staff believe that day care needs in the community are sufficiently met. A few community members surveyed believe that further funding and extended hours are needed for day care.

While some of the NCBR-funded projects impact primarily children (e.g. child nutrition) or their parents (e.g., support to parents or home-to-work transition projects), the cultural enrichment NCBR projects clearly benefit the entire community. Cultural enrichment projects often involve special celebrations prepared and organized by youth, children and Elders, with a wide variety of activities that engage families and community members of all age groups. In this sense, the NCBR program has a community-wide impact that exceeds that of AL or IA. In addition, considering the importance of cultural activities in healing communities – as stated by Key Informants, community service providers and end users – the community-level impacts of cultural enrichment projects takes on additional depth and relevance.

4.4 Conclusions and Recommendations

4.4.1 Conclusions

INAC's National Child Benefit Reinvestment in First Nations program is relevant, owing to the continuing levels of poverty, high levels of unemployment, low educational attainment levels, scarcity of jobs, and the high percentage of children in First Nations populations. As long as these conditions persist, there will be a continuing need for the kinds of supports to low-income parents and their children that the NCBR funds.

Reporting for the NCBR is insufficient (i.e., is only annual) and inaccurate (i.e., the same family or child can be counted numerous times if participating in several programs). In addition, the reports lack clearly defined and measurable outcomes; rather, they report outputs only. Despite the inability to quantify the extent to which the NCBR programs are assisting children in low-income families, however, the evaluation found that NCBR programs are highly valued by the communities they serve and respond to community-defined needs due to their flexibility in design and execution. Programs that provide hot breakfasts and/or lunches to children and cultural teaching programs, in particular, are greatly valued by parents and educators. EQAO scores in the community of Walpole Island First Nation (Bkejwanong) provide compelling testimony to the positive impacts of the diverse suite of NCBR programs aimed at child development in the community.

Attributing outcomes for the NCBR program is – and will likely remain – a major challenge, owing to the nature of the program itself. With its diverse, five-area mandate that targets low-income children, their parents and the community as a whole; covering basic needs (i.e., food), employment readiness (i.e., training) and culture – each with its own set of distinct outcomes; and dealing with the complex, long-term and multifaceted aspects of poverty alleviation, the program has few possible short-term, quantifiable outcomes. It is a unique program and may therefore require a unique approach to identifying and collecting realistic "outcomes."

4.4.2 Recommendations

The evaluation team has three recommendations for improving the effectiveness of the NCBR, as follows:

1. Initiate a formal discussion with First Nations organizations and INAC regional staff on the most effective way to address reporting issues so that meaningful outcomes can be measured.
   
   
2. Recommend to regions that they adopt a management regime similar to Saskatchewan region, which does the following:
   • Outlines clear expectations;
   • Sets targets in collaboration with First Nations;
   • Communicates the intent of NCBR; and,
   • Provide project proposal support.
3. Revise reporting mechanisms to avoid multiple counting of program participants.
   
   

5.1 Performance Measurement

Evaluators requested National, regional, and community level AL data for a ten-year period between 1997/1998 and 2007/2008. The data sought included funding expenditures, number of clients accessing programs, and the types of services accessed through the program.

5.1.1 Performance Measurement Framework

Table 8 below outlines the data reporting system for each of the three programs based on the Recipient Reporting Guide (RRG)[Note 50] and feedback from preliminary consultations. The table identifies and links each of the programs to its corresponding reports, funding agreements, data collection and report preparation body, report audience, reporting period, and the data being reported (by output and outcome).

5.1.2 AL Data Gaps

The data seen in Table 9 below represents, according to both the RRG and what was heard in the preliminary consultations, the AL output and outcome data INAC ideally intends to collect annually.

National Level

A review of the national level AL data gaps showed that significant outcome indicators, which could speak to program success and overall performance, were missing. Salient data gaps at the national level include:

• Total Annual Care Days data (missing for the years of 1997-2008);
  
  
• Total Clients (Disabled) data (missing for the years of 1997-2008);
  
  
• Funding breakdowns for Institutional Care On-Reserve and Off-Reserve data (missing for the years of 1997-2008);
  
  
• Funding breakdowns for Foster Care On-Reserve and Off-Reserve data (missing for the years of 1997-2008);
  
  
• Total Population of 49+ On-Reserve[Note 52] data (missing for the years of 1997-2008); and
  
  
• Service Delivery Funding data (missing for the year of 2007-2008).

Regional Level

Similar to the national level, a review of regional level data gaps showed that outcome indicators which could provide insight into program performance and success were missing. Salient data gaps at the regional level include:

• Total Annual Care Days data (missing for the years of 1997-2008);
  
  
• Total Clients Served data (missing for years 1997-2008);
  
  
• Total Clients (Disabled) data (missing for years 1997-2008 for all regions except the Yukon which is missing data for the years 1997-2001, 2005-2006, and 2007-2008);
  
  
• Breakdown data for Funding, Clients, and Annual Care Days for Foster Care On-Reserve and Off-Reserve data (missing for the years of 1997-2008); and
  
  
• Total Population 49+ On-Reserve (data not available for the years of 1997-2008).

Community Level

The case study communities were asked to provide any data reports that they generate. Those reports were reviewed and data (where it existed) was entered into the community level administrative and financial data review templates. Data gaps were then identified for each community. Given the difference in the range and level of data provided by each community, the data gaps are different for each community.[Note 53]

Salient data gaps at the community level include:

• Cumulative Days of Service Provided (missing for the years of 1997-2008);
  
  
• Quantity of Clients Served data (missing for years 1997-2008);
  
  
• Total Funding Received data (missing for the years of 1997-2008); and
  
  
• Total Population 49+ On-Reserve data (missing for the years of 1997-2008).

5.1.3 Ability of Data to Measure Outcomes

Data gaps pose a challenge to assessing program performance and success at all levels (national, regional, and community) and hinder the ability to determine whether program outcomes are being achieved. The lack of data on total care days and funding by program component makes it difficult to assess program performance. The absence of 49 years+ population data makes it difficult to assess the capacity of the program to service the potential target population of people at risk.

5.1.4 Consistency and Comparability of Data

• It is suggested that in some cases the means test is not being applied consistently for eligibility criteria;
  
  
• It is unclear how the reporting system operates. It appears as though some regions (i.e. Quebec and Saskatchewan) use a fully functioning electronic data system to enter program data; however, it is unknown whether that system is comparable and consistent with the electronic systems operating in other regions.

5.1.5 Reporting Capacity

This section outlines evaluation findings with respect to the capacity of staff and systems to adequately report on the program.

National Level

At current staffing levels of one staff member, headquarters staff do not have the capacity to manage data or report back to the regions and communities about the significance of data collected.

Regional Level

Regional staff noted that, in many instances, they do not have the in numbers or skill to manage and interpret data effectively.

Community Level

During the case study visits, program administrators were asked to provide program reports to the evaluation team. Of the 10 communities visited, 70% did not provide the evaluation team with program reports. The reasons for not providing these reports were as follows:

• INAC regional offices had already received the requested reports from the communities; therefore, it was recommended that the evaluation team contact the regional offices directly for the documents (mentioned by 42.9% of communities who did not provide the team with program reports);
  
  
• In British Columbia, the Assisted Living program is considered a part of the IA program; therefore, it was recommended that the evaluation team contact INAC regional office's IA administration for the reports requested (applies to 28.6% of communities who did not provide the team with program reports).

5.2 Program Effectiveness

5.2.1 Access to Services

The program is designed that clients access the program through an assessment of their care needs by a "designated social or health professional" according to the care assessment criteria of the relevant province or territory; in practice, this is often the home care nurse in a community, or the Social Development Worker (B.C.). Qualitative evidence from the evaluation (end-user surveys and service provider interviews) indicates that this requirement is widely met; but the evaluation did not ask the exact nature of the assessment or the criteria being employed.

In the case of children, a formal assessment is required, and the child must not fall under the responsibility of any other agency or program for such services. The point is moot, however, as the AL program has the authority, but not the resources to fund children's AL services at present. Parents whose children need assisted living supports either provide the supports themselves or move off-reserve. It was reported that in some cases, parents are giving care of their child to Child and Family Services in order to access disability services through that route. The exception to this is B.C. region, where children's services are covered through the provincial Community Living B.C., which delivers support and services to people with developmental disabilities and children with special needs.

Program access is also intended to be contingent on the end-user's personal and family financial circumstances. In practice, this eligibility criterion does not appear to be applied. At least one key informant, a home care nurse, indicated that to do so would be highly insulting to Elders, who clearly need the service.

5.2.2 Linkages / Integration with Complementary Services

The AL program has close links and complementarity with Health Canada's First Nations and Inuit Home and Community Care (FNIHCC) Program. At case study sites, the two programs are de facto integrated in most cases; that is, home support workers of the in-home component of AL work out of the same offices and are supervised by the home care coordinators. The two sources of funding remain separate and two sets of reports are required.

In Ontario, the Homemakers program is closely linked to the local Community Care Access Centres (CCACs); and in B.C. and Manitoba, to the Regional Health Authorities.

5.2.3 Service Gaps or Overlap

A number of service gaps have been identified by previous evaluations and reports. The most significant gap is in children's services; although these are part of the AL program authority, they have not been funded. Other service gaps noted by the Assisted Living Expert Panel, and in key informant interviews are group homes; supportive housing, palliative care, respite care, and evening and week end coverage. Foster care is rarely implemented (only 1% of service profile).

5.2.4 Comparability with Provincial Services

The Assisted Living program authority is guided by the principles that AL services will be delivered at "standards reasonably comparable to the reference province/territory of residence; and that recipients must be "ordinarily resident on reserve."[Note 54] Both key informants and expert panellists noted that the AL is not matching provincial levels of service on a number of fronts:

• Services for children are neither defined or funded
  
  
• Most provinces are investing more in supportive housing
  
  
• Most provinces are investing more in personal care needs
  
  
• Provincial level services are much better integrated and provide "single point of access"
  
  
• AL program does not have the clear standards that provincial programs have
  
  
• AL Program's institutions (PCHs) are not funded for higher levels of care
  
  
• Some regions (Alberta and New Brunswick) have no funding for institutional care
  
  
• Some INAC regions do not fund AL services for mentally handicapped or brain injured clients
  
  
• Provinces provide respite care, while AL program does not
  
  
• The INAC program provides very little adult day care compared to provinces
  
  
• The INAC program provides very little Foster Care/Group Home services compared to the provinces
  
  
• Salaries for home support workers, nurses and Personal Support workers are lower than provincial rates
  
  
• A much higher percentage of INAC's AL programs are delivered in small, remote communities with the attendant problems of staffing and transportation

The Joint Working Group on continuing care has worked toward the integration of the AL and Home Care Services programs under Health Canada's program authority for the FNIHCC program as one of the means of addressing the numerous and significant shortfalls in comparability and in an attempt to mirror the integration of two programs at the provincial level,. The Directors General of the two programs are currently considering this option. Comparability was one of the key factors behind the initiation of this discussion.

5.3 Program Impacts

The following section outlines the evidence of program impacts for the AL program.

5.3.1 Extent to which Needs are Being Met

Funding Trends

Program funding is determined based on population, rather than on documented needs. Assisted Living services for children, while officially part of the AL program authority, are not funded.

Average growth in expenditures of 3% appearing in the survey period seems to be consistent with increases in the economy generally, and/or inflationary trends. However, a large increase of 16% in FY02/03 seems to be an anomaly. In that year, Types 1 & 2 Institutional Care and Disabled program funding also spiked. The variances in 2002/03 can be explained by a $10 million one-time funding transfer to the Yukon and a $2 million decrease in the Atlantic region.[Note 55]

Funding expenditures for the AL program have increased by nearly 33% in the ten year review period, from $66.8 million to $88.5 million; Figure 16).

In addition, service delivery costs grew 29% from FY05/06 to FY06/07, a major variance from the annual average of 9.8%. Service delivery fell sharply in FY01/02 and increased thereafter but ended as a lower percentage of the total than in the initial period – indicating that funding for service delivery is not keeping pace with program expenditures.

Figure 16: AL Funding Expenditures (1997-2008 Current $)

Annual Care Days and number of AL Facilities are used to indicate the volume of the AL program nationally. Over the past three fiscal years, annual care days have increased (Figure 17), although the number of Type 1 care days appears to be level (Figure 20) and the number of Type 2 care days is at the lowest point of the survey period (Figure 21).

Funding levels have not kept pace with the needs of the AL program client population. Annual rates of growth have been less than half of those in provinces / territories for similar services. Levels of service as illustrated by Annual Care Days (Figure 17) have steady declined over the past nine years from their peak point in 1998-1999.

Figure 17: Assisted Living (All Components) Total Annual Care Days, 1997-2008[Note 56]

Funding for In-Home Care

The largest component of Assisted Living is In-Home Care. Funding expenditures have grown from $32.3 million (fiscal year 1997-1998) to $46.9 million (fiscal year 2007-2008); however, client volume has dropped from a high of over 11,000 in fiscal year 2000-2001 to just under 6,600 in fiscal year 2007-2008. While the reasons for this precipitous decrease are not entirely clear, it is possible that clients who were accessing the AL program in the early years of this period transferred to the FNIHCC program, which began in 1999. Additionally, a primary reason for this, as expressed by regional staff, is that since institutional care has a significant exogenous cost component it is becoming an increasing cost demand on the regional AL budgets, resulting in less funds being available for in-home care and consequently a decrease in service.[Note 57]

Figure 18: Assisted Living Program, In-Home Care Funding, 1997-2008

Although the AL expenditures have grown over the survey period, funding for disabled individuals has been decreasing except for fiscal year 2002-2003 – fiscal year 2003-2004 where a five-fold increase is noticed.[Note 58] The number of disabled individuals is not captured in the data template; however, an overall decrease in expenditures from the beginning of the period under review is recognized. (see Figure 19). Without knowing the numbers of individuals accessing the program, it is not possible to say whether it is a drop in client numbers or a decrease in benefits that would account for the current lower levels of funding.

Figure 19: Funding for Disabled (AL Program), 1997-2008

Annual Care Days and number of AL Facilities are used to indicate the volume of the AL program nationally. Over the past three fiscal years, annual care days have increased (Figure 17), although the number of Type 1 care days appears to be level (Figure 20) and the number of Type 2 care days is at the lowest point of the survey period (Figure 21).

Figure 20: Institutional Care (Type 1) Clients – Annual Care Days, 1997-2008

Figure 21: Institutional Care (Type 2) Clients – Annual Care Days, 1997-2008

Identifying patterns or trends in client volumes is challenging. Institutional care data is broken down by level of care (Type I or Type II) and by location of institution (on or off reserve), and as is the case for in-home and foster care, data is provided as annual totals as well as person days for the year. Generally, the data is provided by First Nations as per the reporting requirements but in the case of Saskatchewan the regional office manages a separate paylist[Note 59] for First Nations under CFAs and reports directly on the care that it manages.

One challenge in interpreting the data is that many provinces do not provide programs that directly compare to Type I and Type II classifications. For instance, there is no Type II Care reported in Quebec and the Yukon or until 2005-2006 in Atlantic region; as well, there is no Type I care in Ontario or for BC. The data therefore provided limited insight into the Type of care First Nation people require or receive.

Program eligibility also varies across regions and time; the program was initially limited to status Indians on reserve, and has been expanded to residents on reserve, although no evidence of non-Indian recipients was discovered.[Note 60]

5.3.2 Impacts on end-users

To determine whether clients have achieved "functional independence" as a result of the AL program is not possible given the lack of indicator data. Determining whether "functional independence" has been achieved is virtually impossible at present; first of all because the term has not been defined, and secondly, because it would take a formal assessment by a health professional to determine whether this has in fact been achieved. Much more refined indicators are required to determine impacts. That being said, the evaluation collected qualitative evidence of the impacts on end-users.

The evaluation employed three principal lines of evidence to assess impacts on end-users: expert opinion by means of an expert panel; expert opinion at the community level by means of interviews with service providers; and a survey of AL end-users.

Qualitative evidence gathered in this evaluation indicates that the AL program is having positive impacts on end-users. The 32 AL clients surveyed[Note 61] expressed satisfaction overall with the services they receive from the AL program, and overwhelmingly responded that they are getting all the help they need to live independently at home. The problems that were noted by a minority of clients surveyed were related to length of time spent by home support workers; in other words, clients would prefer the home support worker to spend more time in the home. The other problem noted was with meal preparation; a small number of clients remarked that home support workers do not prepare the kind of food they prefer, particularly traditional foods and diabetic diets.

Expert Panelists and some home support supervisors expressed concern over the "huge" safety issues that are a risk to the Assisted Living Program. In some communities without the option of institutional care on reserve, it was reported that clients who need around-the-clock supervision are relying on family members to provide this at night and on week ends.

5.3.3 Impacts on Community Program Personnel

Community AL programs are understaffed and no service delivery funding is provided to the program. Interviews with service providers, both supervisors and home support workers themselves, show that in most instances, the program is understaffed and that home support workers are working long hours and frequently on week ends to meet client support needs. Although workers are noted as very dedicated by both clients and supervisors, morale is low. Home support workers note that their salaries are lower than pay scales off-reserve, and that they do not receive sufficient compensation for transportation. In some instances, employees are using their own vehicles to take clients grocery shopping or to appointments, without being adequately reimbursed for doing so. Workers also noted that their personal well being would be enhanced if they were paid enough to be able to outfit themselves with safe and secure footwear for the heavy work required.

Recruitment and retention of staff is a continuing challenge. The community of Eskasoni reported that of approximately fifteen community members who had been trained as home support workers, only four took employment in the AL program; respondents surmised that the reasons for this were that the salary was too low to provide a good incentive for relinquishing welfare benefits; and that the work is "hard" and "not for everybody." The Personal Care Home in Opaskwayak Cree Nation expressed deep concern over the wage inequities with off-reserve pay scales, and faces a continual challenge to recruit and retain nursing staff for their facility.

5.3.4 Community-Level Impacts

Community service providers noted that families are stressed by looking after their frail/elderly family members, as little or no respite care available. Communities without institutions whose frail elderly must go off-reserve to access institutional care are losing the positive moral and cultural impact of having Elders in the community, and the body of traditional knowledge they carry with them.

Families with children who have special needs for AL services may move from the community in order to access the needed services.

5.4 Program Cost Drivers

Outlined below are the economic, health, regulatory, and demographic trends likely to affect the AL program on-reserve, along with examples of best practices in place likely to facilitate in addressing cost drivers.

• Regulatory issues faced by the program include licensing requirements for institutions, which may pose to be a hurdle for some First Nations; inconsistent funding models across the regions, and the complexity of provincial legislation and the fact that legislation is subject to regular change and may not be consistent with First Nations needs (KTA Inc., 2007; INAC, 2008b).
  
  
• The 2007 program review (KTA 2007) points to a shortage of funding for the AL program as a whole and to the fact that small facilities or those with low occupancy rates are often not able to cover their operations and maintenance costs (KTA Inc., 2007). Financial viability of on-reserve facilities will be of concern if the demand for on-reserve services increases (KTA Inc., 2007). Funding is also related to ability to attract and retain qualified staff while competing with other facilities for personnel.
  
  
• Remote / small communities will have higher costs and different needs than other communities. Access is and will continue to be a problem;
  
  
• Life spans are shorter, rates of communicable diseases increased, limited access to healthy food, environmental health worsening;
  
  
• First Nations clients are now accessing Continuing Care services 10 years earlier than general population. This trend is linked to the increasing incidences of chronic disease;
  
  
• Program funding needs to increase to in order to match off-reserve services. One expert estimates that program funding would need to be doubled in order to meet the needs of the expanding "near elderly" demographic;
  
  
• Many elderly clients are opting to stay in homes regardless of whether services are available. Programs are being forced to provide care in response to this trend;
  
  
• Addressing the gaps related to facilities or supportive housing would require an investment of $200 - $300 million dollars.

AL Best Practices:

• Standard assessment tool;
• Single point of access;
• Collaboration with other programs; and
• Evidence-based care.

5.5 Conclusions and Recommendations

5.5.1 Conclusion

The Assisted Living Program is vitally relevant in First Nations communities. The evaluation found that AL services being delivered in communities are meeting a real need and are highly valued by community members and AL end-users alike, as a means of assisting the elderly and disabled to remain in their homes and have an improved quality of life. Aboriginal health and demographic trends indicate that the need for the program will rapidly increase in the near future, as the "near-elderly" cohort reaches the age where such services are normally accessed. Health status differences between Aboriginal and other Canadians mean that First Nations members can be expected to access such services approximately ten years earlier than non-Aboriginal Canadians.

The program is under-funded, resulting in significant service gaps. Children's special needs for such services have been part of the AL program authority since 2003, but have no funding. Other noted service gaps are services for developmentally disabled and brain-injured residents, palliative and respite care, and supportive housing. None of the case study communities visited in this evaluation reported providing these types of care.

Community-based AL programs, both the in-home and institutional components, face critical challenges in recruiting and retaining staff, due in part to wage scales that are significantly lower than those off-reserve. These HR shortages have impacts on existing staff, who have heavy work loads, long hours, and the resulting stresses of such working conditions.

At the community level, the AL in-home services, in their regional variations, are de facto integrated with Health Canada's First Nations and Inuit Home and Community Care Program (FNIHCC) at the service delivery, if not the funding, level. This may be due in part, to lack of a service delivery funding component, but is most likely the practical and common sense solution devised by First Nations service providers to efficiently meet client needs. The requirements of double reporting and keeping funding separate are reported as onerous in some cases. Discussions are ongoing between INAC and FNIHB to integrate the in-home component of AL with the FNIHCC program.

The program does not have an evaluation framework with defined outcome indicators. While supporting clients to "functional independence" is a program objective, the term is neither defined nor does it have supporting indicators. Even with such a framework, attribution of outcomes would require assessments according to a standard assessment tool, and would still be difficult in light of the complexity of interacting factors related to functional independence. More refined indicators would make attribution more achievable. Alignment with the provincial move to an integrated, continuum of care approach would bring the AL service standard closer to equity with those provided in the provinces.

The following section provides recommendations for addressing the challenges highlighted by the evaluation of the IA program.

5.5.2 Recommendations

The evaluation team has four recommendations for improving the effectiveness for the AL program.

1. Continue the initiative to devolve the funding and authority for the in-home component to the FNIHCC program;
   
   
2. Secure Treasury Board funding for children's AL services, to resource the program authority in place since 2003;
   
   
3. Coordinate discussions at the Federal / Provincial / Territorial and First Nations level to address other AL service gaps, resolve jurisdictional issues and develop an integrated approach to a full continuum of care model;
   
   
4. Fund community-based AL needs assessments and use the information as a basis for reviewing current funding levels.

Project Title: Impact Evaluation of Income Assistance, Assisted Living and National Child Benefit Reinvestment
Project: #0812
Region or Sector: Social Policy and Program Branch

Overall Recommendations

*Recommendations from the 2007 evaluations of the Income Assistance program and National Child Benefit Reinvestment initiative, as well as those from the program-led review of the Assisted Living program, have been included for reference purposes only.

**The above-noted time frames will be reviewed on a quarterly basis and adjusted accordingly.

Income Assistance Recommendations

National Child Benefit Reinvestment Recommendations

Assisted Living Recommendations

Approved by:

___________________________________
Christine Cram
Assistant Deputy Minister
Education and Social Development
Programs and Partnerships

_________________________
Date

1. OCAP refers to First Nations ownership, control, access to and possession of data. (return to source paragraph)
2. See Walpole Island Case Study report for further details. (return to source paragraph)
3. Indian and Northern Affairs Canada. (2005). Income Assistance Program National Manual. Ottawa: Ministry of Public Works and Government Services Canada. (return to source paragraph)
4. INAC-Social Services and Justice Directorate. (August 2008). Draft Results-based Management and Accountability Framework: Income Assistance, Assisted Living and National Child Benefit Reinvestment. (return to source paragraph)
5. Ibid. (return to source paragraph)
6. Ibid. (return to source paragraph)
7. Ibid. (return to source paragraph)
8. INAC-Social Services and Justice Directorate. (August 2008). Draft Results-based Management and Accountability Framework: Income Assistance, Assisted Living and National Child Benefit Reinvestment. AND Indian and Northern Affairs Canada. (2005). Income Assistance Program National Manual. Ottawa: Ministry of Public Works and Government Services Canada. This document has now evolved into a Performance Measurement Strategy for the department; however, data collected for this evaluation was done so under the previous objectives, and hence, this report does not include the objectives revised in late 2008. (return to source paragraph)
9. INAC-Social Services and Justice Directorate. (August 2008). Draft Results-based Management and Accountability Framework: Income Assistance, Assisted Living and National Child Benefit Reinvestment. AND Indian and Northern Affairs Canada. (2005). Income Assistance Program National Manual. Ottawa: Ministry of Public Works and Government Services Canada. (return to source paragraph)
10. Indian and Northern Affairs Canada. (2007). Income Assistance Program. National Manual. Retrieved Sept 2008. (return to source paragraph)
11. Indian and Northern Affairs Canada. (January 10, 2008). Draft Integrated RMAF / RBAF for Income Assistance, Assisted Living, and National Child Benefit Reinvestment Programs. (return to source paragraph)
12. INAC. (2007). Evaluation of the Income Assistance Program. Ottawa: INAC - Audit and Evaluation Sector. (return to source paragraph)
13. This includes those currently on-reserve, but also those who have been off-reserve less than 12 months. (return to source paragraph)
14. Indian and Northern Affairs Canada. (2005). National Child Benefit Reinvestment Initiative National Manual. Ottawa: Ministry of Public Works and Government Services Canada. (return to source paragraph)
15. INAC-Social Services and Justice Directorate. (August 2008). Draft Results-based Management and Accountability Framework: Income Assistance, Assisted Living and National Child Benefit Reinvestment. AND Indian and Northern Affairs Canada. (2005). National Child Benefit Reinvestment Initiative National Manual. Ottawa: Ministry of Public Works and Government Services Canada. (return to source paragraph)
16. Ibid. (return to source paragraph)
17. Ibid. (return to source paragraph)
18. Ibid. p. 31. (return to source paragraph)
19. Ibid. (return to source paragraph)
20. Indian Residential School Survivors, (2007: 2), National Conference and Workshops - Final Report: Preparing Survivors and Communities to Move Forward, Winnipeg, 2007. (return to source paragraph)
21. Finding from the preliminary consultations. (return to source paragraph)
22. Ibid. (return to source paragraph)
23. Ibid. (return to source paragraph)
24. Ibid. (return to source paragraph)
25. INAC-Social Services and Justice Directorate. (August 2008). Results-based Management and Accountability Framework: Income Assistance, Assisted Living and National Child Benefit Reinvestment. AND Indian and Northern Affairs Canada. (2005). Assisted Living Program National Manual. Ottawa: Ministry of Public Works and Government Services Canada. (return to source paragraph)
26. Indian and Northern Affairs Canada. (January 10, 2008). Draft Integrated RMAF / RBAF for Income Assistance, Assisted Living, and National Child Benefit Reinvestment Programs. (return to source paragraph)
27. Services to Mentally Handicapped. (return to source paragraph)
28. Treasury Board Secretariat, 2001. Guide for the Development of Results-based Management and Accountability Frameworks. p. 21-22. (return to source paragraph)
29. Recipient Reporting Guide (RRG), 2008-2009, INAC. (return to source paragraph)
30. CFNFA: Canada/First Nations Funding Agreement CFA: Comprehensive Funding Arrangement. (return to source paragraph)
31. Recipient Reporting Guide (RRG), 2008-2009, INAC. (return to source paragraph)
32. See Section 3.1.4 for discussion on reasons why data may be missing from the community level. (return to source paragraph)
33. Recipient Reporting Guide (RRG), 2008-2009, INAC. (return to source paragraph)
34. Recipient Reporting Guide (RRG), 2008-2009, INAC. (return to source paragraph)
35. Indian and Northern Affairs Canada. 2004. Income Assistance Program National Manual. (return to source paragraph)
36. Ibid. (return to source paragraph)
37. According to the 1965 Indian Welfare Agreement between Ontario and the federal government, Ontario is responsible for on-reserve social assistance, and INAC reimburses the province (at 93%). (return to source paragraph)
38. Indianand Northern Affairs Canada. Audit and Evaluation Sector. December 2007. Evaluation of the Income Assistance Program. (return to source paragraph)
39. Evaluation of the Income Assistance Program. Audit and Evaluation Sector, INAC. December 2007. (return to source paragraph)
40. Audit and Evaluation Sector Indian and Northern Affairs Canada. (Dec. 2007). Evaluation of the Income Assistance Program. (return to source paragraph)
41. Recipient Reporting Guide (RRG), 2008-2009, INAC. (return to source paragraph)
42. Recipient Reporting Guide (RRG), 2008-2009, INAC. (return to source paragraph)
43. See Section 4.1.4 for discussion on reasons why data may be missing from the community level. (return to source paragraph)
44. The Vice Principal's claim is substantiated by NCBR Annual Reports (2007/2008) for Youth Healthy Lifestyles, Speech and Language, Breakfast and Lunch Program, and Art and Play Therapy, which mentioned the school. (return to source paragraph)
45. Home-to-Work Transition Projects 'offer training opportunities to increase the skill level of parents and/or eligible youth which can increase their chances of obtaining work (Examples include: employment and training programs; youth summer work programs; and personal development workshops (e.g. job-readiness training)). Source: INAC. (n.d.) First Nations National Child Benefit Reinvestment (NCBR) Initiative: Proposal Development & Reporting Guide. (return to source paragraph)
46. "Community Enrichment" includes programs such as "teaching traditional culture, support projects for youth, celebrations, peer support groups and group activities that build community awareness" (INAC, 2005e, p.73). (return to source paragraph)
47. Project Providing Support to Parents 'offer early intervention for parents to help their children with a healthy start in life (e.g. parenting programs; drop-in centers for parents and children; and parent and child activities)'. Source: INAC. (n.d.) First Nations National Child Benefit Reinvestment (NCBR) Initiative: Proposal Development & Reporting Guide  (return to source paragraph)
48. It is important to note, however, that some NCBR programs that are delivered outside the school (e.g., My Father/Family Reads to Me; Good Munchies, Good Reads; and Operation Read - all of which were delivered in the public library in 2007/2008) likely contribute to improved academic performance in school. (return to source paragraph)
49. A full list of NCBR reports provided by the Walpole Island First Nation for this evaluation is provided in the Walpole Island Case Study Report - one of the technical reports submitted as part of this evaluation. (return to source paragraph)
50. Recipient Reporting Guide (RRG), 2008-2009, INAC. (return to source paragraph)
51. Recipient Reporting Guide (RRG), 2008-2009, INAC. (return to source paragraph)
52. It should be noted that INAC did not intend to collect this data. (return to source paragraph)
53. See Section 5.1.4 for discussion on reasons why data may be missing from the community level. (return to source paragraph)
54. Indian and Northern Affairs Canada. 2004. Assisted Living Program National Manual. (return to source paragraph)
55. Figure 15; Indian & Northern Affairs Canada Assisted Living Policy & Program Review DRAFT REPORT, Submitted September 2007 (KTA Inc.). (return to source paragraph)
56. Annual Care Days is the sum (National) of: In-Home Care, Homemaker Service, Other Services, Institutional Care (Type 1 & 2, On- and Off-Reserve), and Foster Care days. (return to source paragraph)
57. Indian & Northern Affairs Canada Assisted Living Policy & Program Review DRAFT REPORT, Submitted September 2007 (KTA Inc.), page 71; Annex A, page 24. (return to source paragraph)
58. The increase is explained by a one-time transfer to Yukon, and a decrease in funding to Atlantic region the same year. (return to source paragraph)
59. This is a list of persons with various disabilities or difficulties functioning in their own home, typically elderly members who have moved off-reserve or younger persons with severe handicaps. This is done in Saskatchewan, but not reported by other regions. (return to source paragraph)
60. Indian & Northern Affairs Canada Assisted Living Policy & Program Review DRAFT REPORT, Submitted September 2007 (KTA Inc.); Annex A, page 7. (return to source paragraph)
61. See Appendix VI for a summary of AL survey responses. (return to source paragraph)