Measurable Assessment in Recreation
for Resident-Centered Care
(MARRCC)

Reliability and Validity
Conceptualization
The Measurable Assessment in Recreation for Resident-Centered Care (MARRCC) was developed to provide recreational professionals working in Long Term Care (LTC) a standardized clinical assessment. The MARRCC provides a measurement of resident's functional level within four domains (Cognitive, Social, Physical, and Emotional) as they relate to recreation involvement. The underlying conceptual framework for the MARRCC may be found within the Leisure Ability Model (Peterson and Gunn, 1984) and the Model of Selective Optimization and Compensation (Baltes and Baltes, 1990). The Leisure Ability Model operationalizes the delivery of therapeutic recreation practice and consists of three program categories: Treatment, Leisure Education, and Recreation Participation. Treatment services have as their focus the improvement of physical, cognitive, social, and/or emotional functional abilities to enhance a client's/patient's ability to participate in leisure experiences. Leisure Education focuses on the development and acquisition of skill, attitudes, and knowledge related to leisure participation and leisure lifestyle development. Recreation participation services provide the opportunity for clients to engage in structured group recreation experiences for enjoyment or self-expression. (Peterson and Gunn, 1984).

The Model of Selective Optimization and Compensation (Baltes and Baltes 1990) is a developmental theory of successful aging/adaptation. It is based on the premise that successful individual development (including aging) is a process that involves three components: selection, optimization, and compensation. Selection refers to the restriction in the amount and variety of one’s life pursuits due to a decrease in functional abilities. This decrease in functioning is caused largely by impairment of the individual’s ability to adapt to environmental or intrinsic demands. The process of selection implies that an individual will choose to participate in life pursuits in which the environmental demands match personal motivations and remaining resources, skills, and abilities (Baltes & Baltes). Optimization refers to the propensity of an individual to capitalize on remaining personal strengths and abilities as well as environmental resources while engaging in leisure and life pursuits. Thus, even though the number, and perhaps variety, of life pursuits are decreased due to the aging process, the potential for the individual to experience a sense of satisfaction and control while engaging in these pursuits is maintained at the highest level possible (Baltes & Baltes). Compensation refers to the use of one or any combination of psychological cues, social/environmental adaptation, and technological adaptive devices to compensate for those functional abilities and behavioral capacities that are lost due to the aging process. These adaptations enable the individual to continue to participate in chosen life pursuits with a level of independence and mastery as similar as possible to that which was experienced prior to the onset of a decrease in functioning (Baltes & Baltes).

Application of the Theories
Both the Leisure Ability Model (Peterson and Gunn, 1984) and the Model of Selective Optimization and Compensation (Baltes and Baltes, 1990) lend support for the development of an assessment of resident/client/patient physical, cognitive, and psycho-social functioning. To clarify, within the Treatment element of the Leisure Ability Model, "Individuals with a disability or illness are assessed to determine their physical, mental, social, and emotional assets and deficiencies. Specific functional deficiencies that would limit or interfere with leisure involvement become areas for possible program intervention. Programs are then designed that have goals related to the identified functional improvement." (Peterson and Gunn, 1984). Similarly, the Selection component of the Selective Optimization and Compensation Model focuses on those domains of functioning that match environmental demands with the individual's motivations, skills, and abilities. "We cannot predict what any given individual's successful aging will look like until we know the domains of functioning and goals that that individual considers important, personally meaningful, and in which he or she feels competent." (M. Baltes and L. Carstensen, 1996).

The importance of accurate client/patient/resident assessment is paramount within the therapeutic environment. So too is the application of assessment outcomes to the development of treatment protocols and clinical care plans. The MARRCC provides an assessment of physical, cognitive, locial and emotional functioning to facilitate efforts which ultimately support individuals in their efforts to achieve, maintain, and/or regain optimized leisure lifestyles.

Description of the Domain Scales
The MARRCC includes 4 domain scales: Cognitive, Social, Physical, and Emotional. Each scale is composed of 10 items with corresponding indicators. The indicators have specific minimum requirements that must be met for the item to be considered "True". If a resident does not completely fulfill an indicator's requirements, the item is considered "False". The functional score for the domain can be assessed on an index from -1.85 to 1.85, with a score of zero as the median. The associated functionality level for each domain can be assessed using the following guidelines:

Low Functionality
-1.86 to -1.29

Moderate Functionality
-1.28 to +1.39

High Functionality
+1.40 to +1.86

Development of the MARRCC
PILOT PHASE: To establish face validity, a panel of experts was chosen and asked to respond to several questions regarding the MARRCC. The panel of experts consisted of three therapeutic recreation educators, one therapeutic recreation practitioner with education at the Ph.D. level, one LTC assessor, and ten individuals with 10 or more years of clinical experience in the field. The results were as follows:

1. Is the MARRCC an adequate assessment for use in LTC? On a scale of 1 (poor) to 5 (excellent), the MARRCC-FAD received an overall rating of 4.8.
2. Are there any items in MARRCC that are difficult to understand? 82% said "No".
3. Are there any unnecessary items included in MARRCC? 90% said "No".
4. Are there any items that should be added to MARRCC? 82% said "No".
5. Please rate the overall usefulness of MARRCC for Therapeutic Recreation Specialists (TRSs) working in LTC. On a scale of 1 (poor) to 5 (excellent) the panel rated the usefulness of the MARRCC as 4.72.

In addition to the above, a separate list of questions was developed specific to the domain scales within the tool.

1. "Please rate the overall readability of the four domain scales." On a scale of 1 (poor) to 5 (excellent) the MARRCC Domain scales received an overall rating of 4.5.
2. "Please rate the quality of the scaling in within the domain scales." On a scale of 1 (poor) to 5 (excellent) the MARRCC-Domain Scales received an overall rating of 4.4.
3. "Please rate the quality of items chosen to measure the functioning level for each domain." On a scale of 1 (poor) to 5 (excellent):
   Cognitive Domain - 4.30
   Social Domain - 4.30
   Physical Domain - 4.40
   Emotional Domain - 4.25
   

To determine the reliability of the assessment between different assessors, percentage agreement between the ratings of two assessors were computed in the following manner. After receiving the assessment tool and the manual that accompanies it, the assessors independently rated the same residents for all four domain scales. Percentage agreements were calculated by dividing the number of exact agreements by the number of exact agreements plus disagreements. A total of seven assessors, resulting in 21 different pairings, were involved. Percentage agreements between the assessors averaged 88%. The physical and emotional domain scales of the MARRCC were modified based on information derived from the panel of experts as well as the simplified reliability test. The pilot phase was completed in September 2000.

TRIAL PHASE: The trial phase entailed extensive multi-site implementation of the MARRCC to obtain practical information regarding its usefulness and feasibility. During the trial phase more extensive validity and reliability testing was conducted. In addition, during this time, the MARRCC was further modified to ensure consistency with the MDS - version 2.0 and the Care Plan Development and Quarterly Progress Notes sections of the assessment package were added. The trial phase was completed in June 2002.

Reliability and Validity of the MARRCC
Participants in the study included 66 residents from 11 Skilled Nursing Facilities. To be included in the study, participants were required to have been residents in the facility for at least 6 months. No other restrictions were made when selecting the type of resident.

Reliability: Test re-test reliability was examined by assessing participants on two separate occasions - with a time period of three days between assessments.

Validity: To examine the validity of the MARRCC, the MARRCC was compared to the Multidimensional Observation Scale for Elderly Subjects (MOSES). The MOSES (Helmes, 1987) provides a comprehensive, valid, and reliable assessment of resident physical, cognitive, and psycho-social functioning and was included in the testing of the MARRCC to provide an anchor for criterion-related validity. For the purpose of testing the validity of the MARRCC the Cognitive domain of the MARRCC was compared to the Disorientation scale of the MOSES, the Social domain of the MARRCC was compared to the Withdrawal scale of the MOSES, the Emotional domain of the MARRCC was compared to a combined score for the Depresses/ Anxious and Irritable scales of the MOSES, and finally the Physical domain of the MARRCC was compared to the Self-care scale of the MOSES.

PROCEDURE: The MARRCC was completed by Recreation Services Directors (RSDs) who were involved with direct resident care and were well acquainted with the participants being assessed. No attempt was made to select raters because a typical sample of skill, education, and experience was desired. The RSDs received detailed written and verbal instructions regarding how to obtain a random sample of participants as well as timelines for the completion of the assessments. No training was provided beyond the written MARRCC User's Guide for the MARRCC and the written instruction for the implementation of the MOSES. Eleven Recreation Service Directors at 11 different Skilled Nursing Facilities rated six residents each, utilizing both the MARRCC and the MOSES on the same day. The RSDs were then asked to rate the same six residents three days later utilizing only the MARRCC. One RSD did not complete the re-test; therefore, data from that site was not included in the results of the test-retest study. Upon completion of the assessments, the forms were returned for scoring and statistical analysis.

Results
Test/Retest Reliability
Test/retest reliability (intra-rater reliability) was assessed utilizing the Pearson Product Correlation Coefficient. Correlation coefficient values for each of the raters were all significant (p less than 0.0001). Coefficient correlation was as follows: Cognitive 0.99, Social 0.98, Physical 0.97, and Emotional 0.95.

Scale Reliability
Within MARRCC the domain scores are determined from a modified averaging technique. Therefore, to determine if a Cronbach's alpha of scale reliability was appropriate an additive scale (adding all items in each domain) was created and plotted against the domain scores calculated from the same data. Correlations between the two scores were very high, ranging between 0.98 and 0.99. Therefore, it was considered appropriate to calculate values for Cronbach's Alpha Reliability Coefficient utilizing the additive scale. The reliability results ranged from acceptable to very good: Emotional (alpha=0.72), Social (alpha=0.74), Physical (alpha=0.83) and Cognitive (alpha=0.90).

Validity
To assess the validity of the MARRCC, each of the four MARRCC domains were compared to the four MOSES domains. For the validity test, the MOSES Self Care subscale was modified by removing the following items for the test: Dressing, Bathing, Incontinence and Using the Toilet. This was done for two reasons:

1. The primary focus of the MARRCC physical domain is to assess the resident's physical mobility and ability to physically participate in leisure activities of choice; it does not address incontinence, bathing, dressing, or grooming.
2. The resident's ability to participate in activities of daily living may be determined more by the staff's willingness to enable the residents to engage in self-care than by the residents actual physical functioning. It is generally understood that it often takes less time to totally dress a resident than it does to provide assistance to the resident while he or she dresses him/herself. To examine the criterion related validity of the MARRCC, Pearson Product Correlation Coefficients for the MARRCC and MOSES domain scores were calculated. Results were as follows: Cognitive 0.88 (P less than 0.0001), Social 0.62 (P less than 0.0001), Physical 0.68 (P less than 0.0001), and Emotional 0.39 (P = 0.0015).
   
   DISCUSSION: MARRCC shows high intra-rater reliabilities, ranging from a high of 0.99 for the Cognitive domain to a low of 0.95 for the Emotional domain. These results clearly indicate that the MARRCC can be successfully completed by staff with minimal training other than the instructions provided in the manuals of the MARRCC. Internal consistencies were satisfactory, averaging approximately 0.80. For ten-item scales this is quite good, especially for raters who are relatively inexperienced in using the scale. With regard to the validity of the MARRCC, in general, the MARRCC domain scales compare favorably with the MOSES subscales (Cognitive 0.88 p less than 0.0001, Social 0.62 p less than 0.0001, Physical 0.68 p less than 0.0001) with the exception of the Emotional domain (0.39 p=.0015).
   
   When regarding the values for the Emotional Domain, the following factors should be considered: First, although the MOSES does represent a reliable and valid geriatric assessment, the Depressed/Anxious Mood subscale within the MOSES demonstrated an initial inter-rater reliability of 0.58 (Helmes et al. 1987) which is significantly lower than the inter-rater reliability of the other subscales (Self-Care 0.97, Disorientation 0.84, Irritability 0.72, and Withdrawal 0.75). Second, emotion and mood-related scales generally tend to have lower reliabilities than scales that rate more objective behaviors such as cognitive and physical functioning (Helmes et al. 1987). This is further clarified by Minium and Clark, "Except for certain areas of research, the relationships studied by psychologists and other behavioral investigators seldom give correlations as high or higher than 0.71. Thus, the proportions of variance typically accounted for often fall well below 0.50. (Minium & Clarke, 1982, Elements of Statistical Reasoning, John Wiley and Sons, New York, NY.) Finally, it is also possible that the low correlation between the MARRCC Emotional Domain and the MOSES subscales were a result of the manner in which the Depressed/Anxious Mood and Irritable Behavior subscales of the MOSES were reconstituted to facilitate this particular study of criterion-related validity between the MARRCC and the MOSES.
   
   BETA TEST: A beta test was conducted to establish usability and feasibility of the MARRCC-software version. Four test sites participated in the beta test. Each site received the MARRCC-software version and all accompanying manuals at the onset of the test. Test sites were asked to use the software extensively and answer a weekly questionnaire regarding the usability and functionality of the software. The beta test was conducted for a period of four weeks. Based on feedback derived from the beta test sites, additional fields were added to the software and some navigational fields were slightly altered to maximize ease of navigation for users.
   
   For more detailed information regarding extensive reliability and validity testing on the MARRCC, please contact Sienna Boothman at Boothman@telus.net.