Evidence of Validity of the Brazilian Version of ADS: Assessment of Attitudes towards Disabilities

Keywords

Disabilities; Attitudes; ICF; Scales; WHO

Abstract

Introduction: The number of people with disabilities in Brazil and worldwide has grown considerably in recent decades. However, prejudice and stigma faced by this population have not decreased yet. Negative attitudes towards people with disabilities can impose barriers to functionality and quality of life. Cross-cultural measures of attitudes towards disability can help identify these barriers and contribute to the development of intervention strategies. Objectives: To provide evidence of validity of the Brazilian Portuguese version of a World Health Organization cross-cultural instrument designed to assess attitudes towards disability (Attitudes to Disability Scale, ADS) from the perspective of people with physical disabilities (ADS-D) and Intellectual Disabilities (ADS-ID).

Methods: A total of 162 people with physical disabilities and 156 with intellectual disabilities participated in the study. Classical psychometrics was used to analyze the two samples independently. Evidence of criterion validity (concurrent type) was obtained by Mann-Whitney test for non-normal distributions. Evidence of reliability was calculated with Cronbach alpha for the instrument scales and subscales. Test-retest reliability was assessed for people with intellectual disabilities through intraclass correlation coefficient and Wilcoxon test.

Results: ADS-D showed better levels of internal consistency than ADS-ID. Evidence of discriminant validity was verified. Evidence of test-retest reliability was not conclusive. Conclusion: Results suggest the maintenance of the factor structure revealed in the cross-cultural study to assess the attitudes towards disabilities in the Brazilian population. Studies with larger samples are needed for the investigation of additional evidence of validity and reliability.

Citation

Bredemeier J, Agranonik M, Perez TS and Fleck MPA. Evidence of Validity of the Brazilian Version of ADS: Assessment of Attitudes towards Disabilities. SM J Community Med. 2015; 1(1): 1002

Introduction

The number of people with disabilities is growing worldwide every year, with “disability” being the umbrella term adopted by the World Health Organization (WHO) to define “impairments, activity limitations and participation restrictions” caused by a health condition [1], whether it is impairment or a chronic illness. Currently, it is estimated that over one billion people are living with disabilities [2]. This growth is due both to the increase in life expectancy (in countries where the mean life expectancy reaches the age of 70, it is estimated that individuals spend, on average, 11 years of their lives with some form of disability) and to the occurrence of accidents, urban violence, war, stress, drug abuse, HIV/AIDS, and malnutrition, among others. Despite the increase in the disabled population, the prejudice and stigma faced by these people has not decreased as expected, and studies show that the less knowledge about disabilities, the greater the frequency of biased and stigmatizing attitudes [3-7]. Attitudes may be defined as evaluations and favorable or unfavorable reactions to different aspects of the world. In other words, attitudes are affinities or aversions to people, groups, situations, and any other aspect of the environment, including abstract aspects [8]. According to the WHO, attitudes can represent important barriers to the adaptation of people with disabilities, with noticeable losses in areas such as access to work, education, and health care [1]. Conversely, positive attitudes can enhance their functionality. Researches in university settings have showed that knowledge and information about disabilities led to a higher frequency of positive attitudes towards disabled people among health care professionals, indicating that attitudes can be improved [9-11]. In view of this context, studies have sought to develop questionnaires and/or scales to measure attitude [12] and stigma [13], most of them focusing on intellectual disabilities. Overall, surveys on attitudes towards people with intellectual disabilities are more frequent than studies on attitudes towards physical disabilities and usually analyze the attitudes of health care and education professionals towards patients and students with intellectual disabilities [4,14-17]. In this sense, cross-cultural measures of attitudes towards disabilities can help identifying barriers and contribute to the development of intervention strategies to facilitate functionality and qualityof life. The present study was linked to the DISQOL project, an acronym for “Quality of Care and Quality of Life Project for People with Intellectual and Physical Disabilities: Integrated Living, Social Inclusion and Service User Participation”, whose aim was to develop not only instruments to assess attitudes towards disabilities but also to assess quality of life and quality of care from the perspective of people with disabilities.

The cross-cultural and simultaneous development of the WHO instrument known as Attitudes to Disability Scale (ADS) was described in detail by Power, Green, and the WHOQOL-DIS Group [18], while this report focuses on the work of the Brazilian branch of the WHOQOL-DIS Group. This study aims to present the evidence of validity of the Brazilian Portuguese version of instruments for the assessment of attitudes towards people with disabilities. These instruments took into account the perception of disabled individuals themselves, family members, caregivers, and health care professionals. T he study was carried on under the responsibility of the WHOQOL DIS-Brazil Group. Our specific objectives were to provide evidence of 1) construct validity, through factor analysis and internal consistency testing (Cronbach alpha), 2) criterion validity through discriminant validity, and 3) test-rest reliability. Results on the development of versions of the Quality of Life Instrument for People with Disabilities (WHOQOL-Dis) and of the Quality of Care Instrument for People with Disabilities (QOCS) have been published elsewhere [19, 20]. Data collection in field trial of WHOQOL-Dis, QOCS, and ADS happened at the same time.

Materials and Methods

Inclusion and exclusion criteria

As previously described [19,20], all participants were required to attend or be institutionalized at a facility specializing in the care of people with disabilities, a non-governmental organization, a school, or a health care facility. Participants had to be 18-65 years old. As the concept of disability has not yet been widely disseminated in Brazil, a flowchart was devised and administered to each potential subject with physical disabilities prior to inclusion in the sample [19]. Intellectually disabled people were included only when cared for at a service dealing specifically with the care of persons with intellectual disabilities. In addition, a screening step was used to evaluate the ability of participants to respond to study questionnaires in an assertive manner, ensuring that only persons with mild intellectual disability were included in the sample. Two instruments were used for screening: Test of Acquiescence [21], which determines whether the subject merely tends to agree with the interviewer’s questions (acquiescent responding) or is capable of providing actual answers even to reverse-scored questions; and the Test for Discriminative Competence [22], which seeks to ascertain whether the participant is able to discriminate his/her chosen response on a three-point scale. Participants with intellectual disability were excluded from the sample after failure in both screening tests.

Instruments

Concept exploration led to the development of a pilot version of the ADS, which underwent a cross-cultural analysis by different judges [18]. Psychometric analyses of data obtained by the 15 participating centers of the pilot study were carried out at the coordinating center [18]. The results of this analysis suggested a need for the development of a distinct instrument for participants with intellectual disabilities (i.e., cognitively simpler), and that the use of a three-point rather than a five-point Likert scale would improve the psychometric performance of the instrument in this sample [23].As previously described [19, 20].

The following instruments were used in the DISQOL research protocol:

Whodas II: measure of disability developed by WHO [24] within the International Classification of Functioning, Disability and Health framework [1]. The 12-item version is available both as a self-administered questionnaire and as an interviewer-administered instrument. Cronbach alpha: 0.98. Lowest and highest scores: 5 and 60 respectively. As to the intellectual disabilities sample, minor semantic changes plus the addition of a visual aid (“smiley faces”) were performed as to improve the poor comprehension observed during the pilot study. This modified version was approved by the DISQOL Group and entitled Experimental Brazilian Version for Persons with Intellectual Disabilities.

Socio demographic questionnaire: questions on socio demographic data and participants’ perceptions of their health status and disability.

ADS-D [25]: assesses attitudes towards disability from the perspective of people with physical disabilities using questions scored on a five-point scale. Its cross-cultural version is made up of 16 items designed to analyze the perception of people with disabilities and their family members and caregivers on attitudes towards disability [26]. The higher the score, the more positive the respondent’s perception is with regard to other people’s attitude towards disability.

ADS-ID [25]: assesses attitudes towards disability from the perspective of people with intellectual disabilities. Questionnaire items were the same as those of ADS-D, but the response scale was changed to a three-point system and a visual aid (“smiley faces”) was added [26].

Satisfaction with Life Scale (SWLS) [27, 28]: consists of f ive questions scored on a scale of 1 to 5, so that the lowest possible score (poorest satisfaction with life) is 5 and the highest possible score (greatest satisfaction with life) is 25. Cronbach alpha: 0.82 [27].

Beck Depression Inventory – version II (BDI-II) [29]: 21 item depression assessment scale translated and validated into Brazilian Portuguese [30]. Scoring: 0 (zero) to 63. Cronbach alpha: 0.92 [29].

Due to practical reasons, retesting was restricted to the intellectual disability sample.

Ethical Aspects

The project was approved by the Hospital de Clínicas de Porto Alegre Research Ethics Committee (process no. 06-021). The wording of the Written Informed Consent (WIC) form stressed the possibility of dropping out of the study at any time if desired. For participants in the sample with intellectual disability, WIC was also required from one of the participant’s parents or legal guardians. Participants with intellectual disability who were excluded from the study due to failure at screening tests were not told why their participation was being terminated. An intervention protocol was devised for participants with moderate-to-severe depression or a score of 2 or 3 on the suicidal ideation item (question 9) of the BDI-II. Protocol interventions ranged from notifying the participant of the need for in-depth assessment for depression to notification of the care team in participants positive for suicidal ideation.

This protocol was the subject of extensive discussions between the investigators and the Hospital de Clínicas de Porto Alegre Research Ethics Committee. Research was carried out in accordance with the standards of the Declaration of Helsinki [31], with participant’s anonymity and confidentiality being preserved at all times. These procedures were the same for the development of WHOQOL-Dis [19] and QOCS [20].

Data analysis

All analyzes were based on classical psychometric methods and were conducted independently for the two study samples. Shapiro Wilk and Kolmogorov-Smirnov tests were used to evaluate the normality of distribution and revealed that all variables had a non normal distribution. Confidence interval was set at 95%. Missing data were replaced with the medians of nearby points. Exploratory factor analyses were carried out using principal component analysis with Varimax rotation and Kaiser Normalization. Evidence of criterion validity (discriminant) was obtained by means of Mann Whitney test for variables with a skewed distribution in relation to the following variables: depression (BDI-II), Satisfaction With Life (SWLS), functionality (WHODAS- II), health status, disability status, age, income, and, in the case of the group with physical disabilities, years of study. Cronbach alpha was calculated to obtain evidence of reliability for the instrument scales and subscales. In the intellectual disability sample, test-retest reliability was analyzed by means of the intraclass correlation coefficient (ICC, average measures, two-way random) and Wilcoxon test for paired samples (domain averages). All analyses were carried out in the Statistical Package for the Social SciencesTM (SPSS) version 18.0

Results

A total of 162 people with physical disabilities (98 females) and 156 people with intellectual disabilities (55 females) participated in the study. Mean age was 45.48 years for the group with physical disabilities Standard Deviation (SD 12.26) and 30.53 years for the group with intellectual disabilities (SD 9.42). Visual impairment (15.4%), hearing impairment (6.8%), and stroke sequelae (2.5%) were the most common disabilities in the physical disabilities sample. Overall, missing data rates were at 0.08% for individuals with physical disabilities and 0.68% for those with intellectual disabilities.

In the latter group, two subjects were responsible for 58% of the missing data. Among the physical disabilities sample, missing data were equally distributed and there was no more than one case of missing data for the same item. Among the intellectual disability sample, the highest rate of missing data (1.9%) was found in item 9 (“I achieve more because of my disability”). Exploratory factor analysis of the group of people with physical disabilities (Table 1) revealed that the change in the model in relation to the cross-cultural model was much more related to changes in factor ordering that in item grouping. In this exploratory model, factors 2 (Inclusion) and 3 (Prospects)

Table 1: Exploratory factor analysis: physical disability subsample (n=162)

KMO: 0.825. Bartlett’s test: p<0.001; Explained variance: 53.242% * F1: Inclusion; F2: Discrimination; F3: Gains; F4: Prospects. Extraction method: Principal component analysis; Rotation method: Varimax with Kaiser normalization.

Table 2: Exploratory factor analysis forced into 4 factors: physical disability subsample (n=162).

KMO: 0.825; Bartlett’s test: p<0.001; Explained variance: 59.31%. *F1: Inclusion. F2: Discrimination; F3: Gains; F4: Prospects. F3 F3 F1 F1 F1 F1 F4 F4 F4 Extraction method: Principal component analysis; Rotation method: Varimax with Kaiser normalization

Table 3: Exploratory factor analysis: intellectual disability subsample (n=156).

KMO: 0.711; Bartlett’s test: p<0.001; Explained variance: 54.524%. F1: Inclusion; F2: Discrimination; F3: Gains; F4: Prospects. Extraction method: Principal component analysis; Rotation method: Varimax with Kaiser normalization.

represent the most and least important factor in explaining variance, respectively. In the model forced into four factors (Table 2), factors 1 and 4, which had been previously merged, were organized so as to become independent factors. Exploratory factor analysis of the intellectual disability sample already showed a solution with four factors (Table 3). The results of internal consistency analysis of models constrained to four factors are shown in (Table 4).

Table 4: Evidence of internal consistency.

Cronbach alpha was satisfactory [32] across all ADS items in the intellectual disability sample and remained below 0.70 only in factors, which all have only 4 items. In the physical disability sample, alpha was satisfactory (> 0.70) across all ADS items and factors. Tables 5 and 6 show the results of discriminant validity analysis for people with physical and intellectual disabilities, respectively. Of the 96 participants with intellectual disabilities who completed retesting, 27 were excluded from the retest reliability analysis due to the occurrence of significant life events (positive or negative) during the interval between test and retest. The results of this analysis are shown in (Table 7).

Table 5: Discriminant validity of ADS for people with physical disabilities.

(a) Factors consistent with those identified in the cross-cultural project (global sample); Levels of significance: *p < 0.05; **p ≤ 0.001.

(b) Stratified by quantils. BDI-II = Beck Depression Inventory – version II; SWLS = Satisfaction with Life Scale.

Table 6: Concurrent validity of ADS for people with intellectual disabilities, according to depression and satisfaction with life.

(a) Factors consistent with those identified in the cross-cultural project (global sample); Levels of significance: *p <0.05; ** p ≤ 0.001.

(b) Stratified by quantils. SD = Standard Deviation

Table 7: Test-retest reliability among participants with intellectual disabilities.

(a) Mean interval: 28 days (standard deviation 13.58); Levels of significance: *p < 0.05; **p ≤ 0.001 (two-tailed). df = Degrees Of Freedom.

Discussion

Analysis of results revealed discriminative competence for the depression variable as defined by the BDI-II in both versions of the instrument (for people with physical disabilities and with intellectual disabilities) (except for factor 3 in the physical disability sample), which may suggest that depression plays an intervening role in the way disabled individuals perceive other people’s attitudes. As for the income variable, the two groups showed different results. No significant difference was found when the intellectual disability sample was stratified by income. Conversely, highly significant differences were observed when the physical disability sample was stratified by income. This may suggest that social class plays a mediating role in the perception of positive attitudes in the latter group, in which the highest ADS scores (representing more positive attitudes) were found in the group classified as average, slightly above or well above average. Other clinically significant differences were also observed. For example, there were significant differences among people with intellectual disabilities with regard to the health status variable (also except for F3). As for reliability, Cronbach alpha showed satisfactory results in the physical disability sample and lower coefficients in the intellectual disability sample, which is in agreement with findings on other instruments showing that alpha coefficients lower than 0.70 are to be expected for scales with a few items [33]. It is worth emphasizing that all ADS factors have only 4 items. Analysis of test retest reliability, carried out in subjects with intellectual disabilities, revealed highly significant intra-class correlation coefficients only for factor 3. Inconsistently, a significant difference between t1 and t2 was found precisely in this factor when using the Wilcoxon test. The authors believe that this may have been influenced by the low number of participating subjects. Further studies with larger sample sizes are required. Regarding the scope of the developed instruments, the authors understand that the ADS-D and the ADS-ID include aspects not previously measured by available tools to assess attitudes, which favored the assessment of health care professionals and did not pay attention to the fact that the attitudes of society as a whole may act as facilitators in the lives of people with disabilities [1]. As for recent questions on whether intellectually disabled individuals are able to provide valid self-reports [18], the authors believe that the validity evidence presented in this study speak in favor of the hypothesis that they are. In any case, further research is needed to give insight on this topic. In this sense, it is worth informing that the DISQOL project also sought to triangulate all information obtained from people with intellectual disabilities with data provided by their caregivers and relatives. These findings will be the subject of a future publication.

Conclusion

The present study sought to present evidence of validity of the Brazilian Portuguese version of an instrument developed by the WHO and designed to assess attitudes towards disabilities. Based on the results presented here, the factor model found in the international sample of the DISQOL project showed a similar performance when applied to Brazilian samples. At the same time, the authors believe that the administration of ADS-D and ADS-ID in different samples may contribute to the validation of the best model to be used in the Brazilian population. The availability of the Brazilian Portuguese version of the ADS with evidence of basic satisfactory psychometric characteristics may be an important stimulus for research in this area and contribute to the development of interventions focusing on reducing stigma and prejudice. In addition, ADS can represent a tool The present study sought to present evidence of validity of the Brazilian Portuguese version of an instrument developed by the WHO and designed to assess attitudes towards disabilities. Based on the results presented here, the factor model found in the international sample of the DISQOL project showed a similar performance when applied to Brazilian samples. At the same time, the authors believe that the administration of ADS-D and ADS-ID in different samples may contribute to the validation of the best model to be used in the Brazilian population.

The availability of the Brazilian Portuguese version of the ADS with evidence of basic satisfactory psychometric characteristics may be an important stimulus for research in this area and contribute to the development of interventions focusing on reducing stigma and prejudice. In addition, ADS can represent a tool for the evaluation of interventions and therefore for the collection of useful information for the development of public policies for disabled people with focus on removing barriers and improving tools to facilitate adaptation, functionality and quality of life. Limitations of this study reside primarily in sample size and in convenience sampling. Therefore, the results must be taken with parsimony until additional research with other samples is performed.

Acknowledgement

The authors thank CNPq, CAPES and FIFE-HCPA for funding this project; their research assistants; the study participants; and the participating institutions: Abrigo Cônego Nadal, Associação Canoense de Portadores de Deficiências, Centro Abrigado Zona Norte, Centro de Atenção Integrada, Centro de Orientação e Prep. p/ o Trabalho, Centro Louis Braille, Coopersocial, Escola E.E.Cristo Redentor, Escola E.E.Nazareth (APAE), Escola E.E.Renascença, Escola Lília Mazzeron, Fundação de Amparo e Desenvolvimento de Políticas Públicas para PcD e PcAH no RS, Grupo Hospitalar Conceição, Hospital de Clínicas de Porto Alegre, Hospital São Lucas da PUC/RS, and Projeto Rumo Norte.

Reference

1. Organização Mundial Da Saúde (2003). Classificação Internacional de Funcionalidade, Incapacidade e Saúde - CIF. Original publicadoem 2001.

2. World Health Organization. World report on disability. Malta. 2011.

3. Mankar MJ, Joshi SM, Velankar DH, Mhatre RK, Nalgundwar AN. A Comparative Study of the Quality of Life, Knowledge, Attitude and Belief About Leprosy Disease Among Leprosy Patients and Community Members in Shantivan Leprosy Rehabilitation centre, Nere, Maharashtra, India. J Glob Infect Dis. 2011; 3: 378-382.

4. Satchidanand N, Gunukula SK, Lam WY, McGuigan D, New I, Symons AB, et al. Attitudes of healthcare students and professionals toward patients with physical disability: a systematic review. Am J Phys Med Rehabil. 2012; 91: 533-45.

5. Fonseca LC, Tedrus G M.A.S, Freire costa AC, Luciano PQ, Costa KC. Knowledge and attitude toward epilepsy among health area students. Arq. Neuro-Psiquiatr. 2004; 62: 1068-1073.

6. Gasparetto, Maria Elisabete R. Freire, Temporini ER, Monteiro de Carvalho KM, Jose NK. Students’ visual difficulties: knowledge and actions of teachers of elementary schools working with low vision students. Arq.Bras. Oftalmol. 2004; 67: 65-71.

7. Dantas FD, Cariri GA, Gustavo AC, Ribeiro Filho RV et al. Knowledge and attitudes toward epilepsy among primary, secondary and tertiary level teachers. Arq.Neuro-Psiquiatr. 2001; 59: 712-716.

8. Atkinson e Hilgard’s. Introdução à psicologia.São Paulo: Cengage Learning, 2009.

9. Morgan PE, Lo K. Enhancing positive attitudes towards disability: evaluation of an integrated physiotherapy program. Disabil Rehabil. 2013; 35: 300-305.

10. Instituto Brasileiro De Geografia E Estatística (Ibge). Censo Demográfico 2010. Características gerais da população, religião e pessoas com deficiência. Rio de Janeiro: IBGE, 2012.

11. Thompson TL, Emrich K, Moore G. The effect of curriculum on the attitudes of nursing students toward disability. Rehabil Nurs. 2003; 28: 27-30.

12. Morin D, Crocker AG, Beaulieu-Bergeron R, Caron J. Validation of the attitudes toward intellectual disability: ATTID questionnaire. J Intellect Disabil Res. 2013; 57: 268-278.

13. Ali A., Strydom A., Hassiotis A., Williams R. & King M. A measure of perceived stigma in people with intellectual disability.British Journal of Psychiatry. 2008; 193: 410–415.

14. Willems AP, Embregts PJ, Bosman AM, Hendriks AH. The analysis of challenging relations: influences on interactive behaviour of staff towards clients with intellectual disabilities. J Intellect Disabil Res. 2014; 58: 1072 1082.

15. Knotter MH, Wissink IB, Moonen XM, Stams GJ, Jansen GJ. Staff’s attitudes and reactions towards aggressive behaviour of clients with intellectual disabilities: A multi-level study. Res Dev Disabil. 2013; 34: 1397-1407.

16. Ozer D, Nalbant S, Alamıș E, Baran F, Kaya Samut P, Aktop A, et al. Physical education teachers’ attitudes towards children with intellectual disability: the impact of time in service, gender, and previous acquaintance. J Intellect Disabil Res. 2013; 57: 1001-1013.

17. Werner S, Stawski M, Polakiewicz Y, Levav I. Psychiatrists’ knowledge, training and attitudes regarding the care of individuals with intellectual disability. J Intellect Disabil Res. 2012 .

18. Power, MJ; Grenn, AM; WHOQOL-DIS Group. The Attitudes to Disability Scale (ADS): development and psychometric properties. J Intellect Disabil Res. 2010; 54: 860-874.

19. Bredemeier, Juliana; Wagner, Gabriela Peretti; Agranonik, Marilyn; Perez, Tatiana Spalding; Fleck, Marcelo P. The World Health Organization Quality of Life instrument for people with intellectual and physical disabilities (WHOQOL-Dis): evidence of validity of the Brazilian version. BMC Public Health. 2014; 14: 538.

20. Bredemeier, Juliana;Agranonik, Marilyn; Perez, TatianaSpalding ;Fleck, Marcelo Pio de Almeida. Brazilian version of the Quality of Care Scale: the perspective of people with disabilities. Revista de Saúde Pública. 2014; 48: 583-593.

21. Cummins RA. Comprehensive Quality of Life Scale – Intellectual/Cognitive Disability. 5th Edition. School of Psychology: Deakin University, 1997.

22. Dalton, AJ, McVilly KR. Ethics Guidelines for International Multicenter Research Involving People with Intellectual Disabilities. JPPID, 2004; 1: 57-70.

23. Fang J, Fleck MPA, Green A, McVilly K, Hao Y, Tan W, et al. The response scale for the intellectual disability module of the WHOQOL: 5-point or 3-point? J Intellect Dev Disabil. 2011 Jun; 55: 537-549.

24. World Health Organization. WHODAS-II Disability Assessment Schedule: Training Manual – A Guide to Administration. Geneva: World Health Organization, 2000.

25. Power MJ, Green AM; WHOQOL-DIS Group. The Attitudes to Disability Scale (ADS): development and psychometric properties. Journal of Intellectual Disability Research. 2010; 54: 860–874.

26. Mattevi BS, Bredemeier J, Fam C, Fleck MPA. Quality of care, quality of life, and attitudes toward disabilities: perspectives from a qualitative focus group study in Porto Alegre, Brazil. Rev Panam Salud Publ. 2012; 31: 188 196.

27. Diener E, Emmons R, Larsen J, Griffin S. The Satisfaction With Life Scale. J Pers Assess, 1985; 49: 71-75.

28. Hutz, CS, Giacomoni, CH. Adaptação da Escala de Satisfação de Diener para o Brasil. Porto Alegre: UFRGS, 1998.

29. Beck, AT, Steer RA, Brown, GK. Manual for the Beck Depression Inventory II. San Antonio, TX: Psychological Corporation, 1996.

30. Gorenstein C, Pang WY, Argimon IL, Werlang BSG. Manual do Inventário de Depressão de Beck – BDI II – Adaptação Brasileira. Casa do Psicólogo. 2011.

31. World Medical Association. Declaration of Helsinki: ethical principles for medical research involving human subjects updated version. World Medical Association, 1964/2008.

32. Cortina, JM. What is coefficient alpha? An examination of theory and applications. J. of Applied Psych, 1993; 78: 98-104.

33. Streiner DL. Starting at the beginning: an introduction to coefficient alpha and internal consistency. J Pers Assess. 2003; 80: 99-103.