Generalizability theory (G-Theory).ppt

Testing 05 Reliability Errors Reliability Errors in the test cause unreliability. The fewer the errors, the more reliable the test Sources of errors: Obvious: poor health, fatigue, lack of interest Less obvious: facets discussed in Fig. 5.3 Reliability Validity Reliability is a necessary condition for validity. Reliability validity are complementary aspects of the measurement. Reliability: How much of the performance is due to measurement errors, or to factors other than the language ability we want to measure. Validity: How much of the performance is due to the language ability we want to measure. Reliability Measurement Reliability measurement includes: logical analysis and empirical research, i.e. identify sources of errors and estimate the magnitude of their effects on the scores. Logical Analysis Example of identification of source of errors: Topic in an oral interview: business negotiation Source of error: if we want to measure the test taker’s ability of general topics. Indicator of the ability: if we want to the test taker’s ability of business English. Empirical Research Procedures are usually complex. Three kinds of theories Classical true score theory (CTS) Generalizability theory (G-Theory) Item Response Theory (IRT) Factors on Test Scores Characteristics of factors general vs. specific lasting vs. temporary systematic vs. unsystematic Variance Standard Deviation s: standard deviation of the sample σ: standard deviation of the population s2: variance of the sample σ2: variance of the population s=√∑(X-Xˉ)2/n-1 where X: individual score Xˉ: mean score n: number of students Correlation Coefficient (相关系数) Covariance (COV): two variables, X and Y, vary together. COV(X,Y)=1/(n-1)∑(Xi－Xˉ)(Yi－Yˉ) Correlation Coefficient (Pearson Product-moment Correlation Coefficient 皮尔逊积差相关系数) r(x,y)=COV(x,y)/sxsy r(x,y)= 1/(n-1)∑(Xi－Xˉ)(Yi－Yˉ)/ sxsy Correlation Coefficient Where n: number of items Xi: individual score of the first half Xˉ: mean of the scores