Appendix K - Significance testing of the results of the reading tests

(English (Word Find), and Nyanja V.01)

Introduction

The results (see p. 49) estimate the effects on test score of a number of different factors:- urban or rural school (URBAN), sex of child (SEX) grade (YEAR) and whether Nyanja was the home language of the child (HOMEL). When analysing test scores in English the last of these variables (HOMEL) is inappropriate and is not included in the analysis.

Variable codes

Variable YEAR	Categories 3	Codes
YEAR	3	346
SCHOOL	5	C D J K T
SEX	2	B (boy) G (girl)
URBAN	2	R (rural) U (urban)
HOMEL	2	H (home language) X (other)

The data collected for this study has a hierarchical structure, children within classes within schools, and this complicates the form of the analysis required. Each level in the hierarchy has its own degree of random variation which has to be taken into account e.g. a class may have a particularly good (or bad) teacher with the result that the scores of all the children in the class are better (or worse) than expected. This extra variation is over and above the usual child-to-child variation in scores. When comparing grades, i.e. years 3, 4 and 6, the extra variation should be estimated and taken into account, otherwise the differences between the years will be found to be more significant than they really should be. Similarly at the school level, when comparing urban and rural schools the variation between schools needs to be used. The procedure MIXED in the computer package SAS is designed to deal with this type of data and has been used to produce the analyses.

Three sets of results are provided for each analysis:-

1) estimates of variance parameters - these are the estimated values for the random sources of variation described above, child, class and school.

2) estimates of fixed effects - these are the estimates of the effects of the factors listed above (year, sex, urban/rural). Since the analysis is concerned with differences due to the different categories of the factors then one category is taken as the baseline (and attributed 0.0000) and the difference between this and all other categories is presented (eg year 6 is attributed 0.0000 compared with year 4 which gets -16.8213, ie nearly 17 points less than Year 6). In addition to the estimated effects an overall test of whether the factor has a significant effect is given.

3) least squares means - these are adjusted mean scores for each of the factors examined. Adjustment is made for all other factors in the analysis. Thus the least squares means for urban and rural schools are the scores to be expected in such schools if each class had the same number of pupils and equal numbers of boys and girls.

Significance Tests: Findings

1. Sex differences

Overall there is little evidence of differing achievement by the two sexes. There were no statistically significant sex differences.

2. Urban/rural differences

The estimated differences in score between urban and rural schools is in most cases large (7.2 points for English) but none of these estimates is significant. The problem is the relatively small number of schools involved in this study. It is possible that real differences between urban and rural schools do exist but a larger sample of schools (not pupils) would be needed to confirm this.

3. Year differences

Estimated differences between years 3, 4 and 6 are large and strongly significant, indicating a progressive improvement in ability with age for English and Nyanja.

4. Language differences

Nyanja home language pupils doing the Nyanja test do not seem to have a significant advantage. However, the difference is in the expected direction (ie Nyanja home language speakers perform slightly better overall in the Nyanja test than do non-Nyanja home language children).

English test scores.

Variance Parameter Estimates

Parameter	Ratio	Estimate	Std Error	Z	P
SCHOOL	0.5605	42.3909	37.0318	1.14	0.2523
CLASS	0.0857	6.4888	4.5651	1.42	0.1552
Residual	1.0000	75.6285	5.1214	14.77	0.0000

Estimates for Fixed Effects

Parameter	Estimate	Std Error	DF	T Value	P
INTERCEPT	32.8958	4.9055	436	6.71	0.0000
YEAR 3	-23.4253	1.9230	436	-12.18	0.0000
YEAR 4	-16.8213	1.9079	436	-8.82	0.0000
YEAR 6	0.0000
SEX B	-1.2056	0.8300	436	-1.45	0.1471
SEX G	0.0000
URBAN R	-7.2126	6.1526	436	-1.17	0.2417
URBAN U	0.0000

Tests of Fixed Effects

Source	NDF	DDF	Type III F	Pr>F
YEAR	2	8	79.39	0.0000
SEX	1	436	2.11	0.1471
URBAN	1	3	1.37	0.3257

Least Squares Means

Level	LSMEAN	Std Error	DF
YEAR 3	5.2613	3.2759	436
YEAR 4	11.8654	3.2685	436
YEAR 6	28.6867	3.2644	436
SEX B	14.6683	3.1048	436
SEX G	15.8740	3.1031	436
URBAN R	11.6648	3.8970	436
URBAN U	18.8774	4.7608	436

Nyanja test scores

Variance Parameter Estimates

Cov Parm	Ratio	Estimate	Std Error	Z Value	Pr > |Z|
SCHOOL	0.00000000	0.00000000
YEAR* SCHOOL	0.29499321	15.58507453	7.42402862	2.10	0.0358
Residual	1.00000000	52.83197777	3.58155772	14.75	0.0000

Estimates for Fixed Effects

Parameter	Estimate	Std Error	DF	T Value	P
INTERCEPT	20.0530	2.3507	435	8.53	0.0000
YEAR 3	-18.2916	2.6489	435	-6.91	0.0000
YEAR 4	-13.8498	2.6392	435	-5.25	0.0000
YEAR 6	0.0000
SEX B	-0.1826	0.6959	435	-0.26	0.7931
SEX G	0.0000
URBAN R	1.7541	2.2784	435	0.77	0.4418
URBAN U	0.0000
HOMEL H	0.4222	1.0732	435	0.39	0.6942
HOMEL X	0.0000

Tests of Fixed Effects

Source	NDF	DDF	Type III F	Pr>F
YEAR	2	8	26.04	0.0003
SEX	1	435	0.07	0.7931
URBAN	1	3	0.59	0.4974
HOMEL	1	435	0.15	0.6942

Least Squares Means

Level	LSMEAN	Std Error	DF
YEAR 3	2.75830942	1.90838830	435
YEAR 4	7.20006093	1.89271955	435
YEAR 6	21.04991977	1.88954775	435
SEX B	10.24478534	1.18049684	435
SEX G	10.42740809	1.17133003	435
URBAN R	11.21316059	1.49854693	435
URBAN U	9.45903283	1.69511988	435
HOMEL H	10.54723965	1.14218611	435
HOMEL X	10.12495377	1.33969706	435