William Sealy Gosset’s paper on the Lanarkshire experiment is nine pages of Biometrika, volume 23, pages 398 to 406, published in December 1931 under the name he had used since 1908, “Student.” It is read today mostly in summary, as the paper that found the fatal mistake, and the summary is right; but the paper is also a model of how to tell a government department that its largest experiment has failed without once raising the voice, and it has a shape worth following from beginning to end.

How it came to be written

The chronology is in chapter five of Egon Pearson’s biography of Gosset, as Stephen Senn summarises it. Fisher and Bartlett’s letter appeared in Nature on April 18, 1931. On July 14 Gosset wrote to Karl Pearson, the editor of Biometrika, that he was examining the Department of Health’s report together with the Nature note. On July 23 he sent Pearson a draft. Pearson returned detailed comments on July 26; Gosset answered them on July 30; the corrected proofs went back on August 18; the paper was in the December number. Senn’s comment is that “one can only marvel at the speed with which our scientific forbears managed these things in an era before electronic communication.” The typescript letters, with the Nature reprint enclosed and a copy of one of Pearson’s replies, are in the Pearson papers at University College London, where the catalogue also notes that they discuss “the ‘z’ test paper,” the other business between the two men that summer.

Gosset was fifty-five, the head experimental brewer at Guinness in Dublin, and the brewery’s rule was that its brewers published under pseudonyms and did not mention beer. He had been Pearson’s student for two terms in 1906 and 1907 and had sent his papers to Biometrika ever since; he had also corresponded with Fisher since 1912, and had been, as much as anyone, the man who brought the two camps of British statistics into speaking terms. The Lanarkshire paper is the one place where he reviewed a piece of Fisher’s in print.

The opening

The first page sets out the experiment in four sentences and then does something that every later critic has copied: “It need hardly be said that to carry out an experiment of this magnitude successfully requires organisation of no mean order and the whole business of distribution of milk and of measurement of growth reflects great credit on all those concerned.” Then the hinge. “It may therefore seem ungracious to be wise after the event and to suggest that had the arrangement of the experiment been slightly different the results would have carried greater weight, but what follows is written not so much in criticism of what was done in 1930 as in the hope that in any further work full advantage may be taken of the light which may be thrown on the best methods of arrangement by the defects as well as by the merits of the Lanarkshire experiment.”

The first defect: one milk to a school

“The 20,000 children were chosen in 67 schools, not more than 400 nor less than 200 being chosen in any one school, and of these half were assigned as ‘feeders’ and half as ‘controls,’ some schools were provided with raw milk and the others with pasteurized milk, no school getting both.” He grants the reason at once: “This was probably necessary for administrative reasons, owing to the difficulty of being sure that each of as many as 200 children gets the right kind of milk every day if there were a possibility of their getting either of the two.” But “this does introduce the possibility that the raw and pasteurized milks were tested on groups of children which were not strictly comparable.” He will come back to it.

The second defect: the substitution

He quotes the report’s account of the head teachers’ selection, “the actual method of selection being important,” down to the ballot and the alphabet. “So far so good, but after invoking the goddess of chance they unfortunately wavered in their adherence to her for we read: ‘In any particular school where there was any group to which these methods had given an undue proportion of well fed or ill nourished children, others were substituted in order to obtain a more level selection.’ This is just the sort of after-thought that most of us have now and again and which is apt to spoil the best laid plans. In this case it was a fatal mistake, for in consequence the controls were, as pointed out in the Report, definitely superior both in weight and height to the ‘feeders’ by an amount equivalent to about 3 months’ growth in weight and 4 months’ growth in height.”

Then the hypothesis that has carried the paper into the textbooks. “Presumably this discrimination in height and weight was not made deliberately, but it would seem probable that the teachers, swayed by the very human feeling that the poorer children needed the milk more than the comparatively well to do, must have unconsciously made too large a substitution of the ill-nourished among the ‘feeders’ and too few among the ‘controls’ and that this unconscious selection affected, secondarily, both measurements.”

The third defect: the clothes

“Thirdly, it was clearly impossible to weigh such large numbers of children without impedimenta. They were weighed in their indoor clothes, with certain obvious precautions, and the difference in weight between their February garb and their somewhat lighter clothing in June is thus necessarily subtracted from their actual increase in weight between the beginning and end of the experiment.” Had the selection been random “this fact, as pointed out in the Report, would have mattered little, both classes would have been affected equally, but since the selection was probably affected by poverty it is reasonable to suppose that the ‘feeders’ would lose less weight from this cause than the ‘controls.’ It is therefore not surprising to find that the gain in weight of ‘feeders’ over ‘controls,’ which includes this constant error, was more marked, relatively to their growth rate, than was their gain in height, which was fortunately not similarly affected.”

The fourth defect: the pooled controls

“Fourthly, the ‘controls’ from those schools which took raw milk were bulked with those from the schools which took pasteurized milk. Now with only 67 schools, at best 33 against 34, in a district so heterogeneous both racially and socially, it is quite possible that there was a difference between the averages of the pupils at 33 schools and those of the pupils at another 34 schools both in the original measurements and in the rate of growth during the experiment. In that case the average ‘control’ could not be used appropriately to compare with either the ‘raw’ group or the ‘pasteurized’ group.” The selection of controls “can hardly have been carried out in a uniform manner in different schools,” which made it worse. And then the sentence that would wait ninety years: “Fortunately it would still be possible to correct this, for the figures for the different schools must still be available in the archives.”

The diagrams

Four diagrams fill pages 400 and 402. Each plots the average height or weight of the controls, the raw-milk feeders and the pasteurised feeders against age, with a point for February and, four months to the right, a point for June, for each of the seven age groups, and the three groups’ lines drawn through them. In the height diagrams the June points sit on the growth line: “not any of the points diverge very much from their appropriate line of growth whether ‘controls,’ ‘raws’ or ‘pasteuriseds.’” In the weight diagrams, “there is, after the first two ages, a very decided dip, especially in the later ages. The weights at the end of the experiment are too low. This might be accounted for by a tendency in older children to grow normally in height and subnormally in weight during the spring, but I think it much more likely that older children wear about 1 lb. more clothes in February than they do in June, while in the case of younger children a more limited wardrobe permits of fewer discards.” He also notices that the five-year-olds, few in number, sit off the line, “either because only the older (or larger) children are sent to school between 5 and 6 or because the teachers did not think that the smaller children would be able to play their part.”

Then he turns to the report’s defence, the correlations of Table 5. “Now this might have been a perfectly good argument had the height and weight been selected directly, but if, as I have indicated was very likely the case, the selection was made according to some unconscious scale of well being, then it is surely natural to suppose that the relatively ill nourished ‘feeders’ would benefit more than their more fortunate school mates, the ‘controls,’ would have done by the extra ¾ pint of milk per day.”

The three conclusions

“That being so how are we to regard the conclusions of the Report.”

The first, that milk increased growth in height and weight, “was probably true; the average increase for boys’ and girls’ heights was 8 per cent. and 10 per cent. over ‘controls’ and for boys’ and girls’ weights was 30 per cent. and 45 per cent., respectively, and though, as pointed out, the figures for weights were wholly unreliable it is likely enough that a substantial part of the difference in height and a small part of that in weight were really due to the good effect of the milk. The conclusion is, however, shifted from the sure ground of scientific inference to the less satisfactory foundation of mere authority and guesswork by the fact that the ‘controls’ and ‘feeders’ were not randomly selected.”

The second, that the younger children had not gained more than the older, he thought the report had understated against itself, and he gave his table from Bartlett’s regrouped figures to show that in weight the older children had apparently gained far more than the younger while in height the younger boys had done slightly better. “In the light of previous criticism, however, we must be content to say that apparently the differential shedding of clothes between the ‘feeders’ and the more fortunate ‘controls’ is more marked with older children (and possibly with girls than with boys), and that there is some probability that younger boys gain in height more than older.”

The third, that the two milks were equal, “has been challenged by Capt. Bartlett, and by Dr Fisher and Capt. Bartlett, who conclude that there is definite evidence of the superiority of raw over pasteurised milk in both height and weight. Even they however point out that the raw and pasteurized milk were not supplied to the same schools, and their conclusion amounts to saying: ‘If the groups of children taking raw and pasteurized milk respectively were random samples from the same population, the observed differences would be decisively in favour of the raw milk.’ Unfortunately they were not random samples from the same population: they were selected samples from populations which may have been different, and moreover the ‘controls’ with which they were compared were not appropriate to either group; and so — again it is a matter of guess and authority — I would be very chary of drawing any conclusion from these small biased differences. That is not to say that there is no difference between the effect of raw and pasteurised milk — personally I believe that there is and that it is in favour of raw milk — but that this experiment, in spite of all the good work which was put into it, just lacked the essential condition of randomness which would have enabled us to prove the fact.”

That is the whole of what the paper says about Fisher: that he and Bartlett had drawn a conclusion the design could not support, and that Student privately agreed with the conclusion. The two men’s long argument about whether field plots should be laid out at random or in balanced patterns was still to come; Student’s last paper, in 1938, was on the other side of it from Fisher. On Lanarkshire, where the question was not how to arrange a fair comparison but whether one had been made at all, there was no distance between them.

The summary

The paper ends with a summary that is the shortest fair account of the experiment anyone has written. “The Lanarkshire experiment devised to find out the value of giving a regular supply of milk to children, though planned on the grand scale, organised in a thoroughly business-like manner and carried through with the devoted assistance of a large team of teachers, nurses and doctors, failed to produce a valid estimate of the advantage of giving milk to children and of the difference between raw and pasteurized milk. This was due to an attempt to improve on a random selection of the controls which in fact selected as controls children who were on the average taller and heavier than those who were given milk. The hypothesis is advanced that this was due not to a selection of the shorter, lighter children as such to take the milk, but to an unconscious bias leading the teachers to pick out for this purpose the needier children whom the milk would be most likely to benefit. This hypothesis is supported by the fact that while the advantage derived from the milk was only 8–10 per cent. of the gain in height, without much variation for age, it was 30–45 per cent. of the gain in weight, varying from 9–13 per cent. in the younger children (who do not seem to have shed much clothing in the summer) up to 73–78 per cent. in the older children — who obviously did.”

What he proposed instead, a large experiment on random lines and a small one on pairs of twins, is in the companion piece “Fifty pairs of twins.”

What the paper did not do

It did not use the cards, which it had not seen, and it did not ask for them; it asked that the school-by-school figures be looked at, and nobody did. It did not recompute anything from scratch; its one table was built on Bartlett’s regrouping, with a warning that the probable errors were “subject, as his are, to his having interpreted the methods of the original Report correctly.” It did not say that the experiment had shown nothing. It said, three times in different words, that the experiment had probably shown what it claimed and could not prove it, and that the difference between those two things was the whole of the matter. Senn’s reading, ninety years on, is that the points it made “are perceptive and still valuable,” and that the one it made in passing, about the pooled controls, is the one a modern analyst would put first.

Sources

• “Student,” “The Lanarkshire Milk Experiment,” Biometrika 23 (3–4), December 1931, pp. 398–406. Every quotation above. gwern.net, doi:10.2307/2332424
• Stephen Senn, “Student and the Lanarkshire milk experiment,” European Journal of Epidemiology 38, 2023, pp. 1–10, for the chronology of the letters from E. S. Pearson, Student: A Statistical Biography of William Sealy Gosset (Clarendon Press, 1990), chapter five, which was not reached. PMC9867657
• UCL Special Collections, Pearson Papers, PEARSON/7/180/1, “Letters from William Gosset (‘Student’),” 1931. archives.ucl.ac.uk
• R. A. Fisher and S. Bartlett, “Pasteurised and Raw Milk,” Nature 127, April 18, 1931, pp. 591–592; only the opening paragraph was read. doi:10.1038/127591c0
• William Sealy Gosset, Wikipedia, for the brewery’s rule, the terms with Pearson in 1906 and 1907, and the correspondence with Fisher. en.wikipedia.org