Student’s paper on the Lanarkshire experiment would be remembered if it had stopped at its diagnosis. It did not. “This note would be incomplete without some constructive proposals in case it should be considered necessary to do further work upon the subject,” he wrote, and the two pages that follow are a design for the experiment that should have been run, and a second design for one that would have been better and cheaper. Both were taken up, in part, within a few years. This piece gives the proposals as he wrote them, what Ethel Elderton could and could not do with the cards, how the Milk Nutrition Committee ran the experiment again, and how a statistician of our own day reads the whole affair.

The large experiment, done properly

“If it should be proposed to repeat the experiment on the same spectacular scale,” the first rule was pairs. “The ‘controls’ and ‘feeders’ should be chosen by the teachers in pairs of the same age group and sex, and as similar in height, weight and especially physical condition (i.e. well or ill nourished) as possible, and divided into ‘controls’ and ‘feeders’ by tossing a coin for each pair. Then each pair should be considered to be a unit and the gain in weight and height by the ‘feeder’ over his own ‘control’ should also be considered as a unit for the purpose of determining the error of the gain in weight or height. In this way the error will almost certainly be smaller, perhaps very much smaller, than if calculated from the means of ‘feeders’ and ‘controls.’” Note what the teachers are asked to do: exactly what they had done in 1930, judge which children were well or ill nourished, but before the coin and not after it, so that their judgement sorts the pairs and the coin sorts the treatment. “If in addition the social status of each pair be noted (well to do, medium, poorly nourished or some such scale) further useful information will be available for comparing pasteurized and raw ‘feeders.’”

If that was too difficult, “a perfectly good comparison can be made by adhering to the original plan of the 1930 experiment and drawing lots to decide which should be ‘controls’ and which ‘feeders’ (this is better than an alphabetical arrangement), but the error of the comparison is likely to be larger than in the plan outlined above.” The second rule was that “if it is at all possible each school should supply an equal number of raw and pasteurised ‘feeders,’ again by selection of similar children followed by coin tossing, but I fear that this is a counsel of perfection.” The third: “Some effort should be made to estimate the weight of clothes worn by the children at the beginning and end of the experiment: possibly the time of year could be chosen so that there would be little change in this respect.”

The small experiment

The second proposal starts from a concession. “If it be agreed that milk is an advantageous addition to children’s diet — and I doubt whether any one will combat that view — and that the difference between raw and pasteurised milk is the matter to be investigated, it would be possible to obtain much greater certainty at an expenditure of perhaps 1–2 per cent. of the money and less than 5 per cent. of the trouble.” The footnote to “money” reads: “This is a serious consideration: the Lanarkshire experiment cost about £7500.”

The arithmetic is in the text. “For among 20,000 children there will be numerous pairs of twins; exactly how many it is not easy to say owing to the differential death rate, but, since there is about one pair of twins in 90 births, one might hope to get at least 160 pairs in 20,000 children. But as a matter of fact the 20,000 children were not all the Lanarkshire schools population, and I feel pretty certain that some 200–300 pairs of twins would be available for the purpose of the experiment. Of 200 pairs some 50 would be ‘identicals’ and of course of the same sex, while half the remainder would be non-identical twins of the same sex. Now identical twins are probably better experimental material than is available for feeding experiments carried out on any other mammals, and the error of the comparison between them may be relied upon to be so small that 50 pairs of these would give more reliable results than the 20,000 with which we have been dealing.”

The procedure: “The proposal is then to experiment on all pairs of twins of the same sex available, noting whether each pair is so similar that they are probably ‘identicals’ or whether they are dissimilar. ‘Feed’ one of each pair on raw and the other on pasteurised milk, deciding in each case which is to take raw milk by the toss of a coin. Take weekly measurements and weigh without clothes.” And a precaution that tells you he had thought about children: “Some way of distinguishing the children from each other is necessary or the mischievous ones will play tricks. The obvious method is to take finger-prints, but as this is identified with crime in some people’s minds, it may be necessary to make a different indelible mark on a fingernail of each, which will grow off after the experiment is over.” With so few children “further information about the dietetic habits and social position of the children could be collected and would doubtless prove invaluable,” and “the comparative variation in the effect in ‘identical’ twins and in ‘unlike’ twins should furnish useful information on the relative importance of ‘Nature and Nurture.’”

What Elderton could do

Nobody ran the twins experiment. What the Department of Health for Scotland did was lend the cards to Karl Pearson, and Elderton’s paper of 1933 begins by saying why she did what she did: “‘Student’ suggested that the experiment should be carried out on identical twins and if identical twins were more numerous and could with ease and certainty be discriminated from other twins they would be ideal subjects for such an experiment. In the absence of such data Professor Pearson suggested that, from the original cards, enough children of each class — controls, raw milk feeders, pasteurised milk feeders — could be found and paired who would have the same initial height and weight within reasonable limits.” She found them, some 340 pairs of each sex for each milk against the controls, and compared them as Student had said pairs should be compared. Her last conclusion was that she “heartily” endorsed his plan, “it is the method I have tried in this paper, but the weakness in my work is that the undergrown and overgrown children have been omitted, and that no knowledge of the general physical condition of the children was available.” Pairing after the fact could match two numbers on a card. It could not undo what a teacher had seen in a child’s face in February 1930, and it threw away every child too small or too large to have a match, which is to say the children the experiment was for. Her figures are in the companion piece “The twelve tables.”

The experiment run again

The Milk in Schools Scheme began in October 1934, a third of a pint for a halfpenny in every elementary school in England and Wales that would take it, paid for by the Milk Marketing Board and the Treasury. Lord Astor, who chaired the Board’s advisory committee on school milk, had proposed that the scheme be tested as it ran, and a Milk Nutrition Committee was formed under him with the Ministry of Health’s Dr H. E. Magee as secretary. Its expert sub-committee was chaired by Sir John Boyd Orr, who had run the Scottish test of 1927 and 1928, and its members included Sir Edward Mellanby, the secretary of the Medical Research Council, Professor H. D. Kay, the director at Shinfield, and, for Scotland, Dr P. L. McKinlay, the co-author of the Lanarkshire report, who was by then Superintendent of Statistics at Register House. The historian Peter Atkins, who read the committee’s files in the National Archives, found that the original plan had been for 5,000 children in London, a depressed area, a rural district and a district in Scotland, that the cost grew from an expected 4,000 pounds to 15,000, and that “from the outset there was a concern that the scientific planning of the survey should allow for a better sampling frame than had been available in Lanarkshire, no doubt because of the painful memory of the criticisms that had come from statisticians at that time.”

The committee’s final report, published by the National Institute for Research in Dairying at Shinfield in 1939, describes what was done. The objects were “to compare the nutritive values of raw and pasteurised milk” and “to ascertain by tests on school children the growth and health promoting properties of supplements of 1/3 or 2/3 pint of milk. These amounts are the same as those which figure in the Milk-in-Schools Scheme; the investigation was therefore a practical trial of the scheme.” It was done in five places, Luton, Wolverhampton, Burton-on-Trent, Renfrewshire and Huddersfield, with laboratory work on rats and a feeding trial on calves beside it. There were 8,435 children aged five to fourteen, about equally divided between the sexes, and every one of them was in a school that had all four treatments.

The allocation is described in a paragraph that reads as if written with the Lanarkshire report open on the desk. “At the commencement of the investigation the children in each school were randomly divided into four feeding groups, the procedure being as follows. The children were taken by classes and No. 1 child was placed in the biscuit group, No. 2 in the 1/3 pint pasteurised, No. 3 in the 2/3 pint pasteurised, No. 4 in the 2/3 pint raw, No. 5 in the biscuit, No. 6 in the 1/3 pint pasteurised group, and so on. The classification was made before the children were medically examined and no account was taken of their physical condition or social or economic status.” It was counting off, not a coin; a statistician of Fisher’s school would not have called it random. But it was done before anyone had looked at the children, and nothing was substituted afterwards. The biscuit group was the control, the biscuit’s “nutritive value was negligible” and it “served merely to satisfy the children.” Four specially trained doctors weighed, measured and examined every child at the start, at the end and twice between; the teachers graded each child’s schoolwork at the same four points. Of the 8,435, 6,097 were present at all four examinations and 695 of the oldest at the first two; the remaining 1,643 were dropped, and “statistical tests showed that the randomness of the sampling was not upset by the absences.”

The results were that every milk ration produced “in both sexes, irrespective of age, increments in height, weight and chest circumference greater than those which occurred in the control or biscuit groups,” that two-thirds of a pint did more than a third, that the gains in weight came mostly in the first months, and that “no constant differences could be detected between the growth-promoting effects of 2/3 pint pasteurised and 2/3 pint raw milk in either height, weight or chest circumference.” The one oddity ran the other way from Bartlett’s: among boys of eleven to fourteen the raw-milk group gained less in a test of muscular strength than any other group, “but the evidence is not regarded as conclusive.” The committee’s practical conclusion was that “under purely practical conditions, there is no difference in nutritive value between them,” while allowing that a test on babies fed on nothing else, which it had planned and could not staff, might have found one. Of 1930 it wrote only that in “a four-month test on about 12,000 children” Leighton and McKinlay “were unable to detect any significant difference in the average rate of growth of the two groups,” and that “the present test confirmed these findings.”

A modern reading

Stephen Senn, a statistician who has spent a career on the design of clinical trials, went back to Student’s paper in 2022 at the suggestion of the epidemiologist George Davey Smith. His description of the Lanarkshire design is a phrase from a vocabulary that did not exist in 1930: “a cluster allocated incomplete block design,” schools being the clusters, the two milks the treatments that no school received together, and the control the one treatment every school had. He compares it to a trial of two arthritis drugs against a third in 2004, with 18,244 patients in 849 centres, in which, for practical reasons, some centres compared one pair of treatments and some the other, and in which the analysts kept the two sub-studies apart as a term in the model and compared each drug only with the control that had sat beside it.

That, he argues, is where Leighton and McKinlay went most wrong, and where Student, almost in passing, was most right. By pooling the controls of the raw-milk schools with those of the pasteurised-milk schools they threw away the comparison the design had actually made, feeder against control within a school, and replaced it with one across schools that the design had never made fair. The right way to compare the two milks was the double contrast: raw minus its own controls, pasteurised minus its own controls, and the difference of the two. Senn’s appendix works out that with 67 schools and about 135 children of each kind in each, almost any variation between schools makes the pooled comparison worse than the local one, and that the report’s probable errors, however they were computed, were almost certainly too small. Elderton, he notes, “appears to have made no use of the fact that the data were clustered in schools: not even to the extent of resolving the mystery of how many there were of each type.”

He also set out, for anyone with the time, how he would analyse the cards: boys and girls separately; sub-study as a fixed factor with two levels, because the milks were not allocated to schools at random; school as a fixed factor; the February measurement as a covariate, with age as a continuous variable beside it. George Davey Smith told him that the cards still existed in 1988. His conclusion is in two sentences. “By the standards of modern randomised trials the study is inadequate. It is not clear, to me, however, that the data would be inferior to what one might expect from the sort of observational study that is frequently used to draw epidemiological conclusions.” And he adds a thought the Lanarkshire teachers would have understood: that many observational studies may hold hidden clusters of their own, so that their standard errors are too small in just the way the report’s were, and nobody has tossed a coin to find out.

Sources

• “Student,” “The Lanarkshire Milk Experiment,” Biometrika 23, December 1931, pp. 404–406, for the proposals. gwern.net
• Ethel M. Elderton, “The Lanarkshire Milk Experiment,” Annals of Eugenics 5, October 1933, pp. 326–338. gwern.net
• Milk Nutrition Committee, Milk and Nutrition: New Experiments Reported to the Milk Nutrition Committee, Part IV: The Effects of Dietary Supplements of Pasteurised and Raw Milk on the Growth and Health of School Children (Final Report), National Institute for Research in Dairying, Shinfield, 1939: the committee, the objects, the allocation by classes, the numbers, the conclusions and the paragraph on Lanarkshire. Wellcome Collection’s scan, Public Domain Mark, read by optical character recognition. wellcomecollection.org
• Peter J. Atkins, “Fattening children or fattening farmers? School milk in Britain, 1921–1941,” Economic History Review 58, 2005, pp. 57–78, for the committee’s origins, the five towns, the cost and the “painful memory.” pjatkins.webspace.durham.ac.uk
• Stephen Senn, “Student and the Lanarkshire milk experiment,” European Journal of Epidemiology 38, 2023, pp. 1–10, for the modern reading, the TARGET comparison, the recommended analysis and the cards in 1988. PMC9867657