The survival
error in social science
Matt Berkley
Draft released as “People, words
and numbers”, 5 February 2006
Retitled 13 February 2007
Summary
This document takes as a starting
point an error in the theory of social science: failure to take account
of survival rates when aggregating outcomes.
Several Millennium Goal
indicators, other things being equal, show “better” progress if people die
earlier.
This logical truth may indicate a
need for reform in the education of social scientists in several
disciplines.
The document goes on to examine the use of some words in large-scale social
science.
Part
1: A suggestion on the aims of international development
Email to Professor Jeffrey
Sachs,
chair of the World Health Organisation Commission on
Macroeconomics and Health
The text
of the email of 11 April 2001 begins below at “Subj:
Economics of Survival”.
Quotations:
“Here
are some theoretical points about mortality rates and the International Development
Goals....
To me, no outcome measure is humane unless it takes into account what happened
to people who started the period but didn't make it to the end. If
the poorest die, the average income of those alive at the end of the period
will be higher....
My
suggestion is this: For any outcome measure....account needs to be taken
of those within the relevant group who did not achieve the target, whether
through death or any other path. ”
Introductory comments
In mid-2000 I became aware of a
common error in economics: to claim to have aggregated outcomes for
individuals without considering survival rates. Strictly
speaking, the tradition in macroeconomics has been to confuse “the average
rise” with “the rise in the average”, and “poverty alleviation” with “poverty
reduction”. If some people die, the figures look
better. I raised this problem with several senior academics and in
2001 with officials in the UK Government, the OECD and the World Bank.
In April 2001 my reading of material
from the World Health Organisation Commission on Macroeconomics and Health had
not revealed any attention to this problem. I wrote to Professor
Sachs explaining the problem.
In
December 2001 the Commission’s Report and at least one of its background papers
did make the distinction in relation to national economic
statistics. As an approach to aggregating benefits to people, the
idea of looking at trends for people is clearly right, while the idea of
looking at trends for statistics about populations in the abstract is clearly
wrong. The Commission chose the former.
In
relation to survival rates, other questions arise relating to:
a) how in future governments might agree clearer goals;
b)
the need or otherwise for revisions of existing
internationally agreed goals, targets and indicators;
c)
how scientists might accurately describe their findings where
data and/or demonstrably reliable inferences on relevant survival rates are not
available;
d)
how various social science disciplines should be reformed in
this respect;
e)
the implications of the existence of the error for the
education of scientists, policy-makers or the public.
The Commission sensibly did not
set an absolute value on years of life.
However, another general problem
in the Commission’s approach was that its ascriptions of benefit, like many
claims from economists, were strictly speaking based on a
confusion between income and profit. Neither GDP per
capita nor income adjusted by life length measures prosperity. One
reason is that neither of those measures need for
expenditure.
These two areas
- the cost of dying and the cost of living - form the
basis of some remarks about Millennium Goal indicators in part 2 of this
document. Other remarks relate to aspects of demography other than
life length.
Some
thoughts have recently occurred to me in relation to the Commission’s assigning
of monetary values to years of life. These thoughts may turn out to
be fundamentally mistaken.
The
Commission estimated that if certain inputs were made, there would be at least
$186 billion of savings in income and increased life for people.
One
thought runs along these lines:
“Other
things being equal - if the Commission’s
idea is accepted that financial values can be given in a meaningful sense to
years of life, which is not necessarily uncontroversial - the
figure of $186 billion of savings seems far too low, for a reason not connected
to the value of years, but to the value of currency.
Suppose
A. Two
people die early, in the
B. The
US citizen was living on $1000 a year, and the Indian citizen on $333 a
year.
C.
Because of exchange rates, they could both afford the same (and everything else
is equal).
D. They
each lose 30 years of life.
It seems
to me that the logic of the Commission would be that the
The
value of wages lost by a person who dies early might perhaps reasonably be
given in real money (though on its own it is perhaps not of great
interest).
The
question of assigning a value to the lost years is a different question.
Without prejudging whether this is a sensible question or not, we might think
about what the approach would be if we decided that it was a sensible
question. At the minimum, it would seem sensible to use units which
relate to comparable purchasing power in different
countries. Otherwise, the value of life would depend on exchange
rates.
In
practice, international purchasing power estimates for people living on such
amounts are not generally available.
If we ignore that problem, and for
the sake of argument assume (perhaps unreasonably) that “purchasing power
parity” estimates for whole countries are roughly right and appropriate for
people’s basic needs, then other things being equal, if the rest of the
Commission’s logic is accepted, the minimum estimate for savings from inputs
into health care should have been over $500 billion rather than $186 billion”.
I do not mean here to imply that
life can be given a monetary value. I do mean to imply that it is
sensible to value people’s lives in a similar way to one’s own life in the
absence of a convincing argument otherwise.
The general
area of exchange rates spurs me on to another thought. Where in economic
theory is the following aspect of real life modelled?
Person A
has an excess of the money equivalent of 1 bowl of rice beyond their needs.
Persons
B and C lack 1 bowl of rice each.
Person A
lives in a country where both wages and prices are double those where B and C
live.
So if A
gives up 1 extra bowl, B and C each gain an extra bowl.
A second
question along these lines might be, how would a theory of utility cope with such
a situation?
The
point of that part was to highlight the fact that money of people in rich
countries is worth more to people in poor countries not only because of
a)
diminishing returns (an extra bowl of rice is worth more to a starving person
than to an obese person)
but also because of
b) the fact that the money buys the person in the poor country more than the
person in the rich country can buy with it. (a bowl of rice is worth more
than the half bowl which the same money would buy in the rich country).
I think
this may have some bearing on discussions about the morality of -
to the extent that moral discussions make sense - both
international aid policies and policy decisions which affect populations of
countries where money is worth widely differing amounts.
Subj: Economics
of survival
11 April
2001
Dear Professor Sachs
I wonder if this of interest to the WHO Commission.
Here are some theoretical points about mortality rates and the International
Development Goals, and then some practical points which are less simplistic.
Among the International Development Goals, progress has been faster on reducing
the proportion of people in extreme poverty, and slower on child mortality.
The question no-one seems to be asking is this: Is the proportion of
poor people getting smaller partly because child mortality is worse than
we hoped?
Most of the goals [later correction: “several indicators”
- 21 indicators for the Millennium Goals] are
susceptible to the problem that if the worst-off die, we are closer to the
target.
There are good grounds for thinking that the child mortality goal being on
track provides a statistical safeguard among the goals - if this
goal goes according to plan, it ensures that we do not get a false impression
of progress towards the other goals simply through high death rates among the
poorest.
Grounds for believing this include the following. Firstly, child
mortality is concentrated among the poorest, so an improvement in the total may
well reflect improvement among the target groups. Secondly, the child
mortality rate is believed to give an indication of the rate of early deaths
among adults Policies which reduce child mortality are likely to also
reduce early mortality among adults.
If all this is true, then the closer we are to the required rate of progress on
child mortality, the more poor people there are. Slow progress on
child mortality (as now) makes for fewer poor people, so the poverty goal looks
closer - simply because fewer poor people are alive, not because
more of the survivors have raised their living standards.
My hunch is that slower progress on total child mortality means much slower
progress on child mortality among the poorest. If this is so, then
the effect is stronger.
To me, no outcome measure is humane unless it takes into account what happened
to people who started the period but didn't make it to the end. If
the poorest die, the average income of those alive at the end of the period
will be higher than the average when the group included the poorest, even if
none of the survivors' income has gone up. It even looks higher, if
enough of the poorest die, when the average among the survivors goes down
somewhat - simply because the poorest are no longer there to pull
the average down.
If we measure the income of those alive in 1995 and then the income of those
alive in 2000, we will not notice the decline in income of someone who died in
1998. The average income of those alive will be exactly the
same as if he had survived and raised his income to the average of the group.
In fact, since most people in poor countries work on the land,
vulnerability is seasonal, and therefore the people who die may have a
declining income for a few weeks or months before they die. This is too
fast for measurements taken every five years.
My suggestion is this: For any outcome measure - reducing
poverty, achieving 100% schooling - account needs to be taken
of those within the relevant group who did not achieve the target, whether
through death or any other path.
[Later note: “Reducing poverty” is strictly speaking not an outcome
measure but an expression of an opinion. If poverty is a lack of
prosperity, and prosperity is subjective, then by definition poverty cannot be
measured.].
In practice, the relationship between child mortality and statistical progress
on the goals would appear to need careful research (see below).
Practical considerations
In real life, there may not be such a clear division between the poorest and
the less-poor. However, DHS data seem to point to assets as
important determinants of child mortality - the lowest 10% can be
far more vulnerable than the next 10% (Bonilla-Chacin
and Hammer, "Life and Death among the Poorest", 1999, revised version
2001 forthcoming, World Bank). There may be a clear division, for
example, in some geographical areas, between the landed and the landless.
In real life, policies which reduce the proportion of people living under $1 a
day may also save the most vulnerable from death. This cannot be assumed,
and may depend on the relative vulnerability of the poorest (see previous
point).
In real life, the poorest may produce more children to replace those who have
died. The total number of poor children could conceivably be the same in 2015
whatever the child mortality rate. But if adults as well as
children die in hard times this is unlikely.
The statistical relationship between mortality and outcome measures can only be
determined by careful research, together with an intimate knowledge of
household behaviour.
Statistical progress on the goals needs to be translated into human terms.
If there is any suspicion that apparent progress on any of the
goals is helped by lack of progress on any of the others, then this is an
argument for tackling the goals that are furthest behind, not the ones that are
furthest ahead.
---------------------
Later notes:
1. A better title for the
email would perhaps have been “statistics and survival”. My suggestion
above to Professor Sachs related to any outcome measure, which implies
“in any social science”. Also, the scope of theoretical
points made in the email concerning economic data was wider than
internationally agreed goals.
2. Perhaps a better hypothesis
than
“the proportion of people may be falling due to child mortality”
might have been something like this:
“The puzzle that
fast reported progress on economic
indicators which are traditionally thought to indicate aggregate outcomes for
people
has co-existed with
slow progress on child mortality
may be partially explicable by
a) the logical truth that the
economic indicators do not in reality aggregate outcomes for people, but for
populations,
combined with
b) a hypothesis that current bad
progress on child survival may be indicating low adult survival as well, among
the target population for the economic indicators (the “poor”).
Part
2: Notes on Millennium Goal indicators
The Millennium Goal indicators
listed below do not take survival rates into account. They are not in
themselves guides to the aggregate progress of individuals. They
risk being mistaken for such.
The points below relate to the
interpretation of country-level trends as well as global trends.
a) Common sense is needed to
consider which indicators may be problematic for which countries and time
periods. For example, indicator 22 on malaria may not be
problematic at all in context: another indicator deals with malaria deaths.
b) The application of common sense
does not necessarily mean applying traditional inferences in social
science. The guiding principle might reasonably be one of caution
in inferring benefits to people without evidence on survival
rates.
c) Indicator 18 on HIV would seem
to be more problematic than that on malaria, given the absence of any indicator
relating to HIV deaths.
d) The text of the letter to
Professor Sachs, and notably the suggestion, deal with a fundamental principle
of social science. A reasonable line of argument might
be this: it is important for social scientists to understand the
principles, so that methods can deal with any reasonably foreseeable
circumstances.
e) The letter to Professor Sachs
and this document do not make specific claims that particular indicators have
shown the wrong trend as a result of mortality differences. Nor do
they claim that any will do so in the future.
f) However, the context is of
falls in life expectancy in some countries during the 1990s. The
effects of AIDS on demographic projections in some African countries have been
large. Reasonably foreseeable circumstances which would affect the
statistics may already have come to pass.
g) It is perhaps worth noting here
a point I made to Professor Frances Stewart of
h) The mortality flaw is a flaw in
the logic of arguments used by social scientists.
i) Two aspects to the mortality flaw are mentioned in the
letter to Professor Sachs.
One is that aggregate outcomes are
not measurable without considering survival rates. Again, common
sense should override this rule where necessary. Exceptions to this rule
are where variations in survival rates have previously been excluded on
reasonable grounds as insignificant or irrelevant. The
point here is that some people’s outcomes are left out.
Another aspect is that differences
between statistics across time, place or circumstance may be caused by
differences in survival rates between people at different levels of the
variable being studied.
A third aspect of the mortality
flaw is that the effects of deaths on people’s happiness may not be limited to
those who die. Parents generally prefer their children to stay
alive. Where children die, the work put into being pregnant
and raising them is wasted. In
j) Note that indicators may also
be affected by birth rates - not just by the effects of birth rates
on age structure, but also by the direct effects of birth rates on population
numbers. It is important to understand the difference between
“there are x million fewer people in a category now” and “x million people went
from one category to another”. More detail is given
below.
This list comprises 21 of the 48 indicators. The descriptions are those provided by the United Nations Statistics Division.
1. Proportion of population below $1 (1993 PPP) per day (World Bank)
2. Poverty gap ratio [incidence x depth of poverty] (World Bank) [* see note by MB below]
3. Share of poorest quintile in national consumption (World Bank) [** see note below]
4. Prevalence of underweight children under five years of age (UNICEF-WHO)
5. Proportion of population below minimum level of dietary energy consumption (FAO)
6. Net enrolment ratio in primary education (UNESCO)
8. Literacy rate of 15-24 year-olds (UNESCO)
15. Proportion of 1 year-old children immunized against measles (UNICEF-WHO)
17. Proportion of births attended by skilled health personnel (UNICEF-WHO)
18. HIV prevalence among pregnant women aged 15-24 years (UNAIDS-WHO-UNICEF)
19. Condom use rate of the contraceptive prevalence rate (UN Population Division)
19a. Condom use at last high-risk sex (UNICEF-WHO)
19b. Percentage of population aged 15-24 years with comprehensive correct knowledge of HIV/AIDS (UNICEF-WHO)
19c. Contraceptive prevalence
rate (UN Population Division)
20. Ratio of school attendance of orphans to school attendance of non-orphans aged 10-14 years (UNICEF-UNAIDS-WHO)
22. Proportion of population in malaria-risk areas using effective malaria prevention and treatment measures (UNICEF-WHO)
29. Proportion of population using solid fuels (WHO)
30. Proportion of population with sustainable access to an improved water source, urban and rural (UNICEF-WHO)
31. Proportion of population with access to improved sanitation, urban and rural (UNICEF-WHO)
32. Proportion of households with access to secure tenure (UN-HABITAT)
45. Unemployment rate of young people aged 15-24 years, each sex and total (ILO)
46. Proportion of population with access to affordable essential drugs on a sustainable basis (WHO)
47. Telephone lines and cellular subscribers per 100 population (ITU)
48. Personal computers in use per 100 population and Internet users per 100 population (ITU)
Source: http://millenniumindicators.un.org/unsd/mi/mi_goals.asp .
* Note on the word “poverty”:
Indicator 2 is described as, “Poverty gap ratio [incidence x depth of poverty]”.
It is not clear why the United Nations Statistics Division used the word “poverty” here - or why indicator 1 might be thought of as indicating poverty, even in the absence of problems relating to life length. The data are mostly on what people said they spent. They are not data on economic shortfall relative to need.
Most people might think poverty is excess of needs over resources. Let us explore some thoughts on the assumption that this is a sensible definition of poverty.
Suppose a researcher has information about resources. How might they come to a judgement about poverty without looking at needs? Needs include needs for food, rent, water and so on. It is not clear how an argument might be advanced in favour of the idea that lowness or highness of resources might measure prosperity or poverty without reference to needs. Surely, if you earn X-plus-1 units but need to spend 2 more units on health or rent or food or water, you are worse off than your friend who earns X units.
One type of argument in favour of looking at resources without looking at needs might stem from hypotheses about general trends in real life. A person might say “in the real world, income differentials, across time, place and level, are so big that they overwhelm the effects of things like people moving to cities and needing to pay rent”. Intuitively, I do not find this line of reasoning very appealing. If landlessness is a big problem in some countries, then its most immediately obvious effects on command of material resources - lack of assets - may be compounded by the fact that people in cities may have to pay rent, and/or live in worse conditions than others who stay in villages. Quantifying the value to people of living in a slum versus a village seems to me an enterprise which would have a significant element of subjectivity. But even ignoring this problem, the idea that the costs of rent can safely be left out, in an era of fast urbanisation, on the grounds that incomes will be rising fast enough to render this kind of thing insignificant, seems to me to be implausible, or at least in need of empirical examples. In any case, the burden of proof is not on the sceptic who points out the scientist’s assumptions. The sceptic in such a case points out what the scientist must show. In this kind of situation, the burden of proof is on the scientist to show that their claim is reasonable.
Additionally, in this case, mostly the data are not on resources, but on spending. What kind of argument might be advanced in support of the idea that “if you spent 1% more, this shows you got 1% richer”? One argument could be of the form “on average people spend more when they have more”. One problem is that this type of argument would not tell us about any particular period of history, or any particular place. In this particular period of history, for instance, in some places AIDS might reasonably be thought to raise need for expenditure.
A question arises as to how social scientists might describe the existing statistics more accurately. “Poverty” seems wrong, since the data give no specific information about either prices or needs for the relevant people. Even if there had been data on prices, and judgements about relevant prices and relevant needs which were indisputable, income, spending and the money value of consumption could not measure economic poverty as a whole, even in a narrow sense related to legally-defined personal command over resources, since they leave out assets and debts. Statistics on income, spending or the money value of consumption could not by themselves measure income poverty or consumption poverty, since they do not measure consumption need or relevant prices. They do not measure necessary expenses.
A more accurate description of this indicator would be “spending gap ratio [prevalence x depth of spending gap]”. However, this would still not be entirely accurate even ignoring the demographic problems. This is so for at least three reasons.
The first reason why “spending gap” would not be accurate is this: the data are not on what people spent, but on what they said they spent.
The second reason is that people who are destitute may not be reachable. In order to understand the potential importance of this factor, it is necessary to understand how the mathematics works.
The whole point of these types of measures - traditionally claimed to show the “depth of poverty” - is to place more mathematical emphasis on people who are further from the official poverty line (“poorer”). So, for instance, measures like this look at how far each person’s number is from the official line, and add the numbers up in a way that gives more weight to people further from it. This makes sense in theory, at least from the philosophical standpoint implied by the traditional language of development macroeconomics, since the stated aim is to find out not only “how many poor people there are” but also “how poor they are”.
I say the stated aim, because from a philosophical standpoint where needs are not objectively quantifiable, any claim to have measured prosperity or poverty is logically false.
Part of the argument for this standpoint could be this: The concept of a “need” has no meaning except by reference to a particular purpose. You and I do not “need” anything in an absolute sense. We only need things if we want other things to happen. I don’t “need” to live. But I do need certain conditions to be met in order to live.
Beyond bare survival, the relative values of purposes reduce to value judgements about what activities are important.
A related argument could be this: In practice, assessing consumption needs of different people is too complex to be feasible at any reasonable cost. For instance, to come to a judgement about even one person’s food consumption needs you would need to know at least their size, age, gender, workload, the nutritional value of each component of their diet and how the components worked more or less successfully together to make a balanced diet for their body and their activities. For practical reasons, in one sense it is not surprising that official statements about poverty at the national level tend to be based on the cost of calories rather than costs of a balanced diet. Calories do not tell me about how well nourished you are. They are a measure of energy consumption. In another sense, it is surprising that anyone might think calories measured physical well-being.
For mathematical procedures such as those used for what are traditionally called “measures of the depth of poverty”, the absence of destitute people may be important. Without knowing about homeless people, even in the absence of other philosophical, theoretical or data problems, it is not clear how social scientists might justifiably claim to know what has happened to the poorest or to have measured the “poverty gap”.
The third reason why “spending gap” would not be accurate is that the data are not all on what people said they spent. Some data are on what people said they earned. Some are on money values which researchers assigned to people’s answers about what they grew for themselves to eat.
In the official description of indicator 2, the word “incidence” is strictly speaking, according to the most common usage, incorrect.
The tradition in economics is to use the word “incidence” for what people usually call “prevalence”.
Usually:
Incidence is the number of incidents over a period. It is “how often something happens to real people over time”.
Prevalence is “the prevailing rate in the population at one time”.
If incidence rises and so does mortality, prevalence can fall.
Incidence is about real people.
Prevalence is about populations, not what happens to real people.
The discrepancy between the terminology in economics and other social sciences can be seen in the list above.
** Note on indicator 3, “share of poorest quintile in national consumption”:
A “quintile” is a fifth.
It is not clear why the UN Statistics Division referred to “consumption” rather than “consumption expenditure”..
The data compiled by the World Bank from national statistical agencies were on financial amounts, not consumption amounts. The data are mostly on what people said they spent. It is important to distinguish “consumption” from “consumption expenditure”. If I have data on people’s spending, then my figures will relate to what might be called the “share” of spending, not the share of consumption. Economists sometimes shorten “consumption expenditure” to “consumption” rather than “spending”, and the reason is not clear. The word “consumption” in the phrase is not a noun, but an adjective.
Again, this is in addition to the problem about mortality, which makes the use of the word “share” problematic in any case. Because of the problem about mortality, the indicator could not show how much people in the quintile (fifth of the population) consumed even if the statistics had been on food, water, fuel and shelter. This is strictly speaking a confusion between a fifth of the population and a fifth of people. If the “poorest” consume less and some die, the “poorest quintile” now contains people from the next quintile up, who were consuming more in the first place. In such a case, the “poorest” people have had a lower share but the traditional macroeconomist would in the absence of other data say they have had a higher share.
Another aspect of demographic change is birth rates.
Two relevant aspects of falling birth rates are 1) while reducing total food need they increase average food need, and 2) they lower the count of people anyway.
One line of argument might be this: none of the
financial indicators for the Millennium Goals measures poverty, because they do
not consider food and shelter needs. Globally, the proportion of
children is falling. Among countries, one example is
this: other things being equal macroeconomists have underestimated
current poverty in
A counterargument might be that life lengths have risen, so
counteracting some of the effects of falling birth rates on age structure and
efficiency. In the case of
In addition, neither the World Bank nor FAO methods consider the difference between “fewer babies being born” and “people eating better”.
This would still be the case even if they did actually measure consumption adequacy (which is not really possible even in principle, as it is partially a subjective concept) in different years.
The severity of hunger is less if
fewer babies are born to hungry families. But that is different
from saying “people ate better”, which would be the usual inference drawn, if
not by social scientists or politicians, then by the public.
The use by a couple of family planning, so that a baby is not born, is not the
same thing as someone “rising out of poverty”.
Such a change, even in the absence
of other methodological, theoretical and philosophical problems, would not show
how much people’s consumption increased. Nor would it show
how much their consumption adequacy increased.
Proposals for clearer language
from social scientists appear at:
Social science and government aims .
....