USBIG Blog: “Social Experiments 101: A Short Primer for UBI Observers”

Social Experiments 101: A Short Primer for UBI Observers

By: Michael Anthony Lewis (Associate Professor of Social Work, CUNY)

I’ve gotten through most of Karl Widerquist’s new book A Critical Analysis of Basic Income Experiments for Researchers, Policymakers, and Citizens. It’s a must read for those interested in the universal basic income (UBI) experiments currently proliferating around the world. Much of what Widerquist says is familiar to anyone who’s done coursework in basic statistics and research methods. What he does that’s different is relate this material to recent UBI experiments.

In principle, I’m a UBI supporter and have been for about 20 years. I qualify my support by saying “in principle” in order to signal that I recognize there are questions surrounding how such a plan would work in practice. At least some of them will need some pretty clear answers before we should take the plunge and implement the plan. And it may be that if we can’t answer some key questions, such as how to finance a UBI in a way that wouldn’t make currently low-income people worse off, we shouldn’t implement a UBI at all. As someone who supports UBI and who’s read most of Widerquist’s book, I’m worried about some of the commentary I’ve seen on the recently released findings from Finland’s basic income experiment.

A textbook case of what Widerquist says can go wrong can be seen by reading the recent article by Tedra DeSue entitled Free Money for Everyone Sounds Great, But Finland Proves Basic Income is a Bust. The first problem is the sensationalist title of this piece. Finland hasn’t proven anything about UBI. Second, even though the article mentions in passing some of the other findings (e.g. UBI recipients had higher levels of happiness and well-being than non-recipients), it focuses primarily on the fact that researchers found no evidence that recipients of “free money” were more likely to work than those who didn’t receive the stipend. This second finding, according to DeSue, is what makes basic income a bust. It’s true that UBI recipients were no more likely to work than those who didn’t receive the UBI. But neither were than any less likely to do so. That is, the Finland study found no evidence that UBI increases or decreases work. With commentators claiming that a UBI would destroy the incentive to work, this is an important point to keep in mind.

One of the points Widerquist drives home in his book is that social scientists face difficulties in communicating the findings of complex social experiments to a lay audience. This is always a problem, but when it comes to a policy as controversial as UBI, it becomes an even bigger one. He also tells us that one source of the problem is that social scientists are mainly trained to communicate to each other, not to laypeople. But Widerquist does think social scientists should still try — the UBI is too important a topic for them just talk to each other while leaving it to politicians and citizens to engage in a broader discussion. This post is my attempt to sort of follow his advice.

I said “sort of” because I’m not mainly going to try to communicate the preliminary findings of Finland’s study to a lay audience, although I’ll do some of that. What I’ll do instead is communicate how social scientists think about the design of studies such as Finland’s. The hope is that an understanding of some of these basics may help lay people make sense of articles such as DeSue’s, along with the many others that have been (and will be) released.

The first thing I should say is that social scientists don’t see experiments as proving anything. To be charitable, I suspect DeSue didn’t mean to use the phrase “proves” in its technical sense. I’ll assume that by “proves basic income is a bust” he meant something like “provides strong evidence that basic income is a bust.” But since this could still be a source of misunderstanding, I’ll say a bit about what a proof is. Once this is understood and I’ve gone into how social experiments are designed, it should become clear that experiments prove nothing.

A proof is a type of deductive argument. An argument is a set of statements, one of which is the conclusion and at least one other of which is a premise meant to support that conclusion. That is, the premise is meant to provide a reason for believing that the conclusion is true. A deductive argument is one where if the premise(s) is (are) true, the conclusion must be true as well. Here is a trivial example of a proof:

Premise One: 2+2 = 4

Premise Two: 3+1 = 4

Conclusion: Therefore, 2+2 = 3+1

This is a proof/deductive argument because if both premise one and two are true (which they are), then the conclusion must be true (which it is).

The experiments designed by social scientists aren’t deductive arguments, so they cannot prove anything. Instead, these are exercises in inductive arguments. This is a type of argument where the truth of the premises doesn’t guarantee the truth of the conclusion but only makes the conclusion probably true (to some degree of probability that’s less than 1 or 100%.) Here’s an example of an inductive argument:

Premise One: The stipend in Finland’s UBI experiment didn’t encourage participants to work more

Conclusion: Therefore, if a nation wide UBI were implemented no one who received a UBI would work more.

Even if premise one of this argument is true, it doesn’t mean the conclusion must be true. Now that it’s clear what a proof is, let me move on to what social scientists are trying to do when they design experiments.

Social scientists design experiments when they want to figure out what causes some outcome of interest to them. What’s meant by cause is a philosophical minefield, but many social scientists have adopted something like the following idea: X causes Y if had X not occurred, Y wouldn’t have occurred. Applying this to the Finland experiment, receipt of a UBI causes an increase in labor supply if had the person not received a UBI, they wouldn’t have increased their labor supply. This is a deceptively simple idea, so let’s take it again more slowly.

Imagine someone named Buffy. We consider her in two different possible worlds: the one where she receives a UBI and the one where she doesn’t. If in the one where she receives a UBI, she increases her labor supply while in the one where she doesn’t receive it, she doesn’t increase her labor supply, then receipt of a UBI causes increased labor supply. The problem is that in the actual world, Buffy either receives a UBI or she doesn’t — it’s impossible for her to receive and not receive a UBI at the same time. This is where social experiments come in.

In experiments, social scientists randomly assign people to receive X (e.g. a UBI) or not receive X. The group that receives X is usually called the experimental group while the one which doesn’t receive it is called the control group. Social scientists don’t usually focus on whether the values of Y differ between each experimental and control group pair but on the difference in the average values of Y between the two groups. If there’s a big enough difference in these averages, they conclude there’s a causal effect of X on Y. Let’s dig a little deeper into this to get a better sense of what’s going on.

I said social scientists compare the average Y values between the experimental and control groups. The logic employed in how they typically do so is a bit odd. I’ll illustrate this logic by focusing on the Finland study — just keep in mind that everything I’ll say applies more generally.

The way the reasoning works is that at the beginning of the experiment, researchers assume there’s no difference in the average values of labor supply between the experimental and control groups. That is, the initial assumption is that those assigned to receive UBI don’t have a higher or lower average labor supply than those denied it — the average labor supplies of the two groups are assumed to be equal. In general, this assumption of no difference is called the null hypothesis.

The experiment then unfolds — the experimental group receives UBI while the control group doesn’t. After a specified period of time, the average labor supplies of the two groups are compared. Suppose at this time, that the average labor supplies of the two groups differ. A question is then asked: if there really is no difference between the average labor supplies of UBI recipients and non-recipients, how probable is it we’d obtain a difference at least as large (in absolute value) as the one observed in the experiment? If that probability is small enough, the researchers would conclude there’s a difference in the average labor supplies of the two groups. And, given this conclusion, if the observed difference in the experiment is such that the average labor supply is higher for the UBI recipient group than for the control group, the conclusion would be UBI causes an increased labor supply. When the assumption of no difference is set aside and researchers conclude they’ve found a causal effect, this is called rejecting the null hypothesis.

At this point, a question might arise: if there really is no difference between the average labor supplies of the two groups, how could a difference be observed in the experiment? The answer is simple: probability. There’s a non-zero chance that a difference in the average labor supplies between the two groups will be observed in the experiment, even if there’s no real difference in these averages. This is precisely why when an experimental difference is observed, social scientists ask the question I referred to earlier: how likely is it that a difference at least as large as the observed one would be seen, assuming there really is no difference between the average outcomes of the two groups? And, as I also said earlier, if the chance of such a difference is small enough, social scientists assume the difference observed in the experiment indicates a causal effect.  

Why do social scientists randomly assign people to experimental and control groups? They do this because they want those in the control group to be as similar as possible to those in the experimental group (other than the difference in UBI receipt), and random assignment is thought to be the best way of attaining such similarity. Recall the conception of causality from earlier: X causes Y if had X not occurred, Y wouldn’t have occurred. And recall that in the real world we can’t assign someone X and deny it to them at the same time–they either receive X or they don’t. In other words, to see if X causes Y what we really want to do is observe what happens to someone when they receive X and at the same time when they don’t. We can’t do that, so random assignment is the next best thing. What randomization does it create “statistical clones” in the sense that members of the control group are “stand ins” for members of the experimental group. That is, members of the control group are meant to help us find out what the labor supply of members of the experimental group would’ve been had they not received a UBI.    

Let’s consider the finding from the Finland study that UBI recipients neither worked more nor less on average than members of the control group. I’m not privy to the details of the study, but knowing what I do about social experiments, I suspect this means that any observed difference between the average labor supply of UBI recipients and non-recipients wasn’t large enough to reject the assumption of no difference between the two groups. That is, any observed difference between the two groups in average labor supply values was small enough that it could’ve been due to chance, instead of a causal effect of the UBI on labor supply.

When social scientists employ the experimental reasoning I’ve been discussing, there are two possible errors they can make. One is they can assume they’ve observed a causal effect when in fact they haven’t; that is, any observed difference in average values of the outcome is due to chance, not causality. The other is they can assume any observed difference is due to chance, when in fact what they’ve observed is a causal effect.

Having gotten to this point, I can now explain why earlier I said that social experiments prove nothing. Recall that a proof is a deductive argument, where the truth of its premises guarantees the truth of its conclusion. We can think of a social experiment as a kind of argument. The argument’s conclusion might be “there’s no causal effect of X on Y.” The argument’s premises might be 1) there’s a difference in the average value of Y between the experimental and control groups and 2) this difference isn’t large enough to reject the null hypothesis of no casual effect. But I said above that this type of reasoning is vulnerable to two possible errors. The one relevant here is concluding there’s no causal effect of X on Y when in reality there is such an effect. The fact that either of these types of errors is possible, indicates that the premises of experimental arguments can be true and yet a researcher’s conclusion false. This combination of true premises and a false conclusion is impossible in a proof. Therefore, social experiments can’t prove anything.

Let’s apply what I said in the previous paragraph to the Finland UBI experiment. Based on preliminary results, the researchers have concluded that UBI didn’t affect labor supply, one way or the other. But this conclusion could be in error. It may be that UBI really increases or decreases labor supply, although this study’s results make it appear this isn’t the case.  

Let me say one other thing about social experiments that’s especially relevant to Finland’s UBI study. Researchers, and when it comes to controversial topics like UBI, politicians and citizens as well, often want a study’s findings to apply to a broad group of people. But participants in an experiment are often merely proper subset of this broader group. For example, in the Finland study, a UBI was granted only to the unemployed. This fact suggests that even calling the grant a UBI is a misnomer, since it wasn’t granted universally. Yet the eyes of the world are on Finland because folks think they may be able to learn something about how a UBI would work in their nation. But findings from a study of Finland’s unemployed may not even tell us much about how a UBI would work if given to the entire population of Finland, let alone the populations of other nations. So, DeSue’s contention, based on what happened in Finland, that basic income is a bust, is a bit premature. Differences between Finland’s unemployed and members of the U.S. population or between Finland’s labor markets and the U.S.’s may mean that the findings in Finland tell us almost nothing about what would happen if a UBI were implemented in the U.S.

The issues I’ve discussed in this post don’t exhaust the ones observers of UBI studies should keep in mind. Here are a few others. As Widerquist says in his book, the labor market, as well as broader economic, effects of a UBI given to a few people may be vastly different from a full scale UBI given to an entire population. Also, participants in Finland’s study knew they would only receive a UBI for a relatively short period of time. The effects on their behavior may be very different from what we’d see if recipients expected to receive a UBI indefinitely. Another thing to keep in mind is that social scientists’ focus on differences in group averages between experimental and control groups can mask the fact that causal effects can vary within groups. Even though the Finland researchers, by comparing group averages, found no evidence of a causal effect of UBI on labor supply one way or the other, it’s possible that some members of the UBI group worked more, some worked less, and some didn’t change their labor supply at all.  

The DeSue article I mentioned earlier is an example of the kind of spin we’re likely to see as the results of high profile UBI studies start coming out. They (to be gender neutral) seem to be opposed to UBI. But there will no doubt be UBI proponents spinning as well. Hopefully, at least a few commentators will simply be trying to figure out what particular studies can and can’t tell us about the impact of a fully implemented UBI. Perhaps this post can serve as a guide to such observers not trained in the design and interpretation of social experiments.    

—

Photo: Sunset in Helsinki, CC BY 2.0 Giuseppe Milo