I recently saw an article from Pew Research that was entitled, “Obama Tops Public’s List of Best President in Their Lifetime, Followed by Clinton, Reagan.” This will make some of you pretty happy, and some of you may not jump on board with this conclusion. The survey itself was rather straightforward. 2002 people were phone interviewed and asked to name the President who has done the best job during their lifetime. This question was followed up by asking for the President who had done this second-best job during their lifetime. President Obama not only received the most votes for the best President, but when the percentages for best and second-best were added together, he also came out on top of that tally.
As I was reading this article, a highly significant problem jumped out at me, and I just want to say upfront I am far from an expert on survey methods. Yes, I was an undergraduate statistics major, so I know a little something about this field, but the people at Pew Research are much better at this than I am. Also, when I reached out to Pew for some information and clarification, team members were very helpful and prompt in providing the information I requested for research purposes I want to thank them for that.
At the same time, I want to be honest with all of you. The way the research question is phrased betrays a vital weakness in this research that renders it useless. It destroys any type of objective measurement and makes any type of conclusion impossible.
Notice that this is a quantitative survey design. It is the type of survey where there are a limited number of answers that a person can choose. In this case, the acceptable range of answers is the group of Presidents that were alive during your lifetime. For me, the options would have been George H.W. Bush, Bill Clinton, George W. Bush, Barack Obama and Donald Trump. I would choose two of those answers, one being the best and one being the second-best.
If you asked the same question to my grandmother, she would be able to choose from a lot more Presidents, from Herbert Hoover forward. She would similarly be able to choose the best President and second-best during her lifetime, but her answers would not be constrained to the same five I had the option to choose from.
If she was limited to the same five that I was, her answers very well might be different than it would be if she was able to choose from the entire set since Herbert Hoover. She might really like John F. Kennedy and would be free to say that in this survey. I would not have the option to choose John F. Kennedy because he was not born within my lifetime.
This is a substantial problem in terms of survey design.
The survey is divided into four generations: Millennials (ages 22-37), Gen X (ages 38-53), Boomer (ages 54-72) and Silent (ages 73-90).
For the latter three groups, Ronald Reagan was the favorite in both best and the combination of best and second-best. However, among my generation, he was only voted best 8% of the time and a combined total of 13% for best and second-best.
Ronald Reagan was president from 1981 through 1989. That means that he was only a President during the lifetime of Millennials who are between 29 and 37. There are seven years of Millennials, myself included, who would not have had the option to choose Ronald Reagan as our favorite President even if we wanted to. However, we must put our votes somewhere, so if we are conservative, we might choose Donald Trump or George W. Bush.
You can see the same phenomenon with the other generations as well. Lyndon B. Johnson was President from 1963 until 1969. Not that he was an overly popular president anyway among the people who were alive to see his Presidency, but it is not fair to measure his popularity among Gen X members who were born beginning in 1965. We could say it is ridiculous that he only got 1% of the vote combined as best President, but Gen Xers who were born later in 1969 until 1980 would not have had the option to choose Lyndon B. Johnson because the survey constrained their answers to people during their lifetimes. Therefore, how do we know how this generation really feels about him? We only know how members of this generation born in the first four years feel. Maybe 1% is not all that bad especially since he was not an option for a majority of the people of that group in the survey.
Even within the same age brackets, the same options are not available to the members of the group. Therefore, the survey would never know that Ronald Reagan is indeed my favorite President from 1981 until the present because the way the question is worded would cause me to only consider Presidents from 1991 until the present.
There is the obvious problem. This survey would report that I, a Millennial, said George W. Bush was my favorite President in my lifetime. The way the data is compiled, it would appear that I did not like Ronald Reagan, but that is not true. I did not have the ability to choose him. Therefore, my being labeled a Millennial who loves George Bush more than all other Presidents since 1981 is false. The outline does not accurately characterize the frame that I was working within.
You can multiply this problem by all 2002 people who took the survey. Not every member of a given generation had the ability to vote for every President that was in office during their classified generation. Believe it or not, not every Millennial was born on January 1, 1981. Some people were actually born on December 31, 1980, so their information would be classified as Gen X, but they would not be able to choose any of the additional Presidents that other members of their generation had the ability to.
I hope that you can see this problem especially because, and this is the information that the extraordinarily helpful staff at Pew Research shared with me, in the top survey where President Obama garnered 44% of the combined best and second-best votes, that is a weighted total based on population representation. The 2002 people consisted of 416 Millennials, 459 Gen Xers, 740 Boomers and 250 Silents. These were then weighted based on the demographic splits of the United States population. That is 27% Millennial, 28% Gen X, 28% Boomer and 9% Silent.
Doing this kind of weighting is not a problem. It is done in surveys all the time. Obviously, the numbers that were gathered on the phone were not representative of the population at large, and Pew Research wanted to make sure that their survey was actually representative of the American population. I respect that and don’t blame them for doing it.
However, they are using the problematic assumption that these groups are fair, and as I have shown above, they are not.
The data virtually means nothing because it lacks all objective standards of measurement. It is impossible to compare one person to another even within the same generation, and when you try to aggregate the data through the generations, the entire thing simply becomes a mess that makes very little sense.
What is potentially more frightening is that despite a great deal of media coverage for this survey result, no one seems to mention this problem. If you hadn’t read this article, you may not have thought about these problems.
The Washington Times, certainly not a publication that would celebrate this result, failed to draw out this problem in design. They did mention the fact that while President Reagan did so well in every other group, he faltered among Millennials, but they attributed to a lack of memory rather than a fundamental flaw in the study. “But among Millennials, most of whom would have little or no memory of him, he was the second least-favorite, ahead only of his vice president, George H.W. Bush.” It really doesn’t matter if we have a memory of him or not. I have a memory of Ronald Reagan because I have read about history, but that memory doesn’t matter because I could not choose him given the constraints on the survey.
ABC News doesn’t really even address this problem.
Politico acknowledges that, “The survey would, of course be tilted against presidents of the more distant past,” but it doesn’t explain why this is the case. It doesn’t blame memory, but by speaking about the, “distant past,” I wouldn’t be surprised if that is the implication some people draw from that phrasing.
Time gets closest in their response by attributing the results to, “changing demographics, since the question asked respondents for presidents during their lifetimes. For example, only about a tenth of adults named John F. Kennedy — but that number is around a quarter for those who were alive to see his presidency.” However, I don’t know that changing demographics really is the problem. The problem is the question having different answers for people with different demographic features. It is not the fact that the demographics are changing. Maybe I am misunderstanding what the author at Time intended, but they are not far off. They seem to see the problem, but they don’t seem to play it out quite right.
New York Magazine, writing from a very liberal perspective, also seems to understand what is going on here by and large in their commentary. However, like the Washington Times above, they still misattribute these results to bad memory. “No doubt much of this [gap between liberal and conservative Presidents] reflects not only the liberal leanings of younger voters but also the gap in experience: Only the oldest millennials remember much from Reagan’s era.” It doesn’t matter how much we remember. We have history books to help us remember the past. I remember that Reagan was a good president because I have read about what he did. However, I could not choose him because he was not within the scope of possible answers for someone like me born in 1991.
I tried to pull media sources from left, right and center to illustrate my point. No matter where you read about this survey (except for here), it is quite possible that you did not hear about the true problem of this study.
So Who Cares?
You might wonder why this matters. Why did I bother writing something like this especially on a website that is normally concerned with Christian culture and the problems facing Christians in the world today. It seems off-topic or irrelevant.
I tell you this because we all talk a lot about the reliability of the government or the media. We all wonder about survey results and the data that is constantly thrown at us. When we see a number that we like, we hit the ground running to go prove our point to whoever may have had the nerve to doubt us before. After all, when a survey officially and definitively shows that President Obama is the most popular president in America, I have to go throw that number in everyone’s face who ever thought that he was anything but the best President ever.
I went through this exercise, not to judge President Obama one way or the other, but I want to encourage you to think before you look at numbers and automatically assume that the support your preconceived opinion no matter where you land on any issue really. There are plenty of ways that data can go wrong, and some of them are as simple as faulty question design. That’s what we saw here, and we need to be diligent.
Caution is a virtue. Jumping on board with a particular trend, especially if it reinforces our preconceived notion, is always a dangerous place to be in. We don’t want to fall for falsehood. As Christians, we have a particular commitment to Truth, so we need to do our best to evaluate what we see and not fall prey to inaccurate information.