Modern science is hard, complex, and built from many layers and many years of hard work. And modern science, almost everywhere, is based on computation. Save for a few (and I mean very few) die-hard theorists who insist on writing things down with pen and paper, there is almost an absolute guarantee that with any paper in any field of science that you could possibly read, a computer was involved in some step of the process.
Whether it’s studying bird droppings or the collisions of galaxies, modern-day science owes its very existence—and continued persistence—to the computer. From the laptop sitting on an unkempt desk to a giant machine that fills up a room, “S. Transistor” should be the coauthor on basically all three million journal articles published every year.
The sheer complexity of modern science, and its reliance on customized software, renders one of the frontline defenses against soft and hard fraud useless. That defense is peer review.
The practice of peer review was developed in a different era, when the arguments and analysis that led to a paper’s conclusion could be succinctly summarized within the paper itself. Want to know how the author arrived at that conclusion? The derivation would be right there. It was relatively easy to judge the “wrongness” of an article because you could follow the document from beginning to end, from start to finish, and have all the information you needed to evaluate it right there at your fingerprints.
That’s now largely impossible with the modern scientific enterprise so reliant on computers.
To makes matters worse, many of the software codes used in science are not publicly available. I’ll say this again because it’s kind of wild to even contemplate: there are millions of papers published every year that rely on computer software to make the results happen, and that software is not available for other scientists to scrutinize to see if it’s legit or not. We simply have to trust it, but the word “trust” is very near the bottom of the scientist’s priority list.
Why don’t scientists make their code available? It boils down to the same reason that scientists don’t do many things that would improve the process of science: there’s no incentive. In this case, you don’t get any h-index points for releasing your code on a website. You only get them for publishing papers.
This infinitely agitates me when I peer-review papers. How am I supposed to judge the correctness of an article if I can’t see the entire process? What’s the point of searching for fraud when the computer code that’s sitting behind the published result can be shaped and molded to give any result you want, and nobody will be the wiser?
I’m not even talking about intentional computer-based fraud here; this is even a problem for detecting basic mistakes. If you make a mistake in a paper, a referee or an editor can spot it. And science is better off for it. If you make a mistake in your code... who checks it? As long as the results look correct, you’ll go ahead and publish it and the peer reviewer will go ahead and accept it. And science is worse off for it.
Science is getting more complex over time and is becoming increasingly reliant on software code to keep the engine going. This makes fraud of both the hard and soft varieties easier to accomplish. From mistakes that you pass over because you’re going too fast, to using sophisticated tools that you barely understand but use to get the result that you wanted, to just totally faking it, science is becoming increasingly wrong.
What’s more, the first line of defense for combatting wrongness—peer review—is fundamentally broken, with absolutely no guarantees of filtering out innocent mistakes and guilty fraud. Peer reviewers don’t have the time, effort, or inclination to pick over new research with a fine-toothed comb, let alone give it more than a long glance. And even if a referee is digging a little too deeply, whether revealing some fatal flaw or sniffing at your attempt at fraud, you can simply go journal shopping and take your work somewhere more friendly.
With the outer walls of scientific integrity breached through the failures of peer review, the community must turn to a second line of defense: replication.
Replication is the concept that another scientist—preferably a competing scientist with an axe to grind because they would love nothing more than to take down a rival—can repeat the experiment as described in the published journal article. If they get the same result, hooray! Science has progressed. If they get a different result, hooray! Science has progressed. No matter what, it’s a win.
Too bad hardly anybody bothers to replicate experiments like this, and when they do, they more often than not find that the original results don’t hold up.
This is called the “replication crisis” in science. It first reared its ugly head in the 2010s in the field of psychology, but I don’t think any field is immune.
There’s a complex swirling septic tank of problems that all contribute to the replication crisis, but the first issue is that replication isn’t sexy. You don’t get to learn new things about the world around us; you just get to confirm whether someone else learned new things about the world around us. As an added ugly bonus, nonresults often don’t even get published. Novelty is seen as a virtue, and if you run an experiment and it doesn’t provide a positive result, journals are less likely to be interested in your manuscript. Additionally, because replication isn’t seen as sexy, when it is done, it isn’t read. Replication studies do not get published in high-impact-factor journals, and authors of replication studies do not get as many citations for their work. This means that their h-index is lower, which lowers their chances of getting grants and promotions.
Second, replication is becoming increasingly hard. With sophisticated tools, mountains of data, complex statistical analyses, elaborate experimental design, and lots and lots of code, in many cases, you would have no hope of attempting to repeat someone else’s work, even if you wanted to.
The end result of all this, as I’ve said before, is that fraud is becoming increasingly common, increasingly hard to detect, and increasingly hard to correct.
The lackluster mirage that is peer review doesn’t adequately screen against mistakes, and with the increasing complexity of science (and with it of scientific discourse), fraud just slips on by, with too many layers between the fundamental research and the end product.
It looks like science is getting done, but is it really?
Sure, retractions and corrections appear here and there, but they are far too few and far between given the amount of fraud, both hard and soft, that we know exists. The conditions in every field of science are ripe for fraud to happen, and nobody has developed a decent enough policing system to root it out—in one case, a journal issued a retraction 35 years after being notified. The kind of investigations that lead to full retractions or corrections don’t come often enough, and meanwhile, there’s just too much science being published. A perusal of the Retraction Watch website reveals instance of fraud after fraud, often coming too late, with little to no major repercussions.
Nobody has the time to read every important paper in their field. It’s a nonstop tidal wave of analysis, math, jargon, plots, and words, words, words. Every scientist writes as much as possible, without really digging into the existing literature, adding to the noise. Heck, take my field, astrophysics, for example. Every single day there are about forty to sixty brand-new astronomy and astrophysics papers birthed into the academic world. Every single day, even holidays! According to surveys, scientists claim that they read about five papers a week. In my not-so-humble opinion, that’s a lie, or at least a big stretch, because there are different definitions of the word “read,” varying from “reading the abstract, the conclusions, and the captions of the prettier pictures” to “absorbing this work into the fabric of my professional consciousness.”
The vast majority of science published is not being consumed by the vast majority of its intended audience. Instead, most scientists just skim new titles, looking for something that stands out, or dig up references during a literature search to use as citations. We’re just running around publishing for the sake of publishing, increasing our chances of striking gold (having a major paper stand out among the noise and get a lot of citations) or at least building a high h-index by brute force of an overwhelming number of publications.
This is exactly the kind of environment where fraud not only exists but abounds. Who cares if you fudge a number here or there—nobody’s actually going to read the thing, not even the referee! And the conclusions you draw in your paper will form the basis of yet more work—heck, maybe even an entire investigative line.
So why doesn’t it stop? We all know and agree that fraud is bad, that uncorrected mistakes are bad, that impossible-to-replicate results are bad. If we all agree that bad things are bad, why don’t we make the bad things go away?
Put simply, there’s no reason to do better. The incentive in science right now is “keep publishing, dang it,” not to waste your time doing careful peer reviews, making your code public, or replicating results.
I’m not saying that most scientists are attempting to pull off willful fraud. I do believe that most scientists are good people trying to do good things in the world. But with all the incentives aligned the way they are, it’s easy to convince yourself that what you’re doing isn’t really fraud. If you refuse to make your code public for scrutiny, are you committing fraud? Or righteously protecting your research investment? If you had to twist the analysis to get a publishable result, are you committing fraud? Or just finally arriving at the answer you knew you were going to get anyway? If you stop checking your work for mistakes, are you committing fraud? Or are you just... finally done, after months of hard work? If you withdraw your article from submission because the referee was getting too critical, are you committing fraud? Or are you protesting an unreasonable peer reviewer?
Scientists aren’t really incentivized to change any of these problems. The system is already in place and “works”; it allows winners and losers to be chosen, with the winners receiving grant funding, tenured positions, and fancy cars. Well, maybe not the fancy cars. Universities aren’t motivated to change these practices because, so long as their winners continue to bring in grants, it lets them roll around in giant taxpayer-funded money pits (I don’t know if this image is totally accurate, but you get the idea).
The very last people who aren’t really motived to change any of this are the publishers themselves. Most publishers make the scientists pay to get their research printed, and they don’t have to pay for the peer review that makes science all scientific. But the big bucks come from the subscriptions: in order to read the journal, you need to either cough up a few dozen bucks to access a single article or pay prohibitively high fees to get annual access. The only people who can afford this are the big universities and libraries, locking most of our modern scientific progress behind gilt-ivory doors.
Altogether, the scientific and technical publishing industry rakes in about ten billion dollars a year, often with double-digit profit margins. Of course they don’t want this ship to change course. I can’t really blame them; they’re just playing by the accepted rules of their own game.
Excerpted from the book Rescuing Science: Restoring Trust In a Digital Age by Paul M. Sutter. Used by permission of the publisher Rowman & Littlefield. All rights reserved.
