notes-science-scientificMethodOutsideOfScience

These are some opinions i have about the supposed (mis)application of the scientific method outside of science, they may not be correct.


Myth: if there is no scientific consensus on something, it's a toss-up

In order for something to reach scientific consensus, the standard for certainty is higher than in most other areas of life. Frequently a very good educated guess can be made about the answer to a question even when the science is not yet conclusive.

Myth: if the science is unsettled, we shouldn't act

It often takes decades for science to resolve a question (and even then, the supposedly conclusive resolution can always be changed later with new evidence, as happened with eg Newtonian physics). In the meantime, we should make educated guesses and take action based on these, while bearing in mind the risk that our guesses are wrong.

Imagine if eg no company dared to introduce a new product until there was 'scientific certainty' that the product would sell; we would probably still not have eg personal computers (perhaps we would eventually though, after the studies completed). Sometimes it's better to take a risk rather than wait for (and pre-pay for the collection of) conclusive evidence.

And on the other hand, if something may or may not be dangerous, sometimes it's better not to take the risk rather than to use it until it is conclusively proven to be dangerous. Imagine if you discovered a new kind of mushroom. Do you eat it? There is no specific evidence that it is poisonous...

Myth: There is no evidence for X means X is probably not true

Often you hear that 'there is just no (scientific) evidence for X'. This does NOT mean that X is false; it just means what it says, that there is no scientific evidence for it. In the absence of scientific evidence, is X a good guess? Maybe.

Often the best guess depends on the application. How much do you lose if you guess that X is false and it is actually true, vs. how much do you lose if you guess that X is true and it is actually false?

Myth: Scientists are careful and rational; their opinions on scientific matters are reliable; they are the sort of people that you trust to be right

On average, scientists seem to be somewhat more careful and rational than your average person, and i would somewhat trust the average scientist to be right about science more than i would trust the average person.

BUT I have met multiple well-regarded scientists who would come off as "cranks" if you didn't know they were famous scientists. I have met multiple well-regarded scientists who seem to have become emotionally attached to their scientific theories and who can't fairly evaluate contrary information. I have met multiple well-regarded scientists who personally insult others who don't share their scientific views, and met multiple well-regarded scientists who feel personally insulted if you disagree with their scientific views.

There is a saying that by Max Planck, "A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it." In my opinion the certainty of science does not come from individual scientists being rational, but rather from (a) the peer review process, (b) the focus on evidence, (c) the focus on falsifiability, (d) the freedom to disagree, and (e) the insistence that even the most picky details of a topic can be examined over and over again until it's right, even if that takes decades.

Myth: Whenever you look at evidence, if you make hypotheses and then check them, then that's the scientific method

No; the scientific method needs peer review. Individual scientists make mistakes and come to the wrong conclusions about the evidence that have seen. You need an open peer review process to allow the community to (slowly, eventually) notice and fix these mistakes.

For example, i hear people in companies say that they tried some thing and then collected data to see if it worked, and then based on their conclusions they either kept doing the thing, or stopped doing it. That's not the scientific method. Did they publish their results? Was there peer review by many people all over the world who had no incentive to agree with them (and in fact, every incentive to disagree with them)? Did these people reproduce their work? Did they wait months or years to see if anyone anywhere in the world disagreed, and if so, argue about it for decades?

Of course not. That would take too long. What these people did was not science; although it was based on evidence and may have involved experiments, it was missing open peer-review. And i think that's great; they got the benefits of an evidence-based, experimental approach, but they avoided spending the time and effort of publishing their results thru a peer review process. They will end up with much less certain results, at a much lower cost, and much quicker. That's exactly what they should have done. But don't call it science.