Peer evaluation is an honor-based system – Daniel Lemire’s weblog

It might take too lengthy to show all the flaws of peer evaluation, listed here are some:
- some work is simply flat flawed as a result of the reviewers can not analyze all the mathematical outcomes, and since they can’t redo the experiments;
- quite a few researchers cheat, typically in small methods (“2 out of three experiments agree with by concept, allow us to drop the third one”), typically in huge methods (“I don’t have time to run these experiments, so let me make up some knowledge”);
- peer evaluation might perpetuate some biases and stop researchers from placing into query some basic questions (“we determined that that is the precise means, in the event you query it, you’re a loony”).
Nevertheless, for all its faults, peer evaluation stays important in science. I need different researchers to learn and criticize my work. I take pleasure in it very a lot when folks attempt to discover flaws in my work. I feel that my work is severe sufficient that when folks level out flaws, I’m often conscious of them at some stage and I can reply simply (and benefit from the course of).
The kind of peer evaluation I don’t take pleasure in is the country-club strategy: 1) does the paper agrees with the targets and views of the reviewers 2) is the paper written by somebody we will respect? Thankfully, you may navigate the system and keep away (largely) from country-club peer evaluation.
However why do I nonetheless like peer evaluation regardless of its apparent flaws? As a result of I see it as an honor-based system. In such a system, it’s a must to settle for that there shall be cheaters. Plenty of them. And there’ll a lot of errors. All we’ve got to do is be open about it. That’s, you can’t say “however my work was peer reviewed so you can’t query it!” or “I’m superb, have a look at these prestigious publications!”. The peer evaluation is there to assist the authors. It’s not, nevertheless, an insurance coverage towards fraud or errors. I like peer evaluation as a result of it helps me change into higher, however I don’t use the system to find out how good another person is.
So, what can we do if we wish to know the way good somebody is? You learn his work. You reproduce his experiments. You redo his math. After all, this scales poorly. If it’s a must to rent somebody, you can’t learn the work of fifty or 500 candidates. So? I feel we’ve got to be sensible. It’s onerous to know the way good somebody is. You will get to know 10 or 20 researchers in your life. That’s about all.
Hiring processes are flawed. You’ll rent cheaters. Recover from it.
Printed by
