Much more is now known about the biases that cause people to overestimate their or underestimate their skill levels in different areas. Kahnaman talked about it his book.
Basically, if something is felt to be 'easy' such as driving a car, then most people overestimate how good they are at it. OTOH, if something is considered 'hard' then most people underestimate how good they are at it.
In other words, its not simply a matter of competence or not. It may be a matter of misestimation due to cognitive bias.
Or, it may be more complicated that either model proposes, which seems the most likely possibility to me.
I suppose a professional psychologist is someone who gets paid for it, whereas in reality they're all amateurs.
Nope. It doesn't cover "humans don't work on reason as much as they would like to think they do". BTW, it wasn't me who used "peer reviewed published research" strawman reply first, if you get my drift.
Only "respond" to 1 out of 4 questions? You must have a lot to hide.
I've heard several academics say that research papers are published prematurely, being often poorly peer reviewed, because the researchers are chasing the next research finance.
That's a pretty unfair thing to say, unless you want to go fully down that line of thought and say we're all a bunch of amateurs at our jobs, based on competence at least. Just because a field is less mature doesn't mean the people in it are less professional.
You owe yourself a bit of reading on the subject -- it's been tested across a much larger swath of population. And the bigger takeaway is we all suffer from it (the less you know about a subject the worse your ability estimation is).
Yes, when the data is poor/sparse then we have to borrow from wherever we can. Gut/heuristics is an addictive drug, though, and l am becoming more and more experience inheriting issues where this was the MO versus being data-driven, which is expensive in terms of resources, especially time.
Being involved, in another life, up to my neck in the academia, I can confirm this being true then, and I have no reason to believe today is anywhere better. Some of the peer reviewed papers, in particular those appearing low ranking journals, are true garbage.
However, beyond attempts to rank journals (ISI, SJR, etc...) nothing replaces common sense and the critical thinking of the readers. Even the citation indexes can be misleading, since there's nothing telling if the citations are positive/constructive or critical (a typical example on topic would be the Oohashi paper, widely cited from a critical perspective). Like it or not, the peer review process is the best we could do, and if anybody has a better idea, I am sure Nature, Physical Review or New England Journal of Medicine would be all ears.
Well, Mark has been accused of "amateur psychology" before now, but who's to know if his analysis is more or less accurate than a professional's?
Just don't start on 'superforecasters'. We had a laughable incident this year where the shower in power recruited one. His forecasting powers were so good he didn't think that the media would immediately trawl his social media history and point out the fact that he supported various nasty things including eugenics. He lasted less that a day in the job!
Having left the circus only a couple month ago, you're spot on. I would be more damning than your assessment but I'll keep my mouth shut. We are seeing (pockets) of improvement in research publication integrity but it's a long road.
*Low ranking journals may have high ranking statistics (standard metrics have been hijacked). E.g. a lot of highly specialized journals have low rank stats due to their minimal reach (but with high integrity). But, yes, there's a ton of crap and for-profit predatory journals.
Edit -- Bill, not sure I know anything about "superforecasters" but I can feel my blood pressure rising just on title alone.
Last edited:
On another forum a few months ago was posted a link to a Radio interview which was filmed. It featured a senior lecturer and a student who had made up a paper in the social sciences containing a thesis which was completely fabricated. Not only did it get the masters degree, but also a special award, and the two were wondering how to get out of this situation, especially the lecturer.
While flawed, peer review is certainly still better than none. I'm glad biorxiv and medrxiv exist, but just take a look at the torrent of garbage that the pandemic has spawned. We're seeing even normally credible news outlets citing preprints that are total nonsense.
The method of preventing Covid 19 has long been known. Burn all the witches, of course after a peer reviewed trial.
Having reviewed material submitted for publication, it is not an easy job. To be truly accurate I would have to do experiments in the style of the submission to be fully aware of the actual issues. The idea is folks reading the articles will test the material presented and then form a group consensis. Just because a paper is peer reviewed does not make it right. It has been presented for discussion.
Of course one does get a smile when dumping out garbage. However as reviewing is done blind as to the author I have reviewed material positively from folks I am not fond of. (In one case I did point out a huge hole in the data base.)
Having reviewed material submitted for publication, it is not an easy job. To be truly accurate I would have to do experiments in the style of the submission to be fully aware of the actual issues. The idea is folks reading the articles will test the material presented and then form a group consensis. Just because a paper is peer reviewed does not make it right. It has been presented for discussion.
Of course one does get a smile when dumping out garbage. However as reviewing is done blind as to the author I have reviewed material positively from folks I am not fond of. (In one case I did point out a huge hole in the data base.)
Last edited:
I submitted a paper to a conference and one of the reviewers ran to the patent office with the content, it never mattered those that can put up make the others shut up.
This guy has some interesting (IMHO) views on psychology (basically that it's unscientific despite perennial claims otherwise). He's added stuff since I last read his site years ago, and even back then it (just the psychology writings, and that's just one part of his site) was approaching book-length. I think the "true science minded" people here might enjoy reading his psychology section:
* arachnoid.com
* arachnoid.com
I worked with a PhD solid state physics guy (one of hundreds of at Philips back in the day in all sorts of areas of expertise) that told me exactly the same thing. A lot of this stuff is low value or just plain garbage. Not to say these people are not smart (and more so than I will ever be) but the pressure to publish is immense. The value of an academic seems to be counted in the number of citations.
@benb my wife has a psychology degree. Plenty of stats and loads of experiments that are tested.
Saying psych is not a science when we know how to fix mental illness in many cases, or in others, bend the perceptions of huge swathes of the population by applying methods gleaned from research into group behavior seems incongruous to me.
🙂
Saying psych is not a science when we know how to fix mental illness in many cases, or in others, bend the perceptions of huge swathes of the population by applying methods gleaned from research into group behavior seems incongruous to me.
🙂