In this course, we talk a lot about the REPLICATION CRISIS •What is replication crisis? • •Can you describe it in your own words? • •I want to be a psychotherapist – is this relevant for me? • •How about other disciplines? The Sin of Bean Counting The Sin of Corruptibility Who is this guy? A picture containing text, person, person, indoor Description automatically generated Who is this guy? A picture containing text, person, person, indoor Description automatically generated Boek 'Ontsporing' van fraudehoogleraar Diederik Stapel staat online - Omroep Brabant Who is this guy? Boek 'Ontsporing' van fraudehoogleraar Diederik Stapel staat online - Omroep Brabant Psychology must learn a lesson from fraud case (Wicherts, 2011) •“The interim report of the investigating committee revealed that Stapel often refused to share his research data with colleagues, even co-authors on papers. To scientists in other fields, this may seem extraordinary; to psychologists it is sadly common practice.” • •almost 3/4 of researchers who had published a paper in a high-impact psychology journal had not shared their data (more with the Sin of Data Hoarding) •it is not unusual for data that are shared to list variables only as VAR00001 through VAR00019, with no further explanation. •co-authors rarely verify a study's analysis • Psychology must learn a lesson from fraud case (Wicherts, 2011) •readers of published papers are shown dense summaries of results •sharing data: there are practical problems, including the need to keep data or participants confidential •these can be solved by embargoes on releasing data for longitudinal studies, guidelines for pre-processing raw data, proper anonymity and exemptions where necessary •a 'co-pilot' model, in which we share data between us for double-checking and preventing embarrassing errors But why would they do it? A picture containing text, cat, sign, standing Description automatically generated A picture containing text, mollusk, invertebrate, snail Description automatically generated Diagram Description automatically generated Impact factor •A well-intentioned measure of journal quality •Started in 1979, now calculated by a private company Clarivate • IF = number of citations / number of publications in two preceding years Impact factor – examples Nature: 50 Science: 47 European Journal of Psychological Assessment: 3 Czechoslovak Psychology: 0.5 Testforum: NA Article Influence Score •An alternative to IF that measures influence of a journal •Recently implemented as the metric that decides funding in Czech Republic •Mean AIS = 1 •Omits journal-level self-citations • AIS = journal’s citation influence / number of articles over five years AIS – examples Nature: 24 Science: 22 European Journal of Psychological Assessment: .96 Czechoslovak Psychology: .06 Testforum: NA h-index •Measure of author quality, comparable within fields •Started in 2005 •Intedned to be better than raw number of papers or raw number of citations... but correlates r = .90 with the raw number of papers, oops. Among all publications of an author, at least h of them received at least h citations. h-index – examples Albert Einstein: 117 Albert Bandura: 208 Eric-Jan Wagenmakers: 101 Jordan Peterson: 55 David Šmahel: 28 Stanislav Ježek: 19 Erdös number •The only measure of author quality that makes sense •The only measure of author quality that measures something real E_N = How many co-authors separate you from the mathematician Pál Erdös On 20 September 1996, at the age of 83, he had a heart attack and died while attending a conference in Warsaw.[17] These circumstances were close to the way he wanted to die. He once said, I want to be giving a lecture, finishing up an important proof on the blackboard, when someone in the audience shouts out, 'What about the general case?'. I'll turn to the audience and smile, 'I'll leave that to the next generation,' and then I'll keel over.[17] Other idiosyncratic elements of Erdős's vocabulary include:[32] •Children were referred to as "epsilons" (because in mathematics, particularly calculus, an arbitrarily small positive quantity is commonly denoted by the Greek letter (ε)). •Women were "bosses" who "captured" men as "slaves" by marrying them. Divorced men were "liberated". •People who stopped doing mathematics had "died", while people who died had "left". •Alcoholic drinks were "poison". •Music (except classical music) was "noise". •To be considered a hack was to be a "Newton". •To give a mathematical lecture was "to preach". •Mathematical lectures themselves were "sermons".[33] •To give an oral exam to students was "to torture" them. Erdös number – examples Albert Einstein: 2 Albert Bandura: NA Eric-Jan Wagenmakers: 3 Jordan Peterson: NA David Šmahel: NA Stanislav Ježek: NA Text Description automatically generated Chart, line chart Description automatically generated https://www.nature.com/articles/d41586-019-03308-7?proof=t%3B A "popular" article has been published in nature. The graph maps articles in the domains covered by nature and its "daughter" journals: - articles are mapped to citations - the likelihood is high that papers from a certain discipline are cited by their own disciplines - the lower we go, the less cross-disciplinary citation QUESTION 1: Are measures that are heavily based on citations (without a correction) comparable across disciplines? QUESTION 2: Having a look at the picture, how is psychology faring in the business? •Reason distinguishes a “person approach” to error from a “system approach” (Reason, 2000): • •He notes that “high reliability” organizations (including air traffic control centers, nuclear power plants, and nuclear aircraft carriers) are characterized by a focus on error management at the systems level more than the individual level. • •Reason's logic has also been adopted in widespread attempts to reduce medical errors and a focus on moving medicine from a “blame and shame” cultural perspective to one of a “reporting and feedback” cultural perspective, with public reporting of individual and organizational performance being crucial (Leape, 2010). • 6 core reasons for DRP •(1) career and funding pressures •(2) institutional failures of oversight, •(3) commercial conflicts of interest, •(4) inadequate training, •(5) erosion of standards of mentoring, and •(6) part of a larger pattern of social deviance. • Epic fraud: How to succeed in science (without doing any) | Ars Technica Take home message •Scientists are incentivized to engage in fraud •Metrics that incentivize them are fatally flawed •Fraud is not merely an issue of individual bad apples, but rather a larger flaw of the whole academic culture •Resisting all bad research practices is hard due to institutional (and personal ) pressures • > https://forms.gle/jL8D3pVxKddxWEGu8 Thank you for your attention! Don’t forget that you will be quizzed on the sins of Corruptibility & Bean counting next week. And please fill in the feedback form. J