How should evaluation be? Is a good evaluation of research also just? Towards the implementation of good evaluation

In this paper we answer the question of how evaluation should be by proposing a good evaluation of research practices. A good evaluation of research practices, intended as social practices a la MacIntyre, should take into account the stable motivations and the traits of the characters (i.e. the virtues) of researchers. We also show that a good evaluation is also just, beyond the sense of fairness, as working on good research practices implies keep into account a broader sense of justice. After that, we propose the development of a knowledge base for the assessment of 'good' evaluations of research practices to implement a questionnaire for the assessment of researchers' virtues. Although the..

Sociology of Economics

The interdisciplinarity dilemma: public versus private interests

In this paper, we investigate how the choice to conduct interdisciplinary work affects a researcher's career. Using data on 23,926 articles published by 6,105 researchers affiliated with the University of Florida in the period 2008-2013, we show that synthesizing knowledge from diverse fields pays off in terms of reputation. However, if combining too-distant research fields, the impact of a work is penalized. Moreover, research conducted balancing the contribution of different scientific fields has a negative impact on the reputation of scientists in terms of the number of citations but a positive impact on the diffusion of knowledge across other disciplines. Our findings are robust to a num..

Sociology of Economics

Paving the Road for Replications: Experimental Results from an Online Research Repository

Are users of a bibliographic database interested in learning about replications? Can we motivate them to learn? To answer these questions, we performed an experiment on a RePEc (Research Papers in Economics) website: Using randomized stratification, we allocated 324 replications and their corresponding original studies to clusters; we then drew from those clusters to select treatment and control groups. We added brightly colored tabs to the relevant webpages to alert visitors to the existence of a replication study or to the original study of a replication. We monitored traffic over three phases lasting several months: a) no treatment, b) treatment on one group, c) treatment on both groups. ..

Sociology of Economics

The Impact of Research Independence on PhD Students' Careers: Large-scale Evidence from France

This study investigates the effect of research independence during the PhD period on students' career outcomes. We use a unique and detailed dataset on the French population of STEM PhD students who graduated between 1995 and 2013. To measure research independence, we compare the PhD thesis content with the supervisor's research. We employ advanced neural network text analysis techniques evaluating the similarity between student's thesis abstract and supervisor's publications during the PhD period. After exploring which characteristics of the PhD training experience and supervisor explain the level of research similarity, we estimate how similarity associates with the likelihood of pursuing ..

Sociology of Economics

Women in Science. Lessons from the Baby Boom

How do children affect women in science? We investigate this question using rich biographical data, linked with patents and publications, for 83,000 American scientists in 1956 at the height of the baby boom. Our analyses reveal a unique life-cycle pattern of productivity for mothers. While other scientists peak in their mid-thirties, mothers become more productive after age 35 and maintain high productivity in their 40s and 50s. Event studies show that the output of mothers increases after 15 years of marriage, while other scientists peak in the first 10 years. Differences in the timing of productivity have important implications for tenure and participation. Just 27% of mothers who are aca..

Sociology of Economics

Right to Research and Copyright Law: From Photocopying to Shadow Libraries

Academic research and publishing are facing a crisis. The importance of access to academic literature in an interconnected world, the ever-growing cost of subscriptions to this literature, different revenue models of journals, and reducing or stagnant library budgets are pushing research and the academic community to find alternatives for academic research and publications. In its 25 years of existence, the open access movement and models which sought to contain the crisis have become the subject of considerable criticism. At the same time, a significant portion of academic literature remains locked behind steep paywalls. This has led to the growth of pirate websites and shadow libraries whi..

Sociology of Economics

Why is the statistical revolution not progressing? Vicious cycle of the scientific reform

To analyze the results of the research, behavioral scientists widely use a statistical rule that sets the significance level to 0.05. Recently, two recommendations on how to improve statistical inference were published: to redefine statistical significance to 0.005, and to select and justify the alpha. We analyzed the empirical work that cited the original recommendation papers, as well as the papers published by the scientist that co-authored the publications. About half of the numerous papers citing these recommendations adhered to them already in the first year since their publication. What is striking, the original authors that proposed the recommendations followed their own recommendati..

Sociology of Economics

What makes a journal questionable? An analysis using China’s early-warning list

There is widespread agreement that questionable journals pose a threat to the integrity of scholarly publishing and the credibility of academic research. However, there is currently no agreed upon definition of what constitutes a questionable journal. The characteristics of questionable journals have not been delineated, standardized, nor broadly accepted. A series of policy initiatives by the central Chinese government has culminated in the now Early Warning List of International Journals, released by the National Science Library of the Chinese Academy of Sciences – 65 journals that Chinese scholars should be wary of publishing in. Taking this List as a litmus test, we analyze the charact..

Sociology of Economics

The impact of the COVID-19 pandemic on academic productivity

'Publish or perish' is an expression describing the pressure on academics to consistently publish research to ensure a successful career in academia. With a global pandemic that has changed the world, how has it changed academic productivity? Here we show that academics are posting just as many publications on the arXiv pre-print server as if there were no pandemic: 168,630 were posted in 2020, a +12.6% change from 2019 and $+1.4\sigma$ deviation above the predicted 162,577 $\pm$ 4,393. However, some immediate impacts are visible in individual research fields. Conference cancellations have led to sharp drops in pre-prints, but laboratory closures have had mixed effects. Only some experimenta..

Sociology of Economics

Similar-to-me effects in the grant application process: Applicants, panelists, and the likelihood of obtaining funds

We analyse if and how the characteristics of grant research panels affect the applicants' likelihood of obtaining funding and, especially, if particular types of panels favor particular types of applicants. We use the award decisions of the UK's Engineering and Physical Sciences Research Council (EPSRC). We show that not only applicants' but also panels' characteristics matter. Panels of higher quality, in terms of prior research performance, for instance, as well panels that include more female members or members of Mongoloid origin, are tougher than others. Our main results indicate that panel members tend to favor more (or penalise less) applicants with similar characteristics to them, as..

Sociology of Economics

Pluralist Economics as a Democratizing Force: A Review Essay about The Routledge Handbook of Heterodox Economics and Democratizing the Economics Debate: Pluralism and Research Evaluation

The Routledge Handbook of Heterodox Economics and Democratizing the Economics Debate: Pluralism and Research Evaluation, two recently published books about heterodox economics and its role in broader academic and policy discourses, serve as an antidote to some recent popular narratives equating economics and economists with policies that are inherently pro-market, anti-regulation, and based in neoclassical theories. These texts illuminate challenges in current economic discourse about (1) the place of economic pluralism, (2) the role economics and economists should play in guiding policy relative to other social science disciplines, and (3) the consequences of the reliance of policy-makers o..

Sociology of Economics

Changes in Women's Representation in Economics: New Data from the AEA Papers and Proceedings

The shortage of women and historically underrepresented racial and ethnic groups in the economics profession has received considerable public attention in the past several years. The American Economic Association (AEA), the professional organization for economists, has been taking steps to address criticism that the economics discipline is unwelcoming to women and underrepresented minorities.

Sociology of Economics

Gender differences among active reviewers: an investigation based on Publons

Peer review of scientific manuscripts before publication is essential in scholarly publishing, and most active researchers hold relationships with a number of journals as both an author and a reviewer. There have been several studies focusing on gender balance in academic research and authorship, but fewer studies on our role as reviewers. Publons is a commercial website run by Clarivate Analytics that allows researchers to track and verify their peer review activities and be recognized for it. The platform features over 2 million researchers and 6.9 million reviews for more than 5,000 partnered journals, listing the most active reviewers as “top reviewers”. Our study focuses on gender r..

Sociology of Economics

Fifty shades of QE: comparing findings of central bankers and academics

We compare the findings of central bank researchers and academic economists regarding the macroeconomic effects of quantitative easing (QE). We find that central bank papers find QE to be more effective than academic papers do. Central bank papers report larger effects of QE on output and inflation. They also report QE effects on output that are more significant, both statistically and economically, and they use more positive language in the abstract. Central bank researchers who report larger QE effects on output experience more favorable career outcomes. A survey of central banks reveals substantial involvement of bank management in research production. JEL Classification: A11, E52, E58, G..

Sociology of Economics

Publishing Economics: How Slow? Why Slow? Is Slow Productive? Fixing Slow?

Publishing in economics proceeds much more slowly on average than in the natural sciences, and more slowly than in other social sciences and finance. It is even relatively slower at the extremes. We demonstrate that much of the lag, especially at the extremes, arises from authors' dilatory behavior in revising their work. The marginal product of an additional round of re-submission at the top economics journals is productive of additional subsequent citations; but conditional on re-submission, journals taking more time is not productive, and authors spending more time is associated with reduced scholarly impact. We offer several proposals to speed up the publication process. These include no..

Sociology of Economics

Congestion on the Information Superhighway: Does Economics Have a Working Papers Problem?

Publishing takes a long time in economics. Consequently, many authors release “working” versions of their papers. Using data on the NBER working paper series, we show that the dissemination of economics research suffers from an overcrowding problem: An increase in the number of weekly released working papers on average reduces downloads, abstract views, and media attention for each paper. Subsequent publishing and citation outcomes are harmed as well. Furthermore, descriptive evidence on viewership and downloads suggests working papers significantly substitute for the dissemination function of publication. These results highlight inefficiencies in the dissemination of economic research e..

Sociology of Economics

Auctions and Prediction Markets for Scientific Peer Review

Peer reviewed publications are considered the gold standard in certifying and disseminating ideas that a research community considers valuable. However, we identify two major drawbacks of the current system: (1) the overwhelming demand for reviewers due to a large volume of submissions, and (2) the lack of incentives for reviewers to participate and expend the necessary effort to provide high-quality reviews. In this work, we adopt a mechanism-design approach to propose improvements to the peer review process. We present a two-stage mechanism which ties together the paper submission and review process, simultaneously incentivizing high-quality reviews and high-quality submissions. In the fir..

Sociology of Economics

Is there a differentiated gender effect of collaboration with supercited authors? Evidence from early-career economists

Several inequalities between genders have been reported over the last decades in academia. Female researchers tend to have a lower pay, write fewer articles and receive fewer cites than their male counterparts, among other disparities. Co-authorship with highly cited scholars tend to give an advantage to early career researchers. Indeed, the impact of researchers that collaborate with super-cited (SC) authors at their early career stage tends to be greater than for those scientists who do not. The question of whether this advantage is favors male or female scientists has not been addressed yet. By conditioning on career length (at least ten years), we study the effect on male and female econ..

Sociology of Economics

Gender differences in the aims and impacts of research

This study uses mixed methods – classical citation analysis, altmetric analysis, a survey with researchers as respondents, and text analysis of the abstracts of scientific articles – to investigate gender differences in the aims and impacts of research. We find that male researchers more often value and engage in research mainly aimed at scientific progress, which is more cited. Female researchers more often value and engage in research mainly aimed at contributing to societal progress, which has more abstract views (usage). The gender differences are observed among researchers who work in the same field of research and have the same age and academic position. Our findings have implicati..

Sociology of Economics

Grounded in Methodology, Certified by Journals: The Rise and Evolution of a Mainstream in Economics

In this paper, we introduce a new understanding of the mainstream notion in economics. Its distinct character is based on a set of methodological standards deemed compulsory in the theoretical or empirical practice of the discipline. We contend that a theoretical mainstream arose around the 1980s, when the prevailing methodological standards in microeconomics and game theory – mathematical language, equilibrium discipline, and ‘explicit micro-foundations’ – came to be adopted in theoretical papers across a wide range of fields and specializations. We further argue that the 1990 period witnessed the surge of a distinct empirical mainstream and the emergence of a joint mainstream, the ..

Sociology of Economics

Article Processing Charges based publications: to which extent the price explains scientific impact?

The present study aims to analyze relationship between Citations Normalized Score (NCS) of scientific publications and Article Processing Charges (APCs) amounts of Gold Open access publications. To do so, we use APCs information provided by OpenAPC database and citations scores of publications in the Web of Science database (WoS). Database covers the period from 2006 to 2019 with 83,752 articles published in 4751 journals belonging to 267 distinct publishers. Results show that contrary to this belief, paying dearly does not necessarily increase the impact of publications. First, large publishers with high impact are not the most expensive. Second, publishers with the highest APCs are not nec..

Sociology of Economics

What’s Worth Knowing? Economists’ Opinions about Economics

We document economists, opinions about what is worth knowing and ask (i) which research objectives economic research should embrace and (ii) which topics it should study. Almost 10,000 economic researchers from all fields and ranks of the profession participated in our global survey. Detailed bibliometric data show that our sample represents the population of economic researchers who publish in English. We report three main findings. First, economists' opinions are vastly heterogeneous. Second, most researchers are dissatisfied with the status quo, in terms of both research topics and objectives. Third, on average, respondents think that economic research should become more policy-relevant, ..

Sociology of Economics

What’s Worth Knowing? Economists’ Opinions about Economics

We document economists’ opinions about what is worth knowing and ask (i) which research objectives economic research should embrace and (ii) which topics it should study. Almost 10,000 economic researchers from all fields and ranks of the profession participated in our global survey. Detailed bibliometric data show that our sample represents the population of economic researchers who publish in English. We report three main findings. First, economists’ opinions are vastly heterogeneous. Second, most researchers are dissatisfied with the status quo, in terms of both research topics and objectives. Third, on average, respondents think that economic research should become more policy-releva..

Sociology of Economics

What's Worth Knowing? Economists' Opinions about Economics

We document economists' opinions about what is worth knowing and ask (i) which research objectives economic research should embrace and (ii) which topics it should study. Almost 10,000 economic researchers from all fields and ranks of the profession participated in our global survey. Detailed bibliometric data show that our sample represents the population of economic researchers who publish in English. We report three main findings. First, economists' opinions are vastly heterogeneous. Second, most researchers are dissatisfied with the status quo, in terms of both research topics and objectives. Third, on average, respondents think that economic research should become more policy-relevant, ..

Sociology of Economics

What is the Media Impact of Research in Economics?

Many research institutions aim to have a strong public impact but little evidence exists on the extent to which research findings reach a wider audience. Using a large sample of studies released in the working paper series of the National Bureau of Economic Research, I identify online coverage of research findings in 6 major news outlets. The analysis shows significant coverage rates in most newspapers in the first month after study release. Overall, about every 11th working paper is covered at least once during this period. I also find that media reporting is correlated with several author and study characteristics. While differences in coverage between most research areas are modest, empir..

Sociology of Economics

Academic engagement with industry: the role of research quality and experience

This work explores the role of university department characteristics in academic engagement with industry. In particular, we investigate the role played by research quality and previous experience in academic engagement across different scientific disciplines. We test our hypotheses on a dataset of public sponsored university-industry partnerships in the United Kingdom, combined with data from the UK Research Assessment Exercises 2001 and 2008. Our analysis reveals a negative link between academic quality and the level of engagement with industry for departments in the basic sciences and a positive relationship for departments in the applied sciences. Our results further show that the role o..

Sociology of Economics

Refuse to be a fool

Science has been held on high moral ground, and precisely because of that, researchers have to try to satisfy everyone. They are universities, funders, industries, journals, publishers, editors, reviewers, and even the public. Universities and funders often demand prestigious studies in prestigious journals. Meanwhile, journals want submitted manuscripts to be novel and original, despite the fact that it could be the 54,560th papers to write about a topic, and the journal is one in a few hundred journals in the same discipline. Editors and reviewers can be more personal with their intentions, but the infamous Reviewer 2 will always be around the corner. The industries will probably pay you h..

Sociology of Economics

Gender Differences in Peer Recognition by Economists

We study the selection of Fellows of the Econometric Society, using a new data set of publications and citations for over 40,000 actively publishing economists since the early 1900s. Conditional on achievement, we document a large negative gap in the probability that women were selected as Fellows in the 1933-1979 period. This gap became positive (though not statistically significant) from 1980 to 2010, and in the past decade has become large and highly significant, with over a 100% increase in the probability of selection for female authors relative to males with similar publications and citations. The positive boost affects highly qualified female candidates (in the top 10% of authors) wit..

Sociology of Economics

Gender Differences in Peer Recognition by Economists

We study the selection of Fellows of the Econometric Society, using a new data set of publications and citations for over 40,000 actively publishing economists since the early 1900s. Conditional on achievement, we document a large negative gap in the probability that women were selected as Fellows in the 1933-1979 period. This gap became positive (though not statistically significant) from 1980 to 2010, and in the past decade has become large and highly significant, with over a 100% increase in the probability of selection for female authors relative to males with similar publications and citations. The positive boost affects highly qualified female candidates (in the top 10% of authors) wit..

Sociology of Economics

Game theory and scholarly publishing: premises for an agreement around open access

Actors in research and scientific publishing are gradually joining the Open-Access (OA) movement, which is gaining momentum to become nowadays at the heart of scientific policies in high-income countries. The rise of OA generates profound changes in the chain of production and dissemination of knowledge. Free access to peer-reviewed research methods and results has contributed to the dynamics of science observed in recent years. The modes of publication and access have also evolved; the classic model, based on journal subscriptions is gradually giving way to new economic models that have appeared with the arrival of OA. The objective of this article is twofold. First, propose a model for the..

Sociology of Economics