Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data
详细信息 查看全文
- 作者：Michail Kovanis ; Raphaël Porcher ; Philippe Ravaud ; Ludovic Trinquart
- 关键词：Peer review ; Publishing ; Computer simulation ; Complex systems ; Agent ; based model
- 刊名：Scientometrics
- 出版年：2016
- 出版时间：February 2016
- 年：2016
- 卷：106
- 期：2
- 页码：695-715
- 全文大小：912 KB
- 参考文献：Alberts, B., Hanson, B., & Kelner, K. L. (2008). Reviewing peer review. Science, 321(5885), 15. doi:10.1126/science.1162115 .CrossRef
Allesina, S. (2012). Modeling peer review: An agent-based approach. Ideas in Ecology and Evolution, 5(2), 27–35. doi:10.4033/iee.2012.5b.8.f .CrossRef
Arns, M. (2014). Open access is tiring out peer reviewers. Nature, 515(7528), 467. doi:10.1038/515467a .CrossRef
Auchincloss, A. H., & Diez Roux, A. V. (2008). A new tool for epidemiology: The usefulness of dynamic-agent models in understanding place effects on health. American Journal of Epidemiology, 168(1), 1–8. doi:10.1093/aje/kwn118 .CrossRef
Bohannon, J. (2013). Who’s afraid of peer review? Science, 342(6154), 60–65. doi:10.1126/science.342.6154.60 .CrossRef
Bohannon, J. (2014). Google Scholar wins raves—But can it be trusted? Science, 343(6166), 14. doi:10.1126/science.343.6166.14 .CrossRef
Bonabeau, E. (2002). Agent-based modeling: Methods and techniques for simulating human systems. Proceedings of the National Academy of Sciences of the United States of America, 99(Suppl 3), 7280–7287. doi:10.1073/pnas.082080899 .CrossRef
Brown, T. (2004). Peer review and the acceptance of new scientific ideas. London: Sense About Science.
Cals, J. W., Mallen, C. D., Glynn, L. G., & Kotz, D. (2013). Should authors submit previous peer-review reports when submitting research papers? Views of general medical journal editors. The Annals of Family Medicine, 11(2), 179–181. doi:10.1370/afm.1448 .CrossRef
Chhatwal, J., & He, T. (2015). Economic evaluations with agent-based modelling: An introduction. PharmacoEconomics, 33(5), 423–433. doi:10.1007/s40273-015-0254-2 .CrossRef
Day, T. E. (2015). The big consequences of small biases: A simulation of peer review. Research Policy, 44(6), 1266–1270. doi:10.1016/j.respol.2015.01.006 .CrossRef
Epstein, J. M. (2006). Generative social science: Studies in agent-based computational modeling. Princeton, NJ: Princeton University Press.
Farmer, J. D., & Foley, D. (2009). The economy needs agent-based modelling. Nature, 460(7256), 685–686. doi:10.1038/460685a .CrossRef
Galea, S., Riddle, M., & Kaplan, G. A. (2010). Causal thinking and complex system approaches in epidemiology. International Journal of Epidemiology, 39(1), 97–106. doi:10.1093/ije/dyp296 .CrossRef
Gannon, F. (2001). The essential role of peer review. EMBO Reports, 2(9), 743. doi:10.1093/embo-reports/kve188 .CrossRef
Gura, T. (2002). Scientific publishing: Peer review, unmasked. Nature, 416(6878), 258–260. doi:10.1038/416258a .CrossRef
Herron, D. M. (2012). Is expert peer review obsolete? A model suggests that post-publication reader review may exceed the accuracy of traditional peer review. Surgical Endoscopy, 26(8), 2275–2280. doi:10.1007/s00464-012-2171-1 .CrossRef
Hopewell, S., Collins, G. S., Boutron, I., Yu, L.-M., Cook, J., Shanyinde, M., et al. (2014). Impact of peer review on reports of randomised trials published in open peer review journals: Retrospective before and after study. BMJ. doi:10.1136/bmj.g4145 .
Houry, D., Green, S., & Callaham, M. (2012). Does mentoring new peer reviewers improve review quality? A randomized trial. BMC Medical Education, 12(1), 83. doi:10.1186/1472-6920-12-83 .CrossRef
Laakso, M., Welling, P., Bukvova, H., Nyman, L., Björk, B.-C., & Hedlund, T. (2011). The development of open access journal publishing from 1993 to 2009. PLoS ONE, 6(6), e20961. doi:10.1371/journal.pone.0020961 .CrossRef
Lee, B. Y., Brown, S. T., Cooley, P., Grefenstette, J. J., Zimmerman, R. K., Zimmer, S. M., et al. (2010). Vaccination deep into a pandemic wave: Potential mechanisms for a “third wave” and the impact of vaccination. American Journal of Preventive Medicine, 39(5), e21–e29. doi:10.1016/j.amepre.2010.07.014 .CrossRef
Look, H., & Sparks, S. (2010). The value of UK HEIs contribution to the publishing process: Summary report. JISC Collections. https://www.jisc-collections.ac.uk/Global/report%20on%20HEIs%20noncash%20contribution%20to%20the%20journal%20publishing%20process%20final.pdf .
Maglio, P. P., & Mabry, P. L. (2011). Agent-based models and systems science approaches to public health. American Journal of Preventive Medicine, 40(3), 392–394. doi:10.1016/j.amepre.2010.11.010 .CrossRef
Marshall, B. D., & Galea, S. (2015). Formalizing the role of agent-based modeling in causal inference and epidemiology. American Journal of Epidemiology, 181(2), 92–99. doi:10.1093/aje/kwu274 .CrossRef
Mulligan, A., Hall, L., & Raphael, E. (2013). Peer review in a changing world: An international study measuring the attitudes of researchers. Journal of the American Society for Information Science and Technology, 64(1), 132–161. doi:10.1002/asi.22798 .CrossRef
Nandi, A., Megiddo, I., Prabhakaran, D., & Laxminarayan, R. (2013). An agent-based simulation modelling approach to extended cost-effectiveness analysis of health interventions. Lancet, 381, S96. doi:10.1016/S0140-6736(13)61350-1 .CrossRef
Nicholas, D., Watkinson, A., Jamali, H. R., Herman, E., Tenopir, C., Volentine, R., et al. (2015). Peer review: Still king in the digital age. Learned Publishing, 28(1), 15–21. doi:10.1087/20150104 .CrossRef
Paolucci, M., & Grimaldo, F. (2014). Mechanism change in a simulation of peer review: From junk support to elitism. Scientometrics, 99(3), 663–688. doi:10.1007/s11192-014-1239-1 .CrossRef
Park, I.-U., Peacey, M. W., & Munafo, M. R. (2014). Modelling the effects of subjective and objective decision making in scientific peer review. Nature, 506(7486), 93–96. doi:10.1038/nature12786 .CrossRef
Patel, J. (2014). Why training and specialization is needed for peer review: A case study of peer review for randomized controlled trials. BMC Medicine, 12, 128. doi:10.1186/s12916-014-0128-z .CrossRef
Rennie, D., & Flanagin, A. (2014). Research on peer review and biomedical publication: Furthering the quest to improve the quality of reporting. JAMA, 311(10), 1019–1020. doi:10.1001/jama.2014.1362 .CrossRef
Review Rewards. (2014). Nature. doi:10.1038/514274a . http://www.nature.com/news/review-rewards-1.16138
Rigotti, N. A., & Wallace, R. B. (2015). Using agent-based models to address “wicked problems” like tobacco use: A report from the institute of medicine using agent-based models to address tobacco use. Annals of Internal Medicine, 163(6), 469–471. doi:10.7326/m15-1567 .CrossRef
Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., & Smith, R. (2008). What errors do peer reviewers detect, and does training improve their ability to detect them? Journal of the Royal Society of Medicine, 101(10), 507–514. doi:10.1258/jrsm.2008.080062 .CrossRef
Siler, K., Lee, K., & Bero, L. (2015). Measuring the effectiveness of scientific gatekeeping. Proceedings of the National Academy of Sciences of the United States of America, 112(2), 360–365. doi:10.1073/pnas.1418218112 .CrossRef
Squazzoni, F. (2010). The impact of agent-based models in the social sciences after 15 years of incursions. History of Economic Ideas, 18(2), 197–234.
Squazzoni, F., & Gandelli, C. (2013). Opening the black box of peer review: An agent-based model of scientist behaviour. Journal of Artificial Societies and Social Simulation, 16(2), 3.CrossRef
Stahel, P. F., & Moore, E. E. (2014). Peer review for biomedical publications: We can improve the system. BMC Medicine, 12, 179. doi:10.1186/s12916-014-0179-1 .CrossRef
Thurner, S., & Hanel, R. (2011). Peer-review in a world with rational scientists: Toward selection of the average. European Physical Journal B: Condensed Matter and Complex Systems, 84(4), 707–711. doi:10.1140/epjb/e2011-20545-7 .CrossRef
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers’ recommendations: A randomised trial. BMJ, 318(7175), 23–27. doi:10.1136/bmj.318.7175.23 .CrossRef
Vespignani, A. (2012). Modelling dynamical processes in complex socio-technical systems. Nature Physics, 8(1), 32–39. doi:10.1038/nphys2160 .MathSciNet CrossRef
Ware, M. (2013). Peer review: An introduction and guide. Bristol: Publishing Research Consortium.
Ware, M., & Mabe, M. (2015). The STM report: An overview of scientific and scholarly journal publishing (4th ed.). The Hague: International Association of Scientific, Technical and Medical Publishers.
Wilson, J. (2012). Peer review: The nuts and bolts. London: Sense About Science. - 作者单位：Michail Kovanis (1) (2)
Raphaël Porcher (1) (2) (3)
Philippe Ravaud (1) (2) (3) (4) (5)
Ludovic Trinquart (1) (4)
1. INSERM U1153, 1 Place du Parvis Notre Dame, 75004, Paris, France
2. Université Paris Descartes – Sorbonne Paris Cité, Paris, France
3. Centre d’Epidémiologie Clinique, Hôpital Hôtel-Dieu, Assistance Publique-Hôpitaux de Paris, Paris, France
4. Cochrane France, Paris, France
5. Department of Epidemiology, Columbia University Mailman School of Public Health, New York, NY, USA - 刊物主题：Information Storage and Retrieval; Library Science; Interdisciplinary Studies;
- 出版者：Springer Netherlands
- ISSN：1588-2861
文摘
Scientific peer-review and publication systems incur a huge burden in terms of costs and time. Innovative alternatives have been proposed to improve the systems, but assessing their impact in experimental studies is not feasible at a systemic level. We developed an agent-based model by adopting a unified view of peer review and publication systems and calibrating it with empirical journal data in the biomedical and life sciences. We modeled researchers, research manuscripts and scientific journals as agents. Researchers were characterized by their scientific level and resources, manuscripts by their scientific value, and journals by their reputation and acceptance or rejection thresholds. These state variables were used in submodels for various processes such as production of articles, submissions to target journals, in-house and external peer review, and resubmissions. We collected data for a sample of biomedical and life sciences journals regarding acceptance rates, resubmission patterns and total number of published articles. We adjusted submodel parameters so that the agent-based model outputs fit these empirical data. We simulated 105 journals, 25,000 researchers and 410,000 manuscripts over 10 years. A mean of 33,600 articles were published per year; 19 % of submitted manuscripts remained unpublished. The mean acceptance rate was 21 % after external peer review and rejection rate 32 % after in-house review; 15 % publications resulted from the first submission, 47 % the second submission and 20 % the third submission. All decisions in the model were mainly driven by the scientific value, whereas journal targeting and persistence in resubmission defined whether a manuscript would be published or abandoned after one or many rejections. This agent-based model may help in better understanding the determinants of the scientific publication and peer-review systems. It may also help in assessing and identifying the most promising alternative systems of peer review. Keywords Peer review Publishing Computer simulation Complex systems Agent-based model