A propensity score approach in the impact evaluation on scientific production in Brazilian biodiversity research: the BIOTA Program

详细信息    查看全文

• 作者：Fernando A. B. Colugnati ; Sergio Firpo ; Paula F. Drummond de Castro…
• 关键词：Quasi ; experiment ; Propensity score ; Impact evaluation ; Biota program ; Bibliometrics ; 62P25
• 刊名：Scientometrics
• 出版年：2014
• 出版时间：October 2014
• 年：2014
• 卷：101
• 期：1
• 页码：85-107
• 全文大小：287 KB
• 参考文献：1. Abadie, A. (2005). Semiparametric difference-in-differences estimators. / The Review of Economic Studies, / 72(1), 1-9. p://dx.doi.org/10.1111/0034-6527.00321" target="_blank" title="It opens in new window">CrossRef
  2. Azoulay, P., Zivin, J. S. G., & Manso, G. (2011). Incentives and creativity: Evidence from the academic life sciences. / RAND Journal of Economics, / 42(3), 527-54. p://dx.doi.org/10.1111/j.1756-2171.2011.00140.x" target="_blank" title="It opens in new window">CrossRef
  3. B?hmer, S., & von Ins, M. (2009). Different—Not just by label: Research-oriented academic careers in Germany. / Research Evaluation, / 18(3), 177-84. p://dx.doi.org/10.3152/095820209X466865" target="_blank" title="It opens in new window">CrossRef
  4. Carney, J., Smith, W., Parsad, A., Johnston, K., & Millsap, M. (2008). / Evaluation of the Faculty Early Career Development (CAREER) Program. Bethesda, MD: Abt Associates Inc.
  5. Chen, S., Mu, R., & Ravallion, M. (2008). Are there lasting impacts of aid to poor areas? Evidence for rural China. / Policy Research Working Paper 4084, World Bank, Washington, DC.
  6. Crump, R. K., Hotz, J., Imbens, W., & Mitinik, O. A. (2009). Dealing with limited overlap in estimation of average treatment effects. / Biometrika, / 96(1), 187-99. p://dx.doi.org/10.1093/biomet/asn055" target="_blank" title="It opens in new window">CrossRef
  7. D’Agostino, R. B, Jr, & Rubin, D. B. (2000). Estimating and using propensity scores with partially missing data. / Journal of the American Statistical Association, / 95, 749-59. p://dx.doi.org/10.1080/01621459.2000.10474263" target="_blank" title="It opens in new window">CrossRef
  8. Ferraro, P. J. (2009). Counterfactual thinking and impact evaluation in environmental policy. / New Directions for Evaluation 122 (Special Issue: Environmental Program and Policy Evaluation: Addressing Methodological Challenges).
  9. Ferraro, P. J., & Pattanayak, S. K. (2006). Money for nothing? A call for empirical evaluation of biodiversity conservation investments. / PLoS Biology, / 4(4), e105. doi:pan class="a-plus-plus non-url-ref">10.1371/journal.pbio.0040105 . p://dx.doi.org/10.1371/journal.pbio.0040105" target="_blank" title="It opens in new window">CrossRef
  10. Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. / Annals of Statistics, / 29(5), 1189-232. p://dx.doi.org/10.1214/aos/1013203451" target="_blank" title="It opens in new window">CrossRef
  11. Frondel, M., & Schmidt, C. M. (2005). Evaluating environmental programs: The perspective of modern evaluation research. / Ecological Economics, / 55(4), 515-26. p://dx.doi.org/10.1016/j.ecolecon.2004.12.013" target="_blank" title="It opens in new window">CrossRef
  12. Hastie, T., Tibshirani, R., & Friedman, J. (2001). / The elements of statistical learning. New York: Springer. p://dx.doi.org/10.1007/978-0-387-21606-5" target="_blank" title="It opens in new window">CrossRef
  13. Hirano, K., Imbens, G. W., & Ridder, G. (2003). Efficient estimation of average treatment effects using the estimated propensity score. / Econometrica, / 71, 1161-189. p://dx.doi.org/10.1111/1468-0262.00442" target="_blank" title="It opens in new window">CrossRef
  14. Hornbostel, S., B?hmer, S., Klingsporn, B., Neufeld, J., & von Ins, M. (2009). Funding of young scientist and scientific excellence. / Scientometrics, / 79(1), 171-90. p://dx.doi.org/10.1007/s11192-009-0411-5" target="_blank" title="It opens in new window">CrossRef
  15. Hoshino, T. (2008). A Bayesian propensity score adjustment for latent variable modeling and MCMC algorithm. / Computational Statistics & Data Analysis, / 52(3), 1413-429. p://dx.doi.org/10.1016/j.csda.2007.03.024" target="_blank" title="It opens in new window">CrossRef
  16. Ihaka, R., & Gentleman, R. (1996). R: A language for data analysis and graphics. / Journal of Computational and Graphical Statistics, / 5(3), 299-14.
  17. Inglesi-Lotz, R., & Pouris, A. (2011). Scientometric impact assessment of a research policy instrument: The case of rating researchers on scientific outputs in South Africa. / Scientometrics, / 88(3), 747-60. p://dx.doi.org/10.1007/s11192-011-0440-8" target="_blank" title="It opens in new window">CrossRef
  18. Jacob, B. A., & Lefgren, L. (2011). The impact of NIH postdoctoral training grants on scientific productivity. / Research Policy, / 40, 864-74. p://dx.doi.org/10.1016/j.respol.2011.04.003" target="_blank" title="It opens in new window">CrossRef
  19. Jalan, J., & Ravallion, M. (2003). Estimating the benefit incidence of an antipoverty program by propensity-score matching. / Journal of Business and Economic Statistics, / 21(1), 19-0. p://dx.doi.org/10.1198/073500102288618720" target="_blank" title="It opens in new window">CrossRef
  20. Joly, C. A., Rodrigues, R. R., Metzger, J. P., Haddad, C. F. B., Verdade, L. M., Oliveira, M. C., et al. (2010). Biodiversity conservation research, training, and policy in S?o Paulo. / Science, / 328, 1358-359. p://dx.doi.org/10.1126/science.1188639" target="_blank" title="It opens in new window">CrossRef
  21. Khandker, S. R., Koolwal, G. B., & Samad, H. A. (2009). / Handbook on impact evaluation quantitative methods and practices. Washington, DC: World Bank. p://dx.doi.org/10.1596/978-0-8213-8028-4" target="_blank" title="It opens in new window">CrossRef
  22. Liu, X., Zhang, L., & Hong, S. (2011). Global biodiversity research during 1900-009: A bibliometric analysis. / Biodiversity Conservation, / 20(807), 807-26. p://dx.doi.org/10.1007/s10531-010-9981-z" target="_blank" title="It opens in new window">CrossRef
  23. Lynch, L., Gray, W., & Geoghegan, J. (2007). Are farmland preservation program easement restrictions capitalized into farmland prices? What can a propensity score matching analysis tell us? / Applied Economic Perspectives and Policy, / 29(3), 502-09.
  24. McCarey, D., Ridgeway, G., & Morral, A. (2004). Propensity score estimation with boosted regression for evaluating adolescent substance abuse treatment. / Psychological Methods, / 9(4), 403-25. p://dx.doi.org/10.1037/1082-989X.9.4.403" target="_blank" title="It opens in new window">CrossRef
  25. Mena-Chalco, J. P., & Cesar, R. M, Jr. (2009). scriptLattes: An open-source knowledge extraction system from the Lattes platform. / Journal of the Brazilian Computer Society, / 15(4), 31-9. p://dx.doi.org/10.1007/BF03194511" target="_blank" title="It opens in new window">CrossRef
  26. Mutz, R., & Daniel, H.-D. (2012). The generalized propensity score methodology for estimating unbiased journal impact factors. / Scientometrics, / 92(2), 377-90. p://dx.doi.org/10.1007/s11192-012-0670-4" target="_blank" title="It opens in new window">CrossRef
  27. Nelder, J., & Wedderburn, R. (1972). Generalized linear models. / Journal of the Royal Statistical Society. Series A (General), / 135(3), 370-84. p://dx.doi.org/10.2307/2344614" target="_blank" title="It opens in new window">CrossRef
  28. Ravallion, M. (2008). Evaluating anti-poverty programs. In T. P. Schultz & J. Strauss (Eds.), / Handbook of development economics (4th ed., pp. 3787-846). Amsterdam: North-Holland.
  29. Ridgeway, G. (1999). The state of boosting. / Computing Science and Statistics, / 31, 172-81.
  30. Ridgeway, G. (2005). GBM 1.5 package manual. p://cran.r-project.org/doc/packages/gbm.pdf" class="a-plus-plus">http://cran.r-project.org/doc/packages/gbm.pdf.
  31. Ridgeway, G. (2006). Assessing the effect of race bias in post-traffic stop outcomes using propensity scores. / Journal of Quantitative Criminology, / 22(1), 1-9. p://dx.doi.org/10.1007/s10940-005-9000-9" target="_blank" title="It opens in new window">CrossRef
  32. Ridgeway, G., McCaffrey, D., & Morral, A. (2006). twang: Toolkit for weighting and analysis of nonequivalent groups. Software for using matching methods in R. Available at p://cran.r-project.org/web/packages/twang/index.html" class="a-plus-plus">http://cran.r-project.org/web/packages/twang/index.html.
  33. Robins, J. M., & Rotnitzky, A. (2001). Comment on “inference for semiparametric models: Some questions and an answer-by P. Bickel and J. Kwon. / Statistica Sinica, / 11, 920-36.
  34. Rodrigues, R. R., & Bonomi, V. L. R. (Eds.). (2008). / Diretrizes para conserva??o e restaura??o da biodiversidade no estado de S?o Paulo. S?o Paulo: SMA-SP & FAPESP.
  35. Rosenbaum, P., & Rubin, D. (1983). The central role of the propensity score in observational studies for causal effects. / Biometrika, / 70(1), 41-5. p://dx.doi.org/10.1093/biomet/70.1.41" target="_blank" title="It opens in new window">CrossRef
  36. Rubin, D. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. / Journal of Educational Psychology, 66(5), 688-01.
  37. Rubin, D. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. / Health Services and Outcomes Research Methodology, / 2, 169-88. p://dx.doi.org/10.1023/A:1020363010465" target="_blank" title="It opens in new window">CrossRef
  38. Salles-Filho, S., Avila, F. D., Sepulveda, J., & Colugnati, F. A. B. (2010a). Multidimensional assessment of technology and innovation programs: The impact evaluation of INCAGRO-Peru. / Research Evaluation, / 19(5), 361-72. p://dx.doi.org/10.3152/095820210X12904313498360" target="_blank" title="It opens in new window">CrossRef
  39. Salles-Filho, S., Bonacelli, M. B. M., Zackiewicz, M., Castro, P. F. D., & Bin, A. (2007). Development and application of a methodology for evaluating S,T&I Programs: The decomposition method. In / Workshop Internacional sobre Inova??es Metodológicas na Avalia??o de Impacto da pesquisa Agropecuária, 2008, Brasília, XII Seminário Latino-Iberoamericano de Gestión Tecnológica - / ALTEC 2007, Buenos Aires, 2007 (Vol. 1, pp. 2-).
  40. Salles-Filho, S., Castro, P. F. D., Bonacelli, M. B. M., Zeitoum, C., & Sá, F. P. (2010b). Indicators for evaluation of results and impacts of research program in conservation and sustainable use of biodiversity, the case of BIOTA/FAPESP. In / International conference: Getting post 2010 biodiversity targets right, Bragan?a Paulista.
  41. Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. / Statistical Science, / 25(1), 1-1. p://dx.doi.org/10.1214/09-STS313" target="_blank" title="It opens in new window">CrossRef
  42. White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. / Econometrica, / 48, 817-30. p://dx.doi.org/10.2307/1912934" target="_blank" title="It opens in new window">CrossRef
  
• 作者单位：Fernando A. B. Colugnati (1) (4)
  Sergio Firpo (2)
  Paula F. Drummond de Castro (1)
  Juan E. Sepulveda (3)
  Sergio L. M. Salles-Filho (1)
  
  1. Laboratory of Studies on the Organization of Research and Innovation -GEOPI, Department of Science and Technology Policy, Institute of Geosciences, University of Campinas, Caixa Postal: 6152, Campinas, SP, CEP 13083-970, Brazil
  4. Department of Clinical Medicine, Federal University of Juiz de Fora, Rua José Louren?o, 1300 - Cid Universitária, Juiz de Fora, MG, 36036-330, Brazil
  2. S?o Paulo School of Economics/FGV, R. Itapeva 474 - Bela Vista, ?S?o Paulo, SP, 01332-000, Brazil
  3. Institute of Economics/UNICAMP, R. Pitágoras, 353 - Bar?o Geraldo, Campinas, SP, 13083-857, Brazil
  
• ISSN：1588-2861

文摘

Evaluation has become a regular practice in the management of science, technology and innovation (ST&I) programs. Several methods have been developed to identify the results and impacts of programs of this kind. Most evaluations that adopt such an approach conclude that the interventions concerned, in this case ST&I programs, had a positive impact compared with the baseline, but do not control for any effects that might have improved the indicators even in the absence of intervention, such as improvements in the socio-economic context. The quasi-experimental approach therefore arises as an appropriate way to identify the real contributions of a given intervention. This paper describes and discusses the utilization of propensity score (PS) in quasi-experiments as a methodology to evaluate the impact on scientific production of research programs, presenting a case study of the BIOTA Program run by FAPESP, the State of S?o Paulo Research Foundation (Brazil). Fundamentals of quasi-experiments and causal inference are presented, stressing the need to control for biases due to lack of randomization, also a brief introduction to the PS estimation and weighting technique used to correct for observed bias. The application of the PS methodology is compared to the traditional multivariate analysis usually employed.