Introduction
The Puerto Rico Clinical and Translational Research Consortium (PRCTRC) was established on September 2010 with the idea of implementing productive collaborations among local institutions and to have a broader impact across all levels of research in health disparities in Puerto Rico. Supported through the National Center for Research Resources and the National Institute of Minority Health and Health Disparities (NIMHD) of the National Institutes of Health (NIH) (U54 RR026139, P20 RR011126, U54 MD007587), the PRCTRC is a research infrastructure and capacity building partnership among the University of Puerto Rico Medical Sciences Campus (UPR-MSC), a public academic health sciences center as the leader; the Ponce Health Sciences University (PHSU), and Universidad Central del Caribe (UCC) [Reference Estape-Garrastazu, Noboa-Ramos, De Jesus-Ojeda, De Pedro-Serbia, Acosta-Perez and Camacho-Feliciano1]. PRCTRC has fostered a synergistic environment to support research across the clinical and translational research continuum in four primary clinical focus areas: HIV/AIDS, neuroscience, cardiovascular and metabolic disease, and cancer.
Research publications represent the public cumulative record of science, documenting empirical results and providing a forum for theorizing, debate, and the gradual advance of scientific knowledge [Reference Schneider, Kane and Rainwater2]. Although publications are not the end point of scientific research, it is impossible to imagine translating research into knowledge or practice without the manifestation of research in publications [Reference Schneider, Kane and Rainwater2]. The quantitative analysis of publications is known as bibliometrics. Bibliometrics can be defined as “the application of quantitative analysis and statistics to publications such as journal articles and their accompanying citation counts” [3]. Bibliometrics (sometimes called scientometrics) turns the main tool of science, quantitative analysis, on itself. The complexity is in the analysis and use of the numbers, for the statistics obtained can be understood as indicators of achievement or lack thereof [3].
Scientometric analysis provides a structured process to understand the dynamics of the sciences [Reference Börner, Glänzel, Scharnhorst and van den Besselaar4]. It is an important method of health science informatics research, which can be used for quantitative analysis of the intellectual structures and connections of scientific articles in a specific field [Reference Chen, Hu, Liu and Tseng5,Reference Zhang, Diao and Wang6]. This tool has the potential to contribute to a better understanding of the quantity, impact, and types of research that the PRCTRC supports. Collaboration between researchers is of great importance in the development of subject areas and in the dissemination of research results. It is a powerful form of interaction that allows for effective communication as well as the sharing of competence and other resources [Reference Sangam and Arali7]. Collaboration metrics can help identify where collaboration has and has not taken place [3]. Such analysis can also reveal dynamic and influential points of collaboration, as well as those that have been less influential [3].
The Clinical and Translational Science Awards (CTSA) program is one of the most important initiatives in translational medical funding [Reference Zhang, Wang and Diao8]. To provide a quantitative evaluation of the CTSA program, Zhang and collaborators [Reference Zhang, Diao and Wang6,Reference Zhang, Wang and Diao8] performed studies based on science mapping, scientometric analysis, and information extraction techniques. One of these studies quantitatively analyzed the scientific articles funded by the CTSA program. The results of the study showed that the quantitative productivities of the CTSA program had a stable increase since 2008. Also, the emerging trends of the research funded by the CTSA program covered clinical and basic medical research fields [Reference Zhang, Wang and Diao8]. The quantitative evaluation of the efficiency and performance of the CTSA program has a significant referential meaning for the decision making of global translational medical funding [Reference Zhang, Wang and Diao8]. The second study attempted to show the scientific output and impact, identify the specific core field and institute, and understand the academic status and benefit of translational medicine to evaluate quantitatively the efficiency and performance of translational medicine [Reference Zhang, Diao and Wang6]. The research showed that translational medicine had significant scientific output and impact, core field and institute, and academic status and benefit [Reference Zhang, Diao and Wang6]. Llewellyn and collaborators also evaluated publication and citation patterns for articles supported by CTSA hub investment over the first decade of the CTSA program [Reference Llewellyn, Carter, Rollins and Nehl9]. The authors collected bibliometric data from PubMed, Web of Science InCites, and NIH iCite for articles citing any CTSA hub grants published from hub inception through 2016. They compiled data on publication and citation rates, and indexes of relative citation impact aggregated by hub funding year cohort. The study showed that the CTSA program is yielding a robust and growing body of influential research findings with consistently high indices of relative citation impact [Reference Llewellyn, Carter, Rollins and Nehl9]. Another study of CTSA used complementary bibliometric approaches to assess the scope, influence, and interdisciplinary collaboration of publications supported by single CTSA hubs and those supported by multiple hubs, characterizes the CTSA consortium’s contributions to clinical and translational science, identifies content areas of strength, and provides evidence for the success of multihub collaborations. The two articles by Zhang focused on the top 10 CTSA funded institutions and the top 20 agencies for translational research and Puerto Rico was not part of that group. None of the abovementioned studies include the research at Puerto Rican institutions.
In Puerto Rico, two prior studies assessed the scientific research productivity of Puerto Rican cancer researchers, so prior bibliometric studies have been conducted but limited to cancer [Reference Ortiz, Calo, Suárez-Balseiro, Maura-Sardo and Suárez10,Reference Calo, Suárez-Balseiro, Suárez, Soto-Salgado, Santiago-Rodríguez and Ortiz11]. One of the articles presents the characteristics and trends of cancer publications in Puerto Rico’s biomedical journals and their relationship with the island’s cancer mortality [Reference Ortiz, Calo, Suárez-Balseiro, Maura-Sardo and Suárez10]. This first study limits its analysis to the papers published in the Puerto Rico Health Sciences Journal and the Boletín de la Asociación Médica de Puerto Rico from 1903 to 2005. Bibliometric indicators studied included the number of authors and references by article, first author’s institutional affiliation and country, document type, and language [Reference Ortiz, Calo, Suárez-Balseiro, Maura-Sardo and Suárez10]. A study by Calo and collaborators characterizes the trends in cancer-related research publications by authors affiliated to Puerto Rican institutions in recent decades [Reference Calo, Suárez-Balseiro, Suárez, Soto-Salgado, Santiago-Rodríguez and Ortiz11]. They used the Science Citation Index (SCI) database from 1982 to 2009. Search criteria were that the author’s affiliation field contained some institution located in Puerto Rico and that the manuscripts were related to cancer research (according to keywords from the National Cancer Institute’ cancer definition). The indexes measured in the analysis included number and type of manuscript, scientific collaboration, author’s affiliation, and journal visibility [Reference Calo, Suárez-Balseiro, Suárez, Soto-Salgado, Santiago-Rodríguez and Ortiz11].
In the current study, we are incorporating other metrics to understand the big picture of publication productivity supported by the Puerto Rico Consortium for Clinical and Translational Research (PRCTRC) using Scientometric and collaboration indicators. Our aim is to present a detailed scientific analysis of the PRCTRC impact as a generator of important contributions addressing health questions in an underserved population.
Here we present a summary of the research impact of the PRCTRC from 2010 to 2018. Measuring the research impact was done by assessment of the researchers’ performance and their contribution to the creation of knowledge and technology, using citation analysis and objective, quantitative data [3].
This examination has the potential to contribute to a richer understanding of the quantity, impact, and types of research the PRCTRC supports, providing evidence of the accomplishments and capacities of the institution.
Methods
Peer-reviewed publications published from 2010 to 2018 were retrieved from the SCI database (October 2019). Our search criteria included publications that (1) acknowledge the PRCTRC support (i.e., award numbers U54RR026139, P20RR011126, U54MD007587) and (2) have a PMID number. To try to minimize the situation about the misprint award numbers, we created an algorithm with a series of combination of the awards numbers and then we verified one by one the papers retrieved.
For this study, the Scientometric indicators included citation indicators [i.e., papers count, citation counts, h-index (HI), and average citation (AC) per item) and collaboration indicators (i.e., collaboration index (CI), degree of collaboration (DC), and collaboration coefficient (CC)]. We stand by the use of the HI, as a simple and useful way to characterize the scientific output of a researcher which has been extensively used by others to establish comparisons of scientists, scientific journals, research teams, and research institutions and countries [Reference Mester12].
The citation indicators were calculated using the Web of Science Platform and iCite [Reference Hutchins, Hoppe, Meseroll, Anderson and Santangelo13] program, and the collaboration indicators were calculated using Stata software for statistical inquiry. Joinpoint Trend Analysis Software and Excel were used to make the trends analysis for citation indicators. We also describe the principal research areas of the PRCTRC publications.
The indicators definitions are listed here [3,Reference Sangam and Arali7] (Table 1).
Table 1. Indicators definitions
The Web of Science Platform include the following databases: CCC (Current Contents Connect), WOS (Web of Science), INSPEC (Institution of Engineering and Technology), ZOOREC (Zoological Record), SCIELO (Scientific Electronic Library Online), DIIDW (Derwent Innovation Index), MEDLINE (National Library of Medicine premier life sciences database), RSCI (Russian Science Citation Index), KJD (Korean Journal Database), DRCI (Data Citation Index), BCI = Book Citation Index (Science). The Web of Science database is one of the most authoritative and multidisciplinary platforms in the world, covering nearly all the leading scholarly articles and high-quality citation data.
Results
A total of 341 peer reviewed PRCTRC supported publications were identified from 2010 to 2018 from all databases, with an average of 38 publications per year. During this period, a significant growth [annual percent change (APC): 17.76% P < 0.05] of scientific production was observed (Fig. 1). The overall HI was 31. The average number of citations per publication was 11.04. And the overall ACs per year was 376. The total number of times that all articles have been cited was 3763, but when self-citations were excluded it was 3569, for a difference of 194 publications. The relative citation ratio (RCR) showed a mean of 1.08 with a maximum of 9.77.
Fig. 1. PRCTRC publication impact over time, 2010–2018.
In terms of collaboration indicators, only three papers (0.87%) were single authored. Papers with four or more authors (87.1%) dominate the PRCTRC collaboration pattern. The average CC for the PRCTRC researchers during the studied time frame was 0.82. The highest CI with an average of 10.28 authors per paper was seen in year 2016 (Table 2).
Table 2. Quantification of publications by numbers of authors, collaboration coefficient (CC), and collaboration index (CI)
The nine-year CI in PRCTRC papers was 8.59. And the CI (grouped after 4) was 3.81. The average DC between researchers in the investigated papers was 0.99 (Table 3).
Table 3. Authorship pattern with degree of collaboration measures (DC)
The 12 principal research areas of the PRCTRC publications are: Immunology, Biochemistry/Molecular biology, Infectious Diseases, Ethnic/Studies, Geriatrics/gerontology, Pharmacology/Pharmacy, Genetics/Heredity, Health Care Sciences Services, Pediatrics, Mathematics, Oncology, Behavioral Sciences. Each of these research areas appears on at least 17% of the PRCTRC publications (Fig. 2).
Fig. 2. Top 25 PRCTRC research areas publications (341 publications).
Discussion
In this study, scientometric analysis and collaboration measures were integrated into the evaluation of the efficiency and performance of the PRCTRC. In terms of Citation Indicators, the overall HI was 31, meaning that 31 papers in the list have 31 or more citations. This number reflects the productivity of authors based on their publication and citation records. The HI reflects not just the number of papers, or the number of citations; it also indicates the number of well-cited papers [14], and it is not influenced by a single highly cited paper because the HI can be determined for any population of articles. Very productive researchers in subject areas with high volume of publications and citations can show HI values over 100 at the peak of their scientific careers. Newer researchers in smaller subject areas can have HIs under 10 [14]. The HI is a particularly simple and useful way to characterize the scientific output of a researcher but also was applied to compare scientists, scientific journals, research teams, and research institutions and countries [Reference Mester12]. In terms of influence the RCR has a mean of 1.08 with a maximum of 9.77. The RCR indicates how a publication has been cited relative to other publications in its co-citation network, and this is assumed to be reflective of the article’s area of research. Our results showed an RCR of 1.0 which means that our papers on average have received the same number of citations as would be expected based on the NIH-norm [Reference Hutchins, Hoppe, Meseroll, Anderson and Santangelo13].
According to this research, the overall quantitative productivities of the PRCTRC program had a stable increase since 2010. The PRCTRC publications underwent significant growth from 2010 to 2017 with the highest productivity peak observed during 2016. It is worth noting that during this period the PRCTRC incorporated a new service for Scientific Writing Editing. Professional Development Core’s Scientific Writing was involved in writing and editorial services for manuscripts and grants written by new/early investigators. In addition, the core provided short-term online courses and workshops to encourage and provoke new/early investigators to write and submit grant proposals and seminars targeted to clinical specialties to engage clinicians in research. We theorize that the support provided by this service could be an important factor to explain in part the fast increase during the years 2015–2016.
Unprecedented destruction from recent hurricanes in 2017, not seen since the 1930s, resulted in widespread damage and disruption of the infrastructure throughout the whole Puerto Rican territory, causing short-term and long-term impacts which will extend for many years [Reference Alcorn15,Reference Zorrilla16]. The island faced the upsetting panorama of power failures and widespread interruption of cellphone and Internet service, which lasted, depending on the geographical location, weeks or months and in some areas almost a year (September 2017–August 2018). As we reviewed the publication’s data, we notice a marked reduction of published papers during 2017–2018 period, which could be explained by the abovementioned catastrophe.
Prior studies in Puerto Rico were in cancer [Reference Ortiz, Calo, Suárez-Balseiro, Maura-Sardo and Suárez10,Reference Calo, Suárez-Balseiro, Suárez, Soto-Salgado, Santiago-Rodríguez and Ortiz11]. In the first study, a total of 369 articles were retrieved during the period of 1903 to 2005. The primary results showed that the institutions with most publications were universities (39.6%), English was the predominant publication language (72.1%), and the principal document type was original papers (69.6%). Epidemiologic studies were the dominant study type (62.1%), and the most studied cancers were digestive (15.4%) and gynecologic (9.6%). Although the publication percent has increased since 1913 (APC = 1.2%), the mortality percent increased at a faster pace (APC = 2.7%). They concluded that although a growth in the number of cancer publications is observed in these journals, it does not parallel the increase in proportional mortality [Reference Ortiz, Calo, Suárez-Balseiro, Maura-Sardo and Suárez10]. The second study showed that from 1982 to 2009, cancer-related papers authored by scientists located in Puerto Rico came to 451. Over the last three decades, the scientific production underwent significant growth (APC = 6.4%, P < 0.05) with the highest peak between 2000 and 2009 (61.4% of all articles). Universities are the local institutional sector with the highest number of authors (81.4%), and the University of Puerto Rico is the most active center in this regard (68.5%). Forty-three percent of the manuscripts (n = 195) were published in 20 journals from which 14 are observed to have high visibility when compared to similar thematic journals [Reference Calo, Suárez-Balseiro, Suárez, Soto-Salgado, Santiago-Rodríguez and Ortiz11].
The present study expands the knowledge of scientific impact and collaboration in Puerto Rico. As mentioned above, results highlight a strong scientific collaboration among PRCTRC affiliated investigators as shown by the high average CC of 0.82 and the high average DC of 0.991 that were found in our results. This could be explained by the multidisciplinary nature of the PRCTRC and of clinical and translational science [Reference Ameredes, Hellmich and Cestone17–Reference Siamaki, Geraei and Zare-Farashbandi20]. The research areas of the PRCTRC publications evaluated in this study covered clinical and basic medical research, including the PRCTRC focus areas of HIV/AIDS, neuroscience, cardiovascular and metabolic disease, and cancer.
The main innovations of this study were to evaluate the efficiency and performance of the PRCTRC by using scientometric analytic methods, author collaborations, and to reveal the quantitative productivities and the research areas preferences. As shown, the application of the scientometric indicators will allow the PRCTRC to assess the research productivity, impact, and collaboration to design and implement interventions according to the program experience and needs. This study provided a better understanding of the quantity, impact, and types of research that the PRCTRC supports and provided evidence of the accomplishments and capacities of the institution. Results allow for the assessment of the progress resulting from the provided support and to plan further strategies accordingly.
The present article contributes to the literature when considering that other academic centers would like to gauge their output and progress against a comparison center. The paper also contributes in terms of filling a research gap into the change in publication output over time within a single center. However, there are limitations of the research. The purpose of this first paper is to present a description of the behavior of the scientific productivity using bibliometrics indicators. A limitation on the presented data and results is that it does not include independent variables that would be valuable in understanding why the selected outcome metrics change over time. In future analyses, we will include comparison analyses to try to explain the changes over time. Another limitation is that our study used traditional metrics for assessing the bibliometric impact. In future studies, we will use alternative metrics for assessing impact. Other limitation of this study is that it was very difficult to calculate the indicator at author level because the authors do not have a standardized way to be cited. For future studies, we must analyses each author and group the different ways of cites for the same author.
A complete understanding of citing, publishing, and collaboration patterns in Puerto Rico is critical to researchers, policy makers, and heath care professionals in order to make informed decisions about research priorities and to guide the Puerto Rico Clinical and Translational Research Consortium. Although our study provides important information on publication patterns in the publications that mention the number of the PRCTRC grant number and that are in Web of Science database, a complete analysis of PRCTRC publications published in other databases by researchers affiliated with PRCTRC must be done. To fully understand the total spectrum of clinical and translational research in Puerto Rico, it is important that future bibliometric studies also evaluate other scientific materials such as monographs, books, and thesis dissertations and presentation [Reference Calo, Suárez-Balseiro, Suárez, Soto-Salgado, Santiago-Rodríguez and Ortiz11]. We also recommend that future bibliometric studies of clinical and translational research publications include citation analysis for authors affiliated to the PRCTRC. Finally, the scientometrics and collaboration characteristics of Puerto Rico should be compared to that of other Clinical and Translational Centers, to further understand our level of development in the area of Clinical and Translational Research. The evaluation of the scientometric measures will serve to other similar research consortia to determine areas of excellence, identify possible interventions to improve the landscape, and to define areas in which research has a room to grow.
Acknowledgments
The research reported was supported by National Institute on Minority Health and Health Disparities (NIMHD) and the National Institute of Allergy and Infectious Diseases (NIAID) of the National Institutes of Health under Award Number U54MD007587. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Disclosures
The authors have no conflicts of interest to declare.
Open access
