Critical reflections about the three 'r', or the barred Brazilian papers

Mauricio Rocha e Silva

Rocha-e-Silva M. Clinical Hospital of University of São Paulo's Medical School. Email: mrsilva36@hcnet.usp.br Phone: 55 11 3069-6235

CLINICS has recently proposed to debate the idea that QUALIS 2010 was likely to be perfected by the three-r concept (Remove review papers, Recognize other methods of evaluation, Reevaluate Brazilian papers). 1 Submitted to debate in pairs, on the occasion of the II Satellite Seminar for Full Editors (ABEC, November 2010), it has been concluded that it would rather be focused on only one 'r', namely have SCImago (cites/document) and SciELO (Impact Factor) evaluation methods recognized by CAPES for 2013.

We have previously demonstrated that ISI THOMSON Impact Factor, the only method recognized by CAPES, is virtually identical to SCImago cites/documents. The general correlation by sampling between the two indexes is bigger than 0.9 and the angular coefficient cannot be differentiated into the unit. SCImago Journal & Country 2009 Rank's2 2009 collection's recent divulgation now joins those already divulged JCR-ISI and SciELO Impact Factors for that year and allows the three indexes to be collected in real time. We emphasize that such a collate is not applied to any QUALIS table, since 2010 table is history and 2013 Table shall be decided upon in the future. In short, the collate hereinafter related to the year 2009 is offered on an argumentative basis proper to demand course correction regarding the exclusion of SCImago and SciELO indexes.

Let us move to the 2009 scenario as a simulation of a hypothetical Qualis table: In the Journal of Citations Report - ISI's 2009 version, the Brazilian representation jumped from 31 to 71 papers; however, in SCImago Journal Ranking, this representation remained invariable in 235 papers. There are, accordingly, 164 Brazilian papers (out of which 138 have an impact above zero) not present in the JCR-ISI table. Had Qualis rule been maintained in 2010, these 138 papers would fall into 'no impact factor' categories. Figure 1 shows the identity between ISI and SCImago for the 64 magazines included in both indexes: the angular unit coefficient and the high coefficient of correlation mean that, if you know one of the indexes, you can estimate the other one with a 95% probability against a chance of error under 5%.

Figure 1. Correlation between impact factors - J. Citation Reports (151 - Thomson) and SCImago (2009) for 64 Brazilian papers represented in both indexes with values above zero. The identity between measures is nearly absolute. The angular coefficient indicates a 45o angle (what means identity) and the correlation (R2 = 0,932; p <0.001) is almost perfect.

Table 1 shows the relation of theses 138 Brazilian papers with Cites/Documents SCImago > zero, but without a JCR-ISI impact factor. It should be emphasized that they are not papers with an impact clearly lower than those of JCR-ISI collection. The first four ones show an impact > 1.00, what would rank them as the best 15 in Brazil. Other 10 show an impact above 0.50, above the average of the Brazilian papers in JCR-ISI. All 138 titles would unquestionably deserve the 'impact factor' classification.

Another interesting correlation occurs between SCImago and SciELO. Firstly, it should be noted that there is a broad compliance: SCImago collection contains 235 Brazilian titles and SciElo collection has 223. Inclusions are not 100% compliant: SCImago contains 69 papers not included in SciElo collection; SciElo collection, in turn, contains 56 papers not included in the SCImago collection. This convergence is enough to reveal the quality uniformity of the papers included in SciELO collection.

Figure 2. SCImago and SciELO (2009) impact factors for 142 Brazilian papers represented in both indexes with values above zero. It is observed that when SCImago > SciELO, the difference is often big (as indicated by the vertical arrow related to Oswaldo Cruz Memorial Institute), but it is invariably reduced when SciELO > SCImago.


It is easy to assume that, for any Brazilian paper included in both collections, it must be expected that SCImago impact is higher than SCIELO because SCImago collection has 18,732 papers and there are only 759 papers in SciELO collection. Surprisingly, however, the effect of this huge disproportion between databases is smaller than expected, as you can see from the figure 2: out of the 142 Brazilian papers included in the two collections, only 88 (62%) show SciELO > SCImago and nine are equal. This inconsistency between what is expected and what is observed deserves further bibliometric study, but a reasonable hypothesis would be that the Brazilian articles cite other Brazilian articles more intensely by reasons of a deeper concern about a specific location. The correlation between the impacts (Figure 3) is similarly instructive: The angular coefficient (0.54) suggests that the average SciELO impact is only 40% lower than SCImago impact. The high ratio of correlation (r = 0.62; p <0.01), however, shows that the two measures evaluate the same parameter in very different databases.

Figure 3. Correlation between SciELO and SCImago (2009) impact factors for 142 Brazilian papers represented in both indexes with values above zero. The angular coefficient indicates a 28º angle and a medium SciELO impact 40% lower than SCImago. The significant correlation indicates that both parameters have the same measure in different databases.



In case CAPES recognizes Cites/Document SCImago, the majority of the Brazilian papers with an impact > 0 shall be rescued. Table 2, nevertheless, shows that, were the decision made at this moment, there would be 35 Brazilian papers left with a SciELO impact above zero but not included in JCR-ISI and SCImago. Here we also found minor impacts: four papers showed impacts above the average ISI collection. Not to mention the strong possibility that these 35 SciELO impacts shall overestimate what ISI or SCImago impacts would be.

From this simulation, we can conclude that we would have 173 Brazilian papers with IMPACT > ZERO regarded as 'WITH NO IMPACT' by QUALIS, in case the evaluation was made now and the 2010 criteria were repeated. We know that this 'simulation of exclusion' is not static: when Qualis table 'closes' for the next evaluation, a plenty of things shall have changed: some of them would surely be ISI-JCR, SCImago and SciELO, tables as well as (hopefully!) CAPES criteria of evaluation. But we stand that recognizing only ISIJCR Impact factor would be illogical. Hence, we understand that this simulation is the rational basis of an in-time warning to adopt new criteria.


NOTE: this editorial is published by CLINICS and it is exempt from copyright restrictions. We offer them to the Brazilian scientific papers for full or partial reproduction. We alternatively suggest that they support this idea in original editorials. Such actions shall express our politic will of exerting the republican right of petitioning before the public entity to defend what is deemed to be a legitimate and generalized demand of the Brazilian scientific editorial community.


REFERENCES

1. Rocha e Silva M. Qualis 2011-2013: os três erres. Clinics. 2010;65: 935-6.

2. SCImago. (2007). SJR - SCImago Journal & Country Rank. Retrieved January 26, 2011, from http://www.scimagojr.com.