Evaluate to Evolve: the evolution of Scientific Evaluation

Reach for the clouds (Credit: Photo by Samuel Zeller on Unsplash)


Are you productive enough? Are the jobs accomplished up to the mark?


In the professional setting, these questions always cross our minds, making the imminent journey a tad turbulent!


Feedback appears as a remedy here.


Starting from a trembling toddler to a Nobel Laureate, everyone needs feedback. A wise man once said, “Feedback is the breakfast of a champion.”


In science, feedback is indispensable; they emanate in the form of peer-reviews, still as a paradigm. Or, at times a brainstorming discussion in the lab-meeting meets the need.

Feedback is necessary throughout the whole science-cycle; while laying out the plan, securing grants, publishing the findings, or safeguarding your tenure.


Research teams are units of an institution, appropriate evaluation of these units propels the overall productivity of the organization.

However, it is difficult to evaluate research consortiums like CEITEC, where a wide variety of scientific disciplines are in action. The traditional metric systems like impact factors or h-index, solely, are unsuitable to judge the research laboratories spanning across diverse disciplines (from cell biology to robotics).


Science is a potboiler, where we try to unravel something

unknown every day


There is no magic potion, but fresh ideas and a fresh workforce are the cornerstones to keep the voyage progressing.


By clinging only at the ‘established’ number-based metric system (number of publications, citation, patents, etc.), we are unfair to the young brains, who might possess remarkable visionary concepts but still not backed by these indicators.


"Not everything that counts can be counted,

and not everything that can be counted counts"

- William Bruce Cameron


The situation requires a more elaborate form of evaluation with an unbiased judgment. CEITEC has exactly matched that demand, already for the 3rd time in a row.


The 3rd Scientific Evaluation of CEITEC (Photo: Emil Gallík) 


Creating a unique formula within EU, the scientific evaluation not only judges the bibliometric competence of 48 research groups (evaluated in 2018), it gauges their cross-functionality (as a team), collaborations, vision, funding, technical expertise, and interdisciplinarity, etc. as indicators.

.....


It's Evolving!


The Metric Tide1, Report of the Independent Review of the Role of Metrics in Research Assessment and Management”, set up by stakeholders across the UK research system, proposes a model where peer review and the metric system can act conjointly.

Neither can substitute the need of the other but instead, complement.


The newer drift in Europe is trying to tackle the question, “how do we evaluate the broader impact of the researcher, with the team, contributing to the academic community, as well as the society at large?“ There is no yardstick yet, however, some countries are going in this direction.


The Netherlands is chaperoning the matter robustly. The Dutch Government, back in 2014 issued a blueprint - Vision for Science 2025 2, 3 (Wetenschaps-visie 2025), with the efficient tie between Dutch science and society, as a focal ingredient.

It suggests two methods to determine social impact of science: First, ex-post assessment, measures impact a certain period after the research is accomplished; and, ex-ante assessment, where the valuation of the probable impact is carried out before the actual research 3.  

Ex-post assessments should be performed on an institutional/organisational level, because individual projects or research groups may not result in a significant societal change.

On the other hand, ex-ante assessments could be done at all levels; a research group can identify a particular societal issue while planning a research grant and sketch out the research-path, or at the organisational level – while designing and updating the research portfolio.



“How do we evaluate the broader impact of the researcher,

contributing to the academic community,

as well as the society at large?“



The uniqueness of CEITEC


As a research consortium, this lesson of segmenting the assessment periods into many fractions, would help steer the route to greater impact, accordingly.

In CEITEC, the blend of life sciences, advanced materials and nanotechnologies could answer multi-disciplinary research questions, giving us more openings to fulfill the need of the society by tackling such challenges from different angles.



           

                

      

Scientific Evaluation : the past, present, and future 



How does our research impact the lives of citizens?


We are trying to offer tangible solutions to existing problems (e.g., a device which can detect heavy metals in water, research on plant-stress amidst the growing problem of climate-change, tackling debilitating diseases like tick-borne encephalitis and many more themes).

Additionally, we are spreading the acquired knowledge and bringing science closer to society through the regular open days at CEITEC or the fun-filled 'Science Party Brno.'

We might need to embrace more criteria to evaluate the impact of science.

A hoisted outreach of the findings should be the goal, where science communication will appear as a prominent player. Without engaging the public, the biggest section of stakeholders, dissemination of the acquired knowledge is not complete.

If we want to achieve this, we can‘t perceive it as an obligation, but try to meet a broader audience, at a common platform; currently, social media is an ideal vehicle.   


By employing different layers of altmetrics , scientometrics, and methods to impact society, the future could ask to redraw the science-evaluation-framework, where the young and vibrant CEITEC could play an important part.



References

  1. Wilsdon, J., et al. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. DOI: 10.13140/RG.2.1.4929.1363
  2. 2025 – Vision for Science choices for the future (2014). Ministry of Education, Culture and Science. Government of the Netherlands. https://www.government.nl/documents/reports/2014/12/08/2025-vision-for-science-choices-for-the-future
  3. KNAW (2018). Maatschappelijke impact in kaart (Mapping social impact), Amsterdam, KNAW. https://www.knaw.nl/en/news/publications/maatschappelijke-impact-in-kaart



Written by Somsuvro Basu

Reviewed by Markus Dettenhofer and Zdeňka Pavlačková 


Publication date: 29.03.2019