Science - Evaluation and Financing
Jan Obdržálek
Fl MU
DUVOD, October 1, 2013
MINISTRY OF EDUCATION, OPEducatlon
YOUTH AND SPORTS for Competitiveness
INVESTMENTS IN EDUCATION DEVELOPMENT
european
social fund in the czech republic
EUROPEAN UNION
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
1 122
Part I
Journal evaluation
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
2/22
Impact Factor
a defined for journals
• the ratio of the number of citations to the previous 2 years of the journal divided by the number of articles in those years
• essentially the average number of recent citations per article
• only for journals indexed in Journal Citation Reports
• accessed from the Web of Science
The formula
O A = total cites in 2012
O B = 2012 cites to articles published in 2010-11 (B c A) O C = number of articles published in 2010-11 O D = B/C = 2012 impact factor
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
3/22
Impact Factor - problems
• computed by private company (Thomson Reuters)
• items not in JCR do not count
• strongly influenced by editorial policies
European Association of Science Editors (EASE) (November 2007)
.. Journal impact factors are used only - and cautiously - for measuring and comparing the influence of entire journals, but not for the assessment of single papers, and certainly not for the assessment of researchers or research programmes
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
4/22
Impact Factor - problems
• computed by private company (Thomson Reuters)
• items not in JCR do not count
• strongly influenced by editorial policies
European Association of Science Editors (EASE) (November 2007)
.. Journal impact factors are used only - and cautiously - for measuring and comparing the influence of entire journals, but not for the assessment of single papers, and certainly not for the assessment of researchers or research programmes
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
4/22
Impact Factor - problems (contd.)
International Council for Science (ICSU) (July 2008)
Suggests considering a limit number of publications per year to be taken into consideration for each scientist, or even penalising scientists for an excessive number of publications per year.(e.g. more than 20).
Deutsche Forschungsgemeinschaft (February 2010)
Guidelines to evaluate only articles and no bibliometric information on candidates to be evaluated in all decisions concerning "...performance-based funding allocations, postdoctoral qualifications, appointments, or reviewing funding proposals, [where] increasing importance has been given to numerical indicators such as the h-index and the impact factor".
National Science Foundation (US) Research Assessment Exercise (UK)
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
5/22
Impact factor - related measures
• 5-year impact factor
• journal immediacy index- the number of citations that year to articles published the same year
• journal citing half life - the median age of the articles that were cited by the articles published in the journal that year
• journal cited half life - the median age of the articles in the journal that were cited by other journals during the year
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
6/22
Part II
Institutional evaluation
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
7/22
ČR: Hodnocení výzkumu a vývoje (VaVal)
Main characteristic
• point-based system (bobnci, kafemlejnek)
• on organizational level
• faculties evaluated separately however money awarded to university
• way of splitting money within the university is not defined Results for 2012
http://www.vyzkum.cz/FrontClanek.aspx?idsekce=650 812 Tabulka c. 5
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
8/22
ČR: Hodnocení VaVal - points table
http://www.výzkum.cz/FrontClanek.aspx?idsekce=650 022 Metodika, page 33
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DŮVOD, Fall 2013
9/22
ČR: Hodnocení VaVal - journal results
Jimp = 10 + 295 x Faktor
where
• Faktor = (1 - A/)/(1 + (A//0.057))
» N je normované pořadí časopisu, N = (P - ^/{Pmax- 1)
• P = pořadí časopisu v daném oboru podle Journal Citation Report v řadě seřazené sestupně podle IF
• Pmax = celkový počet časopisů v daném oboru dle Journal Citation Report
a Je používána hodnota IF platná v roce uplatnění výsledku, není používána hodnota IF-5.
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
10/22
ČR: Hodnocení VaVal - criticism
• point value of different results keeps changing
• leads to short-term optimization - skived results
• this needs to be corrected in the next term
• various constants, which are subject to change
• rules keep changing (e.g. software)
• tends to be used to evaluate individuals and their research
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013 11/22
UK: Research Assessment Exercise (RAE)
• approximately every 5 years
• to evaluate the quality of research undertaken by British higher education institutions
• grouped by subject (Computer Science, Chemistry ...)
• evaluated by subject specialist peer-review panel
• assessment focuses on evaluating:
• quality of research outputs (journal and conference papers) 9 research environment
• indicators of esteem
a for RAE 2008, each full-time staff member could submit at most four publications from 2001-2007
• full-time researchers not included
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
12/22
UK: Research Assessment Exercise (RAE) - results
The scale	
4*	Quality that is world-leading in terms of originality, significance and rigour
3*	Quality that is internationally excellent in terms of originality significance and
	rigour but which nonetheless falls short of the highest standards of excellence
2*	Quality that is recognized internationally in terms of originality significance and
	rigour
1*	Quality that is recognized nationally in terms of originality, significance and
	rigour
Unclassified	Quality that falls below the standard of nationally recognized work. Or work
	which does not meet the published definition of research for the purposes of
	this assessment.
Results
University	Average	%4*	% 3* plus	staff 2008	staff 2001	% staff 2001	subject
Cambridge	2.975	32.0	71.2	2040.39	1826.1	96.49%	48
Oxford	2.959	31.8	70.3	2245.83	2023.83	94.89%	48
LS E	2.957	34.9	68.4	490.36	431.57	97.19%	14
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
13/22
Part III
Personal evaluation
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
14/22
h-index
• named after Jorge E. Hirsch (physicist, UCSD)
• for an individual scientist
• measures productivity and impact of the published work The formula
• Np - the number of papers published by a scientist
a h - the number of papers that have at least h citations each
• Np - h - the number of papers with less than h citations each
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
15/22
h-index (contd.)
• useful only for comparing in the same field
• grows with academic age
• demonstrated to have high predictive value for National Academy membership or the Nobel Prize
Original Hirch's suggestion: (for physics!) 12 advancement to tenure (associate professor) 18 full professor 15-20 membership in the US National Academy of Sciences
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
16/22
Counting publications
Where to take the data from?
• Web od Science (WoS)/Web of Knowledge
• Scopus
• Google Scholar a DBLP
Problems
• completely different numbers
• errors in data
a spelling of Czech/Slovak names
• multiple people with the same name
Jan Obdržálek (Fl MU) Science - Evaluation and Financing DUVOD, Fall 2013 17/22
Part IV
Funding bodies
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
18/22
Funding bodies
ČR - primary sources
• Grantová agentura České republiky (GAČR) basic research
• Technologická agentura České republiky (TAČR) applied research
• Ministerstvo školství, mládeže a tělovýchovy (MŠMT) EU
« EU Framework Programme for Research and Technological Development
• FP7: 2007-2013, FP8/Horizon: 2014-2020
a European Research Council (ERC) grants (part of FP7)
• Starting grants (2-7 years after PhD)
• Advanced grants
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
19/22
GAČR
Project types
• Standard (2-3 years)
• Postdoc (2-3 years, within 4 years of obatining Ph.D.)
• International/bilateral (2-3 years, Germany, Korea, Taiwan)
• Centres of Excellence (7 years)
Evaluation
• expert panel
• university/AV CR researchers
• possibility of reducing the budget
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
20/22
TA ČR
Příjemci: výzkumné organizace a podniky Programy
• centra kompetence
• Program ALFA
Aplikovaný výzkum a experimentální vývoj. Oblasti: O Progresivní technologie, materiály a systémy O Energetické zdroje a ochrana a tvorba životního prostředí O Udržitelný rozvoj dopravy
a Program BETA
Výzkum, experiementální vývoj a inovace pro potřeby státní správy.
a Program OMEGA
Aplikovaný výzkum a experimentální vývoj, jejichž výsledky mají vysoký potenciál pro uplatnění v řadě oblastí celospolečenského života obyvatel České republiky.
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DUVOD, Fall 2013
21 /22
MŠMT
Typy projektů
• Operační programy
• prostředky ze strukturálních fondů EU
• OP VK - Vzdělávání pro Konkurenceschopnost (2007-2013) (IDS)
• OP VaVpl - Věda a Výzkum pro Inovace (2007-2013) (CERIT)
• Fond rozvoje vysokých škol (FRVŠ)
A Inovace a rozvoj laboratoří, ateliérů a pracovišť pro praktickou
výuku a informačních technologií ve vysokoškolském vzdělávání B Podpora pedagogické práce akademických pracovníků do 35 let G Tvůrčí práce studentů směřující k inovaci vzdělávací činnosti
• KONTAKT - mezinárodní spolupráce ve výzkumu a vývoji
• EUREKA - evropská spolupráce v aplikovaném výzkumu «...
Jan Obdržálek (Fl MU)
Science - Evaluation and Financing
DŮVOD, Fall 2013
22/22