Print this page

NWO Institutes prepare for SEP evaluation

In the last quarter of this year, all nine NWO Institutes will receive an evaluation visit from a special committee. This so-called SEP evaluation (Strategy Evaluation Protocol) takes place once every six years.
"It gives the institutes the opportunity to demonstrate what they do and, in particular, how they carry out their research. The evaluations help us to look ahead," says Peter Spijker, Head Strategic Support and Chair Management Team at the NWO-I Office. He explains the process.

No more scores

Why does NWO-I conduct these evaluations* once every six years? Spijker explains that the most important reason is that these evaluations help the institutes to improve. The aim is for every institute to gain insight into the best course to follow by reflecting on the past and by looking ahead. This evaluation is different from previous SEP evaluations: in the past, institutes were assessed with scores for, amongst other things, research quality and relevance. However, now that is no longer the case in the Netherlands.
Spijker: "In the past, using scores meant there was too strong a focus on obtaining the highest possible score. Now, each institute is assessed on its own merits using the quality criteria stated in the protocol, and each institute receives critical and useful feedback".
*The SEP protocol is an evaluation guideline used not only for the NWO Institutes but also for all Dutch universities. For further information, please see the NWO website.

Submitting numerous lists

The route to the final assessment by the committee is not simple. "Preparing for the evaluation involves an awful lot of work," says Spijker. "Institutes must provide a considerable amount of information and statistics that are interpreted with a bibliometric analysis. One example is output data such as publications. The analyses were realised under considerable time pressure. All of the analysed data provided input for the self-evaluation. Each institute had to compile this information by July of this year so that the evaluation committees have sufficient time to read it. Therefore, the institutes are to be commended for managing to complete this task on time".

Each Institute its own committee

Spijker explains that each institute will receive its own international committee made up of six people. The institutes make suggestions for the composition of the committees, which are mainly made up of researchers but sometimes also contain representatives from industry or people with a position in society. The criteria for participating in a committee are that there are no conflicts of interest with the institute concerned and that a member is deemed to possess expertise concerning the broader research field in which the institute is active. In addition, experience in specific areas is desired as well, such as knowledge about Open Science or the training of PhD students. Each institute submits a proposal for committee members to the NWO-I board, and the committee is appointed in close consultation.

Self-evaluation report

The committee uses the institute’s self-evaluation report to prepare for the site visit. That report provides a review of the previous six years, the institute’s strategy and an evaluation of the advice provided by the committee during the previous evaluation. Each institute also outlines its strategy for the coming period of six years. Spijker: "The institutes take a good look in the mirror and are critical about their own functioning. All nine self-evaluation reports have been approved by the NWO-I board. Each committee is asked to reflect on three subjects: research quality, societal relevance and viability. The NWO-I board added four other themes to this list".

The subjects of the SEP evaluation:
1) Research quality: scientific relevance of the research, academic reputation and leadership within the field. Position in the (inter) national research landscape.
2) Societal relevance: in terms of impact, public engagement and uptake of the research in economic, social, cultural, educational or other relevant terms.
3) Viability: the extent to which the goals will remain scientifically and societally relevant in the coming six years. Objectives, strategy and the foresight of the leadership, including management and resources. The national role.
4.1) Open Science: openness towards third parties. Open Access and FAIR data policy, and a software policy.
4.2) PhD policy and training: supervision and training of PhDs.
4.3) Academic culture: openness, (social) safety, diversity and inclusion in the research environment. Policy for research integrity.
4.4) Human resources policy: talent management, recruitment policy and training and development.

All facets of the Institute

The site visits form the climax of the evaluation. During a period of two to three days - each institute determines that independently - the committee will be shown all facets of the institute. Spijker: "Of course, the institutes prepare well for this so that they give as good an impression of themselves as possible. Every employee is involved in this process. A real buzz develops as if the king is coming to visit. The committee visits each research group, but who it speaks to differs per institute. The committee values informal conversations with a broad cross-section of employees so that it can form a picture of the institute's culture. The aim is an honest story".

Institute

Site-visit

NIOZ

13 and 14 September 2023

ASTRON

3 to 6 October 2023

SRON

5 and 6 October 2023

CWI

1 to 3 November 2023

ARCNL

8 to 10 November 2023

Nikhef

20 to 22 November 2023

DIFFER

22 and 23 November 2023

NSCR

29 and 30 November 2023

AMOLF

11 and 12 December 2023

 

Critical in a positive way

For the coordination and supervision of the site visits and the production of the committees' reports, NWO-I has engaged Academion, a company specialised in supervising such evaluations. Spijker’s Strategic Support department colleagues will be responsible for the connection with NWO-I.
Spijker: "For example, sometimes questions can arise about NWO-I policy that the external experts cannot answer. At the time of the visit, the committee makes a provisional report on the last day and often shares its findings with the entire institute. Those findings are always critical in a positive way", according to Spijker. Once all of the reports have been completed, the board will formulate its response.

Often small changes

To what extent do the committee's findings lead to a change of course? Spijker says that the committees are made up of fundamentally critical scientists who always present findings that help to finetune the course of an institute. Sometimes, the committee advises to follow a direction with growth potential and that can provide just the impetus needed to fill an open tenure track position in the area of this theme.
Spijker: "Based on the evaluation outcomes, the institutes sometimes conclude that a slight adjustment to the course is a good idea. This critical self-reflection is an important part of the evaluation process. More often, the advisory reports lead to smaller changes, which are always made from the perspective of the scientific context. Due to the input from the committee, new collaborations also commonly emerge after an evaluation". Spijker is looking forward to the evaluations: "They are an important highlight for the whole of NWO-I".

Rosemarie Aben (strategic adviser and board member Nikhef) about the SEP evaluation

Evaluation bears much fruit
After gaining her PhD from Nikhef in 2015, Rosemarie Aben found a job elsewhere. But on 1 March 2023, she returned to the institute as a strategic adviser and board member.
The first important task was clear: ensure that a new strategic plan and self-evaluation according to the SEP criteria are produced. Aben: "We did that together with scientists and other colleagues, of course. We examined the outcomes for the objectives formulated in the previous strategy. Then you can really see how much progress has been made in the past six years. Our agenda for the site visit will definitely include subjects such as 'How do you deal with Open Science and how do we supervise our PhDs?', but we expect that the committee will mainly focus on our strategy for the future: 'In which experiments will we participate and how will Nikhef do that? And how does the institute deal with the risks of short-term funding that we do not control ourselves?'"
Aben states that a decision about the location for the Einstein Telescope will probably be taken in 2026. The strategy includes scenarios for different locations: in the Dutch province of South Limburg or elsewhere. She thinks that the committee will further ponder this question. Aben is positive about the outcome of the evaluation process. "It forces you to think and - independent of the visit - bears much fruit. Nevertheless, I'm still a bit nervous about the site visit. The committee is made up of renowned scientists who do not avoid asking critical questions, but that is fine too." Nikhef also added 'sustainability' as a subject to the self-evaluation strategy. Aben: "It's a subject that we feel is very important, and we also have a sustainability task force. Everybody flew to CERN during my PhD, whereas now colleagues travel there by train".

Jessica Dempsey (director ASTRON) about the SEP evaluation

The evaluation provides a boost towards the future
"I've learned an awful lot about ASTRON and I enjoy the evaluation process," says Jessica Dempsey, who has been director of the Institute in Dwingeloo for a year now. Just like all the other institutes, ASTRON wrote a self-evaluation report to prepare for the visit of the evaluation committee. For the review, Dempsey had to depend upon the information provided by colleagues: "Their historical memory was vital in this entire process and I really appreciated being able to rely on this. The evaluation is very much a joint, bottom-up process. We allocated the assessment per subject to members of the middle and higher management. They discussed this with their teams to ensure that all relevant experiences were considered. In this way, everybody became the owner of the assessment".
For Dempsey, the evaluation was an ideal way of quickly gaining insight into the institute's achievements during the past six years. Now, all of that information will help determine the way forward. A 'dream  committee' will visit ASTRON in October: "It is a superb panel with a fantastic portfolio of scientific and technological experience and skills". The committee members will talk with everybody at each level of the organisations: "More conversations and fewer presentations". Dempsey hopes that the committee will recognise the ambition and performances of ASTRON and see that the skills of the employees match this ambition. She expects that the members of the committee will come up with constructive suggestions to strengthen the institute.
Dempsey thinks the evaluation will give individual ASTRON employees a boost of confidence towards the future: "Academics, researchers and also colleagues in support positions sometimes tend not to realise just how good their work is. And it is fantastic that an external committee can provide them with positive feedback about their performance. Needless to say, we will be celebrating that".

Text: Anita van Stel

Newsletter Inside NWO-I, September 2023.
You can find the archive of the newsletter Inside NWO-I on the NWO-I website.

Confidental Infomation