Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes

• Published date: 01 June 2021
• Author: Robert A. Buckle,John Creedy,Ashley Ball
• Date: 01 June 2021
• DOI: http://doi.org/10.1111/1467-8462.12415

The Australian Economic Review, vol. 54, no. 2, pp. 208–230 DOI: 10.1111/1467-8462.12415

Fifteen Years of a PBRFS in New Zealand: Incentives and

Outcomes

Robert A. Buckle, John Creedy and Ashley Ball*

Abstract

This article examines the transformation of

New Zealand universities following the intro-

duction in 2003 of a performance‐based

research fund system. The analysis, based on

a social accounting framework, utilises long-

itudinal researcher data available from three

full assessment rounds, in 2003, 2012 and

2018. This enables identification of the entry,

exit and quality transformation of researchers

and their contribution to changes in university

and discipline research quality. Changes in

the discipline composition of universities

made a negligible contribution compared to

improvements in the quality of researchers.

The dynamics are closely related to the new

incentives created by the system.

JEL CLASSIFICATION

I2; I23; I28; L38

1. Introduction

This article examines the transformation of

New Zealand universities following the in-

troduction of a performance‐based research

fund system (PBRFS).

1

There have been three

full assessment rounds, in 2003, 2012 and

2018.

2

The PBRFS was designed to allocate

research funds based on research performance

rather than number of students.

3

The purpose of the …[PBRFS] …is to ensure that

excellent research in the tertiary education sector is

encouraged and rewarded. This means assessing the

research performance of tertiary education organisations

(TEOs) and then funding them based on their perfor-

mance. (The Tertiary Education Commission, 2019,

p. 11)

A distinctive feature of the system is that

performance is based on peer‐reviewed eva-

luation of each individual researcher. Funding

to universities flows from the assessment of

each researcher.

The system changed the incentives facing

individuals, departments and universities. It

gave rise to an extensive debate about the

perceived consequences and merits of the

change. This debate is worldwide, in view of

the widespread adoption of performance‐

based schemes for allocating research

funding. Issues include the possible effects

on academic freedom, the diversion of energy

* Buckle: School of Economics and Finance, Victoria

University of Wellington, Wellington 6140, New

Zealand; Creedy: School of Accounting and

Commercial Law, Victoria University of Wellington,

Wellington, New Zealand; Ball: School of Economics and

Finance, Victoria University of Wellington, Wellington,

New Zealand. Corresponding author: Buckle, email:

<bob.buckle@vuw.ac.nz>. We are grateful to the New

Zealand Tertiary Education Commission (TEC) for

providing the data and thank Sharon Beattie, Morgan

Healey, Paul Lawrence and Susan McFadyen for helpful

discussions and support. We are grateful to Leon Bakker

and Philip Roderick for helpful discussions, and have

benefited from comments by Norman Gemmell, and the

Editor and referees of this journal. The research for this

article was supported by a Victoria University of

Wellington research grant.

© 2021 The University of Melbourne, Melbourne Institute: Applied Economic & Social Research,

Faculty of Business and Economics

Published by John Wiley & Sons Australia, Ltd

from teaching, the introduction of a competi-

tive rather than collegial atmosphere, the

discouragement of longer‐term scholarly pro-

jects, the nature of the metric devised to

measure research quality, and compliance

costs.

4

However, the present article is con-

cerned with an analysis of the incentives and

the nature of the changes that have been

induced by those incentives. When the NZ

PBRFS was introduced, there was very little

international evidence regarding the effective-

ness of these systems. They were relatively

new and had varying designs. Even now, a

consensus has not yet been reached regarding

their effects (see Checchi, Malgarini and

Sarlo 2019).

A university can improve its research

quality in three ways, although strong con-

straints are placed on the changes that can be

made. Changes in average quality depend on

the exits and entries of individuals (to and

from other universities in New Zealand, or

international movements), and the extent to

which remaining individuals can improve

their measured quality. A university can also

influence its average quality by changing its

discipline composition in favour of higher‐

quality groups. Institutions do not have

complete control over these changes.

Competition for research skills, contractual

and legal obligations, and individuals’career

choices and retirement decisions impinge on

the extent to which these transitions can be

managed. Hence, many observed flows may

result from other forces, but at the margin

the institution can be expected to have an

influence through human resource manage-

ment policies such as recruitment decisions,

performance management, training and the

support of a strong research environment.

This study examines these changes in detail,

using a special dataset that contains anon-

ymised longitudinal information about every

researcher who participated in any assessment

round from 2003 to 2018. The data are not

publicly available and were provided by the

Tertiary Education Commission (TEC) fol-

lowing a confidentiality agreement. The em-

phasis here is on the transitions since the

introduction of a PBRFS. The extremely

difficult problem of assessing the scheme in

terms of a cost–benefit analysis is beyond the

scope of this article.

5

One important concern is whether observed

changes can be attributed to the introduction

of a PBRFS. It is sometimes suggested that

universities have always striven to improve

research quality, that academics are highly

self‐motivated, and a competitive environ-

ment (both among researchers seeking promo-

tion and universities wishing to attract stu-

dents and high‐quality academics) exists

without the need for a formal evaluation

process. Typically, no empirical measures of

the metrics designed by PBRFSs exist until

the formal process begins. Although attempts

have been made to look for structural breaks

in series of bibliometric measures, there is a

low degree of concordance between such

measures and the metrics used in PBRFSs.

6

The present study, by focusing on the precise

nature of the incentives and the routes by

which performance can be improved, is able

informally to test a number of hypotheses

relating to changes that can be attributed

directly to the PBRFS.

7

In discussing research quality, the metric

used by the NZ system is taken at face value,

and hence the term ‘high‐quality researcher’

relates specifically to the official quality

measure described below. This is consistent

with the attempt to disentangle the precise

incentives and the ways in which universities

have been transformed over the 15‐year

period of the PBRFS. This should not imply

agreement with the metric: indeed, it has

unusual properties and no rationale was

provided by the TEC or the Department of

Education.

Section 2 explains the essential features

of this system, and the longitudinal data.

8

The incentives created by the system are

discussed in Section 3 together with hy-

potheses regarding consequent organisa-

tional changes. Section 4 explains how the

data can be utilised to trace changes in

researcher quality and the entry, exit and

quality transformation (QT) of researchers,

and how these contribute to changes in

overall quality. Section 5 applies a

209Buckle, Creedy and Ball: Fifteen Years of a PBRFS in New Zealand

© 2021 The University of Melbourne, Melbourne Institute: Applied Economic & Social Research,

Faculty of Business and Economics