On reviewer training and guidelines

BY CRAIG BATTY — Do we agree on what we are looking for in research assessment in creative disciplines? As a DASSH survey in 2018 revealed, assessors (at least those surveyed) had mixed views about what was important – from theoretical contributions, to industry contributions, to hybrid contributions, and so on – the caveat ‘it depends’ came up strongly.

There is still, perhaps surprisingly given the advances in the literature we have (especially in Australia), no consensus on what research excellence looks like. We know what creative practice research is and why it’s undertaken, but when and how do we know it is excellent (or not)? 

Might we be bold and put out a statement (like the Manifesto, 2023) via the Australian Research Council which is bolder and more specific in asserting the criteria? Have we been too accommodating of nuance and interpretation? Has this inadvertently undone us? 

I will draw here on the difference between what I call research-enabled practice, and practice-enabled knowledge (see Batty and Zalipour 2024).

This is the difference between a creative artefact that contributes knowledge in and of itself, and creative works/iterations/experiments that form a method from which to draw knowledge about process and practice. There are different contributions in these models, found in different places or artefacts, so when judging excellence in a creative artefact, are we looking for the same thing?

If we want robust data and a level playing-field for research evaluation – excellence and impact – then we need to have a consistent and reliable dataset.

We currently do not have this. Each university has its own way of collecting and assessing a creative research output, from sophisticated digital systems to in-person committees where people literally bring in piles of materials and notes to table and discuss. And let’s not ignore personal agendas here – for those who have been involved in assessing creative research works, who can, hand on heart, say that they have never brought in or seen from others personal opinion (rather than balanced critique) about someone’s work? I don’t have enough hands on which to count how many times I have experienced this.

How can we be confident that we are all coming to evaluation on the same basis, with the same data, and therefore are producing a national dataset that is able to do a good, robust job of research evaluation?

This is where we need a national, fit-for-purpose NTRO (or creative research output) system – informed and built by experts, with sector-wide consultation – to ensure that we’re all putting in the same data, to the same level, assessed to national standards, which would then lead to a better outcome for all universities participating in whatever the new Excellence in Research for Australia becomes.

On reviewers: this proposed national system – a national standard, if you like – ought to then give reviewers, experienced and emerging, more confidence in how they assess research. And by association, university research committees and research offices confidence in the verifiability of their research output collection. For example, how many people here today have been sent a work to assess – in whatever system, university committee or creative journal – and have struggled with what to say? How many can say, hand on heart, that they are crystal clear about the criteria – and that academic buzz words (e.g., affordances, intersectionality, methodological innovation), and that name-dropping, industry awards and other accolades haven’t swayed their thinking? Are we sure that we are looking for the research contribution only, despite how many awards the book won, how many international festivals the film played at, or how many aged care facilities have taken on the co-design framework? I suggest that most of us here today oscillate between ‘this looks great’ and ‘this sounds pretty standard practice’; between ‘I have no idea what this research statement means, so it must be genius’ and ‘this research statement has clearly been written by someone else, to make it sound better than it probably is’. I won’t ask for a virtual show of hands, but you get my drift.


Craig Batty is Professor and Executive Dean of UniSA Creative, University of South Australia. Craig is also President of the Australian Council of Deans and Directors of Creative Arts (DDCA). Craig is an award-winning educator, researcher and supervisor in the areas of screenwriting, creative writing and screen production, and the broader field of creative practice research, which includes the creative doctorate. 

More from this issue

Artists in academia

BY BEATA BATOROWICZ — provocations on traversing research and industry success within creative practice.

The ‘tension’ between industry and

Read More +

More from this issue

BY SMILJANA GLISOVIC — On August 9, 2024 the DDCA held a National Forum to generate discussion on the shape of the future of creative practice research in Australia (and beyond). The particular focus of the event was on research evaluation and assessment, chosen because of the current reviews of ERA (Excellence in Research for Australia).
BY MIA LINDGREN — I asked AI to give me a list of words including the prefix ‘non’: non-profit, non-negotiable, non-essential and so on. The non prefix is used to indicate the opposite, absence or exclusion of the root words, meaning it signals a deviation from the standard, typical or expected.
BY JESSICA WILKINSON — In the ERA 2018 exercise I was invited to be an assessor for the Creative Writing field. Of the five universities assigned to me for assessment of submissions within this code, I encountered wildly different approaches to how each university collated the 'top 30%' of representative samples.
BY BEATA BATOROWICZ — provocations on traversing research and industry success within creative practice. The ‘tension’ between industry and academia, in addition to having diverse roles within the broader creative arts research ecology of development and contribution, also describes an interconnectedness: they both feed into each other in building notions of success.
BY DAVID CROSS — Oh, to be world standard. To have reached the peak of global creative practice. To have left behind the parochialism of local concerns and made it in the places, contexts and ruthlessly competitive environments that truly matter.
Thank you to all that so generously and respectfully contributed to the conversation on the day of the National Online Forum, both ‘on mic’ and in ‘the chat’. The contributions in the below text are not assigned to individuals but rather the general threads and themes are summarised. For more nuance (and less unintended interpretive valence from me) I do encourage you to watch the recording of the forum here.
BY JULIA PRENDERGAST and JEN WEBB — Let us begin by introducing ourselves: we are Associate Professor Julia Prendergast, AAWP President/Chair, and Distinguished Professor Jen Webb, AAWP Treasurer – accepting the invitation to contribute on behalf of the Australasian Association of Writing Programs (AAWP), the peak academic body representing the discipline of creative writing (Australasia).
BY VERONIKA KELLY and CHARLES ROBB for ACUADS — The Australian Council of University Art and Design Schools (ACUADS) is the nation’s peak organisation representing the interests of art and design schools within Australian higher education. Here, ACUADS draws attention to issues surrounding the interpretation and positioning of ‘world standard’ in creative practice research.
BY SUSAN KERRIGAN for ASPERA — Australian Screen Production Education and Australian Screen Production Education and Research Association (ASPERA) has contributed greatly to the creation and assessment of Creative Practice Research (CPR) in Screen Production disciplines. This work began with the creation of the peak disciplinary body two decades ago, at that time only one person in the gathering held a PhD and was considered to be a legitimate researcher by the academy.
BY CHARLES ROBB — When news broke that Excellence in Research for Australia (ERA) 2023 had been cancelled, a palpable wave of relief swept through Australian universities – no more laborious compilation of packages, impact statements, and ranking spreadsheets.
BY CAT HOPE — Despite an increasing number of artist scholars in the performing arts – those who have higher degree qualifications featuring the creative project/ exegesis model – are being employed in universities, it seems as if scholarly recognition for the so called ‘non traditional research output’ (NTRO) is in decline.
BY SMILJANA GLISOVIC and CRAIG BATTY — The discussion amongst colleagues at the DDCA National Forum on evaluation and assessment of creative practice research – where more than 100 from a range of disciplines were in attendance – was informed, considered and encouraging.
BY ANDREA RASSELL and JO POLLITT — In thinking about the development of a standardisation of assessment of creative research, we, as interdisciplinary artist scholars practising respectively in filmmaking/media and choreographic writing/dance/feminist environmental humanities, are constantly reforming our identities as researchers and artists.