Parity in ERA submissions across universities

BY JESSICA WILKINSON — In the ERA 2018 exercise I was invited to be an assessor for the Creative Writing field. Of the five universities assigned to me for assessment of submissions within this code, I encountered wildly different approaches to how each university collated the 'top 30%' of representative samples.

The lack of consensus on quality and how non-traditional research outputs were valued by the respective institutions was obvious. To give a few examples: one university put up a folio of creative works – i.e. one countable output – that constituted upwards of 60 poems, plus full books of poems. As a poet myself, and knowing the difficulties of publishing, I was surprised by the suggestion that such enormous quantities were required to constitute equivalence with a single research output such as a peer-reviewed scholarly article. This approach suggested that the poet researcher’s outputs were more-or-less gathered up and retrospectively lassoed by a research statement, without demonstrating strong practice research purpose; or, this was a possible gaming strategy on the part of the university. 

In another example from a different university, the submitted sample of creative works included images of the covers of outputs such as published books, along with industry indicators of quality such as review excerpts and lists of prizes and accolades that the respective works had garnered. But there were no excerpts of the creative works themselves.

The work itself, I thought, must surely be necessary to demonstrate quality in creative practice research, rather than relying purely on an author’s reputation or industry success, which we all know can be fickle markers.

In that same assessment round, there were other approaches to submitting samples and ‘evidence’, too, demonstrating a very broad spectrum of understanding of quality creative practice research, and how to demonstrate that quality. 

I have heard similar anecdotes from colleagues and peers who have also been assessors. We are clearly faced with problems of understanding the relationship between quality and quantity, and with how to demonstrate quality in fields that haven’t developed in the academy according to ‘traditional’ methods.

What I enjoy about ‘evaluation’ of creative practice research in all its contexts – whether that be examination of PhDs, peer-reviewing for publications, assessing for ERA, or even evaluating my own outputs and writing a statement for internal processes – is the call to connect with the work itself and the extent to which this work stretches not only practice or idea, but articulation of practice and idea. This ‘connecting’ with the breadth of research by peers tempers the ‘labour’ for me when it comes to evaluation tasks (to an extent, of course!). The attentiveness required for proper evaluation of creative practice research is a privilege that we ought to emphasise in developing ‘national standards’. But where we perhaps stray is in ‘evaluating’ according to arbitrary industry markers that don’t necessarily have much to do with research – for example, awards and other accolades, number of reviews and so on. These markers can be important factors in terms of whom one’s work reaches and how its impacts might be measured, but they can’t be leaned on as labour ‘work-arounds’ in place of a more well-rounded focus on the quality of the research offered by the work itself. So, what standards might be set nationally to ensure that submissions focus on the quality of the research itself?

Some considerations and questions towards the next generation of ERA: First is the labour involved in writing research statements, preparing submissions, and assessing both internally (at a university level) and nationally.

Our current methods are labour and cost intensive; the significant administrative burden was, of course, outlined in the ARC submission to the interim report of the Universities Accord.

Yet peer-review is, really, a non-negotiable. What processes might we collectively put forward to guide university collection and national assessment?

Second, what is the relationship between creative practice research and the industries that support and disseminate their public engagement and impact lifecycles? To what extent do these external or industry markers of approval indicate that good research has occurred? To what extent are we leveraging those markers to retrofit strong research statements around our creative practice outputs, and what are the implications of this retrofitting for research quality?

Third, to what extent do we want universities to have autonomy over their own priorities when it comes to assessing creative practice research? We know that universities assign points and metrics to outputs in different ways, which is usually driven by institutional priorities. How might national standards enable and/or restrict this institutional autonomy?

Fourth, is it possible to compare our creative and artistic rigour with traditional academic rigour? As I have gleaned from my own encounters within institutional settings, there persists an attitude that creative practice researchers and research can be naive. Why is this? Perhaps because we’re made to squeeze practitioner knowledges into other modes of reporting that aren’t necessarily a good fit. National guidelines for assessing quality would help to establish standards and dispel such attitudes.

And finally, are there some creative practices that are better suited to academe? How so and why? And how might assessment processes help us to expand academic frontiers to encompass, rather than to exclude or ignore certain practices? 

To close, I wanted to offer a few provocations towards the future of assessment and what might be possible:

  • What if we allowed ourselves to go deeper into exploratory territory and to recognize that the goal is ‘knowledge creation’, not overproduction?
  • What if discipline fields came together to strategise ‘our way’ of saying what is worth doing and how might be the best way to go about it? Of course, this should not preclude interdisciplinary discussion. 
  • How might we place emphasis on the enablers of quality creative practice research? What do people need for their disciplines and creative contributions to thrive?
  • And a final, wild thought: Could we extract ourselves from metrics to instead focus on researcher projects and creative practice research ecologies? What if we shifted emphasis from ‘points accrual’ towards evaluation of sustained research narratives that demonstrate longer term goals and interdisciplinary thought leadership?

Jessica Wilkinson is Associate Professor in Creative Writing at RMIT University. Jessica is a writer, critic, scholar and editor whose research interests include: poetry and poetics; contemporary poetry; poetic biography; ‘nonfiction poetry’; experimental/radical writing; literary theory. She is the founding and Managing Editor of ‘Rabbit: a journal for nonfiction poetry’ (2011-present) and of the Rabbit Poets Series. 

More from this issue

Artists in academia

BY BEATA BATOROWICZ — provocations on traversing research and industry success within creative practice.

The ‘tension’ between industry and

Read More +

More from this issue

BY SMILJANA GLISOVIC — On August 9, 2024 the DDCA held a National Forum to generate discussion on the shape of the future of creative practice research in Australia (and beyond). The particular focus of the event was on research evaluation and assessment, chosen because of the current reviews of ERA (Excellence in Research for Australia).
BY MIA LINDGREN — I asked AI to give me a list of words including the prefix ‘non’: non-profit, non-negotiable, non-essential and so on. The non prefix is used to indicate the opposite, absence or exclusion of the root words, meaning it signals a deviation from the standard, typical or expected.
BY BEATA BATOROWICZ — provocations on traversing research and industry success within creative practice. The ‘tension’ between industry and academia, in addition to having diverse roles within the broader creative arts research ecology of development and contribution, also describes an interconnectedness: they both feed into each other in building notions of success.
BY CRAIG BATTY — Do we agree on what we are looking for in research assessment in creative disciplines? As a DASSH survey in 2018 revealed, assessors (at least those surveyed) had mixed views about what was important – from theoretical contributions, to industry contributions, to hybrid contributions, and so on – the caveat ‘it depends’ came up strongly.
BY DAVID CROSS — Oh, to be world standard. To have reached the peak of global creative practice. To have left behind the parochialism of local concerns and made it in the places, contexts and ruthlessly competitive environments that truly matter.
Thank you to all that so generously and respectfully contributed to the conversation on the day of the National Online Forum, both ‘on mic’ and in ‘the chat’. The contributions in the below text are not assigned to individuals but rather the general threads and themes are summarised. For more nuance (and less unintended interpretive valence from me) I do encourage you to watch the recording of the forum here.
BY JULIA PRENDERGAST and JEN WEBB — Let us begin by introducing ourselves: we are Associate Professor Julia Prendergast, AAWP President/Chair, and Distinguished Professor Jen Webb, AAWP Treasurer – accepting the invitation to contribute on behalf of the Australasian Association of Writing Programs (AAWP), the peak academic body representing the discipline of creative writing (Australasia).
BY VERONIKA KELLY and CHARLES ROBB for ACUADS — The Australian Council of University Art and Design Schools (ACUADS) is the nation’s peak organisation representing the interests of art and design schools within Australian higher education. Here, ACUADS draws attention to issues surrounding the interpretation and positioning of ‘world standard’ in creative practice research.
BY SUSAN KERRIGAN for ASPERA — Australian Screen Production Education and Australian Screen Production Education and Research Association (ASPERA) has contributed greatly to the creation and assessment of Creative Practice Research (CPR) in Screen Production disciplines. This work began with the creation of the peak disciplinary body two decades ago, at that time only one person in the gathering held a PhD and was considered to be a legitimate researcher by the academy.
BY CHARLES ROBB — When news broke that Excellence in Research for Australia (ERA) 2023 had been cancelled, a palpable wave of relief swept through Australian universities – no more laborious compilation of packages, impact statements, and ranking spreadsheets.
BY CAT HOPE — Despite an increasing number of artist scholars in the performing arts – those who have higher degree qualifications featuring the creative project/ exegesis model – are being employed in universities, it seems as if scholarly recognition for the so called ‘non traditional research output’ (NTRO) is in decline.
BY SMILJANA GLISOVIC and CRAIG BATTY — The discussion amongst colleagues at the DDCA National Forum on evaluation and assessment of creative practice research – where more than 100 from a range of disciplines were in attendance – was informed, considered and encouraging.
BY ANDREA RASSELL and JO POLLITT — In thinking about the development of a standardisation of assessment of creative research, we, as interdisciplinary artist scholars practising respectively in filmmaking/media and choreographic writing/dance/feminist environmental humanities, are constantly reforming our identities as researchers and artists.