Is it time for the arts sector to lead the impact agenda?

By Professor Brydie-Leigh Bartleet — Impact is something that cuts across the lives of artists both outside and inside the academy.

Considerations of impact have always been core to creative artists as they have activated the power of the arts to influence people, places, practices, and politics across many diverse cultural contexts and time periods. What is far newer is a bureaucratic turn towards measuring these impacts for the sake of accountability and measuring return on public investment.

Recently we have seen major shifts in our national research landscape driven by the Government’s innovation and science agenda, which presents both possibilities but also risks to the arts ecology here in Australia. The government’s new impact evaluation exercise has emerged as a significant driver in this landscape. As the government says: “While the success of university research can be viewed in measures of excellence, it can also be found in its economic, social, and environmental impacts. Assessing and reporting on how our investments in university research translate to tangible benefits for Australia will help show where collaboration with industry and other partners could bolster and more quickly deliver these benefits.”

The kind of impact accountability that is pervading higher education is not new to artists outside the academy. For many decades arts funding bodies have had various manifestations of criteria around impact or broader societal contribution, and/or required evaluations that speak to the impact of the project and their return on investment. As the director of a research centre, I frequently get requests from artists and arts organisations to evaluate the impact of their projects to meet the requirements of these funding bodies. What they are asking for is not always in-depth research, but rather proof that what they have done has had some kind of impact. Here evidence gets confused with advocacy, where arts organisations often want metrics that prove their desired impact rather than trying to understand it.

In these recent developments in higher education and the arts sector, we have also seen a tendency towards quantifying and measuring the instrumental impact of the arts. By their very nature, these exercises seem to focus more on measuring the ways that the arts serve social, economic and environmental agendas, rather than trying to understand the intrinsic, conceptual, and indeed aesthetic processes that are central to them. I have certainly felt these tensions in explaining the multidimensional impact of my community-engaged, relationship-focused, arts-based work in the one-dimensional text-based form required by the Government’s Impact and Engagement Exercise. This form uses transactional terminology such as end-users and beneficiaries, and privileges a limited definition of impact. Significantly, it asks for this case to be made in a small number of words and numbers only. The resulting truncated narrative and metrics only provide one part of the picture.

A major risk in quantifying and narrowly measuring the social and economic benefits of the arts in this way is that measurement becomes prized and rewarded over understanding how these benefits actually work. You might say it is a case of “the tail wagging the dog.” As Meyrick, Phiddian and Barnett (2018) argue in their recent book, What Matters? Talking Value in Australian Culture, as culture gets more complex, so must the ways we evaluate it. They suggest that we are going backwards in our obsession with quantification, and turning arts and culture into something to be scaled, measured and benchmarked, rather than trying to better understand its interconnected instrumental and intrinsic value.

Within the field of cultural policy, there has been significant critique of this limited focus on the instrumental impact in the arts, and its resulting metrics and measures. It has pointed out that such an approach to thinking about measuring impact is highly problematic because the arts operate as part of complex ecosystems of forces and factors, where cause and effect explanations are rarely sufficient, and systems of data collection struggle to find effective ways of measuring the preventative role of the arts. Moreover, responses to the arts are always personal and individual (some people are radically transformed by a particular arts experience while others are left unmoved), and capturing the subjective impact is difficult. Current thinking in this field advocates for a much more integrated way of thinking about measuring impact in the arts (see for example, Reframing the Debate About the Value of the Arts.

This raises questions about how we respond to these external pressures to measure our impact when we know that it is a far more complex process than the government (or indeed some funding bodies) sometimes perceive it to be. We could ignore it, and hope it goes away, but unfortunately that seems unlikely and maybe even a missed opportunity. Rather, it might be worth thinking about how we challenge and turn this impact exercise on its head. My sense is that we are not going to find inspiration nor wisdom in government guidelines, but rather rethinking definitions of impact in our field, and how we can develop and implement robust methods and measures that can in turn enhance our practices and how we understand them.

We don’t need to reinvent the wheel. There is a significant amount of work that has already been done on this topic here in Australia. This work focuses on a wide range of methods and approaches to understanding, measuring and communicating impact. MacDowall, Badham, Blomkamp and Dunphy’s (2015) pivotal collection essays explores the expanding field of cultural measurement and presents new approaches and frameworks for accounting for culture in local, national and international contexts. Gattenhoff’s research (2017) explores ways to move beyond narrow, acquittal-focused evaluations of impact and cultural value to far more dynamic and participatory partnerships between the evaluators and arts organisations. Likewise, Meyrick, Phiddian and Barnett’s research (2018) shares a number of examples of cultural practice that are defying the current reliance on metrics. The Australia Council for the Arts’ latest national arts participation survey “Connecting Australians” has focused on the impact of the arts, in both instrumental and intrinsic ways (see table).

Connecting Australians: Results of the National Arts Participation Survey (2017).Connecting Australians: Results of the National Arts Participation Survey (2017).

A recent report from the AHRC on understanding the value of the arts and culture also suggests speaking to the question of metrics in a way that resonates with more holistic understandings of impact in the arts (Crossick & Kaszynska, 2016). It advocates for more formative and participatory evaluations of impact, as opposed to more summative and non-participatory approaches. It suggests that accounting for human experiences of the arts calls for multi-criteria analyses and a range of approaches from artistic research methods, peer review, case studies, benchmarking, cost-benefit analysis, hindsight studies, and so on. In essence these arguments and methods suggest that we need to find more rounded ways of integrating instrumental and intrinsic impact and value.

As these examples show, within the arts sector we have the know-how to turn the current impact agenda around from its rather narrow focus on quantitative metrics, fragmented measures, and instrumentalised outcomes. Our experience and insights position us perfectly to advocate for more holistic approaches to evaluation and measurement that combat such constricted and oftentimes short-term thinking. However, this know-how seems to be struggling to cut through conceptually in the current political landscape. The pressing challenge for us seems to be how we translate these nuanced and sophisticated ways of engaging with impact in the current environment, and play a leading role in transforming prevailing ways of measuring impact in both higher education and the arts sector more broadly.

References

Crossick, G. & Kaszynska, P. (2016). Understanding the value of arts & culture. Swindon, UK: Arts and Humanities Research Council.

Gattenhof, S. (2017). Measuring impact: Models for evaluation in the Australian arts and culture landscape. London: Palgrave.

MacDowall, L., Badham, M., Blomkamp, E., & Dunphy, K. (Eds.). (2015). Making culture count: The politics of cultural measurement. London: Palgrave.

Meyrick, J., Phiddian, R., & Barnett, T. (2018). What matters?: Talking value in Australian culture. Melbourne: Monash University Publishing.

Professor Brydie-Leigh Bartleet is Director of the Queensland Conservatorium Research Centre, Griffith University, Australia. She is known worldwide for her research in community music and community engagement and has led many projects that explore the social impact of the arts. She has worked in partnership with a wide range of NGOs, arts and community organizations, and colleagues across Australia and internationally to design, drive and deliver innovative and highly complex projects. This work has led to new and interdisciplinary approaches to music research that intersect with health and wellbeing, corrections and criminology, Indigenous and cultural policy, social justice and regional arts development, and most recently human rights. She has worked on five nationally competitive grants, five consultancies and three prestigious fellowships (totalling over $1.2 million), as well as 140 research outputs in high-level national and international publications, and keynotes in Aotearoa New Zealand, Australia, Canada, Japan, Germany and Ireland. In 2014 she was awarded the Australian University Teacher of the Year and in 2018 she was awarded a highly competitive Art for Good Fellowship from the Singapore Foundation.

More from this issue

More from this issue

By Jenny Wilson — As a former university research administrator, one of my favourite books is Management Fads in Higher Education: Where they come from, What they Do, Why they fail[1] by US writer Robert Birnbaum. At the same time that Australian universities were enthusiastically adopting new management practices, Birmbaum’s book was clearly documenting their failure in the US.
By Professor Su Baker AM — Thirty years, conventionally, represents a working life, and if that is still true, then that has been mine, so far.
By Professor Marie Sierra — Now that the Federal election is over, we can likely expect the next ERA to be 2021, instead of some later year. While it’s an enormous amount of work for what some deem a “beauty pageant”, more infrequent ERA exercises mean having to manage a larger, more unwieldy data set. And it’s all about the data.
By Professor Larissa Hjorth — When NISA released their innovation report that led to the ARC developing the Engagement and Impact (E&I) framework, people were palpably nervous. Not yet another framework which already over-worked academics had to consider in their research trajectory.
By Professor Simon Biggs — How do the current criteria we use to evaluate the quality, engagement and impact of research relate to the priorities of creative arts research? What do these criteria capture and what do they miss?
By Professor Craig Batty — While the sector has a pretty broad understanding what creative practice research is – and how its outcomes align with the ARC’s definition of research (e.g., new materials, devices, processes, understandings) – there are still conflicting views about where the new knowledge resides, even from those doing the work.
By Professor Jen Webb and Professor Ross Gibson — In 2017 the Deans of Arts, Humanities and Social Sciences agreed to survey decision makers in creative arts disciplines about the perception that “While every other FoR has increased its average score in each of the ERA rounds … FoR19 is the only code in which the average score across the sector has decreased in each ERA round” (McKee 2018).[1]
Professor Clive Barstow interviews Professor Jill Durey — Within the broad definition of practice-led research, how has contemporary literature fared in terms of its categorisation, measurement and funding compared with the visual and performing arts? I interview Professor Jill Durey, previous head of English at ECU and now retired, and who has lived through the various incarnations of ERA.