Skip to content
  • Accessibility options:
 
Menu

How commissioners use research evidence

Health and Social Care Services Research

27.03.18

doi: 10.3310/highlight-02685

Please note that this summary was posted more than 5 years ago. More recent research findings may have been published.

Researchers want their work to be used and useful, but may not always understand the context in which decisions are made. Most health and care organisations aim to base decisions on the best available evidence, but accessing and interpreting the right evidence at the right time is hard. Researchers need to do what they can to make their research as useful as possible to those making decisions under pressure.

The NIHR has funded six particular studies in the past five years on the use of evidence by commissioners. Some of this research may also be relevant to service managers in hospital trusts and other care providers and systems.

This highlight includes studies into the behaviour of individual managers and the way in which commissioning organisations make sense of and use research information when making decisions. The findings provide some practical pointers for researchers to make their work more accessible and relevant to commissioners and managers.

Evidence at a glance on how commissioners use research

Most health and care organisations aim to base decisions on the best available evidence. This is enshrined as a principle for commissioners in the Health and Social Care Act of 2012. Research will play only a small part in decisions that are made. Service leaders will draw on context (including available resources) and values, as well as data of different kinds.

Core findings and questions arising from the research

  • Managers of all backgrounds find it hard to make sense of and apply evidence in their everyday work
  • Studies show that managers tend to make less use of formal research. They value examples and experience of others, as well as local information and intelligence
  • Senior managers rely on a small conversational circle and trusted colleagues to identify and interpret evidence
  • Evidence does not speak for itself. Organisations need skills, not just technical around critical appraisal capacity, but also to engage experts and frame research for different audiences
  • One study evaluating a service providing evidence briefings for commissioners found poor uptake and use of research for decision-making
  • Having skilled individuals, like public health staff, on the spot to contextualise and interpret evidence helps managers use evidence when making decisions about systems and services
  • Timing is key - having good enough evidence at the right time trumps perfect research which arrives too late for decision makers to use

Questions for researchers

    • How have you involved service planners or leaders in your project - from determining the research topic and question through to dissemination activities?
    • Are your study findings presented in a format which will appeal to managers (like using case studies, quotes and examples to bring the research to life)?
  • Have you made use of toolkits and best practice in communicating research?
  • Have you looked for opportunities to present your findings face-to-face to relevant audiences and meetings of managers? Do you know what trusted sources they use (such as thinktanks or professional bodies) and how you could work with them?
  • Does your research address the context (local and national) for managers, including policy drivers?
  • Does your research have anything to offer managers in their quality improvement or service transformation activities?

Questions for staff and commissioning organisations

  • Do we pause when making strategic decisions to see if and how evidence could be used?
  • How can we best bring research together with other evidence (such as health needs or population data) to make decisions?
  • Who could help us inside or outside the organisation to make sense of research for particular important problems?
  • Can staff access training or expertise about understanding and using evidence? What library or information services can we use?
  • Do we share our learning, including unexpected results of service changes, with others?

What research evidence do health care managers use?

From surveys, interviews, focus groups and observational research of different kinds, we have learned that managers rarely seek formal, research-based information from academic journals or other sources.

Learning from experience

A study by Sue Dopson in 2013 used comparative case studies in six health settings to look at the use of management and organisational evidence. Findings suggest health care managers are most oriented towards knowledge from their own experience and that of their communities of practice.

Research often offers answers for healthcare providers and clinicians easier than commissioners, as it is more black and white – is drug A better than drug B? As commissioners, we have thornier issues, such as what one or two interventions are going to make the greatest impact for the population we serve? Is it affordable? How do we implement it effectively?

Rachel Anthwal, Bristol Clinical Commissioning Group

Read the blog at the end of this article

Formal research-based knowledge and management journals were the lowest source of influence. Given the importance of knowledge by experience, the study confirmed the value of formative spaces like action learning sets. These provided a space for reflection where managers can 'transpose' knowledge into practice.

Another study by Christine Edwards in 2013 combined case research in five sites with a large survey of over 2000 managers in commissioning and provider organisations. Over two thirds of respondents found it difficult to access information. As well as reinforcing the importance of learning by personal experience, both studies show the way managers access and use evidence is complex and non-linear. This does not fit well with existing models for providing information in formal 'products’ or library type evidence services.

Similarly, a project by Jacqueline Swan in 2012 looked at how commissioning managers, public health experts, finance managers and clinicians used information. Survey results showed the source of evidence most used in commissioning decisions was local public health intelligence and examples of good practice from other healthcare sites.

Making sense of research

A key finding was that 'evidence does not speak for itself', but needs to be mobilised by the right people, at the right time, to affect decisions. The Swan study found that knowledge does not exist independently as intact products, but emerges through 'co-production' by managers in their own networks and groups, who make sense of key findings and frame evidence around local context and real issues. This chimes with the useful notion of clinical 'mindlines' or communities of peers creating and building knowledge (Gabbay and LeMay 2004).

Traditional ways that researchers share findings are not likely to work for decision-makers in health and care. One rich comparative case study of four commissioning organisations by Lesley Wye in 2015 found that managers drew on a wide range of sources to make decisions. These included best practice guidance, local expert views and examples from other sites. Little evidence was drawn from academic research.

Local data and evaluations seemed to trump national or formal research-based information. Managers selected evidence pragmatically to make the best case for action. They conveyed information through conversations and stories, best suited to fast-paced decisions.

Researchers like to write, but commissioners like to talk. Through conversations, discussions and stories, commissioners can get timely, relevant, adaptable, contextually-specific information quickly. This suits their information needs and working environment.

Lesley Wye, Senior Research Fellow, University of Bristol

Read the blog at the end of this article

How do health care organisations use research evidence?

We do not know enough about the reasons why some organisations are better than others at using evidence to make decisions. Attention has shifted in recent years from a focus on how individuals seek and use information, to how organisations draw on evidence and what support they need to do this well.

Are we learning?

The capability of organisations to use evidence is known as ‘absorptive capacity’. This concept of the readiness of organisations to use research was developed outside health, linked to concepts of learning organisations (Davies 2000). A study by Jacqueline Swan published in 2017 used comparative case studies in eight locations to look at how commissioners use NICE and other evidence to redesign services.

An important finding was that the capabilities needed by organisations to make sense of evidence were not just technical (in reviewing research). The ability to engage experts and know how to frame and interpret findings which would resonate with target audiences, were also required.

Another study, not yet published, is looking at the absorptive capacity or critical use of evidence in six commissioning networks, with a focus on decisions about keeping people safely out of hospital. It includes the evaluation of a tool for commissioners to assess and improve their own ability to make appropriate use of evidence in everyday decisions.

Evidence briefing service?

Some have argued that organisations, particularly those commissioning services and making substantive investment decisions, need specific support to find and package relevant evidence.

One study by Paul Wilson evaluated different forms of support for commissioners. Nine organisations tested options, from a full briefing service on demand through to an unsolicited push of non-tailored evidence, with an intermediary level of service with some contact.

Although the follow-up response was low, findings published in 2017 suggested that access to a demand-led evidence briefing service did not improve the uptake and use of research evidence by commissioners, compared to less intensive alternatives.

The authors concluded that managers were well intentioned but inconsistent users of evidence. The informal nature of much decision-making also made it hard to track how evidence was being used.

A number of studies have emphasised the importance of having experts with credibility to interpret and make sense of evidence 'in the room' when decisions are made. Studies note the role of public health staff as trusted critical friends and intermediaries in this capacity. Public health staff were seen as able to combine local intelligence (data and knowledge of services and populations) with 'harder' evidence to inform commissioning and other decisions. This appeared to be true even under new arrangements with relocation of public health function to local authorities.

Many of these studies emphasise that timing is key. Good enough information at the right time trumps high quality evidence which arrives too late for decision makers to use.

About the research on how NHS commissioners use research evidence

What is the research that this highlight is based on?

This Highlight is based on the following NIHR-funded published studies:

And one project which has not yet completed:

Do you want to read more?

This section gives a bit more context about wider research, including NIHR studies and other landmark studies which have shaped what we know about how managers use evidence. It also signposts general resources for individuals and organisations, including training and networks supported by NIHR.

There is a growing evidence base on how evidence is used. This field is variously known as knowledge mobilisation, knowledge transfer and implementation science. NIHR-funded studies have contributed to this wider evidence base, from an influential 2004 review of diffusion of innovations (Greenhalgh) to a scoping review of management theories (Crilly 2013).

There is an interesting debate about the differences between the use of evidence for clinicians and for managers. While some have argued that evidence-based management is overdue (Rousseau), a seminal paper (Walshe and Shortell 2001) noted distinct differences in how different the evidence base and decision types were for clinicians and managers.

A recent NIHR study also highlighted the pressures on middle and frontline managers in the NHS, many of which could be characterised as 'extreme jobs' given intense and changing demands and pressures (Buchanan 2013). This is part of a wider body of NIHR research on management and leadership, including clinical hybrid management roles.

A related and much studied field is around the uptake of guidelines and mechanisms to facilitate this, such as audit and feedback. Resources in terms of published reviews are available through the NIHR-funded Cochrane review group on effective practice and organisation of care (EPOC), with a thread on implementation strategies.

A number of organisations have produced useful guidance for researchers to make their research more attractive to managers and other health service leaders. This includes a recent toolkit on communicating research by the Health Foundation in July 2017.

What else has NIHR done to support greater use of evidence by managers?

The NIHR has also funded collaborations between universities and health organisations, in the form of Collaborations for Leadership in Applied Health Research and Care (CLAHRCs), which include an investment in the knowledge broker role. These new forms of partnership have also been evaluated in NIHR-funded studies, including a focus on the knowledge mobilisation role (Rycroft-Malone).

NIHR has also invested in particular knowledge mobilisation fellows, supporting projects and people interested in how evidence is used in practice. More details are given here about these projects – and a useful background reading section, which provides a good introduction to the field.

From 2007-2013, the NIHR also funded a network for managers, hosted by the NHS Confederation, including research briefings for managers and linkage events, like research seminars and forums for chief executives.

Many CLAHRCs and others offer tailored training for managers on making use of evidence and reviewing research critically. Academic Health Science Networks (AHSNs) also play a role in spreading innovation and connecting healthcare managers to research.

There is increasing interest in how commissioners use evidence, with a ten-point manifesto developed by Sian Jones and Alison Turner.

You can find more information about planning how to share your research with the people who can use it to effect change.

References

Buchanan D et al (2013) How do they manage? A qualitative study of the realities of middle and front-line management work in health care. NIHR Journals Library; 2013 Jun. Health Services

Davies HTO, Nutley SM (2000). Developing learning organisations in the new NHS. British Medical Journal 2000, 320:998-1001

Ferlie E, Crilly T, Jashapara A (2010). Research utilization and knowledge mobilization: a scoping review of the literature. SDO project 08/1801/220.

Gabbay, J., le May, A. (2004). ‘Evidence based guidelines or collectively constructed "mindlines?" Ethnographic study of knowledge management in primary care’. BMJ 329(7473):1013.

Greenhalgh T, Robert G, Macfarlane F, Bate P and Kyriakidou O (2004). Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations. Milbank Quarterly, 82: 581–629

Rousseau, D. M. (2006) Is there such a thing as evidence-based management"? Academy of Management Review, 31,256-269.

Walshe, K., Rundall, T. (2001). ‘Evidence based management: from theory to practice in healthcare’. Milbank Quarterly, 79(3):429-457

Blog - 'Evidence-based policy-making: the view from a commissioner'

I have always been of the mindset that to be the best commissioner I can, I need to use the best available evidence, information and best practice from across the country and abroad. But I am not sure I am typical as a commissioner.  As commissioners we are influenced by a vast range of competing factors. Some of these are:

  • clinical leads having a sole interest / solution to a problem they encounter which usually no amount of counter research evidence can shift
  • financial architecture – we have ‘payments by results’ and block contracts, which means transforming services and pulling the money out of one healthcare provider and putting into another, such as from an acute hospital into community services, is complex and fraught with difficulty
  • the many and varied competing priorities CCGs encounter – currently we have locally Sustainability and Transformation Plan (STP) formation and delivery; The Quality, Innovation, Productivity and Prevention(QIPP) Programme; turnaround and merging of our exec team with two other Clinical Commissioning Groups (CCGs); CCG assurance; performance of constitutional standards
  • the state of NHS finances - many CCGs are in deficit, or at the very least have extreme cost saving requirements year on year.
  • Local, national pressure groups or local councillors / committees

Research evidence is one of these factors, of which I value highly, but it doesn’t always help us out as commissioners. RCTs are not often applicable to pockets of our population. In addition research often doesn’t answer specific problems, e.g. is it possible to expand the patient cohort and get positive results for a population? Research often offers answers for healthcare providers and clinicians easier than commissioners, as it is more black and white – is drug A better than drug B? As commissioners, we have thornier issues, such as what one or two interventions are going to make the greatest impact for the population we serve? Is it affordable? How do we implement it effectively?

Another factor which often is overlooked by researchers is implementation which is complex even when the research evidence is clear. An example of this would be anticoagulation for patients with Atrial Fibrillation to prevent stroke. Despite knowing we should make this happen as commissioners, the research doesn’t help us overcome some of the practical hurdles to make it happen because:

  • We rely on GPs to implement this. But this is more work for GPs who are on a block budget held by another commissioner (NHS England), so we have no levers to make GP practices implement.
  • Which drug do we use to anticoagulate? Can we afford the NICE approved drug?
  • Will people diagnosed adhere to the treatment regime; if not how can we best ensure this happens?
  • If implemented, this could tip over our prescribing spend on the best drug for these patients, but we won’t see the cost savings as we are counting ‘strokes avoided’. So we won’t save any actual money on the balance sheet. This makes justifying the proposed change even harder.

In summary, our job as commissioners is to change systems (many and varied) to improve care - despite being up against people’s views and beliefs, payment systems, competing priorities and time pressure. I firmly believe that research evidence, although not perfect, can help us make better decisions. I would also, however, implore researchers to come and talk to us, to understand our pressures, our areas of worry, and help us build research evidence that answers our commissioning questions, with us.

For more information about the working world of commissioners, please see ‘evidence-based policy-making and the art of commissioning’

About the author:

Rachel Anthwal, Delivery Director, Bristol Clinical Commissioning Group

Blog - 'Finding research that answers questions of importance to CCGs can be a challenge'

The NIHR Dissemination Centre’s recent Highlight on Evidence for Commissioners really chimed with me and my experience over the past couple of years in supporting my commissioning colleagues to find, generate and use evidence of all types in the Clinical Commissioning Group’s (CCG) decision making processes.

Commissioners utilise a range of evidence, of which research evidence is only one element. However, actually finding research evidence that answers the questions of importance to CCGs can be a challenge. Commissioners need evidence that can support complex transformation programmes and to inform best use of resources for their local populations in challenging financial circumstances. They are increasingly commissioning on a wider scale as CCGs work together with local partners across the health and care system.

Barriers to overcome

However, even when relevant research evidence is available, there are barriers to accessing, interpreting, summarising and synthesising it with other types of evidence. This is a skill, and like any other, it isn’t something that can just be picked up without training and ongoing support.

It is also important that there is an organisational culture of enquiry where evidence is valued. From my experience last year of training commissioning staff in my CCG about finding and using evidence (in conjunction with library and public health colleagues) there isn’t any systemic lack of interest in knowing what the evidence says. It’s more the practicalities of doing so in a fast paced world and finding pragmatic solutions, that we need to work on. The idea of knowledge mobilisation and ‘trusted critical friends’ are therefore powerful.

The way forward

There is still more to be done on understanding each other’s worlds, with commissioners and researchers working together for mutual benefit. We in CCGs need to take advantage of the opportunities to feed into regional and national research priority setting exercises to influence the research that the NIHR commissions. We also need to work in partnership with our local academic colleagues so that we are helping them to do real-world, timely research that we can then benefit from.

In turn, researchers need to talk with us about our key commissioning challenges to inform the research they do. They can also help us by presenting their research findings in more user friendly formats that can sit alongside peer reviewed publications. We need to know what the evidence says, what it doesn’t say, where the gaps are, and what we should be focussing on for maximum impact for patients.

Whether you are a commissioner or researcher, the NIHR Dissemination Centre’s Highlight is a good place to start!

About the author

Rachel Illingworth, Head of Research, Evaluation and Evidence, Nottingham City Clinical Commissioning Group

Blog - 'Researchers: To make an impact, write less and talk more!'

I have been a researcher for over two decades. In that time, lamentations about the limited influence of research evidence have grown. But I think we researchers are largely to blame. We steadfastly insist on disseminating our knowledge in ways that we know don’t work.

Researchers usually write scientific papers, because publication is a key career performance metric. But scientific papers are read and digested by other scientists, not those who can act on our findings. Our ethnographic study showed how and why research doesn’t reach policymakers, like healthcare commissioners.

We found that local healthcare commissioners cannot retrieve papers from many scientific journals, as they often do not have passwords or subscriptions. Although open access publication helps, commissioners usually use Google, where scientific papers often do not appear - even if open access. If a commissioner can access a potentially relevant paper, the scientific jargon, ‘intro-methods-results’ structure and sheer length may be baffling, time-consuming and intimidating.

The next hurdle is in identifying and applying relevant findings. Often, there’s no clear, practical message so it’s discouraging to spend precious time wading through with a conclusion of ‘more research needed’. What’s more, successful interpretation requires someone who understands the research and can work with local commissioners to translate it into the local context. Research evidence published in scientific journals doesn’t get to commissioners because commissioners can’t access, understand, interpret or apply it.

What about guidelines?

Many researchers think that if their research informs guidelines, then job done. Commissioners often look at guidelines (especially from NICE), when revamping a service. But if the service is not under scrutiny, then the guidelines aren’t consulted. What’s more, few guidelines actually are acted on. Commissioners tend to implement the ‘doable’ ones, defined as those that align with current services that don’t cost any extra money. So as vehicles for transferring research evidence, guidelines have patchy success.

And evidence briefs?

What about short, punchy summaries of research evidence? Again, access is a challenge. Commissioners’ trusted sources include the King’s Fund, Nuffield Trust and the Health Foundation. But these ‘think tanks’ tend to distribute their own reports, not summaries of research evidence. Other national organisations producing research summaries are not often on commissioners’ radar. And anyway, evidence briefs don’t mitigate the challenge of translating research findings to the local context, especially without a research-savvy interpreter. Sometimes, commissioners have access to a local evidence brief service. In observing meetings for our study, these were appreciated, as it meant that the research evidence had actually been consulted. But evidence briefs made little difference to decision-making, even with a tailor-made service. Commissioners tended to glance over the brief quickly, pick out the findings that concurred with their own views and then move on. Again, often no one was available to translate or lead a discussion on the implications.

Researchers like to write, but commissioners like to talk

So what does this mean for evidence-based decision-making?  Well, researchers rely almost entirely on the written word to disseminate their findings, yet systematic reviews consistently find that personal contact between researchers and decision makers is crucial. Researchers like to write, but commissioners like to talk. Through conversations, discussions and stories, commissioners can get timely, relevant, adaptable, contextually-specific information quickly. This suits their information needs and working environment.

Commissioners also need ‘research translators’ to help interpret findings. But researchers stubbornly insist on pumping more written documentation into the system, thereby perpetuating the ‘research-practice gap’. We are our own worst enemies, driven to write mounting stacks of unread scientific papers to progress our academic careers.

So far researchers have had limited success in changing the behaviour of commissioners, because they are outside our scope of influence. As we can’t change them, we need to change ourselves. If research is to make a difference, then researchers have to start talking to commissioners.

A first step would be recognising that local ‘research translation’ is vital. In Bristol, we set up a team of embedded commissioners (into academia) and researchers (into commissioning) that, according to independent evaluators, had substantial success in stimulating conversations between researchers and commissioners. For more information see: www.bristol.ac.uk/primaryhealthcare/km.

Regardless of how those conversations are fostered, the wider research community needs to start making substantial cultural shifts. If we genuinely want our research to benefit society, then researchers need to write less and talk more. Now.

About the author

Lesley Wye, NIHR Knowledge Mobilisation Fellow & Senior Research Fellow, Centre for Academic Primary Care

Blog - 'The journey to evidence informed commissioning decision'

Commissioners need to deliver more radical service transformation against a backdrop of tighter finances and increasingly complex population needs. At the same time, commissioners are producing challenging Quality, Innovation, Productivity and Prevention (QIPP) projects and are undertaking more difficult decisions about what to commission and decommission.

The use of robust external evidence from research and other sources is becoming even more important, so the timeliness of the NIHR Dissemination Centre’s helpful Highlight is most welcome.

Joint working

Who is on the journey with commissioners to bring, critique and push this evidence so it is more visible and better used? The last major NHS reconfiguration resulted in public health staff moving into local authorities and library services moving into the acute sector.

Although staff and decision-makers in public health and the NHS do undertake joint work and sit together on relevant committees, they don’t share the same physical space. This distance from one another limits integral approaches, informal relationships, conversations and understanding.

Forward thinking individuals and approaches need to consider how the expertise and knowledge of public health and information services staff can be more joined up with commissioning and Sustainability and Transformation Partnership (STP) decision-makers.

In addition, at system level (mainly in NHS providers), there is a wealth of expertise in the librarian’s workforce. Health Education England are positively supporting staff development  of these professionals, with a recent policy aiming to ‘develop NHS librarians and knowledge specialists to use their expertise to mobilise evidence obtained from research and organisational knowledge to underpin decision-making in the National Health Service in England’. This brings a timely opportunity for these professions to add value in the system to support evidence decision-making.

Evidence mobilisation

In the North East and Cumbria, key individuals with a shared vision of mobilising knowledge, aligning skills and avoiding duplications across a system are taking the first steps on this journey. They represent the North of England Commissioning Support Unit (NECS), NHS Librarians and sectors of public health who, together with colleagues, have formed a Sustainability and Transformation Evidence Mobilisation (STEM) club.

By working with the leaders and STP priority themes, they are actively linking into the conversations, making the evidence components more visible, supporting the finding of relevant evidence and plan to produce this in a way that is useful and usable. It is early days, but everyone recognises this is a journey with difficult questions along the way.

It has become clear that conversations are at the centre of successful relationships. We are all keen to keep the conversations about research evidence going, acknowledging their importance to transforming care and, ultimately, to arriving at a destination of evidence informed commissioning decisions.

About the author

Shona Haining, Head of Research & Evidence, North of England Commissioning Support

Produced by the University of Southampton on behalf of NIHR through the NIHR Dissemination Centre

NIHR Evidence is covered by the creative commons, CC-BY licence. Written content and infographics may be freely reproduced provided that suitable acknowledgement is made. Note, this licence excludes comments and images made by third parties, audiovisual content, and linked content on other websites.

  • Share via:
  • Print article