Centre for Local & Regional Government Research

‘Decisive Moment’

The independent review of the Best Value Audit process

Final report to the Accounts Commission and Audit Scotland  

May 2007
Clive Grace, Sandra Nutley, James Downe and Steve Martin

Foreword

Over the last two years, some two thirds of local authorities in Scotland have been reviewed through the process of Best Value audits.  Some of these audits have reported good or very good performance against the statutory Best Value criteria; others have been less favourable.  Some councils have welcomed the auditors’ findings, but others have contested them vigorously.  For all concerned - in Audit Scotland, which undertakes the audits, as well as in the councils themselves - the Best Value audit process has been a demanding and challenging one, and has represented a significant investment of resource in terms of staff and councillors’ time, as well as money.

In the tradition of ‘physician heal thyself’, it is now appropriate for the Best Value audit approach to be subject to independent scrutiny and review in order to evaluate its strengths and weaknesses to date, and make recommendations about its future course and development.  This is the purpose of this report.  The study was commissioned as an independent evaluation from the Universities of Cardiff and Edinburgh.  It was led by Dr Clive Grace and Professor Sandra Nutley, with Dr James Downe and Professor Steve Martin completing the research team.

It has been a fascinating and rewarding study to undertake, and as a research team we have learned a lot ourselves, as well as hopefully contributing to knowledge and understanding about the Best Value audit approach.  We would like to thank all those who gave so generously of their time and views in interviews and by responding to the survey we carried out.  Special thanks go to Carol Calder, our key contact at Audit Scotland, who was immensely helpful throughout the work.

The overwhelming impression from all the material we collected was of a local government  ‘family’ who were all strongly committed to improving local authority services and community leadership - most importantly the councils themselves, but also COSLA, the Improvement Service for Scotland and SOLACE, as well as Audit Scotland, the Scottish Executive, and other stakeholders.  This is undoubtedly a decisive moment for local public services in Scotland with the changes ushered in by the new electoral arrangements, with the arrival of a new administration in the Scottish Executive, with local authority performance indicators being revised, and with the Crerar Review of scrutiny due to report in the near future.  We hope that our report will help to contribute to this momentum for change and strengthen the shared endeavours of all stakeholders to secure improved services and governance.

May 2007

Contents
			
									       page
		   
Foreword			2

Acknowledgements and Authors		4

Summary and Recommendations		5

1.0	Introduction		10
	
2.0	Background and Policy Context		14

3.0	Assessment of the Best Value Audit Process		20

4.0	Key Findings		28

5.0	The Way Forward		51

Annex 1   The Research Team		60
Annex 2   Interview and case study topic guide		63
Annex 3   Survey questions		69
Annex 4   Bibliography		72

Acknowledgments

We acknowledge with gratitude the assistance of the many national stakeholders and officers and councillors in local authorities who agreed to be interviewed in the course of the study.  We are also grateful to the officers who completed our survey, and we want to place on record our appreciation of the invaluable help and guidance that we have received from Audit Scotland.

The views expressed are however our own and do not necessarily reflect those of Audit Scotland, the Accounts Commission or the case study councils.

The authors
Clive Grace is Honorary Research Fellow in the Centre for Local & Regional Government Research, Cardiff University

Sandra Nutley is Professor of Public Management at the University of Edinburgh and Director of the Research Unit for Research Utilisation

James Downe is Senior Research Associate in the Centre for Local & Regional Government Research, Cardiff Business School

Steve Martin is Professor of Public Policy and Management and Director of the Centre for Local & Regional Government Research at Cardiff University

Summary and Recommendations

S1.	This independent review of the Best Value audit (BVA) process was commissioned by the Accounts Commission as part of its wider review of the progress of the audits of Best Value and Community Planning.  The study was undertaken by a team from the Centre for Local & Regional Government Research at Cardiff Business School and the School of Management at Edinburgh University.   The overall objectives of the independent review were to:
•	Evaluate the effectiveness of the Best Value audit approach to date in making judgements about Best Value and promoting improvement in council performance;
•	Identify relevant knowledge and experience from the research literature and from practice in the rest of the UK; and,
•	Make recommendations for improvement of the BVA process, with a view to ensuring that the audits are proportionate and focused on service performance and the experience of citizens.

S2.      The approach we took was to consider six main areas of investigation:
•	How well are current Best Value audit methods working? 
•	How effective are Best Value audit reports?
•	What are the views of local authorities and other stakeholders on the role of the Accounts Commission?
•	What impact are Best Value audits having on councils?
•	What impact have audits had on citizens, service users and other stakeholders?
•	How should the Best Value audit approach develop in future, both in itself and in relation to other existing and potential new methodologies of providing assurance and supporting improvement and in the wider context of public services reform in Scotland?

	We undertook our investigations between mid December 2006 and the end of March 2007.  

S3.	Best Value and Community Planning were introduced as statutory duties for local councils by the Local Government in Scotland Act 2003 which requires them to:
•	Secure Best Value (defined as achieving continuous improvement in the performance of functions); and
•	Initiate, facilitate and maintain Community Planning to ensure organisations work together to provide better public services and engage local communities in making decisions that affect them.

S4.	In developing its BVA approach, the Accounts Commission (and Audit Scotland) drew on its prior experience of the Local Government Management Arrangements audits.  The key principles which underpinned the resulting BVA process were:
•	Responsiveness to local context; 
•	No league tables or scores;
•	Corporate arrangements; and
•	Continuous improvement. 

S5.	Key features of the way in which BVAs are currently undertaken include audit by a central specialist team and a cyclical audit.  The first Best Value audit report was published in 2004.  Reports on 18 of the 32 councils had been published by 31 March 2007.  

S6.	Audits have taken place against the backdrop of a number of very important wider policy developments including the Public Service Reform agenda. The Scottish Executive's Public Services Reform document sets out an approach which seeks to drive up quality and encourage innovation and to ensure that services are increasingly user focused and personalised.

S7.	Our overall assessment is that the BVA approach is now established, has gained credibility, and in broad terms has been effective.  The best test of this is that even those who have significant criticisms of the BVA approach almost invariably argue that it needs to be fine tuned rather than abandoned. It is also warranted by comparison with the broadly equivalent methodologies in Wales and in England as reflected in the WPI and CPA.  

S8.	There are some strong similarities between the BVA approach and both of these regimes.  BVA, the WPI and CPA draw on the same underlying rationale i.e. the assumption that continuous improvement in a council’s functions and services is only possible over the medium term if the corporate processes are working well.  This model of improvement has particular relevance as councils are increasingly required to address complex ‘wicked’ issues which cross traditional service and organisational boundaries.  

S9.	Quality of BVA reports - There has been an overall improvement in the quality of the reports as the BVA process has developed and matured.  We compared published BVA reports with what we took to be ‘best practice’ examples of CPA reports (WPI reports are not public documents) and found no fundamental differences in quality.  

S10.	Professionalism of audit teams - One of the most important aspects of the BVA approach has been the degree of expertise and professionalism of the audit teams which have carried out the work.  There is convincing evidence from a number of sources that the perceived credibility of teams has varied.  Many of the weaknesses appear to us to have been in the early phases of the development of BVA. It is plain from Audit Scotland's own evidence to us that they have recognised the issue and have invested, and continue to invest, considerable efforts to strengthen the Best Value audit teams across all of these dimensions.  The changes to the Best Value audit teams are fundamental to the success and development of BVA and the contribution it is capable of making.  

S11.	Clarity of method - Because the audit teams’ theory of improvement has not been made explicit, there has been a lack of clarity about the criteria that they are using to judge councils.  This is an important area for attention because it fuels concern about the transparency of the process and the issues on which judgments are based.  

S12.	In our view a future round of the BVAs will therefore need to be based on a more explicit understanding of how improvement happens and will need to ensure that the:
•	Framework for assessing the forces and factors which actually contribute to a council’s ‘equation for change and improvement’ is aligned with the theory and model of improvement employed by Audit Scotland;
•	Self-assessment process by councils in turn tunes into that framework; and
•	Best Value audit process then provides a challenge to and test of an authority’s self assessment by a team with the skills and experience required to make independent judgements which command confidence and credibility.
 
S13.	Consistency - The issue of perceived inconsistency arises in a number of forms. The first is  that councils that have gone through the process later have experienced it as more demanding and challenging than those which went through it earlier.  The second ‘inconsistency’ relates to judgements made between councils. The third perceived ‘inconsistency’ that we have found is between the BVA report and other audit reports which a council has received previously.  The fourth ‘inconsistency’ is that some councils report that there has been a significant difference between the verbal feedback given to them by audit teams at the end of their work on-site and the written report which has followed (the latter is seen as having been more critical) and then in some cases a further gap between the judgements made by Audit Scotland, and those reached by the Accounts Commission.    

S14.	The issue of ‘impact’ - It is clear that BVAs have had significant positive impacts on councils, but that their effects have been uneven.  Nearly all councils have benefited from the process of self-assessment and the conscious and systematic review of their own processes and performance that it has entailed.  

S15.	There is no evidence that BVAs have had a direct impact on service users.  Given the nature of the process and the way in which it has been conducted, this is to be expected, but this needs to be made clearer to councils and others.  The impact on councils’ partners has also been negligible because to date BVAs have focused on councils’ corporate capacity rather than on community planning.  But this is clearly an area that demands more attention in the future.

S16.	Despite the improvements which have been made as the BVA process has developed the evidence that we have gathered points to a number of issues which still need to be tackled in addition to those that we have highlighted above.  These include the strong managerial focus of BVA reports;  relatively little attention to community planning; insufficient linkage between the Best Value audit work and the work of other inspectorates, and the annual audit process; the phrase Best Value Audit has caused confusion;  limitations in the use and interpretation of performance data and of the statutory performance indicators and in the extent to which the BVA reports deal with questions of probity and propriety and with the ‘joining up’ agenda; and, BVA reports have limited value in terms of public reporting.  

S17.	The way forward - Our recommendations are informed by ‘two faces of improvement’:
•	The first ‘face of improvement’ is improvement in council performance.  This may be stimulated by a range of factors including internal forces for change, external scrutiny, sanctions and incentives, and public expectations; and  
•	The second ‘face of improvement’ is the change which takes place in the methods and approach to external scrutiny in response to improvements in councils and/or as a result of learning and development by audit bodies.  

S18.	Our recommendations address three key areas:
•	Developing the purpose and the method of BVA in light of experience to date;
•	Developing and adapting BVAs in the light of changes taking place in councils; and
•	Developing BVAs in the light of developments in the overall strategy for reform and regulation of public services.

S19.	Developing the purpose and method of BVAs - In our view, the future development of the purpose and method of BVAs needs to:
•	Build on the baselines provided by the first round of BVAs;
•	Clarify the purpose, scope and theory of improvement associated with BVAs;
•	Enhance the credibility and capacity of BVA teams;
•	Strengthen the links between BVA and other forms of external scrutiny; and,
•	Improve reporting, publication and communication of the findings of BVAs.

S20.	The BVA approach is now sufficiently well established and well regarded that the first round of audits can be seen as providing credible baselines against which to measure future progress by councils.  There are a number of ways in which we believe that the Accounts Commission and Audit Scotland need to clarify the purpose, scope and methods of BVAs.  In our view the credibility and capacity of the BVA team needs to be developed as a matter of priority.  This should involve putting in place a Development Plan for the Team as a whole and supplementing the existing team with senior ‘associate’ peers drawn from right across the UK. There is ample evidence from councils and other evidence that BVAs and other forms of audit and inspection are not currently sufficiently co-ordinated.  There are a number of ways in which this issue could be addressed including improved co-ordination of the timetabling of inspections and better sharing of information between inspectorates; the introduction of relationship managers or a lead agency to be the first point of contact with councils on behalf of all inspections; and the merger of some or all of the inspectorates that oversee councils.  There is a need in future to Improve the links between BVAs and the annual audit process, and improve the links between BVAs and other inspections.  In our view there are a number of areas in which reporting of BVAs might be improved including removing the anomaly around publication, increasing public involvement in BVAs, improving reporting to the public, and a more active engagement of the media. 

S21.	Developing and adapting BVAs in the light of changes taking place in councils - In our view the future development of BVAs should involve:
•	A more proportionate and flexible approach;
•	An increased emphasis on self assessment;
•	A greater emphasis on corporate outcomes;
•	An increased emphasis on community leadership; and
•	Better use of benchmarks and existing good practice.

S22.	Developing BVAs in the light of developments in the overall strategy for reform and regulation of public services - There are at least four important developments that we believe could help to shape the context within which future BVAs are implemented:
•	Making better use of the resources within and available to the ‘local government family’ itself;
•	The findings of the Crerar Review;
•	A restatement of key regulatory principles; and
•	The establishment of a task force to help produce and develop future guidance relating to BVAs.

Section 1: 	Introduction 

1.	This independent review of the Best Value audit (BVA) process was commissioned by the Accounts Commission as part of its wider review of the progress of the audits of Best Value and Community Planning.  The study was undertaken by a team from the Centre for Local & Regional Government Research at Cardiff Business School and the School of Management at Edinburgh University.  It is intended to complement the public consultation which has been undertaken by the Accounts Commission and which involved submissions from a wide range of interested parties. The consultation and the independent review between them will help to inform the Commission’s consideration of how Best Value audits should develop in the future.

2.	The Commission considered that the time was right to review the BVA approach and any key changes that might be needed in the second round of audits once all 32 councils have completed the first round. A key objective for the Commission was to move on from current systems and processes to achieve a more rounded assessment of service delivery and outcomes which takes more account of the views and experience of citizens and service users.  

3.	The overall objectives of the independent review were to:
•	Evaluate the effectiveness of the Best Value audit approach to date in making judgements about Best Value and promoting improvement in council performance;
•	Identify relevant knowledge and experience from the research literature and from practice in the rest of the UK; and,
•	Make recommendations for improvement of the BVA process, with a view to ensuring the audits are proportionate and focused on service performance and the experience of citizens.

4.	In reaching our conclusions we were asked by the Commission to take account of the:
•	Commitment to self assessment;
•	Importance of the local government democratic mandate;
•	Importance of local priorities;
•	Effectiveness of corporate governance of councils;
•	Need to minimise the burden and maximise the impact of Best Value audits by building on existing approaches and adopting a selective, risk based approach; and 
•	Need for effective dialogue between councils and the Commission to agree improvement plans.
5.	Experience from BVAs to date had suggested that the methodology might need to become more flexible, risk-based and proportionate, in order to recognise the differing levels of progress being made by councils, and their capacity to sustain continuous improvement.  Other issues that we were particularly asked to consider included: the perception that there were differences between the Commission’s findings and the conclusions contained in the Controller of Audit’s reports; whether or not to introduce some form of scoring or rating of councils; the potential use of peers in the audit process; and ways of enabling more opportunity for the views of service users to be reflected in the audit process. 
 
6.	We were asked to address a number of key areas for inquiry but were not required to confine our analysis only to these questions if other important issues presented themselves during the course of our work or if our own experience and analysis pointed to other questions.   In the event, during the course of the review we introduced a number of different ways of thinking about the BVA process, and these are reflected in our recommendations.

7.	The review was based on in-depth interviews with key national stakeholders and senior officers and councillors in a range of different councils which had contrasting experiences of BVAs.  As academic researchers we were able to bring independence to the study combined with substantial experience of policy relevant research and an in-depth knowledge of inspection of local government across the UK.  The Commission and Audit Scotland made it very clear that there was a genuine willingness on their part to hear contrary, challenging and uncomfortable views if the evidence called for that.  

8.	The approach we took was to consider six main areas of investigation:

•	How well are current Best Value audit methods working? 
•	How effective are Best Value audit reports?
•	What are the views of local authorities and other stakeholders on the role of the Accounts Commission?
•	What impact are Best Value audits having on councils?
•	What impact have audits had on citizens, service users and other stakeholders?
•	How should the Best Value audit approach develop in future, both in itself and in relation to other existing and potential new methodologies of providing assurance and supporting improvement and in the wider context of public services reform in Scotland?

9.	We addressed these questions by gathering evidence from eight main sources:
•	A detailed analysis of the BVA reports that had been completed by the time of the study.
•	An analysis of approaches to audit and inspection in other parts of the UK.
•	An analysis of research and policy reports, papers and other documents in Scotland and elsewhere in the UK.
•	In-depth semi-structured interviews in seven councils.
•	In-depth semi-structured interviews with senior staff in the Commission. 
•	In-depth semi-structured interviews with a range of key stakeholders at the national level. 
•	A workshop with Best Value audit staff.
•	A survey of a range of corporate and service officers in all authorities which had been the subject of Best Value audits by the time of our study. 

We undertook our investigations between mid December 2006 and the end of March 2007.  

Section 2:  Background and Policy Context

10.	Best Value and Community Planning were introduced as statutory duties for local councils by the Local Government in Scotland Act 2003 which requires them to:
•	Secure Best Value (defined as achieving continuous improvement in the performance of functions); and
•	Initiate, facilitate and maintain Community Planning to ensure organisations work together to provide better public services and engage local communities in making decisions that affect them.

11.	Prior to the 2003 Act, the duty of Best Value existed on a voluntary basis, having been introduced in 1999/2000 as a quid pro quo for a moratorium on compulsory competitive tendering.  The 2003 Act and the supporting statutory guidance and advisory notes built on the practices that had been developed since 2000, and significantly extended the powers of the Accounts Commission and the Controller of Audit's reporting duties to include the duty of Best Value and Community Planning.  

12.	In developing its BVA approach, the Accounts Commission (and Audit Scotland) drew on its prior experience of the Local Government Management Arrangements audits.  It also sought to learn from the experience of the equivalent systems in England and Wales (Comprehensive Performance Assessment (CPA) and the Wales Programme for Improvement (WPI) respectively). It was keen to ensure that the Best Value audits provided an overall assessment of an authority’s corporate capacity but, at the same time, it was keen to avoid a ‘rules-based assessment’ or one that ended up with an overall label or score for performance.  This stance was primarily one of principle but there was also a pragmatic recognition that the CPA approach could not have been transferred to Scotland at the time, even if it had been deemed desirable, because of the lack of requisite performance data.

13.	The key principles which underpinned the resulting BVA process were:

•	Responsiveness to local context – defining the goals that local authorities should aim for but allowing them discretion on the methods and routes they use to achieve these, taking into account the needs of local communities.
•	No league tables or scores – the provision of a descriptive account of each council’s strengths and weaknesses, focusing on areas where improvement is most needed.
•	Corporate arrangements – a focus on a council's corporate arrangements and how well these support Best Value. This includes examining how elected members meet their strategic and scrutiny responsibilities and how a council’s corporate management team fulfils its strategic role.
•	Continuous improvement – examining how councils can demonstrate they are achieving continuous improvement in performance.

14.	Key features of the way in which BVAs are currently undertaken include:
•	Audit by a central specialist team within Audit Scotland, with local auditor input;
•	A cyclical audit, which aims to cover all councils over a period of three years;
•	A selective and tailored audit – which responds to the identified strengths and weaknesses of each council (which means that the BVA report on each council may cover different areas in detail and, as a result, the reports are not precisely comparable); and
•	Using and building on existing scrutiny information, including inspectorate reports and the annual work of the local external auditor.

15.	Box 1 provides an overview of the key features of current BVAs.

Box 1: Overview of the Best Value audit process

In operational terms the audit process begins with a set up meeting with the local council.  The council then presents its audit submission, which is a self assessment of how well it thinks it is addressing the statutory Best Value characteristics of:
•	commitment and leadership
•	responsiveness and consultation
•	sound governance at a strategic, financial and operational level
•	sound management of resources
•	use of review and options appraisal
•	competitiveness, trading and the discharge of authority functions
•	sustainable development
•	equal opportunities arrangements
•	joint working, and
•	accountability.

The audit team uses the submission and information from other external sources - such as inspection reports, statutory performance indicators and information from the council's website - to decide the scope of the audit.  The audit scope determines where audit resources will be concentrated in undertaking more detailed work.  

Following the detailed on-site audit (which includes document reviews, interviews and observation of meetings), the audit team produce a draft report which is sent to the council to confirm factual accuracy.  The final report is presented to the Accounts Commission by the Controller of Audit.  The Controller of Audit's report is a public report and it is published with the Commission's findings after the Commission meeting at which it has been considered.

The Accounts Commission ask the council for a detailed improvement plan, which should reflect the improvement agenda identified in the Controller of Audit’s report.  The council’s local external auditor monitors progress on the council’s improvement plan in the period between Best Value audits.

There are several checks and balances within the audit process, including two internal moderation meetings during the audit.  In addition, in instances where the Accounts Commission has particular concerns, it may direct the Controller of Audit to carry out further investigations and/ or hold a public hearing. In all cases, members of the Commission visit the council and meet with elected members after the Best Value report has been published. 

16.	The first Best Value audit report was published in 2004.  Reports on 18 of the 32 councils had been published by 31 March 2007.  During this time the Best Value audit approach has been developed and adapted.  Key developments have included:
•	introducing new tools and processes to capture the evidence to support audit judgements;
•	developing the risk assessment process which guides decisions about the scope of each audit;
•	developing sharper and more focused audit reports.

17.	Alongside these ‘technical’ changes, Audit Scotland have also established the ‘seamless audit project’ to develop a better integration between annual audit work and the triennial BVAs.  
18	The Best Value audit process has also been taking place against the backdrop of a number of very important wider policy developments including the Public Service Reform agenda. The Scottish Executive's Public Services Reform document sets out an approach which seeks to:
•	drive up quality and encourage innovation; and,
•	ensure that services are increasingly user focused and personalised.

19.	To achieve these outcomes the Executive has emphasised the importance of:
•	improving efficiency and productivity;
•	joining up services; and
•	strengthening accountability.

20.	Several key initiatives have been launched in pursuit of these broad objectives.  They include:
•	An independent review of regulation, audit, inspection and complaints handling of public services in Scotland, which was launched in June 2006.  The purpose of the Scrutiny Review is to evaluate the existing scrutiny system and make recommendations to ministers on a framework for the future external scrutiny of public services in Scotland.  The review is examining the purposes of scrutiny, how it can better support continuous improvement in public services, and how it can be made more efficient and better co-ordinated.  It is due to report in summer 2007.
•	Discussions around a new performance framework, which includes agreement about the roles and responsibilities of national government and local government, and of the other members of the local government family (e.g. COSLA, SOLACE, Improvement Service, Audit Scotland).  To date, this discussion has focused on narrowing down the range of things that national government will set as priorities and allowing more space for local discretion on activities around these priorities.  It also includes the development of a new framework of performance indicators, which envisages a modest number of national indicators, supplemented by a large menu of indicators which local councils can use in their own performance management systems.
•	The efficient government agenda, which seeks to improve public services productivity.  It has five main workstreams:
-	procurement
-	managing absence
-	asset management
-	shared support services
-	streamlining bureaucracy.
•	The last of these encompasses the Scrutiny Review, the proposed local government performance information system (both of which are discussed above), and ways of tackling the concerns raised by the Community Planning Task Force.
•	Discussions around joint working arrangements, particularly focusing on the shared services agenda, community planning and working with the voluntary sector.

21.	The changes ushered in - at both national and local level - by the elections in May will also of course be important in shaping the future context for BVAs.  In particular the one-off severance scheme for local councillors and the introduction of Proportional Representation for the local government elections in 2007 has led to the retirement of large numbers of experienced councillors and an influx of new members, as well as changes in political control.  

Box 2:  Interim Recommendations of the Crerar Review 

•	A national scrutiny plan setting out priorities to be agreed between scrutiny bodies and in consultation with delivery bodies
•	A national timetable, agreed in conjunction with delivery bodies, that sets out which scrutiny body is going where, within a defined period, to avoid overlapping visits etc
•	An agreed core dataset to be provided by each delivery body, which all scrutiny bodies use in the course of their work, with the proviso that any additional information request should meet a specific set of criteria before being complied with The development of a common self-assessment framework for delivery bodies to enable greater focus on self assessment reports by scrutiny bodies
•	Scrutiny bodies to report on how their activities/work focuses on the experience and outcomes for service users and/or citizens
•	All scrutiny and delivery bodies to develop common public reporting language and assessments in their reports

Section 3:  Assessment of the Best Value Audit Process

22.	This section of the report presents our independent assessment of the BVA process based on the evidence that we have gathered from in-depth interviews at local and national level, the survey of local authorities that have undergone BVAs, our analysis of research evidence, and our assessment of audit and inspection practice in other parts of the UK.  The detailed findings from our interviews and survey are presented in Section 4.   Our recommendations about the future of BVAs are presented in Section 5.

23.	This section first provides a short overall assessment of the BVA approach and then gives a more detailed analysis of the quality of reports, audit team professionalism, clarity of the BVA method, consistency in approach, impact, and a range of other issues that our findings suggest will need to be addressed as BVA is developed in the future.

Overall assessment of the BVA approach

24.	Our overall assessment is that the BVA approach is now established, has gained credibility, and in broad terms has been effective.  The initial legitimacy of the approach was underpinned by its statutory basis and the subsequent guidance from the Scottish Executive, and it was further extended into the stakeholder community through the work of the Best Value Task Force. Getting the approach established beyond that has been no mean feat, and it has required substantial effort on the part of councils and of Audit Scotland.  The extent of its success for individual councils and the degree of success overall is explored further below.  But what is important to note at the outset is that BVAs are accepted by most councils and other stakeholders, and owned by Audit Scotland and the Accounts Commission, as a viable and credible instrument of scrutiny through which to assess councils' performance of their Best Value and Community Planning responsibilities.  

25.	The best test of this is that even those who have significant criticisms of the BVA approach almost invariably argue that it needs to be fine tuned rather than abandoned.

26.	In our view the widespread perception that the approach has credibility and legitimacy is justified, not only by the broad endorsement and acceptance on the part of the various ‘customers’ of the approach.  It is also warranted by comparison with the broadly equivalent methodologies in Wales and in England as reflected in the WPI and CPA.  There are some strong similarities between the BVA approach and both of these regimes.  In particular, as with CPA and WPI, the particular value added by the BVA approach has been its focus on a corporate overview of a council’s capacities.  

27.	BVA, the WPI and CPA draw on the same of underlying rationale i.e. the assumption that continuous improvement in a council’s functions and services is only possible over the medium term if the corporate processes are working well.  The (largely implicit) ‘theory of improvement’ which underpins the three regimes is therefore that whilst individual pockets of good service may persist and flourish, and even major services may do well for a while, for a council continuously to improve across the range of its functions it has to be well led, use its resources effectively, have good governance, and so on.  

28.	This model of improvement has particular relevance as councils are increasingly required to address complex ‘wicked’ issues which cross traditional service and organisational boundaries.  Councils require good corporate processes if they are to tackle these wicked issues.

29.	The emphasis on the corporate functions and processes was, of course, the central feature of the guidance on Best Value and the work of the Task Force.  Audit Scotland has translated that guidance into the BVA methodology, and used it as the basis for the public reports.  In turn, councils have strengthened their own attention to the Best Value characteristics such as leadership, governance, the management of resources, and performance management - either through self-assessment prior to the Best Value audit and/or during the BVA process and in its aftermath.  That this has proved beneficial is acknowledged even by those with strong criticisms of the approach.  

Detailed assessment

Quality of BVA reports 

30.	There has been an overall improvement in the quality of the reports as the BVA process has developed and matured.  We compared published BVA reports with what we took to be ‘best practice’ examples of CPA reports (WPI reports are not public documents) and found no fundamental differences in quality.  The BVA reports are more specific on key issues such as leadership and improvement than the equivalent CPA reports.  Such other differences as we found were mainly formal, although in our judgment CPA reports have a slightly clearer structure and logic, which we attribute principally to the more explicit and detailed framework which is applied in the CPA process.

31.	Some of the criticisms that councils make of the BVA approach (see section four) are in our judgement misplaced.  In particular, whilst there may be a case in future for shifting the focus of the BVA approach from corporate processes towards services and outcomes (as some councils suggested to us), the current focus is necessitated by the statutory framework and associated guidance and flows from the underlying ‘theory of improvement’ adopted by the Accounts Commission.  In our view this theory needs to be articulated more clearly and explicitly so that councils understand the reasons for the focus on corporate processes.

Professionalism of audit teams

32.	One of the most important aspects of the BVA approach has been the degree of expertise and professionalism of the audit teams which have carried out the work.  There is convincing evidence from a number of sources that the perceived credibility of teams has varied.  This seems in part to be related to the severity of criticism in the messages that they delivered through their reports, but this is not the whole answer.  Some councils which received critical reports nonetheless praised the professionalism of the team that carried out the work, whereas some of those which received favourable reports were nevertheless critical of the team that delivered them.  

33.	Many of the weaknesses appear to us to have been in the early phases of the development of BVA when the audit team was still finding its feet, when methodologies were still being developed and tested, and where the procedures to ensure consistency between teams and the full recording of evidence were not yet fully developed.  It is plain from Audit Scotland's own evidence to us that they have recognised the issue and have invested, and continue to invest, considerable efforts to strengthen the Best Value audit teams across all of these dimensions.  

34.	The changes to the Best Value audit teams are fundamental to the success and development of BVA and the contribution it is capable of making.  Therefore (as explained in section five) we believe that there is a need to strengthen the seniority of the teams, and to ensure that teams include people with experience of running services and/or organizations at a senior level.

Clarity of method  

35.	Because the audit teams’ theory of improvement has not been made explicit, there has been a lack of clarity about the criteria that they are using to judge councils.  This is an important area for attention because it fuels concern about the transparency of the process and the issues on which judgments are based.  But in our view there is an important and delicate balance to be struck.  A more explicit set of audit criteria would make for a more transparent process but also imply a more prescriptive approach which would be likely to be less sensitive to local needs and priorities.  The CPA process in England uses a more explicit assessment framework than BVAs and in this sense is more transparent.  But as a result it leans towards a ‘one size fits all’ model of improvement.  By contrast the WPI is focused on an authority’s individual improvement progress.  It is therefore more sensitive to local issues and context but is a less transparent approach and makes common standards and comparisons on a like for like basis virtually impossible.  The BVA approach sits in the middle ground between the English and Welsh regimes.

36.	There are though in our view improvements that can be made.  At present there is a degree of ambiguity as to what ‘Best Value’ actually means.  In some documents, it is referred to as the continuous improvement in a council’s functions, and in others as the continuous improvement of council services.  In addition, the position that has been taken by Audit Scotland, at least, ostensibly, has been a combination of ‘one size fits all’ (in terms of the size of the audit teams and the amount of audit work undertaken irrespective of the council’s size or other characteristics) with ‘any shape you choose’ (in terms of what councils choose to present by way of their own self-assessment, and how they choose to do so).  That combination may well have fitted the moment when BVA was first introduced, and it may have been right for the first round when a benchmark was being established, and everybody - councils themselves as well as Audit Scotland – were ‘learning the ropes’.  But that moment will certainly have passed by the time that the first round of BVAs are completed.

37.	In our view a future round of the BVAs will therefore need to be based on a more explicit understanding of how improvement happens and will need to ensure that:
•	the framework for assessing the forces and factors which actually contribute to a council’s ‘equation for change and improvement’ is aligned with the theory and model of improvement employed by Audit Scotland;
•	the self-assessment process by councils in turn tunes into that framework; and,
•	the Best Value audit process then provides a challenge to and test of an authority’s self assessment by a team with the skills and experience required to make independent judgements which command confidence and credibility.

Consistency

38.	The issue of consistency arises in a number of forms, not all of which should be seen as problematical.  The first ‘inconsistency’ is the fact that councils that have gone through the process later have experienced it as more demanding and challenging than those which went through it earlier.  This in our view is principally a function of the development of the BVA methodology and should not be regarded as a significant problem or a criticism.  
39.	The second ‘inconsistency’ relates to judgements made between councils. This is especially evident in the case of councils that are known to be performing relatively poorly in terms of service based statutory performance indicators but have been judged to have good and improving underlying corporate processes and therefore received more favourable reports than councils that have a better track record of service delivery but lack the corporate characteristics that BVAs are looking for.  This perception of ‘inconsistency’ stems from the lack of clarity around the implicit theory of improvement which underlies the BVA approach and a resulting lack of understanding of the purpose and focus of BVAs.  This is one of the reasons why we believe that the rationale for BVAs needs to be spelled out more clearly in future.  

40.	The third perceived ‘inconsistency’ that we have found is between the BVA report and other audit reports which a council has received previously.  We have not been in a position to independently verify this, but circumstantial evidence suggests that it has some force.   The development by Audit Scotland of the ‘seamless audit’ will help to address the problem, and it is also a topic we return to below.  

41.	The fourth ‘inconsistency’ is that some councils report that there has been a significant difference between the verbal feedback given to them by audit teams at the end of their work on-site and the written report which has followed (the latter is seen as having been more critical) and then in some cases a further gap between the judgements made by Audit Scotland, and those reached by the Accounts Commission.  Again, this need not be problematical in the sense that it is the function of Audit Scotland to quality assure the work of its different teams and it is responsibility of the Accounts Commission to challenge and test the reports of the Controller of Audit, and to substitute their own judgement where they see fit to do so.  But the difficulty is that councils and even some BVA teams are currently unclear about the basis on which judgements are revised.   The Chairman of the Accounts Commission told us that the members of the Commission have developed their understanding of what constitutes a ‘good’ Council in the course of considering many BVA reports, through internal discussions at the Commission and by drawing on their own considerable individual experience and expertise.  However, that understanding has not yet been set out in a way which enables councils or BVA teams to understand or indeed to challenge it.

The issue of ‘impact’

42.	It is clear that BVAs have had significant positive impacts on councils, but that their effects have been uneven.  Nearly all councils have benefited from the process of self-assessment and the conscious and systematic review of their own processes and performance that it has entailed.  It is extremely unlikely that such self assessments would have been undertaken with the same rigour and determination in the absence of the prospect of subsequent testing through a process of external audit.  It is also encouraging that as the BVA process has developed and matured, councils have become more likely to engage in more extensive self-assessment.  This is partly because experience and understanding of the process has developed and been transferred around the local government family, and partly because resources and capacities have become available to support councils in their preparation.  It is also the result of a growing awareness among councils of the consequences of receiving a critical BVA report.

43	Clearly, this increases the risk that councils will try to use self-assessments to construct a ‘narrative’ which is designed to convince BVA teams that the council is better than it actually is.  But we believe that this risk is relatively small, principally because our case studies showed a consistent commitment to genuine improvement rather than simply seeking a positive report.  The absence of a published ‘score’ through the BVA process also reduces the risk.

44.	The BVA process has had the most impact in those councils which have had inadequate corporate processes, and has been felt most sharply by chief executive officers and senior management teams.  But is has also engaged some local politicians and encouraged them to be more aware of and engaged in the improvement process.  It has been less helpful in explaining the improvement process as practised by councils themselves or as understood through the BVA approach.  And it has had little direct impact on services or service outcomes.  This is entirely understandable and indeed appropriate given the statutory guidance and the purpose of BVAs, but this has not been widely understood by councils, partly perhaps because of the multiple Best Value definitions which are used, including defining it as continuous improvement in services.

45.	There is no evidence that BVAs have had a direct impact on service users.  Given the nature of the process and the way in which it has been conducted, this is to be expected, but this needs to be made clearer to councils and others.  Similarly, there has been little direct impact on the general public beyond the short-term effect of quite limited and usually negative local press coverage.  Unsurprisingly, critical reports have attracted much more media attention than positive ones, but it is worth emphasising that the media as well as national politicians are supportive of the BVA process, regarding reports as credible and important judgements on a council’s capacity and performance.

46.	The impact on councils’ partners has also been negligible because to date BVAs have focused on councils’ corporate capacity rather than on community planning.  But given the increasing importance of partnerships in enabling councils and their partners to tackle wicked and complex issues which cross service and organisational boundaries, this is clearly an area that demands more attention in the future.

Other issues

47.	Despite the improvements which have been made as the BVA process has developed, and the improvements in the experience and expertise of the BVA teams, the evidence that we have gathered points to a number of issues which still need to be tackled in addition to those that we have highlighted above.  

•	BVA reports focus primarily on managerial leadership.  Rightly or wrongly this understandably leaves them open to the criticism that audit teams fail to take account of the importance of local political leadership and the realities of managing organisations that are strongly influenced by party political considerations.  There is a need for BVAs to demonstrate a greater awareness of the role which politicians can play in facilitating (or impeding) improvement;
•	Linked to the first point and as noted above, BVA reports give relatively little attention to community planning, or to equalities and sustainability. This is surprising given the emphasis on community planning in the statutory guidance, and the wider importance of equalities and of sustainability.  Whilst some councils have not yet fully embraced community planning, some now regard it as being centre stage.  The BVA process appears to be lagging behind them and it is unclear how the current methodology will rise to the challenge of auditing community planning processes;
•	We have found that at present there is insufficient linkage between the Best Value audit work and the work of other inspectorates, and the annual audit process.  
•	The phrase Best Value Audit has caused confusion.  Because of the use of this terminology BVA teams have been subject to the misplaced (but entirely understandable) criticism that they lack the skills usually associated with audit work.  There is therefore a need to clarify the purpose of BVAs and promote a clearer understanding of the nature of the methodology used and skills needed by BVA teams, which like CPA and WPI have more in common with inspection than traditional audit. We do not favour any hard and fast distinction in this regard, but a greater clarity of purpose.  If there were to be a terminological change, the use of Best Value Review might be more apt.
•	There are limitations in the use and interpretation of performance data and of the statutory performance indicators.  This may reflect the limitations of the current data sets but it is an issue which needs to be addressed;
•	There are also limitations in the extent to which the BVA reports deal with questions of probity and propriety and with the ‘joining up’ agenda; 
•	Finally, reports have limited value in terms of public reporting.  They are written as if for an audience which is familiar with local government functions and services.  It would be difficult for even a well informed layperson to get from them an accessible and user-friendly account of their local council’s record of continuous improvement.

Section 4:  Key findings

48.	This section presents our key findings.  As noted in section 2, the study drew upon a range of different kinds of data.  But three sources of evidence proved particularly important:
•	In-depth interviews with senior officers, councillors and partners in a range of case study authorities. These councils had contrasting experiences in terms of the conclusions of their BVA reports and in terms of the timing of their BVA, and comments made reflect experiences at different stages of a process which has changed significantly over time ;
•	A survey of all councils that had undergone BVAs at the time of our study; and,
•	Interviews with a range of national stakeholders with expert knowledge of and an interest in BVAs and the wider policy context.

49.	Between them these different sources enabled us to make a comprehensive and rounded assessment of BVAs.  The surveys enabled us to gather views from a wide range of different councils; the interviews in the case studies enabled us to probe the issues in detail; and the national stakeholders provided a broader perspective linked to the overall policy context.

Case studies

Case study authorities and interview methods

50.	Six case study authorities were chosen to reflect a range of small and large councils, urban and rural councils, and councils that had received favourable and critical BVA reports, and had been audited at different stages in the implementation of BVA (Table 4.1).  

51.	We conducted in-depth, semi-structured face to face interviews with between seven and ten interviewees in each of these authorities.  BVA provided special challenges in West Dunbartonshire and led to a public hearing, so we also paid a short visit to this authority and conducted interviews with a senior officer and a senior councillor to ensure that their perspective as an authority which experienced the ‘appeal’ aspect within the BVA process was taken into account, although the hearing itself was held at the instigation of the Accounts Commission in light of the seriousness of the issues and its wish to hear directly from the council itself before making findings on such serious matters.

Table 4.1:  In-depth interviews with councils

	Officers	Members	External partners	Total	
Argyll & Bute	7	2	-	9	
East Renfrewshire	5	2	-	7	
Edinburgh	4	3	2	9	
Glasgow	8	1	2	11	
Highland	5	2	-	7	
Inverclyde	7	3	-	10	
West Dunbartonshire	1	1	-	2	
	37	14	4	55	

52.	In each council we asked a series of questions drawn from our brief and set out in a topic guide (see Annex 2).  This ensured that we took a consistent approach and covered the same broad issues in every case study.  

53.	Views about BVAs varied between the councils and sometimes between different interviewees in the same council.  But some very consistent messages emerged.   Most interviewees valued at least some aspects of BVAs but not surprisingly most tended to focus primarily on those elements of the process that they felt needed to be improved.  As a result, our analysis gives more attention to the criticisms than the plaudits.  This should not however be interpreted as indicating that councils’ views of the process were, even on balance, negative.

54.	Our analysis of the evidence from the case studies is grouped around five main issues, the:
•	BVA approach;
•	BVA reports;
•	Impact of BVAs; 
•	Role of the Accounts Commission; and
•	Future of BVAs.

BVA approach

55.	Corporate focus - The corporate focus of BVAs was generally welcomed and seen as having been useful, but interviewees expressed some concerns about the lack of attention given to service outcomes 

56.	The way in which BVAs focus on corporate management and governance was seen as a real strength of the process.  Interviewees reported that no other inspection or audit process took this strategic and holistic view and that it had helped them to draw together an overall picture of their authority’s corporate capacity and systems.  However, the majority of interviewees believed that the next round of BVAs needed to pay more attention to service outcomes.  This was particularly the case in councils where BVAs had judged corporate processes to be weak but where services were relatively high performing.  

57.	Some interviewees expressed concern that the services that are not covered by the ‘big five’ audit and inspection bodies are not subject to rigorous external scrutiny and some believed that it would be helpful if BVAs reported on the overall performance of an authority’s services and highlighted underperforming services.

58.   Audit methods and criteria - There were concerns about the absence of an explicit audit method and transparent criteria

59.	Many interviewees believed that there had been a lack of clarity about the criteria used by BVA teams to reach their judgements.  Some thought that this was because the criteria did not exist.  Others believed that auditors were working with an implicit set of criteria about what makes a ‘good’ council but were unwilling to share this with councils.  In this regard, BVAs were compared unfavourably with the HMIE INEA inspections, which were considered to have a much more robust methodology which is based on an explicit inspection framework and is in the hands of staff with experience of running services – albeit that there are significant differences insofaras INEA inspections have a much narrower focus on a single service and can draw on much stronger performance data.  They have also, of course, been much longer established, now being well into Phase 2. 

60.	Interviewees also called for greater clarity about whether BVAs were concerned primarily with the progress of individual authorities or were comparing performance across councils.  Interviewees acknowledged that Audit Scotland had stated clearly that it was concerned with the former but they felt that in practice the judgements reached by BVA teams were influenced by their experiences of other authorities.

61.	Self assessments -The self-assessment process was widely welcomed but interviewees believed that it needed to be facilitated more effectively.

62.	All but one council described the self-assessment process as helpful, but interviewees also commented that it had taken up a great deal of time.  It would, they believed, have been less onerous if there was an agreed template for them to work to.  They acknowledged that Audit Scotland had advised councils not to spend too long preparing self-assessments but believed that it was naive to expect councils not to prepare thoroughly.  All of them claimed to have taken the BVA process extremely seriously because BVA reports were seen as having considerable potential to influence a council’s external reputation.  

63.	Interviewees in several councils complained that they had been encouraged to provide an honest self-assessment, but then felt that they had been penalised for providing a ‘warts and all picture’.  With hindsight they questioned whether they should have been quite so candid.

64.   Involvement of stakeholders - A range of stakeholders has been involved in BVAs but interviewees believed that more needed to be done in future to include partners and the public in the process.

65.	Several interviewees were impressed by the efforts of audit teams to get feedback from a wide range of people.  The involvement of councillors was particularly welcomed, although in two case studies elected members felt that they had not been sufficiently involved and, as noted above, there were concerns that audit teams took a somewhat naïve, manageralist view of the ways in which decisions are made in local government.

66.	Most interviewees believed that it was important that BVAs took more account in future of service users’ and citizens’ views, especially if BVAs become more focused on service outcomes (as opposed to managerial and governance processes).  Most accepted that audit teams were dependent on the data that councils collected about user’s and citizen’s views, although interviewees in one case study thought it odd that the audit team did not check this.  Others observed that many councils do not yet have adequate procedures for assessing user and citizen satisfaction and that this limited the ability of BVA teams to comment on this issue.

67.	Engagement with non-council bodies was widely seen as having been superficial.  Only a small number of community planning partners had been interviewed and most local authority interviewees believed that assessment of the community planning process and partnership working needed to become much more rigorous in future.  Several advocated separate audits of community planning, which analyse performance across all of the partnerships and partners in a locality.  Some believed that the next round of BVAs should embrace this wider, more ‘cross-cutting’ focus, as opposed to focusing primarily on councils.

68.  Links between inspectorates - The link with the annual audit and the inspectorates needs to be improved.

69.	All councils believe strongly that they are currently over inspected and over audited and that this imposes an unnecessary burden on them.  We were told of numerous instances where different inspection or audit teams had asked for the same information, particularly about leadership and management issues. 

70.	Many councils also commented on what they perceived to be a tension between the annual audit and the BVA processes.  In several cases, the BVA was seen as being out of step with their annual audit reports.  Liaison between the two was described as poor in one council, and some interviewees believed that their auditors and their BVA teams seemed to lack respect for each other and to be disinclined to cooperate.

71.	Management of BVAs - There were concerns about the failure of audit teams to keep to agreed timetables and about the quality of draft reports

72.	Many interviewees complained of audit teams’ failure to keep to the agreed timetable.  Some felt that they had been pushed hard by audit teams to provide data on time but that the audit team had not ‘kept its side of the bargain’.  

73.	Ongoing delays had knock-on effects on councils, including squeezing the time which they had to respond to draft reports.   Some were also concerned that delays and protracted processes of revisions meant that reports were often several months out of date by the time that they were published.  As a result councils were sometimes criticised in reports for issues that they were well on the way to addressing.  In some cases this was seen as having undermined the credibility of the report as a whole and/or having a negative impact on staff morale.

74.	Nearly all of the interviewees reported that they had enjoyed a constructive dialogue with the audit teams whilst they were on site, but that the verbal feedback which they received at the end of the audit process had been more favourable than the draft report which they received subsequently.  Many were puzzled by this apparent change of tone.  

75.	Aside from this perceived inconsistently, many councils expressed disappointment with the quality of draft reports.  In the majority of cases, the audit team had been responsive to the comments made by the council on draft reports, but the process of pointing out errors and arguing for revisions had imposed an additional burden on them.  

76.	Inclusion of peers - Interviewees believed that BVA teams should include peers with experience of running large, complex organisations

77.	There were positive comments about the skills of the audit teams which were seen as having conducted themselves in a thorough and professional manner.  But the general view was that there was a need for more input from senior staff.  Interviewees complained that audit teams had lacked people with experience of tackling large, cross cutting issues such as equalities, which feature prominently in the Best Value criteria.  They also felt that given the focus of BVAs, members of audit teams should have experience of running large, complex organisations and managing corporate processes.  

78.	There was widespread support for senior peer input to BVAs.  It was thought that this would enable audit teams to offer suggestions about how to improve, which in turn would make BVAs more ‘useful’ to councils.  But it was accepted that if this was not handled properly it could run the risk of creating a ‘cosy’ relationship between auditors and councils which would undermine the perceived independence of the process.

BVA reports

79.	Perceived fairness of reports - There were contrasting views about the fairness of the audit reports

80.	Most interviewees thought that the audit report on their own council had been fair.  However, there were concerns that they focused more on the negatives than the things that were working well – ‘all gruel and no gravy’.  In three councils there was a strong sense across a range of interviewees that their audit report had been unduly critical, usually because they disagreed with the way in which audit teams had weighted the relative strengths and weaknesses of the council.  Some interviewees believed that audit teams had largely made up their minds before site visits.  They were said to arrive with a set of preconceptions and assumptions such as ‘small councils are inefficient’ and ‘rural councils are insular’ which they then sought to confirm during interviews.  

81.	Regardless of overall perceptions of fairness, many interviewees expressed concerns about the lack of transparency about the basis on which some of the judgements were reached in the reports and they called for more information on the evidence base for judgements (‘more evidential footnotes’).  There were also concerns about the audit teams’ use and interpretation of statistics and survey data in the reports.

82.	Consistency - Interviewees believed that there has been a lack of consistency between reports on different councils

83.	Many interviewees commented on what they perceived to be a lack of consistency in the conduct of BVAs and tone of BVA reports in different councils.  There was a widespread feeling that some authorities had ‘got off more lightly than they should have done’ whilst others had been treated harshly - the same councils were mentioned in both categories by a range of interviewees in different case studies.  

84.	Interviewees felt that different reports covered different issues and they were uncertain why.  They were unsure what role risk assessments played in determining the focus of reports and believed that even when ‘similar issues are covered they get different treatment in different councils’.  

85.	Many believed that the composition of the audit team determined the tone and focus of reports – and that you could pick out who the lead auditor was by reading the report.  

86.	Smaller councils complained that because all audit teams were of similar size, larger councils were not scrutinised as closely as smaller authorities.  There was, they felt, a tendency for the BVA process to treat large and more complex organisations in a more superficial fashion because there was insufficient time to ‘dig beneath’ the self assessment.

87.   Performance rankings - There was cautious, but not universal, support for an explicit performance ranking 

88.	Many interviewees believed that there was already in practice a kind of informal league table of BVA results.  A couple of interviewees in separate councils said they had been told unofficially by a member of the Accounts Commission or Audit Scotland what their relative standing was, for example, ‘you are the third best council so far’.   Moreover, BVA reports were widely read by the Executive and by others in the local government family, so everyone knows who has been judged favourably and whose performance is seen to be inadequate.  
  
89.	A significant number of interviewees in several of the councils expressed cautious support for introducing an explicit performance classification or ranking.  It was felt that this would sharpen up the BVA reports and reduce the need for lengthy discursive text.  There was also a view that it would act as a discipline on the auditors which would require them to justify their judgements.  

90.	Not all interviewees shared this view however.  Some argued that given the wide variations in the size of councils and the rurality and deprivation of their areas, it was meaningless to rank councils; there was no firm basis for like-for-like comparisons.  Others felt that the introduction of a league table would make authorities more defensive and the BVA process more combative.  Some believed that what they regarded as simplistic scoring systems would damage public perceptions of the ‘local government brand’. 

Impact of BVAs

91.	Positive impact - Many interviewees believed that BVAs had positive impacts in their council.

92.	Nearly all interviewees cited important benefits and impacts of the BVAs.  A frequent comment was that it focused councillors’ attention on corporate issues and galvanised the council into action.  The process was also valued as a chance to ‘sit down and systematically consider what we are doing’. 

93.	In several cases it was said to have speeded up the implementation of a performance management system, resulting in improved reporting to councillors.  In another council it had led to a new staff survey.  In some cases, authorities had taken action in advance of BVAs because they knew that if they did not address issues in advance the audit team would highlight them as deficiencies.  Examples included the adoption of a more systematic approach to equalities issues and changes to political management structures.  

94.	Two councils valued the impact that BVAs had on their external reputations because ‘it provided external validation for the council’.  In two cases where reports had been critical, interviewees believed that BVAs had given the authority ‘the shake that it needed’.

95.	However, there were some interviewees who reported that the BVAs had relatively little impact.  BVA reports had, they said, not told them anything that they did not already know.

96.	Costs and benefits of BVAs - Interviewees were not convinced that the benefits outweighed the costs.

97.	The indirect costs of BVAs are considered to be substantial, estimated at 900 person hours in one council - thus representing  something more than one full time equivalent member of staff for over a year, devoted to a process designed to take place once every three years.  There were concerns that these costs are not proportionate to the risks posed by the council – ‘there doesn't seem to be any payback for being a good performer’.  And several councils felt that the beneficial impacts of BVAs did not justify the time taken up by the process.  Some complained of a lack of real added value given that several reports reflected the submissions made by the councils.

98.	In contrast, a couple of councils had the opposite experience of feeling that their self-assessment had been a wasted effort because they could not see any evidence that the audit team had used it as part of the audit.

99.	Some interviewees compared BVAs unfavourably with service based inspections which they felt provided more practical advice on how to improve.  

100. BVAs as drivers of improvement - There were mixed views about the importance of BVAs as a driver of improvement

101.	BVAs were more likely to be considered as a catalyst for significant improvement in councils that had received unfavourable audit reports.  Some of the other councils felt that whilst it had driven the improvement of corporate processes, it had had little or no impact on services.  A couple of councils did not see the BVA as a major driver of improvement of corporate processes or service improvement.  

102. Improvement plans and support for improvement - There were mixed views about the value of improvement plans

103.	One council commented that the improvement plan was the most valuable thing to have emerged from their BVA, and a majority of interviewees said the plan had been of some use.  However, a couple of councils were doubtful about its value.  In one case it was considered to be a low-key document that had not been monitored very rigorously.  

104.	Two councils had positive experiences of the support provided by COSLA and the Improvement Service around the delivery of the improvement plan, but others expressed concern about the capacity of the Improvement Service to support councils in implementing their improvement plans, and contrasted this unfavourably with the work of the IDeA in England.  In comparison with the IDeA, the Improvement Service was seen as lacking a clear focus and being under resourced. It was also felt that it lacked sufficient staff with practical, hands on experience of running councils and council services.   

105. Impacts on the public - The impact on service users and citizens has been very limited.

106.	In most cases the impact on users and citizens was considered to be negligible and there was an overwhelming view that the BVA had not improved accountability to the public.  

107.	Any impact that there had been tended to be seen as negative – the result of adverse media coverage of Accounts Commission press releases on reports, although the impact of such media coverage had been short lived.  The media role was not seen as having improved accountability.

108.	Interviewees believed that the only way to reach the public would be to produce much shorter and ‘punchier’ reports specifically for this audience.  Some believed a league table might help to attract public interest.  But others argued that there was little public appetite for reports on corporate processes of the kind which BVAs focus on, and there was therefore no point in trying to ‘drum up’ interest with ‘glossy reports’.

Role of the Accounts Commission

109.	The role of the Accounts Commission is not well understood

110.	Most interviewees commented that the role of the Accounts Commission is not well understood in local government.  They were seen as being ‘a check on the Audit Scotland report’ but there was uncertainty about the objectives of the commissioners’ meetings with councils - ‘the meeting with the commissioners was fine… but it wasn't clear what its purpose was’.

111.	The publication of separate findings by the Accounts Commission was seen as being helpful in some instances – ‘they moderated the comments in the audit report’ - but others found this dual reporting confusing, especially when different issues were emphasised in the two reports.

112.	Finally, several councils were unhappy with the way the Accounts Commission handled the press release associated with the publication of the BVA report and Commission Findings.  The press releases were said to focus on the negatives and interviewees believed this to be an attempt by the Accounts Commission and Audit Scotland to grab media attention.  This was seen as being unhelpful for councils and potentially undermining public confidence in local government services.

The future of BVAs

113.	Interviewees made a range of recommendations about the future of BVAs ranging from incremental improvements to existing processes to much more radical suggestions for change. 

114. Enhancing the role of BVAs in public accountability - Interviewees suggested a range of incremental improvements that would improve the public accountability function of BVAs. In particular there was support for:
•	Greater clarity and transparency of criteria and method;
•	Better standardisation of judgements possibly using a classification system;
•	Greater emphasis on service outcomes, including putting greater store on what a local community thinks is important;
•	More benchmarking of councils against similar councils (including those in England) - although there were some strong dissenting voices on this issue who argued that the BVA needed to stick with the remit of assessing individual progress rather than comparative performance;
•	Peer input into audit teams;
•	More condensed site visits by audit teams; and
•	Different reports for different audiences.

115.	If BVAs were developed along these lines, then a three to four-year cycle was thought to be about right, although interviewees believed that there was a need to ‘flex’ the cycle and the intensity of the process depending on the performance of the council.

116. Enhancing the role of BVAs in driving improvement - Interviewees also suggested that there was a need for changes to enhance the contribution that BVAs make to improvement.  There were frequent calls for Audit Scotland to facilitate the sharing of best practice by virtue of their knowledge of what different councils are doing and their assessment of these practices through the BVAs.  However, there was disagreement about whether audit teams should go as far as giving advice.  Some interviewees called for an approach which focused less on public accountability and concentrated instead on supporting internal improvement and local scrutiny.  Some argued that audit teams (or perhaps the Improvement Service) needed to do much more to facilitate benchmarking between councils in Scotland with English authorities.

117. Joining up and reducing the burden of scrutiny - Many interviewees argued that there was a need to ‘join up’ BVAs and other audit and inspection activity.  This was seen as a way of reducing the burden of inspection and also a necessary response to increasing partnership working through the community planning process.  There were hopes that the Crerar Review would lead to better integration of audit and inspection, including the development of common frameworks for audit and inspection, co-inspection visits, and co-reporting regimes, although some interviewees commented that they had heard similar promises in the past and ‘would believe it when they see it’.

118.	Many interviewees believed that the increased importance of partnership working and shared services, meant that it did not make sense any longer to audit councils in isolation from other agencies.  Increasingly local councils facilitate the delivery of local services through working with other public bodies and voluntary and private sector providers.  Therefore the remit of the audits of community planning should now include all partners and service providers.  It was recognized that this would entail extending the duty of Best Value to all public service organisations, but many interviewees believed that this was desirable.  In urban areas in particular, it was felt that in future the BVA needs to be at a more strategic city-wide or even region-wide level because, it was argued, councils cannot be held solely accountable for the delivery of outcomes in their areas.

119.	Councils which had been judged to be doing well argued strongly that they should be subject to much ‘lighter touch’ inspection than other authorities next time round.  However, councils that had received less favourable reports believed that they should not be subject to more intensive scrutiny if they could demonstrate in advance that they had made significant improvements since the first report.

120.	There was a strong plea from smaller councils that more account is taken in future of their limited capacity to respond to audit and inspection activity.  

Survey of authorities

Methodology

121.	In order to assess the views of a wider range of officers than those in the seven case study councils we undertook a survey of senior managers in all sixteen authorities which had been the subject of audits completed by early March 2007.  The survey was undertaken using a similar methodology to that we have employed successfully in other recent studies in England and Wales.  We wanted to obtain a range of perspectives on BVAs and therefore sent it to senior officers with corporate responsibilities as well as to the heads of seven key services as follows:
•	the Chief Executive
•	Head of Policy
•	Director of Finance 
•	Head of Democratic Services
•	Director of Education
•	Director of Children’s Services
•	Director of Housing
•	Director of Planning
•	Director of Leisure and Culture
•	Director of Revenues and Benefits, and 
•	Director of Environment.

122.	We sent the survey to 173 officers and received 53 replies.  This is a response rate of 31% which is encouraging given the very short time span in which we had to complete the survey.  It includes replies from all but one council and a range of responses from different kinds of officers.  The results therefore provide a valuable means of cross checking the findings from the in-depth interviews in the case studies reported above.

123.	The survey covered the same key issues as the interviews in the case study authorities including the:
•	BVA approach;
•	BVA reports;
•	Impact of BVAs;
•	Role of the Accounts Commission; and
•	Future of BVA.

124.	In line with the approach that we have used successfully on other studies, the survey asked respondents to state the degree to which they agreed or disagreed with a series of statements about BVAs on a seven point Likert scale ranging from -3 (‘strongly disagree’) to +3 (‘strongly agree’).  In addition, we asked open ended questions which gave space for respondents to expand on their views about key aspects of BVAs. All survey questions are listed in Annex 3.

BVA approach

125.	Survey respondents were generally positive about the BVA approach.  Almost two-thirds agreed that ‘the Best Value audit team was credible’ in terms of the skills and experience that it possessed, and almost 60% felt that ‘the process was transparent’.

127.	In contrast to the interviewees in case studies, more than half (60%) of survey respondents believed that ‘the balance of attention given between members and officers was about right’. 

128.	As reported above, interviewees in several of the case studies felt strongly that BVAs did not pay enough attention to the performance of services, but this message was not reflected in the survey findings.  Two-thirds of survey respondents believed that their service area had been covered in sufficient depth.  

130.	Two main areas of concern about the approach emerged from the survey:
•	Like the interviewees in the case study authorities, many respondents felt that community planning was the ‘junior partner’ in BVAs. Only 4% believed strongly that community planning was covered in sufficient depth in the BVA of their council and 29% disagreed to some extent that this was the case.  Similarly, only a quarter of respondents believed that BVAs ‘engaged extensively with non-council bodies e.g. health service, voluntary sector etc.’ and half stated that they did not do so. 
•	The second area where large numbers of survey respondents felt that there was room for improvement was in the way in which BVAs work with other audit and inspectorates.  Nearly half of respondents believed that the coordination with other scrutiny bodies was not effective and more than a third felt that the links made with the annual audit were not effective (Table 4.2). 

Table 4.2	The Best Value Audit Approach

Question	– 3	– 2	– 1	0	+ 1	+ 2	+ 3	
Community planning was covered in sufficient depth	2.1%	14.6%	12.5%	14.6%	31.3%	20.8%	4.2%	
My service area (if appropriate) was covered in sufficient depth	4.1%	14.3%	14.3%	2.0%	20.4%	26.5%	18.4%	
The balance of attention given between members and officers was about right	0.0%	4.1%	22.4%	12.2%	30.6%	24.5%	6.1%	
The links made with the annual audits were effective	4.3%	12.8%	19.1%	19.1%	17.0%	23.4%	4.3%	
The coordination with other scrutiny bodies was effective	8.7%	15.2%	23.9%	17.4%	23.9%	8.7%	2.2%	
The audits engaged extensively with non-council bodies e.g. health service, voluntary sector etc.	7.5%	20.0%	22.5%	17.5%	25.0%	5.0%	2.5%	
The Best Value audit team were credible (skills, experience etc. of staff)	2.0%	0.0%	18.4%	14.3%	26.5%	28.6%	10.2%	
The process was transparent	0.0%	11.8%	15.7%	13.7%	21.6%	25.5%	11.8%	

BVA reports

131.	Most respondents were content with BVA reports.  A very large majority (86%) agreed that reports were ‘readable’ and 84% said that they were an appropriate length.  Almost three quarters (73%) felt that the reports covered the local context well and 69% believed that reports were ‘fair and consistent’ (Table 4.3).

132.	Some respondents said that reports did not cover services and cross-cutting issues, such as equalities and sustainability, adequately. The lack of a consistent coverage on equalities was a complaint made to the Accounts Commission in their consultation exercise. And a third of respondents believed that reports did not make adequate use of material from other scrutiny bodies.

Table 4.3 	BVA Reports

Question	– 3	– 2	– 1	0	+ 1	+ 2	+ 3	
Fair and consistent	2.0%	0.0%	17.6%	11.8%	19.6%	31.4%	17.6%	
Readable	0.0%	0.0%	5.9%	7.8%	21.6%	41.2%	23.5%	
Covered the local context well	0.0%	2.0%	15.7%	9.8%	17.6%	39.2%	15.7%	
The appropriate length	0.0%	0.0%	2.0%	13.7%	19.6%	49.0%	15.7%	
Services (part 3)	2.0%	5.9%	17.6%	5.9%	35.3%	29.4%	3.9%	
Processes (part 2)	0.0%	2.0%	10.0%	10.0%	38.0%	38.0%	2.0%	
Equalities	0.0%	2.0%	6.0%	20.0%	40.0%	30.0%	2.0%	
Sustainability	0.0%	7.8%	11.8%	19.6%	37.3%	21.6%	2.0%	
The use of available data (e.g. Statutory Performance Indicators)	0.0%	8.0%	10.0%	14.0%	22.0%	42.0%	4.0%	
Material from other scrutiny bodies	4.2%	0.0%	29.2%	14.6%	27.1%	22.9%	2.1%	

133.	Respondents made a range of suggestions for improving reports, most of which echoed comments made by interviewees in the case studies.  The main suggestions were:
•	Increase the focus on service delivery and outcomes and give less attention to processes.
•	Give more emphasis to the local context - Although a majority of respondents believed that reports took account of local contexts, some suggested that there was a need to give more attention to differences in levels of deprivation.
•	More engagement with staff, partners and elected members -  Like the interviewees in case study authorities, survey respondents believed more should be done to take account of a wider range of views and in particular that BVA teams should have a greater level of engagement with staff, elected members and private and voluntary sector partners.
•	Dissemination of good practice – Like interviewees in the case studies, survey respondents argued that BVA teams should do more to disseminate good practice.

•	More balanced reporting – Like interviewees in case studies, many survey respondents believed that reports gave too much emphasis to negatives which contributed to adverse and unbalanced media reporting.  It was suggested that all summaries should highlight good performance as well as areas for improvement.  One officer called for the Accounts Commission’s findings to be separated from the report because they did not reflect the findings in the report.  
•	Better use of evidence - A number of officers raised concerns about the evidence used to support the judgements made in reports, particularly in relation to Statutory Performance Indicators (SPIs) – but rather different ones.  As three respondents put it:
‘I would have expected more use of SPIs and other performance data’ (Director of Finance).

‘Too much credibility is attached to SPIs, many of which are confusing, ambivalent or meaningless’ (Director of Service).

 ‘The Audit - if relying on SPIs - should look at trends in performance on these SPIs over time rather than a snapshot of one year - as was the case in our audit.’ (Director of Service).

Impact of BVAs

134.	Views of the impact of BVAs were generally very positive.  Relatively few respondents reported that they encouraged a focus on national goals at the expense of local priorities or led councils to prioritise short-term actions (Table 4.4).  But one chief executive questioned the need for BVAs at all:

‘The Best Value process has confirmed to elected members and the public that our Council is a high performing organisation. However, we do not require external scrutiny to improve our performance. That drive was already in our culture’ (Chief Executive).

135.	But a large majority of respondents believed that BVAs had helped to drive improvement in their councils.  Three quarters (76%) reported it had acted as a catalyst for improvement.  The same proportion believed that the self-assessment process had been instrumental in helping to drive improvement.  

Table 4.4:  Impact on Councils

Question	– 3	– 2	– 1	0	+ 1	+ 2	+ 3	
Improved accountability to politicians	0.0%	14.0%	6.0%	20.0%	26.0%	26.0%	8.0%	
Improved accountability to the public	2.0%	14.3%	6.1%	18.4%	30.6%	20.4%	8.2%	
Provoked media interest	2.0%	2.0%	2.0%	16.0%	44.0%	16.0%	18.0%	
Identified instances of failure	4.1%	4.1%	4.1%	30.6%	36.7%	16.3%	4.1%	
Diagnosed the reasons for failure	6.1%	2.0%	16.3%	34.7%	26.5%	12.2%	2.0%	
Acted as a catalyst for improvement (better quality and more efficient services)	2.0%	4.0%	2.0%	16.0%	30.0%	26.0%	20.0%	
Struck the right balance between 'holding to account' and 'helping to improve'	2.0%	10.2%	22.4%	12.2%	24.5%	26.5%	2.0%	
Led my council to focus on national priorities at the expense of local priorities	5.9%	17.6%	33.3%	25.5%	7.8%	7.8%	2.0%	
Led my council to concentrate on short-term goals rather than longer-term issues	3.9%	15.7%	43.1%	17.6%	11.8%	5.9%	2.0%	
Encouraged my council to develop a greater capacity for self-evaluation and inter- and intra-organisational learning	0.0%	5.9%	5.9%	17.6%	43.1%	21.6%	5.9%	
My council's improvement plan has been instrumental in helping to drive improvement	2.0%	5.9%	7.8%	17.6%	39.2%	15.7%	11.8%	
My council has received support from external organisations to help deliver actions in our improvement plan	18.2%	9.1%	20.5%	20.5%	18.2%	6.8%	6.8%	
The self-assessment process has been instrumental in helping to drive improvement	0.0%	3.9%	3.9%	15.7%	45.1%	17.6%	13.7%	
The overall Best Value audit approach has led to improvement in my council	0.0%	2.0%	12.0%	12.0%	46.0%	14.0%	14.0%	

136.	Seven in ten respondents reported that the BVA had encouraged their council to develop a greater capacity for self-evaluation and inter- and intra-organisational learning, and 67% believed that the improvement plan had been instrumental in producing improvement.  

137.	BVAs were seen as having had slightly less of an impact on accountability than on improvement.  Almost eight out of ten (78%) believed that the BVA of their council had attracted media interest, but much smaller majorities (60% and 59% respectively) believed that BVAs had increased accountability to politicians and the public.  Only half reported that BVAs currently struck the right balance between accountability and improvement.

138.	Like interviewees in the case studies, survey respondents pointed to a lack of support for improvement.  Fewer than half believed BVAs had diagnosed the reasons for failure or that they had received support from external organisations to help deliver actions in their improvement plan.

139.	We asked respondents to give specific examples of where BVAs had led to improvement in their councils.  Some said that there had been no tangible or direct improvements or that it was too early to identify specific changes or improvements.  Others reported that the BVA had helped to focus attention on issues that their council was already aware of but would not have addressed so quickly in the absence of the audit.  Examples of tangible improvements included:

•	Improvements in corporate processes - Many respondents said that the BVA had highlighted weaknesses in their authorities’ corporate processes, and had helped focus attention of addressing these through implementing a performance management system and establishing better links between corporate plan and service plans.  But, in a small number of cases, the BVA was seen as having impeded improvement:

‘All improvement plans were already in place within Council. Nothing new added or learned as result of audit. It simply delayed implementation of some of our improvement plans because of capacity to respond to audit itself’ (Head of Policy).       	
•	Changes in internal management - In some authorities, the BVA had led to management changes:

‘Resulted in a new senior management team with a renewed focus directed at improvement.’ (Director of Service).	

•	Impact on elected members - Some officers suggested that BVAs had benefits for elected members through, in the words of one respondent, ‘strengthening decision making and scrutiny arrangements’(Director of Service).
•	Service improvement – Only a very small number of respondents cited examples of direct impacts on services.

Role of the Accounts Commission

140.	A large majority (78%) of respondents believed that their council’s relationship with the Accounts Commission was good.  80% believed that the meeting with the Accounts Commission had been good, and almost three quarters that communications with the Commission were effective (Table 4.5).  

Table 4.5: Role of the Accounts Commission

Question	– 3	– 2	– 1	0	+ 1	+ 2	+ 3	
The relationship between my council and the Accounts Commission was good	0.0%	6.5%	6.5%	8.7%	28.3%	32.6%	17.4%	
The meeting between my council and the Accounts Commission was good	0.0%	2.6%	5.1%	12.8%	28.2%	35.9%	15.4%	
The communication between the Accounts Commission and my council was effective	2.4%	0.0%	14.3%	9.5%	28.6%	38.1%	7.1%	

Future of BVAs 

141.	Like interviewees in the case studies, survey respondents were concerned about the overall burden of audit and inspection and believed that there was a lack of integration between BVAs and other forms of scrutiny.  

142.	Only just over a quarter (28%) believed that the benefits of external inspection outweigh the costs and less than a third reported that the activities of the various inspectorates had become more joined-up over the last few years’ (Table 4.6).  

Table 4.6: Future of BVAs

Question	– 3	– 2	– 1	0	+ 1	+ 2	+ 3	
The activities of the various inspectorates have become more joined-up over the last few years	8.3%	27.1%	29.2%	10.4%	22.9%	2.1%	0.0%	
There has been a shift in inspection from assessing compliance to supporting service improvement in the last few years	8.3%	14.6%	33.3%	14.6%	27.1%	2.1%	0.0%	
The benefits of external inspection outweigh the costs in terms of improvements in my council	14.9%	8.5%	17.0%	31.9%	19.1%	8.5%	0.0%	
My council has the capacity to respond effectively to Scottish Executive initiatives	3.9%	13.7%	5.9%	7.8%	29.4%	23.5%	15.7%	
Audit Scotland should continue not to use league tables or scores	2.0%	2.0%	3.9%	11.8%	17.6%	29.4%	33.3%	
Future audits should include an indication of a council's direction of travel in terms of improvement	0.0%	0.0%	0.0%	2.0%	29.4%	41.2%	27.5%	
Future audits need to be more flexible, risk- based and proportionate to risk	0.0%	2.0%	0.0%	2.0%	31.4%	29.4%	35.3%	
The involvement of peer members/officers in the Best Value audit team would improve the process	2.0%	2.0%	2.0%	12.2%	30.6%	32.7%	18.4%	
Greater reliance on self regulation would help to encourage improvement	0.0%	4.1%	8.2%	8.2%	42.9%	10.2%	26.5%	

143.	Unlike some of the case study interviewees who expressed support for more explicit rankings, 80% of survey respondents were opposed to the introduction of league tables based on BVA reports.  

144.	Almost all respondents (98%) believed that in future BVAs should include an indication of a council's direction of travel in terms of improvement and 96% agreed that they needed to be ‘more flexible, risk- based and proportionate to risk’.   Like the interviewees in case studies, there was support from survey respondents for using peer review in BVAs and for an increased emphasis on self-assessment – both developments were advocated by 80% of respondents.
145.	The main changes in the next round of BVAs suggested by respondents were:
•	Increased focus on service delivery - As noted above, some respondents felt that there is too much emphasis on processes rather than outcomes in the first round of the audits, and that this needed to be reversed in future audits.
•	Improve co-ordination with other inspectorates - Respondents suggested that more needs to be done to improve the co-ordination with other inspectorates in order to learn from their ‘good practice’ and reduce the burden of scrutiny. In turn, the inspectorates need to place more reliance on Audit Scotland’s work.
•	Increased focus on Community Planning – including more in depth reviews of the performance and contributions of other key partners rather than just focusing on the Council as lead authority.
•	More emphasis and reliance placed on self-assessment to allow the overall approach to be streamlined.
•	More transparency about the cost of the process so that the value for money of BVAs can be assessed.
•	Focus on weaknesses – areas of greatest risk
•	A lighter touch for better councils
•	Greater sharing of findings at an earlier stage in the process.

146.	Overall, there was a recognition that the future of BVA needs to be in tune with the changing agenda for local government:

‘Key drivers for the Council in terms of efficiency and improvement are the Scottish Executive's Service Reform Agenda and Efficient Government. At some point we will be required to report back to the Executive on efficiency. There needs to be some thinking done on how these agendas tie in with Best Value and the Best Value Audit as they have become significant drivers for change’ (Chief Executive).

National stakeholders

147.	We interviewed a range of stakeholders including representatives of national organisations in the local government family, journalists, senior civil servants, the Scottish Consumer Council, and staff from private audit firms with an involvement in BVA.  Unsurprisingly, there was a range of different perspectives on BVAs.  Here we give an overall summary which represents the centre of gravity of the views expressed to us.  We have grouped these around the same six main issues as we explored through the interviews in case study authorities and survey, namely the:

•	BVA approach; 
•	BVA reports;
•	Impact of BVAs;
•	Role of Accounts Commission; and 
•	Future of BVA.

BVA approach

148.	Overall, the BVA approach was well regarded by the national stakeholders and opinion formers to whom we spoke.  As with survey respondents and interviewees from the case studies, they saw self assessment as being particularly valuable.  However some other aspects were subject to criticism.  In particular, like many of the interviewees in our case study authorities, some national stakeholders believed that there was too much emphasis on corporate managerial processes and that more attention needed to given to service outcomes.  BVA teams were seen as having identified the right issues in most authorities, but interviewees thought that this was often in spite of rather than because of the BVA methodology, which was seen as being too loose.  Some national stakeholders also believed that the audit teams lacked skills and expertise they needed to judge authorities’ corporate capacity.

BVA reports

149.	BVA reports were generally seen as being readable, covering the right issues and being the right sort of length.  Some interviewees believed that there was a degree of inconsistency between reports in terms of their coverage.  The strongest criticisms focused on what was seen as the unduly negative tone of summaries and press releases.  There was a suggestion that there was a ‘hierarchy of negatives’ from the reports, which were the most positive in tone, through to the summaries, onto the views of the Accounts Commission, and into the press releases.  ‘Uncontrolled escalation’ was how one interviewee put it.

Impact of BVAs

150.	There was a broad consensus that BVAs have had a significant impact on councils, especially in relation to corporate management issues and awareness of the need to improve.  However, views varied as to whether these beneficial consequences should be credited principally to the self-assessment process or to the BVA as a whole.  There was some concern about the level of resource being absorbed by the BVA process, and a view that the process was taking up more time and other resources than had originally been intended.

151.	Like the interviewees in the case studies, national stakeholders believed that the public was really only aware of BVAs when there was negative media coverage.  But insofar as the media and national politicians and the Scottish Executive can be taken to be a ‘proxy’ for impact on citizens and service users, then it is clear that BVA has had quite a significant influence on thinking and knowledge about the performance of local government.  National interviewees regarded BVA reports as authoritative and valuable sources of assessment and saw them as being of considerable interest to national policy makers and commentators as well as local politicians and local authority officers.

Role of the Accounts Commission

152.	There was a general acknowledgement that BVAs had raised the profile of the Accounts Commission, both through the general role played by the Commission in relation to BVA and, most notably, through the publicity attracted by the public hearing on West Dunbartonshire.  There was also uncertainty and even confusion about the roles and identities of the Accounts Commission and Audit Scotland.  Both of the journalists who we interviewed were unsure about what point a BVA report becomes a public document.

Future of BVA

153.	This was perhaps the area about which stakeholders’ view varied most.  There was a general consensus that BVAs were valuable and should be retained.  Some interviewees set out quite an ambitious agenda for their future development, which have also been reflected in their formal responses to the wider consultation carried out by the Accounts Commission.  However, most argued for more modest adjustments which would correct the perceived deficiencies which we described earlier.

Section 5:  The way forward

154.	The brief for the study asked us to make recommendations ‘with a view to ensuring that audits are proportionate and focused on service performance and the experience of citizens’.  This final section therefore considers ways in which the BVA approach can be improved and changed - not only in its own terms but in tandem with current developments both within local government and the wider context of public services reform in Scotland.  

155.	Our recommendations are informed by ‘two faces of improvement’:
•	The first ‘face of improvement’ is improvement in council performance.  This may be stimulated by a range of factors including internal forces for change, external scrutiny, sanctions and incentives, and public expectations.  
•	The second ‘face of improvement’ is the change which takes place in the methods and approach to external scrutiny in response to improvements in councils and/or as a result of learning and development by audit bodies.  

156.	This review is itself an expression of the interplay between these two faces of improvement.  There have been improvements in councils over the last two years and BVAs have evolved as Audit Scotland and the Accounts Commission have learnt from the experience of the early audits.  Our recommendations seek to identify the best way forward in the light of these developments and the wider context of public services reform including in particular the need for more joined up working between local public service providers.

157.	Our recommendations address three key areas:
•	Developing the purpose and the method of BVA in light of experience to date - in other words doing better what the Accounts Commission and Audit Scotland have already set out to achieve;
•	Developing and adapting BVAs in the light of changes taking place in councils – including improvements in their performance, new policy demands placed upon them, and changes in the environment in which they work - social, demographic, and technological; and 
•	Developing BVAs in the light of developments in the overall strategy for reform and regulation of public services.

158.	To draw hard and fast distinctions between these three areas would be counterproductive because the successful future development of BVAs in any one is dependent on progress being made in the other two.  But separating them in this way does hopefully provide greater clarity about what needs to be done, when and by whom.

Developing the purpose and method of BVAs

159.	In our view, the future development of the purpose and method of BVAs needs to:
•	Build on the baselines provided by the first round of BVAs;
•	Clarify the purpose, scope and theory of improvement associated with BVAs;
•	Enhance the credibility and capacity of BVA teams;
•	Strengthen the links between BVA and other forms of external scrutiny;
•	Improve reporting, publication and communication of the findings of BVAs.

Build on the baseline provided by the first round of BVAs

160.	In our view, the BVA approach is now sufficiently well established and well regarded that the first round of audits can be seen as providing credible baselines against which to measure future progress by councils.  This does not rule out the need for some significant changes in BVAs in the future, but it does reflect our view that they are essentially fit for the purpose for which they were designed.  

Clarify the purpose, scope and theory of improvement

161.	There are a number of ways in which we believe that the Accounts Commission and Audit Scotland need to clarify the purpose, scope and methods of BVAs. 
•	Purpose - It is important that the Accounts Commission and Audit Scotland make it clear to councils that BVAs are intended both to increase accountability and encourage improvement and explain how they are intended to achieve this.
•	Scope – It would also be useful if the boundaries of BVAs were made explicit.  It should be made clear that audit teams identify ‘what’ aspects of a council’s performance needs to improve and explain ‘why’ such improvements are required, but stop short of recommending ‘how’ improvements are to be made (other than perhaps pointing to exemplars of good practice examples and identifying the kinds of capacities which councils may need in order to achieve improvement).  Put another way, audit teams ‘hold councils to account for their improvement’ but will not get involved in helping councils to improve.  A third possible formulation of this distinction is that the role of BVA teams is to assess the ‘business risk’ that councils will not be able to improve and to pinpoint the main risk factors (as happens in the WPI). 
•	Definition of ‘audit’ – There should be clarity as to what is meant by ‘audit’ in the context of BVAs, so that councils are clear that the methods used are not those usually associated with the traditional forms of audit with which local government is perhaps more familiar.  
•	Theory of improvement - We believe that it is important that the understanding (or theory) of improvement which underpins BVAs is articulated more clearly.  As a minimum it is important to make more explicit the assumptions about the link between corporate processes and service outcomes.  It would also be useful to explain how the various elements of Best Value as set out in the statutory guidance and task force documentation act both independently and together to enable councils to fulfil their Best Value responsibilities.  It may not have been possible to do this at the outset, but the extensive experience that the Accounts Commission, Audit Scotland and councils have accumulated means that they should now be able to articulate this.
•	Roles and responsibilities - There needs to more clarity about the respective roles and responsibilities of the Accounts Commission and Audit Scotland.  Perhaps the simplest formulation of the ‘division of labour’ between them is that the Accounts Commission is the explicit owner and steward of the characteristics that make for a good ‘Best Value’ council (and its associated theory of improvement) and that Audit Scotland is the delivery body which assesses councils against the model.  A statement along these lines would, we believe, be helpful to councils and other stakeholders.

Enhance the credibility and capacity of BVA teams

162.	In our view the credibility and capacity of the BVA team needs to be developed as a matter of priority.  This should involve at least two actions:
•	Development plan - There is a need to put in place a properly resourced development plan for the team as a whole and for its individual members.  The plan should take account both of the need to complete the current round of BVAs and also to develop and implement the methods needed in future rounds.  
•	Associate peers - The existing team should be supplemented with senior ‘associate’ peers drawn from right across the UK to strengthen its seniority, capacity and credibility. 

Strengthen the links between BVA and other forms of external scrutiny

163.	There is ample evidence from councils and other evidence that BVAs and other forms of audit and inspection are not currently sufficiently co-ordinated.  There are a number of ways in which this issue could be addressed including:
•	Improved co-ordination of the timetabling of inspections and better sharing of information between inspectorates;
•	The introduction of relationship managers or a lead agency to be the first point of contact with councils on behalf of all inspections; and,
•	The merger of some or all of the inspectorates that scrutinise councils.

164.	These kinds of changes to the wider infrastructure of external scrutiny are worth considering, but they have wider ramifications and are a matter for the Crerar Review rather than for our study with its narrower focus on BVAs.   However, it is clear that, whatever form the solution takes, there is a need in future to:
•	Improve the links between BVAs and the annual audit process – There were good reasons for adopting a three-year cycle for the first round of BVAs.  But, in our view, there could be more frequent but much lighter touch BVAs once the first round has been completed.  Potentially this could involve linking BVAs more closely to the annual audit process, a step which would seem to us to be a natural development of the Audit Scotland project to develop a ‘seamless audit’.  
•	Improve the links between BVAs and other inspections – We believe that moves towards a more joined up approach to BVA and the annual audit process should go hand in hand with a determined effort to link up BVA with other inspections and external scrutiny.  We would like to see Audit Scotland and the inspectorates providing a composite picture of councils with joint or at least ‘federal’ reporting (in line with the recommendations of the report on the ‘The role of external review in improving performance’ produced by Sir Michael Lyons and Sir Ian Byatt).

165.	Clearly, it would not be possible for councils to go through on an annual basis the same kind of extensive self assessment exercises that have been involved in the first round of BVAs. Nor would it be possible for Audit Scotland to undertake the same level of intensive fieldwork which currently informs BVA reports.  Instead we envisage an approach that has the advantage of providing ‘real time data’ and involves a level of external scrutiny which is more proportionate to the perceived risks that a council will fail to deliver its duty of Best Value.  

166.	This is likely to have staffing and organisational implications for Audit Scotland.  It may be that BVA audit teams will increasingly be integrated with or linked to ‘mainstream’ audit.  But we believe that it will be important to maintain at least a small central BVA team which is able to undertake specialised, focused and more intensive work where a council is failing to improve or to deliver Best Value.

167.	Strengthening the links with other inspectorates in this way will also make it easier over time to develop a composite picture not only of individual local services in an area, but also of the public services picture in that area as a whole.  This would be part of a developing focus on achieving outcomes for the ‘place’ which a local authority is responsible for, and would constitute a development of the current BVA assessment of the community planning function.  This ‘territorial’ assessment and perspective would be further strengthened if the Best Value duty is extended fully to other public services, so that the ‘local’ contribution of national agencies could also form part of the composite picture. This would also help to inform citizens and service users, who are perhaps more likely to relate to the kind of overall picture that such a composite could help to provide.  

168.	There would be a considerable amount of work to bring all of these strands together, and there is a judgement to be made about how quickly and effectively it would be possible to move in this direction, especially in the current absence of a corresponding duty of Best Value on non-local government public services.  We see consideration of this issue taking place in the new Task Force which we envisage, as set out in later recommendations.

Improve reporting, publication and communication

169.	In our view there are a number of areas in which reporting of BVAs might be improved:  
•	Removing the anomaly around publication - The Accounts Commission should take the lead in removing the anomaly which leads to the BVA report on a council being technically in the public domain before it is published in practice after the Accounts Commission has reviewed the report of the Controller of Audit and come to its own judgement.  It should seek the support of the Scottish Executive and Parliament to effect an appropriate legislative amendment.
•	Increasing public involvement in BVAs - Given that the focus on BVAs is on corporate processes (rather than service outcomes) it is not surprising that the level of public awareness and involvement is currently very low.  Indeed, it is difficult to see why the public would want to be engaged in discussions about corporate processes or what value this would add to BVAs.  However, we recommend that all authorities collect data about satisfaction with their overall performance on a consistent and regular basis and that BVA teams should take this into account in reaching their judgements, and should make an assessment of the scope, relevance and reliability of the performance data.
•	Improving reporting to the public –The publication of council league tables might help to attract more media coverage, but establishing a scoring system would be resource intensive and attract opposition and hostility from some councils.  There are also legitimate concerns about making comparisons among the very different kinds of councils that make up local government in Scotland; about the validity of composite scoring systems (research has cast doubt on the validity of both the CPA and ‘star ratings’ used in the past in the NHS); and the impact on public confidence of these simplistic forms of reporting.  We do not therefore recommend that BVAs go down this route.  An alternative approach would be for Audit Scotland and/or councils to produce short summary reports designed specifically for service users and the wide public which could be distributed in a variety of ways including via the web.  This might increase awareness among some sections of the population, although we are not convinced that there is in reality much public appetite for this kind of information.  
•	A more active engagement of the media – Audit Scotland and the Accounts Commission should consider whether they have sufficiently invested in and developed their communications and media strategies.  This could include periodic briefings by senior Audit Scotland personnel to the media to try and encourage them to adopt and portray a wider perspective on local government improvement. We have not formed any detailed independent assessment of whether this is needed but there was sufficient evidence to indicate that some systematic reflection on their part along these lines would itself be a useful investment.

Developing and adapting BVAs in the light of changes taking place in councils 

170.	In addition to improving BVAs in the light of experience to date, it is also important to adapt and develop the approach in the light of changes in councils and in the environments in which they operate.  In our view the future development of BVAs should involve:

•	A more proportionate and flexible approach;
•	An increased emphasis on self assessment;
•	A greater emphasis of corporate outcomes;
•	An increased emphasis on community leadership; and
•	Better use of benchmarks and existing good practice.

171.	A more proportionate and tailored approach – The first round of BVAs has established benchmarks against which each council’s progress can be monitored in future.  Once these are in place it should be possible to develop a more flexible approach which is tailored to the performance, capacity and challenges facing each council.  

172.	Greater use of self-assessment – The existence of baselines also means that it is possible to move to a greater reliance on self-assessment.  Councils should be asked to articulate clearly their ‘improvement journey’, their targets for improvement and how they will achieve them.  BVA teams should assess the accuracy of self assessments, the level of ambition reflected in targets for improvement, and the capacities and tools that councils are marshalling to achieve the improvements they have set themselves.

173.	A continued focus on corporate capacity and processes, and also on corporate outcomes – In our view it is important that BVAs continue at least in part to focus on corporate capacity rather than inspecting individual services, although the intensity of the focus should itself be tailored to the degree of risk in individual councils.  In order to ensure that councils understand the rationale for this approach, it is important that (as recommended above) the Accounts Commission and Audit Scotland spell out their theory of improvement and the link which they believe exists between corporate processes and the performance of services.  However, there can and should be a shift towards a greater focus on the outcomes which councils have set themselves, especially in relation to the complex and boundary spanning ‘wicked’ issues which are designed to ‘place-shape’ in areas such as economic development, regeneration, social justice, and sustainability.    The BVA focus need not and should not be a matter of either a corporate or a service/outcome approach.   

174.	An increased emphasis on community leadership – Because of the increasing importance of community planning and partnership working in tackling complex, cross cutting issues, future BVAs need to give much more attention to the extent to which councils are providing effective community leadership which ensures proper orchestration of the work of the full range of local service providers.   Councils cannot be held to account for the performance of other agencies, but they can and should be judged in terms of their ability to lead the development of a vision for their area, to bring together other stakeholders and to mobilise effective partnership working between them.

175.	Making better use of benchmarks and expertise from beyond Scotland - There is a need for councils to make much more use of comparisons with approaches to corporate governance issues in other authorities, both in Scotland and beyond.  This is important both as a means of challenging their current performance and of identifying ways of improving their corporate capacity.  There is a need for Audit Scotland or other parts of the local government family to help councils to identify authorities from which they can learn and there is need for greater investment in support for capacity building of this kind.  Scottish authorities would also benefit from developing stronger links to the activities of the IDeA which represents a significant resource in terms of knowledge of good practice, the provision of leadership training and a source of expert peers.

Developing BVAs in the light of developments in the overall strategy for reform and regulation of public services 

176.	In addition to taking account of changes in councils, the future development of BVAs also needs to respond to changes in the wider context of reform and regulation of Scottish public services. There are at least four important developments that we believe could help to shape the context within which future BVAs are implemented:

•	Making better use of the resources available to the ‘local government family’;
•	The findings of the Crerar Review;
•	A restatement of key regulatory principles; and
•	The establishing of a renewed Task Force to help produce and develop future guidance relating to BVAs.

177. Making the most of the resources available to the ‘local government family’ – We believe that BVAs need to be owned as the core methodology for assessing whether councils are achieving their duty to deliver Best Value, community planning and continuous improvement by the whole of the local government family in Scotland (by which we mean COSLA, the Improvement Service for Scotland, SOLACE, the Scottish Executive, and the Accounts Commission/Audit Scotland).  The BVA approach was constructed in part through collaboration between these organisations and it is our view that this collaboration needs to be refreshed.  The local government family as a whole appears to have relatively few resources to support council improvement as compared to the position in Wales and England.  It is striking that Audit Scotland contains a substantial share of the available experience, data, and understanding of the improvement process.  It should therefore have a more explicit role in sharing this capacity and knowledge.  We do not consider that such arrangements would inhibit Audit Scotland’s independence, or lead to it giving what might amount to ‘consultancy’ advice, which would clearly be inappropriate.  It is surprising to us that whilst there are bi-lateral and tri-lateral contacts there is no forum in which senior representatives of all of the organisations in the ‘family’ meet on a regular basis.  We believe that as a minimum it would be appropriate for there to be an annual review seminar at which they pool the learning and test developing thinking about council improvement.  

178. Crerar review - It is very important to recognize the context provided by the Crerar review and by the development of views of public services reform on the part of the Scottish Executive, and associated enterprises such as that to drive a reduction in the number of performance indicators.  Taken together with the impacts of new electoral arrangements in local government and the new administration in the Scottish Executive, all of this adds up to a decisive moment in which there is the potential for significant change in the role of the Accounts Commission and of Audit Scotland and their stewardship of BVA.  This will become all the more important if the Best Value duty were to be extended to all public services.

179. Restatement of regulatory principles – We believe that there would be benefit in a clear articulation of regulatory principles which restates the principles of independence, the need to maintain an appropriate ‘relational distance’ between regulators and regulated bodies and the importance of (in the words of the Auditor General) ‘holding to account and helping to improve’.  This is especially important if our recommendations for a closer, more ‘corporate’ approach to local government improvement by the local government family as a whole are adopted. It would also be an opportunity to work through the balance and relationship between accountability and improvement as legitimate purposes for audit as well as inspection in the light of the BVA experience to date and also of the Crerar review.

180. New Task Force – We believe that there would be merit in re-establishing the Task Force that helped produce and develop the BVA guidance.  The re-formed task force should bring together and reflect on the experience of BVA to date and produce a statement outlining the Accounts Commission’s and Audit Scotland’s thinking about what will follow at the end of the first round of BVA.  This will provide an opportunity to articulate a clear ‘theory of improvement’ and could draw on the findings of this report, the Crerar review, the early outputs from the work on performance indicators and also take account of the implications of the outcomes of the national and local elections (including for example the need for some focused and supported learning about Best Value and BVA to be provided for the large numbers of first time councillors).  

May 2007 

Annex 1   The Research Team
Dr. Clive Grace

Clive is an Honorary Research Fellow at the Centre for Local & Regional Government Research, Cardiff Business School.  He has an outstanding senior track record of success in public services as, inter alia, Chief Executive of Torfaen CBC and Deputy Auditor-General for Wales. He was Chairman of SOLACE from 1998 to 2002 and led the re-construction of the Society as a modern, engaged, and forward looking professional body.  He also devised and leads the SOLACE Foundation Imprint, the key ‘thought-leadership’ vehicle of the Solace family.  He serves on CIPFA’s Council and Chairs its International Panel.  He has recently been appointed as the first Chairman of the Local Better Regulation Office.  He is also Chairman of Supporta plc, an AIM listed company providing services to the private and public sectors.

He has conducted and led a range of academic and policy related research projects. At Cardiff he pursues interests in public services change and improvement, in public sector accountancy and corporate governance, and in leadership and public policy.  He has commercial interests as Chairman of Supporta Plc and as an Adviser to the Board of BT Wales. His consultancy clients include both metropolitan and district councils; the IDeA; the Research Councils of the UK; the Institute of Public Finance; major public corporations; Cardiff Business School; and the United Nations Development Programme.

He has an academic track record which includes three books, numerous articles in a wide range of academic and policy-related publications, and editorship of a legal journal and a legal encyclopedia.

Professor Sandra Nutley

Sandra is Professor of Public Management at the University of Edinburgh and Director of the Research Unit for Research Utilisation.  She joined Edinburgh in October 2006 from the University of St Andrews, where she was Professor of Public Policy and Management and Director of the Centre for Public Policy and Management.  She has eight years experience in local government in England, working in planning and research roles.  Since moving into higher education, her career has been interspersed with periods of secondment to work with various public service organisations including the Scottish Executive and an internal audit service within the NHS.  She has also worked with a number of public service organisations (government departments, NHS and local government) as a researcher and consultant.

Sandra has an international reputation for her work on research use and evidence-based policy and practice and she also has expertise in comparative public management reform and performance management.  She has directed and contributed to a series of studies for a range of agencies including the Economic and Social Research Council, Social Care Institute for Excellence, Learning and Skills Development Agency, NHS Health Scotland, the Scottish Executive, and the probation service.

Sandra chairs the Scottish Funding Council’s Knowledge Transfer into Public Policy Action Group. In 2005 she was appointed as a Non-Executive Director of the Social Work Inspection Agency.  She is a member of the Scottish Executive's Changing Lives: Practice Governance group. She is also a member of expert panel for the Independent Review of Regulation, Audit, Inspection and Complaints Handling of Public Services in Scotland.

Dr James Downe

James is a Senior Research Associate in the Centre for Local and Regional Government Research, Cardiff Business School. He has completed a wide range of studies on the modernisation of local government, including research for the Department for Communities and Local Government (DCLG) on turnout in local government elections (2000), Best Value (2001) and the Beacon Council Scheme (2001, 2002, 2003). He is currently managing a long-term evaluation of the Local Government Modernisation Agenda in England for DCLG. (See http://www.clrgr.cf.ac.uk/lgma/index.html for more details).

James has a background in researching public engagement in the democratic process. His PhD focused on analysing the determinants of turnout in English local elections and he has written a number of articles in this field. He recently published a co-authored book analysing the political effects of boundary changes in English Local Government.

James has recently completed two projects for the Joseph Rowntree Foundation. The first explored political leadership under the new political arrangements and the second studied the changing role of Audit Commission inspection of local government. Current research projects include one analysing the development and impact of performance regimes operating in England, Scotland and Wales (ESRC funded) and a study for the Welsh Assembly which focuses upon the role and functions of elected members in Wales.   

Professor Steve Martin

Steve is the Director of the Centre for Local & Regional Government Research at Cardiff University. He is an expert in public policy and management and a close observer of current attempts to modernise local government and improve local public services.  
Over the last twenty years Steve has directed a wide range of major studies for central government departments, the European Commission, the Audit Commission, the IDeA, the National Assembly for Wales, the Welsh Development Agency and the Rural Development Commission.  

He is associate director of the research team commissioned by the Department for Communities and Local Government to undertake a long-term overall evaluation of the local government modernisation agenda and he is contributing to the DCLG study of the lessons that can be learnt by the UK from international approaches to local and regional governance and the long-term impact of the Best Value regime. 

Steve is a member of the Board of the IDeA and co-chair of the DCLG’s Local and Regional Governance Research Network, and he was academic adviser to the recently completed Review of Local Service Delivery in Wales under the chairmanship of Sir Jeremy Beecham.  

Annex 2   Interview and case study topic guide
Two topic guides were produced – one for stakeholders and one for case study authorities. The questions below were posed to interviewees in the case studies. A similar set of questions were used for stakeholders with the phrasing of some questions adjusted to take account of the interviewee. We have indicated below where questions were posed to stakeholders only.

Policy Context and Background (stakeholders only)

1.	What was the policy background to the development of Best Value audits? What were the key political and policy considerations that shaped its development?

2.	To what extent, if at all, was CPA/WPI taken into account, either to follow, avoid, or learn from?

3.	What was the role of:
•	the Scottish Executive?
•	the Scottish Parliament?
•	Audit Scotland?
•	Accounts Commission?
•	the Auditor-General?
•	the Improvement Service?
•	COSLA?
•	Councils?
•	Solace and other officer/professional groups?
•	Other stakeholders?

4.	Which individuals played the major development roles, and what were they? (Were there changes in personnel during this period – did this have any impact?)

5.	What are/were the key objectives of the introduction of Best Value audits?

6.	How confident was the Accounts Commission of success, what were regarded as the key risks, and how, if at all, were these risks mitigated?

7.	How did and does the Best Value audit policy connect to the wider Public Service Reform policy if at all?

The Effectiveness of the Best Value Audit Approach
1.	In your experience, how well would you say the current Best Value audit methods are working? 

2.	In your view, what are the main strengths and weaknesses of the current approach?
3.	If not already covered:
•	How well is community planning covered in the process?
•	Is the balance of attention between members and officers right?
•	How are the viewpoints of citizen taken into account?
•	Are the links with annual audits effective?
•	Are the links with the inspectorates effective?
•	Do the audits engage appropriately with non-council bodies e.g. health service, voluntary sector etc?
•	Would some form of peer input improve the process?

4.	Do you think that the Best Value audit process has changed over time? If so, in what ways?

5.	In what ways (if any) has the Best Value audit approach led to improvement in your council?

6.	What support have councils received, if any, from external organisations to help them deliver actions in the improvement plans? (stakeholders only)

7.	Do you have any other comments on the Best Value audit approach?

The Effectiveness of the Best Value Audit Reports

1.	What are your views of the Best Value audit report that your council received?
•	Is the report fair and consistent? 
•	Does it cover the local context well? 
•	Is it readable and the appropriate length?
•	How well does it cover services (part 3) as opposed to processes (part 2)? 
•	Does it provide adequate coverage of community planning, equalities, sustainability?
•	Does it provide adequate coverage of the other Best Value criteria? (Are there gaps in coverage of other key subjects?)
•	Are Statutory Performance Indicators used well in the report?
•	Is material from other scrutiny bodies used appropriately?

2.	What changes have been made to the reporting process over time and how successful have these changes been? (stakeholders only)

3.	In what ways (if any) has the report led to improvement?  (Please give specific examples and evidence of improvement).  Do you believe that it will lead to further improvements in the future? (If yes, please give examples).

4.	What changes could be made to improve the impact of the Best Value audit report on the public?

5.	Do you have any other comments on the Best Value audit report?
The Role of the Accounts Commission

1.	What are your views on the role of the Accounts Commission? (stakeholders only)

2.	Do you have any comments on the improvement plans? How would you describe the relationship between your council and the Accounts Commission? (stakeholders only)

3.	How effective was the meeting with your council following the Best Value audit?

4.	How useful has the improvement plan been to your council?

5.	Did your council consider appealing against the report and asking for a hearing?  If you went ahead with an appeal what was the outcome and were you content with the process and outcome of the hearing?

6.	To what extent does the work of the Accounts Commission represent Best Value? 

7.	Do you have any other comments on the powers and roles of the Accounts Commission and the Controller of Audit? (stakeholders only)

The Impact on Councils

1.	Has the Best Value audit led to positive internal changes (e.g., in culture, processes, structures, self-confidence etc.) in your council?  (If so, how?)

2.	In your experience has the Best Value audit:
•	Improved accountability to politicians and the public?
•	Provoked media interest?
•	Acted as a catalyst for improvement (better quality and more efficient services?)
•	Identified instances of failure?
•	Diagnosed the reasons for failure and success?
•	Prevented failure in advance by encouraging your council to take greater care than they might have done if their activities were not subject to external scrutiny?
•	Encouraged your council to develop a greater capacity for self-evaluation and inter- and intra-organisational learning?

3.	What have been the drawbacks of the Best Value audit? Have these been outweighed by the benefits of the audit?  

4.	What resource demands did the Best Value audit place on your council? What, if anything, would you like to see done to reduce its costs?
5.	What support has your council received, if any, from external organisations to help you deliver actions in your improvement plan? Is more support needed?

6.	Do you have any comments on the following, if not covered already:
•	The effectiveness of the publication and media arrangements
•	How effective is the coordination with other scrutiny bodies?
•	How effective was the communication between the Accounts Commission/Audit Scotland and your council – before, during, after the audit?
•	How transparent is the process?

7.	Do you have any other comments in relation to the impact of the Best Value audit on your council?

The Impact on Stakeholders

1.	What is the impact of Best Value audits on citizens, service users and other stakeholders (e.g. increased public confidence/accountability)? What evidence is there on this?

2.	In what ways have the Best Value audits placed a focus on the experience of citizens?

3.	What evidence is there about the impact of Best Value audits on the Scottish Parliament, the Scottish Executive and any other public bodies?

4.	What is the impact of the Audit Scotland’s improved tools for engaging stakeholders (e.g. better use of focus groups and use of surveys)?

5.	Do you have any other comments in relation to the impact of Best Value audits on citizens, service users and other stakeholders?

Evaluating the Best Value Audit Approach

1.	Overall, have the Accounts Commission/Audit Scotland accomplished what they set out to achieve? 

2.	When you consider the underpinning Best Value audit principles, do these raise issues that you have not covered already?
•	Self assessment
Is the balance between self assessment and external audit correct?
•	No league tables or scores
Is this the best approach?
•	Local context
Does the process allow elected members to determine local priorities?
•	Corporate responsibility
Is the balance between examining service performance and corporate governance of councils’ right?
•	Three year cycle
Is a three year cycle appropriate?
•	Central audit team
Are the audits credible (skills, experience etc. of staff)?  
•	Selective approach
Are the audits flexible, risk-based and proportionate to risk? (Are more incentives needed?)
•	Linked to other scrutiny processes
Are the different parts of the scrutiny process in local government now more ‘joined-up’ (or co-ordinated) than they were 2-3 years ago? 
•	Built on existing audit processes
Does Best Value build on the annual audits reports?
•	Continuous improvement
Does Best Value encourage continuous improvement?

3.	Do you have any other comments in relation to whether Accounts Commission /Audit Scotland have accomplished what they set out to achieve?

The Future Best Value Audit Approach

1.	How should the Best Value audits develop in the future (the approach, the reports, the role of the Accounts Commission, the impact on councils, and the underpinning principles)?

2.	How can the impact of Best Value audits be widened?

3.	How should community planning and increasing joint/shared services be covered?

4.	What implications, if any, does the efficient government project have for the Best Value audits? (Should Best Value audits have a ‘use of resources’ component?)

5.	Do the existing Statutory Performance Indicators need to be improved?

6.	How should Best Value audits be developed in relation to other forms of scrutiny? 

7.	How can the Best Value audits provide added value (e.g. an annual digest of good practice produced by the Improvement Service and COSLA, workshop on selected topics, networks of officers etc?)

8.	What are the alternatives to the current approach?

9.	What changes would you like to see in the next round of audits?

10.	Do you have any other views on future developments?

Annex 3   Survey questions
Section A: The Best Value Audit Approach
To what extent do you agree with the following statements about the Audit of Best Value and Community Planning?
A1	Community planning was covered in sufficient depth
A2	My service area (if appropriate) was covered in sufficient depth
A3	The balance of attention given between members and officers was about right
A4	The links made with the annual audits were effective
A5	The coordination with other scrutiny bodies was effective
A6	The audits engaged extensively with non-council bodies e.g. health service, voluntary sector etc.
A7	The Best Value audit team were credible (skills, experience etc. of staff)
A8	The process was transparent

Section B: The Best Value Audit Reports
To what extent do you agree that the Best Value audit report your council received was:
B1	Fair and consistent
B2	Readable
B3	Covered the local context well
B4	The appropriate length

To what extent do you agree that the Best Value audit report adequately covered
B5	Services (part 3)
B6	Processes (part 2)
B7	Equalities
B8	Sustainability
B9	The use of available data (e.g. Statutory Performance Indicators)
B10	Material from other scrutiny bodies

B11	What changes could be made to improve the impact of the Best Value audit report?

Section C: The Role of the Accounts Commission
To what extent do you agree with the following statements about the role of the Accounts Commission?
C1	The relationship between my council and the Accounts Commission was good
C2	The meeting between my council and the Accounts Commission was good
C3	The communication between the Accounts Commission and my council was effective
C4	The publication and media arrangements were effective
C5	My council consider appealing against the report

Section D: The Impact on Councils
To what extent do you agree that the Best Value audit undertaken in your council:
D1	Improved accountability to politicians
D2	Improved accountability to the public
D3	Provoked media interest
D4	Identified instances of failure
D5	Diagnosed the reasons for failure
D6	Acted as a catalyst for improvement (better quality and more efficient services)
D7	Struck the right balance between 'holding to account' and 'helping to improve'
D8	Led my council to focus on national priorities at the expense of local priorities
D9	Led my council to concentrate on short-term goals rather than longer-term issues
D10	Encouraged my council to develop a greater capacity for self-evaluation and inter- and intra-organisational learning
D11	My council's improvement plan has been instrumental in helping to drive improvement
D12	My council has received support from external organisations to help deliver actions in our improvement plan
D13	The self-assessment process has been instrumental in helping to drive improvement
D14	The overall Best Value audit approach has led to improvement in my council

D15	In what ways, if any, has the Audit of Best Value and Community Planning led to improvement in your council? (Please give examples)

Section E: The Current Scrutiny Regime and the Future Best Value Audit Approach
To what extent do you agree with the following statements about the current scrutiny regime?
E1	The activities of the various inspectorates have become more joined-up over the last few years
E2	There has been a shift in inspection from assessing compliance to supporting service improvement in the last few years
E3	The benefits of external inspection outweigh the costs in terms of improvements in my council
E4	My council has the capacity to respond effectively to Scottish Executive initiatives

To what extent do you agree with the following statements about the future Best Value approach?
E5	Audit Scotland should continue not to use league tables or scores
E6	Future audits should include an indication of a council's direction of travel in terms of improvement
E7	Future audits need to be more flexible, risk- based and proportionate to risk
E8	The involvement of peer members/officers in the Best Value audit team would improve the process
E9	Greater reliance on self regulation would help to encourage improvement

E10	What changes would you like to see in the next round of Best Value and Community Planning audits?

Other comments
Please use the space below to make any other comments about your experience and impact of the Best Value and Community Planning audits.

Annex 4  Bibliography

Accounts Commission (2003) The new audit of Best Value Consultation Paper.
Audit Commission (2005) CPA: the harder test (Audit Commission, London)
Audit Commission (2005) Wales Programme for Improvement 2004 Annual Report (Audit Commission in Wales, Cardiff) 
Audit Scotland (2004) Background briefing: The Audit of Best Value http://www.audit-scotland.gov.uk/accounts/pdfs/BestValuebriefingnote.pdf
Audit Scotland (2006) Overview of the local authority audits 2005, (Audit Scotland: Edinburgh)
Audit Scotland (200?) Annual Report 2005/6: Accountability and Performance, (Audit Scotland: Edinburgh)
Audit Scotland (200?) A Quick Guide to the Auditor General, Accounts Commission and Audit Scotland, (Audit Scotland: Edinburgh)
Audit Scotland (200?) Audit Scotland. Corporate Plan: 2005-08, (Audit Scotland: Edinburgh)
Black, B. (2004) Holding to account and helping to improve, (Audit Scotland: Edinburgh)
Black, B. (2006) Public Service Improvement. The conditions for success and the Scottish experience, (Audit Scotland: Edinburgh)
Broadbent, J. (2003) The crock of gold at the end of the performance rainbow?, Public Money & Management, 23, 1, 5-7
Bundred, S. (2005) Regulating Reformed Public Services in Bichard, M and Grace, C (ed.),  Perspectives on the Third Term: The Way Ahead,  (Solace Foundation Imprint, London)
Byatt, I. and Lyons, M. (2001) The Role of External Review in Improving Performance, (H.M. Treasury, London)
Cope, S. and Goodship, J. (2002) The Audit Commission and Public Services: Delivering for Whom?, Public Money & Management, 22, 4, 33-40
Cutler, T. and Waine, B. (2003) Advancing public accountability: the social services ‘star ratings’, Public Money & Management, 23, 2, 125-128
Davis, H., Downe, J. and Martin, S. J. (2004) The changing role of Audit Commission inspection of local government, (Audit Commission, London)
Grace, C. (2003) Regulation: The Modern Idiom, Public Money & Management, 23, 2, 73-75
Grace, C. (2005) Change and Improvement in Audit and Inspection:  A Strategic Approach for the 21st Century, Local Government Studies, 31, 5, 575-596
Hood, C., Scott, C., James, O., Jones, G. and Travers, T. (1999) Regulation inside government, (Oxford University Press, Oxford)
Hood, C., James, O. and Scott, C. (2000) Regulation of Government: has it increased, is it increasing, should it be diminished?, Public Administration 78, 2, 283-304
Humphrey, J. (2001) Bewitched or Bewildered? ‘Facts’ and ‘Values’ in Audit Commission Texts, Local Government Studies, 27, 2, 19-43
Jacobs, R. and Goddard, M. (2007) How Do Performance Indicators Add Up? An Examination of Composite Indicators in Public Services, Public Money & Management 27, 2, 103-110
Kelly, S. and Rodden, C. (1998) From Watchdog to Guidedog? The Evolving Role of the Accounts Commission for Scotland, Faculty of Business Working Paper no. 10 (Glasgow Caledonian University)
Martin, S. J. (2002) The modernisation of UK local government: markets, managers, monitors and mixed fortunes, Public Management Review, 4, 3, 291-307
Martin, S. J. (2005) Public service improvement: current developments and future research agendas, Local Government Studies, 31, 2, 531-540
Midwinter, A. and McGarvey, N. (2001) In Search of the Regulatory State: Evidence from Scotland, Public Administration, 79, 4, 825-849
Office of the Deputy Prime Minister (2005) Securing Better Services: Developing a New Performance Framework, (Office of the Deputy Prime Minister, London)
Office of Public Services Reform (2002) Principles of Public Services Reform, (Cabinet Office, London)
Office of Public Services Reform (2003a) Inspecting for Improvement, (Cabinet Office, London)
Office of Public Services Reform. (2003b) The Government’s policy on inspection of public services (Cabinet Office, London)
Platt, D. (2002) Why stars are underrated, Community Care, 27, 27 June – 3 June 2002, 36-38
Power, M, (1994) The Audit Explosion, (Demos, London)
Power, M. (2003) Evaluating the Audit Explosion, Law and Policy, 25, 3, 185-202
Rhodes, R. A. W. (1994) The Hollowing out of the State: the changing nature of the public service in Britain, The Political Quarterly, 65, 2, 138-151
Scottish Executive (2004) The Local Government in Scotland Act 2003: Best Value Guidance, (Scottish Executive, Edinburgh)
Scottish Executive (2006) Transforming Public Services: The Next Phase of Reform, (Scottish Executive, Edinburgh)
Scottish Executive (2007) Independent Review of Regulation, Audit, Inspection And Complaints Handling Of Public Services In Scotland: Discussion paper for the Academic Seminar to be held on 17th April 2007 (Unpublished)
Wales Audit Office (2006) Wales Programme for Improvement Annual Report 2004/5, (Welsh Audit Office, Cardiff)
Welsh Assembly Government (2004) Making the Connections: Delivering Better Services for Wales, (Welsh Assembly Government, Cardiff)
Welsh Assembly Government (2005) Regulation and Inspection of Public Services in Wales, National Assembly for Wales Local Government and Public Services Committee (Welsh Assembly Government, Cardiff)
Welsh Assembly Government (2005) Wales Programme for Improvement: Guidance for Local Authorities 2005 (Welsh Assembly Government, Cardiff)
Welsh Assembly Government (2006) Making the Connections: Beyond Boundaries, (Welsh Assembly Government, Cardiff)

May 2007