Audit Scotland
Impact – annual report 2013/14

Audit Scotland is a statutory body set up in April 2000 under the Public Finance and Accountability (Scotland) Act 2000. We help the Auditor General for Scotland and the Accounts Commission check that organisations spending public money use it properly, efficiently and effectively.

Introduction
1.	Our vision is to be a world-class audit organisation that improves the use of public money. Through our reports to the Auditor General and the Accounts Commission, we provide independent assurance to the people of Scotland that public money is spent properly and provides value for money. We aim to achieve this by:
•	carrying out relevant and timely audits of the way the public sector manages and spends money
•	reporting our findings and conclusions in public
•	identifying risks, making clear and relevant recommendations for improvement, and following these up.
2.	All our audit work aims to improve the use of public money, and the recommendations we make in our reports are an essential element of this. We focus our work on those areas which are most relevant to this overall aim – governance and financial management, value for money, financial sustainability and improving the transparency of financial and performance reporting. 
3.	During the year, an external independent review of our work found that the source of Audit Scotland's impact is built on a number of features including:
•	our statutory role and status: the carrying out of an independent audit process with right of access and which is open and transparent creates the pressure to do things well and with integrity
•	providing assurance to the public and wider stakeholders in ways which ensure confidence in the public sector in Scotland and therefore our wider society
•	the quality and credibility of our reports
•	the identification and articulation of risk
•	the clarity of our recommendations and their clear foundation in evidence 
•	active following through on the implementation of recommendations and clear encouragement to take action
•	the guidance we offer to support the implementation of recommendations, including checklists and good practice. 
4.	Measuring the impact of our work gives us assurance that we are doing the right thing. During 2013/14, our corporate impact group supported individual business groups strengthen their approach to planning and promoting our audit work and measuring its impact. We produced impact reports on our financial audit work and on individual performance audits throughout the year and these demonstrate that our work makes a positive impact. 

5.	This review summarises the findings from these reports. It is split into three sections: 
•	How we measure impact
•	What impact our reports have made
•	What we have learned.

How do we measure impact?
6.	To measure and report impact, we use a framework which sets out the four broad areas where we expect our work to make a difference. These broad areas are financial sustainability, value for money, governance and financial management, and transparency of reporting. They have been collectively identified by Audit Scotland as the public sector audit dimensions and are used to plan, report and measure the impact of our work (Exhibit 1). 
7.	We are reporting on the impact we measured during 2013/14. This will not always relate to reports published during 2013/14. Reports published in previous years may have continued impact as audited bodies implement the recommendations, use the findings for baseline information or use checklists provided to support improvement. The full impact of a report may not emerge until many months or years after publication, and we may never be able to capture all elements of impact.
8.	We produce a wide range of local and national reports about the performance and financial management of Scotland's public bodies. These can be split into two types of publication:
•	Annual audit reports for all the public bodies we are responsible for auditing, as well as the audit certificate on the financial statements. Auditors give detailed reports directly to the organisations they audit, as well as many other pieces of work during the financial year.
•	A wide range of reports on matters of public interest. These include overview reports on how different sectors perform during each financial year, performance audits on how particular services are performing, or reports which focus on specific issues, either in an individual organisation or across a sector.
9.	At the planning stage of all audits, we consider what impact we want to achieve using the four public sector audit dimensions, who the target audience for our work is and what methods could be used to reach that audience. After the audit has been completed we measure the actual impact, again using the four public sector audit dimensions. However, we use slightly different approaches for measuring the impact of annual audit reports and public interest reports to reflect their diverse scope, content and audience. 
10.	Annual audit reports on public bodies in Scotland are produced by both our internal teams and by appointed auditors (private accountancy firms appointed by the Accounts Commission and the Auditor General for Scotland to conduct around 40 per cent of our audits). We measure the impact of the annual audits conducted by our own auditors by reviewing the action points identified by the auditor and how the audited body responded to these (for example, whether the improvements were implemented). We collate this information every year in sector-based impact reports (health, local government and central government) and report this internally. This year we also reviewed the impact that audits delivered by the firms had in the local government sector. 
11.	The impact of performance audit and other public interest reports, which tend to make recommendations to a number of different bodies, is captured both through an impact report, completed 12-18 months after a report has been published (performance audits) and ongoing analysis of report downloads and media coverage. 
12.	Also, the impact of both annual audits and public interest reports is measured in our annual survey of audited bodies. Audited bodies are surveyed on a rotating sector basis. The most recent surveys (2012/13 audits) covered the central government and local government sectors and part of the survey asked about impact of audit work. 
What impact have we made?
13.	The work of Audit Scotland on behalf of the Auditor General and the Accounts Commission, forms a critical part of the accountability system for public services in Scotland. Put simply, our job is to improve the use of public money. Our work supports trust in public services by providing assurance that financial management and control is sound. We can shine a spotlight on areas which do not meet the highest standards of governance and financial management. And through our performance audit work, we assess whether public services can demonstrate value for money. We spend around £24m per year to audit over £40 billion pounds of public expenditure in Scotland. 
14.	Our impact can be felt in many ways: improvements to how public money is accounted for; encouraging continuous improvement and the delivery of best value in the public sector; improvements to the way in which whole systems of public service delivery work; and changes to the way public policy is planned and implemented.
15.	In this section we capture the impact of our work during 2013/14. Because of the diversity of our work, we summarise the impact of our annual audits and that of our public interest reports separately. 
 Annual audits 
16.	The main public method by which auditors achieve change, or have an impact, is by agreeing action points for improvement on the significant risks identified with management. It is not easy to capture the other impacts that auditors have at audited bodies, say through discussions with management, though it should be noted that these 'softer impacts' can be very important. 
17.	We have collated information on the action points for improvement made by local auditors (measured during 2013/14 although it relates to the 2012/13 audits). All action points where listed together with a management response were agreed. 
1.	Sixty-seven per cent of all action points (239 out of 358) made by local auditors in their 2012/13 annual audit reports related to governance and financial management (Exhibit 2).  This is to be expected given the nature of local audit work which is primarily focused on the audit of financial statements. The smallest proportion (13 per cent) made reference to value for money, although the number of action points made in relation to transparency of reporting and financial sustainability was relatively low too.
18.	Similarly, when we asked audited bodies about the contribution auditors made across the four public sector audit dimensions, they reported that the biggest contribution was in improving governance and financial management (Exhibit 3). 
19.	Exhibit 4, overleaf, sets out examples of impact resulting from action points made in annual audit reports against the four public sector audit dimensions.
Public interest reports
20.	We produce a wide range of reports on matters of public interest. These include overview reports on how different sectors perform during each financial year. Other reports focus on particular issues, either in an individual organisation or across a sector. Many of our reports are considered by Scottish Parliament committees. In 2013/14 we published 23 public interest reports, including: 
•	Reshaping care for older people
•	Modern apprenticeships 
•	The Highland Council: Caithness Heat and Power follow-up statutory report
•	Management of patients on NHS waiting lists – audit update
•	Police reform: progress update 2013
•	Argyll & Bute Council – targeted best value work
•	An overview of local government in Scotland
•	Scotland's colleges. 
21.	In 2013/14 there were around 180,000 downloads of Audit Scotland's reports. The reports with the most downloads in the year were Health Inequalities in Scotland published in December 2012 (10,036), Overview of Scotland's justice system from September 2011 (6,580) and Prescribing in general practice in Scotland from January 2013 (4,043). These figures will be influenced by the date of publication. For example, a report published in April 2013 is likely to have been downloaded more times than one published in March 2014. As can be seen from the dates of the reports noted above, impact can also be sustained over a number of years. 
22.	There were over 1,200 media items for Audit Scotland's reports in 2013/14. The report with the most media items was Police reform published November 2013.
23.	When we asked audited bodies for feedback on our local government performance audit reports using the four public sector audit dimensions, the highest scoring impact area was improved governance and financial management with an average 80 per cent agreement rate, transparency of reporting of financial and other performance was second with 78 per cent, better understanding of financial sustainability was third with 75 per cent and better value for money was lowest with 73 per cent. This ranking mirrors the impact categories made by the local auditors. 
24.	The report with the greatest impact across the four dimensions was Managing performance: Are you getting it right? This had an average impact score of over 88 per cent, closely followed by An overview of local government in Scotland (average 88 per cent). 
25.	Exhibit 5 provides examples of how performance audit and other public interest reports contributed to improvements in the four public audit dimensions. 

What have we learned?
26.	In recent years, we have made good progress in measuring impact and the evidence we currently gather provides positive assurance about the impact we make. However, we recognise there is still much more we could do and so we engaged Rocket Science to review our approach and they presented their report to us in May 2013. 
27.	Areas where Rocket Science acknowledged we were making a particularly positive impact included:
•	Where we describe the 'whole system'. An example of this working well was demonstrated by the demand for the A3 Justice System poster, one of the products of our report An overview of Scotland's criminal justice system, published in 2011. This gave a graphic representation of the different elements of the justice system and how they connected.
•	Where we identify the scale of a problem and provide information on the way it is likely to change.
•	Where we compile information that allows organisations to compare their performance.
28.	However, Rocket Science also identified ways in which the impact of Audit Scotland's work could be enhanced. These included developing our communication methods and gaining insights from organisations about the use and impact of Audit Scotland findings and recommendations. 
29.	We accepted all of Rocket Science's recommendations and have made good progress in implementing them. For example, there have been significant improvements in how we work together as one organisation and we have held a number of events with relevant stakeholders to promote our performance audits and public interest reports. Over the coming year, we will continue to enhance cross-team working to gain greater local impact from our national audit work. We will also revise the Code of Audit Practice to clarify expectations around the focus of our work and how improvements and changes in the use of public money should be reported. 

Conclusions
30.	This report provides some tangible examples of the impact of our work during 2013/14. Across Scotland, all our action points in annual audit reports were accepted by the audited bodies, our public interest reports were presented and discussed by the Accounts Commission and Committees of Scottish Parliament, in particular the Public Audit Committee, and many of our reports featured prominently in the media. 
31.	However, we are still developing our approach to measuring and improving impact and are looking at ways of extracting further value from our work. Our audit planning guidance for audits currently being undertaken asked all auditors to consider the four public sector audit dimensions as part of their audit planning and reporting. We are diversifying the product range of our public interest reports by, for example, producing specific guides for operational managers, checklists for board members and easy-access leaflets for the public. 
32.	As we move forward, we intend to build on the good progress we have made to date by improving cross-team working, capitalising on skills and intelligence across all parts of our organisation. In addition, we will collect feedback more systematically from audited bodies to ensure we are achieving our goal to improve the use of public money.