Performance management in Scottish Enterprise

Key messages/Prepared for the Auditor General for Scotland
March 2006

Introduction

1. Scottish Enterprise is the main agency for economic development in Scotland. It delivers its functions through a network of 12 Local Enterprise Companies (LECs). Over the period 2001/02 to 2004/05, it has spent just over £2 billion.

2. In January 2001, the Scottish Executive published A Smart, Successful Scotland, (SSS), 
a comprehensive policy statement which sets out the strategic direction for Scottish Enterprise, Highlands and Islands Enterprise and their networks of LECs. 

3. It set out a strategy for improving the competitiveness and productivity of the Scottish economy focusing around three key themes: 

•	Growing Businesses
•	Global Connections
•	Skills and Learning. 

4. Scottish Enterprise provides a range of services and projects to deliver these three themes in partnership with others.

The study 

5. Scottish Enterprise is currently making changes to the way it manages performance. This study reviews the current position and comments on the likely impact of the changes underway.

6. The study follows earlier work undertaken by Audit Scotland. 
It covers: 

•	the performance of the Scottish economy and progress towards A Smart, Successful Scotland 
•	the framework for measuring performance and the information available on Scottish Enterprise’s achievements
•	how Scottish Enterprise develops, monitors and evaluates its activities and how it manages performance to encourage continuous improvement.

7. Measuring the impact of an economic development agency is complex and there is no easy way to assess the impact of its investment on overall economic performance. Similar agencies throughout the world also struggle with these issues and Scottish Enterprise’s approach appears to be well advanced.

Scottish Enterprise’s achievements need to be set in the context of wider economic performance
8. Scottish Enterprise’s activities should all support the growth of the Scottish economy. However, the performance of the Scottish economy reflects a wide range of influences, many of which are beyond the control of Scottish Enterprise. 

9. In recent years, Scotland’s economic performance has been mixed. Progress has been made in narrowing the gaps between Scottish performance and that of UK and international comparators, but performance remains weak in a number of areas and there is considerable variation between Scottish regions. 

The performance measurement framework provides a limited picture of Scottish Enterprise’s achievements
10. The Scottish Executive focuses on the overall performance of the Scottish economy and uses a range of 38 outcome indicators to monitor progress towards achieving the ambitions set out in SSS. Scottish Enterprise’s annual Operating Plan attempts to link its project activities with these ambitions.

11. Scottish Enterprise’s Annual Report provides a summary of achievements against ten key progress measures identified in the operating plan which focus on the activities and outputs of the Network. Because there is no explicit link to the SSS outcomes in the Annual Report, it is difficult to assess how Scottish Enterprise activities are contributing to the overall strategy. 

12. Internally, Scottish Enterprise uses a much wider range of information to assess performance. The ‘balanced scorecard’ approach provides a more complete picture, but the emphasis has again tended to be on processes rather than ‘impact’. 

13. The lack of impact data largely reflects the complexities of measuring outcomes in relation to economic development activity.

14. Scottish Enterprise has attempted to measure the overall impact of its activities, most recently in 2003 but the results should be treated with caution. It estimated that it had added some £1.6 billion to Scottish GDP over three years, as a result of its activities in 2001/02. 

Scottish Enterprise needs to continue to improve how it develops, monitors and evaluates its activities
15. Effective performance management relies on high-quality management information and effective use of this information. Underlying this, a well-designed framework for developing, monitoring and evaluating activities is essential.

16. Our review of a small sample of projects highlighted areas for improvement in relation to appraisal, approval, monitoring and evaluation:
 
•	The strategic rationale for projects was not supported by factual evidence in 40 per cent of the cases we examined.
•	In over half the cases reviewed, alternative project options were not adequately considered.
•	Performance targets have been based on past performance and available resource without a fuller review of the wider economic environment.
•	Monitoring tends to focus on data about activities and outputs rather than outcomes.
•	Although four-fifths of the projects we examined had monitoring frameworks in place, less than half were fully measuring appropriate data and only one-quarter were used to assess the effectiveness of project delivery.
•	Information from monitoring and evaluation is not always used effectively to manage and improve performance or to inform wider operations, strategy or policy.
17. Scottish Enterprise has introduced new processes and guidance which should help address these issues, improve the strategic fit of their activities and promote efficient allocation of resources.

Further improvement in the quality and focus of evaluation work is required 
18. Activity and output data should not be used in isolation, but needs to be analysed within the context of outcome data to enable a comprehensive assessment of performance. At a project level, the impact of a particular intervention (or group of interventions) is typically assessed through an evaluation study. 

19. The quality of Scottish Enterprise’s evaluation evidence is mixed and approaches to impact assessment vary widely. 

•	Half of the evaluations we examined had not mapped out all the activity delivered by the project concerned.
•	Less than half assessed net additional impact and only a quarter assessed value for money.
•	Over half the evaluations did not test the strategic rationale of the project.

20. Greater emphasis should be placed on broader, strategic evaluation studies. Mechanisms for disseminating and responding to findings also need to be enhanced.

21. Gross value added (GVA) is a potentially useful impact measure that helps focus on outcomes, although the concept needs to be clearly understood and used in conjunction with other SSS indicators.

Scottish Enterprise must continue to move towards a performance improvement culture 
22. A number of changes have been made, or are underway, that should encourage improved performance management within Scottish Enterprise. These should sharpen the strategic focus of its activities and improve understanding of their contribution towards intended outcomes. Changes include:

• 	a gateway system and training for project appraisal
• 	a customer relationship management system
•	greater focus on economic impact using GVA
•	focusing on key industries and a metropolitan approach.

23. However, Scottish Enterprise is large and dispersed, and there is some way to go before staff throughout the organisation understand the positive role of performance management.

Summary of key recommendations

• 	In assessing and reporting on performance, Scottish Enterprise should work with the Scottish Executive to focus on the contribution being made to SSS outcome measures. 
•	Scottish Enterprise should further investigate the potential of econometric models to estimate the combined impact of its activities.
•	Targets should be informed by external evidence, rather than past performance.
•	Monitoring frameworks for projects should reflect a focus on outcome measures. 
•	Scottish Enterprise should continue to work to improve the quality and consistency of evaluation work and ensure that learning effects are disseminated effectively and are used to challenge the rationale for activities.
•	Evaluation activity should focus on broader, strategic studies that can deliver a wider picture of impact, based on the range of indicators presented in SSS, rather than on GVA alone.
•	Scottish Enterprise should continue working to ensure that the positive changes at management level are cascaded throughout the organisation.

If you require this publication in an alternative format and/or language, please contact us to discuss your needs. It is also available on our website: www.audit-scotland.gov.uk

Audit Scotland 
110 George Street 
Edinburgh EH2 4LH 

Telephone 
0131 477 1234 
Fax 
0131 477 4567
 
www.audit-scotland.gov.uk 

ISBN 1 905634 12 9

AGS/2006/4