Performance management in Scottish Enterprise

Prepared for the Auditor General for Scotland
March 2006

Auditor General for Scotland
The Auditor General for Scotland is the Parliament's watchdog for ensuring propriety and value for money in the spending of public funds.

He is responsible for investigating whether public spending bodies achieve the best possible value for money and adhere to the highest standards of financial management.

He is independent and not subject to the control of any member of the Scottish Executive or the Parliament.

The Auditor General is responsible for securing the audit of the Scottish Executive and most other public sector bodies except local authorities and fire and police boards.

The following bodies fall within the remit of the Auditor General:

• departments of the Scottish Executive eg, the Health Department

• executive agencies eg, the Prison Service, Historic Scotland

• NHS boards 

• further education colleges

• Scottish Water

• NDPBs and others eg, Scottish Enterprise.

Summary

Introduction

1. Scottish Enterprise is the main agency for economic development in Scotland. It delivers its functions through a network of 12 Local Enterprise Companies (LECs). Over the period 2001/02 to 2004/05, it has spent just over £2 billion.

2. In January 2001, the Scottish Executive published A Smart, Successful Scotland, (SSS), a comprehensive policy statement which sets out the strategic direction for Scottish Enterprise, Highlands and Islands Enterprise and their networks of LECs. 

3. It set out a strategy for improving the competitiveness and productivity of the Scottish economy focusing around three key themes: 

•	Growing Businesses
•	Global Connections
•	Skills and Learning. 

4. Scottish Enterprise provides a range of services and projects to deliver these three themes in partnership with others.

Summary of key findings

Economic context
5. Scottish Enterprise’s achievements need to be set in the context of wider economic performance. Although Scottish Enterprise’s activities should all support the growth of the Scottish economy, the performance of the Scottish economy reflects a wide range of influences, many of which are beyond the control of Scottish Enterprise. 

6. In recent years, Scotland’s economic performance has been mixed. Progress has been made in narrowing the gaps between Scottish performance and that of UK and international comparators, but performance remains weak in a number of areas and there is considerable variation between Scottish regions. 

The performance measurement framework 
7. The Scottish Executive uses a range of 38 outcome indicators to monitor progress towards achieving the ambitions set out in A Smart, Successful Scotland. Scottish Enterprise’s Operating Plan attempts to link its project activities to these ambitions and the Annual Report provides a summary of achievements against ten key progress measures. These measures focus on the activities and outputs of the network. 

8. There is no explicit link between the Annual Report progress measures and the SSS outcomes. This makes it difficult to assess how Scottish Enterprise activities are contributing to the overall strategy. 

9. Information published in the Annual Report provides a limited picture of Scottish Enterprise’s achievements. Internally, Scottish Enterprise uses a much wider range of information to assess performance. The ‘balanced scorecard’ approach provides a more complete picture, but the emphasis has tended to be on ‘process’ rather than ‘impact’ measures. 

10. The lack of impact data largely reflects the complexities of measuring outcomes in relation to economic development activity. Other economic development agencies in the UK and internationally face similar challenges, and no simple solution exists, although Scottish Enterprise’s approach appears to be well advanced.

11. Scottish Enterprise has attempted to measure the overall impact of its activities, most recently in 2003, but the results should be treated with caution. It estimated that it had added some £1.6 billion to Scottish GDP over three years, as a result of its activities in 2001/02.

Developing, monitoring and evaluating activities
12. Our review highlighted some concerns in relation to appraisal, approval, monitoring and evaluation: 

•	The strategic rationale for projects was not supported by factual evidence in 40 per cent of the cases we examined.
•	In over half of cases reviewed, alternative project options were not adequately considered.
•	Performance targets have been based on past performance and the resources available, with limited consideration of the wider economic environment.
•	Monitoring tends to focus on data about activities and outputs, rather than outcomes.
•	Information from monitoring and evaluation is not always used effectively to manage and improve performance or to inform wider operations, strategy or policy.
•	The quality of evaluation evidence is mixed and approaches to impact assessment vary widely.
•	Over half of the evaluations reviewed failed to test the strategic rationale for intervention, less than half provided an assessment of net additional impact and only a quarter assessed value for money.
13. Gross value added (GVA) is being used more widely and is a potentially useful impact measure that helps focus on outcomes. However, the concept needs to be clearly understood and used in conjunction with other SSS indicators.

14. Scottish Enterprise has introduced new processes and guidance which should help address these issues, improve the strategic fit of their activities and promote efficient allocation of resources.

Changes underway
15. A number of changes have been made, or are underway, that should encourage improved performance management within Scottish Enterprise. These should sharpen the strategic focus of its activities and improve understanding of its their contribution towards intended outcomes. Changes include:

•	a gateway system and training for project appraisal
•	a customer relationship management system
•	greater focus on economic impact using GVA
•	focusing on key industries and a metropolitan approach.

16. However, Scottish Enterprise is large and dispersed, and there is some way to go before staff throughout the organisation understand the positive role of performance management.

Summary of key recommendations

• 	In assessing and reporting on performance, Scottish Enterprise should work with the Scottish Executive to focus on the contribution being made to SSS outcome measures. 
• 	Scottish Enterprise should further investigate the potential of econometric models to estimate the combined impact of its activities.
• 	Targets should be informed by external evidence, rather than past performance.
• 	Monitoring frameworks for projects should reflect a focus on outcome measures. 
•	Scottish Enterprise should continue to work to improve the quality and consistency of evaluation work and ensure that learning effects are disseminated effectively and are used to challenge the rationale for activities.
•	Evaluation activity should focus on broader, strategic studies that can deliver a wider picture of impact, based on the range of indicators presented in SSS, rather than on GVA alone.
•	Scottish Enterprise should continue working to ensure that the positive changes at management level are cascaded throughout the organisation.
The study

17. The study follows earlier work undertaken by Audit Scotland. It covers: 

•	the performance of the Scottish economy and progress towards A Smart Successful Scotland
•	the framework for measuring performance and the information available on Scottish Enterprise’s achievements
•	how Scottish Enterprise develops, monitors and evaluates its activities and how it manages performance to encourage continuous improvement.

Part 1. Introduction

18. Growth in Scotland’s economy was recognised as the top priority in the Scottish Executive’s 2003 Partnership Agreement1, which identified a successful economy as the key to future prosperity and a prerequisite for building first-class public services, social justice and a Scotland of opportunity. 

19. Alongside its sister organisation, Highlands & Islands Enterprise (HIE), Scottish Enterprise is the main public agency for supporting economic development in Scotland. The investment it makes is, however, a small proportion of the overall investment in the economy in any year. Its role is therefore to act as a catalyst to enable more, better or faster investment than would otherwise take place to realise Scotland’s full potential.

20. Scottish Enterprise and HIE were established under the Enterprise and New Towns (Scotland) Act 1990. The Act defines Scottish Enterprise’s functions as to:

•	further the development of Scotland’s economy and, in that connection, provide, maintain and safeguard employment
•	enhance skills and capacities relevant to employment in Scotland and assist persons to establish themselves as self-employed persons there
•	promote Scotland’s industrial efficiency and international competitiveness
•	further improve the environment of Scotland.

21. Scottish Enterprise and HIE deliver their functions through national teams and networks of Local Enterprise Companies (LECs). LECs were established in 1991 as companies limited by guarantee with local business-led boards. Many of Scottish Enterprise’s and HIE’s powers were delegated to the LECs under operating contracts to enable LECs to better assess and deliver services to meet local needs. LECs became wholly-owned subsidiaries of Scottish Enterprise and HIE in 2000. There are 12 LECs in the Scottish Enterprise area.

22. An organisational review is currently under way at Scottish Enterprise, which may result in structural changes. In part, this is in response to Scottish Enterprise’s emerging ‘city regions’ agenda. This entails greater focus on the economic performance of key metropolitan areas. The details of any changes have yet to be announced, but the metropolitan regions are likely to centre around Glasgow in the West and incorporate Edinburgh, Dundee and Aberdeen in the East.

23. Scottish Enterprise spends around £500 million each year delivering services related to its statutory responsibilities. Some £400 million of its budget comes from the Scottish Executive Enterprise Transport and Lifelong Learning Department (SEETLLD); the rest comes from its own business income (for services provided and from property and investments) and from grants from the European Union.

24. The framework under which Scottish Enterprise operates is set out in a management statement drawn up by SEETLLD in consultation with Scottish Enterprise. The framework covers:

•	rules and guidelines relevant to the exercise of Scottish Enterprise’s functions, duties 
and powers
•	the conditions under which any public funds are paid to Scottish Enterprise
•	how Scottish Enterprise is to be held to account for its performance.
Strategic direction

25. In January 2001, the Scottish Executive published A Smart, Successful Scotland (SSS), a comprehensive policy statement which sets out the strategic direction for Scottish Enterprise, HIE and their networks of LECs. It offers a strategy for improving the competitiveness and productivity of the Scottish economy focused around three key themes – Growing Businesses, Global Connections and Skills and Learning. Twelve specific priority areas are associated with these themes.

26. A refreshed version of the document, published in November 2004, retained the themes and priority areas identified in the earlier version, but emphasised that Scottish Enterprise and HIE needed to work in partnership with other parts of the public sector if improvements are to be delivered.

27. Over the period 2001/02 to 2004/05, Scottish Enterprise has spent just over £2 billion. Total annual expenditure has varied very little, although the composition of the annual total has changed over time. The biggest change has been a fall in expenditure within the Global Connections theme. 
 
The study

28. In November 2003, the Auditor General for Scotland produced a special audit report on Scottish Enterprise. The report covered a number of areas where Scottish Enterprise performance activity (as reported in its Annual Report to Parliament) had been the subject of media speculation. The Parliament’s Audit Committee made a number of recommendations for improving performance management in Scottish Enterprise, and supported a more detailed review of Scottish Enterprise’s performance management in the future.

29. The overall aim of this study is to determine whether the performance management framework for Scottish Enterprise provides a complete and accurate record of strategic and operational achievements. It covers:

•	the performance of the Scottish economy and progress towards A Smart Successful Scotland (Part 2)
•	the framework for measuring performance and the information available on Scottish Enterprise’s achievements (Part 3)
•	how Scottish Enterprise develops, monitors and evaluates its activities and how it manages performance to encourage continuous improvement (Part 4).

30. The results of the study provide a review based on information available at this stage. But measuring the impact of assistance to economic development is difficult. There is no easy way to assess the extent to which investment by an economic development agency influences overall changes in economic performance. 

31. We compared Scottish Enterprise’s approach to performance measurement with 
that of other international organisations undertaking similar functions (Appendix 1). We found that other organisations also struggle with these issues and that Scottish Enterprise’s approach is well advanced. Wherever possible, we used that information to place Scottish Enterprise’s activity in context. 

32. Scottish Enterprise is changing its approach to performance management, including a number of initiatives to improve the quality of its performance information. These initiatives are at different stages in development and many have yet to deliver results. The report therefore comments on what is currently in place and recognises the changes that are under way. 

1. A Partnership for a Better Scotland set out the key principles guiding policy development and implementation for the current Parliament.

Part 2. Scottish economic context

Key messages

Scottish Enterprise’s activities should all support the growth of the Scottish economy. However, the performance of the Scottish economy will be subject to a wide range of external influences that are beyond the control of Scottish Enterprise, so that Scottish Enterprise’s performance needs to be considered within the wider context of the performance of the Scottish, UK and international economies.

In recent years, Scotland’s economic performance has been mixed. Progress has been made in narrowing the gaps between Scottish performance and that of UK and international comparators, but performance remains weak in a number of areas and there is considerable variation between Scottish regions.

33. To set the context for Scottish Enterprise’s performance, this part of the report looks at:

•	the overall performance of the Scottish economy
•	progress towards the goals set out in A Smart, Successful Scotland
•	regional variations in economic performance.
Scotland’s performance relative to the UK

34. For much of the last decade, Scotland’s economic performance has been weaker than that of the UK as a whole. With the exception of 2002, when Scotland grew marginally faster than the UK as a whole, 2 the Scottish economy has grown at a slower rate than the UK in every year since 1995. 

35. The gap between Scottish and UK gross value added (GVA) per head has also widened. GVA measures the total output of an economy and GVA per head is a key indicator of economic prosperity.3 In 2003, Scottish GVA per head stood at 94 per cent of the UK level, compared with 99 per cent in 1995.4

36. The UK-wide figure is heavily influenced by the strong performance of London and the south-east of England. Scotland’s GVA per head remains higher than many English regions, and higher than Wales and Northern Ireland. Nonetheless, the figures indicate a widening gap between Scotland and the UK average, that sets a challenge for Scottish Enterprise in helping to close this gap.

Progress towards A Smart, Successful Scotland

37. A Joint Performance Team (JPT) comprising representatives from the Scottish Executive, Scottish Enterprise and HIE was established in 2001 to measure progress against SSS and develop an evaluation plan to assist in assessing the impact of the Enterprise Network interventions. The team publishes annual progress reports assessing performance against each of the SSS indicators and comparing results with those recorded elsewhere in the UK and in other OECD countries. 

38. The most recent report, published in December 2005,5 identified progress in a number of areas. The gap between Scotland’s performance and that recorded by the rest of the UK has narrowed for all but two of the indicators – GDP per capita and productivity levels. 

39. Although changes in the definition of indicators used makes comparison with other OECD countries difficult, a review of information presented in the report published by the Scottish Executive suggests progress. Scotland continued to narrow the gap in performance for eight of the twelve indicators assessed. There was no change for one indicator (business research and development as a percentage of GDP) and a widening of the gap for three of the indicators (percentage of businesses trading online, productivity levels and percentage of employers exporting). 

40. As a result of the progress recorded, latest data position Scotland in the top quartile for four of the lead indicators:

•	percentage of businesses 
trading online
•	percentage of employers exporting
•	employment rate
•	percentage of those in employment undertaking training.
41. However, Scotland has not moved higher than the third quartile for:

•	new business starts per capita
•	business research and development as a percentage 
of GDP
•	net migration as a percentage of the population
•	proportion of 16 to 19-year-olds not in employment, education or training (NEET).

Regional variations in performance

42. Economic performance across the regions of Scotland varies considerably. Scottish Enterprise uses a wide array of data at LEC level to assess and compare regional performance against SSS indicators. Scottish Enterprise’s latest estimates suggest that, in 2002, GVA per head ranged from just under £9,000 in Dunbartonshire to over £20,000 in Glasgow.6 

43. This type of analysis is influencing the way Scottish Enterprise considers the regional distribution of resources.There is increasing acceptance that this is, to some extent, inevitable. Economic activity will always tend to gravitate towards major centres of population. Rather than trying to counter this effect, Scottish Enterprise is developing a ‘city regions’ approach that focuses on how surrounding areas can contribute to and benefit from the growth in major conurbations.

2.	As measured by gross value added (GVA) at current basic prices.
3.	GVA is linked to gross domestic product (GDP) as follows: GDP = GVA + taxes - subsidies.
4.	Regional Gross Value Added, National Statistics First Release, 22 December 2004.
5.	Measuring Progress Towards a Smart Successful Scotland, Scottish Executive, 2005.
6.	These data are estimates and to some extent reflect commuting patterns eg, Glasgow’s figure is boosted by the fact that many people commute into work there, which raises the GVA figure (which is workplace-based) without increasing the population base (which is residence-based).

Part 3. The performance measurement framework

Key messages

The Scottish Executive uses a range of 38 indicators to monitor progress towards achieving the ambitions set out in A Smart, Successful Scotland. The framework focuses on the overall performance of the Scottish economy.

Scottish Enterprise provides a summary of achievements against ten key progress measures in its annual report, focusing on the activities and outputs of the network. These progress measures give a limited picture of achievements. The performance data make no explicit link to the SSS outcomes and it is therefore difficult to assess how Scottish Enterprise activities are contributing to the overall strategy. 

Scottish Enterprise uses a wider range of information internally to assess performance. The ‘balanced scorecard’ approach provides a more complete picture, 

but the emphasis has again tended to be on ‘process’ rather than ‘impact’ measures. 

To a large extent, the lack of impact data reflects the complexities of measuring outcomes in relation to economic development activity. Other economic development agencies in the UK, and internationally, face similar challenges, and no simple solution exists. Scottish Enterprise has attempted to measure the overall impact of its activities, most recently in 2003. This exercise, which should be interpreted with caution, estimated that its activities had added some £1.6 billion to Scottish GDP over three years, as a result of activities in 2001/02. 

44. This section examines:

•	how the Scottish Executive measures progress towards A Smart, Successful Scotland
•	how Scottish Enterprise reports on progress against key performance measures
•	additional information used to provide a ‘balanced scorecard’ of performance for Scottish Enterprise
•	the issues and complexities involved in measuring outcomes in economic development.

Measuring progress towards A Smart, Successful Scotland 
45. The Scottish Executive sets the strategic direction for Scottish Enterprise through A Smart Successful Scotland. 

46. A Smart Successful Scotland sets out a range of indicators for monitoring Scotland’s economic performance. There are two headline measures of progress supported by 12 lead indicators and 24 sub-measures, giving a total of 38 indicators. They are designed to provide a framework that can be benchmarked against comparative economic performance in the UK as a whole and in other OECD countries. There have been only very minor changes to the measures since 2001, although the Scottish Executive is expected to complete a consultation exercise on the appropriateness of the measures currently in use early in 2006. 

47. The Executive’s annual Measuring Progress reports present a snapshot of the relative performance of the Scottish economy and provide some analysis of the reasons underlying Scotland’s relative position against each indicator. However, they do not comment on the contribution of Scottish Enterprise, or the role of Scottish Enterprise in seeking to influence performance against these indicators. 
 
Scottish Enterprise’s performance measurement framework

Scottish Enterprise’s progress measures
48. Scottish Enterprise’s Operating Plan sets out how it intends to tackle the challenges identified in A Smart Successful Scotland. It is a rolling three-year plan, updated annually. Within the context of SSS and Scottish Enterprise’s planned activities, the plan also sets out ‘in year’ progress measures. The latest plan (2005-08) 7 identifies ten progress measures against which performance is to be reported:

•	business starts assisted
•	major high-growth business 
start-ups assisted
•	growth businesses improving their performance
•	knowledge transfer supported 
by the network
•	working age population assisted with career planning
•	number participating in occupational training programmes
•	number achieving positive outcomes from occupational training
•	companies helped to develop their workforce
•	high-potential organisations in priority industries helped to do business internationally
•	planned high-value jobs brought to Scotland as a result of inward investment support.

49. The current set of ten measures represents a significant rationalisation in the range of headline measures used in the past (from a total of 28 measures in 2003/04).

50. As well as a reduction in the number of measures used, there have also been a number of changes in the measures themselves, including both fundamental changes and more subtle changes in definitions. Some of these changes reflect policy developments; others represent refinements to improve the focus of the measures and mitigate any undesirable effects they might create. 

51. Examples of changes include:

•	In the first three years there was a measure to record the number of spin-out businesses generated by company and/or academic research activity. This definition changed in 2004/05 to knowledge transfer supported by the network, reflecting recognition that knowledge transfer may occur in the absence of a spin-out being formed. The unit of measure was expanded accordingly to include licences acquired by businesses to exploit intellectual property, and new collaborative ventures between business and academia with the aim of knowledge transfer, in addition to spin-outs. 
•	The measure for the number of companies achieving the Investors in People standard has been broadened to SMEs engaged in workforce development to better reflect the overall objective. Again, this newer definition captures a wider range of activity, making year-on-year comparisons impossible. 

52. The changes in the measures make it impossible to provide a time series of performance information for all measures for all years. 

Scottish Enterprise’s reported achievements

53. Scottish Enterprise’s Annual Report to Parliament includes both qualitative and quantitative analysis. Its focus is on annual achievements rather than a review of performance and impact over a longer timeframe.
 
Qualitative analysis of performance
54. The qualitative analysis describes key achievements during the year, highlighting successful projects and investments. It describes the rationale for the main activities supported and the allocation of funds. Areas for future focus are also identified.

55. This analysis provides a helpful picture of the key areas of activity and achievements. Scottish Enterprise has introduced quarterly reviews of progress against performance measures and these reviews are scrutinised by the board. The Scottish Executive receives half- yearly updates which summarise progress made and form the basis of its assessment of performance. However, these performance reviews are not published and the annual report remains the key statement on Scottish Enterprise’s activities, outputs and impact. It highlights areas of good performance rather than presenting a balanced picture of performance across the year.

Quantitative analysis of performance
56. The quantitative analysis presented in the annual report provides evidence about performance on the ten progress measures. The measures are grouped into the three strategic themes; for each measure an expected output range is set and performance is reported against these. 

57. Two measures have remained unchanged through the whole period: 

•	new business starts
•	high-growth business starts (although the measure relating to high-growth starts has changed for 2005/06 so further time series analysis will not be possible).
58. Scottish Enterprise assisted a steadily increasing number of business starts each year between 2000/01 and 2003/04, although the total fell back slightly in 2004/05. The target also increased steadily over this period. Assisted high-growth starts showed greater variability over the same time period, although the target also fluctuated.
 
59. Reported achievements against the full set of ten measures for 2004/05 are presented in Exhibit 19. In isolation, these figures give a very limited picture of Scottish Enterprise’s achievements. Without contextual data on activity in the wider economy, or time series data for many of the measures, it is difficult to interpret the figures in any meaningful way; they simply provide a snapshot of activities and outputs at a point in time.

Are the measures ‘fit for purpose’?
60. The ten performance measures and the highlights of achievements in Scottish Enterprise’s Annual Report do not capture the full range of Scottish Enterprise’s activities and are supported internally by other management information and research and evaluation evidence to provide a more complete picture. The performance measures focus on activities and outputs rather than impacts; although Scottish Enterprise met or exceeded all its headline targets, this does not provide a picture of the impact that the activities have had on the economy.

61. The relationships between inputs, activities and outputs can be used to assess the economy and efficiency of an organisation and are important operational tools. However, an assessment of effectiveness requires an understanding of how inputs, activities and outputs relate 
to outcomes. 

62. Scottish Enterprise’s Operating Plan seeks to link its activities and progress measures with the SSS priorities and indicators, but these linkages are weak or non-existent in some cases. Exhibits 20, 21 and 22 show the linkages between SSS priorities and indicators and Scottish Enterprise’s activities and measures, as presented in the operating plan.
 
Growing Businesses 
63. There are four Scottish Enterprise ‘in year’ progress measures in the Growing Businesses theme. Three of these measures relate to the SSS priority of entrepreneurial dynamism and creativity and one relates to increasing research and development. There are no measures for two of the SSS priorities: global success in key sectors and creating more e-business.

64. The weaknesses with the growing businesses measures largely reflect the fact that they are ‘in year’ measures that are not designed to provide information on outcomes, which need to be assessed over a much longer timeframe:
 
• 	There is often no direct link to show how the activity and output measures identified will contribute to SSS outcomes being achieved. 
•	While the measures are specific to the activity being delivered, only one of the four (growth businesses showing improved performance) attempts to measure qualitative improvement. 
•	There is no use of benchmark data eg, total number of business starts in Scotland in a year relative to the number assisted by Scottish Enterprise.
•	The measures make no attempt to address issues of additionality and attribution ie, how much of any observed output is wholly due to the intervention of Scottish Enterprise?
•	None of the growing business measures address cost issues, ie, what inputs have gone into achieving an output? 

Skills and Learning 
65. The second of Scottish Enterprise’s three themes is Skills and Learning. Four of the ten headline measures monitored are within this theme. Again, one of the four SSS priorities – narrowing the gap in unemployment – does not have an associated measure (although measures such as participation in training which appear under other priorities will also influence this priority). 

66. Many of the same limitations apply to those in the Skills and Learning theme. In addition, the Skills and Learning measures tend to be quantitative in focus and have limited reference to qualitative achievements that may have been realised.

Global Connections 
67. The remaining two measures relate to the strategic theme of Global Connections. This is the weakest theme in terms of the relationship between SSS strategic priorities and Scottish Enterprise progress measures. Measures exist for only one of the four strategic priorities, and a number of key activities with substantial funding levels, such as major infrastructure projects, are not included in the measurement framework. 

68. Internally, Scottish Enterprise uses a much wider range of information to assess performance. For example, some of the information gaps highlighted above are addressed periodically through evaluation studies which allow for a much more in-depth analysis of performance in relation to specific interventions. Also, use is made of contextual and qualitative data to inform internal analysis eg, assisted business starts are reviewed in the context of all business starts and information is gathered on the survival rates of new businesses. But this type of information is not always collected in a strategic manner, and it does not feature in public performance reporting, which remains heavily focused on activities and outputs. 

A balanced scorecard for Scottish Enterprise

69. To present a balanced view of an organisation’s performance, objectives and information reflecting a range of perspectives need to be considered. Scottish Enterprise has adopted a balanced scorecard approach and considers performance from four perspectives. 

•	Stakeholder: How is Scottish Enterprise contributing to the strategic objectives envisaged by the Scottish Executive?
•	Customer: How are Scottish Enterprise’s services perceived by its customers?
•	Process: How effectively and efficiently is Scottish Enterprise delivering its services?
•	People: How do the staff support the achievement of the Scottish Enterprise objectives and how satisfied are they?

70. This allows for a more rounded analysis of performance. It also balances the need for regular on-going monitoring of activities and outputs (‘process’ perspective) with the need for longer-term strategic analysis of impact (‘stakeholder’ perspective).

71. Scottish Enterprise’s set of ten progress measures focuses essentially on the ‘process’ aspect of the list above. Additional information is collected within Scottish Enterprise to inform the assessment of wider aspects of performance and provide a more complete picture. This includes much more detailed process information, as well as information to address the other perspectives within the balanced scorecard. This exhibit does not aim to present a full picture of the information available (which is extensive), but simply highlights examples of the type of information available within each perspective.

72. The balanced scorecard approach provides a more rounded picture of Scottish Enterprise’s performance, from a range of perspectives. This information is used as part of quarterly reviews with the Executive board, which were introduced in Autumn 2005. 

73. Although process information is important and valuable, it needs to be balanced against other aspects of performance. For Scottish Enterprise, the key challenge remains in understanding the combined impact of all activities. This is a highly complex area, but forms an important element of a comprehensive assessment of performance. Three board committees were established in Autumn 2005 (Performance, Investment and Operations) and the Performance Committee will be seeking to address these complex issues as part of its remit.

Linking inputs to outcomes

74. The Scottish Executive’s monitoring of SSS indicators is primarily concerned with outcomes and changes in Scottish economic performance, while Scottish Enterprise’s published performance information focuses on its own activities and outputs. Exhibit 24 shows the linkages between inputs, activities, outputs and outcomes for Scottish Enterprise. Scottish Enterprise’s Operating Plan shows the link between its planned activities and SSS outcomes, but there is limited evidence of linkages when performance is assessed. This would enable the Scottish Executive to challenge Scottish Enterprise more effectively on its contribution towards the goals contained within SSS.

Measuring outcomes

75. The difficulties in providing information on Scottish Enterprise’s overall impact should not be underestimated. In the field of economic development, measuring outcomes is a highly complex task. Routinely-collected performance information on activities and outputs does not provide for an assessment of outcomes, as the linkages between outputs and outcomes are complex and require careful analysis and interpretation. But measuring outcomes is a critical element of performance management and these complexities cannot be ignored. 
 
76. The type of information required for a fuller picture of Scottish Enterprise’s achievements is neither cheap nor easy to collect. A range of issues make the task of measuring the impact of an economic development agency particularly complex. In particular:

•	Attribution – changes in economic performance will reflect the combined influence of a wide range of factors, including many which are beyond the control of Scottish Enterprise, such as interest rates and exchange rates. Also, a business or individual may have received a number of different types of support from Scottish Enterprise and other bodies. Isolating the influence of any single factor is impossible without making a range of assumptions, which may be subject to some degree of error.
•	Additionality – identifying how much of any observed change is due to a particular intervention by Scottish Enterprise involves assessing what might have happened in the absence of that intervention. Again, this requires a range of assumptions to be made.

77. This means that activities and outputs can be measured using data that are relatively easy to collect and analyse, but impact can only be assessed through evaluation studies which consider the impact of a particular intervention in detail. However, evaluation can only take place some time after the intervention, to allow impacts to have been achieved. As a result, evaluation evidence will only be available after some time – often several years.

78. Scottish Enterprise has attempted to measure its overall impact, building on the extensive range of evaluation evidence it holds. This is a complex exercise, subject to a wide margin of error and highly dependent on the extensive range of assumptions made, so the results should be treated with a degree of caution. Nonetheless, it is important in providing the scale of impact. 
 
79. A two-stage exercise was undertaken in 2003, relating to activity in 2001/02. This was a detailed and comprehensive exercise, the second stage of which used an econometric model to simulate effects on both the Scottish and UK economies. It estimated that Scottish Enterprise’s activities in 2001/02 generated additional GDP in Scotland of some £1.6 billion over the course of three years.8

80. Scottish Enterprise acknowledges that the methodologies employed require improvement and the exercise highlighted some of the methodological difficulties involved. There are no immediate plans to repeat such an exercise; the current focus is on improving the quality of evaluation and using the results of evaluation work more effectively.

International experience
81. The lack of evidence of impact at organisational level is not unique to Scottish Enterprise. As part of this study, we reviewed a number of agencies in the UK and internationally to compare approaches to performance measurement and management and determine if lessons learned could be applied to Scottish Enterprise. The review examined approaches to performance measurement and management and – where it was made explicit by the organisations concerned – the rationale for measuring performance. Organisations in the following countries were included in the review:

•	Australia
•	Canada
•	Denmark
•	England
•	Ireland
•	New Zealand
•	USA.

82. The review confirmed that other agencies struggle with the same issues and there is no simple solution. These examples highlight the importance of gathering good-quality, reliable information in any attempt to measure impact. The Canadian example demonstrates the next stage in the process, using the information in an attempt to measure impact through the use of an econometric model.

83. The Danish Government’s Regional Growth Strategy published in 2005 presents benchmark data on a number of economic and social performance measures for its regions and between Denmark and other countries.

Recommendations

Recommendation 1: A clearer understanding of the contribution Scottish Enterprise is making to achieving the ambitions of SSS is required, and the Scottish Executive should work with Scottish Enterprise to achieve this. 

Recommendation 2: In assessing performance, Scottish Enterprise should make greater use of longitudinal assessment and use this to make more explicit reference to the contribution being made to SSS outcome measures. 

Recommendation 3: Scottish Enterprise should further investigate the potential of econometric models to estimate the combined impact of its activities.

7.	http://www.scottish-enterprise.com/publications/se-operating-plan05-08.pdf
8.	This compares with total GDP (at basic prices) of £70.4 billion in 2000 (Source: ONS).

Part 4. Developing, monitoring and evaluating activities

Key messages

The processes for developing, monitoring and evaluating activities are critical to effective performance management. Scottish Enterprise has introduced new procedures for appraising and approving projects which should help improve strategic fit and promote efficient allocation of resources.

Targets have a valid role in monitoring activities and outputs, but they must be carefully selected and should not be the main driver for activity.

The use of GVA as an impact measure should help focus on outcomes, although the concept needs to be clearly understood and used in conjunction with 
other measures.

Evaluation is a critical element of impact assessment and the current variation in approaches and quality needs to be addressed. Greater emphasis should be placed on broader, strategic evaluation studies, and more effective mechanisms for disseminating and responding to findings need to be introduced.

There is evidence of positive change within Scottish Enterprise in moving towards a performance improvement culture. However, Scottish Enterprise is large and dispersed, and there is some way to go before staff throughout the organisation understand the positive role of performance management.

84. Effective performance management relies on high-quality management information and effective use of this information. Underlying this, a well-designed framework for selecting, monitoring and evaluating activities is essential.

85. This part of the report looks at how Scottish Enterprise selects activities and subsequently monitors and evaluates the chosen activities 
by considering:

•	developing activities
•	target setting
•	monitoring progress
•	evaluating performance.
•	changes under way at Scottish Enterprise.

Developing activities

Appraisal and approval
86. If Scottish Enterprise’s activities are to maximise impact in relation to the strategic objectives set out in SSS, resources must be allocated with the ultimate objectives of SSS in mind, with clear linkages between the individual activities and the objectives towards which they aim to contribute.

87. The process for appraising and approving new projects is therefore an essential element of the performance management process. If these processes are effective, resources will be allocated efficiently and outcomes will be maximised. Once activities are identified, it is crucial that good-quality monitoring information is available to support subsequent evaluation to inform future delivery.

88. To understand how effectively the appraisal and approval process is operating within Scottish Enterprise, we reviewed a small sample of 78 project appraisals. The sample included projects of a range of sizes and from a range of LECs across the network. In relation to appraisal and approval, the review considered the following stages:

•	identification of the strategic rationale
•	objective setting
•	options appraisal
•	detailed appraisal of the selected option
•	approval process.

89. Each project was scored against a framework, developed in conjunction with Scottish Enterprise. The framework criteria were based on guidance issued by Scottish Enterprise. Where projects were approved prior to the latest guidance being issued, assessments were based on good practice.

90. Identification of strategic rationale. This is the first step for Scottish Enterprise in determining whether to intervene in the market place where it is failing to operate effectively. The majority (79 per cent) of projects clearly stated a rationale for the project in the appraisal paperwork, but this rationale was based on market failure considerations in little over half of projects and only 40 per cent of projects gave factual evidence to support the rationale. 

91. Objective setting. All projects should have clearly stated objectives setting out the changes that will be achieved through delivery. Generally projects scored well on this element of assessment, with the majority having clearly stated objectives and three-quarters linking objectives to change. But only 32 per cent of projects had measurable objectives. 

92. Options appraisal. It is critical that an options appraisal is undertaken prior to deciding how to intervene in the market to ensure that the right type and level of intervention is achieved. This was the weakest stage in the project appraisal cycle. Only six per cent of projects fully appraised the options to determine the preferred approach and the overall process was considered ineffective in more than half (55 per cent) of cases reviewed. 

93. Appraisal of preferred option. The preferred option generated through the option appraisal process should be subject to rigorous assessment to determine the risks and benefits associated with delivery. This assessment should cover the inputs, management structure and capabilities required to deliver the project, and an exit strategy on completion of the project. In general, appraisal papers scored well on assessing delivery requirements – 87 per cent of projects adequately assessed inputs and 75 per cent had sufficient consideration of the management capabilities required.

94. The main weaknesses were in identifying potential benefits (only 38% of projects did this fully) and how these could be attributed to Scottish Enterprise’s intervention (only 16 per cent of projects considered additionality). There was also limited consideration of what would happen at the end of the project – only 36 per cent of projects had a clear exit strategy.
 
95. Approval process. Appraisal papers are submitted for formal approval to the relevant body. The process differs depending on the nature of the project: large national projects are reviewed by the Scottish Enterprise board, while larger LEC-specific projects are considered by the LEC board, and smaller projects do not require board approval. The board should review the appraisal papers and challenge the assumptions. Most projects are approved with limited changes. From our analysis of 78 projects, only three projects were returned for further comments and four were rejected outright. For ten projects (13 per cent), there was no approval paper available.

96. Despite this high level of first-time approval, only 38 approval papers were considered to be effective and compliant with the available guidance. A further 27 were assessed as being partly effective, with the remaining 13 approvals assessed as ineffective. This suggests that in some cases there is insufficient rigour in the process leading to approval of projects, without adequate review and enhancement.

The Gateway review process
97. In September 2003, Scottish Enterprise introduced a much more rigorous appraisal, approval and evaluation process, known as the Gateway review process. This is a five-stage process providing a clearly structured format for the development and delivery of projects which is in accordance with HM Treasury guidance. It incorporates:

•	strategic review
•	project appraisal
•	investment decision
•	implementation review
•	evaluation.

98. There is now comprehensive and clear guidance for all stages in the project lifecycle. The system applies to all projects, from large infrastructure projects down to smaller-scale interventions (although smaller projects use a much simplified version). Extensive training in the Gateway process has been delivered to staff at all levels to ensure that they fully understand the processes involved and their importance to the development of high quality projects. This has been accompanied by the development of comprehensive support materials and peer networks.

99. In the past, the appraisal and approval process has been viewed by some staff as a barrier to getting on with project delivery, rather than a potentially constructive process that can help in developing a better quality project. These perceptions undoubtedly still persist to some extent across the network, but Scottish Enterprise anticipates that training will help to change attitudes and ensure that project appraisal is seen in a more positive light. However, it is too early to judge whether the new procedures are having a positive impact on the quality of the appraisal and approval process.

Allocating resources
100. Until this year, budgets were allocated to each broad theme at the start of the financial year, and then broken down further to individual LECs. This may have created an incentive for LECs to spend up to their budget allocation, with a risk that some projects may have been selected by LECs with this objective in mind, rather than on the basis of economic justification or strategic fit. 

101. In 2005/06, Scottish Enterprise introduced a major change in the way it allocates funds to projects. Budgets have not been allocated in advance either by theme or geography. Instead, project staff apply for funds as project proposals are approved during the course of the year. Applications are reviewed throughout the year and approved or rejected following review. There is no set budget limit for specific areas of activity. Financial control relies on robust forecasting and strategic alignment relies on quarterly management reviews.

102. This rolling system should help ensure that projects are driven by economic justification rather than budgetary limits. As a result, the selected projects should more closely reflect the ultimate objectives and strategic intent of the organisation, provided that the appraisal and approval system works effectively. However, this is the first financial year that this approach has been used, and management reports to the Scottish Enterprise Board indicate that demand has exceeded the resources available. It is too early to comment on the effectiveness of management action to contain expenditure, but audited accounts will be available later in 2006.

Setting performance expectations

103. Each of Scottish Enterprise’s ten progress measures has an associated output range, published in the operating plan. Until 2004/05, Scottish Enterprise’s Operating Plan included targets for each performance measure. However, because Scottish Enterprise considered that the targets were driving activity in too narrow a way, they were replaced in 2004/05 with indicative output ranges. This allows for greater flexibility in the distribution of resources and should help avoid excessive focus on a single figure. 

104. Output ranges are developed by Scottish Enterprise and agreed with the Scottish Executive.9 They are reviewed annually. In the past, the review of targets was primarily based on past performance and available resource, seen within the context of the policy objectives. Since Scottish Enterprise generally outperformed against many of its headline targets, the targets tended to increase over time.

105. There appears to be an implicit assumption that, for many areas of activity, a higher level of activity is better and will achieve greater impact. For example, the target for new business starts assisted by Scottish Enterprise has risen from 6,600 in 2000/01 to a target range of 9,000-10,000 for 2005/06. However, it is not clear that the review of this target takes account of the wider economic environment in which Scottish Enterprise operates.

106. Although a higher number of business starts is generally considered to be a desirable aim, this does not necessarily imply that a higher number of assisted business starts is desirable. A steadily increasing target is not necessarily appropriate because, with higher levels of Scottish Enterprise intervention in the new business market, there is an increasing risk that Scottish Enterprise funds will be used to support businesses that would have started up regardless. To make effective use of public funds, support should be targeted at achieving outcomes that would not be achieved without assistance. It is not easy to identify and support only those that would not otherwise start up, but this should be the guiding principle.
 
107. Greater emphasis is now being placed on the use of external evidence to inform the setting of output ranges. Output ranges should reflect evidence about the appropriate level of intervention to maximise the impact of public sector support. This will require research to investigate the effectiveness of different types of intervention, and Scottish Enterprise is seeking to gather better information in this respect.

108. Targets (or output ranges) have a valid role in informing the monitoring of activities and outputs through the year and play a part in performance management, but should not become an end in themselves.
 
Monitoring progress

Management information
109. Within Scottish Enterprise, information relating to headline measures and a wide range of supporting measures is reported through two systems – the Knowledge Management Information System (KMIS) and the Corporate Training System (CTS). This information is used extensively for management reporting at LEC level and at national level. It provides a basis for senior management to review progress against targets on a quarterly basis (or more regularly at LEC level). Areas of poor performance can be highlighted and addressed, and examples of good practice shared.

110. The monitoring data are reviewed in the context of the relevant geographies or markets to provide an informed analysis of the data. Nonetheless, there has been a tendency in the past to focus on targets, using management information to assess whether a business unit or team is managing to achieve given levels of activity with the allocated resources and within a specified timeframe. This type of review takes no account of the effectiveness of activities.

Quality of management information
111. The value of monitoring information will depend on the quality of the information collected. Previous reports by the Auditor General for Scotland have highlighted concerns over the quality of monitoring data.10 We analysed the quality of the information contained within KMIS and CTS in respect of the ten headline measures used by Scottish Enterprise. A sample of cases was investigated to verify that there was clear evidence for each claimed output.

112. Data on activities and outputs in relation to training programmes are robust and there are clear procedures for recording the data via CTS. In part, this reflects the clear cut nature of these activities and outputs. An internal team (National Compliance Service) is responsible for checking the data, primarily because the evidence is used to support payment claims by external training providers.

113. For other measures, there were a few issues with regard to the timing of claims, but data generally seemed robust. However, there are a number of concerns about the quality of the KMIS data in relation to business starts:

•	Business starts claimed without clear evidence of the business actually trading. 
•	Business starts claimed where the LEC assistance appears to have been provided after the business was set up (although this is in some cases allowed within the guidance).
•	Lack of evidence of the nature or duration of the LEC assistance.
•	Inconsistency in the criteria used to judge whether an output has been achieved.
 
114. In almost half (43 per cent) of the 120 business start claims investigated, one or more of the above issues was raised. These issues partly reflect concerns that have been addressed through improved guidance. Any remaining concerns similarly relate to the guidance available at Network level and within the Business Gateways.

115. This indicates a need for clear and unambiguous guidance on how outputs are recorded, particularly in relation to business starts where the Business Gateways have an important role in the delivery of services. It also reinforces the need for careful consideration of the type of measures used and the incentives that they may create.

Project-level monitoring
116. Effective monitoring at the project level requires an appropriate framework for collection of information. This framework needs to be established at the outset of a project. The Gateway process requires that project staff define a monitoring framework as part of the project development process. This means that all projects should have a monitoring framework in place from the outset which defines the type of information to be collected, frequency of collection, responsibilities for collection and analysis requirements.

117. We reviewed a total of 49 projects to assess the quality of the monitoring activity associated with those projects:

•	78 per cent of projects had a monitoring framework in place.
•	42 per cent of frameworks were judged to be fully measuring the appropriate data, although a further 47 per cent had only partial data.
•	In only 26 per cent of cases was the framework used by management to assess the effectiveness of project delivery. 

118. Overall, this exercise highlighted that project-level monitoring tends to focus on activity data, with limited reference to output or outcome data. Also, monitoring information on individual projects is not used effectively by management to improve performance or inform wider operations, strategy or policy. In both areas, there is room for improvement.

119. Project staff need to fully understand the potentially valuable role of well-designed monitoring activity. The key here is in careful design – better monitoring does not necessarily imply collection of more data. The volume of data is not the issue – it is the relevance and usefulness of the data that is the key factor. The data collected should reflect the main risk areas and allow these to be adequately assessed. Senior management need to ensure that data are effectively used to influence future activity, and staff must be fully engaged in the active use of monitoring information, so that they can appreciate its value and see its practical application.

Evaluation of performance

The role of evaluation
120. Activity and output data should not be used in isolation, but need to be reviewed within the context of outcome data to enable a comprehensive assessment of performance. At a project level, the impact of a particular intervention (or group of interventions) is typically assessed through an evaluation study. This usually takes place on completion of a project, or some time after implementation, to allow for impacts to have been partially, if not fully, achieved. This means that outcome data are not generally available as early as activity and output data.

121. An economic evaluation study should consider issues such as:

•	the rationale for assistance
•	observed changes in performance of the target group eg, assisted businesses
•	the extent to which observed changes can be attributed to the particular intervention (rather than external factors, other interventions, support from other agencies, etc)
•	what changes might have been observed in the absence of any intervention (the counterfactual) 
•	the efficiency, economy and effectiveness of delivery.
122. These are complex issues which cannot be easily measured or quantified. For this reason, evaluation studies – which can be costly – are not carried out for each and every intervention on an annual basis. Rather, evaluations are carried out according to needs and priorities, using a risk framework to determine the evaluation plan for a given year.

Quality of evaluation
123. Scottish Enterprise evaluation studies are generally conducted by external consultants, because of the level of resources and the specialist skills required. This also ensures independence and neutrality. However, the use of a wide range of consultants inevitably results in variations in the methodological approaches used, the nature of the assumptions made and the quality of results.

124. We examined the evaluation activity associated with 49 projects across the network. This included the extent to which the key stages in evaluation were completed and whether feedback was provided to the organisation as a whole in order that learning can take place. We found:

•	Just over half (51 per cent) of projects reviewed had an evaluation that mapped all the activity delivered – and significantly fewer projects recorded all inputs (30 per cent) or all expenditure (20 per cent). 
•	Well over three-quarters of evaluations (85 per cent) either fully or partially assessed outputs and outcomes, but only 40 per cent calculated net additional impact achieved.
•	Less than one in four evaluations included a value-for-money assessment.
•	Over half the evaluations did not test the strategic rationale (55 per cent). 
•	There was mixed evidence that lessons learnt were effectively disseminated and acted upon – in only half of cases was there evidence of feedback being acted upon.

125. Scottish Enterprise is aware that there is variance in the quality of the evaluation evidence and is seeking to address the issue, initially through training staff, so that it can design better consultancy briefs, provide more critical comment on the consultancy reports and challenge the findings. Extensive guidance has been produced for staff. This should help improve the quality and consistency of the evaluation reports, although much will still depend on the ability of the commissioned consultants to deliver the standard of outputs required.

Developing the approach to evaluation in Scottish Enterprise
126. Scottish Enterprise’s approach to evaluation is evolving. With the involvement of the JPT, greater emphasis is now being placed on more strategic evaluations covering broad themes, rather than evaluations of individual interventions or those focused on local areas. Examples include the business birth rate review, the review of clusters and industries and the current evaluation of Skillseekers and Modern Apprenticeships. These are all large-scale, Network-wide investigations designed to challenge the nature and scale of Scottish Enterprise’s interventions in these areas and to understand their impact.

127. Each year, an evaluation plan is prepared identifying priorities for evaluation and research based on an analysis of risks. This balances the evaluation priorities against available resources, recognising that the broader strategic evaluations are costly, time-consuming exercises and there will be a limit to the number that can be undertaken in any given year.

128. Greater use of such evaluation evidence should help in providing a more complete picture of the impact of a group of interventions, taking into account their interactions and hence the combined impact on the businesses assisted. This will take Scottish Enterprise a step further in addressing the complexities of impact measurement across the whole organisation, rather than simply at a project level.
 
The use of GVA in evaluation
129. Since early 2004, Scottish Enterprise’s board and senior management team has placed much greater emphasis on the use of GVA as a means of assessing impact. GVA is a measure of the output of an economy that (unlike GDP) can be calculated for individual economic units, such as businesses, and for small geographic units or regions. Because it can be calculated for small economic units, it is a valuable tool for assessing the economic impact of interventions at a micro-level. As GVA is closely linked to GDP,11 it represents an appropriate measure for assessing Scottish Enterprise’s ultimate objective of supporting growth in the Scottish economy.

130. The aim of using GVA is to encourage project staff to focus on outcomes, rather than on activities and outputs. Net additional GVA should also provide a single measure for comparing impact across a wide range of different types of intervention, both at the project appraisal stage and at the evaluation stage. This should help inform resource allocation decisions and direct resources to where greatest impact can be achieved. However, the complexities involved mean that the concept is often poorly understood and there is a risk that it will be misused. It is essential that its meaning is clearly explained and communicated throughout Scottish Enterprise so that it is used in an informed manner, alongside supporting evidence and information.

131. It could be misleading to use GVA in isolation and Scottish Enterprise acknowledges this. Certain activities, such as business support, would be expected to show a direct contribution to GVA, but other activities such as regeneration or training will influence GVA in a much less direct manner and over a longer timeframe. This means that using GVA as a single measure of impact over a short timeframe might unfairly disadvantage certain types of project. The range of 12 lead indicators presented in SSS represent a more balanced basis for analysis. Ultimately, all of Scottish Enterprise’s activities should have an impact on GVA, but in some cases this may be a long-term impact and, in the short to medium-term, impact can only reasonably be measured in terms of a range of indicators.

Changes under way
132. A number of changes have been made, or are under way, that should encourage improved performance management within Scottish Enterprise. The key areas of positive development are highlighted in Exhibit 31.

133. These changes should sharpen the strategic focus of Scottish Enterprise’s activities and improve understanding of their contribution towards intended outcomes. However, this understanding will remain partial unless further work is done to allow for more regular assessment of impact at organisational level.

134. Scottish Enterprise is a large organisation with over 2,500 staff across 13 locations, and cultural change will only happen gradually. Among some Network staff, there remains a view that appraisal, monitoring and evaluation processes are essentially administrative tasks rather than tools to assist continual improvement. The training currently under way will go some way to addressing this, but the message needs to be reinforced through clear communication of the practical application of appraisal, monitoring and evaluation findings.

Recommendations

Recommendation 4: Target-setting should be informed wherever possible by external evidence, rather than based solely on past performance.

Recommendation 5: The focus of reporting should be on longer- term outcome measures, clearly linked to SSS progress indicators, with activity and output data used to monitor progress towards objectives.

Recommendation 6: Clearer guidance on definitions for recording activities and outputs should be prepared for staff to ensure the quality of the monitoring data.

Recommendation 7: Monitoring frameworks for projects should be designed at the outset to include outcome measures and the monitoring information should influence and inform future delivery.

Recommendation 8: Scottish Enterprise should continue to work to improve the quality and consistency of evaluation work and ensure that learning effects are disseminated effectively so as to influence future activity.

Recommendation 9: Evaluation activity should be redirected to deliver a wider picture of impact and should provide a more fundamental challenge to the rationale for activities.

Recommendation 10: Better use should be made of evaluation evidence to allocate resources and challenge the nature and scale of activities. 

Recommendation 11: Assessment should be based on the range of indicators presented in SSS, rather than on GVA alone, recognising that the paths to impact on GVA will vary considerably between projects.

Recommendation 12: Scottish Enterprise should continue working to ensure that the positive changes at management level are cascaded throughout the organisation.

9.	With the exception of the target for Modern Apprenticeships, which is set by the Scottish Executive and forms an important component of the occupational training programmes’ participants’ target.
10.	 Scottish Enterprise: Special audit examination, Audit Scotland, December 2003; Scottish Enterprise: Account management services to high-growth businesses, Audit Scotland, June 2004.
11.	GDP = GVA + taxes – subsidies

Appendix 1. A review of international performance measurement approaches

This Appendix presents the key messages from a review undertaken by SQW Ltd on behalf of Audit Scotland. The purpose of the study was to review international approaches to the performance measurement of economic development agencies and departments with similar remits to Scottish Enterprise.

Variability of approaches
1. As explained below, the review identifies both many different approaches, but also striking
similarities. For example, the agencies or government departments researched all had
relatively similar objectives, many around innovation, supporting employment, global connectedness, inward investment. Most set out their activities within the context of country or regional performance generally, often benchmarked against other countries using OECD data.

2. However, the tools for measuring performance did vary more widely – the detailed guidance developed at a federal level in Australia; substantial and in-depth consideration of approach at state-level in the USA; use of balanced scorecard approaches alone or in combination in the USA and Australia; sophisticated logic models in Canada; and use of large-scale surveys in Canada and Ireland. In Canada and in the US, there were formal reporting structures to which agencies had to adhere in order to fit with wider city reporting systems applied to a number of different agencies. In most cases, this was necessary in order to report to parliament/council and to demonstrate effective and efficient use of resources to national or state government.

No ‘silver bullet’
3. The emphasis of most reporting was on what is usually termed inputs, activities and/or outputs. This was usually in the form of the value of investment that had been made and, for example, the number of companies or projects supported, or training places created. This led to what we consider to be a logic gap or to two sets of performance information running in tandem.

4. On the one hand, a detailed assessment of the economic conditions with benchmarks is often
provided in performance reports, accompanied by information on agency inputs, activities and
outputs. The gap occurs at the interface between identifying the outputs resulting from an
agency’s activities with changes in the wider economic conditions.

5. We did not find any evidence of a ‘silver bullet’ here – we have not found structures which effectively bridge this gap or link convincingly and quantitatively the activities and outputs of an agency with changes in the wider economy of its area. However, we did find examples, notably in the logic models and survey work being deployed in Canada, where this gap was being narrowed and the link strengthened.

No sense that Scottish Enterprise in ‘thinking’ and approach is behind its peers
6. From our knowledge of Scottish Enterprise’s approach and from discussions during this work, we do not consider SE, although adopting often a different approach, to be behind its peers. From the many documents we reviewed, it is clear that most other agencies are wrestling with exactly the same issues.

7. It is also not easy to make direct comparisons between agencies for a number of reasons. At the level we were looking at, it can be difficult to tell whether some of the apparently more advanced models, for example around the use of a balanced scorecard or a logic model, are actually being implemented effectively by the organisations concerned. Also, although at a national level there are often attractive models described, reporting by the agencies in that jurisdiction that are supposed to be using the model is often disappointing.

Limited treatment in organisation-level reports of attribution and additionality
8. From our own work and knowledge of SE, we know how important the treatment of attribution and additionality to be in evaluation work. However, we found that among the agencies we examined, there was, at best, only a limited treatment of these issues in their performance reporting. This may be because we were looking primarily at the higher levels of reporting and that treatment of attribution and additionality is mainly handled at project or programme level through evaluation. Even so, many agencies appear to report the gross rather than the net effect of their interventions eg, the number of companies they assisted or the employment that they supported. Our sense is that SE probably pays as much or greater attention to these adjustments than appears to be the case among other agencies.

Limited use of GVA
9. One of the issues raised in the consultants’ brief was the use of GVA by other countries in
measuring the performance of their agencies. It is worth clarifying here the definition of GVA and GDP. GVA is the contribution to the economy of each individual producer, industry or sector in for example the UK. Approximately, GVA represents the income generated by businesses out of which is paid wages and salaries, the cost of capital investment and financial charges, before arriving at a figure for profit. It includes taxes onproduction (e.g. business rates), net of subsidies but excludes subsidies and taxes on products (eg, VAT and excise duty). So, GVA + taxes on products – subsidies on products = GDP.

10. We found very little evidence that GVA is used at all. Most agency strategies and reports use GDP as an indicator of the condition of the economy and as a benchmark for comparison with other countries and regions. Only in Atlantic Canada did we find contribution to GDP used as a measure of the agency’s performance. In Ireland, a survey of assisted businesses also provides a measure of change in net GVA per employee at a firm level.

11. Although there was limited evidence of its use, GVA is becoming more important as a measure of economic performance and potentially provides a link between firm level and regional or national economic performance, as well as being a measure of productivity. The Allsopp report draws a number of conclusions on the quality and methodological issues surrounding the collection and use of GVA statistics at different levels in the UK. In terms of the current study, this only highlights the importance of GVA as a measure of economicperformance but not specifically of an organisation’s performance. It also raises the importance of collecting data in a form that can be used with other sources or as an input into other models.

Benchmarking economic progress against peers using GDP and other metrics is common
12. Most agencies introduce their strategies and objectives using regional or national statistics that can be benchmarked against other countries or regions. This is not just true of the economic development function, but of many other areas of policy. The most common economic measures are GDP; GDP per head; employment and unemployment; business investment in R&D and exports. Often data on key sectors are drawn out (biotechnology and ICT are particularly popular). Despite often detailed analysis of the
current condition relative to competitors, this was not tied back convincingly to attributable,
additional contributions of an agency in high-level reporting.

Explicit treatment of partner contributions
13. In some cases agencies are required to work closely with partners and this is reflected in the way in which their performance is reported. The RDAs in England are one example, although in the case the profile of partner contributions to date is highest in strategy documents and action plans rather than in performance reports. In Canada, agencies deliver a large number of cross-cutting initiatives which they report on in order to contribute to national targets. This forms a large part of many of the performance reports and gives a sense of how partners contribute to the wider economic development objectives.

More obviously part of a wider performance measurement process followed by other agencies and departments
14. In several countries, Australia and Canada in particular, there was strong guidance on how agencies or departments should report performance. This meant that the performance data collected could be both compared across departments and agencies, but also more importantly could be aggregated or rolled-up to produce a state level or federal performance report. The Canadian guidance produced by the Treasury stresses the logic chains linking activities and outputs to strategic outcomes. In the US some states collect data from a number of agencies in order to produce a state-level report on performance. Although in practice this consistency is valuable, it still does not help understand individual agencies performance other than to reinforce the importance of making the links between activities and strategic outcomes.

Subject area of current concern to UK central government
15. Finally, these issues and the measurement of the performance of economic development agencies is a current topic in England. SQW has recently been commissioned by the Department of Trade and Industry to develop a framework for performance measurement of the English RDAs. This has already raised issues around:

• 	development of programme/project evaluations and the development of consistent logic chains from activities through to PSA targets
• 	statistical beneficiary analysis – as a way to look for broad patterns over time or for comparisons with control groups
• 	partners’ contributions – RDAs work with partners and through delivery agents more and more, which will raise double counting issues unless there is apportionment of spend/effort and of outputs/outcomes
• 	strategic added value (SAV) – RDAs are placing a lot of emphasis on the contribution they make through influencing others, but there has been limited work done on assessing this.

If you require this publication in an alternative format and/or language, please contact us to discuss your needs. It is also available on our website: www.audit-scotland.gov.uk

Audit Scotland 
110 George Street 
Edinburgh EH2 4LH 

Telephone 
0131 477 1234 
Fax 
0131 477 4567
 
www.audit-scotland.gov.uk 

ISBN 1 905634 11 0

AGS/2006/4