Performance management in the Scottish Qualifications Authority

Prepared for the Auditor General for Scotland
November 2006

Auditor General for Scotland
The Auditor General for Scotland is the Parliament’s watchdog for ensuring propriety and value for money in the spending of public funds.

He is responsible for investigating whether public spending bodies achieve the best possible value for money and adhere to the highest standards of financial management.

He is independent and not subject to the control of any member of the Scottish Executive or the Parliament.

The Auditor General is responsible for securing the audit of the Scottish Executive and most other public sector bodies except local authorities and fire and police boards.

The following bodies fall within the remit of the Auditor General:

•	departments of the Scottish Executive eg, the Health Department
•	executive agencies eg, the Prison Service, Historic Scotland
•	NHS boards
•	further education colleges
•	Scottish Water
•	NDPBs and others eg, Scottish Enterprise.

Summary
Introduction

1. The Scottish Qualifications Authority (SQA) is the national awarding body for Scotland, responsible for the development, accreditation, assessment and certification of qualifications (other than degrees and some professional qualifications). It has an annual turnover of around £51 million and employs around 650 full-time equivalent (FTE) staff. Its qualifications are taken by around 500,000 candidates each year and around 1,470 centres are approved to offer its qualifications. 

2. The SQA is responsible for a large range of academic and vocational qualifications. This includes the annual set of national examinations known as the Diet. The results of these examinations are issued in August and include Standard Grades and Highers.

3. In 2000, the SQA encountered significant problems with processing the summer exam results, leading to inaccuracy and incompleteness in 2.7 per cent of the results. A total of 17,000 candidates were affected.

4. The overall aim of this study was to assess the extent to which the organisation has recovered from its position in 2000. We considered whether the SQA’s current risk and performance management framework is appropriate, whether the organisation displays the principles of good governance and whether it demonstrates a commitment to best value through continuous improvement.

Summary of key findings

5. Since 2000, the SQA has been successful in re-establishing customer and stakeholder confidence in its core functions. In response to the problems encountered, the SQA has developed a much better understanding of its key business activities. It has gone beyond simply recovering from 2000, demonstrating an ongoing commitment to continuous improvement and it is now in a position to more critically challenge its operational processes.

6. The SQA has received additional government funding since 2000 and has also increased the income generated by charges for its qualifications. This has helped to secure the SQA’s recovery and also to support additional costs created by the more complex educational environment in which the SQA now operates. Customers and stakeholders have indicated that they are increasingly confident in the SQA’s performance since 2000.

7. The board and executive team have shown strong leadership in driving forward improvements. This has taken the form of ‘hands-on’ management with detailed monitoring and involvement in all stages of activity. There is, however, a risk that concentrating on detailed monitoring can detract from strategic-level thinking. The board has recognised this by agreeing to adopt a more strategic approach from June 2006. The executive team also needs to continue to develop a more strategic approach, particularly in relation to the management of risk.

8. The SQA’s corporate planning process has established clear objectives. There is still work to do in linking performance management information to these objectives. The SQA’s key objectives are backed by corporate and business objectives and they are expressed in the corporate plan. Progress is, however, monitored by management mainly at the level of detailed business objectives, with an emphasis on process.

9. The SQA’s annual report includes 20 performance indicators. As they stand, these indicators do not present a clear picture of performance. The SQA recognises that the current performance management framework needs reviewing and is introducing a balanced scorecard approach to performance management in order to provide a clear and coherent picture of its performance. The development of key performance indicators (KPIs) and the balanced scorecard provides an opportunity to give a clearer picture of performance for both internal and external users.

10. The SQA is developing a deeper understanding of the link between costs and activities and is beginning to benchmark its functions against other organisations. Continued progress in these areas will allow the SQA to explore its scope for efficiency savings and to further improve its understanding of its cost base.

Summary of key recommendations

11. The key recommendations from our study are:

•		The SQA should use its reviews of staffing, qualifications portfolio and business areas to provide a more critical challenge of its products and processes.
•		The board and senior management of SQA should continue to develop a more strategic focus based on the key risks facing the business.
•		The SQA should develop its key performance indicators, linked to key objectives, as part of its development of a balanced scorecard.
•		The SQA should continue to explore the scope for efficiency savings including opportunities presented by the SQA’s modernisation agenda.
•		The SQA should continue to refine its work linking costs with activities, to further improve its understanding of its cost base and to inform the setting of charges for qualifications.
•		The SQA should explore the opportunities for greater use of benchmarking to inform performance management activity.

Part 1. Introduction
The role of the Scottish Qualifications Authority

12. The SQA is the national awarding body for Scotland, responsible for the development, accreditation, assessment and certification of qualifications other than degrees and some professional qualifications. It was established under the Education Act (Scotland) 1996 which was subsequently amended by the Scottish Qualifications Authority Act 2002. This Act outlines the functions and the governance of SQA along with the composition and structure of its board. 

13. The SQA was formed following the merger of two existing bodies, the Scottish Examination Board and the Scottish Vocational Education Council. It is an executive non-departmental public body (NDPB) sponsored by the Scottish Executive Education Department. 

14. The SQA has an annual turnover of around £51 million and a staff of around 650 FTEs. Its qualifications are offered at approximately 1,470 centres, including schools, colleges, employers and private training providers in Scotland and elsewhere. In 2005, the SQA received more than 2.5 million entries from around 500,000 candidates.

15. The SQA’s functions are to: 

•	devise, develop and validate qualifications, and keep them under review
•	accredit qualifications
•	approve education and training establishments as being suitable for entering people for these qualifications
•	arrange for, assist in, and carry out, the assessment of people taking SQA qualifications
•	quality assure education and training establishments which offer SQA qualifications
•	issue certificates to candidates.
Qualifications
16. The SQA is responsible for a large range of academic and vocational qualifications split into three main types: units, courses and group awards.

17. Most Scottish qualifications are covered by a single framework known as the Scottish Credit and Qualifications Framework (SCQF) which was developed by the SQA along with Universities Scotland, the Quality Assurance Agency Scotland and the Scottish Executive. This framework brings together all the qualifications awarded and accredited by SQA and by degree-awarding institutions. The SCQF provides a means for comparing different qualifications by their level and credit points. The level of qualification shows how difficult it is and the credit points show how much learning has to be done to achieve it.

Background to the report

18. One of the SQA’s key functions is to set, deliver and mark the annual set of national examinations known as the Diet. The results of these examinations are issued in August and include Standard Grades and Highers.

19. In 2000, the SQA encountered significant problems with the processing of the summer exam results. As a consequence, 2.7 per cent of results and 17,000 candidates were affected by missing or incomplete data. The following problems contributed to the errors:

•	The late delivery of software for SQA’s new Awards Processing System computer system.
•	Mismanagement of candidate data.
•	Lack of properly trained staff to process data.
•	Failure to recruit sufficient script markers within the required 
time frame.
•	Poor reporting of progress to the management board. 
•	Lack of clarity of the role of the board in dealing with the concerns which were presented to them. 
20. The problems with the Diet in 2000 and subsequent media criticism led to investigations into the management and performance of SQA being conducted by the Scottish Executive (using Deloitte & Touche) and the Scottish Parliament’s Enterprise and Lifelong Learning and Education, Culture and Sport Committees. The reviews identified weaknesses and recommended action across a number of areas including:

•	governance
•	planning
•	staffing
•	communication
•	operational processes
•	exam procedures
•	data management
•	the Awards Processing System.
21. The SQA developed an action plan to address these issues and all subsequent Diets have been completed successfully. 

22. A set of performance measures relating specifically to the Diet has been agreed annually with the Scottish Executive since 2001. These detailed measures cover areas such as the accuracy and timeliness of results. All measures have been met or exceeded in each year since 2001.

The study

23. The overall aim of this study was to assess the extent to which the organisation has recovered from its position in 2000. We considered whether the SQA’s current risk and performance management framework is appropriate, whether the organisation displays the principles of good governance and whether it demonstrates a commitment to best value through continuous improvement.

Key messages

Since 2000, the SQA has been successful in re-establishing customer and stakeholder confidence in its core functions.

The board and executive team have shown strong leadership in driving forward improvements.

24. To set the context for the SQA’s performance, this part of the report looks at:

•	the SQA’s response to the problems of 2000
•	the additional resources received by the SQA since 2000
•	changes in the wider environment within which the SQA operates and the associated funding implications
•	customer and stakeholder confidence in the authority
•	the role of the board and executive team
•	the SQA’s approach to achieving the principles of sound governance, including its approach to risk management.
Response to the problems of 2000

25. Following the problems of 2000, the SQA undertook an extensive mapping of business processes. This has led to a much better understanding of the processes underlying its core functions, which are now linked to individual responsibilities and accountabilities. 

26. The SQA’s approach to managing the annual cycle of activities leading up to the issue of examination certificates in August has significantly improved. Risks are highlighted and managed on an ongoing basis, and critical events and milestones through the year are clearly understood and regularly monitored. An annual certification plan has been prepared in each year since 2000 which sets out key tasks, owners and timescales. This is updated annually following the August certification process to reflect new developments and lessons learned. Weekly meetings of the Certification Planning Group focus on managing this process and involve senior managers. 

27. The SQA has monitored progress against the recommendations made by the Scottish Executive following Diet 2000. Other developments in response to issues highlighted post-2000 include:

•	internal restructuring, including a smaller executive and senior management team and a reduced number of board members
•	improved communications with centres, including the introduction of customer account managers linked to each centre
•	more active stakeholder communication and engagement
•	improved management of those appointed to mark examination papers, including a 50 per cent increase in payments to these markers in 2001.
28. The SQA has gone beyond simple recovery from the problems it faced in the immediate aftermath of 2000. It is now demonstrating a commitment to continuous improvement, as evidenced in a wide range of areas of operation, including regular reviews of staffing, a major review of the portfolio of qualifications and a rolling programme of business area performance reviews. A Best Value review in 2006 by the SQA’s internal auditors concluded that the organisation was able to demonstrate a commitment to best value principles. The auditors cited a number of examples of good practice such as its detailed corporate planning cycle and its use of a quality management framework. An important current area of development for the SQA involves exploring the potential for e-enablement of various areas of activity, for example marking and processing of exam scripts. This has the potential to deliver considerable efficiency savings.

29. Having reached a point where the organisation is confident that core processes will be delivered to schedule and to the standard required, the SQA is now in a position where it can use the review process to more critically challenge its operational processes.

Recommendation 1
The SQA should use its reviews of staffing, qualifications portfolio and business areas to provide a more critical challenge of its products and processes.

Additional resources

30. The SQA is funded by the fees it charges for its qualifications and assessments and by government grants. Since 2000 it has received additional resources to help ensure the successful delivery of its core functions. Exhibit 3 shows the increase in the SQA’s income and its expenditure since 1999/2000. Government funding, including additional grant funding, has increased significantly (by 160 per cent) during this period. 

31. The SQA’s increased expenditure reflects a range of factors. For example, the more diverse range of qualifications delivered by SQA has led to additional costs associated with assessing these qualifications. Current levels of activity could not be sustained without support from government. ‘Additional grant funding’ covers non-core expenditure on activities such as the development of new qualifications or e-enablement of processes. However, a certain level of government support (labelled ‘Government funding’) is also required to support the core activities associated with delivering and assessing qualifications.

32. In 2005/06, the SQA generated £33.8 million in revenue through, for example, examination fees and consultancy assignments. The revenue generated largely covers the SQA’s core functions of assessment and certification. In addition, the SQA received £10.6 million additional grant funding and £6.9 million of other government funding. The additional grant funding is principally to cover ‘development costs’, which include qualifications development. Total government funding reduced from a peak of £23.8 million in 2003/04 to £17.5 million in 2005/06.

33. The SQA has increased income generated through charges for its core work by 50 per cent since 2000. Much of this is accounted for by increases in charges for exams, set in agreement with the Scottish Executive. Between 2002/03 and 2005/06, charges for National Qualifications and Scottish Vocational Qualifications increased by 48 per cent and charges for Higher National Qualifications increased by 20 per cent. This reflects a move towards setting charges at a level that fully recovers the costs of core service delivery.

Changes in the wider environment

34. The SQA’s need for increased resources is not simply a product of its need to recover from the problems it faced in 2000. The educational environment in which the SQA operates has become much more complex since then as a result of new initiatives and developments in education. The SQA now offers a much broader range of qualifications, and a greater number of options is available to candidates to provide a system that recognises a wider range of abilities and skills than in the past.

35. The Skills for Work initiative, to encourage young people to become familiar with the world of work, is a recent example of expansion to the portfolio of qualifications. The SQA is also involved in developing and distributing 5-14 National Tests to schools as part of the Scottish Executive’s 5-14 Programme, and in the development of e-assessment and wider application of new technology throughout its processes. Other potential future developments include more than one opportunity during the year for sitting National Courses, such as Highers, involving a move away from a single annual Diet of examinations. 
 
36. The SQA is also committed to developing an international profile with a vision ‘to be recognised nationally and internationally as a leader in qualifications and assessment’. The SQA’s international strategy is based on three strands of activity:

•	International benchmarking and intelligence.
•	Promoting Scottish education
and training.
•	Commercial activities to support education and training development.
37. The SQA participates in projects and initiatives funded by the UK Government, the European Commission and other agencies. The SQA’s strategy for these and other commercial activities is to ensure that all projects should at least break-even and should aim to make a reasonable return on investment where possible.

38. Despite increased income generation, it is apparent that without additional government funding the SQA could not continue to deliver its current range of services. On the basis of its current workload, the SQA is likely to require even more government funding in future as Scottish Executive forecasts show falling numbers in the school-age population in Scotland over the next few years.1 The SQA is concerned that the gap between its expenditure and the income it generates through charges for exams will widen as a result of this. 
 
39. The SQA’s modernisation agenda offers some opportunities for efficiency savings through, for example, increased application of new technology to existing processes and activities. This is explored further in Part 4.

Customer and stakeholder confidence in the SQA 

40. Customer and stakeholder confidence suffered as a result of the problems of 2000 and the SQA has worked to address this situation through a range of initiatives. Successful delivery of the Diet has obviously been a major factor in restoring confidence, but the SQA has also focused on areas such as communication, openness, responsiveness and accessibility.

41. To monitor the success of these efforts and direct future activities, the SQA has carried out three customer surveys since 2000, reporting findings in 2004, 2005 and 2006. In the 2006 survey, overall customer confidence was rated at 80 per cent, putting the SQA in the second quartile when compared to other educational bodies.2 Customer satisfaction levels improved from 78 per cent in 2004 to 80 per cent in 2006 and are higher for schools than they are for colleges. The areas with which customers expressed most satisfaction were:

•	qualifications
•	effectiveness of account management
•	the SQA’s expertise
•	contact and openness with customers
•	provision of information.
42. The SQA plans to develop its customer surveys through increased frequency and by focusing on key issues identified by its customers. The aim is to link the findings from these surveys into the corporate planning process, helping the SQA to become more directly accountable to its customers.

43. The SQA is developing a stakeholder engagement plan for broader consultation on the SQA’s performance and is proposing to meet with stakeholders formally once a year. We consulted the following bodies as part of our study:

•	Scottish Executive
•	Headteachers Association Scotland
•	Sector Skills Alliance Scotland
•	Careers Scotland
•	Scottish Enterprise
•	Scottish Parent Teacher Council
•	Scottish Funding Council
•	Scottish Council for Independent Schools
•	Educational Institute of Scotland
•	Scottish Further Education Unit
•	Association of Scottish Colleges
•	Association of Directors of Education Scotland
•	HM Inspectorate of Education
•	Highlands & Islands Enterprise 
•	Learning and Teaching Scotland
•	UCAS.
44. In general, stakeholders were positive about the progress that the SQA has made since 2000 and about its ability to deliver its core functions. The SQA’s engagement and relations with its customers, its communications and the robustness of its core business were all cited as strengths by stakeholders. The main concerns expressed were about the SQA’s levels of funding and about its ability to sustain its core business in future without being distracted by commercial activity.

Role of the board and senior management

45. Stakeholders also spoke positively of the role of the SQA’s board and senior management. Following the problems of 2000, the board and executive team developed a very ‘hands-on’ approach to management, with detailed monitoring and involvement in all stages of activity. The improvements in the SQA’s understanding of its key business activities has been led and overseen by senior management. This has allowed the SQA to drive forward change and instil a commitment to continuous improvement. However, it has also meant that the board and executive team have remained actively involved at a detailed level of activity.

46. There is a risk that concentrating on detailed monitoring can detract from strategic level thinking. The executive team meet on a weekly basis and the management team hold monthly review meetings which are supported by detailed information on all aspects of the business. While summaries are provided, which use a priority system to help users focus on areas of concern, the information would be more transparent and useable if it was reported on an exception basis. In addition, a wide range of committees and groups meet on a regular basis to address specific areas of business, such as qualifications management and commercial activity. There are a number of regular meetings that senior management routinely attend, ensuring that core business is tightly managed. However, as these groups and committees typically involve senior level staff, there is some overlap and duplication of issues considered. 
 
47. The board regularly receives updates from business group areas or committees that focus on particular aspects of the organisation. A review of board effectiveness was reported to the board in June 2006. The paper argued that, of necessity, the board had adopted a hands-on approach post 2000 but that it could now safely devote more time to strategic issues and less time to scrutinising operational matters. The paper was supported by the board and argued that:

•	decision-making powers needed to be reviewed both for the board and the executive team
•	agenda items should be scrutinised before meetings to identify matters for noting rather than discussion
•	committee chairs should consider issues that would warrant extended discussion by the board
•	consideration should be given as to how the balanced scorecard and risk register can be used to put more emphasis on challenges and strategy
•	a review of the patterns of meetings and agendas for 2007 should include a different approach to the board’s engagement with the
corporate plan.
48. This proposed approach is consistent with the principles of best value and good governance and should promote greater board effectiveness. It will require streamlining of the information presented to the board and a greater focus on strategic risk assessment. This will allow the board to develop even further the SQA’s commitment to continuous improvement and its approach to best value. The executive team should continue to develop a more strategic approach now that systems and processes are well developed and functioning effectively. This will involve a greater concentration on key risks and issues, rather than regular review of such an extensive array of information. 

49. Although much has been achieved, the latest staff survey suggests that there remains scope for further improvement in the role of management. In the 2005/06 staff survey, only 37 per cent of staff felt that senior management provided staff with a clear sense of direction, reinforcing the need for a more strategic approach by the senior management team.

The SQA’s approach to good governance and the management of risk

50. In 2005, the Independent Commission on Good Governance in Public Services published The Good Governance Standard for Public Services. The standard detailed six core principles of good governance.

51. The SQA has made significant progress as an organisation over the last six years, partly in response to the problems it faced in 2000, but also through an ongoing commitment to continuous improvement. The principles of good governance are evident within the organisation and where there are areas for further development, there is a commitment and willingness to work to make the required improvements. For example, the SQA has worked hard to engage with customers and stakeholders since 2000 and it is committed to developing even more effective and meaningful consultation.

52. However, a key area that the SQA could develop further is that of the management of risk. The SQA’s risks are managed through a risk register and there are regular meetings across all areas of activity to ensure comprehensive coverage of issues. The SQA continues to develop its approach to risk management through the development of a risk management policy statement, a risk management framework and risk management guidelines. At present, the processes in place highlight risks, but there needs to be greater focus on prioritising risks and developing responses to address the key risk areas. Development of a more strategic approach to risk management will be important in supporting the board in its more strategic approach to management.

Recommendation 2
The board and senior management of SQA should continue to develop a more strategic focus based on the key risks facing the business.

1. Results of Teacher Workforce Planning for 2005/06 Scottish Executive National Statistics
2. The comparative bodies include other UK examinations authorities, accountancy bodies and other examinations providers such as City & Guilds.

Part 3. The SQA’s performance management framework
Key messages

The SQA’s corporate planning process has established clear objectives, but performance management information is not clearly linked to these objectives at present.

53. In order to assess whether the SQA is operating effectively, it is important to have access to appropriate performance information, which in turn depends on the robustness of its performance management systems. Effective performance management relies on high-quality management information and the effective use of this information within the framework of a well developed and coherent corporate plan. This section examines:

•	the effectiveness of the SQA’s performance indicators and measures
•	how well developed the SQA’s corporate planning processes are
•	whether the SQA’s performance management framework is fit for purpose.
How effective are the SQA’s performance indicators and measures? 

54. The SQA’s annual report includes 20 performance indicators. On the basis of those indicators, the SQA’s performance has generally improved since 2001/02.

55. The first indicator summarises a subset of indicators linked specifically to the delivery of the summer Diet of examinations, which were developed and agreed with the Scottish Executive post-2000. 

56. Exhibit 7 shows how the objectives and indicators have been linked in the 2005/06 annual report. Although most of the indicators can be linked to a particular objective, many of them do not present a clear picture of performance in a given area of activity. The indicators tend to relate to activities rather than outcomes or impacts. For example, the objective to inform and support national policy development and implementation (Objective 2) is only supported by indicators that refer to satisfaction ratings for partners and the development of strategic arrangements with partners. These indicators do not provide a sense of what the SQA is seeking to achieve or the extent to which it is meeting this objective.

Are corporate planning processes well developed?

57. The SQA has developed a sound corporate planning process which has supported the development of clear objectives for the organisation. The planning cycle is now an integrated process bringing together the corporate plan, the business plan and resource planning. The corporate plan is a rolling three-year plan, the latest of which covers the period 2006-09 and is combined with the SQA’s business plan for 2006/07. The corporate plan sets out the vision, mission and values of the SQA, and its five key objectives. The five key objectives in achieving its vision to be recognised nationally and internationally as a leader in qualifications and assessment are to:

•	provide qualifications and support to match the needs of individuals, society and the economy of Scotland
•	inform and support national policy development and implementation
•	redesign our services and processes to make them better to use and more efficient
•	develop a highly skilled, motivated and effective workforce
•	increase the use of SQA qualifications and services nationally and internationally.
58. The annual business plan provides detailed objectives to support the key objectives. The key objectives are backed by 19 corporate objectives and 73 business objectives which are allocated to named employees with an indicative timescale for their achievement. Appendix 1 (page 23) sets out
these objectives.

59. Stakeholders are consulted during the development of the corporate plan and 86 per cent of staff stated they understood how their work contributed to SQA’s strategies and goals. There is now a clear timescale for development of the plan. This integrated planning process worked effectively in 2005, with the corporate plan for 2006-09 presented to the Scottish Executive for approval on schedule in December 2005.

Is the SQA’s performance management framework fit for purpose?

60. The SQA operationally manages its performance by producing monthly reports to the executive team and high-level quarterly reports to the board and Scottish Executive. The reports to the executive team are very detailed and concentrate on the 73 business objectives detailed in the SQA annual business plan.

61. This example highlights how performance management within the SQA relies on detailed monitoring of progress against a large number of very specific business activities. While such monitoring has its place, there is also a need for a higher-level picture of progress towards key objectives and a means for assessing the impact that the organization is achieving.

62. The emphasis of the business objectives against which performance is reported is largely on the processes of the SQA rather than its outputs and the impact of its work. However, there is a risk that providing such a large volume of operational information to management on a regular basis can detract from strategic level thinking. In reviewing detailed activity-related information, management and the board spend a significant amount of time on operational management and are therefore less able to devote time to strategic development and risk management. The framework is comprehensive and appropriate for use within individual teams and as a basis for exception reporting to management. However, the performance management framework needs to offer a higher-level picture of performance against key objectives to give management and external stakeholders an understanding of overall achievement. 

63. Performance indicators should be specific, measurable, achievable, realistic and time bound. They should also contribute to the measurement of continuous improvement and should, ideally, be suited to benchmarking with other organisations. The SQA recognises that the current set of indicators is not fit for purpose and is in the process of reviewing them as part of its work in developing a balanced scorecard approach to performance management.

64. The SQA is demonstrating a commitment to continuous improvement. It is important therefore that its performance management framework is adequate to support that commitment and to be able to demonstrate that the organisation is achieving its aims. The introduction of a balanced scorecard offers an opportunity for the SQA to develop its performance management framework to provide information of this nature (see Part 4).

3. The current definition of self-financing used by the SQA refers to ensuring that the costs of its core services are met by charges and other income.

Part 4. The SQA’s use of 
performance information
Key messages

The SQA is introducing a balanced scorecard approach to performance management in order to provide a clear and coherent picture of its performance.

The SQA is developing a deeper understanding of the link between costs and activities and is beginning to benchmark its functions against other organisations.

65. Having examined and assessed the robustness of the SQA’s performance management systems, this part of the report assesses how well the SQA uses performance information. It is important that the SQA is able to demonstrate its performance to internal and external stakeholders and the development of a balanced scorecard approach will help it to do so. Within this framework, benchmarking is a valuable tool and can help in providing assurance to management and stakeholders that the SQA is operating efficiently and effectively. Information to support benchmarking needs to be backed up by robust costing of the SQA’s activities.

66. This section of the report examines:

•	the SQA’s development of a balanced scorecard approach
•	the SQA’s approach to the analysis of its costs and activities
•	the SQA’s approach to the development of benchmarking information.
The SQA’s development of a balanced scorecard approach

67. To present a balanced view of an organisation’s performance, information reflecting a range of perspectives needs to be considered. In 1998 the Accounts Commission published a discussion paper setting out an approach to the development of the balanced scorecard approach. It described the overall purpose of the scorecard approach being ‘to enable managers to develop a robust set of performance measures that provides a comprehensive view of the overall performance of the organisation but that is also visibly linked to the key strategies and priorities of the organisation.’4

68. The SQA needs to provide a high-level picture of performance which internal and external stakeholders can use to assess performance. The SQA plans to introduce a balanced scorecard approach to performance management in order to provide a clear and cohesive performance measurement framework. The aim is for the scorecard to:

•	support the organisation’s objectives
•	provide intelligence for decisions
•	effectively communicate SQA’s performance to internal and external stakeholders.
69. The executive team plans to present the balanced scorecard to the board in November 2006. The process gives the SQA the opportunity to focus on key measures of impact, linked to its five key objectives, rather than the more process-driven indicators that form the focus for performance reporting at present. The development of key performance indicators (KPIs) and the balanced scorecard provides an opportunity to give a clearer picture of performance for both internal and external users.

70. In formulating its approach to developing a scorecard, the SQA asked Audit Scotland to comment on the approach being taken. The key issues for the SQA in developing their approach are a need for:

•	clarity between:
1.	Objectives – which should describe the desired outcome 
eg, develop the right qualifications.
2.	Measures – which should set out a quantifiable means for assessing progress towards the objective eg, uptake of qualifications.
3.	Targets – which define the desired level for the measure eg, x per cent of relevant age group entered for SQA qualifications.
•	development of a concise set of clearly defined and quantifiable measures, each of which has a clear purpose.
71. The process needs to start from the top down to ensure the linkages are clear. The high-level KPIs should be capable of providing a concise picture of performance to both internal and external audiences.

72. Further details of the advice provided are reproduced in Appendix 2. 

Recommendation 3
The SQA should develop its key performance indicators, linked to key objectives, as part of its development of a balanced scorecard. 

The SQA’s approach to the analysis of costs and activities 

73. In the past two years the SQA has made progress in linking costs with activities and income generated (for example, through fees for examinations). This is giving the SQA a clearer understanding of the financial impact of its work and helping it to align plans and targets with available resources.

74. The continued development and refinement of this work is particularly important in developing a set of key performance indicators that can be used in performance management. The SQA is introducing a new financial system which will help its approach to this work.

75. The SQA should continue to adopt a more strategic approach, and explore systematically the scope for efficiency savings. The SQA’s modernisation agenda will highlight some opportunities for savings and the organisation is already working towards generating efficiency savings through e-learning and e-assessment. However, many of these will require upfront investment in order to realise future savings.

Recommendation 4
The SQA should continue to explore the scope for efficiency savings including opportunities presented by the SQA’s modernisation agenda. 

Recommendation 5
The SQA should continue to refine its work linking costs with activities, to further improve its understanding of its cost base and to inform the setting of charges for qualifications.

The development of benchmarking information

76. One of the main ways in which an organisation can demonstrate its commitment to continuous improvement is by searching for examples of better practice in other organisations that can be integrated into its own service delivery. An Accounts Commission publication5 summarises benchmarking as follows:

‘Benchmarking is best thought of as a structured and focused approach to comparing with others how you provide services and the performance levels you have achieved. The purpose of such comparison is to enable you to identify where and how you can do better. It is about understanding why there are differences in performance between organisations and this involves looking in detail at the way services are delivered and managed and at the processes and activities involved in service delivery that lie behind the benchmark measures of performance.’

77. The document also gives details on a number of different approaches to benchmarking and describes a four-stage process as follows:

Planning
•	Decide what you want to benchmark and why.
•	Decide what type of benchmark you are going to start with.
•	Decide who is going to be involved.
Analysis
•	Understand your own performance.
•	Undertake data benchmarking.
•	Assess key performance gaps.
•	Understand your own processes.
•	Choose your benchmarking partners.
•	Complete benchmarking.
Action
•	Decide what you want to improve and how you are going to do it.
•	Implement the improvement plan.
Review
•	Monitor progress and achievements.
•	Decide what comes next.
78. To date, the SQA has undertaken limited benchmarking of its performance with the Qualifications and Curriculum Authority in England, although it proposes to extend this activity. The information obtained should be incorporated in the balanced scorecard.

79. As part of this study, Audit Scotland carried out a high-level benchmarking exercise with a number of other organisations that are concerned with the provision of qualifications in the United Kingdom. The object of the exercise was to:

•	establish the extent of comparative information available for benchmarking purposes
•	assess the degree to which other qualifications authorities are engaged in, or would be willing to engage in, benchmarking
•	where possible, produce benchmarking information.
80. Our exercise initially gathered information readily available from public sources such as organisational websites and published documents. We met with two bodies; Edexcel and the Council for the Curriculum, Examinations and Assessment (CCEA), to discuss the availability and comparability of benchmarking information. We also consulted with the Qualifications and Curriculum Authority (QCA). 

81. Our exercise identified that there are complexities in benchmarking the SQA with other UK organisations as roles and responsibilities vary considerably. However, the exercise did highlight potential areas for further investigation, such as unit costing for types of examinations. We have derived high-level unit costs as a starting point for further work on benchmarking but further discussions will need to be held between SQA and other bodies to arrive at meaningful information. Further results of the exercise are presented at Appendix 3. Our key findings were:

•	There are fundamental differences in the ways in which qualifications to degree level are administered in the different parts of the UK, which limits the opportunities for benchmarking.
•	The ability to compare information is made more complex by the confidentiality requirements of private sector providers Edexcel, the Assessment and Qualifications Alliance (AQA) and OCR, which operate largely in England.
•	There was, however, willingness expressed by the bodies we spoke to as part of this exercise (CCEA and Edexcel) to explore further opportunities for benchmarking costs and processes with the SQA.
•	Some detailed information was obtained to allow limited comparisons for some areas of activity, such as unit costing for types of examination. Further work would have to be undertaken to refine costing information if a similar approach was to be extended to other business activities. Information on specific costs for private sector providers is commercially sensitive but Edexcel has indicated its willingness to discuss activity costs with SQA on a confidential basis.
•	Audit Scotland will share the detailed information it has obtained during this exercise with the SQA. Audit Scotland will follow up any work done on benchmarking in due course.

Recommendation 6
The SQA should explore the opportunities for greater use of benchmarking to inform performance management activity.

Next steps

82. The SQA has made good progress in its recovery from the problems it faced in 2000. It now has a better understanding of its key business activities and it is demonstrating an ongoing commitment to continuous improvement.

83. Much of this success has been driven by the board and executive team, and customers and stakeholders have indicated that they are increasingly confident with the SQA’s performance. The SQA now has clear objectives generated through a robust corporate planning process.

84. There is still room for further improvement. The ‘hands-on’ management of the board and executive team can present a risk by detracting from strategic level thinking, which the board has recognised. The SQA’s performance information needs to be more clearly linked to its key objectives and it is working to introduce a balanced scorecard approach to performance management. This should provide a clearer and more coherent picture of its performance. It is also working on establishing a greater understanding of the links between its activities and its costs. Progress with this will allow the SQA to effectively benchmark its costs and activities with other bodies.

85. Our recommendations reflect this position and support the SQA in its efforts towards further improvement.

4. The measures of success – Developing a balanced scorecard to measure performance, Accounts Commission, 1998.
5. Measuring up to the best – A manager’s guide to benchmarking, Accounts Commission, 1999.6. Since these comments were provided, the SQA has revised the model to provide four perspectives.

Appendix 2. Advice on the development of a balanced scorecard approach
The notes below are summarised thoughts from the Audit Scotland team involved in the SQA performance management study. They were intended as an overview of key points from our review of the material available in relation to the SQA’s balanced scorecard (BSC) development. The notes were provided to the SQA in July 2006 and were used to help inform the SQA’s ongoing work in this area. It should be noted that the SQA has continued to develop its approach since this review was undertaken.

Existing performance management framework

At present, the focus of performance management is on business objectives (as set out in the business plan). Progress towards achievement of these is monitored on a monthly basis at monthly management team meetings and also reported to the board.

The 20 KPIs developed post-2000 relate to a range of areas of activity and there is an additional set of indicators relating specifically to the Diet results – performance against these is published in the annual report, but they are not used for regular monitoring purposes aside from this. 

The introduction of a BSC offers an opportunity to define a more appropriate set of KPIs that: 

•	are clearly linked to key objectives
•	provide a clear picture of progress and impact
•	can present a summary of performance to inform both internal and external audiences.
Corporate plan/business plan

The current practice of combining these two documents into one has merits in providing linkage between objectives and activities, but there is a danger that objectives become too activity-focused, rather than focusing on desired outcomes.

The current document has three levels of objectives:

•	Key objectives (5) – these provide a good balance, reflect the nature of the organisation, and address both internal/external perspectives. They provide a helpful starting point for development of the BSC.
•	Corporate objectives (19) – this is a reasonable number of objectives against which to develop KPIs, but the current set includes several which are outputs rather than objectives (eg, implement an integrated HR and financial system).
•	Business objectives (73) – in the latest corporate plan, these describe activities and/or outputs required eg, implement benchmarking strategy rather than desired outcomes. Ownership for each objective is identified and timescales are set out. The number of business objectives is too large to be a manageable set for managing performance at a corporate level. These activities/outputs are appropriate for team-level management.
In the business plan, it would be helpful to have a clearer distinction between objectives, outputs and activities – using the term ‘objective’ at all levels is potentially confusing.

Balanced scorecard development

In introducing the BSC, it is important that there is clear linkage with the SQA’s corporate plan, so as to avoid creating an extra layer of objectives. The five key objectives set out in the latest corporate plan should be the starting point and the BSC model should flow from this.

The current model uses three perspectives – qualifications, working with others, management – rather than the traditional four.6 This is fine in principle, but it is not clear how this links to the five key objectives in the corporate plan. Similarly, the objectives set out within these three categories are different from those in the corporate plan. To ensure transparency and clarity, the objectives should be consistent throughout.

The model as described in the SQA’s implementation progress summary does not refer to impact or continuous improvement – these are likely to be captured within ‘qualifications’ and ‘management’ respectively, but it would be helpful to make these important considerations more explicit in
the objectives.

In the current model, there is not a clear enough distinction between objectives, measures and targets:

•	Objectives – should describe the desired outcome eg, develop the right qualifications.
•	Measures – should set out a quantifiable means for assessing progress towards the objective eg, uptake of qualifications.
•	Targets – defines the desired level for the measure, eg x per cent of relevant age group entered for SQA qualifications.
The measures described in the latest version of the model are often objectives rather than measures. Measures should describe the quantifiable factor that is being used to assess progress. For example, ‘Improve commercial focus, capability and profitability’ is not a measure, this is a desired outcome. A measure might be ‘Improve profitability of commercial contracts’ (which currently appears as a target). Meanwhile the target might be ‘five per cent increase in average profitability of commercial contracts’.

The measures and associated targets need to be challenging and focus on quality. Measures should also be clearly defined and unambiguous. Where possible, targets should avoid simple yes/no responses as this gives no basis for assessing the scale of achievement. For example:

•	Credibility rating for qualifications – why should Standard Grades have a lower target for credibility than Highers? How is credibility defined (and if this is based on survey results, will ‘credibility’ mean the same thing to everyone?).
•	Research output – number of bulletins produced does not capture quality, better to identify a measure that indicates the value of these bulletins to users.
•	Become self-financing – this is a milestone; a better measure might be non-government income as a percentage of expenditure, with an associated target of 100 per cent.
Ideally, the BSC approach should be developed from the top down, not bottom up. It should start from the corporate plan and work downwards, but always showing clear linkage from one level to the next. There is a risk that, in starting at individual team level, there will be gaps, overlap or inconsistencies at higher levels.

The Corporate Strategy and Performance Team (CSPT) model is very detailed – it may not be appropriate for each team to have a performance measure against every single objective, as different teams will make contributions in different areas. The main requirement is that there is coverage of all objectives at organisational level, rather than team level (this should be guaranteed if a top-down approach is used).

The focus should be on defining measures that are informative and valuable in managing performance, not measuring everything that is done by a particular team or the organisation as a whole. The CSPT model is heavily focused on measuring activity, but does not give much indication of the quality of outputs. For example, measuring the number of staff reviews conducted says nothing about the quality of these reviews – a better measure might relate to staff satisfaction with the review system.

There is a lot of information in the team level model – for each measure, it is important to think about the value of data, ie how will the information be used? Who needs to know? What action might result from the information gathered?

Summary of key points

•		Make sure there is a clear linkage through from corporate objectives to measures and targets.
•		Work from top down to help ensure this linkage.
•		Make sure that measures are clearly defined and quantifiable.
•		Think about desired impact and outcomes when defining measures.
•		Avoid measuring activity, instead aim to capture the quality of outputs.
•		Focus on a concise set of indicators and avoid duplication.
•		Each indicator should have a clear purpose.
•		At top level, the KPIs should give a balanced picture of performance and provide a useful summary for both internal and external audiences.
•		More detailed sets of KPIs may be appropriate for individual teams, but should still aim to be concise and manageable.

If you require this publication in an alternative format and/or language, please contact us to discuss your needs. It is also available on our website: www.audit-scotland.gov.uk

Audit Scotland 
110 George Street 
Edinburgh EH2 4LH 

Telephone 
0845 146 1010
Fax 
0845 146 1009
 
www.audit-scotland.gov.uk 

ISBN 1 905634 36 6
AGS/2006/11