Performance management in the Scottish Qualifications Authority

Key messages/Prepared for the Auditor General for Scotland
November 2006

Auditor General for Scotland
The Auditor General for Scotland is the Parliament’s watchdog for ensuring propriety and value for money in the spending of public funds.

He is responsible for investigating whether public spending bodies achieve the best possible value for money and adhere to the highest standards of financial management.

He is independent and not subject to the control of any member of the Scottish Executive or the Parliament.

The Auditor General is responsible for securing the audit of the Scottish Executive and most other public sector bodies except local authorities and fire and police boards.

The following bodies fall within the remit of the Auditor General:

•	departments of the Scottish Executive eg, the Health Department
•	executive agencies eg, the Prison Service, Historic Scotland
•	NHS boards
•	further education colleges
•	Scottish Water
•	NDPBs and others eg, Scottish Enterprise.

Introduction

1. The Scottish Qualifications Authority (SQA) is the national awarding body for Scotland and its main functions are set out in the Education (Scotland) Act 1996. SQA was formed in 1997, following the merger of the Scottish Examination Board and the Scottish Vocational Education Council. 

2. It is a non-departmental public body (NDPB) sponsored by the Education Department of the Scottish Executive and has responsibility for the development, accreditation, assessment and certification of qualifications in Scotland (other than degrees and some professional qualifications).

3. In 2005, the SQA received more than 2.5 million entries from around 500,000 candidates. Qualifications awarded ranged from National Units to higher education qualifications. Around 1,470 centres are approved to offer SQA qualifications, including schools, colleges, employers and private training providers in Scotland and elsewhere. 

4. Since it was established, the environment in which the SQA operates has become more complex. A much wider range of qualifications is now on offer and the organisation has developed new areas of business through the expansion of its commercial and international interests. 

5. The SQA’s expenditure has risen by 69 per cent since 1999/2000 and this has required additional government funding. The increased expenditure reflects a range of factors. For example, the more diverse range of qualifications delivered by SQA has led to additional costs associated with assessing these qualifications. 

6. In 2005/06, the SQA generated £33.8 million in revenue through, for example, examination fees and consultancy assignments. The revenue generated largely covers the SQA’s core functions of assessment and certification. In addition, the SQA received £10.6 million additional grant funding and £6.9 million of other government funding. The additional grant funding is principally to cover ‘development costs’, which include qualifications development. Total government funding reduced from a peak of £23.8 million in 2003/04 to £17.5 million in 2005/06. Over the same period, income generated has increased by £6.7 million.

The study

7. One of the SQA’s key functions is to set, deliver and mark the annual set of national examinations known as the Diet. The results of these examinations are issued in August and include Standard Grades and Highers. 

8. In 2000, the SQA encountered significant problems with processing the summer exam results, leading to inaccuracy and incompleteness in 2.7 per cent of the results. A total of 17,000 candidates were affected.

9. Several investigative reports recommended action across a number of areas, including: governance, planning, staffing, communication, operational processes, examination procedures, data management and the Awards Processing System. The SQA developed an action plan to address these issues and all subsequent Diets have been completed successfully. 

10. A set of performance measures relating specifically to the Diet has been agreed annually with the Scottish Executive since 2001. These detailed measures cover areas such as the accuracy and timeliness of results. All measures have been met or exceeded in each year since 2001.

11. The overall aim of this study was to assess the extent to which the organisation has recovered from its position in 2000. We considered whether the SQA’s current risk and performance management framework is appropriate, whether the organisation displays the principles of good governance and whether it demonstrates a commitment to best value through continuous improvement.

Since 2000, the SQA has been successful in re-establishing customer and stakeholder confidence in its core functions

12. The SQA mapped and documented its key business activities in response to the problems it encountered in 2000. This has led to a much better understanding of the processes underlying its core functions, which are now linked to individual responsibilities and accountabilities. 

13. The SQA has received additional resources since 2000 to help ensure the successful delivery of its wider range of core functions. Between 2002/03 and 2005/06, charges for National Qualifications and Scottish Vocational Qualifications increased by 48 per cent and charges for Higher National Qualifications increased by 20 per cent, in agreement with the Scottish Executive. This reflects a move towards setting charges at a level that fully recovers the costs of core service delivery.

14. Customer surveys and the views of stakeholders indicate that the organisation has become more responsive and customer-focused. The SQA’s 2006 survey of its customers gave an overall satisfaction index of 80 per cent, putting it in the second quartile when compared to other educational bodies.1 Respondents to the survey also gave the SQA credit for its openness and its communications.

15. The SQA has demonstrated its commitment to continuous improvement by introducing reviews of staffing, its portfolio of qualifications and the performance of its business areas. These reviews can form the platform for a more critical challenge of the SQA’s operational processes.

The board and executive team have shown commitment and leadership in driving forward improvements

16. Stakeholder confidence in the SQA’s work has improved markedly since 2000 and the SQA’s board and senior management are highly regarded by external stakeholders.

17. The SQA’s senior management has taken a ‘hands-on’ approach to managing the organisation’s core functions in order to achieve the necessary improvements. The board has closely monitored that work. Risks are managed through a risk register and there are regular meetings across all areas of activity to ensure comprehensive coverage of issues.

18. There is, however, a risk that concentrating on detailed monitoring can detract from strategic level thinking. The board has recognised this and in June 2006 agreed to adopt a more strategic approach, supporting the principles of good governance. This will require streamlining of the information presented to the board and a greater focus on strategic risk assessment. 

19. The executive team also needs to continue to develop a more strategic approach now that systems and processes are well developed and functioning effectively. This will involve a greater concentration on key risks and issues.

20. Although much has been achieved, the latest staff survey suggests that there remains scope for further improvement in the role of management. In the 2005/06 staff survey, only 37 per cent of staff felt that senior management provided staff with a clear sense of direction, reinforcing the need for a more strategic approach by the senior management team.

The SQA’s corporate planning process has established clear objectives, but performance management information is not clearly linked to these objectives at present.

21. The SQA produces a rolling three-year corporate plan which is combined with its annual business plan. This document, approved annually by Ministers, clearly sets out the SQA’s vision, mission, values and key objectives. 

22. Five key objectives are backed by 19 corporate objectives and 73 business objectives. 

23. The SQA now involves its stakeholders in the development of its corporate plans and 86 per cent of staff stated they understood how their work contributed to SQA’s strategies and goals. 

24. Performance against objectives is reported monthly to the executive team and quarterly to the board and the Scottish Executive. Progress is, however, monitored by management mainly at the level of detailed business objectives, with an emphasis on process. 

25. The annual report includes performance indicators. On the basis of those indicators the SQA’s performance has generally improved since 2001/02. Exhibit 3 (overleaf) shows the SQA’s progress with each of its current performance indicators.

26. The SQA recognises that the current set of performance indicators is not fit for purpose. The indicators are not used as active tools in managing performance and do not link clearly to objectives. They focus more on the organisation’s processes than on its impact.

27. The SQA needs to provide a high-level picture of performance which internal and external stakeholders can use to assess performance. 

28. The SQA is reviewing its performance indicators as part of its development of a balanced scorecard approach to performance management. The scorecard format will be presented to the board in November 2006.

29. The introduction of a balanced scorecard gives the SQA the opportunity to focus on key measures of impact, linked to its five key objectives, rather than more process-driven indicators. This will provide a clearer picture of performance for both internal and external users.

The SQA is developing a deeper understanding of the link between costs and activities and is beginning to benchmark its functions against other organisations

30. In the past two years the SQA has made progress in linking costs with activities and income generated (for example, through fees for examinations). This is giving the SQA a clearer understanding of the financial impact of its work and helping it to align plans and targets with available resources. 

31. The continued development and refinement of this work is particularly important in developing a set of key performance indicators that can be used in performance management.

32. The SQA should continue to adopt a more strategic approach, and explore systematically the scope for efficiency savings. The SQA’s modernisation agenda will highlight some opportunities for savings and the organisation is already working towards generating efficiency savings through e-learning and assessment.

33. The SQA has started to benchmark its functions against other comparable organisations and the information obtained should be incorporated in the balanced scorecard. 

34. Audit Scotland has also undertaken a benchmarking exercise comparing key aspects of the SQA’s operations and activities with other organisations concerned with the provision of qualifications in the UK.
 
35. There are complexities in benchmarking the SQA with other UK organisations as roles and responsibilities vary considerably. However, the exercise has highlighted potential areas for further investigation such as unit costing for types of examinations. It has also identified willingness on the part of all organisations involved to pursue this further.   
 
Key recommendations

•		The SQA should use its reviews of staffing, qualifications portfolio and business areas to provide a more critical challenge of its products and processes.
•		The board and senior management of SQA should continue to develop a more strategic focus based on the key risks facing the business. 
•		The SQA should develop its key performance indicators, linked to key objectives, as part of its development of a balanced scorecard. 
•		The SQA should continue to explore the scope for efficiency savings including opportunities presented by the SQA’s modernisation agenda.
•		The SQA should continue to refine its work linking costs with activities, to further improve its understanding of its cost base and to inform the setting of charges for qualifications.
•		The SQA should explore the opportunities for greater use of benchmarking to 
inform performance management activity.
1. The comparative bodies include other UK examinations authorities, accountancy bodies and other examinations providers such as City & Guilds.

If you require this publication in an alternative format and/or language, please contact us to discuss your needs. It is also available on our website: www.audit-scotland.gov.uk

Audit Scotland 
110 George Street 
Edinburgh EH2 4LH 

Telephone 
0845 146 1010
Fax 
0845 146 1009
 
www.audit-scotland.gov.uk 

ISBN 1 905634 37 4
AGS/2006/11