Managing ICT contracts in central government: An update
Prepared by Audit Scotland
June 2015

Summary
Key messages
1.	Information and Communication Technology (ICT) underpins the operation of every public sector body and offers the opportunity to transform the way public services are delivered. It is critical that investment in ICT is managed effectively. Our report Managing ICT contracts [PDF]  in August 2012 made a number of recommendations for the Scottish Government to improve its strategic oversight and support of central government ICT programmes. The Scottish Government introduced new assurance and oversight arrangements, but these arrangements have not been fully effective. The assurance framework lacked clarity and the Information Systems Investment Board was limited in its ability to provide effective oversight because it did not have sufficient capacity or the necessary information. The Scottish Government is now taking action to further improve arrangements. 
2.	Public bodies still encounter difficulties in identifying, recruiting and retaining staff with the range of skills required to manage ICT programmes. The Scottish Government did not perform a skills gap survey across the public sector until August 2014. It recognises this is a significant recurring problem for bodies and is developing a new approach that pools and shares resources in a central government Digital Transformation Service. Governance and operational arrangements for this Service are yet to be finalised. This is an ambitious project that requires strong leadership and the right skills to ensure that it can quickly address the skills shortage facing the sector. 
3.	Some central government bodies can demonstrate good processes for managing aspects of their ICT programmes. This includes identifying benefits at the start of a programme and investing in contract management. Other bodies did not have processes in place to clearly define benefits, collect baseline information from the start of the programme, or invest in contract and supplier management throughout the project lifecycle. 
4.	Central government bodies are using appropriate programme management techniques in line with the framework. However, some bodies used Agile to manage aspects of their ICT programmes without the necessary skills and experience in place at the start of the programme. These bodies have since trained their staff in this technique. 
Introduction
1.	Information and Communication Technology (ICT) underpins the operation of every public sector organisation and many of the services that they deliver. It provides the opportunity to transform the way services are delivered, to provide better services to the public, reduce costs and meet future demands. The difficulties in managing ICT programmes in both the public and private sector are well documented. It is critical that this ICT investment is properly managed to ensure that the public and users get the full benefits of the investment.
2.	The Scottish Government published its digital strategy in September 2012.1 This set out its vision to use digital technologies to deliver integrated public services innovatively by working across organisations to meet users’ needs. The public sector is expected to deliver all services online where feasible. 
3.	ICT services in the public sector are provided by in-house teams, external providers, or a combination of both. In 2013/14, the Scottish public sector spent £739 million on suppliers of ICT-related goods and services of which at least £153 million was central government spend (Exhibit 1). This figure does not include the cost of in-house service provision; it is based on 21 central government bodies that account for 60 per cent of the total central government budget. 
4.	Central government spend on ICT-related goods and services has increased by 15 per cent between 2011/12 and 2013/14, reflecting the move to digital public services. Spend on ICT programmes, whether delivered in-house or by contractors, is likely to increase as public bodies develop and align their digital strategies to that of the Scottish Government and seek to deliver public services in a more efficient and coordinated way. 
Background
5.	In August 2012, the Auditor General for Scotland published a report focusing on three central government ICT projects that had encountered difficulties.2 The three bodies were: Registers of Scotland, Crown Office and Procurator Fiscal Service (COPFS), and Disclosure Scotland. The report looked at the factors that contributed to the problems encountered and whether any lessons could be learned for public bodies undertaking ICT programmes in the future.
6.	The report also found that a key factor in the failure to deliver was a lack of specialist skills and experience. It recommended that the Scottish Government should improve its oversight of individual ICT programmes.
•	The report found that there were significant weaknesses in the management of the three programmes, which contributed to their failure to deliver. For example:
•	Variable quality of business cases and lack of defined benefits.
•	Governance structures were not always clear and followed.
•	Failings in financial control and progress reporting.
•	Findings of independent assurance reviews were not acted on.
7.	The report also found that a key factor in the failure to deliver was a lack of specialist skills and experience. It recommended that the Scottish Government should improve its oversight of individual ICT programmes.
8.	The publication of the Scottish Government’s digital strategy, Scotland’s Digital Future: Delivery of Public Services, in September 2012 provided a platform for implementing the recommendations of our August 2012 report across the public sector. This is an overarching strategy to which each sector (central government, NHS, and local government) have aligned their digital strategies. The Digital Public Services Strategy Assurance Board is responsible for the Scottish Government’s digital strategy and is supported by sectoral boards that oversee their respective strategies (Exhibit 2). The Strategic Corporate Services Board (SCSB) provides governance and oversight of the central government digital strategy and is supported in its role by the Information Systems Investment Board (ISIB). 
9.	Our 2012 report made a number of recommendations for the Scottish Government and central government bodies to consider. Exhibit 3 (page 8) summarises what actions the Scottish Government and central government bodies took in response to our recommendations. A timeline is provided to demonstrate when specific actions were taken.
About this report
10.	This report reviews the progress that the Scottish Government and central government bodies have made against the recommendations in our previous report on managing ICT contracts.3 We selected ICT programmes in 12 central government bodies (Appendix 1, page 37) for review to establish to what extent lessons have been learned since August 2012. 
11.	The auditors of the 12 central government bodies used the scrutiny checklist from the August 2012 report to gather information on specific ICT programmes between June 2014 and November 2014. Key documents including independent assurance reports, such as Gateway Review reports, and internal audit reports were reviewed to inform this process. Auditors selected the questions to ask each public body based on the types of risks identified and the stage of the project. The information gathered provided a snapshot of the ICT programme at a point in time.
12.	We have revisited the three organisations featured in our previous report – Registers of Scotland, COPFS, and Disclosure Scotland – to consider what actions they have taken to improve how they manage ICT programmes. Appendix 2 (page 40) summarises what has happened at these bodies since August 2012. We have also drawn upon Audit Scotland reports on specific ICT programmes across the public sector in Scotland since August 2012. 
13.	The report provides a high-level summary of the main themes arising from the local audit work and uses a selection of case studies to highlight issues that central government bodies have encountered and how they have managed them. In doing so, we hope to share examples of what has worked and further lessons to be learned that the Scottish Government and other central government bodies can use when planning and implementing future ICT programmes. The case studies look at specific parts of selected ICT programmes and we do not give an opinion on whether the overall delivery of the programmes has been, or will be, a success.
14.	The report has three parts:
•	In Part 1, we consider what actions the Scottish Government has taken to improve strategic oversight of ICT programmes.
•	In Part 2, we consider what skills gaps and shortages exist and what the Scottish Government is doing to address this issue.
•	In Part 3, we consider how effectively central government bodies are managing ICT programmes.
15.	Appendix 1 provides details of the audit methodology used for this audit. 

Part 1
Strategic oversight
Key messages
1.	The Scottish Government has a key role to play in the strategic oversight of ICT programmes by ensuring investment planning is coordinated and aligned to its digital strategy and that appropriate processes are in place to manage them. It introduced an ICT assurance framework in February 2013 to increase the likelihood of programmes achieving intended outcomes. ICT technical assurance reviews were added to strengthen the framework in August 2013. The framework was not clear enough and has not enabled the Scottish Government to fulfil its oversight role. 
2.	The Information Systems Investment Board’s (ISIB) role was to oversee the implementation of the framework but it did not have sufficient information or capacity to perform this role effectively. It did not receive all the ICT investment and assurance information required from central government bodies, and it did not have the staff to pursue this lack of information. ISIB has not shared the information from lessons-learned reports more widely to help improve the management and implementation of future projects.
3.	The Scottish Government recognised the need to improve the framework and the role of ISIB and took action to address this. A revised framework was introduced in April 2015 which provides clearer instructions and guidance. The Office of the Chief Information Officer will now support ISIB but the split of roles and responsibilities has not yet been finalised. The opportunity to make improvements will be missed if arrangements are not clear.
4.	The Scottish Government has strengthened the independent assurance process. It has developed an additional assurance product specifically aimed at ICT programmes, and has developed the Gateway Review process to ensure that recommendations are implemented.
The Scottish Government has a key role to play in the strategic oversight of ICT programmes
16.	The Scottish Government has a key role to play in ensuring that ICT programmes are aligned to its digital strategy, are improving the way that public services are delivered and are meeting the needs of users. It is also responsible for ensuring that central government bodies are well supported and have appropriate arrangements in place to effectively manage their ICT programme. It is important that the Scottish Government has oversight of all central government ICT programmes to ensure investment planning is coordinated and to maximise opportunities to improve the quality of services through better use of ICT.
17.	Our August 2012 report made a number of recommendations to the Scottish Government to improve its strategic oversight of significant central government ICT programmes (Exhibit 3, page 8). In addition, it recommended that the Scottish Government should seek assurances that weaknesses in management identified in our report were not present in other ICT programmes, for example by ensuring that suitable assurance and scrutiny processes were in place for each central government ICT programme.
The Scottish Government introduced an ICT assurance framework in February 2013
18.	The Scottish Government’s Digital Directorate introduced the ICT assurance framework (the framework) in February 2013. Its purpose is to provide support to the central government body and provide the Scottish Government with information to give it better oversight of ongoing and proposed ICT investments. The Senior Responsible Owner (SRO) and Accountable Officer (AO) are responsible for the delivery of a project.5 The Scottish Government has a key role in the oversight of programme delivery.
19.	The framework was overseen by ISIB and is designed to:
•	allow baseline information on ICT programmes to be collected and monitored regularly in a central investment log, including information on which central government bodies had ICT programmes and their value
•	ensure ICT investments are aligned to the digital strategy and consider opportunities for collaboration
•	provide guidance on, and ensure that appropriate assurance processes are in place for, ICT programmes. If appropriate arrangements are not in place, or there are concerns with specific programmes, ISIB should escalate this to SCSB.
20.	The framework includes an ICT investment plan checklist for central government bodies to use based on the scrutiny checklist included in our August 2012 report. This helps central government bodies and ISIB assess different elements of an ICT programme. It should be used when the outline business case is being developed, and can be used throughout the programme’s lifecycle to assess if suitable arrangements remain in place. We found that ISIB and most central government bodies that we looked at used this checklist at the start of a programme but not at other stages. 
Independent assurance processes have been strengthened
21.	The framework supplements other established Scottish Government independent assurance processes, which are required by the Scottish Public Finance Manual. This includes the Gateway Review process which is used across the UK public sector for all major projects and provides assurance at key decision points in a project’s life (Exhibit 4, page 12).6 The Scottish Government’s Programme and Project Management Centre of Expertise (PPM-COE) is responsible for the Gateway Review process. 
22.	The Scottish Government introduced the ICT technical assurance review in August 2013 to provide central government bodies with specific technical assurance on ICT programmes throughout their lifecycle. The review looks at technical aspects of ICT programmes in greater depth than Gateway Reviews and can be used either in isolation or alongside the standard Gateway Review process (Exhibit 4, page 12).
23.	This is a good additional assurance process, particularly for complex ICT programmes. Since its inception, four ICT technical assurance reviews have been performed. These were at Registers of Scotland, Scottish Environmental Protection Agency, Scottish Government Rural Payments and Inspection Directorate and Scottish Government Learning Directorate. 
24.	The Scottish Government has also strengthened the Gateway Review process. Under the new arrangements, any central government body whose Gateway Review report is rated red or amber/red must develop an action plan within three months. This action plan is then independently reviewed to see if the proposed actions increase delivery confidence. The action plan is also copied to ISIB, which will take action if it considers that the central government body is not taking adequate action. To date ISIB has not done this.
25.	All the central government bodies that we looked at were familiar with the Gateway Review process but not all were aware of the ICT technical assurance review. The Scottish Government should engage more with central government bodies to ensure the best available independent assurance processes are used.
The 2013 framework did not explain some of its requirements clearly enough
26.	The type and level of independent assurance required differs according to the value and risk of the programme. Programmes costing more than £5 million should follow the Gateway Review process. We found that the framework, as originally developed, did not explain clearly enough the programme costs that should be considered for independent assurance decisions. The framework states that the full-life costs should be considered. This means initial capital cost, as well as delivery, operational, maintenance, repair, upgrade and eventual disposal costs. This was not explained clearly enough in the original framework. This means that some bodies may not have assessed their programme correctly against the framework, and fewer ICT projects may have been reported to ISIB than should have.
27.	The framework was not clear about the assurance process for programmes with a value of between £1 million and £5 million, and it did not explain how the public body should consider the level of risk of the programme in the process. For example, a small central government body could have an ICT investment of less than £1million in value but its failure could be detrimental to the whole organisation and the services that it provides. Over 60 per cent (around 200 projects) of ICT projects on ISIB’s investment log are under £1 million in value, and would not have required independent assurance according to the framework.
28.	ISIB and central government bodies should consider the significance of an ICT programme not just in monetary terms but in terms of how failure would affect its core business and the service it provides. 
The Scottish Government changed the role of ISIB in June 2013 to make it responsible for overseeing the framework
29. The Scottish Government extended the role of ISIB in June 2013 to ensure that the framework was being adhered to. This gave ISIB a dual role:
•	Assurance and oversight. It is responsible for ensuring that central government bodies follow the framework and have suitable assurance processes in place.
•	Investment approval and monitoring. It is responsible for approving core Scottish Government ICT programmes and maintaining a log of central government investments. It developed a quarterly investment return for bodies to complete for this purpose. ISIB reviews investment and outline plans to ensure that the programme fits with the digital strategy and encourages collaboration between bodies.
ISIB does not have sufficient information and capacity to perform its role effectively
30.	ISIB requires knowledge of all ICT programmes in central government bodies for its role to be successful. However, some bodies are not completing investment returns, the required checklist, or responding to requests for information from ISIB. In January 2015, ten bodies had not provided responses to ISIB queries. Some of these queries dated back to August 2013. ISIB cannot provide effective support and scrutiny because it has incomplete information.
31.	Central government bodies may not be supplying the right information to ISIB because they do not understand the value of the framework. SCSB has acknowledged that the role of ISIB is not fully understood and that the assurance process for central government bodies needs to be clearer.
32.	ISIB does not have the required support staff to pursue every non-respondent. The membership of ISIB consists of senior officers from the Scottish Government and three central government bodies. ISIB has had difficulty finding the time to review the information and returns that is has received. 
Findings from lessons-learned reports from ICT programmes have not been shared widely
33.	Learning lessons from projects (successful or otherwise) can improve the implementation of future projects. We reported in August 2012 that lessons arising from the three programmes we looked at should be shared more widely with central government bodies. In November 2012, the Scottish Government issued a letter highlighting the main issues from our report to all central government bodies so that lessons could be learned. Our scrutiny checklist was shared as a tool to help central government bodies assess future ICT programmes and now forms part of the framework. We found that ISIB reviews lessons-learned reports prepared by central government bodies and requests their attendance at board meetings. It has not shared lessons from these reports more widely. 
34.	In March each year, the Scottish Government’s Programme and Project Management Centre of Expertise publishes a lessons-learned statement on its website. This gives a broad overview of findings from Gateway and other assurance reviews on all types of projects performed during the year, and it provides guidance on what bodies should be doing. While this is useful, the statements are brief and high level and do not give details about how, for example, a body successfully managed a particular type of programme. Although assurance reviews are confidential it should be possible to extract more detailed lessons learned to share. For example, anonymised case studies could be used to provide real life examples of what worked well, and what did not.
The Scottish Government updated the framework and governance arrangements in April 2015
35.	The Scottish Government has acknowledged that ISIB and the framework are not operating as effectively as they could. It reviewed both of these in February 2015 in response to feedback from central government bodies. It launched a revised framework in April 2015 which included the following improvements:
•	There is a clearer explanation of what is meant by the full-life cost of a programme. 
•	The assurance requirements for programmes of different values are more clearly explained.
•	It clearly sets out the escalation route to SCSB and when it will be used.
•	Flow diagrams show the processes to be followed.
•	Workshops will be used to share lessons learned.
36.	In addition, the Scottish Government created the Office of the Chief Information Officer (OCIO) in February 2015. This sits within the Scottish Government’s Digital Directorate, and will support ISIB by monitoring returns and following up any non-respondents. Having a designated team for this role should improve the Scottish Government’s capacity to ensure that all central government bodies are following the framework and offer appropriate support. Exhibit 5 sets out the roles and responsibilities of each of these functions as at April 2015. 
37.	The creation of the OCIO adds to an already complex support and governance map. There is a risk that if roles and responsibilities are not fully understood by all parties that governance arrangements will not operate effectively and the opportunity to make improvements will be missed. The new framework explains the role of ISIB more clearly but does not explain the role of the OCIO in detail.
38.	The roles and responsibilities of these three bodies have not yet been finalised. The Scottish Government hopes to coordinate the oversight of the revised framework with a new Digital Transformation Service to address the shortage in ICT skills that central government bodies continue to experience (Part 2). The implementation of these initiatives will have to be carefully managed and will require strong leadership, particularly to balance its support role with its oversight role.
The Scottish Government will need to improve engagement with central government bodies 
•	As the Scottish Government launches the revised framework and governance arrangements there are a number of actions it needs to take:
•	Stakeholder engagement will be critical to ensuring that bodies understand and use the framework. The benefits of the framework need to be demonstrated to all central government bodies. Case studies and workshops could be used to highlight where ICT programmes have benefited from the process. 
•	Clear ownership and strong leadership of the framework will be needed to ensure it is implemented successfully. The roles and responsibilities of SCSB, ISIB, OCIO, the Digital Transformation Service, and central government bodies need to be clearly communicated and understood. 
•	A coordinated approach is needed. The OCIO will need to coordinate its activities with the PPM-COE and the Digital Transformation Service to ensure that guidance and support is clearly linked and coordinated. 
•	OCIO will need to have the capacity to fulfil its role. It will need to have the people to support central government bodies and provide appropriate information to ISIB. This will include following up actions included in the framework, and stakeholder engagement.
•	Detailed lessons learned from ICT programmes should be shared more widely. Case studies and workshops could be used to provide details of what has worked well for some ICT programmes.
•	ISIB should provide SCSB with an annual report on ICT programme investments including details of bodies that have failed to follow the framework. The escalation route set out in the framework needs to be used more effectively.
•	The Scottish Government will need to assess if the framework is providing bodies with the support they need. It needs to set out a number of criteria that it will use to monitor the effectiveness and success of the framework.
Recommendations
To help improve strategic oversight, the Scottish Government should:
•	promote the refreshed ICT assurance framework to central government bodies and provide guidance on how it should be used (paragraphs 35–39)
•	clearly set out and communicate the roles and responsibilities of Information Services Investment Board (ISIB), Strategic Corporate Services Board (SCSB) and the Office of the Chief Information Officer (OCIO) (paragraphs 36–39)
•	oversee ICT programmes more robustly through ISIB and OCIO, by holding central government bodies to account if they do not follow the ICT assurance framework. In doing so, they will need to balance their role between providing oversight and offering support (paragraphs 30–32 and paragraphs 36–39)
•	explain clearly in the framework that bodies should consider the significance of a programme, not just in monetary terms, but in terms of the risk to business continuity, and service delivery (paragraph 28)
•	implement the revised assurance framework effectively (paragraph 39).
To help improve the management of ICT programmes, the Scottish Government should:
•	develop a more detailed approach to reviewing lessons learned, and use interactive and innovative ways to disseminate them, so that bodies can learn from other projects. This could include using workshops, case studies, and further developing online resources (paragraphs 30–35 and 39).
To improve and enhance programme management, central government bodies should:
•	fully comply with the ICT assurance framework and work with ISIB and the OCIO from the scoping stage of an ICT programme through to business-as-usual (paragraphs 35–39)
•	calculate the full-life cost of the programme and assess the significance of the programme in terms of risk to business continuity and service delivery to help determine what assurance processes it should use (paragraph 28)
•	proactively seek opportunities to share and learn lessons from other organisations’ ICT programmes to benefit from their experience (paragraphs 33–34).

Part 2
Addressing skills gaps and shortages
Key messages
1.	Our August 2012 report highlighted that a lack of skills was a key factor in the failure of central government bodies’ ICT programmes. The Scottish Government set up a workforce workstream in March 2013 to explore options to help address this issue across the public sector. It did not perform a skills gap survey across the public sector until August 2014. The action plan published in December 2014 may help to address the shortages in ICT skills in the longer term. 
2.	The Scottish Government recognises that the recurring lack of skills is a significant problem for central government bodies. It is now developing a new approach to help address the problem by pooling and sharing resources in a central government Digital Transformation Service. This is an ambitious project and detailed arrangements are still being developed. It is too early to assess how effective this will be.
3.	Some bodies are not assessing the skills they require to deliver ICT programmes at the start of the process. Those that had done were able to plan to address the gaps and recruit specific expertise to key roles.
4.	Central government bodies are still finding it difficult to access the skills required to manage ICT programmes due to internal skills gaps and shortages in the market. Some have used short-term contractors to fill identified skills gaps. This means that knowledge and expertise may be lost when the contracts end if effective knowledge transfer plans are not put in place. 
The Scottish Government did not perform a skills assessment across the public sector until August 2014
40.	Our 2012 report recommended that the Scottish Government assess the skills required for future ICT programmes to ensure these skills were accessible to public bodies. This included carrying out a strategic review of ICT skills available across the public sector.
41.	The Scottish Government set up a cross-public sector workforce workstream in March 2013 to meet commitments set out it its digital strategy, including:
•	Working with public sector bodies to develop a workforce with the skills and confidence to support the delivery of digital services.
•	Working with public sector bodies to maximise the impact of employees with ICT skills and use these skills efficiently. 
•	Developing shared approaches to enhancing the skills of the ICT employees in the public sector.
42.	The actions from this workstream included introducing a digital leaders’ programme in July 2013, and examining the use of ICT Modern Apprentices in organisations. Between September and December 2013, the Scottish Government carried out a feasibility study on developing a register for ICT employees to register their skills and capacity across the public sector. The study concluded that a public sector-wide register was not practical but that sector-specific solutions should be considered.
43.	The Scottish Government developed a skills development investment plan covering the wider digital technologies sector in March 2014.7 It commissioned a skills gap survey across the public sector in August 2014 as part of its digital strategy. In the two-year period between our report and this survey, central government bodies were still experiencing difficulties in finding appropriately skilled staff for ICT programmes. In our view the skills gap survey should have been done earlier to enable the Scottish Government to take quicker action to address the problem in the short and long term. 
The Scottish public sector continues to have gaps in the skills needed to deliver ICT programmes
44.	The survey looked at current and future skills gaps and shortages:
•	Skills gaps are defined as those within the organisation, that is, the skills and experience of existing employees. 
•	Skills shortages are defined as a shortage of skills and experience outside the organisation; in other words, in the external market. 
45.	The survey looked at specific ICT and digital skills as well as business skills such as contract and project management. The Scottish Government sent surveys to 159 public sector bodies with 66 responding (42 per cent); 29 out of 55 central government bodies responded (53 per cent). 
46.	The results of the survey were published in November 2014.8 The results (Exhibit 6, page 21) showed that there are gaps in relation to specific ICT skills as well as general business skills such as contract and supplier management. 
47.	The results of the survey are consistent with the findings from our work. Central government bodies continue to face difficulties in recruiting the specialist ICT, project and contract management skills required to implement ICT programmes. Central government bodies are constrained by public sector pay scales and cannot compete with the private sector salaries on offer; for example, the Scottish Qualifications Authority (SQA), Registers of Scotland, COPFS, and Police Scotland all experienced difficulties in attracting and retaining skilled and experienced staff because they could not compete with salaries in the private sector. The roles these bodies were trying to fill ranged from senior IT directors and project managers to software developers and design analysts. Software developers in the private sector can earn between £45,000 to £50,000, compared to £20,000 to £25,000 in the public sector. 
48.	Shortages in the external market add to these challenges, making it difficult to find the required skills, and pushing up the salaries for these skilled jobs.
Central government bodies are using short-term solutions to fill skills gaps
49.	Central government bodies use short-term solutions, such as short-term contracts or agency staff to fill identified gaps. Using short-term contracts may be the only way to quickly access these skills and supplement a programme team but there are risks associated with this approach:
•	The body may be too reliant on short-term contractors, reducing continuity in the programme team.
•	Knowledge and expertise, including detailed knowledge and understanding of a contract or system, may be lost when the short-term contractors leave.
•	The cost of using short-term contract arrangements is generally much higher than employing permanent staff. This increases the pressure on budgets.
50.	We found evidence of ICT programmes where the majority of staff were on short-term contracts across all levels of the programme, including the programme leadership (Case study 1, page 22). 
51.	Although bodies gain the expertise they need in the short term (generally two years), that knowledge and expertise may be lost at the end of the contract. Bodies need to put knowledge transfer plans in place from the outset to ensure this expertise is shared with permanent members of the team and is not lost. This applies at all levels of the organisation, whether a software developer or digital director. Registers of Scotland, COPFS, and Police Scotland have put succession and knowledge transfer plans in place. Knowledge transfer should occur not just within the central government body but across the sector.
52.	There are also significant cost implications to consider. For example, the daily rate for a project manager employed through a short-term contract can range from around £300 to £850 (excluding VAT). This equates to an annual cost of between £78,000 and £221,000 (including VAT). A specialist ICT role, such as a web developer, would cost around £90,000 annually using a short-term contract. This highlights the importance of addressing the skills gap quickly.
53.	It is also important for central government bodies to consider the business-as-usual skills and resource requirements alongside those required for the specific programme. For example, some staff may have responsibility for managing a programme while maintaining their usual day-to-day role. It is important that management find a balance between the staffing requirements for both the ICT programme and day-to-day operations during busy periods. The Scottish Fire and Rescue Service did this and were able to identify where project timelines could be adjusted and where external consultants could be brought in to deal with pressures points where specific staff and resources were in high demand. This has helped to keep the overall programme on track.
Some central government bodies are not identifying the skills they need to deliver ICT programmes at the start
54.	We reported in August 2012 that Registers of Scotland, COPFS, and Disclosure Scotland did not have the in-house skills and expertise they needed to manage their ICT programmes. For example, for some programme team members, their role was an add-on to their day-to-day business responsibilities. This often led to conflicting time pressures. We also found that some programme board members and SROs lacked specialist ICT skills and experience. The lack of skills was identified as a key factor in the failure to deliver. If the right skills are not in place from the start of the programme there is a risk that it will not be planned and scoped properly, and benefits will not be clearly defined. Our report recommended that central government bodies carry out a detailed skills assessment before starting any ICT investment programme. 
55.	Central government bodies that performed a skills assessment at the start of their ICT programme were able to put action plans in place to fill these gaps (Case study 2). 
56.	Some organisations in our sample had not performed a skills assessment at the start of the ICT programme. This meant that they did not have the required skills in place to deal with the complexities of the programme. For example, National Records of Scotland did not perform a formal skills assessment at the start of its project. A Gateway Review identified that they did not have the skills to provide sufficient technical challenge to feed into the decision-making process resulting in the scope of the business case being changed.
57.	A skills assessment should consider the skills at all levels of the organisation including the board. In response to our 2012 report, Registers of Scotland recruited a non-executive board member with experience in IT projects. This gave the board the capability to scrutinise the delivery of its ICT programmes more effectively.
The Scottish Government has put a skills gap action plan in place which may help in the long term
58.	The Scottish Government’s skills gap action plan, published in December 2014, set out four high-level actions to deal with identified gaps and shortages. The plan focuses on ICT-specific skills such as software development (Exhibit 6, page 21) and includes the following actions:
•	Develop collaborative projects to support skills development and training, for example by developing a collaborative ICT Modern Apprenticeship pilot.
•	Develop an ICT skills and training notice board to advertise skills and training events.
•	Make public sector ICT employment more attractive, for example, by developing case studies to make working in public sector ICT more appealing, and establishing a network of ambassadors to promote ICT careers to young people and schools and colleges.
•	Increase engagement with the education sector as a potential source of training and entry level staff.
59.	These actions focus on training and development opportunities to address the gaps and shortages in the long term. The pace of change is fast and the education system will need to be able to keep pace with the changing demands of the public sector and beyond. The use of case studies to make public sector ICT employment more attractive will be a start but will not be enough on its own to attract experienced ICT professionals to the sector. 
The Scottish Government is developing a new solution to make skills more available to central government bodies
60.	The Scottish Government recognises that a lack of digital and business skills is a recurring and significant problem for central government bodies. The public sector is no different to the private sector in this respect. There is a shortage of some digital skills across all sectors, but the public sector can often not compete with private sector salaries. In response to this, the Scottish Government plans to develop a central government Digital Transformation Service (the Service).
61.	The aim of this Service is to help strengthen, and provide access to, digital skills across the sector. It will be managed centrally within the Scottish Government, and plans to offer four different types of service to central government bodies (Exhibit 7, page 25). The Service plans to:
•	help with developing business cases and project design, including benefits
•	provide support in implementing the programme, including attending programme boards
•	help bodies with their overall digital strategy
•	support collaboration between central government bodies and across the public sector.
62.	The Service will employ people centrally and also access people and skills from the contractor market and other central government bodies. The full specification of the Service is being finalised, but it has been granted funding of £2 million in 2015/16 to get it established and to recruit some of the key people and skills required. It is planned that the Service will be financially self-sustainable in the long term.
63.	This is a positive step by the Scottish Government. It is an ambitious project with potential to significantly change how central government bodies access the key skills they require for ICT programmes. The Service will need a good understanding of the skills and experience that bodies require, and will need to know what experience exists and can be shared. This will require all central government bodies to fully participate in and engage with the Service. 
64.	The Scottish Government intends the Service to support central government bodies find and successfully implement new digital solutions. For example, where central government bodies have similar requirements for a Customer Relationship Management system, the Service will help bodies to find a common solution and help them move to the new system.
65.	The Service will be governed by a new steering group; its remit is yet to be finalised. The Scottish Government is proposing that the steering group take on the assurance role that ISIB currently has (Exhibit 5, page 15). This is already a significant and challenging project, and the addition of the assurance role adds more complexity to the governance and support arrangements. If this Service is to have the required impact and be implemented effectively, it is critical that the Scottish Government sets out plans to ensure that: 
•	Strong leadership is in place to deliver the Service. Leadership of the Service will need to effectively engage with the sector and sell the benefits of the Service. Strong leadership will be required to ensure that the Service is coordinated and effective. 
•	There is good stakeholder engagement. The team need to plan carefully how to engage all central government bodies, and how to get buy-in to the benefits of the new Service. 
•	Strong governance arrangements are in place. The Scottish Government will need to review the governance arrangements for ICT programmes to ensure that the new Service fits appropriately within the framework. This includes finding a balance between the Scottish Government’s assurance role and the support aim of the new Service. 
•	It is clear what success looks like. This involves defining what benefits the Service will bring and setting up performance monitoring and customer feedback systems. This is vital to ensure that the Service delivers value for money.
•	The right people and the right skills are in place from the start. The Service will need to make better use of the ICT investment log to carefully identify the priorities to make an impact immediately. It should ensure that it has access to the skills to provide this support. This will help to get engagement and buy-in to the new Service.
•	Central government bodies are persuaded about the need for a change in culture. This is a significant change for central government bodies that are used to recruiting staff independently. The Service will need to work with bodies to help them recognise the benefits of the new way of working.
•	The pace of change is ambitious but realistic. The Service needs to ensure that it only commits to actions that it can deliver, and should be careful not to be too optimistic about timescales. The Service will need a clear plan with milestones and communicate and agree this with central government bodies.

Part 3
Managing programmes

Key messages
1.	Some central government bodies have processes for identifying the benefits of a programme at the start, and measuring them throughout. Other bodies did not have processes in place to clearly define benefits or collect baseline information at the start of their programme. 
2.	The Scottish Government and central government bodies are using appropriate project management techniques in line with the framework. Five central government bodies used Agile techniques to manage parts of their ICT programmes. We found that some had limited experience of using an Agile methodology when they embarked on this approach. These bodies have since invested in Agile training for staff. It is important that all appropriate staff within the organisation have the required skills and experience to use and understand new project management techniques.
3.	Central government bodies that invested in contract and supplier management were better able to hold the supplier to account. This helped ensure that the supplier delivered what was expected. Some central government bodies still do not recognise the value of investing in contract management throughout the lifecycle of a programme.
There is still evidence of benefits not being clearly defined at the start of a programme
66.	We reported in August 2012 that the business cases of the three ICT programmes we looked at lacked detail in some sections. In particular, benefits were not clearly defined, and it was therefore difficult to measure and monitor the extent to which they were being realised. 
67.	The business case establishes from the outset the overall direction, impact and value for money of the programme. It should clearly define the programme’s benefits and how they will measure them. If bodies do not fully understand at the beginning what benefits a programme is expected to achieve there is a risk that the programme will be poorly defined and opportunities missed. Evaluating the benefits realised is essential in assessing whether a programme has achieved value for money. Bodies need to monitor and measure benefits through the life of a programme (Case study 3, page 28). 
68.	Collection of baseline information is a critical part of being able to assess the benefits and value for money of a project. The Scottish Public Pensions Agency (SPPA) and the Scottish Children’s Reporter Administration (SCRA) were unable to collect baseline information at the start of their programmes owing to lack of resources or inadequate systems. SPPA subsequently appointed a benefits manager in October 2014 to progress work on benefits realisation for the rest of the programme. 
69.	National Records for Scotland (NRS) did not clearly identify planned benefits and how they would be realised until partway through the programme. This meant that baseline information could not be collected. 
Central government bodies are using recognised programme and project management techniques in line with the framework
70.	Programme management involves overseeing and coordinating a number of projects to achieve outcomes and strategic benefits. It involves identifying how projects depend on each other, and managing these interdependencies to minimise risk and maximise benefits. 
71.	Project management involves creating plans, coordinating resources such as money, skills, and equipment, monitoring progress and managing risk to deliver a project within a given timeframe. Contract management, including supplier management, can be an important part of programme and project management. 
72.	Programme and project management techniques include PRINCE2 (PRojects in Controlled Environments) and Agile. PRINCE2 is the Scottish Government’s preferred approach and is the most widely recognised form of project management in the public sector. 
73.	Agile is a project management approach which emerged from the software industry and is used mainly for ICT projects. Agile is a flexible, incremental approach where teams work in short bursts, called sprints, on small-scale launches of a functioning product. Teams test each software launch against the user’s requirements throughout the project. In theory, the end product is more likely to meet the requirements of the organisation and end-user because stakeholders have been involved throughout and changes can be made more easily. Exhibit 8 provides a comparison between Agile and PRINCE2 methodologies.
74.	The framework requires appropriate programme management processes to be in place; both PRINCE2 and Agile are acceptable. We found that most central government bodies use PRINCE2 to manage projects in a structured and controlled manner (Summary of the 12 bodies, page 39). Some central government bodies that we looked at, particularly those involved in developing software, are using Agile or a hybrid of PRINCE2 and Agile. These include the Scottish Government, Registers of Scotland, Revenue Scotland, Accountants in Bankruptcy, and the Scottish Housing Regulator. 
ICT programmes are complex and central government bodies should assess how best to manage them
75.	Central government bodies should consider which programme or project management method, or combination of methods, will best suit each aspect of their programme; there is no one size fits all solution. All levels of the organisation involved in the programme should understand the chosen approach and have the expertise and skills to manage it. The Agile methodology is very different to PRINCE2, and providing training in Agile is essential to ensure bodies fully understand all the processes involved so that the technique can be used effectively. Case study 4 highlights how the Scottish Housing Regulator combined both techniques to manage its programme.
Central government bodies did not always have the appropriate skills and experience to use different project management techniques
76.	Agile is a relatively new project management technique in the public sector, and it is unlikely that bodies will have used and fully understand the methodology. It is equally important for a body that has outsourced some of its ICT programme to an external contractor to know how Agile project management works if the contractor is using it. This will ensure that the body has the skills and expertise to challenge the supplier on progress and hold them to account. We found that Registers of Scotland and the Scottish Government had limited internal experience of dealing with this methodology, including dealing with contractors using Agile.
77.	Registers of Scotland have started investing in Agile training for staff at all levels of the organisation (Case study 5). The Scottish Government is using specialist Agile coaches for the futures programme. Revenue Scotland sourced a programme director with experience of managing change, including large ICT projects using an Agile methodology. Fully understanding this methodology will help ensure that such projects are managed and governed effectively.
78.	Decision-makers need to understand how Agile methodologies will be used and managed, and the information they can expect to receive. The National Audit Office reviewed the governance of some Agile projects in the private sector in order to provide some practical guidance to public bodies considering the use of Agile. It identified and identified good practice, including: 9 
•	Governance should be proactive and focus on the activity and tasks taking place at a point in time, and the value they are adding. This is in contrast to PRINCE2 where progress is checked against a detailed plan and what is being done to reduce variations between baseline and forecast spend.
•	Senior management identify the ‘must have data’ and performance information they require to oversee the project. The project team is then responsible for producing this at specified times.
•	Senior management approves spend by the project team at a particular rate over a set period of time to achieve the desired outcome. The decision- makers and project team should define what standards and tests they will use and what results are acceptable at the start of each task.
Effective contract and supplier management is essential but effective arrangements are not always in place
79.	Contract management is a key aspect of programme and project management where a contract or agreement is in place between two or more bodies. It starts with negotiating the contractual agreement, and extends through using the contract to manage delivery, to managing the end of the contract.
80.	Investing in contract management from the outset can help to keep projects on track and contain costs (Case study 6). Contract management requires close working with both the supplier and key internal stakeholders, such as the programme board and management team. Historic Scotland had to re-scope a contract with a supplier after it was found not to meet requirements. This was caused by poor controls at the start of the contract including sign off, authorisation and contract development. This led to budgeted costs increasing from £2.6 million to £3 million. The auditor recommended that processes be improved to ensure appropriate oversight and approval was in place.
81.	Another aspect of contract management is supplier management. This involves ensuring that the supplier meets its obligations, milestones and the performance indicators set out in the contract. Central government bodies should have a designated person in the project team to deal with the supplier. They should meet the supplier regularly to ensure the supplier is performing to the required standard and that timescales are achievable.
82.	We found that central government bodies understood the importance of having regular meetings with the supplier and holding them to account, but we found inconsistent application of this. In some cases the monthly or quarterly meetings with the supplier had stopped for a period of time. We found that a lack of capacity was the main reason for supplier meetings having stopped in the bodies that we looked at. This increases the risk to successful project delivery.
83.	The Scottish Government offers guidance on supplier management including providing information on tools, such as balanced scorecards, that can be used as part of the monitoring and management process. A balanced scorecard measures performance in terms of quality, cost, sustainability, and service. Key Performance Indicators (KPIs) should be set to enable an organisation to measure the supplier’s, as well as its own, performance against the scorecard and strategic goals. For example, the Scottish Government’s futures programme did not have effective tools for contract and supplier management at the start. Once the programme had recruited the relevant expertise, it created a full balanced scorecard to manage the supplier. 
Central government bodies have responded when issues arise
84.	Part of good programme management involves responding quickly when the programme or project is showing signs of slippage, or when significant issues arise. We found examples of how central government bodies identified, responded to, and managed issues as they arose, including:
•	using highlight reports to identify issues with specific projects and tasks
•	weekly or daily meetings between programme director and programme manager to identify actions to be taken to address identified issues
•	bringing in support from other areas of government for a short ‘rapid response’ approach. This approach can help to focus attention and support on the areas requiring intervention, while not stopping progress with the programme.
85.	Revenue Scotland used some of these techniques when managing its IT implementation project (Case study 7).
86.	If a programme experiences significant difficulties and delays it may be necessary to put a detailed recovery plan or experienced team in place to get it back on track (Case study 8, page 35).
87.	Having effective processes in place to identify and respond to problems quickly is necessary to keep the delivery of a programme on track and prevent increases in costs. Regular communication between project teams and senior management can help find solutions including identifying the right people to deal with the issue. 
Recommendations
To improve and enhance programme management, central government bodies should:
•	ensure they define benefits clearly at the start of a project, and monitor and measure them throughout, so that the body has a clear focus and can demonstrate value for money (paragraphs 66–69)
•	ensure appropriate people within the organisation have the skills and understanding of the chosen project management technique, such as Agile. This includes programme staff, senior management and board members (paragraphs 75–78)
•	ensure that they have the required contract and supplier management skills to negotiate, monitor and effectively manage the contract throughout the programme’s lifecycle (paragraphs 79–83).

Appendix 1
Methodology

The focus of our work was to assess what progress the Scottish Government, Registers of Scotland, Crown Office and Procurator Fiscal Service, and Disclosure Scotland had made against the recommendations in our 2012 report. We also selected a number of central government ICT programmes to explore areas including:
•	what problems they encountered when managing ICT programmes
•	what techniques they have used to manage their programmes.
We set out to find examples that could be used as cases studies to help other central government bodies learn how best to deal with issues that they encounter. 
We sent an initial questionnaire to the auditors of central government bodies in January 2014. This was to gather information on their ICT programmes and contracts and establish where best to target the follow-up work.
•	We reviewed the initial returns and selected 12 central government bodies that had significant ICT contracts or programmes in place. The criteria included:
•	how significant in size or complexity the programme was for that body
•	if it involved implementing new legislation, or was a new type of programme for that body
•	if it required strong governance and management arrangements.
•	We agreed an approach with local auditors, which included:
Auditor returns 
Local auditors completed a return between June and November 2014 based on the scrutiny checklist from the 2012 report. Auditors were asked to select the questions that best suited the stage of the programme and the risks identified by the auditor. We reviewed these returns between December 2014 and February 2015 and identified common themes and approaches that we thought were useful. The returns provided a snapshot of the ICT programme at a point in time.
We did not look at the selected programmes in detail, and only looked at aspects that were considered of interest for the purposes of this report. For this reason, we have not given an opinion on whether the overall delivery of these programmes has been, or will be, a success. All the programmes are at different stages and there will have been developments since we completed our audit work in April 2015. We will report further if necessary.
Further information about the 12 programmes or projects is given on (Summary of the 12 bodies, page 39). 
Interviews
We obtained updates on progress and further information on specific programmes by interviewing senior people within the following bodies:
•	The Scottish Government
•	Registers of Scotland
•	Crown Office and Procurator Fiscal Service.
•	Disclosure Scotland
•	Police Scotland
•	Revenue Scotland.
Desk research
•	We reviewed key documentation including:
•	the ICT assurance framework 
•	ISIB and SCSB minutes
•	the Scottish Government digital strategy.

Appendix 2
Update on the three central government bodies included in our August 2012 report 

Registers of Scotland (RoS) 
Background
RoS started a ten-year Strategic Partnership Agreement in 2004 with BT involving ongoing ICT service provision and a transformation programme. RoS terminated the agreement in 2012 because it no longer considered the outsourcing of all ICT as appropriate.
Update
Since we reported in August 2012, RoS ended the agreement with BT and brought all ICT activities in-house on 1 December 2012. RoS paid £1.2 million in compensation to BT in 2012/13.
Auditors reported that RoS managed the transition well without disrupting its service provision. RoS has strengthened governance arrangements, financial control and progress reporting over the past two years. Some actions include:
•	appointing non-executives with ICT skills and expertise to both the board and Audit Committee
•	using regular independent assurance reviews to inform and challenge how it delivers programme delivery
•	regularly providing budget tracking documents to programme boards.
RoS has had difficulty recruiting and retaining senior ICT professionals to lead development work and ICT services. The chief technology officer left unexpectedly in February 2014. RoS took this as an opportunity to review its ICT operations.
RoS recognised it lacked the skilled staff needed to develop its ICT systems to meet the requirements of new legislation; the Land Registration (Scotland) Act 2012 (LRA). It introduced an alternative approach in the spring of 2014. This included the following:
•	RoS prioritised what the ICT system needed to do by identifying the minimum requirements that the new law and customers required.
•	It took on two existing external contractors to support its intelligent client function and project delivery functions (ICT professional lead role).
•	It used a contractor working within its operations to fulfil the role of professional ICT technical lead. Two RoS managers were appointed as ICT lead and ICT business manager to oversee the work of contractors.
In September 2014, RoS appointed a new digital director, and other specialist senior ICT roles on short-term contracts using the Scottish Government’s interim manager’s framework. RoS has processes in place to minimise the risk of knowledge being lost when these contracts end. 
RoS’s digital strategy was approved in February 2015. This represents a key milestone, with agreed plans now in place to support its delivery.
Conclusion
ICT provision is now in a better position than it was in 2012. RoS successfully implemented the legislative requirements of the LRA on the designated day and has delivered its IT responsibilities for the implementation of the Land and Buildings Transaction Tax. 
Priority was given to the successful delivery of these key projects. RoS had to take on agency staff to help deal with its day-to-day operations as a result. RoS recognises it needs to have the capacity to maintain service delivery and manage major projects going forward and it has put processes in place to manage this.
Crown Office and Procurator Fiscal Service (COPFS)
Background
COPFS ended the Phoenix case management programme in 2010 owing to its increased cost and reduced capital budget.
Update
Since we reported in August 2012, COPFS reviewed the decision to design and build a completely new case management system (CMS) and decided on an alternative approach of upgrading and enhancing the existing ICT infrastructure and systems. This allowed individual tasks with clear objectives and costs to be identified presenting a lower-risk management option. 
Auditors reported that progress with this new approach was initially held up because there was an absence of senior leadership in the IT systems division owing to unforeseen circumstances. A new director of IT was appointed in June 2013, which provided the stability to change governance and organisational structures required to develop the IT systems. A strategic plan was put in place to structure and prioritise the development programme and improvement initiatives. A training plan was put in place to ensure that staff developed the range of skills and experience to deliver strategic priorities.
Upgrading of the CMS has been done using existing resources (a combination of an ongoing supplier contract and in-house staff). A significant upgrade went live in January 2015 with no issues reported. 
COPFS has had difficulty hiring experienced software developers, and has had to recruit staff using short-term contracts. COPFS has put processes in place to ensure that knowledge is transferred to the in-house team over the period of these contracts. IT resources are complemented where necessary with additional expert technical resource and specialist supplier provisions to meet strategic and project delivery needs.
COPFS is currently developing its digital strategy which will be aligned to the Scottish Government’s Justice digital strategy.1 It worked collaboratively with the Scottish Government, Scottish Prison Service, Scottish Court Service, Police Scotland and other justice sector partners during the development of the national strategy. It was a sensible approach for COPFS to have waited until this was finalised before developing its own digital strategy. 
Conclusion
COPFS IT systems division has been through a period of change since we last reported. It has taken the new director of IT time to recruit the required staff and to evaluate the processes and systems to support and deliver the strategic development plan. A portfolio of ICT-enabled business improvements has been identified and implementation is on-going. It is too early to say if these will deliver the intended benefits. 
Disclosure Scotland
Background
Disclosure Scotland introduced the Protecting Vulnerable Groups (PVG) programme to meet the requirements of the Protecting Vulnerable Groups (Scotland) Act 2007. The programme included a project to design and build a new ICT system to support the new arrangements. The PVG system was delivered by BT through an amendment to an existing service contract. The programme experienced a number of significant problems when it went live in February 2011, and a recovery project was initiated.
Update
Disclosure Scotland has worked to improve governance since the previous report. One action was the formation of the PVG supplier group. This group met weekly to track the progress of the PVG recovery project and helped to provide an early indication of any potential issues. 
Disclosure Scotland took on contractors to strengthen the skills available, including increased legal support for contract management. It also commissioned additional external assurance of the PVG system to ensure that the system being delivered met contractual requirements. A contract variation agreement was signed in 2014, which included defined financial penalties for deadlines and service targets not being met, and included the option to extend the contract for a year to April 2015. Disclosure Scotland also increased communication with the supplier, and improved programme reporting.
The recovery project ended in April 2014, and all applications are now processed on the PVG system. All legacy systems have been migrated onto PVG and have been shut down.
Disclosure Scotland planned to replace the BT service contract with a new contract for the care and maintenance of the PVG system which would better meet the needs of the organisation. It awarded a new contract to Atos in May 2014 with the service due to transfer in December 2014. The delivery and transfer of the service by the new supplier did not take place as planned and to protect the business critical operations, Disclosure Scotland continued the existing service contract with BT until the end of March 2015. Disclosure Scotland awarded a new contract to BT for the care and maintenance of the PVG system starting April 2015.
Disclosure Scotland is in discussions with Atos around the non-completion of the contract awarded in May 2014. Given this, we have not commented on Disclosure Scotland’s management of the procurement and contract award process at this time.
Conclusion
Disclosure Scotland has successfully managed the end of the recovery programme, moving the PVG system into business-as-usual. 
We are considering the risks and challenges of replacing the BT service contract further and will report as appropriate.
In the longer term Disclosure Scotland is planning for a significant decrease to the volume of its business in 2017, arising from changes to basic disclosure applications in England and Wales. As part of this planning it will need to fully consider what its future requirements will be, and ensure that it has systems in place to meet these.