.png)
1 month ago
7
Analysis In 2020, the UK's Office for National Statistics (ONS), which provides data vital to form public sector policy and allocate resources, launched a plan to integrate government data and provide "high quality analysis that reflects the diversity of economic and social experience in our country."
Five years later, an independent review revealed that the budget of £240.8 million had been used – with approval from His Majesty's Treasury – to fund more general tech and data costs as the ONS struggled to get off legacy IT systems. In the spring spending review, the Treasury cut the rope.
The muddled plan and missed objectives left the government with three different platforms for data sharing, just as it faces seemingly insurmountable policy challenges in tax, spending, borrowing, housing, healthcare, social care, education, immigration, policing, planning, and more. It has also launched ambitious plans for AI, which will depend on data.
In its business plan for 2020-25 [PDF], the ONS promised its Integrated Data Platform Program – the name it used at the time – would support "the integration of Government data, [which] enables inclusive analysis on national issues and facilitates enhanced dissemination of data and analysis."
"The platform will deliver a strategic, secure and user-friendly environment and with integrated and categorized micro-data made available for research, analysis and presentation to users and the public via controlled access routes," it promised.
Last year, the ONS continued to promise that the Integrated Data Service (IDS) – which resulted from the program – was in its "BETA maturity phase."
"The IDS will grow at pace over the remainder of the Program with a pipeline of additional and transformational capability, data, projects and users as well as improved and accredited data integration and cross-sector collaboration enabled through cloud technologies that will drive significant uptake," it said in its strategic business plan.
However, this assessment belied the difficulties the program faced.
Last year, the Infrastructure and Projects Authority (IPA) – a joint Cabinet Office-Treasury unit that tracks major government projects – revealed it had increased the risk rating of the project from amber to red because it had completed only three of the 12 actions recommended by the government's gateway review process for major projects. It also slashed nearly £500 million from the agreed benefits of the program. The ONS plowed on, promising to "re-engage with other government departments... to capture their analytical efficiencies and statistical financial benefits for the public good."
But in the equivalent report for the 2024-25 financial year, the National Infrastructure and Service Transformation Authority (NISTA) – which replaced the IPA – indicated not enough progress had been made and continued to rate the project a red risk following another gateway review in March 2025.
The sustained risk was attributed to the program "being unable to demonstrate sufficient progress in unlocking cross-government data sharing; limited case studies of analysis undertaken on IDS; and the future of linked data; lack of a replacement Program Director," the report said.
It also effectively ended plans to build new services, although existing services would continue to be available, largely internally to the ONS, missing one of the major objectives.
NISTA's report said that a previously agreed budget of £53 million a year until 2029/30 would end in March 2026 as a result of the government's Spending Review in June. The total, or whole-life cost, of the project was duly cut from £525 million to £284.6 million, ending in March 2026. Any funding beyond next year would be subject to the next stage in the Spending Review.
In April this year, the ONS gave an indication of underlying challenges with its technology stack. It said that the government's "flat" financial settlement for its overall budget meant it was "obliged to protect budgets (in nominal terms) for our range of economic statistics and essential security to protect the data we hold."
Inflation and civil service pay awards mean it was required to make "tough choices" including putting a number of programs on the back burner as it transitions away from legacy IT systems, which it acknowledged would have a financial downside in the long term.
Yet the alarm over legacy systems – the ONS still runs HCL (née Lotus) Notes and Domino for the time being – comes after it managed to siphon off the budget meant for the IDP/IDS project and use it for core spending.
An independent review by Sir Robert Devereux, a retired senior civil servant, found "deep-seated" issues with the ONS and called for radical changes, including splitting it into two organizations, but also pointed out what went wrong with the IDS, which it said "proved the most difficult program for ONS."
Although the IDS had "delivered some ground-breaking data linkages," the benefits compared to the costs were low, few government analysts outside the ONS used it, and it had failed in its original plan to replace the ONS's aging Secure Research Service (SRS).
"It is also the case that substantial sums initially ringfenced for IDS have – with the HMT approval – been used to fund more general technology and data costs. And despite the IDS business case originally being in part predicated on the need urgently to replace the Secure Research Service, the two services are now running in parallel," it said.
- 'Money-saving' UK procurement platform racks up monster tab
- UK.gov ditching 'Red' risk data sharing project after slashing £0.5B budget in half
- 'Technical debt' in police database built to respond to child murders causing a 'failing service'
- Britain jumps into bed with Palantir in £1.5B defense pact
It also struggled to win friends in wider government. "Given that IDS also relied heavily on securing data from other departments, several people – in and outside ONS – thought the program initially adopted a 'high-handed' approach, essentially demanding data, and downplaying others' concerns for example around security," the Devereux review said.
In its business plans for the current year, the ONS says the IDS will continue to provide a "unifying the approach to data linkage," creating a discoverability layer – including a Google-style data catalog – and will revolutionize how "linked data can be used through a scalable cross-government data product."
While both the IDS and the legacy SRS are set to continue, the government is adding a third source of data sharing into the mix. The Labour Party manifesto promise of a National Data Library (NDL) was confirmed in the autumn budget after it came to power.
Controlled by the Department for Science, Innovation and Technology (DSIT), the NDL hopes to "provide simple, ethical and secure access to public data assets, giving researchers and businesses powerful insights that will drive growth and transform people's quality of life through better public services and cutting-edge innovation, including AI," a budget document says.
A later discussion document said the NDL could use data to "improve public services and policy making."
Having three data-sharing services inevitably raises questions about who should use which service, and to what end.
According to a missive from His Majesty's Revenue & Customs, the SRS – which has just celebrated its 21st birthday – offers curated, research-ready data sets to accredited researchers. The IDS, meanwhile, is "primarily used as an internal ONS environment but has been made available to a limited number of external users... though it is not currently accepting new applications for external research."
That was not exactly the plan, but the relationship with the National Data Library seems even more muddied. As NISTA noted, there is "ongoing uncertainty in the relationship with NDL and the future of linked data." The Register has contacted DSIT on this point.
Next month, UK finance minister Rachel Reeves is set to deliver her autumn budget, setting out plans for tax and spending. Wherever she's taking the nation, she's going to need a map.
Data sharing is supposed to inform policy, support research, and provide fodder for AI, which the government hopes will increase public sector productivity and limit the appetite for spending.
After being appointed chief data officer, Craig Suckling said his mission was to "unlock access and break down silos." He might also want to ask how the government ended up with three different systems for doing just that. ®
