Statistics NZ data pipes have burst, and it’s as bad as any burst sewer

Comment

When municipalities fail to fix their pipes, everyone notices the sewage flowing into the harbour. The failure of the outdated NZ.Stats database has less visible but equally worrying impacts on health and well-being.

Opinion: This is the standard criticism of local government, and it is often true. Councilors and mayors are more interested in spending big bucks on glitzy convention centers than on maintaining the drains. Deferred maintenance can be a problem for future board. Ribbon cutting in new places brings benefits today.

But it’s not just local government.

On February 4, a major data pipe from Statistics New Zealand burst. Or had to be turned off. In all cases, the indescribable notice went up: “February 4, 2022: NZ.Stat is currently unavailable”.

I didn’t notice it at the time.

But five days later, Tze Ming Mok was in despair that he could no longer do his job because Statistics NZ had taken down NZ.Stat, without notice, and with no apparent plan to replace the service.

She posted a screenshot of the update it received from Stats NZ about the outage. Stats NZ told her, and she told the rest of us, that Stats NZ took the tool offline following a security review and “because the infrastructure supporting NZ.Stat is become obsolete”. The tool would be unavailable “for the foreseeable future”.

And a howl went off from New Zealand data geeks on Twitter.

Dr. Emily Harvey said“If only they had told us, we could have fetched all the data at the finest resolution preemptively. Is it time for Twitter data to pool their stored data snippets?”

Economist Brad Olsen made an apt comparison: “It’s the data nerd equivalent of Instagram going down. A few worried looks and disbelieving looks in the @InfometricsNZ office! #NZDotStatDown”.

NZ.Stat and Infoshare are the two primary means by which the public, whether journalists, researchers and academics or interested citizens, can access critical statistics held by Statistics NZ.

They are also the main means by which officials access the country’s statistics.

Infoshare is the squeaky workhorse that holds most of the statistics that economists would use most of the time. I don’t know how old it is, but I was using it in the mid 2000s. The interface is outdated but it still works somehow.

Ten years ago, Statistics NZ announced the launch of NZ.Stat in its Annual Reportdescribing it as “a world-class web-based tool for publishing statistics that allows businesses to quickly and easily find relevant information”.

Some of New Zealand’s vital statistics are only available through NZ.Stat. If you want the results of the annual household income survey, you can find them in NZ.Stat. Tons of household data is only available on NZ.Stat.

The collapse of the system is not good.

As an interim measure, Statistics NZ is offering a form asking the public to let them know what data they need and whether they need it in 2-3 days, 4-5 days or more than 5 days. The form asks researchers who they are and why they need the data, presumably to help Stats triage requests.

I’m sure the agency is doing everything possible to make the best of a very bad situation. I’ve had nothing but great interactions with their staff, who will go out of their way to provide stats to people who need them.

But it should never have come to this.

After the front fell off from NZ.Stat, a knowledgeable data analyst David Friggens reported that the system was running on five-version outdated software. NZ.Stat is based on OECD.Stat. The Australian Bureau of Statistics, according to its report, has version 9 of OECD.Stat in beta. And while Statistics NZ had recently launched a project to upgrade to version 10, the version of NZ.Stat that failed was version 5.

I asked Statistics NZ if Friggens was right. Statistics NZ confirmed, by email, that it was using “an older version of the OECD.Stat software, with an ongoing project to upgrade it”, but did not say which version had been used.

To its credit, Statistics NZ had recognized some of the risks it faced. The Agency’s Statement of Strategic Intent 2021-2025 has defined areas of work aimed at ensuring that “basic IT systems are less at risk of failure”. It aimed to “identify risks to core systems and track the effectiveness of mitigation measures to ensure the stability of these systems”.

Unfortunately, he arrived a little too late.

For years, Statistics NZ has, like the proverbial local government, pursued shiny new goals while largely ignoring the critical infrastructure that is needed to keep the whole ship running.

The problem is not only budgetary.

The problem is also the priorities.

In 2018, Statistics NZ launched a new project, Indicators Aotearoa New Zealand. The Agency held meetings across the country and an online campaign, asking Kiwis what wellness meant to them. Data experts then attempted to distill the essence of these meetings into indicators that could eventually be measured.

It culminated in a meeting of about 150 people at the Michael Fowler Center that included two hours of introductory remarks and a game of bingo-check-your-privilege. Then followed hours of discussion of potential new stats in a range of wellness areas, including spiritual wellness.

New initiatives of this type can be very useful if the core business is not neglected. But while attendees at the Indicators Aotearoa New Zealand event in Wellington were play a clapping gameStatistics NZ was rush the census and the core data infrastructure was breaking down.

the Flags Aotearoa New Zealand website is now live and active – although they have yet to find a way to measure spiritual well-being.

More recently, Statistics NZ has put a lot of effort into better measures of child poverty. Before Covid, the government made child poverty reduction a key part of its welfare agenda. The Child Poverty Reduction Act 2018 requires the government statistician to report annually on ten measures of child poverty. The Agency thus produced a series of working documents explaining how it defined the measures. This too greatly expanded the household economic survey to provide the necessary data.

Trying to measure spiritual well-being is inherently laughable.

Measuring child poverty is certainly not.

But undertaking expensive new projects without taking care of the infrastructure supporting the basic statistics is very risky. The Household Economic Survey, which provides measures of child poverty, was only available through NZ.Stat – and is now only available by special request.

Due to NZ.Stat being shut down without notice, a few other economists and I started to check if we could build an external mirror of Statistics NZ’s Infoshare archive. Losing access to NZ.Stat is bad. Losing Infoshare would be a disaster. And Infoshare is even older and more decrepit than NZ.Stat before the front broke away from it.

Scraping and mirroring the giant spreadsheets underlying Infoshare would be valuable insurance in case Statistics NZ decides to abruptly end this service as well.

The global problem is not limited to Statistics NZ. Upgrading departmental back-end systems that enable both business as usual and new initiatives is never as appealing as announcing new things.

The central government likes to criticize the local government for taking care of all the priorities other than boring upgrades and maintaining underground pipes.

But when the local government fails to fix its pipes, everyone eventually notices sewage flowing into the port. This wastewater is quickly making headlines.

The central government ignored its data pipes and they started breaking. The consequences are just as real as bursting sewage pipes, but they are harder to see.

Perhaps the central government and the local government could for once try to ensure that the basic infrastructure is up to specification before taking on any interesting new initiatives.