Local government data warehouse

2019.11.25

In the ASP service center, the state provides IT support for the most important components of local government operation and the possibility for electronic administration for close to 3200 municipalities and their customers. The owner of the ASP system, the Hungarian Treasury (MÁK), as per the relevant government directive of February 2018, launched a new development: they wanted to create a data warehouse, whose sources are the special local government systems working in ASP, the local governments obtaining exemption from using it, operating on an interface basis, and certain specialized MÁK systems.
„The task of this warehouse is, among others, to provide support for local government decision preparation, developing central and local analyses, and for the timely recognition of indebtedness processes. Its users consist not only of municipalities and MÁK employees, but also relevant ministries, and colleagues of KSH and the State Audit Office”, told Gabriella Vikárius,  MÁK subproject manager, about the project.
During the development, several challenges had to be tackled. Although the data warehouse itself receives and processes the data of “only” nine source systems – of which the biggest are specialized local tax and management systems – still, it integrates data from close to 3200 local governments, coming from 52 systems of various development backgrounds. In case of the special management system, the number of independent organizations exceeds 11 thousand, since it also contains data from local government budgetary institutions and associations. Sometimes, 50 thousand data packages are coming in at once.
The so called “interfaced local governments” represented a significant integration and consolidation challenge. 39 large cities and districts in the capital could keep their specialized systems, with the permission of the Ministry of Interior (BM), and did not have to join ASP. But the data provision obligations still apply to these local governments: they have to send, on  a daily basis, data to the local government data warehouse, via their proprietary interfaces. This was authorized, if the given local government could prove that they were able to satisfy the above conditions. „Because of this, already by the beginning of the summer of 2018 we had to come up with a prototype, in which we could check whether the local governments in question could continuously send data in a structured way, according to the required formats. This, however, meant that during the project, close cooperation and communication was needed with the vendors of altogether 52 source systems of different development backgrounds. We made progress through a multi-round pilot, in line with the agile methodology, until the results were perfect”, explained Zsolt Frunza, Hungarian project manager of T-Systems. 
The result was a hybrid data warehouse, unique on a regional level, in which the classic data warehouse technology was combined with Big Data technologies. The latter support not only storing data, but also their transformation and processing.
The data warehouse works in the Government Data Center (KAK) operated by the National Infocommunication Service Company Ltd. (NISZ Zrt.), and on its cloud infrastructure, added Attila Csaba Sárközi, senior expert at NISZ Zrt. 
While setting up the data warehouse, NISZ Zrt. constructed the hardware and virtualization infrastructure of the data warehouse, supported its completion and integration with the ASP Framework system, operated by them.
The local government data warehouse was the first similar system in KAK, and since it requires an especially great bandwidth and processing capacity, it often provided important information for fine-tuning the infrastructure of the data center as well.

Each year, T-Systems nominates three projects for the Project of the Year award, that are considered larger than average, not only due to their value or volume, but because they stand out on the basis of their novelty or complexity, representing a real professional challenge.