Gaining control on data quality step-by-step

KWR’s work on data QC for the Water Framework Directive (KRW)

Over the last years, the insight has grown that the quality and reliability of (for  instance) monitoring data are essential for their usage. In every day practice, however, data unfortunately often are messy and error prone, whether they are collected manually or with the use of sensors. The quality of data not only needs to be sufficiently high ‘on average’, at least as important is that the quality is of a ’guaranteed and known’ level. For these reasons, the guidelines of the European Water Framework  Directive on monitoring state that the use of clear and standard methods and procedures for data quality control (QC) is obligatory.

The QC methods and steps presented to the workshop participants

KWR, TNO and the Dutch Provinces have taken the lead in developing a standard procedure or protocol for QC of groundwater head data. As they were unavailable, as first steps, a framework and methods for QC were developed that is applicable and may be valuable for all sorts of monitoring data. Recently, a new project has started in which the QC framework, methods and protocol are put to the test.

A QC workshop and QC Wizard for the Dutch Provinces

Aim of this new pilot project is to implement, apply and test the methods and procedures that are part of the QC protocol developed. Using the results and experiences gained, the protocol should be evaluated, optimized and finalized. In order to do so, the methods and steps of the QC protocol, that may be rather abstract and difficult for those involved in monitoring in practice, were translated by KWR into a simple but powerful QC tool or QC Wizard. As practical test for the  protocol, a workshop was organized for the Dutch Provinces where the QC Wizard was used to apply the various automated and visual QC checks and steps, and visualize and demonstrate their results and effectiveness.

Although visual data checking is perceived by most (including ourselves!) to be the most powerful instrument for data QC available, the combined use of automated checks and different views and visualizations proves to open up a new world. It proves to be highly effective in discovering various sorts of measurements and patterns that clearly are either erroneous, implausible or questionable. The enthusiasm of the workshop participants means that there is work to do for all in implementing and applying these QC methods and tools. Gaining control on data quality step-by-step, will it lead to a future be where data of high and controlled quality are predominant?