TsQuality: Measuring Time Series Data Quality in Apache IoTDB

Time series has been found with various data quality issues, e.g., owing to sensor failure or network transmission errors in the Internet of Things (IoT). It is highly demanded to have an overview of the data quality issues on the millions of time series stored in a database. In this demo, we design...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings of the VLDB Endowment 2023-08, Vol.16 (12), p.3982-3985
Main Authors: Qiu, Yuanhui, Fang, Chenguang, Song, Shaoxu, Huang, Xiangdong, Wang, Chen, Wang, Jianmin
Format: Article
Language:English
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Time series has been found with various data quality issues, e.g., owing to sensor failure or network transmission errors in the Internet of Things (IoT). It is highly demanded to have an overview of the data quality issues on the millions of time series stored in a database. In this demo, we design and implement TsQuality, a system for measuring the data quality in Apache IoTDB. Four time series data quality measures, completeness, consistency, timeliness, and validity, are implemented as functions in Apache IoTDB or operators in Apache Spark. These data quality measures are also interpreted by navigating dirty points in different granularity. It is also well-integrated with the big data eco-system, connecting to Apache Zeppelin for SQL query, and Apache Superset for an overview of data quality.
ISSN:2150-8097
2150-8097
DOI:10.14778/3611540.3611601