Lite2: A Schemaless Zero-Copy Serialization Format

In the field of data transmission and storage, serialization formats play a crucial role by converting complex data structures into a byte stream that can be easily stored, transmitted, and reconstructed. Despite the myriad available serialization formats, ranging from JSON to Protobuf, each has lim...

Full description

Saved in:
Bibliographic Details
Published in:Computers (Basel) 2024-04, Vol.13 (4), p.89
Main Authors: Chen, Tianyi, Guan, Xiaotong, Shuai, Shi, Huang, Cuiting, Aibin, Michal
Format: Article
Language:English
Subjects:
B trees
Communication
Copying
data formats
data serialization
Data structures
Data transmission
Flexibility
Format
JavaScript
Popularity
Programming languages
Protocol
Python
schemaless
zero-copy
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In the field of data transmission and storage, serialization formats play a crucial role by converting complex data structures into a byte stream that can be easily stored, transmitted, and reconstructed. Despite the myriad available serialization formats, ranging from JSON to Protobuf, each has limitations, particularly in balancing schema flexibility, performance, and data copying overhead. This paper introduces Lite2, a novel data serialization format that addresses these challenges by combining schemaless flexibility with the efficiency of zero-copy operations for flat or key–value pair data types. Unlike traditional formats that often require a predefined schema and involve significant data copying during serialization and deserialization, Lite2 offers a dynamic schemaless approach that eliminates unnecessary data copying, optimizing system performance and efficiency. Built upon a contiguously stored B-tree structure, Lite2 enables efficient data lookup and modification without deserialization, thereby achieving zero-copy operations.
ISSN:2073-431X
2073-431X
DOI:10.3390/computers13040089