Index–Trie: Efficient archival and retrieval of network traffic

• Published in: Computer networks (Amsterdam, Netherlands : 1999) Netherlands : 1999), 2017-09, Vol.124, p.140-156
• Main Authors: Xie, Gaogang, Su, Jingxiu, Wang, Xin, He, Taihua, Zhang, Guangxing, Uhlig, Steve, Salamatian, Kave
• Format: Article
• Language: English
• Subjects: Archive; Archiving; Communications traffic; Computer centers; Computer Science; Flow; Indexing; Information retrieval; Network security; Networking and Internet Architecture; On-line systems; Packet; Retrieval; Retrieve; Traffic control; Traffic engineering; Traffic speed
• ISSN: 1389-1286; 1872-7069
• DOI: 10.1016/j.comnet.2017.06.010

Historical network traffic traces, both at the flow and packet level, play a significant role in many research and engineering areas, such as network security, traffic engineering and accounting. To retrieve the specific entries at a higher speed from large traces, each packet or flow should be indexed using multiple query fields during archiving. This brings challenges both in terms of archiving speed and storage consumption. We propose a network traffic indexing and querying method based on Index–Trie, to achieve fast archiving, low storage space of the indexes, and fast retrieval. We implemented a system for online trace archival and retrieval. Our experiments, performed both offline and online on backbone, campus and datacenter network traffic, demonstrate that our method outperforms the popular FastBit method. For packet traces, the Index–Trie based method can obtain an improvement up to 72% for the archiving rate, 56% lower storage consumption, and 14 times faster retrieving time. For flow traces, compared to FastBit, our system is up to 15 times faster in term of the archiving rate, 42% less storage, and 100 times faster retrieving speed. Furthermore, we extend the application of Index-Tries to log file indexing and retrieving.