Efficient attribute-oriented generalization for knowledge discovery from large databases

• Published in: IEEE transactions on knowledge and data engineering 1998-03, Vol.10 (2), p.193-208
• Main Authors: Carter, C.L., Hamilton, H.J.
• Format: Article
• Language: English
• Subjects: Algorithmics. Computability. Computer arithmetics; Applied sciences; Artificial intelligence; Computer science; Computer science; control theory; systems; Computer Society; Data mining; Exact sciences and technology; Government; Information systems. Data bases; Intelligent robots; Learning and adaptive systems; Memory organisation. Data processing; Partitioning algorithms; Radio access networks; Runtime; Software; Testing; Theoretical computing
• ISSN: 1041-4347; 1558-2191
• DOI: 10.1109/69.683752

We present GDBR (Generalize DataBase Relation) and FIGR (Fast, Incremental Generalization and Regeneralization), two enhancements of Attribute Oriented Generalization, a well known knowledge discovery from databases technique. GDBR and FIGR are both O(n) and, as such, are optimal. GDBR is an online algorithm and requires only a small, constant amount of space. FIGR also requires a constant amount of space that is generally reasonable, although under certain circumstances, may grow large. FIGR is incremental, allowing changes to the database to be reflected in the generalization results without rereading input data. FIGR also allows fast regeneralization to both higher and lower levels of generality without rereading input. We compare GDBR and FIGR to two previous algorithms, LCHR and AOI, which are O(n log n) and O(np), respectively, where n is the number of input tuples and p the number of tuples in the generalized relation. Both require O(n) space that, for large input, causes memory problems. We implemented all four algorithms and ran empirical tests, and we found that GDBR and FIGR are faster. In addition, their runtimes increase only linearly as input size increases, while the runtimes of LCHR and AOI increase greatly when input size exceeds memory limitations.