Research of network data mining based on reliability source under big data environment

• Published in: Neural computing & applications 2017-12, Vol.28 (Suppl 1), p.327-335
• Main Authors: Li, Jinhai, He, Youshi, Ma, Yunlei
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial Intelligence; Big Data; Computational Biology/Bioinformatics; Computational Science and Engineering; Computer Science; Data management; Data mining; Data Mining and Knowledge Discovery; Feasibility studies; Image Processing and Computer Vision; Network reliability; Original Article; Probability and Statistics in Computer Science; Search engines
• ISSN: 0941-0643; 1433-3058
• DOI: 10.1007/s00521-016-2349-x

In the era of big data, facing vast amounts of network data, only identifying the reliable data source can the researchers extract the original data that can be used in scientific research. Building reliable network data mining model based on the improvement of PageRank algorithm with applying each improved algorithm. Then the model is divided into three modules: the first, use PageRank and TrustRank to eliminate cheating webpages; then, refine webpages which related to research topic highly by TC-PageRank which combined with the topic relevancy between webpages and weight of time difference; finally, determine the authoritative webpages of the original data source by the improved HITS which considered the influence of the similarity between webpage and research topic and the amplification of webpage links to the authoritative webpages. Meanwhile, the partitioning of matrix operation based on MapReduce reduces the time and space complexity of the algorithms. And the feasibility and accuracy of the method are verified by comparative analysis of the algorithms.