The exact probability law for the approximated similarity from the Minhashing method

We propose a probabilistic setting in which we study the probability law of the Rajaraman and Ullman \textit{RU} algorithm and a modified version of it denoted by \textit{RUM}. These algorithms aim at estimating the similarity index between huge texts in the context of the web. We give a foundation...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2022-09
Main Authors: Dembele, Soumaila, Gane Samb Lo
Format: Article
Language:English
Subjects:
Algorithms
Similarity
Statistical analysis
Webs
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We propose a probabilistic setting in which we study the probability law of the Rajaraman and Ullman \textit{RU} algorithm and a modified version of it denoted by \textit{RUM}. These algorithms aim at estimating the similarity index between huge texts in the context of the web. We give a foundation of this method by showing, in the ideal case of carefully chosen probability laws, the exact similarity is the mathematical expectation of the random similarity provided by the algorithm. Some extensions are given. \noindent \textbf{R\'{e}sum\'{e}.} Nous proposons un cadre probabilistique dans lequel nous \'{e}tudions la loi de probabilit\'{e} de l'algorithme de Rajaraman et Ullman \textit{RU} ainsi qu'une version modifi\'{e}e de cet algorithme not\'{e}e \textit{RUM}. Ces alogrithmes visent \`{a} estimer l'indice de la similarit\'{e} entre des textes de grandes tailles dans le contexte du Web. Nous donnons une base de validité de cette m\'{e}thode en montrant que pour des lois de probabilit\'{e}s minutieusement choisies, la similarit\'{e} exacte est l'esp\'{e}rance math\'{e}matique de la similarit\'{e} al\'{e}atoire donn\'{e}e par l'algorithme \textit{RUM}. Des généralisations sont abordées.
ISSN:2331-8422
DOI:10.48550/arxiv.2209.10031