A fuzzy approach for ambiguity reducing in text similarity estimation (case study: Persian web contents)
محورهای موضوعی : Natural Language Processing
حمید آهنگربهان
1
,
غلامعلی منتظر
2
1 -
2 - دانشگاه تربیت مدرس
کلید واژه: Text similarity, Similarity Metric, Fuzzy Sets, Lexical Similarity, Structural Similarity, Persian Text, , ,
چکیده مقاله :
Finding similar web contents have great efficiency in academic community and software systems. There are many methods and metrics in literature to measure the extent of text similarity among various documents and some its application especially in plagiarism detection systems. However, most of them do not take ambiguity inherent in word or text pair’s comparison as well as structural features into account. As a result, pervious methods did not have enough accuracy to deal vague information. So using structural features and considering ambiguity inherent word improve the identification of similar contents. In this paper, a new method has been proposed that taking lexical and structural features in text similarity measures into consideration. After preprocessing and removing stopwords, each text was divided into general words and domain-specific knowledge words. Then, the two lexical and structural fuzzy inference systems were designed to assess lexical and structural text similarity. The proposed method has been evaluated on Persian paper abstracts of International Conference on e-Learning and e-Teaching (ICELET) Corpus. The results shows that the proposed method can achieve a rate of 75% in terms of precision and can detect 81% of the similar cases.
Finding similar web contents have great efficiency in academic community and software systems. There are many methods and metrics in literature to measure the extent of text similarity among various documents and some its application especially in plagiarism detection systems. However, most of them do not take ambiguity inherent in word or text pair’s comparison as well as structural features into account. As a result, pervious methods did not have enough accuracy to deal vague information. So using structural features and considering ambiguity inherent word improve the identification of similar contents. In this paper, a new method has been proposed that taking lexical and structural features in text similarity measures into consideration. After preprocessing and removing stopwords, each text was divided into general words and domain-specific knowledge words. Then, the two lexical and structural fuzzy inference systems were designed to assess lexical and structural text similarity. The proposed method has been evaluated on Persian paper abstracts of International Conference on e-Learning and e-Teaching (ICELET) Corpus. The results shows that the proposed method can achieve a rate of 75% in terms of precision and can detect 81% of the similar cases.
