Confidence measure estimation for Open Information Extraction
محورهای موضوعی : Machine learningVahideh Reshadat 1 , Maryam hourali 2 , Heshaam Faili 3
1 - Malek-Ashtar University of Technology
2 - Malek-Ashtar university of Technology
3 - Tehran University
کلید واژه: Information Extraction, , Open Information Extraction, , Relation Extraction, , Knowledge discovery, , Fact Extraction,
چکیده مقاله :
The prior relation extraction approaches were relation-specific and supervised, yielding new instances of relations known a priori. While effective, this model is not applicable in case when the number of relations is high or where the relations are not known a priori. Open Information Extraction (OIE) is a relation-independent extraction paradigm designed to extract relations directly from massive and heterogeneous corpora such as Web. One of the main challenges for an Open IE system is estimating the probability that its extracted relation is correct. A confidence measure shows that how an extracted relation is a correct instance of a relation among entities. This paper proposes a new method of confidence estimation for OIE called Relation Confidence Estimator for Open Information Extraction (RCE-OIE). It investigates the incorporation of some proposed features in assigning confidence metric using logistic regression. These features consider diverse lexical, syntactic and semantic knowledge and also some extraction properties such as number of distinct documents from which extractions are drawn, number of relation arguments and their types. We implemented proposed confidence measure on the Open IE systems’ extractions and examined how it affects the performance of results. Evaluations show that incorporation of designed features is promising and the accuracy of our method is higher than the base methods while keeping almost the same performance as them. We also demonstrate how semantic information such as coherence measures can be used in feature-based confidence estimation of Open Relation Extraction (ORE) to further improve the performance.
The prior relation extraction approaches were relation-specific and supervised, yielding new instances of relations known a priori. While effective, this model is not applicable in case when the number of relations is high or where the relations are not known a priori. Open Information Extraction (OIE) is a relation-independent extraction paradigm designed to extract relations directly from massive and heterogeneous corpora such as Web. One of the main challenges for an Open IE system is estimating the probability that its extracted relation is correct. A confidence measure shows that how an extracted relation is a correct instance of a relation among entities. This paper proposes a new method of confidence estimation for OIE called Relation Confidence Estimator for Open Information Extraction (RCE-OIE). It investigates the incorporation of some proposed features in assigning confidence metric using logistic regression. These features consider diverse lexical, syntactic and semantic knowledge and also some extraction properties such as number of distinct documents from which extractions are drawn, number of relation arguments and their types. We implemented proposed confidence measure on the Open IE systems’ extractions and examined how it affects the performance of results. Evaluations show that incorporation of designed features is promising and the accuracy of our method is higher than the base methods while keeping almost the same performance as them. We also demonstrate how semantic information such as coherence measures can be used in feature-based confidence estimation of Open Relation Extraction (ORE) to further improve the performance.
[1] L. Yao, A. Haghighi, S. Riedel, and A. McCallum, "Structured relation discovery using generative models," in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2011, pp. 1456-1466.
[2] J. Piskorski and R. Yangarber, "Information extraction: Past, present and future," in Multi-source, Multilingual Information Extraction and Summarization, ed: Springer, 2013, pp. 23-49.
[3] B. Min, S. Shi, R. Grishman, and C.-Y. Lin, "Towards Large-Scale Unsupervised Relation Extraction from the Web," International Journal on Semantic Web and Information Systems (IJSWIS), vol. 8, pp. 1-23, 2012.
[4] R. C. Bunescu and R. J. Mooney, "A shortest path dependency kernel for relation extraction," in Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, 2005, pp. 724-731.
[5] A. Culotta, A. McCallum, and J. Betz, "Integrating probabilistic extraction models and data mining to discover relations and patterns in text," in Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, 2006, pp. 296-303.
[6] N. Kambhatla, "Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations," in Proceedings of the ACL 2004 on Interactive poster and demonstration sessions, 2004, p. 22.
[7] M. Banko, O. Etzioni, and T. Center, "The Tradeoffs Between Open and Traditional Relation Extraction," in ACL, 2008, pp. 28-36.
[8] C. C. Xavier, V. L. S. de Lima, and M. Souza, "Open information extraction based on lexical semantics," Journal of the Brazilian Computer Society, vol. 21, p. 4, 2015.
[9] M. Schmitz, R. Bart, S. Soderland, and O. Etzioni, "Open language learning for information extraction," in Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, 2012, pp. 523-534.
[10] S. Soderland, B. Roof, B. Qin, S. Xu, and O. Etzioni, "Adapting open information extraction to domain-specific relations," AI Magazine, vol. 31, pp. 93-102, 2010.
[11] P. Gamallo and M. Garcia, "Multilingual open information extraction," in Portuguese Conference on Artificial Intelligence, 2015, pp. 711-722.
[12] V. Reshadat, M. Hoorali, and H. Faili, "A Hybrid Method for Open Information Extraction Based on Shallow and Deep Linguistic Analysis," Interdisciplinary Information Sciences, vol. 22, pp. 87-100, 2016.
[13] L. Del Corro and R. Gemulla, "ClausIE: clause-based open information extraction," in Proceedings of the 22nd international conference on World Wide Web, 2013, pp. 355-366.
[14] A. Culotta and A. McCallum, "Confidence estimation for information extraction," in Proceedings of HLT-NAACL 2004: Short Papers, 2004, pp. 109-112.
[15] A. McCallum and D. Jensen, "A note on the unification of information extraction and data mining using conditional-probability, relational models," 2003.
[16] F. Wu and D. S. Weld, "Open information extraction using Wikipedia," in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 2010, pp. 118-127.
[17] O. Etzioni, A. Fader, J. Christensen, S. Soderland, and M. Mausam, "Open Information Extraction: The Second Generation," in IJCAI, 2011, pp. 3-10.
[18] L. Qiu and Y. Zhang, "Zore: A syntax-based system for chinese open relation extraction," in Proceedings of EMNLP, 2014.
[19] Y.-H. Tseng, L.-H. Lee, S.-Y. Lin, B.-S. Liao, M.-J. Liu, H.-H. Chen, et al., "Chinese open relation extraction for knowledge acquisition," EACL 2014, p. 12, 2014.
[20] C. Castella Xavier, S. de Lima, V. Lúcia, and M. Souza, "Open information extraction based on lexical-syntactic patterns," in Intelligent Systems (BRACIS), 2013 Brazilian Conference on, 2013, pp. 189-194.
[21] P. Cimiano and J. Wenderoth, "Automatically learning qualia structures from the web," in Proceedings of the ACL-SIGLEX workshop on deep lexical acquisition, 2005, pp. 28-37.
[22] A. Akbik and J. Broß, "Wanderlust: Extracting semantic relations from natural language text using dependency grammar patterns," in WWW Workshop, 2009.
[23] A. Akbik and A. Löser, "Kraken: N-ary facts in open information extraction," in Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction, 2012, pp. 52-56.
[24] F. Mesquita, J. Schmidek, and D. Barbosa, "Effectiveness and efficiency of open relation extraction," Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, vol. 500, pp. 447–457, 2013.
[25] H. Bast and E. Haussmann, "Open information extraction via contextual sentence decomposition," in Semantic Computing (ICSC), 2013 IEEE Seventh International Conference on, 2013, pp. 154-159.
[26] H. Bast and E. Haussmann, "More informative open information extraction via simple inference," in Advances in information retrieval, ed: Springer, 2014, pp. 585-590.
[27] A. Fader, S. Soderland, and O. Etzioni, "Identifying relations for open information extraction," in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2011, pp. 1535-1545.
[28] J. Christensen, S. Soderland, and O. Etzioni, "An analysis of open information extraction based on semantic role labeling," in Proceedings of the sixth international conference on Knowledge capture, 2011, pp. 113-120.
[29] V. Punyakanok, D. Roth, and W.-t. Yih, "The importance of syntactic parsing and inference in semantic role labeling," Computational Linguistics, vol. 34, pp. 257-287, 2008.
[30] R. Johansson and P. Nugues, "The effect of syntactic representation on semantic role labeling," in Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, 2008, pp. 393-400.
[31] M. H. Kim and P. Compton, "Improving open information extraction for informal web documents with ripple-down rules," in Knowledge Management and Acquisition for Intelligent Systems, ed: Springer, 2012, pp. 160-174.
[32] P. Gamallo, M. Garcia, and S. Fernández-Lanza, "Dependency-based open information extraction," in Proceedings of the Joint Workshop on Unsupervised and Semi-Supervised Learning in NLP, 2012, pp. 10-18.
[33] P. G. Otero and I. G. López, "A grammatical formalism based on patterns of part of speech tags," International journal of corpus linguistics, vol. 16, pp. 45-71, 2011.
[34] M. Banko, M. J. Cafarella, S. Soderland, M. Broadhead, and O. Etzioni, "Open information extraction for the web," in IJCAI, 2007, pp. 2670-2676.
[35] T. Scheffer, C. Decomain, and S. Wrobel, "Active hidden markov models for information extraction," in Advances in Intelligent Data Analysis, ed: Springer, 2001, pp. 309-318.
[36] D. Downey, O. Etzioni, and S. Soderland, "A probabilistic model of redundancy in information extraction," DTIC Document2006.
[37] E. Agichtein, "Confidence estimation methods for partially supervised relation extraction," in Proc. of SIAM Intl. Conf. on Data Mining (SDM06), 2006.
[38] F. Mesquita, "Clustering techniques for open relation extraction," in Proceedings of the on SIGMOD/PODS 2012 PhD Symposium, 2012, pp. 27-32.
[39] M. Röder, A. Both, and A. Hinneburg, "Exploring the space of topic coherence measures," in Proceedings of the eighth ACM international conference on Web search and data mining, 2015, pp. 399-408.
[40] D. M. Powers, "Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation," 2011.
