Web-page cleaning Sample Clauses

Web-page cleaning. Apart from a main textual content, a typical web page also contains certain ―noise‖ including navigation links, advertisements, disclaimers, etc. (often called boilerplate) of only limited or no use for the purposes of training an MT system. Such irrelevant parts should be removed and only the main content should be kept in order to produce good-quality language resources. This is the most challenging task of the CNC and special attention will be paid to it in WP4.

Web-page cleaning. Current state of the art in the area of corpus cleaning is briefly described in Spousta et al. (2008): Interest in web page cleaning originated in the area of web mining and search engines (see e.g. Xxxxxx at al. (1999) or Xxx et al. (2000)). Xxx-Xxxxxx and Xxxxxxxxxxx (2002) introduced a notion of 'pagelet' determined by the number of hyperlinks in the HTML element used to segment a web page; pagelets with high frequency of hyperlinks were removed. Xxx and Xx (2002) extracted keywords from each block of text to compute entropy, low-entropy blocks were removed. In Xx et al. (2003) and Xx and Xxx (2003), a tree structure was introduced to capture the common presentation style of web pages and similarly as in the previous work entropy of its elements was computed to determine which element should be removed. Xxxx et al., (2006) proposed another two-stage cleaning method: in the first phase, web pages are segmented into blocks and blocks are clustered according to their layout features, in the second phase, the blocks with similar layout style and content are deleted. Several new methods and approaches were introduced during the CLEANEVAL 2007 contest (xxxx://xxxxxxxxx.xxxxxx.xxx.xx/) organized by the ACL Web as Corpus interest group. Competitive systems used both heuristics (often based on the observations that HTML tag density within boilerplate text is higher than within the main content and that main content is usually longer than boilerplate text) as well as sound machine learning methods (often requiring annotated data to optimize parameters of the systems), such as Support Vector Machines (Bauer et al., 2007), decision trees, genetic algorithms, and language models (Xxxxxxx and Weerkamp, 2007). The best performing system was described in Xxxxx et al. (2007) and later improved by Xxxxxxx et al. (2008). It employes multi-feature sequence labelling of textual blocks based on Conditional Random Fields.