Full text weblog spider. Full text weblog spiders enable gathering of content from a weblog site. Such a weblog spider may consist of two elements. Firstly, a spider component for finding blog sites and blog feeds and, secondly, a component that acquires the list of identified blogs and indexes them extensively to include all available content. This way of capturing is usually contrasted with RSS crawling that is limited to the data distributed via web feeds. Full text crawling provides a possibility of capturing content in full for blog posts, comments, links, attachments, graphics, author information and metadata. Recent developments in the area of weblog crawling The World Wide Web continues to grow and evolve. New technologies, services and standards emerge periodically, affecting the development of the blogosphere. Among the most recent 2 xxxx://xxxxxxxxx.xxx/extend/plugins/search.php?q=feed&sort= 3 xxxx://xxxxxxxxxx.xxx developments that may have direct implications for the development of the BlogForever project are the developments of Microdata, crawling of JavaScript by Google, and HTML5: • Microdata (xxxxxx.xxx) is a standard for enriching web content. This standard is recognised and has been agreed by Microsoft, Google and Yahoo since July 2011. Microdata can be very valuable when crawling the web and weblogs for different items and types of content. • Capturing comments on blog posts often requires running some JavaScript code. This forms one of the challenges when crawling social media websites. However, the recent announcement of Google highlighted the launch of a Google spider that is capable of executing AJAX/JavaScript code and capturing comments on social media sites, including Facebook [1]. The announcement raises a discussion on what the implications of this recent development may be in relation to the blogosphere. It is clear, however, that capturing content that has remained out of the reach of spiders is now becoming accessible. • HTML5 is another substantial development in the area and constitutes a significant improvement on its predecessor version. Although earlier versions of HTML are still being used widely, the situation may change in the near future. The use of HTML5 can enable a more extensive extraction and processing of semantic entities.
