Development challenges and future work. The spider prototype is the first version of the BlogForever spider. The platform is based upon well- known methods and modules. However, further tools, as well as tuning and scaling, are needed. The developed prototype captures and populates a repository with weblog content using a combination of weblog specific technologies such as ping servers and RSS feed monitoring, as well as traditional web data extraction techniques such as scheduled crawls. The key challenge of the objective is to capture and identify automatically all type of content elements and metadata across the variation of blog hosts. Analysing blog pages and extracting information is demanding both on resources as well as on handling all types of exceptions. For instance, blog comments and some blog elements are not exposed to the spiders due to JavaScript coding. Another related challenge is scaling. The more analysis is required for finding and identifying blog elements, the longer processing and more hardware is required. This can be handled through multiple servers, but the objective for the prototype has been to enable this on a single server. One of the key challenges of next development is the implementation of the multi-server architecture. The prototype spider handles the above challenges, although further work is required to improve both analysis and scalability, as described in Section 4.7. This will be done in the next stages of the project in WP4, especially Task 4.2 (Design of the weblog spider component) and Task 4.3 (Implementation of the weblog spider component). Further work planned towards WP4 includes:
