Examples and methods. WP1 mentions the possibility of relating the demand for specific jobs to specific areas. The location of companies advertising jobs could be used for that. This would enable geographic information about job vacancies to be published. Interesting information could be provided when compare company residence and job vacancy location. They could differ. The topic has however not been studied in WP1. In WP2 geographic information, such as addresses, were collected from enterprise websites. This could be used to update this information in the Business Register. In WP3 geocoded data has been used to link electricity consumption derived from smart meters to geographic areas. The geolocation of a metering point data needed to be derived via the address associated with it. In Estonia this was done at a massive scale via the Estonian Land Board’s web based Massgeocoding service (WP3 Del 3.1). About 90% of the metering point addresses could be geo-coded automatically. In WP4 the major component of the data studied is geolocation data. A subset of the Automatic Information System (AIS) messages includes data on the geolocations of a ship (WP4 Del 4.1). However, because these messages are radio signals there are typical ways in which the signal can be disturbed. This causes errors in the geolocations transmitted, for example, resulting in ships in the Sahara. A cleaning procedure was developed enabling the construction of the journeys of ship and their location in harbours (WP4 Del 4.2, Del 4.3). In WP5 mobile phone data is studied. Here the location of mobile phone masts provide important geo-information on the mobile phones connected. The area covered by a base station, the overlap between such stations and the accuracy by which a phone is connected to a particular masts are important considerations (WP5 internal documents). Such data can, for example, be used to study Day Time Population (Tennekes and Offermans, 2014) or for population density (Xx Xxxxxxxxx et al., 2016). In WP6 road sensors are studied in Slovenia. These have a geolocation assigned indicating their position in the country (WP6 Del 6.1). These have not been linked to other data sources in the WP.
1). Other messages may be assigned by the location of the user (location field) or from the message content (because of the object they describe). In this way topics discussed or sentiment could be assigned to specific areas. When Tourism/border crossings are studied various sources could be used that provid...
Examples and methods. In WP1 the relation between job vacancies and job advertisements is discussed (WP1, Del 1.1). Not every vacancy may result in an online advertisement, some advertisements are placed on multiple web sites, some advertisements may stand for in multiple vacancies and it has even been found that some advertisements are not directly linked to a job. It was found that the data needed to discern between these cases is not always available. It can also be problematic to identify the enterprise unit that advertises the jobs needed. Company name matching is used for that. In WP2 considerable effort was put into retrieving the correct website of a company. This is done in an automated way but requires some manual checking (WP2, Del 2.2). The name of the company and address and location data or important input for the approach developed. In WP3 units in the smart meter dataset refer to so-called metering points. However, how these units relate to an address, a dwelling, a household or a company is not always clear. The electricity consumption data of a metering point was used as a feature to discern between households and companies (WP3 Del 3.2). Some metering points were found to correspond to multiple households or companies.
Examples and methods. No WP in the ESSnet Big Data has reached the stage that a complete Big Data process has been setup, however, but both WP2 and 4 are almost there (WP2 del 2.2; WP4 del 4.3). The only example of a Big Data based statistical process known to date is the Traffic Intensity statistics process of Statistics Netherlands (Stat. Neth., 2015; Puts et al., 2016). The general steps described below apply to each of these examples.
Examples and methods. In the ESSnet Big data no workpackage has applied SMC. The first practical implementation of SMC was in 2008 when 1200 Danish farmers used it to determine the market price of sugarbeets contracts without having to reveal their (sensitive) selling and buying prices and without resorting to an external ty trusted party (Xxxxxxxx et al., 2008).
Examples and methods. WP1 searches for possibility for producing job vacancy statistics based on searching through job portals. WP1 deals with the population of companies which publish job ads online. Do these companies represent the whole business population? A survey in Germany suggests that big companies employ online job ads much more often than small and medium (WP1, Deliverable 1.2). We can reasonably assume online job vacancies survey under covers SMEs and covers big companies well. One way to cope with SME under coverage is to make a job vacancy survey on SME. Combining online and sample survey could improve job vacancy statistics. Another option is to use online job vacancies statistics as they are since the big companies are included. Internet technology develops and it is reasonable to expect more SMEs to hire online. Recently Facebook developed an application to facilitate companies to hire and job seekers. Within Facebook companies could post job ads and job seekers could post job alerts for positions they are interested in. Companies could track application and communicate directly with the applicants. The application is useful for small companies. Web scrapping should spread to social media to grasp job ads posted in the newly developed application. WP2 aims at finding URL addresses of enterprises via web scrapping techniques. The population subject to web scrapping is all companies that have a web site. A URL identification method may be more efficient for companies that have an e-commerce activity. Possibly, URL identification is biased to sites that provide online trade as well. There are at least two different ways to deal with this potential bias component. A first one is, in a full model based approach, to adopt an estimator that makes use of weights obtained by pseudo-calibrating units for which it is possible to scrap their websites with known totals pertaining to the whole population of interest. Another possible solution is, whenever also survey data are available, to adopt a combined estimator that makes use of predicted values for units belonging to the sub-population of units with successfully scraped websites, and of observed values for the sub-population of units with unassessed websites; both components of the estimator can make use, as in the previous case, of weights obtained by calibrating with respect to the two sub-populations. WP3 is a pilot study to produce data about electricity consumption of buildings equipped with smart meters. The popu...
Examples and methods. WP1: machine learning is used for text analysis: specifically they used Python’s gensim model for text analysis and deduplication and similarity functions to determine professions. WP2: enterprise website link retrieval was executed with use of logistic models, random forests and neural nets fitted according to known links. Some use Naïve Bayes classifiers on bag-of-words data for features classification. There are two goals of using machine learning. The one is to determine whether the enterprise uses e-commerce on their website. The second is to determine the type of social media presence of enterprise (e.g., marketing, commercial, enterprise image, etc. based on classification from ICT in enterprises survey). WP3: ARIMA models are used for modelling smart meter data and random forest for estimation/forecasting of vacant living spaces. Cell-wise outlier detection methods were used for anomaly detection. WP4: work is focused on using ships’ AIS data as a source which they do not use in conjunction with any machine learning methods. WP5: data was unable to be collected and therefore no machine learning techniques have been used in this work package. WP6: machine learning is extensively used to integrate different sources of data or to quickly calculate estimates on large data in a timely fashion. A variety of methods are used, from the simplest linear regressions, through different optimization methods such as principal component analysis (PCA, a dimensionality reduction method), to ensemble methods. Every method in the R package Caret was used for testing purposes. WP7: for satellite image classification K-means methods, decision trees, K-Nearest Neighbours and SVM were used while for population happiness a range of classification methods for text analysis were used. Naïve Bayes is used to determine the sentiment for Population use case (e.g., happy, sad, calm, angry, discouraged, and depressed).
Examples and methods. The two approaches mentioned above by which Big Data can be used are observed in the WP’s of the ESSnet Big Data. Usually the focus is (initially) on Big Data as the main source but sometimes Big Data is used as an additional data source. From the description given in table 2 (in section1.1) it’s obvious that WP1, WP3 and WP4 clearly focus on using their Big Data sources, job vacancies on web sites, smart meters and AIS-data, respectively, as the main source of input. In WP2 this seems not to be the case; improving general information on enterprises is mentioned in table 2. WP6 focuses on combining various sources. In WP7 some studies as Big Data as the main source of information. This reveals that a whole range of approaches to infer from Big Data have been used in the ESSnet Big Data.
Examples and methods. The aim of WP1 is to demonstrate which approaches are most suitable to produce statistical estimates in the domain of job vacancies. In on-line job advertisements, there is over-coverage (some advertised vacancies are out of scope for purposes of official statistics) and under-coverage (not all job vacancies are advertised on-line), there can be duplicates (e.g. the same vacancy is advertised in two different job portals), some data can be missing (e.g. closing date) and there can be errors because of misclassification of job advertisements. Job vacancy survey data often differ a lot from on-line job vacancy data. In WP1 the issues affecting accuracy and selectivity for on-line job vacancy data were well understood but no solution has been identified so far. An internal report discussed many quality issues (WP1 2017). A general approach that was explored for assessing selectivity (particularly by the UK) is the idea of linking reporting units in the job vacancy survey to the company names in the on-line data. In principal, if one can understand the differences between the survey at the level of the individual enterprise, then one should be able to better understand the biases in the on-line data. However, this linking process is itself difficult and prone to error. This is discussed in section 4.2.3 of the abovementioned internal document and, in a general framework, in section 2.8 of this report. Furthermore, even for the subset of reporting units where it is possible to achieve a good match the patterns between the survey and the on-line data for individual enterprises are often not consistent. Some examples illustrating this problem are provided by WP1: the time series pattern for the survey data is often very different from on-line data and different sources of on-line data may vary between themselves. Basically, WP1 does not yet have a method for measuring selectivity due to the fact that there are many confounding factors with on-line job vacancy data. WP2 performed web scraping experiments with the aim of deriving experimental statistics on enterprises from information found on the web, especially the websites of enterprises. There were four use cases, referring to URL Retrieval, E-commerce, Job vacancies ads on enterprises’ websites and Social Media Presence. The assessment of accuracy can be carried out in a different way distinguishing the first use case (URL retrieval) from the others. For URLs retrieval, accuracy can be measured at the unit lev...
Examples and methods. The enormous sources of data can be classification in different ways indicating their potential application or may just indicate an overall population of sources. An UNECE task team on big data suggested the following classification: • Social networks (human-sourced information): Facebook Twitter, Blogs and comments, Personal documents, Pictures, Videos, Internet searches, etc. These data are loosely structured; • Traditional Business systems (process-mediated data): these processes record and monitor business events of interest, such as registering a customer, manufacturing a product, taking an order, etc. These sources produces well-structured information; • Internet of Things (machine-generated data): derived from the phenomenal growth in the number of sensors and machines used to measure and record the events and situations in the physical world. The output of these sensors is machine-generated data, and from simple sensor records to complex computer logs, it is well structured. The IoT class has two subclasses: data from sensors and data from computer systems. It is possible to make another BD classification for instance on that’s based on the type of data included, i.e. numbers, texts, pictures, movies or sound. Other classifications may refer to specific needs. Existing statistical classification like ISIC could be of help for a particular need. In all cases, however, the enormous growth in number of potential data sources and the differentiation of these sources clearly indicates that researchers have to learn how to extract information from all kind of sources of which some have never been used for official statistics. For the use of texts and pictures/movies, it’s also clear that a whole range of new methods may be needed. Advantage is that these may be borrowed or derived from areas of science that already have been studying these types of sources for a considerable time. In all WP’s new data sources were studied (see section 1.1).
