Data ingestion into OKA database Sample Clauses

Data ingestion into OKA database. The second API handles the parsing and upload of EAR metrics from the CSV files into OKA Elasticsearch database. This API must be called with a cluster parameter which is the name of the cluster where the job ran. The data parameter of the HTTP request sent to the API must contain the csv data passed on using the application and loops keys. The data parameter also contains complementary job features like the size of the input bucket and the application of the job (passed on using the job_features key). This API will be called at the end of each job execution and both application and loops data of a job must be sent at the same time in the API call. This API is available through the /ingest_ear_files route. It executes the following steps: • Get cluster name from the cluster parameter. • Retrieve last accounting logs from the job scheduler of the selected cluster. This is done through OKA by running a pipeline that calls the job scheduler to get the logs The HEROES project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 956874. The JU receives support from the European Union's Horizon 2020 research and innovation programme and France, Spain, Italy. and parses them into the right format to be saved into OKA database. This pipeline gets accounting logs starting from the last job submit date saved in OKA database to avoid reloading all the jobs history at each call. • Parse and enhance application data. New features are computed from the application and loops data: o Consumed energy, using power and execution time contained in application data. o Minimum/maximum power consumption, using loops data. • Save the enhanced application data into OKA database. Application data are saved into a specific index in OKA Elasticsearch database. • Update the job accounting logs. From the enhanced energy data, multiple features are added to the job accounting data retrieve in the second step: o Maximum, minimum and average power consumption are averaged between job steps and summed across all nodes. o Energy consumption is accumulated between job steps and nodes. o Percentage of time the job spent in a computational phase: memory bound, CPU bound or mix bound (not memory or CPU bound) or doing IO (IO bound). o The main phase where the job spent most of its time. o Multiple IDs related to the HEROES platform: HEROES_ORGANIZATION_ID, HEROES_USER_ID, HEROES_TEMPLATE_WORKFLOW_ID and HEROES_INSTANCE_WORKFLOW_ID. Th...