Interactive Data Analysis Sample Clauses

Interactive Data Analysis. XXX describes a scientist interacting with a dataset in real time to perform their analyses. That is, they type commands or manipulate controls, and observe the results that are produced or the figures that are displayed. Contrast this with batch processing, discussed in §2.3.7. The processes and tools required for XXX differ substantially from field to field and from facility to facility. For example, the way that data from the Square Kilometre Array (SKA) will be analyzed is very different to the processes applied to data from the Large Hadron Collider (LHC). It is therefore essential that ESAP implement a flexible capability for interfacing with a variety of XXX services. The architecture described in §2.3.1, together with the data management system described in §2.3.3, are de‐ signed to make this possible. Specifically, this will be implemented by developing APIs through which ESAP can provide elements of the “shopping basket” to the XXX system — including both data and software specifi‐ cations — and then by accepting appropriately authenticated updates from the XXX system as the user saves their analysis. The expectation is that the XXX system will write substantial data products (such as output im‐ ages) to bulk storage (such as the DIOS data lake), and return references to them to ESAP for further analysis.

Interactive Data Analysis. ‌ The current version of ESAP directly embeds information which enables it to execute a limited number of workflows, described in the form of Jupyter notebooks. A user interface exists to make it possible to execute these notebooks on a number of different analysis systems, but the existing test system only provides access to MyBinder12. Upon choosing an appropriate workflow and analysis service, the user is redirected to the notebook environ‐ ment. In that environment, a Python library — initially developed specifically to address Zooniverse classifica‐ tion data, but now adapted to a wide range of data types — makes it possible for them to access their ESAP shopping basket, and hence to download or otherwise manipulate the data that they have selected. Selecting the workflow and execution service, then working in the notebook, are shown in Fig. 7. 12xxxxx://xxx.xxxxxxxx.xxx/ There some usability and polish issues remain outstanding regarding shopping basket integration in the note‐ book: currently, the user must manually use the Python API to access data, including supplying an appro‐ priate authentication token (copied‐and‐pasted from the ESAP interface). Ultimately, this process should be improved: data should be automatically staged to the notebook system, and no additional authentication should be required. This system is not yet integrated with the WP3 OSSR, or any other external software or service collections. Discussions are currently underway between WP3 and WP5 to clarify these interfaces. When complete, this integration will provide a much richer experience for the user to discover and schedule workflows. Although the current XXX system focuses on Jupyter notebooks, we expect to extend this service to address other forms of interaction in future work, likely making use of the Rosetta system (§2.4).