Data Staging Sample Clauses

Data Staging. When scientific communities use data-intensive applications, their data inputs are typically staged onto computational resources prior to job execution. Following execution, output data sets are generally staged out of compute nodes and onto storage systems. Additional data transfer operations may be required to move intermediate data products from nodes where they are produced to nodes where they will be consumed by other jobs. If the storage space on a node is limited, these stage-out operations need to be performed before other jobs can run. In distributed computing, where resources are geographically dispersed, efficient data staging is essential. Starting with this assumption, the objective of the data staging service is to enable research communities to stage data from EUDAT repositories to external HPC facilities for further processing. The “Data Staging” service can be considered as the fundamental service for exchanging data between the EUDAT domain of globally registered data and the domain of data consumers and producers (e.g. scientific instruments, HPC or HTC facilities). Three EUDAT core communities (VPH, ENES and EPOS) explicitly requested this service as they need to perform statistical model analyses on stored data. This service area differs from the safe replication scenario, which was discussed in the previous section, on three main points.  The replicated data are discarded when the analysis application ends.  The PID references are not applied to data replicated into scratch workspaces.  The users initiate the process of replicating data, whereas in the safe replication scenario, data are replicated automatically on a policy basis. A typical utilization scenario is depicted in Figure 14. 10xxxx://xxxx.xxxxxx.xxx/p/irodspython/issues/detail?id=11 CINECA EPOS 3 1 EUDAT Storage Community Storage 2 2 XXXXX XXXX 3 4 EUDAT Storage HPC Facility HPC Facility PID HPC Facility Figure 14: Dynamic replication utilization scenario steps
Sample 1
AutoNDA by SimpleDocs
Data Staging. Data staging imposes the following requirements on the AAI.  It needs to have EUDAT-federation credentials accepted across community (or non-EUDAT) resources as well as EUDAT. Identity mappings, and in particular permissions, need to be consistent.  The EUDAT federated AAI must provide access to the replicated copies of the DOs.  It should work with target data interfaces at non-EUDAT resources.  It needs support for data transfer protocols (e.g. GridFTP, UFTP, or similar). GridFTP is currently known to be working with X.509 certificates. UFTP, as used by Unicore, should work with certificates. Support for other types of credentials (e.g. XXXX assertions) is highly desirable.  Data staging requires command line support.  GridFTP, being the preferred protocol, suggests the use of X.509 certificates. Indeed, third party copying will normally require RFC3820 “proxies” (or the older versions, GT2 or GT3 – some implementations are sensitive to the version of the proxy).
Sample 1
Data Staging. The Data Staging service works on top of the EUDAT storage services (e.g. iRODS) provided by the Safe Replication service. Due to the nature of the different communities requesting this type of service (namely VPH, EPOS and ENES), different solutions have been evaluated and demonstrated. These solutions ranged from basic iRODS tools and third party transfers via GridFTP, to higher level tools such as Globus On-Line and the XSEDE file manager. To make it possible to access GridFTP from iRODS, iRODS was extended with the Xxxxxxx GridFTP server. The Data Staging service is regarded as a user-level service that allows individual users to easily manage the staging of large amounts of data (that consists of many files) and that results in many file transfers. Consequently the Data Staging service depends on the services developed by the AAI task force to provide federated access to the EUDAT data services. The time plan of the AAI TF, does not predict that a federated AAI solution will be available in the short term. Because the data staging solutions are all based on standard authentication methods (e.g. X509, username/password), it is possible to provide access in a less federated way (e.g. separate EUDAT or local credentials). As well as continuing to work on the selected technologies, the Data Staging TF will focus on developing a practical solution to enable user access during the coming months. During the service-building process, the Data Staging task force collaborated with the following organisations. ▪ PRACE The communities actively participating in the Data Staging task force (VPH and EPOS) also use the PRACE HPC computing facilities. To provide an optimal solution, the EUDAT data service should be accessible from the PRACE supercomputers. At the moment, EUDAT and PRACE are holding ongoing strategic discussions for the purpose of collaborating. At time of writing, EUDAT and PRACE had agreed to conduct some pilots. ▪ XSEDE XSEDE is the successor of the TeraGrid project in the US, and is funded by the National Science Foundation. The XSEDE organization develops and maintains the XSEDE file manager. The XSEDE file manager is the tool to use for moving large amounts of data between XSEDE supercomputers. XXXXX has collaborated with XSEDE to adopt the XSEDE file manager to make it work with the EUDAT data services.
Sample 1
Data Staging. The question that remains is how to integrate the architecture with the outside world. We have already seen that we make use of external infrastructures for AAI. For the moment, we assume data is being transferred with GridFTP. In this case, three entities must have X.509 certificates (and private keys, and credentials): the source, the destination, and the controller (namely the user client). One or more of these could be the same. But, more importantly, the certificates must be signed by CAs that is trusted by each entity. This means that EUDAT cannot necessarily run its own CA and expect PRACE (say) to accept it44. The proposal of the AAI TF is to secure the infrastructure itself with known and trusted certificates from NRENs, IGTF, and commercial CAs (particularly for browser-facing services) – these are all (usually) trusted by e- and cyber-infrastructures across the world. However, earlier we proposed to generate federation credentials on behalf of users, to be used internally in EUDAT. Such a service, if X.509 based, would most likely not be trusted outside of EUDAT (see footnote 44 on page 43). Consequently, data staging would have to be done by an automated service with a trusted certificate. However, it would also need to preserve, or at least respect, permissions and rights associated with the user. 42 The case is equivalent in general for command line login. 43 See e.g. the IETF SCIM working group. Moonshot (ABFAB) will also be a potential alternative. 44 That is not to say this is impossible: it can be done but it requires a lot of work (defining a CP/CPS, see RFC3647), buying a Hardware Signing Module for protection of an online CA; checking and validating the CA against the IGTF SLCS profile, and seeking accreditation – the latter process can take 2-3 years – which is not compatible with the timescale of this project.
Sample 1
Data Staging. Creation of a database that contains all the data in the format required by the new system. This data is updated regularly and can be used by the legacy application as well as the new system, thus providing a two-fold benefit to the agency.
Sample 1
Data Staging. The Data Staging service is designed to facilitate transfers of large data sets to and from the EUDAT infrastructure. The building of this service has been continuing since the start of the project, and reached a high level of maturity recently when its core component was completely renovated to overcome performance limitations experienced with the previous technology. As reported in deliverable D5.2.1, the EUDAT Data Staging service was originally based on the Xxxxxxx technology, a GridFTP server – written entirely in JAVA  being able to interact with iRODS resources. After a long period of testing, and upgrading the underlying iRODS Java library (Jargon37) to the latest release, serious performance issues were revealed that significantly reduced the benefits of GridFTP (see Annex I for further technical information)38,39. To address this issue, CINECA has build a new component in order to simplify the software stack (the Java layer causing a big transfer overhead was removed), optimize the performance of the transfer, and thus make the software more stable and reliable. The new component was developed in accordance with the GridFTP Data Storage Interface (DSI) specifications which allow a GridFTP server to be a transfer interface to as many data storage systems as possible, like iRODS. Besides the work done on the server side of the service, the Data Staging script was also extended to support new functionalities such as the capability to handle data using its assigned PIDs. The Data Staging service is currently employed by the VPH community to ingest and stage their data sets onto the EPCC storage resources that are part of the EUDAT federation. 33 Home page iRODS: xxxx://xxx.xxxxx.xxx 34 xxxxx://xxxxxxxxxx.xxx.xx/download/attachments/28836527/EUDAT-DEL-WP2-D2.2.2-EUDAT_Services_Rolling_Plan_2012- 2015_%28update%29.pdf 35 xxxx://xxxx.xxx.xxxxxx.xxx/AmazonS3/latest/API/Welcome.html 36 xxxx://xxx.xxxx.xxx/tech_activities/standards/curr_standards/cdmi 37 xxxxx://xxx.xxxxx.xxx/index.php/Jargon 38 xxxxx://xxxx.xxxxx.xxx/gf/project/jargon/tracker/?action=TrackerItemEdit&tracker_item_id=1243&start=0 39 xxxxx://xxxxxx.xxxxxx.xxx/forum/#!topic/irod-chat/tgtsvVXDCNI The following subsections report further details about the new DSI GridFTP component and the extensions made to the Data Staging script respectively.
Sample 1

Related to Data Staging

  • Data Storage Where required by applicable law, Student Data shall be stored within the United States. Upon request of the LEA, Provider will provide a list of the locations where Student Data is stored.

  • Links If The Services are made available through the Internet, the Financial Institution’s website may provide links to other websites, including those of Third Parties who may also provide services to You. You acknowledge that all those other websites and Third Party services are independent from the Financial Institution’s and may be subject to separate agreements that govern their use. The Financial Institution and Central 1 have no liability for those other websites or their contents or the use of Third Party services. Links are provided for convenience only, and You assume all risk resulting from accessing or using such other websites or Third Party services.

  • MOBILE DATA COMPUTERS 20 1. As part of the law enforcement services to be provided to CITY, COUNTY 21 has provided, or will provide, mobile data computers (hereinafter called 22 “MDCs”) that are or will be mounted in patrol vehicles and motorcycles, 23 designated by COUNTY for use within CITY limits.

  • Network Access TENANT may find it necessary to purchase a network interface card, wireless PC card or other hardware in order to connect to the internet service. LANDLORD is not responsible for the purchase of these items and LANDLORD cannot guarantee compatibility with any device TENANT may have. The computer and network card must have software installed that supports the Internet Protocol commonly referred to as TCP/IP. Any conflicts between the software compatibility of the network and the TENANT’S computer operating system or any other feature will be the responsibility of the TENANT to resolve. LANDLORD will not be responsible for software issues related to the user’s personal computer.

  • Infrastructure Vulnerability Scanning Supplier will scan its internal environments (e.g., servers, network devices, etc.) related to Deliverables monthly and external environments related to Deliverables weekly. Supplier will have a defined process to address any findings but will ensure that any high-risk vulnerabilities are addressed within 30 days.

  • Network Etiquette You are expected to abide by the generally accepted rules of network etiquette. These include but are not limited to the following:

  • LIDB 10.1 BellSouth LIDB stores current information on working telephone numbers and billing account numbers.

  • Network Interconnection Architecture Each Party will plan, design, construct and maintain the facilities within their respective systems as are necessary and proper for the provision of traffic covered by this Agreement. These facilities include but are not limited to, a sufficient number of trunks to the point of interconnection with the tandem company, and sufficient interoffice and interexchange facilities and trunks between its own central offices to adequately handle traffic between all central offices within the service areas at P.01 grade of service or better. The provisioning and engineering of such services and facilities will comply with generally accepted industry methods and practices, and will observe the rules and regulations of the lawfully established tariffs applicable to the services provided.

  • Configuration Management The Contractor shall maintain a configuration management program, which shall provide for the administrative and functional systems necessary for configuration identification, control, status accounting and reporting, to ensure configuration identity with the UCEU and associated cables produced by the Contractor. The Contractor shall maintain a Contractor approved Configuration Management Plan that complies with ANSI/EIA-649 2011. Notwithstanding ANSI/EIA-649 2011, the Contractor’s configuration management program shall comply with the VLS Configuration Management Plans, TL130-AD-PLN-010-VLS, and shall comply with the following:

  • Network PHARMACY is a retail, mail order or specialty pharmacy that has a contract to accept our pharmacy allowance for prescription drugs and diabetic equipment or supplies covered under this plan. NETWORK PROVIDER is a provider that has entered into a contract with us or other Blue Cross and Blue Shield plans. For pediatric dental care services, network provider is a dentist that has entered into a contract with us or participates in the Dental Coast to Coast Network. For pediatric vision hardware services, a network provider is a provider that has entered into a contract with EyeMed, our vision care service manager.

Time is Money Join Law Insider Premium to draft better contracts faster.