Data Staging Sample Clauses

Data Staging. The Data Staging service is designed to facilitate transfers of large data sets to and from the EUDAT infrastructure. The building of this service has been continuing since the start of the project, and reached a high level of maturity recently when its core component was completely renovated to overcome performance limitations experienced with the previous technology. As reported in deliverable D5.2.1, the EUDAT Data Staging service was originally based on the Xxxxxxx technology, a GridFTP server – written entirely in JAVA  being able to interact with iRODS resources. After a long period of testing, and upgrading the underlying iRODS Java library (Jargon37) to the latest release, serious performance issues were revealed that significantly reduced the benefits of GridFTP (see Annex I for further technical information)38,39. To address this issue, CINECA has build a new component in order to simplify the software stack (the Java layer causing a big transfer overhead was removed), optimize the performance of the transfer, and thus make the software more stable and reliable. The new component was developed in accordance with the GridFTP Data Storage Interface (DSI) specifications which allow a GridFTP server to be a transfer interface to as many data storage systems as possible, like iRODS. Besides the work done on the server side of the service, the Data Staging script was also extended to support new functionalities such as the capability to handle data using its assigned PIDs. The Data Staging service is currently employed by the VPH community to ingest and stage their data sets onto the EPCC storage resources that are part of the EUDAT federation. 33 Home page iRODS: xxxx://xxx.xxxxx.xxx 34 xxxxx://xxxxxxxxxx.xxx.xx/download/attachments/28836527/EUDAT-DEL-WP2-D2.2.2-EUDAT_Services_Rolling_Plan_2012- 2015_%28update%29.pdf 35 xxxx://xxxx.xxx.xxxxxx.xxx/AmazonS3/latest/API/Welcome.html 36 xxxx://xxx.xxxx.xxx/tech_activities/standards/curr_standards/cdmi 37 xxxxx://xxx.xxxxx.xxx/index.php/Jargon 38 xxxxx://xxxx.xxxxx.xxx/gf/project/jargon/tracker/?action=TrackerItemEdit&tracker_item_id=1243&start=0 39 xxxxx://xxxxxx.xxxxxx.xxx/forum/#!topic/irod-chat/tgtsvVXDCNI The following subsections report further details about the new DSI GridFTP component and the extensions made to the Data Staging script respectively.
Sample 1
AutoNDA by SimpleDocs
Data Staging. Creation of a database that contains all the data in the format required by the new system. This data is updated regularly and can be used by the legacy application as well as the new system, thus providing a two-fold benefit to the agency.
Sample 1
Data Staging. The Data Staging service works on top of the EUDAT storage services (e.g. iRODS) provided by the Safe Replication service. Due to the nature of the different communities requesting this type of service (namely VPH, EPOS and ENES), different solutions have been evaluated and demonstrated. These solutions ranged from basic iRODS tools and third party transfers via GridFTP, to higher level tools such as Globus On-Line and the XSEDE file manager. To make it possible to access GridFTP from iRODS, iRODS was extended with the Xxxxxxx GridFTP server. The Data Staging service is regarded as a user-level service that allows individual users to easily manage the staging of large amounts of data (that consists of many files) and that results in many file transfers. Consequently the Data Staging service depends on the services developed by the AAI task force to provide federated access to the EUDAT data services. The time plan of the AAI TF, does not predict that a federated AAI solution will be available in the short term. Because the data staging solutions are all based on standard authentication methods (e.g. X509, username/password), it is possible to provide access in a less federated way (e.g. separate EUDAT or local credentials). As well as continuing to work on the selected technologies, the Data Staging TF will focus on developing a practical solution to enable user access during the coming months. During the service-building process, the Data Staging task force collaborated with the following organisations.  PRACE The communities actively participating in the Data Staging task force (VPH and EPOS) also use the PRACE HPC computing facilities. To provide an optimal solution, the EUDAT data service should be accessible from the PRACE supercomputers. At the moment, EUDAT and PRACE are holding ongoing strategic discussions for the purpose of collaborating. At time of writing, EUDAT and PRACE had agreed to conduct some pilots.  XSEDE XSEDE is the successor of the TeraGrid project in the US, and is funded by the National Science Foundation. The XSEDE organization develops and maintains the XSEDE file manager. The XSEDE file manager is the tool to use for moving large amounts of data between XSEDE supercomputers. EUDAT has collaborated with XSEDE to adopt the XSEDE file manager to make it work with the EUDAT data services.
Sample 1
Data Staging. When scientific communities use data-intensive applications, their data inputs are typically staged onto computational resources prior to job execution. Following execution, output data sets are generally staged out of compute nodes and onto storage systems. Additional data transfer operations may be required to move intermediate data products from nodes where they are produced to nodes where they will be consumed by other jobs. If the storage space on a node is limited, these stage-out operations need to be performed before other jobs can run. In distributed computing, where resources are geographically dispersed, efficient data staging is essential. Starting with this assumption, the objective of the data staging service is to enable research communities to stage data from EUDAT repositories to external HPC facilities for further processing. The “Data Staging” service can be considered as the fundamental service for exchanging data between the EUDAT domain of globally registered data and the domain of data consumers and producers (e.g. scientific instruments, HPC or HTC facilities). Three EUDAT core communities (VPH, ENES and EPOS) explicitly requested this service as they need to perform statistical model analyses on stored data. This service area differs from the safe replication scenario, which was discussed in the previous section, on three main points.  The replicated data are discarded when the analysis application ends.  The PID references are not applied to data replicated into scratch workspaces.  The users initiate the process of replicating data, whereas in the safe replication scenario, data are replicated automatically on a policy basis. A typical utilization scenario is depicted in Figure 14. 10xxxx://xxxx.xxxxxx.xxx/p/irodspython/issues/detail?id=11 1. The desired data are replicated from community storage to an EUDAT node using the “safe replication” solution. 2. Next the data are replicated between two EUDAT nodes, since the destination node is close to the target HPC facility. The corresponding PID record contains all the relevant URLs for the copies. 3. The data are then staged into a HPC facility that is either close to the EUDAT node or available elsewhere, for example, as part of the PRACE infrastructure11. 4. Finally the replicated data are staged into the local HPC facility. Afterwards, the analysis results will be staged out to the original source, and the results can be copied back to the community storage.
Sample 1
Data Staging. The question that remains is how to integrate the architecture with the outside world. We have already seen that we make use of external infrastructures for AAI. For the moment, we assume data is being transferred with GridFTP. In this case, three entities must have X.509 certificates (and private keys, and credentials): the source, the destination, and the controller (namely the user client). One or more of these could be the same. But, more importantly, the certificates must be signed by CAs that is trusted by each entity. This means that EUDAT cannot necessarily run its own CA and expect PRACE (say) to accept it44. The proposal of the AAI TF is to secure the infrastructure itself with known and trusted certificates from NRENs, IGTF, and commercial CAs (particularly for browser-facing services) – these are all (usually) trusted by e- and cyber-infrastructures across the world. However, earlier we proposed to generate federation credentials on behalf of users, to be used internally in EUDAT. Such a service, if X.509 based, would most likely not be trusted outside of EUDAT (see footnote 44 on page 43). Consequently, data staging would have to be done by an automated service with a trusted certificate. However, it would also need to preserve, or at least respect, permissions and rights associated with the user. 42 The case is equivalent in general for command line login. 43 See e.g. the IETF SCIM working group. Moonshot (ABFAB) will also be a potential alternative. 44 That is not to say this is impossible: it can be done but it requires a lot of work (defining a CP/CPS, see RFC3647), buying a Hardware Signing Module for protection of an online CA; checking and validating the CA against the IGTF SLCS profile, and seeking accreditation – the latter process can take 2-3 years – which is not compatible with the timescale of this project.
Sample 1
Data Staging. Data staging imposes the following requirements on the AAI.  It needs to have EUDAT-federation credentials accepted across community (or non-EUDAT) resources as well as EUDAT. Identity mappings, and in particular permissions, need to be consistent.  The EUDAT federated AAI must provide access to the replicated copies of the DOs.  It should work with target data interfaces at non-EUDAT resources.  It needs support for data transfer protocols (e.g. GridFTP, UFTP, or similar). GridFTP is currently known to be working with X.509 certificates. UFTP, as used by Unicore, should work with certificates. Support for other types of credentials (e.g. XXXX assertions) is highly desirable.  Data staging requires command line support.  GridFTP, being the preferred protocol, suggests the use of X.509 certificates. Indeed, third party copying will normally require RFC3820 “proxies” (or the older versions, GT2 or GT3 – some implementations are sensitive to the version of the proxy).
Sample 1

Related to Data Staging

  • Data Storage Where required by applicable law, Student Data shall be stored within the United States. Upon request of the LEA, Provider will provide a list of the locations where Student Data is stored.

  • Links If The Services are made available through the Internet, the Financial Institution’s website may provide links to other websites, including those of Third Parties who may also provide services to You. You acknowledge that all those other websites and Third Party services are independent from the Financial Institution’s and may be subject to separate agreements that govern their use. The Financial Institution and Central 1 have no liability for those other websites or their contents or the use of Third Party services. Links are provided for convenience only, and You assume all risk resulting from accessing or using such other websites or Third Party services.

  • Network Access During its performance of this Contract, Contractor may be granted access to Purchaser’s computer and telecommunication networks (“Networks”). As a condition of Network use, Contractor shall: (a) use the Networks in compliance with all applicable laws, rules, and regulations; (b) use software, protocols, and procedures as directed by Purchaser to access and use the Networks; (c) only access Network locations made available to Contractor by Purchaser; (d) not interfere with or disrupt other users of the Networks;

  • Network Interconnection Architecture Each Party will plan, design, construct and maintain the facilities within their respective systems as are necessary and proper for the provision of traffic covered by this Agreement. These facilities include but are not limited to, a sufficient number of trunks to the point of interconnection with the tandem company, and sufficient interoffice and interexchange facilities and trunks between its own central offices to adequately handle traffic between all central offices within the service areas at a P.01 grade of service or better. The provisioning and engineering of such services and facilities will comply with generally accepted industry methods and practices, and will observe the rules and regulations of the lawfully established tariffs applicable to the services provided.

  • Configuration Management The Contractor shall maintain a configuration management program, which shall provide for the administrative and functional systems necessary for configuration identification, control, status accounting and reporting, to ensure configuration identity with the UCEU and associated cables produced by the Contractor. The Contractor shall maintain a Contractor approved Configuration Management Plan that complies with ANSI/EIA-649 2011. Notwithstanding ANSI/EIA-649 2011, the Contractor’s configuration management program shall comply with the VLS Configuration Management Plans, TL130-AD-PLN-010-VLS, and shall comply with the following:

  • Network PHARMACY is a retail, mail order or specialty pharmacy that has a contract to accept our pharmacy allowance for prescription drugs and diabetic equipment or supplies covered under this plan. NETWORK PROVIDER is a provider that has entered into a contract with us or other Blue Cross and Blue Shield plans. For pediatric dental care services, network provider is a dentist that has entered into a contract with us or participates in the Dental Coast to Coast Network. For pediatric vision hardware services, a network provider is a provider that has entered into a contract with EyeMed, our vision care service manager.

  • Interface A defined set of transmission facilities that separate Load Zones and that separate the NYCA from adjacent Control Areas. Investor-Owned Transmission Owners. A Transmission Owner that is owned by private investors. At the present time these include: Central Xxxxxx Gas & Electric Corporation, Consolidated Edison Company of New York, Inc., New York State Electric & Gas Corporation, Niagara Mohawk Power Corporation, Orange and Rockland Utilities, Inc., and Rochester Gas and Electric Corporation.

  • Network Access Control The VISION Web Site and the Distribution Support Services Web Site (the “DST Web Sites”) are protected through multiple levels of network controls. The first defense is a border router which exists at the boundary between the DST Web Sites and the Internet Service Provider. The border router provides basic protections including anti-spoofing controls. Next is a highly available pair of stateful firewalls that allow only HTTPS traffic destined to the DST Web Sites. The third network control is a highly available pair of load balancers that terminate the HTTPS connections and then forward the traffic on to one of several available web servers. In addition, a second highly available pair of stateful firewalls enforce network controls between the web servers and any back-end application servers. No Internet traffic is allowed directly to the back-end application servers. The DST Web Sites equipment is located and administered at DST’s Winchester data center. Changes to the systems residing on this computer are submitted through the DST change control process. All services and functions within the DST Web Sites are deactivated with the exception of services and functions which support the transfer of files. All ports on the DST Web Sites are disabled, except those ports required to transfer files. All “listeners,” other than listeners required for inbound connections from the load balancers, are deactivated. Directory structures are “hidden” from the user. Services which provide directory information are also deactivated.

  • Access Toll Connecting Trunk Group Architecture 9.2.1 If ECI chooses to subtend a Verizon access Tandem, ECI’s NPA/NXX must be assigned by ECI to subtend the same Verizon access Tandem that a Verizon NPA/NXX serving the same Rate Center Area subtends as identified in the LERG. 9.2.2 ECI shall establish Access Toll Connecting Trunks pursuant to applicable access Tariffs by which it will provide Switched Exchange Access Services to Interexchange Carriers to enable such Interexchange Carriers to originate and terminate traffic to and from ECI’s Customers. 9.2.3 The Access Toll Connecting Trunks shall be two-way trunks. Such trunks shall connect the End Office ECI utilizes to provide Telephone Exchange Service and Switched Exchange Access to its Customers in a given LATA to the access Tandem(s) Verizon utilizes to provide Exchange Access in such LATA. 9.2.4 Access Toll Connecting Trunks shall be used solely for the transmission and routing of Exchange Access to allow ECI’s Customers to connect to or be connected to the interexchange trunks of any Interexchange Carrier which is connected to a Verizon access Tandem.

  • Data Access Access to Contract and State Data The Contractor shall provide to the Client Agency access to any data, as defined in Conn. Gen Stat. Sec. 4e-1, concerning the Contract and the Client Agency that are in the possession or control of the Contractor upon demand and shall provide the data to the Client Agency in a format prescribed by the Client Agency and the State Auditors of Public Accounts at no additional cost.

Draft better contracts in just 5 minutes Get the weekly Law Insider newsletter packed with expert videos, webinars, ebooks, and more!