Error Reconciliation. The Archive will report any problems or errors with file integrity, file name, checksum validation, or other errors that inhibit the data ingest and archive to the Provider. A new corresponding submission manifest will be required for files re-submitted by the Provider.
Error Reconciliation. The Archive will report any problems or errors with file integrity, file name, checksum validation, or other errors that inhibit the data ingest and archive to the Provider. A new corresponding submission manifest will be required for files re-submitted by the Provider. Receipt Confirmation: The Archive will provide an inventory of the data ingested once it is completed or as requested by the Provider. Quality Assurance: No quality checks on the submitted data are planned. Archive File Packaging: Archival Information Packages (AIPs) will be organized into one AIP per platform type. The data files will be exact duplicates of the data that was posted to the FTP server. Archive Storage Archive File Type Name: Descriptive name for this archive file type Archive File Attributes/IDs: Archive attributes of each archived file type. Attribute/ID Type Value None Attribute/ID value Archive Updates New, never-before seen data files will be archived based on which buoy they are: each buoy will be assigned an accession number. New, data from a previously submitted buoy: The AIP for that buoy will be updated (NCEI's major-revision) with the new data file. Revised, data that was previously submitted that needs to be updated: If the naming conventions match and the checksums do not match, then the most recent submission of that file will be assumed to be the latest and greatest submission and will replace the previous file.
Error Reconciliation. The Archive will report any problems or errors with file integrity, file name, checksum validation, or other errors that inhibit the data ingest and archive to the Provider. A new corresponding submission manifest will be required for files re-submitted by the Provider. Receipt Confirmation: The Archive will provide an inventory of the data ingested once it is completed or as requested by the Provider. Quality Assurance: No quality checks on the submitted data are planned. Archive File Packaging: Description of file packaging or re-naming by the Archive upon ingest. Archive Storage Archive File Type Name: Descriptive name for this archive file type Archive File Attributes/IDs: Archive attributes of each archived file type. Attribute/ID Type Value Attribute/ID value Attribute/ID value Archive Updates None planned. Retention Schedule The data will be retained in the Archive for long-term preservation in accordance with NOAA data management standards. Information on data usage and archive value may be used for making decisions on continuing the duration of the archive.
Error Reconciliation. NCEI automatically tracks problems or errors with file integrity, file name, checksum validation, or other errors that inhibit the L0 data ingest and archive. These errors are reported to LZSS, and NCEI tries to pull data a second time to complete the record. At times NCEI requests LZSS staff create new checksums if data files have missing or incorrect checksums.
Error Reconciliation. Validation of the BOEM-provided checksums against the NCEI-created checksums revealed 5 directories which did not match the expected directory name pattern. NCEI checksums were generated for those 5 additional directories. 1 directory, JSL_2787_MPG contained files that had completely broken names. The BOEM-provided checksums had the correct names, but the BOEM-provided files had different names. For example, the BOEM-provided checksums listed a file called "JSL_2787 Clip 01.mpg" but the directory contains a file incorrectly named "LEQSF 2787 Clip 01.mpg". Those files were renamed to match the BOEM-provided checksums. Otherwise, it appears that all files match their BOEM-provided checksums. Receipt Confirmation: Confirmation email was sent to provider on October. 16, 2017. Quality Assurance: No quality checks on the submitted data are planned. Archive File Packaging: Each dive folder containing all video clips captured on that day's dive will receive a unique 7-digit accession file number. The data/0-data folder associated with that dive folder will contain all the video clips as well as a a submitter- provided metadata spreadsheet whose rows correspond to only those video clips contained in this dive's folder. Archive Storage
Error Reconciliation. The Archive will report any problems or errors with file integrity, file name, checksum validation, or other errors that inhibit the data ingest and archive to the Provider. A new corresponding submission manifest will be required for files re-submitted by the Provider. Receipt Confirmation: The Archive will provide an inventory of the data ingested once it is completed or as requested by the Provider. Quality Assurance: No quality checks on the submitted data are planned. Archive File Packaging: Submission Information Packages (SIP) will be organized into 'bags'. Each 'bag' will contain data, metadata, and manifest files which fully document the files intended to be submitted. The 'bags' will be folders on xxxx://xxxx.xxxxxx.xxx/ncei/ohsucmop/ which correspond to the name of the platform. E.g. abpoa/, riverrad/, saturn01/, etc. Within the station folder (or 'bag') there will be four standard files with the following names: bag-into.txt, bagit.txt, manifest-sha256.txt, and tagmanifest-sha256.txt as well as a data/ directory which will contain folders for all of the netCDF files to be submitted. Each of the folders within the data/ directory represent an instrument/instrument deployment. NCEI will organize the Archival Information Packages (AIP) by station. Each time a new station arrives, a new AIP will be generated. If a station follows the 'bag' convention and has a name which matched a previously submitted package, NCEI will update the AIP and append the data from the new submission. If the new submission has files with the same name as what we previously submitted, NCEI will assume that the most recent submission should replace the previous submission. Only the files that have the same name will be replaced with the newly submitted file. Archive Storage Archive attributes of each archived file type. Attribute/ID Type Value Attribute/ID value Archive File Type Name: Descriptive name for this archive file type Archive File Attributes/IDs: Archive Updates New, never-before seen data files will be archived based on which station they are: each station will be assigned an accession number. New, data from a previously submitted station: The AIP for that station will be updated (NCEI-MD's major-revision) with the new data file. Revised, data that was previously submitted that needs to be updated: If the naming conventions match and the checksums do not match, then the most recent submission of that file will be assumed to be the latest and greatest submission...
Error Reconciliation. Archive will notify the Provider via email to reconcile any errors. Receipt Confirmation: To be determined. Quality Assurance: Each drawing will be checked visually by the Data Technician. For each file that has been gzipped for the Archive, NGDC will perform a data integrity check of the internal checksum contained within the gzipped file as part of the archive workflow process. This test will uncover any file corruption at the point at which the file was gzipped. Archive File Packaging: Files are stored in their native format and gzipped. For example: boul_neutl_fd_yyyymmdd_time.jpg.gz.
Error Reconciliation. The Archive will report any problems or errors with file integrity, file name, checksum validation, or other errors that inhibit the data ingest and archive to the Provider. A new corresponding submission manifest will be required for files re-submitted by the Provider. Receipt Confirmation: The Archive will provide an inventory of the data ingested once it is completed or as requested by the Provider. Quality Assurance: No quality checks on the submitted data are planned. Archive File Packaging: The packages are submitted as a tar file with an associated checksum. NCEI will untar the file to reveal the contents of data.csv, data.xml, and any other supplemental files in a supplemental directory. The tar file and checksum will be discarded after extraction and validation. An automated process extracts various metadata from the ISO 19115-2 XML file included in the package. The gmd:fileIdentifier from the ISO is used to determine the BCO-DMO package ID. Each BCO-DMO package ID corresponds to one NCEI accession. Title, Abstract, and other fields from the ISO metadata are mapped to NCEI accession metadata. If required ISO metadata fields are not present, the package is considered invalid. Archive Storage Archive File Type Name: Descriptive name for this archive file type Archive File Attributes/IDs: Archive attributes of each archived file type. Attribute/ID Type Value Metadata ID Attribute/ID value Archive Updates Data submissions intended to update an existing archive record require adequate notification and justification. Updates can supersede previous data submissions as a newer or improved version, however any previously submitted data will not be removed from the archive for the purpose maintaining version control and traceability in the archive. Retention Schedule The data will be retained in the Archive for long-term preservation in accordance with NOAA data management standards. Information on data usage and archive value may be used for making decisions on continuing the duration of the archive.
Error Reconciliation. The Archive will report any problems or errors with file integrity, file name, checksum validation, or other errors that inhibit the data ingest and archive to the Provider. A new corresponding md5 file will be required for files re-submitted by the Provider. Receipt Confirmation: The Archive will provide an inventory of the data ingested once it is completed or as requested by the Provider. Quality Assurance: No quality checks on the submitted data are planned. Archive File Packaging: Raw Incremental: ~14 daily files/day/satellite will be aggregated in one daily compressed (gzip) tarball and identified by satellite ID and date. Raw Daily File: Same NetCDF4 file, but compressed (gzip) for the archive. Archive Storage Archive File Type Name: Raw Incremental Archive File Attributes/IDs: Archive File Type Name: Raw Daily File Archive File Attributes/IDs: Archive attributes of each archived file type. Attribute/ID Type Value Storage System archive/satellite/poes/data/raw/socc/year/satid/poes_id_YYYYMMDD.tgz Attribute/ID Type Value Storage System archive/satellite/poes/data/raw/ngdc/year/satid/poes_id_YYYYMMDD_raw.n c.gz Archive Updates Data submissions intended to update an existing archive record require adequate notification and justification. Updates can supersede previous data submissions as newer, corrected or improved versions are available. Further discussion will be required about the disposition of previously submitted data, whether or not to version or simply replace the data. All actions with rationale will be captured in NGDC tracking systems.
Error Reconciliation. The Archive will report any problems or errors with file integrity, file name, checksum validation, or other errors that inhibit the data ingest and archive to the Provider. A new corresponding submission manifest will be required for files re-submitted by the Provider. Receipt Confirmation: The Archive will provide an inventory of the data ingested once it is completed or as requested by the Provider. Quality Assurance: No quality checks on the submitted data are planned, data were quality checked by data provider. Archive File Packaging: Description of file packaging or re-naming by the Archive upon ingest. Archive Storage Archive File Type Name: CalCOFI _icthyoplankton Archive File Attributes/IDs: Archive attributes of each archived file type. Attribute/ID Type Value Dataset ID Attribute/ID value Archive Updates Data submissions intended to update an existing archive record require adequate notification and justification. Updates can supersede previous data submissions as a newer or improved version, however any previously submitted data will not be removed from the archive for the purpose maintaining version control and traceability in the archive. Retention Schedule The data will be retained in the Archive for long-term preservation in accordance with NOAA data management standards. Information on data usage and archive value may be used for making decisions on continuing the duration of the archive.
