Approach to Flexibility in EOSDIS Data Format Standards
--H.K. Ramapriyan (Rama.Ramapriyan@gsfc.nasa.gov) or (rama@ulabsgi.gsfc.nasa.gov), ESDIS Project, Code 505, NASA GSFC
It is the intent of NASA's Earth Science Data and Information (ESDIS) Project to establish, maintain, and evolve data standards to support interoperability across Earth science disciplines, to improve and promote easy access to science data, and to empower users and developers of Earth science software and tools. The ESDIS Project believes that Earth Observation System Data and Information System (EOSDIS) support for a minimal set of data format standards supports this intent. The recent rapid growth of the World Wide Web, a system that empowers users and provides an environment that promotes dynamic evolution by adherence to a few basic data standards, validates this purpose.
The ESDIS Project, after years of review and extensive prototyping, established HDF and conventions for geolocated data known as HDF EOS as its baseline data format standard. The ESDIS Project continues to support HDF EOS as its baseline standard data format for new products and for community-directed heritage data upgrades. As EOSDIS's baseline standard, HDF EOS will be fully supported by the ESDIS Project with multi-platform read-and-write libraries and tools, updates to correct deficiencies, and continued development to make it the best solution for the majority of EOSDIS data. The specification for HDF EOS is accessible from the World Wide Web at http://eos.nasa.gov/esdis/InfoArch.
Heritage Data Upgrades
Upgrades of heritage data into the HDF format are based on the EOSDIS Version 0 data migration plans being developed at the direction of the EOSDIS Distributed Active Archive Centers (DAACs) in consultation with their user communities. Where the DAACs and their user communities desire HDF upgrades, they will be performed on a priority basis determined by the user communities. Where the DAACs and their user communities do not desire such upgrades, native formats will be retained for the old data sets after migration.
New Data Products
The ESDIS Project levies no data format requirements on data systems that are not part of NASA's Mission to Planet Earth Program. Data brought into the EOSDIS system for archive are preferred to be HDF EOS (as EOSDIS can provide a higher level of distribution and access service for those products). EOSDIS, however, has always had, and will continue to have the ability to support "native" format data with inventory search and data delivery at the file level, and will accept data in those formats. However, the ESDIS Project will not generally fund an upgrade in services for "native" or other non-HDF EOS format data sets.
The ESDIS Project has established EOSDIS Core System (ECS) requirements to support conversion of a limited number of formats for ingest and distribution. These are identified and handled on a case-by-case basis.
Supporting Evolution
EOSDIS is designed to support evolutionary change and can evolve from HDF EOS for production and archive when and if better methodologies can feasibly support the technical requirements. To ensure that evolution can take place, a change process is now in place and is detailed in the following paragraphs.
Change Process - Identifying Alternatives
In consultation with, and representing the user community, the DAACs may determine what alternative formats should be supported, based on DAAC systems engineering analysis of the need. ESDIS expects the DAACs themselves to initiate this effort. Data Producers who wish to propose alternative formats will work with their host DAAC to accomplish the required system engineering analysis.
Where the science community strongly desires consideration of an alternative format, but is not represented by a DAAC, those users should contact the ESDIS Project and request that the ESDIS Project initiate DAAC or Project analysis of an alternative. Adoption of a user-initiated alternative will follow the same process as a DAAC-initiated alternative.
Change Process - White Paper
To substantiate the need for an alternative format the DAAC or Data Producer will develop and submit to the ESDIS Project a white paper defining and selecting an alternative format and demonstrating an important technical need or resource savings within the Earth science community. The white paper should be coordinated with other DAACs and Data Producers to determine like need. Collaboration among DAACs and Data Producers is recommended.
The white paper should provide a life-cycle cost estimate for the support of the alternative format pinpointing what ECS components will be required to integrate the alternative format and showing estimated costs or cost savings associated with needed ECS tailoring, development, and maintenance. The white paper should also show a timeline of events that can be worked into current mission deadlines. The ESDIS Project will support cost and schedule analysis where needed. The white paper should show limitations of already supported formats which preclude their use. The white paper should also address collection, ingest, access, and usage of metadata required by EOSDIS to meet the requirements of the proposed alternative format.
Change Process - Consultation
The alternative format must have technical and scientific justification. To validate scientific justification, the Project will consult with relevant Project Scientists and Instrument teams.
Change Process - Resources
The ESDIS Project will accept any DAAC-determined alternate format as long as it can be supported by a combination of: application of DAAC-unique funds taken as an aggregate across all DAACs; the ECS Contractor currently levied requirements for conversion; and savings based on using the alternative format.
Change Process - Prioritization
The ESDIS Project will support alternative formats to the extent they are needed by the science community and are higher priority than other EOSDIS services or functions.
If the overall collection of formats desired by the DAACs (representing the user community) and Data Producers is not supportable, or impacts the goal of interopera-bility, then the ESDIS Project will prioritize them based on DAAC, User Community, and Data Producer feedback. The final prioritization will be published to allow impacted parties to provide feedback and comment.
[Table of Contents]
[Previous]
[Next]