The Earth Observer

March/April 1996, Vol.8, No.2

ECS Hosts First Meeting of Ad Hoc Working Group for Consumers

--Lori J. Tyahla (ltyahla@eos.hitc.com), Science Office, ECS Project, Landover, MD.

The EOSDIS Core System (ECS) hosted the first meeting of the Ad Hoc Working Group for Consumers (AHWGC) on March 14, 1996 at the Landover, MD facility. The overall objective of this group is to collect further information about the EOSDIS user community to refine the ECS User Characterization Team's methods and results. The group was formed by the EOSDIS Data Panel and is co-chaired by Bill Emery (University of Colorado) and Dave Emmitt (University of Virginia). Members include representatives from IDS Teams, EOS Instrument Teams (ITs), and EOSDIS Distributed Active Archive Centers (DAACs). Representatives from the ECS Science Office and NASA/GSFC are also active in the group.

The initial efforts of the group have focused on collecting detailed information regarding the data needs of the NASA EOS-funded Interdisciplinary Science (IDS) Teams. This information is used by ECS developers in designing and sizing various components of the ECS, including data servers, distribution hardware, archive structure, and processing requirements. A packet of information was sent to each IDS Team with a request for each team to identify the data required by all of the team members in each of three time periods. The details of the methods and results can be found in Input Data Requirements of EOS-Funded Interdisciplinary Science (IDS) Teams (URL http://ecsinfo.hitc.com/sec1/ahwgc.html).

There were three main objectives of the March 14 meeting:

review analysis of the "volume distributed will equal twice the volume produced" design assumption and provide recommendations;
discuss methods for managing user loads on the system to maximize system performance and provide recommendations; and
review the results of the analysis of the IDS Team data needs.

The outcome of the meeting was a list of assumptions, definitions, and recommendations regarding these issues. In addition, the group provided a list of topics that will need to be addressed at some future time by the ECS Project.

Guiding Principles

Users/providers must sense fairness (non-discrimination), responsiveness, and stability.

Assumptions/Definitions

Users/providers will behave differently tomorrow than they do today.
Emphasis must be on having plans for adapting to the change.
The system will saturate (when we include all potential users) with user requests/demands reducing performance, throughput and data delivery:
- saturation is likely a function of time of day
- subscriptions and media distribution impact saturation differently than does network access
- system is susceptible to saturation due to unreasonable network data requests
X is the volume of data put into, i.e., "push," the archives per year.
N is the ratio of pull/push.
- N refers only to data "pulled" by the following three groups of users: Instrument Teams for QA/QC, IDS Teams for all purposes, and data pulled by general users. Data Pull for product generation is accounted for separately and sufficient resources are available for this purpose.
- N, e.g., 2, is assumed to represent "pull" or data "out the door" without replication.
- N is also a system average but known to vary between data products.
- The ECS architecture is capable of handling 16X given sufficient hardware.
- DAACs are in the best position to choose the appropriate means of managing pull demand that approaches or exceeds their capacity.

Key Findings by the AHWGC

Initial IDS plus Instrument Team QA/QC data pull should not exceed 1.5X and may be closer to 1X, given the history of pre-launch overestimating the volume of data needed by Quality Control.
N = 2 is reasonable for initial design, i.e., 2X.
Estimation of non-EOS research demand is bounded by:
- 10 petabyte/year (33X) extreme
- 2 petabyte/year (7X) reasonable
- 0.5 petabyte/year (2X) probable
10% of data products account for 90% of pull.
Currently, tape and CD ROM are the more frequently requested modes of data delivery -- this is very likely to change as ftp capability becomes more common and network capacity is significantly increased.

Recommended User Load Management Options

Although current estimates of IDS and IT demands are within the ECS 2X resource limit, as the user demand (pull) approaches the operational capacity of ECS (initially 2X), the DAACs (or Earth System Information Partners [ESIPs]) will need to exercise some form(s) of system management that will avoid having the level of service to EOS investigations fall below some minimum level of performance (TBD).

The AHWGC has identified the following options for managing user demands that approach or exceed the existing ECS capacity. It is assumed that system managers will be monitoring push/pull in real time and will have adequate warning (on time scales of weeks) of impending saturation.

Short-Term Options (days to months):
Load leveling to minimum acceptable performance (in terms of turnaround time) multiple copy options:

mastering CDs
ftp staging
WWW sites
PIF (Potential Impact Filter) -- user services interception of requests for services with significant impact on a system's marginal performance

Long-Term Options (months to years):
Temporal charging -- "sooner costs more"

vouchers for core users
bidding between DAACs for new data sets/services
Cooperative Agreement Notices (CANs) to lever age private sector participation in meeting user demands increase N (up to 16X) by purchasing more hardware

Last-Resort Options (more long term than short term):

Prioritization of requests based upon a to-be-developed NASA policy
Pricing designed to limit demand
User authorization resulting in discrimination

There were also many questions raised at the meeting including:

Who will manage system access/performance?
Who (how) will policy be set for these management practices?
How will this policy/practice be reviewed and communicated to the science user community?
How do we collect input on these issues from the science user communities?
Can we have a federation ESIP manage this aspect much as a network arbiter enforces the network Acceptable User Practices (AUP) and resolves conflicts?
Will value-added providers be treated differently from other external users? What happens if the value-added provider trades in timeliness and the system approaches saturation?

Overall, the meeting was very successful with the group reaching consensus on several issues. These recommendations were formally documented by the co-chairs of the group and delivered to the ESDIS Project and NASA Headquarters.