Exhibit 10.9
SOFTWARE MAINTENANCE, DATA SERVICES AND OPERATIONS
SERVICE LEVEL AGREEMENT
SOFTWARE MAINTENANCE, DATA SERVICES AND OPERATIONS SERVICE LEVEL AGREEMENT
("Agreement") entered into as of the 1st day of January, 2001, and amended on
May 15, 2002, by and between ITA Software, Inc. ("Licensor") and Orbitz, LLC
("Licensee").
1. INTRODUCTION
This Software Maintenance, Data Services and Operations Service Level Agreement
covers standards for the provision of software maintenance, data management and
support, and operations by Licensor to Licensee, in connection with that certain
Software License Agreement dated as of July 1, 2000 between Licensee and
Licensor (the "License Agreement"). The License Agreement terminates on
September 30, 2004 (if not renewed) and this Agreement, and the pricing
contained herein, has been structured to run concurrently with the License
Agreement. Capitalized terms used but not defined herein will have the meaning
ascribed thereto in the License Agreement.
In accordance with the License Agreement, Licensor has granted to Licensee a
license to use the Licensed Software (as defined in the License Agreement) for
the purpose of enabling End Users (as defined in the License Agreement) to
obtain information and to make decisions about airline routes and schedules,
airfares and availability, as well as to search for low airfares, at Licensee's
World Wide Web site currently located at xxx.xxxxxx.xxx. Licensee will operate a
portion of the Licensed Software at its location in Chicago, Illinois (the
"Licensee Facility", which shall include any other location to which Licensee
may move its operations center in the future). To support Licensee, Licensor
will: (i) provide maintenance and support for the Licensed Software, pursuant to
Section 2 of this Agreement ("Maintenance Services"), (ii) provide data and data
support relating to the Licensed Software, pursuant to Section 3 of this
Agreement ("Data Services") and (iii) provide operations to effect the
Maintenance Services and the Data Services, pursuant to Section 4 of this
Agreement ("Operations Services").
2. MAINTENANCE SERVICES
2.1 SCOPE OF SERVICES
(a) The following are the Maintenance Services that will be provided by Licensor
in connection with the Licensed Software:
Licensor shall:
(i) develop and provide corrections, changes, or workarounds
("Corrections") for any defects, errors, or malfunctions in the
Licensed Software, including, but not limited to, any
nonconformities with the Specifications (as defined below)
(collectively, "Defects"), discovered by Licensee or Licensor,
on a timely basis, given the nature and scope of the Defect;
(ii) provide to Licensee all improvements, modifications and
enhancements ("Improvements", which term will not include
improvements, modifications or enhancements (i) which contain
significant new or improved functionalities or (ii) which are
developed by Licensor specifically for its other customers which
are specific to the systems or software of such other customers
and which Licensor does not have the right to license to its
licensees) to the Licensed Software which Licensor shall make or
acquire from time to time and which Licensor makes available to
its licensees generally; and
(iii) provide Licensee any upgrade releases ("Upgrade Releases") to
the Licensed Software and all new Versions and Releases of the
Licensed Software, which Licensor makes available to its
licensees generally; provided, however, that if any such Version
or Release contains significant new or improved functionalities,
Licensor shall have the right to make such Version or Release
available to Licensee only upon the payment of additional
license fees, or upon such other terms, as the Licensor requires
of its licensees generally therefor. "Versions" shall be
designated by a change in the digits to the left of the decimal
point, and "Releases" shall be designated by a change in the
digits to the right of the decimal point.
As used herein, "Specifications" means the functionality of the Licensed
Software described in Exhibit A to the License Agreement.
(b) Licensee shall implement all Corrections that do not materially alter or
diminish the functionality of the Licensed Software as described in the
applicable Documentation and Specifications. Licensee's failure to do so within
90 days after receipt of notice from Licensor regarding the availability of such
Corrections shall eliminate Licensor's obligation to provide Maintenance
Services until such Corrections are implemented. In addition, any modifications
to Licensor-provided configuration files or alteration from Licensor's
recommended use of configuration options shall eliminate Licensor's obligation
to provide Corrections or other Maintenance Services for that portion of the
software affected thereby, unless such modifications by Licensee were
occasioned, following consultation between Licensee and Licensor, by Licensor's
failure to modify or correct the Licensed Software as required hereunder or
under the License Agreement.
(c) In the event that changes in industry-standard practices and/or data formats
(such as ATPCO file formats, ATPCO processing, SSIM schedule formats and
government mandates regarding taxes and passenger facility charges (PFCs))
prevent the Licensed Software from functioning as specified in the License
Agreement or Specifications, the Licensor shall deliver updates to the Licensed
Software to permit operation of the Licensed Software in a manner equal to the
functionality described in the License Agreement and Specifications. Such
updates will be considered "Improvements" for the purposes of this Section 2,
and Licensor shall deliver such Improvements to Licensee
within a reasonable time prior to the effective date of such industry change so
that such Improvements may be implemented by the effective date of such change.
Licensee shall implement any Improvements delivered pursuant to this paragraph
(c) as soon as practicable after delivery thereof.
(d) Licensor will provide Licensee any revisions to the existing Documentation
developed or necessary to reflect all Corrections, Improvements, Upgrade
Releases, Versions or Releases for the Software.
(e) All computer programs delivered pursuant to the Maintenance Services shall
be considered "Licensed Software" and subject to all the terms and conditions of
the License Agreement.
(f) Licensor shall provide Maintenance Services directly to Licensee but not to
End Users.
(g) Licensor shall at all times provide Maintenance Services for at least the
current and one (1) previous Release of the Licensed Software; provided, that
Licensor's obligation to maintain a previous Release will terminate 120 days
after the current Release was made available to Licensee; and provided further,
however, that in the case of Improvements delivered pursuant to paragraph (c),
Licensor will not be obligated to provide Maintenance Services with respect to
any prior Release which does not incorporate the industry changes comprised in
such Improvements.
2.2 MAINTENANCE
2.2.1 BASIC MAINTENANCE
The Licensor shall maintain a technical support entry point in Massachusetts,
identified by a dedicated phone number, fax number and e-mail address, which
will be staffed by knowledgeable employees capable of providing technical
assistance regarding the Licensed Software, its functionality, databases,
operations, utilities and supporting documentation. Such telephone or e-mail
assistance will be available to the Licensee on a daily basis during regular
working hours (9:00 a.m. to 6:00 p.m. Eastern time Monday through Friday). This
technical support entry point will also coordinate problem resolution and keep
the Licensee apprised of efforts to remedy any problem situation until complete
restoration of the service. First line of support will be the Licensee's Help
Desk, who will contact and escalate problem to Licensor when necessary.
2.2.2 ENHANCED MAINTENANCE
Enhanced Maintenance will comprise the following: the Licensor shall provide
on-call coverage outside the technical support working hours described above, 24
hours/day, 7 days/week, to support urgent problem resolutions of Severity 1 and
2 Problems (as
defined in Section 2.3); provided that during the 72 hours prior to major
launches of the site (which will not occur more than four times per year)
Licensee may report problems which relate to such launch during non-business
hours even though such problems are not otherwise within the definitions of
Severity 1 or Severity 2. This on-call coverage will be provided via a dedicated
pager number, carried by knowledgeable employees of the Licensor. During normal
working hours, problem reporting will be through the Basic Maintenance entry
point, but the Licensor's response shall be in accordance with Enhanced
Maintenance service standards. In addition, upon request of Licensee, Licensor
shall provide on-site support within twenty-four (24) hours or within a mutually
agreed time frame between the parties where telephone support fails to correct
any Defect within the target time period set forth in Section 2.3. In such
event, Licensor shall provide qualified personnel to work exclusively to correct
such Defects until the same is corrected.
2.3 PROBLEM CLASSIFICATION
Licensor shall respond to and use its reasonable commercial efforts to resolve
problems in accordance with the severity levels indicated below, which severity
levels shall be determined in good faith by Licensee.
SEVERITY PROBLEM RESOLUTION
LEVEL DESCRIPTION RESPONSE TIME TARGET
------------------------------------------------------------------------------------------------
1 Mission-Critical Impact- 30 Minutes (with hourly Within 45 minutes
Licensed Software or System updates thereafter)
down
2 High Systems Impact 1 Hour (with hourly updates Within in 2 hours
thereafter)
3 Business Productivity Impact 2 Hours (with daily updates Within 72 hours
thereafter)
4 Minor Service Impact 24 Hours Next scheduled upgrade
(but not later
than 120 days)
In addition to the foregoing, in the case of Severity Level 2 errors, Licensor
will use reasonable commercial efforts to acknowledge receipt of Licensee's
error report within 30 minutes; and in the case of Severity 3 and 4 errors
(which as set forth above may only be reported during normal business hours),
Licensor will use reasonable commercial efforts to acknowledge receipt of
Licensee's error report within 60 minutes (or, in the case of error reports
received during the last 60 minutes of the business day, within the first 60
minutes of the following business day).
Severity 1 - Mission-Critical Impact:
The system operating the Licensed Software is rendered inoperable due to system
software failure and thus cannot perform and return responses to fare/schedule
queries. Under Enhanced Maintenance, all resources shall be applied 24 hours/7
days a week until the problem is resolved; provided that in the event that
Licensee's system is "rolled back"
and functionality is returned to a level such that it is no longer within the
definition of Severity 1 (although functionality may still be down, corrupted or
severely degraded), then the problem will be reclassified as a Severity 2
problem. Initial response and problem resolution target will be according to the
chart set forth above.
Severity 2 -High Systems Impact:
Availability feed from the Atlanta Facility is interrupted for a period
exceeding two hours, or availability information related to more than four
carriers is interrupted for a period exceeding four hours. For Enhanced
Maintenance, resources shall be applied 24 hours/7 days a week until the problem
is resolved. Initial response and problem resolution target will be according to
the chart set forth above.
Severity 3 - Business Productivity Impact:
Availability feed from the Atlanta Facility is interrupted for a period
exceeding one hour, or availability information related to more than two
carriers is interrupted for a period exceeding four hours. The Licensed Software
commits a pricing error which varies from the correct price by more than $[***]
and which affects more than [***]% of queries to the Licensed Software
(calculated based on a representative sample of queries as reasonably designed
by the parties). Initial response and problem resolution target will be
according to the chart set forth above.
Severity 4 - Minor Service Impact:
Failure of hardware results in a backup or "failover" piece of hardware
operating in place of the failed hardware without material interruption or
deterioration of functionality. The Licensed Software commits a pricing error
which varies from the correct price by more than $[***] and which affects [***]%
or fewer of queries to the Licensed Software. Initial response and problem
resolution target will be according to the chart set forth above.
Notwithstanding the foregoing definitions, problems caused by or arising from
the following will not be considered "problems" for the purposes hereof and will
not be subject to Licensor's obligation to provide Maintenance Services: (i)
failure of a Data Provider (as defined in Section 3.1(d)) to provide data in a
timely fashion; (ii) failure of telecommunications hardware or equipment; (iii)
failure or unavailability of the Worldspan system; (iv) failure of an airline
carrier to provide availability information; (v) Force Majeure (as defined in
Section 7.4).
In the event Licensee has elected not to receive Enhanced Maintenance, then the
response times and problem resolution targets (i.e., the hours during which
Licensor will be obligated to work on problem resolution) will be restricted to
normal working hours (9:00 am to 6:00 pm EST, Monday through Friday).
----------
*** Certain information on this page has been omitted and filed separately with
the Commission. Confidential treatment has been requested with respect to the
omitted portions.
2.4 STANDARD OPERATING PROCEDURES
The procedures for reporting, diagnosing, tracking and responding to problems
are set forth in Licensor's standard operating procedures for Licensee ("SOP"),
which is attached to this Agreement as APPENDIX A, as it may be amended by the
parties from time to time; provided, that Licensor will give Licensee reasonable
prior notice of any anticipated major changes to the SOP.
2.5 CHANGE MANAGEMENT
The parties shall develop a mutually agreeable change management process. At a
minimum, such process shall require Licensor to notify Licensee and obtain
Licensee's approval prior to implementing any material changes to the services
provided by Licensor hereunder or any changes which could materially affect
Licensee's use of the Licensed Software as contemplated in this Agreement and
the License Agreement.
2.6 MAINTENANCE STANDARDS
Licensor shall use its best commercially reasonable efforts to meet the response
times and resolution targets set forth in this Section 2.
3. DATA SERVICES
3.1 SCOPE OF SERVICES
(a) Licensor maintains its data center in the Exodus Communications facility in
Waltham, Massachusetts (the "Waltham Facility", which shall include any other
location to which Licensor may move its data center in the future), from which
Licensor manages data feeds for schedule and faring data and provides such data
to Licensee for use with the Licensed Software. Licensor currently receives fare
data from Airline Tariff Publishing Company ("ATPCO") and schedule data from
Innovata, LLC ("Innovata"). Licensor also receives availability data through the
Worldspan GDS ("Worldspan") and maintains an availability cache at the Waltham
Facility for the purpose of providing availability data to its licensees.
(b) In response to a specific request from Licensee, Licensor has established a
facility at Worldspan in Atlanta, Georgia (the "Atlanta Facility", which shall
include any other location to which Licensor may in the future move its data
center for receiving availability data and operating an availability cache,
provided that Licensee shall not be obligated to pay the monthly fixed charge
set forth in Section 6.4.1 in the event such facility is moved from Atlanta,
Georgia), at which Licensor receives availability data and operates an
availability cache for the provision of availability information to the Licensed
Software. Licensor provides availability data from such availability cache to
another availability cache (which is part of the Licensed Software) operated at
the Licensee Facility. Licensor will also utilize the Waltham Facility as a
backup for availability data
in the event the Atlanta Facility is unable to communicate directly with the
Licensee Facility.
(c) The operation of the Atlanta Facility and the Waltham Facility (including
all software running in such facilities) are the sole responsibility of Licensor
and the operation of the Licensee Facility (including the Licensed Software) is
the sole responsibility of Licensee.
(d) As part of the data management services provided hereunder, Licensor will
(i) receive and manage downloads of fare and schedule data from ATPCO and
Innovata, respectively (or in each case from such other commercially reasonable
industry accepted sources as the parties may mutually agree upon, referred to
herein (together with Worldspan) as "Data Providers"), (ii) process such data
such that it is in a form suitable for use with the Licensed Software, (iii)
receive availability data from Worldspan and maintain the availability cache at
the Atlanta Facility (and, for backup purposes, at the Waltham Facility), and
(iv) transmit such availability, fare and schedule data to the Licensee Facility
as frequently as it receives such data from the relevant Data Providers. In the
event that Licensee desires Licensor to change a particular Data Provider for
purposes of providing services hereunder, Licensor shall consult with Licensee
to determine whether Licensor should make such a change.
3.2 DATA PROVIDERS
Licensor has in place and will use its best efforts to maintain legal agreements
with Data Providers for the provision of availability, schedule and fare data
for the operation of the Licensed Software. Furthermore, Licensor believes that
its commercial arrangements presently in effect with Data Providers should
enable Licensor to provide services in accordance with the terms of this
Agreement. However, Licensee acknowledges and agrees that such Data Providers
are third parties which are not in Licensor's control and which are subject to
delay or failure. While Licensor agrees, as part of its obligations under this
Agreement, to receive, load and manage such data, Licensor does not warrant the
accuracy of such data. In addition, Licensor disclaims any and all liability
resulting from or related to Data Providers' failure to provide data to Licensor
in a timely fashion. In the event a Data Provider breaches the terms of a legal
agreement with Licensor, and such breach has an adverse effect on Licensee,
Licensor shall, upon Licensee's request, consult with Licensee as to Licensor's
enforcement of its rights under such agreement; however, Licensor will retain
complete discretion as to what, if any, action to take as a result thereof.
4. OPERATIONS SERVICES
4.1 WALTHAM FACILITY AND ATLANTA FACILITY
Licensor will operate and maintain the Waltham Facility and the Atlanta
Facility. Such operation and maintenance will include, without limitation, the
acquisition, installation,
maintenance, upgrading, monitoring and all aspects of the operation of all
computer hardware and equipment, and all services related thereto, for the
Waltham Facility and the Atlanta Facility necessary in connection with the
provision by the Licensor of the Data Services provided hereunder.
4.2 COMMUNICATIONS
The Licensor will receive data at the Atlanta Facility and the Waltham Facility
and transmit data from from such facilities via communications channels which
may include the Internet, a virtual private network or dedicated point-to-point
circuits. Licensor shall at all times maintain agreements for high speed
connections for the receipt and transmission of such data (except that Licensee
acknowledges that Licensor does not have direct connections to ATPCO or Innovata
and receives data from such Data Providers via a high-speed internet
connection).
(a) All hardware, software and services associated with communications between
the Atlanta Facility and the Licensee Facility, including maintenance thereof,
will be the responsibility of Worldspan. The cost of such services from
Worldspan will be for the account of Licensee and Worldspan will invoice
Licensee therefor directly. Such hardware, software and services will not be
included within any of the service level or support commitments contained in
this Agreement.
(b) All hardware, software and services associated with communications between
the Waltham Facility and the Licensee Facility, and between the Atlanta Facility
and the Waltham Facility, including maintenance thereof, will be provided by the
Licensor and charged to Licensee pursuant to the provisions of Section 6.4.2
hereof.
(c) Licensee expressly acknowledges that the flow of data to or from the Atlanta
Facility or the Waltham Facility will depend in large part on the performance of
hardware, software and services provided or controlled by third party
communications providers and by Worldspan. Licensor disclaims any and all
liability resulting from or related to Licensee's inability to communicate with
the Atlanta Facility or the Waltham Facility, to the extent such inability is
the result of the failure of hardware, software or services provided by such
third party communications providers and/or Worldspan.
5. PERFORMANCE SERVICE LEVELS
5.1 DATA DELIVERY.
Licensor will exercise reasonable commercial efforts to deliver data in
accordance with Section 3.1. Failures to deliver data will be subject to the
following:
(a) In the event Licensor fails to deliver schedule data at the scheduled
delivery time, such failure will be treated as a "Severity 4" error. If
such failure has not been remedied within twenty-four hours of the
later of (i) the scheduled delivery time
or (ii) the time Licensor actually receives such data from the Data
Provider thereof, then such failure will be treated as a "Severity 2"
error.
(b) In the event Licensor fails to deliver fare data at the scheduled
delivery time, such failure will be treated as a "Severity 4" error. If
such failure has not been remedied within the later of (i) the
scheduled delivery time for the next scheduled delivery or (ii) two
hours of the time Licensor actually receives such data from the Data
Provider thereof, then such failure will be treated as a "Severity 2"
error.
(c) Any failure to transmit availability data will be covered by the
Severity definitions contained in Section 2.3.
Licensor will not be deemed to have failed to deliver data under this Section if
such failure occurs as a result of unplanned telecommunication outages, of Force
Majeure (as defined in Section 7.4), or of acts or omissions of Licensee.
5.2 PRICING ACCURACY.
Licensor will exercise reasonable commercial efforts to ensure that, within the
functionality described in the Specifications, the Licensed Software will
perform pricing calculations in accordance with Airline Tariff Publishing
Company ("ATPCO") rules and other applicable rules (i.e., rules relating to
taxes) as in effect from time to time. For the purposes hereof, the Licensed
Software shall be deemed to have performed a pricing calculation correctly if it
is within $[***] (per passenger) of the correct price. In the event the parties
are unable to agree as to whether the Licensed Software has performed a pricing
calculation (i.e., applied ATPCO rules) correctly, the parties will seek the
interpretation of the airline whose fare calculation is disputed and the
interpretation of such airline with respect thereto shall be conclusive. The
parties will work together to identify and investigate pricing errors and to
determine the priority with which such errors should be addressed by Licensor
(including cases in which errors need not be addressed because of the
infrequency with which they arise). Pricing errors will be responded to and
resolved by the Licensor in accordance with the Severity Levels described in
Section 2.3; provided, that any pricing error will be considered as Severity 4
unless it varies from the correct price by more than $[***] and affects more
than [***]% of queries to the Licensed Software, in which event it will be
considered as Severity 3.
5.3 DISASTER RECOVERY
Licensor will comply with the provisions of the Disaster Recovery Plan (DRP)
attached hereto as Appendix B, and shall provide Licensee with notice of any
material change to such DRP.
6. FEES AND EXPENSES
----------
*** Certain information on this page has been omitted and filed separately with
the Commission. Confidential treatment has been requested with respect to the
omitted portions.
6.1 BASIC MAINTENANCE
THERE IS NO CHARGE FOR BASIC MAINTENANCE.
6.2 ENHANCED MAINTENANCE
Beginning January 1, 2001, if the Licensee elects to receive Enhanced
Maintenance, the fee therefor will be $[***] per month, plus the increase in the
Consumer Price Index for Boston (as reported in the Wall Street Journal) from a
base of January, 2001 through January of the year in which the Enhanced
Maintenance is provided ("CPI Increase"). Any CPI Increase (under this Section
6.2 or under Section 6.3 or 6.4.1) will be calculated in January of each year
beginning 2002 and will remain in effect for the balance of such calendar year.
Licensee may discontinue Enhanced Maintenance at any time upon 90 days' prior
written notice to Licensor.
6.3 DATA SERVICES
Beginning January 1, 2001, the charge for the Data Services described in Section
3.1(d) is [***] per month, plus the CPI Increase. Licensee may discontinue Data
Services at any time upon 90 days' prior written notice to Licensor.
6.3.1 AVAILABILITY DATA
Beginning January 1, 2001, Licensee will pay Licensor a fee of $[***] per month,
in respect of the cost of obtaining availability data. In the event that
Licensor is able to allocate the cost of Worldspan availability data among a
larger number of licensees, Licensor shall reduce the portion of such cost that
is being paid by Licensee pursuant to this paragraph.
Notwithstanding the foregoing, in the event Licensee may negotiate with
Worldspan, L.P. or another third party for relief from or abatement of such fee,
Licensor shall xxxxx or adjust such fee as directed by Licensee; provided,
however, that if obtaining such relief or abatement necessitates a change in the
availability vendor, the parties shall mutually agree upon the costs associated
with such change.
6.3.2 ADDITIONAL CHARGE FOR AVAILABILITY DATA
(a) In the event Licensee uses Worldspan availability data but ceases to
process in the Worldspan system substantially all its air travel
transactions which are generated from the Licensed Software, then
Licensee will pay an additional charge of $[***] per PNR (as defined in
the License Agreement) created by Licensee's use of the Licensed
Software; provided, that in the event Licensee may negotiate with
Worldspan a relief from or abatement of such fee, Licensor will xxxxx
such fee if so directed by Worldspan.
(b) If the charge described in paragraph (a) is applicable, Licensee will
certify to Licensor, within 30 days of the end of each calendar
quarter, the number of PNRs created on Worldspan during such quarter.
Notwithstanding the provisions of
----------
*** Certain information on this page has been omitted and filed separately with
the Commission. Confidential treatment has been requested with respect to the
omitted portions.
Section 6.5, Licensee will pay such charge simultaneously with its
delivery of such certification.
6.3.3 INCREASE IN DATA COSTS
The costs to Licensor of obtaining fare and schedule data provided hereunder is
included in the license fee payable by the Licensee. However, in the event there
is a material increase in the cost to Licensor of the data feeds which are
provided hereunder from those presently in effect, Licensor reserves the right
to impose a separate fee for such data upon 90 days' prior written notice to
Licensee; provided that such fee shall be limited to the actual cost increase
incurred by Licensor for data provided to Licensee. Such fee will only be
assessed as part of a general increase by Licensor in its pricing to its
licensees resulting from such a cost increase.
6.4 OPERATIONS SERVICES
6.4.1 ATLANTA AND WALTHAM FACILITIES
Licensor will charge Licensee in respect of the costs for establishing and
maintaining the Atlanta Facility and a rateable portion of the costs for
maintaining the Waltham Facility. Such charge consists of (i) a single, one-time
charge of $150,000, relating to the initial (non-recurring) cost of establishing
the Atlanta Facility (including acquisition of hardware and equipment and
amounts paid to Worldspan), which will be payable in accordance with the payment
schedule set forth in Exhibit G to the License Agreement, (ii) a fixed charge of
$[***] per month plus the CPI Increase, relating to the recurring acquisition,
replacement, upgrading, operation and maintenance costs of the hardware,
equipment, rack space, bandwidth, facility services and performance monitoring
for the Atlanta Facility and Licensee's rateable portion of such services from
the Waltham Facility. The monthly cost set forth in clause (ii) has been
calculated based upon the assumption of a three-year term for this Agreement.
6.4.2 COMMUNICATIONS COSTS
All hardware, software and services associated with communications between the
Waltham Facility and the Licensee Facility, including maintenance thereof, will
be contracted for by Licensor and the cost thereof is included in the monthly
fixed charge described in Section 6.4.1(ii).
6.4.3 OTHER COSTS
In the event that Licensor's personnel are obligated to travel outside of the
Greater Boston area in connection with the provision of Maintenance Services,
Data Services or
----------
*** Certain information on this page has been omitted and filed separately with
the Commission. Confidential treatment has been requested with respect to the
omitted portions.
Operations Services pursuant to this Agreement, Licensee will pay Licensor, with
respect to such travel, all out-of-pocket expenses thereof; provided that (1)
Licensor obtains Licensee's prior written approval before incurring such
reimbursable expenses; or (2) such expenses are incurred in accordance with
Licensee's then-current policy regarding such reimbursable expenses. Licensor
agrees to provide Licensee with access to such original receipts, ledgers, and
other records as may be reasonably appropriate for Licensee or its accountants
to verify the amount and nature of any such expenses.
6.5 PAYMENT TERMS
All fees and expenses described in this Section 6 will be payable, prior to
December 31, 2001, in accordance with the payment schedule set forth at Exhibit
G to the License Agreement, subject to Section 8(j) of the License Agreement. In
accordance with the provisions set forth in Section 8(h) of the License
Agreement and except as otherwise set forth herein, Licensor will invoice
Licensee for all such amounts at least thirty (30) days prior to the due date;
provided, that Licensor shall not be obligated to invoice any amounts (such as
the fee for availability data described in Section 6.3.1) the amount and timing
of which are fixed by this Agreement and/or by Exhibit G. Each such invoice will
be payable thirty (30) days after receipt by Licensee. In addition, without
limiting the foregoing, in the event Licensee fails to pay any properly invoiced
and undisputed amounts when due hereunder, and such failure continues for more
than ten business days after notice thereof by Licensor, then (i) Licensor will
be excused from all further performance of its obligations under this Agreement
until such amounts are paid in full, and (ii) Licensee will remain obligated to
pay the amounts set forth in Section 6.3 and 6.4, when and as specified therein,
for services actually performed. Any such failure will not affect the parties'
rights and obligations under the License Agreement except as specifically set
forth therein.
6.6 SERVICE LEVEL CREDITS
6.6.1 FAILURE TO RESPOND TO SEVERITY 1 OR SEVERITY 2 ERRORS OTHER THAN
RELATING TO DATA DELIVERY:
For each failure to respond or apply resources to a Severity 1 or Severity 2
Error (other than errors relating to data delivery), Licensee will be granted a
Service Level Credit (i) in the amount of $[***] if Licensor fails to respond to
a Severity 2 error pursuant to the terms of Section 2.3 and (ii) in the amount
of $[***] if Licensor fails to respond to a Severity 1 error pursuant to the
terms of Section 2.3, provided that the aggregate amount of credits for failure
to respond to a Severity 1 or Severity 2 Error (other than errors relating to
data delivery) shall not exceed a maximum of $[***] during any single month.
----------
*** Certain information on this page has been omitted and filed separately with
the Commission. Confidential treatment has been requested with respect to the
omitted portions.
For the purposes hereof and of the two succeeding sections, Licensor will be
deemed to have failed to respond and apply resources to a problem if Licensee
shall have submitted a request to the primary and secondary numbers listed in
Section 2 of the SOP and (i) Licensor shall have failed to have a Licensor
technician respond within the time period listed in Section 2.3 hereof and (ii)
Licensor fails to apply resources to resolution of such problem as required in
Section 2.3. Any such failure shall only occasion [***] Service Level Credit
unless, in the case of a Severity 1 or Severity 2 problem, such failure shall
continue for more than [***] hours, in which event Licensee shall be entitled to
an additional Service Level Credit for each [***]-hour period that such failure
continues. In no event shall Licensee be entitled to more than [***] Service
Level Credit for any [***] failure to respond to a Severity 3 or Severity 4
error, regardless of the period of time such failure continues.
6.6.2 FAILURE TO DELIVER SCHEDULE OR FARE DATA:
For each failure to deliver schedule or fare data pursuant to the provisions of
Section 5.1(a) or Section 5.1(b) respectively, Licensee will be granted a
Service Level Credit in the amount of $[***] (up to a maximum of $[***] of
credits during any single month) if Licensor fails, after such failure has
become a Severity 2 error pursuant to the terms of such section, to respond to
such error as required by Section 2.3.
6.6.3 FAILURE TO DELIVER AVAILABILITY DATA:
For each failure to deliver availability data pursuant to the provisions of
Section 5.1(c), Licensee will be granted a Service Level Credit (i) in the
amount of $[***] if Licensor fails, after such failure has become a Severity 3
error pursuant to the terms of Section 2.3, to respond to such Severity 3 error
as required by Section 2.3 and (ii) in the amount of $[***] if Licensor fails,
after such failure has become a Severity 2 error pursuant to the terms of
Section 2.3, to respond to such Severity 2 error as required by Section 2.3;
provided that the aggregate amount of credits for failure to deliver
availability data shall not exceed a maximum of $[***] during any single month.
6.7 ABUSE OF MAINTENANCE SERVICES BY LICENSEE
In the event Licensee abuses the Maintenance Services (E.G., by declaring a
problem Severity Level 1 when it was only Severity Level 3, unless such
distinction could not reasonably have been determined, or by reporting problems
which are not Severity 1 or 2 during non-business hours) (an "Abuse Incident"),
then Licensee shall pay Licensor fees for such abuse in accordance with this
Section 6.7. There shall be no charge for the first Abuse Incident occurring in
a calendar month. For each of the second through seventh Abuse Incident
occurring in such calendar month, Licensee shall pay to Licensor a fee of $150.
Thereafter, Licensee shall pay Licensor a fee of $500 for each additional Abuse
Incident occurring in such calendar month; provided that the aggregate amount of
fees for Abuse Incidents shall not exceed $2900 during any single calendar
month.
----------
*** Certain information on this page has been omitted and filed separately with
the Commission. Confidential treatment has been requested with respect to the
omitted portions.
7. GENERAL PROVISIONS
7.1 WARRANTIES
Licensor shall perform the Maintenance Services, Data Services, Operations
Services and all other services specified herein in a good, workmanlike and
professional manner using qualified personnel fully familiar with the Licensed
Software.
7.2 DESIGNATED ACCOUNT MANAGERS
Licensee and Licensor shall each appoint an account manager ("Account Managers")
who shall be responsible for all administrative matters pertaining to this
Agreement. The Account Managers will serve as primary point of contact for the
other party for any matter regarding this Agreement.
The Account Managers will initially be the following individuals:
For Licensor:
Name: Xxxxxx Xxxxx
Title: Customer Relations Manager
E-Mail: xxxxxxxxxxxxxxx@xxxxxxxxxxx.xxx
Phone: (000) 000-0000
Fax: (000) 000-0000
For Licensee:
Name: Xxxx Xxxxxxx
Title: Chief Technology Officer
E-Mail: xxxx@xxxxxx.xxx
Phone: (000) 000-0000
Fax: (000) 000-0000
Either party may replace the appointed Account Manager upon delivery, prior to
such change, to the other party of written notice of such change.
7.3 TERM
The term of this Agreement will commence as of January 1, 2001 and will
terminate contemporaneously with the termination of the License Agreement,
unless earlier terminated in accordance with this Section 7.3.
(a) Licensee may, upon 90 days' prior written notice to Licensor, terminate
Enhanced Maintenance under Section 2.2.2, in which event Licensee will,
following such
termination, be relieved of further responsibility for payment of the
charge described in Section 6.2.
(b) Licensee may, upon 90 days' prior written notice to Licensor, terminate
Data Services under Section 3.1 with respect to schedule, fare and
availability data, in which event Licensee will, following such
termination, be relieved of further responsibility for payment of the
charge described in Section 6.3 (but will remain responsible for
payment of the charges described in Sections 6.3.1, 6.3.2 and 6.4). In
the event Licensee terminates the Data Services with respect to
availability data, Licensee shall be permitted to use its own data
source in connection with the Licensed Software, subject to the
provisions of Section 6(b) of the License Agreement.
(c) Licensee may, upon one year's prior written notice (or such shorter
notice period, no less than 90 days, equivalent to the then-remaining
commitment Licensor has to Worldspan with respect to the Atlanta
Facility) to Licensor, terminate Operations Services under Section 4.1
and 4.2, in which event Licensee will, following such termination, be
relieved of further responsibility for payment of the charge described
in Section 6.4 (but will remain responsible for payment of the charges
described in Sections 6.3.1 and 6.3.2).
(d) Either party may terminate this Agreement in its entirety in the event
the other party materially breaches any of its obligations hereunder
and fails to cure such breach within thirty (30) days after receipt of
notice of such breach by the non-breaching party.
7.4 FORCE MAJEURE
Licensor will not be responsible or liable for, and will be excused from, any
non-performance or delay in the performance of any of its obligations under this
Agreement if and to the extent that such non-performance or delay (i) is caused
by an act of God, natural disaster, civil disturbance, war, fire, earthquakes,
changes in law, regulation or government policy, or non-performance by any third
party (including vendors or suppliers), or any other factor beyond the control
of Licensor, whether or not foreseeable ("Force Majeure"), and (ii) could not
have been prevented by Licensor's taking normal and customary precautions. In
the event that Licensor is excused from the performance of its obligations
pursuant to this Section 7.4, then Licensor will use its best commercial efforts
that are practicable under the circumstances to resume performance of its
obligations as soon as feasible.
7.5 EXCLUSIVITY OF REMEDIES
The provisions of this Agreement and the License Agreement constitute the
exclusive provisions applicable to Licensor's maintenance and support of the
Licensed Software and the provision and support of data and operations therefor.
Licensor disclaims any and all warranties, express or implied, including but not
limited to warranties of merchantability or fitness for a particular purpose,
except as specifically set forth herein.
7.6 NOTICES
Any notices given hereunder shall be given pursuant to and as provided in the
License Agreement. In addition, Licensee agrees that Licensor may provide
invoices to Licensee by means of facsimile or email transmission.
7.7 LIMITATION OF LIABILITY
The parties' limitations of liability set forth in Section 13 of the License
Agreement shall apply to this Agreement.
7.8 DEFAULT
If Licensor fails to perform any of its obligations under this Agreement, and
such failure continues for more than thirty (30) days following receipt of
written notice from Licensee, then Licensee, in addition to its other remedies,
shall have the right to correct the failure or to have the failure corrected by
a third party. Licensor shall reimburse Licensee for all costs incurred by
Licensee in effecting such correction, less the amount of any Service Level
Credit received by Licensee pursuant to Section 6.6 hereof. In the event such
failure relates to Severity 3 or Severity 4 errors, this Section will only apply
if such errors are chronic or continuous.
7.9 MISCELLANEOUS
(a) If any provision of this Agreement is declared by a court of competent
jurisdiction to be invalid or unenforceable, such determination shall not affect
the validity or enforceability of any other provision hereof.
(b) This Agreement, together with the License Agreement, represents the entire
agreement of the parties with respect to the subject matter hereof and any other
previous understanding, commitments, or agreement, oral or written, between
Licensee and Licensor with respect to the subject matter hereof.
(c) No failure by either party to insist upon the strict performance of any
covenant, term or condition of this Agreement, or to exercise any right or
remedy, shall constitute a waiver of such right or remedy on any subsequent
occasion.
(d) The validity, construction, scope and performance of this Agreement shall be
governed by the laws of the State of
Delaware, exclusive of its choice of law
provisions.
(e) This Agreement may not be amended except in writing executed by duly
authorized representatives of both Licensor and Licensee.
(f) This Agreement may not be assigned by either party except in connection with
and under the circumstances permitted under the License Agreement. Subject to
the foregoing, this Agreement will be binding on the parties and their
respective successors and permitted assigns.
(g) This Agreement may be signed in one or more counterpart copies, all of which
together shall constitute one Agreement and each of which shall constitute an
original.
IN WITNESS WHEREOF, the parties have executed this Agreement as of this 2nd day
of March, 2001.
ITA Software, Inc. Orbitz, LLC
By: /s/ Xxxxxx Xxxxxxxxxx By: /s/ Xxxxxxx Xxxx
----------------------- -------------------------------
Name: Xxxxxx Xxxxxxxxxx Name: Xxxxxxx Xxxx
--------------------------- ----------------------------
Title: President & CEO Title: President & CEO
--------------------------- ----------------------------
APPENDIX A
[ITA SOFTWARE INC. LOGO]
ITA Software, Inc.
Standard Operating Procedures
(SOP)
for
On Call Operating Support to
Orbitz, LLC.
[ORBITZ LLC. LOGO]
1. INTRODUCTION
The purpose of this Standard Operation Procedure (SOP) document is to establish
a defined approach for the reporting and resolution of problems related to the
operational use of software (the "Licensed Software, as defined in the License
Agreement referred to below) licensed to Orbitz, LLC by ITA Software, Inc.
pursuant to a Software License Agreement dated as of July 1, 2000 and a Software
Maintenance, Data
Services And Operations Service Level Agreement dated as of
January 1, 2001 (the "SLA"). Primary objectives are:
1. To facilitate the rapid communication of problems by Orbitz personnel
to appropriate ITA personnel.
2. To document problem symptoms, manifestations, and impacts in sufficient
detail to facilitate rapid problem diagnosis and resolution.
3. To provide a methodology for tracking logged Orbitz problems, their
current status, and their interim and final resolutions.
The primary goal of ITA Software's Operational support team is to provide
unparalleled customer satisfaction through the timely diagnosis and resolution
of operations related problems.
A. SCOPE OF PROBLEMS
In the context of this SOP, a problem refers to some part of the Licensed
Software, supporting user interfaces, or data delivery components that are not
working as intended, in turn causing some functionality to be lost or severely
impaired, data to not be delivered, or connectivity to be unavailable.(1)
Sometimes users may experience operational difficulties that can be attributed
to "operator error" or lack of understanding with regards to system features and
capabilities. Although incidents of this type do not indicate a problem with the
software per se, the "incident" will still be logged and tracked. Review of the
user assistance logs will help identify possible deficiencies in user
documentation, training, or even the user interface itself.
2. HOW A PROBLEM IS REPORTED
User problems can be reported in one of two ways, depending on the severity
level of the problem. In cases where problems are non-critical (severity levels
3 and 4 as defined in the SLA) and thus can be deferred for resolution during
normal working hours (9:00 am to 6:00 pm EST, Monday through Friday), the best
reporting vehicle is an email to xxxx@xxxxxxxxxxx.xxx. The following information
must be included in all e-mails:
- Contact Information (NAME, ORGANIZATION, PHONE NUMBER, E-MAIL)
----------
(1) One distinction that can be made is that some issues are not resolvable via
a problem resolution process but are instead due to a lack of a particular
feature from the software. However, once a problem has been entered into the
problem tracking system, it will remain in the system and will be reclassified
as a feature request if appropriate.
- System Being Used (SOFTWARE COMPONENT OR DATA FEED COMPONENT)
- Problem Description (INCLUDING SYMPTOMS AND MANIFESTATIONS).
- Frequency With Which The Problem Can Be Repeated (ALWAYS, SOMETIMES,
RARELY, NEVER).
- Impact Of The Problem (FATAL, SEVERE, MINOR, COSMETIC)
- Date/time problem first occurred.
For problems that are considered mission critical or high impact (severity
levels 1 and 2 as defined in the SLA)) and need immediate attention, the most
appropriate reporting vehicle is ITA Software's Problem Reporting Hotline. The
following Hotline numbers are to be used:
Primary: [***]
Secondary Numbers:
[***] (Xxxxx Xxxxxx - fare/schedule loads and LFS)
[***] (Xxxxx Xxxxx - availability system and monitoring systems)
[***] (Xxxxx Xxxxxxxxx - networking or security issues)
[***] (Xxx Xxxxxxxx)
During normal working hours an ITA representative will be available to answer
the primary number. If for some unforeseen reason a technician is not available
to answer immediately, the system will automatically forward the incoming call
to a numeric pager or cellular phone. If connected to a pager number, a prompt
will request entry of a number for call back. If connected to a cell phone voice
mail system, a prompt will request the caller to leave a voice message. Should
the primary number fail (telephone service unavailable), the secondary numbers
can be used. If immediate technical support is required during off-duty hours
(severity levels 1 or 2), the same Hotline number should be used. The system
will automatically page or forward the call to an on-call technician who will
respond within the time frames outlined in the SLA. Again, should the primary
number fail (telephone service unavailable, e.g.), the secondary numbers can be
used.
Once connected with the ITA technician, provide an explanation of the problem in
the same format as would be provided via email. In all cases, reported problems
will be documented in the problem-tracking system and assigned to the
appropriate personnel for resolution.
Once the incident is logged into the problem-tracking system, a ticket number
will be generated and provided to the submitter of the problem. Additional
amplifying information may be requested at that time.
3. HOW A PROBLEM IS DIAGNOSED
----------
*** Certain information on this page has been omitted and filed separately with
the Commission. Confidential treatment has been requested with respect to the
omitted portions.
ITA Software's Operations staff has developed a well defined, tailored, and
consistent procedure for the diagnosis of operational problems related to the
use of the Licensed Software. These procedures will also be applied for problems
associated with system interfaces or data feeds.
A. SUMMARY OF THE DIAGNOSTIC PROCEDURE
The diagnostic procedure will consist of five main phases:
Phase One: establish communications
Phase Two: classify the problem
Phase Three: perform appropriate tests
Phase Four: diagnose the problem
Phase Five: agree on an appropriate course of action
In the course of coordinating problem resolution, our interaction with Orbitz
personnel may not take such a clear-cut path. However, all of these elements
will be implicit in the interaction, and will be documented in the problem
tracking log.
B. PHASE ONE: ESTABLISH COMMUNICATIONS
In order to help diagnose specific problems, ITA Software will need to be
notified of the problem and provided sufficient detail to start investigating an
acceptable solution. This should be done via the procedures described above. It
is recognized, however, that there may be situations where detailed information
related to the problem might not be available. Best judgment may dictate that
the problem be reported immediately, without in-depth diagnosis or gathering of
information for a comprehensive problem report submission.
C. PHASE TWO: CLASSIFY THE PROBLEM
The next step in the diagnostic process, after receiving a problem report, is to
classify the problem, in two categories, by TYPE and COMPLEXITY LEVEL.
The TYPE classification category describes whether the problem is related to
data transmission or a software component. Examples of data transmission
problems would be an inability to receive updated Low Fare Search distributions
or an interruption in the availability data flow.
The second category describes the COMPLEXITY LEVEL (not to be confused with
severity levels) involved in developing a problem solution. This is an initial
estimate of whether the problem is a First-level problem (i.e. it can be solved
on the spot) or a Second- or Third-level problem (requires some level of effort
for resolution).
D. PHASE THREE: PERFORM APPROPRIATE TESTS
The Licensed Software contains some internal status indicators and diagnostic
tests that can be used as first-stage diagnostic tools. ITA may ask Orbitz
personnel to
provide information derived from internal status indicators. In certain cases,
ITA may require test results via prearranged electronic methods (e.g. ftp,
e-mail). Based on the results, ITA may request that Orbitz run additional tests.
E. PHASE FOUR: DIAGNOSE THE PROBLEM
After ITA has analyzed the results of the diagnostic tests(s) performed by
Orbitz personnel a technical analysis will be performed to determine potential
causes. This effort will seek to identify internal or external causes, and will
seek to pinpoint the specific trouble area within a particular software
component or data transmission procedure. In order to effectively analyze the
problem it may be necessary to gain access to systems at the Orbitz facility.
Ideally, this will be done remotely but there may be occasions where an on-site
visit will be required.
F. PHASE FIVE: AGREE ON AN APPROPRIATE COURSE OF ACTION
Once a problem has been isolated and defined, discussions with Orbitz personnel
will be held to determine an appropriate course of action to resolve the
problem.
If the problem has been classified as a First-level problem, then immediate
steps can be taken to address the problem. Even though the problem may be
solvable "on the spot", the established reporting procedures will still be
followed so that the problem tracking system is updated for
recording/administrative purposes and to aid in future diagnosis and trend
analysis. Some trends may point towards deficiencies in the usage of the
software that can be addressed via further communications or changes in software
procedures or operations.
In cases where a specific problem is classified in the Second- or Third- level
category, i.e., changes are required that involve updates to a software
component, or major network hardware or procedural changes, a proposed problem
resolution schedule and approach will be submitted to Orbitz for review and
approval.
4. PROBLEM TRACKING SYSTEM
In order to maintain customer satisfaction and meet service level requirements,
it is necessary to resolve problems as quickly as possible, while keeping track
of the status of the resolution. The assignment of resources to a problem takes
into account the type of problem, the available resources, and the impact level
of the problem.
ITA will categorize, track, and document on-going Orbitz user problems using a
customized version of Bugzilla. This package has been modified to meet the
unique requirements posed by Orbitz's licensing of ITA's product offerings.
A. DEFINITIONS
A "PROBLEM" refers to either a disruptive event, or an incident of
non-functioning
hardware or software that indicates some part of the SYSTEM(2) is not working
as designed, or points to a desire for new functionality. Alternatively, an
Orbitz user may encounter a "problem" when he or she does not use the system
properly. In this case the solution is user education, but on-site help systems
and documentation may need to be improved.
An "INCIDENT" refers to a specific instance of a problem, and how it affects the
rest of the system, data feeds, and customer. If an incident points to an
operational First- or Second- level problem, the resolution will fall into the
category of "Incident Response".
B. CATEGORIZATION OF PROBLEMS (NOT TO BE CONFUSED WITH SEVERITY LEVELS)
Problems will be categorized into first-level problems, second-level problems,
and third-level problems. These are defined as follows:
FIRST-LEVEL PROBLEMS are those that can be solved "on-the-spot" by an operator
or staff member, or are problems that are as a result of user error. In cases of
user error, we will work with Orbitz personnel to ensure a full understanding of
the functionality and capabilities of the supported systems. In any case,
First-level Problems are those that are fixable in a short amount of time and do
not require extensive effort. They do not need to be tracked if the problem can
be fixed "on-the-spot".
SECOND-LEVEL PROBLEMS are those that cannot be fixed "on-the-spot" or must be
referred to another staff member in order to be fixed. However, the problem does
not take an extensive amount of effort to be fixed and the solution is within
the scope of a staff members' current responsibilities, capabilities, or within
the current budget for the particular resources that will be required.
THIRD-LEVEL PROBLEMS are problems that require a non-trivial amount of effort or
new resources to solve. The solution to these problems requires the assignment
of a dedicated staff resource or the allocation of unbudgeted resources.
C. FIRST-LEVEL PROBLEM MANAGEMENT LOGGING
First-Level problem management is the most basic method of tracking implemented
by the Bugzilla system. This Bugzilla feature allows for a TICKET to be created,
along with a ticket number. The basic information about the problem is stored
with the ticket. In addition, the problem resolution and any suggested but
non-necessary changes to documentation, procedures, or software components are
stored. Since, in most cases, a problem is resolvable "on-the-spot", the ticket
status is stored as RESOLVED. Furthermore, the PROBLEM RESOLUTION CODE is also
recorded as FIXED, INVALID, WONTFIX, or LATER. See "Descriptions of Problem
Resolution Codes" below for a description of these codes.
----------
(2) SYSTEM refers to the many components that make up the services ITA provides:
data feeds from suppliers, data feeds to customers, the web site, the Low Fare
Search engines, and all of the other software components that work together to
deliver low fare searches.
D. ESCALATION MANAGEMENT - SECOND- OR THIRD- LEVEL PROBLEM RESOLUTION
If a problem cannot be solved "on-the-spot" (i.e. the problem is a Second- or
Third- level problem), the results of initial efforts to resolve the problem
will be recorded in the ticket and the problem will be escalated to appropriate
technical personnel. Also recorded in the initial ticket will be a proposed
"next step" for resolution. This may be as simple as "problem will be/has been
escalated". Problems that are escalated will be dealt with in accordance with
the procedures described in the "Incident Response" section below. Finally, the
ticket will be assigned a status code as described in the "Description of STATUS
CODES" section below.
E. SUMMARY REPORT CAPABILITIES
Bugzilla's problem tracking system also gives our operations staff the
capability to print summary reports based on a number of criteria, including
ticket number, keyword, and ticket status code. The items in each report can
include the status code, intermediate resolution or final resolution, as well as
other information stored for each ticket.
F. DESCRIPTIONS OF PROBLEM RESOLUTION CODES
INFORMATION
This problem was due to misuse of the software component or data feed
and was resolved by giving the user information.
FIXED
A fix for this bug has been implemented into the software component or
data feed and tested by the person marking it FIXED.
INVALID
The problem described is not a bug, or not a bug in an ITA Software
component or data feed.
WONTFIX
The problem described is a bug that will never be fixed.
LATER
The problem described is a bug that will not be fixed in this version
of the product.
REMIND
The problem described is a bug that will probably not be fixed in this
version of the product, but might still be.
DUPLICATE
The problem is a duplicate of an existing bug. Marking a bug duplicate
requires the bug number of the duplicating bug and will add a
comment with the bug number into the description field of the bug it is
a duplicate of.
WORKSFORME
All attempts at reproducing this bug were futile. If more information
appears later, please re-assign the bug...for now, file it.
MOVED
The bug was specific to a particular derivative distribution and didn't
affect ITA code. The bug was moved to the bug database of the
distributor of the affected derivative.
G. DESCRIPTIONS OF STATUS CODES
UNCONFIRMED
The initial problem report has been received, but no contact has been
made with the reporter of the bug, nor has the problem been
independently checked.
NEW
A contact has been made with the reporter of the bug to make sure this
is not a First-level problem and to clarify and misunderstandings about
the initial problem report. The efforts described in the Diagnostic
Procedures document have led to a conclusion that the problem report is
valid.
ONGOING
A problem has been assigned for resolution. Progress is being made and
a schedule for completion has been set.
RESOLVED
When a problem will not have any further work performed on it during
this software release, due to the problem being fixed or for some other
reason, the ticket is given this Status Code and the Problem Resolution
Code is updated appropriately (see Appendix A for a description of the
Problem Resolution Codes).
CLOSED
When a new software version or data delivery method is released that
includes the resolution to the problem, the status is changed from
RESOLVED to CLOSED.
REOPENED
If some event has caused the resolution to the problem described in the
ticket to become invalid, the ticket's status is changed to REOPENED.
5. INCIDENT RESPONSE
Incident response begins with discovery of a problem, either by ORBITZ
personnel, or as a result of data obtained through the use of ITA monitoring
tools. Once information is received that points to an ongoing operational
problem, on-call operations engineers will apply systematic diagnostic
procedures to triage the problem and determine whether or not there are any
available workarounds. If it is confirmed to be a valid problem of an
operational nature that either has an immediate adverse impact or has the
potential to present a future adverse impact, the problem will be classified as
an "Incident".
The work to resolve incidents is started as soon as possible, using best
commercial practices. If additional personnel resources are required to address
the incident, they are notified. The Reporting Procedures are still followed
(i.e. the problem receives a ticket number and its status is still tracked,
etc.).
After a problem has been reported and diagnosed, it will either be given a
status of "New" or "Resolved". For any problems that are ongoing, an
INTERMEDIATE RESOLUTION will be recorded in the problem tracking system. If the
customer wishes, this intermediate resolution can include a notation that the
problem has been escalated.
Following resolution of the problem, ORBITZ personnel will be notified prior to
officially closing out the trouble ticket. ORBITZ personnel will be afforded the
opportunity to test the system to verify resolution prior to authorizing
close-out.
6. POINTS OF CONTACT
The following administrative points of contact are provided to answer any
questions related to ongoing support as described in this SOP.
Xxx Xxxxxxxx, VP - Operations: [***]
Xxxxx Xxxxxx, Senior Network Engineer: [***]
Xxxxx Xxxxx, Director - Datacenter Operations: [***]
----------
*** Certain information on this page has been omitted and filed separately with
the Commission. Confidential treatment has been requested with respect to the
omitted portions.
APPENDIX B
[ITA SOFTWARE LOGO]
ITA
DISASTER
RECOVERY
PLAN
(DRP)
TABLE OF CONTENTS
I. INTRODUCTION.......................................................................29
II. RESPONSIBILITIES...................................................................29
A. Network Infrastructure..........................................................29
1. SPECIFIC RESPONSIBILITIES....................................................29
B. Datacenter Management...........................................................30
1. SPECIFIC RESPONSIBILITIES....................................................30
C. Data Support....................................................................30
1. SPECIFIC RESPONSIBILITIES....................................................30
D. Quality Assurance...............................................................31
E. Marketing and Public Relations..................................................31
1. SPECIFIC RESPONSIBILITIES....................................................31
F. Customer Relationship Management (CRM)..........................................31
2. SPECIFIC RESPONSIBILITIES....................................................31
III. IMPACT ANALYSIS (RISKS).........................................................32
G. Minimum Acceptable Levels of Service:...........................................32
IV. DETAILED CONTINGENCY PLAN.......................................................32
A. Disaster Detection and Determination............................................33
B. Disaster Notification...........................................................33
C. Initiation of the DRP...........................................................33
D. Activation of a Designated Hot Site.............................................33
E. Dissemination of Public Information.............................................33
V. DISASTER RECOVERY STRATEGIES AND RESPONSES.........................................34
A. General Contingency Strategy....................................................34
B. Specific Contingency Strategies.................................................34
1. LOSS OF MASTER AVAILABILITY CACHE SYSTEM (MAVS) AT THE ATLANTA DATACENTER....34
2. LOSS OF WALTHAM DATACENTER...................................................34
VI. RESTORATION OF SERVICES.........................................................35
I. INTRODUCTION
Uninterrupted quality of service to ITA Software's customers is the most
important focus of ITA Software's Operations Team. As such, operational
continuity, in the aftermath of a disaster, requires comprehensive disaster and
contingency planning. This planning must result in the development of
methodical, yet flexible, pre-planned responses, which can be implemented
rapidly so as to minimize downtime. Pre-staging of resources that can be quickly
installed, configured, and activated is also a critical ingredient of an
effective disaster response. Preparation for, response to, and recovery from a
disaster will require the cooperative efforts of many support organizations who
are critical partners with ITA Software in providing the superior functional
capabilities our customers have grown to expect. This document provides the plan
to coordinate disaster recovery efforts and describes contingencies and computer
recovery facilities that will be employed in the case of catastrophic outages to
critical infrastructure.
II. RESPONSIBILITIES
This Disaster Recovery Plan (DRP) specifies the responsibilities of ITA
Software's Disaster Response Team (DRT), whose mission is to establish and
implement procedures to ensure the continuity of our customer's business
functions. In the event of a disaster affecting any of the operational
functional areas, the DRT will serve as the liaison between ITA Software's
operational component organizations and partnering support organizations. These
services include data providers, telecom providers, datacenter hosting services,
and internally, ITA Software's Marketing, Quality Assurance (QA), and Customer
Relationship Management (CRM) departments.
ITA Software's Vice President for Operations, with assistance from key
operations support personnel, is responsible for developing the DRP and overall
coordination of disaster recovery efforts. Development and support of individual
component DRPs, and coordination of their implementation, are the responsibility
of each of the functional area coordinators.
A. NETWORK INFRASTRUCTURE
Ensuring that the DRP reflects ongoing changes to network topologies,
communications infrastructure, hosting support arrangements, and resources is
crucial. This task includes updating the plan and revising this document to
reflect updates. ITA Software's Network Engineer/Administrator is responsible
for this comprehensive task.
1. SPECIFIC RESPONSIBILITIES
a. Provides alternate voice and data communications capability in the
event normal telecommunication lines and equipment are disrupted by the
disaster. Evaluates the requirements and selects appropriate means of
reestablishing ITA Software's telecommunications network connectivity.
b. Coordinates all efforts in support of the restoration of network
services and support facilities. This includes adjustments/changes to
the network topology and addressing schema, reconfiguration of routers
and firewalls, and notifications to outside contractors and service
providers to ensure that
replacement equipment and materials are available for timely delivery
and installation.
c. Coordinates all efforts for the restoration of data and electrical
systems and structural integrity. Assesses damage and makes a prognosis
for occupancy of the structure affected by the disaster.
B. DATACENTER MANAGEMENT
The Director of Datacenter Operations has the primary role of ensuring the
transition of datacenter operations to the designated backup facility within
established timeframes (normally 48 hours).
1. SPECIFIC RESPONSIBILITIES
a. Following the assessment of damage, the Director of Datacenter
Operations will salvage equipment, data, and supplies following a
disaster; identify which resources remain; and determine their future
utilization in rebuilding the datacenter and recovering from the
disaster.
b. The Director of Datacenter Operations will make arrangements for
transporting personnel, equipment, and materials to back-up sites as
necessary. Pre-staged resources at the backup facility will be
augmented by resources salvaged from the primary site, if required.
c. Provides data processing installation and configuration resources
necessary for restoration of operational capability to end users.
Assesses computing resources available and reconfigures as necessary to
restore functional capability.
d. Responsible for activating the designated backup site or back-up
resources.
C. DATA SUPPORT
The Data Support Director is responsible for coordinating the rerouting or
redistribution of all critical data required for sustaining ITA Software's
functional support to end users.
1. SPECIFIC RESPONSIBILITIES
a. Coordinates support for data processing and delivery at the primary
datacenter and the designated backup sites. Initiates discussion with
data vendors (Innovata, ATPCO, and Worldspan) to provide backup routing
information.
b. Coordinates routing changes and modification of delivery procedures to
ensure end-user access to critical operational data.
D. QUALITY ASSURANCE
Testing of the DRP is an essential element of preparedness. ITA Software's QA
Department will be responsible for conducting regular testing of individual
components and recovery plans of specific functional teams. A comprehensive
exercise of our continuity capabilities and support by our designated recovery
facilities will be performed on an annual basis.
E. MARKETING AND PUBLIC RELATIONS
ITA Software's Vice President of Marketing and Public Relations is responsible
for arranging and directing all meetings and discussions with the news media and
the public.
1. SPECIFIC RESPONSIBILITIES
a. Communicates with the news media, public, and staff who are not
involved in the recovery operation.
b. Prepares press releases on a periodic basis for distribution to ITA
Software's media list.
c. Maintains a log of all incoming calls to ensure a quick response to
media and other requests for information.
d. Coordinates follow-up news releases to recap events and describe what
has been done to reestablish operations.
e. Coordinates media strategies and press releases with public relations
staff of vendors and customers as appropriate.
F. CUSTOMER RELATIONSHIP MANAGEMENT (CRM)
The Vice President for CRM will ensure a roster of all critical customer
personnel is maintained, with contact and notification information. The
individual CRM representatives will be responsible for providing immediate
outage notification and status reports to customers for whom they have
responsibility.
1. SPECIFIC RESPONSIBILITIES
a. Individual CRM representatives will establish immediate communication
with supported customers, outlining circumstances of event, impact to
customer,
and restoral timelines.
b. Provide periodic status reports to assigned customers to update status
and plans for restoration.
c. Provide 24-hour contact numbers for customer inquiries and information
requests.
III. IMPACT ANALYSIS (RISKS)
ITA Software recognizes the low probability of severe damage to datacenter
operations supporting both our internal and external customers. Nevertheless,
because of the potential adverse impact to our end-users, a plan for reducing
the risk of damage from a disaster, however unlikely, is vital. This DRP is
designed to reduce the risk to an acceptable level by ensuring the restoration
of critical processing within 48 hours, and all essential production within 2
week(s) of the outage.
This DRP identifies the critical functions of ITA Software and the resources
required to support them. The DRP provides guidelines for ensuring that needed
personnel and resources are available for both disaster preparation and response
and that the proper steps will be carried out to permit the timely restoration
of services.
A. MINIMUM ACCEPTABLE LEVELS OF SERVICE:
In order to ensure the continued availability of services for our customers a
few key components of the ITA suite of sever products must be in place and
operational. These include: QPX servers, Query distributor(s), a query firewall,
AVS caches, and ATPCO and SSIM delivery build servers. With all of these
services running in both the primary and backup datacenters, the availability of
the services offered is ensured.
Although casualty modes of operation will provide continued functionality for
end-user applications, it should be understood that backup services might not
initially provide the same level of performance as that experienced during
normal operations. Degraded operations, due to disasters as those described in
this DRP, will be characterized by identical functionality with degraded
performance.
Both of ITA Software's current datacenters (Waltham and Atlanta) contain
machines in reserve in the unlikely occurrence of a disaster or network failure.
The reserve system, upon failure of a datacenter, will be brought on-line within
48 hours of the declaration of a disaster.
IV. DETAILED CONTINGENCY PLAN
This section describes the required steps to be taken in response to a disaster:
1. Detect and determine a disaster condition
2. Notify persons responsible for initiating the DRP.
3. Initiate the DRP.
4. Activate the designated backup site or services.
5. Disseminate public information.
A. DISASTER DETECTION AND DETERMINATION
The determination of an event that could be classified as a disaster is the
responsibility of the Vice President of Operations and involves obtaining access
to the affected site, determining the extent of the damage, and making
recommendations to upper management regarding possible reactivation and/or
relocation of datacenter or user operations.
B. DISASTER NOTIFICATION
The Vice President of Operations will act as the Disaster Response Team
Coordinator (DRTC) and follow existing procedures, notifying DRT members that
the DRP has been put into effect. The DRC will continually monitor the evolving
situation and direct and or obtain resources as necessary to effect the
transition to the backup facility.
At a minimum, the Disaster Response Team Coordinator will contact the following
people immediately:
1. The Director of Datacenter Operations.
2. The Network Engineer/Administrator.
3. The Director of Data Support Operations.
4. The appropriate CRM Representative.
5. The Vice President of Marketing
C. INITIATION OF THE DRP
Initiation of the DRP and its subcomponent plans is the responsibility of the
DRTC and DRT functional coordinators, respectively.
D. ACTIVATION OF A DESIGNATED HOT SITE
The responsibility for activating the designated backup site or back-up
resources is delegated to the Director of Datacenter Operations (DDCO). Within 2
hours of the event, the DDCO will determine the prognosis for recovery of the
lost functionality through consultation with the other DRT members.
If the estimated recovery of the lost functionality cannot be accomplished
within 8 hours, the transition to the backup facility will begin.
E. DISSEMINATION OF PUBLIC INFORMATION
The Vice President of Marketing and Public Relations is responsible for
directing all meetings and discussions with the news media and the public, and
in conjunction with the Director of Customer Relationship Management, ensuring
that affected customers are immediately notified of the event and its impact. A
dedicated phone number will be established as a means for providing recovery
status and information.
V. DISASTER RECOVERY STRATEGIES AND RESPONSES
Current Service Level Agreements (SLAs) with contracted hosting services and
data providers define well defined escalation procedures to be applied in the
case of system outages. Both of our datacenter vendors, Exodus Communications
and Worldspan, LLC., have significant infrastructure to provide for casualty
modes of operation. Both vendors provide significant capability to sustain
operations in the case of the loss of power, internet connectivity, or other
site related issues. As an initial step in evaluating options for casualty
operations, datacenter providers will be contacted for assistance in
establishing failover operations.
A. GENERAL CONTINGENCY STRATEGY
Due to both the primary and backup site each having a number of machines in
reserve, the restoration of services will require augmenting the failover
datacenter systems with salvaged assets and building the lost services in the
unaffected datacenter. All application code that is necessary for operation will
be kept on imaging servers in each location allowing expedited building of any
type of service. In the case of a loss of connectivity to any of the
datacenters, the DRT will immediately be notified via pager and email by the
multi-homed monitoring software in place. At the point at which a failure in
service is confirmed by any member of the DRT team, the issue will be elevated
to the DRT Coordinator who will ascertain whether or not the situation should be
classified as a disaster. Once determined that a disaster situation has
occurred, the DRT Coordinator will contact functional coordinators with
instructions to implement the DRP.
B. SPECIFIC CONTINGENCY STRATEGIES
1. LOSS OF AVAILABILITY CACHE SYSTEM (AVS) AT THE ATLANTA DATACENTER
The loss of the AVS at the Atlanta Datacenter will adversely impact ITA
Software's capability to provide timely availability data to supported
customers. Upon a confirmed disaster taking place at the Atlanta Datacenter, the
AVS core services will be moved to the Waltham Datacenter. The following tasks
must be completed in order to implement this contingency plan:
a. Determine extent of damage to Kinetics System, if any.
b. If Kinetics damaged, coordinate with Worldspan for direct
access.
c. Coordinate with Worldspan to reroute availability data to
Waltham
d. Bring up the Waltham Availability Cache as Master
e. Reroute AV data feeds to Waltham Datacenter
f. Coordinate with Customers to redirect availability related
communications to Waltham
g. Start AVL and DirNVQ processes at Waltham
h. Update DNS accordingly
2. LOSS OF WALTHAM DATACENTER
The loss of the Waltham Datacenter will have an immediate negative impact on all
customers. The following services will cease to be available to ITA-managed
customer implementations should the Waltham Datacenter suffer catastrophic loss:
Low Fare Search
Web Site Hosting
XML access to Query Distribution System
Primary Availability Cache
Additionally, the following services will cease to be available to ITA-managed
customer implementations, as well as customer-managed implementations that
utilize ITA fare and schedule data:
Fare data access
Schedule data access
Upon a confirmed disaster taking place at the Waltham Datacenter all QPX
services and supported infrastructure will be moved to the Atlanta Datacenter.
Web servers will be relocated as well. Fare and Schedule delivery will be moved
to Worldspan. The following tasks must be completed in order to implement this
contingency plan:
a. Activate current versions of services at Waltham Datacenter on
redundant machines.
b. Adjust monitoring to Waltham Datacenter
c. Update DNS accordingly so as to ensure receipt of fare and
schedule data in Atlanta.
d. Notify customers
VI. RESTORATION OF SERVICES
The time required for recovery of the primary/secondary datacenter and the
affected functional areas, and the eventual restoration of normal processing
depends on the damage caused by the disaster. The time frame for recovery might
vary from several days to several months. In either case, the recovery process
will begin immediately after the disaster and take place in parallel with
back-up operations at the designated backup site. The primary goal is to restore
normal operations as soon as possible.
