STIPULATION AND CONSENT AGREEMENT
Exhibit 99.1
UNITED STATES OF AMERICA
FEDERAL ENERGY REGULATORY COMMISSION
|
Arizona Public Service Company ) |
Docket No. IN14-6-000 |
STIPULATION AND CONSENT AGREEMENT
I. INTRODUCTION
1. Staff of the Office of Enforcement (Enforcement) of the Federal Energy Regulatory Commission (Commission), the North American Electric Reliability Corporation (NERC), and Arizona Public Service Company (APS) enter into this Stipulation and Consent Agreement (Agreement) to resolve a non-public investigation conducted by Enforcement and NERC pursuant to Part 1b of the Commission’s regulations, 18 C.F.R. Part 1b (2013). The investigation examined possible violations of NERC Reliability Standards by APS related to a system event in the Pacific Southwest on September 8, 2011 (September 8 event or event). APS neither admits nor denies that it violated the Reliability Standards described in the Agreement, but agrees to pay a civil penalty of $3,250,000, of which $2,000,000 will be paid to the United States Treasury and NERC, divided in equal amounts, and $1,250,000 will be invested, subject to Enforcement and NERC approval, in reliability enhancement measures identified below that go above and beyond the Agreement’s mitigation commitments or what the Reliability Standards require (Reliability Enhancements). APS also commits to mitigation and compliance measures, subject to compliance monitoring, as detailed in the Agreement.
II. STIPULATED FACTS
2. Enforcement, NERC, and APS hereby stipulate and agree to the following facts.
A. Arizona Public Service
3. APS is a vertically integrated utility that serves a 35,000 square mile territory spanning 11 of Arizona’s 15 counties. Pertinent to this investigation, APS is registered with NERC as a Transmission Operator (TOP) and Balancing Authority (BA). APS owns and operates transmission facilities at the 500, 345, 230, 115, and 69 kV levels, and owns approximately 6,300 MW of installed generation capacity. APS also operates distribution networks and retail energy services, and its peak load in 2011 was 7,087 MW.
B. Event Description
4. During an 11-minute period on the afternoon of September 8, 2011, a system disturbance occurred in the Pacific Southwest, resulting in cascading outages and leaving approximately 2.7 million customers without power, some for multiple hours extending into the next day. The total load loss for the event was in excess of 30,000 MWh. The event started with a three-phase fault which led to the loss of APS’s Hassayampa-N. Gila 500 kV transmission line (H-NG). This transmission line is a segment of the Southwest Power Link (SWPL), a major transmission corridor transporting power in an east-west direction, from generators in Arizona, through the service territory of Imperial Irrigation District (IID), into Southern California.
5. With the SWPL’s major east-west corridor broken by the loss of H-NG, power flows instantaneously redistributed throughout the electric system in the Pacific Southwest and Southern California, increasing flows through lower voltage systems parallel to the SWPL as power continued to flow on a hot day during hours of peak demand.
6. These redistributed flows traveled through IID’s and Western Area Power Administration-Desert Southwest’s (Western-DSW’s) facilities, onto Western Electricity Coordinating Council (WECC)(1) Path 44, an aggregation of five 230 kV transmission lines that deliver power in a north-south direction from Southern California Edison Company’s (SCE’s) territory in Los Angeles to San Diego. The increased power flows parallel to the SWPL, together with lower than peak generation levels in California and Mexico, led to significant voltage deviations and transmission equipment overloads. The flow redistributions, voltage deviations, and resulting overloads had a cascading effect, as transmission and generation equipment tripped offline in a relatively short time period. Just seconds before the blackout, Path 44 carried all flows into San Diego as well as parts of Arizona and Mexico. This excessive loading on Path 44 initiated an intertie separation scheme owned and operated by SCE at the San Xxxxxx Nuclear Generating Station (SONGS) in Southern California. The California Independent System Operator (CAISO) is responsible for many of the TOP functions for SCE under a Coordinated Functional
(1) At the time of the event, WECC was registered with NERC as the Reliability Coordinator (RC) for all of the entities affected by the event, as well serving as the Regional Entity (RE) under delegation agreement with NERC. Since the event, the Regional Entity and Reliability Coordinator functions have been bifurcated, with WECC remaining the Regional Entity, and Peak Reliability becoming the independent Reliability Coordinator. See Order on Compliance, 146 FERC ¶ 61,092 (2014) (accepting compliance filings submitted by NERC and WECC and eliminating all final obstacles to bifurcation). The Agreement will refer to WECC when relevant to the event, and will otherwise refer to the relevant function (RE or RC) rather than using the entity names WECC or Peak Reliability.
Registration.(2) Initiation of the intertie separation scheme at SONGS separated San Diego Gas & Electric (SDG&E) from Path 44, contributed to tripping the SONGS nuclear units offline, and eventually resulted in the complete blackout of San Diego and Comision Federal de Electricidad’s (CFE’s) Baja California Control Area.
7. APS’s role in the event began before the loss of H-NG. APS did not perform unique next-day studies. APS relied on its Arizona Security Monitoring Manual (ASMM) as its next-day study for September 8, because the studies underlying the ASMM considered the most severe likely contingencies affecting APS, including the loss of H-NG, and considered some stressed system conditions (although different from those conditions seen on September 8). Although WECC Path 49, which includes H-NG, was being operated below its nominal system operating limit (SOL) on September 8 before H-NG tripped, after H-NG tripped, neighboring TOPs IID and Western-DSW immediately experienced overloads beyond emergency ratings (IID’s Coachella Valley transformers) and voltage drops of over five percent (Western-DSW’s Blythe and Kofa transformers). After the loss of H-NG, APS operators believed APS would be able to bring H-NG back into operation within minutes, and told the WECC RC operator so. However, immediately after H-NG tripped, the phase angle difference between H-NG’s two terminals increased from 20 degrees to approximately 72 degrees. Because of the excessive phase angle, APS’s synchro-check relay could not have reclosed H-NG until APS changed the relay setting or until generation was backed down or load reduced.
III. INQUIRY AND INVESTIGATION
8. On September 9, 2011, the Commission and NERC announced a joint inquiry to determine how the blackout occurred and to make recommendations to avoid similar situations in the future. The inquiry team, comprised of Commission and NERC staff, used on-site visits and interviews, detailed computer modeling, event simulations, and system analyses to make its findings and recommendations for preventing similar events in the future. The inquiry determined that entities responsible for planning and operating the Bulk-Power System (BPS) were not prepared to ensure reliable operation or prevent cascading outages in the event of a single contingency. On May 1, 2012, the inquiry team published a report entitled Arizona-Southern California Outages on September 8, 2011, Causes and Recommendations (the Report), which is hereby incorporated by reference.(3) The Report discusses a detailed sequence of events, simulations, and findings related to the causes of the cascading outages. The Report also makes twenty-seven
(2) JRO00009 was originally entered into on September 11, 2008 and most recently updated on May 24, 2012. JRO00009 delineates compliance responsibility for the Standards and Requirements associated with the TOP function between CAISO and SCE.
(3) Arizona-Southern California Outages on September 8, 2011, Causes and Recommendations (April 2012), available at xxxx://xxx.xxxx.xxx/xxxxx/xxxxx-xxxxxxx/00-00-0000-xxxx-xxxx-xxxxxx.xxx.
recommendations related to next-day planning, seasonal planning, near- and long-term planning, situational awareness, consideration of Bulk Electric System (BES) equipment, SOLs and Interconnection Reliability Operating Limits (IROLs), and protection systems.
9. Following publication of the Report, Enforcement and NERC reviewed the data gathered during the inquiry for compliance implications. As a result of that review, Enforcement and NERC initiated non-public investigations of several entities, including APS, under Part 1b of the Commission’s regulations, 18 C.F.R. Part 1b (2013). Enforcement and NERC determined that APS violated four Requirements of two Reliability Standards and found that these violations undermined the reliability of the BPS and contributed to the September 8 event. Enforcement and NERC recognized, however, that after the event, and during the inquiry and investigation, APS voluntarily began making improvements in its planning and operations procedures, and implementing recommendations from the Report, that addressed many of the findings arising from the Report. These efforts included developing and implementing human performance and process improvements with regard to switching operations. APS has also worked, in cooperation with other entities, to improve the coordination of transmission system operations in the West. APS established a morning system operators’ conference call with the Pacific Southwest entities and weekly calls with chief dispatchers during peak seasons, increased Inter-Control Center Communications Protocol (ICCP) data exchanges with other TOPs and the RC to increase regional situational awareness, and worked with others in the West to enhance information sharing about equipment outages, load, and system conditions. APS has also undertaken efforts, through an expanded Real-Time Contingency Analysis (RTCA) and other actions, to increase its situational awareness beyond the APS system. In addition, APS fully and comprehensively cooperated with Enforcement and NERC during the investigation. APS was the first entity involved in the September 8 event that attempted to resolve the potential violations without the need for litigation.
10. As part of the investigation, Enforcement and NERC reviewed APS’s compliance program and found that APS satisfies the criteria for an effective compliance program under the Commission’s Penalty Guidelines. Enforcement and NERC considered the elements of APS’s compliance program set forth in this paragraph. APS’s compliance program is supported by a dedicated staff devoted to evaluating and responding to compliance issues. APS has an executive-level committee responsible for compliance, which meets no less frequently than quarterly, and receives updates no less frequently than bimonthly. The Chief Compliance Officer and Director of Regulatory Compliance both have independent access to APS’s CEO and President, as well as to APS’s and its parent Pinnacle West Capital Corporation’s Boards of Directors. APS’s Regulatory Compliance Director is supported by seven full-time employees. APS requires all lines of business to review their policies, standards, procedures and guidelines and update them whenever NERC or WECC issues new or revised Reliability Standards. New hires are
trained on regulatory compliance, and subject matter experts (SMEs) are provided with periodic regulatory updates as well as annual training on the Reliability Standards. Each line of business has SMEs responsible for managing day-to-day compliance with the Reliability Standards for their business, and these SMEs must certify compliance with the Reliability Standards annually. APS has an independent hotline staffed 24/7 for employees to anonymously report potential noncompliance. Finally, APS has a written Regulatory Compliance Program that addresses risk assessment, controls, monitoring and reporting, and evidence of compliance for laws and regulations including the Reliability Standards.
IV. VIOLATIONS
11. Enforcement and NERC determined that APS violated four Requirements of two Reliability Standards: TOP-002-2a R4, TOP-002-2a R6, TOP-002-2a R11, and TOP-008-1 R2. Enforcement and NERC found that APS violated: 1) TOP-002-2a R4 by failing to perform a unique next-day study for September 8 (or to properly assess whether a unique next-day study was required based on current system conditions) and failing to coordinate its day-ahead studies with neighboring TOPs, including IID and CAISO; 2) TOP-002-2a R6 by failing to adequately plan for unscheduled changes in system configuration, specifically following the loss of the H-NG; 3) TOP-002-2a R11 by failing to perform a unique next-day study or current-day study for September 8 (or to properly assess whether a unique next-day or current-day study was required based on current system conditions) to correctly determine SOLs; and 4) TOP-008-1 R2 by not operating in a manner to prevent the likelihood that a disturbance would result in IROL or SOL violations.
V. REMEDIES AND SANCTIONS
12. APS stipulates to the facts as described in Section II of this Agreement, but neither admits nor denies Enforcement and XXXX’s findings that its conduct violated the Reliability Standards specified in Section IV. For purposes of settling any and all civil and administrative disputes within the jurisdiction of the Commission arising from the reliability issues related to the September 8 event and Enforcement’s and NERC’s investigation, APS agrees to the remedies set forth in the following paragraphs.
A. Civil Penalty
13. APS agrees to a civil penalty in the amount of $3,250,000 of which $2,000,000 shall be paid, divided in equal amounts, to the United States Treasury and NERC, within 10 days of the Effective Date. Enforcement and NERC agree to give APS a partial civil penalty offset for the additional $1,250,000 in exchange for APS completing Reliability Enhancements as set forth in Section V.B. The value of the Reliability Enhancements is expected to substantially exceed the amount of the offset.
B. Reliability Enhancements
14. In exchange for the $1,250,000 offset, APS shall complete the following Reliability Enhancements:
a. Install Phasor Measurement Units at Moenkopi, Yavapai, Xxxxxx, Pinnacle Peak, and Cholla (both 345 and 500 kV);
b. Install and/or pay its share of the capacitor bank for Western-DSW’s Kofa substation; and
c. Implement Phase I and Phase II of its ECC Visualization Project, which will provide system operators with a visual geo-spatial representation of RTCA results.
APS shall provide Enforcement and NERC with satisfactory evidence, as determined by Enforcement and NERC, of the completion of the Reliability Enhancements and its investment of amounts exceeding the offset in the Reliability Enhancements. If APS has not spent a minimum of $1,250,000 on the Reliability Enhancements described in 14.a. through 14.c. above by June 30, 2015, it shall pay the remainder of the $1,250,000 in equal shares to the United States Treasury and NERC.
C. Completed and Required Mitigation
15. APS commits to the following actions, designed to mitigate Reliability Standard violations and to improve overall reliability of the BES.(4) As indicated below, APS affirms that it has already completed some of the mitigation measures and shall complete all the mitigation items no later than one year after the Effective Date of the Agreement, unless otherwise stated in Section V.C. In those instances where APS has already implemented mitigation measures prior to entering into the Agreement, it shall continue operating under the practices and procedures implemented as part of the mitigation, until such time as it implements superior or improved practices and procedures, as determined by Enforcement and NERC staff. Enforcement and NERC recognize that successful completion of some of the mitigation actions is subject to cooperation from and coordination with third parties. APS will report on the status of all mitigation items described in Section V.C. and submit evidence of status and progress in its compliance
(4) Bulk Electric System (using the most recent definition approved by NERC and the Commission). The definition slated to take effect July 1, 2014, generally includes all Transmission Elements operated at 100 kV or higher and Real Power and Reactive Power resources connected at 100 kV or higher, subject to certain other inclusions and exclusions.
monitoring reports to be submitted to Enforcement and NERC staff pursuant to Section V.D of this Agreement.
i. Mitigation Related to Seasonal, Next-day, and Current-day Planning
16. Since the September 8 event, APS now performs unique next-day studies for each day and shares its unique next-day studies with neighboring entities and the RC. APS also participates in daily calls with the RC and neighboring entities to discuss potential reliability issues for the coming day.
17. APS shall improve its seasonal, next-day, and current-day studies, plans, and associated procedures in order to ensure additional coordination with other entities and reliable operation of its portion of the BES by identifying external elements necessary for proper planning and requesting and utilizing, where available, planning information from other entities for these elements in a usable format and in a timeframe that allows APS to properly complete its planning studies. For the duration of its compliance monitoring under the agreement, APS shall report to both Enforcement and NERC staff any difficulties in obtaining this required information from other entities, and in all cases shall report any requested information that either cannot be or has not been supplied by a neighbor in an adequate format and within APS’s specified time deadlines. APS shall submit these reports to Enforcement and NERC staff within three months of the initial request for information by APS. Likewise, APS shall timely provide all requested information to other entities for their planning purposes.
ii. Mitigation Related to SOLs and IROLs
18. Following the September 8 event, APS participated in the creation of a SWPL nomogram in order to establish reliable operating limits of the SWPL, which includes H-NG. This nomogram was used until the Kofa capacitor banks were installed and then was retired by WECC. Additionally, APS fully implemented a State Estimator and RTCA tool.
19. APS shall review and improve its SOL and IROL procedures to ensure the resulting limits are valid for the planned and actual conditions and configurations of the BES in the Western Interconnection, including external facilities, sub-100 kV facilities, and generation levels that impact BES reliability. APS shall develop and implement operator procedures that use the results of the RTCA tool and the ASMM, as appropriate, to identify, prepare for, and respond to contingencies.
iii. Mitigation Related to Long-term Planning
20. Within six months of the Effective Date of the Agreement APS shall improve its
long-term planning process to ensure that its long-term planning studies provide for an analysis that fully integrates all available information on neighboring facilities, protection systems, and operating criteria that have a material impact on its transmission planning study results or on which its transmission operations have a material impact. APS agrees to review the WECC regional model as a whole to determine the interrelationship of its facilities with those in neighboring systems to understand mutual system and operations impacts, including but not limited to loop flow, and model them accordingly. APS shall review its planning criteria to include a sufficient range of off-peak conditions to address all critical system concerns.
iv. Mitigation Related to Emergency Operation Plans
21. APS shall review and, as appropriate, revise its procedures for restoring critical lines and facilities within required time limits when large phase angle differences on BES facilities could impede restoration operations. These plans must address any coordination required by other entities. APS trains its personnel on dealing with such phase angle differences and also participates in the annual Desert Southwest Training Advisory Committee training/simulation drill with the RE and neighboring utilities. APS shall provide its procedures for review to Enforcement and NERC staff and shall inform Enforcement and NERC staff when training on these procedures will be provided in the future to allow both Enforcement and NERC staff the opportunity to observe the training.
v. Mitigation Related to Protection Systems
22. Following the September 8 event, APS reviewed and verified the relay loadability settings on its protection systems involved in restoration activities, and as appropriate, revised those settings to ensure that those settings would not unnecessarily restrict transmission loadability during restoration activities. This review was not limited to those facilities involved in the event. APS shall report to Enforcement and NERC staff all protection systems reviewed in this activity and any changes made to those protection systems as a result.
vi. Mitigation Relating to Voltage Control
23. APS shall review and update its voltage support procedures to prevent voltage collapse under all N-1 contingencies. APS shall report to Enforcement and NERC staff how it determined the facilities studied and the N-1 contingencies included in its review and update of its voltage support procedures.
vii. Mitigation Related to Angular Separation
24. APS shall implement tools to determine when planned or actual operations will
result in large phase angle differences on BES facilities that will impede normal, emergency, or restoration operations, including the delayed reclosing of transmission lines. APS trains all system operators to reliably identify and address large phase angle differences, including any coordination with neighbors or the RC, and also participates in the annual Desert Southwest Training Advisory Committee training/simulation drill with the RE and neighboring utilities. APS shall provide a copy of its procedures and training material to Enforcement and NERC staff and inform Enforcement and NERC staff when training on these plans will be provided in the future to allow the opportunity for both Enforcement and NERC staff to observe the training.
viii. Mitigation Related to Modeling
25. APS shall review its models across all operational timeframes for accuracy and shall include in its models all elements, both internal and external, that affect BES reliability (subject to the cooperation of other entities that must provide the necessary information). Such models must also include the effects of planned control devices, including protective relays and special protection systems, on system reliability.
ix. Mitigation Related to Situational Awareness
26. Following the September 8 event, APS has accelerated the development and implementation of its State Estimator and RTCA tools to improve its planning and situational awareness in the current-day and real-time operational horizons. Both of these tools are now operational as of the date of this Agreement. APS shall maintain these tools in full operating condition and fully integrate them into its operations in current-day and real-time operations horizons. APS shall provide in its first compliance report to Enforcement and NERC staff copies of its operator manuals and training plan regarding these tools.
D. Compliance Monitoring
27. APS shall make semi-annual reports to Enforcement and NERC staff for one year following the Effective Date of this Agreement. The first semi-annual report shall cover the first six-month period after the Effective Date of this Agreement and shall be submitted to Enforcement and NERC staff within thirty days later. The subsequent report(s) shall be due in six month increments thereafter. Each report shall detail the following: (1) actions taken as of the date of the report to satisfy the terms of this Agreement, including all mitigation items; (2) actions taken to improve reliability compliance, including investments in new measures and training activities during the reporting period; and (3) any additional possible violations of Reliability Standards that have occurred and whether and how APS has addressed those new violations. The reports must include an affidavit executed by an officer of APS that the compliance reports are true and accurate and also include corroborative documentation or other
satisfactory evidence demonstrating or otherwise supporting the content of these reports. Enforcement and NERC staff may require additional semi-annual reporting if circumstances indicate the need for further monitoring or if APS has not yet completed all the mitigation measures discussed in this Section.
VI. TERMS
28. The “Effective Date” of this Agreement shall be the date on which the Commission issues an order approving this Agreement without material modification. When effective, this Agreement shall resolve all reliability matters relating to the September 8 event within the jurisdiction of the Commission, and that arose on or before the Effective Date, as to APS or any affiliated entity.
29. Commission approval of this Agreement without material modification shall release APS and forever bar the Commission and NERC from holding APS, any affiliated entity, and any successor in interest to APS liable for any and all administrative or civil claims arising out of the reliability issues related to the September 8 event or the conduct addressed and stipulated to in this Agreement that occurred on or before the Agreement’s Effective Date.
30. Failure to make the civil penalty payment, comply with the mitigation, Reliability Enhancements, and monitoring agreed to herein, or any other provision of this Agreement, shall be deemed a violation of a final order of the Commission issued pursuant to the Federal Power Act (FPA), 16 U.S.C. §792, et seq., and may subject APS to additional action under the enforcement provisions of the FPA.
31. If APS does not make the civil penalty payment described above at the time agreed by the parties, interest payable to the United States Treasury and NERC shall begin to accrue pursuant to the Commission’s regulations at 18 C.F.R. § 35.19(a)(2)(iii) (2013) from the date that payment is due, in addition to the penalty specified above and any other enforcement action and penalty that the Commission or NERC may take or impose.
32. The Agreement binds APS and its agents, successors, and assignees. The Agreement does not create any additional or independent obligations on APS, or any affiliated entity, its agents, officers, directors, or employees, other than the obligations identified in this Agreement.
33. The signatories to this Agreement agree that they enter into the Agreement voluntarily and that, other than the recitations set forth herein, no tender, offer or promise of any kind by any member, employee, officer, director, agent or representative of Enforcement, NERC, or APS has been made to induce the signatories or any other party to enter into the Agreement.
34. Unless the Commission issues an order approving the Agreement in its entirety and without material modification, the Agreement shall be null and void and of no effect whatsoever, and Enforcement, NERC, and APS shall not be bound by any provision or term of the Agreement, unless otherwise agreed to in writing by Enforcement, NERC, and APS.
35. APS agrees that the Commission’s order approving the Agreement without material modification shall be a final and unappealable order assessing a civil penalty under section 316A of the FPA, 16 U.S.C. § 825o-1. APS waives findings of fact and conclusions of law, rehearing of any Commission order approving the Agreement without material modification, and judicial review by any court of any Commission order approving the Agreement without material modification.
36. The Agreement can be modified only if in writing and signed by Enforcement, NERC, and APS, and any modifications will not be effective unless approved by the Commission.
37. Each of the undersigned warrants that each is an authorized representative of the entity designated, is authorized to bind such entity and accepts the Agreement on the entity’s behalf.
38. The undersigned representative of APS affirms that he has read the Agreement, that all of the matters set forth in the Agreement are true and correct to the best of his knowledge, information and belief, and that he understands that the Agreement is entered into by Enforcement and NERC in express reliance on those representations.
39. The Agreement may be signed in counterparts.
40. The Agreement is executed in triplicate, each of which so executed shall be deemed to be an original.
|
Agreed to and accepted: |
| |
|
|
| |
|
|
| |
|
/s/ Xxxxxx X. Xxx |
| |
|
Xxxxxx X. Xxx |
| |
|
Director, Office of Enforcement |
| |
|
Federal Energy Regulatory Commission |
| |
|
|
| |
|
Date: |
June 19, 2014 |
|
|
/s/ Xxxxxxx Xxxxxxxxxx |
| |
|
Xxxxxxx Xxxxxxxxxx |
| |
|
Senior Vice President, General Counsel and Corporate Secretary |
| |
|
North American Electric Reliability Corporation |
| |
|
|
| |
|
Date: |
June 19, 2014 |
|
|
|
| |
|
|
| |
|
/s/ Xxxxxx X. Xxxxxxxxxx |
| |
|
Xxxxxx X. Xxxxxxxxxx |
| |
|
Senior Vice President |
| |
|
Transmission, Distribution & Customers |
| |
|
| ||
|
|
| |
|
Date: |
June 16, 2014 |
|
