Capital Chatter: What caused state data network failure?
Published 7:00 am Thursday, June 14, 2018
- Capital Chatter: Dangerous times for the Oregon Legislature
Oregon drivers couldn’t get the latest travel conditions from the state. Oregon State Police couldn’t run background checks on prospective gun purchasers. Truckers couldn’t file their mileage reports. Anglers couldn’t get fishing updates. People calling many state offices were out of luck.
All that, and more, occurred last weekend when the state government’s information super-highway came to a halt.
Trending
It all started when Portland General Electric turned off power early Saturday for work at the Oregon State Capitol.
No big deal. It was a planned power outage. The state has battery systems and backup generators for its key computer hardware, along with a series of redundant routers. If one part fails, another will take over.
Those redundancies worked fine when they last were tested a couple of years ago. For whatever reason, this time a card in one router failed, and the backup card failed to take over. So did the other routers. That caused the state data center’s network to crash. Dozens of state agencies lost their phones, websites and other systems.
“This kind of outage has never happened before where it’s a core network,” said Sandy Wheeler, administrator of the state data center, officially known as Enterprise Technology Services.
The data center is an around-the-clock operation, and additional staff responded within 15 minutes of the incident Saturday morning. It took much of the weekend to determine which part failed, to restore the core network and to have individual agencies reboot their systems. For some, such as Oregon Trucking Online, difficulties continued into this week.
These main routers are nearing the end of their functional life, which is about four years, and their replacements were scheduled for installation. The vendor, Cisco, maintains the routers but the state could have a big overtime bill for the employees who responded last weekend throughout state government.
Trending
The data center will do a post-incident review of what happened, but Wheeler told me that one early lesson is the need for more-frequent testing to ensure the backup hardware takes over when it is supposed to.
Officials said state Police dispatch and patrols are on a separate network, so they were not affected.
• Life before email: Out of curiosity, I checked Cisco System’s website and found this history. The company was founded in 1984: ” Len Bosack and wife Sandy Lerner, both working for Stanford University, wanted to email each other from their respective offices, but technological shortcomings did not allow such communication. A technology had to be invented to deal with disparate local area protocols, and as a result of solving their challenge, the multiprotocol router was born.”
My brother, a top administrator at Stanford, was among the early users of email. I remember his telling me about this new technology called electronic mail, which enabled his boss to communicate with him at work, at home and during evenings and holidays.
O, what email hath wrought …
• Give the state a break: Computer crashes occur in every segment of our lives, not just government. For example, glitches at a well-known national internet provider caused Salem-area phones to temporarily fail at my credit union and other businesses last week.
• Avoiding a Jory Cove vote: Vicki Walker, a former Democratic legislator from Eugene who now runs the Oregon Department of State Lands, told State Land Board members this week that decisions on the controversial Jordan Cove LNG project are hers to make, not theirs.
Her reasoning seemed plausible. But it also means Gov. K ate Brown won’t have to vote on the project before the November election. Democrat Brown has avoided taking a stand on the proposed liquefied natural gas export terminal in Coos Bay, as well as the pipeline that would cross four Oregon counties. Many local officials and legislators support the project.
Besides Brown, the Land Board members are Republican Secretary of State Dennis Richardso n and Democratic State Treasurer Tobias Read.
• More, or less, money for schools: Walker is “quite frustrated” with the budget proposed by the State Department of Forestry for managing 33,073 acres of forest lands in 18 counties on behalf of the Department of State Lands. Proceeds from forest logging support the Common School Fund.
Walker, who noted that the Land Board asked her to be a change agent when she was appointed department director, wants to see whether counties or independent logging companies are interested in managing those lands.
The Forestry Department predicted its management would net $2 million for the Common School Fund.
In another move, the Land Board voted to distribute 3.5 percent of the Common School Fund to schools, down from 4 percent, the mark set in 2009. State and private financial experts recommended the decrease “to maintain a relatively stable real asset value.” The Common School Fund was valued at $1.676 billion at the end of last year.
• Water, algae, water: The Oregon Health Authority canceled its recreational advisory from cyanotoxins in Detroit Lake, then reinstituted the advisory this week. It also added one for Dorena Reservoir.
OHA Director Pat Allen told me he expects algae blooms and cyanotoxins to be a recurring concern. The health agency will require public water systems to test for cyanotoxins, which Salem and towns already are doing.
Meanwhile, Salem kept its water advisory going for two weeks while engineers spread activated carbon on the city’s water filtration areas to absorb the cyanotoxins. After a rough beginning to the city’s water crisis, the city has been communicating much better with the public.
• This calf didn’t bite the dust: The Oregon Department of Fish and Wildlife reports a wolf attacked a 4-month old, 350-pound calf this month on private rangeland in the Reagin Gulch area of Wallowa County.
A veterinarian examined the calf. The ODFW report said: “The live calf was examined and the hair was shaved around the injuries. Bite marks were found on the inside, rear, and outside of both hind legs above the hocks. These bite marks were associated with muscle tissue trauma, and are clear evidence of a predator attack. The size, number, and locations of the bite injuries are similar to injuries observed in calves attacked by wolves. Although individual wolves have been documented, there is no known pack currently using this area. ODFW will continue to monitor wolf activity in the area.”
• Democrats expand: More than 250 Democrats gathered in Pendleton last weekend for the quarterly Democratic State Central Committee meeting. They approved two new groups in the party — a Native American Caucus and a Young Democrats of Oregon Caucus
They also showed support for Jamie McLeod-Skinner, who is running against Rep. Greg Walden, “a longtime Republican congressman who has cozied up to the Trump administration and consistently voted against his district’s interests.” (That’s the Democrats’ description, not mine.)
McLeod-Skinner will have a tough time winning the only Oregon congressional seat held by a Republican. As of April, the 2nd Congressional District had 543,561 registered voters, including 186,969 Republicans, 174,176 voters with no party affiliation, 142,378 Democrats, 28,363 members of the Independent Party and several thousand members of minor parties.
• Housing task force: Oregon Senate President Peter Courtney and House Speaker Tina Kotek have appointed the Task Force on Addressing Racial Disparities in Home Ownership, created by the Legislature’s passage this year of House Bill 4010. The task force is one way the Legislature is responding to decreasing home ownership in the state.
Members are Democratic Rep. Mark Meek, Republican Rep. Richard Vial, Republican Sen. Cliff Bentz, Democratic Sen. James Manning, Debra Neal, Frederick Edwards, Joe Enlet, Julie Nash, Maria Elena Guerra, Maxine Fitzpatrick and Se-Ah-Dom Edmo.
Dick Hughes, who writes the weekly Capital Chatter column, has been covering the Oregon political scene since 1976. Contact him at TheHughesisms@Gmail.com, Hughesisms.com/Facebook, YouTube.com/c/DickHughes or @DickHughes on Twitter.