The $3 million in damage to the M/V Walla Walla happened in a little more than 30 minutes as the ferry's engine crew and Eagle Harbor electricians were working on a main propulsion drive motor on one end of the vessel while tapping power from an unoccupied engine room on the other end of ship.
The problem, according to a just-released investigation of the mishap at Washington State Ferries shipyard on Bainbridge, was that pieces of the other engine that was being used for power had been removed and it had been "locked and tagged out" in anticipation of upcoming maintenance.
With the drive motor locked out and unable to rotate, the multi-million dollar meltdown quickly followed.
According to the investigation into the Walla Walla incident, if a crew had been present in the No. 1 motor room, the problem would have been "detected immediately and the damage to the propulsion drive motor may have been limited or minimized."
Investigators said human error was to blame, and noted that the engine crew members of the ferry were unfamiliar with the specific requirements that were needed for the maintenance work.
"Work crews were overly reliant on inadequate historical institutional knowledge concerning this infrequent complex maintenance work," the investigation report states. "Additionally, there were no written procedures or checklists available for this infrequent complex maintenance work."
The report, released March 11, also includes a comprehensive timeline of the days leading up to the meltdown and the day of the incident.
Though shipyard workers had previously blamed earlier sandblasting and paint work that was down aboard the ship as the cause of the meltdown, the investigators found that work did not cause the mishap.
Ferry officials said Monday that WSF was taking steps to prevent similar incidents in the future.
The timeline of events, as presented by the WSF Investigation Panel into the Walla Walla incident
July 20, 2012
Routine visual inspections by the vessel crew noted discolored commutator surfaces on all four drive motors. The staff chief engineer created a work requisition to have an outside vendor perform a drive motor condition survey. The work requisition was reviewed and approved by the vessel’s port engineer.
July 23, 2012
Walla Walla enters the scheduled commercial shipyard at Vigor Everett for dockside work, with a planned completion date of Oct. 12, 2012. The following major work items were planned to be executed:
- Vehicle deck plate repairs and steel renewal
- Gallery deck steel renewal
- Miscellaneous structural steel renewal
- Preparation and painting of car decks, pickle fork decks, stairways
- Watertight door control upgrade and maintenance
- Bilge and other piping systems renewal
- HVAC control upgrade
- Pilothouse 24 VDC upgrade
Aug. 20, 2012
A WSF purchase order is issued to Delta Electric Motors, Inc. for the vessel work request that was generated on July 20, 2012 to conduct the drive motor condition survey.
Aug. 29, 2012
The No. 2 end motor room and drive motors are dusted with sandblast material. While the lower car deck was being sandblasted by the shipyard in preparations for painting, the containment for the No. 2 end drive motor ventilation ducting was damaged by the sandblasting operation. This containment damage allowed sandblast grit and dust to enter the No. 2 end motor room and drive motors. Vessel crews and WSF inspectors found that the No. 2 end motor room and drive motors were contaminated with sandblasting dust. No sandblasting dust was observed in the No. 1 motor room or drive motors No. 1A and No. 1B.
Sept. 29, 2012
Vigor Everett Shipyard hires Delta Electric Motors, Inc. as a sub-contractor to clean the No. 2 end motor room and drive motors. This work was separate and independent of the drive motor condition survey previously arranged by WSF prior to the vessel entering the shipyard in Everett.
Oct. 5, 2012
As per the work requisition issued by the vessel’s port engineer, Delta Electric Motors, Inc. completed the drive motor condition survey. This survey was completed on all four propulsion drive motors. This survey was contracted to examine the drive motor condition and commutator surfaces. Further information can be found in the body of the investigation report. This condition survey report did not find any evidence of sandblasting grit or paint contamination.
Oct. 12, 2012
The Walla Walla’s original contract completion date was extended and the departure from the shipyard was delayed. This delay was granted due in part to increased structural steel replacement during surface preparations for topside painting. The commercial shipyard time was subsequently extended to Oct. 29, 2012.
Oct. 23, 2012
Delta Electric Motors, Inc. submits the drive motor condition survey report to WSF based on the drive motor condition survey conducted on Oct. 5, 2012.
Vendor report notes discoloration on all propulsion drive motor commutator surfaces due to atmospheric conditions and spring pressure of the brush assemblies.
Continued discussion between the vendor, vessel port engineer, the Walla Walla staff chief engineer and Eagle Harbor electricians took place to determine the next course of action. The possible steps that were being reviewed were polishing, stoning, changing brushes or a combination of the three.
Oct. 29, 2012
The redelivery of the Walla Walla from the shipyard for the second delivery date was not made. The redelivery of the vessel was delayed to a “day-by-day” status. This delay was accepted by WSF.
Oct. 31, 2012
The Walla Walla shipyard work was substantially complete and the vessel was redelivered back to WSF by Vigor Everett. Walla Walla was delivered via towboat to WSF Eagle Harbor Maintenance Facility at approximately 10 a.m.
Nov. 1, 2012
The Walla Walla layup meeting commenced between Eagle Harbor personnel, vessel engine crew and the vessel port engineer. In response and follow up to the Delta Electric Motors, Inc. survey report, the decision to stone the propulsion drive motor commutators was made.
It was decided that all four drive motor commutators would be stoned, per the recommendations of the survey report findings and follow up discussions with the vendor.
The Walla Walla delay had impacted the remaining fleet’s out-of-service needs; the need to return the vessel to operating status was required. Due to these time constraints it was decided that extra electricians from Eagle Harbor would be assigned for accomplishment of this task.
Nov. 2, 2012
Eagle Harbor electricians performed the drive motor pre-work insulation resistance testing of motor windings (this procedure is called “meggering,” it measures the resistance of the motor to ground).
The megger readings showed unacceptably low readings. Because low megger readings can indicate insulation breakdown and possible grounds, it was decided to postpone the work for 24 hours so that heaters could be applied to the drive motors in order to decrease humidity levels and, in turn, improve the condition of the drive motors. All but one brush set was removed by Eagle Harbor electricians from the No. 1A and No. 2B drive motors. This was done in preparation for that weekend’s stoning. Note that there are six sets (12 rows) of brushes on each drive motor.
Nov. 3, 2012
An Eagle Harbor electrician returned to the Walla Walla to megger the drive motors for a second time. The megger readings improved to an acceptable limit. The stoning procedure commenced the following day, Sunday.
Nov. 4, 2012
One half week of additional maintenance out of service time had been added to the Walla Walla’s schedule when the drive motor maintenance increased the scope of work.
WSF staffed up accordingly with extra crews, work on the weekend, and allotted overtime to absorb this increased scope. The staffing was as follows:
In addition to the Eagle Harbor electricians (five), vessel engine crew including the alternate staff chief engineer (four) dispatched for that day, an extra Eagle Harbor electrician crew (three) as well as the vessel’s master and staff chief engineer who was called out on overtime were dispatched to complete the work on the Walla Walla.
A total workforce of 14 were present.
The remaining brushes were removed from the No. 2B drive motor by Eagle Harbor electricians, leaving the motor with no brushes.
The following timeline is compiled from statements of those involved with the incident and represents the investigation panel’s best estimate of the sequence of events.
Approximately 10:15: The A Loop propulsion circuit was energized for stoning preparations by using the #3 main propulsion generator. Because the stoning operation required the shafts to turn, the master was present in the inshore pilothouse to ensure the vessel’s safety while pushing the dock.
Approximately 10:49: The stoning commenced on the No. 2B drive motor. The shaft speed was set to 40 RPM, which held for a period of time before it decreased to 30 RPM. This decrease in RPM was noted by the vessel crew members. The speed was then increased back up to 40 RPM by adjusting the control handwheel in engineers operating station.
Approximately 10:57: No. 1A propulsion motor exhaust air temperature high alarm sounded in the EOS. An oiler was directed to investigate the No. 1 end motor room. Upon arrival at the No. 1 motor room, the oiler discovered smoke and arcing.
The general alarm was sounded.
Vessel crew suited up in firefighting gear but did not attempt to enter the No. 1 motor room. The staff chief engineer decided to wait to evaluate the situation until the space was safe to enter.
Approximately 10:59: The propulsion loop was secured; it was determined that there was no fire in the No. 1 motor room. No firefighting efforts were required. The No. 1 motor room was ventilated.
After approximately 45 minutes the No. 1 motor room was cooled and cleared of smoke.
Initial inspections revealed that the No. 1A drive motor had sustained substantial damage to the commutator surface.
The commutator surface and brushes of No. 1A drive motor had melted due to intense heat created by the arcing.
U.S.Coast Guard, Sector Puget Sound was notified and a "Report of Marine Accident, Injury, or Death" form was filed.
WSF chain of command was notified and a WSF investigation was ordered and commenced.
Nov. 5, 2012
Walla Walla crew and Eagle Harbor electrician statements were taken and interviews were conducted.
General Electric, the original manufacturer of the drive motors, was contacted for a damage assessment.
Two Labor and Industries investigators arrived at Eagle Harbor to investigate the Walla Walla drive motor damage.
Nov. 6, 2012
WSF decided to hire a third party DC motor expert to perform an independent technical investigation of the No. 1A propulsion drive motor damage. Cadick Corporation was chosen as the third party vendor to conduct the technical investigation.
Bob Steele, GE senior field engineer, and Steve Doman, GE sales manager, were also contracted to assess the condition of the No. 1A propulsion drive motor to facilitate a repair and replacement plan for the damaged drive motor.
Nov. 8, 2012
USCG Sector Puget Sound Marine Inspectors visited the vessel to assess the damage.
Nov. 13, 2012
WSF investigation was upgraded to the formal investigation status. Third-party investigator, Michael Turner from Cadick Corporation, commenced his independent investigation.