This is the fourth part of my review of the Final Report of the Commission of Inquiry into the Ottawa LRT fiasco. This article covers the construction of the line up to Substantial Completion and draws from chapters 10-12 of the Inquiry Report.

See also:

• The Ottawa LRT Report Part I
• The Ottawa LRT Report Part II
• The Ottawa LRT Report Part III

Project Delays

Proponents of P3 deals stress the value of “risk transfer” where the cost of unknown problems is absorbed by the contractor. This is advantageous to the owner, the City, but for the contractor it is effectively a bet that things will not go wrong. In this case, they lost the bet, big time. The question then becomes resolving whether the problem was caused by the contractor or by the City, and whether the risk should fall totally to the contractor.

This type of problem also beset the Line 5 Eglinton contract in Toronto where Metrolinx attempted to put responsibility for delays and extra costs entirely on the P3, Crosslinx. Metrolinx lost in court, twice, although they are appealing.

In Ottawa, the biggest issue was a sinkhole on Rideau Street.

“The most significant delay for the project was the Rideau Street sinkhole in 2016, which profoundly disrupted the construction timeline and caused an immediate delay in OLRT-C’s sequencing of the work. While there were other delays at the same time – such as Alstom’s delivery of the vehicles and OLRT-C’s systems engineering and assurance failures – the sinkhole disrupted OLRT-C’s progress at a critical stage of construction. This delay had knock-on effects throughout the project, most significantly in OLRT-C’s ability to deliver the necessary track and other infrastructure to test the vehicles and the train control system. This resulted in a shortened testing schedule and a resequencing of this work to use the available infrastructure.” p12

The Inquiry did not assess blame, a matter for the parties and their insurance companies, but rather looked at how the dispute over responsibility hampered the project.

“The Project Agreement transferred the geotechnical risk – the risk of ground-related problems and conditions – to RTG. This risk transfer was subject to certain limited exceptions described in the Project Agreement. As a result of this risk transfer, RTG was generally responsible for the consequences of any geotechnical issues occurring, including any associated costs and delays.

“[…] While they recognized tunnelling as an area of risk, RTG and OLRT-C were confident in the technology they had and believed it gave them an edge over competitors. The evidence heard in this Inquiry was that constructors do not anticipate or plan for an event like a sinkhole of this magnitude. At least RTG, in its optimism, did not.” [p 247]

In brief, RTG’s position was that a fire hydrant relocation was done improperly by the City causing water to leak and liquefy soil nearby. Eventually, underground ductbanks collapsed into the hole, followed by the road which the ductbanks had supported even as the void underneath formed. RTG’s claim was for $230 million and an extension of 281 days to the Revenue Service Availability date.

(There have been cases in Toronto where sinkholes have formed from underground water leaks or streams, and the only thing holding up the road before complete collapse was the streetcar track.)

The City’s position is completely different. They argue that the tunneling process caused a subsurface collapse in the clay below Rideau Street creating the sinkhole and rupturing the water pipe. This possibility should have been foreseen and protected for by RTG, and hence the sinkhole was their responsibility.

The dispute went to the Independent Certifier for adjudication, and they ruled in the City’s favour. RTG challenged this decision in court, but eventually the dispute was settled by mediation in which “the City and RTG released each other from all claims regarding the Rideau Street sinkhole”. [p 249] RTG’s insurer paid a portion of the costs.

In the aftermath of this dispute, relations between RTG and the City deteriorated.

“Whether the sinkhole was or was not one of the dominant causes of RTG’s failure to achieve RSA on time, it led to significant scheduling complications and a disorderly sequencing of construction and testing.” [p 249]

The sinkhole delay threw the rest of the project schedule out of kilter. On the construction side, the station could not be finished, but more importantly, the track, power and signalling could not be installed for vehicle and systems testing. This forced the testing onto a much smaller test track, and full systems testing was not possible.

“Thales and Alstom required access to the full length of the track to complete validation and integration testing. However, at least in part due to the sinkhole, the full track was not available until September 2018. Jacques Bergeron, of OLRT-C, gave evidence that the main reason the RSA date of May 24, 2018 was missed was because the testing and commissioning of the LRVs and train control system were delayed because the tracks were unavailable. Bergeron attributed part of this delay to the sinkhole, because the tunnel could not be accessed.”[p 250]

The added cost to OLRT-C was between $400 and $500 million, hardly a trivial sum on a contract of $2.1 billion. This provided an incentive to get the line into operation as soon as possible to collect the completion payments, to avoid a late completion penalty of about $125,000 per day, and to begin receiving revenue for operations and maintenance.

“Scheduling became an exercise in trial and error. As a result, OLRT-C repeatedly delivered revised schedules that would then be revised again.

“The common pattern in these revised schedules was that the time available for testing and commissioning – particularly the testing and commissioning that needed to happen at the end of the construction and manufacturing phase – was compressed in order to make up for the delays caused by the sinkhole.” [p 253]

Updated Dec 25 at 9:40 pm: In the following paragraph the word “not” has been inserted so that it now reads “it could not be done”.

Testing and commissioning vehicles and the control system is a complex business, and it could not be done without access to a substantial portion of the system. Moreover, a “burn in” period would be needed to get past early failures and identify needed fixes before revenue service began.

“[…] everyone involved with the OLRT1 project knew – or should have known – that there would be significant reliability issues in the early stages of the OLRT1 system, and that a robust testing and commissioning process, together with meaningful pre-trial running and trial running, were necessary to ensure that these reliability issues were minimized during the public operation of the OLRT1 system.

“What happened instead was that OLRT-C continually compressed much of the testing and commissioning, and in particular the general running period for the vehicles, to address the delays and mounting pressure – financial and otherwise – to complete the OLRT1 project and achieve RSA by the Required RSA Date or as soon as possible after that.” [p 255]

The relationship between the City and RTG deteriorated especially when discussions shifted from the technical staff level to the management and politicians.

“In the first few months after the Rideau Street sinkhole appeared, the City had a co-operative relationship with RTG. But once it became clear that the sinkhole would cause a major delay, the City’s mood changed; its approach to the relationship hardened and the relationship became more strictly contractual. It appears that issues began to be escalated to higher-up officials in the City hierarchy rather than being dealt with co-operatively by technical staff. Although it became nearly impossible for OLRT-C to catch up, according to the evidence, the City “didn’t want to hear about delays,” as RTG’s then-CEO Antonio Estrada put it.” [p 256]

Everything hung on the May 24, 2018 target date for revenue service even though OLRT-C knew by mid-2017 that it was “nearly impossible to meet its RSA date”, but delayed telling the City this. After many changes to the RSA date and specifications including vehicle delivery and availability, the eventual opening on a reduced spec was set for Sept. 14, 2019.

“While delays are understandable on any project, what is inexplicable was RTG’s and OLRT-C’s insistence on providing RSA dates to the City that they had no realistic hope of achieving.” p13

Relationship Between RTG and the City

Through the project, the “partnership” between RTG and the City of Ottawa deteriorated and became antagonistic. Thus, one of the three “P”s disintegrated as the parties attempted to defend their position and claims. A second “P” vanished when the City took over the role of lender to finance the project. Thus a scheme that was designed to shift risk and financial responsibility away from the City to a private sector partner failed to achieve this goal.

“[…] the City took aggressive positions in asserting contractual claims. The City evidently believed financial pressure was the best method to achieve its desired result.

“Another example of this approach is what has been described as the “debt swap,” a financial transaction through which the City stepped into the shoes of RTG’s long-term lenders. The debt swap came about because of the failure of the Project Agreement to effectively provide for the next stage of the LRT project. There were legitimate financial reasons for the City to enter this transaction. However, it is also apparent that the City saw the debt swap as another way to exert financial pressure on RTG.

“The bottom line is that the relationship between the City and RTG was adversarial at critical stages of the construction and maintenance of OLRT1, and this fact contributed to problems with the OLRT1 project.” [p 15]

As the project wore on, the City, with good reason, stopped trusting claims made by RTG about the date when the revenue service would be possible. A year before the target RSA [Revenue Service Availability] date of May 24, 2018, the City knew this could not be met. Slippage continued, and there was no sign of how OLRT-C could get back on time. Correspondence between the various companies involved showed that the May 2018 date could not be met and time extensions had been granted, notably to Thales. However, RTG continued to tell the City that the May 2018 date would be achieved right up to February 2018.

For its part, the City issued a Failure to Maintain Schedule Notice in August 2017 in an attempt to force production of a schedule to achieve the May 2018 date even though it was clear this was impossible. The City was attempting to enforce its contractual position, while RTG and OLTR-C were covering their corporate butts and planning for a later RSA date.

“[…] what the City was looking for, aside from meeting RSA, was schedule certainty. Lorne Gray, the City’s contract manager for the OLRT1 project, explained, “It became … quite a tense relationship because we were looking for certainty on the revenue service availability date.… We just wanted a schedule that we could rely on, that was achievable.

[…]

“The fact that the City could no longer rely on what RTG and OLRT-C were telling it about how the OLRT1 project was progressing, and instead developed its own source of information, shows just how far the communication between the parties had broken down. This lack of trust was further reinforced when the Independent Assessment Team’s assessments generally confirmed that OLRT-C’s schedules were unlikely to be met.” [p 265]

The Inquiry notes that “much of the pressure was of the City’s own making” [p 266] because of promises made and the years of construction disruption to the city and its bus routes. Political claims of an “on time, on budget” created “a public expectation and a political imperative” [p 266]. The situation was viewed rather differently by the two parties, one with a clear love for spin.

“In the words of John Manconi, General Manager of Transportation Services, “There was disappointment. There was anticipation. There was pressure. There was excitement. There was energy….”

[…]

“OLRT-C Project Director Slade said that he had never worked on a project that was “so politically driven,” and that it felt like the project was “front-page news every day.” [p 266]

RTG was required to give six months’ notice of the actual RSA date, a difficult task given the many delays and uncertainties in the project. If the RSA date could be changed through recognition of bona fide delays beyond the P3’s control, this would give them more elbow room. Conversely, if the date was not changed, then liquidated damages would begin to accumulate for late delivery. Keeping the City in the dark about a real RSA target was, in effect, a bargaining tactic. RTG hoped they would get a contract extension while officially holding to the original RSA date.

The City took the position that May 2018 was a hard date, and that they would pursue claims if it were not met.

“[…] the City held RTG to its contractually required RSA date of May 24, 2018, rejected any compromise to testing and commissioning and to the safety and reliability of the system. The City wrote: “All of RTG’s obligations must be fully dispatched with adequate time to deal with issues that may arise and to ensure full readiness for Revenue Service Availability.” [p 269]

Much correspondence passed between the parties with the City taking a hard line, including holding back on milestone contract payments, and the P3 taking the position that they were entitled to relief. Without going into the details, this exchange shows how much the dispute poisoned the relationship even though the parties had agreed to set aside the resolution of sinkhole claims for a time so that the project could continue.

Moreover, RTG attempted to have the City accept Substantial Completion with only 32 of 34 vehicles “as the additional vehicles were only needed for maintenance” [p 272]. The City rejected this position for the obvious reason that maintenance spares are essential. At this point the poor fleet reliability in revenue service was still in the future, and RTG obviously had high hopes they could field the contracted service without the full fleet.

There was an extra wrinkle in these demands because the City became the project’s long term lender. In planning for the second stage of the LRT network, the City wanted the existing P3 to assume maintenance responsibility for it. That would require the original lenders’ consent because it was a major scope change. In turn, the lenders wanted a large addition of equity in the P3 which would be a cost to the City. To get around this, the City assumed responsibility for the debt to the lenders transferring their investment risk to a much safer place, the public sector. This change occurred in mid-2017.

The City’s strategy was not simply to untangle the contracts with respect to maintenance, but to gain access to information and leverage over RTG which was not available to them under the original contract. This put the City in a stronger position, but they had different goals as the buyer than the lenders would have for a long-term investment. Moreover, it unwound part of the “risk transfer” premise at the heart of the P3 structure.

“The debt swap was an uncommon, and by all accounts unprecedented, move in the P3 context. John Traianopoulos, of Infrastructure Ontario, said that he had never seen one before or since. But it raised concerns with RTG. RTG, said Cosentino, felt that the City was “tinkering with an established structure in a way that hasn’t been really done before, outside the mechanism envisioned in the contract.” Infrastructure Ontario learned from this experience; it structured later project agreements differently to hard-wire in consent from the lenders to system extensions, which makes debt swaps less likely to occur.

[…]

“The City became privy to additional information that a client would not typically see. While the City had some understanding of the importance of keeping its roles as client and lender separate, given its internal organizational structure, it would have been difficult to keep these roles apart.” [p 274]

Indeed, the City explicitly wrote to RTG in its role both as project owner and lender, and used the additional powers associated with the loan to add pressure on RTG. For its part, RTG saw the mixing of the two roles and the associated contracts as a lack of good faith.

Safety

Safety is a necessary part of every aspect of a transit system all the way from the infrastructure, vehicles, and control systems through to operating, monitoring and maintenance procedures. The concern begins right at the design stage and continues through the operational lifespan of a system, a period easily measured in decades. Necessarily, no one person or group can be possible for every aspect of safety, but there is an overall requirement for co-ordination and ongoing management.

The structure of the OLRT project was such that many parties had responsibility for aspects of safety, and many of these were not under the City’s direct control. Many subsystems and subcontracted functions’ safety lay in the hands of suppliers and maintenance providers who in turn reported upward through layers of the P3. Although the City may impose contract terms on its primary agency, the P3, there is no guarantee that these will flow downward or be co-ordinated at lower levels.

In the lead-up to revenue service, much of the safety review depended on documentation, and the City hired an Independent Safety Auditor to review and certify system readiness. However, there were three major shortcomings in this process:

• The Auditor was commissioned after the design was underway and, therefore, was not in a position to affect design as it occurred. This resulted in after-the-fact risk reviews and imposition of operating constraints that could have been avoided in design. (This shortcoming was corrected for the second stage of the OLRT.)
• Much of the Auditor’s work involved a review of documents, not of actual processes and work quality. The review assumed that if a document required something, it would be carried out, and done to spec.
• The Auditor’s scope did not extend to reviewing maintenance practices.

Although the City would receive a report certifying the line’s readiness, this would mostly be based on a paper review and a checklist that various safety documents existed. Moreover, the actual reporting structure in which the City bore ultimate responsibility for safety but its contractors carried this out exposed the City to gaps in work plans and quality.

The various plans exist in a hierarchy with an important one, the Engineering and Safety Assurance Case, at the top:

Even with a detailed set of safety plans, there are vital caveats including:

The highest-level safety document prepared by OLRT-C is the ESAC, the engineering safety and assurance case. The purpose of the ESAC – also referred to as the OLRT1 project use case – is to demonstrate that the railway infrastructure is fit for operation from a design and systems engineering perspective. Derek Wynne, of SEMP, described the ESAC as a “map” of the safety assurance evidence for the railway. At a high level, the ESAC is an audit of the safety of both the products used in building the railway and the processes used by OLRT-C to design, build, test, and integrate the overall system.

The ESAC includes a review of all the lower-level safety documents and analyses performed by OLRT-C and its subcontractors to prove the safety of the various components of the system. These lower-level safety documents include those prepared by OLRT-C, such as its Case for Safety and the Operational Restrictions Document, as well as safety cases from Alstom (for the vehicles) and from Thales (for
the signalling system). [p 285]

Of particular concern is that the ESAC does not automatically certify a system for revenue service, but is dependent on continued attention to requirements including:

The completion of all identified safety-related deficiencies designated to be completed prior to Substantial Completion or prior to RSA;

No safety-related events occurring during trial running relating to the infrastructure or LRVs; and

Correct maintenance of the railway throughout the period before revenue service. [p 286]

Having a stack of documents is one thing, but actually achieving their expectations is quite another. When the responsibility is fragmented, there is a good chance things will be missed, glossed over or shunted between groups even without pressure to meet schedules and budgets. For its part, the City cannot simply assume that its contractors will perform to spec, but must actively monitor their work. This is another example of the P3 mythology of risk and responsibility transfer.

Testing & Commissioning

With the many delays during construction, notably the sinkhole and knock-on problems it created, the period for system testing was repeatedly compressed. The City demanded that RTG achieve an impossible opening date and left it to them to figure out how. For its part, as we have already seen, RTG made delivery promises they knew were impossible to meet.

Meanwhile, key work to verify that the system and its many components would actually work reliably was compromised to fit available time. Moreover, the acceptance criteria were altered to make reduce the hurdles facing the new system. As we will see later, this change was not communicated to Ottawa Council.

The report cites Ottawa’s General Manager of Transportation Services John Manconi who

“[…] frequently assured councillors and the public that the City would require strict compliance with provisions of the Project Agreement to ensure that the OLRT1 was safe and reliable. On at least one occasion, he advised Council explicitly that there would be no compromise on the trial running of the system.” [p 16]

That statement was clearly false.

“It is clear from the evidence that everyone knew or should have known that there would be significant reliability issues with the system. Therefore, a robust testing and commissioning process was required to ensure that reliability problems did not arise during public service.

By the summer of 2018, testing and commissioning had been significantly delayed. The vehicles had not been able to run on the full track because of construction delays, which delayed validation testing. As a result, the timeline for integration testing was reduced. Further, there was no winter-specific testing on the track; that testing was limited to a laboratory.

The delays meant that the final stages of testing and commissioning had to be reduced, or the timing for RSA had to be changed again. The parties did not want to change the RSA date, because there was pressure on all concerned to get the system up and running.” [p 17]

The financial incentive to the P3 was substantial. Coupled with the political imperative to open on time, both parties aimed at a target that suited their purposes first, and system reliability second.

“… once the system achieved Substantial Completion and RSA, RTG would receive in excess of $250 million and the significant maintenance payments it expected.

“This is an example of the parties failing to put the interests of the people of Ottawa first. Instead of extending the time for testing and commissioning, they prioritized the swift completion of the project, thereby reducing the time scheduled for these critical activities because it was in their interests to do so. Unfortunately […] this was part of a pattern to get the system open regardless of the consequences.” [p 17]

The collusion went further with a private communication channel among a few key parties, while Council was kept in the dark.

A group of key City personnel including people in the Mayor’s office, City Manager Steve Kanellakos, and General Manager of Transportation Services John Manconi joined a WhatsApp Group to communicate about trial running. These individuals received important, concerning information about the system’s performance that should also have been provided to Council.

A memorandum to Council drafted by Manconi – to provide Council with what he viewed as important information about the OLRT1 system’s struggles and the exceptional decision to pause trial running – was not sent to Council because it was stopped by Kanellakos. Kanellakos and Manconi then provided several updates during trial running that contained none of the important information about what was going on with the system at the time, and Kanellakos then sent a memo to Council at the end of trial running that misled Council about the system’s readiness and performance during trial running. Mayor Watson was aware of the important information and that it was not being provided to Council; Mayor Watson’s conduct frustrated Council’s ability to perform its statutory oversight function. [p 296]

Substantial Completion

Substantial Completion is a key benchmark in any project because it marks not just the point where physical construction is finished, but all of the subsystems making up the project have been integrated and tested to meet the original requirements and specifications.

“In my view, the City’s decision to agree that Substantial Completion was achieved was made because the City was intent on moving the project into trial running, whether it was ready or not. The practical result was that the parties pushed out resolving known problems into the period of system operation.” [p 18]

In 2018 and early 2019, RTG (the head of the P3 structure) lost the City’s confidence through repeated proposed new project schedules, and in turn the City lost public credibility. The time for testing was cut leading to “frustration and tension” between various parties. Reliability through winter-spring of 2019 was poor with problems in track switching and vehicle subsystems.

Declaring that substantial completion was achieved had vital implications both for the City and for the various contractors within the P3.

“Finally, both the City and RTG/OLRT-C faced significant pressure to get the OLRT1 system open – namely, public, political pressure on the City, and financial pressure on OLRT-C. Missing multiple target RSA dates damaged the City’s credibility and caused a loss of public confidence. City Manager Steve Kanellakos agreed that by August 2019 there was “a lot of public pressure, and Council pressure, media pressure with respect to launch of the system.” [Matthew Slade, of OLRT-C] similarly testified that the pressure to hand over the system was “very significant” in the summer of 2019. This intense pressure was felt by both the City and RTG. OLRT-C was suffering financially: the consortium partners had been called upon to inject substantial additional cash into the OLRT1 project due to cost overruns, and […] OLRT-C continued to incur liquidated damages and costs with each day of further delay. OLRT-C was keen to receive the large financial payments associated with achieving the contractual milestone of Substantial Completion and achieving RSA (a total of approximately $250 million) in order to, as Slade put it, stop the bleeding (meaning, they had paid out substantial sums to fund the ongoing work without receiving these substantial payments).” [p 302]

The process involved a formal exchange between RTG, the City, and the Independent Certifier. However, there was a gaping loophole in the Project Agreement which “allowed the City to agree that RTG had achieved Substantial Completion despite the contractual requirements not having been met.” [p 304]

The first attempt at getting Substantial Completion approval by RTG was on April 26, 2019. This was rejected by both the City and the Independent Certifer because the system fell well short of requirements. A key deficiency was that 25 of the 34 vehicles were not fit for use, and OLRT-C could not operate 15 two-car trains as required.

The City outlined 130 issues in Appendix A to its opinion letter, identifying:

• Issues concerning safety, including the incompleteness of the engineering safety and assurance case, or ESAC;

• Issues related to vehicles, including the fact that the City required all vehicles to be ready for use; defects related to brakes, doors, communications, and the line contactor; and several items of outstanding work; and

• Several other issues with the transit operations control centre, the supervisory control and data acquisition system, the communications-based train control (CBTC) system, the stations, winter performance of snow removal and switch heater equipment, and the tunnel ventilation system. [p 305]

Clearly at this point the City was holding RTG’s feet to the fire and flagged a long list of problems with the system. A few months later, on July 22, 2019, RTG made a second attempt even though a long list of issues remained open from the first round. The City chose to agree that “Substantial Completion” had been achieved through reclassification of items and the time when resolution was required.

“The Substantial Completion Agreement was clear that RTG and the City agreed that the deferred work items were not minor deficiencies (which, under the Project Agreement, had a separate process for correction and completion). However, the parties agreed to treat the deferred work items as though they were minor deficiencies under the Project Agreement. The impact of deferring those items and requirements, and their technical classification as “minor deficiencies” under the Project Agreement, was that the Independent Certifier did not review or consider them when assessing whether Substantial Completion had been achieved. The Independent Certifier was made aware of the Substantial Completion Agreement, but the Independent Certifier was not asked to give an opinion on the deferred work.

“From the perspective of the Independent Certifier, the Substantial Completion Agreement profoundly changed the Project Agreement requirements for Substantial Completion. Thus, although the Independent Certifier was aware that many of the same issues that had caused it to reject the earlier submission remained outstanding, the Independent Certifier accepted RTG’s second application and issued the Substantial Completion Certificate because the City and RTG had entered into the Substantial Completion Agreement.” [p 306]

Something vital had changed between April and July, and the City was willing to bend the rules of its contract to help the P3 achieve Substantial Completion. In spite of severe problems with the vehicles, the City pressed ahead. The inquiry was not impressed with this action.

“Given the ongoing, visible problems with the system and the history of issues with RTG in 2018 and 2019, the City’s apparent optimism that the issues it was seeing would be addressed and the OLRT1 system would improve over time was misplaced. The City’s decision to defer certain items in July 2019 so that RTG could reach the milestone of Substantial Completion, when many of those same items had been reason to deny RTG Substantial Completion in May 2019, was made because the City was set on moving the OLRT1 project into trial running, whether it was ready or not.” [pp 307-308]

This opinion plants the responsibility for problems that would come squarely with the City which was bent on getting the system open. The P3 would be paid for incomplete work and appear to be more successful than it actually was.

Trial Running would follow Substantial Completion, but this was fraught with problems on the technical, managerial and political levels. Here all of the compromises, budget pressures and political commitments would come undone. Part V of this series will review that key period in the project’s history.