This is the third part of my review of the Final Report of the Commission of Inquiry into the Ottawa LRT fiasco. This article covers mainly the vehicle and control system manufacturing, and corresponds to chapter 9 of the Inquiry Report.

See also:

• The Ottawa LRT Report Part I
• The Ottawa LRT Report Part II

Part IV will deal with the construction of the line, and Part V will deal with the transition to revenue service.

The entire process is a textbook example of what happens when “on time, on budget”, coupled with an unproven design, forces abandonment of well-established best practices for manufacturing and testing. Moreover, the lack of integration and communication across the project puts the lie to the idea that a private sector consortium will automatically be run like a well-oiled machine rather than a clanking contraption on the edge of collapse.

Updated Dec. 12/22 at 5:30 pm: The discussion of signalling systems used on various lines has been corrected to cite Bombardier’s Cityflo 650 system as the one used on Line 5 Eglinton.

Who Does What

A major part of the inquiry’s findings concerns the lack of co-ordination and clear responsibility among the various parties. As an introduction to the section on construction, the report reprises the list of players and how they were supposed to interact. (This is a recap from Part I of the series for readers just joining in.)

• Rideau Transit Group (RTG) is the top of the P3 pyramid. They contracted the actual construction of the line to Ottawa Light Rail Transit Constructors (OLRT-C), a joint venture of Dragados, EllisDon and SNC-Lavalin. The latter two also were part owners of RTG, but this is a separate entity from OLRT-C.
• In turn, OLRT-C contracted the design and engineering of everything except the trains and control system to RTG Engineering Joint Venture which is a joint venture of SNC-Lavalin and MMM Group. They produced the plans and specs for construction actually carried out by OLRT-C.
• Alstom supplied the trains for the system and had full responsibility for them from design through manufacturing and testing to most of the maintenance.
• Thales supplied the train control system. Note that the choice of this vendor was separate from the train supply contract with Alstom, a situation which created the need for integration between vehicles and signalling.
• Late in the project, in 2017, OLRT-C contracted SEMP, a UK systems engineering and assurance consultant, to verify how well the system elements were actually integrated and to provide a safety assurance case.

There were many players on the City side of things as well. This project was not simply handed to RTG as a blank slate for them to design from scratch.

• The City had a Rail Implementation Office (RIO), later folded into the O-Train Construction Office in 2016.
• The transit operating agency, OC Transpo, had little involvement with the design, construction and manufacturing phases, but advised on service issues.

It is abundantly clear that the City and at least some of its contractors were well out of their depth on this project. Project timetables were unrealistic, but those problems were not acknowledged when they were evident. There is plenty of blame to go around including the Mayor and City management attempting a project constrained by a fixed budget and a tight schedule, and suppliers who committed to impossible deadlines.

The City retained consultants both for initial design and ongoing review including:

• Capital Transit Partners comprising the firms STV, URS, Jacobs Associates and Morrison Hershfield.
• In 2015, Parsons was retained primarily to assist with implementation of the Thales train control system, operations and maintenance plans, and system safety.
• In 2017, the City created an Independent Assessment Team to advise on RTG’s construction progress and the readiness of Rideau Transit Maintenance (RTM) to maintain the system.

Although there are many separate entities, there is overlapping participation and ownership, no doubt to isolate future legal liabilities. The entire structure can be difficult to manage with so many players, and with the P3 itself, RTG, taking on so much responsibility beyond direct management of the project’s owner, the City.

The complete lack of expertise of the City and of its procurement advisors, Infrastructure Ontario and Boxfish, in large transit projects did not help either. As discussed in an earlier article, the P3 contract structure designed by IO was inappropriate for such a large project much different from any of their previous work.

The report observes that the City took a passive role in much of the work on the assumption that its various contractors would bring the necessary expertise. A significant shortcoming was the lack of a “concept of operations” that would inform design work with an expectation of how the system would work in day-to-day operations including failure management and maintenance. This would normally exist before system design, let alone procurement, and would define the owner’s expectations. In fact, this document was not prepared until 2015 when a consultant, Parsons, flagged the need for one, but had to fit the content to what already existed in the design.

Vehicle Manufacture and Testing

The contract with Alstom called for the supply of 34 Light Rail Vehicles (LRVs) to be operated in 2-car sets as 17 trains. The original plan was to produce two prototypes and shake out any problems with them before proceeding to the full manufacturing run. (Toronto readers will remember both the CLRV and Flexity prototypes for their respective fleets.) That plan was not followed due to project delays.

The Thales train control system keeps track of train locations and operates the trains through a combination of equipment along the line and on-board controllers. The basic functions are similar to the system now in use on TTC’s Line 1 subway, but that uses Alstom’s Urbalis train control system. Line 5 Eglinton uses the Bombardier Cityflo 650 system. The future Ontario Line trains will come from Hitachi which expects to purchase Thales Ground Transportation Systems business, including their signalling technology, in 2023. Thales is also supplying the signalling system for Line 6 Finch. (A much earlier version of their Seltrac system runs the SRT Line 3 which is about to be decommissioned.)

There is a long series of tests during the construction of vehicles that I will not detail here, but it ends with integration testing of vehicle components and the signalling package, followed by commissioning to ensure that the vehicle works according to the buyer’s requirements.

If this were a literally “off the shelf” vehicle, a great deal of the process would already be well-understood and the design proven by vehicles already in service. However, the Ottawa design was not “off the shelf”, although its components existed as parts of other fleets. Moreover, the construction occurred in a new shop with inexperienced workers. A great deal of the process started from scratch.

This was the first time Alstom would manufacture vehicles in North America although they had long experience in Europe, and this was not simply a matter of setting up a factory.

Alstom witnesses candidly acknowledged that they had not sufficiently anticipated or planned for these challenges in the early stages of the OLRT1 project.

Perhaps most crucially, Alstom had to set up new supply chains for the LRVs for the OLRT1 project. This was necessary, in part, to respect the Canadian content requirement in the Project Agreement. However, Alstom also explained that, regardless of the Canadian content requirement, the company intended to use the OLRT1 project as a strategic long-term investment to build relationships with suppliers that could be used for other business opportunities in North America.

Alstom did establish a detailed process for selecting suppliers and testing the suppliers’ products for quality. However, as [Yves Declercq, Alstom’s Bid Director] acknowledged, testifying in French, Alstom “underestimated the difficulties of setting up the supply chain, the supplier base, and the qualification of these suppliers.” These supplier issues caused both quality and schedule issues that likely would not have occurred if Alstom had been using an established supply chain. [pp 206-207]

An original plan to build the prototypes in France was abandoned because of the complexity of using North American suppliers for a European product, and this meant the work would be done in a new plant in Canada. In effect, both the plant and the cars would be prototypes.

Part of Alstom’s challenge lay in the tight project schedule after the City disqualified a proposed vehicle from the Spanish manufacturer, Grupo CAF, as not being service-proven. Alstom had only two months to prepare their bid and gain qualification from the City.

This situation shows clearly how the City boxed itself in both with a “service proven” requirement and by bundling the vehicle supply into the overall P3.

Potential builders and their vehicles were not even in the running because they were part of other P3 bidder groups.

If a “service proven” vehicle were required, then the system should have been designed around the vehicle specs, not the other way around. By contrast in Toronto, streetcar specs were dictated by the requirements of the existing streetcar system, and the Flexity design was adapted to it. On the Eglinton project, the line is designed to the specs for the vehicles that will operate there (grades, curve radii, etc).

The need to integrate the Thales signalling system just added to the problems.

In anticipation of the above challenges and to protect its commercial position, Alstom negotiated strict timelines with early deadlines in its subcontract with OLRT-C, particularly with respect to OLRT-C’s obligation to provide the City’s design and radio selections, and to provide Thales’s ICD [Interface Control Document]. Earlier dates would allow Alstom to claim compensation or seek a variation if OLRT-C failed to meet its obligations under the timelines in the contract. […] the earlier dates were accepted by OLRT-C, even though they were unrealistic. Unsurprisingly, then, there were soon delays in the project as these overly optimistic deadlines were missed. [p 207-208]

The challenges and shortcomings of the Alstom-Thales relationship are detailed later in this article.

Even such basic things as the interior design of the vehicle were delayed. The report notes that there is some debate about responsibility, but this delay also factored into the shift in manufacturing location due to lost time in the overall project.

The other major design delay was in the ICD which would specify how the Thales control system would interface with Alstom’s vehicles.

Through a provision in its subcontract with OLRT-C, Alstom was entitled to receive a finalized ICD from Thales by April 26, 2013. However, this was an unrealistic date. It was only two months after the start of the contract. The LRVs were not sufficiently developed in April 2013 for Thales to be able to produce a finalized ICD: the relevant details of the LRVs were simply not yet known at that time. Where the details of the LRVs are unknown, it takes particularly long to finalize the ICD, as it cannot be known in advance that all the signals will communicate correctly between the CBTC and the vehicle. [p 209]

The inquiry chose not to assign blame for these delays, but noted:

What is important to note is that both delays influenced the parties’ decisions about where to manufacture the first LRVs. In turn, those decisions about manufacturing had greater consequences, resulting in the inability of Alstom and Thales to conduct validation testing early in the process. [p 209]

The Shifting Site for Manufacturing

Manufacture of the prototypes was planned for France, then shifted to New York State, and finally to the Maintenance and Storage Facility [MSF] to be built as part of the Ottawa project. Each of these changes was intended to save time in an already-late project. This process interfered with testing, and in particular with completion of the prototypes before starting to manufacture of the balance of the fleet.

The original plan was to use LRV1 and LRV2 for validation testing, with about a year’s gap between the completion of LRV2 and the serial manufacturing of subsequent LRVs in Ottawa. Both Alstom and Thales had been planning to do the early testing on the LRVs in Valenciennes, where Alstom’s facility had a test track that could be used for dynamic testing. Thales also planned early testing on a test track; in its contract with OLRT-C, Thales had planned to install test track equipment in France and deliver the first two VOBCs there so that Thales could conduct validation testing using the first two LRVs. Dynamic testing of the VOBCs on the prototype LRVs at that early stage would have given Thales valuable information about how the LRVs performed and allowed Thales to reduce the engineering time for the rest of its work on the train control system.

However, there was no suitable test track in Hornell [NY], so Thales was unable to do that early dynamic testing. For the same reason, Alstom was also unable to perform validation testing in Hornell to the same extent as it would have done in France. Clearly, when the decision was made to move the manufacturing of LRV1 and LRV2 to Hornell, Alstom did not intend to abandon dynamic validation testing. However, the move occurred without alternate plans in place for that testing. While Alstom raised the possibility of doing dynamic validation testing at the Transportation Technology Center in Pueblo, Colorado, this, too, did not take place. Despite OLRT-C’s initial concerns about the location change, it does not appear that OLRT-C consulted Thales about the change in location and the impact that change would have on Thales’s work.

Ultimately, except for some limited testing on LRV1 in Hornell, most validation testing took place in Ottawa. As delays to LRV design squeezed the schedule, the planned gap of about a year between the completion of validation testing and the start of serial manufacturing did not occur. Instead, serial manufacturing of the rest of the LRVs started right after the completion of LRV2. In another effort to save time, Alstom conducted validation testing on the first four LRVs (rather than the planned two prototype LRVs), which was more resource-intensive. More importantly, validation testing took place at the same time as serial manufacturing and serial testing were occurring. [pp 211-212]

As if all that were not bad enough, OLRT-C was supposed to provide a 4.5 km double track test area on the main line by September 2016. However, this did not occur, and a much shorter single track test facility was provided part way into 2017. This further delayed testing of vehicles and the control system. Vehicle testing continued into 2019 triggering retrofits to the already completed fleet and a further round of testing.

Setting Up a New Production Line

Although the manufacture of 32 or the 34 vehicles was always intended to occur in Ottawa, this scheme was not without its problems. North American supply chains did not exist, and labour was hard to source at least in part because this would be a temporary facility with no long-term job guarantees. At the time of these events, Bombardier had not been acquired by Alstom, and they were part of another of the would-be consortium bidding for the P3 contract. Their existing plants and workforce in Canada were not available for this project, leaving aside any concerns with Bombardier’s abilities as a supplier.

Even the MSF was not available on time, like so much else in the project.

Under OLRT-C’s subcontract with Alstom, OLRT-C was to have the MSF available for manufacturing on July 1, 2015. But the decision to manufacture LRV2 in Ottawa rather than Hornell required an earlier start in Ottawa than initially planned. OLRT-C agreed to move up the timeline so that some parts of the MSF – the final vehicle assembly area and offices for Alstom personnel – would be available in May 2015, for Alstom to begin installing its equipment. According to the revised timeline, the remainder of the MSF would be available for Alstom’s use by July 2015, at which point vehicle assembly could begin. This did not go as planned. Following a review of the site on July 31, 2015, Alstom wrote to OLRT-C saying that the MSF was “still clearly a construction site.” Alstom eventually took occupancy of the final vehicle assembly area on August 26, 2015, although it argued that even in November 2015, the site was still not ready given its state. [p 218]

Even with vehicle assembly underway, the facility was not complete, notably in the absence of power in the Light Maintenance Bay [LMB].

This meant that Alstom could not perform its full suite of tests in the LMB. Vehicles that Alstom manufactured began to accumulate but could not be fully tested until overhead catenary power was available; as a result, issues with the LRVs were not being caught soon after the assembly of each vehicle or before assembly of additional vehicles took place. Even once the LMB was usable for testing, it was subject to competition for space, as Alstom needed to use it for validation testing at the same time. In addition, over nearly three years, the LMB was subject to repeated power outages that prevented testing from taking place. [p 218]

The 25% Canadian content required that the vehicles be assembled in Canada as this is the easiest way to achieve the necessary labour component. Otherwise, Alstom would have done this work at their plant in Hornell, NY. The downside of this is that Ottawa does not have a pool of skilled workers.

Manufacturing the LRVs in Ottawa required hiring and training new personnel for the job, which took significant time and effort. While Alstom transferred some of its own employees to Ottawa, typically for managerial roles, the workforce responsible for manufacturing the LRVs was generally made up of new people hired locally through an agency and sent to Alstom’s facility in New York for training. The new local personnel may have been trained, but they had no experience. [p 216]

Regardless of the location, Alstom also required new North American suppliers who could eventually be part of their breakthrough into this market, primarily in the USA.

[…] while the particular vehicle chosen for the OLRT1 project was related to an established family of vehicles used worldwide, Alstom had to establish new supply chains to produce components for the fairly small fleet of LRVs required for the OLRT1 project. Working with new suppliers sometimes meant adapting the design to what the new suppliers were able to produce. [p 215]

Several of their new suppliers did not produce to the required quality including auxiliary power units, line inductors, bogie [truck] castings and brake calipers. The report observes that:

These supply chain and quality issues may have added to manufacturing delays, and it is likely that there would have been fewer quality issues if Alstom had been able to use its usual suppliers. [p 215-216]

The report talks at some length about Canadian content requirements and the constraints this places on would-be suppliers. There is an obvious difference between a firm with an established manufacturing facility and one starting from scratch, especially with a vehicle that is not quite as “service proven” as the City’s initial specification claimed to require. An obvious question, in general, is how reasonable such a requirement is particularly for small lots of vehicles.

The report does not address the wider options that would have been available if the vehicle suppliers and the overall P3 consortia were not locked in exclusive relationships thanks to the procurement structure. There simply are not enough Canadian manufacturers to flesh out multiple P3 groups. Even so, some international firms might simply decline to set up Canadian assembly plants as the potential volume of work is not worth their investment.

The Alstom-Thales Interface

Although Alstom and Thales were to provide trains and control systems, respectively, without which the line could not function, they had no relationship with each other.

Alstom and Thales did not have a contract with each other; instead, each had a subcontract with OLRT-C. As a result, both subcontractors depended on OLRT-C as their point of contact, for coordinating their different but related activities, and for the integration of their systems. OLRT-C did not perform these tasks well: the schedules in the Alstom and Thales subcontracts were misaligned; the deliverables (what each company was to provide or do) in the two subcontracts were also misaligned; no one at OLRT-C was assigned to manage the integration of these key systems until nearly a year into the contract; and the lagging integration caused operational issues. [p 222]

The City chose to specify a Communication Based Train Control (CBTC) system because the system spec would require a high capacity and close train spacing. That is more easily achieved with an automatic system than one based on the older style block signals because the system knows moment to moment where every train is and how fast it is travelling.

To integrate their systems effectively, Alstom and Thales first had to work together at the level of hardware. […] The more important and difficult task was to then ensure that the software of the two systems could communicate to each other, so that signals and commands sent from one system could be properly responded to by the other. As a simple example, if the VOBC gave a specific brake command, the LRV needed to receive that command and then trigger the right level of braking in the calipers and other brake components.

The importance of getting this Alstom-Thales interface right is obvious. Without proper integration between the signalling system and the vehicles, the vehicles will not accelerate or brake properly, nor can safe headways be maintained. This was a critical interface. [p 223]

A major problem lay with OLRT-C to whom both Alstom and Thales were contracted. The schedules given to each supplier were different. Most critically, the contract with Alstom promised a “frozen” specification for the train control interface on April 26, 2013, whereas the contract with Thales did not require a final ICD until September 2014.

[…] even at the contracting stage, OLRT-C failed to meet a fundamental precondition to establishing a feasible consolidated schedule: negotiating contractual obligations that were consistent with each other. [p 224]

At this point, the report hints at another basic issue with the project: both the early City consultants and the P3 were primarily focused on civil infrastructure, not on railway technology.

[Alstom Bid Director Declercq …] observed that OLRT-C did not have engineers at the table with the necessary skillsets to assess the deadlines and schedules: “There were people with an engineering background, project directors, the bid manager, but no technical experts in front of us.… We never felt the presence of a system engineering [expert] that understood the integration and interface issues between the vehicle and the control train.” [pp 224-225] [“control train” should read “train control”, but the first sequence is a direct quote.]

A further problem lay in the division of project scope between Alstom and Thales at both the assembly and testing stages. The contracts with OLRT-C provided no guidance on how this should be resolved.

Notably, OLRT-C does not dispute that there was room for improvement in its management of the subcontracts with Alstom and Thales. On the contrary, Manuel Rivaya, OLRT-C Executive Committee member, readily agreed with Commission counsel’s suggestion that, “in hindsight … the contractual deliverables in the Thales and Alstom subcontract[s] should have been better aligned.”

The lack of clarity in the Alstom-Thales subcontracts, their misalignment of schedules and deliverables, and the “unnatural division of responsibilities” resulted in conflict and delays. [p 226]

The report criticizes OLRT-C at some length for their inattention to systems integration.

When OLRT-C entered into the subcontracts with Alstom and Thales, it had not hired anyone with the necessary training and expertise to manage this critical interface. Consequently, early in the project, Alstom and Thales felt they had been left on their own to integrate the vehicles with the signalling system, without support and without leadership. To paraphrase Alstom’s senior engineer, Goudge, OLRT-C’s approach to systems integration was to put Alstom and Thales in a room and let them figure it out, especially at the outset of the OLRT1 project. This was far from ideal for two entities that are competitors (which may have made them wary of sharing certain information) and that had no contractual relationship with one another. Indeed, one of the key roles of a systems integrator is to make decisions about how to move forward when conflicts arise. [p 227]

Alstom has its own CBTC train control system, a direct competitor to Thales, and it is hard to imagine the two companies openly sharing information on how their systems worked unless this process were managed by a third party.

OLRT-C’s approach [to managing its suppliers] also ignored the fact that Thales and Alstom were competitors in the train control market. It was evident early in the OLRT1 project that their competition in the market was an issue, because there was a lack of co-operation between the two companies. The issue of Thales and Alstom being competitors was a known risk from the outset of the project and should have been better managed by OLRT-C. [pp 233-234]

In early 2014 – nearly a year after OLRT-C signed subcontracts with both Thales and Alstom – OLRT-C hired Jacques Bergeron as its Director of Integration. Once in place, Bergeron conducted a series of interface workshops between Alstom and Thales. The goal of these workshops, and Bergeron’s work overall, was to develop Alstom’s and Thales’s ICDs so that they were fully integrated. Bergeron was also able to make recommendations to the subcontractors about how to proceed in the face of any disagreements. [p 227]

This is a further example of OLRT-C’s inattention to a critical technical component of their project. Bergeron retired before the project was completed, and the report notes a deteriorating relationship between the two companies from that time onward.

Problems remained right into revenue service due to inconsistencies between the Alstom and Thales subsystems including aspects of each that were unknown to the other vendor. This included issues with emergency braking and door operations. This type of problem was a direct result of the lack of testing which would have uncovered the issues before revenue service began.

Systems Engineering & Integration

The complexity of transit vehicles and control systems demands careful attention to how the various components work together, especially if they are from suppliers who have not done this work before. The systems are safety critical, and are essential to the functioning of a rapid transit line.

With the OLRT1 project, however, systems integration was a critical risk because the project involved several “firsts” and unproven elements in the most important interface on the project: that of the vehicles and train control system. These elements included:

• LRVs that were significantly adapted from Alstom’s existing Citadis Dualis model and that were to be operated at the limits of an LRV’s capabilities;
• The first time Thales and Alstom had, reportedly, integrated their systems for an LRV;
• The first time an automatic train control system with a moving block (a zone of space around each train that moves with the train, thus allowing the trains to run closer together than other types of systems) was used with a Citadis family LRV; and
• The first time a CBTC system was integrated with a low-floor LRV.

[p 232]

All of this was further complicated by the web of companies and contractual links in the project and the myriad hand-offs between parties. Overall responsibility lay with RTG, but they passed it on to OLRT-C who in turn depended on its suppliers to do the work.

As I mentioned earlier, the focus of the P3 was more on construction, their collective field of expertise, than on systems that would operate the line.

Despite the critical importance of systems integration, the OLRT1 project began with a focus on the civil construction works. For the first three years of the project, this involved widening Highway 417 and excavating the downtown tunnel. The civil works were prioritized over systems integration and systems engineering, which became a source of problems as the OLRT1 project progressed. [p 234]

The report details ongoing problems among various companies and groups about where responsibilities, including integration issues, might lie. It is not pretty reading.

OLRT-C candidly acknowledged that the disputes and misunderstandings of the parties’ responsibilities delayed the progression of systems integration. In the end, it does not matter who was right in the disputes regarding contractual responsibility for systems integration. The important point is that they left systems integration and engineering largely unaddressed until too late in the OLRT1 project. This was a failure by OLRT-C, which had the ultimate responsibility to deliver an integrated system to RTG. [p 235]

An outside review of the project by consultant SEMP concluded that:

[…] the level of System Engineering on the project to date is considered to be substantially below the minimum acceptable level for a project of this size and complexity. This is especially evident at the Railway System level and for early phases of the lifecycle (requirements and design). [p 237]

A thorough review of the engineering process resulted in some design work being done, in effect, in reverse to ensure that is was of appropriate quality. The fact this was necessary does not speak well to the companies responsible for the work in the first place.

In conclusion, OLRT-C’s approach to systems integration was too passive, too little, and too late. Where a focus on systems integration did emerge, it was mostly directed at integrating the systems from Thales and Alstom, but overall systems integration, and systems engineering and assurance, suffered. [p 238]

This is a stinging indictment, and one cannot help wondering how these companies get work anywhere.