This is the fifth part of my review of the Final Report of the Commission of Inquiry into the Ottawa LRT fiasco. This article covers the trial operation of the line up to the transition to revenue service and draws from chapter 12 of the Inquiry Report.

Some of the quotations here are extensive as I want there to be no doubt that the voice is that of the Inquiry Commissioner, not my paraphrase.

See also:

• The Ottawa LRT Report Part I
• The Ottawa LRT Report Part II
• The Ottawa LRT Report Part III
• The Ottawa LRT Report Part IV

Trial Running Criteria

The purpose of trial running is to mirror planned service and prove that the system can actually operate at that scale. This is different from testing individual trains and control systems, not to mention other aspects of the infrastructure. A trial should reveal problems that will arise from continuous use of all components.

As the inquiry report had already shown, there was a change in the Substantial Completion criteria in mid-2019 that had allowed the project to progress to Trial Running in the first place. However, RTG also wanted the Trial Running requirements to be relaxed so that the project could get to Revenue Service. Basically, there was an effort to have a system with outstanding deficiencies slip through project milestones so that revenue would flow to the P3 before they had really earned it.

The criteria for a successful demonstration had two versions: the first set written in 2017 while the line was under construction was replaced in 2019. The Inquiry Report describes the 2017 set as “somewhat vague” by comparison to the 2019 set with many aspects “unaddressed” and with generous margins for acceptable performance. [pp 310-311]

Also, by 2019 problems were evident from testing that did not exist in 2019, although they should have been foreseen at least as requirements to be met. Work on the revisions began in December 2018 and an important change was that the new version was based on “the performance expectations set out for the OLRT1 system in the Project Agreement” [p 312]. The surprise here is that the 2017 version was more superficial and would not have enforced the Agreement.

“The Commission finds that the 2019 criteria were (1) developed deliberately and after due consideration of the shortcomings of the 2017 criteria, (2) considered appropriate by the City and RTG, (3) clearly agreed upon by the City and RTG, (4) appropriately aligned to the Project Agreement service requirements, and (5) devised at a time when the parties had the best possible knowledge of the system.” [p 312]

The tightening of criteria is intriguing because it occurred after the line was initially supposed to open in 2018, and after the City had rejected a proposal by Rideau Transit Group (RTG, the head of the P3 structure) to ease the requirements. Officially, the City did not want to assume the risk of accepting a system that could not meet its specs, but this would change due to political pressure to start revenue service.

An important change in the criteria was to set a high value for achieving the target average kilometres of daily operation. Averaging could disguise poor performance on selected days even though the overall 12-day number might look good. Only a very high target for the average would prevent this. The 2017 criteria required 96 percent of the target level of 9 days out of 12, whereas the 2019 criteria required 98 percent on all 12.

(Readers of this site will recognize a common problem with the TTC’s standards for service availability and reliability where averaging masks wide variation in actual performance.)

Eventually, during testing, the project reverted to the less demanding 2017 criteria because it could not meet the 2019 version. This shows a profound change in the City’s demand that RTG meet its original promises.

“Unfortunately, the OLRT1 system did not do well during trial running, repeatedly failing to meet the criteria established in 2019. These failures and the pressure to open the system to public use combined to cause poor decision-making by both the City and RTG. Most significantly, the City and RTG agreed to revert to the easier and flawed 2017 criteria, because the OLRT1 system simply could not pass if they continued to use the 2019 criteria. Rather than acknowledge the problem and seek expert advice about what to do, the City and RTG instead agreed to revert to the easier standard, but there is a lack of any documented expert or other analysis of the reasons for and consequences of doing so.

“As will be seen, the City’s decisions during trial running were being made by a select group of people who were given full and accurate information, rather than by the City Manager (to whom sole authority had been delegated), the FEDCO (the Council committee responsible for the OLRT1 project), or Council.” [p 314]

Although the Independent Certifier signed off on the daily scorecard, they did so on the advice of the City and RTG. Only if the two did not agree would the Certifier get involved in the evaluation. This could give a veneer of “independent” review to a process that was defined and controlled by the two parties. Moreover, the Certifier had no role in creating the acceptance criteria he was supposedly monitoring.

Trial Running Begins

Before the 12-day trial period began on July 29, 2019, there were reliability problems with the system. On July 25 and 26 respectively, only 4 and 5 trains were operating because “others did not meet launch criteria”. [p 316]

To say that initial tests went poorly is an understatement. The report described those days with strong language. An important metric is the Average Vehicle Kilometre Ratio, or AVKR, which is the ratio of actual mileage operated to the scheduled value.

“The start of trial running was disastrous. The first day, July 29, was scored as a “fail” and many individual items on the scorecard were not even close to passing. The AVKR was below 86 percent (the minimum was 90 percent; […] even that threshold was described by Morgan as ruining users’ commute to work). The travel time was too long. The ratio of trains successfully completing trips fell far short of the 2019 criteria requirement. Station subsystems such as the closed-circuit television (CCTV) cameras and passenger information system failed.”

“July 30 was arguably worse, with headway and station ratios badly missing the scorecard criteria, station availability failing, and AVKR again at approximately 86 percent. The day was scored a “repeat.” That same day, Deloitte advised the City that the integrated management information reporting system (IMIRS) was not providing reliable data. (The IMIRS […] is the computer system RTG used to track work orders.) That was a problem because, as the TRRT minutes put it, the City and RTG were “heavily reliant” upon the IMIRS to demonstrate and confirm that the maintenance system was ready for public launch.” [p 316-317]

July 31 was no better, and the TRRT group decided to “pause” the trial running. This would also force the trial to restart from the beginning. This was unsettling to the Mayor, but even worse, the problems were not shared with Council.

Manconi apparently shared that concern and felt that Council should be informed about the poor start to trial running and the need for a restart. Manconi thus drafted a memo dated July 31, 2019 to Council to advise of the OLRT1 system’s struggles, obviously feeling that it was appropriate to inform Council of this significant step (the use of the exceptional “pause and restart” option). However, that memo was never sent to Council. Manconi explained to the Commission that City Manager Kanellakos “reminded” him that they had committed not to update Council unless and until trial running was complete. That alleged “commitment” is the City’s explanation to this Commission for the decision not to send that important
memo to Council, which meant that Council did not have the chance to learn of Mayor Watson’s concern and did not have the opportunity to inform itself and potentially act. [p 317]

The Inquiry Commissioner was quite blunt in his assessment of this situation.

“I find that quashing this memorandum was inconsistent with Kanellakos’s obligation as City Manager to keep Council informed about important matters and was an act of intentionally misleading Council. I further find that Mayor Watson, by not ensuring this information was provided to Council, prevented Council from exercising its statutory oversight authority. Mayor Watson had a duty to “ensure the accountability and transparency of the operations of the municipality, including the activities of the senior management of the municipality.” These obligations are not mere words on a page; keeping Council informed is a critical element of municipal governance.” [p 319]

There is a fascinating comparison here with the conventions of municipal and provincial governance. Cities are expected to be transparent and the behaviour of its officials and politicians is roundly denounced by the Commissioner. By contrast, provincial agencies like Metrolinx operate under a veil of secrecy providing as little information as possible under the rubrics of “commercial confidentiality” in their dealings with P3s, and “cabinet confidentiality” in their dealings with the government.

A Second Start and Diluted Criteria for Success

The trial running resumed with the clock reset. Day one was August 3, 2019, a Saturday at the start of a three-day holiday weekend when the service level, and hence the target performance would be easier to hit. The system managed to achieve a “pass” grade for four days, but on the last day, August 6, the system achieved an AVKR of only 91.6% well below the 98% average, and a level that would represent inferior service for riders.

(An important note about AVKR is that it is measured over a full day’s operation, but the missing service is not necessarily evenly distributed through all hours. For example, if peak period trains are missing, service is degraded at a time of heavy demand even though a better hour-by-hour AVKR might occur in the off peak.)

Things got even worse as the attempt to field weekday level service continued.

The system’s performance then took a significant turn for the worse on August 7 and 8. August 7 was a terrible day, with failures in nearly all categories and an AVKR ratio of just 85.2 percent. August 8 was worse. The moderately improved performance of August 3–6 was gone, and the City and RTG were searching for answers. [p 320]

The search for a fix concentrated not on running better service, but on moving the goalposts so that “success” could be achieved.

Manconi then met on the evening of August 7 with RTG and OLRT-C, and it appears from a contemporaneous email summarizing that meeting from Peter Lauch, of RTG, that Manconi began a discussion about how the parties could use their discretion and make changes to trial running in order to ensure a pass. Lauch cites an email from Slade, which states that OLRT-C expected the score from that day to be a “fail” (which, because of past “fail” days, would require a restart) and that OLRT-C tried to “make a case” that it could be a “repeat” rather than a “restart,” characterizing this as a “favour” it needed from the City. The summary email then notes that Manconi asked “‘what’s in it for me’ to get you a PASS on Trial Running.” At the public hearings, Manconi denied making the “get you a pass” comment. Other witnesses could not recall whether he had said that, though they confirmed that Lauch’s email summary was the best evidence they had of who had said what at the meeting.

The statements attributed to Manconi in the email are entirely consistent with what else was being said and done by Manconi and Kanellakos around that time. On August 8, Manconi was asking his team (via another WhatsApp messaging group he had with City staff and STV) whether August 7 and 8 would be scored a “fail” and, if so, whether trial running could still be successfully completed by August 16. He was advised that there would likely have to be a restart. On August 9, Manconi responded to a question from the Mayor’s office about the impact of the August 7 and 8 results by stating that he had a “bigger strategy” he was bringing to the group. [pp 320-321]

The proposal to revert to the less stringent 2017 trial criteria dates from this period, although there is some debate about where exactly the idea originated. However, the Inquiry observed that this really didn’t matter.

The more important point is that the City and RTG were searching for ways to pass trial running regardless of the reliability problems, each for their own reasons: political, in the City’s case; financial, in RTG’s. They therefore decided together to revert to a set of criteria that were developed earlier in the OLRT1 project’s life, criteria that had been reviewed and rejected just a few months earlier by those responsible on both sides in favour of stricter criteria – and all of this was apparently done with limited analysis. [p 322]

This illustrates a major problem with project delivery whether it is via a P3 or more traditional means. The most basic question – does it work? – is at best a secondary concern when political imperatives take over, and they are compounded by the vendors desire to be paid. “Risk transfer” for politicians is just about impossible without a solid case to lay blame elsewhere, and the question remains “what did you know, and when did you know it”. For the private sector vendors, “risk transfer” is limited by how well they have padded their quotes, the amount of insurance that might be available, and their willingness to absorb losses. “Risk transfer” is a joke when the buyer colludes with the seller to change the acceptance criteria, and especially when this is done in secret.

The City, at least those in the know, were well aware of the problems.

“Manconi is reported to have told RTG and OLRT-C during the meeting on the evening of August 7 that even the days on which the system had passed (August 3–6) would be “horrendous” customer experiences.” [p 322]

Trial running continued from August 9 to 23, but with the 2017 criteria in effect from August 16 onward. The system managed a “pass” on each day from August 16, this was not with flying colours including a low AVKR and failures in maintenance metrics.

Although the target was supposed to be based on twelve “consecutive” days, the agreement allowed for “repeat” days and so the trial actually extended over fifteen days. The AVKR over this entire period was about 93.5%, well under the relaxed average of 96% counted for only 9 of 12 days. In effect, the trial running went for 15 days of which the best 9 were used to “achieve” the target.

“The Commission finds that the determination by the City and RTG that the system had passed trial running was only possible because the criteria were changed and discretion was applied for the purpose of achieving a pass. This conduct was not consistent with the goal of trial running, which was to demonstrate that the system could reliably perform as intended.” [p 323]

Maintenance During Trial Running

An important aspect of the trial running period was the ability of the contractor, RTM (with Alstom as subcontractor actually doing the work), to maintain the vehicles and provide a fleet at the availability level needed for service. Both the City and RTG were “concerned” about this, and yet maintenance was not a focus during trial running. [p 323]

Several other factors further increased the pressure on maintenance going into operations, including a growing maintenance workload, concurrent work and retrofits being done on the vehicles, a disorganized handover of the OLRT1 system, and an under-resourced maintenance team. [p 324]

The competence of maintenance staff was a particular concern. This highlights the problem of starting up operations on a new mode where nobody either on the owner’s side (the City and OC Transpo) nor on the vendor’s side (Alstom) has experience.

[Tom Prendergast, from the City’s expert consultants STV] evaluated RTM’s maintenance readiness as a 3 or 4 out of 10 in an email he sent to high-level OC Transpo staff on June 24, 2019, about a month before the start of trial running. He flagged several issues of particular concern, including lean maintenance resources; Alstom staff’s inexperience in performing inspection, maintenance, and troubleshooting functions; issues managing the MSF yard; and outstanding vehicle issues that would have to be resolved using space and resources in the MSF. [p 324]

Even without the difficulties imposed by overlapping work demands and by “lean” staffing, basic processes such as documentation of vehicle inspections continued. This was compounded by only selective and minimal review of work orders. The TRRT only reviewed five randomly selected work orders each day, and even this review “was more focused on the quality of the work order than on the quality of the actual maintenance activities performed in response to it.” [p 325]

The TRRT had discretion to override failing grades on even the small number of work orders they reviewed.

“The TRRT used that discretion often. For example, maintenance failures were logged on 13 of the 25 days of trial running. On 7 of those 13 days with maintenance failures, an overall “pass” was given by the TRRT. Further, 5 of those 7 days were part of the 12 consecutive days used to grant a “pass” on trial running.” [p 325]

Maintenance was an ongoing problem, and it gets its own chapter in the report which I will review as part of the next article on revenue service.

Decision Making and Transparency

The section of the report on Decision Making during the trial running period is extremely damning. It documents not only the withholding of critical information from Council, but also an attempt to make the OLRT1 project appear to be in much better condition than it actually was.

“Decision-making with respect to the OLRT1 project was delegated to the City Manager by the December 2012 Delegation of Authority. The City Manager was given broad authority to oversee the OLRT1 project, including the power to amend the Project Agreement, with a responsibility to report to Council and seek Council’s direction on significant matters.

“That delegation of authority broke down in the summer of 2019, as evidenced by the creation and operation of the WhatsApp Group. The select membership in the WhatsApp Group – including City Manager Kanellakos, the Mayor (through his staff) and Manconi – were the only senior executives or councillors within the City who had accurate information about trial running, and although Kanellakos denied that the Mayor and Hubley were participating in decision-making, Mayor Watson testified that the group of people on that WhatsApp chat were indeed making decisions about how to proceed, what information to share with Council, and when. Mayor Watson stated that the people on the WhatsApp Group were the “decision makers,” including with respect to deciding when to update Council.” [p 327]

Moreover, at the inquiry hearings, the City took a position that regular updates to Council were explicitly not to occur. This warped a commitment to notify Council of the completion of the Trial Running period into an alleged commitment to give no further updates until this actually happened.

The Inquiry Commissioner rejected this position which was quite clearly a rearguard action by the City to defend actions by its most senior management and the Mayor.

[Documents] of July 10 and July 22 […] state in clear terms that, “once RTG has achieved all Trial Running requirements, staff will inform Council.” Those words cannot reasonably be understood to mean that no updates would be provided for any reason during trial running. [p 332]

One must wonder about the pretzel-shaped logic used by the City and its legal team in advancing this argument and, indeed, just who that team thought they were representing.

“The very notion that Manconi and Kanellakos would make a commitment to withhold information that was vital for councillors to fulfill their statutory obligations is nonsensical and smacks of an obvious attempt to justify the wrongful withholding of information retroactively and dishonestly.” [p 21]

[…]

“The Commission finds that no commitment was made to not update Council during trial running. Instead, decision-making authority and information sharing were informally restricted to the people on the WhatsApp Group. The Commission further finds that Manconi revealed, in a moment of candour, the real reason the July 31, 2019 memorandum was not sent. He testified that if he released it, he feared the Council would “ask too many questions.” [p 22]

The existence of a small, privileged and secret group undermined the whole concept of project management especially because it included the Mayor who had a significant political stake in the project’s apparent success.

“The information sharing and commitment to keeping Council informed also broke down in mid-July 2019 with the creation of what will be referred to here as the WhatsApp Group. (Its members used WhatsApp, a messaging platform in which multiple people can exchange messages in a group via their smartphones.) This was a group of key personnel initially consisting of the Mayor’s Chief of Staff, Serge Arpin; Councillor Allan Hubley, Chair of the Ottawa Transit Commission; City Manager Kanellakos; John Manconi; and two other people in the Mayor’s office. (Mayor Jim Watson was later added to the group.)

“This select group of people was receiving real-time updates throughout the trial running period, and as a result, had knowledge of the events described below. In contrast, Council as a whole did not receive the information shared in the WhatsApp Group, and thus did not have the benefit of any meaningful insight into the trial running results.” [p 315]

[…]

“To be clear, the problem is not that senior City staff were sharing information with or updating the heads of relevant committees. That is undoubtedly a necessary part of a municipality’s operation. What is inappropriate is that there was a small group of people, the existence of which was unknown to others in the FEDCO and Council, sharing information privately (via the WhatsApp platform), and making decisions about not only what action to take but also what information to share with the rest of the FEDCO and Council, and when. The WhatsApp Group was an end run around proper governance. As Kanellakos noted in his evidence, if Mayor Watson wanted to participate in decision making with respect to the OLRT1 project, the only way for him to do so properly was through the appropriate channels at Council.” [p 329]

The report is quite blunt in describing information provided to Council as “inadequate and, in some cases, misleading.” [p 329]

Some members of Council were concerned that they were not receiving full information, but they were unaware of the back-channel via the WhatsApp group. Memos from the City Manager Kanellakos and John Manconi, the General Manager of the Transportation Services Department to Council did not explain what was really happening. Because this section of the report is critical, I will quote it directly at length rather than attempting a paraphrase.

The first relevant memo is dated July 27, 2019 – two days before trial running began. The July 27 memo advised Council only that trial running was about to start. It said nothing about the ongoing problems that were occurring in the immediately preceding days related to launching vehicles, and nothing about the experts’ ongoing statements that the OLRT1 system was not performing effectively and was not reliable. That information, which the Mayor himself testified was concerning and that he wanted and needed to have, was not shared with the rest of Council. [p 330]

At this point it is clear that even with a formal, public report, the intent was to hide information.

The next relevant event is Manconi’s draft memo of July 31, 2019. That memo, if sent, would have informed Council of the disastrous start to trial running and the need to use the “pause” and “restart” function that the Trial Running Test Procedure (the document that captured the 2019 criteria) described as reserved for “exceptional circumstances.” But the July 31 memo was never sent. It was suppressed by Kanellakos, allegedly because of the “commitment” made in the July 10 memo not to update Council until trial running was complete. [p 330]

That “commitment”, which was cited by the City as a justification for management’s actions, was a smokescreen, and again critical information was withheld from Council.

Council received another memo on August 7, which was framed as a regular quarterly update on the progress of the OLRT1 / Confederation Line project and OLRT Stage 2 projects. This memo was sent at a time when Kanellakos, Manconi, and Mayor Watson were alarmed at the OLRT1 system’s poor performance. By August 8, the system had been in trial running for 11 days and had only passed on 4 of those days. Slade told the Commission that, on that very day, Manconi apparently (according to Lauch’s email summary) said to OLRT-C representatives that even the “pass” days would be “horrendous” for customers. The Mayor was sufficiently concerned that he called Manconi into his office for an emergency briefing on August 8, and Manconi appears to have begun considering reverting to the 2017 criteria for trial running. Yet, the August 7 memo to Council did not disclose any of the performance issues that caused the “decision makers” to be so concerned. Instead, the memo noted only that trial running was ongoing and City staff anticipated a public launch in September 2019. [p 330]

This memo continued the deception by providing selective information about the trial and doing so out of context so that the situation appeared better than it was.

Council received another update memo on August 16. This was also an important date, because the OLRT1 system had performed dismally on August 15 and 16, and because August 16 was the date on which RTG confirmed the agreement with the City to revert to the 2017 trial running criteria. Manconi drafted a memo to Council advising that additional time would be required for trial running, the anticipated RSA date would be missed, and the City would levy another $1 million liquidated damages payment against RTG for that failure. Again, however, that draft memo was not sent.

Instead, a shorter memo was written and sent, and that memo gave the impression that everything was fine, in particular by stating that RTG had made “significant progress” toward completing trial running by “exercising the Confederation Line system at full functionality.” Council was told that this was a “complex and rigorous process where high performance standards must be met.” This memo was sent on the very day that RTG confirmed the agreement with the City to decrease the standards because RTG and the City understood that the OLRT1 system could not meet the higher 2019 standards. The August 16, 2019 memo was misleading by its multiple significant omissions. [pp 330-331]

The level of deception here is breathtaking. This is not just a case of keeping Council in the dark, but of feeding them patently “fake news”.

“The final trial running update to Council was sent on August 23, 2019. That memo, sent by Kanellakos, deliberately misled Council. Kanellakos reported that he was “pleased to inform the Mayor and Members of Council that the objectives of Trial Running and the requirements of the Project Agreement have been satisfied.” The memo stated that “the City of Ottawa established targets for the Trial Running period that were based on industry best practices,” including, for example, a requirement to achieve 96 percent AVKR on 9 of 12 days. He wrote that the City recognized that the TRRT required “additional tools” to deal with specific events and had thus provided the TRRT with options such as repeat days. All of this was entirely misleading. Kanellakos did not tell Council that the TRRT had been compelled in the early days to use the pause and restart “additional tool” that was described as being reserved for “exceptional circumstances.” He did not tell Council that repeat days had been required on several occasions. He did not tell Council that discretion had to be exercised to achieve a pass on several of the days, or of the difficulties that existed even on the days that were marked a pass.

“The memo also misled Council about the change in criteria midway through trial running. The memo stated that “RTG … wanted to not only meet these targets [i.e., 96% on 9 of 12 days] but exceed them. RTG targeted a figure of 98% for service availability and wanted to assess if they could reach 98% for the entire twelve (12) day period.” That statement was inconsistent with the reality, which was that the parties had agreed on the 98 percent for 12 consecutive days standard after finding that the 2017 criteria should not be used, and RTG and the City then agreed to revert to the 2017 criteria of 96 percent on 9 of 12 days only because the system could not otherwise pass.

“The memo then stated that the system had achieved an average AVKR of 97 percent over the 12 days of trial running. It did not tell Council that the average over all days during which trial running occurred was approximately 93.5 percent.

“There can be no doubt that Kanellakos knew the memo was misleading. Kanellakos himself conceded during his testimony that in certain respects the memo was “inconsistent with the reality.” This is shocking conduct, which constitutes deliberate wrongdoing, by the most senior public servant employed by the City. [p 331]

The report puts this succinctly in the Executive Summary:

“The Commission finds that the August 23, 2019 memorandum from Kanellakos did not seek to provide information; it sought to disseminate misinformation and hide critical facts from Council so that councillors could not properly exercise their oversight function. The inescapable conclusion is that Kanellakos deliberately misled Council.” [p 22]

Bad enough that the City Manager was “misleading” Council (a delightfully parliamentary expression), but many other people must have known that this memo was both false and designed to mislead. Some of them are likely still working in Ottawa on the expansion of the LRT network.

Coming Next

In Part Six of this series, I will take the story through the OLRT1 acceptance and opening, and the severe problems that afflicted the line during revenue service.