Updated December 14, 2012 at 1:40pm:

Additional information regarding Presto and Metrolinx’ response to the Auditor General’s report has been added at the end of that section in this article.

Original post from December 13:

The Auditor General of Ontario released his Annual Report on December 12, 2012, and it includes a section on Metrolinx.  For those of us who have wrestled with the secrecy of Metrolinx, some of the information and recommendations in this report are a breath of fresh air.

Metrolinx’ overall reaction to the report is much of the same boilerplate about the wonderful job they are doing and how important they are to the region.  In some cases, Metrolinx dodges the questions raised by the Auditor in a way familiar to anyone who has ever attended one of their press conferences.

The report should be read in the context of March 31, 2012, the end of the period to which the audit applies.

Background

The report begins with an overview of Metrolinx, its plans and its funding including a table of the status of various projects as of March 31, 2012.  This is followed by a summary of the Auditor General’s findings.  [Pages 205-208]  The Metrolinx overall response follows [Page 209] and includes the astonishing statement that:

As a regional fare card serving many transit providers, Presto is a unique product and one of the world’s most advanced fare-card systems (similar to London’s Oyster card, The Netherlands’ OV-chipkaart and the Chicago Card).

I will take it on faith that as and when Presto Next Generation proves itself, it will have a robust set of functions and may be spoken of in the same league as Oyster, but Presto as it now exists is in a technological backwater.  We will see more of Presto later, but this is an example of Metrolinx trumpeting things as they may become, not as they are today.

Best practices for regional agencies include [Page 210]:

• seamless, integrated, user-oriented systems;
• fair, clear and rigourous benefits-case analysis to ensure optimal decision making;
• risk management, cost-effective and on-time project delivery;
• sufficient consultation with key stakeholders;
• clear targets and progress reports on meeting plan objectives.

The Auditor General observes:

In reviewing several of the major priority transit projects contemplated within the RTP’s first 15 years and in discussion with GTHA municipalities and transit agencies, we noted that Metrolinx has encountered challenges in successfully implementing some of these practices.

This could be read two ways.  Projects may run aground because they are mismanaged, but also because they encounter unexpected difficulties including political interference or unreasonable expectations.  Both conditions apply in various ways.

In the sections below, I will look at each major area addressed by the Auditor General, but will not attempt to duplicate all of the information in his report.  Readers are urged to browse the full text themselves.

Air Rail Link (aka Union Pearson Express)

The ARL has a projected cost of $456-million although this probably does not include costs that have been included in the Georgetown South project to upgrade the corridor from Union to the Pearson Spur (itself a $1.5-billion project).  Although not mentioned by the Auditor General, the Georgetown South project was begun under GO Transit when it was a separate agency operating under MTO rather than as a Metrolinx division.  Some of the project’s history is outside of Metrolinx per se, but as GO was brought in more or less intact.

The Auditor General’s concern lies with the business model for the ARL and the improbability that it will be able to break even especially if its capital cost is included in the business model.  The sad history of the private sector’s abandonment of this scheme is explained:

In 2003, Transport Canada announced a private-sector group as the successful Public–Private Partnership (P3) proponent that would design, build and operate the ARL. However, the group was unable to secure financing for the venture because its lenders did not feel that they had sufficient protection from “no market” risk (that is, from a situation where, despite all reasonable efforts to attract riders, the service does not generate enough revenues to be a viable business). They perceived this project to be riskier than other infrastructure projects because there was no “pre-existing demonstrated revenue stream.” The group proposed that the province assume the lenders’ risk by purchasing ARL assets if the “no market” scenario arose. The province rejected this proposal, so the group walked away from the project. In 2010, the government decided that the province, through Metrolinx, would build and operate the ARL itself.  [Page 211]

Metrolinx’ own estimate for the annual operating cost of the ARL is $30m, and this may not include all costs as the full nature of the service offering is not yet defined (e.g. station and onboard amenities and staff).  A further $20m would be required to pay down the capital cost.  The Auditor General estimates that based on projected ridership, the break-even fare (including capital costs) would be $28 one way from Union to Pearson.

However, research done by Metrolinx (information curiously absent from the public record until now) shows that at this fare, a large proportion of potential riders would not use the service.  Metrolinx believes that it can achieve its ridership targets, although their response tells us almost nothing beyond the fact they don’t really know what their business model will look like when the service begins operating.

As the ARL launch approaches, a number of significant decisions need to be made. Metrolinx will continue to use best-in-class ridership information to guide our internal decision-making and to inform our business model, and we will continue working with the Ministry of Transportation to finalize the business model.
As we would with any new service, Metrolinx will closely monitor the ARL over its first years of operation and make adjustments based on customer feedback we receive.  [Page 213]

This comment echoes a response I received to a recent query at a press scrum when I asked about a business case analysis for the Downtown Relief Line (DRL).  Metrolinx thought I was asking about the “ARL” and replied as below:

As the Union Pearson Express launch approaches, a number of significant decisions need to be made and we will continue to engage in detailed planning in order to inform and finalize our business model.

And as we would with any new service, Metrolinx will closely monitor the Union Pearson Express over its first years of operation and make adjustments.  More information will be made available at a later date.

The Spur Line at Pearson Airport is a separate construction project that was awarded as an AFP through Infrastructure Ontario.  The process of evaluating this mechanism as opposed to traditional project management by the project owner includes a considerable element of estimating the value of the risk transferred to the private sector builder.  Indeed, this value is essential to the claim that AFP is “cheaper”.  The Auditor General was troubled by the process of making these estimates and the fact that the same people who will benefit from the contract are also part of the evaluation.  He also commented that the degree of risk should be based on comparable transit construction projects, not on the type of work more commonly engaged in by Infrastructure Ontario.

IO’s response essentially was “we know what we’re doing”, and there is no way to determine how this work out until they have actually run a fair number of transit jobs and established a track record.  The Auditor General noted that one risk that is not included in the evaluation is created by the AFP process itself.  Because the specification is locked down earlier in the process than would be the case with a Metrolinx-managed project, there is a greater danger of scope creep relative to the contracted work.

One point not mentioned is that the “risk” is hypothetical.  It may or may not actually be incurred, and the actual risk transferred may not cover all of the possibilities for cost overruns.

I would not be surprised to see Metrolinx talk of cost recovery purely on the operating budget, not including capital, treating the investment as, in effect, a gift from the Province to air travellers (much as capital costs of the commuter rail network are not recovered through fares).  However, elsewhere in the discussion of the high cost of Presto’s implementation, Metrolinx expects to recover some of this expense through chargebacks to the local transit agencies.  Their accounting practices flex depending on their goals.

Presto Farecard System

The Auditor General begins with a background history of Presto that I will not repeat here.  The important points worth noting are that the original goals were for the system to use off-the-shelf technology wherever possible and to be extensible to all GTAH transit providers.  Research began in 2002, and in 2006 MTO entered into a 10-year contract for design, development and operation of Presto.

To put this in an industry context, Hong Kong’s Octopus card was developed and tested from 1993 to 1997 .  London’s Oyster card was developed from 1998 to 2003.  It is unclear why MTO chose to develop its own system when the fare card industry was well-established, but this followed an Ontario tradition of building proprietary systems.

The original 2006 contract for the base system was for $149-million in capital costs and $82.5m for system operation.

In practice, Presto was not suitable for the complex fare structure of the TTC nor for the evolving business market where proprietary cards will start disappearing to be replaced by bank cards and smart phone applications.  Requirements from both Ottawa and Toronto led to a “Presto Next Generation” (PNG) design which is now rolling out in Ottawa and will appear in Toronto starting with the new streetcars.  PNG’s development cost will be $498-million, and the contractor will also be paid $152m for operating the system until 2016.

Although the TTC has been working with Metrolinx on Presto requirements since 2009, they also pursued an alternative strategy under the Miller/Giambrone administration at City Hall.  This would have seen a more open, flexible system implemented initially at no cost to the TTC with the risk borne by a private sector partner.  However, that approach fell victim to the combined effects of government change in Toronto and strong-arm practices by Queen’s Park who threatened to withhold all transit funding if Toronto didn’t sign on to Presto [Page 220].  That formal agreement was concluded recently, although its details have not yet been made public.

Metrolinx opted to develop PNG through change orders to its base contract rather than through a new tender.  CEO Bruce McCuaig defended this decision in a CBC radio interview saying that continuity of the systems was essential.  This is a common problem with proprietary technology — once you select a product, you are stuck with its vendor unless there are overwhelming reasons of cost or functionality to justify a change.  (In a way, this is not unlike the TTC’s long-standing love for cash, tickets and tokens, just updated by a century or so.)

An external review of the decision to stay with the original vendor concluded that if a value-for-money (VFM) review were positive, then there was no “compelling reason” to retender.  [Pages 219-220]  The VFM considered the cost of Presto based on change orders totalling $227m and compared this on a per capita basis with other major systems.

However, if the capital portion of the additional $496 million expected to be incurred had been included in the analysis, Presto base and PNG combined would turn out to be one of the more expensive fare-card systems in the world.  [Page 220]

Metrolinx’ Board was not happy with the level of Presto spending, and asked for a new VFM review.

In April 2012, citing concerns about the request for an additional $496 million in spending authority for PNG, the Board asked Metrolinx’s management to carry out further due diligence on PNG. In response, Metrolinx asked the same consulting firm for a second VFM review of PNG. This second review concluded that, although reprocurement “may drive pricing benefits,” it incurred a significant risk of not meeting the timelines for the development of a new fare card, because procuring, developing and implementing a new system would take more than 24 to 48 months. Furthermore, having a new vendor build the system would introduce significant other project and operational risks that could hinder the
efficient delivery of the entire system. [Page 220]

The Auditor General goes on to observe that the 48-month timeframe was what Presto got anyhow given that PNG development started in 2009 and the system will not go live in Ottawa until 2013.

Metrolinx’ response indicates that they intend to reduce the role of the existing contractor (Accenture) and do more competitive tendering for elements of the fare system.  Moreover, they hope to recoup part of the development costs ($275M) by backcharges to the systems using PNG, notably Toronto and Ottawa.  In effect, the cost of a poor business decision gets passed on to the local system operators whether they want it or not.  [Page 221]

Change order management comes in for criticism by the Auditor General with a lack of clear documentation of whether changes should be no-charge fixes to a system that does not perform to specification, or enhancements that are billable.  Moreover, there is no cost estimate mechanism for changes.  Metrolinx says that they have determined that only enhancements were billed for, but also that they will improve their change order procedures in the future.

Further Presto issues include the system’s ownership and performance.  When the Accenture contract ends in 2016, major portions of the system including the intellectual property would revert to the contractor, a classic problem in technology projects (and one that turned up in other major smart card rollouts).  This problem has now been resolved so that Metrolinx will retain rights and access to their system even if Accenture is out of the picture.  On the performance side, the system (and its controlling contract) include many metrics of performance, but the Auditor General reports that Metrolinx has never held the contractor accountable or imposed penalties allowed in the contract.

The larger issue Metrolinx will have to face in the next decade is the evolution of payment systems from proprietary cards such as Presto to open systems including bank cards and smart phone applications.  Unlike Hong Kong’s Octopus card which developed early (before standards for such things even existed) and became the de facto electronic purse for a wide variety of uses in HK, Presto would be competing with a well-established banking industry for payment systems.  Metrolinx will have to decide whether to move away from Presto as it now exists to embrace a wider selection of technologies.

Originally, Metrolinx had suggested that features such as loyalty discounts would only be made available to Presto card users, but such an approach is impractical given the way payment technology is developing.  They appear to have abandoned this position and a recent report to the Metrolinx Board implies a move to bank card and smart phone support in coming years.

Updated December 14, 2012:

The Star reports that two regional operators, Mississauga Transit and York Region Transit, have come to the defense of Presto saying that the $700m investment will be worthwhile once the TTC is included in the system.  There remain, however, barriers to Presto use by some riders especially those who do not have ongoing banking arrangements that would be used to support transit fares.

Metrolinx CEO Bruce McCuaig defended the Presto system saying that a Value for Money review showed it had a relatively low cost of implementation per rider at full rollout (i.e. including the TTC).  However, the VFM review (dated December 2011 on its cover, but posted by Metrolinx with a March 2012 date) only includes a chart of capital cost per capita [Page 22], does not comment on operating costs, and is based on a lower capital investment in Presto than what will actually occur.  Some of these points were also made by the Auditor General in his review.

A second VFM which was requested by the Metrolinx Board in April 2012 according to the Auditor General and some Presto reports are dated more recently implying that they respond to the April request.  [The reports are listed as “Appendix A” and “Appendix B” on the website, but they are dated August 2012 and should not be confused with the appendices of the same name in the original 2011 report.]  These reports add little to the original review.  In particular, the derivation cost of 14¢ per ride cited by McCuaig in the Star is not described in the reports.  It is unclear whether this refers only to capital or also to operating costs.

The new Appendix A is a VFM for the original Presto system, and it concludes that an integrated region-wide system was clearly preferable to individual municipal system.  This is no surprise considering the level of investment the smaller 905 operators would have to make relative to their ridership and the complexity of interoperating multiple systems.

The obvious issue is that only the TTC, the largest operator by far in the region, would have the economy of scale to implement a farecard system on which others could piggyback.  In effect, the scale of the TTC as a potential component of Presto, would make a “regional” system much more competitive in providing services for the smaller operators.  However, Presto Version 1 did not have the functionality needed for a large operation nor for the TTC’s business requirements, and the TTC was not included.

The new Appendix B is a VFM for Presto Next Generation.  This repeats much of the analysis in the original report supporting a decision to enhance Presto rather than moving to a new technology provider.  The Pan Am Games and a desire for region-wide fare capability were strong drivers for this choice, along with the concern that Metrolinx might somehow lose reputation by changing vendors.  The latter consideration is rather odd in light of the history of arrangements with private providers.

If some else can do the job better, you wave goodbye to the original provider.  This is called “competition” and, provided that the headaches of migration from an old to new provider are worth the upheaval, then this is simply a good business decision.  Presto lost the chance to make the change before its largest rollouts — Ottawa and Toronto — and in the future will find migration almost impossible because of the scale of locked in infrastructure and users.

Also included is a table comparing the capital cost and functionality of various fare card systems.

Union Station Project

The Union Station reconstruction project is jointly undertaken by GO Transit and the City of Toronto.  The train shed itself lies in GO’s jurisdiction together with the contract to rebuild/replace the shed.  The Auditor General reports that this project may come in 25% above the original estimate and notes that work done under the “contingency” provision (which itself has been expanded) included items that were originally part of the base contract.  This shell game allowed the contract to stay “on budget” by moving work from the original agreement to an add-on.

This complex project is a good example of one with high risk of changes through discovery of unexpected site conditions and delays.  Metrolinx chose to retain the risk for itself rather than going with a fixed-price contract to ensure that bidders were not frightened off by potential risks to their own profit margins.  This shows that the idea of risk transfer as advocated by Infrastructure Ontario may not always prove beneficial.

The Auditor General observes:

Significant price changes in contracts can occur because of poor planning, inadequate processes for estimating the initial cost projections, weak monitoring of the project or a combination of these problems. In 2011, the province’s Internal Audit Division reviewed Metrolinx’s budgeting and forecasting process and found that the capital budgeting and forecasting processes were not well established, and also that recent years’ budget-to-actual results suggest that Metrolinx may need to re-evaluate how project costs and/or contingencies are determined.  [Page 224]

Infrastructure Ontario itself has argued that its processes drive better rigour in project specifications by forcing more up-front planning and decision making.  Whether this is practical for contracts with significant unknowns is hard to say, and we will see how successful their AFP approach is as the transit projects roll out.

In the Union Station Rail Corridor (USRC), the project to replace and restructure the very large number of double slip switches completed recently.  These switches allow trains to move easily between tracks without having dedicated crossovers for each possible move.  The Auditor General comments that the original contract with Toronto Terminals Railway for this work was $37.6m in 2006, but that through scope creep, regulatory and staffing changes, the cost grew to $89.6m.  The Auditor General was also concerned about the adequacy of details in records for this project and in invoices for work performed by TTR in general.

Metrolinx plans to change the way it procures such work including moving some of the expertise in house given its large and growing ownership of the GTAH rail  network.

Regional Plan

The Auditor General observed:

In order to effectively carry out [its] mandate, Metrolinx’s decisions regarding transportation and transit planning must be made on the basis of a credible business case supported by objective and sound data.

…

In the recent debate over the City of Toronto’s
transit projects within the RTP, Metrolinx could
have been perceived as not being a strong enough
advocate of what its own analysis suggested was
the right course of action for these projects.

…

Metrolinx’s analyses concluded that the most cost-effective strategy was a mix of light rail with traffic on two of the lines (Sheppard and Finch) and a fully grade-separated rail system on Eglinton Avenue. However, the Eglinton project was approved only as a partially grade-separated project, because there wasn’t enough provincial funding for a fully grade-separated system. [Page 226]

The Auditor General goes on to describe the purported Memorandum of Understanding (MOU) with the City of Toronto that completely revised previously agreed-to priorities, and which went against some of Metrolinx’ previous business case analyses.  The inability of Metrolinx to stick to a consistent position in a politically charged environment is only hinted at, but the Auditor General urges that decisions be made on the basis of analysis.

This is all very well, but it ignores the fact that Metrolinx operates in a political environment as do all public agencies.  If anything, the spinelessness can be traced to the government itself who, terrified of “Ford Nation”, were unwilling to take on the newly-elected Mayor’s bluster.

Moreover, the “analysis” done on various transit proposals is often coloured by assumptions in the methodology and in what proponents of a project consider “good” or “bad”.  If spending a lot of money for job creation is “good”, then this favours expensive options that are further justified by talk of “protecting for future growth”.  If providing mobility and encouraging moderate levels of development are the goals, then spending every available penny on a few transit lines is counterproductive.

The Auditor General reports that the Metrolinx analysis of the Eglinton corridor shows that a totally grade-separated line would be the best choice.  However, this analysis has never been published and is not available among the Benefits Case Analyses that are online.  This is an example of how an unpublished report can inform internal policies and directions at Metrolinx, but without any opportunity for public review or comment.

It is difficult to trust in the appropriateness of business decisions when the information used to make them is withheld from the public.

Funding is a major issue for the regional plan as I have discussed in many previous articles.  The lack of leadership from Queen’s Park which leaves Metrolinx hovering around the edges of discussions organized by others (such as Civic Action) means that we have no sense of when or if new revenues may flow, nor of official advocacy that they are even needed.  It is not even clear how much of the plan will be funded from the pending Investment Strategy and how much from local and/or federal contributions.

Project Management Information System

The project management software used by Metrolinx has severe shortcomings that require managers to keep separate spreadsheets to provide the level of detailed tracking they need.  Metrolinx plans improvements, but it is unclear whether these are band-aids on an existing flawed system, or a new suite of tools providing the required functionality.

Recommendations of the Auditor (paraphrased and condensed)

• Metrolinx and the Ministry of Transportation (MTO) need to define the business model for the Air Rail Link (now known as the Union Pearson Express), and Metrolinx should ensure that its ridership projections are valid.
• The Alternate Financing and Procurement (AFP) model used by Metrolinx and Infrastructure Ontario (IO) depends strongly on estimation of savings through risk transfer to the private sector.  These estimates should be based on actual experience with comparable transit projects.
• Metrolinx should consider the risks involved in pursuing development of Presto Next Generation (PNG) [as opposed to sourcing a new vendor for a smart card system].
• Metrolinx should work with municipal governments and the Province to resolve the problem of fare integration across the GTAH, and to resolve issues that inhibit uptake of Presto by riders on municipal systems.
• The Metrolinx procurement process should ensure that value-for-money and business-case justifications are approved by the Board and by MTO before significant projects are finalized.
• Change orders for the Presto system should clearly identify whether they apply to deficiencies in the system or to new functionality with only the latter being chargeable by the contractor, and internal cost estimates should be prepared as a check against the reasonableness of actual charges.
• Metrolinx should ensure that ownership of the Presto system components and intellectual property is not lost when the current contract expires in 2016.  If performance specifications in the contract are not met, Metrolinx should have valid justification if specified penalties are not levied.
• Metrolinx should ensure that contracts have firm ceiling prices wherever possible, and contracts should be monitored for compliance.  In the Union Station Rail Corridor (USRC), Metrolinx should seek other qualified suppliers or inhouse provision of services [now performed by the Toronto Terminals Railway under contract].
• All projects in the regional plan should be subject to rigourous cost-benefit analysis, and decisions should be made on the basis of that analysis.
• Metrolinx should regularly consult with municipalities and other stakeholders, have clearly defined targets for major projects, and regularly report on project costs and progress.
• Metrolinx should ensure that its project management information system has the functionality necessary for effective project monitoring.

Many of these recommendations arise from circumstances or practices that suggest a less than well-managed organization and, in some cases, one which is driven more by political considerations than by good “business” practices.  Metrolinx accepts some, but not all, of the Auditor General’s criticism, and it is difficult to see just how much this report will change Metrolinx’ behaviour.