At the TTC Board meeting of October 24, 2019, Toronto’s Auditor General, Beverly Romeo-Beehler presented a detailed report on the problems faced by the TTC with the Presto farecard system.

• Report: Toronto Transit Commission’s Revenue Operations: Phase Two – PRESTO/TTC Fare Equipment and PRESTO Revenue
• Video: Toronto Auditor General’s Review of PRESTO and TTC Fare Equipment and PRESTO Revenue
• Auditor General’s Presentation: Review of TTC Revenue Operations: Phase Two
• Metrolinx Letter to the Auditor General

For clarity: In this article, where text is directly quoted with an indented block and a page citation, these are the words of the Auditor General’s report. Where there is emphasis within a quoted section, this is taken from the original report, not added by me. Otherwise, comments here are my own with, in some cases, paraphrases of the report. Page numbers cited are within the linked PDF of the report, not necessarily the page numbers shown in the document because numbering restarts within it.

This is a long article, but nowhere as long as the 123 page report on which it draws. It begins with an overview and then continues with detailed reviews of various aspects of Presto and of the relationship between Metrolinx and the TTC. The sequence is slightly different from the Auditor General’s report in order to group related sections and comments together.

After the Introduction, the Audit Findings section delves into the detail. There is a lot of detail because there is a lot wrong with Presto. The Auditor General’s report is an indictment of a failed project and bad management primarily from Metrolinx, but the TTC does not go unscathed.

If anything is missing, it is a historical review of how the system came to be in its present state. However, that is beyond the remit of an investigation into whether the TTC is getting what it pays for from Presto, and whether shortcomings affect the revenue the TTC should receive through that system.

TTC management has accepted all of the Auditor General’s findings, and their remarks can be found in Appendix 1 beginning on page 110 of the report pdf.

Summary: TTC accepts all of the Auditor General’s recommendations. Most of the recommendations are consistent with TTC management and actions to date and ongoing efforts to address and resolve the issues as identified in the recommendations.

Metrolinx’ response to the findings are generally positive and acknowledge many shortcomings including missing or broken processes that should be in place by both parties. How well this will be sorted out remains to be seen. Probably the most important part of the letter is the view forward to the next iteration of the Presto system, and the recognition that what we have now can be improved for everyone’s benefit.

As you know, PRESTO is embarking on modernization plans that are enabling us to focus even more on delivering exceptional services. This work will result in new offerings that will remove customer pain points and give them new ways to pay, while offering transit agencies valuable features to drive ridership and fare revenue. Consultations with the transit agencies on new payment forms and other improvements included in the roadmap have been underway throughout the summer, including with TTC, and we look forward to their participation in shaping the future of PRESTO.

We are also preparing to retender our current supplier agreement with a view to opening PRESTO’s ecosystem to more market participants, which we will leverage to improve our overall performance. Recommendations stemming from your audit of the TTC will inform our future approach as we go to market for these services.

As with all reports from the Auditor General, there will be a follow-up in a year’s time to report on the progress achieved on her 34 recommendations.

Introduction

This is “Phase Two” of a detailed study of TTC fare collection that began with a review of fare evasion. I reported on this in February 2019. See Fare Evasion on the TTC: The Auditor General’s Report. That review raised questions about the unreliability of Presto equipment as a source of revenue loss and frustration for riders who could not always pay their fare properly.

A central issue here is the length of time Presto has been in place and the number of unresolved or even unacknowledged issues surrounding the system.

In November 2012, TTC contracted with Metrolinx to integrate and operate the PRESTO fare card on its transit network for 15 years, plus options for renewal. As of the end of 2016, PRESTO could be accepted for fare payment across the entire TTC network.

The Master E-Fare Collection Outsourcing Agreement (Master Agreement) stipulates that Metrolinx manage all PRESTO equipment on TTC’s properties, on board TTC vehicles, and where necessary on the street. This includes the design of the hardware and software, and the installation and maintenance of the machines. In return, Metrolinx is compensated with 5.25 per cent, inclusive of HST, of the gross revenue collected through the PRESTO system on TTC. [pp 33-34]

The Auditor General’s office conducted a field study of Presto equipment, looking at real machines on real buses.

In reviewing the functionality of PRESTO fare equipment, we conducted a bus device audit for two days in June 2019. Over 100 TTC operators representing all seven bus garages drove 168 buses and noted any PRESTO issues during their shift. Of the 330 PRESTO issues noted, nearly 300 (91 per cent) of them were frozen PRESTO card readers.

A frozen PRESTO card reader is when a passenger taps their PRESTO card but the reader is stuck and does not always accept the tap. Not only does this result in revenue loss if the passenger doesn’t tap on another reader successfully, but the device may be captured as “in-service” rather than “out-of-service” in the availability calculation and the availability rate could be overstated for this issue. [p 24]

They also dove into the murky world of contracts between the TTC, Presto and vendors who provide services within the Presto system. The results were not pretty, certainly not with the rosy confidence we often hear from Metrolinx/Presto spokespeople making claims of a robust, near-perfect system. Moreover, there is a wide gap between the services actually provided by Presto and those for which the TTC has contracted and is paying through its service fees.

There is a very long list of findings from the audit, and it reveals a complex web of ownership and responsibilities for provision, operation, monitoring and maintenance of the Presto system. One cannot help feeling that these arrangements grew like an unweeded garden where functions were cobbled together on an ad hoc basis to provide new or extended features. If Metrolinx has any desire to consolidate all of this into a new iteration of Presto (we already have “Presto Next Generation”, and so a new name is really required here), this will not be a simple process. Moreover, as noted by the Auditor General, both Metrolinx and its client transit systems must have a clear set of requirements for any new system rather than taking whatever Metrolinx management thinks they will accept willingly or not.

The appalling status of the entire contract is summarized in one paragraph in which we learn:

PRESTO card readers need to be available greater than 99.99 per cent as per the service level identified in the Master Agreement between TTC and Metrolinx. Given SLAs [Service Level Agreements] have not been set up, what goes into the calculation of the service level has not yet been defined or agreed upon. Metrolinx staff have advised us that “they do not have an agreement with TTC on the device service level commitment calculations, nor the consequences for non-performance”. [p 25]

One might reasonably ask why an SLA is not in place after all these years including a penalty mechanism, but given that it is Metrolinx who benefits from this situation it is hard not to assume that they simply do not want to take the responsibility and potential, publicly identified cost that might go with it.

There is more than a little irony that a report exposing bad project management and a difficult relationship between the TTC and Metrolinx should appear just before City Council was asked to approve a much more complex arrangement for the funding, expansion and operation of the rapid transit system.

The greatest problem between the two parties is the political aversion at the City to “rocking the boat”. The Provincial attitude is that they can more or less act as they please because they have the upper hand on funding and political control. The Liberal government threatened to withdraw Gas Tax funding from the TTC if it did not implement Presto, and the Conservative government was on the verge of stripping responsibility for at least the subway network, if not the entire transit system, from the City. This does not make for a level playing field nor honest, transparent public debates.

Quite bluntly, if Metrolinx were not a government agency with the power to do as it pleases, but instead a private vendor, it would have lost this contract years ago and faced claims for non-performance.

The first recommendation is the need for “Foresight” by both the TTC and Metrolinx. The contract has many unfulfilled requirements, but what does the TTC really want its fare system to look like? What are Metrolinx’ goals for Presto as it evolves? Are the various suppliers of parts of the Presto system delivering what Metrolinx and its clients really need?

Overall, and in our view, there must be a strategic refocussing at the top by both TTC and Metrolinx to tackle what matters most – the shared outcomes of customer experience and maximizing revenue.

For this to work, both parties also need to:

• Define clear, agreed upon, and formalized outcomes and Service Level Agreement (SLA) targets
• Seek a win/win for both parties, but acknowledge individual and shared accountabilities and responsibilities in this arrangement – several examples of which are outlined in this report. [p 2]

The second recommendation is the need for “Insight”.

To solve problems you need insight into the root cause. To gain such insight, the right level of information must be analyzed using the right data. Without that, you are solving what you think might be wrong without the evidentiary support to confirm you are addressing the true cause(s) and actual issue(s). [p 2]

One might cynically observe that this statement applies to a lot of what passes for “analysis” and “planning” for transit in Toronto, but that is entirely another article.

The Auditor General noted “fundamental” information gaps:

• Service level agreements are not in place setting expectations and responsibilities for each party seven years after the contract was signed.
• Key information that the TTC needs is either encrypted or purged in a short time-frame contrary to the Master Agreement.
• The current analysis was limited by
  • problems with Presto readers including root cause analysis for frozen machines, problems with the device monitoring tool;
  • accurate monitoring of vending machines on streetcars and regular collection of coins from them to prevent out of service conditions;
  • manual practices for identification of out of service fare gates. [adapted and condensed from pp 2-3]

The third recommendation is the need for “Oversight”. Astoundingly, although the agreement provides for a Joint Executive Committee and an Expert Panel, these either do not exist or have not met for an extended time. One cannot help feeling that the TTC, in the face of an intransigent service provider, has simply given up and treats Presto as “broken as designed”. However, this could also mask resentment of a system forced onto the TTC compounded by little municipal political support to hold a provincial agency publicly to account.

Moreover …

… there needs to be the following to improve oversight of the system as a whole:

• TTC, PRESTO and all vendors working together as one, sharing information, and diagnosing and solving problems together (e.g. coin collection issue on new streetcar vending machines needs to be resolved together despite all vendors staying within defined responsibilities)
• Focusing, measuring and monitoring the desired outcomes
• Controls over PRESTO revenue and assurance provided by PRESTO needs strengthening, including retailer network controls [p 3]

There are 34 recommendations in the report, and I leave it to dedicated readers to peruse the full list.

Looking back to Phase 1 of the study and the ongoing dispute between the TTC and Metrolinx over revenue losses due to Presto, the Auditor General concludes:

We prepared calculations to estimate a range for the overstatement of the PRESTO card reader availability rate and annual revenue loss. Based on the work performed with the information we could obtain, it is our view that TTC’s estimate of $3.4 million in revenue loss for 2018 due to malfunctioning PRESTO fare equipment does not appear to be overstated. TTC’s availability estimates may even be understated given the issues we identified in this report with availability of PRESTO card readers. We are not including revenue loss calculations in this report. It is our view that the information and data gaps identified at this time make it difficult to provide these important numbers with the required level of audit assurance. [p 9]

The two agencies are now going to arbitration over this and other issues, and it will be intriguing to see what effect the Auditor General’s report has on that process. A major problem for quite some time has been that there has been little field work to document how the system is actually behaving or to explore reasons behind the perceived unreliability of equipment. That neither the TTC nor Metrolinx undertook a definitive review says a lot about both organizations, but at least we now have the Audtor General’s work to show the kind of information that is available if only one makes the effort to collect it.

The high level summary of the audit occupies one page [click to enlarge].

Audit Findings

In each section below, there are “Roles and Responsibilities” and a “Checklist of Issues” taken from the Auditor General’s report. This gives a high level view of the players involved in each device, and the variation among devices and processes. Each item in the checklist is keyed to a finding in the report with a reference like this: (A.1.1). My own comments and excerpts from the findings follow the checklist in each section.

I have included the entire lists here so that readers can see just how many problems the TTC has with Presto. Further details for selected problems are included after each list, but the full text is much longer and documents many problems with inconsistent reporting, analysis and data availability.

These are not a few annoyances to be smoothed over, but rather pervasive mismanagement of a major part of our public transit system.

Presto Card Readers [pp 19-20]

Roles and Responsibilities:

• Devices are owned by PRESTO
• PRESTO bought them from Scheidt & Bachmann (S&B)
• Maintenance is done by PRESTO
• Monitoring of out-of-service instances are done by PRESTO’s vendor’s offshore team
• Revenue transactions are recorded in PDS subsystem
• Out-of-service instances are recorded in the device monitoring software tool

Availability Calculation of PRESTO Card Readers (Buses, Streetcars):

• How it is calculated: Using device out-of-service statuses in the device monitoring software tool
• Who calculates the availability? Accenture
• Availability Calculation Issues Include:
  • Availability calculation is only provided for Monday to Friday between 6 AM and 10 PM (A.1.1)
  • Frozen card readers may be captured as “in-service” rather than “out-of-service” in the availability calculation (A.1.1)
  • Vehicles that are in-service but not recorded in NextBus GPS application system (due to a number of different reasons) are excluded from the availability calculation (A.1.2)
  • Vehicles that are improperly included in TTC’s maintenance list and are actually in-service are excluded from the availability calculation (A.1.2)
  • Not all out-of-service device statuses occurring between the 15 minute pings are captured in the availability calculation (A.1.3)
  • Due to issues with the device monitoring software tool during our audit, some devices were not included in the availability calculation and some devices were captured with an incorrect status such as “in-service” rather than “out-of-service” (A.1.4)
  • PRESTO’s vendor appears to be able to make adjustments to the device statuses for the availability calculation. The analysis and support for the weekly rate, including any adjustments made, is not provided to TTC (A.1.4)
  • TTC does not get a daily availability calculation spreadsheet for holidays in India/Canada and weekends in Canada (Note: Daily availability spreadsheet is now provided to TTC for holidays in India) (A.1.5)
  • For daily spreadsheet not provided during holidays in India, PRESTO staff were not aware of the issue and back-up could not be provided to confirm that these days were included in the weekly rate (data purged after 60 days, when it is required to be kept for 7 years per contract) (A.1.5)
• Incident Management Issues Include:
  • Monitoring for out-of-service instances is done by PRESTO’s vendor’s offshore monitoring team 24/7
  • Monitoring team did not always open a service ticket in the incident management system (called ServiceNow) for out-of-service devices (B.1.1)
  • Device monitoring software tool used by PRESTO’s vendor does not have reporting/extracting capability available for the TTC and the data in the device monitoring software tool is purged after 60 days (B.1.2)
  • PRESTO does not maintain a running log of swapped devices, contrary to the Master Agreement (B.1.3)
  • TTC staff need to improve the accuracy of the bus maintenance list provided to PRESTO (B.1.4)
  • TTC staff did not always report and raise service tickets in PRESTO’s incident management system for malfunctioning devices (B.1.5)

What Does “Availability” Mean to Metrolinx and to Riders

Before we even get into the details, there is not even a single definition of “availability” for card readers.

Metrolinx has two definitions of availability in its consolidated device dashboard document – when this audit report refers to availability, we are referring to card reader availability:

1. Card reader availability – The total percentage of PRESTO card readers that are in working order in all in-service vehicles
2. Service availability – The total percentage of in-service vehicles that have at least one working PRESTO card reader in the vehicle [p 38]

The problem of physical access to a working reader has been debated in this and other forums for some time. In short, if there is no working reader where a rider boards a vehicle, and the vehicle is at all crowded, the rider is likely to not “tap on” because it is too inconvenient. Moreover, if readers appear to be in service, but do not acknowledge a tap, riders could get used to a non-responsive device and simply assume this is normal. The effect for TTC revenue collection varies because some riders will have passes on their cards, and others may be taking a multi-stage trip where they will encounter a working machine eventually. Therefore, each missed tap at a defective reader does no translate to lost revenue.

A standard that is based on one working reader per vehicle will overstate availability because it assumes all riders will or can make the attempt to physically reach a working reader, and an organization that counts on such a metric is badly out of touch with the real world of transit riders.

Are The Lights On, But Nobody’s Home?

Next comes the definition of the state of a reader:

Included in Availability Calculation – Device is:

• working
• set to not collect fares (e.g. New Year’s Eve)
• in error state
• in transition state (while bus is in-service, e.g. system update)
• offline

Excluded from Availability Calculation:

• Device is in scheduled maintenance
• Vehicle is not in NextBus (GPS application system with TTC information on scheduled vehicles, used by Metrolinx to determine whether vehicle is in-service or not)
• Device is in the process of booting up upon vehicle start up (PRESTO staff advised this should not generally take longer than 5 minutes, therefore excluded) [p 39]

There are many problems with these definitions.

“Working” state is tested using a network “ping” which is a low-level request that the device acknowledge that it exists. This does not guarantee that the application layer a rider deals with is actually working, only that enough of the device is “alive” to send back a response. It is unclear whether the readers are capable of (or even polled to send) a response indicating specific error conditions.

When a reader is “frozen” in that it appears ready to accept taps but provides no response, there is a good chance it is not actually doing anything.

Our audit work shows that of the issues identified during our bus device audit, nearly 300 (91 per cent) of the total 330 issues identified over a period of two days on 168 buses were due to frozen PRESTO card readers.

We selected 20 per cent of the instances of frozen readers and found that they were all counted as being in-service in the device monitoring software tool and in the card reader availability calculation. This suggests that the availability of these frozen readers is being overstated. [p 43]

Presto’s response to this was something of a moving target suggesting a scramble to come up with a credible story.

PRESTO staff advised us that frozen readers will only be “momentary lapses” and are “isolated incidents”. At the time of audit fieldwork PRESTO staff advised us that frozen readers would show as in-service in the device monitoring software tool. Then while reviewing drafts of this report, PRESTO staff advised us the device status of frozen readers will be corrected overnight for the weekly availability report. However, we were not able to confirm this is the case. The weekly report we reviewed raised additional questions on the reliability of this information and we could not verify whether these issues were later corrected or not. [p 44]

When Is A Bus In Service?

The list of buses in scheduled maintenance depends on data from each of the garages and carhouses. These lists might not be up-to-date, and so the wrong population of vehicles could be included in the availability test. Activity in NextBus does not tell the whole story either because there are conditions under which NextBus will not report the location or activity of a vehicle, depending on how the query to it is framed. Some vehicles do not show up in NextBus, and others vanish from time to time. Unscheduled extras are particularly troublesome for NextBus which is designed for tracking and prediction of scheduled service.

TTC uses the NextBus GPS application to provide its customers with real time information on where buses and streetcars are on each route. TTC provides electronic files on a monthly basis to NextBus of its scheduled vehicles by route. The information does not include vehicles that are in-service but have not been scheduled, e.g. shuttle buses due to service outage on portion of subway.

…

Only the in-service vehicles reflected in the NextBus data are included in the PRESTO card reader availability calculation. However, due to various technical and other limitations, Metrolinx may not be receiving complete and accurate information for all TTC in-service vehicles. It is important to note that NextBus was not designed to be used for purposes extending beyond informing customers of the location of its buses and streetcars …

…

We noted 61 vehicles that reported revenue but were not included in the card reader availability spreadsheet for one day tested. 17 of the 61 vehicles did not show up in NextBus even though they were in-service, which means they were not captured in the availability calculation. The cause of these incidents is not clear.

…

The remaining 44 of the 61 vehicles showed up properly in NextBus but still were not included in the card reader availability spreadsheet. Metrolinx staff advised us that those vehicles were included in the long term maintenance list provided by TTC and as a result were excluded from the card reader availability spreadsheet. However, we noted some vehicles that were not on the long term maintenance list but showed up in NextBus were still excluded from the card reader availability spreadsheet. Metrolinx Staff advised us that this was due to a defect with the device monitoring software tool. [p 45]

Another problem not mentioned by the report is that, at times, the schedule in use by NextBus is out of sync with what the TTC is actually operating. This occurs typically at changeovers between schedule periods when, for some reason, new data is not loaded into NextBus, or occasionally when this is done before the actual date of the change. This can cause strange behaviour in vehicle arrival predictions as riders have found from time to time, but it would also cause at least part of the service not to be counted for Presto reader availability.

Hello? Hello? Is Anyone There?

The situation with “ping” tests of device availability is also subject to problems identified by the audit.

Up until late 2017, the availability rate for PRESTO card readers was calculated using one daily ping snap shot at 7 AM. This provided a very limited view of the true availability of the devices. Currently, more frequent pinging has been implemented, with 65 pings each day, at 15 minute intervals,between 6:00AM and 10:00 PM.

While 65 pings a day is a better measure of the availability rate than just one ping a day, it can still be improved because a device can go through multiple states (working, error, offline, etc.) within that 15 minute window. This is an issue,especially at rush hour, because even if the reader goes out for a few minutes, the impact is much more significant because a larger volume of passengers could be unable to pay, and this would not be reflected in the availability rate.

From the over 330 issues identified from our bus device audit work, the majority of the issues reported didnot occur exactly at a 15 minute interval. For example, some operators noted the device went out of service and came back within a minute (e.g. 7:03 PM) while other operators noted issues that lasted for a few minutes (e.g. 9:51 AM–9:56 AM). In both cases, because the issues occurred within the 15 minute ping interval, the outages were notreflected in the availability rate.

…

Our results were consistent with TTC’s own device results in this area as well. The TTC found that the availability rate would vary if more frequent pinging was used. [p 47]

Metrolinx argues that more frequent ping testing could burden the readers slowing response time for riders, and burden the network. This really sounds like grasping at straws considering that any individual reader is not called on to validate many cards per minute, compared with the sub-second activity necessary to respond to a ping.

The boot-up sequence is important because buses may be turned off for extended periods at terminals. The readers may not be ready to accept fare taps immediately when a bus comes back into service. A fairly common failure mode with readers is a reboot loop where a machine will alternate between being in service (usually briefly) and out of service attempting to restart itself. There is a good chance that a “ping test” that talks to the hardware, not to the application, will not see this problem. The reader will be “available”, but will not be collecting fares.

Presto staff claim that the boot sequence takes about five minutes, but the Audit Report found that “some devices took at least 15 minutes to boot up on an audit day” [p 39].

There were many problems found with the monitoring tool used to track Presto reader availability, but I will leave it to interested readers to peruse the report [starting at p 48].

Part-Time Monitoring

A particularly amusing problem was discovered in that, because the monitoring is actually done by a company in India, the availability reports were not available on days that were Indian holidays. This has been corrected. Meanwhile, the TTC is still not receiving availability reports for weekends and holidays in Canada. [p 50]

The following chart [p 41] summarizes how the process works.

Incident Management: Presto Card Readers

Major problems exist in the operation and support of the network of Presto devices thanks to a badly designed, dysfunctional incident management system and process. One cannot help feeling that this system began with a handful of devices and low user volume on GO Transit at a scale when ad hoc processes with pencil and paper (or possibly stone tablets) were sufficient to keep the system running. As it grew, so did volumes and complexity, but the processes and responsibilities set up to support the system were a patchwork, not a unified system.

Presto has an incident management system, but many events, especially transient ones that appear to correct themselves, are not necessarily logged. This causes under-reporting of device failures and maintenance activities, and could prevent necessary repairs from being made because nobody knows they are required.

The service ticket creation process is a manual process, with no automatic creation of tickets. While TTC can open tickets in PRESTO’s incident management system, the majority of tickets are opened by PRESTO’s vendor. PRESTO’s vendor’s monitoring team observes the screen of the device monitoring software tool 24/7to identify any large anomalies and will only open tickets for and troubleshoot devices that have triggered a critical alert under the “out-of-service” device status on an in-service vehicle. [p 57]

The incident reporting mechanisms for Presto readers are diagrammed in the figure below.

A common factor here is that much of the reporting depends on someone in the field, either a customer or TTC employee, reporting a problem which might eventually find its way into the incident management system. A major problem, however, is that the problems are so pervasive that there is little incentive to report them, and those who might do so feel that Presto is unresponsive to (or worse does not care about) reliability issues on its system.

Under some circumstances, Presto’s support vendor can recover a machine’s functionality remotely, but originally these events were not logged because no hands-on repair was required.

The vendor’s staff advised us that starting in late 2018, they decided to slowly work towards logging all remote recoveries as tickets in PRESTO’s incident management system, regardless of whether or not the team was able to remotely recover the device or not. However, it was not clear from our audit work whether remote recovery is always logged as tickets, as we found instances where devices had an “out-of-service” device status but did not have a ticket opened in PRESTO’s incident management system.

Without logging a ticket in PRESTO’s incident management system, it is difficult to verify whether any action was taken or not to address the malfunction, and if taken, whether it was done in a timely manner.

In addition, if a device appears in-service but is truly not available and the status is not reflected properly in the device monitoring software tool (e.g. frozen card readers), it appears that the defective issue may not be addressed unless raised by TTC staff. Also, only in-service vehicles are prioritized for remote recovery or creating a ticket in PRESTO’s incident management system. Those buses identified by TTC as being in long-term maintenance (greater than seven days) will not be prioritized. If there is inaccuracy in this information, any PRESTO issues with those vehicles would only be addressed if raised by TTC staff. [p 59]

Operators and Forepersons are supposed to report problems with Presto readers, but this is not done consistently. Whether this is a workload issue, or simply an attitude that logging these problems is not worth the effort, is unclear. In any event, the report notes that many problems that had been logged by operators on sign-in sheets (filled in when a vehicle returns to the garage) never made it into the incident management system.

We found that of the over 70 PRESTO issues reported in TTC operator sign-in sheets over four days, no ticket was opened for over 40 per cent of these. As mentioned above, it is possible that the device may have self-recovered automatically or PRESTO’s vendor monitoring team may have remotely recovered them. However, since no ticket was raised, we are unable to verify if any action was taken.

TTC’s Fare Card team also conducted internal device audits in February and June 2019. For both of these audits (with a limited sample size), TTC found that for the majority of the issues identified by TTC Operators and confirmed by PRESTO as being valid issues, there was no ticket opened in the incident management system. [p 59]

Operators do not always report problems, no doubt because they despair of things actually being fixed.

TTC operators are required to record all PRESTO defects observed during their shift on their vehicle sign-in sheet at the end of their shift. However, in our device audit, we found that of the over 330 issues reported, very few of them were noted on the vehicle sign-in sheets. Management interviewed some operators and the most common reason given for not including PRESTO issues on their sign-in sheet was that the PRESTO issues occur so frequently each day, with readers going in and out of service and back in again. [p 63]

…

TTC operators are also required to perform vehicle pre-service checks (referred to as circle checks) before they leave the garage, and this includes identifying and reporting any PRESTO defects. If a front PRESTO card reader is found to be out of service during the vehicle pre-service check, TTC’s policy is to change off that vehicle for another one, as long as there is sufficient supply of vehicles to meet operational needs. However, we noted the policy was not always consistently followed, particularly for the beginning of the early morning shift. [p 64]

As if all this were not bad enough, there is no tracking system for physical devices which might be swapped out with no record of which device was on which bus at what time. Problems reported by vehicle number could be misleading if an unreliable device found its way onto two or more vehicles through maintenance replacements. [See p 62]

There is also a problem with buses flagged as being out of service long-term for repair, and this lies with the TTC, although the problem might be compounded by difficulties with keeping the vehicle list up-to-date.

For one day that we tested, we identified 37 out of 125 in-service buses (30 per cent) that were listed on the “buses out-of-service more than seven days” spreadsheet, but were actually in-service, i.e. on the road and collecting fares. We also checked the following three days of spreadsheets in case these vehicles had been scheduled and repaired on the same day, however they were still listed in long-term maintenance. The spreadsheet provided by TTC to PRESTO showing the out-of-service buses is not always accurate. [p 63]

Vehicles that are thought to be out of service will not have incident tickets written, nor will real-time attempts be made to restore functionality remotely.

This situation is further complicated by the lack of a data analysis tool to review the logs, and the fact that these are purged after 60 days, a practice that interfered with the audit process and which is contrary to the TTC/Presto contract. Presto’s sub-vendor claims:

“our professional services team as per our policy do not write database scripts to access this information as this is not a best practice”

Moreover, the contracted data retention period is seven years, not two months. [Details on pp 60-61]

This suggests that the Metrolinx sub-vendor responsible for the incident management system has very different contractual requirements for the system they provide compared to what the TTC contracted for with Metrolinx.

PRESTO Vending Machines on New Streetcars [p 20]

The problems with machines on board the new streetcars show the problems of split responsibility for various parts of a system and the lack of centralized accountability and monitoring.

Roles and Responsibilities:

• Machines are owned by PRESTO
• PRESTO bought them from S&B
• Maintenance is done by S&B
• Monitoring of out-of-service instances are done by S&B
• Revenue transactions and out-of-service instances are recorded in FareGo subsystem

Availability Calculation of PRESTO Vending Machines on New Streetcars:

• How it is calculated: Based on repair time (calculated from the time the issue was raised in the incident management system to the time it was fixed)
• Who calculates the availability? Scheidt & Bachmann (S&B)
• Availability Calculation Issues Include:
  • Out-of-service machines as a result of “coin box full” are excluded from the availability calculation because it is technically not broken so it is not the vendor’s responsibility (A.2)
  • Out-of-service machines as a result of network connectivity issues are excluded from the availability calculation because connectivity is the responsibility of another vendor (A.2)
  • Prior to July 2019, PRESTO was responsible for the first line maintenance but this repair time was not included in the availability calculation (A.2)
  • Not all out-of-service machines were included in the availability calculation as being “not available” but should have been, according to the definition between PRESTO and its vendor (A.2)

Incident Management Process:

• S&B staff monitor the out-of-service instances in their back-end system (FareGo)
• TTC also raises out-of-service incidents to PRESTO’s incident management system
• S&B is responsible to fix hardware and software issues, Garda is responsible for coin collection, and Telus is responsible for network connectivity issues

A major problem with availability reporting for the onboard fare machines is that the length of time a machine was considered to be out of service could be considerably shorter than the period during which it was unable to collect fares. Some failures were detected remotely, while others were reported manually by TTC staff. This would trigger a “first level” repair attempt by Presto, but if that was unsuccessful, the incident escalated to “second level” repair by staff from the machines’ vendor, S&B. Only at this point did the clock start running, and even then the incident was not counted if S&B determined that it was not actually a machine failure.

We also noted that there were out-of-service incidents that were shown as being in-service in the availability calculation. When we inquired with PRESTO and its vendor’s staff, they advised us that only out-of-service incidents that were a result of a hardware or software problem would be included as out-of-service. Vending machines that are out-of-service as a result of a network problem or “coin box full”would not be captured as out-of-service in the availability calculation, as PRESTO’s vendor is not responsible for network connectivity and coin collection. [p 54]

A common problem with these machines is that if their farebox fills up, the machine takes itself out of service until the box is emptied. This is not done until the streetcar is  back at the carhouse, and then only if (a) the fact the machine needs to be clear is known and (b) TTC staff make the car available to Garda staff rather than parking it in the yard.

One might also observe that the absence of credit/debit card readers on these machines (about which more below) accelerates the rate at which their fareboxes fill making this type of failure more likely. Moreover, it begs the question of whether the regular cycle interval for clearing fareboxes is sufficient to keep up with typical usage patterns.

The audit was complicated by the absence of data for much of July 2019 when the vendor was updating their FareGo software. Try to imagine any reputable online business trying to use that excuse for missing functionality. Even after the upgrade, problems remained.

When we reviewed the August 2019 availability calculation raw data, we noted the same issues persisted. We noted that there were out-of-service incidents that were counted as being in-service in the availability calculation. [p 55]

As for credit/debit functionality, this was removed because it interfered with the proper operation of the machines. There are conflicting statements about its future from vendor and client.

In TTC’s June Board meeting, TTC staff were requested to follow up with Metrolinx on expected timing to restore the credit/debit card functionality of PRESTO vending machines. Metrolinx has informed TTC that it does not plan to restore the credit and debit card functionality and states that TTC agreed to this change being permanent. TTC advised us that the change was temporary and that they did not agree for it to be permanent. TTC is in negotiation with Metrolinx to restore this important functionality for customers to pay their fares. [p 55]

Incident Management: Fare Vending Machines on Streetcars

As noted earlier, a common problem with the on board vending machines is that they are out of service because their fareboxes are full. The resolution of such problems is hampered, putting it delicately, by split responsibility for various aspects of the Presto system. The process is almost comically broken, but the larger question is why this situation has persisted for so long.

When we reviewed August 2019 incident tickets in PRESTO’s incident management system, we noted about 56 per cent of the out-of-service incidents raised for PRESTO vending machines by TTC were all related to “coinbox full” as per vendor’s resolution notes. At the time the ticket was raised by TTC, TTC did not know the vending machine was out-of-service as a result of the coin box being full.

We were informed by TTC and PRESTO staff that PRESTO’s vendor is responsible to collect coins/tokens during every night on all streetcars that are made available to them in TTC’s car house. We noted in one instance that it took seven days to collect coins/tokens after the machine issued the “coin box is full”warning sign. For the same streetcar, the last time the coin collection was done was 14 days prior. TTC staff have advised us that the vehicle was not made available for collection those nights due to the vehicle being in-service.

We noted that for August 2019, the “coin box is full” sign was issued for 188 PRESTO vending machines. On average per day, approximately six PRESTO vending machines were out-of-service as a result of the coin box being full, the highest being 15 vending machines.

…

We noted in one instance,that TTC raised incident ticket ssix times in the incident management system for the same streetcar. The vendor’s resolution for those incident tickets was that the coin box was full and they closed the incident on their end with no action, as coin collection is not their responsibility.

…

From our audit work, it appears that neither TTC nor PRESTO are ensuring that the coin box is emptied on a regular basis for all vending machines on new streetcars. It is important for TTC to not only run “coin box is full” warning sign reports on a daily basis, but also to ensure that the streetcars with these warning signs be made available to PRESTO’s vendor during the night for coin collection. [pp 67-68]

Parkeon Vending Machines on New Streetcars [p 21]

Roles and Responsibilities:

• Machines are owned by PRESTO
• PRESTO bought them from Precise ParkLink
• Maintenance is done by Accenture and Precise ParkLink
• Monitoring of out-of-service instances are done by Accenture
• Revenue transactions are recorded in the Precise ParkLink subsystem
  • Out-of-service instances are recorded in the monitoring tool owned by Precise ParkLink [p 21]

Availability Calculation of Parkeon Vending Machines on New Streetcars:

• How it is calculated: Based on repair time (calculated from the time the issue was raised in the incident management system to the time it was fixed)
• Who calculates the availability? Precise ParkLink
• Availability Calculation Issues Include:
  • Out-of-service machines as a result of “coin box full” are excluded from the availability calculation because it is not technically not broken so it is not the vendor’s responsibility (A.2)
  • Out-of-service machines as a result of network connectivity issues are excluded from the availability calculation because connectivity is the responsibility of another vendor (A.2)
  • Not all out-of-service machines were included in the availability calculation as being “not available” but should have been, according to the definition between PRESTO and its vendor, such as coin jam issues (A.2)

Incident Management Process:

• Accenture staff monitor the out-of-service instances in the monitoring tool owned by Precise ParkLink
• TTC staff also raise out-of-service incidents to PRESTO’s incident management system
• Precise ParkLink is responsible to fix hardware and software issues, Garda is responsible for coin collection, and Telus is responsible for network connectivity issues

As with the S&B fare vending machines, there are many issues where a device reported as “available” is in fact out of service. However, if the fault is not the vendor’s “fault”, it does not count. This is another example of a sub-contractor reporting based only on the scope of their function while the overall purpose of their devices, to collect fares, is lost in the shuffle.

The availability calculation for the Parkeon vending machines is calculated by Precise ParkLink. We reviewed the May 2019 raw data availability calculation and noted that there were out-of-service incidents that were counted as in-service in the availability calculation, and the availability was reported as 100 per cent for May 2019. When we inquired with PRESTO’s vendor staff, they advised us that only out-of-service incidents that were a result of a hardware problem would be counted as out-of-service by Precise ParkLink. However, we noted incidents that were a result of a hardware or a software problem but still were counted as in-service in the availability calculation. For example, the machine was out-of-service as a result of a coin jam issue – this was counted as in-service in the availability calculation. Also, a machine out-of-service as a result of a coin box being full would not be counted as out-of-service by Precise ParkLink, as the problem is not related to hardware or software and Precise ParkLink is not responsible to empty the coin box. [p 56]

Subway Fare Gates [p 22]

Roles and Responsibilities:

• Fare Gates are owned by TTC
• TTC bought them from S&B
• Maintenance is done by TTC and S&B
• Manual identification of out-of-service instances are done by TTC
• Revenue transactions are recorded in FareGo subsystem

Availability Calculation of TTC Subway Fare Gates:

• How it is calculated: Using fare gate out-of-service status in the FareGo subsystem
• Who calculates the availability? TTC
• Availability Calculation Issues Include:
  • Partially available gates (e.g. only exit side works and not the entry side) are excluded from the availability calculation (D.1)
• Incident Management Process Issues Include:
  • Not all out-of-service gates that are stuck in an open position are barricaded to prevent customers from passing through, which could result in revenue loss to TTC (D.1)
  • TTC does not receive automatic alerts from the current FareGo subsystem when the fare gate goes out of service (D.2)
  • TTC staff currently have to manually identify the out-of-service gates (D.2)
  • At the automatic entrances where there is no TTC Staff presence, the gate could potentially stay out of service for a long time (D.2)
  • Escalated issues to TTC’s vendor to fix fare gates were not completed within the targeted timeline as per the SLA (D.3)
  • TTC gets compensated up to a maximum of 25 per cent of the service charge if the availability and maintenance targets are not met by its vendor (D.3)

The Roles and Responsibilities are summarized in the diagram below.

As with other devices in the Presto family, responsibility for various parts of the functionality and support of the fare gates is scattered among different entities, and depends on manual reporting of failures.

When a fare gate goes out of service, the FareGo system does not create an automatic incident ticket. TTC staff (including Station Supervisors and Fare Collectors) are required to raise the incident with Transit Control when they spot an issue with the fare gate functionality. Transit Control then logs the incident in TTC’s work order management system for TTC’s Revenue Equipment Attendant (REA) staff to go out in the field to fix the gates. TTC staff do make efforts to barricade an entrance when it is stuck in the open position until someone comes to fix it. However, there is a higher risk at automatic entrances for broken gates (including those stuck in an open position) to go unreported for longer periods, as there is no TTC staff presence, and it would generally require a station supervisor or REA staff to notice the issue.

TTC staff advised us that the current version of FareGo (version 3.2) does not support automatic alerts when a gate goes out of service. However, TTC is planning to purchase FareGo version 3.9 which allows for automatic email alerts when a gate goes out of service. [p 89]

This primitive reporting structure is symptomatic of functionality that should have existed from day one and is only now being addressed. But this is not all.

Fare gate activities are uploaded to four PRESTO data concentrators before being uploaded to the FareGo system. PRESTO owns the four data concentrators provided to TTC. When the data concentrators are down and/or stop working, FareGo will not receive any activities in real time at all. We were advised by TTC management that the issue with data concentrators is due to the high volume. The lack of consistent real time data makes it harder for TTC staff to monitor the gate functionality in real time or to do inquiry.

TTC staff have advised us that data concentrators will upload the fare gate activities once they are back online and running, and when TTC moves to FareGo version 3.9, it is expected that data concentrators will no longer be required. [p 89]

This explains the mystery of why taps made on the subway sometimes much longer to appear on a rider’s online trip log than those made on surface vehicles.

Fare gate availability is supposed to sit at 99.5%, although this target has not been achieved. Since July 2018 it has stayed above 97%. TTC has worked with the vendor on defect resolution and software issues, and has shifted some first line responsibility to its own staff from the vendor. The TTC also has “increased diligence” in preventative maintenance for the gates. [pp 84-85]

Fare gates can be out of service in various ways, but not all of these represent a potential for lost revenue or prevention of riders from entering the system and paying (using another gate). However, when a gate is stuck open, riders can simply walk into a station without paying.

The field observations by the Auditor General’s team revealed that many gates were impaired in some way, and this calls into question claims for 97% availability.

We conducted four days of fieldwork observation of fare gates across 74 stations (992 total fare gates).

…

Our staff observed 45 out-of-service gates at 27 subway stations:

• 18 gates were fully out of service (showing red “X” sign) with eight of these gates stuck in an open position.
• Seven gates automatically reset back in-service during the staff attendance, with the reset time ranging from 30 seconds to two minutes.
• 20 of the gates were partially available – they were only available from the exit side and did not accept customer taps from the entry side, as described above.

The 20 gates that were only partially available, however, showed as being in-service in the FareGo system. TTC staff advised us that if the gate is partially available (i.e. only exit side works), the gate would still show as being “in-service” in the FareGo system.

…

Of the 18 gates that were fully out-of-service in the morning, we observed:

• 13 gates were fixed; and
• Five gates were not fixed by the time staff went back in the afternoon.

Of the 20 gates that were only partially available:

• Nine gates were fixed by the time staff went back in the afternoon; and
• 11 gates were not fixed. Only one of the 11 was serviced by TTC staff after we observed them in the morning, but went out of service again in the afternoon.

Of the five gates above that were not yet fixed:

• Two of them were serviced by TTC staff after we observed in the morning, but went out of service again in the afternoon;
• Two gates were attended to by TTC staff and escalated for S&B’s second-line maintenance; and
• One gate was fixed after our observation in the afternoon – this work did not have a work order opened in TTC’s work order management system.

Of the eight gates that were stuck in an open position:

• Five gates were fixed by the time staff went back in the afternoon;
• Two were not fixed but had pylons placed in front of the gates and had been escalated by TTC staff for S&B’s second line maintenance; and
• One was serviced by TTC staff after we observed in the morning, but went out of service again in the afternoon and it remained stuck open with no barricade.[pp 85-87]

As long as riders are prevented from physically passing through the open gates, those that simply don’t work do not represent a revenue loss. However, if riders often see out of service gates, this contributes to a reputation that the system as a whole is unreliable.

There can also be an accessibility issue if the wide gates needed for mobility devices are out of service as affected riders cannot make use of a bank of working, normal-sized gates.

If outages are more pervasive than what is reported, the quality of this part of the Presto system is not accurately reported, and complaints can be easily brushed aside with claims of a very high, but fictitious, availability rate. If a high rate can only be achieved with constant attention and maintenance, this is a cost of doing business even if notionally most of the gates are “available” most of the time.

The TTC presents statistics purporting to represent the overall performance of the fare gates, but there are problems in how this is done.

TTC’s definition of service availability as stated in the CEO report is “Percentage of time fare gates are available for use”. TTC selected two codes from the FareGo events to calculate and report its fare gate availability, however these codes do not include fare gates with reduced functionality where customers are unable to tap to make a payment.

[A table in the report] shows what the recorded downtime in FareGo compared to our observed downtime for fully out of service gates and partially available gates. For gates that were fully out of service, the difference in downtime was six hours and 27 minutes. However, for the partially available gates the observed downtime was approximately 20 times more than the recorded downtime in FareGo system. Therefore, TTC’s reported availability rate could be overstated and should be refined further to include partially available gates. [p 88]

Actual maintenance targets for fare gates are not being met by the vendor.

In June 2019, there were almost 3,000 TTC work orders created for fare gates, with approximately 150 of these escalated for SLM. The average resolution time for these June 2019 SLM work orders was just over 27 ½ hours. TTC staff advised us that generally all SLM incidents are considered high to critical priority. [p 90]

There are further problems with tracking systems that cannot distinguish between severity levels in maintenance orders and therefore cannot correlate performance targets with ticket severity.

Contract Management [pp 22-23]

TTC and PRESTO signed the Master Agreement in November 2012.

TTC and Metrolinx Need to Work Together to Resolve Issues Better, including the Following:

• The lack of a governance committee (Joint Executive Committee) for TTC and Metrolinx to address contractual and operational issues directly together represents a governance gap and is impacting the arrangement. SLAs have not been set and the Expert Panel to help set service levels has not been operationalized (C.1)
• At least 40 per cent of contracted deliverables remain outstanding, according to TTC. Metrolinx has indicated that in its view, almost half of them “don’t have public value or cost benefit, or the requirements are not considered relevant” and are not formally planning to deliver them, while TTC does not agree – so further clarity is needed. (C.2)
• SLAs have not yet been signed by both parties after 7 years into the agreement (C.3)
• Metrolinx has not yet paid TTC’s $7.5M revenue loss claim for PRESTO fare equipment issues, stating “it is a result of TTC not providing sufficient data and support on their revenue loss claim”. However, much of the information needed by TTC has not been provided by Metrolinx, e.g. device level data (C.4)
• It appears that Metrolinx has withdrawn funds from TTC’s revenue bank account for items other than its commissions – control improvements are needed (C.5)

Governance and Oversight

The Presto framework and contract envisage three bodies providing oversight and guidance to the system’s development and operations:

• A Joint Executive Committee between the TTC and Metrolinx
• An Expert Panel to deal with the capabilities of the system technology and the requirements in the TTC/Metrolinx contract
• A Scheme Governance Committee of Presto client agencies and Metrolinx in its role as service provider

The Joint Executive Committee has not existed for some time, although it was replaced in early 2018 by another committee.

These Integrated Schedule meetings include the same attendees, who have met on a bi-weekly basis with the focus on projects and their schedules (timelines). Although the meetings included the same executive members, the nature and purpose of these meetings are generally project focused. Operational issues are raised only insofar as it related to current issues in a project. Contracted deliverables yet to be delivered are not addressed. [p 69]

Because a Service Level Agreement still does not exist, the Expert Panel cannot do its job as there is no yardstick against which the system’s performance or possible modifications can be measured. The Panel is also expected to deal with conflicts where Metrolinx and the TTC cannot agree on an SLA. This entire framework is “not operational” [p 70].

The absence of data about Presto’s operations affects is reflected in a lack of understanding by Metrolinx about just how their system is (or is not) working.

In addition, when resolving issues at the Joint Executive Committee (or a similar table) and as part of the SLA set-up, it is paramount that the parties have the right information to understand and diagnose the issues. We observed that both the TTC and Metrolinx have an information gap with regards to the areas we audited. Some of the information we requested for our audit was the first time this information was requested by Metrolinx of its vendors. Metrolinx staff advised us that our audit requests helped them to better realize the information that they should be regularly requesting and monitoring from its vendors, and then sharing with the TTC. The parties need to obtain the necessary information and data to properly identify, diagnose, prioritize, and resolve the various operational issues, including those identified in this audit report. Once the parties have been able to address both the information and governance gaps, our view is that the issues identified in this Phase 2 report can then be resolved by all the parties working together to do so. [p 71]

A further problem is that the TTC is the largest client of Presto with complex requirements, but their needs can be bogged down in the context of Presto’s service to other agencies.

Many of our recommendations from our Phase 1 audit, for example, require TTC to work together with Metrolinx to achieve them, such as changes to improve front-end controls on concession (e.g. child) PRESTO cards and the sale of them. The current contractual arrangement (with the Joint Executive Committee continuing into the operational phase) between the two parties should be sufficient for the parties to be able to work together on many of the required changes to controls, etc. However, many of these recommendations have not yet been addressed, which may be due in part to TTC being limited by the requirement to go through the consensus based governance framework for all PRESTO transit agencies via the second governance committee, the Scheme Governance Committee. There also appears to be lack of clarity for both TTC and Metrolinx on which issues should come to which governance table and it would be helpful to define this. [p 73]

…

… the Scheme Governance Committee table has also created concerns for TTC. Metrolinx is working together with all PRESTO transit agencies on planning the significant foundational changes required to further modernize PRESTO, including addressing obsolescence of field equipment and supporting and enabling future PRESTO features. However, the current draft plan lays out milestone dates for all PRESTO transit agencies other than for TTC.

TTC staff are concerned that the current draft plan does not include sufficient detail to address the remaining contractual deliverables that are included in the Master Agreement between TTC and Metrolinx. Further, the draft plan indicates TTC changes will be made after the implementation dates for the other PRESTO transit agencies, and these foundational changes will affect the features that become available for TTC customers in the future. Although we recognize the other PRESTO transit agencies need to have their aging equipment addressed, it is important to recognize that such a large transit agency as TTC is a key client who needs to be a significant contributor in PRESTO’s roadmap forthe future, particularly in the initial planning stages. TTC staff have advised us they are very concerned as the future of the PRESTO system changes are being designed for the needs of the other smaller transit networks, and may not be able to accommodate the needs of the most complex transit network (i.e. TTC). [pp 73-74]

The complexity of relationships between the parties is shown in the diagram below [p 72].

Undelivered Functionality

According to the TTC, approximately 40% of all contracted requirements have not yet been delivered by Presto, chief among them support for “Open Payment” where use of the system is not tied to a proprietary fare card.

The main contracted deliverable that TTC was seeking at the time ofits arrangement with Metrolinx was to procure an open payment solution for its customers to be able to pay with mobile applications (i.e. cell phone, debit/credit card), as noted on the first page of its contract:

“TTC expressed an interest in using such PRESTO services provided that suitable funding arrangements could be made and the system could be modified to meet its (TTC’s) business requirements.

To better meet such business requirements and the needs of the other (transit agencies) … who use the current PRESTO services, Metrolinx is in the process of modifying and enhancing that service to develop and implement a new generation fare collection solution, defined herein as PRESTO NG, that is based on an open architecture using industry standard tools,which will accommodate open loop financial cards, mobile applications and future technological innovations.”

Open payment is an important customer service that also allows TTC and Metrolinx to benefit by making it easier for customers to pay for services that are part of their everyday lives.

If you can use your cellphone application to pay for a coffee or debit/credit card almost anywhere – most customers would expect one should be able to – as envisioned in the 2012 contract seven years ago – to tap your debit/credit card or cell phone to pay. [p 75]

…

The contract also requires Metrolinx to provide key services, including:

(i) obtaining and maintaining the functionality described in a mutually agreed set of business requirements;

(ii) obtaining and maintaining the equipment, software and network capability necessary for the operation of the PRESTO NG service;

(iii) the implementation, operation, service, maintenance and repair of same, and

(iv) such other services required to be performed to meet the TTC Business Requirements, defined herein collectively as the Managed Services. [p 76]

For its part, Metrolinx does not intend to deliver many items claiming that “they don’t have public value or cost benefit, or the requirements are not considered relevant any longer”.

Some of the items on Metrolinx’s list include:

• visual distinction for PRESTO concession cards (i.e. child, youth, post-secondary student, senior)
• PRESTO card reader device management and monitoring
• single-use tickets for concession fares
• ability to purchase single-use tickets on surface vehicles
• recording of denied PRESTO taps

The parties have not reached common ground on what remains deliverable. TTC needs clarity on exactly what is outstanding – what will be delivered and when, what Metrolinx intends to drop, and the impact and value of those items to the overall deal need to be assessed. Changes should be done through a formal change-order process.

Further, while we recognize that Metrolinx is reliant on many of its vendors and sub-vendors to ensure its contracted deliverables are met, TTC’s contract is with Metrolinx. If there is a deliverable already included in the 2012 Master Agreement between TTC and Metrolinx, TTC should not be billed an additional amount to receive the information supporting the deliverable. [p 77]

The absence of a Service Level Agreement relieves Metrolinx of much responsibility because there is nothing against which they can be held accountable, and hogties the TTC in its relationship with a vendor who should be treating them as a valued, primary customer.

While the Master Agreement was signed in 2012, there is yet to be a suite of SLAs signed between TTC and Metrolinx, seven years after the contract was signed. Other PRESTO transit agencies have a signed SLA with Metrolinx. However, per discussions with Metrolinx and TTC, it does not appear that the two parties are close to achieving an agreement on an SLA.

In our view, it is best that contracts are only signed when an SLA is established, otherwise, there is little leverage to obtain agreement and there needs to be agreement on key deliverables, targets and timelines.

However in the case of this deal, the design, hardware, software and details of PRESTO were not finalized at the time of execution so the Master Agreement included a governance framework (including the Expert Panel discussed earlier) to ensure future agreement on the Service Levels and Service Level Targets.

Setting up SLAs helps to clarify what’s important to measure, how it will be measured, and the target and penalty for not meeting the target. SLAs make it easier to hold all parties accountable to deliver on expected service levels.

If Metrolinx and the TTC should pursue interim SLAs for specific subsystems even before an overarching agreement is reached, and Metrolinx should deliver contracted functionality rather than dismissing items as unnecessary. Process improvements are needed to ensure that broken links in the chain of responsibility are addressed.

By measuring availability and analysing why targets are not being met, issues can be addressed and performance will improve for the benefit of all stakeholders. Going forward, to ensure vending machines on new streetcars are available and to reduce service calls, the new SLA should be measuring if vending machines on TTC’s new streetcars are being emptied of coins/tokens every night.

Other areas that are important to measure and monitor could include the following:

• customer is given the ability to pay (e.g. coins/tokens collected nightly from each streetcar, frozen PRESTO card reader issue resolved);
• risk for customers evading fare is minimized (e.g. bus drivers can view concession type, child card controls improved);
• customer experience is positive (e.g. fare gate availability improved, including when stuck in closed position); and
• revenue transactions are properly captured (e.g. controls improved between device level and subsystem) and deposited (e.g. retailer cash reconciliation controls strengthened).

…

It is also important that there be shared accountabilities. For example, TTC needs to ensure vehicles for repair or coin collection are made available each night because that can impact fare operations. PRESTO needs to acknowledge such things as fare evasion and that working together with its transit agencies to improve front-end controls is partly their responsibility. (e.g. visually distinct PRESTO cards for varying concession types (e.g. child, student, senior) – also one of the contracted business requirements in Master Agreement). [pp 79-80]

In all of this, it is clear that Metrolinx signed a contract to get the TTC on board with Presto, but may have had little real commitment to it, or failed to understand the complexity of achieving what was agreed, or both.

Revenue Accountability [p 23]

TTC Should Receive All Its PRESTO Revenue.

There is no reconciliation between the device level (where the transaction starts) and the subsystems and PRESTO’s central system

TTC Needs More Assurance from PRESTO on their Controls:

• A manual reconciliation is done by PRESTO staff, but only between the subsystem and PRESTO’s central system. The manual reconciliation identifies transactions missing in PRESTO’s central system from the subsystem, but could be prone to human error and may not identify all items (E.1)
• There is no reconciliation between the device level (where the transaction starts) and the subsystems and PRESTO’s central system. This creates a higher risk if there are any missing transactions from the device level to the subsystems and to PRESTO’s central system, as they may not be identified and pushed through (E.1)
• TTC has not been provided with device level data from PRESTO (data is encrypted and purged after seven days) (E.2)
• PRESTO is not provided with a cash and debit/credit card reconciliation from PRESTO’s third party retailer network for sales transactions of TTC’s monthly passes and single-use tickets (E.4)
• TTC is not compensated for sales of TTC’s monthly passes and single-use tickets that do not get into PRESTO’s central system (E.4)

Claims Against Metrolinx

Problems have been brewing between the TTC and Metrolinx on many issues, and from Council’s point of view, a major item is revenue lost through non-functioning Presto readers.

Since the implementation of Metrolinx’s PRESTO fare equipment, TTC has invoiced Metrolinx $7.5 million for the amount of estimated revenue loss for the three years ending December 31, 2018, as a result of PRESTO fare equipment (including PRESTO vending machines on new streetcars (called Single Ride Vending Machines) and PRESTO card readers) not working at agreed-upon levels. Metrolinx has not yet paid TTC for this lost revenue. [p 81]

Although their contract provides a method for calculating lost revenue, Metrolinx and the TTC cannot agree on this. TTC’s position is that the Presto adoption rate (the speed with which fare collection shifts from legacy media to Presto) is much different from what was contemplated in the contract, the methodology for calculating lost revenue also needs to change. This runs headlong into another problem.

Metrolinx’s view is that TTC has not provided adequate supporting information for their revenue loss claim. However, much of this information would be required from PRESTO for TTC’s calculation and it has not been made available to them (e.g. device level data–is not kept beyond seven days). [p 82]

A Provincial Hand in the TTC’s Pocket

Metrolinx has been dipping into TTC’s bank account in a manner not permitted by the contract to recover fees other than their agreed-to commission.

TTC’s revenue bank account was set up to allow Metrolinx to withdraw their commission, as was done by all other PRESTO transit agencies, as per section 2.3 of the Funding and Financial Reporting Agreement.

…

Although TTC’s revenue bank account was set up in this manner, no other amounts should be withdrawn by Metrolinx from TTC’s revenue bank account without TTC permission. [p 83]

At the TTC Board meeting where this report was discussed, staff reported that the TTC had put a stop payment order on their account to block this practice, and Metrolinx had refunded monies taken in this manner. However, Metrolinx subsequently withheld over half a million dollars from revenue owed to the TTC to achieve the same end without actually reaching into the TTC’s account. The exchange between Vice Chair Heisey and TTC Acting Chief Financial Officer Josie La Vita is part of the YouTube record of the meeting.

Getting Money From RIders to the TTC

The diagram below shows how revenue flows from riders who pay for transit service in various ways through Presto and into the TTC’s account. There are many links, potential points of failure, and processes where integrity of the reporting is important.

Controls which, from an audit perspective, should be present are not for all paths where revenue changes hands between vendors, subsystems and the TTC.

Metrolinx contracts an independent auditor to perform an annual CSAE 3416 Type II audit (“3416 report”). The 3416 report provides assurance for the external financial statement auditors of its transit agencies, including TTC, that Metrolinx maintained effective and efficient internal controls related to financial reporting …

The 3416 report is also one of TTC’s business requirements in its contractual arrangement with Metrolinx …

To support TTC with validating PRESTO reported revenues, Metrolinx provides TTC with several key financial reports including financial transaction history, daily bank settlement, and extract reports on tap transactions (usage) and sales and payments.

TTC Finance uses the key financial reports provided by Metrolinx to perform its own monthly reconciliations and financial reporting of PRESTO revenues. TTC’s monthly reconciliations include comparing revenue reported from the financial settlements made by Metrolinx to TTC’s revenue bank account to the PRESTO central system and to the device subsystems (PDS and FareGo). TTC also ties its reconciliation from the financial reports to Metrolinx’s monthly reconciliations. [p 94]

Two types of problems were flagged by the Auditor General:

• missing transactions between the systems; and
• transactions not checked after devices re-connect to the system after being offline. [p 95]

The process of reconciliation between Presto systems is a manual one.

When Metrolinx prepares its monthly reconciliation for TTC, it manually identifies transactions missing in the subsystems but appearing in PRESTO’s central system, and vice versa. When there is a discrepancy identified by PRESTO staff manually, a ticket is raised internally to investigate the issue further. PRESTO Staff advised us that transactions from the subsystem are manually pushed into the central system as a result of the investigation. Manual identification of issues and pushing of transactions from the subsystem to central system are prone to human error. TTC only receives revenues that appear in the PRESTO’s central system. Every month, approximately 10,000 transactions on average from the subsystem do not get uploaded into PRESTO’s central system for TTC. Although this monthly amount of missing items is not considered financially significant, it is important to ensure this risk is addressed so that potential revenue is not missed.

There is no automatic exception reporting produced by the PRESTO systems when transactions do not get reported to either the subsystem or central system. Currently the manual reconciliation is a comparison against what is reported in the subsystem vs. PRESTO central system, so it is an even higher risk when transactions do not get reported to either of the systems (subsystem and PRESTO central system) as this would not be caught by the manual reconciliation. [p 95]

Some transactions take a while to arrive at the Presto central system and thence to the TTC as revenue.

We also noted that there are transactions that do not get uploaded to PRESTO’s system on the day of the transaction occurrence. These delayed transactions occur when the device is offline and the device doesn’t upload the transactions at that time, because it isn’t connected to the network, or there might be a system issue where the transactions do not get pushed from the subsystem to PRESTO’s central system. If there is a loss in connectivity and the device goes offline, the PRESTO card readers are designed so that the device will still successfully accept the taps, and the transactions will be uploaded once online again. However, this delay can sometimes be several days or longer. PRESTO’s systems do not provide exception reporting when a device does not upload its transactions on the day of the transaction occurrence. They also do not provide exception reporting on the delayed transaction receipts. Metrolinx staff advised us that the transaction gets a delayed transaction exception flag only when a transaction is received in the system 30 days later than the actual transaction date and time. [p 96]

The Auditor General even engaged an external auditor to review this.

We identified this risk and control gap – it is important for PRESTO to have additional controls in place to provide assurance that the existing interface controls are working properly and transmitting revenues completely, accurately and timely from device level to subsystems (PDS and FareGo) and subsequently to PRESTO’s central system.

The experienced professionals we engaged from an external audit firm confirmed this, with the control gap related to “device has malfunctioned, resulting in the inability for customers to tap their PRESTO cards” and concluded the following:

“Not all device failures are captured/logged in the monitoring tool. Further, vending machines on new streetcars and fare gates are not subject to the same detailed monitoring controls over PRESTO card reader device failures. … ” [p 97]

Although Presto does produce “3416” reports as required, they are incomplete.

Given risks and potential control gaps we identified through our audit work, we augmented our team with experienced professionals from an external audit firm to perform a gap assessment of TTC’s risks related to PRESTO revenue against the controls covered by the 3416 report.

We found some key controls were not included in the 3416 report; 12 out of 35 controls were either not implemented, not included in the 3416 report, or in one case, insufficient information was available to conclude on the gap. The main issue highlighted was that gaps exist in ensuring all revenue transactions are captured between the fare equipment (devices) and the subsystems.

This main issue includes the following risks that need to be addressed:

1. There are no formalized controls between TTC owned fare gates, PRESTO owned data concentrators, and the subsystem.
2. With regards to TTC monthly passes and single-use tickets, there is a lack of reconciliation controls designed to address variances between proceeds received and the information captured in PRESTO’s central system. Specifically, sales from retailers are not subject to reconciliation controls and similar controls have not been validated for cash received from PRESTO owned Point of Sale (POS) devices. Thus the extent, impact and treatment of potential variances is currently unclear and bears further investigation.
3. Errors that occur at the device level are not consistently identified by the existing controls. In addition, varied vendors and subsystems create further challenges in identifying whether all devices have consistent levels of control. For example, incomplete or failed taps do not always result in errors being logged (i.e. card reader faults). The impact of these “unknown” errors cannot be quantified at this point. [p 98]

Getting Data from Presto Was Challenging

The ability to audit the Presto system was complicated because data needed for this purpose was not available even though the information is supposed to be provided to the TTC under its contract.

For the section on completeness and accuracy of PRESTO revenue transmission to TTC, the Auditor General required and requested the device level data from PRESTO at the April 2019 joint kick-off meeting for the audit. Device level data is at the level of PRESTO card readers in buses and streetcars. Although reports were made available from the daily bank settlements and from the central system – if transactions did not make it from the device to the central system, we would have no way of knowing this without the device level data.

There were no concerns or objections raised by PRESTO at the time of our joint kick-off meeting to provide this data to us. After several follow-up requests and varying reasons provided by PRESTO on why this data could not be provided, we were informed too late in the audit (end of July 2019) that the device level data is available to be decrypted, but is purged after 7 days at the device level, and PRESTO’s vendor’s quote for the additional cost per device to obtain this decrypted data would have been cost prohibitive for the AGO. PRESTO’s vendor’s staff informed us that the high cost was due to the manual process to decrypt each device individually.

We were able to perform some audit procedures to test accuracy, and timeliness of TTC’s PRESTO revenue from PRESTO’s central system. However, there is a risk that not all transactions are properly uploaded from the devices to the subsystem and then to the central system. Therefore our audit scope is limited in the area of completeness of PRESTO revenue, as we were unable to audit this risk given the lack of data made available to us for our audit period.

One of the business requirements in the Master Agreement states that Metrolinx has to:

“capture data in a manner that permits TTC, PRESTO, and an authorized independent auditor to reconstruct, track, and analyze all elements in the transaction process from initiation by a Customer through completion of the exchange of funds between a Customer, PRESTO and TTC.”

Currently, TTC does not receive any device level data from PRESTO, despite requesting it for several years. It wasn’t until our audit request was made that they were told it could be decrypted and made available at an additional cost. Without that data, TTC cannot verify completeness and accuracy of transactions from the device level to the PRESTO systems. TTC’s Farecard team could also use this data in checking against the physical observations from their device audits. TTC has submitted a formal change request to Metrolinx for device level data and this request is currently being assessed for feasibility by Metrolinx.

It is our view that the provision of this data should have already been considered and discussed with TTC considering the requirement in the Master Agreement and the requests made by TTC. [p 100]

Although the TTC is supposed to get a lot of data from Metrolinx, it does not.

The contract requires Metrolinx to retain data for seven years and to keep information in a data warehouse for TTC to access. TTC is not accessing this information and it is not being kept by Metrolinx as required under the contract. For example, device level data (whether there was a successful tap or not and on which machine) is key to identifying ongoing equipment issues, calculating device availability, and reconciling revenue. Under the contract, Metrolinx must keep this and other contract information for seven years, but it does not. The information is encrypted and purged after seven days. This is an area where TTC needs to “drive the bus” to get the information it requires to fulfill its oversight responsibilities to citizens. [p 101]

There are many points in the overall process where the Auditor General commented on the potential for lost revenue, but both the TTC and Metrolinx appeared satisfied that controls already in place were catching the types of error, usually related to late posting of transactions, in question.

Passes and Single Use Tickets

The process for reporting and posting revenue for passes and tickets is different from that for fares charged as a user taps their way through the system. However the method by which the amount owed to the TTC is calculated is under dispute.

PRESTO staff initially advised both us and the expert professionals we engaged that these reconciliations and the related compensation have never been provided to TTC, as contractually TTC is to receive revenue for these items based solely on PRESTO’s central system. However, we reviewed the contract and there is no wording in the contract to this effect. When we inquired further with PRESTO staff, they later advised us it is not stated in the contract, but that there was informal agreement from TTC during the PRESTO roll-out in 2014 and that PRESTO’s internal process narratives include it, but could not provide anything in writing regarding TTC’s agreement. TTC disagrees that this is the agreed upon settlement method for TTC monthly passes and single-use tickets. In our view, TTC should be receiving compensation for these reconciling items for TTC monthly passes and single-use tickets and should work through this issue together with Metrolinx. It is very important for TTC to know how many sales transactions did not make it to PRESTO’s central system to get compensation for potential missing revenues.

In particular for the sale of TTC monthly passes and single-use tickets from PRESTO’s third party retailer network, there are control gaps with the cash and debit/credit reconciliations that need to be addressed. It is possible the retailer network may have some controls in this area, but there are no reconciliation reports provided to PRESTO nor included in its 3416 report. PRESTO relies solely on the sales transactions from its central system for the retailer network and does not receive any type of reconciliation report from them. In our view, this control should be put in place, Metrolinx should be providing assurance on its retailer network controls, and obtain a 3416 report from its retailer network vendor. [p 105]

Postscript

There is always a problem with gigantic reports on important issues in deciding how much to boil down and how much to leave out, let alone any commentary on the actual content. Those with a very keen interest can read the originals, but even if you have read this far, thank you.