Back in the early days of John Tory’s mayoralty, the 2015 budget discussions were overshadowed both by the legacy of the Ford administration and by major issues with project control at the TTC. From the Ford years, Tory inherited a mean-spirited attitude to transit spending and service cuts that the new mayor would come to reverse, for a time at least. A bigger issue, however, was the matter of runaway spending by the TTC on two major projects: the Spadina subway extension (aka TYSSE) and the resignalling contracts for the Yonge-University-Spadina subway (aka Line 1).

Even while the TTC’s CEO was coming to grips with these projects, Council passed a motion asking for a review of how the TTC was managing its business.

145. City Council direct the City Manager to issue a Request For Proposal to expedite a review of Toronto Transit Commission Capital program service delivery including:

a. a review of project management of Toronto Transit Commission Major Capital Projects in the past five years to determine actual project costs and completion dates relative to original schedules and estimated costs;

b. a review of staff reporting mechanisms to the Toronto Transit Commission and City Council related to capital project budget and completion date status; and

c. future organizational options for Transit project management and delivery of Major Capital projects related to Transit expansion and major State of Good Repair projects.

146. City Council direct the City Manager to co-ordinate the review in Part 145 above with the Chief Executive Officer, Toronto Transit Commission and to report to the Toronto Transit Commission no later than the November 23, 2015 Board meeting. [Item EX 3.4 Council meeting of March 10, 2015]

In the fullness of time, considerably later than the November 2015 date in the motion, a report from KPMG landed on the TTC Board’s agenda, the TTC Capital Program Review. This was supplemented at the meeting by a presentation from KPMG and a response from CEO Andy Byford.

The terms of reference for KPMG’s work were somewhat different from the Council motion.

KPMG’s scope was as follows:

• review project management practices at the TTC with respect to the delivery of the Capital Program, and provide recommendations to staff that will assist the organization to improve capabilities for managing capital projects and programs. The TTC Capital Program Review seeks to achieve the following goals:
• Improve the organization’s project and program management performance by learning from past experience;
• Support continuous improvement efforts underway at the TTC, including the continued implementation of the TTC Portfolio Management Office (“PfMO”) established in 2014;
• Assess project governance structure and protocols for reporting of project status, to ensure the appropriate level of transparency and accountability to project sponsors and stakeholders; and
• Provide guidance on project delivery options and project management requirements for projects of varying size, scope, and complexity. [Presentation, p. 4]

Item “a” of the Council motion asked for a comparison of actual costs and completion dates with original plans. KPMG does not provide this information, and even worse, included a table of selected projects that does not clearly explain their history (see below). There is no “deep dive” into any of the projects and, therefore, no specifics that could be tied to “lessons learned”, to practices that created the problem in the first place.

The table below is the closest the report comes to commenting on Council’s request, but read in isolation it can be misleading. It is ironic that in their presentation to the TTC Board, KPMG did not include this table and made no comment  on it. CEO Andy Byford was not so kind, and emphasized that most of the scope changes were perfectly legitimate.

Other projects included in KPMG’s review but not in the table above were:

• Fuel Storage Tank Replacements
• Subway Station Easier Access Phase III
• Surface Track
• On Grade Paving

Of these, only the Easier Access program was flagged as “Challenged” with the others as “Successful”, but the reason for the EAP’s status likely has more to do with the complexity of later stages of the work and a lack of funding than with project management and controls. [See Table 1 at pp. 20-21 of KPMG’s report]

Two of these (track and paving) are the only “Ongoing” programs reviewed by KPMG. These are fundamentally different from “Finite” projects such as the construction of a line or the retrofit of elevators to stations. I will return to this distinction later.

This is a dangerous table in that it shows an apparent growth from $5 billion to almost $8 billion in project costs, a 60% increase.  That was the easy headline in media coverage of this report, and the sort of simplistic comparison that some members of Council would seize on as symptomatic of “waste” at the TTC. Scope change appears as a “primary cause” in five of these projects, but not all such changes arise from the same circumstances.

• TYSSE’s cost grew due to inflation between the original estimate and the eventual approval of the project, and of course because the line was extended from York University to Vaughan Centre.
• There was a considerable delay between the approval and actual start of work thanks to foot-dragging by senior governments in finalizing their contribution agreements. No adjustment to the completion date nor allowance for inflation was added to the estimate.
• Politicians along the line wanted showcase stations, not the standard TTC boxes, and much of the contingency in the project’s budget was consumed by unexpectedly high bids for these structures.
• The TYSSE project suffered from poor organization with many contractors on competing deadlines. This arose in part because the prevailing wisdom at the time was that work should be parceled out in small enough pieces that multiple mid-size contractors could bid. This arrangement was more a political decision than a technical one.
• The original project did not foresee a conversion to Automatic Train Control (ATC) because that was not part of the TTC’s plans at the time. This item was added after the fact under a separate project budget, but this work added yet another layer of complexity to overlaps between many subsystem installation contracts.
• The original fleet plan for TYSSE included the continued use of a portion of the older T1 fleet on the YUS with most, but not all, of the trains coming from the new TR order. After the decision to implement ATC, the T1s were no longer suitable because of the high cost of retrofitting ATC to them, and the TR contract grew accordingly. The TTC has a surplus of T1s, but no place to use them, including the proposed Scarborough subway which will also have ATC. The planned opening date for the SSE occurs well before the planned completion of the replacement of the T1s, a project that will have to be accelerated shifting spending into earlier years than planned.
• The TTC now has a much larger subway car fleet than foreseen a decade ago thanks to the T1 surplus and cars for improved service under ATC that will not actually be used until 2019 and beyond when the ATC project completes. Additional space will be needed for the T1 replacements that will require concurrent storage space and a significantly different carhouse design from Greenwood Shops. These projects are not yet funded, and are not fully included in the cost estimates for the SSE.
• The TR purchase, like all TTC vehicle contracts, includes a base order plus an option for additional vehicles. This is a standard arrangement for procurement because the TTC never receives full funding for its plans in one go, and circumstances change causing quantities to go up, or occasionally down. The complete order now includes enough trains for the complete replacement of non-ATC capable trains on YUS, conversion of the Sheppard subway from T1 to TR operation, the Spadina extension to Vaughan and additional trains to run more frequent service once ATC is in use on YUS.
• The ATC contract evolved out of a dog’s breakfast of plans for new signalling on the YUS. Originally, the plan was simply to replace the aging block signal system on the original part of the line. However, funding for this could only be obtained by misrepresenting necessary maintenance work as something that would enable additional capacity. Eventually, the project grew to have at least two overlapping technologies being installed at the same time so that non-ATC operations could co-exist on an ATC line. This proved unworkable, and the contracts were consolidated into a single ATC project. The extra costs are a direct result of, first, underscoping the project for political reasons, and later from a failure to appreciate the technical complexity of what was being attempted.

The above are only the high points of a much more complex history (available in the linked articles), but what should be evident is that major projects interact with each other even though they are almost always discussed as if they are completely separate entities. This is a major problem in TTC budgeting and in the political context where such projects are debated. KPMG hints at this problem in its review, but does not drag any details out into the open.

The following discussion can only touch on the full report of 273 pages, and is intended to place the report more in the political context of the TTC Board and Council (and through them to the ongoing debates about TTC funding) than KPMG which tended to focus on internal issues of process.

General Recommendations

The report contains many recommendations, and these are summarized in the following table. Of these, the most important are:

• Use of a “gating” process to ensure that projects are reviewed at key points and not allowed to proceed until the requirements for that stage are met.
• Standardization of budgeting processes including risk analysis and an awareness of costs (and effects) external to the project itself.
• Improvement in data reporting to enhance the organization’s ability to track projects.
• Clarification of roles and responsibilities including those of the project management group, senior management and the political bodies to which the TTC reports.

Although these are worthy goals, some of them will run headlong into the political environment in which the TTC operates. For example, the idea of a “gated” project structure is important, and will likely work well for “internal” projects, those with little public exposure. There is certainly a need to better define what some of the TTC’s larger projects are actually intended to achieve, and to measure and control their progress against those goals. “In for a penny, in for a pound” is not a sufficient rationale for continuing with a troubled project.

During the Board’s discussion, staff from the City Manager’s Office were asked about the degree to which other City departments and agencies meet the goals set for the TTC. This, of course, was out of scope, and staff declined to answer the question. However, KPMG does have comments touching the City and notes the absence of “gating” mechanisms at the City to force project review at various stages in its evolution.

This review addresses a project before it actually launches into construction, but is silent on what happens afterward.

High profile projects acquire a demand that they be “on time, on budget” no matter what is actually happening. For years, that was the status claimed for the TYSSE and trumpeted at each TTC Board meeting by the then-chair, a sunny “good news” story. All this came unraveled when the project simply could not make its target date. Sending up a flare in advance would have required a managerial and political acknowledgement not just of current troubles, but of bad decisions that led to them.

(There is a direct parallel to the early 1990s, a time of spending cutbacks, when TTC management repeatedly claimed that the reduced budgets would not compromise the transit system. Then we had the Russell Hill crash, and the true state of TTC maintenance was unveiled.)

In a project plan, the only Key Performance Indicators (KPIs) the pols care about are the frequency of photo ops and the ability to claim “promises kept”. The concepts of gating and project review are utterly foreign to this context. Are the TTC and City Council (let alone Queen’s Park) really prepared to be given bad news, are they prepared to be told “what you want will cost a lot more and take longer than you think”? Will they accept an election campaign in front of a vacant field rather than a shiny new subway station?

Getting to that point is at least as important to good project management at the TTC and other agencies as many of the basic internal controls. If the politicians cannot discipline themselves, how can organizations for which they are responsible?

Will we see some outcome from the improvements already underway when the Scarborough Subway Extension report comes forward to Council and the TTC in December?

What is the TTC Doing Today?

At the outset of his presentation, KPMG’s Gary Webster (no, not the same person as the TTC’s former Chief General Manager) noted that KPMG’s review of TTC found it to be in better shape than they had expected when they were retained by the City. This was in part because in response to the TYSSE and ATC problems, the TTC had already begun working on better project management in house.

In 2014, the TTC established the Portfolio Management Office (PfMO) within the CEO’s office to support the development and implementation of standardized tools and processes for capital project delivery across the organization, and to provide in-house consultant services to areas of the organization that do not have the required program and portfolio management practices. The PfMO was established using existing resources through the reallocation of vacant positions and is currently comprised of 3 full-time capital positions.

Since its introduction, the TTC’s PfMO has focused predominantly on the introduction of processes for business case development and their subsequent approval; project prioritization of capital projects; standardized reporting from departments of capital project performance to the CEO and Executive Team; and improving transparency of project status reporting through the monthly CEO’s Report to the TTC Board. [Main report, TTC section, p. 4]

Some of the reporting tools produced by the PfMO have started to show up in reports to the Board notably the monthly CEO’s Report, but there is still quite a way to go notably in providing more thorough analyses of projects, comparative prioritization, and tracking of project changes during their life cycles. A real test of progress will be the degree to which the 2017 Capital Budget reflects a new approach. However, materials published to date do not seem much different from previous years, in part because work on the 2017 budget actually started in late spring 2016.

The situation is further complicated by the arrival of “new” federal funding which allows many projects to shift into “funded” status. The TTC appears to take the attitude that anything that has passed over that line does not require detailed review, and the new money allows many projects to escape detailed examination. In a worst case situation, there are piles of money and the project list expands to consume whatever is available. That has been the standard way the TTC operated for years with the only “prioritization” being to subdivide the budget into “State of Good Repair” (SOGR) and other categories. The problem was that the first priority, SOGR, consumes about 90% of the total, and “prioritization” is meaningless. A more-detailed approach is required, and the TTC is working on one, but preliminary results have not been very clear.

KPMG reviewed three groups within the TTC:

• Engineering, Construction & Expansion
• Information Technology Services
• Operations

The nature of projects within each group varies considerably and a one-size-fits-all management scheme is not necessarily the best approach. The challenge is to balance standardization and the need for ongoing controls with a recognition that at some scales, an organization can tie itself in knots following a standard program rather than getting work done. This is a common battle within any large enterprise or agency.

KPMG has its own method of scoring and ranking organizations with four tiers: informal, standardized, monitored and optimized.

It is no surprise here that the most informal of the three is “Operations” considering that this group tends to have many relatively small projects that are capital maintenance work, some on an ongoing basis, as opposed to a great deal of one-time contracted work as one would find in the Engineering group. Necessarily when work is to be done outside the organization, and is a “one of” project, the level of detail to be developed up front and contract specifications will be greater than, for example, replacing yet another few hundred metres of subway track.

KPMG fails here in doing a review of the complete project inventory (begging off because there are so many) at least to a level where the number, type and scale of projects would be reviewed to see whether the project mix typical of each group drives the accepted standard of “informality” overall. To get a sense of the number of projects, browse Appendix D of the Proposed 2017-2026 Capital Budget.

There is room for improvement even compared to KPMG’s public sector benchmark, and a big jump to “leading public sector” rank. Unfortunately, KPMG does not tell us (likely due to client confidentiality) what type of existing organization has this exalted status. In other words, where is the role model for the TTC? Is it even in Canada or North America?

KPMG observes that going to “Optimized” could be counter-productive:

The high end of KPMG’s maturity rating scale (see figure 1) identifies “optimized” as the highest standard of project management maturity. However, in KPMG’s experience, it is not applicable to public sector organizations as the return on investment is not justified. [KPMG report p. 6]

To put this less charitably, there is a point where the attempt to manage a project exceeds the value that might be achieved by the effort. Project Management as a practice gets a bad name from the frequency with which a PM group will attempt to push controls as close as possible to the zero-ROI line, if not over it. This makes the supposed benefit hard to fathom by people who are doing “real work”, and PM comes to be something to evade rather than embrace. It is self-evident that a very large project with high levels of organizational impact and risk requires far more control and discipline than a small, unimportant one. The trick is to find the right balance for the scale and complexity of a project.

Project Cost Estimation and Budgets

Over the past year, a new chart has appeared in many City and TTC reports. This shows the AACE (Association for the Advancement of Cost Engineering) classification of project status based on the level of detail in design and estimation of eventual cost. KPMG suggests:

If such a process is followed, project approvals can be staged as the project proceeds from a Class 5 estimate, to a Class 4 estimate, and ultimately to a Class 3 estimate, where the budget is approved (in line with the Stage Gate process (see Recommendation #10). By doing so, the City stakeholders will become familiar with the level of accuracy at various stages, including for example that a Class 4 estimate for a project can have an accuracy range of -15% to +50% (which may be a concept that is currently difficult to effectively communicate to stakeholders). [p. 62 of report]

Getting to a cost estimate that KPMG suggests is sufficient for budget approval requires considerably more detailed work than is now undertaken even at the Transit Project Assessment phase (the mini Environmental Assessment used for transit projects), and would entail considerably more cost simply to get to the point of saying “go/nogo” to a proposal. Debates over major projects take place today on at best a 5% design which leaves a huge margin within which the eventual cost could lie. Why only 5%? Because governments do not want to spend the money necessary to get to a more reliable level especially if multiple projects and options are on the table at once.

KPMG claims:

Inaccurate budgeting and estimating was one of the leading causes of project overruns in the 13 projects examined as a part of this review. Early in a project’s lifecycle there is inherent uncertainty in the information that makes up the estimate. This uncertainty historically has not been communicated sufficiently to manage stakeholder expectations regarding the intended accuracy of the existing estimate and how the estimate will be refined as the scope and design are advanced. [p. 60]

This claim is contradicted by the information in the project cost list above, and by the history of the more “out of control” projects, notably for the TYSSE and signalling work. The problem was less in inaccurate budgeting than in a major change to project scope, to a desire for more complexity than the budget would allow (TYSSE stations), and for a technically unsound implementation (signals). No amount of estimating can make up for political decisions to artificially constrain a budget, or to add major new scope that was not foreseen. It is worth noting that only one paragraph later, KPMG observes:

Although capital project estimating processes do not exist in the ITS or Operations groups, the EC&E group has a robust estimating process in place. For capital projects in groups that do not have a dedicated estimating function (e.g., ITS and Ops), EC&E’s specialist resources are sometimes consulted, but only used if requested. EC&E’s process clearly defines the steps of estimate preparation, the roles and responsibilities of individuals, and the multi-phase development of estimates from early project scope definition through to construction costing. The EC&E process is leading practice compared to many organizations, as attested by a peer review conducted by the American Public Transit Association (APTA). There are however limitations that prevent it from working successfully consistently.

The EC&E estimating process is not rigidly followed by all projects within the EC&E group. In some instances, projects are submitted in the annual budget process with estimates developed without input from the estimating function, either because of budget timelines or a lack of available resources. In other instances, project budgets are developed without a scope of the project having been developed which is sufficient to support generating a sound estimate. On certain occasions, these types of immature estimates are used for project approvals (which includes the establishment of the project budget). [p. 60]

In other words, the TTC knows how to estimate well and how to manage this process, but they do not always actually achieve this, in part because the function is undertaken by multiple parts of the organization. Even this change will not make up for political meddling, including the attempt to fit projects within available funding.

A further problem, one of which the City’s annual budget analyses have repeatedly complained, is the lack of integrated capital and operations budget planning, not to mention effects beyond the TTC organization.

Generally, large complex transit projects evolve into public infrastructure programs fulfilling multiple objectives such as urban renewal, improved access to underserved communities, improvement of the public realm, and upgrading aging utilities. However, the current EC&E estimating process is not designed to address broader project objectives unless they fall within the scope of the TTC’s work. Frequently, the budgets for transit projects are set before the scope of the other infrastructure components (relating to the broader objectives) are substantively defined and costed, and without risk adjustments for to deal with project unknowns. This can result in reputational damage when project costs end up higher than the original (incompletely scoped) estimate that set the stakeholder expectations.

Estimating can also involve lifecycle costs. As discussed earlier, when considering alternative procurement models, a lifecycle approach to estimating is often necessary. For procurement models that include components of operations or maintenance over a time horizon of up to 30 years, the scoping of projects must go beyond initial capital costs to encompass lifecycle costs including operations, ongoing maintenance and periodic capital expenditure (e.g. a replacement roof). Currently, the TTC capital project estimating and business case development processes do not consider initial capital costs alongside lifecycle operations and maintenance costs. In an organization with segregated Operations and EC&E departments, taking a lifecycle approach toproject estimating would represent a significant change. [pp. 60-61]

To be fair, estimating life cycle costs for assets that can last many decades involves a good deal of wishful thinking including assumptions about how robust the original asset was at the outset. For example, the “subways last a century” mythology presumes at a minimum that certain aspects of the system will last more or less forever at least on the timescale of anyone building the project. However, unexpected future events can change this such as the need to prematurely overhaul vehicles, or structural failures of subway infrastructure requiring unplanned-for repairs. Business case assumptions may prove false, or might even be manipulated to produce a desired outcome.

(As one example, every generation of subway cars the TTC buys is supposed to be better than its predecessors, and this should lead to higher reliability and a lower spare ratio, hence a smaller fleet to provide a given service level. In fact, the spare ratio continues to rise, and the reliability numbers do not always pan out as originally hoped. The problem is masked because the fleet tends to be larger than actually needed.)

Life cycle costing is an important factor in the technology debates about rapid transit modes because the very high capital cost of subways brings with it a future infrastructure operations and maintenance cost. Some parts do not begin to affect budgets until decades after a line opens (for example, the deterioration of the tunnel structure between Eglinton and Sheppard stations).

A rather bizarre outcome of the Board’s discussion, one in which KPMG’s presenter concurred, was the idea that cost estimates for projects would not be made public until well after the projects had been approved and the estimates worked up to a high degree of reliability. This idea completely misses the process by which projects are evaluated against each other, notably by a review of whether they are cost-effective relative to alternatives both within the project, and to other spending proposals before governments. In effect, the public would see pretty pictures, lines on a map, but with no idea of what this might cost to build or operate. That would be a most flagrant abuse of basic planning and business principles one could imagine.

If preliminary cost data were hidden from public view, the City could travel down a path that sounds good, only to find it has politically committed (if not budgeted for) far more than it can afford. Does this sound familiar?

KPMG appears to ignore the political context in which transit planning occurs where simply drawing a line on a map is sufficient to influence property values, win elections and skew the focus of planning debates. This may not be good fiscal planning, but it’s how things work. When the Mayor or the Premier announce a project, the plans are set in stone, and critics are dismissed as naysayers. The job of the TTC, and of any consultants advising them, is to make this process work as well as possible even if a project becomes unrecognizable. The brand survives, if not the original promise. See SmartTrack and the Scarborough Subway Extension.

Ongoing Capital Projects

Many TTC capital projects are not one-time affairs such as the construction of a new line, but ongoing major maintenance works that are treated as capital rather than operations for budget and subsidy purposes. A homeowner would be familiar with the need to perform routine fixup jobs to their building (“operations”), but from time to time, one must replace the roof or the furnace, upgrade the plumbing or renovate the bathroom. These are “capital” expenses whose value lasts for many years. If, however, you owned hundreds of houses, there would be some type of major work underway on some of them all of the time.

That is the situation with a fleet of buses or a subway network whose parts date from various eras. All buses receive a major overhaul at least once during their lifespans, and subway cars receive several. At any time, there are always vehicles “in the shop”. Similarly track on the subway and streetcar systems wears out and must be replaced, and this is done on a rolling basis across the network each year. The justification for such projects is much more basic than for a new line: if the work is not done, the system will fall apart. There is a growing backlog of “state of good repair” projects, but these are currently lumped together with no sense of how critical each might be nor of the risk involved in postponing completion.

These are also the projects most sensitive to ongoing attempts to trim the capital budget, and quite recently the TTC invented an accounting fiction that their project inventory’s cost could be reduced by 10-15%. Either the projects were consistently overestimated, or they involve work that is not required and will never be undertaken, or the TTC is trading away contingencies against unexpected problems that could trigger budget crises in years to come.

An obvious question one might ask is could this work be done more “efficiently” (i.e. “cheaply”), and the answer will vary depending on the type of project. Some improvements will occur through changes in processes (surface track, for example, is now built more robustly, but in a way requiring less labour and a shorter disruption of affected streets). Some could come through outsourcing (leaving aside issues of collective agreements), although this could bring a requirement for management of external contractors. For example, repairs to the linear induction motors used on the SRT have always been done externally to the TTC. The fleet is small enough that bringing this capability in house has never been a consideration. The subway and streetcar fleets are much larger, and the rotary motors they use have been maintained in house for as long as Toronto has had a transit system.

Management of ongoing projects differs from the more public construction jobs such as the TYSSE because there is no discrete “start” and “end” to the work except, possibly, “phases” based on the calendar or an arbitrary choice of completion percentages. If a project that is seasonal (such as construction work outdoors) is budgeted on a flat spending rate through the year, it will be “under budget” early in the year when little work is actually done, and then spending at a higher than budgeted rate (but not necessarily final cost) later  on. Some project components (notably for work that must be co-ordinated with City and other utilities’ projects) may be deferred to future years. It is important to distinguish between budget-to-actual comparisons that are simply timing differences and those representing real savings or extra costs.

Responsibility at the Political and Senior Management Levels

Many TTC capital projects are at a level well below the political radar, and they surface only when something goes wrong such as the recent fiasco with broken air conditioning units on the BD subway’s fleet of T1 cars. Even then, the public debate focuses on “how many work today” rather than “why was this overhaul not begun sooner” or “how much will this cost compared to budget”? Reporting requirements considered adequate for internal management use might not look as good at the Board level or in the media.

Among KPMG’s observations, one touches on the lack of expertise at the level of governance structures where project review is purported to occur.

Principles of capital project management and project stewardship appear to be well understood at the entities close to the implementation of projects. However, for those entities whose core expertise may not be capital projects, such as the City-TTC Executive Transit Committee, the TTC Board, or the TTC Executive, this level of expertise is not necessarily present. Accordingly, their Terms of Reference may need to be supplemented with a capital projects toolkit which may include:

• High-level training on the concepts of capital project management and stewardship;
• An overview of the relevant capital project entities, their missions, mandates, and strategic objectives;
• A list of considerations for making decisions;
• Techniques for performing due diligence related to capital projects.

These entities may also wish to consider establishing a capital projects subcommittee and ensuring it has the ability to bring in expert advice as needed. [KPMG Report p. 42]

In a rather delicate way this suggests that Council, the TTC Board and even senior layers of management are not doing, or are not able to perform, the sort of oversight one might expect. It is not sufficient for Councillors to claim that they are “too busy” and slough all responsibility off on management who could well read their role to be minimization of “bad news” rather than provision of reliable information. This was certainly what happened with the TYSSE and signalling projects. These themes are elaborated in the report (pp 42-45).

KPMG makes an important observation about how projects are measured:

Class-leading organizations are shifting away from focusing solely on KPIs that are limited to metrics around cost (cost performance index – CPI), schedule (schedule performance index – SPI) or scope (% changes). They are transitioning to reporting on value to the project’s customer or shareholder, value that is defined through the project’s objectives. This form of reporting is particularly important in the public sector, where broader objectives of a capital program, such as community building, urban regeneration or providing accessibility, cannot simply be measured in dollars spent and time saved. In the public sector, there are arguably many more stakeholders invested in the successful outcome a transit project, where impacts are felt further afield than just in the immediate vicinity of each transit station. [p. 45]

This can be turned around and argued from the point of view that managing to simplistic, uni-dimensional KPIs (Key Performance Indicators) can encourage an organization to game the system and work to maximize their scores on metrics that do not grasp the breadth of a project. Measurement against broader objectives requires, of course, that those objectives be defined and used consistently across all projects so that measures of success and benefit are comparable between them.

A fundamental issue here is that bodies charged with oversight and review should not simply sit around digesting tea and cookies while avoiding the harder work of actually understanding what projects are about, how they work and whether they are in trouble. “I don’t have time” should be an excuse that instantly disqualifies the speaker from any claim to effective oversight.

Evolution of Project Documentation

KPMG notes that the traditional method of reporting on a project within the capital budget is now changing. Historically, there has been a large collection of documentation containing, mainly, Project Summaries and Project Expenditure Summaries together with explanatory material for the more complex work. These are the “Blue Books” (two large binders) to which I have often referred in commentaries about the budget.

The following pages show the Yonge-University-Spadina resignalling project as it appeared in the 2016 budget:

The Project Summary describes the condition of the existing plant and its dire need for replacement, but the text also includes sections left over from the earlier version of this project when overlapping signal technologies were to be deployed. It also includes references to other related projects, but one would have to read through the full budget papers to discover this information. There is also a chain of related projects where new signals affect new trains which affect maintenance and storage facilities, and so on.

Replacing the signals is a “must have” project, but there is no business case beyond this nor a ranking of priority based on corporate goals and risk to the systems integrity. Obviously this is vital work, but there is a lot of “vital” work on the TTC and an ability to set priorities is essential if only to subdivide the mountain of “state of good repair”.

The Project Expenditure Summary shows the current state of project funding and spending. We can see the spending to date, and the planned future work, but almost nothing of the history of how this project evolved. Even the “budget comparison” is only relative to the most recent Council approval in 2015 and it shows no history of how this value might have changed since the original proposal. Similarly, there is little explanation for the sizeable difference between the approved 2015 budget ($78.1m) and the probable actual ($30.3m) beyond “timing”. This of course reflects the major change in project scope and technology, but also shows the limitations of this report format.

A new type of high-level project overview now appears in the monthly CEO’s Report for critical projects. Here is the same YUS signalling project as of August 2016:

This chart gives more current information, but is still dependent on two key points:

• That the reader understands the technical implications particularly under “key issues and risks”, and
• That the information truly reflects what is happening with the project.

It’s a fancy way of saying “on time, on budget”.

The corresponding chart for the TYSSE (Spadina extension) project is somewhat different:

Many sub-projects are tracking behind schedule and a few are critical pieces. Note also the wide variance in anticipated 2016 spending versus budget. This includes a combination of holdbacks that are not expected to be paid until 2017 as well as contingencies. The amount of “yellow ink” in this chart raises a flag, although the overall project outlook is “green”. This is another example of how spending viewed in a narrow time window can be misleading.

Much more detailed tracking is done at the management level, but these charts are not published. KPMG notes overall that projects managed by the EC&E group tend to have better processes already in place than for other groups although improvements are still possible.

A fundamental question for the TTC Board is “how much information is actually useful”. There was a time when the monthly report included reams of project status sheets that nobody ever read because understanding them required an encyclopædic knowledge of the TTC’s workings. That is a difficult balance for which the Board must depend on Management, and in turn Management must have adequate project tracking data. KPMG makes the point that many internal TTC processes, some manual, are required to track project status and spending, and that reporting can run a few months behind current status thanks to this complexity. Some of this will change with the TTC’s move to more modern project and financial tracking.

Data collection to produce status reports and track projects has been a long-time challenge for the TTC.

In an organization as complex as the TTC, with a functional organizational structure, there is an inherent difficulty in getting a complete grasp on the sources of and integration of data. This is particularly the case when trying to aggregate data for reporting on the capital program (as opposed to an individual project). The TTC has recently paid specific attention to customer service data to improve service delivery, but has only begun to investigate ways to use capital program data in the same way. [p. 67]

Data exist in a variety of places and formats, and originate from many sources, some of which involve duplicate effort and the chance for inconsistency. Some charts, including the main budget book pages and the new project status reports (above) are built in Excel rather than existing as a collected set of data that are formatted as needed by report templates.

The TTC does not currently have a system to manage reporting at a portfolio level. Within EC&E Construction team, daily and monthly construction reporting is completed on an online SharePoint form that can create monthly reports for project managers. Reporting above a project manager requires manual Excel-based PSRs that take manual inputs from scheduling, finance, cash flow Excel reports and Excel-based risk registers. These PSRs [Project Status Reports] are then manually consolidated into Excel-based executive dashboards or other ad-hoc reports.

The PfMO is currently investigating a PMIS solution to streamline and centralize some of this reporting. There are some transformative organizational implementations on the horizon, including SAP and an Enterprise Risk Management package, whose integration may dictate particular solutions. [p. 70]

Concluding Thoughts

The KPMG report covers a great deal of territory showing both room for improvement and, in some cases, a good understanding of and commitment to improved project delivery within the TTC. That said, the actual implementation will not be quick or easy because some recommendations have far-reaching implications for the corporate culture, for the relationship between the TTC and other agencies, and at the political level.

Any attempt to do this “on the cheap” with a handful of staff who have no real power, or inadequate resources to provide the centralized services a new management scheme might entail, will be doomed. Conversely, the last thing any agency needs is a new large group with an inflated sense of its own importance and a propensity to demand its own review and checkoff on any activity. The challenge will be to get the correct balance between these extremes.

For their part, the TTC Board and Council cannot simply exist as an ill-informed group whose primary role is to say “make it so” without understanding the implications. Management will have to push back when expectations or demands exceed what can be done. They have a duty to advise and to warn, as well as to do the best they can without compromising the transit system.

The “Gary Webster” effect is still very much with us in Toronto, and all civic staff understand that a simplistic analysis of all problems is that there are “efficiencies” to be found. This absolves Council and the Mayor of actual responsibility, the sort of situation where a back-of-the-envelope transit campaign plan can skew years of city planning.

This report was, on a motion by Deputy Mayor Minnan-Wong, referred to the City’s Executive Committee. Whether its content will be appreciated, or simply used as a basis for grand-standing, remains to be seen.

Does Toronto really want an even more professional civil service, or is it content to leave the TTC as a whipping boy, an excuse for spending less and less on transit while promising more and more?