At it’s September Board meeting, Metrolinx received a report on a strategy for tracking the benefits of their regional plan, The Big Move.  At the time, this and other business was overtaken by their slavish support for Transportation Minister Murray’s alternative to the Scarborough Subway.  Now that the panel reviewing the Transit Investment Strategy has begun its public meetings and will soon report with recommendations on how we will all pay for The Big Move, a look at the measurement tools is in order.

Regular readers here will know that I deeply distrust “Key Performance Indicators” from my own days in public sector management.  They can be an exercise in so compressing information to a single-dimensional value that meaningful oversight is impossible.  Even worse, they may be constructed to validate what management is already doing in an uncritical way.  I have approached the new Big Move KPIs with the same skeptical outlook.

The Baseline Monitoring Report and its appendices are linked from the main Big Move page on the Metrolinx site (not to be confused with the Big Move’s own website).  By looking at the proposed KPIs, we can learn what Metrolinx considers to be a mark of “success” and, conversely, what they don’t bother to measure.

The Full Report gives an overview of the scheme and how it was developed, but the details of the KPIs are found in Appendix A: Monitoring Handbook.

The report notes that circumstances have changed since The Big Move was first published with shifting demographics, population growth, and economic effects that alter how people make choices (if options are available) about travel.  This provides a context for the current set of KPIs, although there is no discussion of how sensitive these might be to major shifts in the economic and transportation context.

The table below shows the questions Metrolinx seeks to answer and, broadly, the types of measurements they would use.  [Page 7]

Having set out to define metrics, the Full Report then turns to the more general area of “progress” and poses these questions:

The Big Move sets out ten strategies with 92 Priority Actions and Supporting Policies to achieve its vision, goals, and objectives. Are we making progress towards each of the Priority Actions? Are we implementing the Supporting Policies? [Page 10]

What follows are many pages listing all of the projects in various stages of planning, execution and completion throughout the GTHA.  What is absent, however, is any quantification of the contribution each of these has made to achieving the overall goals (or will contribute when long-promised projects actually start providing service rather than photo-ops).  While all this has been going on, the progress, as measured by some of the KPIs, of actually improving transportation has been less than stellar.

A major challenge to any planner, operator or politician is that our environment is not static.  Every year, the regional population grows by about 100K, and much of that growth occurs in areas poorly served by transit.  Moreover, the travel demands from this growing population are many-to-many and do not lie conveniently along a handful of corridors where a few strategic upgrades could solve all of our problems.

If the population were static, we could talk about new lines providing real benefits.  However, when the population grows by 1-million in the decade needed to deliver major transit projects, any benefits are swamped in the losses to inaction in those parts of the region where no transit upgrades worth mentioning have occurred.

The past decade’s history is a classic cycle — a boom leading to grand announcements (including The Big Move itself), a bust in which the dreams are shattered and most plans go into limbo, and a tentative recovery where announcements of a bright future collide with “we’ve heard it all before”.  What did not stop was the population growth, and our collective timidity to invest in better transit has left the region ever more dependent, overall, on auto-based transportation.  This is not the work of evil, car-loving maniacs bent on environmental destruction; it is the simple choice of millions of people for whom the car is the only viable option.

Looking Under the Covers: The Monitoring Handbook

How are we moving around the GTHA?

The handbook makes a few points that should be obvious, but which are often lost in the focus on announcements:

• the proportion of trips by auto over two decades has risen slightly, and transit use per capita has fallen slightly;
• the way people travel is “largely correlated with the availability of rapid transit networks”;
• the subjective view of a transit service’s “may vary depending on how the question is framed”; moreover, what is attractive to one rider may be worthless to another.

That last point is particularly important.  It is not enough to point to a map full of pretty lines if the services they represent are inconvenient or uncomfortable.  A highway is there 7×24 (although congestion will vary), but a line on a transit map is no guarantee of frequent, easily accessed, attractive service.  Indeed, a major reason for the popularity of “subways” in Toronto is that they come closest to always available, frequent service while the rest of the transit system must fight against demands for “efficiency” and an abhorrence of the “waste” of empty vehicles.

[With apologies for the colour scheme.  This entire report gets very low marks for lousy graphics.]

Auto mode share gradually rises while transit mode share falls.  It is worth noting that the stronger transit growth of the past five years is not reflected in this chart.

What we do not have here are the raw numbers.  What are the population values?  How many trips does each mode get?  What are the regional breakdowns?  With regional population climbing at about 1.67% annually, transit has to gain at least that many trips just to hold its (assuming no change in overall trips/capita).  The difference between Toronto and the rest of the GTHA shows up clearly in this breakdown:

[In this chart, the legend is incorrect and the colour for “Walk+Cycle” actually shows the transit usage.  Compare to Figure 1 above.]

This is a one-year snapshot that is now seven years old.  It tells us what we already know about Toronto (taken as a whole, never mind its component parts), but does not give a time series for subsets of the 905.  That’s a challenge in its own right because during the 20 years covered by Figure 1, some locations have gone from rural to suburban status and a time series that does not show population, only market shares, would be meaningless out of context.

Updated:  A reader sent me the transit mode share by regional municipality from the TTS database.  Obviously the breakdown exists.  Why does Metrolinx choose to measure itself against a region-wide average?

Transit mode
share          1986     1991     1996     2001     2006

Toronto        25.7%    22.8%    23.2%    23.2%    23.7%    
Durham          9.3%     7.6%     6.8%     6.8%     7.4%
York           12.8%     9.6%     8.5%     8.5%     9.5%
Peel           11.5%     9.6%     9.2%     9.2%    10.2%
Halton          8.4%     7.9%     6.0%     6.0%     6.4%
Hamilton       12.2%     9.5%     8.5%     8.5%     9.6%

Trips per capita fell during the early 1990s, a period of severe recession compared to the modern history of the GTHA, and this number has not recovered even though total trips continue to rise.  The numbers are not subdivided by region, and it is possible that growing demand mainly in Toronto is making up for the lower transit trip rates of growing population in the 905.

I must comment, as a side note, on political statements dropped into what should be a “just the facts” review:

Since 2004, transit agencies have improved services due to Provincial gas tax funding, and as a result, the number of transit trips has been going up.  [Page 7]

Actually, transit service is under fire in some parts of the GTHA including Toronto as a budget balancing exercise especially after the 2008 financial crisis.  Provincial gas tax may reduce this effect, but certainly in Toronto it is a small part of the overall financial picture contributing less than $100m to a $1.5b operating budget.  Toronto has been cutting service quality to cope with rising demand.

Transit trips are going up through economic pressure (the cost of operating cars) and, on a smaller scale, the provision of new or improved services in some GO corridors.  Metrolinx harms the credibility of its “measurements” by singling out one factor among many that may have driven evolving demand for transit service.

A further piece of nonsense appears here:

One of the themes of The Big Move is working with delivery partners to improve satisfaction among all users of the transportation network, and this is strengthened through the Investment Strategy recommendation to develop a common approach to reporting on performance. Future monitoring of this will provide information on whether this goal is being achieved.  [Page 8]

All travelers, be they transit or auto users, are less satisfied today than they have been in the past.  That’s no surprise given the lack of investment in infrastructure and service.  What will Metrolinx do about this?  They will create a “common approach to reporting on performance”.  Run more buses?  The report is silent.  What we will track is how the buses we do push out of the garage perform, whatever that may mean.

Is there more choice in how we travel?

With transit market share falling, it is self-evident that transit is not keeping up with population growth.  Two KPIs are proposed to measure “choice” in travel decisions:

• transit service per capita;
• length of regional rapid transit.

This chart shows that while vehicle hours have risen over two decades, the number of hours per capita fell thanks to service cuts in the early 90s and has never recovered.  There are two big problems with this chart:

• GO Transit is not included;
• vehicle hours are not interchangeable between modes — a subway might replace many bus routes and operate on wildly different service standards not to mention capacity.

[At last! Colour!]

This chart shows us a few things, but also hides much:

• On a quantitative basis, the network stopped growing in the early 1980s with only minimal additions thereafter;
• Recent additions are mainly “BRT Light” which is little more than painting stripes on roadways and declaring a reserved-some-of-the-time bus lane.

What is worse, this chart gives no indication of the quantity or quality of service provided by all of this mileage.  We could build a “commuter line” to Montréal simply by striping Highway 401 (at least to the Québec border), but this would not represent meaningful additional to transit service for the GTHA.

The length of a transit route or of the consolidated network is no measure of its actual benefit or attractiveness.  Two trains each way per day to Kitchener-Waterloo is a very different service from half-hourly all-day operations on the Lake Shore corridor, but the kilometres of track count for the same in this measure.  It is a measure that values building more, not operating more service.

Do more people live and work close to fast and reliable transit?

The measures in this section look at population and jobs within 2km of “rapid transit”.  Ah, there’s the rub — what is “rapid transit”?  An infrequent bus service on a semi-reserved lane?  A train that runs a few times a day one way?  An LRT or subway with frequent all-day service and stops well-served by connecting routes?

This metric continues the mythology of lines on a map as a substitute for transit service.  The question is not whether there is a service nearby, but when is it there and what proportion of the travel demand does it address?  A related problem arise with locations that attract or originate traffic in many directions.  Building a subway line to a university only serves those students and faculty whose trips lie along the subway’s route and catchment area.  Simply saying “we have service within 2km of X” says nothing about how useful that service might be to the residents or workers at that location.

The population within 2km of “rapid transit” has gone from 43% to 46% from 2001 to 2006 (a lot of the data in this report does not contain recent info thanks to the cycle time of major surveys such as the census).  Without knowing just what this means in terms of service quality and usefulness, this number hard to interpret and if anything shows that nothing much was happening.

The report notes that most of the growth is due to BRT-Light in Brampton and York Region.  Sadly, when people hear about “rapid transit” coming to their neck of the woods, they tend to think of something more substantial.  This is an example of a metric that rewards the least significant of network additions just as much as those that cost orders of magnitude more to provide in capital and operating costs.

Some of the growth simply reflects new population and housing clustering in areas with already-good transit, not an active program to expand the network.  This is at best a “win” for planning and development policies, not for the buildout of transit services.

Over the same period, employment within 2km of “rapid transit” grew by 8% largely because of the new BRT-Light services.

As The Big Move and political claims for it go forward, Metrolinx and Queen’s Park need to avoid overplaying their hand by claiming that vast areas of the GTHA are near transit when that really is nothing more than the thinnest of green lines on a map.

Are we providing transportation alternatives for those who need them the most?

This section begins with a comment that shows some skew in the meaning of the question, but also alerts us to some “hard truths” about people who use transit:

A transportation system needs to work for everyone, including those who cannot drive or do not wish to. Living near a bus stop does not necessarily mean that transit is a realistic transportation choice. Transit usage depends on factors such as frequency, reliability, routing, and scheduling. We can create a general picture of transportation choice for those who need it by looking at the travel behaviour and the options available for children, seniors, those with mobility difficulties, and individuals from low-income households. There has been a gradual increase in active transportation for low-income households, but outside of Toronto, automobile usage remains the predominant mode of travel.

As I said earlier, just having a line on a map outside your door does not mean that you have “service”.  This is not just an issue for those with mobility problems, but for travellers in general.  However, for those without access to some form of personal transport, these problems are much more severe because some trips will be impossible.

A vital point here is that it is the local transit services across the GTHA where the first accessibility barrier may be encountered.  If someone cannot get on a bus, or if the stop is poorly located, or the service is infrequent and unreliable, then the most accessible regional service is of no use.  It’s that first link from home or work that makes or breaks “accessibility”.  Local transit is outside of the Metrolinx purview and Queen’s Park does everything in its power to avoid addressing service standards and funding for the local systems.

At the level of those for whom a conventional system is not an option, we must turn to Wheel-Trans.  This service, within Toronto, is 100% funded by the City of Toronto.  Outside of Toronto, accessible services vary in scope and funding levels.  Queen’s Park is great at setting rules for accessibility, but they are unwilling to fund either the capital or operating costs associated with them.  This is a gaping hole in the Big Move’s financial plans.

Metrolinx proposes four metrics to track “accessibility”:

• Mode share by transit for commuters from low-income households;
• Mode share by transit for school trips of 12-16 year olds;
• Mode share by transit for trips by seniors 65+;
• Proportion of GTHA fleets that are accessible.

This is a strange way of looking at “accessibility”.  First off, almost all transit is accessible by law, although systems have until 2025 to complete their migration.  For the TTC, the only large group of non-accessible vehicles is the streetcar fleet, and that will be complete in about five years.  This is hardly a “goal” or a “measure” considering that it will simply happen as fleets are replaced.

The larger issue for TTC is the accessibility of its stations, a project that is chronically underfunded, and for GO Transit.  Such “accessibility” is not just a question of having an elevator, but of ensuring that it is working reliably.  There is no standard requiring either 99% “up time”, or redundancy in stations to ensure 100% performance.  This would not be cheap, but my point is that there is no standard or goal against which the provision of accessibility can be measured.

As for mode share, that is not an indication of accessibility, but rather an indication of the degree to which people choose to travel by transit.  Just because they have a transit option does not mean that they will use it as other factors may form part of their decision.  “Fixing” the problem, altering their choices, may not be something than can be done by building one new transit line, but by revising the entire philosophy of what constitutes “good” transit service.  Again this brings us to local transit which is beyond Metrolinx’ world view.

The mode share does not tell us why riders made their choices nor give any hint what would change these choices.  In particular, the mode share for these three groups says nothing about “accessibility” as the term is traditionally used because it deals primarily with riders who are, at least in part, capable of accessing conventional transit services.

Are we safer as we travel?

The one metric in this section is the count of road-based fatalities and injuries in the GTHA.  Given that the majority of travel in the region is by car, this may be appropriate, but it says nothing about safety for transit riders, cyclists and pedestrians beyond an observation that as their numbers grow, things generally become safer for them.

The stats need to be subdivided by type of individuals affected and the nature and location of the accidents.  “Hot spots” should be identified and corrective measures taken to reduce events at those locations.  Reporting on a region-wide basis with no distinction about the type of accident gives no indication of what might be achieved or how stats are changing in specific areas relative to population and traffic volumes.

Are we reducing the impact of transportation on the environment?

Metrolinx proposes two measures of environmental impact:

• Count of smog advisory days in the GTHA;
• Greenhouse gas emissions from personal vehicles.

It may come as a shock to Metrolinx, but there are activities that affect air quality that have nothing to do with the transportation network.  Indeed, the largest contribution to the improved air quality in Toronto is the shutdown of coal-fired Hydro plants, as well as reductions in industrial activity and emissions in the United States.  Any discussion of trends in smog days must first look at the proportion of emissions that come from each major segment of the economy.

As shown in the report, the per capita emission of greenhouse gases from 1986 to 2006 was more or less stable.  However, the population grew substantially and so the total emitted also grew.

The amount of emissions reduced by transit will be a hard thing to estimate considering that any individual transit project contributes comparatively little to overall travel, and even with greater transit riding, there will be a backfill thanks to population growth.  This sort of number needs to have a “do nothing” comparator, and should be subdivided so that the effects of technology improvements (better mileage, smaller vehicles) and population growth are separated from reductions in auto trips made possible by transit.

Are we better connected across the Greater Toronto and Hamilton Area?

This section proposes three metrics:

• Number of urban centres reachable in 45 minutes by transit;
• Average vehicle speed on selected highways in the am peak;
• Transit mode share for passengers at Pearson Airport.

The first metric is rather dismal.  Only 1% of trips are now made between the centres.  Within the subset of transit trips, the number is slightly higher at 1.7%.  In either case, however, it is self-evident that travel patterns are not centre-to-centre.  Metrolinx proposes to use the number of centres reachable in 45 minutes as a surrogate for connectivity.  However, not all “centres” are equal with the most obvious one being downtown Toronto.

It does not matter if I can reach 15 “centres” in 45 minute if these are not places to which I am likely to be travelling.  To put it another way, if 50% of the jobs are in downtown Toronto, then 50% of the trips want to get there, although the proportions will vary from one part of the GTHA to another.  Saying that I can get to 14 other centres each of which  may only have a few percent of the total job market is another variation of the “lines on a map” problem.  Collectively, not being able to reach centres that have a small share of the total job pool may be less important than capacity constraints on access to the major centres.

Another obvious point of simply geography is that from locations on the periphery of the GTHA, it will not be possible to reach as many “centres” as one could from a more central location.  Someone living in North York is a lot closer to more places than someone living in Hamilton.

This metric just doesn’t make sense.

Average speed on highways will give an indication of that great enemy “congestion”, and it’s one way of looking to see whether all the new projects have any effects.  However, there is a good chance that many high-profile projects will have minimal effects on highway congestion because they address completely different travel markets.

This is an area where Metrolinx plans may be unmasked for contributing little to a pressing issue for many motorists whose journeys do not lend themselves to transit improvement.

Finally passenger trips to Pearson Airport are only part of the total demand to the airport district.  This metric appears designed to show how wonderful the Union Pearson Express is plus whatever new transit demand comes in via BRT systems.  However, if we don’t also address transit accessibility for work trips, we are ignoring half of the total demand.

Is transit provision in the Greater Toronto and Hamilton Area becoming more fiscally sustainable?

This metric attempts to report on “transit efficiency”, but that is a difficult term to measure.  What are we trying to be “efficient” at doing?  Carrying large numbers of riders to diverse destinations at lower personal cost may have economic benefits to workers, employers and students, but it may not look very good on the bottom line.  Recent experience in Toronto and elsewhere shows that transit funders want to cut spending on transit and this inevitably translates to less service and greater crowding, presuming that routes don’t simply disappear.

The efficiency of transit operations is difficult to report in isolation from the wider benefits of greater transit use. Greater cost can result in greater benefits, both in terms of operating costs and capital investment. This issue has often been considered political, based on whether a transit subsidy is deemed a cost or an investment, leading to debates on how to minimize unnecessary costs through the streamlining of operations.  [Page 26]

The metric presented is operating cost per passenger, and the report notes that this cost is rising faster than inflation not unlike costs of service provision in other parts of the economy.  What is missing is a discussion of the goals of a transit network, and how both operating and capital spending helps to achieve these goals.

When we talk about cost per passenger, we speak of operating efficiencies — how many people are on a bus, do service run to satisfy political demands or because there are residences, businesses and services that they connect?  But service quality is an important issue also.  The most common complaint of a motorist is that they cannot trust transit to deliver them on their journey in a reasonably predictable length of time without undue delays enroute and at a price perceived to be fair.

The billions invested in infrastructure to make that journey possible is generally unseen, and a metric that looks only at operating costs misses many other factors as well.  Depending on the length of a trip (and whether we are measuring complete journeys or “linked trips” versus vehicle boardings or “unlinked trips”) the cost per rider will obviously vary.  This will also be affected by vehicle utilization on a route and network where some routes may see strong bidirectional, all-day demand, and others may be heavy on peak period/direction travel.  Do we abandon those services that cannot run full buses and trains up to 11:00 pm?  Of course not — they are part of an overall network serving a variety of demands.

This is a simplistic metric, and one that will be dominated by the cost base of the TTC which carries the vast majority of riders.  The information needs to be reported on a granular basis and compared with the goals that we set, as a region, for what constitutes good transit service.

Conclusion

The idea that the benefits of The Big Move should be tracked is laudable, but the indices constructed for the purpose are poorly thought out to the point of being misleading or meaningless in some cases.

Much more granularity is needed to show specific benefits to which people can relate in their region, in their daily travel, rather than averages over a population of 6-million.

For specific projects within The Big Move, we have to start talking about just how each project will contribute to the network, what will it bring in local and regional improvements, will the real benefit require something more than capital construction (e.g. better operating subsidies for local services)?

Metrolinx really needs to be held accountable not just after the fact, decades in the future, for the betterment of transit, but for their planning today, and the degree to which each component of their plan will contribute meaningfully to the whole.