Thursday, March 20, 2014

Updated: Survey results appear to be inconsistent

Update (March 24): I misinterpreted what "we report on the percentage of survey respondents who give a rating of 6 or higher" meant. It's not a subset of the sample, but it's exactly how it's described: a percentage of those who rated 6 or higher. So, yes, about half of females feel safe waiting for a bus late at night. As I looked through some of the summarized results sent to OC Transpo, which were made available today, from the survey research firm Core Strategies, some of the numbers regarding service weren't consistent with what OC had presented. A rating of 7.3 was given to route planning in OC Transpo presentation slides, but in the newly released document, a 6.7 rating was listed. Stats on safety appear to be consistent, so there's less concern with the results of safety questions now from what I can see. Summary tables of each survey question were published via the Sun's Jon Willing.

OC Transpo presented its 2013 survey results on customer satisfaction to the Transit Commission on Wednesday. The transit agency received a rating of at least "good" from 80% of transit users in a sample of 1525 transit and non-transit users in Ottawa. It's the highest proportion since 2008, when the winter strike began in December. The telephone survey, which has a margin of error of 2.5 percentage points overall and 4 percentage points for transit users, was conducted in November and December.

Ratings on various aspects of service quality rise compared to the past few years. Route planning increased by nearly a whole point in one year (from 6.4 in 2012 to 7.3 in 2013), even though there have been no major routing changes since "route optimization" in 2011. A 7.3 rating for planning, is in fact, the highest in any one year going back to 2008.

This year, OC Transpo has included more questions about safety, which they keep saying is their number one priority. The statistics on customer feelings towards safety are not based on the sample, but on a subset of it. Only those who gave OC Transpo a favourable rating on safety were counted: "We report on the percentage of survey respondents who give a rating of 6 or higher" (on slides 35, 40, 44). So, anyone reporting a "5" or less are inexplicably excluded. It means, for example, the proportion of female customers who feel safe waiting for a bus late at night is probably not 49% as stated. It's very likely to be much lower when the less satisfied respondents are included. Percentages in the high 90's are probably far away from the true values too.

A whole survey section is dedicated to safety and security and somehow, meaningful statistics are hidden from public viewing.

No comments: