Angry in the Great White North

A report from the CBC of an "unmitigated disaster" in a pilot project to study the effectiveness of electronic anklet bracelets to manage parolees:

A federal pilot project to outfit parolees with electronic anklets in hopes of keeping track of them and deterring further crimes has been a costly failure, according to a corrections expert.

"The whole fact is that the program was an unmitigated disaster," said Paul Gendreau, a professor emeritus at the University of New Brunswick who is internationally known for his research in corrections and electronic monitoring.

Nearly two years ago, then public safety minister Stockwell Day announced the pilot project with great fanfare as part of the Conservative government's tough-on-crime stance.

Maybe you need to be someone with training in the scientific method, but this just doesn't read right.  An experiment is not a disaster, unmitigated or otherwise, if it yields useful information, and this one seemed to provide all sorts of interesting results:

The Correctional Service of Canada review of its own program found the technology was faulty and often failed to accurately pinpoint a parolee's whereabouts.

For example, there was only one valid electronic anklet alert out of 19 where a parolee had actually tampered with his anklet strap. Most of the false alarms were due to equipment sensitivity and hardware or software issues. About one-third, or six cases, were caused by accidental jarring at work or during other activities.

And all seven alerts that parolees had tampered with the device itself turned out to be false.

Another problem was so-called GPS "drift," when the monitoring map inaccurately identified the parolee's location by a difference of up to 200 metres, requiring a program reset to sync it back up.

Parolees also complained of an overwhelming number of phone calls about technical issues, such as telling them to charge their batteries to keep the device from emitting alerts and recalibrating the GPS to fix drifting. At least one parolee received more than 30 calls in a month.

Clearly not ready for prime time, but then a pilot project is done early, essentially as a mockup of the real system you propose to build, to establish that it is even possible, and to reveal early on what the potential problems are that will need to be solved.

That is different from, say, a beta test, which is when you take a nearly completed and thoroughly tested system, and put in into the field for live trials.  The expectations are much higher in a beta test.  Indeed, a beta should work nearly flawlessly.

Now to be sure, people are not always strict in their use of terms like pilot, beta, shakedown, and so on.  But according to the Corrections Canada report, a pilot in the true sense of the word is what was executed:

This evaluation was initiated in response to a request for an analysis of CSC's Electronic Monitoring Program Pilot (EMPP) and constituted an implementation evaluation intended to examine the progress of EMPP to date and establish a foundation upon which the design and delivery of EMPP could be realigned. Treasury Board standards for evaluation were used to examine the project's continued relevancy, implementation, success, and cost-effectiveness.

But Paul Gendreau treats this as an actual experiment:

Gendreau says the program was poorly orchestrated, contained too small a sample size, didn't properly collect data, and experienced too many technological breakdowns.

"This [electronic monitoring] project that they ran was so expensive that they would have been better off just to keep people locked up in jail," said Gendreau.

An "experiment" is run to stricter standards than a pilot.  A falsifiable hypothesis is tested under blinded conditions using a randomized set of subjects of sufficient size to yield statistically significant data.

I wouldn't get too upset about a small sample size in a pilot project.  It's just not that important in a pilot.  Indeed, a pilot project can be run with a sample size of exactly one, as long as no one is expecting anything more than proof-of-concept.

Still, Corrections Canada was trying to extract some actual conclusions, so this pilot certainly had some of the qualities of an experiment, so we ought to set the standards higher.  To be fair to Corrections Canada, the pilot-slash-experiment started with 46 volunteers, but only 9 stuck with the bracelets to the end.  That in itself is an interesting data point.

But Gendreau's comment that Corrections Canada "would have been better off just to keep people locked up in jail" instead of running the pilot is weird.  No one was released from prison in order to run this pilot.  Everyone involved was a volunteer. 

As for the cost, the pilot program burned a whopping $856,096.

What of Gendreau's revelations of a poorly orchestrated pilot with too few volunteers that experienced difficulty in collecting data and in getting the technology to work?  How did he figure this out?  Well, he read the Executive Summary, like I did:

It should be noted that the current evaluation has a number of significant limitations. More specifically, the sample size was unavoidably small due in part to EMPP being designed and implemented to monitor a maximum of 30 offenders at any one time during the one-year pilot. Additionally, only 9 offenders accepted to participate in the evaluation interviews and, thus, these offenders' views cannot be considered as representative of all offender participants in the project. Furthermore, there were some challenges encountered during the course of the evaluation due to the quality of data maintained regarding the EMPP participants. For instance, it was difficult to determine the exact number of offenders who participated in EMPP during the time period of interest, as there were inconsistencies regarding the list of participants, start and end dates of participation, and whether an offender was referred but refused to participate in EMPP.

FINDING 4: There were challenges associated with the reliability of the technology used in EMPP with respect to the sustainability of a charged battery (e.g. time to charge, duration of charge), the device (size, comfort and visibility), drift, and frequent false tamper alerts.

FINDING 8: The majority of respondents described several challenges facing the National Monitoring Centre, including its location, the condition of the centre and NMC operators' knowledge of the geography of EMPP coverage area, as well as technical and communication difficulties.

FINDING 10: CSC developed and trained staff and partners who were responsible for the application and/or removal of EM devices, and the monitoring of offenders. However, there were several challenges in the collection, storage, analysis and reporting of data received in the monitoring centre.

Did Paul Gendreau regurgitate what the Corrections Canada report already described in great detail (in the Executive Summary, of all places), and then just added "unmitigated disaster" in the front of it?

Seems like it.

So who is Paul Gendreau?  Well, he seems to be a guy who makes it his business to not actually do his own studies, but instead regurgitates other people's work and adds a few comments of his own.  It's called meta-analysis.  This from Paul Gendreau's biography at the University of New Brunswick website:

Dr. Paul Gendreau is without doubt a top-flight researcher who has received recognition throughout North America - and indeed throughout the world - for his studies into the criminal justice system.

In the past, and today, many approaches to criminal justice policy are based on beliefs and prejudices instead of on learning from past practices. Paul Gendreau has been successful in bringing an empirical approach to the policy domain of criminal justice, one in which we learn from research investigating the success or lack of success of criminal policy initiatives.

Over the past decade, Dr. Gendreau has analysed and combined the results of earlier studies, using a statistical technique called meta-analysis, to obtain an accurate description of research findings. This research technique has involved synthesizing the findings of literally hundreds of research articles. From this synthesis he is able to advise policy-makers on the factors that make the criminal justice system work.

Meta-analysis.  Uh-huh.  Right. 

OK, meta-analysis is legitimate.  You take multiple studies, and combine the data using rigorous statistical techniques.  Indeed, meta-analysis can help address problems that occur when sample sizes are too small, like in the Corrections Canada pilot project.  A researcher like Paul Gendreau combines this small study with others in a way that helps bolster the conclusions.

Meta-analysis, on the other hand, has problems, since by its very nature, it is not a blinded study.  Real scientists do blinded studies.  Meta-analysis can be described as armchair science:

The most severe weakness and abuse of meta-analysis often occurs when the person or persons doing the meta-analysis have an economic, social,or political agenda such as the passage or defeat of legislation. Those persons with these types of agenda have a high likelihood to abuse meta-analysis due to personal bias. For example, researchers favorable to the author's agenda are likely to have their studies "cherry picked" while those not favorable will be ignored or labeled as "not credible". In addition, the favored authors may themselves be biased or paid to produce results that support their overall political, social, or economic goals in ways such as selecting small favorable data sets and not incorporating larger unfavorable data sets.

If a meta-analysis is conducted by an individual or organization with a bias or predetermined desired outcome, it should be treated as highly suspect or having a high likelihood of being "junk science". From an integrity perspective, researchers with a bias should avoid meta-analysis and use a less abuse-prone (or independent) form of research.

Does Paul Gendreau have an agenda?  I don't know, but calling a study that forthrightly described its limitations so as to warn the reader to be careful when drawing conclusions or in making policy decisions an "unmitigated disaster" is a huge red flag.

Another red flag is the "rock star status" that Paul Gendreau seems to enjoy, which is curious for guy whose job description says he analyses other people's work:

The success of Dr. Gendreau's research is shown by the esteem in which he is held around the world. Institutions and organization in a variety of jurisdictions have invited him to give advice on their criminal justice systems, and he has received numerous awards from national and international organizations. Other indicators of his phenomenal success include the number of research grants and contracts he has received and the number of undergraduate and graduate students who have studied with him while conducting thesis research. The sheer number of publications he has to his credit is also most impressive.

OK, in my estimation, the guys who actually ran a working study of the program, and then wrote up an honest report on what worked and what didn't so that future studies can address the shortcomings are the guys we ought to be giving credit to.

Let Paul Gendreau do his meta-analysis thing once the working researchers have collected enough data and published enough reports and peer-reviewed papers.

I really don't understand why Paul Gendreau is so worked up over this.  Framed the way it is, it makes for a fun story for the CBC, and looks bad on Corrections Canada, and yes, politicians tend to oversell these things way to early and set expectations too high, but really, Gendreau's attack seems unjustified.

Unless the fun story is the justification.