Saturday, 2 May 2015

Why not everyone is enthusiastic about the Arrowsmith Program

Recent media reports such as this one in The Age have again drawn attention to the Canadian-based Arrowsmith Program for addressing the needs of struggling learners.


What could possibly be wrong with a program that

"....is founded on neuroscience research and over 30 years of experience demonstrating that it is possible for students to strengthen the weak cognitive capacities underlying their learning dysfunctions through a program of specific cognitive exercises"?

What parent or teacher would not want children to be able to strengthen "weak cognitive capacities", particularly if this occurs via a program of "specific cognitive exercises"? This sounds a bit like the process undertaken in pathology labs, where microbes in a particular bacterial infection are isolated under a microscope, so that antibiotic sensitivities can be ascertained, and targeted drug therapy can be prescribed.

If only that was how learning worked!

The Arrowsmith program makes liberal use of the kinds of words that are designed to hook parents and teachers and get them believing that this is a rigorous, scientifically-based intervention, offering something unique over and above what can be offered in a well-organised, evidence-based classroom curriculum.

Have a look at the Arrowsmith website, and put a dollar on the table every time one of the following terms appears:

  • cognitive
  • neuro
  • brain-based
  • neuronal
  • synaptic
  • neural sciences
  • cognitive-curricular research
  • brain imaging
  • targeted cognitive exercises
  • neuroplasticity

You'll certainly be a lot poorer at the end of this exercise! But perhaps you're not yet convinced of a need to activate your inner sceptic? Well have a listen to the videoclip interview with Dr. Lara Boyd, discussing neuroimaging studies of children and controls undergoing the Arrowsmith program (see homepage link above). Dr Boyd describes what sounds like a rigorous scientific study into the "changed brains" of children who have undergone Arrowsmith training, compared to those who have not (the control group).

What's the problem here?

The problem is that the good folk behind the Arrowsmith program have reduced an incredibly complex (and to a large extent, poorly understood) phenomenon (human cognition and its representation in the cerebral cortex) to a highly over-simplified narrative that mums and dads can "understand". It's a narrative that has strong face appeal, and encourages those without specialist knowledge or training to make a link between supposedly "known" activity in the brain and certain learning exercises.

If you were a parent of a child with learning difficulties, what would best motivate you to enrol your child in an expensive 3-4 year intervention (yes, that's right, 3-4 years)? Would it be the expectation that s/he would be performing at expected levels across the curriculum, or the reassurance (irrespective of academic outcomes), that s/he now has a thicker cerebral cortex and/or more myelinated neural pathways? I know what I'd be wanting as a parent.

To the best of my knowledge, the "evidence" in support of Arrowsmith comprises in-house research reports, conference poster-presentations, and satisfied client testimonials. Studies "in process" (as at 2014) include Effects of the Arrowsmith Program on Academic Performance: A Pilot Study - University of Calgary. One would have to wonder why a program that has existed for some 30 years, is only now at the point of collecting pilot data - yet it has been charging parents thousands of dollars over three decades, in the absence of robust, independent empirical data.

It is also notable that the first-listed publication at the link above is described as a "case study". Case studies are useful tools in elucidating the impact of a condition and also the response of a small number of individuals  ("cases") to an intervention.  Case studies are often an attractive marketing tool, as they are typically devoid of pesky statistics and other "dense" concepts that reduce their accessibility to lay readers.

In this case, clicking on the case study link reveals a PhD dissertation, which looked at 5 students, and concluded:

"Four of the five students experienced large and significant increases in cognitive, academic, emotional, and/or interpersonal functioning following their participation in the LDAS Arrowsmith program. One of the five students had much smaller gains in cognitive and academic functioning and experienced difficulties with emotional and interpersonal functioning following participation in the program".

So notwithstanding the very small sample size (not in itself inappropriate in case-study methodology), we have a scenario in which 20% of the sample not only failed to derive a benefit, but may have been adversely affected.

It should be stressed that case studies, though accepted in the scientific and academic communities as a form of evidence, are regarded as "weak" alongside other readily available, more robust methodologies that enable to us to accept or reject an intervention with far greater confidence, e.g., cross-sectional studies, randomized controlled trials, systematic reviews of well-controlled studies, etc. (See
this link for a quick summary of the evidence hierarchy). Case studies are a useful starting point, and need to be followed by larger, more rigorous, and independent evaluations. Unfortunately, it does not appear that this has occurred in the case of the Arrowsmith program.

What needs to happen?

In order for academics in education, developmental (neuro)psychology, speech-language pathology etc to be able to give unbiased, accurate advice to parents, schools, and policy-makers, peer-reviewed research is needed that controls for the fact that children in programs such as Arrowsmith receive a great deal of intensive, 1:1 time. That in itself should result in improved knowledge and skills.  Well-conducted studies that control for variables such as socio-economic status, comorbidities, and prior instructional environment are needed. It is important that gains made are tracked over time to see if they are maintained, or are an initial halo-effect. Bear in mind that when a child is significantly behind academically, interventions need to accelerate their progress relative to their typically-developing peers. Otherwise they will never catch up, let alone maintain their gains in the face of an ever-more demanding academic curriculum. We also need studies in which those carrying out outcome assessments are "blind" to which study arm a child was in. Not ensuring this opens the findings up to a range of overt and covert biases.

In the absence of such empirical data that people such as myself can draw on, it is disappointing and worrying that two Victorian education sectors have succumbed to (understandable but regrettable) consumer pressure to countenance an introduction of the Arrowsmith Program in a small number of schools (at this stage in only one of the sectors as far as I am aware).

Consumers should expect providers to go to the market with already-tested, replicated, and high-level evidence before they ask people to sign-up to an expensive intervention. We don't expect cancer patients to organise their own randomised controlled trials of new treatments, so why should it be left to schools (who don't typically have the appropriate expertise on staff) to stumble around and try to work out whether an education intervention is an appropriate investment above and beyond what they are already doing
(or could be doing)?

So - how will it be determined that the adoption of Arrowsmith mentioned above (and others that are popping up here and there) has been successful (or not)? What pre-determined criteria will be applied? Will there be any evaluations by independent and appropriately qualified researchers, or is this yet another education-intervention cul-de-sac?

Postscript, September  4, 2015: Interested readers should also check this blog post about the Arrowsmith Program by Professor Dorothy Bishop (University of Oxford).

(c) Pamela Snow 2015

15 comments:

  1. All of this is fine and well but the fact of the matter is, the program works. I hope the research catches up soon so this program will be widely available.

    ReplyDelete
    Replies
    1. If something "works", that's a position that should be supported by independent, empirically verifiable evidence. People should not be profiting from parental anxiety and the capacity of some parents to pay enormous amounts of money on the strength of biased testimonials. We would not accept this in medicine and nor should we in education.

      Delete
    2. Parachutes work. Show me a randomised double-blinded research project that supports this.

      Delete
    3. That's an anti-intellectual, straw man unfortunately "Unknown". Parachutes "work" until someone proposes something better, which will no doubt be subjected to rigorous scientific scrutiny before it is made widely available and promoted for mass use - as it should be. This is what I advocate in relation to the welfare of struggling students too.

      Delete
  2. PS Would you please update the link to Prof. D Bishop's post, I can't find it at the site you have given. Thank you.

    ReplyDelete
    Replies
    1. So sorry about my very tardy reply - but I have updated the link. You might like to have a look at some of Prof Bishop's other posts while you are on her site.

      Delete
  3. having attended this school, and I am stretching that word to its limits for the sake of inclusion, I must say that it is a year I will never get back.

    ReplyDelete
    Replies
    1. Interesting comment - would you care to elaborate? I am sure readers would like to know more.

      Delete
    2. I would like to know more.

      Delete
  4. As an adult I experienced no gains after one year of rigorous work at AS. I wonder how many other mature students share my sentiments? I envy those that benefited from the program. I wonder ,though, if they are deluding themselves??

    ReplyDelete
    Replies
    1. I am an adult who has been contemplating attending the six week summer program that they offer. It will be a financial burden and I really don't have the time to take off work for 6 weeks, however I'm desperately hopeful, but wise enough to know that because I'm an adult I may not benefit as much as younger people with more malleable brains. I'm trying to gather as much into as possible to make an informed choice before I make the leap. Might you be willing to discuss over email? I would so appreciate any insights you may have. Thank you

      Delete
    2. Yes of course - happy to discuss in a general sense, but of course I can't provide individualized advice. My email is p.snow@latrobe.edu.au
      Can you p[lease put "Arrowsmith Program" in the subject line so I don't miss it?
      Thanks :-)
      Pam

      Delete
  5. Interesting comment - thank you. It is always helpful and illuminating to hear from people you have had direct experience of programs such as these, but without the gains described in the promotional hype. One of the problems with over-reliance on testimonials is that the voices of dissatisfied consumers are never included.
    Thanks for joining in on the discussion.

    ReplyDelete
  6. I am "anonymous" for the sake of my son who attended an Arrowsmith School and wishes to remain anonymous. However I would like to ask where the randomised, peer reviewed studies are that prove that children should be grouped by age and sit all day long in the name of "education"? And yet it is generally accepted as the way to educate children. My son has benefitted hugely from the program and has gone from needing significant support in terms of extra tutoring and speech and language therapy - none of which made any difference to the underlying learning difficulties, to needing no support at all. He had a huge spread in his psych-ed assessment and every professional that dealt with him said he had a very unusual profile in having very "high highs" and very "low lows". Now in his recent Allwell testing, he achieved high average or above average across every test and in one of the tests, he had previously been in the 5th-7th percentile (on more than one occasion before you suggest it was testing error). A few months in (after leaving the program), he is managing high school with no support or interventions whatsoever and has already closed academic gaps, achieving above average for his age and grade. Sending him was not without its sacrifices and challenges - not least because he missed a big chunk of regular school during that time but I am writing as I feel saddened that more children will not have the opportunities that my son had - especially more so with the naysayers in the professional world saying it can't work. It does. Maybe not for every child - and perhaps that's where the focus on the studies should go instead of wasting time and money investigating whether neuroplasticity works or not - when there is already a lot of evidence to suggest that it does.

    ReplyDelete
  7. Hello and thank you for contributing to the discussion. First of all, I am sorry that you and your family have had a difficult journey and of course I am pleased that you seem to have found something that appears to work - that is after all, every parent's aim. I agree with you that there is a great deal (too much) in fact that goes on in education that is not evidence-based, though it could be argued that there are some broad developmental principles that underpin the way schools are organised, and of course many schools deal flexibly with this by having multi-age/composite classsrooms for at least part of the day.
    My main point in this post is that the Arrowsmith Program has been around for more than THREE DECADES and yet has not played ball with the scientific method, whilst charging parents thousands of dollars. It may well do some (or even a great deal of) good for some children - but wouldn't it be wonderful if we could find out how many, which ones, under what circumstances, at what treatment intensity, and maybe even why? That is what scientific research is all about - serving the greater good, and not just a fortunate few. Those children who are not going to benefit from this approach should be spending their time and family resources elsewhere. As with all testimonial-based approaches, the voices of the unsuccessful are rarely, if ever heard.

    ReplyDelete