Saturday, 2 May 2015

Why not everyone is enthusiastic about the Arrowsmith Program

Recent media reports such as this one in The Age have again drawn attention to the Canadian-based Arrowsmith Program for addressing the needs of struggling learners.


What could possibly be wrong with a program that

"....is founded on neuroscience research and over 30 years of experience demonstrating that it is possible for students to strengthen the weak cognitive capacities underlying their learning dysfunctions through a program of specific cognitive exercises"?

What parent or teacher would not want children to be able to strengthen "weak cognitive capacities", particularly if this occurs via a program of "specific cognitive exercises"? This sounds a bit like the process undertaken in pathology labs, where microbes in a particular bacterial infection are isolated under a microscope, so that antibiotic sensitivities can be ascertained, and targeted drug therapy can be prescribed.

If only that was how learning worked!

The Arrowsmith program makes liberal use of the kinds of words that are designed to hook parents and teachers and get them believing that this is a rigorous, scientifically-based intervention, offering something unique over and above what can be offered in a well-organised, evidence-based classroom curriculum.

Have a look at the Arrowsmith website, and put a dollar on the table every time one of the following terms appears:

  • cognitive
  • neuro
  • brain-based
  • neuronal
  • synaptic
  • neural sciences
  • cognitive-curricular research
  • brain imaging
  • targeted cognitive exercises
  • neuroplasticity

You'll certainly be a lot poorer at the end of this exercise! But perhaps you're not yet convinced of a need to activate your inner sceptic? Well have a listen to the videoclip interview with Dr. Lara Boyd, discussing neuroimaging studies of children and controls undergoing the Arrowsmith program (see homepage link above). Dr Boyd describes what sounds like a rigorous scientific study into the "changed brains" of children who have undergone Arrowsmith training, compared to those who have not (the control group).

What's the problem here?

The problem is that the good folk behind the Arrowsmith program have reduced an incredibly complex (and to a large extent, poorly understood) phenomenon (human cognition and its representation in the cerebral cortex) to a highly over-simplified narrative that mums and dads can "understand". It's a narrative that has strong face appeal, and encourages those without specialist knowledge or training to make a link between supposedly "known" activity in the brain and certain learning exercises.

If you were a parent of a child with learning difficulties, what would best motivate you to enrol your child in an expensive 3-4 year intervention (yes, that's right, 3-4 years)? Would it be the expectation that s/he would be performing at expected levels across the curriculum, or the reassurance (irrespective of academic outcomes), that s/he now has a thicker cerebral cortex and/or more myelinated neural pathways? I know what I'd be wanting as a parent.

To the best of my knowledge, the "evidence" in support of Arrowsmith comprises in-house research reports, conference poster-presentations, and satisfied client testimonials. Studies "in process" (as at 2014) include Effects of the Arrowsmith Program on Academic Performance: A Pilot Study - University of Calgary. One would have to wonder why a program that has existed for some 30 years, is only now at the point of collecting pilot data - yet it has been charging parents thousands of dollars over three decades, in the absence of robust, independent empirical data.

It is also notable that the first-listed publication at the link above is described as a "case study". Case studies are useful tools in elucidating the impact of a condition and also the response of a small number of individuals  ("cases") to an intervention.  Case studies are often an attractive marketing tool, as they are typically devoid of pesky statistics and other "dense" concepts that reduce their accessibility to lay readers.

In this case, clicking on the case study link reveals a PhD dissertation, which looked at 5 students, and concluded:

"Four of the five students experienced large and significant increases in cognitive, academic, emotional, and/or interpersonal functioning following their participation in the LDAS Arrowsmith program. One of the five students had much smaller gains in cognitive and academic functioning and experienced difficulties with emotional and interpersonal functioning following participation in the program".

So notwithstanding the very small sample size (not in itself inappropriate in case-study methodology), we have a scenario in which 20% of the sample not only failed to derive a benefit, but may have been adversely affected.

It should be stressed that case studies, though accepted in the scientific and academic communities as a form of evidence, are regarded as "weak" alongside other readily available, more robust methodologies that enable to us to accept or reject an intervention with far greater confidence, e.g., cross-sectional studies, randomized controlled trials, systematic reviews of well-controlled studies, etc. (See
this link for a quick summary of the evidence hierarchy). Case studies are a useful starting point, and need to be followed by larger, more rigorous, and independent evaluations. Unfortunately, it does not appear that this has occurred in the case of the Arrowsmith program.

What needs to happen?

In order for academics in education, developmental (neuro)psychology, speech-language pathology etc to be able to give unbiased, accurate advice to parents, schools, and policy-makers, peer-reviewed research is needed that controls for the fact that children in programs such as Arrowsmith receive a great deal of intensive, 1:1 time. That in itself should result in improved knowledge and skills.  Well-conducted studies that control for variables such as socio-economic status, comorbidities, and prior instructional environment are needed. It is important that gains made are tracked over time to see if they are maintained, or are an initial halo-effect. Bear in mind that when a child is significantly behind academically, interventions need to accelerate their progress relative to their typically-developing peers. Otherwise they will never catch up, let alone maintain their gains in the face of an ever-more demanding academic curriculum. We also need studies in which those carrying out outcome assessments are "blind" to which study arm a child was in. Not ensuring this opens the findings up to a range of overt and covert biases.

In the absence of such empirical data that people such as myself can draw on, it is disappointing and worrying that two Victorian education sectors have succumbed to (understandable but regrettable) consumer pressure to countenance an introduction of the Arrowsmith Program in a small number of schools (at this stage in only one of the sectors as far as I am aware).

Consumers should expect providers to go to the market with already-tested, replicated, and high-level evidence before they ask people to sign-up to an expensive intervention. We don't expect cancer patients to organise their own randomised controlled trials of new treatments, so why should it be left to schools (who don't typically have the appropriate expertise on staff) to stumble around and try to work out whether an education intervention is an appropriate investment above and beyond what they are already doing
(or could be doing)?

So - how will it be determined that the adoption of Arrowsmith mentioned above (and others that are popping up here and there) has been successful (or not)? What pre-determined criteria will be applied? Will there be any evaluations by independent and appropriately qualified researchers, or is this yet another education-intervention cul-de-sac?

Postscript, September  4, 2015: Interested readers should also check this blog post about the Arrowsmith Program by Professor Dorothy Bishop (University of Oxford).

(c) Pamela Snow 2015

32 comments:

  1. All of this is fine and well but the fact of the matter is, the program works. I hope the research catches up soon so this program will be widely available.

    ReplyDelete
    Replies
    1. If something "works", that's a position that should be supported by independent, empirically verifiable evidence. People should not be profiting from parental anxiety and the capacity of some parents to pay enormous amounts of money on the strength of biased testimonials. We would not accept this in medicine and nor should we in education.

      Delete
    2. Parachutes work. Show me a randomised double-blinded research project that supports this.

      Delete
    3. That's an anti-intellectual, straw man unfortunately "Unknown". Parachutes "work" until someone proposes something better, which will no doubt be subjected to rigorous scientific scrutiny before it is made widely available and promoted for mass use - as it should be. This is what I advocate in relation to the welfare of struggling students too.

      Delete
    4. Okay. It works. That's all? What else can you say about it? How come only the maker of the Arrowsmith program talks, and nobody else is?

      Delete
  2. PS Would you please update the link to Prof. D Bishop's post, I can't find it at the site you have given. Thank you.

    ReplyDelete
    Replies
    1. So sorry about my very tardy reply - but I have updated the link. You might like to have a look at some of Prof Bishop's other posts while you are on her site.

      Delete
  3. having attended this school, and I am stretching that word to its limits for the sake of inclusion, I must say that it is a year I will never get back.

    ReplyDelete
    Replies
    1. Interesting comment - would you care to elaborate? I am sure readers would like to know more.

      Delete
    2. I would like to know more.

      Delete
    3. A different Anonymous : I can tell by unknowns one sentence comment above, that they would never have been to the school. The simple fact that theyve found your site, had to have read your content, understood it and then replied in the correct section, and in such a well structured and concise single sentence which involves a high degree of articulation and comprehension of the english language is undeniable evidence that they dont have a learning disability, or processing disorder. Probably never have. Just a troll attempting to muddy the waters.

      Delete
  4. As an adult I experienced no gains after one year of rigorous work at AS. I wonder how many other mature students share my sentiments? I envy those that benefited from the program. I wonder ,though, if they are deluding themselves??

    ReplyDelete
    Replies
    1. I am an adult who has been contemplating attending the six week summer program that they offer. It will be a financial burden and I really don't have the time to take off work for 6 weeks, however I'm desperately hopeful, but wise enough to know that because I'm an adult I may not benefit as much as younger people with more malleable brains. I'm trying to gather as much into as possible to make an informed choice before I make the leap. Might you be willing to discuss over email? I would so appreciate any insights you may have. Thank you

      Delete
    2. Yes of course - happy to discuss in a general sense, but of course I can't provide individualized advice. My email is p.snow@latrobe.edu.au
      Can you p[lease put "Arrowsmith Program" in the subject line so I don't miss it?
      Thanks :-)
      Pam

      Delete
    3. Did you see it work for others?

      Delete
  5. Interesting comment - thank you. It is always helpful and illuminating to hear from people you have had direct experience of programs such as these, but without the gains described in the promotional hype. One of the problems with over-reliance on testimonials is that the voices of dissatisfied consumers are never included.
    Thanks for joining in on the discussion.

    ReplyDelete
  6. I am "anonymous" for the sake of my son who attended an Arrowsmith School and wishes to remain anonymous. However I would like to ask where the randomised, peer reviewed studies are that prove that children should be grouped by age and sit all day long in the name of "education"? And yet it is generally accepted as the way to educate children. My son has benefitted hugely from the program and has gone from needing significant support in terms of extra tutoring and speech and language therapy - none of which made any difference to the underlying learning difficulties, to needing no support at all. He had a huge spread in his psych-ed assessment and every professional that dealt with him said he had a very unusual profile in having very "high highs" and very "low lows". Now in his recent Allwell testing, he achieved high average or above average across every test and in one of the tests, he had previously been in the 5th-7th percentile (on more than one occasion before you suggest it was testing error). A few months in (after leaving the program), he is managing high school with no support or interventions whatsoever and has already closed academic gaps, achieving above average for his age and grade. Sending him was not without its sacrifices and challenges - not least because he missed a big chunk of regular school during that time but I am writing as I feel saddened that more children will not have the opportunities that my son had - especially more so with the naysayers in the professional world saying it can't work. It does. Maybe not for every child - and perhaps that's where the focus on the studies should go instead of wasting time and money investigating whether neuroplasticity works or not - when there is already a lot of evidence to suggest that it does.

    ReplyDelete
  7. Hello and thank you for contributing to the discussion. First of all, I am sorry that you and your family have had a difficult journey and of course I am pleased that you seem to have found something that appears to work - that is after all, every parent's aim. I agree with you that there is a great deal (too much) in fact that goes on in education that is not evidence-based, though it could be argued that there are some broad developmental principles that underpin the way schools are organised, and of course many schools deal flexibly with this by having multi-age/composite classsrooms for at least part of the day.
    My main point in this post is that the Arrowsmith Program has been around for more than THREE DECADES and yet has not played ball with the scientific method, whilst charging parents thousands of dollars. It may well do some (or even a great deal of) good for some children - but wouldn't it be wonderful if we could find out how many, which ones, under what circumstances, at what treatment intensity, and maybe even why? That is what scientific research is all about - serving the greater good, and not just a fortunate few. Those children who are not going to benefit from this approach should be spending their time and family resources elsewhere. As with all testimonial-based approaches, the voices of the unsuccessful are rarely, if ever heard.

    ReplyDelete
  8. The "regular" education system is not designed for the greater good. There are very few studies done supporting it, yet we accept it as better than the Arrowsmith program simply because it's free ?

    Maybe if we supported schools like Arrowsmith there would be subsidy provided for families that feel the need to have their child learn in a tailored environment.

    ReplyDelete
  9. Thanks for your comment "Unknown". I agree that mainstream education needs to be more evidence-informed. If you read other posts on my blog, you will see many examples to that effect. However lack of research evidence in the mainstream does not justify or excuse the entry of a new player that lacks an evidence base AND charges thousands of dollars AND requires parents to sign confidentiality agreements. In my opinion, such a combination is simply unethical.

    ReplyDelete
  10. My 9 yo son just completed his first year of AS. So far, I have seen so many improvements that we to signed him up for a second year. He himself has observed that he is capable of understanding things differently than he used to. I have noticed that he is more confident, more self-motivated, and he is able to perceive things more clearly. And yes, his academics have improved. I appreciate that you would like more peer reviewed scientific evidence - I would too. Might I point out that your article was heavily based on your opinion, and not based on science, which is somewhat ironic to me.
    As someone who works in a scientific field, I understand that it costs a lot of money to thoroughly research a topic. Usually, there has to be a financial interest in funding the research. Perhaps that will come in time, but until then, as far as I can tell you don't have any peer-reviewed science that can disprove the methods of the program. So that begs me to ask: Why such a critical article about something that is helping children, such as my son? Have you considered the harm you might be causing the families who choose not to enroll their children based on this article? Parents who are going through this do not need a bunch of pseudo-facts, such as this article, to come up as the first item in their Google search. This article raises lots of questions, but does not provide any real answers. In my experience there are two types of people in the world: people who see a problem and fix it, and people who see a problem and cause more problems. Which one is is Barbara Arrowsmith Young? Which one are you? Based on the condescending tone of this article, I think we have our answer.
    I for one encourage anyone considering Arrowsmith to give it a shot for a year. In our experience it has been the best thing we could have done for our son.

    ReplyDelete
    Replies
    1. Could you be more specific in how it worked?

      Delete
  11. Dear Anonymous
    Thanks for contributing to the discussion. I am pleased that you have found something that seems to be producing gains for your son. I would encourage you though, to read over my responses to some similar posts above. If the AP is beneficial to some children, as evidenced by robust (ideally independent) research, then I will be happy to share that information via my blog. You ask me what evidence I have that it is not effective, but that is the precisely the problem - there is no scientific evidence about this program one way or the other.

    You are fortunate to be in a position to pay the not inconsiderable sums of money to engage with this program, but what of the parents who are not able to bankroll this commitment? They are left feeling guilty and wondering, but in reality the program may not be a panacea at all. I reiterate that this program has been around for over 30 years, but has not played ball with the scientific method.

    I don't think it's ethical to charge money and play on parents' hopes, for something that has plenty of time to be validated or dismissed. Confidentiality agreements prevent us from hearing the "testimonials" from parents whose children have not benefitted from the program.

    I could not agree more that we need better research funding in this space, but I don't see how that legitimises three decades of a successful business model without an empirical basis.

    ReplyDelete
  12. I believe that Arrowsmith has survived over 30+ years with the pioneer's hard work , struggle, courage and believe. I wish Arrowsmith to be successful in business, then more kids can be supported through and beneficial. I decided to put my kid in to count on.

    ReplyDelete
  13. As someone who is considering putting their child in the Arrowsmith Program, I would so like to see studies to either support or not support this program. I have scoured the internet, and find nothing regarding studies or negative reviews. And even most positive reviews are so general in nature. I've seen parents boast about their child's ability to read a clock. That's not enough for me considering they spend hours doing that exact thing. I've read about increased confidence many times. My daughter went from mainstream classroom and then to special day class and her confidence and happiness were drastically improved because she was able to work at her own speed, and was not the lost kid who didn't understand anything anymore. So improved confidence could just mean not being in a mainstream classroom and being lost anymore. And I hear parents say it's helped dramatically, but no detailed information. How has it helped? Will my daughter improve her speech after years of speech pathology? Will she be understood? Will she be able to do math afterwards? I don't care about testing. I want to know how it's changed things in the real world. Do they fit in with the mainstream kids now? Or are they still out of the loop? Sometimes I see two pieces of writing where one is improved, but after two to three years of schooling, wouldn't it be improved anyway? So my dilemma is do I take her out of a school she loves with art and music and some friends, a small classroom setting where she's doing better but far behind her peers or forego all of that and spend the money and see what happens? It's a question that keeps me up nights.

    ReplyDelete
  14. To the readers:
    Why are you relying on bloggers that have no science background or understanding of neuroplasticity?
    And why are you criticizing the program on its cost?
    The cost of the program in Australia is nothing compared to the thousands of dollars we spent over 7 years ( $150- 200 per hour per session) seeing psychologists, speech pathologists, occupational therapists, paeditricians and tutors with no real gain or benefit but immense stress and heartache to our family and our beautiful intelligent boy.
    In the Arrowsmith program for the fist time we are seeing real results with our son and it's only being a year. I can't wait to see what happens in the next 2!!!



    ReplyDelete
    Replies
    1. I am pleased you have found something that appears to work for your child, however that does not detract from my concerns about the program's lack of empirical research evidence.
      To set the record straight, I do in fact have a background in neuroscience. I have a Bachelor of Applied Science in Speech Pathology, a Graduate Diploma in Communication Disorders (Neuropsychology and Aphasia) and completed a PhD on acquired brain injury. I am also a registered psychologist.

      Delete
  15. All excellent questions! I hope the Arrowsmith machine can answer them for you soon - with independent empirical research, not testimonials.

    ReplyDelete
  16. As a parent who went to Toronto to try the Arrowsmith program for 6 weeks and stayed for a year, I would like to say we like other parents have spent thousands of dollars with all the 'recommended' specialists and told our child would never talk, handwrite, go to a normal school and should be put on medication. Not just once, but for 9 years!

    We had tried everything we were recommended by the specialists and I have never once asked any of the specialists we saw for peer reviews or research backing it. Nothing worked, we were desperate as were half of the students that went to the summer school were from Australia. We saw the most incredible results that we stayed a year to access the program because all the schools that offer it were full. My child now handwrites, has in a year gone from not being able to count to passing Year 6 Maths (after one year in the program and being told they would never do Maths) and will be going into Grade 7 next year and I feel confident she is going to excel academically. Which 2 years ago was so far from reality!

    I find it incredible to understand why knock it when all parents want is to help their child. Let people try it and see if themselves. We were so much poorer after following the normal route more so than doing the Arrowsmith program. You can have all the research you want but in the end, it is just research and it is the results that I was after and the Arrowsmith program proved it 100% more than other programs that has been researched, peer reviewed that we had tried.

    I ask all specialists have a heart for these kids ( and the parents who are just wanting to help their children) and not knock something you that you truly don't know about. I ask do you have a child with learning difficulties? Or is it just a profession and job?

    It is different being the parent with your child who you are desperate to want to help because you see they are bright and it just seems like it is locked in there and the Arrowsmith program unlocks it! There is research and brain scanning happening but it all takes time and money.

    It should be about helping these children overcome their learning difficulties and working together in this industry, not saying what you do is better because it has more research. If it works for some children celebrate that. Just let parents try! People thought we were crazy but now, they are amazed at the transformation and I get asked at least twice a week about the program because like us, nothing else had worked.

    The Arrowsmith program was life changing for our child and our whole family and I whole heartedly recommend it. We moved to Canada for it (and would do it again) and that is why hundreds of other families are moving even within Australia to access it. Research or not, it is outcomes we were and we got it! I am eternally grateful to Barbara Arrowsmith and thank God she was brave enough to stand up against the nay-sayers because she has changed so many kids lives and continues to do so.

    ReplyDelete
  17. I think I have replied to most of the points here previously. The key one is that if people are going to spend thousands of dollars (that in many cases they don't actually have) then it seems only fair for there to be some independent evidence that it has a better than maybe's chance of working, and that it works for reasons that are above and beyond the level of intensity and the optimism surrounding the child and family - both of which are important, but we should be able to provide them in standard classrooms, (especially if they turn out to be the active ingredients for those children for whom it produces results).

    If you read my other posts on this blog, you will see that I call for this evidence-based approach in relation to many other interventions as well, not just Arrowsmith. My line of thinking is also borne out in greater detail in this 2017 book for parents, co-authored by Caroline Bowen and me: http://www.jr-press.co.uk/making-sense-of-interventions-for-childrens-developmental-disorders.html

    ReplyDelete
  18. We have attended a 6 week intensive AS program and are looking into what we can do next for our son. I'm looking at all avenues. The problem I see with our son, and this is also information I have gathered from talking to other parents, which I realise, from your point, is completely invalid, is that our children are not A-typical and do not fit into a diagnosis. We are left to wander around the fringe of trying to find something that helps them. Every child presents with generally similar problems but to various extremes and no two profiles are the same. It is an expensive exercise but so is one on one tutoring and so are my school fees, as I choose to pay for my sons education and any interventions that are needed. I HAVE to fund these as I don't qualify for any subsidy as he doesn't have a diagnosis that will enable financial help as he does not fall into a category of mainstream disability. I can't even explain his difficulties to our friends or therapists as they try to categorise him. It is so much more than just a reading problem - in fact he doesn't have a reading problem, quite the opposite. He just doesn't comprehend most of what he reads or remembers it or can't put it into context. Anyway I digress. I don't worry about my child getting through school, although that will be a difficult experience for all of us, but I do worry about his outcomes after school. About his left path after school. Yes I understand your sceptacisim. There always has to be a devils advocate to make us look at all aspects and not just grab a rope as the ship sinks. What I don't see here is a true understanding of the children and the problems they experience. I wonder if you have taken the time to research the parents of children with learning difficulties. Surely you understand that there is not one fix that works for all and my take is that most parents I have spoken to - and there are many - that have tried everything, expensive and not so expensive. I am wondering what you suggest we do for our children. I see your criticism of the program. I don't see any other constructive suggestions. Also you mention on a number of occasions, that we, the parents that have attended an Arrowsmith program, have signed confidentiality agreements. Perhaps you need to research this as well as, quite the opposite, we are told, tell the world about your experience, positive or negative. I was asked to not talk about research that had not been completed as yet as it has not been completed. Free speech is encouraged. Please don't make AS sound like a cult. I don't know if it works for everyone but then I don't think there is anything out there that ticks all of those boxes. I just know that the curriculum, other interventions and school learning support are not working either.

    ReplyDelete
    Replies
    1. I have read all the posts here as I ponder what to do with my ADHD child. He is in the public school system and is being bullied by the administration of his public school in Toronto. I attended an orientation with the school last year and coming form a double degree background in Psychology myself, I was not impressed with the administration and their ability to sell me on the system. I asked for metrics and for a few references I could speak with and nothing was given to me of any value. With that said, no one so far has give me any alternatives and I am desperate to find one (just one). MY child is border line ADHD with reading LD's and needs emotional support as well as physical exercise for him to engage. I even started an after school and camp program to address the fact that many schools punish children with ADHD and behavioral challenges instead of emotionally support them. The school system is comprised of teachers who want children to adapt their methodology rather than the other way around. Even without empirical evidence I would rather my child receive the understanding and support of a group of people who understand the condition and use proper positive reinforcement techniques to clear the negative cobwebs and allow some available brain power to be utilized toward engagement and therefore learning. I am not sure if Arrowsmith is the place for this - I sure wish I knew because I am DESPERATE for a solution. But I have to say if it is the only alternative to the public system my child has endure for 9 years - then I would beg borrow and steal to place him there. If anyone has a better alternative I sincerely hope you will help me by sharing as I need to make a decision soon! Thank you!!

      Delete