r/ABA • u/twelvefifityone • Jun 23 '23
Journal Article Discussion ABA is based on research, but there's a huge bias in research toward publishing only successful interventions.
I've never read an article where the intervention failed to have an effect. All the research seems biased toward affirming a researcher's original intervention. There might be 100 studies on showing Intervention A is effective, but we have no information on if there are 1000 unpublished studies showing that Intervention A is not effective. We'd have no way of knowing because non-effective research is not published.
24
u/newyorklogic BCBA Jun 23 '23
Me and a colleague argued for the “journal of failed interventions “ for years. First as a joke ( mainly due to his doctoral thesis falling into this category) but then as something that could be invaluable to the field.
7
2
u/GirlLunarExplorer Jun 24 '23
Little bit different but Spectrum News publishes a "null and noteworthy" post every week or so on its blog: https://www.spectrumnews.org/?s=Null+and+noteworthy+&orderby=newest
14
u/mynameisnotmorgan Jun 23 '23
If you’re curious— looking into the replication crisis may be up your alley. It’s a HUGE issue across the entire field of science.
3
u/Due_Guidance408 Jun 24 '23
The replication crisis is real, but behavior analysis research is probably uniquely suited to guard against this due to using repeated observations of a smaller group of participants, using single case designs, etc... All of this as opposed to what one sees in more mainstream personality and social psych literature, in which participants fill out (self report) on survey instruments, inferential statistics are used, etc... I would guess that the further away from direct observation and measurement one gets, the higher the likelihood of failing to replicate a particular finding. Also, I'm not saying that the latter research designs are wholly inappropriate or not useful.
1
u/davinia3 Jun 24 '23
While true, ABA takes it way further than the rest of the field. This cognitive distortion gets radically intense around ABA practices because people don't want to believe it's harmful.
7
u/ExaminationNice616 Jun 23 '23
I majored in Pharmacy and actively participated in research (this was before moving to the US and pursuing ABA) and a lot of times negative or unexpected outcomes are not published because it does not interest the magazines. Once we were testing pesticides and most of the flies survived. I said well can't we still publish this unsuccessful result? Now we know this combination of flavonoids doesn't work and we can save others the time of researching. My tutor said I could put together something but to not get my hopes up. Science magazines just don't want to publish that unless your hypothesis is to disprove or demerit something in the first place.
5
u/SharpBandicoot4437 Jun 24 '23
I actually saw Hanley yesterday and he was saying exactly this. For every one successful intervention article, there are 100 unsuccessful ones that are kept in a file cabinet somewhere because there is a huge bias towards exciting articles.
4
u/Jason_C_Travers_PhD Jun 24 '23
Relevant paper I published with a colleague that folks might find interesting: https://www.dropbox.com/s/r0vexgvof1zx89n/Travers%20%26%20Tincani%20%282017%29%20Studies%20without%20Experimetnal%20Control.pdf?dl=0
4
u/BehaviorProf Jun 24 '23
If you look closely at the conditions that are tested in a reversal design you will see that very often there are two or more that are not effective before the researchers hit on something that does work. This is where the "unsuccessful" interventions are published, right next to what does work.
3
u/Eager-Emu Jun 23 '23
Yes!!! This 1000 times!!
I literally was talking about this to my mom the other day! I WANT to know which interventions didn't work, with what type of client characteristics, and possible reasons why it didn't work. We can learn so much from when things don't go as planned. I think it would also help to know that when you applied a specific intervention and it didn't work like it did in the research, that doesn't necessarily mean you did it wrong!
We should absolutely have access to articles of when interventions did not have the desired outcomes!
1
3
u/melsar Jun 24 '23
That and also the bias of having participants with generally more skills. I have a lot of research to demonstrate progress but only in children who are on the more “skilled” end of the spectrum. Much less research exists on the more “severe” end of the spectrum. Also lack of research on the core deficits of autism and treatments for those areas alone
3
u/Mitteer Jun 24 '23
That's why approaches like CCCS are so important: https://doi.org/10.1002/jaba.691
3
u/Mr-Seabreath BCBA Jun 24 '23
I take it this JABA article doesn't count, along with it's more recent replications?
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1311997/
On a serious note, this 2018 "replication" of the Unsuccessful Treatment of Writer's Block, published in BAP, is interesting, and shows the professional community agrees with you that more replications and more failed studies need to be published.
https://link.springer.com/article/10.1007/s40617-018-00290-w
While BAP's editors, at least in theory, recognize this publication deficit, I have not found any serious published articles lately that discuss actual unsuccessful interventions. While unsuccessful interventions are less helpful to the typical practitioner, there should be a place for them that is well known so you can check if people have had failures with interventions you may be thinking of attempting. I believe it would be good for ABAI and for the typical ABA practitioner to have an official journal where such studies are published.
7
Jun 24 '23
Not to mention the lack of diversity in those who are studied. 🙃😒
2
u/twelvefifityone Jun 24 '23
To be fair, ABA researchers usually don't seek out participants. Participants go to them for behavioral health.
6
Jun 24 '23
That isn’t fair though. I mean it’s something that’s better than nothing I guess . But if we look at what it takes to seek out a behavioral health professional including education , money , transportation, language, trust, time and well the list goes on, some families may not have access. For example I worked witch a couple teens who were homeless (in the public school system ABA class) and their families didn’t have any access to services due to the above barriers . So where does that leave them😭 ?
3
u/isolatednovelty Jun 24 '23
Yep. Socioeconomic and gender disparities are all over our research. It's fine if it's proportional to the population, but it's not okay to forgo study on specific demographics due to ease and ability. At least it's not right to me.
4
u/Independent-Blood-10 Jun 23 '23 edited Jun 24 '23
I would like to add one more piece to this. Alot of these studies/interventions are done under ideal perfect conditions,with very well trained staff. No way are they replicable in a public school.
When I saw iwata for two day conference all of his stuff was done in his lab, great results but the average practitioner cannot do that
2
u/dangtypo Jun 23 '23
Not to mention most “peer-reviewed” research may not even be thoroughly reviewed. One of my undergrad professors was a Ph.D and had multiple articles published. She said a lot of journals just approve articles quickly from people who they’ve reviewed a few times and approved without much issue.
4
u/Due_Guidance408 Jun 24 '23
I'm curious to hear what journals your professor was referring to. I'm not an academic, but have many friends and colleagues who are, and the peer review process for journals like JABA, Behavioral Interventions, Perspectives on Beh Science, etc... is pretty rigorous.
1
u/Vast-Chemical-4434 Jul 10 '23
JABA is supposed to be pretty good. Has an impact factor of 2.1. That is dismal to be honest.
1
u/Due_Guidance408 Jul 11 '23
Impact factor is a different topic than peer review. And there are probably high impact factor journals with less rigorous review processes, and vice versa.
1
u/Vast-Chemical-4434 Jul 11 '23
I know what I am talking about. I’ve published more than 15 articles, all in pretty high impact biology (neuroscience) journals. But an article (esp on human behaviour) that can be published with a n = 1 is … a case study. It’s like someone writing a diary at night. Nothing else.
But, I am still looking for the most cited celebrated article in ABA that is not flawed. Can someone point me out to it?
2
u/Due_Guidance408 Jul 11 '23
I was answering a question about the peer review process (again, in the flagship journals of ABA, it's fairly rigorous, while I'm sure exceptions exist)
1
u/Vast-Chemical-4434 Jul 11 '23
Thank you for your reply. Yeah I mean, can you please share an article that you think is the most rigorously vetted? Happy to take this on chat because I am really interested to understand and make an informed decision.
2
u/Due_Guidance408 Jul 11 '23
Here's a few:
Eldevik, S., Hastings, R. P., Hughes, J. C., Jahr, E., Eikeseth, S., & Cross, S. (2010). Using participant data to extend the evidence base for intensive behavioral intervention for children with autism. American Journal on Intellectual and Developmental Disabilities, 115, 381-405. doi.org/10.1352/1944-7558-115.5.381
Howard, J. S., Sparkman, C. R., Cohen, H. G., Green, G., & Stanislaw, H. (2005). A comparison of intensive behavior analytic and eclectic treatments for young children with autism. Research in Developmental Disabilities, 26, 359-383. doi.org/10.1016/j.ridd.2004.09.005
Howard, J. S., Stanislaw, H., Green, G., Sparkman, C. R., & Cohen, H. G. (2014). Comparison of behavior analytic and eclectic early interventions for young children with autism after three years. Research in Developmental Disabilities, 35, 3326-3344. doi.org/10.1016/j.ridd.2014.08.021
Klintwall, L., Eldevik, S., & Eikeseth, S. (2015). Narrowing the gap: Effects of intervention on developmental trajectories in autism. Autism, 19, 53-63. doi.org/10.1177/1362361313510067
Stanislaw, H., Howard, J., & Martin, C. (2019). Helping parents choose treatments for young children with autism: A comparison of applied behavior analysis and eclectic treatments. Journal of the American Association of Nurse Practitioners, 32(8), 571-578. doi.org/10.1097/JXX.0000000000000290
Steinbrenner, J. R., Hume, K., Odom, S. L., Morin, K. L., Nowell, S. W., Tomaszewski, B., Szendrey, S., McIntyre, N. S., Yücesoy-Özkan, S., & Savage, M. N. (2020). Evidence-Based Practices for Children, Youth, and Young Adults with Autism. The University of North Carolina at Chapel Hill, Frank Porter Graham Child Development Institute, National Clearinghouse on Autism Evidence and Practice Review Team. https://ncaep.fpg.unc.edu/sites/ncaep.fpg.unc.edu/files/imce/documents/EBP%20Report%202020.pdf
1
u/Vast-Chemical-4434 Jul 11 '23
Okay so the last one is not really a peer reviewed paper. So I am going to have to ignore it.
The rest are quite tiny in terms of n numbers. Two studies (both from Howard JS) is the same cohort of children. Obviously the conclusions are similar. Also we take more mice for tests in our behaviour studies. Also there is no description of statistical power in the studies. I’d call both these studies quite flawed.
The Eldevik and Klintwall are the exact same cohort of subjects and the same folks who publishing the papers (there seems to be a theme to this). When I looked deeper, it’s a meta analysis of a meta-analysis drawing the same conclusion from the same set of folks. Very very fishy!
Looking forward to a counter though. One paper which is not flawed would be awesome to see.
2
u/Due_Guidance408 Jul 11 '23
I am not looking for a debate, so I'll close with a few thoughts and you can take or leave them. With vulnerable human subjects who have critical developmental periods, it is nearly impossible to do a large N, randomized control study. The ethics of assigning participants to a treatment as usual group vs. an ABA group without parent input is dicey. Measurement of progress also has challenges due to the heterogenous nature of ASD. As you know, there's no blood panel, brain scan, EEG reading, etc... that can measure ASD severity. ABA treatment is also long and researchers have to control for development as a potential confound. Long story short, there is no perfect study on the treatment of ASD to my knowledge, nor will there likely be one.
It's kind of like what I think Churchill said about democracy. To paraphrase, ABA is perhaps the least-worst intervention for ASD that we have.
→ More replies (0)
-1
u/Academic_Grade_5672 Jun 24 '23
I quit aba after studying to become a BCBA and getting into debt and job insecurity due to cancellations. What a scam of a industry
2
u/TheZambianBCBA Jun 24 '23
Oh wow. I'm sorry to hear that. Did you only work with children with private insurance?
-6
Jun 24 '23
I’m sure ABA, as the pseudoscience it is, would be under much more scrutiny than it already is if they did share all the failed trials. All the PTSD, depression, and anxiety that comes from ABA therapy. It would shed light on what’s really going on, forced coercion by using tactics that they use to get a confession out of someone. But instead of a few days, it’s their childhood, taken from them. If you think full physical prompting is science, you’re mad. Physically prompting, or forcing someone to do it the “right way” sounds pretty fucked up. As a former RBT, I drank the koolaid, I thought this was helping. It took one month to realize this is ableism. I get it, the parents want there kids to be “normal” but there is more humane ways to go about this. I wouldn’t send my dog to a kennel for 40 hours a week, knowing they were being put in uncomfortable situations, so much so, they scream and meltdown. But hey that’s ABA, it’s not up for the kid to decide.
6
u/anonymoustexas123 Jun 24 '23
Hey - BCBA here.
I hear you. And I’m sorry about your experience. That’s not okay and there’s a lot of practitioners that are trying to help move the field in a more compassion and ethical way… but I recognize there was BCBAs out there that are still utilizing “old school” ABA which is so disappointing.
Physical prompting for a motivation deficit is never okay. Children should assent to physical prompts for a skill deficit and the least intrusive prompt should be used. Ethical ABA shouldn’t work to make a child mask or be neurotypical - but help them access things that make them happy and live their fullest, most independent life possible.
Thank you for sharing your opinion- we can’t continue to grow without recognizing how much work is still left to do, and your negative experience shines light on that there is still sooo much work left to do.
3
u/isolatednovelty Jun 24 '23
Thank you for making this into a positive. It's definitely not okay and sometimes it's hard to be grouped with such ridicule as a person who hates conflict and does no harm. I'm always worried I'm going to do something I'll cringe at in a few years, but I guess that keeps me mindful of what I'm doing. I'm nervous but excited to be a BCBA someday so I can share alll the ethical practices with my RBTs.
2
u/anonymoustexas123 Jun 24 '23
I spent some time in 2018 diving into criticism of ABA and wouldn’t let myself reason it away or ignore it. It was my goal to fully embrace the criticism and listen to learn to be a better practitioner. It was not easy but it definitely de-sensitized me to the jerk reaction it is easy to feel. ❤️
Your mindset is going to go along ways for your career. Give yourself grace- you will look back and regret somethings but if we continue to embrace feedback and practice with a know better, do better mindset- you are helping to move our field in the right direction. 🫶
1
u/Admiral_Swoon Jul 03 '23
i know this is old but im currently doing rbt training for my job and am curious about the criticisms you found? I’ve heard from lots of people they don’t like aba, and we’ve had some bad behaviorists at our facility
1
2
u/Awkward-Deer1074 Jun 24 '23
Agree! I've been saying this from the time I started studying ABA. There are strategies out there that are not being used because it goes against the interventions the field has chosen to adopt. How are we truly helping our clients and advancing the field?
1
u/ThomasEdmund84 Jun 25 '23
I think this is mischaracterising the situation somewhat.
Yes there are replication issues, and positive finding bias in publishing - however on the flipside if there are 1000s of studies showing an intervention commonly understood as successful as ineffective then you would in face expect someone somewhere to publish.
It's not like studies are produced by robots with no thought or context - and say even if negative studies aren't published that doesn't mean that academics can't produce correspondence or other media.
What I'm trying to say is that if you have 100s of studies on the efficacy of an intervention - its not entirely accurate to portray this has a black hole that likely contains 1000s of negative studies. scepticism is healthy but to be fair you need to be sceptical of sceptics too
1
u/Vast-Chemical-4434 Jul 10 '23
Well okay, so there is obviously a giant replication crisis. This is really bad for biology (much less for chem and physics). And there many many nice reviews and discussions on this in journals like Nature, Cell and Science. One of the worst affected in biology is: ….. behavioural data. (I am talking about mice/rats and not humans yet).
I was looking at papers in ABA. I am shocked!!! Most papers have very very low n numbers. That in itself for me just means, it’s likely to be BS. I would not believe papers with anything less than 400-500 people, placebo controlled, double blinded and with robust statistics.
2
u/twelvefifityone Jul 11 '23
ABA research often uses "single-subject" research methods. Usually ABA cases are so unique that many controls are difficult to find. There are several experiement designs but here is a simple one that kind of exemplifies how a subject can be their own control. You measure a subject's behavior at baseline, you administer your treatment, you take away the treatment and measure, and then you administer the treatment again (ie. an ABAB) design. If there is elevated (or decreased) behavior only during the intervention, then it is very likely that the intervention is responsible.
It would not be adviseable to apply somewhat generalized research from conventional biology/psychology control groups to subjects who are very unique (as many are with developmental disabilities). are.
1
u/Vast-Chemical-4434 Jul 11 '23
Thanks for engaging me in this discussion. So in that example you gave, the conclusion itself is flawed. It’s like saying: someone has a headache. You gave him asprin, and the headache went away. Then you take them off asprin, and the headache comes back. So you conclude that the headache is due to lack of asprin?
1
u/twelvefifityone Jul 11 '23
In that case the headache went away because of the aspirin.
1
u/Vast-Chemical-4434 Jul 11 '23
Right. But in that case study example, that’s what you would be concluding. That the headache was caused due to asprin. “You take the treatment away and see the behaviour come back and it’s likely the treatment reduced the behaviour and hence behaviour treatment is needed/helped”.
1
u/twelvefifityone Jul 11 '23
You can't conclude that the headache is due to a lack of aspirin because the headache is the initial condition- the baseline condition. You don't know what preceeded it. You only know that in the presence of aspirin, the headache goes away. Also, I shouldn't have used the term behavior, i guess. A headache isn't really having a behavior.
Anyway, the point of the example was just to try to respond to your shock at the lack of controlled studies in ABA research.
2
u/Vast-Chemical-4434 Jul 11 '23
Right. And the point I am making is that even a simple situation like that, there is fallacy in the conclusion. Humans are a looooot more complex.
Now let’s say, someone has a headache. They are unwilling and unable to participate in a social activity. You give them a reinforcer and make them do what you want them to do and they do it. Then you would conclude that the reinforcer helped the behaviour. You then take the reinforcer away and they don’t participate and hence the conclusion is that reinforcer worked. but it doesn’t ‘treat’ the headache that was causing the behaviour in first place. You teach the child to ignore what their body is telling them to do.
70
u/Offbrandcereal123 Jun 23 '23
Unfortunately this is a huge problem is all science really. Publish or perish, and journals don’t want to take research that isn’t exciting.