I can only play in your back yard if you invite me in
I read a post called Statistics in the era of AI by Terry McGlynn. I hadn’t come across Terry before. He self-identifies as an ecologist and I have a lot of respect for ecologists — out of all of the not-statistics disciplines, ecologists seem to be uncommonly good at using R and complex statistical methods. But at the end of the day, Terry is an ecologist, not a statistician.
I say this not to demean or gatekeep, but to highlight that when statistics is outsourced to LLMs, ecologists still get to do ecology. Economists still get to do economics. Psychologists get to do psychology… You get my point.
What do statisticians get to do? We’re humans, living in a society where we have to work to live, doing jobs that we might actually love, too. So what do we get to do when statistics is outsourced to LLMs? Statisticians love John Tukey’s quote about getting to play in everyone’s back yard, but reading Terry’s post made me worry that more and more people are closing their gate and hanging signs saying “No trespassing, ChatGPT on patrol”.
It drives me nuts to be an expert in a discipline that every other discipline needs, yet if you listen to a discussion by statistical consultants about consulting I can guarantee they’ll talk about the difficulty of getting collaborators on board early enough and getting them to budget for your time, and the frustration of being handed shithouse datasets that can’t answer the research question.
Do people actually understand our expertise?
In every statistical consult, the most helpful thing I’ve done was not data cleaning. It wasn’t fitting regression models. It wasn’t writing up the results for the paper.
The single most helpful thing a statistician can do is get you to answer questions you hadn’t even considered
It really didn’t matter what level of statistical expertise the people I worked with had — undergrad students, postdocs, professors with decades of experience and hundreds of publications — there was always something they had not considered that, in answering, changed something important about their research design and data collection.
Humans seem missing from this equation
Terry’s blog post was so striking to me because so often he faces problems or has questions where, to me, the answer is “get more humans involved”. But Terry’s answer is “get ChatGPT involved”. Let me illustrate with some quotes.
So I fed it a dataset that’s been giving me headaches. I just gave it my .csv files. I told it how I wanted to clean up the dataset and prepare it for analyses.
This could just as easily be work for a student or early-career researcher. They’ll overcome the issue you’re having and will learn something about code, data management, analysis, and research in general.
it suggested running a couple additional analyses to decide if I should prefer one parameter over another based on the nature of my question. I was wondering about this, but, frankly, I wasn’t quite sure what approach to take for that issue.
I looked into it, and lo and behold, the approach that was recommended to me made a LOT of sense
So 15 minutes into using ChatGPT to do your analysis you’ve reached the limit of your statistical expertise. Terry makes a point of saying he knows how to google and look stuff up on Stack Overflow — which is a great skill to have. What I wonder though, and this relates to my earlier point of asking questions that had not been thought about, is if your search terms are the words ChatGPT uses to describes its methods, then of course you’re going to get back search results that make sense — because you’re reading the same training data that ChatGPT is using to respond to your queries.
When you’re at the limits of your statistical expertise, you’re very much in “don’t know what you don’t know” territory and I worry about the role confirmation bias plays in this situation — you have an idea of what you might do, ChatGPT does something that looks familiar, the search results are consistent with your ideas, so you’re happy with the output. Would an actual statistical expert give you different advice? You’ll never know because ChatGPT is your expert now.
… one of my three goals was to become proficient at R. It was a slog. I wouldn’t say I became proficient but I became slightly capable.
I want to stress my point here isn’t to dunk on Terry, I’m happy to take at face-value his assertion of being good at statistics (he’s an ecologist after all). But this really feels like a Dunning-Kruger problem and reminds me of much of my consulting work — the medical doctors who knew how to interpret a regression coefficient were typically much harder to work with than those who knew nothing.
But Terry talks about the coding being the problem — not the statistics. I find this point fascinating because it’s like there’s some hierarchy of expertise, where statistics is the important thing to know and coding is just a mundane implementation. To me, as a practising statistician, coding and data management are just as important as knowing the statistical methods
And then it asked me if I wanted it packaged in a Jupyter notebook.
and
Literally all the stats I wanted to do on this project were done for by. In this case, ChatGPT provided a service to me that I would have paid a data wizard a couple thousand bucks to do.
and
If I hired a technician to do the fieldwork instead of doing it myself, would you have an issue with that? If I hired a consultant to write the code when I told them what I needed, would that be a problem? Then, what’s the problem in doing stats with an (AI) consultant?
People can do these jobs, and people learn things by doing them, and eventually these people will replace you when you retire. Your expertise is the result of every single thing you’ve ever learned or done, so what are we doing to the expertise of students and postdocs by outsourcing the work they’d usually do to LLMs?
Where does this lead?
Academia suffers from an ever-rising bar of excellence and you can see it so clearly when you talk to current professors. The academia I hear about from them sounds nothing like what I am currently experiencing, having recently finished my PhD. Needing to have first-author publications to get a scholarship for a degree where you learn how to do research. Publish, publish, publish, but only in Q1 journals if you want a permanent job. Make sure you’re winning grants, but don’t worry we’ve cut your admin support, increased your teaching load, and the grant schemes have miniscule success rates.
I was talking to a friend the other day and he told me that in one week, he was able to use AI-supported coding to do the job that a couple years ago would have taken a postdoc several months to get done. Wow.
This is the exact worry that I have, and Terry sort of comments on this — except he’s not worried, he’s thrilled about not needing to work with “coding wizards”. With the focus on individual success and the superstar researcher, maybe it’s no surprise then that Terry is happy that he can now do everything himself and do it quickly (with ChatGPT’s help of course).
Which brings me to my original worry — where’s the consideration and care for the humans involved? And this isn’t only about statisticians, it’s about students and postdocs too. Teaching statistics and coding in less and less depth because LLMs seem like they can do these things for you is a terrible idea. It’s contrary to the whole point of being a student, a postdoc, or a researcher at any stage of your career — because it’s a choice to not learn any more, but the above quote shows how someone else can make that choice for you.
It’s like instead of teaching kids to ride a bike with training wheels, we’re putting them straight onto a road-bike travelling 50km/h down a hill. The sheer momentum of the bike keeps them upright and flying forward, which looks like success, but if we need them to steer or slow down they’ve got no idea.
