I did an interview with Dr. Inzlicht on this subject. We talked about his recent paper on the Moralization of AI. What they found is just for some subset of the population, the objection they have to AI is essentially window dressing. These individuals have already decided that creating anything with AI is immoral and they plug in their reasons after.
It’s how people have always worked. I think my first exposure to this was John Haidt, though he was just citing earlier research.
The emotional snap judgment comes first, all your cognitive faculties come online after the fact and get to work rationalizing the decision you’ve already made.
If we have disclosure norms, then they might initially benefit dishonest AI users and punish honest AI users. But if it’s discovered that one of the dishonest AI users did use AI, the cost would be greater than if they had disclosed in the first place—because they both used AI and broke the disclosure norm, meaning they were trying to pass the work off as completely their own. If we’re increasing unable to detect AI-generated writing, then the risks to dishonest AI users get lower and lower, but that doesn’t mean the disclosure norms are a bad idea in the meantime.
Right, this makes sense in principle. I agree that if a disclosure norm is clear, and someone violates it, the violation itself can become part of the offense.
I think the devil is in the implementation, though. If someone fails to check an AI-use box when submitting a paper and the paper later turns out to have a hallucinated citation, I'm not sure most institutions would actually punish the non-disclosure more than the underlying failure. And with current detector tools, even good ones, the evidence is still probabilistic. That makes it hard to prove AI use with enough confidence, especially once we move beyond basic prose generation into outlining, translation, editing, coding, data analysis, citation search, and such.
So I'm not against disclosure norms in every context. I just worry they will mostly burden honest users, while the worst cases either won't disclose or will be hard to prove.
Although one thing we have seen a few times is people not ticking they'd used AI. Then submitting a load of hallucinated references. Then when challenged blaming AI (or more usually denying they'd used AI followed by explaining how they "just pasted it" into an LLM to format the references for them. At least with the AI tickbox you have an easy rejection route rather than the tedious back and forth that otherwise ensues.
My sense is that, as the editor, you should always be able to reject submissions without real explanation. If you see some human or AI slop, you can just say that the piece doesn't meet the standards or that the submission volume is high. No need to prove or deal with any suspected AI use :)
On the point about prices and exclusivity - various things I’ve been reading suggest that the tools aren’t actually subsidized much, if at all, any more. Both Anthropic and OpenAI seem to be getting as much revenue as they are spending on inference-time compute. They are still on net losing money because of their investment in the next round of data centers and their next round of training. But by the time those next round of models are in use, their revenue has grown enough that the losses on training and infrastructure are comparable to the cash flows they have on inference-time compute on that next model.
If they hit a bottleneck soon where they can’t add new data centers any more (because of electricity constraints or chip constraints or whatever), then it’s possible that continued user growth will bid up the price to users, and the AI companies will switch to huge profits. But if they can keep growth scaling with demand, then the current pattern suggests that price won’t rise - and for people who are willing to settle for year-old models once models start getting really good, prices may even fall drastically.
Also, on the LLM tells point (and you can wrest my Oxford comma from my cold dead hands) how much do you think they're already seeping into everyday discourse? The use of “steelmanning” in the conversation made me think that because it had seemed a very American (maybe rationalist adjacent) phrase that I only really regularly encounter in LLM output otherwise. You wonder if it could lead to an odd sort of divergence in conversational patterns between regular LLM users and others. And even a social signifier!
Dan and Alex are not wrong to mention The Structure of Scientific Revolutions here; "paradigm shift" has a common-sense meaning which is applicable here as a measure of the intelligence of AI.
But I am going to give my usual rant against the book, because it's a little snowball that has become a devastating avalanche from so many people helping roll it down the hill. It is either a stupid book, or a dishonest one. The entire reason for its fame is its premise that paradigm shifts are objectively neutral--a claim which Kuhn renounced in the preface to the 2nd edition, but which he definitely made in the book, and without which the book is uninteresting.
The problem is, Kuhn's /interpretation/ of his data is the exact opposite of what the data says. Every paradigm shift he studied was a case in which the shift was from one paradigm, to an objectively better paradigm. They were also all cases in which the people considering the 2 paradigms had no problem with incommensurable terminology. Kuhn proved the opposite of what he claimed to have proven.
Now, if you were to apply Kuhn's thesis to the humanities, that would be different. You would have no trouble showing, for instance, that the people who called the New Critics "ahistoricists" never even read their books, let alone understood their terminology. You would run into more of the opposite problem: of the same paradigm being presented as a new paradigm using rhetorical tricks, such as Sartre pretending to have superseded Heidegger, or the French post-modernists pretending to be doing more than just dressing up the Nazi epistemology of subjectivity that Hitler had called for in 1924.
Kuhn is a classic example of the maximalist/minimalist move in the marketplace of ideas. Make a maximalist claim that on the face of it is bold and revolutionary and if true would have profound and wide reaching implications - but is also unfortunately trivially false. Then defend that idea once you've got all the fame and attention you want by retreating to the minimalist claim which is much less exciting and trivially true. This sort of manoeuvre has been pulled by the great and charismatic men (usually men) of ideas for centuries - and still works.
You said, thanks to AI, you have less need to confer with colleagues, so you could be more productive. Contact with colleagues has social and intellectual benefits that go beyond getting the answer to a particular question. Right?
Right. But note that I also said that, since we can now outsource all the grunt work to AI (like the well-resourced folks previously did with their RAs), we can now have more time for meaningful human interaction and networking.
Maybe it's because I'm old enough to remember photocopying in the library stacks pre-online journals, but I'm pretty unimpressed by literature reviews produced by LLMs. And it isn't massively surprising since they don't have access to major domain expertise or pay walled papers. I remember being surprised during the rise of open access at a younger collaborator telling me they didn't read anything that wasn't free to access online from their institution. That sort of attitude would lead to serious gaps in your knowledge of a field - certainly in my area of medicine.
There's also a degree of caution needed for using AI to do statistical analysis. On the one hand it is totally cool to get AI to run a python script to do data analysis but you need to know exactly what it's doing on your behalf and how - there are assumptions you need to understand and naively asking for a graph or analysis could easily lead to completely inappropriate or inaccurate results. Particularly if people are just chucking it in Gemini say where the numbers and figures can be completely wrong (I don't know if they still do it but it wasn't that long ago frontier models would hallucinate a whole dataset for you). I recently had a trainee tell me how he'd run his stats in Gemini and how helpful it had been but when pressed couldn't explain what it had done, how it had done it, or what the assumptions were. Personally I blame the growth of data science as an offshoot of computing as opposed to old fashioned stats and I think we're going to see the same thing more generally - lots of cool tools being deployed by people who never learned how to do things the right way first.
That said, I do agree the modern journal article is not fit for purpose. Trite introduction copied from twenty other papers, methodology underspecified, conclusions tendentious, stats dubious. The future surely has to involve centrally registered datasets and AI-assisted exploration that lets readers zoom in and out depending on their needs. But I fear large-scale data fabrication is just around the corner. Scientific publishing is already awash with slop and dubious data, and the lower the bar falls the worse it will get. We already see egregiously bad AI-generated papers being submitted and occasionally published. The incentive structure produces a lot of noise. The number of papers suggesting drug X might help condition A must outnumber actual empirical trials by a hundredfold - and they keep publishing them even when trial evidence of inefficacy is incontrovertible.
Which is not to say I don't enjoy checking in from my phone while I'm between patients to monitor Codex autonomously designing a large-scale experiment on my laptop, particularly all the tricky coding I never previously got round to.
As for writing emails - maybe my inbox is just very boring but I don't have so many replies to write that it is a chore and anything I do write I want to make sure it is expressed in the right way. I struggle to think of something worth saying but not worth saying enough to bother drafting it out for a minute or two. I tried the agents.md approach and had it read extensively from my own writing but the result sounded like a bad parody of me because it was just amplifying stylistic tics.
It’s a tool that engages in complex information processing that can be helpful at achieving goals in a variety of situations. If you don’t want to call that “intelligence”, fine, but it’s a bit weird to call it “diabolical”. Is there a better term that both acknowledges the way these tools are different from any other tool that came before as well as drawing the distinction that you’d like to draw?
There’s also a natural perspective that says it’s misleading to call these things “language” models, when so much of their development now involves reinforcement learning on tool use for coding, image interpretation, data processing, and other things no one would have described as “language” four years ago.
I did an interview with Dr. Inzlicht on this subject. We talked about his recent paper on the Moralization of AI. What they found is just for some subset of the population, the objection they have to AI is essentially window dressing. These individuals have already decided that creating anything with AI is immoral and they plug in their reasons after.
Interesting. Sounds plausible! Thanks
It’s how people have always worked. I think my first exposure to this was John Haidt, though he was just citing earlier research.
The emotional snap judgment comes first, all your cognitive faculties come online after the fact and get to work rationalizing the decision you’ve already made.
If we have disclosure norms, then they might initially benefit dishonest AI users and punish honest AI users. But if it’s discovered that one of the dishonest AI users did use AI, the cost would be greater than if they had disclosed in the first place—because they both used AI and broke the disclosure norm, meaning they were trying to pass the work off as completely their own. If we’re increasing unable to detect AI-generated writing, then the risks to dishonest AI users get lower and lower, but that doesn’t mean the disclosure norms are a bad idea in the meantime.
Right, this makes sense in principle. I agree that if a disclosure norm is clear, and someone violates it, the violation itself can become part of the offense.
I think the devil is in the implementation, though. If someone fails to check an AI-use box when submitting a paper and the paper later turns out to have a hallucinated citation, I'm not sure most institutions would actually punish the non-disclosure more than the underlying failure. And with current detector tools, even good ones, the evidence is still probabilistic. That makes it hard to prove AI use with enough confidence, especially once we move beyond basic prose generation into outlining, translation, editing, coding, data analysis, citation search, and such.
So I'm not against disclosure norms in every context. I just worry they will mostly burden honest users, while the worst cases either won't disclose or will be hard to prove.
Although one thing we have seen a few times is people not ticking they'd used AI. Then submitting a load of hallucinated references. Then when challenged blaming AI (or more usually denying they'd used AI followed by explaining how they "just pasted it" into an LLM to format the references for them. At least with the AI tickbox you have an easy rejection route rather than the tedious back and forth that otherwise ensues.
My sense is that, as the editor, you should always be able to reject submissions without real explanation. If you see some human or AI slop, you can just say that the piece doesn't meet the standards or that the submission volume is high. No need to prove or deal with any suspected AI use :)
Sadly this seems more common with reasonably well established authors who you need at least not to piss off.
Thanks. Good point
On the point about prices and exclusivity - various things I’ve been reading suggest that the tools aren’t actually subsidized much, if at all, any more. Both Anthropic and OpenAI seem to be getting as much revenue as they are spending on inference-time compute. They are still on net losing money because of their investment in the next round of data centers and their next round of training. But by the time those next round of models are in use, their revenue has grown enough that the losses on training and infrastructure are comparable to the cash flows they have on inference-time compute on that next model.
If they hit a bottleneck soon where they can’t add new data centers any more (because of electricity constraints or chip constraints or whatever), then it’s possible that continued user growth will bid up the price to users, and the AI companies will switch to huge profits. But if they can keep growth scaling with demand, then the current pattern suggests that price won’t rise - and for people who are willing to settle for year-old models once models start getting really good, prices may even fall drastically.
Thanks, this is good to know. I hope you're right!
Also, on the LLM tells point (and you can wrest my Oxford comma from my cold dead hands) how much do you think they're already seeping into everyday discourse? The use of “steelmanning” in the conversation made me think that because it had seemed a very American (maybe rationalist adjacent) phrase that I only really regularly encounter in LLM output otherwise. You wonder if it could lead to an odd sort of divergence in conversational patterns between regular LLM users and others. And even a social signifier!
Dan and Alex are not wrong to mention The Structure of Scientific Revolutions here; "paradigm shift" has a common-sense meaning which is applicable here as a measure of the intelligence of AI.
But I am going to give my usual rant against the book, because it's a little snowball that has become a devastating avalanche from so many people helping roll it down the hill. It is either a stupid book, or a dishonest one. The entire reason for its fame is its premise that paradigm shifts are objectively neutral--a claim which Kuhn renounced in the preface to the 2nd edition, but which he definitely made in the book, and without which the book is uninteresting.
The problem is, Kuhn's /interpretation/ of his data is the exact opposite of what the data says. Every paradigm shift he studied was a case in which the shift was from one paradigm, to an objectively better paradigm. They were also all cases in which the people considering the 2 paradigms had no problem with incommensurable terminology. Kuhn proved the opposite of what he claimed to have proven.
Now, if you were to apply Kuhn's thesis to the humanities, that would be different. You would have no trouble showing, for instance, that the people who called the New Critics "ahistoricists" never even read their books, let alone understood their terminology. You would run into more of the opposite problem: of the same paradigm being presented as a new paradigm using rhetorical tricks, such as Sartre pretending to have superseded Heidegger, or the French post-modernists pretending to be doing more than just dressing up the Nazi epistemology of subjectivity that Hitler had called for in 1924.
Kuhn is a classic example of the maximalist/minimalist move in the marketplace of ideas. Make a maximalist claim that on the face of it is bold and revolutionary and if true would have profound and wide reaching implications - but is also unfortunately trivially false. Then defend that idea once you've got all the fame and attention you want by retreating to the minimalist claim which is much less exciting and trivially true. This sort of manoeuvre has been pulled by the great and charismatic men (usually men) of ideas for centuries - and still works.
You said, thanks to AI, you have less need to confer with colleagues, so you could be more productive. Contact with colleagues has social and intellectual benefits that go beyond getting the answer to a particular question. Right?
Right. But note that I also said that, since we can now outsource all the grunt work to AI (like the well-resourced folks previously did with their RAs), we can now have more time for meaningful human interaction and networking.
Maybe it's because I'm old enough to remember photocopying in the library stacks pre-online journals, but I'm pretty unimpressed by literature reviews produced by LLMs. And it isn't massively surprising since they don't have access to major domain expertise or pay walled papers. I remember being surprised during the rise of open access at a younger collaborator telling me they didn't read anything that wasn't free to access online from their institution. That sort of attitude would lead to serious gaps in your knowledge of a field - certainly in my area of medicine.
There's also a degree of caution needed for using AI to do statistical analysis. On the one hand it is totally cool to get AI to run a python script to do data analysis but you need to know exactly what it's doing on your behalf and how - there are assumptions you need to understand and naively asking for a graph or analysis could easily lead to completely inappropriate or inaccurate results. Particularly if people are just chucking it in Gemini say where the numbers and figures can be completely wrong (I don't know if they still do it but it wasn't that long ago frontier models would hallucinate a whole dataset for you). I recently had a trainee tell me how he'd run his stats in Gemini and how helpful it had been but when pressed couldn't explain what it had done, how it had done it, or what the assumptions were. Personally I blame the growth of data science as an offshoot of computing as opposed to old fashioned stats and I think we're going to see the same thing more generally - lots of cool tools being deployed by people who never learned how to do things the right way first.
That said, I do agree the modern journal article is not fit for purpose. Trite introduction copied from twenty other papers, methodology underspecified, conclusions tendentious, stats dubious. The future surely has to involve centrally registered datasets and AI-assisted exploration that lets readers zoom in and out depending on their needs. But I fear large-scale data fabrication is just around the corner. Scientific publishing is already awash with slop and dubious data, and the lower the bar falls the worse it will get. We already see egregiously bad AI-generated papers being submitted and occasionally published. The incentive structure produces a lot of noise. The number of papers suggesting drug X might help condition A must outnumber actual empirical trials by a hundredfold - and they keep publishing them even when trial evidence of inefficacy is incontrovertible.
Which is not to say I don't enjoy checking in from my phone while I'm between patients to monitor Codex autonomously designing a large-scale experiment on my laptop, particularly all the tricky coding I never previously got round to.
As for writing emails - maybe my inbox is just very boring but I don't have so many replies to write that it is a chore and anything I do write I want to make sure it is expressed in the right way. I struggle to think of something worth saying but not worth saying enough to bother drafting it out for a minute or two. I tried the agents.md approach and had it read extensively from my own writing but the result sounded like a bad parody of me because it was just amplifying stylistic tics.
Who cares? Thanks for a great example of combining both anti-AI sentiment and grammar nazism at the same time :) https://www.popularbydesign.org/p/pangram-policing-is-the-new-grammar
It’s a tool that engages in complex information processing that can be helpful at achieving goals in a variety of situations. If you don’t want to call that “intelligence”, fine, but it’s a bit weird to call it “diabolical”. Is there a better term that both acknowledges the way these tools are different from any other tool that came before as well as drawing the distinction that you’d like to draw?
There’s also a natural perspective that says it’s misleading to call these things “language” models, when so much of their development now involves reinforcement learning on tool use for coding, image interpretation, data processing, and other things no one would have described as “language” four years ago.