Scott's Mixtape · Economics & Policy
TIER 5 Mon, 4 May 2026 10:40:44 +0000
I grew up in Brookhaven Mississippi, a small town of around 20,000 people, about an hour south of Jackson Mississippi, the state capital. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ | | ---|---|--- | | | Forwarded this email? Subscribe here for more --- # Claude Code 46: Verification is the new bottleneck | | scott cunningham --- | May 4 --- | --- --- | | | --- | | --- | | --- | | --- | | READ IN APP --- I grew up in Brookhaven Mississippi, a small town of around 20,000 people, about an hour south of Jackson Mississippi, the state capital. And I was born in 1975, and played outside a lot. Rode my bike everywhere. Lived at the public library reading books. Collected comics. Would ride my bike to our local movie theater, and go see multiple matinees, then come back. It was the best of times in many ways. Well, that is an odd way to motivate this anecdote, but I just wanted to establish this basic fact which was it was Mississippi, I was born in 1975, and we were pretty landlocked culturally. If it did not come via the local movie theater or television then I really don't know quite how I could've learned it, though I'm sure I am also exaggerating it. But it seems unclear anyway how I would've learned what I'm about to share because I just could've sworn for years that I invented this little thing I'm about to share which is this. My entire life, I grew up believing I had invented "jinx". You know jinx. I know you know jinx because probably in 2010 I learned that _everyone_ knows jinx. And that in fact there is no conceivable way in which I invented jinx. And yet, I remember inventing jinx! Well actually let me correct that -- I grew up believing that I invited "and buy me a coke". Let me explain. I remember with my friend Brandon us saying the same thing at the same time, perfectly timed. I was probably 10. Perfectly timed, by accident. He said something, I said it at the same time, he was new to town, so I think probably he brought "jinx" to us. So he said "jinx". But then, my so-called innovation was created. I started counting, he was counting, we got to 10, and then I said "buy me a coke". Which meant I think he couldn't talk until he bought me a coke. Which was never going to happen, because we're 10 and we have no money. Anyway, guess what. I didn't invent that. I learned that I didn't invent it because my son one day when he was tiny did it, I hadn't done it in a million years, he tells me someone at school did it, I said "that's impossible. I invented jinx buy me a coke." And he said, "no, my friend said his cousin invented it." To this day, I still can't fathom how I was wrong about something like that for 30 years. Like I remember inventing it. So how did I not invent it? And it's weird because probably it was something we both heard on a movie or a show, or Brandon brought it with him from where he lived, or who knows honestly. But one thing I do know, I did not invent that game or that phrase. Well, that is a long way of saying that I also am not sure if I heard someone say about Claude Code "verification is the new bottleneck" or if it was something I said on here, or something Claude said to me, but I have been saying it for sure for months now. Verification is the new bottleneck. Production used to be the bottleneck of research, now verification is. And it just seems like all the time, that is becoming more and more apparent. It is more and more apparent to me that verification is going to be the thing that holds back my own productivity, and the profession's collective productivity. It ironically won't be the actual production side, but the verification side. On a micro scale, it's a bunch of weird small things, some of which are thoughts and feelings without a name. I've mentioned it before, and recently mentioned it on the podcast with Caitlin. But let me explicit here about it. One of the things that I have always noticed about myself is that I don't code ahead of time. I code while I code. Just like I write while I write. I have the vaguest of outline, the most opaque idea of what I'm going to say, then I write, I get a lot of the ideas out of me, I work them over and over, I start cutting and moving them around, and then that's how I write. Similarly, I write in large tasks, not an outline, and I figure it out as I go. My dad was a programmer and he did not code that way. I know that because he told me he didn't, and he explained what he did do, my mom explained it too, and it made it a lot clearer that all those years he would be writing on scraps of paper around the house this weird gibberish, he was plotting his code. He was problem solving the code ahead of time such that he would sit down and do it, maybe bringing that scrap of paper with him, or maybe he didn't even need it by then. So he was sort of coding outside of the coding environment, whereas I was coding inside the coding environment. I won't say one is better than another, because it's all idiosyncratic to one's style and objectives. I'll just say it isn't the same thing. Well, I've noticed with Claude Code a weird "missing emotion" and a "new emotion" that I can't quite nam and it goes like this. When you code as you go, without a real outline, and you learn the outlined inductively by coding, and especially for me with my aphanasia stuff meaning I don't have the best mental image of what I'm doing or things, then I tend to memorize the structure of my code through repetition. It's almost like I'm a blind man who learns the layout of his living room by just walking around the living room blindly enough times that I memorize it. And it works really well, though I think it was probably why for a long time I had to also go through a lot of trial and error, backing my way into better understanding how I needed to be very disciplined and organized, often only learning after some fatal coding error or what have you. When that's how you work, two things happen. One, you just know what the code does, even when no one else can. You know what all the parts do. You sort of have this muscle memory. And two, if you put the project down long enough, you have no idea what the code does, and it takes a little to get the memory back. Now go to Claude Code. Where Claude is writing way, way more code, and I in turn am writing way, way less code. And I think you can probably see where I'm going. That muscle memory, that memorizing of the living room as a blind man -- it's human capital. It's human capital accumulated through attention and repetition. The same task done over and over until you just _know_. That's all that human capital is -- it's knowledge, it's skills, it's things that having it makes you more productive the next period, and the next and the next. Well, with Claude Code doing it, not only do I not have the same kind of human capital in the next period, I don't have it in the present period either. It's like I am blind, remain blind, and just dictating to someone else "do this, do that" and they're saying "I did that, I did that", and I am just trusting that they did it without having my own internal antenna that it was done. That's what I mean -- what I mean is that _the emotion of knowing_ isn't quite there. I don't mean the knowing of the facts too. I don't mean the knowing I get from /beautiful_deck where I get to review what was done. There's a range of verification tools I've been building for a while that for sure are doing _something_. I mean the _emotion_ of verification is gone. And that has been bewildering, and even sad. Sometimes anyway. There's a lot of stuff to be sad about in this life, and probably being sad I am missing the emotions of knowing where I am in the code is very low at the bottom. I have to choose a bit which parts of life I'm going to be sad about, and then concentrate not to be sad about it, and code-sad is for sure low on it. But it's definitely a missing emotion. It's a missing something. And that is because I think historically, production of research and verification of research were the same thing. Or if not the same thing, they were bundled and to such a degree that you could not distinguish them or I couldn't anyway. The same code that created the regression output also created a marker of what was and was not done, and what line it was or was not done, and therefore the confidence that it was or was not done. The regression output contained the confidence that it was regression output that I had seen before, and created myself, and so maybe it was wrong, which is a different thing, but I _knew_ it. I _recognized_ it. And that is actually not there. And I'm not saying that it's wrong, too. I'm just saying that the internal confidence is not there in the same way because it's human capital to have it, and you only have it through attention and time and authorship. And yet. This is the new equilibrium. I predict that we as a species "will not code" anymore. Not in the long run. And how quickly we get to the long run is up for grabs, but for some of us, we may already be there or close to it. We will know because of the sheer effort we are putting towards verification itself, and the awareness of what I am saying is in fact happening inside of us which is the _missing emotion of verification._ So, where am I going with this. Well, related to that is that the speed at which you are ready to write a paper once you've fully embraced AI Agents for research purposes is uncanny. If you are writing theoretical models down, that is especially true. Because for me, the toy models that I would often have in my mind, mulling over and over in my research, particularly about sex work, but also about any paper where mechanisms and context were my obsession, and a desire to fit into a price theory or game theory or matching framework existed. I would often have Beckerian like ways of thinking in the sense that it was consistently "applied price theory" types of stories. Much like I'm doing here talking about production of research, or human capital, or whatever. And I could really never write down models very well. It has always frustrated me too. I would study and study these Nash bargaining models in my dissertation, for instance. I could see them, and I just couldn't figure out the first steps to applying them to my context. So they would become inevitably prose models. Not the explicit mathematical ones, but more like a depiction of the story of the models in words. And it worked for me because the profession shifted towards empiricism anyway, and no one wanted models in the paper as it was. So it was fine to talk a certain way, without getting an actual theoretical toy model down on paper. But now I can get those theoretical toy models down on paper. Claude completely understands what I'm trying to do, and he fills in all the missing gaps, he sees how to start, he does then start, he works through the various things, various lemmas and theorems, various proofs. And I have learned how to verify through audits. Multiple spawned agents that go line by line. I send it to refine.ink. I review it over and over. I do it as the blind man, and once it's all done, _then_ I go through and compare the mental model I've been having with what he worked out, less judging what was done than almost giddy seeing if there actually was, all along, an elegant toy model just out of my reach. Well, I have been reviving old papers of mine I had given up. Two on sex work that I had basically abandoned but I had always really loved. I fielded a large survey of internet sex workers from 2008-2009 and had built into the survey various questions about networked references used to discern client type ex ante. I would later publish on this in the _Journal of Human Resources_ but quasi-experimental in nature, not using my survey, which was richer and got into far more details about how these networked references worked. So I revived that one, and started working on it. I had probably 5 different versions of the paper, one I had presented at Cornell in 2011, scraps of paper like my dad of subgame perfect bayesian Nash equilibrium interpretations of what I was doing, because I had developed it for my grad micro course, and therefore it was the example I was using in my mind. It was Spence job market signaling on some days, it was a screening model I'd found on other days, it was sometimes this very complicated duopoly model with all these predictions. It was an extension on other days of Diego Gambetta's _Codes of the Underworld: How Criminals Communicate _on other days. I just was all over the place, editing draft after draft, iterating until the paper would change, even as the empirical results remained the exact same. So now I've been working on getting things distilled, simplified and into draft form and just trying to commit so that I can move on from this part of my life, and I have it in two papers, both of which have inspired me to see the connection between what I was working on with networked references, and modern online dating culture which has many similar networked references for screening new people. So eerily similar are they to one another that I am stunned sometimes that I didn't quite see it. Well, I have two of these projects I'm reviving and I can see that I will get them into manuscript form, and I will submit them, and I will publish them, and I will feel the satisfaction that I can finally close the circle on this part of my sex work research agenda relating sex work to platforms and networks and technology and matching. But without the muscle memory. And with a decisive thinking partner who just cuts through the morass, and helps me commit. But here's the thing I am now thinking. Yes, there is this internal verification thing missing that I am having to replace, and am still trying to figure out precisely what the replacement will be for me. I haven't quite figured it out because it's not just verification of facts. It's the confidence based verification -- the _knowing_. The _feeling_ and the _confidence_ I got from research that only came via the internal human capital I got from my repeated engagement with the topic. And since I was spending less time on the same type of production, then I was getting less human capital mechanically until I could figure out a new way. Because I can't submit something until I have some threshold of confidence that what I have done, in fact I have done it, and it's correct, and that is for me as much an emotion as anything else. But you know what? Let's say I do get there, and I'm confident I will. All of this AI stuff is just solving old problems, creating new problems, and then putting the responsibility on me to solve those new problems -- and the _missing emotion of verification_ is one such problem, and I will solve it. But the thing that is now new is to realize that if I am now making progress, actually fixing broken parts of my pipeline -- and trust me, my pipeline has had kinks in it for a while, and none of them really had to do with the ideas, but they for sure were there. William Shockley, the 1956 winner of the Nobel Prize in physics, has this article on the productivity of scientists and their salaries, which I encourage you to read. Particularly if you're interested in the production function of science and the labor economics of scientists. It was alarmingly right in the wheel house of economics, even though I get the sense he was just trying to argue scientists needed a far different salary structure given the log normal distribution of outputs associated with their work. Here is the paper. | 1957 Shockley2.43MB ∙ PDF file ---|--- | Download --- And here is this one paragraph I think about a lot. Notice the production function for scientists he lays out. See how it is the product of a bunch of inputs, as opposed to the inputs adding together? That's a Cobb-Douglass production function without being called it. The factors of production if you were to log it would then be the sum of those logged productivity terms, though. | | ---|---|--- Well read it carefully. And then ask yourself -- what of these has AI actually changed, and for some of us, completely repaired? Will you be able to come up with a good idea? Yes. So that's F1, fixed and/or massively inflated. Can you do the work? Yes. F2. Can you recognize a good idea? Maybe. That could be harder given all I've said about the _missing emotion of verification_ , but it's solvable. So let's say that one could change somewhat, perhaps even go down, but maybe stay the same and maybe others goes up. And on and on, even down to the revision stages with the journals. Point is, if the sum of those inputs really does determine not just output, but publications, which are actually themselves key inputs in science and innovation -- it isn't just the work, in other words. It appears to be _the publication_ too. It's the publication that translates the private knowledge into that which can be built on by others, and I think we then can imagine that this is indeed the bottleneck. Because if you are becoming more productive, then probably someone else is, and you are now writing 3x as many papers ready for submission, then someone else is, and at some point there is more being sent, more being screened, more being sent to referees, but also more being circulated also beforehand too. That's the other part of it. I am noticing a shifting away from reading as intensively as I had been. I am working intensively with AI to break papers down into digestible forms that can accommodate my ADHD and misallocated attention in general. I am using spawned agents to completely create a learning process that fits me, and I am still being asked to referee, and I am having to split my time between reading them the old way and doing my own learning another way. And that too is creating some challenges within me. So what I am saying? I am saying that the verification is the bottleneck that's what I'm saying because ultimately in Dr. Shockley's paper, the "movement of the paper through submission" and "benefiting from feedback on a paper from referees" ultimately is a mixture of things you can and cannot control. You cannot control what they see, how they respond, what they tell you to do. You can control where you send it, how you respond, your own resilience, but at some point you're managing a larger portfolio of papers, and you have to solve that problem of how do you have more balls in the air when you already felt there were a lot of balls in the air as it was. Not all the inputs can be expanded after all -- the creative capacity and attention of the human mind is not expanding so much as it's getting reallocated and it's not yet clear where and what is the optimal one. It just is obvious that I have the responsibility of verification now that the other parts of that production pipeline is on steroids. So all that is to say the demand for our papers and the supply of our papers is probably not in the truest form of equilibrium. There's the partial equilibrium, for sure, which I think at best would manifest as more papers, a bit of a race to the scarce slots, the slots in the journals staying exactly the same, but acceptance rates changing. Perhaps the taste of the paper selection shifts, but in what direction, who knows. Maybe in aggregate average taste falls, but maybe in aggregate average taste stays the same or goes up even. No one knows. We are really early in the game, and adoption of agents for research despite what we all feel is true because of our echo chambers is in fact bizarrely low. You would be surprised. It almost defies all logic to me that we are not in the long run equilibrium right this moment given how obvious it is that the gains are massive both privately but also for society if we can figure this out as soon as possible. But the long run equilibrium won't just be falling acceptance rates. It won't even be an expansion of issues in journals and it won't even be new journals. It won't be charging more for submissions or the hiring of more editors. Those might happen, but it's entirely possible that the long run equilibrium is almost unrecognizable and hard to predict because ultimately the current system is based on the bundling of production and verification and now they're separated, AI augments one and the other is the more complex task and thorny one to solve. And it involves a lot more experimentation because we don't have off the shelf knowledge that we can just pull down because a lot of what we know about that was borne out of human capital from when they were bundled. Anyway, I think a lot of us are going to have unpublished manuscripts and maybe for even longer than ever. We will have more "resting papers" -- papers we gave up on because we simply couldn't find a home for them. The intense collaboration between an AI agent and its human researcher can lead to a manuscript that is so far outside of what is presently acceptable in the profession that it cannot get published because the paper still must pass peer review, and peer review has people, and they may start to get irritated at the workload agents are imposing on them and who knows how that breaks. Who knows how they would've responded to the exact same paper you just sent out but had sent it five years ago, or even one year ago. I think we are in a place now where this could go any number of ways. Scott's Mixtape Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. Upgrade to paid You're currently a free subscriber to Scott's Mixtape Substack. For the full experience, upgrade your subscription. Upgrade to paid --- | | | Like --- | | Comment --- | | Restack --- (C) 2026 scott cunningham 910 North 17th Street, Waco, Texas 76707 Unsubscribe