Building in AI

“does model selection really matter? I mean, are your clients like picking, oh, okay, I'm going to use this Lama model of this size or Quinn or do they make those decisions? Or is this something that's kind of like handled automatically by the autonomous intelligence system? So model selection matters, but it's also exhausting. So funny thing. There are two interesting phenomena that's so unique to AI. One is the model depreciation is very fast. As you can observe, every couple of weeks, there's …”

“Jen, you just came back from Mexico. You were talking about your dogs. There's a lot of new paperwork you have to have. And so there's this story going viral of this woman that went to an airport with her dog. And then I'll let the video play. It'll explain it. We're back with a dog left alone at the airport in Las Vegas. Video shows the woman leaving the Golden Doodle Mini Poodle at a JetBlue ticket counter earlier this month. She didn't have the right documents to travel with it as a service …”

“And one of the extremely controversial things we did was we felt that we should focus on fine-tuning existing models because we thought fine-tuning was important. Some other folks were doing work contemporaneously with that. So Jason Ussinsky did some really great research. I think it was during his PhD on how to fine-tune models and how good they can be and some other folks in the computer vision world. We were, you know, amongst the first. There was a bunch of us kind of really investing in …”

“holding multiple possibilities in parallel backtracking when needed and converging on solutions that satisfy all rules simultaneously These are precisely the skills needed for countless real challenges in medicine law operations planning and many other spaces domains where you're balancing competing constraints under uncertainty. A system that can reason through these spaces natively, rather than forcing everything into a text-based chain of thought, could eventually do more than summarize …”

“The biggest moat is going to be which companies understand something that's super hard for other people to understand. And if your answer to that is, I don't know, then you maybe could get vibe coded away. Block was one of the first to make a pretty drastic decision in cutting 40% of the workforce. What led up to that decision? There's been this correlation between the number of folks at a company and the output from the company for decades and decades. I think that basically broke. And what …”

“But you've mentioned a few times the communication. I know you're a big voice user, so I'd love to hear what your voice setup looks like or if there's anything special that you've learned that think people should follow your example on. And then you mentioned Slack too. And I'm curious about even just such practical details. Is it like one to one chats with five open calls? Or is it one channel where you like tag which one you want to assign to things and, but they can all see what's going on. …”

“um let's let's shift gears now a little bit chris to talking about inference specifically uh you know we talked earlier in this episode about how inference is most of what's happening at scales with ai models training is an important part but inference is most of what we're doing with gpus these days and so you talk in the book about multi-node inference and disaggregated architectures. What the heck are those? Yeah. Welcome to another chapter that was supposed to be a single chapter or a topic …”

“They basically said, hey, what if we gave it more experts? So we're going to use more memory capacity, but we keep the amount of activated experts the same. We increase the expert sparsity. So we have fewer experts. The ratio of experts activated to number of experts is smaller. And we decrease the number of attention heads. And kind of for context, what we had been seeing was you make models sparser instead. So no one was really touching heads. You're just having. Well, they implicitly made it …”

“Here's my pitch from a computer science point of view. Pretty rare if people ask me this question. That is, if you're working at a vertically integrated company that have an end product for, let's say, for chatbots, for a system, you are working on the vertical slice of the problem. In Infract, you will be working on a abstraction of horizontal layer. And this is similar to operating system, databases, and different kinds of abstraction that people have built over the years. Operating system, …”

“This is great news. To form a joint venture that will distribute enterprise products across the firm's portfolio companies and beyond. the proposed deal as a free money valuation. AI is coming to Fogo de Chao. Bain Capital owns Fogo de Chao, the Brazilian steakhouse. Maybe now they can take Apple Pay. Oh yeah, we got cooked on that. Maybe they deliberately don't. I wonder if Apple Pay is expensive for them and this is actually a cost consideration. Fiji Simo said, this news came out a little …”

“If you fuse too aggressively you make the kernel too complex and that blows your memory register your memory budget and ironically it makes it slower So the AI has to look at your whole graph and plan of your computations to actually execute this properly And anyway there like a whole bunch of challenges associated with this The the the art of knowing how to let data just flow through your system and, and how to spin up, uh, spin up kernels that do what you want efficiently is just really, …”

“Take me through DepreciationGate. How did you process that, and where do we stand now with the fear that GPUs will depreciate precipitously? And H100s will be worthless in 6 to 12 months. It's totally not a problem right now. Like CoreWeave has talked about these things are lasting 5 to 6 years. And they're getting like almost 90, 95% of the pricing. So it could potentially be a problem if this is a bubble. I don't think it's a bubble. But if it's a bubble 2, 3 years from now and there's a …”

“there enough uh victory giant is there enough pcb this is like one of the largest suppliers of pcbs to nvidia and they're a chinese company all the all the pcbs come from china sort of from them um or many of them and anyways they're like do you have enough pcb capacity great oh hey uh memory vendors who has all the memory capacity okay nvidia does great um so when you look at sort of in the same way you know who who is agi pilled enough to buy compute and long timelines at levels that seem …”

“And anyway, a lot to say from the first hand experience of using those models, but for the sake of speed, we should probably move on. And next model release coming from Google, also fast model, Gemini 3.1 Flash Lite, getting an improvement in both cost and speed. They say this is 2.5x faster time to first token. And first token time, by the way, when you're using fast models, typically you want them for shorter tasks. At least often that's the case. You want to have a quick output with a short …”

“Speaking of AI stuff, I noticed you all have both an MCP and a llms.txt, which is just the docs in a text format. What's the MCP doing for you all? It's just doc search. I believe it's using the Cloudflare AI gateway and then Autorag pointed at the doc site itself. So just an easier way, you know, I think if you just like pointed at that server and tell it to install Varlock and do the onboarding, it'll do a pretty good job. But I probably need to retest that because it's been a few weeks. …”

“And they are making physical changes to the booster to help with that exact recovery process, right? Yes. Booster 19 has new hardware. They installed three grid fins that are 50% larger than the previous generation. 50% larger? That's a massive aerodynamic change. It is. And they are mounted lower on the hull and equipped with specific lifting pins. Those lifting pins are the contact points for the mechanical arms. Correct. The entire system is being optimized for rapid capture and turnaround. …”

“So while the PRD is generating, can we take a look at that Cloud MD file? Yep. Let me pull it up for you now. And that trick, guys, that he's doing there to open preview, make sure you do that because it's a pain to look at markdown files with all of those pound symbols. Annoying. Cursor, it's actually on their forums. You'll see me on the forums whinging about it every month. don't know why they can't fix it but there used to be a setting that by default it would open up in preview anyway but …”

“Things were a bit crass back then. Also, speaking of crass, this editorial at the beginning from John sure hits a little different in this modern era of hardware being like wildly unaffordable. And like, here he is going like, hey, make games that require this brand new $500 graphics card. Like, stop supporting the entire rest of the market. Well, but it's, hey... I get it. I mean... Like this was before crisis, right? We didn't know how bad it would be if you released a game that wouldn't run …”

“Or that could be like a cloud skill or something. It could be anything. Yeah. Anything that's writing down steps. Skills can be tools, though, too. I'm less familiar with how Cloud does the skills. But then the last piece is memory right So and memory right now is in the form of files So whenever you talk to your club bot or whatever you calling these things is just it writing down these markdown files and it reading them So the analogy people use is that movie Memento where the guy wakes up …”

“I do think the cursors and the UI makes a huge difference. Because if I use cloud code, I can theoretically do this too, but it's just generally a bunch of code that I can't see. But being able to see this happen is like a game time changer. It's actually really hard. I mean, sure, a lot of these platforms that are now going into parallel mode, you know? Yeah. They're like assigning different roles to different agents and so on. But what I wanted really to do is to have parallelism with same …”