主持人
Um be talking about QM and so QM there at YC we had this general agent that we use and they came out with QM a month ago and it is like meaningfully uh better and is so functional and I use it every day. All right, that's all I got. Thank you so much.
[掌声] 他们会讲 QM。在 YC，我们本来有一个大家在用的 general agent，他们一个月前推出了 QM，明显更好，也好用得多，我每天都在用。好，我就讲这些。非常感谢。[掌声]
嘉宾
Hi everyone. Uh, great to be here tonight. Uh, my name is Seth and I'm going to be presenting Prime Agent,which is a self-improving ROM harness. Um, my job here tonight, I think, is to try to convince all of you to take a very first principles style approach to thinking about how you build your harness. So, we're going to be very, very basic here to begin with. If you think about and I love the uh introduction right from of all the background literature um which I thought was so cool I think is very complimentary the how I think about this as well um but if you think about just
大家好。今晚很高兴能在这儿。我是 Seth，接下来讲 Prime Agent，一个会自我改进的 RLM harness。我今晚的任务，是想说服大家用非常第一性原理的方式来想：你该怎么建自己的 harness。所以一开始我们会非常、非常基础。刚才那段背景文献的介绍我特别喜欢，我觉得它和我的思路非常互补。但如果你只看
嘉宾
the raw LLM itself it's actually just this sequential processor that you know has some fixed weights and has some visible context and it's it's taking some tokens in and it's putting some tokens out uh to make the next decision. Um we don't really think of an LLM that way these days. We have a set of files that it has access to. We give endless programs and tools for it to use. Um, you can even message other sessions of LLMs that are going on and create sub agents in order to do all of these very cool things. But, but in its basics, it's just tokens in, tokens out. It's a neural network making a prediction. The harness itself is the layer between the LLM and the world that adds things like this persistent state tools and compute. Uh, here we have a diagram of how we think about prime agent. um from the human's perspective. So you open up your prime agent uh on your computer just like you would cloud code, codec, pi,
裸的 LLM 本身，它其实只是一个序列处理器：有固定的 weights，有一段可见的 context，吃进一些 token，吐出一些 token，用来做下一个决定。我们现在不太会这样看 LLM。我们会给它一组能访问的文件，给它无穷无尽的程序和工具。你甚至可以给正在跑的其他LLM session 发消息，创建 sub-agent，去做所有这些很酷的事。但在最根本上，它就是 tokens in、tokens out。就是一个在做预测的神经网络。harness 本身是 LLM 和世界之间的那一层，加上持久状态、工具和算力这类东西。这是一张我们怎么看待 Prime Agent 的图，从人的视角。你在电脑上打开 Prime Agent，就像打开 Claude Code、Codex、Pi 一样
嘉宾
etc. Um and it gets you into this agents view and this agents view is an overview of all the agents that you have going on for all your parallel sessions with like a very tight like tight summary of what you're using and then you can hop into one of those and check it out. there you're going to be at this root session and this root session is your basically the project orchestrator over all of these different sub aents that it's controlling and you don't have to ask it to start sub agents it will leverage sub agents when they're useful um and these are all programmatically called because this is all based on uh the recursive language model principle where everything is in this IPython shell um all your tools all your memories all your sub aents uh and and we expose uh for further coordination these uh these messaging paradigms. So you can manage all of these um and then these agents can directly interact with your environment which could just be like the programs and files on your computer or running an H200 node cluster
等等。然后你会进入这个 agents 视图。这个视图是你所有并行 session 里正在跑的 agent 总览，带一份很紧的摘要，说明你在用什么。然后你可以点进去看某一个。你会落在这个 root session 上。[清嗓] 这个 root session 基本上就是项目的 orchestrator，管着它控制的那些不同的 sub-agent。你不必让它去启动 sub-agent，有用的时候它自己会用。这些都是程序化调用的，因为整套都建立在 Recursive Language Model 原则上：所有东西都在这个 IPython shell 里，你的工具、memory、sub-agent。为了进一步协调，我们还暴露了这些消息范式。所以你可以管理所有这些。然后这些 agent 能直接跟你的环境交互，环境可以只是你电脑上的程序和文件，也可以是跑着的 H200 节点集群
嘉宾
maybe for your auto research. Each of these agents are then backed by this persistent dam on your computer. Uh this is so that you know when you close uh your laptop or you you control C out of the session, it's still running in the background. You have to actually stop the session so that make sure that you're continuing to running. And then we also have these other features we exposed from continual harness where it's able to provide like live CRUD operations on all of the components that we mentioned um in order to manage its uh memory skills sub agents um persistent and and prompt its own system prompt persistently. The way I like to think about all of this context that we're building up is that we have this sort of almost like like here I have like L1, L2, L3. is like a cache, right? It's like what is the most accessible information that we're working with and at the very like fastest like readily available information, you know, the
也许是给你做 auto research 的。每个 agent 背后都有你电脑上的一个持久 daemon。这样你合上笔记本，或者对 session 按了 Ctrl+C，它仍在后台跑。你必须真正停掉 session，否则它会继续跑。我们还从 continual harness 里暴露了另一些功能：它能对所有组件做实时 CRUD来管理它的 memory、skills、sub-agent，并且持久地改它自己的 system prompt。我喜欢这样看我们堆起来的这些 context：差不多像这里的 L1、L2、L3，像一层缓存。也就是：我们正在用的信息里，什么是最好拿的。在最快、最现成的那一层
嘉宾
models to be able to retrieve that really quickly. It's the model weights. So, everyone always wants to get all the information in the model weights. Um, but then we said, okay, well, maybe we don't have all the information because we don't want to have to fine-tune every single time to update because that's very expensive. So, we have this uh active input context. So, we're using lots and lots of tokens on the input. we might have some in context examples like we've seen previously in order to add to these different capabilities but at a certain point we run out of context. Um and so the very like earliest form of harnesses that we've seen that are still used to this day even by those who say we want the most minimal harness possible is compaction because compaction is a very generalized tool for the agent to be able to uh summarize its own context history in order to work past its context length working window. You can think about um once we go beyond like what are directly like inputs and
模型要能很快取到，那就是 model weights。所以大家都想把信息全塞进 weights。但后来我们说：也许不必全放进去，因为我们不想每次更新都微调，那太贵了。于是我们有了 active input context。我们在输入上用掉大量 token。我们可能放一些 in-context 例子，就像前面看到的，用来补上这些不同能力。但到了某一刻，context 就用完了。所以我们见过的最早、直到今天还在用的 harness 形态——就算那些声称只要最极简 harness 的人也会用——就是 compaction。因为 compaction 是一种很通用的工具，让 agent 能总结自己的 context 历史，从而越过 context 长度这个工作窗口。可以这样想：一旦我们超出模型直接的输入和
嘉宾
outputs from the the model here into this L2L3. You might be familiar with the L3 which is more of the dispatch state. So if you're working with a file system, you can read and write from uh main memory. Uh if we're at the L2, which is I could think uh at a means in between uh what the active context is and working with your file system, you might have a live uh ripple, which could just be running things directly in Python uh an IPython shell like you're in a Jupyter notebook. And all of those variables are saved directly in your RAM. your agent can then programmatically manipulate them and run all sorts of programs directly on the information there saving tons of tokens rather than putting it directly into context. Uh you can also create sub agents and it's the same thing you're basically saving context here because you can task the agent with a specific set of information in order to perform some operations and the report back at the end. What is interesting here is what we talked about compaction for the active context, right? You have your
输出，进入这个 L2、L3。你们可能熟悉 L3，它更像 dispatch 状态。如果你在跟文件系统打交道，可以从主存里读写。如果在 L2，我会把它想成 active context 和文件系统之间的一层，你可能有一个活的 REPL，可以直接在 Python 里跑东西，一个 IPython shell，就像在 Jupyter notebook 里。而那些变量直接存在 RAM 里。你的 agent 可以程序化地操作它们，直接在那些信息上跑各种程序，省下大量 token，而不必把它们塞进 context。你也可以创建 sub-agent，道理一样：你其实是在省 context，因为你可以给 agent 指定一小份信息去执行操作，最后再汇报回来。有意思的是，我们刚才说的 active context 上的 compaction，对吧？你有你的
嘉宾
context history. This is helping to update it over time. So you can continue to leverage this. But once we go beyond this, we need to be thinking about how are we doing these update. We talked about CRUD. How are we do beyond just creating reading? How are we updating and deleting our context over time beyond our uh or the state over time beyond the context line. Uh I like to think of this at the ripple is this aentic garbage collection where we're just cleaning up the variables in our state as well as like what sub agents could be used. And then so to make sure that our RAM doesn't crash my computer laptop every day. And then on top of that, we have this uh notion of refinement where we're updating and deleting the skills and memories and prompts that are stored on your system. You can think of that so that way you don't crash your actual uh out of space on your hard drive as well. And so this very much is a here's how we express this thing and here's how we revise it over time. The other perspective that I
context 历史。这能帮它随时间更新，让你能继续用这一层。但一旦超出这一层，我们就要想：这些更新怎么做。我们讲过 CRUD。不只是创建和读取，我们怎么随时间更新和删除 context，或者说删除 context 窗口以外的状态。我喜欢把 REPL 这一层想成 agent 式的垃圾回收：我们在清理自己状态里的变量以及哪些 sub-agent 还能用，免得 RAM 每天把我的笔记本搞崩。再往上，我们有 refinement 这个概念：更新和删除存在系统上的 skills、memory 和 prompt。可以把它想成：这样你的硬盘也不会被撑爆。所以这很大程度上就是：我们怎么表达这件事，以及我们怎么随时间改它。我很喜欢的另一个视角
嘉宾
really like to think about and I really trying to push because harnesses are almost going towards this like agentic operating system that we're creating uh is when I think of it um more metaphorically here is that when you look at the raw LLM it kind of looks more like a touring machine where you have this ticker tape uh and you have all these instructions that are going in and then it's performing some set of operations and going out. But when you look at a harness it's looking a lot more vono like a vonoyman computer.
我也一直在推，因为 harness 几乎在走向我们正在做的这种 agent 操作系统。更比喻一点说：看裸的 LLM，它更像一台图灵机，有一条纸带，指令进去，做一组操作，再出来。但看一个 harness，它更像冯·诺依曼计算机。
嘉宾
you're able to do these read and write operations on external memory and that makes it much more powerful and another class of problems than just what a touring machine is able to express on its own. And so yeah, the idea of like how do you build a good hardness? You want it to be the most expressable thing you can imagine. So some some like early harnesses before it gets into the data flywheel where the models can do themselves are very specific. Plan, act,critique, do these exact specific um steps. Well, now har uh the models are able to do that themselves. You can imagine like we we don't have like a react loop that we necessarily need to explicitly impose. The models kind of have natively uh figured this out. But what they haven't figured out is how to um you know they have to be able to have the expressibility to call compact. They have to be able to have a Python ripple so they can run programs. Um they have to have the ability to programmatically
你能对外部 memory 做读写，这让它强大得多，能表达的问题类别也和图灵机自己能表达的不一样。所以，怎么建一个好的 harness？你希望它是你能想象到的、表达力最强的东西。早期一些 harness，在进入模型能自己转起来的数据飞轮之前，是非常具体的。规划、行动、批评，按这些非常具体的步骤来。而现在，模型自己就能做这些了。可以想象，我们不必再显式地套一个 ReAct 循环。模型差不多已经原生学会了。但它们还没学会的是：它们必须有调用 compact 的表达能力；必须有一个 Python REPL，才能跑程序；必须能程序化地
嘉宾
create sub agents and access state and have different feedback mechanisms. Th those are model controlled expressibility features and if you removed one of those you're actually removing a capability that it won't be able to do otherwise. The way we manage um and I'm sure you're all familiar with the RLM paper uh from my co-author Alex um fantastic bit work. What we do beyond what was in the RLM paper is we think about the age sub aents as these persistent subsessions.
创建 sub-agent、访问状态，并拥有不同的反馈机制。这些都是由模型控制的表达力特性。你拿掉其中任何一个，就是在拿掉一种它否则做不到的能力。我们的管理方式——我相信大家都熟悉我的合作者 Alex 的 RLM 论文，那篇工作非常出色。我们在 RLM 论文之上多做的是：把这些 sub-agent 想成持久的 subsession。
嘉宾
So each the parent station can create uh spin up a new RLM sub aent and each of these are then emitted. They run some task and then they finish and report back to some end state to the parent session. These are then idle. They're still working in your RAM. At any point, the parent session can then send a message to one of the sub aents to continue working and it has all that good context that you built up over time so that you're not missing information or have to reuse information that was already developed in a prior context. And then of course you know we don't want to use a lot of RAM. So we can move them offloaded uh in an inactive state which then can be called back at any time by messaging them in this persistent sub agent setup. I talked a little bit about uh continual harness uh which we have in a a prior paper of mine which talks about cuding the entire uh harness state. Um this is another feature that we want in our coding agents leverage all of our prior
所以父 session 可以创建、拉起一个新的 RLM sub-agent，它们被发出去，跑某个任务，完成后把某个结束状态汇报给父 session。然后它们闲着，但还在你的 RAM 里。父 session 随时可以给其中一个 sub-agent 发消息，让它继续干。它还保有你一路攒下来的那些好 context，这样你不会丢信息也不必重用先前 context 里已经长出来的信息。当然我们也不想占用太多 RAM，所以可以把它们卸载成非活动状态，之后随时发消息再召回来，这就是这套持久 sub-agent 的设定。我刚才提了一点 continual harness，那是我之前一篇论文，讲的是对整个 harness 状态做 CRUD。这也是我们希望编码 agent 具备的能力：用上我们先前所有的
嘉宾
history. So you can imagine like some set of trajectories where they have some actions and outcomes or something happened um at a at each turn. And so we just kind of want to expose the ability for the agent to leverage all that information in order to update what the future harness is going to look like. Are we do we need to change our system prompt? Do we need to create some skills? And skills I think of as a set of instructions or a program in order to achieve some some specific goal. uh memory which could just be long-term storage about things that are important as well as the sub aent specifications that we talked about in this very persistent manner. Were there uh certain sub aents that we want to reuse at a later time because the context is useful and just having the ability to do this kind of reflection or refinement um over time. It's very powerful for the models to have. They're not perfect at this right now, but this is one of the the capabilities that you want to you want
历史。可以想象有一组轨迹，每一轮都有一些动作和结果，或者发生了什么。我们只是想把这种能力暴露给 agent，让它用上全部这些信息，去更新未来的 harness 会长成什么样。要不要改 system prompt？要不要创建一些 skills？我把 skills 理解成一套指令或一段程序，用来达成某个具体目标。还有 memory，可以只是关于重要事情的长期存储，以及我们刚才说的、以非常持久的方式保存的 sub-agent 规格。有没有某些 sub-agent 以后还想复用，因为那份 context 有用？再加上随时间做这种 reflection 或 refinement 的能力。这对模型来说非常强。它们现在做得还不完美，但这正是你希望
嘉宾
to build your harness such that it is a bit better than what the current models are able to do. So then you can get those reasoning traces and use that to leverage your next iteration of model and they'll be able to handle the harness and be able to bootstrap themselves into a higher and higher performance. One of the coolest features that we have in uh Prime Agent um that we we've had since the beginning of when I was working on this, this is one of the first things I added um is the ability to message between any any two agents um within like some nuclear family setup, parents, children, uh siblings. Um and the reason why I did this is because I was I was constantly trying to figure out what's the best way to like myself to manage all of the agents I have doing everything for me in five different directions, five billion different directions every day. Um, and it would be so much better if they could just like share their contacts directly with each other and coordinate. And turns out that's fantastic for like typical software engineering and long horizon jobs as well. Uh, the last thing
把 harness 建成比当前模型能力稍高一点的东西。这样你就能拿到那些推理轨迹，用来推动下一轮模型，它们就能驾驭这个 harness，并把自身 bootstrap 到越来越高的性能。Prime Agent 里最酷的功能之一，从我一开始做这事就有，也是我最先加的之一：任意两个agent 之间能发消息，在一种核心家庭结构里：父、子、兄弟。我之所以这样做，是因为我自己一直在想：我每天让这些 agent 朝五个方向、五十亿个方向替我干活，最好的管理方式是什么。如果它们能直接共享各自的 context、互相协调，会好得多。结果发现，这对典型的软件工程和长程任务也同样非常好。最后一件
嘉宾
that we look at when it comes to how did we want to design our harness is we were really thinking about long horizon performance. I want to go run some jobs and I don't want to have to babysit my agents the entire time and when when I'm ready to come back and check in, I can check in with them and see what's going on. And this is a perspective that I also really lack seeing in a lot of the evaluations that we're looking at. Uh a lot of times if you run a model for not enough time or say, oh well the model stopped working after this amount of budgets, but then this other model kept working with using more budgets. Well, first of all, you're not even using the same fixed expenditure to compare the models. But second of all, that could also be hiding performance that you're missing. Uh the way that I look at long horizon performance eval is that I want to see what's the practical plateau. At what point will we only get incremental gains in performance as I throw more test time tokens at it? I have a couple experiments that I'm going to show after
我们在设计 harness 时看的，是长程表现。我想去跑一些任务，不想全程盯着我的 agent。等我准备好回来查看时，我可以去跟它们对一下，看看进展。这个视角，我在很多现有评测里几乎看不到。很多时候，如果你跑模型的时间不够，或者说：哦，这个模型在这么多预算之后就不干了，另一个模型用了更多预算还在继续。首先，你比较模型时甚至没用同一笔固定开销。其次，这也可能把你错过的性能藏起来了。我看长程表现评测的方式是：我想看到实际的平台在哪。从哪一点开始，我再砸更多 test-time token，性能也只是增量上涨？等会儿讲完我们怎么做成
嘉宾
we've shared design philosophy here about how we created uh prime agent. Um we're going to talk a little bit about test time scaling and uh our our TI results as well as looking at um you know does is it actually helpful and why is it actually helpful for our information management for the ripple that we're working on these long contexts and then um when we have these really really long like almost ultra horizon long horizon uh tasks uh how do we sustain these like multi-day work and like what actually goes on when we have these refinements um over like these settings that can last like a week at a time or more. So, this is a result that you probably all seen. We actually have a one additional data point that we added here that we didn't include in our original result uh just to compare across harnesses. We solved this uh we went out, we're trying to figure out what is the the best uh eval that people care about these days when we're running our harnesses and we're like, "Oh, we should do RKGI." I like, "Oh, yeah. Yeah, I remember. I I ran some results
Prime Agent 的设计理念之后，我会展示几个实验。我们会讲一点 test-time scaling，以及我们的 TI 结果；再看它对信息管理、对我们在这些长 context 上用的 REPL，到底有没有用、为什么有用。然后，当我们面对这些非常非常长、几乎是超长程的任务时，我们怎么把这种持续多日的工作撑住以及在这些一次能持续一周甚至更久的设定里，refinement 实际在发生什么。这是一个你们大概都见过的结果。我们这里多加了一个原来结果里没有的数据点，只是为了跨 harness 比较。我们当时在想：现在跑 harness，大家最关心的评测是什么？然后说：哦，应该做 ARC-AGI。我心想：对，我记得，我跑过一些结果
嘉宾
with continual harness and we got 20% with um Gemini Flash uh or sorry, Gemini Pearl." Um so, I I think we can get at least 20%. people who think that's really cool that our like general harness that didn't even like wasn't even structured for ARHI did really well. So I went online I was like okay I need to find a good system prompt because I don't want to make sure that we're losing information. So I found a another community leaderboard called prolong and I just grabbed their system prompt and I was like okay I'm going to grab their system prompt forget the rest and I'm just going to throw this directly into prime agent. Uh and then I ran this and I was like oh my god the first run that I got it hit 99.9% and then I looked at the logs and I was cheating. Okay. So, I was like, "Okay, I got to do proper sandboxing here. Like, let's set this up properly." Uh, and then so I spent another day on this. And then, and then I went back and I was like, "Oh my god, I got 78% with GPT soul. Like, this is going to be a great result." Um, and then we're back. It's like, oh, let's compare a couple other ones. And
是用 continual harness，Gemini Flash——抱歉，Gemini Pro——拿到了 20%。所以我想至少能到 20%。有人觉得这很酷：我们这个通用 harness 甚至不是为 ARC-AGI 专门设计的，却表现得很好。于是我上网，心想得找一份好的 system prompt，别把信息丢掉。我找到另一个社区榜，叫 Prolong，直接把他们的 system prompt 拿过来。我说：好，只要他们的 system prompt，别的不管，直接扔进Prime Agent。然后我一跑，第一轮就到了 99.9%。我去看日志，发现自己在作弊。好吧。我说：得做正确的沙箱，把实验搭干净。于是我又花了一天。然后再跑，我心想：天哪，GPT Soul 拿到了 78%，这会是个很好的结果。然后我们回来，说：再拿几个别的比一比。
嘉宾
so, it's again, we just took the prompt, uh, general prompt that basically says, uh, use a world model to solve ARC AGI 3. Uh, here are the actions that you can take. um you have uh and then the general system prompt for prime agent which is like you have a ripple you can call sub agents uh you can use the it programmatically um and and so we went through I went through the traces and it's basically doing a bunch of different um like calls of the coding in order to like check out these different scenarios and analyzing the images and doing like image processing and it's a lot of really cool um stuff that uh it seems like it was doing reasonable reasoning while leveraging the the ripple that we had um as like one of the main things that was able to enable build this. Uh so I went through and I ran a couple other ones. We did GPT tero 25.7% which is really cool. You can see that compared to like what were the um like the week before we did this uh open AAI was like the guys the harness matters a lot when you're doing
所以还是一样，我们只用了那份 prompt，一份通用 prompt，大意是：用一个世界模型来解 ARC-AGI 3。这些是你可以采取的动作。再加上 Prime Agent 的通用 system prompt：你有一个 REPL，可以调 sub-agent，可以程序化地用它。我把轨迹过了一遍，它基本上就是在做各种编码调用，去查看不同场景、分析图像以及做图像处理，很多很酷的操作。看起来它在做合理的推理，同时把我们的 REPL 当作能把这套东西做起来的主要能力之一。于是我又跑了几个。GPT-5 拿到了 25.7%，这很酷。你可以对比一下：我们做这事的前一周，OpenAI 还在说，做评测的时候 harness 非常重要
嘉宾
evaluations. We use the responses API. This is the result that we got. Um and we we ran Terra and and got almost like we we didn't run to completion this one but we got really good results in comparison. And then we go um that that we're already achieving higher than some of like the GBT soul extra high which was crazy. And then uh we went and we did Opus and hit 95.5%. We're like that's insane. We also compared to a lot of the other harnesses. So some people ask me like did you run this with cloud code? Uh I did. Um unfortunately the results weren't very good. Um, and so rather than having bad results, I just deferred to the the original cloud code results and some other people have run it uh with similar configurations to prime agent and gotten much better results since then. Um, but what's interesting is that a lot of the really popular harnesses don't necessarily do well when prime agent does well. So like for air agent, um, we spent a lot of money very quickly and uh, we had to cut
我们用的是 responses API，这是我们拿到的结果。我们跑了 Terra，虽然这一次没有跑完，但对比下来结果已经很好。然后我们发现，我们已经高于某些 GPT Soul extra high 的成绩了，这很疯狂。接着我们跑了 Opus，到了 95.5%。我们当时就觉得这太离谱了。我们还跟很多其他 harness 比过。有人问我：你用 Claude Code 跑过吗？跑过。可惜结果不太好。所以与其摆出差结果，我直接沿用了 Claude Code 原来的成绩。后来也有人用和 Prime Agent 类似的配置跑，结果好了很多。但有意思的是：很多很火的 harness，在 Prime Agent 表现好的地方，它们未必好。比如 Air Agent，我们很快花掉很多钱，不得不砍掉
嘉宾
it off because I spent like $5,000 without making much performance. Um, not saying this is the best they could do, but it cost a lot of money to do so. Uh so I think that the cost to performance uh ratio is very important and one of the things that does save money is being able to programmatically work with your context. Uh we ran a bunch of long horizon um evals as well like oolong and some coding uh emulator bench which is going to come out soon which is a program bench alternative and we found that it was mainly parody or slightly better than these other harnesses like you across different models versus doing like pimono cloud codecs with glm 5.2 to Opus 5 and 5.6 as our setting. Another one I thought was really cool is we have this like program bench alternative called emulator bench where we're trying to reproduce entire emulators of computer systems or in this case creating like a Game Boy Color and check that out. And we found that what's
因为我花了大概五千美元，性能却没涨多少。不是说他们不可能做得更好，但这样做很贵。所以我觉得性价比非常重要。能真正省钱的一件事，就是能程序化地处理你的 context。我们还跑了一批长程评测，比如 Oolong，还有一些编码的 EmulatorBench，马上会发布，是 ProgramBench 的替代。我们发现它大体持平，或略好于这些其他 harness。我们在不同模型上比过，比如 Prime、Claude Code、Codex，设定是 GLM 5.2 到 Opus 5 和 5.6。另一个我觉得很酷的，是我们这个叫 EmulatorBench 的 ProgramBench 替代：我们试图复现整套计算机系统的模拟器，这个例子里是做一台 Game Boy Color，再去验收。我们发现有意思的是
嘉宾
really interesting is because it has this um ripple access in the RLM, it's able to use these programs in order to kind of do these like out of experiment uh loop designs in order to um try things out in a lot more expressable and free way before submitting the final solution to the greater. Uh we also tried this with uh GPU kernels um and we got about par results uh across different um both soul and kimico. One is better, one is worse. about par um which so we we're not overfit to like any one particular um evaluation here. Um what's interesting for the long horizon stuff is we had some auto research uh experiments that we did with the nano GPT speedrun but we scaled it up. We said let's give it uh 8 by H200 for um a week and see what happens. And you might be like, okay, prime age is going to do so much better, right? Because it's able to do all this
因为它在 RLM 里有 REPL 访问权，它能用这些程序做实验循环之外的设计，用表达力更强、更自由的方式先试，再把最终方案提交给评测。我们也用 GPU kernel 试过，结果大致持平，Soul 和 Kimi 上都是。一个更好，一个更差，总体持平。所以我们并没有过拟合到某一个评测上。长程这边有意思的是，我们做了一些 auto research 实验，基于 nanoGPT speedrun，但规模放大了。我们说：给它 8 张 H200，跑一周，看看会怎样。你可能会想：Prime Agent 会好得多，对吧？因为它能做所有这些
嘉宾
programming. Uh, it's a little high variance. We can't attribute um any of the benefits to with the harness versus the model there because it's a very hard task. But what we can do is inspect a lot of the behavior that we've seen. And what's really interesting is that we're seeing models like deep 6v4, GLM 5.3, and Kim K3. Um, you can tell these were done a little more recently than our first results. Uh, and we took these and they were doing like what we call out of loop experiments. So we were trying to say how can I run experiments on like the CPU and like look at the parameterization and do hyperparameter search and analyze the data so that I don't have to spend like all my time running expensive H200 experiments uh because that takes the majority of the time. So it's it's running experiments that are not the main experiment in order to optimize them. I think that's really cool behavior that we're seeing uh as we we shape what would be what kind of things we need to for the expressability for prime agent. So you can use like really good auto research because you can imagine if it's good at
编程。方差有点高。这个任务太难，我们没法把收益归因到 harness 还是模型。但我们可以去看大量观察到的行为。很有意思的是，我们看到 DeepSeek V4、GLM 5.3、Kimi K3 这类模型——可以看出这些比我们第一批结果更新一点——它们在做我们所谓的循环外实验。也就是在说：我怎么在 CPU 上跑实验，去看参数化、做超参搜索、分析数据，这样就不必把所有时间花在昂贵的 H200 实验上，因为那才占了大部分时间。所以它在跑的不是主实验，而是为了优化主实验的那些实验。我觉得这是我们看到的很酷的行为，也在塑造 Prime Agent 需要什么样的表达力。所以你可以拿它做很好的 auto research。可以想象，如果它擅长
嘉宾
auto research, it'll be good with you. It be even better with a human in the loop to bootstrap your experiments. Uh and finally, we also streamed a 7-day factorial run which used a total of 633 agents um across uh 23 million tok output tokens in order to make like steady uh tech technological advancement across the tech tree to continue to progress over time. And here it uh one of the main benefits is I can use like these sub aents that can divvy up into different tasks in the factory in order to research and build and gather resources and build the next items to design the factory. Um as well as it can use the refinement to leverage what happened in the past in order to help in the future um over these very long context so it doesn't get stuck. And one of the most interesting things here is that it does not get stuck and it continues to make technology progression even at the end of our uh stage. Um this
auto research，它跟你配合也会好；有人在环里帮你 bootstrap 实验，会更好。最后，我们还直播了一场 7 天的 Factorio 运行，一共用了 633 个 agent、2300 万输出 token，为的是在科技树上稳步推进，持续往前走。这里一个主要好处是，我可以用这些 sub-agent，把工厂里的不同任务拆开：研究、建造、收集资源、做出下一件物品，去设计工厂。它也能用 refinement，把过去发生的事用到未来，撑过这些非常长的 context，这样它不会卡住。最有意思的一点是：它没有卡住，即使到了我们这个阶段的末尾，仍在推进科技。这更像
嘉宾
is more like a Gemini plays Pokemon kind of uh conclusion here. If there's one thing that uh I find interesting today u but like what takeaways you should actually add to your own harness. Um I think that you should think about agentic context management. Uh you should think about swarms and looking into further depth RLMs and trying to run standardized eval. All of the results that we can they showed today can be run with our uh verifiers uh package that we have at Prime Inslect. Um and shout out to my collaborators who are fantastic and I love working with. Thanks.
Gemini 玩宝可梦那种结论。如果今天只带一样东西走，也就是你真正该加进自己 harness 的 takeaway：我认为你该想想 agent 式的 context 管理；该想想 swarm，更深入地看 RLM，并试着跑标准化评测。我们今天展示的所有结果，都可以用我们 Prime Intellect 的 verifiers 包来跑。也感谢我的合作者们他们非常出色，我很喜欢和他们共事。谢谢。[掌声]
主持人
All right, next up we have John.
好，下一位是 John。
嘉宾
Hey everybody. I'm super excited to talk about a project um that we've been working on at Stanford. Um, I've been working on this with Ivanka Orion, my my co-lead author, as well as our adviserss Hazeni and Christopher Ray. So, personal AI is everywhere, but it's mostly cloudbound today. Uh, we see lots of different harnesses and projects focused on making daily writing, research, coding, and scheduling. But projects like OpenClaw and Hermes agent typically rely on cloud LMS um for most of the intelligence and for most of the most of the queries. Um, what does this mean? It means that it's pretty costly. You're getting thousands and thousands of dollars in API costs if you aggregate it over a year. Um, it's not private.
大家好。我非常兴奋能讲我们在 Stanford 做的一个项目。我和共同第一作者 Ivanka Orion 一起做的，还有我们的导师 Hazy 这边，以及 Christopher Ré。所以，个人 AI 到处都是，但今天大多还是绑在云上的。我们看到很多不同的 harness 和项目，在做日常写作、研究、编码和日程。但像 OpenClaw 和 Hermes agent 这类项目，智能的大部分、查询的大部分，通常都依赖云端 LLM。这意味着什么？意味着很贵。按一年加起来，API 费用能到成千上万刀。而且不私密。
嘉宾
You're often sending your most personal um, data to LMS up in the cloud and you don't necessarily know where all that data is going. Um, it also requires you to rent your intelligence as opposed to just simply owning it out of the box. And finally, it tends to consume orders of magnitude more energy than just running these LMS on your laptop. And so the local LMS are finally good enough to actually run a lot of these queries that people care about. And so we see that um the the current LMS of today are only 6 to 12 months uh behind whatever is the state-of-the-art frontier models um of before. So you see um LMS today such as Quen 3.8 27B um that achieve roughly the same performance as like Claude 4.6 Opus um back in the day. So that was kind of the state-of-the-art model back in August 2025. Um, and that gap seems to be closing uh more and more as the hardware accelerators that we have um
你常常把最私人的数据送到云上的 LLM，而且不一定知道这些数据会去哪。这还意味着你得租用智能，而不是开箱即拥有。最后，它消耗的能量，往往比直接在笔记本上跑这些 LLM 高几个数量级。而本地 LLM 终于够好了，真能跑很多人在乎的那些查询。所以我们看到今天的 LLM，只落后于此前最前沿模型大约 6 到 12 个月。比如今天的 Qwen 3.8 27B，大致能达到当年 Claude 4.6 Opus 的水平。那大概是 2025 年 8 月的前沿模型。而且这个差距似乎还在不断缩小，因为我们拥有的硬件加速器
嘉宾
for our laptops and for our workstations get better and better. Uh, just this week we saw a new release from Apple um with the new Mac Mini. And so we're seeing this renewed focus from Apple as well as Nvidia to build accelerators specifically for personal use cases. And so with this project, we wanted to explore the the main question of can we build the core of a personal AI stack, namely the model inference, the agent execution, the memory, the learning, basically the parts that are mostly reliant on the cloud today entirely on device while staying competitive with these cloudonly stacks. And so we decided to propose open Jarvis. Um name needs no needs no explanation. Um but we wanted to explore just how much of this we could run on device completely for free uh while preserving uh security, privacy and quality. And so to construct open Jarvis we wanted to create the simplest set of primitives for which you define any sort of harness or or personal AI stack. Um the first
笔记本和工作站上的，都越来越好。就在这周，Apple 发布了新的 Mac Mini。所以我们看到 Apple 和 NVIDIA 都重新把重点放在为个人场景做加速器。于是这个项目想探索的核心问题是：我们能不能把个人 AI 技术栈的核心，也就是模型推理、agent 执行、memory、学习，基本上就是今天大多还依赖云的那些部分，完全放到端上，同时仍能跟这些纯云技术栈竞争。于是我们提出了 open Jarvis。名字不用解释。我们想探索：这里面有多少可以完全免费地在端上跑，同时保住安全、隐私和质量。为了构建 open Jarvis，我们想给出最简单的一组原语，用来定义任何一种 harness 或个人 AI 技术栈。第一
嘉宾
one is whatever user interfaces you need to use. Um the second one is the actual agentic logic around composable reasoning and using different kinds of intelligence and tools. Uh for the intelligence, it's whatever LM you're using as your engine for keeping everything going. Um so this could be Quen, GBDO, OSS, Gemma 3N. Um and then whatever actual inference engine you need to run it. So this could be O Lama, um Llama CBP, VLM, SG Lang, um including whatever hardware you're running it on.
个是你需要用的那些用户界面。第二个是真正的 agent 逻辑：可组合的推理，以及使用不同种类的智能和工具。智能这一层，就是你用来驱动一切的那台 LM 引擎。可以是 Qwen、GPT-OSS、Gemma 3N。然后是实际跑它所需的推理引擎，可以是 Ollama、llama.cpp、vLLM、SGLang，再加上你跑它所在的硬件。
嘉宾
So this could be Apple Silicon, Nvidia, whatever you need. um for actually making all of these agents and intelligence useful you need some set of tools in memory that can be run through a standard MCP protocol um and you need some sort of uh set of primitives for actually doing learning whether it's prompt based techniques like Japa or DSPI um whether it's weight based techniques like gpo and sftt and Laura um you need some way to actually get this agent to improve over time and actually be able to make it more personal and more effective and so to kind of walk through like what opens looks like um we tried to go with all of the standard um interfaces that people are already accustomed to. Um so we wanted to give people the ability to interact with it through a desktop and actually just run it as they would normally expect, but then see all of the savings that they're getting in terms of dollars and energy. Um we also wanted to give people the ability um to run different kinds of continuous agents. So different kinds of agents that are
可以是 Apple Silicon、NVIDIA，你需要什么就是什么。要让这些 agent 和智能真正有用，你需要一组能通过标准 MCP 协议跑的工具和 memory；还需要一组做学习的原语，不管是基于 prompt 的技术，比如 GEPA 或 DSPy，还是基于权重的技术，比如 GRPO、SFT 和 LoRA。你需要某种方式让这个 agent 随时间改进，真正能让它更个人化、更有效。顺着讲一下 open Jarvis 长什么样：我们尽量走人们已经习惯的那些标准界面。我们希望大家能通过桌面跟它交互，按他们平时预期的方式去跑，同时看到自己在钱和能耗上省了多少。我们也希望大家能跑不同类型的持续 agent。也就是不同类型的、