We're Moving from the Age of Scaling to the Age of Research

00:00

You know what's crazy that all of this is real? >> Yeah.

你知道最疯狂的是什么吗？

00:03

Meaning what? >> Don't you think so? >> Meaning what? >> Like all this AI stuff and all this area.

这一切都是真的。

00:09

Yeah.

>> 是啊。

00:10

That it's happen like >> isn't it straight out of science fiction? >> Yeah.

什么意思？

00:14

Another thing that's crazy is like how normal the slow takeoff feels.

>> 你不觉得吗？

00:19

The idea that we'd be investing 1% of GDP in AI, like I feel like it would have felt like a bigger deal, you know, where right now it just feels like

>> 什么意思？

00:28

>> you get used to things pretty fast.

>> 人适应东西很快。

00:30

Turns out Yeah.

确实。

00:31

But also it's kind of like it's abstract like what does it mean?

但另一方面，这也有点抽象，它到底意味着什么？

00:34

What it means that you see it in the news. >> Yeah. >> That such and such company announced such and such dollar amount, >> right? >> That's that's all you see, >> right? >> It's not really felt in any other way so far. >> Yeah.

你只在新闻里看到它。

00:46

Should we actually begin here?

>> 对。

00:48

I think this is an interesting discussion. >> Sure. >> I think your point about well from the average person's point of view, nothing

>> 某某公司宣布了某某金额， >> 对吧？

00:55

is that different will continue being true even into the singularity. >> No, I don't think so.

会被感受到。

01:01

Okay.

>> AI会渗透到经济中。

01:02

Interesting. >> So the thing which I was referring to not feeling different is okay.

有非常强大的经济力量在推动这一点，而且我觉得影响会非常强烈地显现出来。

01:07

So such and such company announced some difficult to comprehend dollar amount of investment >> right >> I don't think anyone knows what to do with that. >> Yeah. >> But I think that the impact of AI is

>> 你预计什么时候会有这种影响？

01:21

going to be felt. >> AI is going to be diffused through the economy.

怎么解释它们在做eval时表现那么好。

01:26

There are very strong economic forces for this and I think the impact is going to be felt very strongly. >> When do you expect that impact?

>> 嗯。

01:37

I think the models seem smarter than their economic impact would imply. >> Yeah, this is one of the very confusing things about

>> 你看那些eval，会觉得，这些eval挺难的， >> 对吧？

01:47

the models right now.

模型一方面能做这些惊人的事，另一方面又在某些情况下重复自己。

01:49

How to reconcile the fact that they are doing so well on evals. >> Mhm.

举个例子，比如你用VIP coding做点东西，然后到了某个地方出了个bug，你告诉模型能不能修一下这个bug？

01:54

And you look at the evals and you go, those are pretty hard evals, >> right? >> They're doing so well, >> but the economic impact seems to be dramatically behind.

>> 对。

02:07

And it's almost like it's it's very difficult to make sense

>> 模型说：“天哪，你说得太对了，我有个bug，让我去修。

02:12

of how can the model on the one hand do these amazing things and then on the other hand like repeat itself twice in some situation in a kind of a an example would be let's say you use VIP coding to do something and you go to some place and then you get a bug and then you tell the model can you please fix the bug? >> Yeah.

>> 然后你告诉它，你有了这个新bug，它又说：“天哪，我怎么能这样，你又对了。

02:30

And the model says, "Oh my god, you're so right.

” >> 然后又把第一个bug带回来。

02:33

I have a bug.

你可以在两者之间来回切换。

02:34

Let me go fix that." And it introduces a second bug. >> Yeah.

>> 对。

02:38

>> And then you tell it you have you have this new second bug and it tells you, "Oh my god, how could I've done it?

>> 这就像，怎么可能？

02:44

You're so right again." >> And brings back the first bug.

>> 对。

02:48

And you can alternate between those. >> Yeah. >> And it's like, how is that possible? >> Yeah. >> It's like I'm not sure.

>> 我不确定。

02:55

But it does suggest that the something strange is going on.

但这确实说明有什么奇怪的事在发生。

02:59

I have two possible explanations.

我有两个可能的解释。

03:00

So here this is the more kind of a whimsical explanation is that maybe a real

第一个比较随意的解释是，也许pre-training

03:05

training makes the models a little bit too single-minded and narrowly focused a little bit too I don't know unaware even though it also makes them aware in some other ways and because of this they can't do basic things but there is another explanation which is back when people were doing pre-training

让模型变得有点过于一根筋、过于专注，有点——我也不确定——甚至缺乏意识，尽管在其他方面又让它们有意识，正因为如此，它们做不了基本的事情。

03:31

the question of what data to train on was answered because the that answer was everything. >> Yeah. >> When you do pre-training, you need all the data.

训练数据的问题其实早就有了答案，答案就是“所有数据”。

03:40

So you don't have to think is it going to be this data or that data. >> Yeah. >> But when people do RL training, they do need to think.

>> 对。

03:49

They say okay, we want to have this kind of RL training for this thing and that kind of RL training for

>> 做pre-training的时候，你需要全部的数据，所以根本不用纠结是用这个数据还是那个数据。

03:56

that thing.

那个东西。

03:57

And from what I hear, all the companies have teams that just produce new RL environments and just add it to the training mix.

据我所知，所有公司都有专门团队不断生成新的RL环境，然后直接加入训练组合。

04:05

And then the question is, well, what are those?

问题在于，这些环境到底是什么？

04:08

There are so many degrees of freedom.

自由度太多了，你能创造出的环境种类极其丰富。

04:10

There is such a huge variety of environments you could produce.

其中一种做法——我觉得这可能是无意中发生的——就是人们从评测标准中获取灵感。

04:14

And one of the one thing you could do, and I think that's something that is done inadvertently,

你会说：“嘿，我希望我们的模型发布时表现特别好，我希望评测结果看起来很棒。

04:20

is that people take inspiration from the evals. you say, "Hey, I would love our model to do really well when we release it.

人们从评估中汲取灵感。

04:29

I want the EVOS to look great." What would be RL training that could help on this task, right?

你会说，“嘿，我希望我们的模型在发布时表现超棒，我希望评估结果看起来特别漂亮。

04:35

I think that is something that happens and I think it could explain a lot of what's going on.

”那什么样的RL训练能帮上这个任务呢，对吧？

04:41

If you combine this with generalization of the models actually being inadequate,

我觉得这种事情确实存在，而且可能解释了很多现象。

04:46

that has the potential to explain a lot of what we are seeing. this disconnect between eval performance and actual real real world performance which is something that we don't today exactly even understand what what we mean by that I I like this idea that the real reward hacking is a human researchers who are too focused on the evals um I think there's two ways to understand or

这有可能解释我们看到的很多现象。

05:13

to try to think about what what you have just pointed out one is look if it's the case that simply by becoming superhuman at a coding competition, a model will not automatically become more tasteful and exercise better judgment about how to improve your codebase.

试着想想你刚才指出的问题——如果仅仅因为在编程竞赛中达到超人水平，模型并不会自动变得更有品味、能更好地判断如何改进你的代码库，那你就应该扩展环境套件，不只是测试它在编程竞赛中的最佳表现。

05:29

Well, then you should expand the suite of environments such that you're not just testing it on having the best performance in coding competition.

它也应该能够为X、Y或Z事情做出最佳类型的应用。

05:38

It should also be able to

另一个你可能在暗示的点是：为什么一开始会认为在编程竞赛中变得超人就一定不会让你成为更有品味的程序员呢？

05:40

make the best kind of application for X thing or Y thing or Z thing.

为X、Y或Z这类东西做出最好的应用。

05:44

And another maybe this is what you're hinting at is to say why should it be the case in the first place that becoming super human at coding competitions doesn't make you a more tasteful programmer more generally.

另一个可能是你在暗示的是，为什么在编程竞赛中变得超级厉害，并不一定让你成为一个更有品味的程序员。

05:56

Maybe the thing to do is not to keep stacking up the amount of environments and the diversity of environments to figure out approach with let you learn

也许关键不是不断堆叠更多的环境和多样性环境，来找出如何从一个环境中学到东西，并提升在另一个环境上的表现。

06:05

from one environment and improve your performance on something else.

从一个环境里提升，然后在别的事情上表现更好。

06:10

So I have I have an analog a human analogy which might be helpful.

所以我有个类比，人类的类比，可能有点帮助。

06:14

So even the case let's take the case of competitive programming since you mentioned that and suppose you have two students one of them work decided they want to be the best competitive programmer.

就拿你提到的竞技编程来说吧，假设有两个学生：一个决定要成为最厉害的竞技程序员，于是花了一万个小时专门练这个领域，把所有题目都刷完，记住所有证明技巧，变得非常非常熟练，能又快又准地实现所有算法。

06:26

So they will practice 10,000 hours for that domain.

这么一来，他成了最顶尖的之一。

06:29

They will solve all the

第二个学生呢，觉得竞技编程挺酷的，可能只练了一百个小时——少得多——但也做得很好。

06:31

problems, memorize all the proof techniques and be very very you know be very skilled at quickly and correctly implementing all the algorithms and by doing by doing so they became the best one of the best student number two thought oh competitive programming is cool maybe they practiced for 100 hours >> much much less and they also did really well which one do you think is going to do better in their career later on >> the second >> right and I think that's basically

问题、记住所有证明技巧，并且非常非常熟练地快速正确实现所有算法，通过这样做他们成为了最顶尖的学生之一。

07:00

what's going on.

怎么回事呢。

07:00

The models are much more like the first student but even more because then we say okay so the model should be good at competitive programming so let's get every single competitive programming problem ever and then let's do some data augmentation so we have even more competitive programming problems >> yes >> and we train on that and so now you got this great competitive programmer and with this analogy I think it's more intuitive I think it's more intuitive with this analogy that yeah okay so if

现在的模型更像第一个学生，但更夸张——我们说，好，模型要擅长竞赛编程，那就把所有竞赛编程题都拿来，再做点数据增强，搞出更多题目 >> 对 >> 然后用这些数据训练，结果你就得到了一个超强的竞赛程序员。

07:26

it's so well trained okay it's like all the different algorithms and all the proof techniques are like right at it at its fingertips and it's more intuitive that with this level of preparation it not would not necessarily generalize to other things. >> But then what is the um analogy for what the second student is doing before they do the 100 hours of fine-tuning. >> I think it's like they have it.

它训练得特别好，对吧？

07:52

I think it's the it factor.

就好像所有不同的算法和证明技巧都信手拈来，而且更直观的是，这种程度的准备不一定能泛化到其他事情上。

07:54

>> Yeah. >> Right.

>> 嗯。

07:55

And like I know like when I was in undergrad, I remember there was there was a student like this that studied with me.

>> 对。

08:01

So I I know it exists. >> Yeah.

而且我知道，比如我读本科的时候，我记得有个同学跟我一起学习，就是这种人。

08:03

I think it's interesting to distinguish it from whatever pre-training does.

所以我明白确实存在。

08:07

So one way to understand what you just said about we don't have to choose the data in pre-training is to say actually it's not dissimilar to the 10,000 hours of practice.

>> 对。

08:17

It's just that you get that 10,000 hours of practice for free because it's already somewhere in the

我觉得有意思的是，要把它跟 pre-training 做的事情区分开。

08:22

pre-training distribution.

预训练分布。

08:24

But it's like maybe you're suggesting actually there's actually not that much generalization for pre-training.

但你的意思可能是，实际上预训练并没有带来那么多泛化能力。

08:30

There's just so much data in pre-training but it's like it's not necessarily generalizing better than RL. >> Like the main the main strength of pre-training is that there is a so much of it. >> Yeah. >> And b you don't have to think hard about what data to put into pre-training. >> And it's a very kind of natural data and it does include in it a lot of what people do.

预训练只是数据量特别大，但它并不一定比RL泛化得更好。

08:50

>> Yeah. people's thoughts and a lot of the features of you know it's like the whole world as projected by people onto text. >> Yeah. >> And pre-training tries to capture that using a huge amount of data.

>> 对。

09:06

It's it's very the pre-training is very difficult to reason about because it's so hard to understand the manner in

人们的想法，还有好多特征，你知道，就是整个世界通过文字投射到人身上的那种东西。

09:16

which the model relies on pre-training data.

预训练分布。

09:19

And whenever the model makes a mistake, could it be because something by chance is not as supported by the pre-training data?

但好像你是在暗示，预训练其实并没有那么多泛化能力。

09:27

You know, and pre support by pre-training is maybe a loose term.

预训练数据量很大，但它不一定比强化学习泛化得更好。

09:32

I I don't know if I can add anything more useful on this, but I don't think there is a human analog to pre-training.

预训练的主要优势在于：第一，数据量特别大；第二，你不用费劲去想该放什么数据进去。

09:40

Um, here's analogies that

而且它本质上是很自然的数据，里面包含了大量人类的行为。

09:42

people have proposed for what the human analogy to pre-training is, and I'm curious to get your thoughts on why they're potentially wrong.

>> 太棒了。

09:51

One is to think about the first 18 or 15 or 13 years of a person's life when they aren't necessarily economically productive, but they are doing something that is making them understand the world better and so forth.

你得继续 scale。

10:05

And the other is to think about evolution as doing some kind

继续 scaling。

10:09

of search for three billion years which then results in a human lifetime instance.

对三亿年的搜索，最终凝结成一个人一生的实例。

10:14

And then I'm curious if you think either of these are actually analogous to pre-training or how how would you think about at least what lifetime human learning is like if not pre-training.

然后我好奇，你是否认为这两者中的任何一个实际上类似于预训练，或者你如何看待人类一生的学习，如果它不算是预训练的话。

10:25

I think there are some similarities between both of these two pre-training and pre-training tries to play the role of both of these >> but I think there are some big

我认为这两者与预训练之间有一些相似之处，预训练试图扮演这两者的角色 >> 但我认为有一些很大的不同。

10:34

differences as well.

差异也很大。

10:36

The amount of pre-training data is very very staggering. >> Yes. >> And somehow a a human being after even 15 years with a tiny fraction of that pre-training data they know much less. >> Yeah.

预训练数据的量非常非常惊人。

10:50

But whatever they do know they know much more deeply somehow and the mistakes like like already at that age you would not make mistakes that are make.

>> 对。

11:00

>> Yeah.

>> 对。

11:01

There is another thing you might say could it be something like evolution and the answer is maybe but in this case I think evolution might actually have an edge like there is this I remember reading about this case where some you know that one thing that neuroscientists do or rather one way in which neuroscientists can learn about the brain is by studying people with brain damage to different parts of the brain

还有一点你可能想说，会不会跟进化有关？

11:28

>> and and so and some people have the most strange symptoms you could imagine.

>> 有些人会出现你能想象到的最奇怪的症状，真的特别有意思。

11:33

It's actually really really interesting.

有一个相关的案例我印象很深。

11:36

And there was one case that comes to mind that's relevant.

我读到过一个人，他因为中风或事故导致脑损伤，影响了他的情绪处理能力。

11:41

I read about this person who had some kind of brain damage that took out I think a stroke or an accident that took out his emotional processing.

所以他完全感受不到任何情绪。

11:52

So he stopped feeling any emotion

所以他不再感受到任何情绪

11:55

and as a result of that you know he still remained very articulate and he could solve little puzzles and on tests he seemed to be just fine but he felt no emotion he didn't feel sad he didn't feel angry he didn't feel animated and he became somehow extremely bad at making any decisions at all it would take him hours to decide on which socks to wear and he would make very bad

>> 结果呢，他依然能说会道，能解小谜题，测试表现也完全正常，但他没有情绪——不悲伤、不愤怒、不兴奋。

12:18

financial decisions And that's very but does what what does it say about the role of our built-in emotions in making us like a viable agent essentially >> and I guess to connect to your question about pre-training >> it's like maybe pre- like maybe if you are good enough at like getting everything out of pre-training you can

>> 这说明了什么呢？

12:45

get you could get that as well but that's the kind of thing which seems is well it may or may not be possible to get that from pre-training.

>> 我想联系你刚才关于预训练的问题。

12:56

What is that clearly not just directly emotion?

那显然不完全是直接的情绪吧？

13:00

And it seems like some almost value function like thing which is giving telling you which decision to be like what the end reward for any

这看起来有点像某种价值函数，在告诉你该做哪个决定，以及每个选择的最终奖励是什么

13:12

decision should be and you think that doesn't sort of implicitly come from >> I think it could I'm just saying it's not one it's not 100% obvious. >> Yeah.

>> 就好像，如果你能从预训练中把一切学得足够好，也许也能得到那种能力。

13:20

But what is that like what how do you think about emotions and what is the ML analogy for emotions? >> It should be some kind of a value function thing. >> Yeah.

但这种事情，能不能从预训练中获得，还不一定。

13:28

But I don't think there is a great ML analogy because right now value functions don't play a very prominent role in uh the things people do. >> It might be worth defining for the audience what a value function is if if

那显然不完全是直接的“情绪”。

13:40

you want to do that. >> I mean certainly I I'll be very happy to do that.

>> 你觉得这不会隐含地从……中得来吗？

13:45

Right.

对。

13:45

So so when people do reinforcement learning the way reinforcement learning is done right now how do they do how do people train those agents?

那么，当人们做 reinforcement learning 的时候，现在 reinforcement learning 是怎么做的？人们是怎么训练那些 agent 的？

13:56

So you have your neural net and you give it a problem and then you tell the model go solve it and the model takes maybe thousands hundreds of thousands of actions

你有一个 neural net，然后给它一个问题，告诉模型去解决它，模型可能会采取成千上万次行动

14:08

or thoughts or something and then it produces a solution.

或者某种想法，然后它就会生成一个解决方案。

14:11

The solution is created and then the score is used to provide a training signal for every single action in your trajectory. >> Mhm.

解决方案被创造出来后，分数就会用来为你的轨迹中的每一个动作提供训练信号。

14:20

So that means that if you are doing something that goes for a long time, if you're training a task that takes a long time to solve, you will do no learning at all until you solve the until you came up with a proposed solution.

>> 嗯。

14:34

That's

所以这意味着，如果你在做一件耗时很长的事情，比如你在训练一个需要很长时间才能解决的任务，那么在你想出解决方案之前，你根本不会学到任何东西。

14:34

how reinforcement learning is done naively.

强化学习最朴素的做法。

14:38

That's how O1 R1 ostensibly are done.

O1和R1据说也是这么做的。

14:40

The value function says something like okay look maybe I could sometimes not always could tell you if you're doing well or badly.

价值函数会说，好吧，也许我有时候——不是总是——能告诉你做得好还是不好。

14:49

The notion of a value function is more useful in some domains than others.

价值函数的概念在某些领域比其他领域更有用。

14:55

So for example when you play chess and you lose a piece you know I messed up.

比如下棋的时候，你丢了一个棋子，你就知道自己搞砸了。

15:00

You don't need to play the whole

你不需要下完整盘

15:03

game to know that what I just did was bad and therefore whatever um whatever preceded it was also bad.

棋才知道刚才那步是坏的，因此之前的所有步骤也都是坏的。

15:10

So the value function lets you short circuit the weight until the very end.

所以价值函数能让你不用等到最后才得到反馈。

15:15

Like let's suppose that you started to pursue some kind of um okay let's suppose that you are doing some kind of a math thing or a programming thing and you're trying to explore a particular solution direction

比如，假设你开始尝试某种数学题或者编程题，你在探索某个特定的解题方向，

15:29

and after let's say after a thousand uh steps of thinking you concluded that this direction is unpromising.

然后，假设经过一千步的思考，你得出结论这个方向没有前途。

15:36

As soon as you conclude this, you could already get a reward signal a thousand time steps previously when you decided to pursue down this path.

一旦你得出这个结论，你其实就可以在一千步之前——也就是你决定走这条路的时候——就获得一个奖励信号。

15:45

You say, "Oh, next time I shouldn't pursue this path in a similar situation long before you actually came up with a proposed

你会说，“哦，下次在类似的情况下，我不应该走这条路”，而根本不用等到你真正提出一个

15:53

solution." M this was in the deepcar one paper is that the space of trajectories is so wide that maybe it's hard to learn a mapping from an intermediate trajectory and value and also given that you know in coding for example you'll have the wrong idea then you'll go back then you'll change something >> this sounds like such lack of faith in deep learning >> like I mean sure it might be difficult

解决方案。

16:19

but >> nothing deep learning can't Yeah. >> So my expectation is that like value functions should be useful and and I fully I fully expect that they will be used in the future if not already.

>> 我的意思是，当然可能很难，但>> 没什么是深度学习搞不定的。

16:38

What was I alluding to with the person whose emotional center got >> um damaged is more that

是啊。

16:47

maybe what it suggests is that the value function of humans is modulated by emotions in some important way that's hardcoded by evolution and maybe that is important for people to be effective in the world. >> That that's the thing I was actually planning on asking you.

是想说，这可能意味着人类的价值函数在某种程度上是由情感调节的，这种调节是进化硬编码的，而且可能对人类在世界上有效行动很重要。

17:04

There's something really interesting about emotions as a value function, which is that it's impressive that they have this

>> 这正是我本来想问你的。

17:12

much utility while still being rather um simple to understand.

实用性，同时却仍然相当简单易懂。

17:17

So I have two responses.

所以我有两个回应。

17:19

I do agree that compared to the kind of things that we learn and the things we are talking about, the kind of ads we talking about, emotions are relatively simple.

我同意，相比我们学到的东西和我们正在讨论的这些事，情绪相对简单。

17:30

They might even be so simple that maybe you could map them out in a human

它们可能简单到甚至可以在人类身上画出来。

17:36

understandable way.

用一种容易理解的方式来说。

17:37

I think it would be cool to do.

我觉得做这件事会很酷。

17:40

In terms of utility though, I think there is a thing where you know there is this complexity robustness trade-off where complex things can be very useful but simple things are very useful in a very broad range of situations.

但从实用性来看，存在一个复杂性与鲁棒性之间的权衡——复杂的东西可能非常有用，但简单的东西在非常广泛的情境下也很有用。

17:58

And so I think what what one way to interpret

所以我认为，对我们目前看到的现象，一种解读方式是：我们拥有的这些情绪，本质上是从哺乳动物祖先那里演化而来的，然后在人类进化阶段稍微做了点fine-tuning。

18:02

what we are seeing is that we've got these emotions that essentially evolved mostly mostly from our mammal ancestors and then fine-tuned a little bit while we were homminids just a bit.

>> 对。

18:14

We do have like a decent amount of social emotions though which mammals may lack but they're not very sophisticated and because they're not sophisticated they serve us so well in this very different world compared to the one that we've been living in.

人们一直在讨论scaling数据、scaling参数、scaling算力。

18:30

Actually they they

有没有一种更通用的方式来理解scaling？

18:31

also make mistakes.

所以，这里有一个视角，我觉得可能是对的。

18:32

For example, our emotions well I don't know does hunger count as an emotion debate.

过去ML的工作方式是，人们只是用各种东西做实验，试图得到有趣的结果。

18:38

It's debatable but I think for example our intuitive feeling of hunger is not succeeding in guiding us correctly in this world with an abundance of food. >> Yeah.

这就是过去一直在发生的事情。

18:50

People have been talking about scaling data, scaling parameters, scaling compute.

然后scaling的洞察出现了，对吧？

18:55

Is there a more general way to think

scaling laws，GPT-3。

18:58

about scaling?

>> 是的。

19:00

What are the other scaling axes?

还有哪些 scaling 的维度？

19:03

So the thing so so here here is a perspective here's a perspective I think might be might be true.

所以，这里有一个观点，我觉得可能是对的。

19:13

So the way ML used to work is that people would just think of it with stuff and try to and try to get interesting results.

以前机器学习的工作方式，就是人们随便拿些东西试试，看能不能搞出有意思的结果。

19:26

That's what's been going on in the past.

>> pre-training的重大突破在于意识到这个recipe是好的。

19:29

Then the scaling insight arrived, right?

所以你说，嘿，如果你把一些算力和一些数据混合进一个特定规模的神经网络，你会得到结果，而且你知道只要把这个recipe scale up，结果就会更好。

19:32

Scaling laws, GPT3.

这也很棒。

19:34

And suddenly everyone realized we should scale.

公司喜欢这一点，因为它提供了一种非常低风险的投资方式。

19:38

And it's just this, this is an example of how language affects thought.

这本身就是一个例子，说明语言如何影响思维。

19:44

Scaling is what just one word, but it's such a powerful word because it informs

“扩展”只是一个词，但它非常有力，因为它指明了方向。

19:51

people what to do.

>> 对。

19:52

They say, "Okay, let's let's try to scale things." And so you say okay so what are we scaling and pre-training was a thing to scale it was a particular scaling recipe. >> Yes >> the big breakthrough of pre-training is the realization that this recipe is good.

他们说：“好，那我们试试把东西做大。”然后你就问，那我们要扩展什么？预训练就是一个可以扩展的东西，它是一种特定的扩展方法。 >> 对 >> 预训练最大的突破，就是人们意识到这个方法是可行的。

20:11

So you say hey if you mix some compute with some data into a neural net of a certain size you will get results

所以你就说，嘿，如果你把一定量的算力和数据，放进一个特定规模的神经网络里，你就能得到结果。

20:19

and you will know that it will be better if you just scale the recipe up.

>> 来投入你的资源。

20:23

And this is also great.

这也很棒。

20:25

Companies love this because it gives you a very uh lowrisk way of investing. >> Yeah. >> Your resources. >> Yeah. >> Right.

公司特别喜欢这一点，因为它提供了一种风险很低的投资方式。 >> 是啊。 >> 去投入你的资源。 >> 对。 >> 对吧。

20:33

It's much harder to invest your resources in research.

相比之下，把资源投在研究上就难多了。

20:36

Compare that.

对比一下。

20:37

You know, if you research, you need to have like go forth researchers and research and come up with something versus get

你知道，做研究的话，你得让研究人员去探索、去发现，然后拿出成果，而不是像预训练那样直接得到东西。

20:45

more data, get more compute.

>> 对。

20:47

You know, you'll get something from pre-training.

而且，确实，根据大家在Twitter上说的各种话，有些人提到，Gemini好像找到了一种方法，能从预训练中榨出更多价值。

20:50

And indeed, you know, it looks like I based on various um um things people say on some people say on Twitter, maybe it appears that Gemini have found a way to get more out of pre-training.

但总有一天，预训练会用完数据。

21:01

At some point though, pre-training will run out of data.

数据显然是有限的。

21:04

The data is very clearly finite.

那接下来怎么办？

21:06

And so then, okay, what do you do next?

要么搞某种升级版的预训练，换个不同的配方。

21:08

Either you do some kind of a souped-up pre-training, different recipe from the

要么你搞一种加强版的预训练，换个不同的配方。

21:13

one you've done before, or you're doing a RL or maybe something else.

你之前做过的事，或者你在做强化学习，或者别的什么。

21:18

But now that compute is big, computer is now very big.

但现在算力变得很大，计算机现在非常大了。

21:21

In some sense, we are back to the age of research.

从某种意义上说，我们又回到了研究的时代。

21:24

So maybe here's another way to put it.

所以也许可以换个角度说。

21:27

Up until 2020, from 201 from 2012 to 2020, it was the age of research.

从2012年到2020年，那是研究的时代。

21:31

Now from 2020 to 2025, it was the age of scaling or maybe plus minus.

而从2020年到2025年，那是scaling的时代，或者差不多。

21:36

Let's add arrow bars to those years because people say this is

我们给这些年加个箭头吧，因为人们会说这太棒了。

21:40

amazing.

>> 实际上可以消耗相当多的 compute。

21:40

You got to scale more.

你知道，你要做非常非常长的 rollout。

21:42

Keep scaling.

>> 对。

21:43

The one word scaling.

所以产生这些 rollout 需要大量 compute，而每个 rollout 能学到的内容相对很少。

21:44

But now the scale is so big.

所以你确实可以花很多 compute，我可以想象，在现阶段，我甚至不会称之为 scaling，我会说，嘿，你在做什么？

21:46

Like is is it is the belief really that oh it's so big but if you had 100x more everything would be so different.

你做的事是不是你能做的最有生产力的事？

21:53

Like it would be different for sure but like is the belief that if you just 100x the scale everything would be transformed.

对。

22:01

I don't think that's true.

>> 你能找到一种更有生产力的方式去用你的 compute 吗？

22:03

So it's back to the age of research again just with

我们之前讨论过 value function 那件事，也许一旦人们把 value function 搞好了，他们就能更高效地利用资源。

22:06

big computers. >> That's a very interesting way to put it.

大型计算机。

22:10

But let me ask you the question you just posed then.

这个说法挺有意思的。

22:14

What are we scaling and what what is what would it mean to have a recipe?

但让我问你刚才提出的问题：我们到底在扩展什么？

22:19

Because I guess I'm not aware of a very clean relationship that almost looks like a law of physics which existed in pre-training that was a power law between data or computer parameters and loss.

如果有一个“配方”的话，那意味着什么？

22:33

What is the kind of relationship

因为据我所知，预训练中那种几乎像物理定律一样清晰的幂律关系——数据量或计算量与参数和损失之间的幂律——并不存在。

22:35

we should be seeking and how how should we think about what this new recipe might look like?

我们应该追求什么？

22:42

So, we've we've already witnessed a transition from one type of scaling to a different type of scaling, from pre-training to RL.

又该如何思考这个新配方可能长什么样？

22:51

Now, people are scaling RL.

我们已经见证了一种 scaling 到另一种 scaling 的转变，从 pre-training 到 RL。

22:53

Now, based on what people say on Twitter, they spend more compute on RL than on pre-training at this point because RL

现在，人们在 scaling RL。

23:02

can actually consume quite a bit of compute.

很大的计算资源。

23:05

You know, you do very very long rollouts. >> Yes.

>> 这个说法很有意思。

23:08

So it takes a lot of compute to produce those rollouts and then you get relatively small amount of learning per roll out.

但让我问你刚才提出的问题。

23:15

So you really can spend you really can spend a lot of compute and I could imagine like I wouldn't at this at this st it's it's more like I wouldn't even call it a scale um scaling I would say hey like what are you doing and is the thing you are doing the the the the

我们在扩展什么？

23:31

most productive thing you could be doing?

你现在能做的最有生产力的事是什么？

23:33

Yeah. >> Can you find a most more productive way of using your compute?

对。

23:38

We've discussed the value function business earlier and maybe once people get good at value functions they will be using their their um resources more productively and if you find a whole other way of training models you could say is this scaling or is it just using your resources I think it becomes a little

>> 你能不能找到一种更高效的方式使用你的算力？

23:59

bit ambiguous in a sense that when people were in the age of research back then it was like people say hey let's try this and this and this let's try that and that and that oh look something interesting is happening and I think there will be a return to that. >> So if we're back in the era of research, stepping back, what is the part of the recipe that we need to think most about?

这句话有点模糊，因为以前在研究时代，大家会说“嘿，试试这个，再试试那个，哦，看，有点意思”，我觉得我们会回到那种状态。

24:17

When you say value function, people are already trying the current recipe, but then having LLM as a judge and so forth, you could say that's a value function,

说到value function，大家已经在尝试现有的方法了，比如用LLM当裁判之类的，那其实也可以算是一种value function。

24:25

but it sounds like you have something much more fundamental in mind.

>> 那如果我们回到研究时代，退一步讲，配方里最需要思考的部分是什么？

24:30

Do we need do we need to go back to should we even rethink pre-training at all and not just add more steps to the end of that process? >> Yeah.

你说到价值函数，现在人们已经在尝试当前的配方了，比如用LLM当裁判之类的，这也可以算是一种价值函数，

24:39

So the the the the discussion about value function I think it was interesting.

所以关于value function的讨论，我觉得挺有意思的。

24:45

I want to like emphasize that I think the value function is something like it's going to make RL more efficient

我想强调一下，我认为value function会让RL变得更高效。

24:52

and I think that makes a difference but I think that anything you can do with a value function you can do without just more slowly. >> Mhm. >> The thing which I think is the most fundamental is that these models somehow just generalize dramatically worse than people. >> Yes. >> And it's super obvious.

我认为这确实有影响，但我觉得任何你能用价值函数做的事情，不用它也能做，只是更慢而已。

25:13

That's that seems like a very fundamental thing. >> Okay, so this is the crux of

嗯哼。

25:18

generalization and there's two sub questions.

但听起来你心里想的是更根本的东西。

25:21

There's one which is about sample efficiency which is why should it take so much more data for these models to learn than humans.

我们是不是需要重新思考预训练，而不仅仅是在那个过程末尾加更多步骤？

25:28

There's a second about even separate from the amount of data it takes, there's a question of why is it so hard to teach the thing we want to a model than to a human which is to say for to a human that we don't necessarily need a verifiable reward to

另外一点是，先抛开数据量的问题不谈，为什么我们想教给模型的东西，教给人类反而更容易？也就是说，对人类来说，我们不一定需要一个可验证的reward。

25:43

be able to you're probably mentoring a bunch of researchers right now and you're you know talking with them you're showing them your code and you're showing them how you think and from that they're picking up your way of thinking and how they should do research.

>> 对。

25:55

You don't have to set like a verifiable reward for them that's like okay this is the next part of the curriculum and now this is the next part of your curriculum and oh it was this training was unstable and we gota there's not this shleppy bespoke process.

所以关于价值函数的讨论，我觉得挺有意思的。

26:08

So perhaps these two issues are

我想强调，我认为价值函数会让RL更高效地泛化，

26:09

actually related in some way but I'd be curious to explore this this second thing which feels more like continual learning and this first thing which feels just like um sample efficiency. >> Yeah.

这里面有两个子问题。

26:23

So you know you could actually wonder one one possible explanation for the human sample efficiency that needs to be considered is evolution and evolution has given us a small

一个是关于样本效率，为什么这些模型学东西需要比人类多这么多的数据。

26:36

amount of the mo the most useful information possible and for things like vision hearing and locomotion I think there's a pretty strong case that evolution actually has given us a lot.

尽可能多的最有用的信息，对于视觉、听觉和运动这类事情，我认为有相当强的证据表明进化实际上已经给了我们很多。

26:50

Mhm. >> So for example, human dexterity far exceeds I mean robots can become dexterous too if you subject them to like a huge amount of training in

嗯。

27:02

simulation.

才能教他们。

27:02

But to train a robot in the real world to quickly like pick up a new skill like a person does seems very out of reach.

你现在大概在指导一群研究员，跟他们聊天，给他们看你的代码，展示你的思考方式，他们就从中学到你的思路和怎么做研究。

27:10

And here you could say, oh yeah, like locomotion all our ancestors needed great locomotion squirrels like so locomotion maybe like we've got like some unbelievable prior.

你不需要给他们设一个可验证的奖励，比如“这是课程的下一个部分，现在这是再下一个部分，哦，这次训练不稳定了，我们得……”——没有那种繁琐的定制过程。

27:21

You could make the same case for vision.

所以也许这两个问题

27:24

You know, I I believe Yan Lakhan made the point, oh,

你知道，我记得Yan Lakhan提过这一点，哦，

27:27

like um children learn to drive after 16 hour after like 10 hours of practice, which is true, but our vision is so good.

实际上在某种程度上是相关的，但我很好奇想探索第二个问题，它感觉更像持续学习，而第一个问题感觉就是样本效率。

27:34

At least for me, when I remember myself being 5 years old, my I was I was very excited about cars back then, and I'm pretty sure my car recognition was more than adequate for self-driving already.

至少对我来说，回想我五岁的时候，那时候我对车特别着迷，而且我敢肯定，我那时候识别车的能力已经足够用来做自动驾驶了。

27:45

As a 5-year-old, you don't get to see that much data as a 5-year-old.

作为一个五岁小孩，你根本看不到那么多数据。

27:49

You spend most of your time in your parents house, so you have very low data

你大部分时间都待在父母家里，所以数据量其实非常少。

27:53

diversity.

>> 对。

27:54

But you could say maybe that's evolution, too.

其实你可以想想，人类样本效率的一个可能解释是进化，进化给了我们一个很小的

27:57

But then language and math and coding, probably not. >> It still seems better than models.

但语言、数学和编程，可能不是这样。>> 这看起来还是比模型强。

28:02

I mean, obviously models are better than the average human at language and math and coding, but are they better at the average human at learning? >> Oh, yeah.

我的意思是，模型显然在语言、数学和编程上比普通人强，但它们在“学习能力”上比普通人强吗？>> 哦，对。

28:12

Oh, yeah.

哦，对。

28:12

Absolutely.

绝对。

28:13

What I meant to say is that language math and coding and especially math and coding suggests that whatever it is that makes

我想说的是，语言、数学和编程——尤其是数学和编程——暗示着，不管是什么让……

28:21

people good at learning is probably not so much a complicated prior but something more some fundamental thing. >> Wait, I'm not sure understood.

擅长学习的人可能并非依赖某种复杂的先验，而是更根本的东西。

28:32

Why should that be the case?

>> 等等，我不太确定我理解了。

28:35

So consider a skill that people exhibit some kind of great reliability or you know um >> if the skill is one that was very useful to our ancestors for many millions of

为什么会是这样？

28:48

years, hundreds of millions of years, you could say you could argue that maybe humans are good at it because of evolution because we have a prior >> an evolutionary prior that's encoded in some very nonobvious way. >> Yeah. >> That somehow makes us so good at it. >> Yeah. >> But if people exhibit great ability,

几亿年，几亿年的时间，你可以说，也许人类擅长这个是因为进化，因为我们有一个先验——一个以某种非常不明显的方式编码的进化先验。

29:14

reliability, robustness, ability to learn in a domain that really did not exist until recently, then this is more an indication that people might have [snorts] just better machine learning period. >> Mhm.

实际上可以消耗相当多的计算资源。

29:30

But then how should we think about what that is?

你知道，你会进行非常非常长的 rollout。

29:33

Is it a matter of Yeah.

>> 是的。

29:35

What is the ML analogy for what?

所以产生这些 rollout 需要大量计算，而每次 rollout 获得的学习量相对较小。

29:38

There's a couple interesting things

所以你真的可以花费大量计算资源，我可以想象，在现阶段，我甚至不会称之为扩展 scaling，我会说，嘿，你在做什么？

29:41

about it.

模型依赖预训练数据。

29:41

It takes fewer samples.

每当模型犯错，是不是因为某些东西碰巧在预训练数据里支持不够？

29:43

It's more unsupervised.

预训练“支持”可能是个比较模糊的说法。

29:44

You don't have to set a ver like a child learning to drive a car.

我不知道还能不能在这个话题上补充更有用的东西，但我觉得预训练没有人类的类比。

29:48

Children are not learning to drive a car.

倒是有一个类比是大型计算机。

29:51

A teenager learning how to drive a car is like not exactly getting some pre-built verifiable reward. it comes from their interaction with the machine and the with the environment.

一个青少年学开车，并不是靠某种预设好的、可验证的奖励机制。它来自于他们与机器和环境的互动。

30:03

Um and yet it takes much fewer samples.

嗯，而且样本量要少得多。

30:05

It

第二点，我也觉得，你比其他人更推崇持续学习，而且我其实认为这很重要也很正确，原因如下——我给你举个例子。

30:06

seems more unsupervised.

可靠性、鲁棒性、在直到最近才真正存在的领域里学习的能力，这更多地表明人们可能只是有了更好的机器学习，仅此而已。

30:07

It seems more robust.

>> 嗯。

30:09

Much more robust.

但我们应该怎么理解这件事呢？

30:10

The robustness of people is really staggering. >> Yeah.

这是不是……机器学习的类比是什么？

30:13

So like Okay.

有几个有趣的点。

30:14

And do you have a unified way of thinking about why are all these things happening at once?

那你有没有一个统一的思路，来解释为什么这些事情会同时发生？

30:20

What is the ML analogy that would that could be could realize something like this?

有什么ML的类比，能让人理解像这样的事情是怎么发生的？

30:25

So, so, so, um, this is where, you know, one of the things that you've been asking about is how can, you know, the

所以，呃，这个嘛，你之前一直在问的一个问题是，怎么才能让……

30:33

teenage driver kind of self-correct and learn from their experience without an external teacher. >> And the answer is well, they have their value function, >> right?

青少年司机那种自我修正和从经验中学习的方式，不需要外部教练。

30:45

They have a general sense which is also by the way extremely robust in people like whatever it is the human value function whatever the human value function is with a few exceptions around addiction

答案就是，他们有自己的价值函数，对吧？

30:59

it's actually very very robust and so for something like a teenager that's learning to drive they start to drive and they already have a sense of how they're driving immediately how badly they're unconfident and then they See?

它实际上非常非常稳健。

31:11

Okay.

比如说一个青少年学开车，他们刚开始开的时候，马上就能感觉到自己开得怎么样，有多不自信，然后他们就会看到？

31:11

And they and then of course the the learning speed of any teenager is so fast after 10 hours you're good to go. >> Yeah.

好吧。

31:17

It seems like humans has some solution, but I'm curious about like well how are they doing it and like why is it so hard to like how do we need to

然后当然，任何青少年的学习速度都很快，10个小时后就能开得很好了。

31:25

reconceptualize the way we're training models to make something like this possible? >> You know that is a great question to ask and it's a question I have a lot of opinions about.

重新构想我们训练模型的方式，让类似的事情成为可能？

31:39

But unfortunately we live in a world where not not all machine learning ideas are discussed freely and this is this is one of them.

>> 你知道吗，这是个很好的问题，也是我有很多看法的问题。

31:48

So there's probably a way to do it.

但不幸的是，我们生活在一个并非所有机器学习想法都能自由讨论的世界里，而这就是其中之一。

31:51

I think it can be done.

我觉得这事儿能成。

31:53

The fact that people are like that I think it's a proof that it can be done.

有人觉得做不到，但恰恰是这种想法本身，反而证明了它是可行的。

31:59

There may be another blocker though which is there is a possibility that the human neurons actually do more compute than we think.

不过可能还有另一个障碍——人类神经元实际做的计算量，可能比我们想象的要大。

32:08

And if that is true and if that plays an important role then things might be more difficult.

如果真是这样，而且这个因素又很关键的话，那事情就会变得更棘手。

32:15

But regardless I do think it points to the

但不管怎样，我确实认为这指向了某个机器学习原理，我对这个原理有些看法。

32:19

existence of some machine learning principle that I have opinions on.

存在一些我有看法的机器学习原理。

32:23

But unfortunately, circumstances make it hard to to discuss in detail.

但不幸的是，环境所限很难详细讨论。

32:28

Even though >> nobody nobody listens to this podcast, Ilia. >> Yeah. >> So, I have to say that prepping for Ilia was pretty tough because neither I nor anybody else had any idea what he's working on and what SSI is trying to do.

尽管>>根本没人听这个播客，Ilia。

32:43

I had no basis to come up with my

>>是啊。

32:45

questions and the only thing I could go off honestly was trying to think from first principles about what are the bottlenecks to hi because clearly Ilia is working on them in some way.

存在某种机器学习原理，我有自己的看法。

32:55

Part of this question involved thinking about RL scaling because everybody's asking how well RL will generalize and how we can make it generalize better.

但不幸的是，环境使得很难详细讨论。

33:02

As part of this I was reading this paper that came out recently on RL scaling and it showed that actually the learning curve on RL looks like a sigmoid.

即使 >> 没人听这个播客，Ilia。

33:10

I found this very

>> 是啊。

33:11

curious.

好奇。

33:12

Why should it be a sigmoid where it learns very little for a long time and then it quickly learns a lot and then it asmmptotes.

为什么它应该是一个sigmoid，在很长一段时间里学得很少，然后突然学得很多，最后趋于平稳。

33:18

This is very different from the power law you see in pre-training where the model learns a bunch at the very beginning and then less and less over time.

这和你在pre-training中看到的power law非常不同——pre-training里模型一开始学得很多，然后随着时间推移学得越来越少。

33:26

And it actually reminded me of a note that I had written down after I had a conversation with a researcher friend where he pointed out that the number of samples that you need to take in order to find a correct answer scales exponentially with how

这其实让我想起之前和一个研究员朋友聊天后记下的一条笔记，他指出，要找到正确答案所需的样本数量，会随着你当前概率分布与目标概率分布之间的差异程度呈指数级增长。

33:38

different your current probability distribution is from the target probability distribution.

存在某种机器学习原理，我对此有些看法。

33:43

And I was thinking about how these two ideas are related.

但不幸的是，环境使得很难详细讨论。

33:46

I had this vague idea that they should be connected, but I really didn't know how.

尽管 >> 没人听这个播客，Ilia。

33:50

I don't have a math background, so I couldn't really formalize it.

>> 是啊。

33:53

But I wondered if Gemini 3 could help me out here.

>> 所以，我得说为Ilia做准备挺难的，因为我和其他任何人都不知道他在做什么，也不知道SSI想干什么。

33:56

And so I took a picture of my notebook and I took the paper and I put them both in the context of Gemini 3 and I asked it to find the connection.

我完全没有基础来想出我的……

34:03

And it thought a bunch and then it [music] realized that the

它思考了一会儿，然后[音乐]意识到……

34:06

correct way to model the information you gain from a single yes or no outcome in RL is as the entropy of a random binary variable.

>> 所以，scaling时代的一个后果是，scaling吸走了房间里所有的空气。

34:13

It made a graph which showed how the bits you gain for a sample in RL versus supervised learning scale as a pass rate increases.

它画了一张图，展示了在RL和supervised learning中，每个样本获得的bits如何随着pass rate的提高而变化。

34:21

And as soon as I saw the graph that Gemini 3 made, immediately a ton of things started making sense to me.

我一看到Gemini 3画的那张图，立刻就有好多事情变得说得通了。

34:27

Then I wanted to see if there was any empirical basis to this theory.

然后我想看看这个理论有没有什么实证基础。

34:31

So I asked Gemini to code my

所以我让Gemini把我的代码写出来。

34:33

experiment to show whether [music] the improvement in loss scales in this way with pass rate.

实验来展示[音乐] loss的改进是否随通过率以这种方式缩放。

34:38

I just took the code that Gemini outputed.

我直接拿了Gemini输出的代码，复制粘贴到Google Colab笔记本里，就能运行这个玩具ML实验并可视化结果，一个bug都没有。

34:40

I copy pasted it into a Google Collab notebook and I was able to run this toy ML experiment and visualize its results without a single bug.

有意思的是，结果看起来和我们预期的相似但不完全一样。

34:46

It's interesting because the results look similar but not identical to what we should have expected.

然后我下载了这张图，放进Gemini里，问它这是怎么回事。

34:51

And so I downloaded this chart and I put it into Gemini and I asked it what is going on here.

我提出了一个假设，我觉得其实是对的，那就是我们在开始时通过固定learning rate限制了supervised learning能改进的程度，实际上我们应该随时间降低learning rate。

34:56

I came up with a hypothesis that I think is actually correct which

这让我们直观理解了为什么在实践中会有learning rate scheduler来随时间降低learning rate。

34:59

is that we're capping how much supervised learning can improve in the beginning by having a fixed learning rate and in fact we should decrease the learning rate over time.

>> 对。

35:09

It actually gives us an intuitive understanding for why in practice we have learning rate schedulers that decrease the learning rate over time.

这其实让我们直观地理解了，为什么在实践中我们会用学习率调度器，让学习率随时间递减。

35:17

I did this entire flow from coming up with this vague initial question to building a theoretical understanding to running some toy ML experiments all with Gemini 3.

我从最初那个模糊的问题开始，到建立理论理解，再到跑一些玩具级别的机器学习实验，整个过程都是用 Gemini 3 完成的。

35:26

This

这个

35:26

feels like the first model where it can actually come up with new connections that I wouldn't have anticipated.

感觉这是第一个能真正建立我预料之外的新联系的模型。

35:32

It's actually now become the default place I go to when I want to brainstorm new ways to think about a problem.

现在它已经成了我默认的首选工具，每当我想用新思路去思考一个问题时，就会去找它。

35:39

If you want to read more about RL scaling, you can check out the blog post that I wrote with a little help from Gemini 3.

如果你想了解更多关于 RL scaling 的内容，可以看看我写的那篇博客，Gemini 3 也帮了点忙。

35:45

And if you want to check out Gemini 3 yourself, go [music] to gemini.google.

如果你想亲自试试 Gemini 3，就去 gemini.google 看看吧。

35:49

I am curious if you say we are back in an era of research.

我很好奇，你说我们是不是又回到了一个 research 的时代？

35:53

You were there from 2012 to 2020 and do do you have Yeah.

你从2012年到2020年都在那里，对吧？

35:57

What what is now the vibe going to be if we go back to the era of research?

那如果我们回到那个研究时代，现在的氛围会是什么样？

36:03

For example, even after Alexet, the amount of compute that was used to run experiments kept increasing and the size of frontier systems kept increasing.

比如，即使在AlexNet之后，用来运行实验的compute量一直在增加，前沿系统的规模也在扩大。

36:14

And do you think now that this era of

你觉得现在这个时代

36:17

research will still require tremendous amounts of compute?

你从 2012 年待到 2020 年，那你觉得……是啊。

36:21

Um, do you think it will require going back into the archives and reading old papers?

如果我们要回到那个 research 的时代，现在的氛围会是什么样的？

36:27

What is maybe what was the vibe of like you were at Google and um OpenAI and Stanford these places when there was like a more of a vibe of research.

比如说，即使在 AlexNet 之后，用来跑实验的 compute 量一直在增加，前沿系统的规模也一直在扩大。

36:38

What what kind of thing should we be expecting in the community?

那你觉得现在这个 research 的时代，是不是仍然需要大量的 compute？

36:42

>> So one consequence of um the age of scaling is that there was this um scaling sucked out all the air in the room. >> Yeah.

>> 正因为scaling吸走了房间里所有的空气，每个人都开始做同样的事情。

36:52

And so because scaling sucked out all the air in the room, everyone started to do the same thing.

我们到了一个地步，世界上公司的数量远远超过了想法的数量。

37:00

We got to the point where uh we are in a world where there are more companies than ideas by quite a

我们到了这样一个阶段：在这个世界里，公司的数量已经远远超过了想法的数量。

37:08

bit. >> Actually on that you know there is this Silicon Valley saying that says that ideas are cheap, execution is everything and people say that a lot. >> Yeah.

>> 所以 scaling 时代的一个后果就是，scaling 把房间里所有的空气都吸走了。

37:21

And there is truth to that.

这话确实有道理。

37:23

But then I saw I saw someone say on Twitter um something like if ideas are are so cheap, how come no one's having any ideas? >> And I think it's true too.

但后来我在推特上看到有人说，如果想法这么廉价，那怎么没人想出什么新点子呢？——我觉得这话也没错。

37:36

I think like

我觉得吧

37:37

if you think about um research progress in terms of bottlenecks, there are several bottlenecks.

>> 是啊。

37:43

If you go back to the if if you and um one of them is ideas and one of them is your ability to bring them to life. >> Yeah. which might be compute but also engineering.

如果你回头看，其中一个因素是想法，另一个是你把它们实现出来的能力。

37:52

So if you go back to the '9s let's say you had people who had had pretty good ideas and if they had much larger computers maybe they could demonstrate that their ideas were viable but they

回到90年代，假设有人有过一些相当不错的想法，如果他们拥有更大的计算机，也许就能证明这些想法是可行的，但他们没有。

38:04

could not.

>> 正因为 scaling 吸走了所有的空气，每个人都开始做同样的事。

38:04

So they could only have very very small demonstration that did not convince anyone. >> Yeah. >> So the bottleneck was compute.

我们到了一个地步，就是公司比想法多得多。

38:14

Then in the age of scaling computers increased a lot and of course there is a question of how much comput is needed but compute is large so compute is large enough such that

到了算力规模化的时代，计算机性能大幅提升，当然这里有个问题：到底需要多少算力？但算力已经足够大，大到像AlexNet只用了两块GPU就建成了。

38:28

it's like not obvious that you need that much more compute to prove some idea like I'll give you an analogy.

>> 实际上，说到这个，硅谷有句老话，说想法不值钱，执行才是一切，而且很多人经常这么说。

38:37

Alexet was built on two GPUs.

那就是它使用的全部算力。

38:40

That was the total amount of compute used for it.

Transformer则用了8到64块GPU。

38:44

The transformer was built on 8 to 64 GPUs.

没有哪篇Transformer论文的实验用了超过这个数量。

38:47

No single transformer paper experiment used more

所以当你再看真正留给研究的部分时，差距就小得多了。

38:52

than 64 GPUs of 2017 which would be like what two GPUs of today.

比2017年的64块GPU，大概相当于今天的两块GPU。

38:58

So the ResNet right many like even even the the um you could argue that the like 01 reasoning was not the most comput heavy thing in the world.

所以ResNet对吧，很多，甚至可以说，那个o1推理其实也不是世界上最计算密集的东西。

39:10

So there definitely for for research you need like definitely some amount of

所以做研究肯定需要一定量的算力，但远非显而易见你需要有史以来最大的算力来做研究。

39:18

compute but it's far from obvious that you need the absolutely largest amount of compute ever for research.

>> 所以scaling时代的一个后果是，scaling吸走了房间里所有的空气。

39:25

H >> you might argue and I think it is true that if you want to build the absolutely best system, if you want to build the absolutely best system, then it helps to have much more compute and especially if everyone is within the same paradigm, then compute becomes one of the big differentiators.

>> 对。

39:44

>> Yeah, I guess while it was possible to develop these ideas, I'm asking you for the history because you were actually there.

>> 对，我想说的是，虽然这些想法是可以发展的，我问你历史是因为你当时确实在场。

39:51

I'm not sure what actually happened, but it sounds like it was possible to develop these ideas using minimal amounts of compute, but it wasn't the transformer didn't immediately become famous.

我不确定实际发生了什么，但听起来这些想法是用很少的算力就能发展出来的，但Transformer并没有立刻出名。

40:00

It became the thing everybody started doing and then started experimenting on top of and building on top of because it was validated at higher and higher levels of compute.

它后来成了每个人都开始做的东西，然后开始在上面实验和构建，因为它在越来越高的算力水平上得到了验证。

40:09

>> Correct.

>> 没错。

40:10

And if you at SSI have 50 different ideas, how will you know which one is the next transformer and which one is you know brittle without having the kinds of compute that other frontier labs have.

而如果你在SSI有50个不同的想法，你怎么知道哪个是下一个transformer，哪个是脆弱的，而你又没有其他前沿实验室那种级别的compute？

40:27

So I can I can comment on that which is the short comment is that you know you

关于这个我可以简短说两句，就是……

40:33

mentioned SSI specifically for us the amount of compute that SSI has for research is really not that small and I want to explain why like a simple math can explain why the amount of compute that we have is actually a lot more comparable for research than one might

>> 没错。

40:55

think.

>> 没错。

40:56

And I'll explain.

如果你在SSI有50个不同的想法，你怎么知道哪个是下一个transformer，哪个是脆弱的，而没有其他前沿实验室拥有的那种compute？

40:57

So SSI has raised $3 billion which is like not small by it's like a lot by any absolute sense but you could say but look at the other companies raising >> much more but a lot of what their a lot of their compute goes for inference like these big numbers these big loans

我可以对此评论一下，简短的回答是，你知道

41:18

it's earmarked for inference.

另一件事是，如果你在做一些不同的事情，你真的需要绝对最大的规模来证明它吗？

41:20

That's number one.

我完全不这么认为。

41:21

Number two, you need if you want to have a product on which you do inference, you need to have a big staff of engineers of salespeople.

我认为在我们的情况下，我们有足够的算力来说服我们自己和其他人我们做的是对的。

41:31

A lot of the research needs to be dedicated for producing all kinds of product related features.

有公开估计说，像OpenAI这样的公司每年在算力上花费大约56亿美元。

41:37

So then when you look at what's actually left for research, the difference becomes a lot smaller.

第二点，我也觉得，你比其他人更提倡continual learning，而且我实际上认为这是重要且正确的方向，原因如下——我给你举个例子。

41:44

Now the other thing is is that if you are doing something different do you really need the absolute maximal scale to prove it?

你当前的概率分布与目标概率分布之间的差异。

41:53

I don't think it's true at all.

我就在想这两个想法是怎么关联的。

41:55

I think that in our case we have sufficient compute to prove to convince ourselves and anyone else that what we're doing is correct.

我隐约觉得它们应该有联系，但我真的不知道具体怎么联系。

42:04

There's been public estimates that you know companies like OpenAI spend on the order of56

我没有数学背景，所以没法把它形式化。

42:10

billion dollars a year even just so far on experiments. >> This is separate from the amount of money they're spending on inference and so forth.

每年光是实验就要花掉几十亿美元。

42:18

So seems like they're spending more a year running exper like research experiments than you guys have in total funding. >> I think it's a question of what you do with it.

>> 这还不包括他们在推理等方面的投入。

42:27

It's a question of what you do with it. like they have a like the more I think in in in their case in the case of others I think there's a lot more

看起来他们一年花在研究实验上的钱，比你们的总融资额还多。

42:35

demand on the training compute there's a lot more different work streams there is there are different modalities there is just more stuff and so it becomes fragmented >> how will SSI make money >> you know my answer to this question is something like we just f right now we just focus on the research and then the answer to that will reveal itself.

训练算力的需求非常大，现在有更多不同的工作流，也有不同的模态，东西越来越多，所以变得很分散。

43:00

I think there will

>> SSI 要怎么赚钱？

43:01

be lots of possible answers. >> Is SSI's plan still to straightshot super intelligence? >> Maybe.

现在另一件事是，如果你在做一些不同的事情，你真的需要绝对最大规模的scale来证明它吗？

43:09

I think that there is merit to it. >> I think there's a lot of merit because I think that it's very nice to not be affected by the day-to-day market competition.

我完全不这么认为。

43:21

But I think there are two reasons that may cause us to change the plan. one is

我觉得在我们的情况下，我们有足够的compute来证明——说服我们自己以及其他人——我们做的是对的。

43:28

pragmatic if timelines turn out to be long which they might and second I think there is a lot of value in the best and most powerful AI being out there impacting the world >> I think this is a meaningfully valuable thing >> but then so why is your default plan to straight shot super intelligence because it sounds like you know openai anthropic

务实一点的话，如果时间线拉得很长，那也有可能；第二，我觉得最强大、最好的AI能真正影响世界，这本身就有巨大价值 >> 我认为这是一件非常有意义的事 >> 但那你为什么默认计划是直接冲刺超级智能呢？

43:54

all these other companies their explicit thinking is look we have weaker and weaker intelligences that the public can get used to and prepare for and why is it potentially better to build a super intelligence directly >> so I'll make the case for and against >> the case for is that you are so one of the challenges that people face when they're in the market is that they have to participate

所有这些其他公司，他们的明确想法是：看，我们正在推出越来越弱的智能，让公众慢慢习惯和准备。

44:21

in the rat race and the rat race is quite difficult in that it exposes you to to to difficult trade-offs which you need to make and there is it is it is nice to say we'll insulate ourselves from all this and just focus on the research and come out only when we are ready and not before but the counterpoint is valid too and those those are opposing forces the

另一件事是，如果你在做一些不同的事情，你真的需要绝对最大的规模来证明它吗？

44:47

counterpoint is hey it is useful for the world to see powerful AI it is useful for the world to see powerful AI because that's the only way you and communicate it. >> Well, I guess not even just that you can communicate the idea, but >> communicate the AI, not the idea.

你当前的概率分布与目标概率分布有多大差异。

45:01

Communicate the AI. >> What do you mean communicate the AI? >> So, okay.

我在想这两个想法是怎么关联的。

45:05

So, let's suppose you read an essay about AI >> and the essay says AI is going to be this and AI is going to be that and it's

我有个模糊的概念，觉得它们应该有联系，但我真不知道具体怎么联系。

45:12

going to be this >> and you read it and you say, okay, this is an interesting essay >> right >> now.

会是这样的 >> 你读到它，然后你说，嗯，这是一篇有意思的文章 >> 对 >> 现在，假设你看到AI在做这个，AI在做那个。

45:19

Suppose you see an AI doing this and AI doing that. >> It is incomparable.

>> 这没法比。

45:24

Like basically I think I think that there is a big benefit from AI being in the public and that would be a reason for us to not be quite straight shot. >> Yeah.

基本上我觉得，我认为AI在公众视野中有很大的好处，这也是我们不应该走得太激进的一个理由。

45:36

Well, I guess it's not even that

>> 是啊，我想也不完全是这个原因。

45:39

which I but I do think that is an important part of it.

但我确实认为这是其中重要的一部分。

45:42

The other big thing is I can't think of another discipline in human engineering and research where the end artifact was made safer mostly through just thinking about how to make it safe as opposed to why are airplane crashes per mile so much lower today than they were decades ago?

另一个大问题是，我想不出人类工程和研究领域还有哪个学科，最终产物是通过思考如何让它更安全而变得更安全的，而不是像飞机每英里事故率为什么比几十年前低那么多？

46:00

Why is it so much harder to find a bug in Linux than it would have been decades ago?

为什么现在在Linux里找bug比几十年前难那么多？

46:05

And

我认为主要是因为这些系统被部署到了世界上。

46:06

I think it's mostly because these systems were deployed to the world. you noticed failures, those failures were corrected and the systems became more robust.

存在某种机器学习原理，我对此有看法。

46:15

Now, I'm not sure why AGI and superhuman intelligence would be any different, especially given, and I hope we can talk, we're going to get to this.

但不幸的是，环境所限，很难详细讨论。

46:23

It seems like the harms of super intelligence are not just about like having some malevolent uh paper clipper out there, but it just like this is a really powerful thing and we don't even

尽管这个播客没人听，伊利亚。

46:34

know how to conceptualize how people interact with it, what people will do with it and having gradual access to it seems like a um better way to maybe spread out the impact of it and to help people prepare for it.

知道如何设想人们与它的互动方式、人们会用它做什么，并且逐步开放访问权限，这似乎是一种更好的方式，可以分散它的影响，并帮助人们做好准备。

46:49

Well, I think I think on this point even in the straight shot scenario, you would still do a gradual release of it is how I would imagine it.

嗯，我觉得在这个问题上，即使是直线发展的情景，我也会选择逐步发布，这是我的设想。

47:00

The the gra gradualism would be an inherent inherent component of any plan.

我觉得主要是因为这些系统被部署到了现实世界中。

47:05

It's just a question of what is the first thing that you get out of the door.

你发现了问题，那些问题被修正了，系统就变得更稳健了。

47:10

That's number one.

我不太明白为什么AGI和超人类智能就会不一样，尤其是——我希望我们能聊到这一点——看起来超级智能的危害不仅仅是有个恶意的回形针制造机在外面，而是这东西实在太强大了，我们甚至都不清楚。

47:11

Number two, I also think you know I believe you have advocated for continual learning more than other people >> and I actually think that this is an important and correct thing and here is why so one of the things so I'll give you

整个问题的核心在于算力。

47:27

another example of how thinking how language affects thinking and in this case it will be two words two words that have shaped everyone's thinking I maintain F first word AGI second word pre-training let me explain.

另一个例子是语言如何影响思维，这次是两个词——两个塑造了所有人思维方式的词。

47:46

So the word the term AGI, why does this term exist?

我认为第一个词是AGI，第二个是pre-training。

47:51

It's a very

我来解释一下。

47:52

particular term.

特定术语。

47:54

Why does it exist?

为什么存在？

47:55

There's a reason.

是有原因的。

47:57

The reason that the term AGI exists is in my opinion not so much because it's like a very important essential descriptor of of of some end state of intelligence but because it is a reaction to a different term that existed and the term is narrow

AGI这个术语存在的原因，在我看来，与其说是对某种智能终局状态的重要本质描述，不如说是对另一个已有术语的反应——那个术语就是“狭义AI”。

48:20

AI.

渐进主义应该是任何计划中固有的组成部分。

48:20

If you go back to ancient history of gameplay AI, of checkers AI, chess AI, computer games AI, everyone would say, look at this narrow intelligence.

问题只是你第一步先拿出什么东西来。

48:30

Sure, the chess AI can beat Casper off, but it can't do anything else.

这是第一点。

48:35

It is so narrow, artificial narrow intelligence.

第二点，我也觉得，我知道你比其他人更提倡continual learning >> 而且我其实认为这是很重要且正确的一点，原因如下：我给你举另一个例子。

48:38

So in response, as a reaction to this, some people said, well, this is not good.

所以作为回应，有些人就说，嗯，这不太好。

48:43

It

我觉得目前这个……

48:44

is so narrow.

所有这些其他公司，他们的明确想法是：看，我们有越来越弱的智能，让公众可以逐渐适应和准备。

48:45

What we need is general AI. general AI, an AI that can just do all the things.

那么，为什么直接构建一个超级智能可能更好呢？

48:52

The second and and that term just got a lot of traction. >> Yeah. >> The second thing that got a lot of traction is pre-training.

>> 所以我来陈述支持和反对的理由。

49:03

Specifically, the recipe of pre-training.

>> 支持的理由是，你面临的一个挑战是，当人们在市场中时，他们必须参与……

49:06

I think the current the

然后当你思考，好吧，假设我们成功了，造出了一个安全的超级智能，某种安全的超级智能。

49:09

way people do RL now is maybe um un is undoing the conceptual imprint of pre-training.

现在人们做RL的方式，某种程度上是在抹去预训练留下的概念印记。

49:16

But pre-training had the property. you do more pre-training and the model gets better at everything more or less uniformly.

但预训练有一个特性：你做得越多，模型在几乎所有任务上就会越均匀地变好。

49:26

Yeah, >> general AI pre-training gives AGI but the thing that happened with AGI and

对，>> 通用AI预训练带来了AGI，但AGI和预训练之间发生的事，某种意义上是因为他们过度冲击了目标——因为如果你仔细想想AGI这个词，尤其是在预训练的语境下，你会发现人类并不是AGI。

49:34

pre-training is that in some sense they overshock the target because by the kind if you think about the term AGI you will realize and especially in the context of pre-training you will realize that a human being is not an AGI because a human being Yes, there is definitely a foundation of skills.

另一个关于语言如何影响思维的例子，这次是两个词，两个塑造了所有人思维的词，我坚持这么认为。

49:57

A human being, a human being lacks a huge amount of

第一个词是AGI，第二个词是pre-training。

50:02

knowledge.

想法比公司少，但我认为有更值得构建的东西，而且我相信每个人都会想要它。

50:02

Instead, we rely on continual learning.

那就是一个稳健对齐、特别关心有感知生命的AI。

50:04

We rely on continual learning.

我认为尤其可以论证的是，构建一个关心有感知生命的AI反而更容易。

50:06

And so then when you think about okay, so let's suppose that we achieve success and we produce a safe super some kind of safe super intelligence.

问题在于，你怎么定义它？

50:14

The question is but how do you define it?

它会在持续学习曲线上的哪个位置？

50:16

Where on the curve of continual learning is it going to be?

我造出了一个超级聪明的15岁小孩，特别 eager 想要行动，然后你说，好吧，我要……他们其实什么都不知道。

50:20

I produce like um a super intelligent 15 year old that's very eager to go and you say okay I'm going to they don't know very much at all the

整个问题就在于 power。

50:28

great student very eager you go and be a programmer you go and be a doctor go and learn so you could imagine that the deployment itself will involve some kind of a learning trial and error period >> it's a process as opposed to you drop the finished thing >> okay I I I I see so you're you're suggesting that the thing you're pointing out with

好学生，非常渴望学习，你去做程序员，你去做医生，去学东西。

50:51

super intelligence is not some finished mind which knows how to do every single job in the economy cuz the way say the original I think openi charter or whatever defines AGI is like it can do every single job that a every single thing a human can do.

超级智能并不是某种已经完成、知道如何做经济中每一项工作的固定思维，因为按照最初OpenAI宪章或类似文件对AGI的定义，它指的是能做人类能做的每一件事。

51:08

You're proposing instead a mind which can learn to do any single every single job. >> Yes. >> And that is super intelligence.

而你提出的则是一种能够学会做每一项工作的思维。

51:17

And then

>> 是的。

51:17

but once you have the learning algorithm, it gets deployed into the world the same way a human laborer might join an organization. >> And it seems like one of these two things might happen.

一旦你有了这个学习算法，它就会被部署到世界中，就像人类劳动者加入一个组织一样。

51:33

Maybe neither of these happens.

>> 看起来可能会发生两种情况之一，也可能两者都不发生。

51:35

One, this super efficient learning algorithm becomes superhuman becomes as good as you and potentially even better at the

第一，这个超高效的学习算法变得超人类，变得和你一样好，甚至可能在机器学习研究任务上比你更强。

51:45

task of ML research.

你有很多这样的模型，那么就会有一股强大的力量推动它们被部署到经济中。

51:46

And as a result the algorithm itself becomes more and more superhuman.

除非有某种监管阻止它，顺便说一句，这也有可能。

51:50

The other is even if that doesn't happen if you have a single model I mean this this is explicitly your vision.

但我认为，在一段时间内出现非常快速的经济增长，从广泛部署来看是非常可能的。

51:55

If you have a single model or instances of a model which are deployed through the economy doing different jobs learning how to do those jobs continually learning on the job picking up all the skills that any human could pick up but actually picking them all up at the same time and then

另一个问题是它会有多快。

52:10

amalgamating the learnings. you basically have a model which functionally becomes super intelligent even without any sort of recursive self-improvement in software right because you now have one model that can do every single job in the economy and humans can't merge our minds in the same way and so do you expect some sort of like intelligence explosion from broad deployment >> I think that it is likely that we will

整合这些学习成果后，你基本上会得到一个模型，它在功能上变得超级智能，甚至不需要任何软件层面的递归自我改进，对吧？

52:37

have rapid economic growth I think the broad deployment And like there are two arguments you could make which are conflicting.

实现快速经济增长，我认为广泛部署是关键。

52:49

One is that look if indeed you get once indeed you get to a point where you have an AI that can learn to do things quickly

这里有两个相互矛盾的观点可以讨论。

53:02

and you have many of them then they will then there will be a strong force to deploy them in the economy.

新的转录工具，我们以后所有集都可以用。

53:09

Unless there will be some kind of a regulation that stops it, which by the way there might be.

这只是Labelbox如何在理念层面满足客户需求，并在整个旅程中与他们合作的一个例子。

53:15

But I think the idea of very rapid economic growth for some time, I think it's very possible from broad deployment.

如果你想了解更多，或者想亲自试用这个转录工具，请访问labelbox.com/barcash。

53:23

The other question is how rapid it's going to be.

在我看来，这是一个非常不稳定的处境，

53:27

So I think this is hard to know because on the one hand you have this very efficient worker. on the other hand there is the world is just really big and there's a lot of stuff and that stuff moves at a different speed but then on the other hand now the AI could you know so I think very rapid economic growth is possible and we will see like all kinds of things like different countries with different rules and the ones which have the friendlier rules the economic growth will be faster

所以我觉得这很难预测，因为一方面你有这种极其高效的劳动力，另一方面世界真的很大，有太多东西，这些东西的运行速度各不相同，但再换个角度，AI现在又能做到很多事。

53:54

hard to predict >> some people in our audience like to read the transcripts instead of listening to the episode and so we put a ton of effort into making the transcripts read like they are standalone essays.

而且如果你有很多这样的模型，那么就会有一股强大的力量推动它们部署到经济中。

54:05

The problem is that if you just transcribe a conversation verbatim using a speech to text model, it'll be full of all kinds of fits and starts and confusing phrasing.

除非有某种监管阻止它——顺便说一句，这也有可能。

54:14

We mentioned this problem to Labelbox and they asked if they could take a stab.

但我认为，在一段时间内出现非常快速的经济增长，这种可能性是存在的，源于广泛部署。

54:18

Working with them on this is probably the reason that I'm most

另一个问题是它会有多快。

54:22

excited to recommend Labelbox to people.

很难预测。

54:24

It wasn't just, oh, hey, tell us what kind of data you need and we'll go get it.

我们有些听众喜欢阅读文字稿而不是听节目，所以我们花了很多精力让文字稿读起来像独立的文章。

54:28

They walked us through the entire process from helping us identify what kind of data we needed in the first place to assembling a team of expert aligners to generate it.

问题是，如果你直接用语音转文字模型逐字转录对话，里面会充满各种停顿、重复和混乱的措辞。

54:37

Even after we got all the data back, Labelbox stayed involved.

我们向Labelbox提了这个问题，他们问能不能试试看。

54:40

They helped us choose the right base model and set up auto QA on the model's output so that we could tweak and refine it.

和他们合作这件事，可能是我最愿意向别人推荐Labelbox的原因。

54:46

And now we have a

他们不只是说“嘿，告诉我们你需要什么数据，我们去弄来”，而是从帮我们确定最初需要什么数据开始，到组建一队专家标注员来生成数据，全程指导我们。

54:47

new transcriber tool that we can use for all our episodes moving forward.

AGI及其未来的力量源于这样一个事实：很难想象未来的AI会有所不同。

54:52

This is just one example of how Labelbox meets their customers at the ideas level and partners with them through their entire journey.

它会很强大。

55:02

If you want to learn more or if you want to try out the transcriber tool yourself, go to labelbox.com/barcash.

确实，整个问题——AI和AGI的问题是什么？

55:09

It seems to me that this is a very precarious situation to be in where

整个问题就是力量。

55:14

looking the limit we know that this should be possible because if you have something that is as good as a human at learning but which can merge its brains merge there are different instances in a way that humans can't merge already.

在我看来，这是一个非常不稳定的局面。

55:28

This seems like a thing that should physically be possible.

从极限来看，我们知道这应该是可能的，因为如果你有一个像人类一样擅长学习、但又能以人类无法做到的方式合并不同实例的智能体，这从物理上似乎是可行的。

55:32

Humans are possible, digital computers are possible.

人类是可行的，数字计算机也是可行的。

55:35

You just need both of those combined to produce this thing.

你只需要把这两者结合起来，就能产生这种东西。

55:38

And it also seems like this kind of thing is extremely um powerful

而且这种东西似乎极其强大，经济增长只是其中一种说法。

55:43

and economic growth is one way to put it.

经济增长是一种说法。

55:45

Um I mean Dyson spear is a lot of economic growth but another way to put it is just like you will have potentially a very short period of time because a human on the job can you know you you're hiring people at SSI in six months they're like net productive probably right um a human like learns really fast and so this thing is becoming smarter and smarter very fast what is how do you think about making that go well and why is SSI positioned to do that well or what is SSI's plan there basically is what I'm trying to

嗯，我的意思是，Dyson spear 带来了很多经济增长，但另一种说法是，你可能会经历一个非常短的时期，因为人类在职场上，你知道，你在 SSI 招人，六个月后他们基本上就能净产出，对吧，嗯，人类学东西非常快，所以这个东西也在飞速变聪明。

56:08

ask >> yeah so one of the one of the ways in which my thinking has been changing is that I now place more importance on AI being deployed incrementally and in advance.

嗯，所以我其中一个想法上的变化是，我现在更看重AI能够逐步地、提前地部署。

56:33

One very

AI一个非常棘手的地方在于，我们讨论的是还不存在的系统，很难去想象它们。

56:34

difficult thing about AI is that we are talking about systems that don't yet exist and it's hard to imagine them.

很难预测。

56:45

I think that one of the things that's happening is that in practice it's very hard to feel the AGI.

>> 我们的一些听众喜欢阅读文字稿而不是听节目，所以我们投入了大量精力让文字稿读起来像独立的文章。

56:54

It's very hard to feel the AGI.

问题是，如果你直接用语音转文字模型逐字转录对话，里面会充满各种停顿、重复和混乱的措辞。

56:56

We can talk about it, but it's like it's

我们向Labelbox提过这个问题，他们问是否能尝试解决。

57:00

like talking about like the long f like imagine like having a conversation about like how is it like to be old when you're like old and and frail and you can have a conversation.

AI 的难点在于，我们在谈论还不存在的系统，很难想象它们。

57:14

You can try to imagine it, but it's just hard and you come back to reality.

我觉得，实际情况是，很难真正感受到 AGI。

57:20

Well, that's not the case.

很难真正感受到 AGI。

57:22

And I think that a lot of the issues around

我们可以讨论它，但这就像——

57:26

AGI and its future power stem from the fact that it's very difficult to imagine future AI is going to be diff different.

政府和我认为我们会看到的一些事情——你已经开始看到一些激烈竞争的公司开始合作，在AI安全方面合作。

57:35

It's going to be powerful.

你可能已经看到OpenAI和Anthropic迈出了第一步，但之前并不存在这种情况。

57:37

Indeed, the whole problem, what is the problem of AI and AGI?

这实际上是我大约三年前在一次演讲中预测过的。

57:42

The whole problem is the power.

但大脑皮层里的脑区和神经元，它们大多只跟邻近的细胞交流。

57:44

The whole problem is the power.

整个问题就是算力。

57:47

When the power is really big, what's

当算力真的很大的时候，那又会怎样？

57:50

going to happen?

就像在谈论，比如，想象一下，当你老了、虚弱的时候，讨论变老是什么感觉。

57:51

And one of the one of the ways in which I've changed my mind over the past year and so that that change of mind may back may I'll say I I'll I'll hedge a little bit may back propagate into into the plans of our of our company is that so if it's hard to imagine what do you do you got to be showing the

你可以试着想象，但就是很难，然后你又回到现实。

58:16

thing you got to be showing the thing and I maintain that I think I think most people who work Con AI also can't imagine it because it's too different from what people see on a day-to-day basis.

……新的转录工具，我们以后所有的节目都能用。

58:32

I do maintain here is something which I predict will happen.

这只是Labelbox如何在理念层面与客户对接，并在整个旅程中与他们合作的例子之一。

58:37

That's a prediction.

如果你想了解更多，或者想亲自试试这个转录工具，请访问labelbox.com/barcash。

58:39

I maintain

在我看来，这是一个非常不稳定的局面……

58:40

that as AI becomes more powerful then people will change their behaviors and we will see all kinds of unprecedented things which are not happening right now and I'll give some examples.

过去一年多里，我改变想法的一个方面——这个改变可能会，我稍微保留一下，可能会反向影响到我们公司的计划——那就是，如果很难想象，那该怎么办？

58:54

I do like I I think I think for better or worse the the frontier companies will play a very important role in what happens as will the

你得展示这个东西，你得展示这个东西。

59:04

government and the kind of things that I think we'll see which you see the beginnings of companies that are fierce competitors starting collaborate to to collaborate on AI safety you may have seen open AI and anthropic event doing a first small step but that did not exist That's actually something which I predicted in one of my talks about three years ago

想法比公司少，但我认为有某种东西更值得构建，而且我认为每个人最终都会想要它。

59:29

that such a thing will happen.

这种事会发生。

59:31

I also maintain that as AI continues to become more powerful, more visibly powerful, there will also be a desire from governments and the public to do something and I think that this is a very important force of showing the AI.

我也认为，随着AI变得越来越强大、越来越明显地强大，政府和公众也会产生采取行动的意愿，而我认为这是展示AI的一个非常重要的力量。

59:48

That's number one.

这是第一点。

59:49

Number two, okay, so then the AI is being built. what needs to what needs to

第二点，好，那么AI正在被构建。

59:55

be done.

所以呢，我一直坚持的一个观点是，现在做AI的人觉得AI不够强大，是因为它还会犯错。

59:56

So one thing that I maintain that will happen is that right now people who are working on AI I maintain that the AI doesn't feel powerful because of its mistakes.

但我认为，总有一天AI会开始让人觉得它真的很强大，而一旦到了那个时刻，所有AI公司在安全方面的态度都会发生巨大转变——他们会变得非常谨慎。

60:08

I do think that at some point the AI will start to feel powerful actually and I think when that happens we will see a big change in the way all AI companies approach safety.

我这么说，算是作为一个预测吧，我们走着看对不对。

60:22

they'll become much more paranoid.

他们会变得更加多疑。

60:24

I think I I say this as a predict as a as a as a prediction that we will see happen.

我觉得我这么说，算是作为一个预测吧，我们会看到这种情况发生。

60:29

We'll see if I'm right, but I think this is something that will happen because they will see the AI becoming more powerful.

等着看我是不是对的，但我认为这一定会发生，因为他们会看到AI变得越来越强大。

60:36

Everything that's happening right now, I maintain is because people look at today's AI and it's hard to imagine the future AI.

现在发生的一切，我坚持认为，是因为人们看着今天的AI，很难想象未来的AI会是什么样。

60:43

And there is a third thing which needs to happen.

还有第三件事需要发生。

60:46

And I think this is this this

我觉得这其实……

60:48

and I'm talking about it in in broader terms not just from the perspective of SSI because you ask me about our company but the question is okay so then what should what should the companies aspire to build >> what should they aspire to build and there has been one big idea that actually every that um everyone has been locked in locked into which is the the self-improving AI and why why did it happen because there

>> 所以，就算你让AI去关心有感知的生命，而且说实话，如果你真的解决了对齐问题，我也不确定这到底是不是你应该追求的目标。

61:14

is fewer ideas than companies But I maintain that there is something that's better to build and I think that everyone will actually want that.

我认为这主要是因为这些系统被部署到了全世界。

61:24

It's like the AI that's robustly aligned to care about sentient life specifically.

你注意到了失败，这些失败被修正了，系统变得更稳健。

61:30

I think in particular it will be there's a case to be made that it will be easier to build an AI that cares about sentient

现在，我不确定AGI和超人类智能会有什么不同，尤其是考虑到——我希望我们能聊到，我们马上会谈到这一点。

61:39

life than an AI that cares about human life alone because the AI itself will be sentient.

>> 确实如此。

61:45

And if you think about things like mirror neurons and human empathy for animals which is you know you might argue it's not big enough but it exists.

我觉得它可能不是最好的标准。

61:55

I think it's an emerging property from the fact that we model others with the same circuit that we used to model ourselves because that's the most efficient thing to do.

我想说两点。

62:06

>> So even if you got an AI to care about sentient beings and it's not actually clear to me that that's what you should try to do if you solved alignment.

……这比公司的想法要少一些。

62:16

It would still be the case that most sentient beings will be AIS.

但我坚持认为，有某种更好的东西值得去构建，而且我觉得每个人最终都会想要它。

62:20

There will be trillions eventually quadrillions of AIs.

就像那种稳健对齐、特别关心有感知生命的AI。

62:24

Humans will be a very small fraction of sentient beings.

我认为尤其可以论证的是，构建一个关心有感知生命的AI会更容易。

62:27

So, it's not clear to me if the goal is some kind of human control over

所以我不太清楚，目标是不是某种人类对它的控制。

62:32

this future civilization that this is the best criterion. >> It's true.

这个未来文明，这是最好的标准。

62:38

I I think that it's possible it's not the best criterion.

>> 确实。

62:43

I'll say two things.

我，我觉得可能这不是最好的标准。

62:44

I think that thing number one I think that if there so I think that care for sentient life I think there is merit to it.

我想说两点。

62:54

I think it should be considered.

第一，我认为如果有的话，我觉得对有情生命的关怀是有价值的。

62:57

I think that it

我认为应该被考虑。

62:59

will be helpful if there was some kind of a short list of ideas that then the companies when they are in this situation could use.

如果能有一个简短的思路清单，公司在遇到这种情况时可以参考，那会很有帮助。

63:09

That's number two.

这是第二点。

63:11

Number three, I think it would be really materially helpful if the power of the most powerful super intelligence was somehow capped because it would address a lot of these concerns.

第三点，我认为如果最强大的超级智能的能力能在某种程度上被限制，那会非常实质性地有用，因为这能解决很多担忧。

63:26

The question of how to do it, I'm not sure, but I think that would be materially helpful when you're talking about really really powerful systems. >> Yeah.

>> 嗯。

63:34

Um, before we continue the element discussion, I I want to double click on that.

呃，在我们继续讨论元素之前，我想深入探讨一下这个问题。

63:39

How much room is there at the top?

顶部还有多少空间？

63:41

How do you think about super intelligence?

你怎么看待超级智能？

63:43

Do you think I mean using this learning efficiency idea maybe is just extremely fast at learning new skills or new knowledge and does it just

我的意思是，用这种学习效率的概念，它可能只是学习新技能或新知识的速度极快，还是它只是拥有更大的策略池？

63:51

have a bigger pool of strategies?

>> 所以这是一个不同人有不同直觉的领域。

63:53

Is there a single cohesive it in the center that's more powerful or bigger?

是否存在一个单一的、凝聚的核心，比别的更强大或更大？

63:58

And if so, do you do you imagine that this will be sort of godlike in comparison to the rest of human civilization? or does it just feel like another agent or another cluster of agents? >> So this is an area where different people have different intuitions. >> I think it will be very powerful for

如果是的话，你觉得它会像神一样，与人类文明的其他部分相比高高在上吗？还是它只是像一个智能体，或者一群智能体的集合？

64:17

sure.

>> 我认为它肯定会非常强大。

64:17

I think that what I think is most likely to happen is that there will be multiple such AIS being created roughly at the same time.

我认为最可能发生的情况是，会有多个这样的AI大致同时被创造出来。

64:28

I think that if the cluster is big enough, like if the cluster is literally continent sized, that thing could be really powerful indeed, right?

我认为如果集群足够大，比如如果集群真的有大陆那么大，那东西确实会非常强大，对吧？

64:41

If you literally

如果你真的有一个大陆大小的集群，那些AI可以非常强大。

64:42

have a continentsized cluster, like those those AIs can be very powerful.

>> 所以，就算你让一个AI去关心有感知生命，我也不太清楚如果你解决了对齐问题，这是否就是你应该尝试去做的事。

64:47

And I like all I can tell you is that if you're talking about extremely powerful AIs, like truly dramatically powerful, then yeah, it would be nice if they could be restrained in some ways or if there was some kind of an agreement or something because I think that if you are saying

即便如此，大多数有感知生命仍然会是AI。

65:08

hey like if if you really like what what is the the concern of super intelligence?

嘿，如果你真的，呃，超级智能的担忧到底是什么？

65:13

What is one way to explain the concern?

怎么解释这种担忧？

65:15

If you imagine a system that is sufficiently powerful, like really sufficiently powerful, and you could say, okay, you need to do something sensible like care for sentient life, let's say, in a very single-minded way, we might not like the results.

想象一个足够强大的系统，真的非常强大，然后你说，好吧，你需要做点合理的事，比如关心有感知的生命，用一种非常单一的方式，结果可能我们并不喜欢。

65:30

That's really what it is.

其实就是这么回事。

65:31

And so maybe, by the way, the answer is that

所以，顺便说一句，答案可能是，你不应该构建一个单一的，你不应该构建一个通常意义上的RL agent。

65:34

you do not build a single you do not build an RL agent in the usual sense.

你构建的不是一个通常意义上的RL agent。

65:39

And actually, I'll point I'll point several things out.

实际上，我要指出几点。

65:42

I think human beings are a semi agent.

我认为人类是一种半agent。

65:45

You know, we pursue a reward and then the emotions or whatever make us tire out of the reward.

我们追求奖励，然后情绪之类的东西让我们对奖励感到厌倦，我们又去追求不同的奖励。

65:51

We pursue a different reward.

市场有点像一种非常短视的agent。

65:53

The market is like kind it's like a very shortsighted kind of agent.

进化也是如此。

65:57

Evolution is the same.

进化在某些方面非常智能。

65:58

Evolution is very intelligent in some

>> 所以在这个问题上，不同的人有不同的直觉。

66:01

ways but very dumb in other ways.

>> 确实如此。

66:03

The government has been designed to be a never- ending fight between three parts which has an effect.

我觉得这可能不是最佳标准。

66:09

So I think things like this another thing that makes this discussion difficult is that we are talking about systems that don't exist that we don't know how to build right that's the other thing and that's actually my belief I think what people are doing right now will go some distance and then peter out it will

我想说两点。

66:28

continue to improve but it will also not be it so the it we don't know how to build and I think that a lot h a lot hinges on understanding and in reliable generalization and I'll say another thing which is like you know one of the things that you could say is that cause alignment to be difficult is that human val that it's it's um

能力会持续提升，但也不会是终点。

66:54

your ability to learn human values is fragile then your ability to optimize them is fragile will you actually learn to optimize them and then can't you say are these not all instances of unreliable generalization why is it that human beings appear to generalize so much better.

你不会构建一个单一的，你不会构建一个通常意义上的RL智能体。

67:10

What if generalization was much better?

实际上，我会指出几点。

67:12

What would happen in this case?

我认为人类是一个半智能体。

67:14

What would be the effect?

你知道，我们追求奖励，然后情绪之类的东西让我们对奖励感到厌倦。

67:15

But those we can't we can't like those questions are right now still unanswerable.

我们追求另一个奖励。

67:20

>> Um how [clears throat] does one think about what AI going well looks like because I think you've scoped out how AI might evolve.

>> 嗯，[清嗓子] 你怎么看待AI发展顺利是什么样子？

67:29

We'll have these sort of continual learning agents.

因为你已经勾勒出了AI可能演变的路径。

67:32

AI will be very powerful.

我们会拥有这种持续学习的智能体。

67:34

Maybe there will be many different AIs.

AI会非常强大。

67:37

How do you think about lots of continent computes size intelligences going around?

也许会有很多不同的AI。

67:42

How dangerous is that?

你怎么看待大量自主计算智能体到处存在的情况？

67:44

How do we make that

这有多危险？

67:46

less dangerous?

不那么危险？

67:47

And how do we do that in a way that protects a equilibrium where there might be misaligned AIs out there and bad actors out there?

而且我们该如何在保护平衡的同时，应对可能存在的不对齐AI和恶意行为者？

67:59

So, one reason why I liked the AI that cares for sentient life, >> you know, and we can debate on whether it's good or bad, but

所以，我喜欢那个关心有感知生命的AI的一个原因是，你知道，我们可以争论这是好是坏，但如果这些戏剧性系统的前N个真的关心、热爱人类或类似的东西，关心有感知生命，显然这还需要实现。

68:10

if the first N of these dramatic systems actually do care for, you know, love humanity or something, you know, care for sentient life.

如果前N个这些戏剧性的系统真的在乎，比如，热爱人类或什么的，在乎有感知的生命。

68:20

Obviously, this also needs to be achieved.

显然，这也需要实现。

68:24

This needs to be achieved.

这必须实现。

68:26

So if this is achieved by the first n of those systems then then I can see it go well at least

所以如果前N个系统实现了这一点，那我觉得至少结果会不错。

68:33

for quite some time and then there is the question of what happens in the long run what happens in the long run how do you achieve a long run equilibrium >> and I think that there there is an answer as well and I don't like this answer but it needs to be considered In the long run, you might say, okay, so if you have a world where powerful AI

我不喜欢这个解决方案，但它确实是一个解决方案。

68:58

exist.

如果你学习人类价值观的能力很脆弱，那么你优化它们的能力也很脆弱。

68:58

In the short term, you could say, okay, you have universal high income.

你真的学会了优化它们，然后你不能说，这些不都是不可靠泛化的例子吗？

69:03

You have universal high income and we all doing well.

为什么人类看起来泛化得好得多？

69:06

But we know that what do the Buddhists say?

如果泛化好得多会怎样？

69:09

Change is the only constant.

在这种情况下会发生什么？

69:10

And so things change and there is some kind of government political structure thing and it changes because these things have a shelf life. you know some new new government thing comes up and it functions and then after some

会有什么影响？

69:25

time it stops functioning that's something that you see happening all the time and so I think that for the long run equilibrium one approach you could say okay so maybe every person will have an AI that will do their bidding and that's good and if that could be maintained indefinitely that's true but the downside with that is okay so then the AI goes and like

它停止运作的那一刻，这种事情你经常能看到。

69:50

earns earn earn you know earns money for for the person and you know advocates for their needs in like the political sphere and maybe then writes a little report saying okay here's what I've done here's the situation and the person says great keep it up but the person is no longer a participant and then you can say that's a precarious place to be in but so I'm going to preface by saying

赚钱赚钱赚钱，你知道，为那个人赚钱，然后在政治领域为他们的需求发声，可能还会写个小报告说，好了，这是我做的，这是目前的情况。

70:16

I don't like this solution but it is a solution And the solution is if people become part AI with some kind of neural link++ because what will happen as a result is that now the AI understands something and we understand it too like because now the understanding is transmitted wholesale.

多巴胺神经元连接到气味传感器。

70:35

So now if the AI is in some situation now it's like you are involved in the situation yourself

>> 嗯。

70:41

fully and I think this is the answer to the equilibrium.

我在想，情绪是在数百万年甚至数十亿年前、完全不同的环境中演化出来的，却仍然如此强烈地指导着我们的行为，这本身是不是一个对齐成功的例子？

70:46

I wonder if uh the fact that emotions which were developed millions or in many cases billions of years ago in a totally different environment are still guiding our actions so strongly is an example of

具体来说，脑干里有一个——我不知道该叫它价值函数还是奖励函数更准确——但脑干有一个指令，说要和更成功的人交配。

71:04

alignment success to maybe spell out what I mean the brain stem has these I don't know if it's more accurate to call it a value function or reward function but the brain stem has a directive where it's saying mate with somebody who's more successful.

关于对齐的成功，我来具体解释一下我的意思。

71:18

The cortex is the part that understands what does success mean in the modern context but the brain stem is able to align the cortex and say however you recognize success to be and I I'm not smart enough

脑干里有某种——我不知道该叫它价值函数还是奖励函数更准确——但脑干有一个指令，就是让你跟更成功的人交配。

71:30

to understand what that is.

>> 我觉得这里有一个更普遍的观点。

71:32

You're still going to pursue this directive. >> I think I think there is so I think there's a more general point.

实际上，大脑如何编码高级欲望这件事非常神秘。

71:39

I think it's actually really mysterious how the brain encodes high level desires.

抱歉，应该说，进化如何编码高级欲望。

71:45

Sorry, how evolution encodes high level desires. >> Like it's pretty easy to understand how evolution would would endow us with the desire for food that smells good cuz

>> 比如，很容易理解进化为什么会赋予我们对闻起来好吃的食物的欲望，因为气味是一种化学物质，所以只要追求那种化学物质就行。

71:56

smell is a chemical and so just pursue that chemical.

——对。

72:00

It's very easy to imagine such a me evolution doing such a thing.

>> 我认为它会非常强大，因为

72:05

But evolution also has has endowed us with all these social desires like we we really care about being seen positively by society.

进化在某些方面非常聪明。

72:14

We care about being in a good standing.

我们在意的是保持良好信誉。

72:17

We like all these social intuitions that we

我们喜欢所有这些社会直觉，而且

72:21

have.

>> 对。

72:21

I feel strongly that they are baked in and I don't know how evolution did it because it's a high level concept.

我强烈觉得它们是天生的，我不知道进化是怎么做到的，因为这属于高级概念。

72:29

It's represented in the brain. like what people think like let's say you are like you care about some social thing.

它在人脑中有对应的表征。比如人们会想，假设你在意某件社会性的事情。

72:38

It's not like a low-level signal like smell.

这不像嗅觉那种低级信号。

72:41

It's not something that for which there's a sensor like the brain needs to

这不是那种有专门传感器的东西，大脑需要自己去处理。

72:46

do a lot of processing to piece together lots of bits of information to understand what's going on socially and somehow evolution said that's what you should care about. >> Yes. >> How did it do it?

——它是怎么做到的？

72:58

And it did it quickly too. >> Yeah. because I think all these sophisticated social things that um we care about I think they evolved pretty recently.

而且做得很快。

73:07

So evolution had an easy time hardcoding this high level desire and

所以进化要硬编码这种高级欲望其实并不容易。

73:12

>> I maintain or you know at least I'll say I'm unaware of good hypothesis for how it's done.

我保持这个观点，或者说至少我还没看到能解释这个现象的好假设。

73:17

I I had some ideas I was kicking around but none of them none of them uh are satisfying. >> Yeah.

我之前有一些想法在琢磨，但没有一个能让我满意。

73:23

And what's especially impressive is if it was a desire that you learned in your lifetime, it kind of makes sense because your brain is intelligent.

嗯。

73:32

It makes sense why we be able to learn intelligent desires.

特别厉害的是，如果这是一种你一生中学会的欲望，那还挺合理的，因为你的大脑是智能的。

73:35

But your point is that the desire is maybe this is not

我们能学会智能的欲望，这说得通。

73:39

your point, but one way to understand it is the desire is built into the genome and the genome is not intelligent, right?

——是啊。

73:45

But it's able to you're somehow able to describe this feature that requires like it's not even clear how you define that feature and you can get it into the you can build it into the genes.

因为我觉得所有这些我们所在乎的复杂社会性东西，都是最近才进化出来的。

73:55

Yeah, essentially, or maybe I'll put it differently.

所以进化很容易就把这种高级欲望硬编码进去了。

73:58

If you think about the tools that are available to the genome, it says, okay, here's a recipe for building a brain.

如果你想想基因组可用的工具，它只是说，好，这里有一份构建大脑的配方。

74:04

And you could say, here is a recipe for connecting the

你可以说，这里有一个连接的方法。

74:07

dopamine neurons to like the smell sensor. >> Yeah. >> And if the smell is a certain kind of, you know, good smell, you want to eat that.

如果这些戏剧性系统中的前N个确实关心、热爱人类或关心有感知生命，显然这还需要实现。

74:15

I could imagine the genome doing that.

如果前N个系统实现了这一点，那么我认为事情会顺利发展。

74:17

I'm I'm claiming that it is harder to imagine.

我——我是在说，这更难想象。

74:20

It's harder to imagine the genome saying you should care about some complicated computation that your entire brain that like a big chunk of your brain does.

更难想象基因组会告诉你，你应该关心某个复杂的计算，而你的整个大脑——或者说大脑的一大块——都在做这个计算。

74:29

That's all I'm claiming.

这就是我全部的观点。

74:30

I I can tell you like a speculation.

我可以告诉你一个猜测。

74:32

I was wondering how it could be done.

我一直在想，这到底是怎么做到的。

74:34

And let

但大脑区域和皮层里的神经元，它们基本上只跟邻居交流。

74:35

me offer a speculation and I'll explain why the speculation is probably false.

你说得对，但一种理解方式是：这种欲望是写在基因组里的，而基因组本身并不智能，对吧？

74:40

So the speculation is okay.

但它却能描述出这样一个特征——甚至都不清楚你怎么定义这个特征——然后你还能把它塞进基因里。

74:42

So the brain it's like the brain has those regions.

对，基本上就是这样。

74:45

You know the brain regions.

或者换个说法，如果你想想基因组可用的工具，它说：好，这里有一份建造大脑的配方。

74:47

We have our cortex, right? >> Yeah. >> It has all those brain regions and the cortex is uniform.

然后你可以说：这里有一份把多巴胺神经元连接到气味传感器的配方。

74:53

But the brain regions and and and the neurons in the cortex, they kind of speak to their neighbors mostly.

但大脑区域和皮层里的神经元，它们主要跟邻居交流。

74:59

And that's explains why you get

所以这就解释了为什么你会得到——

75:02

brain regions because if you want to do some kind of speech processing, all the neurons that do speech need to talk to each other and they can and because neurons can only speak to their nearby neighbors for the most part, it has to be a region.

大脑区域，因为如果你要做某种语音处理，所有负责语音的神经元需要互相交流，而它们确实可以，但由于神经元大多只能和邻近的邻居通信，所以必须形成一个区域。

75:15

All the regions are mostly located in the same place from person to person.

这些区域在不同人之间基本都位于相同的位置。

75:19

So maybe evolution hardcoded literally a location on the brain.

所以也许进化硬编码了大脑上的一个具体位置。

75:22

So it says, "Oh, like when when like you know the GPS of the brain, GPS coordinates, such and such, when that

它就像在说，“哦，就像大脑的GPS，GPS坐标，某某位置，当那个区域激活时，那就是你应该关注的东西。

75:28

fires, that's what you should care about." Like maybe that's what evolution did cuz that would be within the toolkit of evolution.

所以，我其实完全同意这一点。

75:38

Yeah.

我认为这个理论还有一个更强的反驳点，那就是想想人类，有些人在童年时期被切除了半个大脑。

75:39

Although there are examples where for example people who are born blind have that area of their cortex adopted by another sense and I have no idea but I'd be surprised if the desires or the reward functions which require visual

虽然也有例子，比如天生失明的人，他们大脑皮层的那个区域会被另一种感官接管，我也不确定，但我怀疑那些需要视觉的欲望或奖励函数——

75:56

signal no longer worked.

对。

75:57

You know people who have their different areas of their cortex co-opted.

你知道，那些大脑皮层不同区域被重新利用的人。

76:02

For example, if you no longer have vision, can you still feel the sense that I want people around me to like me and so forth, which usually there's also visual cues for. >> So, I actually fully agree with that.

比如，如果你失去了视觉，你还能感受到“我希望身边的人喜欢我”之类的感觉吗？这些通常也有视觉线索。 >> 其实我完全同意这一点。

76:14

I I think there's an even stronger counter argument to this theory, >> which is like if you think about people, so there are people who get half of

我、我觉得这个理论还有一个更强的反驳点，>> 就是如果你想想人类，有些人会失去一半的——

76:22

their brain removed in childhood. >> Yeah. and they still have all their brain regions, but they all somehow move to just one hemisphere, which suggests that the brain regions the the location is not fixed.

但他们仍然拥有所有的大脑区域，只是这些区域都转移到了一个半球上，这表明大脑区域的位置并不是固定的。

76:34

And so that theory is not true.

所以那个理论不成立。

76:35

It would have been cool if it was true, but it's not.

如果它成立会很酷，但事实并非如此。

76:38

And so I think that's a mystery, but it's an interesting mystery.

所以我认为这是一个谜，但也是一个有趣的谜。

76:42

Like the fact is somehow >> evolution was able to endow us to care about social stuff very very reliably.

事实是，不知为何，进化非常可靠地赋予了我们关心社交事物的能力。

76:48

And even people who have like all kinds of strange mental conditions and deficiencies and emotional problems tend to care about this.

甚至那些有各种奇怪心理状况、缺陷和情绪问题的人，通常也会在意这件事。

76:56

Also, AI tools like defakes, voice clones, and agents have dramatically increased the sophistication [music] of fraud and abuse.

而且，像deepfakes、voice clones和agents这类AI工具，已经大幅提升了欺诈和滥用的复杂程度。

77:04

So, it's more important than ever to actually understand the identity and intent [music] of whoever or whatever is using your platform.

所以，现在比以往任何时候都更需要真正了解使用你平台的任何人的身份和意图。

77:12

That's exactly what

这正是Sardine能帮你做到的。

77:14

Sardine helps you do.

Sardine 能帮你做到。

77:15

Sardine brings together thousands of device behavior and identity signals to help you assess risk.

Sardine 整合了数千种设备行为和身份信号，帮助你评估风险。

77:21

Everything from how a user types or moves their mouse or holds their device to whether they're hiding their true location behind a [music] VPN to whether they're injecting a fake camera feed during KYC selfie checks.

从用户打字、移动鼠标或握持设备的方式，到他们是否通过VPN隐藏真实位置，再到他们在KYC自拍验证时是否注入虚假摄像头画面。

77:34

Sardine combines these signals with insights from their network of almost 4 billion

Sardine 将这些信号与来自近40亿台设备的网络洞察相结合，比如用户的欺诈历史或与其他高风险账户的关联，这样你就能在坏家伙造成损害之前发现他们。

77:39

devices. things like a user's history of fraud or their associations with other high-risisk accounts so you can spot bad actors before they do damage.

SSI 计划做些什么不同的事？

77:48

This would literally be impossible if you only use data from your own [music] application.

所以，你的计划大概是成为这个时代到来时的前沿公司之一，那么你创办 SSI 大概是因为你觉得，“我有一种方法可以安全地做到这一点，而其他公司没有。

77:54

Sardine doesn't stop at detection.

”这个区别是什么？

77:56

They offer a suite of agents to streamline onboarding checks and automate investigations. [music] So, as fraudsters use AI to scale their attacks, you can use AI to scale your

他们提供了一套agent来简化入职审核并自动化调查。[音乐] 所以，当欺诈者用AI来扩大攻击规模时，你也可以用AI来扩大你的——

78:06

defenses.

我的描述方式是，有一些我认为有前景的想法，我想去研究它们，看看它们是否真的可行。

78:07

Go to sardine.ai/warcash.

就这么简单。

78:08

AI/Swarcash to learn more and download their guide on AI fraud detection.

这是一次尝试。

78:14

What is SSI planning on doing differently?

我认为如果这些想法被证明是正确的，就是我们讨论过的那些关于理解的想法。

78:17

So presumably your plan is to be one of the frontier companies when this time arrives and then what is presumably you started SSI because you're like I I think I have a way of approaching how to do this safely in a

所以，你的计划大概是成为这个时代到来时的前沿公司之一，然后你创办SSI大概是因为你觉得——我觉得我有办法安全地做到这一点，在——

78:32

way that the other companies don't.

其他公司没有的方式。

78:34

What what is that difference?

那区别是什么？

78:36

So the way I would describe it as there are some ideas that I think are promising and I want to investigate them and see if they are indeed promising or not.

我描述的方式是，有一些想法我觉得有前景，我想去研究它们，看看它们是否真的可行。

78:46

It's really that simple.

就这么简单。

78:48

It's an attempt.

这是一种尝试。

78:49

I think that if the ideas turn out to be correct, these ideas that we discussed around understanding

我认为如果这些想法被证明是正确的，我们讨论的这些关于理解的想法……

78:56

generalization, >> if these ideas turn out to be correct, then I think we will have something worthy.

泛化能力。

79:03

Will it turn out to be correct?

如果这些想法被证明是正确的，那我觉得我们就会拥有一些有价值的东西。

79:05

We are doing research.

它会被证明正确吗？

79:06

We are squarely age of research company.

我们正在做研究。

79:09

We are making progress.

我们是一家彻头彻尾的研究型公司。

79:11

We've actually made quite good progress over the past year.

我们正在取得进展。

79:15

But we need to keep making more progress, >> more research. >> And that's how I see it.

过去一年里，我们实际上已经取得了相当不错的进展。

79:21

I see it as an

但我们需要继续取得更多进展，更多研究。

79:22

attempt to be an attempt to be a voice and a participant.

是的。

79:27

Um people have asked uh your co-founder and previous CEO left to go to Meta recently and people have asked well if there was a lot of breakthroughs being made that seems like a thing that should have been unlikely.

关于这个，我只会提醒几个可能被遗忘的事实。

79:44

I wonder how you respond.

我认为这些事实提供了背景，它们能解释这个情况。

79:46

>> Yeah.

当你拥有超人智能，并且你有一些关于如何让超人智能顺利发展的想法，但其他公司也会尝试他们自己的想法。

79:47

So I in for for this I will simply remind a few facts that may have been forgotten and I think this these facts which provide the context I think they explain the situation.

SSI让超级智能顺利发展的方法有什么独特之处？

80:01

So the context was that we were fundraising at a 32 billion valuation and then Meta um came in and offered to

SSI的主要独特之处在于它的技术路线。

80:10

to acquire us and I said no but my former co-founder like in some sense said yes and as a result he also was able to enjoy from a lot of near-term liquidity and he was the only person from SSI to join Meta.

收购我们时我说不，但我之前的联合创始人在某种意义上说了是，结果他也享受到了大量短期流动性，他是SSI唯一加入Meta的人。

80:25

It sounds like SSI's plan is to be a company that is at the frontier when you get to this very important period in human history

听起来SSI的计划是成为一家在人类历史这个极其重要的时期处于前沿的公司——当超级智能出现时，你们有关于如何让超级智能顺利发展的想法，但其他公司也会尝试他们自己的想法。

80:35

where you have superhuman intelligence and you have these ideas about how to make superhuman intelligence go well but other companies will be trying their own ideas.

我觉得大概是5到20年。

80:46

What distinguishes SSI's approach to making super intelligence go well?

是什么让SSI在让超级智能顺利发展方面与众不同？

80:51

The >> the main thing that distinguishes SSI is its technical approach.

SSI最核心的区别在于它的技术路线。

80:56

So we have a different technical approach that I think is worthy

我们采用了一套不同的技术路线，我认为这很有价值。

81:01

and we are pursuing it.

5到20年。

81:03

I maintain that in the end there will be a convergence of strategies.

我始终认为，最终各种策略会走向趋同。

81:07

So I think there will be a convergence of strategies where at some point as AI becomes more powerful it's going to become more or less clearer to everyone what the strategy should be.

所以我觉得策略会逐渐收敛——随着AI越来越强大，大家会越来越清楚该走哪条路。

81:19

And it should be something like, yeah, you need to find some way to talk to each other.

最终大概就是，你得想办法让系统之间能互相沟通。

81:25

And you want your

而且你希望你的……

81:27

first actual like real super intelligent AI to be aligned and somehow be, you know, care for sentient life, care for people, democratic, one of those, some combination of thereof.

嗯。

81:45

And I think this is the condition that everyone should strive for and

我认为这是所有人都应该追求的状态。

81:53

that's what SSI is striving for and I think that with time if not already all the other companies will realizing that they're striving towards the same thing and we'll see.

所以我想展开说说你可能会如何看待世界的发展。

82:02

I think that the world will truly change as AI becomes more powerful. >> Yeah. >> And I think a lot of these forecasts will like I think things will be really different and people will be acting really differently.

就像，我们还有几年时间，这些其他公司继续走当前的路，然后这条路会停滞不前，停滞不前在这里的意思是，它们最多只能赚到几千亿的低位。

82:14

What speaking of forecast what are your forecasts to this

说到预测，你对未来有什么判断？

82:17

system you're describing which can learn as well as a human and subsequently as a result become superhuman. >> I think like uh 5 to 20 >> 5 to 20 years. >> Mhm. >> So I just want to unroll your how you might see the world coming.

如果这些戏剧性系统中的前N个真的关心，你知道，热爱人类或什么，关心有情生命。

82:32

It's like we have a couple more years where these other companies are continuing the current approach and it stalls out and stalls out here meaning they earn no more than low hundreds of billions in

显然，这也需要实现。

82:45

revenue or how do you think about what stalling out means? >> Yeah, I think the re I think it could I think it could stall out and I think stalling out will look like it will all look very similar. >> Yeah. >> Among all the different companies something like this.

营收，或者说你怎么看“停滞”意味着什么？

83:01

I'm not sure because I think I think I think even with I think even I think even with stolen out I think these companies could make a stupendous stupendous revenue maybe not profits because they will be

>> 嗯，我觉得它可能会停滞，而且停滞的样子看起来都会很相似。

83:14

it will be they will need to work hard to differentiate each other from themselves but revenue definitely >> but there's something in your model implies that the when the correct solution does emerge there will be convergence between all the companies and I'm curious why you think that's the case >> well I was talking more about converg convergence on their larger strategies. >> I think eventual convergence on the technical approach is probably going to happen as well but I I was alluding to convergence to the larger strategies.

他们会需要拼命想办法让自己跟别人不一样，但收入肯定会…… >> 不过你的模型里隐含了一个意思：当正确的解决方案出现时，所有公司会走向趋同。

83:42

What what what exactly is the thing that should be done? >> I I just want to better understand how you see the future on rolling.

到底到底到底该做什么？

83:48

So currently we have these different companies and you expect their approach to continue generating revenue.

>> 我我我只是想更好地理解你怎么看待未来的发展。

83:53

Yes. >> But not get to this humanlike learner. >> Yes. >> So now we have these different forks of companies.

所以现在我们有这些不同的公司，你预计它们的方法会继续产生收入。

83:59

We have you we have thinking machines.

是的。

84:00

There's a bunch of other labs. >> Yes. and maybe one of them figures out the correct approach >> but then the release of their product makes it clear to other people how to do this thing.

>> 但不会达到这种类人学习者的水平。

84:10

>> I think it won't be clear how to do it thing but it will be clear that something different is possible >> right >> and that is information and I think people will will then be trying to figure out how how that's how that works.

>> 我认为具体怎么做可能还不清楚，但大家会意识到某种不同的做法是可能的。

84:25

I do think though that one of the things that's that I think you know not addressed here not discussed is that with each increase in the AI's

>> 对。

84:34

capabilities I think there will be some kind of changes but I don't know exactly which ones in how things are being done.

>> 我觉得不会让人明白具体怎么做，但会让人明白某种不同的东西是可能的。

84:42

So like I think it's going to be important yet I can't spell out what that is exactly. >> And how how are the by default you would expect the company that has the model company that has that model to be getting all these gains because they have the model that is learning how to do all has the skills

>> 对。

85:01

and knowledge that it's building up in the world.

>> 我觉得从经验来看，事情会这样发展。

85:04

What is the reason to think that the benefits of that would be widely distributed and not just end up at whatever model company gets this continuous learning loop going first? >> Like I think that empirically what happen so here here is what I think is going to happen.

第一，我们看看实际会发生什么——顺便说一句，我们讨论的是“好世界”对吧？

85:22

Number one, I think empirically when let's let's look at let's look at how

什么是好世界？

85:28

things have gone so far with um the AIs of the past.

>> 那么默认情况下，你怎么会预期拥有那个模型的公司，那个拥有那个模型的公司，会获得所有这些收益，因为他们拥有那个正在学习如何做所有事情、并在世界上积累技能和知识的模型。

85:30

So one company produced an advance and the other company scrambled and produced some competi some some similar things after some amount of time and they started to compete in the market and push their push the prices down >> and so I think from the market perspective I think something similar will happen there as well even if someone okay so okay so okay so okay so okay so okay so okay so okay so okay so

有什么理由认为这些好处会广泛分布，而不是最终落到那个率先启动持续学习循环的模型公司手里？

85:53

okay so okay we talking about the good world by the way where what's the good world.

竞争喜欢专业化，你在市场上能看到，在进化中也能看到。

85:59

What's the good world?

所以会有很多不同的利基市场，很多不同的公司占据不同的利基。

86:01

Where we have these powerful humanlike learners that are also like and by the way maybe there there's another thing we haven't discussed on the on the the spec of the super intelligent AI that I think is worth considering is that you make it narrow

在这种世界里，你可以说，一家AI公司在某个复杂经济领域确实比另一家强很多，而另一家公司则在另一个领域更擅长。

86:21

can be useful and narrow at the same time.

>> 我觉得从经验上看，事情是这样的——所以这里是我认为会发生的事情。

86:24

So you can have lots of narrow super intelligent AIs.

第一，我认为从经验上看，我们来看看过去AI的发展情况。

86:27

But suppose you have many of them and you have some and you have some company that's producing a lot of um profits from it and then you have another company that comes in and starts to compete and the way the competition is going to work is through specialization.

一家公司取得了进展，另一家公司手忙脚乱，过了一段时间也推出了类似的东西，然后它们开始在市场上竞争，把价格压下来。

86:44

I think what's going to happen is that

>> 所以从市场角度来看，我认为类似的情况也会发生，即使有人——好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧好吧

86:47

the way competition like competition loves specialization and you see it in the market, you see it in evolution as well.

但到了某个高点，别人就会说：我不想从头学你已经学会的东西。

86:54

So you're going to have lots of different niches and you're going to have lots of different companies who are occupying different niches in in this kind of world where you might say yeah like one AI company is really quite a bit better at some area of really complicated economic activity and a different company is better at another

>> 我猜这需要很多家公司同时从人类级持续学习智能体起步，这样它们才能在不同分支上展开研究。

87:14

area and the third company is really good at litigation and that's contradicted by what humanlike learning implies is that like it can learn >> it can but but you have accumulated learning you have a big investment.

领域，第三家公司非常擅长诉讼，而这与人类式学习所暗示的相矛盾——它能够学习 >> 它确实能学，但你已经积累了学习成果，有了巨大的投入。

87:26

You spent a lot of compute to become really really really good really phenomenal at this thing and someone else spent a huge amount of comput and a huge amount of experience to get really really good at some other thing >> right >> you apply a lot of human learning to get

你花了大量算力在某件事上变得极其极其出色、极其惊人，而另一个人花了海量算力和大量经验在另一件事上变得极其极其擅长 >> 对 >> 你投入了大量人类学习才达到那个水平，但现在你站在这个高点上，别人会说，你看，我不想从头学你学过的那些东西 >> 我想这需要很多家公司同时起步，拥有那种人类式持续学习智能体，这样它们才能在不同分支上开始各自的研究。

87:41

there but now like you you are at this high point where someone else would say look like I don't want to start learning what you've learned to go >> I guess that would require many different companies to begin at the human like continual learning agent at the same time so that they can start their different research in different branches.

>> 对，这是个合理的论点。

88:01

But if one company, you know, gets that agent first or gets

但我强烈直觉告诉我事情不会这样发展。

88:05

that learner first, it does then seem like well, you know, they could like if you just think about every single job in the economy, you just have uh instance learning each one seems tractable for a company. >> Yeah, that's that's that's a valid argument.

人类团队比AI团队更有多样性。

88:20

My my strong intuition is that it's not how it's going to go.

但我们怎么在AI中引出有意义的多样性？

88:24

My strong intuition is that yeah like the argument says it will go this way. >> Yeah.

我觉得单纯提高temperature只会输出胡言乱语。

88:29

>> But my strong intuition is that it will not go this way that this is the you know in in theory there is no difference between theory and practice.

>> 但我强烈的直觉是，事情不会这样发展。

88:37

In practice there is and I think that's going to be one of those >> a lot of people's models of recursive self-improvement literally explicitly state we will have a million Ilias in a server that are coming in with different ideas and this will lead to a super intelligence emerging very fast.

理论上，理论和实践没有区别，但在实践中是有区别的。

88:53

Do you have some intuition about how

我觉得这就会是那种情况 >> 很多人对递归自我改进的模型，明确地描述说我们会有一百万个Ilia在服务器里，各自带着不同的想法，然后很快就能催生出一个超级智能。

88:55

parallelizable the thing you are doing is?

但想法不同的人，那才是你想要的。

88:58

How how what are the gains from making copies of Ilia? >> I don't know.

>> 为什么你看不同的模型，哪怕是完全不同的公司发布的，训练用的数据集可能也不重叠，但LLM之间却惊人地相似？

89:03

I think I think there'll definitely be there'll be diminishing returns because you want you want people who think differently rather than the same.

>> 也许数据集并不像看起来那么不重叠。

89:13

I think that if they were literal copies of me, I'm not sure how much more incremental value you'd get.

但某种程度上，即使单个人类可能不如未来的AI高效，也许人类团队比AI团队拥有更多多样性。

89:21

I think that

但我们要怎么在AI之间引发有意义的多样性呢？

89:22

but people who think differently that's what you want. >> Why is it that it's been if you look at different models even released by totally different companies trained on potentially non-over overlapping data sets it's actually crazy how similar LLMs are to each other. >> Maybe the data sets are not as non-over overlapping as it seems.

因为它们在同样的数据上预训练。

89:39

But there's there's some sense there's like even if an individual human might be less productive than the future AI.

而RL和post-training才是开始出现差异的地方，因为不同的人会设计不同的RL训练。

89:46

Maybe there's something to the fact that human

>> 对。

89:48

teams have more diversity than teams of AIs might have.

存在一些我有看法的机器学习原理。

89:51

But how do we elicit meaningful diversity among AI?

但不幸的是，环境所限很难详细讨论。

89:54

So I think just raising the temperature just results in gibberish.

尽管 >> 没人听这个播客，Ilia。

89:58

I think you want something more like >> different scientists have different different prejudices or different ideas. [clears throat] How do you get that kind of diversity among AI agents?

>> 是啊。

90:08

So the reason there has been no diversity I believe is because of pre-training.

>> 所以，我得说为Ilia做准备相当困难，因为我和其他任何人都不知道他在做什么，也不知道SSI到底想干什么。

90:13

All the pre-trained models are the same

我完全没有依据来提出我的

90:15

pretty much because the pre-train on the same data.

基本上是因为它们在同样的数据上做预训练。

90:19

Now RL and postraining is where some differentiation starts to emerge because different people come up with different RL training. >> Yeah.

而RL和post-training才是开始出现差异的地方，因为不同的人会设计不同的RL训练方法。

90:28

And then I've heard you hint in the past about selfplay as a way to either get data or match agents to other agents of equivalent intelligence to

>> 对。

90:38

kick off learning.

开始学习。

90:40

How should we think about why there's no public um proposals of this kind of thing and working with LLM? >> I would say there are two things to say.

我们应该怎么理解为什么没有公开的这类提案，以及和LLM合作的情况？

90:50

I would say that the reason why I thought selfplayful is interesting is because it offered a way to create models using compute only without data, right?

>> 我想说两点。

91:01

And if you think that data is the

我觉得selfplay之所以有趣，是因为它提供了一种只用算力、不用数据来创建模型的方法，对吧？

91:04

ultimate bottleneck, then using compute only is very interesting.

最终的瓶颈，那么只用算力就非常有意思。

91:09

So that's what makes it interesting.

这就是它吸引人的地方。

91:12

Now the the thing is that selfplay at least the way it was done in the past when you have agents which are somehow compete with each other it's only good for developing a certain set of skills it is too narrow.

现在的问题是，selfplay——至少过去那种让智能体互相竞争的方式——只适合培养特定技能，太狭窄了。

91:27

It's only good for like negotiation

它只适用于像谈判、

91:30

uh conflict certain social skills strategizing that kind of stuff.

冲突、某些社交技能、策略规划这类东西。

91:36

And so if you care about those skills then selfplay will be useful.

所以如果你在意这些技能，selfplay会有用。

91:41

Now actually I think that selfplay did find a home but just in a different form in a different form.

实际上，我认为selfplay确实找到了归宿，只是换了一种形式。

91:49

So things like debate prove a verifier.

比如debate、prove a verifier。

91:53

You have some kind of an LLM as a judge which is also

你有一个LLM作为裁判，它也有

91:57

incentivized to find mistakes in your work.

动力去发现你工作中的错误。

92:00

You could say this is not exactly selfplay but this is you know a related adversarial setup that people are doing. believe >> and really selfplay is an example of um is a special case of more general like um competition between between agents, >> right?

你可以说这不完全是selfplay，但这是人们在做的一种相关的对抗性设置。

92:13

The response the natural response to competition is to try to be different.

>> 而且selfplay其实是更广义的智能体间竞争的一个特例，>> 对吧？

92:17

And so if you were to put multiple agents and you tell them, you know, you all need to work on some problem and you're an agent and you're

对竞争的自然反应是试图变得不同。

92:25

inspecting what everyone else is working, you're going to say, well, if they already taken this approach, it's not clear I should pursue it. they should pursue something differentiated and so I think that something like this could also create an incentive for um a diversity of approaches. >> Yeah.

在检查别人在做什么，你会说，如果他们已经在用这个方向，我不确定我该继续。

92:46

Um final question, what is research taste?

他们应该走差异化的路线。

92:49

You're obviously the person in the world who is

所以我认为类似的东西也能激励多样化的方法。

92:53

considered to have the best taste in doing research in AI. you were uh the co-author on many of the biggest the biggest things that have happened in the history of deep learning from Alex net to GBT3 to so on what is it that how do you characterize how you come up with these ideas >> I can answer so I can comment on this for myself >> I think different people do it

被认为在AI研究上品味最好的人。

93:18

differently >> but one thing that um guides me personally is an aesthetic of how AI should be >> by thinking about how people are but thinking correctly >> like it's very easy to think about how people are incorrectly but what does it mean to think about people correctly >> so I'll give you some examples

不同。

93:45

the idea of the artificial neuron is directly inspired by the brain and it's a great idea why because you say sure the brain has all these different organs has the faults but the faults probably don't matter M >> why do we think that the neurons matter?

人工神经元的概念直接受大脑启发，这是个好主意。

93:58

Because there's many of them.

为什么？

93:59

It kind of feels right.

因为你说，大脑有各种不同器官，有缺陷，但缺陷可能不重要。

94:01

So you want the neuron. >> Yeah. >> You want some kind of local learning rule that will change the connections.

为什么我们认为神经元重要？

94:06

You want some local learning rule rule that will change the connections between the neurons,

因为有很多神经元。

94:11

>> right?

对吧？

94:12

It feels plausible that the brain does it.

大脑这么做听起来挺合理的。

94:14

The idea of the distributed representation, the idea that the brain, you know, the brain responds to experience or neural network should learn from experience, not response.

分布式表征这个概念，就是说大脑——你知道的，大脑对经验做出反应，或者说神经网络应该从经验中学习，而不是对刺激做出反应。

94:24

The brain learns from experience. the neural network of experience and you kind of ask yourself is some is something fundamental or not fundamental how things should be >> and I think that's been guiding me a

大脑是从经验中学习的。

94:37

fair bit kind of thinking from multiple angles and looking for almost beauty beauty simplicity ugliness there's no room for ugliness it's just beauty simplicity elegance correct inspiration from the brain and all of those things need to be present at the same time and the more they are present the more confident you can be in a top- down belief.

>> 我觉得这很大程度上在引导我，从多个角度思考，寻找那种美、简洁、优雅——没有丑陋的空间，只有美、简洁、优雅、正确，以及来自大脑的启发。

94:58

And then the top down belief is the thing that sustains you when the

所有这些元素需要同时存在，它们越齐全，你就越能对一个top-down信念有信心。

95:02

experiments contradict you.

>> 但你不知道有bug啊。

95:03

Because if you just trust the data all the time, well, sometimes you can be doing a correct thing, but there's a bug. >> But you don't know that there is a bug.

你怎么判断有bug？

95:11

How can you tell that there is a bug? >> How do you know if you should keep debugging or you conclude it's the wrong direction?

你怎么知道有bug？>> 你怎么判断是该继续调试，还是该认定方向错了？

95:18

Well, it's the top down.

嗯，这是自上而下的思路。

95:19

Well, how should you can say the things have to be this way?

那你怎么能说事情非得这样呢？

95:22

Something like this has to work.

类似这样的东西必须能行得通。

95:24

Therefore, we got to keep going.

所以，我们得继续干下去。

95:25

That's the top down.

这就是自上而下的方式。

95:26

And it's based on this like multifaceted beauty

而且它是基于这种多层面的美感。

95:29

and inspiration by the brain. >> All right, we'll leave it there. >> Thank you so much. >> Thank you so much. >> All right.

>> 你怎么知道是该继续debug，还是该断定方向错了？

95:36

Appreciate it. >> That was great. >> Yeah, I enjoyed it. >> Yes, me too. >> Hey everybody, I hope you enjoyed that episode.

嗯，这就是top-down。

95:42

If you did, the most helpful thing you can do is just share it with other people who you think might enjoy it.

你怎么能说事情必须是这样？

95:48

It's also helpful if you leave a rating or a comment on whatever platform you're listening on.

比如“这个东西必须work，所以我们得继续干下去”。

95:53

If you're interested in sponsoring the podcast,

这就是top-down。

95:56

you can reach out at dwarcash.com/advertise.

>> 好，我们就聊到这儿吧。

96:00

Otherwise, I'll see you on the next one.

不然的话，咱们下期见。

We're Moving from the Age of Scaling to the Age of Research

Deep Analysis · 深度解析

Reality of AI Progress

AI进展的现实性

Resource Allocation Challenges

资源分配挑战

Compute Demand in Research Era

研究时代的算力需求

Mindset Shift and Backpropagation into Strategy

思维转变与策略反向传播

Theory Invalidation and Reality Check

理论失效与现实检验

Key Moments · 关键时刻

Play Queue