TWIML AI
Your Ranking Model Is Optimizing the Wrong Thing with Zoe Weil - #780
00:003164
00:00
Security teams don't need another list of alerts.
安全团队不需要又一份 alerts 列表。
00:04
They need to know which vulnerabilities are actually reachable, exposed, and safe to fix.
他们需要知道哪些 vulnerabilities 是真的 reachable、exposed，而且可以安全修复。
00:09
That's what hyperscale remediation from Blitzy does.
这就是 Blitzy 的 hyperscale remediation 能做的事。
00:13
Blitzy understands your entire software state, then finds known and previously undiscovered vulnerabilities in your codebase.
Blitzy 了解你整个 software state，然后在你 codebase 里找出已知的、以及以前没被发现过的 vulnerabilities。
00:20
It traces each one through the execution paths, checks the dependencies, and validates the fix.
它会沿着 execution paths 追踪每一个，检查 dependencies，并验证 fix。
00:26
Your engineers approve the plan, and Blitzy delivers batches of tested pull requests ready to review.
你的工程师批准方案后，Blitzy 会交付一批经过测试的 pull requests，随时可以 review。
00:33
Less alert noise, faster remediation, and confidence that fixes are validated as safe before you review them.
更少的 alert 噪音、更快的 remediation，而且你有信心这些 fixes 在你 review 之前已经被验证为安全。
00:41
Visit Blitzy.com to see it in action, that's B-L-I-T-Z-Y.com.
访问 Blitzy.com 看它实际演示，也就是 B-L-I-T-Z-Y.com。
00:49
TikTok is known for getting you to buy things you never knew you wanted.
TikTok 出名的地方，就是能让你买下你根本不知道自己想要的东西。
00:53
Online retailers would love that kind of impact, but they don't have TikTok's data.
线上零售商当然想要那种效果，可他们没有 TikTok 的 data。
00:58
So, they try to show the shopper what they're looking for, and hope that sales will follow.
所以，他们会试着把购物者在找的东西展示给购物者，然后指望销量会跟着来。
01:03
Optimizing for what you can measure is not unique to e-commerce.
针对你能衡量的东西做优化，并不是 e-commerce 独有的。
01:06
AI teams often settle twice.
AI 团队常常会两次退而求其次。
01:09
They lean on signals that are easy to collect, like a click or a thumbs up,
他们依赖容易收集的 signals，比如一次点击或一个点赞，
01:13
and they optimize for metrics that are easiest to evaluate, like relevance or benchmark accuracy.
然后针对最容易评估的 metrics 做优化，比如 relevance 或 benchmark accuracy。
01:18
Then, they hope it all adds up to what they actually care about.
接着，他们希望这一切加起来，就是他们真正在意的东西。
01:22
If you're building a RAG system, a co-pilot, or an agent, you may be making the same bet.
如果你在构建 RAG 系统、co-pilot 或 agent，你可能也在下同样的赌注。
01:27
Zoe Wall's answer, at least in retail, is to look between the clicks.
Zoe Wall 的答案，至少在零售行业，是去观察点击之间发生了什么。
01:31
The company she co-founded in Leeds, Sequin, is built on signals most systems ignore.
她在 Leeds 联合创办的公司 Sequin，建立在大多数系统忽略的信号之上。
01:36
How long a shopper hesitates, how fast they scroll, and how many times they go back and forth between two products before adding one to the card.
购物者犹豫多久、滑动多快，以及在把某件商品加入购物车之前，会在两个商品之间来回多少次。
01:44
And, it judges the results not by what people click, but by what they buy.
而且，它评估结果不是看人们点击了什么，而是看他们买了什么。
01:49
I push back on two things, whether this is just age-old upsell and cross-sell with better data,
我质疑两点：这是不是只是用更好的数据做的老一套 upsell 和 cross-sell，
01:54
and where it starts to look like surveillance.
以及它从哪里开始变得像监控。
01:57
But she made a point about the playbook, most teams follow that I hadn't heard put this bluntly.
但她关于大多数团队都在遵循的 playbook 提出了一个观点，我之前没听过有人把话说得这么直白。
02:02
The biggest innovators in the space are social media companies,
这个领域里最大的创新者是社交媒体公司，
02:07
and they publish a lot in this, right?
而且他们在这个领域发表了很多，对吧？
02:11
But they have a wealth of data that none of our customers have.
但他们拥有大量 data，而我们的客户一个都没有。
02:16
So, if you're a scientist at Wayfair, or a price line,
所以，如果你是 Wayfair 的科学家，或者是 price line 的科学家，
02:21
you're going to read these papers and you're going to try to implement it.
你会去读这些论文，然后会试着把它 implement 起来。
02:23
And you know what? Oftentimes, you're successful at implementing it,
而且你知道吗？很多时候，你确实能成功把它 implement 出来，
02:26
except guess what? It doesn't drive the same outcomes for you as it does for them
但猜怎么着？它不会给你带来和给他们一样的结果，
02:30
because you don't have the data that they have.
因为你没有他们拥有的那些 data。
02:33
And, Sequin's big unlock is that we're finally able to get you that unlock without you having that data.
而且，Sequin 最大的突破在于，我们终于能在你没有那些数据的情况下，也让你实现这种解锁。
02:45
I'm Sam Charrington, and this is the Twimmel AI podcast.
我是 Sam Charrington，这里是 Twimmel AI podcast。
02:49
For over a decade, I've been exploring the ideas and innovations shaping the future of AI
十多年来，我一直在探索塑造 AI 未来的想法和创新，
02:54
through conversations like this one that help you understand what's real, what's next, and what matters.
通过像这样的对话，帮你理解什么是真实的、接下来会发生什么，以及什么才重要。
02:59
Let's jump in.
我们开始吧。
03:09
So, anything that you see on the internet is ranked, right?
所以，你在互联网上看到的任何东西都是被排序过的，对吧？
03:13
Whether it's a pair of pants that you're looking for at a retailer hotels
不管是你在一家零售商那里想找的一条裤子，还是酒店
03:17
that you're looking for for your extrable contents, media,
你为了自己的 extrable 内容、媒体而寻找的
03:23
even the ads you see online are powered by an auction.
就连你在网上看到的广告，也是由 auction 驱动的。
03:27
And these seem maybe to most people to be unrelated,
而在大多数人看来，这些可能看起来彼此没什么关系，
03:31
but every dollar of online spending consumption goes through a ranking model somewhere.
但线上消费支出的每一美元，都会在某个地方经过一个 ranking model。
03:39
And we power those ranking models for the world's largest consumer companies.
而我们为全球最大的消费类公司驱动着这些 ranking model。
03:45
Walk me through kind of the highlights of what you all are doing,
给我大概梳理一下你们正在做的事情里的亮点，
03:51
what makes your approach unique, and you know, with an eye towards or ear towards the kind of technical innovations.
是什么让你们的 approach 独一无二，而且，你知道，也带着对 technical innovations 的关注，或者说去倾听。
04:01
Unsurprisingly, when people think about ranking, they think about information retrieval.
毫不意外，当人们想到 ranking 时，就会想到 information retrieval。
04:06
In fact, I went to graduate school for information retrieval, which is literally the study of search systems, right?
事实上，我研究生读的就是 information retrieval，它字面上就是研究 search systems，对吧？
04:12
How do I retrieve the most relevant items or documents for a given query?
我要怎么针对一个给定的 query，retrieve 出最相关的 items 或 documents？
04:18
Whether that's in a search bar or some kind of relevance module somewhere out there.
不管是在 search bar 里，还是在外面某个地方的某种 relevance module 里。
04:25
And historically, and again, quite, I would say intuitively,
而从历史上看，而且，再说一次，我会说，凭直觉，
04:30
the wisdom here has been that utility should be measured by whether or not I'm able to retrieve documents that are most
这里的共识一直是，utility 应该用我能不能 retrieve 到
04:40
semantically relevant to a user's query, right?
跟用户 query 最 semantically relevant 的 documents 来衡量，对吧？
04:42
That's a very technical side event.
这其实是个非常偏技术的附带话题。
04:44
And that works incredibly well, and I would say by and large,
而且这套方法效果出奇地好，我会说，大体上，
04:49
maybe it's not a solved area of research, but there have been so many advances on it.
也许它还不算是一个已经解决的研究领域，但在这上面已经有很多进展了。
04:54
It's been pretty remarkable.
这真的挺了不起的。
04:57
I mean, the frontier labs have it, you know, pat down or down pat.
我是说，frontier labs 已经把它，你知道，pat down 还是 down pat 了。
05:03
I forget what the idiom is, but they're, they're pretty good at it.
我忘了这个习语具体是什么，但他们，他们确实很擅长。
05:07
It's not really on its own and interesting, I would say area of research.
单就它本身而言，它其实并不是一个有趣的研究领域，我会这么说。
05:12
However, in a commercial context, you know, modern recommender systems,
但是，在商业语境下，你知道，现代 recommender systems，
05:17
if you think about TikTok, Instagram and the like,
如果你想想 TikTok、Instagram 之类的，
05:20
they're not really recommending content anymore,
它们其实已经不再是在推荐内容了，
05:23
and they're definitely not trying to help people find things.
而且它们绝对不是在帮人们找东西。
05:26
Instead, they're bending the end user's will in subtle ways,
相反，它们在用微妙的方式左右 end user 的意愿，
05:32
over time, to make them want things that they otherwise wouldn't want, right?
随着时间推移，让他们想要一些本来不会想要的东西，对吧？
05:37
That destination vacation, you know, the latest fashion that you might not need,
那种目的地度假，你知道，那些你可能并不需要的最新时尚，
05:43
you know, whatever it is that you can think of that people end up just, you know,
你知道，不管你能想到什么，人们最后就只是，你知道，
05:47
buying on Instagram or TikTok.
在 Instagram 或 TikTok 上买。
05:49
They do that not because Instagram and TikTok are, you know, really good
它们这么做，不是因为 Instagram 和 TikTok，你知道，很擅长
05:54
at helping you find what you're looking for.
帮你找到你在找的东西。
05:57
No, they're very, very good at influencing your behavior.
不，它们非常非常擅长影响你的行为。
06:02
And the reason that they're able to do that is that billions of people
而他们之所以能做到这一点，是因为有几十亿人
06:06
use their apps and are frankly addicted to those apps.
在用他们的 app，而且坦白说，对这些 app 上瘾。
06:10
They open their apps, you know, 10 to 21 times a day.
他们打开 app，你知道，一天 10 到 21 次。
06:14
And when billions of people do that, you have an insane amount of data on them.
而当几十亿人都这么做时，你手上就有关于他们的多到离谱的 data。
06:19
And so you're able to create these incredibly immersive,
于是你就能创造出这些极其沉浸式的、
06:23
I would even say, addictive experiences that are hyper personalized and dynamic.
我甚至会说，让人上瘾的体验，而且是 hyper personalized、dynamic 的。
06:29
And in fact, the apps we act in real time to the user's feedback, right?
而事实上，这些 app，我们会 real time 地对用户的 feedback 做出反应，对吧？
06:35
So even, you know, use M. If you, what you see on the site or the app in the day,
所以甚至，你知道，用它们。如果你，你白天在网站或 app 上看到的东西，
06:42
throughout the day will vary based on your interactions with the app.
一天下来，这会根据你和 app 的互动情况而有所不同。
06:47
But the truth is that consumer companies, regardless of how large they are,
但事实是，消费类公司，不管它们有多大，
06:53
you know, regardless of how many customers they have,
你知道，不管它们有多少客户，
06:56
wafer as a good example, price sign is another really good example.
wafer 就是个很好的例子，price sign 也是个非常好的例子。
07:00
These are very large companies that have incredibly large user bases
这些都是非常大的公司，拥有极其庞大的用户群
07:03
to tens of billions of dollars of online transactions.
以及数百亿美元的线上交易。
07:06
But again, people don't use wafer the same way that they use Instagram or TikTok.
但话说回来，人们用 wafer 的方式并不像用 Instagram 或 TikTok 那样。
07:13
And so regardless of how big you are as a consumer business,
所以，不管作为一家消费类企业你有多大，
07:17
you know, you are going to be data starved.
你知道，你会缺 data。
07:23
And so in an environment where you don't have as much data on the user,
所以，在一个你手头没有那么多用户 data 的环境里，
07:28
where people are not addicted to your app,
在这个环境里，人们并没有对你的 app 上瘾，
07:31
how do you influence their behavior in a moment of high intent
你要怎么在 high intent 的时刻影响他们的行为，
07:36
to increase their basket size, increase their average order value,
来提高他们的 basket size、提高他们的 average order value，
07:41
to increase their engagement on your site?
来提高他们在你网站上的 engagement？
07:44
Again, these aren't like big changes.
再说一次，这些并不是什么大改动。
07:46
So on wafer, for example, if you're going to buy a hundred dollar count,
所以，比如说在 wafer 上，如果你要买一个一百美元的 count，
07:51
it's not like sequin can come in and get you to buy a five hundred dollar count
这并不是说 sequin 进来就能让你买一个五百美元的 count
07:56
and also a thousand dollar coffee table.
还顺带买一张一千美元的咖啡桌。
07:59
My point is I can get people to buy a hundred two dollars,
我的意思是，我能让人买一百零二美元的东西，
08:04
but I can do it consistently in that scale.
但我能在这个规模上稳定地做到。
08:07
And when you do that, that translates to forexing their target profit
当你做到这一点时，这就等于 forexing 他们今年的目标利润
08:13
for the year in less than a month.
在不到一个月内。
08:16
Up so, cross sell, like we've been, you know, doing this from a data science perspective,
Upsell、cross sell，就像我们一直，你知道，从 data science 的角度在做这件事，
08:22
pre data science, you know, we've been doing this for a long, long time,
pre data science，你知道，我们做这件事已经很久很久了。
08:26
trying to get people to buy more, what's different about the approach?
想让人买更多，这种做法有什么不同？
08:31
The TLDR, as I was saying earlier, in fact, with the entire history of this field of research,
TLDR 就是，就像我之前说的，其实，纵观这个研究领域的整个历史，
08:38
people have been trying, the wisdom has been,
人们一直在尝试，而大家一直以来的共识是，
08:41
if I can make it easier for people to help, to find things,
如果我能让人们更容易帮上忙、更容易找到东西，
08:46
then by extension, I can get them to convert better.
那么推而广之，我就能让他们 convert 得更好。
08:51
That has been the prevailing wisdom forever.
这一直以来都是主流共识。
08:54
The company that's ever existed in this domain has been trying to solve search
这个领域里存在过的每一家公司，一直都在试图解决 search
08:59
and relevance from the perspective of search and relevance.
以及 relevance，而且是从 search 和 relevance 的角度出发。
09:04
Make it easier for people to find things faster.
让人们更容易、更快地找到东西。
09:07
That is not what sequin does at all.
sequin 完全不是做这个的。
09:10
We are not at all trying to help people find things.
我们完全不是想帮人们找东西。
09:12
We're trying to influence their behavior in a moment of time when they have high intent.
我们想的是，在他们有 high intent 的那个时刻影响他们的行为。
09:18
How do we do that?
那我们怎么做呢？
09:20
How do we go about doing that?
我们该怎么着手去做这件事？
09:21
Which is I think the meat of the matter.
我觉得，这才是问题的核心。
09:22
We started the company with a black box solution that was powered by what we call large event models.
我们创立公司时，用的是一套由我们所谓的 large event models 驱动的 black box solution。
09:29
You can think of it as, first of all, it's not a large language model.
你可以把它理解成：首先，它不是 large language model。
09:33
We actually don't use large language models anywhere.
我们其实在任何地方都不用 large language models。
09:36
That's not our business whatsoever.
那压根不是我们的业务。
09:38
We don't use generated AI either.
我们也不用 generated AI。
09:41
In any case, we serve with large event models.
不管怎样，我们是用 large event models 来提供服务的。
09:45
The idea behind event models is that the same way that large language models are able to generalize.
Event models 背后的思路是，就像 large language models 能够泛化的那种方式。
09:51
To text, including text that they haven't seen before ever.
泛化到文本，包括它们以前从未见过的文本。
09:58
Large event models are able to generate a generalized to human behavior and streams of events all in real time.
Large event models 能够生成、泛化到人类行为和事件流，而且全部实时进行。
10:06
And creating something like that is incredibly difficult to do for a number of reasons.
而要做成这样的东西，因为很多原因，难度大得惊人。
10:14
One, you need to be able to create extremely ultra low latency systems that are able to keep track of data at a very, very fine-grained level.
第一，你得能构建出极其 ultra low latency 的系统，能够在非常非常 fine-grained 的层面上追踪数据。
10:26
And not only talking about clicks and conversions and the traditional good stuff, although we use that as well.
而且不只是说 clicks 和 conversions 以及那些传统的好东西，虽然我们也会用这些。
10:32
But I'm talking about hover scrolls.
但我说的是 hover scrolls。
10:35
I'm actually talking even deeper than that.
其实我讲的比这还要更深。
10:37
The speed of a scroll.
scroll 的速度。
10:40
The dwell time on a hover.
hover 的 dwell time。
10:42
How many times a user goes back and forth?
用户来回多少次？
10:46
Creating systems that are able to digest data in frankly nanoseconds.
打造出能够以——说实话——纳秒级速度消化数据的系统。
10:52
Transform this data also in nanoseconds.
同样在纳秒级内转换这些数据。
10:56
Put it through a model.
把它送进一个 model。
10:58
Several layers of ranking models in milliseconds.
用毫秒级的时间跑好几层 ranking models。
11:01
And serve dynamic rebanked items to a user as their browsing aside all in under 50 milliseconds.
并在 50 毫秒内，把动态 rebanked items 作为用户浏览时的 aside 提供给他们。
11:10
For thousands of requests per second.
还要应对每秒数千次的 requests。
11:13
Is incredibly difficult to do.
这做起来简直难到离谱。
11:17
In fact, we created what we refer to internally as sequin ranking clusters to be able to even make this feasible at all.
事实上，我们创建了内部称为 sequin ranking clusters 的东西，才能让这件事真正可行。
11:27
And so there was a lot of work done there.
所以在这方面做了大量的工作。
11:30
Not only from an algorithm perspective, but also from a data engineering perspective.
这不仅是从 algorithm 的角度，也是从 data engineering 的角度。
11:35
Specifically for ranking and a ton of ML ops work and infrastructure work that was done.
具体来说，是为了 ranking，还有大量已经完成的 ML ops 和 infrastructure 工作。
11:43
It's hard because you need to do three things that are very, very difficult to do on their own.
这很难，因为你需要做三件本身就已经非常非常难的事情。
11:49
You need to have high fidelity leakage free data pipelines, which is a really difficult problem on its own.
你需要有 high fidelity、leakage free 的 data pipelines，而这本身就是一个非常困难的问题。
11:58
You need to have obviously models that are personalized to our user and continue to improve on those models.
显然，你还需要有针对我们的用户 personalized 的 models，并持续改进这些 models。
12:04
And then you need to be able to serve them fast enough such that rankings are able to react to live user feedback and signals.
然后，你还需要能够足够快地 serve 它们，让 rankings 能够对实时的用户 feedback 和 signals 做出反应。
12:14
So building this system at all requires you to do these three things really, really, really, really well.
所以，光是搭建这个系统，就要求你把这三件事做得非常、非常、非常、非常好。
12:22
And there are very few people on the planet that can do this for the explicit objective of generating outcomes for businesses.
而且这世上很少有人能明确以给企业创造成果为目标来做这件事。
12:35
Right.
对。
12:36
And then so that's one part of the problem that sequence solves.
然后，这就是 Sequence 解决的问题的一部分。
12:41
Two, even if you manage to build that system at all, improving anything within that system is really, really difficult because it's gated not only by how fast your scientists can do experiments and do research.
第二，就算你总算把那个系统建起来了，要改进那个系统里的任何东西也真的非常非常难，因为它不仅受限于你的科学家做实验和做研究的速度。
12:57
But the ML ops component, right, the business component of it, right, like how much am I trading off diversity or freshness of products or, you know, bookings or media content for direct impact to my bottom line.
但 ML ops 那块，对吧，它的业务那块，对吧，比如我为了直接影响自己的 bottom line，到底牺牲了多少产品多样性或新鲜度，或者，你知道，预订量或媒体内容。
13:19
And so the ability to have this unified ranking interface that is focused and brings all these disparate parts of the organization together is incredibly helpful on its own.
所以，拥有这个 unified ranking interface，它很聚焦，能把组织里所有这些分散的部分整合到一起，这件事本身就已经非常有帮助。
13:33
But the real secret sauce and where we started the business was with a collection of proprietary ranking algorithms and harnesses that are designed specifically again, not to help people find things.
但真正的秘诀，也是我们创业的起点，是一组专有的 ranking algorithms 和 harnesses，它们被专门设计出来——再说一次——不是为了帮人们找到东西。
13:50
And I know I keep repeating this like I'm aware that I keep repeating this, but that is really the golden thread because once you realize it and you see, hey, even though you're operating in the search space, you're not trying to help people find things.
我知道我一直在重复这一点，好像我知道自己在重复一样，但那真的是那条关键主线，因为一旦你意识到它，并且看到，嘿，即使你是在 search space 里运作，你也不是在试图帮人们找东西。
14:07
You can't unsee it.
你没法当没看见。
14:10
You said that earlier, and I think I had this like and I maybe expressed a little bit of a degree of dissonance and that like that doesn't sound new to me and certainly you've lived the space more than I have, but, you know, we've been talking about kind of next best offer and propensity to buy and like basket size optimization and all these things have been around forever.
你之前说过，我觉得我也有这种感觉，可能表达出了一点不协调，觉得这对我来说并不新鲜，而且你在这个领域肯定比我待得更久，但是，你知道，我们一直在聊类似 next best offer、propensity to buy，还有 basket size optimization 这些东西，这些早就存在了。
14:36
And so is it like is it like old habits die hard and like, you know, they approach the folks reproachiness in a traditional way approach it like a search problem and really the key is to, you know, be like maniacally focused on business metrics, which are kind of harder to do than, you know, focus on like traditional search metrics or.
所以，是不是就像旧习难改，而且，你知道，他们会以一种传统的方式去处理人们的抵触情绪，把它当作 search problem 来对待，而真正的关键是，你知道，要疯狂地专注于 business metrics，这其实比，你知道，专注于传统的 search metrics 要更难，或者。
15:05
It or what yeah yeah yeah so actually the first part of it is is true that you know they're really old habits are hard and they continue to see it as a search problem and try to solve it from a search perspective, let me help people find things more easily.
或者什么？对对对，所以其实第一部分是真的，你知道，他们的旧习惯真的很难改，他们还是继续把它看作一个 search problem，试图从 search 的角度去解决，就是让我帮人们更容易找到东西。
15:20
That is definitely a part of it the other part of it is that like the biggest innovators in the space are social media companies and they publish a lot in this right.
这肯定是其中一部分，另一部分是，这个领域里最大的创新者其实是社交媒体公司，而且他们在这方面发表了很多东西，对吧。
15:32
Amazon used to publish quite a bit about this too, but they have a wealth of data that none of our customers have.
Amazon 以前也发表过不少这方面的东西，但他们拥有大量 data，而我们的客户都没有。
15:40
So if you're a scientist at Wafer or a price line or any of the companies that we work with, you're going to read these papers and you're going to try to implement it and you know what oftentimes you're successful at implementing it except guess what it doesn't drive the same outcomes for you as it does for them because you don't have the data that they have.
所以如果你是 Wafer 或者 price line，或者任何跟我们合作的公司
16:01
And sequence big unlock is that we're finally able to get you.
而 sequence 的 big unlock 是，我们终于能给你。
16:09
That unlock without you having that data and is that a kind of aggregating customers across we don't we don't we don't we do not share any customer data between our customers and that is what makes us.
那个 unlock，在你没有那些数据的情况下也能实现，而且这是不是一种跨客户聚合——我们不，我们不，我们不，我们不会在我们的客户之间共享任何 customer data，而这就是让我们成为我们的原因。
16:25
Truly truly innovative is that by relying so much on true in session live signaling.
真正真正具有创新性的是，通过如此依赖真实的 in session live signaling。
16:34
I get it so within the same customer interaction you're extracting so much more data than the typical.
我明白，所以在同一个客户互动中，你提取的数据比典型情况多得多。
16:42
And so we've made ourselves very very independent of that.
所以我们已经让自己非常非常独立于那一点。
16:46
How do you respond the folks who kind of see that as like you know going deeper into this like surveillance capitalism kind of rabbit hole and like extracting more data about folks you know as they're trying to do pretty pedestrian things like you know that whole I'm sure you've encountered that.
你怎么回应那些人，他们有点像把这看成——你知道的——越陷越深，掉进 surveillance capitalism 那种兔子洞，像是在人们正做着挺普通的事情时提取更多关于他们的数据，你懂的，那一整套，我敢肯定你遇到过这种情况。
17:10
Oh yeah, I hate it. I hate it. I'm very I'm one of the people that's the most vocal opponent of that kind of I love what you call that surveillance capitalism.
哦对，我讨厌它。我讨厌它。我非常，我是那种最直言不讳的反对者之一，我很喜欢你把它叫作 surveillance capitalism 这个说法。
17:21
No, I absolutely hate it and we don't do that.
不，我绝对讨厌它，我们不会那么做。
17:24
And for several reasons one and this is what distinguishes us from social media companies we're not trying to create addictive experiences whatsoever we're not trying to get people addicted to anything.
而且有几个原因，第一，这也是我们和社交媒体公司的区别：我们根本不想制造让人上瘾的体验，我们根本不想让任何人上瘾。
17:35
We don't store people's information it's all in a live transaction.
我们不存用户的信息，一切都实时交易里完成。
17:40
And I cannot tell you the number of times even I've been online and I'm just trying to find well now I have a kid and I'm trying to find toys for my kid but before I try to find toys for my kid my friends kids.
我跟你说，我自己都数不清有多少次，我上网就是想找——现在我有孩子了，我想给我的孩子找玩具，但在给我的孩子找玩具之前，还得给我朋友的孩子找。
17:56
And it's insanely difficult to do that right and I can actually I'm good at verbalizing what I'm looking for and it's still difficult to find that that thing that inspires you.
而且这事儿难到离谱，对吧？我其实挺能把想找的东西说清楚，但还是很难找到那个能启发你的东西。
18:10
So what makes us different is that you know we're not surveilling people we're not like chasing them from sight to sight to sight we're not doing any of that you already have intense which is why you're on my fair dot com you're on it already because you want to do something.
所以我们不一样的地方在于，你知道，我们不是在监视别人，我们不是从一个网站追到另一个网站再到另一个网站，这些我们都不做。你已经有 intent 了，这就是为什么你在 myfair.com 上——你已经在了，因为你想做点什么。
18:27
We just make it easier for you to actually get to the thing that maybe you can't even verbalize to create moments of inspiration for you spontaneous inspiration for you.
我们只是让你更容易真正触达那个东西——也许你甚至没法用语言表达出来——从而为你创造灵感时刻，给你自发的灵感。
18:39
And so it does yes you know price linens up making a lot of money we friends up making a lot of money but the customers end up walking away happy we're not doing anything like price gouging or like weird stuff if anything the difference is like maybe cents on the dollar per transaction that we're able to increase.
所以它确实会，是的，你知道，价格能抬上去，赚很多钱，我们也能赚很多钱，但顾客最后是满意地离开。我们不会做价格欺诈之类的怪事。如果说有什么不同，那就是每笔交易我们也许能多赚几分钱。
18:58
The sense is like you're using the data is available through that interaction like a sales person might in a store.
感觉就像你在利用那次互动里能拿到的数据，就像商店里的销售员可能会做的那样。
19:07
I notice you were looking at that so for a little bit longer I'm going to kind of steer you the things related to that and maybe maybe you actually you know you're looking at the cell phone what about this case also you know and but we actually don't care about the identity of the user we don't care about your name we don't care about your last name we don't care about your location we don't really care about like you know we really care about that interaction the live interactive component.
我注意到你刚才在看那个，所以再稍微多一会儿，我会稍微把你引导到和那个相关的东西上，也许——也许你其实，你知道，你在看那部手机，那这个手机壳怎么样？另外，你知道，但我们其实并不关心用户的身份，我们不关心你的名字，不关心你的姓氏，不关心你的位置，我们真的不太关心，你知道，我们真正关心的是那次互动，那个实时互动的部分。
19:34
You mentioned ml ops several times when you think about the kind of evolution of ml there and ml ops like what are some of the big things that you and your team have learned in trying to put this this you know platform or environment in place.
你提到过好几次 ml ops；当你想到 ML 在那边的这种演进，以及 ml ops 的时候，比如你和你的团队在试着把这样一个、你知道，platform 或 environment 落地时，学到的一些最大的东西是什么？
19:52
I think we got lucky in a sense because maybe like six seven years ago rust didn't have a ton of support in like open store.
我觉得从某种意义上说我们挺幸运的，因为大概六七年前，rust 在像 open source 这样的东西里还没有特别多的支持。
20:03
Open source AI libraries like hugging face that began to change rapidly in the past I would say three to four years and you know this is why I always say whenever you make a successful startup like there's quite a bit of luck there as well.
像 hugging face 这样的 open source AI libraries，在过去——我会说，三到四年里——开始快速变化；你知道，这就是为什么我总是说，每当你做出一家成功的 startup，里面也有相当多的运气。
20:22
And we got but yeah timing is everything we got very very lucky because we started building this company at a time when.
而我们——但没错，timing 就是一切，我们真的非常非常幸运，因为我们开始做这家公司的时候，正好是……
20:29
Support for rust and many many open source.
对 rust 以及很多很多 open source frameworks 的支持，
20:33
Frameworks I am all for and I'm not have done it in Python and no I don't I genuinely don't I don't think that we could have because we.
我是完全支持的；而我并不是说我们能用 Python 把它做出来，不，我真的不觉得我们能，因为我们……
20:44
You know ultra ultra low latency feature transformations serving are so core to what we do I do not think we could have done it in Python certainly we do a lot of R&D in Python but the core of our back end is all in rust so no I can definitively say we could not have done it in Python at all and we're very very lucky to benefit from a lot of the open source contributors that have you know for free.
你知道，ultra ultra low latency 的 feature transformations serving 对我们做的事来说太核心了，我不觉得我们能用 Python 做成；当然，我们在 Python 里做很多 R&D，但我们 back end 的核心全都是 rust，所以不，我可以肯定地说，我们根本不可能用 Python 做成。而且我们非常非常幸运，能受益于很多 open source contributors，他们你知道，免费地……
21:14
Devoted their time and efforts to to bringing support for languages like rust i'm imagining that another you know timing tailwind is the availability of coding agents now you know you may have needed to build a lot of supporting infrastructure before because it's just wasn't available for what you're trying to do but now you can just ask an agent to build that for you for sure.
投入自己的时间和精力，把对 rust 这类语言的支持带起来。我在想，另一个你知道的 timing tailwind，就是现在有 coding agents 可用了；你知道，以前你可能得搭很多 supporting infrastructure，因为对你正在尝试做的事情来说，它就是没有现成的，但现在你直接让一个 agent 帮你搭出来就行，肯定的。
21:39
I mean 1000% you mentioned that way fair was an early adopter of transformers are your models transformer based we have some that are transformer based we have we have I would say all of our models certainly use self attention self attention mechanism I can say that without a doubt.
我是说，1000%——你提到 Wayfair 是 transformers 的早期采用者。你们的 models 是基于 transformer 的吗？我们有一些是基于 transformer 的，我们有，我们有，我会说我们所有的 models 肯定都用了 self attention，self attention mechanism，这点我可以毫无疑问地说。
21:58
All of them do we have many mixture of experts models now that but they all rely on self attention I couldn't say that they use the transformer architecture across the board because I mean now we have so many proprietary models that we have built that kind of it's kind of like an orchestra of many many many models it's like mixture of experts but for ranking.
它们全都这样。我们现在有很多 mixture of experts models，但它们都依赖 self attention。我没法说它们全部都在使用 transformer architecture，因为我是说，现在我们自己构建了太多 proprietary models，某种程度上，它有点像由很多很多很多 models 组成的一个管弦乐团，就像 mixture of experts，但是用于 ranking。
22:28
Is the idea at a system level that you've got all this data coming in and you are applying a bunch of different models you is it routing or it's like your simultaneously passing the same data through the bunch of different models to kind of come up with new features.
从 system level 来看，这个想法是不是：所有数据都进来，然后你应用一堆不同的 models，这是 routing 吗？还是说你同时把同样的数据传给一堆不同的 models，来产生新的 features？
22:47
Actually both right so there's definitely a routing component to it because if you're like in the advertising space things are going to be different for you then you know if you're just like in traditional commerce but if you're if we're operating within an ads module let's just say wafer there are things that the more specialized advertising models that we would use to power our.
其实两者都有，对吧。所以里面肯定有一个 routing component，因为如果你是在广告领域，情况对你来说会不一样，然后你知道，如果你只是像在传统商业里。但如果你是——如果我们是在一个 ads module 里运作，就比如说 Wayfair，有些东西是那些更专门的 advertising models，我们会用来驱动我们的……
23:15
The bidding platforms you might be helpful there so i'm.
那些 bidding platforms，你可能在那儿会帮上忙，所以我是……
23:21
In that case there's some bit of routing involved but also there's a simultaneous kind of rumation sharing multi-tasking that we also do the thing that I was trying to get at is a theme that comes up on the podcast quite a bit and it's.
在那种情况下，会涉及一些 routing，但同时也有一种同时进行的、类似 rumation sharing 的 multi-tasking，我们也会做。我一直想说的是，这是播客里经常出现的一个主题，而且它是……
23:37
This tension between like we're going to take a bunch of data and train some big mongo model and to end and it's going to like taking everything and spit out the one answer.
这种张力就是，比如我们会拿一大堆数据，train 一个超大的 mongo model，end-to-end，然后它会像把所有东西都吃进去，吐出一个答案。
23:49
Versus we've got a bunch of data we're going to apply a bunch of models to it that you know in some ways are you know creating features and then those features are going to be used by some other model you know to or aren't in you know or some set of models or independently it's like this.
而另一边，我们有一堆数据，我们会把一堆 models 应用到它上面，这些 models 在某种程度上，你知道，是在创建 features，然后这些 features 会被某个别的 model 使用，你知道，用来……或者说不是，你知道，或者被某组 models 使用，或者独立使用，就像这样。
24:07
Kind of you know one big end to end thing versus you know we kind of know a lot about this problem like we're going to model you know individual aspects of the problem the way we might think about it as data scientists and.
有点像，你知道，一个大的 end to end 的东西，而你知道，我们对这个问题其实了解很多，就像我们会像 data scientists 那样去 model，你知道，问题的各个单独方面，然后。
24:23
You know that has you know some advantages around explainability model size et cetera et cetera like.
你知道，那样做，你知道，在 explainability、model size 等等等等方面有一些优势，之类的。
24:29
We're closer to the second because we do have these mega models for like advertising domain for you know different domains.
我们更接近第二种，因为我们确实有这些 mega models，用于比如 advertising domain，用于你知道的不同 domains。
24:38
You know but they tend to be on on the whole much smaller than you know what you would see as large shy numerous models in the industry because the the mammoth models nowadays are so so big our largest models are tiny tiny tiny fraction of that.
你知道，但它们总体上，总体上要小得多，比你在行业里看到的 large shy numerous models 要小得多，因为现在那些 mammoth models 特别特别大，我们最大的 models 只是其中极小极小极小的一部分。
24:57
What are some examples of like you know a model and put output transformation like or I'm trying to get a sense for the granularity.
有没有一些例子，比如你知道，model 和 put output transformation 之类的，或者我想感受一下这个 granularity。
25:06
I can give you a couple of examples here so you know creating even if you look at like creating embeddings right for queries right very very simple text queries tend to be smaller size queries in different contexts and travel in.
我可以在这里给你举几个例子，所以你知道，创建，就算你看比如为 queries 创建 embeddings，对吧，非常非常简单的文本 queries，在不同语境下往往是比较小规模的 queries，以及 travel in。
25:25
And travel hospitality and media in in retail can mean completely different things and so using kind of generalized language these giant giant language models to be able to create these holistic representations are not as useful to us in our context because again keep in mind we're a ranking business we do one thing in.
而且 travel、hospitality 和 media 在在 retail 里可以意味着完全不同的东西，所以用那种泛化的语言，这些巨大巨大的 language models 来创建这些 holistic representations，在我们这个语境里对我们没那么有用，因为再说一次，记住我们是做 ranking 的业务，我们只做一件事，in。
25:55
We do it incredibly well and that's re banking so that's an example of where a lot of people were just use you know one kind of embedding model really really great embedding model for text that is able to contain the most amount of.
我们做得非常非常好，那就是 re banking，所以这就是一个例子，说明很多人只是用，你知道，一种 embedding model，非常非常棒的用于文本的 embedding model，它能够包含最多的……
26:09
Layton knowledge about intent etc etc but like queries are like what one two words maybe there is it like there is a lot it's very information star so.
关于 intent 的 latent knowledge 等等等等，但 query 往往就像，什么，一两个词，也许有，就像有很多，它非常 information star，所以。
26:22
So our ability to narrow that to specific context and see just like two or three token queries in a very specific dimension within the users overall journey because the users.
所以，我们能把那收窄到 specific context，并看到在用户整体 journey 中某个非常具体维度里的两三个 token 的 query，因为用户……
26:39
The first class it is in its equine it's not and by the way when I say user I don't mean the user identity I mean the user is a collection of events in a session right when I think about a user think about it as a collection of events in a session I don't care if you're don't Smith or Adam whatever.
这个 first-class citizen 是一个 entity，它不是——顺便说一句，当我说 user 时，我不是指 user identity，我是指 user 是 session 里的一堆 events，对吧？当我想一个 user 时，就把它想成 session 里的一堆 events，我不在乎你是 Don Smith 还是 Adam 什么的。
26:59
To see when you're just a collection of events in a live session and how I'm able to best represent you in in in that context is what sets me aside from every other provider out there and so we take these very very raw streams of events but where that gets routed to in other words how I end up.
看到当你只是 live session 里的一堆 events 时，以及我如何能在那个 context 里最好地代表你，这就是我和外面其他所有 provider 不同的地方。所以我们拿这些非常非常原始的 events stream，但它被 route 到哪里，换句话说，我最终怎么……
27:25
Representing that information to what then goes on to the final re rancor is a big part of our secret sauce on what makes us really really special is there a re rancor or are there re rancors for.
把这些信息表示给接下来进入 final reranker 的东西，是我们 secret sauce 的很大一部分，也是让我们真正、真正特别的地方。是有一个
27:42
I'm envisioning like you know we've talked about wafer you go to wafer you start for blue sofa you got your primary list and that needs to be re ranking re rank but then there's like you may also be interested in this there's like a bunch of other stuff on the page that we power all of it by the way.
我想象的是，就像，你知道，我们聊过 wafer，你上 wafer，你开始找 blue sofa，你拿到你的 primary list，然后那个需要 re-ranking、re-rank，但接着还会有像“你可能也会对这个感兴趣”，页面上还有一大堆别的东西，顺便说一句，这些全是我们提供支持的。
28:00
And there are different they're very different so I'll give you an example even a very tangible example that I think you'll be able to that any listener technical or less technical will be able to follow.
而且它们不一样，非常不一样，所以我给你举个例子，甚至是一个非常具体的例子，我觉得你肯定能听懂，任何听众，不管有没有技术背景，都能跟得上。
28:14
Think about like complimentary products right so let's say you bought a tennis racket and i'm not going to show you other tennis rackets you just added it to your heart right.
想想像 complementary products 这种，对吧，比如说你买了一个网球拍，我不会再给你看其他网球拍，你只是给它点了个心，对吧。
28:26
Um in fact that will most likely dry down convertible you would think but then we also have the classic I just bought a refrigerator on amazon showing me refrigerators like why is that and that's why you would use sequin because you would never do that.
嗯，其实那很可能会拉低 conversion，你会觉得是这样，但然后我们还有那个经典情况：我刚在 Amazon 上买了个冰箱，结果它给我展示冰箱，像，为什么会这样？所以这就是为什么你会用 sequin，因为你绝对不会那么做。
28:41
That's exactly no i'm serious but anyway you go on the as a cart page and we show you complimentary items so we don't show you other tennis rackets we show you hey what about this like.
就是这样，不，我是认真的。但不管怎样，你进到 cart page，我们会给你展示 complimentary items，所以我们不会给你展示其他网球拍，而是给你展示，嘿，这个怎么样，像——
28:51
Sweatband or this tennis ball but which tennis band and which.
护腕，或者这个网球，但具体是哪个网球护腕，以及哪个——
28:56
Sorry which sweat band or ball we show you is going to vary for.
抱歉，我们给你展示哪个护腕或哪个球，会因人而异，
29:02
From user to user but the point i'm trying to get at is that that complimentary module exists throughout many of these sites so it'll exist under product display page but it will also exist on a not cart page.
用户不同会不一样，但我想说的是，那个 complimentary module 存在于很多这类网站中，所以它会出现在 product display page 下面，但它也会出现在 cart page 上。
29:15
And how you rank items on a product display page even though it's still a complimentary module will vary greatly from how you rank items on a complimentary module and at the cart page and that's because by the time the user gets to the at the cart page they've already committed to the tennis racket right.
而且，你在 product display page 上给商品排序的方式，即使它仍然是 complimentary module，也会和在 cart page 上的 complimentary module 里给商品排序的方式差别很大，因为当用户到达 cart page 时，他们已经决定要那个网球拍了，对吧。
29:36
Whereas i need to be able to show greater diversity on the product display page because they haven't necessarily committed to that tennis racket yet does that make sense.
而在 product display page 上，我需要能够展示更大的多样性，因为他们还不一定已经决定要那个网球拍。这说得通吗？
29:49
Yes and no greater diversity and that's just one example but the point is that they're both saying like if I search for tennis racket i mostly want to see tennis rackets i don't want diversity in that list.
是也不是。更大的多样性，这只是个例子，但重点是，它们都在说，比如如果我搜网球拍，我主要想看到网球拍，我不想要那个列表里有多样性。
30:00
You don't but once you click on it you click on a specific tennis racket then you don't know what i'm going to need next exactly well.
你不会，但一旦你点进去，你点了一个具体的网球拍，然后你并不知道我接下来到底需要什么，嗯。
30:10
We're betting that we have an idea before you do are we can get you to to click and purchase well so this isn't interesting thread by how.
我们赌的是，我们比你先有想法，或者我们能让你去点击并购买，好吧，所以这不是个有趣的线索，怎么说呢。
30:21
So i'm you know i'm in this journey i search for tennis racket bunch of tennis rackets have come up.
所以，你知道，我在这段旅程里，我搜索网球拍，一堆网球拍就冒出来了。
30:28
I choose a tennis racket i added to my cart and then you show me a bunch of other stuff how does any of the information you've collected about 12 time and.
我选了一个网球拍，把它加进购物车，然后你给我展示一堆别的东西，你收集到的关于 12 次以及
30:38
All these other things while i'm looking for tennis racket supply to you know which headband which set of balls which you know anything else like a ton of things oh my god you give you give the site so much information you might not even realize so like.
所有这些别的东西，当我在找网球拍的时候，提供给你，你知道，哪条头带、哪套球、哪个，你知道，任何别的东西，像一大堆东西，天哪，你给，你给这个网站太多信息了，你可能都没意识到，所以就像。
30:52
How how did you end up deciding between the blue tennis racket let's say in the pink tennis racket how long does it take you.
你是怎么，怎么最终在蓝色网球拍和，比如说，粉色网球拍之间做决定的？你花了多长时间
31:01
To decide that and how does that compare to every other sequence of events that every other user has ever had like is it classic like collaborative filtering you know folks who.
来决定那个，然后这跟其他每个用户曾经有过的每一个事件序列相比怎么样，就像它是不是经典的，像 collaborative filtering，你知道，那些人谁
31:13
Well because in this case keep in mind the identity of the user actually doesn't matter right and it's all in real time right so i look at the sequence of events that i've collected in real time when people usually let's just say.
嗯，因为在这个情况下，记住用户的 identity 其实不重要，对吧，而且这全都是 in real time，对吧，所以我看的是我 in real time 收集到的事件序列，当人们通常，我们就说。
31:26
Let's just say hover for x and out of time go back and forth x x number of times before settling on this 10 times in a live session and implicit personality modeling like this is a.
我们就说，先 hover x 那么长时间，再来回 x x 次，最后在 live session 里定下来，这样搞 10 次，而像这样的 implicit personality modeling 是一种
31:42
Overly analytical and only retentive person he keeps flipping back and forth between these two things he's.
一个过度分析、又特别抠细节的人，他会在这两件事之间来回反复，他
31:49
scrolling over these lists kind of you know he's like scrolling over the specs or something like that so that he probably wants you know something different from the person who like is flipping through all the pictures.
在这些列表上滑来滑去，你知道，他就像在翻 specs 之类的，所以他大概想要的东西，跟那种一直在翻所有图片的人不一样
32:03
No it's it's the difficult thing to do again more than anything else is like keep in mind the data the environments we operate in are incredibly data star right so with his little information is possible what can I do.
不，最难做到的，再说一遍，比什么都重要的是，要记住：我们所在的环境，data 是极度稀缺的，对吧。所以在信息尽可能少的情况下，我能做什么
32:15
And most of the reliance comes from streaming in session events and getting all of that to a model clean it first of all the data comes raw in a live session so you have to clean up all that data and a lot in like literally nanoseconds.
而且大部分依赖都来自把 session events 流式传进来，再把所有这些送进 model。首先得清洗，因为在 live session 里，data 是 raw 地进来的，所以你得把所有这些 data 清理干净，而且很多真的就是在纳秒级完成
32:32
And it has to go into a model with has to then spit it out in no seconds.
而且它还得进入 model，然后 model 必须在零点几秒内把它吐出来
32:36
So it's a it's a lot when you think about the entire journey that the raw data has to go through but yeah the idea is based on how you got to finally adding that pink or blue or whatever color tennis racket to your card i'll show you the just the tennis ball or a sweatband or bag that you're going to you're going to buy.
所以，当你想想 raw data 要经历的整个旅程，这工作量很大。但没错，这个思路是基于你是如何最终把那个粉色、蓝色或不管什么颜色的网球拍加进购物车的。我会给你展示你正要买的那个网球、止汗带或者包
33:03
And a fact one of the one of the things that we are very very very good at is increasing basket size like regularly the other thing that we're really really good at is typically historically.
而事实上，我们非常非常非常擅长的一件事，就是提高 basket size，而且是经常性地提高。另一件我们真的真的很擅长的事，从历史上看，通常就是
33:19
You know as revenue and organic revenue a considered competing metrics which makes sense typically usually when had revenue goes up it goes at the expense of organic because you at the end of the day have a list of items and either organic goes to the top rods or whatever.
你知道，ad revenue 和 organic revenue 被认为是相互竞争的指标，这通常也说得通，通常当 ad revenue 上升时，它往往是以 organic 为代价，因为归根结底你有一个 item 列表，要么 organic 排到最上面，或者随便什么。
33:36
And what we've been able to consistently shows that we're actually able to drive both up together so that one doesn't cannibalize the other which allows us to compound these massive massive gains on incredibly mature relevant stacks.
而我们一直能持续证明的是，我们实际上能把两者一起推上去，所以一个不会蚕食另一个，这让我们能在非常成熟、相关的 stacks 上叠加这些巨大巨大的收益。
33:52
Elaborate on that for me when you talk about ad revenue are you talking about add spend that's getting the user to the side are you talking about like an Amazon where there's promoted so promoted items versus.
跟我展开讲讲，当你说 ad revenue 的时候，你是指让用户来到站点的 ad spend，还是像 Amazon 那样有 promoted items，所以 promoted items 对比……？
34:05
Okay right so I guess I think about that from an Amazon perspective because it's you know very you know it's it's surface is you know more or less transparent.
好，对，所以我猜我是从 Amazon 的角度来想这件事，因为你知道，它……它的界面，你知道，或多或少是透明的。
34:21
You know a little bit more than less you know whereas I don't know that when I think about way fair I think about sponsored items versus non spider sponsored items but.
你知道，稍微多一点透明，少一点不透明吧，你知道；而我不知道的是，当我想到 Wayfair 时，我会想到 sponsored items 对 non-sponsored items，但是。
34:30
You know a side of that scale is probably doing that whether or not I know about it is kind of the.
你知道，那种规模的站点可能都在这么做，不管我知不知道，大概就是那个……
34:37
I think they're fairly transparent about it I just think that you see it more on Amazon because Amazon has a much much larger ads business and they're a huge company.
我觉得他们对这件事相当透明，我只是觉得你在 Amazon 上更常看到它，因为 Amazon 有一个大得多得多的 ads business，而且他们是一家巨型公司。
34:46
And so you're saying that you know typically when when an organization is kind of they optimize ad revenue as a.
所以你是说，你知道，通常当……当一个组织有点……他们把 ad revenue 优化成一个……
34:59
As the primary objective conversion tends to go down yeah yeah okay.
作为主要目标的 conversion 往往会下降，嗯嗯，好吧。
35:04
Because people will because they're optimizing for people clicking on stuff as opposed to buying things as opposed to buying things I would see one has proven that we can do time and time again.
因为人们会，因为他们优化的是让人点击东西，而不是买东西，而不是买东西，我看有一件事已经证明了，我们可以一次又一次做到。
35:13
Is we can get people to click significantly more and get them to also buy after the fact why is that is the same thing we're talking about just more data like is it.
就是我们能让人点击显著更多，还能让他们事后也买，为什么会这样？这跟我们说的是不是同一件事，只是更多 data，是这样吗。
35:24
That you're individually optimizing these things and so they both go up or is it some cross no they're they both go up we're just creating like much more we're showing people things that they actually want to see.
是你把这些东西分别单独优化，所以它们俩都涨了，还是说有什么 cross？不，它们俩都涨了，我们只是在创造，像是多得多——我们在给人们展示他们真正想看的东西。
35:36
Based on what we call micro feedback that we get.
基于我们所谓的、我们拿到的 micro feedback。
35:43
Things like scroll speed and stuff yeah it's funny you you say that because I was thinking just a bit ago about how.
比如 scroll speed 之类的，嗯，挺有意思的，你你这么说，因为我刚刚还在想，怎么会……
35:51
You know typically the the big kind of aha moment for relevance is if you can find you know some feedback like any feedback and usually it's like.
你知道，通常来说，relevance 的那种重大 aha moment，就是如果你能找到，你知道，某种 feedback，像任何 feedback，通常它就像是……
36:06
Or you know historically has been in and i'm thinking about like rag systems for example like you know did the user click like you know that's a the user click on the thing that we showed them.
或者说，你知道，历史上一直是——比如我在想 rag systems 这种，你知道，用户有没有点击，你知道，那就是——用户点了我们给他们展示的那个东西。
36:20
Like the user thumb up or thumb down like you know there's i would say we kind of start for like you know any kind of feedback and like they you know have very few levers or this explicit feedback and now like you're.
就像用户点赞或点踩，你知道，我会说我们差不多是从任何形式的 feedback 开始的，而他们你知道，能用的杠杆很少，或者就只有这种 explicit feedback，而现在你……
36:38
Just creating way more implicit way more feedback implicit feedback yeah.
只是在产生多得多的 implicit，多得多的 feedback，implicit feedback，对。
36:44
Yeah no and we rely I would say overwhelmingly on implicit feedback data.
只是在创造多得多的 implicit、多得多的 feedback，implicit feedback，对。
36:49
Overwhelmingly it's crazy I would say I was talking to I was talking to and i'm not going to name because you know I don't want to embarrass them but I was talking to see level executives from like a very very major bank.
总的来说，这太疯狂了，我会说，我当时在和——我当时在和，我不会点名，因为你知道我不想让他们难堪，但我当时在和一家非常非常大的银行的 C-level executives 聊天。
37:01
So they're like you know you'd be like you know work with us where this giant bank and we have all this transaction data and blah blah blah how's like actually this like that's the least.
对，没错，我会说我们压倒性地依赖 implicit feedback data。
37:10
Interesting data you could give me because you're only showing me a sliver of human behavior that transaction data is useless because.
你能给我的有趣数据，因为你只给我看了人类行为的一小部分，那些 transaction data 是没用的，因为……
37:21
99% of people don't actually convert i'm interested in the 99% of data you actually don't have and that's what I have and you don't have that.
压倒性地，太疯狂了。我会说，我当时在跟——我当时在跟，我不会点名，因为你知道我不想让他们难堪，但我在跟一家非常非常大的银行的 C-level executives 聊天。
37:31
We didn't get very far in stack like we mentioned rust like i'm the things that come to mind as you describe the system or questions are like analytical databases no sequel.
我们在 stack 上没聊太深，就像我们提到的 Rust，我——当你描述这个系统时，我想到的东西，或者问题，就像 analytical databases、NoSQL。
37:45
You know cues confluence kind of thing like what are some of the infrastructural the data infrastructure elements that allow you to do this.
你知道的，就是那种 cues Confluence 之类的东西，比如说有哪些基础设施的、数据基础设施的要素能让你做到这一点。
37:55
Yeah so we definitely use a lot of no sequel for sure distributed databases we definitely use rust you rely heavily on.
是的，所以我们肯定大量使用 NoSQL，肯定用分布式数据库，我们肯定用 Rust，你非常依赖。
38:07
We rely very very heavily on open source technology.
我们非常非常依赖开源技术。
38:12
We use a lot of hugging face huge fans of hugging face here.
我们大量使用 Hugging Face，这里的人都是 Hugging Face 的超级粉丝。
38:18
I can tell you also what we don't use much of we almost do not use elements at all.
我还可以告诉你我们不怎么用什么，我们几乎完全不用 elements。
38:25
It's just not relevant to our business we're not trying to do gender of anything we're not a lab we're not trying to reinvent the wheel there.
它只是跟我们的业务不相关，我们不是在试图做任何性别的东西，我们不是实验室，我们不是在那里重新发明轮子。
38:36
So yeah honestly it's just a it's a lot of ranking models it's actually like the thing is that getting these models to work requires you to have an innate understanding of what features drive value and that is not written in any recent favor.
所以是的，老实说，它只是...它有很多 ranking models，其实就像，问题是让这些模型工作起来需要你对什么特征驱动价值有内在的理解，而这并没有写在任何最近的 favor 里。
38:55
And so what we have done is we've created a harness library with proprietary institutional knowledge that we've codified what we call residual knowledge.
所以我们所做的是，我们创建了一个 harness library，其中包含我们编码的专有机构知识，我们称之为 residual knowledge。
39:08
That matter has not republished about that Google has never published about that Pinterest has never published about their biggest money makers.
这件事没有被重新发表过，Google 从未发表过相关内容，Pinterest 也从未发表过他们最赚钱的那部分。
39:19
Even the papers you see them publish like they're not the scientists that work on them have to wait for months to get legal approval and the version of those papers that actually ends up getting published is a very very watered down.
就算你看到他们发表的那些论文，那些真正做这些研究的科学家也得等好几个月才能拿到法律审批，而最终真正发表出来的论文版本是非常非常被稀释过的。
39:36
Version of those of that line of work and so what we've done is that we have created a harness library and a knowledge base of some of the most effective proprietary methodologies to drive outcomes in ranking.
就是那条工作线里那些内容的稀释版。所以我们做的事情是，我们建了一个 harness library 和一个 knowledge base，里面是一些最有效的 proprietary methodologies，用来在 ranking 上驱动结果。
39:58
And some of it has come from hiring folks from these institutions and you know gathering residual knowledge that's not published everywhere and then second of all we have forward deployed engineers embedded in all the customers we work with so we actually don't need feedback data from the customers.
其中一些来自从这些机构招人，你懂的，收集那些不是到处都会公开的 residual knowledge；其次，我们在所有合作的客户那里都嵌入了 forward deployed engineers，所以我们其实不需要客户提供 feedback data。
40:18
But residual information that we gather about what works what doesn't work is all codified.
但我们收集到的关于什么有效、什么无效的 residual information，全都被 codified 了。
40:24
You mentioned or you referred to is the harness and recursive like the same concept or is the harness a sub concept or harness is one component of our the entirety of a recursive stack so you can think of the recursive stack as like being this engine with many many different components.
你提到或者说你指的是，harness 和 recursive 是同一个概念吗，还是 harness 是一个子概念，或者 harness 是我们整个 recursive stack 的一个组件？所以你可以把 recursive stack 想成是一个有很多很多不同组件的引擎。
40:46
We have a proprietary knowledge store we have a harness library we have what we call frontier ranking models all of which work together to power the recursive stack.
我们有一个 proprietary knowledge store，有一个 harness library，还有我们称之为 frontier ranking models 的东西，它们全都协同工作，为 recursive stack 提供动力。
40:58
And so our core IP is the recursive stack itself that produces these insane making models that are able to you know for five 10x fortune 500 companies revenue in just a matter of weeks.
所以我们的 core IP 就是 recursive stack 本身，它产出这些疯狂赚钱的 models，能够在短短几周内把 Fortune 500 公司的 revenue 提升 5 到 10 倍。
41:16
And that's why we are able to allow them to then own the ranking models right because the real meat of the matter is the ability to continuously build ranking models better and better and better and better and better ranking models that continue to drive revenue.
这就是为什么我们能够让他们之后拥有 ranking models，对吧，因为真正关键的地方在于，能够持续不断地把 ranking models 建得越来越好、越来越好、越来越好、越来越好、越来越好，而这些 ranking models 能持续带来营收。
41:33
And so when you talk about this harness what makes it a harness is just a library and like harness is the cool word or is there something harnessy about it in the same way that you know codex or chat you know codex or cloud code is a harness yeah so it's actually collection of harnesses that we have definitely not certainly not a cool word at the time.
所以当你谈到这个 harness 时，让它成为 harness 的原因其实就只是一个 library，harness 是个很酷的词，还是它本身真有什么 harness 特质？就像你知道的，codex 或者 chat，你知道的，codex 或者 cloud code 就是一个 harness，对吧。所以它其实是一堆 harnesses 的集合，我们当时肯定、绝对不是把它当成一个很酷的词。
42:00
And that we started building it in we just needed a way to specifically have what we call evil harnesses to find what we call the commercial ground truth of interactions and so that's why we started to build these harnesses and we quickly realized in short order that even you know in the in the
而我们开始构建它的时候，我们只是需要一种方式，专门拥有我们称之为 evil harnesses 的东西，来找到我们称之为交互的 commercial ground truth，所以这就是我们为什么开始构建这些 harnesses，而我们很快就意识到，即使在，你知道，在——
42:30
space of evaluating these ranking models how we're able to do so offline in let's say the travel space which is you know considered um obviously higher ticket items has a ton of overlap with the furniture space for example which also has higher ticket items.
评估这些 ranking models 的领域里，我们如何能够 offline 地做到这一点，比如说 travel 领域，你知道，这被认为是，嗯，显然客单价更高的品类，它和比如 furniture 领域有大量重叠，因为 furniture 领域也有更高的客单价。
42:50
But guess what it doesn't have a lot of overlap with with the ad space because the user is just clicking on things and that translates directly to revenue but it's not coming at a cost to a user so how you evaluate these different kinds of ranking models.
但猜怎么着，它和 ad 领域并没有太多重叠，因为用户只是在点东西，而这会直接转化为营收，但它并不会让用户付出成本，所以你要怎么评估这些不同类型的 ranking models。
43:05
You know requires to have a very very rigorous approach and so we came up with a collection of these evaluation harnesses and that's that was the start of our of our harness that very.
你知道，这需要有一套非常非常严谨的方法，所以我们想出了一系列这样的 evaluation harnesses，而那就是我们的 harness 的起点，那个非常……
43:20
And you call the tool or the platform recursive like what's recursive about it recursive ranking intelligence is kind of first first name is that is recursive just a name or as recursion a part of what you're doing actually if I were to be very very specific about it it's most similar to like tail recursion meaning that you're able to instead of having to rebuild the entire ranking iterate on the entire ranking.
而你把这个工具或平台叫做 recursive，那它哪里 recursive 呢？recursive ranking intelligence 算是第一个——第一个名字吧，recursive 只是一个名字，还是 recursion 其实是你正在做的事情的一部分？其实，如果要非常非常具体地说，它最像 tail recursion，意思是你可以不必重建整个 ranking、不必在整个 ranking 上 iterate。
43:50
So you're able to learn continuously and make minor modifications here and there and in doing so you're able to compound gains that are much faster rate at significantly less at a fraction of the compute.
所以你能够持续学习，在这里那里做一些小的修改，这样一来，你就能以快得多的速度、显著更少的——只需要一小部分 compute，来复利式地累积收益。
44:00
So that is if I were to get very technical that is why we call it recursion is that we don't have to rebuild ranking models previously.
所以，如果我要讲得非常技术一点，这就是为什么我们把它叫做 recursion——就是我们不需要像以前那样重新构建 ranking models。
44:10
When we started the business we actually did so we would train the ranking models daily sometimes hourly but it would be like from scratch.
我们刚开始做这个业务的时候，其实就是这么做的：我们会每天训练 ranking models，有时候每小时一次，但基本上都是从零开始。
44:20
And it took us quite a bit of time actually to figure out a way how can I have how can I.
而且我们其实花了挺长时间，才想出一种办法：我怎么能有，我怎么能。
44:30
What I'll quote iterate on this without having to rebuild it from scratch every single either day or hour or whatever cadence and.
我所谓的 iterate 这个，而不用每天、每小时或者不管什么 cadence 都从零重建，而且。
44:40
And insert that recursive element in there we talked about this a little bit earlier but kind of with the context of the full conversation thus far.
然后把那个 recursive 元素插进去；我们之前稍微聊过这个，但算是结合到目前为止整个对话的语境。
45:08
What is large event models is that just an instance of one of your ranking models or is that a particular architecture or.
large event models 是什么？它只是你们某个 ranking model 的一个实例，还是一种特定的架构，还是……
45:18
It's one kind it started as the thing that we did let me be very clear that in itself was insanely difficult for us to do to create these models that were able to generalize to stream of events.
它是其中一种，一开始就是我们做的那个东西。让我说清楚，这件事本身对我们来说就极其困难——要创建出能够 generalize 到 stream of events 的这些模型。
45:35
It was extremely intuitive and it is but getting it to generalize across customers and domains is insanely difficult like intuitively it makes a lot of sense.
它非常直观，现在也是，但让它跨客户和 domains 去 generalize 却极其困难，就是，直觉上这非常说得通。
45:46
But to get it to work in travel and media and retail at a different cross points different hierarchies without relying on any end user identity is really really difficult to do so that was the heart one of the hardest things that we can we began to work on.
但要让它在旅游、媒体和零售里，在不同 cross points、不同 hierarchies 上都能跑通，而且不依赖任何 end user identity，这真的真的非常难。所以这就是核心，是我们开始着手做的最难的事情之一。
46:02
It's the input is a stream of events and each event is kind of has a label and a value for example yeah so it would be like hover combinations right so it would be the query the embedding for the query and that query could either be literally the the source item or it could be an actual query of words that users search for whatever.
它的输入是一串 events，每个 event 大概都有一个 label 和一个 value。比如说，对，就像 hover 的组合，对吧？所以它会是一个 query、这个 query 的 embedding，而这个 query 要么字面上就是 source item，要么是用户实际搜索的词组成的 query，之类的。
46:28
It was the first very very first thing that we worked on and it was the black box model and top of which we built a rankers.
这是我们最早、最早着手做的东西，它是一个 black box model，我们在它上面搭了 rankers。
46:37
Then we built what we what we called large mesh models which were like large event models but they're much more relationally focused.
然后我们构建了我们所说的 large mesh models，它们有点像 large event models，但更偏重 relational。
46:46
So now I'm looking not only at streams but I'm looking at mainstream simultaneously and looking at the relationship between those streams together and trying to create generalized representations out of that right then we started creating different classes of what we called frontier ranking models which were independent of large event models and large match models but which large match models and large event models contributed to right.
所以现在我不只看 streams，还会同时看 mainstream，观察这些 streams 之间的关系，并试图从中生成 generalized representations。对，然后我们开始创建不同类别的、我们称之为 frontier ranking models 的东西，它们独立于 large event models 和 large match models，但 large match models 和 large event models 会为它们做贡献，对吧。
47:15
And so we built these collection of many many many now in the hundreds of thousands of frontier ranking models that we have that are proprietary that are not published anywhere a lot of them with very novel architectures and that then became the first building block of the recursive stack.
所以我们建起了这一大堆 frontier ranking models，很多很多，现在已经有几十万个了，都是我们专有的，任何地方都没有公开，其中很多都有非常新颖的 architectures，然后它们就成了 recursive stack 的第一个 building block。
47:38
Over time we started to develop harnesses and knowledge store et cetera et cetera to what degree does the idea of time series models or time series modeling plan to what you're doing do you think of these events as time series already just kind of capture a frame of events and kind of process it.
随着时间推移，我们开始开发 harnesses 和 knowledge store 等等等等。time series models 或 time series modeling 这个概念，跟你正在做的事情有多大关系？你会不会已经把这些 events 当成 time series，只是捕捉一帧 events 然后处理它？
48:01
Independent of you know longer time sequences earlier I will tell you that earlier it played a much much bigger role earlier in our journey now I would say it's a small part of a growing stock.
独立于，你知道，更长的时间序列。早先——我会告诉你，早先它在我们旅程中扮演的角色要大得多得多；现在我会说，它只是不断增长的整体里的一小部分。
48:19
And I'll tell you why I think I can give an example that is perhaps intuitive for people to catch on to you know time series.
我来告诉你为什么我觉得可以举个例子，可能大家比较容易理解，你知道，time series。
48:30
And sequence based modeling works really really well for languages and generalize reason because you can show a follows and sorry b follows from a and there is a general order to things and even though we're in the ordering business and all our whole businesses ranking i'll take you back to the fundamental concept that the first class citizen at sequin is the user.
而 sequence based modeling 对语言和 generalize reason 真的非常非常有效，因为你可以展示 a 跟着 b——抱歉，b 从 a 推出，而且事情有一个总体顺序；尽管我们做的是 ordering 的生意，整个业务都是 ranking，但我会把你带回那个基本概念：在 sequin，first class citizen 是用户。
48:57
And the user does not behave as cleanly or logically as as you would expect there's a lot of madness and people's behavior people aren't like languages language despite how however creative that you can get has a certain repetition to it that human beings simply do not have right you could be in the middle of buying a couch right.
而用户的行为并不像你期望的那样干净或有逻辑，人的行为里有很多疯狂；人不像语言——不管语言能多有创造力，它都有某种重复性，而人类根本没有，对吧？你可能正在买沙发，对吧。
49:24
And your dog just comes walking in and completely was your train of thought and then you suddenly realize i'm going to buy a dog back to you know there is something inherently unpredictable about human behavior to the degree that we've talked about like a big unlock being like capturing more information from the user capturing me the events of the browser is certainly complex but doable but then you talked about like the infrastructure to.
然后你的狗走进来，完全打乱了你的思路，接着你突然意识到，我要买只狗，回到——你知道，人类行为有一种内在的不可预测性，到了我们谈过的那种程度，比如一个大的 unlock 就像是捕捉用户更多的信息，捕捉浏览器里的事件当然复杂但可行，但接着你谈到比如 infrastructure 来。
49:53
Manage those events are scale is it's a lot it's a lot and then transform all of those in real time and then know what to transform in it yes which by the way we have dedicated models to do that so you'll literally have streaming data that's going in not to one model but it's going into like hundreds of models.
管理那些事件、scale 起来，这工作量很大，很大；然后还要在 real time 里转换所有这些，并且知道要把什么转换进去。是的，顺便说一句，我们有专门的 models 来做这件事，所以你会真的有 streaming data 进去，不是进一个 model，而是进几百个 models。
50:15
What all have to then output that data reconstructed put it into the final rinker and have it all be served to the user in sub 20 sub 50 sub 100 milliseconds it's an enormous infrastructure and modeling undertaking with that in mind when you think about kind of the technical horizon or the innovation opportunities like what are the big.
所有这些东西还都得输出，把数据重建，放进最终的 rinker，然后以 sub 20、sub 50、sub 100 milliseconds 的速度全部服务给用户，这是一个巨大的 infrastructure 和 modeling 工程。考虑到这一点，当你想到 technical horizon 或创新机会时，比如最大的...
50:43
Opportunities to go after that will help you kind of push your system and approach further there are several routes that we're looking at right now without revealing too much I think.
可以去追的机会，能帮你把系统和 approach 进一步推进。我们现在正在看几条路线，不过我觉得，不能透露太多。
50:58
You know.
你知道。
50:59
One of the one of the biggest debates we've had internally is how much do we invest in what well represent traditional representation learning of like images or texts and the like versus actually continuing to innovate the ranking models themselves and the nuances there.
我们内部最大的争论之一，就是到底要投入多少在所谓的传统 representation learning 上，比如图像或文本之类的东西，还是要继续创新 ranking models 本身以及其中的细微之处。
51:22
And I was talking about how are evil harnesses and the library there is absolutely critical to our success and so I would say right now at this point in time anyway we're very very focused I would say on the ladder rather than focusing on things like that we think are going to be very commoditized if if not already.
我当时在说，我们的 eval harnesses 和那里的 library 对我们的成功绝对至关重要，所以我会说，现在这个时间点，无论如何，我们非常非常专注，我会说是专注在后者上，而不是那些我们认为将会、或者说已经高度商品化的东西上。
51:46
You know in terms of having multimodal representations of like hero images on company websites or why not.
你知道，就比如公司网站上的 hero images 这类东西的 multimodal representations 来说，或者为什么不可呢。
51:56
So I would say that is probably where we're most focused on are on things that actually make the ranking the core ranking model itself better rather than the auxiliary components that are that involve more traditional representation learning we're going to leave that to the big image.
所以我会说，这大概就是我们最专注的地方，也就是那些真正能让 ranking、让核心 ranking model 本身变得更好的东西，而不是那些涉及更传统 representation learning 的辅助组件；我们会把那些留给大型图像。
52:15
Or video or other frontier labs to do and open source projects and will continue to benefit from that research as it comes out.
或视频或其他 frontier labs 去做和 open source projects，并且会继续从这些研究中受益，随着它们出现。
52:25
Well, it's always been great chatting with you and learning a bit about you know what you're up to and how you're approaching ranking.
嗯，跟你聊天一直都很棒，也能稍微了解一下你在做什么，以及你如何做 ranking。
52:33
It's been super interesting.
这真的非常有意思。
52:35
Thank you so much for having me.
非常感谢你邀请我。
52:37
It's been a huge pleasure and a huge fan of your podcast so.
非常荣幸，也是你播客的忠实粉丝。
52:41
It's an honor to be here.
很荣幸能来到这里。