I recently moved to a new team working on Southeast Asia TikTok Shop marketing and user growth. I had not worked as a growth engineer before, so I used the team’s existing projects and GPT to learn the area. The biggest thing I learned is that data is one of the most important things in growth engineering. These are my notes from that learning process, written by hand.

Engineering and drawing pages are two different things, and they produce different results.

Many engineers follow a product idea, add a module to the homepage, or add a popup at some transaction step, but they do not know why they are doing it. The result is often poor, and the benefit is hard to verify. An engineering approach is closer to debugging: find where the user drops, where the user gets stuck, and why the user fails to convert into transaction value. Then optimize that step and use an experiment to verify whether the optimization meets the expectation.

The most important value an engineer brings to growth work is turning a business idea into a verifiable causal chain.

Take ecommerce as an example. The core metric is GMV, gross merchandise value. Suppose your manager says GMV is below the industry average and needs to increase by xx%. You get busy. You guess that the homepage does not show enough campaign information, so you add a campaign module. What happens? GMV does not increase much. Even if it does increase, Zihan and Liming may also be shipping features at the same time. It is hard to explain that the GMV increase came from your change.

The reason is that GMV is a highly abstract metric. Its rise or fall is the combined result of many factors. You do not know whether the cause is product quality, price, system performance, or something else. If you make decisions only from this top-level number, your change may miss the root cause. It is the same as debugging a long chain without logs. If you only look at the final result and guess the root cause, efficiency is low.

A better approach is to turn the whole growth process into a verifiable system: use data to drive decisions, verify decisions, and measure impact.

Figure: Growth engineering validation loop. A business idea becomes a metric hypothesis, tracking design, experiment, decision, and impact measurement. generated by gpt-image-2.

Ecommerce growth data engineering

Below I use ecommerce growth engineering, the kind of project I work on, to explain how data-driven growth engineering works.

Node decomposition

The final metric for ecommerce transactions is GMV. It represents the transaction scale generated on the platform. It is highly abstract. Other metrics only affect it indirectly.

Engineering decomposition

To make analysis easier, we can break the metric down by process.

To increase GMV, we need as many users as possible to reach a successful transaction with a high enough average order value. We can turn this process into a sequence of metrics, track the user count at each important step, and form a funnel:

Figure: Ecommerce conversion funnel from entry traffic to order success and GMV. generated by gpt-image-2.

We can divide downstream users by upstream users to get each step’s conversion rate. UV, the starting user count, multiplied by all conversion rates becomes GMV:

Each layer has a different meaning:

Operating decomposition

We can also break it down from the user operating perspective, to understand which user group’s GMV is rising or falling and which lever should be pulled.

By new and existing users, or by region and age:

Then we can judge who caused the GMV change:

1
2
3
4

Existing user activity drops -> retention / recall problem
New users increase but do not buy -> new user landing problem
Buyer count is stable but order frequency drops -> repurchase / campaign problem
Order count is stable but AOV drops -> assortment / price / bundle problem

Connecting the nodes

Breaking down nodes helps us see where user loss is largest and helps us find the root cause. But a root cause is not enough. We also need to ensure that the fix eventually affects the final target, GMV. That means the whole chain must be connected in the data. For example, when a user opens the homepage, we need to know whether this visit enters a PDP, whether it places an order, and whether it contributes GMV.

Without that connection, it is easy to reach the wrong decision. If we only care about the conversion from homepage open to product detail page, the extreme solution is a redirect: once the user enters the homepage, we automatically jump to a product detail page. That makes the conversion rate 100%, but it will almost certainly damage GMV.

The same applies to some app download tactics. Downloads may increase on the surface, but if they do not help users place orders or become retained users, ROI is still very low.

So a feature change should be guided by whether it can transmit value to GMV, not by one local step metric.

Several analysis methods

Funnel analysis

Funnel analysis answers: “At which step did users drop?”

Figure: Web and app growth path. A promo click can continue through web checkout or branch into app activation, order, and retention. generated by gpt-image-2.

Funnel analysis mainly locates responsibility:

• Low CTR: frontend display, campaign benefit point, product ranking, recommendation strategy.
• Low PDP arrival: navigation, performance, product status, deeplink, tracking.
• Low PDP add-to-cart: price, stock, promotion, review, trust.
• Low checkout payment: login, address, payment, shipping fee, risk control.
• Low first order after app open: inconsistent landing, benefit gap, login friction, lost product context.

Cohort / retention analysis

Cohort analysis answers: “What happened to this group of users later?” You cannot rely on same-day app opens alone, because many clicks are accidental or low-quality traffic.

1
2
3
4
5
6
7
8

Group by the date the user enters the app:
D0 app open
D0 login
D0 first order
D1 return visit
D7 return visit
D30 return visit
D7/D30 GMV

If an A/B plan increases app opens by 20%, but D7 retention and first order do not improve, that is not good growth. It is traffic transport.

Attribution analysis

Attribution answers: “Who should get credit for this GMV?”

Common models:

A/B Test

A/B answers: “Did this change cause the result?”

A qualified experiment must have:

• Primary Metric: only one, used to decide win or loss. For example, Web-assisted App GMV per UV, order conversion rate, or buyer rate.
• Secondary Metrics: explain why it wins or loses. For example, CTR, PDP arrival rate, add-to-cart rate.
• Guardrail Metrics: metrics that cannot be harmed. For example, LCP, CLS, bounce, refund rate, complaints, other-channel GMV.
• Stable Randomization: bucket by user or device stably. Do not change the group every refresh.
• Enough Sample Size: do not draw conclusions when samples are insufficient.
• Run Full Cycle: cover a full business cycle. Ecommerce usually cannot run for only half a day.
• No Peeking: stopping the experiment halfway because the number looks good is error-prone.
• SRM Check: if experiment/control traffic ratio is abnormal, the result is not trustworthy.

An example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Experiment: show an app landing page when MWeb users click purchase

Control: existing flow
Treatment: after Buy Now, enter app landing, guide the user to open the app and preserve product / benefit context

Primary:
  Web-assisted App paid GMV per exposed UV within 7 days

Secondary:
  App open rate
  App landing success rate
  App login rate
  App first order rate

Guardrail:
  Web direct GMV
  total paid GMV
  bounce rate
  page performance
  complaint/refund

Full case: use App Landing after MWeb purchase click

The sections above covered GMV decomposition, funnels, attribution, retention, and A/B. If these concepts are viewed separately, they still feel scattered. Below is a real project scenario that connects them and shows how data-driven decisions work.

Idea: after Toko Mobile Web users click purchase, should we use an App Landing Page to guide them into the app to finish the purchase?

1. Business problem

From the data, Toko has a typical structural problem:

• Toko Web: DAU is high and stable, but GMV is low.
• Toko App: GMV is high, but DAU is declining.

This means Web may carry a lot of product browsing, SEO, search, and campaign entry traffic, but it has not converted that traffic into transactions well. Mobile Web is especially suitable for app guidance when users browse products on their phones, because the app usually has better login state, payment experience, benefit continuity, and repurchase capability.

But the current problem is that Web has an app banner, yet the banner lacks a clear scenario and user motivation. Users do not necessarily feel they should tap just because they see “open app.” A better scenario is: when a user has clicked Buy Now or shown purchase intent, use a landing page to explain that opening the app can preserve the current product, keep the current benefit, and continue the purchase.

So this project needs to verify:

Can high-intent MWeb users be converted through better app landing into app first order, retention, and long-term GMV?

Business hypothesis

The hypothesis behind this requirement can be split into three layers:

Hypothesis 1: there is a group of high-purchase-intent users on MWeb.

Evidence: they clicked Buy Now / Add to Cart / coupon claim / Flash Sale product.

Hypothesis 2: the current Web -> App guidance efficiency is low.

Reasons: generic banner, weak motivation, incomplete deeplink handoff, product / coupon / price context may be lost.

Hypothesis 3: if App Landing is used after purchase intent happens.

Result: app open rate, login rate, first order rate, later retention, and GMV will improve.

Metric chain

The complete metric chain for this project should be:

The diagram below splits MWeb eligible UV to D30 GMV / LTV into Web intent and App value.

Funnel:

Figure: MWeb-to-App measurement funnel for the app landing experiment. Value is measured per eligible MWeb UV rather than total GMV alone. generated by gpt-image-2.

Each layer answers a different question:

Experiment design

We can design an A/B experiment:

1
2
3
4
5
6
7
8
9
10

Experiment population:
Mobile Web users who click Buy Now / Add to Cart / high-intent CTA

Control:
Keep the existing flow, such as continuing the Web purchase flow or current app banner logic

Treatment:
After Buy Now, enter App Landing Page
Landing Page shows current product, price, campaign benefit, and app open button
After click, open the app through deeplink and preserve product, coupon, campaign, and attribution_id

Experiment structure:

Figure: A/B experiment split for the app landing page. Stable buckets keep control and treatment comparable while metrics and guardrails decide rollout. generated by gpt-image-2.

Metric design

The Primary Metric can be:

Why this metric?

Because it constrains three things at the same time:

Secondary Metrics:

1
2
3
4
5
6
7
8

Landing Page View Rate
Deep Link Click Rate
Deep Link Success Rate
App Landing Success Rate
Login Rate
First Order Rate
D7 Retention
D7 Repeat Order Rate

Guardrail Metrics:

1
2
3
4
5
6

Web Direct GMV
Total (Web+App) Paid GMV
Bounce Rate
Page Performance
Refund / Cancel / Complaint
SEO Traffic

Tracking and attribution requirements

Whether this experiment can be judged clearly depends less on the page and more on attribution.

There must be a stable web_attribution_id that passes from MWeb all the way to the app order:

Figure: Cross-channel attribution handoff. The web touchpoint carries an attribution_id through landing, deeplink, app landing, and app order. generated by gpt-image-2.

Key events include:

1
2
3
4
5
6
7

mweb_buy_now_click
app_landing_page_view
deeplink_click
deeplink_success
app_target_page_view
app_login_success
app_order_success

Key parameters include:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

web_attribution_id
user_id
device_id
session_id
product_id
sku_id
campaign_id
entrance
source_page
target_page
btm_chain
experiment_id
variant_id
app_installed_flag
login_state

Without web_attribution_id or a similar attribution field, even if app orders increase, we cannot prove that the increase came from the Web change.

Decision method

After the experiment ends, do not roll out just because one metric increased. Judge by situation.

The ideal result is:

1
2
3
4
5
6

Treatment group
Web-assisted App GMV / eligible UV increases
Total Paid GMV increases
Web Direct GMV is not materially damaged
D7/D30 retention is normal
Refund, complaint, performance, and bounce rate do not materially worsen

最近转进一个新的组，主要是做东南亚 TikTok Shop 营销/用户增长的，由于自己之前没有接触过增长工程师这个专业，所以借着团队内现有项目和 GPT 学了一轮，我了解到了数据在增长工程里几乎是最重要的东西，以下是我的学习增长数据心得，本文纯手写，请放心食用。

做工程和画页面是两件完全不同的事，最后产生的效果也不同。

很多工程师会 follow 产品的想法，在首页加几个模块、在某个交易节点加一个弹窗，但不知道为什么要这么做，最后往往效果不好，也无法验证收益。而一个工程化的解决思路，就像 Debugging 一样，是要弄清楚用户在哪一步流失了，卡住了，导致最后没有转化成成交额，并针对这一步进行优化，同时用实验验证优化是否达到预期。

工程师在增长业务里最重要的价值，是把业务想法变成可验证的因果链路。

拿电商举例子，电商看的核心指标是 GMV（商品交易总额），那么今天老板跟你说，GMV 低于行业平均水平，要提升 xx%。于是你开始忙活起来，拍一拍脑袋，觉得首页没有展示足够的大促信息，你加上了大促信息模块，结果呢？GMV 并没有明显上涨。就算上涨了，同一时间，程序员子涵、李明也在开发新功能，你也很难跟老板解释这个 GMV 的上涨就是你贡献的。

出现这个问题，本质上是因为 GMV 是一个高度抽象的指标，它的上升或下降是很多不同因素共同作用的总和。你不知道是因为商品质量、价格、还是系统性能等各种原因造成的。所以单纯基于这个指标，你做的改动和决策很可能没办法抓到根因。就像 debug 一样，一条很长的链路，你没有打 log，看结果猜根因，效率也很低。

更科学的做法，是把整个增长流程变成一个可验证的系统：用数据驱动决策、验证决策、衡量收益。

Figure: Growth engineering validation loop. A business idea becomes a metric hypothesis, tracking design, experiment, decision, and impact measurement. generated by gpt-image-2.

电商增长数据工程化

下面拿电商增长工程（我负责的项目）举例子，跟大家讲一讲怎么做数据驱动增长工程。

节点拆分

电商交易最终指标是 GMV（商品交易总额），它代表了在该平台上产生交易的规模有多大，它是一个高度抽象的指标。其他的指标都是间接作用于该指标上的。

工程视角拆分

为了方便我们分析，我们可以把指标按流程进行拆解：

要提升 GMV，就要让尽可能多的用户以高客单价到达成交。我们可以把这个流程转化成一系列指标，追踪到达每个关键流程的用户数量，形成一个漏斗图：

Figure: Ecommerce conversion funnel from entry traffic to order success and GMV. generated by gpt-image-2.

我们可以下游用户量除以上游用户量，形成该节点的转化率，UV（最开始的用户量）乘所有转化率就是 GMV：

每一层都有不同含义：

经营视角拆分

我们也可以按用户经营视角拆分，了解是哪类用户成交额在上涨/下滑，应该拉哪个杠杆。

按新老客分（也可以按地区、年龄等）：

判断 GMV 变化到底是谁导致的：

1
2
3
4

老用户活跃下降 -> 留存/召回问题
新客变多但不买 -> 新客承接问题
买家数没变但订单频次下降 -> 复购/活动问题
订单数没变但 AOV 下降 -> 货盘/价格/凑单问题

节点串联

拆分节点可以帮助我们看清哪个环节用户流失量最大，帮助我们排查根因。但只有根因还不够，我们还要确保修复最终能作用到我们的终极目标，也就是 GMV 上，这时候，我们需要从数据上串联整个链路。比方说，一个用户打开了首页，我们要知道他这次打开页面是否点进 PDP，是否下单，最后贡献给 GMV。

没有这个关联性保证，最容易得出错误的决策和结论。比如我们只关注用户从打开首页到商品详情的转化率，那我们最极端可以做一个重定向，用户一进首页，我们就自动跳转到某个商品详情。这样做转化率是 100%，但是最后肯定会严重伤害 GMV。

同样的，某些诱导用户下载 App 的手段，表面上下载量上升了，但是最后对用户下单、成为留存用户没有帮助，ROI 也是非常低。

所以，功能改动一定要以能否传导到 GMV 为准，而不是只看某个环节指标。

几种分析方法

漏斗分析

漏斗回答的是：“用户在哪一步掉了？”

Figure: Web and app growth path. A promo click can continue through web checkout or branch into app activation, order, and retention. generated by gpt-image-2.

漏斗分析主要是为了定位责任：

• CTR 低：前端展示、活动利益点、商品排序、推荐策略。
• PDP 到达低：跳转、性能、商品状态、deeplink、埋点。
• PDP 加购低：价格、库存、优惠、评价、信任。
• Checkout 支付低：登录、地址、支付、运费、风控。
• App 打开后首单低：landing 不一致、权益落差、登录摩擦、商品丢失。

Cohort / 留存分析

Cohort 回答的是：“这批用户后面怎么样？” 你不能只看当天打开 App，因为很多点击只是误触或低质量流量。

1
2
3
4
5
6
7
8

按进入 App 的日期分 cohort：
D0 打开 App
D0 登录
D0 首单
D1 回访
D7 回访
D30 回访
D7/D30 GMV

如果 A/B 方案让 App 打开提升 20%，但 D7 留存和首单没有提升，这不是好增长，是流量搬运。

归因分析

归因回答的是：“这个 GMV 应该算谁的贡献？”

常见模型：

A/B Test

A/B 回答的是：“这个改动是不是导致了结果变化？”

一个合格实验必须有：

• Primary Metric：只能有 1 个，决定胜负。例如 Web-assisted App GMV per UV、订单转化率、buyer rate。
• Secondary Metrics：解释为什么赢或输。例如 CTR、PDP 到达率、加购率。
• Guardrail Metrics：不能伤害的指标。例如 LCP、CLS、bounce、退款率、投诉、其他频道 GMV。
• Stable Randomization：按用户/device 稳定分桶，不要每次刷新变组。
• Enough Sample Size：样本不足不下结论。
• Run Full Cycle：至少覆盖完整业务周期，电商通常不能只跑半天。
• No Peeking：中途看到涨了就停，很容易出错。
• SRM Check：实验组/对照组流量比例异常，结果直接不可信。

一个例子：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

实验：MWeb 点击购买时展示 App 承接页

Control：现有链路
Treatment：点击 Buy Now 后进入 App landing，引导打开 App 并保留商品/权益

Primary：
  7 天内 Web-assisted App paid GMV per exposed UV

Secondary：
  App open rate
  App landing success rate
  App login rate
  App first order rate

Guardrail：
  Web direct GMV
  total paid GMV
  bounce rate
  page performance
  complaint/refund

完整案例：MWeb 点击购买后用 App Landing 承接

前面讲了 GMV 拆解、漏斗、归因、留存和 A/B。如果这些概念分开看，还是有点散。下面用一个真实项目场景把它们串起来，看看数据驱动怎么做决策。

想法：Toko Mobile Web 用户点击购买后，是否应该通过一个 App Landing Page 引导用户进 App 完成购买？

1. 业务问题

通过观察数据，现在 Toko 有一个很典型的结构性问题：

• Toko Web：DAU 高且稳定，但 GMV 很低
• Toko App：GMV 高，但 DAU 在下降

这说明 Web 可能承载了大量商品浏览、SEO、搜索、活动入口流量，但没有很好地把这些流量转化成交易。尤其是 Mobile Web，用户在手机上浏览商品时，天然更适合被引导到 App 内完成交易，因为 App 里通常有更完整的登录态、支付体验、权益承接和复购能力。

但现在的问题是：Web 上虽然有引导进 App 的 banner，但它缺少明确的场景和用户动机。用户只是看到“打开 App”并不会觉得自己应该点。相比之下，更合理的场景是：当用户已经点击 Buy Now 或表现出购买意图时，再用一个 landing page 承接，告诉用户“进 App 可以保留当前商品、领取当前权益、继续完成购买”。

所以这个项目要验证：

高意图 MWeb 用户能否通过更好的 App 承接转化成 App 内首单、留存和长期 GMV

业务假设

这个需求背后的假设可以拆成三层：

假设 1：MWeb 上有一批高购买意图用户

证据：他们点击了 Buy Now / Add to Cart / 领取券 / 查看 Flash Sale 商品

假设 2：当前 Web -> App 引导效率低

原因：banner 泛化、动机弱、deeplink 承接不完整、商品/券/价格可能丢失

假设 3：如果在购买意图发生后，用 App Landing 承接

结果：App 打开率、登录率、首单率、后续留存和 GMV 会提升

指标链路

这个项目的完整指标链路应该是：

下面这张图把 MWeb eligible UV 到 D30 GMV / LTV 拆成 Web intent 和 App value 两段。

漏斗图：

Figure: MWeb-to-App measurement funnel for the app landing experiment. Value is measured per eligible MWeb UV rather than total GMV alone. generated by gpt-image-2.

每一层回答的问题不同：

实验设计

可以设计一个 A/B 实验：

1
2
3
4
5
6
7
8
9
10

实验对象：
Mobile Web 上点击 Buy Now / Add to Cart / 高意图 CTA 的用户

Control：
维持现有链路，例如继续走 Web 购买链路或现有 App banner 逻辑

Treatment：
点击 Buy Now 后进入 App Landing Page
Landing Page 展示当前商品、价格、活动权益、App 打开按钮
点击后通过 deeplink 打开 App，并保留商品、券、活动和 attribution_id

实验结构：

Figure: A/B experiment split for the app landing page. Stable buckets keep control and treatment comparable while metrics and guardrails decide rollout. generated by gpt-image-2.

指标设计

Primary Metric 建议用：

为什么是这个？

因为它同时约束了三个东西：

Secondary Metrics：

1
2
3
4
5
6
7
8

Landing Page View Rate
Deep Link Click Rate
Deep Link Success Rate
App Landing Success Rate
Login Rate
First Order Rate
D7 Retention
D7 Repeat Order Rate

Guardrail Metrics：

1
2
3
4
5
6

Web Direct GMV
Total（Web+App) Paid GMV
Bounce Rate
Page Performance
Refund / Cancel / Complaint
SEO Traffic

埋点和归因要求

这个实验能不能判断清楚，关键不在页面，而在归因链路。

必须有一个稳定的 web_attribution_id，从 MWeb 一直传到 App 订单：

Figure: Cross-channel attribution handoff. The web touchpoint carries an attribution_id through landing, deeplink, app landing, and app order. generated by gpt-image-2.

关键事件包括：

1
2
3
4
5
6
7

mweb_buy_now_click
app_landing_page_view
deeplink_click
deeplink_success
app_target_page_view
app_login_success
app_order_success

关键参数包括：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

web_attribution_id
user_id
device_id
session_id
product_id
sku_id
campaign_id
entrance
source_page
target_page
btm_chain
experiment_id
variant_id
app_installed_flag
login_state

如果没有 web_attribution_id 或类似归因字段，即使 App 订单涨了，我们也没法证明是 Web 改造带来的。

决策方式

实验结束后，不是看某一个指标涨了就上线，而是分情况判断。

最终比较理想的结果是：

1
2
3
4
5
6

Treatment 组
Web-assisted App GMV / eligible UV 提升
Total Paid GMV 提升
Web Direct GMV 没有明显受伤
D7/D30 留存正常
退款、投诉、性能、跳出率无明显恶化

最小实现很简单。前端每隔几秒发一个心跳包，服务端保存 last_seen_at。如果超过一段时间没有收到心跳，就把这个人标成离线。

这个版本能点亮一个在线灯。要支撑派单、排班、监控和工时统计，它还差很多上下文。运营是否在线，需要同时看他是否有任务、属于哪个技能组、当前状态能不能接单、是否在灰度规则里、异常状态是否已经通知过、下游系统是否已经拿到最新状态。

我维护的 HCM/Heartbeat 系统服务 47,149 个 agent、7,000 个 skill group、约 4,000 QPS 和 141 个上游调用方。这篇文章从一个最朴素的实现开始，一层一层把系统推出来。

先看最终目标：

图 0：最终目标是一套状态平台。图片由 gpt-image-2 生成。用户动作先经过统一工作台 SDK、WS-API 和 Frontier 长链，再进入 Heartbeat、MQ、Compute、HCM 和下游状态传播链路。

下面从 last_seen_at 开始，一步步把这张图搭出来。

每加一层，都先问一个问题。

问题 1：怎么知道运营有没有活跃？

第一版可以直接写状态。

图 1：最朴素的心跳实现。图片由 gpt-image-2 生成。它可以记录最近心跳时间，但缺少任务数、技能组、业务规则和 region 信息。

这个方案的数据结构大概是：

1
2
3

agent_id
last_seen_at
status

前端每隔 N 秒发心跳。服务端更新 last_seen_at。定时任务扫描超时的人，把状态改成 OFFLINE。

这个模型适合很轻的场景，比如显示一个在线灯。先不讨论任务数、技能组和规则，单看 UI 到 Heartbeat 的接入层，就已经有一组工程问题：

• 一个工作台页面里有多个业务模块，不能每个模块都自己建一条通道。
• 用户动作是突发的，连续输入不能变成同步写入风暴。
• 浏览器长链会断，重连期间的消息要能补回来。
• 长链推送和短链补偿可能送到同一条逻辑消息。
• Heartbeat 服务重启后，输入流不能丢。
• 状态计算要能独立发布，不应该和浏览器 SDK 绑死。

last_seen_at 只能提供一个时间戳。下一步要解决的是：怎么把浏览器动作变成可靠的事件流。

所以第一版适合作为思维起点，生产系统还要继续往前推。

问题 2：UI 到 Heartbeat 的事件怎么可靠进入系统？

下一步先补 UI 到 Heartbeat 的接入层，再把事件放进 Raw MQ。

图 2：统一工作台先用 SDK、WS-API 和 Frontier 承接浏览器里的动作事件，再交给 Heartbeat 和 Raw MQ。图片由 gpt-image-2 生成。状态计算留到下一层。

前端上报的内容不只有定时心跳，还有点击、输入、鼠标、URL 变化、打开工单、回复工单、完成工单、转交工单等动作。

这些动作的含义不同，但它们有一个共同点：都只能说明“这个 agent 在某个时间点发生过动作”。

在统一工作台里，UI 到 Heartbeat 之间还有一层长链通道：

1
2
3
4
5
6

Agent UI
-> Workbench SDK
-> WS-API
-> Frontier
-> Heartbeat Service
-> Raw MQ

Workbench SDK 在一个 window 内维护一个长链接。业务模块按 module + entity 注册，deviceID 标识这条通道。module + entity 的粒度要控制好，粒度太细，切换 entity 时容易出现短暂未建联，推送就会掉在窗口外。

WS-API 负责初始化长链、注册/注销 module，也负责短链补偿。Frontier 负责长链通道。Frontier 异常时，端上可以通过短链轮询补未 ack 的消息，默认间隔是 30 秒，可以用配置调整。

重复消息也在这一层先处理一遍。业务后端传 ReqID，SDK 保存最近一批 ReqID 做去重；RPC 推送服务生成 msgID，长链推送和短链补偿都按 msgID 做兜底去重。业务需要顺序时，可以带 Index，排序语义由业务自己维护。

每个节点都对应一个明确的问题：

Raw MQ 里的事件是输入数据。

如果消费端拿到一个 keyup 事件就直接把人标在线，拿不到事件就标异常，系统还是会误判。因为事件缺上下文：它不知道当前状态、任务数、技能组、业务线规则，也不知道这条异常是否已经处理过。

接下来才轮到业务上下文问题：这个 agent 能不能接单，规则覆盖哪些 skill group，异常是否已经处理过，哪个 region 可以发状态变更。这些不是接入层的问题，要交给 Compute 和 HCM。

这就需要下一层。

问题 3：怎么把事件变成状态判断？

我们加一个 Compute 层。

图 3：Compute 消费原始事件，回查 HCM 拿当前事实，再生成异常候选。图片由 gpt-image-2 生成。

Compute 层只做一件事：把“事件”转成“候选状态变更”。

常见的事件类型可以分成几组：

Compute 收到原始事件后，不直接写状态。它先问 HCM：

1

GetWorkStatus(agent_id, tenant_id, channel)

HCM 返回当前工作状态、任务数、技能组关系等信息。Compute 再结合规则判断这个 agent 是否进入异常候选。

这一步的核心是把判断拆开：

1

事件 + 当前事实 + 规则 = 候选

候选只是待确认结果。比如一个 agent 在 10:00 后没有动作，Compute 在 10:10 认为他可能要转异常。但 10:10:01 他可能刚接到新任务。最终能不能改状态，还要由 HCM 再读一次事实。

Compute 层的边界要硬。它负责算候选，不负责提交最终状态。

版本 2 引入这些东西，各自有明确边界：

问题 4：怎么计算“多久没有动作”？

很多心跳规则都带时间窗口：

最直觉的实现是给每个 agent 建一个 timer。

这个方案很快会变麻烦。进程重启时 timer 丢失，扩缩容时 timer 分散到不同实例，跨 region 切流时还要处理 timer 归属。对几万个 agent 来说，进程内 timer 会把复杂度绑到实例生命周期上。

我们用 Redis zset 表达时间窗口。

图 4：候选 agent 按事件时间写入 Redis zset。图片由 gpt-image-2 生成。Cron 扫描到期数据，再把异常消息送回状态更新流程。

Redis 里维护几类队列：

1
2
3
4
5
6
7

agent_no_action_for_8_min
agent_no_action_for_10_min
queue_agent_no_action_for_15_min
agent_in_abnormal_status_more_then_10_min
queue_agent_in_abnormal_status_more_then_15_min
agent_not_on_app_for_5_min
queue_agent_not_on_app

score 存事件时间戳。到期扫描时，用 zrangebyscore 取出 now - threshold 之前的 agent。

这个结构有几个好处：

Cron 每 2 秒扫一次队列。扫描之前先检查当前 IDC 是否允许发送，再抢一个短 TTL 的全局锁，避免多个实例重复发同一批异常消息。

这里有个容易踩坑的点：业务窗口和防重锁时间要分开设置。

比如 10 分钟无操作触发提醒，业务窗口是 600 秒，但 agent 维度的防重锁可以设成 840 秒。这样同一阶段可以减少刷屏，后续 ABNORMAL -> OFFLINE 也还有计算机会。

版本 3 引入的是时间窗口计算层：

问题 5：谁可以写最终状态？

Compute 算出来的是候选。最终状态只能由 HCM 写入。

原因很简单：HCM 才拥有 agent、skill group、任务数、当前状态和状态变更日志。状态写入必须集中，否则下游系统会看到多个版本的事实。

座席状态本身是一组枚举：

1
2
3
4
5
6
7
8
9
10
11

ONLINE(1000)
TRAINING(1001)
BREAK(1002)
MEETING(1003)
ABNORMAL(1004)
IDLE(1005)
LUNCH(1006)
OFFLINE(1007)
OTHER(1008)
BUSY(1009)
STANDBY(1010)

HCM 消费异常候选后，会重新读取当前状态，然后做二次判断：

1. 当前状态是否还匹配规则。
2. 当前任务数是否允许切换。
3. 当前 agent 是否属于规则覆盖的 skill group。
4. 当前 region 是否允许发送状态变更。
5. 这次异常是否已经通知或处理过。

规则配置可以长这样：

1
2
3
4
5
6
7
8
9
10
11
12
13

{
  "access_party_ids": [2, 3, 9, 45, 46],
  "status": [1000, 1005, 1009],
  "no_heart_time_limit": 600,
  "status_to": 1004,
  "status_change_note": "No action for 10min, automatically changes to abnormal.",
  "notifies": [
    {
      "type": "Lark",
      "title": "abnormal hint"
    }
  ]
}

这段配置同时影响状态变更和通知。Compute 的 TCC、Redis 队列和 HCM 规则共同约束最终行为。

这套设计里，HCM 的职责很清楚：

• 拒绝过期候选。
• 根据当前事实判断能不能改状态。
• 写 status table。
• 写 unified_work_status_log。
• 把状态变化交给下游传播链路。

版本 4 的第一部分是建立状态权威：

问题 6：下游怎么拿到一致的状态？

状态写进 HCM DB 后，还要继续传播给下游。

WFM、RouteQueue、数据侧和其他业务通常通过状态变更消息更新自己的视图。

最终链路长这样：

图 5：最终链路保留了前面几个版本的组件，但职责被拆开：事件接收、候选计算、时间窗口、状态写入、状态传播。图片由 gpt-image-2 生成。

状态传播走的是：

1
2
3
4
5

HCM UpdateWorkStatus
-> status_table
-> DBus / binlog
-> status change MQ
-> WFM / Routing / Analytics

这个设计让所有下游都从 DB 变更事实出发。代价是链路变长了，任何 binlog handler、事件分发、MQ 堆积或下游消费延迟，都会让用户看到旧状态。

我们踩过这个坑。

一次 US-TTP 事故里，WFM 的 Omni-channel view 没有反映实时状态。排查时 HCM DB 里的 agent 已经 offline，HCM 到 WFM 的消息看起来也发成功了。

最后问题出在更前面：binlog 到 HCM MQ 链路堆了约 300k 条消息。work status handler 没有变慢，另一个依赖 ES 的 handler 超时了。多个 handler 共用一条消费链路，慢 handler 把其他状态传播一起拖住了。

这个事故之后，我们补了 handler 维度耗时指标，也把“消费 binlog 后直接发 RMQ，减少中间事件分发阻塞”列进改造项。

这类问题在设计阶段就应该被问出来：哪个 handler 可以拖住主链路？哪个消费组有共享故障面？哪个指标能说明下游真的收到状态？

版本 4 的第二部分是状态传播边界：

问题 7：跨 region 切流时，事件怎么不断？

前面的链路在单 region 内已经能工作。新的问题来自容灾和切流。

Heartbeat 事件是状态计算的输入。切流时，如果流量调度先把请求打到目标 region，但目标 region 的 Raw MQ 里没有前一段时间的事件，Compute 会缺少时间窗口里的上下文。结果可能是 agent 刚切过去就被判断成无操作，或者异常计算延迟一段时间才恢复。

所以跨 region 的第一层改造是 MQ mirror。

图 6：Raw MQ 在 region 之间做 mirror。图片由 gpt-image-2 生成。目标 region 在接管流量前，已经有一份可消费的事件流。

这层解决的是事件连续性。

1

Region A Raw MQ <-> Region B Raw MQ

切流前，两个 region 都能拿到最近的心跳事件。真正调度流量时，目标 region 不需要从空队列开始计算。它已经能看到 agent 最近的点击、输入、心跳、工单动作和状态事件。

切流 SOP 也围绕这条链路设计：

1. 先看当前 region 的 Heartbeat、HCM、Raw MQ、binlog MQ、status MQ 指标。
2. 用条件化调度切一小段流量，比如 2%。
3. 看 from_dc 维度，确认目标 region 收到预期流量。
4. 看 mirror 后的 Raw MQ lag、错误率和消费速率。
5. 异常时删除调度配置，把流量切回。

MQ mirror 让事件更难丢。新的问题也随之出现：两个 region 都可能看到同一条事件。

事件重复本身还好，真正危险的是重复触发状态变更。比如同一条 no action 候选在两个 region 各发一次，HCM 可能收到两条异常消息，下游也可能收到两次状态变化。

所以下一层要处理重复。

版本 5 引入的是跨 region 事件连续性：

问题 8：mirror 之后，重复事件怎么处理？

重复事件要分两段处理。

第一段在发送侧：只有一个 region 能发送异常消息。

第二段在状态侧：HCM 写状态前再读一次当前事实，做幂等判断。

图 7：mirror 保证事件连续，active region 和 dedupe lock 控制状态变更只提交一次。图片由 gpt-image-2 生成。

Compute 可以在两个 region 都消费事件，但 Cron 扫描到期队列时会先检查 active_idc：

1
2

active_idc = true  -> can emit exception message
active_idc = false -> calculate only, no emit

发送前再加一层 dedupe lock。锁的粒度不能太粗，否则不同规则会互相挡住；也不能太细，否则同一阶段的重复消息挡不住。比较合适的是把 agent、规则和时间窗口放进同一个去重键：

1

agent_id + rule_id + window_start

Redis zset 也会帮忙收敛一部分重复候选。相同 member 重复写入时，后写入会更新 score。Cron 真正发消息时，还要按 agent/rule/window 抢锁。

HCM 是最后一道保护。

收到异常消息后，HCM 重新读取当前状态、任务数、skill group 归属和规则配置。只有当前事实还满足规则，才写 status_table 和 unified_work_status_log。如果 agent 已经有任务，或者状态已经被别的路径改走，这条候选会被丢掉。

跨 region 之后，状态链路多了三类开关：

这三个开关要分开：请求入口、事件复制、状态发送权可以按不同节奏切换。这样切流、回滚和演练才有操作空间。

版本 6 把 mirror 后的事件变成安全状态变更：

问题 9：TT 和 IES 拆分时，怎么不打断老链路？

HCM/Heartbeat 后来同时服务 IES 和 TT 侧业务。两边共用 HCM、Heartbeat、DB、MQ 和 routing 更新链路时，发布、容量和容灾都会绑在一起。

TT 内容侧要拆出去，最简单的做法是新建 TT HCM、TT Heartbeat、TT DB 和 TT MQ，然后让上游一次性切走。

这个方案风险很高。HCM 连着登录、状态更新、skill group、任务数、路由、WFM 和 binlog 传播。一次性切干净，任何一个字段、过滤条件或 MQ 消费组出错，都会影响接单和状态同步。

所以第一阶段先做共存。

图 8：TT upstream 先迁到 TT HCM，写入事实仍由 IES HCM 承担。图片由 gpt-image-2 生成。dsyncer 和转发逻辑保留回滚空间。

第一阶段的链路是：

1
2
3
4
5
6

TT upstream
-> TT HCM
-> forward RPC to IES HCM
-> IES DB
-> dsyncer
-> TT DB

这一步的目标很明确：先把调用入口迁走，写入事实先留在旧链路。

为了让旧 HCM 知道哪些数据要转给 TT，需要加 ownership filter。判断路径是：

1
2
3
4
5

agent_id
-> agent_skill_group_rel
-> skill_group
-> access_party
-> TT or IES

判断结果写 Redis cache。cache miss 时查 DB，拿到 access party 后再写回。工作状态、agent-skillgroup 关系和 routing 更新都可以按这个结果过滤。

这层改造带来的好处是可回滚。TT HCM 出问题，可以继续走 IES 原链路。filter 出问题，也可以通过配置关掉转发，先保住旧路径。

版本 7 是迁移桥：

问题 10：什么时候可以真的拆开？

共存阶段跑稳之后，才进入第二阶段。

第二阶段要把写入、事件、MQ 和 routing 更新都拆开：

图 9：拆分完成后，TT 和 IES 各自拥有 upstream、HCM、DB、MQ 和 routing 更新链路。图片由 gpt-image-2 生成。

拆分动作包括：

1. TT HCM 停止把 RPC 转发到 IES HCM。
2. IES 到 TT 的 dsyncer 停止。
3. IES heartbeat/MQ 停止消费 TT agent 消息。
4. TT routing 只消费 TT 侧状态更新。
5. IES routing 只消费 IES 侧状态更新。

拆完后，旧 HCM 侧减少约 3,000 个 agent 和约 2,000 QPS。

这里要验证两件事：服务已经拆出来，两边的状态事实也各自闭合。

• TT agent 的心跳只进入 TT 侧。
• TT agent 的状态只由 TT HCM 提交。
• TT 状态变化只进 TT routing/WFM/data 链路。
• IES 侧保留原有路径，不被 TT 发布和切流影响。

这时 TT/IES 拆分才算完成。

版本 8 是把迁移桥拆掉：

问题 11：前端动作没进来，系统怎么发现？

状态计算依赖输入事件。后端 RPC 全绿，只能说明后端还活着，说明不了用户动作真的进了 Heartbeat。

GCP -> NO1A 切流后，出现过 agent 频繁 abnormal。人还在工作台操作，系统却判断他长时间没动作。最后查到的问题在工作台长链路配置：keyup / keydown 事件没有稳定进入 Heartbeat。

HCM 错误率覆盖不到这段输入链路。状态计算前面要加一层输入健康检查。

图 10：状态计算之前先确认用户动作事件稳定进入系统。图片由 gpt-image-2 生成。Workbench SDK、WS-API、Frontier、Heartbeat 接收、Raw MQ 都要上报输入健康指标。

输入健康可以按几个维度看：

工程上更稳的做法是把输入健康接进规则层。

如果某个 region 或工作台版本的输入健康已经异常，自动 abnormal 规则应该降级：延长窗口、暂停自动切异常，或者只发提醒。等输入链路恢复后，再回到正常计算。

这样做会牺牲一部分自动化程度，但能避免“因为采集链路断了，把正在工作的人切成异常”。

版本 9 补的是输入可观测性：

问题 12：HCM 已经写状态，下游还是旧状态怎么办？

状态提交成功以后，下游还要拿到同一个事实。这里出过一个很典型的事故：US-TTP ES down 时，WFM 的 Omni-channel view 没有反映实时状态。

排查时，HCM DB 里的 agent 已经 offline，HCM 到 WFM 的消息看起来也发成功了。最后发现 binlog 到 HCM MQ 堆了约 300k 条消息。work status handler 本身没慢，另一个依赖 ES 的 handler 超时了。多个 handler 共用一条消费链路，慢 handler 把工作状态传播拖住了。

这个问题把 V4 的状态传播继续往前推一层：状态变更要有独立车道。

图 11：binlog 下游按业务语义拆成独立 handler。图片由 gpt-image-2 生成。Search handler 变慢时，work status handler 仍然可以把状态送到 WFM 和 routing。

更好的目标形态是把状态变化当成一等事件处理：

1
2
3
4
5
6

HCM transaction
-> status_table
-> status_change_outbox
-> work_status_dispatcher
-> status MQ
-> WFM / Routing

status_change_outbox 和状态写入放在同一个事务里。dispatcher 只负责把 outbox 里的状态事件送到状态 MQ。

Search、analytics、audit 这些消费者可以继续消费 binlog 或订阅自己的事件流，但它们不应该和 work status propagation 共用同一个阻塞点。慢 handler 要进入自己的 retry / DLQ，不能拖住 WFM 和 routing。

这层还要补端到端指标：

状态系统最怕“主库已经对了，下游还错着”。只有把提交、出站、消费分开看，才能知道慢在哪一段。

版本 10 隔离的是状态传播链路：

问题 13：热读接口把 HCM 打爆怎么办？

GetWorkStatus 是这条链路里的热接口。Compute 要查它，上游也要查它。一次 OOM 事故里，downstream error rate spike 和 GetWorkStatus 流量峰值对齐。

这说明状态系统还需要容量防线。

图 12：GetWorkStatus 前面加 quota，内部按调用方隔离线程池，后面用短 TTL cache 和自动扩容信号保护 DB。图片由 gpt-image-2 生成。

这里的目标是把不同调用方隔开。单个上游出问题时，故障范围停在自己的预算里。

GetWorkStatus 至少需要这几层保护：

Heartbeat Compute 侧也要配合。比如同一个 agent 在短时间内连续上报动作，不需要每条事件都 RPC 查一次当前状态。可以按 agent 做小窗口合并，或者读取本地/Redis 状态快照，再把最终提交交给 HCM recheck。

这层改造把 GetWorkStatus 从“谁来都查 DB”变成“有预算、有隔离、有降级的状态读服务”。

版本 11 把热读接口变成受保护的服务：

问题 14：查询改动怎么安全上线？

HCM 4.3 的 DB query rollback 暴露的是发布防线。

canary 阶段看起来正常，ROW 后出现新错误日志。原因是两个 LEFT JOIN 放大了 count SQL 的数据量，最后打到 DB timeout。

“先 canary 再全量”覆盖不了这类问题。canary 流量小，数据分布也可能偏，复杂查询要单独过成本检查。

图 13：查询改动先经过 feature flag、canary、shadow query、EXPLAIN 和慢 SQL 监控。图片由 gpt-image-2 生成。异常时切回 old query。

查询发布可以按这条路径走：

1
2
3
4
5
6

query change
-> feature flag
-> canary
-> shadow query
-> row traffic
-> auto rollback

shadow query 不参与线上返回，只比较结果和成本。发布系统要看几类信号：

对查询路径来说，feature flag 不能只开关整个功能。更实用的是按 filter 字段选择 old query 或 join query：

1
2

simple filter -> old query
join filter   -> join query

这样某个复杂 filter 出问题时，可以只回退这条查询路径，不影响其他查询。

版本 12 给数据访问加发布门禁：

后续还要解决什么？

上面的 V9 到 V12 把系统推成一个状态平台。输入链路、计算链路、提交链路、传播链路、容量链路、发布链路都要能被单独观测和单独降级。

后面还可以继续扩：

最终设计

回到最开始的问题：怎么知道一个运营是否在线？

回到开头那张最终目标图，可以按两条线看：数据面和控制面。

数据面负责把用户动作变成状态事实：

1
2
3
4
5
6
7
8
9
10
11
12
13

Agent UI
-> Workbench SDK
-> WS-API
-> Frontier
-> Heartbeat Service
-> Raw MQ
-> Compute
-> Redis zset
-> Exception MQ
-> HCM
-> DB + Outbox
-> Status MQ
-> WFM / Routing / Analytics

控制面负责让这条链路在异常时还能收住：

• 前端负责上报动作和心跳。
• Workbench SDK 负责长链单例、断链重连、短链补偿和端上去重。
• 输入健康确认动作事件真的进入 Heartbeat。
• MQ mirror 保留跨 region 事件连续性。
• Redis zset 和 dedupe lock 控制候选重复。
• active_idc 控制哪个 region 能发送异常消息。
• HCM recheck 收口最终状态提交。
• Outbox 和 handler 隔离保护工作状态传播。
• quota、cache、bulkhead 和 circuit breaker 保护热读接口。
• query gate 控制复杂查询发布。
• TT/IES ownership 控制 agent、skill group、MQ 和 routing 更新归属。

这个设计看起来比第一版复杂很多。每一层都是被上一个问题推出来的：

最开始的问题只是“这个人还在不在线”。最后系统回答的是一组更具体的问题：这个 agent 最近有没有动作，输入事件链路是否健康，当前有没有任务，属于哪个 skill group，规则是否覆盖他，哪个 region 可以发异常，状态能不能提交，下游是否拿到了同一个事实，热读接口是否在预算内，查询发布是否能回滚，TT 和 IES 的 ownership 是否已经分开。

The smallest version is almost boring. The frontend sends a heartbeat every few seconds. The backend stores last_seen_at. If the timestamp is too old, the agent becomes offline.

That version can light up a green dot. It is nowhere near enough for dispatching, workforce management, routing, monitoring, and time tracking. In the system I worked on, the HCM and Heartbeat path served 47,149 agents, 7,000 skill groups, about 4,000 QPS, and 141 upstream services. At that scale, “online” becomes a state platform.

Here is the target shape first.

Figure 0: The final target is a state platform. User activity enters through the Workbench SDK, WS-API, and Frontier long-link path, then flows through Heartbeat, MQ, Compute, HCM, and downstream state propagation. Generated by gpt-image-2.

The rest of this article builds the system from last_seen_at to that final design, one engineering problem at a time.

Can we tell whether an agent is active?

Version 0 writes state directly.

Figure 1: The naive heartbeat design records the latest heartbeat timestamp. It has no task count, skill group context, business rules, or region ownership. Generated by gpt-image-2.

The data model is tiny:

1
2
3

agent_id
last_seen_at
status

The frontend reports heartbeat events. The backend updates last_seen_at. A scheduled job scans expired records and marks agents as OFFLINE.

This model works for a lightweight presence indicator. Before we even talk about task count or skill groups, the intake path already has problems:

• A workbench page has multiple modules. They should not each open their own channel.
• User activity is bursty. A typing burst should not become a synchronous write storm.
• Browser connections drop. A reconnect should not make the system lose recent activity.
• Long-link push and short-poll compensation can deliver the same logical message twice.
• Heartbeat service restarts should not erase the input stream.
• State calculation should be deployable without changing the browser SDK.

last_seen_at gives us a timestamp. The next problem is to turn browser activity into a reliable event stream.

How do UI events enter Heartbeat reliably?

The next layer moves activity intake out of direct writes and into an event stream.

Figure 2: Workbench activity enters through SDK, WS-API, and Frontier before Heartbeat writes the raw event stream to MQ. State calculation comes later. Generated by gpt-image-2.

The frontend does more than send a periodic heartbeat. It reports clicks, key events, mouse activity, URL changes, ticket open, ticket reply, ticket completion, handoff, escalation, and other workbench actions.

These events mean different things, but they share one property: they prove an agent did something at a specific time.

Between the agent UI and Heartbeat, the workbench uses a long-link path:

1
2
3
4
5
6

Agent UI
-> Workbench SDK
-> WS-API
-> Frontier
-> Heartbeat Service
-> Raw MQ

The Workbench SDK keeps one long-link connection per window. Business modules register by module + entity, and deviceID identifies the channel. That granularity matters. If entity is too fine, a user switching entities can hit a short unregistered window and lose a push.

WS-API handles long-link initialization, module registration, deregistration, and short-link compensation. Frontier owns the long-link channel. When Frontier has problems, the client can use short polling to fetch messages that were not acked. The default interval is 30 seconds and can be lowered by configuration during an incident.

We also dedupe early. Business messages carry ReqID, and the SDK keeps a recent ReqID cache. The RPC push service generates msgID, so long-link delivery and short-poll compensation can dedupe the same logical message. If a business flow needs ordering, it carries an Index; ordering semantics stay with the business.

Each node earns its place:

The queue contains input, not truth.

If a consumer turns every keyup into ONLINE, and every missing event into ABNORMAL, the system will still misclassify people. Events lack task count, skill group membership, work status, rule scope, and notification history.

That leads to the next set of questions, which are about business context: can this agent receive a ticket, which skill groups does the rule cover, has the abnormal status already been handled, and which region can emit a state change? Compute and HCM own those decisions.

That leads to the compute layer.

How do events become state candidates?

Compute turns raw activity into candidate state changes.

Figure 3: Compute consumes raw events, reads current facts from HCM, and produces candidate status changes. Generated by gpt-image-2.

Compute has one job: convert “an event happened” into “this agent may need a state change.”

Event types fall into a few groups:

Compute does not write the final status. It first asks HCM for current facts:

1

GetWorkStatus(agent_id, tenant_id, channel)

HCM returns current work status, task count, skill group membership, and related facts. Compute combines those facts with the rule configuration:

1

event + current facts + rule = candidate

A candidate is still only a candidate. An agent may have no action after 10:00, so Compute may produce an abnormal candidate at 10:10. At 10:10:01, the agent may receive a new ticket. HCM must re-read the facts before committing the state.

Keep that boundary hard: Compute calculates candidates. HCM commits facts.

Version 2 adds these pieces for specific reasons:

How do we calculate “no action for N minutes”?

Many heartbeat rules are time-window rules:

The obvious implementation is one timer per agent.

That becomes painful quickly. Timers disappear on process restart. Autoscaling spreads timers across instances. Region cutover now has to move timer ownership. For tens of thousands of agents, in-process timers couple business state to instance lifetime.

We used Redis zset to represent time windows.

Figure 4: Candidate agents are written into Redis zsets by event timestamp. Cron scans due records and sends exception messages back into the state update path. Generated by gpt-image-2.

Redis keeps several queues:

1
2
3
4
5
6
7

agent_no_action_for_8_min
agent_no_action_for_10_min
queue_agent_no_action_for_15_min
agent_in_abnormal_status_more_then_10_min
queue_agent_in_abnormal_status_more_then_15_min
agent_not_on_app_for_5_min
queue_agent_not_on_app

The zset score is the timestamp. A due scan uses zrangebyscore for everything older than now - threshold.

This data structure fits the problem:

Cron scans every 2 seconds. Before emitting messages, it checks whether the current IDC can send and grabs a short TTL global lock to avoid duplicate emission from multiple instances.

One detail matters more than it looks: the business window and the dedupe lock window should be different.

For a 10-minute no-action reminder, the business threshold is 600 seconds. The agent-level dedupe lock can be 840 seconds. That reduces reminder spam while still leaving room for later transitions such as ABNORMAL -> OFFLINE.

Version 3 adds these pieces:

Who writes the final status?

Compute emits candidates. HCM writes the final status.

HCM owns the agent, skill group, task count, current status, and state-change log. If several services can write status independently, downstream systems will see competing facts.

The status enum looks like this:

1
2
3
4
5
6
7
8
9
10
11

ONLINE(1000)
TRAINING(1001)
BREAK(1002)
MEETING(1003)
ABNORMAL(1004)
IDLE(1005)
LUNCH(1006)
OFFLINE(1007)
OTHER(1008)
BUSY(1009)
STANDBY(1010)

When HCM consumes an exception candidate, it reads the current facts again and checks:

1. Does the current status still match the rule?
2. Does the current task count allow this transition?
3. Is the agent in a covered skill group?
4. Is this region allowed to send state changes?
5. Has this exception already been notified or handled?

A rule can look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13

{
  "access_party_ids": [2, 3, 9, 45, 46],
  "status": [1000, 1005, 1009],
  "no_heart_time_limit": 600,
  "status_to": 1004,
  "status_change_note": "No action for 10min, automatically changes to abnormal.",
  "notifies": [
    {
      "type": "Lark",
      "title": "abnormal hint"
    }
  ]
}

The rule affects both the state transition and the notification. Compute configuration, Redis queues, and HCM rules all constrain the final behavior.

HCM’s responsibilities stay small and strict:

• Reject stale candidates.
• Recheck current facts.
• Write status_table.
• Write unified_work_status_log.
• Hand the state change to the downstream propagation path.

Version 4 introduces a state authority:

How do downstream systems get the same state?

After HCM commits a status, WFM, routing, analytics, and other consumers still need the same fact.

Figure 5: The single-region pipeline splits intake, candidate calculation, time windows, state commit, and downstream propagation. Generated by gpt-image-2.

The propagation path is:

1
2
3
4
5

HCM UpdateWorkStatus
-> status_table
-> DBus / binlog
-> status change MQ
-> WFM / Routing / Analytics

This gives consumers a DB-backed fact stream. The cost is a longer path. A slow binlog handler, event dispatcher, MQ backlog, or downstream consumer can make users see stale state.

We hit that failure mode.

During one US-TTP incident, WFM’s omni-channel view did not show the current status. HCM DB already had the agent offline, and the HCM-to-WFM send path looked successful.

The real delay was earlier. The binlog-to-HCM-MQ path had about 300k messages queued. The work status handler was not the slow part. Another handler that depended on ES was timing out, and multiple handlers shared one consumption path. The slow handler dragged status propagation with it.

The fix direction was obvious after the incident: add handler-level latency metrics, and move work status propagation closer to a direct binlog-to-RMQ path so unrelated event distribution cannot block it.

The design question is simple: which handler can stall the state path, which consumer group shares a failure domain, and which metric proves the downstream view received the state?

The propagation part of Version 4 adds a second set of boundaries:

How do events survive a region cutover?

The single-region pipeline works until disaster recovery and traffic cutover enter the picture.

Heartbeat events are the input to state calculation. If traffic moves to a target region before that region has recent Raw MQ events, Compute loses the time-window context. Agents can be marked no-action right after cutover, or abnormal calculation can pause until enough fresh context accumulates.

The first cross-region change is MQ mirror.

Figure 6: Raw MQ is mirrored between regions so the target region already has a consumable activity stream before it receives traffic. Generated by gpt-image-2.

Mirror protects event continuity:

1

Region A Raw MQ <-> Region B Raw MQ

Before cutover, both regions can see recent heartbeat events. When traffic actually moves, the target region is not starting from an empty queue. It already sees recent clicks, key events, heartbeats, ticket actions, and status events.

The cutover runbook then follows the data path:

1. Check Heartbeat, HCM, Raw MQ, binlog MQ, and status MQ in the current region.
2. Move a small traffic slice, for example 2%.
3. Check from_dc dimensions to verify the target region receives expected traffic.
4. Check mirrored Raw MQ lag, error rate, and consumption rate.
5. If anything looks wrong, remove the routing config and cut traffic back.

MQ mirror reduces event loss. It also creates a new problem: two regions can see the same event.

Duplicate events are tolerable. Duplicate state changes are dangerous. The same no-action candidate emitted in two regions can send two exception messages to HCM, and downstream systems may receive repeated state changes.

That takes us to dedupe.

Version 5 adds event continuity across regions:

How do we handle duplicates after mirror?

Duplicate control has two stages.

The sender side ensures only one region emits exception messages.

The state side makes HCM re-read current facts and apply idempotent checks before writing.

Figure 7: Mirror keeps events continuous, while active-region ownership and dedupe locks keep state changes single-writer. Generated by gpt-image-2.

Compute can consume mirrored events in both regions. Cron checks active_idc before emitting due candidates:

1
2

active_idc = true  -> can emit exception message
active_idc = false -> calculate only, no emit

Before sending, Cron also takes a dedupe lock. The lock must match the state transition boundary. Too broad, and unrelated rules block each other. Too narrow, and repeated messages for the same stage leak through.

A practical lock key is:

1

agent_id + rule_id + window_start

Redis zset helps collapse part of the duplicate input, because the same member written twice updates the score. The actual send path still needs the agent/rule/window lock.

HCM remains the final guard. When it receives an exception message, it re-reads current status, task count, skill group membership, and rule config. It writes status_table and unified_work_status_log only if the current facts still match the rule. If the agent already has a task, or the status changed through another path, HCM drops the candidate.

After cross-region support, the state path has three independent switches:

Keep those switches separate. Request routing, event replication, and state emission ownership often need different rollout and rollback timing.

Version 6 turns mirrored events into safe state changes:

How do we split TT and IES without breaking the old path?

The system later served both IES and TT business lines. Sharing HCM, Heartbeat, DB, MQ, and routing update paths tied release risk, capacity, and disaster recovery together.

The tempting split is to create TT HCM, TT Heartbeat, TT DB, and TT MQ, then move all TT upstreams in one shot.

That is a bad bet. HCM touches login, status update, skill groups, task count, routing, WFM, and binlog propagation. A field mismatch, filter bug, or consumer group mistake can affect dispatch and state sync.

The safer first phase is coexistence.

Figure 8: TT upstreams first move to TT HCM, while writes still land in IES HCM. Forwarding and dsyncer leave room for rollback. Generated by gpt-image-2.

The phase-one path is:

1
2
3
4
5
6

TT upstream
-> TT HCM
-> forward RPC to IES HCM
-> IES DB
-> dsyncer
-> TT DB

The goal is narrow: move the call entry point first, keep the write truth in the old path.

To let the old HCM know which data should also flow to TT, add an ownership filter:

1
2
3
4
5

agent_id
-> agent_skill_group_rel
-> skill_group
-> access_party
-> TT or IES

Cache the result in Redis. On cache miss, read DB, resolve access party, then write the cache. Work status, agent-skill-group relations, and routing updates can all use this ownership result.

This phase is rollback-friendly. If TT HCM has a problem, traffic can continue through the original IES path. If the filter has a problem, forwarding can be disabled by config while the old path keeps serving agents.

Version 7 is a migration bridge:

When can the split become real?

After coexistence is stable, the second phase separates writes, events, MQ, and routing updates.

Figure 9: After the split, TT and IES own separate upstream, HCM, DB, MQ, and routing update paths. Generated by gpt-image-2.

The split includes:

1. TT HCM stops forwarding RPCs to IES HCM.
2. The IES-to-TT dsyncer stops.
3. IES heartbeat and MQ stop consuming TT agent messages.
4. TT routing consumes only TT status updates.
5. IES routing consumes only IES status updates.

After that, the old HCM path sheds about 3,000 agents and about 2,000 QPS.

There are two things to prove: the services are separated, and the state facts are closed inside each side.

• TT agent heartbeat enters only the TT side.
• TT agent status is committed only by TT HCM.
• TT status changes go only to TT routing, WFM, and data consumers.
• IES keeps its original path without TT release and cutover risk.

Only then is the split actually done.

Version 8 removes the bridge after ownership is proven:

How do we notice when frontend activity stops entering the system?

State calculation depends on input events. Green backend RPC metrics only prove backend services are alive. They do not prove user activity reached Heartbeat.

After one GCP-to-NO1A cutover, agents frequently became abnormal while they were still working in the workbench. The failure was in long-link configuration: keyup and keydown events were not reliably entering Heartbeat.

HCM error rate could not see that part of the input path. We needed input health before state calculation.

Figure 10: Before calculating state, the system verifies that user activity enters through Workbench SDK, WS-API, Frontier, Heartbeat intake, and Raw MQ. Generated by gpt-image-2.

Input health needs several dimensions:

The stronger design feeds input health into the rule layer.

If one region or workbench version has unhealthy input, automatic abnormal rules should degrade: extend the window, pause auto-abnormal, or send reminders only. When input recovers, normal calculation resumes.

That reduces automation during incidents, but it prevents a worse failure: marking working agents abnormal because the collection path broke.

Version 9 adds input observability:

What if HCM is correct and downstream is stale?

Once HCM commits a status, downstream systems still need to receive it. This was the US-TTP ES incident in another form: HCM DB had the correct offline state, but WFM still displayed the old state.

The binlog-to-HCM-MQ path had about 300k queued messages. The work status handler was not slow. An ES-dependent handler timed out and blocked shared consumption.

The design response is to give state propagation its own lane.

Figure 11: Binlog consumers are split by business meaning. A slow search handler no longer blocks work status delivery to WFM and routing. Generated by gpt-image-2.

The target shape treats status change as a first-class event:

1
2
3
4
5
6

HCM transaction
-> status_table
-> status_change_outbox
-> work_status_dispatcher
-> status MQ
-> WFM / Routing

status_change_outbox is written in the same transaction as the status update. The dispatcher only delivers outbox status events to the status MQ.

Search, analytics, and audit can keep consuming binlog or subscribe to their own event streams. They should not share the same blocking point as work status propagation. Slow handlers get their own retry and DLQ.

End-to-end metrics need to follow the same split:

The worst state-system failure is a correct source of truth with stale consumers. Split commit, outbound delivery, and consumption, then measure each segment.

Version 10 isolates propagation:

How do we keep hot status reads from taking HCM down?

GetWorkStatus is a hot read in this pipeline. Compute calls it. Upstreams call it. During one OOM incident, downstream error rate spikes aligned with GetWorkStatus traffic spikes.

The state system needs a capacity guard.

Figure 12: GetWorkStatus gets quota in front, caller isolation inside, and short-TTL cache plus autoscaling signals behind it. Generated by gpt-image-2.

The goal is caller isolation. If one upstream goes bad, its failure should stay inside its budget.

GetWorkStatus needs several defenses:

Heartbeat Compute should also reduce pressure. If the same agent reports many events in a short window, Compute does not need to call GetWorkStatus for every event. It can merge by agent in a small window, read a local or Redis snapshot, and still let HCM recheck before final commit.

The path changes from “every caller hits DB-shaped truth” into a read service with budgets, isolation, and degradation.

Version 11 turns a hot read into a protected service:

How do query changes ship safely?

One HCM 4.3 rollback exposed a release problem.

Canary looked fine. After ROW, new error logs appeared. Two LEFT JOINs amplified a count SQL path and caused DB timeouts.

Canary alone did not cover this class of failure. Canary traffic is smaller and its data distribution may be kind. Expensive queries need their own cost gate.

Figure 13: Query changes pass through feature flags, canary, shadow query, EXPLAIN checks, slow SQL monitoring, and rollback gates. Generated by gpt-image-2.

A query rollout can follow this path:

1
2
3
4
5
6

query change
-> feature flag
-> canary
-> shadow query
-> row traffic
-> auto rollback

shadow query does not affect online response. It compares result and cost. The release system watches:

Feature flags should be finer than “use new query.” A better split is by filter shape:

1
2

simple filter -> old query
join filter   -> join query

If a complex filter path breaks, only that path rolls back.

Version 12 adds release gates around data access:

What remains open?

By this point, the heartbeat design has become a state platform. Intake, compute, commit, propagation, capacity, release, and ownership boundaries can each be observed and degraded separately.

There is still more to build:

Final design

The original question was simple: how do we know whether an agent is online?

The final design has two planes.

The data plane turns user activity into state facts:

1
2
3
4
5
6
7
8
9
10
11
12
13

Agent UI
-> Workbench SDK
-> WS-API
-> Frontier
-> Heartbeat Service
-> Raw MQ
-> Compute
-> Redis zset
-> Exception MQ
-> HCM
-> DB + Outbox
-> Status MQ
-> WFM / Routing / Analytics

The control plane keeps the data plane bounded during failure:

• The frontend reports activity and heartbeat events.
• Workbench SDK owns the long-link singleton, reconnect, short-link compensation, and client-side dedupe.
• Input health proves activity reached Heartbeat.
• MQ mirror preserves event continuity across regions.
• Redis zset and dedupe locks collapse repeated candidates.
• active_idc decides which region can emit exception messages.
• HCM recheck closes the final status commit.
• Outbox and handler isolation protect work status propagation.
• Quota, cache, bulkheads, and circuit breakers protect hot reads.
• Query gates control risky SQL changes.
• TT/IES ownership separates agent, skill group, MQ, and routing boundaries.

The design is much larger than last_seen_at, but each layer earns its place. A heartbeat system that drives routing and workforce management is not a ping loop. It is an event intake system, a time-window calculator, a state authority, a propagation pipeline, a multi-region failover path, and a set of release and capacity guardrails around the same fact: what state is this agent in right now?

但它在关键地方是错的。let a = 1 绑定的是 primitive number，不是 Number 对象。对象只会在访问属性或方法时，以临时 wrapper 的语义出现一下。

这里说“解释器”，更准确一点是 JS 引擎的执行语义。现代引擎会有 bytecode 和 JIT 优化，但这不改变 ECMAScript 规定的那套行为。

图：let a = 1; 的生命周期。先创建词法绑定，绑定在初始化前处于 TDZ；执行到声明语句时才把 primitive number 1 写入绑定。generated by gpt-image-2.

let a = 1 不是 new Number(1)

先看最容易误会的地方：

1
2
3
4

let a = 1;

typeof a; // "number"
a === 1; // true

如果它等价于 new Number(1)，结果就完全不同：

1
2
3
4

const b = new Number(1);

typeof b; // "object"
b === 1; // false

更麻烦的是：

1

Boolean(new Number(0)); // true

对象永远是真值，哪怕它包装的是 0。所以实际业务代码里基本不要写 new Number(...)、new String(...)、new Boolean(...)。它们制造的是 wrapper object，不是普通 primitive。

进入作用域时，a 已经存在，但还不能读

当 JS 引擎处理到一个 block、script 或 module 时，会先建立当前作用域里的词法绑定。let a = 1 里的 a 在这个阶段已经被登记到 Lexical Environment 里。

但它不是 undefined。

它是 uninitialized。

这就是 TDZ，Temporal Dead Zone：

1
2
3
4

{
  console.log(a); // ReferenceError
  let a = 1;
}

这和 var 不一样：

1
2
3
4

{
  console.log(v); // undefined
  var v = 1;
}

let 的设计不是为了让“提升”消失，而是把“声明已被登记”和“值已经可用”分开。这样读一个还没初始化的变量会直接报错，而不是给你一个看起来能用、其实很容易埋 bug 的 undefined。

执行到这一行时，才把 1 写进绑定

真正执行到：

1

let a = 1;

右边的数字字面量 1 会被求值成一个 primitive number。然后引擎把这个值写进 a 这个词法绑定里。

可以把它想成：

1
2
3

Lexical Environment

a -> 1

但不要想成：

1

a -> Number object { [[NumberData]]: 1 }

前者是 let a = 1。后者更接近 new Number(1)。

这里还有一个小细节：1 本身是 immutable，但 a 这个绑定如果用的是 let，可以被重新赋值。

1
2

let a = 1;
a = 2;

这不是把数字 1 改成 2。这是让绑定 a 从指向 primitive value 1，变成指向另一个 primitive value 2。

那为什么 a.toString() 能跑？

问题就在这里：

1
2

let a = 1;
a.toString(); // "1"

如果 a 不是对象，它为什么有方法？

答案是：访问属性或方法时，JS 会对 primitive 走一次临时 wrapper 语义。规范里更准确的说法是把 primitive 做 ToObject，让这次属性查找可以落到 Number.prototype 上。

你可以先用这个模型理解：

1

Number(a).toString();

但要马上补一句：这只是教学模型，不是 let a = new Number(1)，也不代表引擎一定真的在堆上分配了一个对象。现代引擎经常会把这种临时对象优化掉。

更接近语义的过程是：

1. 读到 primitive number 1
2. 访问 .toString
3. 临时按 Number wrapper 的方式查找属性
4. 找到 Number.prototype.toString
5. 调用完以后，临时 wrapper 消失
6. a 仍然是 primitive number

图：let a = 1 绑定的是 primitive。a.toString() 只是在方法访问时走临时 wrapper 语义；new Number(1) 才会创建持久对象。generated by gpt-image-2.

同样的逻辑也适用于字符串：

1
2

const s = "abc";
s.toUpperCase(); // "ABC"

"abc" 不是 String 对象。但访问方法时，JS 可以临时用 wrapper 语义去找 String.prototype.toUpperCase。

这也解释了为什么 primitive 不能挂持久属性：

1
2
3

const s = "abc";
s.x = 1; // sloppy mode: ignored; strict mode: TypeError
console.log(s.x); // sloppy mode: undefined

给临时 wrapper 塞进去的东西，下一行已经没了。

为什么要这样设计？

我觉得这里的设计其实很务实。

primitive 要便宜。数字、字符串、布尔值太常用了，如果每一个都变成对象，内存、GC 和比较语义都会变复杂。1 === 1 应该就是值比较，不应该变成两个对象引用之间的比较。

但开发者又需要统一的调用体验。你当然希望能写：

1
2

"abc".toUpperCase();
(1).toString();

而不是每次都显式创建对象。临时 wrapper 正好把这两件事接起来：值本身保持轻量和不可变，方法则挂在共享的 prototype 上。

TDZ 解决的是另一类问题。let 和 const 是块级作用域，代码读起来应该接近“声明之后才能用”。如果像 var 一样提前给个 undefined，很多错误会变成运行到更后面才爆。TDZ 选择早点报错。

所以这套机制背后不是“JS 偷偷把所有东西都变成对象”。更准确的理解是：

let a = 1 建立一个词法绑定，把 primitive number 放进去。primitive 保持 primitive；只有访问属性或方法时，才临时借用对应 wrapper 和 prototype 的能力。

这个模型记住以后，很多看似奇怪的 JS 行为就顺了：

1
2
3
4
5
6
7
8

typeof 1; // "number"
typeof new Number(1); // "object"

1 === Number(1); // true
1 === new Number(1); // false

Boolean(0); // false
Boolean(new Number(0)); // true

最后那行尤其适合提醒自己：wrapper object 是对象，不是值。

参考

• ECMAScript: let and const declarations
• ECMAScript: ToObject
• MDN: Primitive

这句话没错，但太粗。真正有用的理解是：GC 回收的是从 roots 不可达的对象；新生代和老生代用的策略不同；同一套 JS 引擎放进浏览器和 Node，最容易出问题的地方也不一样。

图：以 V8 为例，GC 从 stack、global、closure 等 roots 出发追踪可达对象。浏览器宿主和 Node 宿主会通过 DOM、listener、Buffer、EventEmitter 等把对象继续留在图里。generated by gpt-image-2.

GC 不是回收“你觉得没用”的对象

GC 只看引用图。

如果一个对象还能从 stack、global、closure、模块缓存、DOM listener、EventEmitter、timer 或 pending async task 走到，它就是活的。业务上你觉得它“已经没用了”，对 GC 没意义。

这也是很多内存泄漏难查的原因。难点通常在引用链上：有人还在很远的地方引用它。

为什么要分新生代和老生代

分代的目的不是给对象贴标签。它是为了用不同算法处理不同年龄的对象。

GC 设计里有一个很实用的观察：大部分对象死得很快。一次渲染里的临时数组、map/filter 生成的中间对象、函数里临时创建的小对象，很多活不过下一轮 GC。

所以新对象先进 young generation。这里的关键在于尽快处理大量短命对象，别每次都扫全堆。

图：新生代 GC 常用 Scavenge 思路。把 From-space 里还活着的对象复制到 To-space，死对象跟着整块旧空间一起丢掉，复制时顺手完成压缩。generated by gpt-image-2.

新生代适合 copying GC。假设 100 万个对象里只有 2 万个还活着，复制 2 万个活对象比扫描、清理、压缩 100 万个位置更划算。

对象如果连续几轮都活下来，就会被提升到 old generation。老生代里的对象死亡率没那么高，再用复制算法就不划算了：如果 90% 都活着，复制成本会非常重。

图：老生代更接近 Mark-Sweep-Compact。先标记可达对象，再把不可达对象放回 free list；如果碎片太多，再移动活对象并更新指针。generated by gpt-image-2.

这里最容易漏掉的是 compact。Sweep 之后内存可能变成一块一块的洞，能复用，但局部性差，也可能放不下大对象。Compact 会把活对象往一侧移动，把空闲空间整理成连续区域。代价是对象地址变了，所以引用这些对象的指针也要更新。

现代 V8 不会傻等一次超长 stop-the-world。Orinoco 里有 parallel、incremental、concurrent 这些手段，把一部分工作并行化、切片化或者放到后台做。核心目标很朴素：少卡主线程。

宿主不同，坑也不同

JavaScript 语言层说的是可达性，宿主决定了很多引用从哪里冒出来。

浏览器里，我会先看 DOM 和交互生命周期：

Node 里更常见的是长生命周期对象：

所以排查方向也不一样。浏览器先看页面生命周期和 detached DOM；Node 先看进程级缓存、连接、listener、Buffer 和 heap snapshot。

Map 和 WeakMap 的边界

WeakMap 很适合给对象挂 metadata，尤其是 DOM node 这类生命周期不完全由你控制的对象。

但它不是“自动防泄漏按钮”。只有 key 是弱引用，value 里如果又被别的地方强引用，照样活着。WeakMap 也不能枚举 key，因为一旦能枚举，你就能观察 GC 什么时候发生，语义会变得不确定。

我现在更愿意这样记：

我会记住的几件事

学 GC 不需要背每个 V8 版本的细节。版本会变，算法实现也会继续优化。

更稳定的是这几条：

1. GC 看可达性，不看业务语义。
2. 分代 GC 是用对象年龄换算法效率。
3. 新生代复制活对象，老生代更关注标记、清扫、碎片和压缩。
4. 对象会移动，移动后必须更新指针。
5. 宿主会制造额外引用：浏览器是 DOM 和事件，Node 是进程级资源和 external memory。
6. 泄漏排查先画引用链，再谈工具。

如果只能记一句，我会记这句：内存泄漏通常说明你还给 GC 留了一条路。

参考资料

• Trash talk: the Orinoco garbage collector
• Orinoco: young generation garbage collection
• Memory management - JavaScript | MDN
• WeakMap - JavaScript | MDN