Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 bigcode
|
a2e1f0d923 | ||
|
bigcode
|
329d0e4c5d | ||
|
bigcode
|
053d3347f3 | ||
|
bigcode
|
642b370c7a | ||
|
bigcode
|
f17eef3f74 | ||
|
bigcode
|
f0444e5879 |
@@ -0,0 +1,142 @@
|
||||
# dp6 24小时实测优化报告(fetch+浏览器双路径)
|
||||
|
||||
- **运行区间**: 2026-11-01 01:22:43 UTC → 2026-11-02 01:21:43 UTC(24h 跑满,进程正常退出,final 已写)
|
||||
- **版本**: SerpMax v1.78.2 + 压测客户端控制器 v4(`stress/serp-dualpath-optimize.ts`,runtime=auto 双路径,workers=12,速率区间 [3,400])
|
||||
- **数据源**: `/opt/serpmax/data/dp6-final.json`、`dp6.jsonl`(分钟)、`dp6-req.jsonl`(逐请求 117,724 条)、`dp6.log`;对照 `dp5-final.json`、`dp5-req.jsonl`。全部为服务器落盘的真实测量值,无任何外推/估算。
|
||||
- **对照说明**: dp1–dp4 为短程调优试探(服务器上无 24h final 文件);完整 24h 对照只有 dp5(v1.78.1+控制器 v3)与 dp6(v1.78.2+控制器 v4)。
|
||||
|
||||
---
|
||||
|
||||
## 1. 结论摘要
|
||||
|
||||
| 指标 | dp5(v3) | dp6(v4) | 变化 |
|
||||
|---|---|---|---|
|
||||
| 总请求 | 80,974 | **104,440** | **+29.0%** |
|
||||
| 成功 | 80,301 | 102,461 | — |
|
||||
| 成功率 | 99.17% | **98.11%** | −1.06pp,仍 >98% ✓ |
|
||||
| 日均速率 | 56.3/min | **72.6/min** | **+29.0%** |
|
||||
| captcha 命中 | 335(0.41%) | 1,537(1.47%) | 当日波次更密 |
|
||||
| 门壳结果 gate | 270 | 246 | — |
|
||||
| 池空 poolEmpty | 0 | 75(最差 1 小时 23 次) | v1.78.2 轮转取缓存代价 |
|
||||
| httpErr / netErr | 10 / 7 | 65 / 0 | httpErr 全部集中在 23 时波浪 |
|
||||
| fetch / browser 分流 | 41,773 / 38,849 | 48,573 / 54,190 | browser 承接溢出 52.7% |
|
||||
|
||||
**核心结论**: 在 >98% 成功率约束下,控制器 v4 把 24h 真实日均吞吐从 56.3 提到 **72.6 请求/分钟(+29%)**,代价是成功率贴着约束线收在 98.11%。这是约束边界上的实测最优交换。
|
||||
|
||||
---
|
||||
|
||||
## 2. dp6 全天逐小时实测(UTC)
|
||||
|
||||
| 小时 | 请求 | 成功率 | captcha | gate | poolE | fetch | browser | 均值/min |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| 01日01时(38min) | 3,762 | 99.39% | 19 | 3 | 0 | 1,558 | 2,185 | 99 |
|
||||
| 02时 | 5,214 | 98.68% | 59 | 8 | 0 | 2,415 | 2,740 | 87 |
|
||||
| 03时 | 5,416 | 98.47% | 54 | 25 | 0 | 2,265 | 3,097 | 90 |
|
||||
| 04时 | 4,266 | 96.91% | 115 | 12 | 1 | 2,094 | 2,056 | 71 |
|
||||
| 05时 | 1,667 | 98.86% | 9 | 10 | 0 | 1,632 | 26 | 28 |
|
||||
| 06时 | 5,098 | 99.86% | 2 | 4 | 1 | 2,189 | 2,906 | 85 |
|
||||
| 07时 | 5,063 | 99.09% | 32 | 14 | 0 | 2,006 | 3,025 | 84 |
|
||||
| 08时 | 5,189 | 98.50% | 67 | 10 | 0 | 2,003 | 3,119 | 86 |
|
||||
| 09时 | 5,146 | 98.15% | 72 | 18 | 1 | 2,105 | 2,968 | 86 |
|
||||
| 10时 | 4,583 | 98.23% | 68 | 13 | 0 | 2,206 | 2,309 | 76 |
|
||||
| 11时 | 3,601 | 98.03% | 45 | 14 | 7 | 2,014 | 1,531 | 60 |
|
||||
| 12时 | 4,470 | 99.22% | 17 | 13 | 3 | 2,093 | 2,356 | 74 |
|
||||
| 13时 | 4,678 | 97.26% | 101 | 14 | 8 | 1,792 | 2,773 | 78 |
|
||||
| 14时 | 3,505 | 97.60% | 55 | 5 | 2 | 2,196 | 1,231 | 58 |
|
||||
| 15时 | 3,231 | 97.62% | 70 | 6 | 0 | 2,165 | 996 | 54 |
|
||||
| 16时 | 5,352 | 97.42% | 112 | 14 | 0 | 2,080 | 3,158 | 89 |
|
||||
| 17时 | 4,509 | 96.70% | 116 | 6 | 14 | 2,094 | 2,282 | 75 |
|
||||
| 18时 | 2,708 | 98.08% | 46 | 3 | 0 | 1,575 | 1,087 | 45 |
|
||||
| 19时 | 4,712 | 98.58% | 61 | 2 | 2 | 2,025 | 2,624 | 79 |
|
||||
| 20时 | 5,228 | 97.42% | 111 | 15 | 3 | 2,037 | 3,076 | 87 |
|
||||
| 21时 | 4,673 | 98.44% | 57 | 8 | 6 | 2,325 | 2,285 | 78 |
|
||||
| 22时 | 4,752 | 97.92% | 91 | 7 | 0 | 1,910 | 2,751 | 79 |
|
||||
| 23时(最差) | 4,114 | **94.90%** | **146** | 14 | **23** | 1,650 | 2,268 | 69 |
|
||||
| 02日00时 | 1,613 | 98.64% | 12 | 6 | 1 | 1,374 | 224 | 27 |
|
||||
| 02日01时(21min) | 1,840 | 99.73% | 0 | 1 | 3 | 747 | 1,090 | 88 |
|
||||
|
||||
- jsonl 口径合计 104,390;final.json 权威口径 **104,440 / 98.11%**(差值为跨小时边界分钟)。
|
||||
- 全天净错误 netErr=0,无一次网络层失败。
|
||||
- 最差小时 23时(94.90%)对全天累计仅拖累约 0.13pp——深度削减把该小时暴露量压到最小,是"深切保护累计"策略的实测验证。
|
||||
|
||||
---
|
||||
|
||||
## 3. 速率段分布与控制器行为结构(实测)
|
||||
|
||||
**速率段时间分布(1,439 分钟):**
|
||||
|
||||
| 速率段 | 分钟数 | 发出请求 | 请求占比 |
|
||||
|---|---|---|---|
|
||||
| <50/min | 250 min | 7,119 | 6.8% |
|
||||
| 50–100 | 252 min | 17,691 | 16.9% |
|
||||
| 100–200 | 517 min | 44,402 | 42.5% |
|
||||
| 200–300 | 337 min | 28,047 | 26.9% |
|
||||
| 300–400 | 53 min | 4,356 | 4.2% |
|
||||
| =400 | 30 min | 2,775 | 2.7% |
|
||||
|
||||
**行为结构(来自 dp6.log 全量速率变更史,共 113 次调率):**
|
||||
- **5 次深切瀑布**(×0.7 连锁到 <30/min):04:50→23.9、09:32→47、14:10→39、17:54→11.1、23:57→6.3/min(末次为 ×0.5 四连)。每次都在波浪过后由 clean-fast ×1.3 链恢复。
|
||||
- **10 次 clean-fast ×1.3 恢复链**,最快 9 拍从 40 拉到 331/min(14:46–14:53),典型 6–7 拍。
|
||||
- **×1.08 常规爬坡**接在恢复链后,爬到 280–400 区间;全天共 228 个 ramp 序列。
|
||||
- **12 次验证码软切**(×0.7)、**2 次池空软切**(×0.85)、多次 30 分钟入场切(<98% 成功)。
|
||||
- 完整周期 1.5–2h:波浪 → 切(最深 ×0.5)→ 低速持守到 captcha==0 → ×1.3 链 → ×1.08 爬坡 → 重复。
|
||||
- 结尾 00:56–01:13 恢复链一路拉回 400/min,最后 21 分钟以 352–400/min 跑出 99.73%。
|
||||
|
||||
**饱和天花板(重要实测发现)**: 速率上限虽设 400/min,但 =400 段实际只发出 92.5 请求/min(2,775/30min)。瓶颈是 workers=12 × 平均延迟(ok 均值 7.5s)→ 理论上限 ≈96/min;实测高速率段稳定落在 100–135/min。**400 的桶只是天花板标记,有效吞吐由 worker 数与延迟决定**——这是 v5 最直接的扩容方向。
|
||||
|
||||
---
|
||||
|
||||
## 4. 双路径经济学(逐请求实测)
|
||||
|
||||
| 路径/结果 | dp6 条数 | dp6 均延迟 | 多次尝试占比 | dp5 对照 |
|
||||
|---|---|---|---|---|
|
||||
| fetch/ok | 48,467 | **5,160ms** | 0.70% | 41,663 / 4,465ms / 0.90% |
|
||||
| browser/ok | 53,994 | **9,668ms** | 2.26% | 38,638 / 7,446ms / 2.89% |
|
||||
| fetch/gate | 106 | 10,873ms | 100% | 110 / 10,649ms |
|
||||
| browser/gate | 140 | **39,677ms** | 100% | 160 / 30,851ms |
|
||||
| browser/empty | 56 | 31,469ms | — | 51 / 24,546ms |
|
||||
| captcha(未分派) | 1,537 | 4,530ms | — | 335 / 4,927ms |
|
||||
| poolEmpty / httpErr | 75 / 65 | 31.2s / 42.2s | — | 0 / 10 |
|
||||
|
||||
- browser 单请求耗时是 fetch 的 1.87 倍,但承接了 52.7% 的已分派流量(dp5 为 48.0%)——高速率段 fetch 桶(60/min 封顶)只够供给约 15%,溢出全部由浏览器热池消化,溢出设计按预期工作。
|
||||
- 门壳尝试 874 次(0.83%,dp5 为 0.95%):browser 门壳每次烧 39.7s,是最贵的失败形态;v1.78.2 的指纹轮换使门壳占比略降。
|
||||
- captcha 检测均耗时 4.5s,本身便宜;真正昂贵的是它触发的降速与爬坡时间。
|
||||
|
||||
---
|
||||
|
||||
## 5. 稀释论的实测验证
|
||||
|
||||
**论题**: captcha 供给由 Google 侧按波次发放(逐小时 2–146 个,与本地速率弱相关),因此同等波次下速率越高、captcha 占比越低、窗口成功率越高;**波浪中降速反而抬高 captcha 占比、恶化窗口成功率**。
|
||||
|
||||
实测证据:
|
||||
- 配对小时:19时(79/min,captcha 61)→ 98.58%;17时(75/min,captcha 116)→ 96.70%。速率相近,成败由波次强度决定。
|
||||
- 高速率窗口:400/min 段窗口成功率 99.6–99.9%,captcha 短窗占比 0(capPctS=0)。
|
||||
- 低速率反例:05:24 在 31.3/min 处,窗口内 3 个 captcha 就构成 1.13% 占比触发软切——同样的 3 个 captcha 在 400/min 下只占 0.09%。
|
||||
- 但波浪总量确实随速率升高而增多(dp6 1,537 vs dp5 335):提速换来的是分母稀释,不是 captcha 绝对数减少。
|
||||
|
||||
---
|
||||
|
||||
## 6. 控制器 v4 的三个残余缺陷(实测例证,已列入 v5 待修)
|
||||
|
||||
1. **ongoing 切不感知量(量盲)**: 低速率下小分母放大占比。05:24 因"验证码 3 个占比 1.13%"把 31.3 切到 25/min;00:02–00:03 波浪实际已近尾声(~00:20 结束),仍以窗口 93.4%/93.2%(每窗仅几十个请求)连续 ×0.5 切到 6.3/min 并滞留到 00:30。代价:05时全天第二低的 28/min 却有 98.86% 成功率——明显压不满。
|
||||
2. **低速恢复锁死**: 恢复条件要求 10 分钟短窗 successS≥0.995,低速涓流(~1% captcha 占比)下永远达不到;实际恢复全靠 captcha==0 清洁段触发 clean-fast。05:00–06:10 在 24–42/min 滞留 70 分钟,11:35–12:14 滞留 39 分钟。
|
||||
3. **累计盲**: 控制器只看窗口不看全天累计。23时 94.90% 把累计从 98.19% 压到 98.06%,控制器没有任何累计感知的削波/放行;反过来,结尾累计已锁定 >98% 时,00:43 仍因 4 个 captcha(0.96%)从 73.3 切到 51.3,未能利用富余度。深切能保护累计是"暴露量最小化"的副产品,不是设计使然。
|
||||
|
||||
---
|
||||
|
||||
## 7. v5 优化方向(待下一个真实 24h 验证,本报告不做任何数字预测)
|
||||
|
||||
1. **恢复条件按量归一**: 低速时用绝对失败数而非占比判恢复(如短窗 captcha≤1 且 sent≥N,或改用指数衰减基线),消除 05时/11时式滞留。
|
||||
2. **ongoing 切加量下限**: 窗口 sent 不足阈值时只记信号不切速,避免小分母瀑布。
|
||||
3. **累计感知**: 把全天累计成功率与 98% 线的余量引入削波深度(余量大→浅切)与恢复速度(余量大→快爬)。
|
||||
4. **扩 workers**: 实测饱和 ≈96–135/min 由 workers=12 × 延迟决定;提高 worker 数是把 100–400 段天花板变为真实吞吐的最短路径,同时配合 fetch 桶扩容评估。
|
||||
5. 池空 75 次来自 v1.78.2 节点轮换取缓存,最差小时 23 次;可对波浪期保留最小缓存水位以归零该类错误。
|
||||
|
||||
---
|
||||
|
||||
## 8. 口径备注
|
||||
|
||||
- 成功率 = ok/sent;sent 含 captcha/gate/empty/poolEmpty/httpErr 全部结果,netErr 另计。
|
||||
- dp6 全天 fetch 48,573 + browser 54,190 = 102,763 条有路径归属;其余 1,677 条为未按路径记录的结果(池空/httpErr 等)。
|
||||
- dp1–dp4 仅有试探性产物(dp1 26KB / dp2–dp3 ~1.7KB / dp4 137KB jsonl),无 24h final,不参与定量对照。
|
||||
- 服务器监控(serp-monitor)h24 口径混有 dp6 之外的生产流量,本报告一律不采用;只用 dp6 系列文件自身口径。
|
||||
@@ -0,0 +1,368 @@
|
||||
/** # 脚本说明
|
||||
* SERP 双路径(fetch+浏览器)24h 自适应吞吐优化测试(用户 2026-10-30 目标;
|
||||
* 2026-11-02 用户要求可用率约束上调至 >99%):
|
||||
* 在保证客户端成功率 >99% 的约束下,自适应逼近系统可持续的最大每分钟请求数。
|
||||
* 与 serp-profile-capacity.ts(单号 sticky+fetch,绕过池/令牌)不同,本脚本走生产
|
||||
* 双路径原形态(runtime=auto,无 sticky,全池公平负载选号):
|
||||
* GET /api/serp/json/search?q=...&num=10
|
||||
* 令牌内走 go-fetch(全局限流 fetchPerSec),限流外并发走温池浏览器 —— 组合即生产形态。
|
||||
* AIMD 控制器 v5(dp6 全天实测教训驱动;v4 基础上改四点):
|
||||
* 硬约束(成功率即约束本身):首次跌破 99%(失败≥2)×0.7、<96% ×0.5;退化段内续砍
|
||||
* 需短窗 <96% 且全程增量新失败 ≥2 —— dp5 教训:同一波旧失败被短窗反复引用,
|
||||
* 五分钟连砍五刀 175→30/min,直接打灭浏览器溢出路径;
|
||||
* 软信号按事件段(episode)迟滞 + 绝对数护栏(低速率下 1~2 个噪音 = 高比率,不得触发):
|
||||
* 验证码 10min 窗 ≥3 个且占比>2% ×0.5、≥3 个且>0.5% ×0.7;门壳 ≥3 且占比>15% ×0.85;
|
||||
* 池空>5 ×0.85;软砍下限 SOFT_FLOOR=25 —— 背景验证码供给恒定,速率越低比率越高,
|
||||
* 比率规则在低位永真(dp5 实测 12:00-17:00 自 52/min 自残至 3/min);
|
||||
* 干净爬坡:不在退化段 且 短窗干净 且 无池空 且(验证码 0 个或样本 ≥300)→ 向「干净
|
||||
* 水位」(30min 窗 ≥99.5% 时的最高速率)以 ×1.3/拍快速回收,水位之上每 5 拍 ×1.08。
|
||||
* v5 修 dp6 实测「低速恢复锁死」:短窗成功率 ≥99.5% 在低速涓流下永不可达
|
||||
* (05:00-06:10 滞留 70min 在 24-42/min,该小时实际 98.86%)→ 增加备选条件
|
||||
* 「验证码 ≤1 个且短窗 ≥98.5%」(稀释论:同量涓流在高速率下占比可忽略,可爬)。
|
||||
* 速率区间 [3, 400] req/min;并发 worker 默认 12(真实并发让溢出自然流向浏览器路径)。
|
||||
* 判定(客户端,JSON 出口):ok = 200 且 organic>0;gate = 200 且 organic=0 且
|
||||
* attempts 含 gate;empty = 200 且 organic=0 无门壳尝试;错误按响应 code 分类
|
||||
* (CAPTCHA_DETECTED / SERP_POOL_EMPTY / 其余 httpErr);网络层失败记 netErr。
|
||||
* 输出:data/<label>.jsonl 每分钟聚合、<label>-req.jsonl 逐请求、<label>-live.json
|
||||
* 每 5min 快照、<label>-final.json 终局;控制台每 5min 摘要。
|
||||
* 用法:SERP_TOKEN=xxx bun serp-dualpath-optimize.ts --hours 24 --label dp1 --rate0 15
|
||||
*/
|
||||
import fs from "node:fs";
|
||||
|
||||
/** # 成员变量 服务基址(同机直连 admin) **/
|
||||
const BASE = "http://127.0.0.1:53100";
|
||||
const args = process.argv.slice(2);
|
||||
function argOf(k: string, d: string): string { const i = args.indexOf(k); return i >= 0 ? args[i + 1] : d; }
|
||||
/** # 成员变量 测试时长(小时) **/
|
||||
const HOURS = Number(argOf("--hours", "24"));
|
||||
/** # 成员变量 输出标签(文件名前缀) **/
|
||||
const LABEL = argOf("--label", "dp1");
|
||||
/** # 成员变量 初始速率(req/min) **/
|
||||
const RATE0 = Number(argOf("--rate0", "15"));
|
||||
/** # 成员变量 速率下限/上限(req/min) **/
|
||||
const RATE_MIN = Number(argOf("--min", "3"));
|
||||
const RATE_MAX = Number(argOf("--max", "400"));
|
||||
/** # 成员变量 并发 worker 数 **/
|
||||
const WORKERS = Number(argOf("--workers", "12"));
|
||||
/** # 成员变量 控制器窗口(分钟)与加速步间隔(分钟) **/
|
||||
const WINDOW_MIN = 30;
|
||||
const RAMP_EVERY_MIN = 5;
|
||||
const TOKEN = process.env.SERP_TOKEN ?? "";
|
||||
if (!TOKEN) { console.error("缺少 SERP_TOKEN"); process.exit(1); }
|
||||
|
||||
/** # 成员变量 输出目录(data/,相对脚本) **/
|
||||
const DATA = new URL("./data/", import.meta.url).pathname;
|
||||
fs.mkdirSync(DATA, { recursive: true });
|
||||
const JSONL = `${DATA}${LABEL}.jsonl`;
|
||||
const REQ_JSONL = `${DATA}${LABEL}-req.jsonl`;
|
||||
const LIVE = `${DATA}${LABEL}-live.json`;
|
||||
|
||||
/** # 成员变量 长尾查询词表(4 维组合,2026-10-30 用户反馈改版):
|
||||
* 旧 A×B×C(2000 组合)相似度太高,/goto 解析缓存命中虚高,同 Profile 并发 /goto
|
||||
* 的真实负载被洗掉。改为 修饰×品类×限定×城市 4 维(40×60×40×60 = 576 万组合),
|
||||
* 随机不重复消费 —— 每次搜索结果集与 /goto 链接基本不同,缓存命中回落到真实水平。
|
||||
*/
|
||||
const A = ["best", "top rated", "cheap", "review of", "how to choose a", "buy", "compare", "2026 guide for", "price of", "warranty for", "where to buy", "discount on", "is it worth buying", "reliability of", "problems with", "alternatives to", "user reviews for", "long term test of", "repair costs for", "energy use of", "setup guide for", "maintenance tips for", "refurbished", "open box deals on", "wholesale price of", "used value of", "safety rating of", "recalled models of", "accessories for", "replacement parts for", "manual pdf for", "dimensions of", "weight limit of", "installation cost of", "monthly deals on", "black friday price for", "cyber monday deal for", "student discount for", "military discount for", "trade in value of"];
|
||||
const B = ["coffee maker", "running shoes", "laptop stand", "rice cooker", "air fryer", "standing desk", "office chair", "mechanical keyboard", "monitor arm", "webcam", "espresso machine", "vacuum cleaner", "electric kettle", "gaming mouse", "noise cancelling headphones", "desk lamp", "water bottle", "yoga mat", "blender", "space heater", "air purifier", "electric toothbrush", "smart thermostat", "robot vacuum", "ergonomic mouse", "usb-c hub", "portable ssd", "wireless charger", "dash cam", "action camera", "trail running shoes", "hiking backpack", "camping tent", "sleeping bag", "kayak paddle", "bicycle helmet", "electric scooter", "board games", "3d printer", "drawing tablet", "studio microphone", "audio interface", "midi keyboard", "electric guitar", "drum pad", "telescope", "binoculars", "bird feeder", "grill smoker", "pizza oven", "garden hose", "leaf blower", "chainsaw", "pressure washer", "snow shovel", "garage door opener", "stud finder", "laser level", "tool set", "sewing machine"];
|
||||
const C = ["for home use", "for small office", "under 100 dollars", "in 2026", "with long warranty", "for beginners", "pro model", "on sale", "quiet operation", "energy efficient", "with fast shipping", "highly rated", "compact size", "heavy duty", "for students", "for professionals", "with extra accessories", "easy to clean", "portable version", "premium edition", "budget option", "luxury model", "with 2 year warranty", "top ranked", "editor's choice", "best seller", "new release", "on clearance", "with free returns", "for tall people", "for small spaces", "made in usa", "eco friendly", "with app control", "voice assistant compatible", "wifi enabled", "with hepa filter", "waterproof model", "lightweight version", "industrial grade"];
|
||||
const D = ["in Austin tx", "in Denver co", "in Seattle wa", "in Portland or", "in Chicago il", "in Boston ma", "in Miami fl", "in Phoenix az", "in Dallas tx", "in Atlanta ga", "in New York ny", "in Los Angeles ca", "in Houston tx", "in Philadelphia pa", "in San Diego ca", "in Las Vegas nv", "in Nashville tn", "in Detroit mi", "in Minneapolis mn", "in Charlotte nc", "in Raleigh nc", "in Omaha ne", "in Boise id", "in Salt Lake City ut", "in Columbus oh", "in Indianapolis in", "in Milwaukee wi", "in Kansas City mo", "in St Louis mo", "in Cincinnati oh", "in Pittsburgh pa", "in New Orleans la", "in Memphis tn", "in Louisville ky", "in Albuquerque nm", "in Tucson az", "in Fresno ca", "in Sacramento ca", "in Portland me", "in Burlington vt", "in Providence ri", "in Hartford ct", "in Newark nj", "in Baltimore md", "in Richmond va", "in Charleston sc", "in Savannah ga", "in Orlando fl", "in Tampa fl", "in San Antonio tx", "in El Paso tx", "in Oklahoma City ok", "in Tulsa ok", "in Wichita ks", "in Des Moines ia", "in Little Rock ar", "in Jackson ms", "in Billings mt", "in Fargo nd", "in Charleston wv", "in Eugene or"];
|
||||
const COMBO_TOTAL = A.length * B.length * C.length * D.length;
|
||||
/** # 成员变量 已消费组合索引(随机不重复;24h 全程用量远低于组合空间,碰撞回退微乎其微) **/
|
||||
const usedCombos = new Set<number>();
|
||||
function nextQuery(): string {
|
||||
let idx = Math.floor(Math.random() * COMBO_TOTAL);
|
||||
while (usedCombos.has(idx)) idx = (idx + 1) % COMBO_TOTAL;
|
||||
usedCombos.add(idx);
|
||||
const d = idx % D.length;
|
||||
const c = Math.floor(idx / D.length) % C.length;
|
||||
const b = Math.floor(idx / (D.length * C.length)) % B.length;
|
||||
const a = Math.floor(idx / (D.length * C.length * B.length)) % A.length;
|
||||
return `${A[a]} ${B[b]} ${C[c]} ${D[d]}`;
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// 自适应令牌桶(速率由控制器动态调整)
|
||||
// ============================================================================
|
||||
|
||||
/** # 成员变量 当前速率(req/min) **/
|
||||
let rate = RATE0;
|
||||
/** # 成员变量 桶状态:令牌数/上次补充时刻(容量 = max(1, ceil(每秒速率))) **/
|
||||
let bucket = { tokens: 0, lastMs: Date.now() };
|
||||
|
||||
/** # 函数
|
||||
* 取一枚发送令牌(不足时等待;速率调整即时生效,无锁 —— 单线程事件循环内安全)。
|
||||
*/
|
||||
async function acquire(): Promise<void> {
|
||||
for (;;) {
|
||||
const rps = rate / 60;
|
||||
const cap = Math.max(1, Math.ceil(rps));
|
||||
const now = Date.now();
|
||||
bucket.tokens = Math.min(cap, bucket.tokens + ((now - bucket.lastMs) / 1000) * rps);
|
||||
bucket.lastMs = now;
|
||||
if (bucket.tokens >= 1) { bucket.tokens -= 1; return; }
|
||||
await Bun.sleep(200);
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// 计数与控制器
|
||||
// ============================================================================
|
||||
|
||||
/** # 成员变量 单分钟计数桶 **/
|
||||
type Counter = {
|
||||
sent: number; ok: number; gate: number; empty: number;
|
||||
captcha: number; poolEmpty: number; httpErr: number; netErr: number;
|
||||
fetch: number; browser: number; attemptsTotal: number; gateAttempts: number;
|
||||
};
|
||||
function newCounter(): Counter {
|
||||
return { sent: 0, ok: 0, gate: 0, empty: 0, captcha: 0, poolEmpty: 0, httpErr: 0, netErr: 0, fetch: 0, browser: 0, attemptsTotal: 0, gateAttempts: 0 };
|
||||
}
|
||||
/** # 成员变量 当前分钟计数 / 滚动窗口(最近 WINDOW_MIN 个分钟桶) **/
|
||||
let cur = newCounter();
|
||||
const window: { ts: string; c: Counter }[] = [];
|
||||
/** # 成员变量 全程累计(终局汇总用) **/
|
||||
const totals = newCounter();
|
||||
/** # 成员变量 控制器决策日志 **/
|
||||
const decisions: Record<string, unknown>[] = [];
|
||||
/** # 成员变量 成功指纹计数(分布公平性观察用) **/
|
||||
const fpMap = new Map<string, number>();
|
||||
|
||||
/** # 函数
|
||||
* 窗口聚合求和(参数 n:取最近 n 个分钟桶;不传 = 全窗口)。
|
||||
*/
|
||||
function sumWindow(n = WINDOW_MIN): Counter {
|
||||
const s = newCounter();
|
||||
for (const { c } of window.slice(-n)) {
|
||||
for (const k of Object.keys(s) as (keyof Counter)[]) s[k] += c[k];
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
/** # 成员变量 软信号事件段状态(迟滞:进段砍一次,退段重置;防同一段波逐分钟连砍) **/
|
||||
let captchaWave = false;
|
||||
let gateWave = false;
|
||||
let poolEmptyWave = false;
|
||||
/** # 成员变量 硬约束退化段(进段砍一次;段内连砍需增量新失败;30min 窗恢复≥99% 退段) **/
|
||||
let degraded = false;
|
||||
/** # 成员变量 上次硬砍时的全程累计失败数(段内连砍要求全程新失败 ≥2,dp5 教训:同一波旧失败被短窗反复引用,10:24-10:28 五分钟连砍五刀 175→30/min) **/
|
||||
let failsAtLastCut = 0;
|
||||
/** # 成员变量 干净水位:30min 窗 ≥99.5% 且短窗 ≥99% 时达到过的最高速率(深跌后向水位快速回收;dp5 教训:×1.08/5 拍从 3/min 爬回需 3 小时,温池 150 ready 全程空转) **/
|
||||
let maxCleanRate = 0;
|
||||
/** # 成员变量 软砍速率下限(req/min):背景验证码供给恒定,速率越低比率越高,比率规则在低位永真 → 钉死地板(dp5 实测 12:00-17:00 自 52/min 自残至 3/min)。软规则不砍破此位;真退化由硬规则(无下限)兜底 **/
|
||||
const SOFT_FLOOR = 25;
|
||||
|
||||
/** # 函数
|
||||
* 控制器决策(每 60s 调用):按窗口成功率/内部压力调整速率,返回决策记录或 null。
|
||||
* v5(dp6 教训):目标上调 99%;爬坡备选条件「验证码 ≤1 且短窗 ≥98.5%」修低速恢复锁死。
|
||||
* v4(dp5 教训):①硬砍连砍需「增量新失败」(全程失败数较上次砍时新增 ≥2),同一波旧失败不再反复引用;
|
||||
* ②软信号加绝对数护栏(10min 窗内验证码 ≥3 / 门壳 ≥3 才允许比率规则),低速率噪音不再触发;
|
||||
* ③软砍下限 SOFT_FLOOR,消除比率陷阱钉死地板;④爬坡:向干净水位快速回收 ×1.3/拍,水位之上 ×1.08/5 拍。
|
||||
*/
|
||||
function controlTick(): Record<string, unknown> | null {
|
||||
const wH = sumWindow(30); // 硬约束窗口:成功率即约束本身
|
||||
const tot = wH.sent;
|
||||
if (tot < 10) return null; // 信号不足,保持
|
||||
const wS = sumWindow(10); // 软信号短窗:波动退去快,防旧波滞留连砍(dp1 教训)
|
||||
const success = wH.ok / tot;
|
||||
const gateRatio = wH.attemptsTotal > 0 ? wH.gateAttempts / wH.attemptsTotal : 0;
|
||||
const gateRatioS = wS.attemptsTotal > 0 ? wS.gateAttempts / wS.attemptsTotal : 0;
|
||||
const capPctS = wS.sent > 0 ? wS.captcha / wS.sent : 0;
|
||||
const from = rate;
|
||||
let to = from;
|
||||
let reason = "hold";
|
||||
// 硬约束:首次跌破 99% 即砍;段内续砍需「进行中」证据 —— 短窗 <96%(比入段门槛更强的
|
||||
// 活跃失败信号,波尾噪声不算)且全程增量新失败 ≥2(高速率下 2 个新失败转瞬即攒,单靠
|
||||
// 增量计数仍会逐分钟级联,dp5 教训:10:24-10:28 五分钟连砍五刀 175→30/min 打灭浏览器溢出)
|
||||
const fails = tot - wH.ok;
|
||||
const allFails = totals.sent - totals.ok;
|
||||
const successS = wS.sent > 0 ? wS.ok / wS.sent : 1;
|
||||
if (success >= 0.99) degraded = false;
|
||||
const entry = !degraded && success < 0.99 && fails >= 2;
|
||||
const ongoing = degraded && wS.sent >= 10 && successS < 0.96 && allFails - failsAtLastCut >= 2;
|
||||
if (entry || ongoing) {
|
||||
degraded = true;
|
||||
failsAtLastCut = allFails;
|
||||
to = success < 0.96 ? from * 0.5 : from * 0.7;
|
||||
reason = `success=${(success * 100).toFixed(1)}%<${success < 0.96 ? 96 : 99}%(失败 ${fails}${entry ? "" : ";短窗<96%+新失败≥2"}) ×${success < 0.96 ? 0.5 : 0.7}`;
|
||||
} else {
|
||||
// 软信号:事件段迟滞 + 绝对数护栏(低速率下 1~2 个噪音即高比率,不得触发;dp5 教训)
|
||||
if (wS.captcha >= 3 && capPctS > 0.005 && !captchaWave) {
|
||||
captchaWave = true;
|
||||
const heavy = wS.captcha >= 5 && capPctS > 0.02;
|
||||
to = Math.max(SOFT_FLOOR, from * (heavy ? 0.5 : 0.7));
|
||||
reason = `验证码 ${wS.captcha} 个占比 ${(capPctS * 100).toFixed(2)}% ${heavy ? "×0.5" : "×0.7"}`;
|
||||
} else if (wS.captcha === 0) captchaWave = false;
|
||||
if (to === from && wS.gate >= 3 && gateRatioS > 0.15 && !gateWave) {
|
||||
gateWave = true;
|
||||
to = Math.max(SOFT_FLOOR, from * 0.85);
|
||||
reason = `门壳 ${wS.gate} 占比 ${(gateRatioS * 100).toFixed(1)}%>15% ×0.85`;
|
||||
} else if (gateRatioS < 0.05) gateWave = false;
|
||||
if (to === from && wS.poolEmpty > 5 && !poolEmptyWave) {
|
||||
poolEmptyWave = true;
|
||||
to = Math.max(SOFT_FLOOR, from * 0.85);
|
||||
reason = `池空 ${wS.poolEmpty} ×0.85`;
|
||||
} else if (wS.poolEmpty === 0) poolEmptyWave = false;
|
||||
}
|
||||
// 干净水位记账:30min 窗 ≥99.5% 且短窗 ≥99% 时的速率即为「实测可行」
|
||||
if (success >= 0.995 && successS >= 0.99 && wS.sent >= 10) maxCleanRate = Math.max(maxCleanRate, from);
|
||||
// 干净爬坡:退化段中不爬(30min 窗未恢复 99%);深跌后向水位快速回收 ×1.3/拍,水位之上 ×1.08/5 拍;
|
||||
// 爬坡验证码条件:主条件短窗成功率 ≥99.5%,备选「验证码 ≤1 且短窗 ≥98.5%」修 dp6 实测
|
||||
// 低速恢复锁死(涓流 1~2 个验证码在低速小分母下永远压住 99.5%,高速率下占比可忽略);
|
||||
// 短窗 0 个验证码恒可爬,或样本 ≥300 时比率已无意义(绝对护栏统一交给软砍规则)
|
||||
if (to === from && !degraded && wS.sent >= 10) {
|
||||
const cleanS = successS >= 0.995 || (wS.captcha <= 1 && successS >= 0.985);
|
||||
if (cleanS && gateRatioS < 0.02 && wS.poolEmpty === 0 && (wS.captcha === 0 || wS.sent >= 300)) {
|
||||
rampSeq++;
|
||||
if (maxCleanRate > 0 && from < maxCleanRate * 0.75) {
|
||||
to = from * 1.3 + 0.2;
|
||||
reason = `clean-fast(水位 ${maxCleanRate.toFixed(0)}) ×1.3`;
|
||||
} else if (rampSeq % RAMP_EVERY_MIN === 0) { to = from * 1.08 + 0.2; reason = `clean(success=${(successS * 100).toFixed(1)}%,gate=${(gateRatioS * 100).toFixed(1)}%) ×1.08`; }
|
||||
}
|
||||
}
|
||||
to = Math.round(Math.min(RATE_MAX, Math.max(RATE_MIN, to)) * 10) / 10;
|
||||
rate = to;
|
||||
const rec = { ts: new Date().toISOString(), from: +from.toFixed(1), to: +to.toFixed(1), reason, rampSeq, rampTick: rampSeq % RAMP_EVERY_MIN, window: { sent: tot, ok: wH.ok, gate: wH.gate, captcha: wH.captcha, poolEmpty: wH.poolEmpty, success: +(success * 100).toFixed(2), gateRatio: +(gateRatio * 100).toFixed(2), capPctS: +(capPctS * 100).toFixed(2) }, waves: { captcha: captchaWave, gate: gateWave, poolEmpty: poolEmptyWave } };
|
||||
decisions.push(rec);
|
||||
if (decisions.length > 500) decisions.splice(0, decisions.length - 500);
|
||||
if (to !== from) console.log(`[${rec.ts}] 速率 ${rec.from} → ${rec.to} req/min(${reason})`);
|
||||
return rec;
|
||||
}
|
||||
/** # 成员变量 控制器节拍序号(加速步按其取模) **/
|
||||
let rampSeq = 0;
|
||||
|
||||
// ============================================================================
|
||||
// 请求发射
|
||||
// ============================================================================
|
||||
|
||||
/** # 函数
|
||||
* 记逐请求行(紧凑:仅分类所需字段)。
|
||||
*/
|
||||
function recReq(o: Record<string, unknown>): void {
|
||||
try { fs.appendFileSync(REQ_JSONL, JSON.stringify(o) + "\n"); } catch {}
|
||||
}
|
||||
|
||||
/** # 函数
|
||||
* 发射单次搜索请求并记账(成功/门壳/空结果/验证码/池空/上游失败/网络失败)。
|
||||
* 路径与指纹取自响应体(search_parameters.runtime / search_metadata.profile.id),
|
||||
* 内部门壳压力取 timing_stages.attempts 的 outcome 统计。
|
||||
*/
|
||||
async function fire(): Promise<void> {
|
||||
const q = nextQuery();
|
||||
const t = Date.now();
|
||||
try {
|
||||
const res = await fetch(`${BASE}/api/serp/json/search?q=${encodeURIComponent(q)}&num=10`, {
|
||||
headers: { authorization: `Bearer ${TOKEN}` },
|
||||
signal: AbortSignal.timeout(240_000),
|
||||
});
|
||||
const ms = Date.now() - t;
|
||||
if (res.status === 401) { console.error("Token 无效(401),终止"); process.exit(1); }
|
||||
if (!res.ok) {
|
||||
const j = await res.json().catch(() => null) as any;
|
||||
const code = String(j?.code ?? `HTTP${res.status}`);
|
||||
// 发出与结果同刻记账:分钟桶边界处的在途请求计入完成那一桶,避免「发而不中」虚报
|
||||
cur.sent++; totals.sent++;
|
||||
if (code === "CAPTCHA_DETECTED") { cur.captcha++; totals.captcha++; }
|
||||
else if (code === "SERP_POOL_EMPTY") { cur.poolEmpty++; totals.poolEmpty++; }
|
||||
else { cur.httpErr++; totals.httpErr++; }
|
||||
recReq({ ts: new Date().toISOString(), kind: code === "CAPTCHA_DETECTED" ? "captcha" : code === "SERP_POOL_EMPTY" ? "poolEmpty" : "httpErr", code, ms });
|
||||
return;
|
||||
}
|
||||
const j = await res.json().catch(() => null) as any;
|
||||
const organic = Array.isArray(j?.organic) ? j.organic.length : 0;
|
||||
const path = j?.search_parameters?.runtime === "browser" ? "browser" : "fetch";
|
||||
const fp = String(j?.search_metadata?.profile?.id ?? "");
|
||||
const attempts: any[] = Array.isArray(j?.timing_stages?.attempts) ? j.timing_stages.attempts : [];
|
||||
const gAtt = attempts.filter(a => a?.outcome === "gate" || a?.outcome === "redirect").length;
|
||||
cur.sent++; totals.sent++;
|
||||
cur.attemptsTotal += Math.max(1, attempts.length); totals.attemptsTotal += Math.max(1, attempts.length);
|
||||
cur.gateAttempts += gAtt; totals.gateAttempts += gAtt;
|
||||
cur[path]++; totals[path]++;
|
||||
if (fp) fpMap.set(fp, (fpMap.get(fp) ?? 0) + 1);
|
||||
if (organic > 0) {
|
||||
cur.ok++; totals.ok++;
|
||||
recReq({ ts: new Date().toISOString(), kind: "ok", path, fp, ms, attempts: attempts.length, organic });
|
||||
} else if (gAtt > 0 || attempts.some(a => a?.outcome === "gate")) {
|
||||
cur.gate++; totals.gate++;
|
||||
recReq({ ts: new Date().toISOString(), kind: "gate", path, fp, ms, attempts: attempts.length, gateAttempts: gAtt });
|
||||
} else {
|
||||
cur.empty++; totals.empty++;
|
||||
recReq({ ts: new Date().toISOString(), kind: "empty", path, fp, ms, attempts: attempts.length, organic: 0 });
|
||||
}
|
||||
} catch (e) {
|
||||
cur.sent++; totals.sent++;
|
||||
cur.netErr++; totals.netErr++;
|
||||
recReq({ ts: new Date().toISOString(), kind: "netErr", error: String(e).slice(0, 140), ms: Date.now() - t });
|
||||
}
|
||||
}
|
||||
|
||||
/** # 函数
|
||||
* worker 主循环:取令牌 → 发射,直到结束时刻。
|
||||
*/
|
||||
async function worker(endAt: number): Promise<void> {
|
||||
while (Date.now() < endAt) {
|
||||
await acquire();
|
||||
if (Date.now() >= endAt) break;
|
||||
await fire();
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// 快照与汇总
|
||||
// ============================================================================
|
||||
|
||||
/** # 函数
|
||||
* 组装快照(窗口+全程+路径占比+指纹 Top20+决策尾部)。
|
||||
*/
|
||||
function snapshot(): Record<string, unknown> {
|
||||
const w = sumWindow();
|
||||
const topFps = [...fpMap.entries()].sort((a, b) => b[1] - a[1]).slice(0, 20).map(([fp, n]) => ({ fp, n }));
|
||||
return {
|
||||
at: new Date().toISOString(),
|
||||
hoursElapsed: +((Date.now() - startedAt) / 3600_000).toFixed(2),
|
||||
rateReqMin: +rate.toFixed(1),
|
||||
window: { ...w, successPct: w.sent ? +((w.ok / w.sent) * 100).toFixed(2) : null, gateAttemptPct: w.attemptsTotal ? +((w.gateAttempts / w.attemptsTotal) * 100).toFixed(2) : null },
|
||||
totals: { ...totals, successPct: totals.sent ? +((totals.ok / totals.sent) * 100).toFixed(2) : null, gateAttemptPct: totals.attemptsTotal ? +((totals.gateAttempts / totals.attemptsTotal) * 100).toFixed(2) : null },
|
||||
pathSplit: { fetch: totals.fetch, browser: totals.browser },
|
||||
topFps,
|
||||
decisions: decisions.slice(-20),
|
||||
};
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// 主流程:并行 worker + 每分钟聚合 + 控制器 + 每 5min 快照;到时终局收尾
|
||||
// ============================================================================
|
||||
|
||||
const startedAt = Date.now();
|
||||
const endAt = startedAt + HOURS * 3600_000;
|
||||
fs.writeFileSync(JSONL, "");
|
||||
fs.writeFileSync(REQ_JSONL, "");
|
||||
console.log(`开始:label=${LABEL} hours=${HOURS} rate0=${RATE0} 区间=[${RATE_MIN},${RATE_MAX}] workers=${WORKERS} 出口=json(auto 双路径)`);
|
||||
|
||||
for (let i = 0; i < WORKERS; i++) void worker(endAt);
|
||||
|
||||
/** # 成员变量 每分钟定时器:滚动窗口推进 + 控制器决策 + 聚合行落盘 **/
|
||||
const minuteTimer = setInterval(() => {
|
||||
const row = { ts: new Date().toISOString(), rate: +rate.toFixed(1), ...cur };
|
||||
try { fs.appendFileSync(JSONL, JSON.stringify(row) + "\n"); } catch {}
|
||||
window.push({ ts: row.ts, c: cur });
|
||||
if (window.length > WINDOW_MIN) window.shift();
|
||||
cur = newCounter();
|
||||
controlTick();
|
||||
}, 60_000);
|
||||
|
||||
/** # 成员变量 每 5 分钟快照与控制台摘要 **/
|
||||
const snapTimer = setInterval(() => {
|
||||
const snap = snapshot();
|
||||
fs.writeFileSync(LIVE, JSON.stringify(snap, null, 2));
|
||||
const t = snap.totals as any; const w = snap.window as any;
|
||||
console.log(`[${snap.at}] ${(snap.hoursElapsed as number).toFixed(1)}h rate=${snap.rateReqMin}/min | 窗口 成功率=${w.successPct}% 门壳尝试=${w.gateAttemptPct}% sent=${w.sent} (ok=${w.ok} gate=${w.gate} captcha=${w.captcha} poolEmpty=${w.poolEmpty} httpErr=${w.httpErr} netErr=${w.netErr}) | 全程 成功率=${t.successPct}% 路径 fetch=${snap.pathSplit.fetch}/browser=${snap.pathSplit.browser}`);
|
||||
}, 5 * 60_000);
|
||||
|
||||
const endTimer = setTimeout(() => {
|
||||
clearInterval(minuteTimer); clearInterval(snapTimer);
|
||||
const snap = snapshot();
|
||||
fs.writeFileSync(`${DATA}${LABEL}-final.json`, JSON.stringify(snap, null, 2));
|
||||
console.log("测试结束,final 已写:" + `${DATA}${LABEL}-final.json`);
|
||||
process.exit(0);
|
||||
}, HOURS * 3600_000);
|
||||
endTimer.unref?.();
|
||||
Reference in New Issue
Block a user