1.77.6: SERP sorry-重定向按验证码语义(不换号+15min冷却);浏览池保温释放跳过+60min持有上限;速率-门壳实验与监控工具

This commit is contained in:
BigCode
2026-10-29 15:54:37 +08:00
commit 05d9462414
25 changed files with 3973 additions and 0 deletions
+74
View File
@@ -0,0 +1,74 @@
# ProfileMax(CloakB + ipmart.io-US)并发常驻浏览器压测报告
- 日期:2026-10-09
- 服务器:OVH-FR `37.187.92.169`(8 核 / 31GB 内存 / /dev/shm 16G / 同机还跑 mysql、clickhouse、plausible 等其它服务)
- 链路:ProfileMax admin(127.0.0.1:5310)→ 节点 OVH-FR(node :3106,systemd 直跑,无 Docker)
- 测试身份:MCP 用户 phptits(id=2);Profile 全部归属该用户
- 方式:节点 API `POST /profiles/:id/lease(ttlMs=2h)` → `POST /profiles/:id/browser/start` → CDP `/json/new` + WS `Page.navigate` 开真实页面;每 20s 扫 `/proc` 按 `--user-data-dir` 聚合每 Profile 进程树
- 内存口径:**PSS**(smaps_rollup,去共享重复);RSS 会把共享页重复计数(报告注明)
## 配置
| 项 | 值 |
|---|---|
| 浏览器内核 | CloakB(cloak,146.0.7680.177.5) |
| 代理 | `ipmart.io-US`(socks5 gateway.ipmart.io:9999,用户名含 `{sid}` → 每指纹独立 US 住宅 sticky 出口) |
| 标签 | `SERP` |
| 模式 | 无头(用户中途指示全部改无头) |
| 保活 | `BROWSER_IDLE_TIMEOUT=0`(禁用 10 分钟空闲自动关闭)+ 每 Profile 2 小时 lease |
| Profile 池 | 10,002 个(`sz-serp-*`,CloakB + ipmart.io-US + SERP) |
## 结果总览
| 指标 | 测试1:3 标签(google/yahoo/x) | 测试2:1 标签(google) |
|---|---|---|
| **同时常驻浏览器上限** | **82 个** | **189 个** |
| 常驻时长 | ≥40 分钟(最老的) | ≥40 分钟(最老的) |
| 系统峰值 PSS | 25.0GB | 25.5GB |
| 内存最低可用(avail) | **542MB(耗尽)** | 2793MB |
| swap 峰值 | **9.2GB 全满(耗尽)** | 7.6GB(触发停止线) |
| load 峰值(8 核) | **118(kswapd 抢转,CPU 也耗尽)** | 46 |
| 总进程/线程数峰值 | 1280 进程 | 1702 进程 |
| 每浏览器进程数(均值) | 15.1(重载页到 50+) | 9.0 |
| 每 Profile 内存(稳定期) | p50=125 / p90=906 / max=1560 MB;总量均值 ≈305MB | p50=152 / p90=158 / max=179 MB;总量均值 ≈129MB |
| 每 Profile swap | p50=20MB | p50=2MB |
| 每 Profile CPU(约 40 分钟常驻) | 均值 ≈164s | 均值 ≈46s |
| 每 Profile 磁盘(user-data) | p50=25MB / max=115MB | p50=10MB / max=115MB |
**结论:当前服务器(31GB 内存,其余 3.7GB 被 mysql/clickhouse/系统占用)**
- 3 标签(轻量页)可常驻 **约 80 个** Profile;3 标签里出现 yahoo/x 重页时重载 Profile 会拖到 1.5GB/个,是方差的主因。
- 1 标签(google)可常驻 **约 190 个** Profile(停止在 swap 7.2GB 保守线,真实极限约 200±10)。
- 内存是第一瓶颈,PSS 是衡量口径;3 标签场景 CPU(kswapd + 软渲染)会成为第二瓶颈。
## 每轮细节
### 测试1(3 标签)
- 发射阶段:并发 4,fd 修复后从 58 → 82 一路推进;`MemAvailable<1.5GB` + swap 满后页面加载/启动开始超时与抖动(load 118),停止发射。
- 残留 82 个全部驻留至数据采集完成(带 OOM 保护策略:浏览器主进程 oom_score_adj=200,其它服务进程=-600,整个压测期间 mysqld/clickhouse/PG 等零损伤)。
- 重载 Profile(x.com/yahoo 富页)实测会把进程树拉到 50+ 进程 / 1.5GB PSS。
### 测试2(1 标签)
- 清场回收后重跑同一 10002 池;发射 25 分钟到 189 个,唯一失败 1 次;swap 7458MB ≥ 7200MB 触发保守停止(avail 仍有 3.3GB,真实极限略高)。
- 单标签 google 的内存分布极稳(129MB 均值,极差 179)——google 轻页没有长尾。
## 过程中修复的 Docker/系统/产品问题
1. **`fs.file-max=65535` 系统级文件句柄上限**——58 个浏览器即吃满(65216/65535),新 chrome 启动静默失败 → 启动超时。已放宽:`fs.file-max=2097152`(已写入 /etc/sysctl.conf)。
2. **inotify 额度**:max_user_instances 128→1024、max_user_watches 261756→524288(chrome 每 Profile 多个 watcher,128 实例上限也会卡死启动)。
3. **swap 仅 1GB**:追加 8G swapfile `/testswap`(priority 10),现共 9GB,保留作为长期优化(防 OOM 误伤同机其它服务)。
4. **10 分钟空闲自动关闭**(节点 `BROWSER_IDLE_TIMEOUT` 默认 600s):测试期间置 0 禁用,已恢复默认。
5. 有头模式(浏览器经 VNC/WebDesk 跑在每 Profile Xvfb 上,llvmpipe 软渲染)在批量场景会瞬间把 8 核打满(load 96),**批量常驻必须用无头**;webdesk 二进制与 x11vnc 测试期间暂移、已恢复。
6. 产品坑(记录进记忆):节点有「UI 误关自动复活」(SIGKILL 杀浏览器会被复活,重启风暴);停浏览器必须走 API 显式 stop(closing 路径)。`browser/start` 返回的 `processId` 是包装进程(拉起 chrome 后退出),不能当存活判据;存活判据应按 `--user-data-dir` 进程树聚合。响应体统一 `{success,data}` 包装。
7. admin `DELETE /api/profiles/:id` 只删库行、**不清理节点侧 Profile 数据**(节点清理只在转移链路触发)——压测清理需要另对节点调 `DELETE /profiles/:id?force=true`(本次已处理)。
## 清理与恢复
- 测试 Profile:admin 侧 10,002 个全部删除;节点侧已启动过的 190 个数据目录全部强删(`force=true`);节点现存 139 个目录均为合法 Profile(未动)。
- 配置:webdesk 二进制、x11vnc、`BROWSER_IDLE_TIMEOUT`、全部恢复;`fs.file-max`/inotify/8G swapfile 保留(系统优化项)。
- 终态:节点在线(0 running)、admin 正常、内存回收(avail 29GB)、swap 余量 6.2GB。
## 复现物
- 服务器 `/root/serpmax-stress/`:`create-profiles.ts / run-stress.ts / sampler.ts / stop-browsers.ts / report.ts / cleanup-profiles.ts / node-wipe-stress.ts`
- 数据:`data/test{1,2}-summary.json|md`、`data/test1-stats-final.jsonl`(154 轮)、`data/test2-stats.jsonl`(117 轮)、各轮 run 日志
- 本地:`/home/bigcode/data/dev/bun/SerpMax/stress/REPORT.md`(本文件)+ 同套脚本与 summary JSON
+63
View File
@@ -0,0 +1,63 @@
/** # 脚本说明
* 压测收尾:删除全部测试 Profile(SERP 标签)。
* 分页拉取(限 500)→ 并发 DELETE /api/profiles/:id(admin 级联:节点数据删除/扩展绑定清理)。
* 404 视为已删;最后复查 admin DB 与节点 profile 目录残留数。
* 用法:bun cleanup-profiles.ts
*/
const fs = require("fs") as typeof import("fs");
const ADMIN = "http://127.0.0.1:5310";
const DATA = `${new URL(".", import.meta.url).pathname}data`;
/** # 函数:docker exec 查 admin DB 单值 **/
function sqlOne(sql: string): string {
const p = Bun.spawnSync(["docker", "exec", "serpmax-pg", "psql", "-U", "postgres", "-d", "serpmax_admin", "-tAc", sql], { stdout: "pipe", stderr: "pipe" });
const out = p.stdout.toString().trim();
if (p.exitCode !== 0 || !out) throw new Error(`SQL 失败`);
return out;
}
const TOKEN = sqlOne("SELECT api_token FROM users WHERE id=2");
// 分页拉取 SERP 标签 Profile(标签过滤走服务端,兜底本地再筛)
const fps: { id: string; fp: string }[] = [];
let page = 1;
while (page <= 40) {
const res = await fetch(`${ADMIN}/api/profiles?limit=500&page=${page}`, {
headers: { authorization: `Bearer ${TOKEN}` },
signal: AbortSignal.timeout(30000),
});
if (!res.ok) throw new Error(`列表 ${res.status}`);
const j: any = await res.json();
const data = j.data ?? j;
const items: any[] = data.items ?? [];
for (const it of items) {
const tags: string[] = it.tags ?? [];
if (tags.includes("SERP") && (it.name ?? "").startsWith("sz-serp-")) {
fps.push({ id: it.id, fp: it.fingerprint ?? "" });
}
}
const total = data.total ?? 0;
if (items.length < 500 || fps.length >= total) break;
page++;
}
console.log(`[cleanup] 目标 ${fps.length} 个`);
let deleted = 0, skipped = 0;
let cursor = 0;
const workers = 16;
async function worker(): Promise<void> {
while (cursor < fps.length) {
const i = cursor++;
const t = fps[i];
if (!t.id) { skipped++; continue; }
try {
const res = await fetch(`${ADMIN}/api/profiles/${t.id}`, { method: "DELETE", headers: { authorization: `Bearer ${TOKEN}` }, signal: AbortSignal.timeout(60000) });
if (res.ok || res.status === 404) deleted++;
else { skipped++; if (skipped < 10) console.error(`[cleanup] 删 ${t.fp} 状态 ${res.status}`); }
} catch (e) { skipped++; }
if (deleted % 500 === 0 && deleted > 0) console.log(`[cleanup] deleted=${deleted} skipped=${skipped}`);
}
}
await Promise.all(Array.from({ length: workers }, () => worker()));
console.log(`[cleanup] 完成 deleted=${deleted} skipped=${skipped}`);
console.log(`[cleanup] admin 残留 SERP=${sqlOne("SELECT count(*) FROM profiles WHERE tags @> ARRAY['SERP']::text[]")}`);
console.log(`[cleanup] 节点目录残留=${Bun.spawnSync(["bash", "-c", "ls /opt/serpmax/node/data/profiles/ | wc -l"], { stdout: "pipe" }).stdout.toString().trim()}`);
+84
View File
@@ -0,0 +1,84 @@
/** # 脚本说明
* 压测准备:批量在 admin(127.0.0.1:5310)创建 N 个 Profile。
* - 归属当前 MCP 用户(users.id=2,phptits),Token 由脚本直取 DB 不落盘不回显
* - 浏览器内核 CloakB(cloak)、代理 ipmart.io-US、标签 SERP、节点 OVH-FR
* - 输出 data/pool.json:[{ id, fingerprint, name }]
*
* 用法:bun create-profiles.ts [N=10000]
*/
import { $ } from "bun";
/** # 成员变量 admin 基址 **/
const ADMIN = "http://127.0.0.1:5310";
/** # 成员变量 目标代理(pressure 测试固定 ipmart.io-US,{sid} 每指纹独立出口) **/
const PROXY_ID = "19e3ab5c-c61e-4936-8a7f-d78f6635f57c";
/** # 成员变量 归属节点 **/
const NODE = "OVH-FR";
/** # 成员变量 标签 **/
const TAGS = ["SERP"];
const N = Number(process.argv[2] ?? 10000);
/** # 函数
* 通过 docker exec 查询 admin DB,取单值文本。
* 参数 sql:完整 SQL 语句
* 返回值:第一行第一列,去尾部换行
*/
function sqlOne(sql: string): string {
const p = Bun.spawnSync(["docker", "exec", "serpmax-pg", "psql", "-U", "postgres", "-d", "serpmax_admin", "-tAc", sql], { stdout: "pipe", stderr: "pipe" });
const out = p.stdout.toString().trim();
if (p.exitCode !== 0 || !out) throw new Error(`SQL 失败: ${p.stderr.toString().slice(0, 300)}`);
return out;
}
/** # 成员变量 admin API Token(phptits,id=2;仅内存) **/
const TOKEN = sqlOne("SELECT api_token FROM users WHERE id=2");
const results: { id: string; fingerprint: string; name: string }[] = [];
let okCount = 0;
let errCount = 0;
const t0 = Date.now();
/** # 函数
* 创建单个 Profile,成功返回 {id, fingerprint, name},失败返回 null。
*/
async function createOne(i: number): Promise<{ id: string; fingerprint: string; name: string } | null> {
const name = `sz-serp-${String(i + 1).padStart(5, "0")}`;
try {
const res = await fetch(`${ADMIN}/api/profiles`, {
method: "POST",
headers: { "content-type": "application/json", authorization: `Bearer ${TOKEN}` },
// 浏览器外观随机浅色/深色(用户 2026-10-20 要求 SERP 号不再全浅色)
body: JSON.stringify({ name, browserKind: "cloak", browserTheme: Math.random() < 0.5 ? "light" : "dark", proxyId: PROXY_ID, tags: TAGS, node: NODE }),
signal: AbortSignal.timeout(20000),
});
if (res.status === 409) return null; // 指纹冲突(极小概率),跳过
if (!res.ok) throw new Error(`HTTP ${res.status} ${(await res.text()).slice(0, 120)}`);
const j = (await res.json()) as any;
const d = (j.data ?? j) as { id?: string; fingerprint?: string };
okCount++;
return { id: d.id ?? "", fingerprint: d.fingerprint ?? d.id ?? "", name };
} catch (e) {
errCount++;
if (errCount < 20) console.error(`create fail ${i}: ${String(e).slice(0, 140)}`);
return null;
}
}
/** 主循环:16 并发创建 N 个 */
const workers = 16;
let cursor = 0;
async function worker() {
while (cursor < N) {
const i = cursor++;
const r = await createOne(i);
if (r) results.push(r);
}
}
await Promise.all(Array.from({ length: workers }, () => worker()));
await Bun.write(new URL(import.meta.url).pathname.replace(/[^/]+$/, "") + `data/pool.json`, JSON.stringify(results));
console.log(`done: created=${okCount} failed=${errCount} elapsed=${((Date.now() - t0) / 1000).toFixed(1)}s -> data/pool.json`);
+16
View File
@@ -0,0 +1,16 @@
/** # 调试脚本:复算杀伤误判(服务器上一次性使用) **/
const fs = require("fs");
const launched = JSON.parse(await Bun.file("data/test1-launched.json").text());
const alive = new Set<string>();
for (const pid of fs.readdirSync("/proc").filter((d) => /^\d+$/.test(d))) {
try {
const cmd = fs.readFileSync(`/proc/${pid}/cmdline`, "utf8");
const m = /--user-data-dir=\S*\/profiles\/([\w.]+)\/browser/.exec(cmd);
if (m) alive.add(m[1]);
} catch { /* ignore */ }
}
console.log("alive trees:", alive.size);
const dead = launched.filter((e: any) => Date.now() - Date.parse(e.startedAt) > 60000 && !alive.has(e.fp));
console.log("dead count:", dead.length, dead.slice(0, 5).map((e: any) => e.fp));
const sample = launched[0] && [...alive].slice(0, 5);
console.log("alive sample:", sample);
+35
View File
@@ -0,0 +1,35 @@
/** # 脚本说明
* 节点侧清理:按 pool.json 清单对节点逐个 DELETE /profiles/:fp?force=true
* (admin API 删除不清节点数据;本脚本只处理压测池内的指纹,不动合法 Profile)。
* 用法:bun node-wipe-stress.ts
*/
const fs = require("fs") as typeof import("fs");
const NODE = "http://127.0.0.1:3106";
const DATA = `${new URL(".", import.meta.url).pathname}data`;
const NODE_KEY = /^NODE_KEY=(.*)$/m.exec(fs.readFileSync("/opt/serpmax/node/.env", "utf8"))?.[1]?.trim() ?? "";
const pool = (await Bun.file(`${DATA}/pool.json`).json()) as { fingerprint: string }[];
const fps = [...new Set(pool.map(p => p.fingerprint).filter(Boolean))];
console.log(`[wipe] 池内指纹 ${fps.length} 个`);
let ok = 0, miss = 0, err = 0;
const workers = 12;
let cursor = 0;
async function worker(): Promise<void> {
while (cursor < fps.length) {
const i = cursor++;
const fp = fps[i];
try {
const res = await fetch(`${NODE}/profiles/${fp}?force=true`, {
method: "DELETE",
headers: { "x-node-key": NODE_KEY },
signal: AbortSignal.timeout(30000),
});
if (res.ok) ok++;
else if (res.status === 404) miss++;
else { err++; if (err < 8) console.error(`[wipe] ${fp} -> ${res.status}`); }
} catch { err++; }
}
}
await Promise.all(Array.from({ length: workers }, () => worker()));
console.log(`[wipe] ok=${ok} miss(本就不在)=${miss} err=${err}`);
const left = Bun.spawnSync(["bash", "-c", "ls /opt/serpmax/node/data/profiles/ | wc -l"], { stdout: "pipe" }).stdout.toString().trim();
console.log(`[wipe] 节点目录剩余=${left}`);
+89
View File
@@ -0,0 +1,89 @@
/** # 脚本说明
* 存量 SERP Profile 浏览器外观随机化:标签含 SERP 的 Profile 主题随机浅色/深色(50/50)。
* 两步:① admin DB 批量 UPDATE browser_theme;② 逐个 PATCH 节点
* /profiles/:指纹/browser-env {theme}(下一次浏览器启动生效;节点侧 ID 统一为指纹)。
* 认证:节点侧 x-node-key = nodes.key;节点寻址 nodes.base_url。
* 幂等可重跑(每次重新随机)。
* 用法:在 admin 机上执行 bun randomize-themes.ts [并发=24]
* (bun 自动加载 admin/.env 的 DATABASE_URL;本地默认 serpmax_admin@127.0.0.1:5432)
*/
import { SQL } from "bun";
/** # 成员变量 admin Postgres 连接(DATABASE_URL 覆盖;默认本机 serpmax_admin) **/
const sql = new SQL(process.env.DATABASE_URL ?? "postgres://postgres:postgres@127.0.0.1:5432/serpmax_admin");
/** # 成员变量 节点 PATCH 并发数 **/
const CONCURRENCY = Math.max(1, Number(process.argv[2] ?? 24));
/** # 函数
* 主流程:拉取 SERP Profile 与节点信息 → 每号随机主题 → 批量落 admin DB → 并发推节点。
*/
async function main(): Promise<void> {
// 节点寻址与密钥(name → base_url / key)
const nodes = await sql`SELECT name, base_url, key FROM nodes`;
const nodeById = new Map<string, { baseUrl: string; key: string }>();
for (const n of nodes) {
nodeById.set(String(n.name), { baseUrl: String(n.base_url ?? "").replace(/\/$/, ""), key: String(n.key ?? "") });
}
// SERP Profile 池(admin 侧 ID 为 UUID;指纹是节点侧 ID)
const rows = await sql`SELECT id::text AS id, fingerprint, node, browser_theme FROM profiles WHERE tags @> ARRAY['SERP']::text[]`;
console.log(`[randomize] SERP Profile 共 ${rows.length} 个`);
if (!rows.length) return;
// 逐号随机主题并按目标节点分组
type Item = { id: string; fingerprint: string; theme: string };
const byNode = new Map<string, Item[]>();
for (const r of rows) {
const item: Item = {
id: String(r.id),
fingerprint: String(r.fingerprint ?? r.id),
theme: Math.random() < 0.5 ? "light" : "dark",
};
const list = byNode.get(String(r.node ?? "")) ?? [];
list.push(item);
byNode.set(String(r.node ?? ""), list);
}
// ① admin DB 批量落库(2000/批,UPDATE...FROM VALUES)
let dbUpdated = 0;
for (const items of byNode.values()) {
for (let i = 0; i < items.length; i += 2000) {
const chunk = items.slice(i, i + 2000);
const params = chunk.flatMap((it) => [it.id, it.theme]);
const ph = chunk.map((_, i) => `($${i * 2 + 1}, $${i * 2 + 2})`).join(",");
await sql.unsafe(`UPDATE profiles AS p SET browser_theme = v.theme FROM (VALUES ${ph}) AS v(id, theme) WHERE p.id::text = v.id`, params);
dbUpdated += chunk.length;
}
}
console.log(`[randomize] admin DB 已更新 ${dbUpdated} 个`);
// ② 并发推节点(失败不阻塞,最后汇总)
let okCount = 0;
let failCount = 0;
const cursor = { n: 0 };
const all = [...byNode.entries()].flatMap(([node, items]) => items.map((it) => ({ node, ...it })));
async function worker(): Promise<void> {
while (cursor.n < all.length) {
const it = all[cursor.n++];
const target = nodeById.get(it.node);
if (!target?.baseUrl) { failCount++; continue; }
try {
const res = await fetch(`${target.baseUrl}/profiles/${encodeURIComponent(it.fingerprint)}/browser-env`, {
method: "PATCH",
headers: { "content-type": "application/json", "x-node-key": target.key },
body: JSON.stringify({ theme: it.theme }),
signal: AbortSignal.timeout(8000),
});
if (res.ok) okCount++; else failCount++;
} catch {
failCount++;
}
if ((okCount + failCount) % 1000 === 0) console.log(`[randomize] 节点推送 ${okCount + failCount}/${all.length}`);
}
}
await Promise.all(Array.from({ length: CONCURRENCY }, () => worker()));
console.log(`[randomize] 完成:节点推送成功 ${okCount},失败 ${failCount}`);
}
await main();
+125
View File
@@ -0,0 +1,125 @@
/** # 脚本说明
* 压测数据汇总(服务器上运行):
* 输入 data/<label>-stats.jsonl(采样器输出)+ data/<label>-launched.json(发射清单)
* 输出:
* - 峰值/残存浏览器数、发射总量、系统峰值(avail 最低/swap 最高/PSS 最高)
* - 每 Profile 稳定期均值(PSS/swap/CPU/进程数)与分布(p50/p90/p99/max)
* - 每 Profile 磁盘占用(du data/profiles/<fp>/browser)
* - top 明细表(MD 片段)→ data/<label>-summary.json + data/<label>-summary.md
* 用法:bun report.ts --label test1 [--tail 10]
*/
const fs = require("fs") as typeof import("fs");
import { $ } from "bun";
const args = process.argv.slice(2);
function argOf(k: string, d: string): string { const i = args.indexOf(k); return i >= 0 && i + 1 < args.length ? args[i + 1] : d; }
const LABEL = argOf("--label", "test1");
const TAILN = Number(argOf("--tail", "10"));
const DATA = `${new URL(".", import.meta.url).pathname}data`;
/** # 成员变量 采样行类型 **/
type Sample = { ts: string; browsers: number; totalPssMB: number; totalRssMB: number; totalSwapMB: number; totalProcs: number; availMB: number; swapUsedMB: number; load: number[]; detail: any[] };
const lines = fs.readFileSync(`${DATA}/${LABEL}-stats.jsonl`, "utf8").trim().split("\n").filter(Boolean).map((l) => JSON.parse(l) as Sample);
console.log(`samples=${lines.length}`);
const peak = {
browsers: Math.max(...lines.map(s => s.browsers)),
pssMB: Math.max(...lines.map(s => s.totalPssMB)),
minAvailMB: Math.min(...lines.map(s => s.availMB)),
maxSwapMB: Math.max(...lines.map(s => s.swapUsedMB)),
maxLoad: Math.max(...lines.map(s => s.load?.[0] ?? 0)),
maxProcs: Math.max(...lines.map(s => s.totalProcs)),
};
const last = lines[lines.length - 1];
console.log(`peak: ${JSON.stringify(peak)}`);
console.log(`last: browsers=${last.browsers} pss=${Math.round(last.totalPssMB)}MB avail=${Math.round(last.availMB)}MB`);
/** # 成员变量 稳定期尾窗样本(后 N 个) **/
const tail = lines.slice(-TAILN);
/** # 函数:取多组数分布(p50/p90/p99/max) **/
function dist(values: number[]): { p50: number; p90: number; p99: number; max: number } | null {
if (!values.length) return null;
const v = [...values].sort((a, b) => a - b);
const q = (p: number) => v[Math.min(v.length - 1, Math.floor(p * (v.length - 1)))];
return { p50: q(0.5), p90: q(0.9), p99: q(0.99), max: v[v.length - 1] };
}
/** # 函数:收集每 Profile 稳定期聚合(窗口内出现的样本取均值;现存取最后样本) **/
const per = new Map<string, { fp: string; pss: number[]; swap: number[]; procs: number[]; cpu: number[]; udd: string; lastTs: string }>();
for (const s of tail) {
for (const t of s.detail) {
const e = per.get(t.fp) ?? { fp: t.fp, pss: [], swap: [], procs: [], cpu: [], udd: t.userDataDir ?? "", lastTs: s.ts };
e.pss.push(t.pssKB / 1024);
e.swap.push((t.swapPssKB ?? t.swapKB) / 1024);
e.procs.push(t.procs);
e.cpu.push((t.cpuDeltaMs ?? t.cpuMs) / 1000);
e.lastTs = s.ts;
e.udd = e.udd || t.userDataDir || "";
per.set(t.fp, e);
}
}
const avg = (a: number[]) => (a.length ? a.reduce((x, y) => x + y, 0) / a.length : 0);
const rows = [...per.values()].map(e => ({
fp: e.fp,
pssMB: avg(e.pss),
swapMB: avg(e.swap),
procs: avg(e.procs),
cpuSec: avg(e.cpu),
}));
/** # 成员变量 每 Profile 磁盘占用(du user-data 目录) **/
const disk: Record<string, number> = {};
try {
const paths = rows.map(r => r.udd).filter(Boolean).slice(0, 400);
const chunks: { stdout: string }[] = [];
for (let i = 0; i < paths.length; i += 200) {
const p = Bun.spawnSync(["du", "-sb", ...paths.slice(i, i + 200)], { stdout: "pipe" });
chunks.push(p as unknown as { stdout: string });
}
for (const p of chunks) {
for (const line of (p as any).stdout.toString().split("\n")) {
const m = /^(\d+)\t(.+)$/.exec(line);
if (m) disk[m[2].split("/").slice(-2)[0]] = Number(m[1]);
}
}
console.log(`disk rows=${Object.keys(disk).length}`);
} catch (e) { console.error("du 失败", e); }
/** # 成员变量 汇总分布 **/
const dPss = dist(rows.map(r => r.pssMB));
const dSwap = dist(rows.map(r => r.swapMB));
const dProcs = dist(rows.map(r => r.procs));
const dCpu = dist(rows.map(r => r.cpuSec));
const dDisk = dist(rows.map(r => disk[r.fp] ?? 0));
const aliveFps = new Set(last.detail.map(t => t.fp));
const summary = {
label: LABEL,
samples: lines.length,
tailSamples: tail.length,
peak,
last: { ts: last.ts, browsers: last.browsers, pssMB: Math.round(last.totalPssMB), availMB: Math.round(last.availMB), swapUsedMB: Math.round(last.swapUsedMB), load: last.load, procs: last.totalProcs },
aliveRows: rows.length,
distPerProfile: { pssMB: dPss, swapMB: dSwap, procs: dProcs, cpuSec: dCpu, diskMB: dDisk ? { p50: dDisk.p50 / 1048576, p90: dDisk.p90 / 1048576, max: dDisk.max / 1048576 } : null },
top10: rows.sort((a, b) => b.pssMB - a.pssMB).slice(0, 10).map(r => ({ fp: r.fp, pssMB: Math.round(r.pssMB), swapMB: Math.round(r.swapMB), procs: Math.round(r.procs), diskMB: Math.round((disk[r.fp] ?? 0) / 1048576) })),
};
await Bun.write(`${DATA}/${LABEL}-summary.json`, JSON.stringify(summary, null, 2));
const md = [
`## ${LABEL} 汇总`,
`- 采样 ${lines.length} 轮(尾窗 ${tail.length} 轮);峰值浏览器 **${peak.browsers}**;残存 **${summary.last.browsers}**`,
`- 峰值系统:PSS ${Math.round(peak.pssMB)}MB / avail 最低 ${Math.round(peak.minAvailMB)}MB / swap 最高 ${Math.round(peak.maxSwapMB)}MB / load ${Math.round(peak.maxLoad)} / 总进程 ${peak.maxProcs}`,
`- 每 Profile(稳定期均值,MB):PSS p50=${dPss?.p50.toFixed(0)} p90=${dPss?.p90.toFixed(0)} p99=${dPss?.p99.toFixed(0)} max=${dPss?.max.toFixed(0)}`,
`- 每 Profile:swapMB p50=${dSwap?.p50.toFixed(0)} / 进程数 avg=${(rows.reduce((a, b) => a + b.procs, 0) / rows.length).toFixed(1)} / CPU 秒 avg=${(rows.reduce((a, b) => a + b.cpuSec, 0) / rows.length).toFixed(0)}`,
`- 每 Profile 磁盘(user-data 目录,MB):p50=${dDisk ? (dDisk.p50 / 1048576).toFixed(0) : "-"} max=${dDisk ? (dDisk.max / 1048576).toFixed(0) : "-"}`,
"",
markdownTable(summary.top10),
].join("\n");
await Bun.write(`${DATA}/${LABEL}-summary.md`, md);
console.log(md);
/** # 函数:top10 markdown 表 **/
function markdownTable(rows: any[]): string {
const head = "| fp | PSS MB | swap MB | procs | disk MB |\n|---|---|---|---|---|";
return [head, ...rows.map(r => `| ${r.fp} | ${r.pssMB} | ${r.swapMB} | ${r.procs} | ${r.diskMB} |`)].join("\n");
}
+283
View File
@@ -0,0 +1,283 @@
/** # 脚本说明
* 压测发射器(在 37 服务器以 root 运行):
* 从 pool.json 逐个(并发 workers)执行:
* 节点导入(幂等)→ POST lease(ttlMs=2h,规避 10 分钟空闲自动关闭)
* → POST browser/start → 按标签数经 CDP /json/new 开真实页面。
* 直到「资源耗尽」触发:MemAvailable<1.5GB / swap已用≥7.2GB /
* 连续启动失败≥10 / 已开浏览器被 OOM 杀≥10 / 达到 --max 封顶。
* 触发后停止发射(已开浏览器保持常驻),结论写 data/<label>-stop.json。
* 断点续跑:进度在 data/<label>-launched.jsonl,重启脚本会跳过已开指纹。
*
* 用法:bun run-stress.ts --pool data/pool.json --tabs 3 --label test1 [--max 500] [--concurrency 6]
*/
/** # 成员变量 辅助:进程内同步读文本(供 /proc 扫描) **/
const fs = require("fs") as typeof import("fs");
/** # 成员变量 服务基址(同机) **/
const ADMIN = "http://127.0.0.1:5310";
const NODE = "http://127.0.0.1:3106";
/** # 成员变量 基目录与数据目录 **/
const BASE = new URL(".", import.meta.url).pathname;
const DATA = `${BASE}data`;
/** # 成员变量 3 标签站点;1 标签测试只取第一个(google) **/
const URLS3 = ["https://www.google.com/", "https://www.yahoo.com/", "https://x.com/"];
/** # 成员变量 Lease TTL:2 小时(远大于 10 分钟空闲关闭窗口) **/
const LEASE_TTL_MS = 7_200_000;
/** # 成员变量 资源耗尽触发阈值(连续失败放宽,配合冷却防瞬时抖动误停) **/
const MEM_FLOOR_MB = 1500;
const SWAP_CEIL_MB = 7200;
const MAX_CONSEC_FAIL = 20;
const MAX_OOM_KILLED = 10;
/** # 成员变量 发射节流:load1 超过核数×3(37 为 8 核 → 24)暂停发射 **/
const LOAD_GATE = 24;
// ---- 参数解析 ----
const args = process.argv.slice(2);
function argOf(k: string, d: string): string { const i = args.indexOf(k); return i >= 0 ? args[i + 1] : d; }
const TABS = Number(argOf("--tabs", "3"));
const LABEL = argOf("--label", "test1");
const POOL_FILE = argOf("--pool", `${DATA}/pool.json`);
const MAX = Number(argOf("--max", "0"));
const CONCURRENCY = Number(argOf("--concurrency", "6"));
/** # 成员变量 本轮要开的真实站点列表 **/
const OPEN_URLS = TABS === 1 ? URLS3.slice(0, 1) : URLS3;
type Entry = {
fp: string; name: string; startedAt: string;
cdpPort: number | null; mainPid: number | null; tabs: string[];
leaseId?: string; userDataDir?: string;
};
// ---- 密钥(仅内存,不落盘不回显) ----
/** # 函数:通过 docker exec 查询 admin DB 取单值文本 **/
function sqlOne(sql: string): string {
const p = Bun.spawnSync(["docker", "exec", "serpmax-pg", "psql", "-U", "postgres", "-d", "serpmax_admin", "-tAc", sql], { stdout: "pipe", stderr: "pipe" });
const out = p.stdout.toString().trim();
if (p.exitCode !== 0 || !out) throw new Error(`SQL 失败: ${p.stderr.toString().slice(0, 300)}`);
return out;
}
const TOKEN = sqlOne("SELECT api_token FROM users WHERE id=2");
const envText = fs.readFileSync("/opt/serpmax/node/.env", "utf8");
const NODE_KEY = /^NODE_KEY=(.*)$/m.exec(envText)?.[1]?.trim() ?? "";
/** # 成员变量 ipmart.io-US 代理四元组(DB 直取) **/
const PROXY_ID = "19e3ab5c-c61e-4936-8a7f-d78f6635f57c";
const prow = sqlOne(`SELECT type,host,port,coalesce(username,''),coalesce(password,'') FROM proxies WHERE id='${PROXY_ID}'`).split("|");
/** # 成员变量 节点公共头(与 admin relayHeaders 一致:Token+节点密钥双证) **/
function nodeHeaders(extra: Record<string, string> = {}): Record<string, string> {
return { "content-type": "application/json", authorization: `Bearer ${TOKEN}`, "x-node-key": NODE_KEY, ...extra };
}
// ---- 状态 ----
const pool = (await Bun.file(POOL_FILE).json()) as { fingerprint: string; name: string }[];
const LAUNCHED_FILE = `${DATA}/${LABEL}-launched.json`;
/** # 成员变量 已启动记录(含续跑读取) **/
let launched: Entry[] = [];
try { launched = JSON.parse(await Bun.file(LAUNCHED_FILE).text() || "[]"); } catch { launched = []; }
const doneSet = new Set(launched.map((e) => e.fp));
console.log(`[run] pool=${pool.length} tabs=${TABS} 续跑已开=${launched.length}`);
const persist = () => Bun.write(LAUNCHED_FILE, JSON.stringify(launched));
// ---- 统计/控制 ----
let consecutiveFail = 0;
let totalFail = 0;
let oomKilled = 0;
let stopReason: string | null = null;
const t0 = Date.now();
/** # 函数:读内存水位(MB) **/
function meminfo(): { availMB: number; swapUsedMB: number } {
const txt = fs.readFileSync("/proc/meminfo", "utf8");
const get = (k: string) => Number(new RegExp(`${k}:\\s+(\\d+)`).exec(txt)?.[1] ?? 0);
return { availMB: get("MemAvailable") / 1024, swapUsedMB: (get("SwapTotal") - get("SwapFree")) / 1024 };
}
/** # 函数:统计已开浏览器中被杀死的数量。
* 判据:该 Profile 的 user-data-dir 下已无任何 chrome 进程,且记录时长超过 60s
* (startBrowser 返回的 processId 是包装进程,会自然退出,不能逐 PID 判)。
**/
function countKilled(): number {
const alive = new Set<string>();
for (const pid of fs.readdirSync("/proc").filter((d) => /^\d+$/.test(d))) {
try {
const cmd = fs.readFileSync(`/proc/${pid}/cmdline`, "utf8");
const m = /--user-data-dir=\S*\/profiles\/([\w.]+)\/browser/.exec(cmd);
if (m) alive.add(m[1]);
} catch { /* 进程已退出 */ }
}
const now = Date.now();
let dead = 0;
for (const e of launched) {
if (now - Date.parse(e.startedAt) < 60_000) continue;
if (!alive.has(e.fp)) dead++;
}
return dead;
}
/** # 函数:能否继续发射;返回触发原因或 null,顺带刷新 oomKilled **/
function budget(): string | null {
if (stopReason) return stopReason;
if (consecutiveFail >= MAX_CONSEC_FAIL) return `连续启动失败 ${consecutiveFail} 次(共失败 ${totalFail})`;
oomKilled = countKilled();
if (oomKilled >= MAX_OOM_KILLED) return `浏览器主进程消失(疑似 OOM)${oomKilled} 个`;
if (MAX > 0 && launched.length >= MAX) return `达到封顶数 ${MAX}`;
const m = meminfo();
if (m.availMB < MEM_FLOOR_MB) return `MemAvailable ${m.availMB | 0}MB < ${MEM_FLOOR_MB}MB`;
if (m.swapUsedMB >= SWAP_CEIL_MB) return `swap 已用 ${m.swapUsedMB | 0}MB ≥ ${SWAP_CEIL_MB}MB`;
return null;
}
async function nodeFetch(path: string, init: RequestInit = {}, timeout = 120000): Promise<Response> {
return fetch(`${NODE}${path}`, { ...init, headers: nodeHeaders(init.headers as Record<string, string> | undefined), signal: AbortSignal.timeout(timeout) });
}
/** # 函数:解开 {success, data} 包装,拿 payload **/
function unwrap<T>(j: T | { data: T }): T {
return (j as any)?.data ?? (j as any);
}
/** # 函数:节点导入(幂等;载荷与 admin relay 的 importProfileToNode 等价) **/
async function ensureImported(fp: string, name: string): Promise<void> {
const chk = await nodeFetch(`/profiles/${fp}`, {}, 10000);
if (chk.ok) return;
if (chk.status !== 404) throw new Error(`节点查询 ${fp} 状态 ${chk.status}`);
const res = await nodeFetch("/profiles", {
method: "POST",
headers: { ...nodeHeaders(), "idempotency-key": `stress-import-${fp}` },
body: JSON.stringify({
id: fp, name,
browser: { kind: "cloak", theme: "light" },
network: {},
proxy: { id: PROXY_ID, protocol: prow[0], host: prow[1], port: Number(prow[2]), username: prow[3], password: prow[4] },
metadata: { importedBy: "stress-run" },
}),
}, 15000);
if (!res.ok && res.status !== 409) throw new Error(`导入 ${fp} 失败 ${res.status}`);
}
/** # 函数
* 通过 WS 对指定标签发送 Page.navigate(内核 /json/new?url= 不生效,需逐标签导航)。
* 参数 port:CDP 端口;targetId:标签 id;url:目标网址
* 返回值:是否发出成功(10s 超时视为失败,不阻塞)
*/
async function navigateTab(port: number, targetId: string, url: string): Promise<boolean> {
return new Promise((resolve) => {
let ws: WebSocket | null = null;
const done = (ok: boolean) => { try { ws?.close(); } catch { /* ignore */ } resolve(ok); };
const timer = setTimeout(() => done(false), 10000);
try {
ws = new WebSocket(`ws://127.0.0.1:${port}/devtools/page/${targetId}`);
ws.onopen = () => ws!.send(JSON.stringify({ id: 1, method: "Page.navigate", params: { url } }));
ws.onmessage = (ev) => {
try {
const m = JSON.parse(String(ev.data));
if (m.id === 1) { clearTimeout(timer); done(!m.error); }
} catch { /* ignore */ }
};
ws.onerror = () => { clearTimeout(timer); done(false); };
} catch { clearTimeout(timer); done(false); }
});
}
/** # 函数:lease(2h)→ start → 开标签;返回每 Profile 记录 **/
async function launchOne(fp: string, name: string): Promise<Entry> {
await ensureImported(fp, name);
// 已有未过期 lease(如脚本重启续跑)则复用其 leaseId,否则新拿一个
let leaseId = "";
const detail = await nodeFetch(`/profiles/${fp}`, {}, 10000);
if (detail.ok) {
const d = unwrap<{ lease?: { leaseId: string; expiresAt: string } | null }>(await detail.json());
if (d.lease && new Date(d.lease.expiresAt).getTime() > Date.now()) leaseId = d.lease.leaseId;
}
if (!leaseId) {
const r = await nodeFetch(`/profiles/${fp}/lease`, { method: "POST", body: JSON.stringify({ consumerId: "stress", ttlMs: LEASE_TTL_MS }) }, 15000);
if (!r.ok) throw new Error(`lease ${r.status} ${(await r.text()).slice(0, 100)}`);
leaseId = unwrap<{ leaseId: string }>(await r.json()).leaseId;
}
const start = await nodeFetch(`/profiles/${fp}/browser/start`, { method: "POST", headers: { "x-lease-id": leaseId }, body: "{}" }, 150000);
if (!start.ok) throw new Error(`start ${start.status} ${(await start.text()).slice(0, 130)}`);
const s = unwrap<{ cdp?: { endpoint?: string; port?: number }; processId?: number; userDataDir?: string }>(await start.json());
const entry: Entry = { fp, name, startedAt: new Date().toISOString(), cdpPort: s.cdp?.port ?? null, mainPid: s.processId ?? null, tabs: [], leaseId, userDataDir: s.userDataDir };
// OOM 角色反转:浏览器主进程调「易杀」,系统其它进程由采样器置「受保护」
if (entry.mainPid) { try { fs.writeFileSync(`/proc/${entry.mainPid}/oom_score_adj`, "200"); } catch { /* 进程已退出 */ } }
// 开标签:CDP /json/new 建空页(Chrome 111+ 要求 PUT;405 回退 POST)→ WS Page.navigate 逐个导航
if (entry.cdpPort) {
for (const u of OPEN_URLS) {
const url = `http://127.0.0.1:${entry.cdpPort}/json/new`;
try {
let t = await fetch(url, { method: "PUT", signal: AbortSignal.timeout(15000) });
if (t.status === 405) t = await fetch(url, { method: "POST", signal: AbortSignal.timeout(15000) });
if (t.ok) {
const j: any = unwrap(await t.json().catch(() => null) ?? {});
const tid = j?.id ?? "";
entry.tabs.push(tid);
if (tid && !j?.url?.startsWith(u)) void navigateTab(entry.cdpPort, tid, u);
}
} catch { /* 单个标签失败不阻塞 */ }
}
}
return entry;
}
// ---- 主 workers ----
let cursor = 0;
/** # 成员变量 每 Profile 启动失败计数(≥2 判代理/数据坏,节点强删后跳过) **/
const failByFp = new Map<string, number>();
/** # 函数:读 1 分钟负载(发射节流用) **/
function load1(): number {
try { return Number(fs.readFileSync("/proc/loadavg", "utf8").split(/\s+/)[0]) || 0; } catch { return 0; }
}
async function worker(): Promise<void> {
while (!stopReason) {
if (cursor >= pool.length) return;
// 发射节流:系统负载超过门限则等待(防 CPU 打满导致的启动超时雪崩)
while (load1() > LOAD_GATE && !stopReason) await new Promise((r) => setTimeout(r, 10_000));
const i = cursor++;
const item = pool[i];
if (!item || doneSet.has(item.fingerprint)) continue;
const b = budget();
if (b) { stopReason = stopReason ?? b; return; }
try {
const entry = await launchOne(item.fingerprint, item.name);
doneSet.add(item.fingerprint);
launched.push(entry);
consecutiveFail = 0;
if (launched.length % 10 === 0) {
const m = meminfo();
console.log(`[run] launched=${launched.length} avail=${m.availMB | 0}MB swapUsed=${m.swapUsedMB | 0}MB t=${((Date.now() - t0) / 60000).toFixed(1)}min`);
persist();
}
} catch (e) {
consecutiveFail++;
totalFail++;
console.error(`[run] fail(${consecutiveFail}) ${item.fingerprint}: ${String(e).slice(0, 160)}`);
// 坏 Profile(嫌疑代理不通/数据脏,非超时类错误)失败两次 → 节点强删后跳过
// 启动超时(Timeout)不删:属系统瞬时拥塞,续跑时还会重试
const msg = String(e);
if (!msg.includes("Timeout")) {
const n = (failByFp.get(item.fingerprint) ?? 0) + 1;
failByFp.set(item.fingerprint, n);
if (n >= 2) {
try { await nodeFetch(`/profiles/${item.fingerprint}?force=true`, { method: "DELETE" }, 30000); } catch { /* best-effort */ }
doneSet.add(item.fingerprint);
console.error(`[run] drop ${item.fingerprint}(失败 ${n} 次,节点已强删)`);
}
}
// 连续失败冷却:6 次后等 45s 再继续(躲开发射抖动期)
if (consecutiveFail === 6) { console.error(`[run] 连续失败,冷却 45s`); await new Promise((r) => setTimeout(r, 45_000)); }
}
const b2 = budget();
if (b2 && !stopReason) stopReason = b2;
}
}
await persist();
await Promise.all(Array.from({ length: CONCURRENCY }, () => worker()));
await persist();
const m = meminfo();
const final = { stopReason, launchedCount: launched.length, totalFail, oomKilled, tabs: TABS, minutes: +((Date.now() - t0) / 60000).toFixed(1), availMB: m.availMB | 0, swapUsedMB: m.swapUsedMB | 0, stoppedAt: new Date().toISOString() };
await Bun.write(`${DATA}/${LABEL}-stop.json`, JSON.stringify(final, null, 2));
console.log(`[run] STOP: ${JSON.stringify(final)}`);
+113
View File
@@ -0,0 +1,113 @@
/** # 脚本说明
* 压测采样器(与发射器并行跑,同机 root):
* 每 20s 扫描 /proc:
* 1) 按 --user-data-dir 路径的指纹段聚合「每浏览器进程树」:
* 进程数 / RSS 合计 / swap 合计 / CPU 累计(utime+stime, ms)
* 2) 系统快照:MemAvailable / swap 已用 / loadavg / 浏览器树总数
* 3) OOM 保护:非浏览器进程(且非本采样器自身)写 oom_score_adj=-600,
* 让内核 OOM 时优先杀「易杀」的浏览器(发射器已把新浏览器主进程置 200)
* 写 data/<label>-stats.jsonl(每轮一行 JSON)。
*
* 用法:bun sampler.ts --label test1 [--interval 20]
*/
const fs = require("fs") as typeof import("fs");
const args = process.argv.slice(2);
function argOf(k: string, d: string): string { const i = args.indexOf(k); return i >= 0 ? args[i + 1] : d; }
const LABEL = argOf("--label", "test1");
const INTERVAL = Number(argOf("--interval", "20")) * 1000;
const BASE = new URL(".", import.meta.url).pathname;
const OUT = `${BASE}data/${LABEL}-stats.jsonl`;
/** # 成员变量 本采样器自身进程组(不碰 oom_score_adj) **/
const SELF_PIDS = new Set([process.pid, process.ppid ?? 0]);
/** # 成员变量 每浏览器树快照类型 **/
type Tree = { fp: string; procs: number; rssKB: number; pssKB: number; swapKB: number; cpuMs: number; userDataDir: string };
/** # 函数:一轮 /proc 扫描,聚合浏览器进程树并返回系统快照 **/
function scanOnce(): { trees: Map<string, Tree>; availMB: number; swapUsedMB: number; load: number[]; cpuCount: number } {
const trees = new Map<string, Tree>();
const mySig = `run-stress.ts`; // 自身及其 bash 父链保护
const txt = fs.readFileSync("/proc/meminfo", "utf8");
const get = (k: string) => Number(new RegExp(`${k}:\\s+(\\d+)`).exec(txt)?.[1] ?? 0);
const cpuCount = osCpus();
const load = fs.readFileSync("/proc/loadavg", "utf8").split(/\s+/).slice(0, 3).map(Number);
for (const pid of fs.readdirSync("/proc").filter((d) => /^\d+$/.test(d))) {
let cmd: string;
try { cmd = fs.readFileSync(`/proc/${pid}/cmdline`, "utf8").replace(/\0/g, " "); } catch { continue; }
const udd = /--user-data-dir=(\S+)/.exec(cmd)?.[1] ?? null;
if (udd) {
// 目录形如 .../data/profiles/<fp>/browser,指纹取「browser」的父目录名
const seg = udd.replace(/\/browser\/?$/, "").split("/").pop() ?? udd.split("/").pop() ?? "?";
const fp = seg;
const st = readStat(pid);
const t = trees.get(fp) ?? { fp, procs: 0, rssKB: 0, pssKB: 0, swapKB: 0, cpuMs: 0, userDataDir: udd };
t.procs++;
t.rssKB += st.rssKB;
t.pssKB += st.pssKB;
t.swapKB += st.swapKB;
t.cpuMs += st.cpuMs;
trees.set(fp, t);
// 浏览器树整体保持「易杀」(主进程已设 200,子进程继承;这里给漏网进程补救)
try { fs.writeFileSync(`/proc/${pid}/oom_score_adj`, "200"); } catch { /* ignore */ }
} else {
// 非浏览器进程:OOM 保护(排除自身/内核线程写失败忽略)
const p = Number(pid);
if (SELF_PIDS.has(p)) continue;
if (cmd.includes(mySig) || cmd.includes("sampler.ts") || cmd.includes("run-stress")) continue;
try { fs.writeFileSync(`/proc/${pid}/oom_score_adj`, "-600"); } catch { /* ignore */ }
}
}
const swapUsedMB = (get("SwapTotal") - get("SwapFree")) / 1024;
const availMB = get("MemAvailable") / 1024;
return { trees, availMB, swapUsedMB, load, cpuCount };
}
/** # 函数:读单进程 RSS/PSS/swap/CPU(kB、ms) **/
function readStat(pid: string): { rssKB: number; pssKB: number; swapKB: number; cpuMs: number } {
let rssKB = 0, pssKB = 0, swapKB = 0, cpuMs = 0;
try {
const s = fs.readFileSync(`/proc/${pid}/stat`, "utf8");
// stat 第 14/15 列为 utime/stime(tick);comm 含空格,取最后一个 ')' 之后按列号回算
const tail = s.slice(s.lastIndexOf(")") + 2).split(" ");
const utime = Number(tail[11]) || 0, stime = Number(tail[12]) || 0;
cpuMs = ((utime + stime) * 1000) / 100; // CLK_TCK=100
} catch { /* 进程已消失 */ }
try {
// smaps_rollup 是内核快速汇总:Pss = 独占内存(去共享重复),Swap 含交换的共享分摊
const roll = fs.readFileSync(`/proc/${pid}/smaps_rollup`, "utf8");
pssKB = Number(/^Pss:\s+(\d+)/m.exec(roll)?.[1] ?? 0);
rssKB = Number(/^Rss:\s+(\d+)/m.exec(roll)?.[1] ?? 0);
const swapPss = Number(/^SwapPss:\s+(\d+)/m.exec(roll)?.[1] ?? 0);
const swapDirect = Number(/^Swap:\s+(\d+)/m.exec(roll)?.[1] ?? 0);
swapKB = swapPss || swapDirect;
} catch { /* ignore */ }
return { rssKB, pssKB, swapKB, cpuMs };
}
/** # 函数:CPU 核数 **/
function osCpus(): number {
try { return require("os").cpus().length; } catch { return 8; }
}
// 主循环
while (true) {
const t0 = Date.now();
const { trees, availMB, swapUsedMB, load, cpuCount } = scanOnce();
const treesArr = [...trees.values()];
const line = JSON.stringify({
ts: new Date().toISOString(),
browsers: treesArr.length,
totalRssMB: +(treesArr.reduce((a, b) => a + b.rssKB, 0) / 1024).toFixed(1),
totalPssMB: +(treesArr.reduce((a, b) => a + b.pssKB, 0) / 1024).toFixed(1),
totalSwapMB: +(treesArr.reduce((a, b) => a + b.swapKB, 0) / 1024).toFixed(1),
totalProcs: treesArr.reduce((a, b) => a + b.procs, 0),
availMB: +availMB.toFixed(1), swapUsedMB: +swapUsedMB.toFixed(1),
load, cpuCount,
detail: treesArr,
});
try { fs.appendFileSync(OUT, line + "\n"); } catch { /* ignore */ }
const wait = Math.max(0, INTERVAL - (Date.now() - t0));
await new Promise((r) => setTimeout(r, wait));
}
+80
View File
@@ -0,0 +1,80 @@
/** # 脚本说明
* SERP 24h 监控采样器(在 admin 服务器本机运行,零额外负载:只读管理接口):
* nohup bun serp-monitor.ts > data/monitor.log 2>&1 &
* 每 60s 采样 /api/serp/stats(hours24)+ /api/serp/pool(浏览器池)+ /api/serp/cache(缓存池),
* 追加 JSONL 到 data/serp-monitor.jsonl;每 10 分钟输出一行摘要到 stdout(进 monitor.log)。
* 用途:跑一天边测边修的长期证据链(吞吐/错误率/门壳/池水位)。
*/
import fs from "node:fs";
/** # 成员变量 基址与令牌(本机) **/
const BASE = "http://127.0.0.1:53100";
const TOKEN = process.env.SERP_TOKEN ?? "";
if (!TOKEN) { console.error("缺少 SERP_TOKEN"); process.exit(1); }
/** # 成员变量 输出目录(data/,相对脚本) **/
const DATA = new URL("./data/", import.meta.url).pathname;
fs.mkdirSync(DATA, { recursive: true });
const JSONL = `${DATA}serp-monitor.jsonl`;
/** # 成员变量 上一次快照(用于计算每分钟增量) **/
let prev: { total: number; ok: number; failed: number; captcha: number; gate: number; at: number } | null = null;
/** # 成员变量 每 10 分钟摘要计数器 **/
let sinceSummary = 0;
/** # 函数
* 带超时 GET JSON。
*/
async function getJson(path: string): Promise<any> {
const res = await fetch(`${BASE}${path}`, { headers: { authorization: `Bearer ${TOKEN}` }, signal: AbortSignal.timeout(15000) });
if (!res.ok) throw new Error(`HTTP ${res.status} ${path}`);
const j = await res.json() as any;
return j?.data ?? j;
}
/** # 函数
* 单轮采样:统计/池/缓存 → 增量计算 → JSONL 追加 → 10 分钟摘要打印。
*/
async function sample(): Promise<void> {
const now = Date.now();
const [stats, pool, cache] = await Promise.all([
getJson("/api/serp/stats").catch(() => null),
getJson("/api/serp/pool").catch(() => null),
getJson("/api/serp/cache").catch(() => null),
]);
if (!stats) return;
const h = stats.hours24 ?? {};
const row = {
at: new Date(now).toISOString(),
h24: { total: h.total ?? 0, ok: h.ok ?? 0, failed: h.failed ?? 0, captcha: h.captcha ?? 0, gate: h.gate ?? 0, avg_ms: h.avg_ms ?? 0, p95_ms: h.p95_ms ?? 0 },
delta: null as any,
browserPool: null as any,
cachePool: null as any,
};
if (prev && now - prev.at > 30000) {
row.delta = {
total: (h.total ?? 0) - prev.total,
failed: (h.failed ?? 0) - prev.failed,
gate: (h.gate ?? 0) - prev.gate,
captcha: (h.captcha ?? 0) - prev.captcha,
};
}
prev = { total: h.total ?? 0, ok: h.ok ?? 0, failed: h.failed ?? 0, captcha: h.captcha ?? 0, gate: h.gate ?? 0, at: now };
if (pool?.totals) row.browserPool = { ...pool.totals, targets: pool.nodes.map((n: any) => `${n.node}:${n.total}/${n.target}`).join(",") };
if (cache) row.cachePool = { count: cache.count, maxCache: cache.maxCache };
fs.appendFileSync(JSONL, JSON.stringify(row) + "\n");
sinceSummary++;
if (sinceSummary >= 10) {
sinceSummary = 0;
const d = row.delta;
const errRate = d && d.total > 0 ? +(((d.failed) / d.total) * 100).toFixed(1) : null;
console.log(`${row.at} Δ10min total=${d?.total} failed=${d?.failed} gate=${d?.gate} captcha=${d?.captcha} err=${errRate}% avg=${row.h24.avg_ms}ms browserPool=${row.browserPool ? `${row.browserPool.total}(ready${row.browserPool.ready})` : "?"} cache=${row.cachePool?.count}/${row.cachePool?.maxCache}`);
}
}
/** # 主流程:60s 一轮,永久运行(systemd/nohup 托管) **/
console.log(`serp-monitor started ${new Date().toISOString()}`);
for (;;) {
await sample().catch((e) => console.error(`${new Date().toISOString()} sample error: ${String(e).slice(0, 160)}`));
await Bun.sleep(60000);
}
+119
View File
@@ -0,0 +1,119 @@
/** # 脚本说明
* SERP go-fetch 速率-门壳实验发射器(在 admin 服务器本机运行,消除跨网噪声):
* SERP_TOKEN=xxx bun serp-rate-test.ts --rate 2 --minutes 5 --label r2
* 以指定速率发射 GET /api/serp/html/search?...&runtime=fetch(强制 go-fetch,绕过令牌桶,
* 用于测「该速率下 Google 是否发门壳」),逐条记录 HTTP 状态 / x-serp-gate / attempts / 耗时,
* 写 data/<label>.jsonl 与 data/<label>-summary.json。
* 门壳判定:x-serp-gate:1(admin 全候选门壳兜底);HTTP!=200 记错误码;网络异常单独计。
*/
import fs from "node:fs";
/** # 成员变量 服务基址(同机直连) **/
const BASE = "http://127.0.0.1:53100";
const args = process.argv.slice(2);
function argOf(k: string, d: string): string { const i = args.indexOf(k); return i >= 0 ? args[i + 1] : d; }
const RATE = Number(argOf("--rate", "1"));
const MINUTES = Number(argOf("--minutes", "5"));
const LABEL = argOf("--label", `r${RATE}`);
/** # 成员变量 runtime 覆写:fetch=强制 go-fetch(绕令牌桶,测速率阈值);auto=双模式生产路由 **/
const RUNTIME = argOf("--runtime", "fetch");
const TOKEN = process.env.SERP_TOKEN ?? "";
if (!TOKEN) { console.error("缺少 SERP_TOKEN 环境变量"); process.exit(1); }
/** # 成员变量 输出目录(data/,相对脚本) **/
const DATA = new URL("./data/", import.meta.url).pathname;
fs.mkdirSync(DATA, { recursive: true });
const JSONL = `${DATA}${LABEL}.jsonl`;
const SUMMARY = `${DATA}${LABEL}-summary.json`;
fs.writeFileSync(JSONL, "");
/** # 成员变量 长尾查询词表(A×B×C 组合,按序消费避免重复) **/
const A = ["best", "top rated", "cheap", "review of", "how to choose a", "buy", "compare", "2026 guide for", "price of", "warranty for"];
const B = ["coffee maker", "running shoes", "laptop stand", "rice cooker", "air fryer", "standing desk", "office chair", "mechanical keyboard", "monitor arm", "webcam", "espresso machine", "vacuum cleaner", "electric kettle", "gaming mouse", "noise cancelling headphones", "desk lamp", "water bottle", "yoga mat", "blender", "space heater"];
const C = ["for home use", "for small office", "under 100 dollars", "in 2026", "with long warranty", "for beginners", "pro model", "on sale", "quiet operation", "energy efficient"];
const combos: string[] = [];
for (const a of A) for (const b of B) for (const c of C) combos.push(`${a} ${b} ${c}`);
/** # 成员变量 洗牌后的查询队列指针 **/
let qi = 0;
for (let i = combos.length - 1; i > 0; i--) { const j = Math.floor(Math.random() * (i + 1)); [combos[i], combos[j]] = [combos[j], combos[i]]; }
function nextQuery(): string { const q = combos[qi % combos.length]; qi++; return q; }
/** # 成员变量 统计桶 **/
const stat = { sent: 0, ok200: 0, gate: 0, httpErr: 0, netErr: 0, inflight: 0, done: 0 };
const totalMsArr: number[] = [];
const clientMsArr: number[] = [];
/** # 函数
* 单次记录:追加 JSONL 与统计。
*/
function rec(o: Record<string, unknown>): void {
fs.appendFileSync(JSONL, JSON.stringify(o) + "\n");
stat.done++;
if (typeof o.totalMs === "number" && o.totalMs > 0) totalMsArr.push(o.totalMs);
if (typeof o.ms === "number") clientMsArr.push(o.ms);
}
/** # 函数
* 发射单次搜索(runtime=fetch 强制 go-fetch)。60s 客户端超时;失败分类:HTTP!=200=错误,
* x-serp-gate:1=门壳兜底,catch=网络异常。
*/
async function fire(): Promise<void> {
const q = nextQuery();
const url = `${BASE}/api/serp/html/search?q=${encodeURIComponent(q)}&num=10${RUNTIME ? `&runtime=${RUNTIME}` : ""}`;
const t = Date.now();
stat.sent++; stat.inflight++;
try {
const res = await fetch(url, { headers: { authorization: `Bearer ${TOKEN}` }, signal: AbortSignal.timeout(60000) });
const gate = res.headers.get("x-serp-gate");
const attempts = res.headers.get("x-serp-attempts");
const totalMs = Number(res.headers.get("x-serp-total-ms") ?? 0);
const nodeMs = Number(res.headers.get("x-serp-node-ms") ?? 0);
const profile = res.headers.get("x-serp-profile") ?? "";
const node = res.headers.get("x-serp-node") ?? "";
let code = ""; let msg = "";
if (!res.ok) {
const j = await res.json().catch(() => null) as any;
code = String(j?.code ?? ""); msg = String(j?.message ?? "").slice(0, 120);
}
const kind = !res.ok ? "httpErr" : gate === "1" ? "gate" : "ok200";
stat[kind]++;
rec({ q, status: res.status, kind, gate: gate === "1", attempts, totalMs, nodeMs, profile, node, code, msg, ms: Date.now() - t });
} catch (e) {
stat.netErr++;
rec({ q, status: 0, kind: "netErr", error: String(e).slice(0, 160), ms: Date.now() - t });
} finally {
stat.inflight--;
}
}
function pct(arr: number[], p: number): number {
if (arr.length === 0) return 0;
const s = [...arr].sort((a, b) => a - b);
return s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))];
}
/** # 主流程:恒定速率发射(每 tick 发 1 发),结束后等在途清空并写摘要 **/
console.log(`开始:rate=${RATE}/s minutes=${MINUTES} label=${LABEL}`);
const interval = 1000 / RATE;
const endAt = Date.now() + MINUTES * 60_000;
const MAX_INFLIGHT = Math.max(30, Math.ceil(RATE) * 20);
while (Date.now() < endAt) {
const t0 = Date.now();
if (stat.inflight < MAX_INFLIGHT) void fire();
const wait = interval - (Date.now() - t0);
if (wait > 0) await Bun.sleep(wait);
}
const drainDeadline = Date.now() + 70000;
while (stat.inflight > 0 && Date.now() < drainDeadline) await Bun.sleep(500);
const summary = {
label: LABEL, rate: RATE, minutes: MINUTES,
sent: stat.sent, ok200: stat.ok200, gate: stat.gate, httpErr: stat.httpErr, netErr: stat.netErr,
gateRate: stat.done ? +(stat.gate / stat.done * 100).toFixed(2) : 0,
errRate: stat.done ? +((stat.httpErr + stat.netErr) / stat.done * 100).toFixed(2) : 0,
totalMs_p50: pct(totalMsArr, 50), totalMs_p95: pct(totalMsArr, 95),
clientMs_p50: pct(clientMsArr, 50), clientMs_p95: pct(clientMsArr, 95),
at: new Date().toISOString(),
};
fs.writeFileSync(SUMMARY, JSON.stringify(summary, null, 2));
console.log(JSON.stringify(summary, null, 2));
+98
View File
@@ -0,0 +1,98 @@
/** # 脚本说明
* 压测清场:把某一轮发射器开过的浏览器全部停止。
* 1) 读 data/<label>-launched.json 的每条记录(--scan 模式则改从节点列表取全部 IN_USE Profile)
* 2) 逐个 GET /profiles/:fp 拿当前 leaseId → POST /profiles/:fp/browser/stop(x-lease-id;
* 显式停止走 closing 路径,不会触发「UI 误关自动复活」)
* 3) 兜底:对仍存的 user-data-dir 进程 SIGKILL(等待复活限频窗口过期,前后共两轮)
* 用法:bun stop-browsers.ts --label test1 [--scan]
*/
const fs = require("fs") as typeof import("fs");
const args = process.argv.slice(2);
function argOf(k: string, d: string): string { const i = args.indexOf(k); return i >= 0 && i + 1 < args.length ? args[i + 1] : d; }
const LABEL = argOf("--label", "test1");
const SCAN = args.includes("--scan");
const NODE = "http://127.0.0.1:3106";
const DATA = `${new URL(".", import.meta.url).pathname}data`;
/** # 函数:docker exec 查 admin DB 单值(取 Token 用) **/
function sqlOne(sql: string): string {
const p = Bun.spawnSync(["docker", "exec", "serpmax-pg", "psql", "-U", "postgres", "-d", "serpmax_admin", "-tAc", sql], { stdout: "pipe", stderr: "pipe" });
const out = p.stdout.toString().trim();
if (p.exitCode !== 0 || !out) throw new Error(`SQL 失败`);
return out;
}
const TOKEN = sqlOne("SELECT api_token FROM users WHERE id=2");
const NODE_KEY = /^NODE_KEY=(.*)$/m.exec(fs.readFileSync("/opt/serpmax/node/.env", "utf8"))?.[1]?.trim() ?? "";
const LEASE_TTL_MS = 120000;
const headersBase: Record<string, string> = { "content-type": "application/json", authorization: `Bearer ${TOKEN}`, "x-node-key": NODE_KEY };
/** # 成员变量 待停清单:label 文件或节点扫描(IN_USE/有活动 lease 的 Profile) **/
let targets: { fp: string }[] = [];
if (SCAN) {
const res = await fetch(`${NODE}/profiles`, { headers: headersBase, signal: AbortSignal.timeout(60000) });
const j: any = await res.json();
const rows: any[] = j?.data?.profiles ?? j?.profiles ?? [];
targets = rows.filter((r) => {
const lease = r?.lease ?? null;
return (lease && new Date(lease.expiresAt).getTime() > Date.now()) || ["IN_USE"].includes(r?.state);
}).map((r) => ({ fp: String(r.id) }));
console.log(`[stop] 扫描到 ${targets.length} 个在用 Profile`);
} else {
let list: { fp: string }[] = [];
try { list = JSON.parse(await Bun.file(`${DATA}/${LABEL}-launched.json`).text() || "[]"); } catch { list = []; }
targets = list.map((e: any) => ({ fp: String(e.fp) }));
}
console.log(`[stop] 目标 ${targets.length} 个`);
/** # 函数:显式停止一个 Profile 的浏览器(先拿当前 leaseId,过期则补一个短 lease) **/
async function stopOne(fp: string): Promise<void> {
const headers = { ...headersBase } as Record<string, string>;
try {
const d = await fetch(`${NODE}/profiles/${fp}`, { headers: headersBase, signal: AbortSignal.timeout(10000) });
const j: any = await d.json();
const lease = j?.data?.lease ?? j?.lease;
if (lease?.leaseId && new Date(lease.expiresAt).getTime() > Date.now()) headers["x-lease-id"] = lease.leaseId;
} catch { /* detail 拿不到就直接停 */ }
if (!headers["x-lease-id"]) {
const r = await fetch(`${NODE}/profiles/${fp}/lease`, { method: "POST", headers: headersBase, body: JSON.stringify({ consumerId: "stress-stop", ttlMs: LEASE_TTL_MS }), signal: AbortSignal.timeout(10000) });
if (r.ok) { const j2: any = await r.json(); headers["x-lease-id"] = (j2?.data ?? j2)?.leaseId; }
}
await fetch(`${NODE}/profiles/${fp}/browser/stop`, { method: "POST", headers, body: "{}", signal: AbortSignal.timeout(30000) }).catch(() => null);
}
const workers = 10;
let cursor = 0;
let stopped = 0;
async function worker(): Promise<void> {
while (cursor < targets.length) {
const e = targets[cursor++];
await stopOne(e.fp);
stopped++;
if (stopped % 25 === 0) console.log(`[stop] ${stopped}/${targets.length}`);
}
}
await Promise.all(Array.from({ length: workers }, () => worker()));
console.log(`[stop] API 显式停完 ${stopped}`);
/** # 函数:兜底统计存活压测浏览器进程数 **/
function countLeft(): number {
let n = 0;
for (const pid of fs.readdirSync("/proc").filter((d) => /^\d+$/.test(d))) {
try { if (fs.readFileSync(`/proc/${pid}/cmdline`, "utf8").includes("--user-data-dir=")) n++; } catch { /* 已退出 */ }
}
return n;
}
for (const round of [1, 2]) {
const left = countLeft();
console.log(`[stop] round${round} 残存进程 ${left}`);
if (left === 0) break;
for (const pid of fs.readdirSync("/proc").filter((d) => /^\d+$/.test(d))) {
try {
if (fs.readFileSync(`/proc/${pid}/cmdline`, "utf8").includes("--user-data-dir=")) process.kill(Number(pid), "SIGKILL");
} catch { /* ignore */ }
}
await new Promise((r) => setTimeout(r, 8000));
}
console.log(`[stop] done`);