Files

T

phptits 851ccf1d7f v2 platform runtime (docs 22 Phase 2/4/7 + 17): Project/Environment + Config + Storage + Observability

- Project/Environment model with default environments (dev/staging/prod)
- Config Runtime: cascade resolution (platform→org→project→env), secrets via vault
- Storage Runtime: local provider, sha256 dedup, integrity check, file list/upload/delete
- Observability: execution query, per-action summary (success rate, avg, p95), health
- Realtime SSE: workspace event stream endpoint (GET /api/v1/v2/realtime/events)
- Phase 1 acceptance: core.health.check, core.user.create, core.user.list
- Actions: core.project.*, core.config.*, core.file.*, core.user.* (62 total)

2026-09-11 21:11:07 +00:00

64 KiB

Raw Permalink Blame History

`17_OBSERVABILITY_AUDIT_LOGGING_RUNTIME.md`

Version: 1.0 Status: Implementation Specification Target: Zcode Depends on:

03_RESOURCE_ACTION_MODEL.md

06_MCP_SKILL_AGENT_RUNTIME.md

09_TASK_JOB_QUEUE_RUNTIME.md

12_MIGRATION_VERSIONING_COMPATIBILITY.md

14_DATA_QUERY_SEARCH_INDEX_RUNTIME.md

15_FILE_ARTIFACT_STORAGE_RUNTIME.md

16_EVENT_STREAMING_NOTIFICATION_REALTIME_RUNTIME.md

1. Document Purpose

本文件定义 DevOS 统一的：

Logging Runtime
Metrics Runtime
Tracing Runtime
Audit Runtime
Error Runtime
Health Runtime
Diagnostic Runtime
Usage / Cost Runtime
Performance Runtime
SLO / SLA Runtime
Alert Runtime

目标：

DevOS 中所有 Core、Plugin、MCP、Agent、Task、Job、Action、Worker、Scheduler、Storage、Search、Realtime 都必须使用统一的 Observability Runtime。

最终形成：

                    DevOS Observability Runtime
                              │
          ┌───────────────────┼───────────────────┐
          │                   │                   │
       Logging             Metrics            Tracing
          │                   │                   │
          └───────────────────┼───────────────────┘
                              │
          ┌───────────────────┼───────────────────┐
          │                   │                   │
        Audit               Error              Health
          │                   │                   │
          └───────────────────┼───────────────────┘
                              │
          ┌───────────────────┼───────────────────┐
          │                   │                   │
     Diagnostics          Usage/Cost          Performance
                              │
                              ↓
                            Alert
                              │
                              ↓
                  Notification / Event Runtime

核心目标不是“增加日志”。

而是：

让 DevOS 中任何一次行为都可以被定位、解释、审计、统计、诊断和追踪。

2. Core Principles

DevOS Observability 必须遵循以下原则。

2.1 Everything Is Correlated

所有 Runtime 行为必须能够关联：

requestId
correlationId
traceId
spanId
executionId
taskId
jobId
agentRunId
pluginId
actionId
resourceId
actorId
organizationId
projectId
environmentId

最少要求：

request
  ↓
trace
  ↓
execution
  ↓
task/job
  ↓
action
  ↓
event

可以完整还原。

3. Observability Context

定义统一：

export interface ObservabilityContext {
  requestId?: string

  correlationId: string

  traceId?: string
  spanId?: string

  executionId?: string
  taskId?: string
  jobId?: string

  agentRunId?: string
  agentStepId?: string

  pluginId?: string
  pluginVersion?: string

  actionId?: string
  actionVersion?: string

  resourceType?: string
  resourceId?: string

  actorId?: string
  actorType?: 'user' | 'agent' | 'service' | 'system'

  organizationId?: string
  projectId?: string
  environmentId?: string

  metadata?: Record<string, unknown>
}

任何 Runtime 都必须能够接收这个 Context。

4. Context Propagation

Context 必须自动传播。

例如：

HTTP Request
    ↓
RequestContext
    ↓
Action Runtime
    ↓
Task
    ↓
Job
    ↓
Worker
    ↓
Execution
    ↓
External API

每一层都不能重新生成一套互不关联的 ID。

5. Request ID

requestId 表示：

一次进入 DevOS 的请求。

例如：

HTTP
CLI
Webhook
MCP
Agent
Scheduler
Plugin

都可以产生 request。

6. Correlation ID

correlationId 表示：

一组相关操作。

例如：

用户创建 SEO Scan
       ↓
Task
       ↓
10 Jobs
       ↓
10 Executions
       ↓
100 Events

全部共享：

correlationId

7. Trace ID

traceId 表示：

一次完整分布式调用链。

推荐使用：

OpenTelemetry

作为 Trace 标准。

DevOS 不应该自己重新发明分布式 tracing 协议。

8. Span

一个 Runtime 操作可以产生 Span。

例如：

Trace
 ├── HTTP POST /tasks
 │
 ├── action.task.create
 │
 ├── task.enqueue
 │
 ├── redis.publish
 │
 └── worker.execute
      ├── action.seo.scan
      ├── dns.lookup
      ├── http.request
      └── artifact.create

9. Logging Runtime

定义统一 Logger：

export interface Logger {
  debug(message: string, data?: LogData): void
  info(message: string, data?: LogData): void
  warn(message: string, data?: LogData): void
  error(message: string, data?: LogData): void
  fatal(message: string, data?: LogData): void

  child(context: Partial<ObservabilityContext>): Logger
}

10. Structured Logging

禁止：

logger.info(
  `User ${userId} executed ${actionId} on ${resourceId}`
)

必须使用结构化数据：

logger.info('action executed', {
  actionId,
  resourceId,
  actorId,
  executionId,
  durationMs,
})

11. Log Entry

统一：

export interface LogEntry {
  id: string

  timestamp: string

  level:
    | 'debug'
    | 'info'
    | 'warn'
    | 'error'
    | 'fatal'

  message: string

  context: ObservabilityContext

  category:
    | 'system'
    | 'request'
    | 'action'
    | 'task'
    | 'job'
    | 'agent'
    | 'plugin'
    | 'mcp'
    | 'database'
    | 'queue'
    | 'worker'
    | 'storage'
    | 'search'
    | 'security'
    | 'integration'

  data?: Record<string, unknown>

  error?: LogError

  durationMs?: number

  source?: {
    service?: string
    module?: string
    file?: string
    line?: number
  }
}

12. Log Levels

标准：

DEBUG
INFO
WARN
ERROR
FATAL

使用规则：

DEBUG

开发调试信息。

INFO

正常业务行为。

WARN

异常但系统仍然正常。

ERROR

操作失败。

FATAL

服务无法继续运行。

13. Sensitive Data Protection

日志绝对不能直接输出：

password
token
apiKey
secret
privateKey
cookie
authorization
session
creditCard

例如：

logger.info('plugin configured', {
  apiKey: 'xxx'
})

必须自动变成：

apiKey: "[REDACTED]"

14. Redaction Runtime

定义：

export interface RedactionRule {
  path: string

  action:
    | 'redact'
    | 'hash'
    | 'mask'
    | 'remove'
}

例如：

{
  path: 'authorization',
  action: 'redact'
}

支持：

global rules
tenant rules
plugin rules
resource rules

15. Logging Sink

Logger 不直接决定日志存储位置。

定义：

export interface LogSink {
  write(entry: LogEntry): Promise<void>

  writeBatch(entries: LogEntry[]): Promise<void>

  flush(): Promise<void>
}

可以支持：

Console
PostgreSQL
Object Storage
Loki
OpenSearch
Cloud Logging

DevOS Runtime 只定义抽象。

16. Log Runtime

export interface LogRuntime {
  logger(context?: ObservabilityContext): Logger

  query(input: LogQuery): Promise<LogQueryResult>

  stream(input: LogStreamInput): AsyncIterable<LogEntry>
}

17. Log Query

export interface LogQuery {
  levels?: string[]

  categories?: string[]

  service?: string

  pluginId?: string

  actionId?: string

  executionId?: string

  taskId?: string

  jobId?: string

  traceId?: string

  actorId?: string

  organizationId?: string

  from?: string
  to?: string

  search?: string

  limit?: number
}

18. Metrics Runtime

Metrics 用于回答：

系统运行得怎么样？

例如：

Action execution count
Action failure count
Action latency
Queue depth
Worker utilization
API request rate
LLM token usage
Storage usage
Search latency
Database latency

19. Metric Types

支持：

Counter
Gauge
Histogram
Summary

20. Counter

例如：

devos_action_executions_total
devos_action_failures_total
devos_http_requests_total
devos_agent_runs_total

21. Gauge

例如：

devos_queue_depth
devos_worker_active
devos_storage_bytes
devos_active_websocket_connections

22. Histogram

用于：

HTTP latency
Action latency
Database query latency
Queue wait time
Job duration
LLM latency

例如：

devos_action_duration_ms

23. Metric API

export interface MetricsRuntime {
  counter(
    name: string,
    value?: number,
    attributes?: Record<string, string | number | boolean>
  ): void

  gauge(
    name: string,
    value: number,
    attributes?: Record<string, string | number | boolean>
  ): void

  histogram(
    name: string,
    value: number,
    attributes?: Record<string, string | number | boolean>
  ): void
}

24. Metric Cardinality

禁止产生无限 Cardinality。

危险：

userId
requestId
executionId
URL
query

不能直接作为 Metric label。

允许：

actionId
pluginId
status
method
provider
environment

必须提供 Cardinality Protection。

25. Tracing Runtime

export interface TracingRuntime {
  startSpan(
    name: string,
    options?: SpanOptions
  ): Span
}

26. Span API

export interface Span {
  setAttribute(
    key: string,
    value: string | number | boolean
  ): void

  addEvent(
    name: string,
    attributes?: Record<string, unknown>
  ): void

  recordException(error: unknown): void

  setStatus(
    status: 'ok' | 'error'
  ): void

  end(): void
}

27. Action Runtime Integration

这是必须实现的。

所有：

Action

执行自动生成：

Span
Log
Metric
Audit
Execution

例如：

action.seo.scan

自动：

Trace
 └── action.seo.scan
      ├── validation
      ├── authorization
      ├── database
      ├── external-http
      ├── storage
      └── result

业务 Action 不需要自己实现这些。

28. Task / Job Integration

Task：

task.created
task.started
task.completed
task.failed

Job：

job.queued
job.started
job.retrying
job.completed
job.failed

自动生成：

metrics
logs
traces

29. Worker Observability

Worker 必须暴露：

worker.started
worker.ready
worker.busy
worker.draining
worker.unhealthy
worker.stopped

Metrics：

worker_jobs_total
worker_jobs_failed_total
worker_active_jobs
worker_utilization
worker_heartbeat_age
worker_queue_wait_ms
worker_job_duration_ms

30. Queue Observability

必须记录：

queue depth
enqueue rate
dequeue rate
wait time
processing time
retry count
DLQ count

例如：

devos_queue_depth{queue="default"}
devos_queue_depth{queue="ai"}
devos_queue_depth{queue="browser"}

31. Audit Runtime

Audit 和 Log 不是一回事。

Log：

系统运行发生了什么？

Audit：

谁对什么资源做了什么具有业务/安全意义的操作？

32. Audit Requirements

Audit 必须：

durable
immutable
queryable
tenant-isolated
actor-aware
resource-aware

不能把普通日志当 Audit。

33. Audit Record

export interface AuditRecord {
  id: string

  timestamp: string

  actor: {
    id?: string
    type: 'user' | 'agent' | 'service' | 'system'
  }

  organizationId?: string
  projectId?: string
  environmentId?: string

  actionId: string

  actionVersion?: string

  resourceType?: string
  resourceId?: string

  operation:
    | 'create'
    | 'update'
    | 'delete'
    | 'read'
    | 'execute'
    | 'approve'
    | 'reject'
    | 'login'
    | 'logout'
    | 'install'
    | 'upgrade'
    | 'configure'

  result:
    | 'success'
    | 'failure'
    | 'denied'

  reason?: string

  before?: unknown

  after?: unknown

  changes?: unknown

  approvalId?: string

  requestId?: string
  correlationId?: string
  executionId?: string

  metadata?: Record<string, unknown>
}

34. Audit Example

用户修改 Plugin 配置：

Actor:
user_123

Action:
plugin.configuration.update

Target:
plugin.seo

Before:
{
  "enabled": false
}

After:
{
  "enabled": true
}

Audit：

user_123
    ↓
plugin.configuration.update
    ↓
plugin.seo
    ↓
enabled false → true

35. Audit Immutable

Audit 创建以后：

UPDATE 禁止
DELETE 禁止

如果发现错误：

create correction audit

不能修改原始记录。

36. Audit Storage

37. Audit Query

支持：

actor
resource
action
operation
result
time range
organization
project
execution

例如：

谁修改了 Plugin？

谁删除了服务器？

谁安装了 MCP？

哪个 Agent 执行了危险操作？

这个 Resource 最近发生了什么？

38. Security Audit

以下操作必须 Audit：

login
logout
password change
role change
permission change
API key create
API key revoke
secret access
plugin install
plugin uninstall
plugin upgrade
MCP install
MCP configuration
agent policy change
approval
deployment
server operation
data export
file deletion

39. Agent Audit

Agent 执行：

User
 ↓
Agent
 ↓
Action

Audit 必须能看出：

actorType = agent
agentRunId = xxx
agentStepId = xxx
parentUserId = xxx

例如：

User
  ↓
SEO Agent
  ↓
seo.website.scan
  ↓
Execution

不能只记录：

system executed action

必须保留责任链。

40. MCP Audit

MCP 调用：

User
 ↓
Agent
 ↓
MCP Capability
 ↓
Action

必须记录：

mcpServerId
mcpToolId
capabilityId
actionId
executionId

41. Error Runtime

DevOS 必须统一错误模型。

export interface DevOSError {
  code: string

  category:
    | 'validation'
    | 'authentication'
    | 'authorization'
    | 'not_found'
    | 'conflict'
    | 'rate_limit'
    | 'timeout'
    | 'dependency'
    | 'network'
    | 'storage'
    | 'database'
    | 'internal'
    | 'unknown'

  message: string

  userMessage?: string

  retryable: boolean

  statusCode?: number

  cause?: unknown

  metadata?: Record<string, unknown>

  fingerprint?: string
}

42. Error Code

错误必须拥有稳定 Code。

例如：

CORE_AUTH_001
CORE_PERMISSION_001

TASK_NOT_FOUND
TASK_CANCELLED

JOB_LEASE_EXPIRED
JOB_RETRY_EXHAUSTED

ACTION_TIMEOUT
ACTION_PERMISSION_DENIED

PLUGIN_INCOMPATIBLE
PLUGIN_MIGRATION_FAILED

MCP_CONNECTION_FAILED

STORAGE_OBJECT_NOT_FOUND
STORAGE_QUOTA_EXCEEDED

SEARCH_INDEX_FAILED

43. Error Categories

错误至少分：

Validation
Authentication
Authorization
Not Found
Conflict
Rate Limit
Timeout
Dependency
Network
Storage
Database
Internal
Unknown

44. Retryable Error

每个 Error 必须明确：

retryable: boolean

例如：

HTTP 500
network timeout
temporary DNS failure
rate limit

可能：

retryable = true

而：

invalid input
permission denied
resource not found

通常：

retryable = false

45. Error Fingerprint

相同错误需要聚合。

例如：

DATABASE_TIMEOUT

10000 次不能产生 10000 个独立 Incident。

通过：

fingerprint

聚合：

error fingerprint
    ↓
Issue
    ↓
Occurrences

46. Error Handling Pipeline

Error
 ↓
Normalize
 ↓
Classify
 ↓
Fingerprint
 ↓
Log
 ↓
Metric
 ↓
Trace
 ↓
Audit if required
 ↓
Create/Update Issue
 ↓
Alert if threshold reached

47. Health Runtime

Health 必须区分：

Liveness
Readiness
Startup
Dependency Health

48. Liveness

回答：

进程还活着吗？

例如：

GET /health/live

只检查：

process
runtime

不能检查数据库。

49. Readiness

回答：

当前实例是否可以接收工作？

例如：

GET /health/ready

检查：

database
redis
queue
critical providers
migration state

50. Startup

启动检查：

configuration
database
migration
plugin registry
critical dependencies

51. Health Check

export interface HealthCheck {
  id: string

  name: string

  type:
    | 'liveness'
    | 'readiness'
    | 'startup'
    | 'dependency'

  check(): Promise<HealthResult>
}

52. Health Result

export interface HealthResult {
  status:
    | 'healthy'
    | 'degraded'
    | 'unhealthy'

  latencyMs: number

  message?: string

  metadata?: Record<string, unknown>
}

53. Dependency Health

例如：

DevOS
 ├── PostgreSQL      healthy
 ├── Redis           healthy
 ├── Storage         healthy
 ├── Search          degraded
 ├── OpenAI          healthy
 └── GitHub          healthy

54. Plugin Health

每个 Plugin 可以注册 Health Check。

例如：

plugin.registerHealthCheck({
  id: 'seo.google-search-console',
  check: async () => {}
})

但 Plugin 不得创建自己的 Health Runtime。

必须注册到：

DevOS Health Runtime

55. MCP Health

MCP Server 必须能够报告：

connected
disconnected
connecting
degraded
error

指标：

mcp_connection_count
mcp_request_count
mcp_error_count
mcp_latency

56. Diagnostic Runtime

Health 只回答：

有没有问题？

Diagnostic 回答：

为什么有问题？

57. Diagnostic Snapshot

定义：

export interface DiagnosticSnapshot {
  id: string

  createdAt: string

  system: {
    version: string
    runtimeVersion: string
    environment: string
  }

  health: unknown

  workers: unknown

  queues: unknown

  plugins: unknown

  mcp: unknown

  database: unknown

  storage: unknown

  search: unknown

  metrics?: unknown

  recentErrors?: unknown

  metadata?: Record<string, unknown>
}

58. Diagnostic Bundle

用户可以执行：

devos diagnostics collect

生成：

diagnostic bundle

例如：

diagnostics/
├── system.json
├── health.json
├── workers.json
├── queues.json
├── plugins.json
├── mcp.json
├── database.json
├── storage.json
├── recent-errors.json
└── metrics.json

59. Sensitive Diagnostic Data

Diagnostic Bundle 默认：

不包含：
password
token
apiKey
secret
cookie
private key
完整 Authorization

并进行：

redaction

60. Usage Runtime

DevOS 必须统一统计资源使用量。

例如：

LLM Tokens
LLM Requests
API Calls
Browser Minutes
CPU Time
Memory
Storage
Bandwidth
Database Queries
Search Queries
Agent Iterations
Action Executions

61. Usage Record

export interface UsageRecord {
  id: string

  timestamp: string

  organizationId?: string
  projectId?: string

  actorId?: string
  actorType?: string

  agentRunId?: string

  taskId?: string
  jobId?: string
  executionId?: string

  resourceType: string

  metric: string

  quantity: number

  unit: string

  provider?: string

  model?: string

  metadata?: Record<string, unknown>
}

62. LLM Usage

例如：

provider = openai
model = gpt-x
inputTokens = 10000
outputTokens = 2000
totalTokens = 12000

记录：

agentRunId
taskId
executionId

从而知道：

哪个 Agent / Task / Action 消耗了多少 Token。

63. Cost Runtime

Usage 不等于 Cost。

Usage：

10000 tokens

Cost：

$0.03

定义：

export interface CostRecord {
  usageId: string

  currency: string

  amount: number

  provider?: string

  model?: string

  pricingVersion?: string
}

64. Cost Attribution

成本必须能够归属：

Organization
Project
User
Agent
Task
Job
Execution
Plugin
Action
Provider
Model

例如：

Project A
 ├── Agent SEO
 │    ├── Task #1
 │    ├── Task #2
 │    └── Task #3
 │
 └── Total:
      12,500,000 tokens
      $24.13

65. Budget Integration

与 06、09 的 Budget Runtime 集成。

例如：

Agent Budget
 ├── maxTokens
 ├── maxExecutionTimeMs
 ├── maxTasks
 ├── maxMoney
 └── maxCredits

Usage Runtime 持续记录。

Budget Runtime 根据 Usage 判断：

continue
warning
pause
stop

66. Performance Runtime

Performance 负责回答：

DevOS 哪些地方慢？

重点指标：

HTTP latency
Action latency
Task queue wait
Job wait
Job duration
Database latency
Search latency
Storage latency
MCP latency
LLM latency
Worker utilization

67. Slow Operation

系统必须自动识别慢操作。

例如：

Action > 3s
DB Query > 1s
HTTP > 2s
Search > 1s
MCP > 5s

阈值必须配置化。

68. Performance Record

export interface PerformanceRecord {
  id: string

  type:
    | 'http'
    | 'action'
    | 'task'
    | 'job'
    | 'database'
    | 'search'
    | 'storage'
    | 'mcp'
    | 'llm'

  name: string

  durationMs: number

  thresholdMs?: number

  slow: boolean

  context: ObservabilityContext

  metadata?: Record<string, unknown>
}

69. Database Performance

Data Runtime 自动记录：

query duration
rows
query type
resource
provider
cache hit

不能默认记录完整 SQL 参数，因为可能包含敏感数据。

70. Search Performance

Search Runtime 自动记录：

search mode
resource
provider
query duration
result count
index
organization

禁止默认记录完整敏感搜索词。

71. SLI

DevOS 可以定义：

Availability
Latency
Error Rate
Queue Delay
Task Success Rate
Action Success Rate

例如：

Action Success Rate
=
successful executions
/
total executions

72. SLO

定义：

export interface SLO {
  id: string

  name: string

  target: number

  window:
    | 'hour'
    | 'day'
    | 'week'
    | 'month'

  indicator: string
}

例如：

API availability >= 99.9%

Action success >= 99%

Task completion <= 10 min >= 95%

73. Alert Runtime

Alert 用于：

系统出现需要关注的问题。

74. Alert Rule

export interface AlertRule {
  id: string

  name: string

  enabled: boolean

  source:
    | 'metric'
    | 'log'
    | 'error'
    | 'health'
    | 'audit'
    | 'usage'
    | 'slo'

  condition: AlertCondition

  severity:
    | 'info'
    | 'warning'
    | 'critical'

  cooldownSeconds: number

  deduplicationKey?: string

  notificationChannels: string[]
}

75. Alert Condition

例如：

queue depth > 1000

或者：

action failure rate > 10%

或者：

database health = unhealthy

或者：

LLM cost > $100/day

76. Alert State

inactive
    ↓
pending
    ↓
firing
    ↓
resolved

77. Alert Deduplication

例如：

5000 次数据库 timeout

不能发送：

5000 notifications

应该：

Issue #123
Occurrences: 5000

并根据：

deduplicationKey
cooldown
aggregation

控制通知。

78. Alert → Notification

Alert 不直接发送 Email。

使用：

Alert Runtime
      ↓
Event Runtime
      ↓
Notification Runtime
      ↓
Email / Webhook / Push / In-App

这与 16_EVENT_STREAMING_NOTIFICATION_REALTIME_RUNTIME.md 保持一致。

79. Issue Runtime

建议建立：

core.issue

用于聚合：

errors
alerts
incidents
performance problems

80. Issue

export interface Issue {
  id: string

  fingerprint: string

  title: string

  severity:
    | 'info'
    | 'warning'
    | 'error'
    | 'critical'

  status:
    | 'open'
    | 'acknowledged'
    | 'investigating'
    | 'resolved'
    | 'ignored'

  firstSeenAt: string

  lastSeenAt: string

  occurrenceCount: number

  service?: string

  pluginId?: string

  actionId?: string

  environmentId?: string

  metadata?: Record<string, unknown>
}

81. Issue Lifecycle

new
 ↓
open
 ↓
acknowledged
 ↓
investigating
 ↓
resolved

也允许：

ignored

82. Incident

Issue 与 Incident 不完全相同。

Issue：

某类错误持续出现。

Incident：

一个实际影响系统/业务的事件。

例如：

Issue:
Database timeout

Incident:
Production database unavailable for 8 minutes

83. Incident Resource

建议：

core.incident

包含：

severity
status
startedAt
resolvedAt
impact
timeline
relatedIssues
relatedTasks
relatedExecutions
relatedAlerts
relatedAuditRecords

84. Incident Timeline

可以自动聚合：

09:01 Database latency increased
09:02 Error rate increased
09:03 Alert fired
09:04 Incident created
09:05 Worker degraded
09:08 Database recovered
09:09 Incident resolved

来源：

Event
Log
Metric
Trace
Audit
Health

85. Observability Dashboard

DevOS UI 必须提供统一：

Observability

导航：

Observability
├── Overview
├── Logs
├── Metrics
├── Traces
├── Audit
├── Errors
├── Issues
├── Incidents
├── Health
├── Diagnostics
├── Performance
├── Usage
├── Cost
├── SLO
└── Alerts

86. Overview Dashboard

至少展示：

System Health

Requests / min

Error Rate

Action Success Rate

Queue Depth

Worker Status

Active Tasks

Active Jobs

Agent Runs

LLM Usage

LLM Cost

Storage Usage

Open Issues

Active Incidents

87. Trace Explorer

用户输入：

traceId

看到：

Trace
 ├── HTTP
 ├── Action
 ├── DB
 ├── Redis
 ├── MCP
 ├── LLM
 └── Storage

支持：

duration
attributes
events
errors
logs

88. Log Explorer

支持过滤：

level
service
plugin
action
task
job
execution
actor
trace
time
search

并支持：

Jump to Trace
Jump to Execution
Jump to Task
Jump to Job
Jump to Audit

89. Audit Explorer

支持：

Actor
Action
Resource
Operation
Result
Organization
Project
Time

点击 Audit 后：

Actor
Action
Target
Before
After
Reason
Approval
Execution
Trace

90. Execution Detail Integration

打开：

Execution #123

必须能够看到：

Execution
├── Status
├── Input
├── Output
├── Error
├── Logs
├── Trace
├── Metrics
├── Audit
├── Events
├── Artifacts
├── Usage
└── Cost

这是 DevOS 非常重要的统一观察入口。

91. Task Detail Integration

Task 页面：

Task
├── Overview
├── Jobs
├── Progress
├── Timeline
├── Logs
├── Errors
├── Events
├── Usage
├── Cost
└── Artifacts

92. Agent Run Detail

Agent Run：

Agent Run
├── Goal
├── Plan
├── Steps
├── Capabilities
├── Actions
├── Tasks
├── Errors
├── Logs
├── Usage
├── Cost
├── Approvals
└── Final Result

必须能够追踪：

User
 ↓
Agent Goal
 ↓
Agent Plan
 ↓
Skill
 ↓
Capability
 ↓
Action
 ↓
Task
 ↓
Execution

93. Plugin Observability

Plugin 可以注册：

logs
metrics
traces
health checks
audit definitions
usage metrics
alerts
dashboards

例如：

export interface PluginObservabilityExtension {
  registerLogger?(): void

  registerMetrics?(): void

  registerHealthChecks?(): void

  registerAuditDefinitions?(): void

  registerUsageMetrics?(): void

  registerAlertRules?(): void
}

但是：

Plugin 只能扩展 DevOS Runtime，不能自己创建独立 Observability Runtime。

94. Plugin Metrics Namespace

Plugin：

seo

只能产生：

seo_scan_total
seo_scan_duration
seo_scan_errors

或者统一命名：

devos_plugin_seo_scan_total

不能污染 Core：

devos_database_xxx

95. MCP Observability

MCP Gateway 自动产生：

mcp_request_total
mcp_request_error_total
mcp_request_duration
mcp_connection_status

每次 MCP 调用必须关联：

mcpServerId
mcpToolId
capabilityId
actionId
executionId
agentRunId

96. Agent Observability

Agent Runtime 自动记录：

agent.run.started
agent.run.completed
agent.run.failed
agent.plan.created
agent.plan.replanned
agent.step.started
agent.step.completed
agent.capability.selected
agent.action.requested
agent.task.created
agent.approval.requested

97. Do Not Store Chain of Thought

Observability Runtime：

不保存模型隐藏思维链。

可以保存：

decision summary
reason
selected capability
selected action
execution result
replan reason
error

例如：

{
  "decision": "use seo.website.scan",
  "reason": "website health information is required",
  "result": "completed"
}

而不是保存：

模型内部完整思维过程

98. Event Integration

与：

16_EVENT_STREAMING_NOTIFICATION_REALTIME_RUNTIME.md

集成。

重要 Observability Event：

observability.error.created
observability.issue.created
observability.issue.resolved
observability.alert.fired
observability.alert.resolved
observability.incident.created
observability.incident.resolved
observability.health.changed

99. Event vs Audit

必须明确：

Event
    = 系统发生了什么

Log
    = Runtime 运行过程中记录了什么

Audit
    = 谁对什么进行了什么重要操作

Metric
    = 系统运行数量/状态/趋势

Trace
    = 一次调用链怎么执行

Issue
    = 某类问题的聚合

Incident
    = 某个实际影响事件

禁止混用。

100. Database Schema

建议：

logs
audit_records
issues
incidents
alert_rules
alerts
usage_records
cost_records
slo_definitions
diagnostic_snapshots

Metrics / Traces 如果使用 OpenTelemetry Collector / 外部存储，则不强制全部进入 PostgreSQL。

101. Recommended Storage

默认：

PostgreSQL
    ├── Audit
    ├── Issues
    ├── Incidents
    ├── Alert Rules
    ├── Usage
    └── Configuration

Redis
    ├── Realtime metrics
    ├── Alert state
    ├── temporary aggregation
    └── distributed coordination

Object Storage
    ├── Diagnostic Bundles
    └── archived logs

OpenTelemetry-compatible backend
    ├── Traces
    └── Metrics

Log backend
    └── Logs

DevOS 不应该把所有 Observability 数据强行存进 PostgreSQL。

102. Retention Policy

不同数据必须拥有不同 retention。

例如：

Debug Logs:
7 days

Normal Logs:
30 days

Audit:
1 year+

Metrics:
90 days

Trace:
7-30 days

Diagnostic Bundle:
30 days

Usage:
1 year+

Cost:
1 year+

实际值必须配置化。

103. Archive

日志/审计等长期数据可以：

PostgreSQL
 ↓
Archive Job
 ↓
Object Storage

Archive 过程使用：

Task
 ↓
Job
 ↓
Worker

而不是 HTTP Request 长时间执行。

104. Observability API

建议：

GET /api/observability/logs
GET /api/observability/logs/stream

GET /api/observability/traces/:traceId

GET /api/observability/audit
GET /api/observability/audit/:id

GET /api/observability/errors

GET /api/observability/issues
GET /api/observability/issues/:id

GET /api/observability/incidents
GET /api/observability/incidents/:id

GET /api/observability/health

GET /api/observability/metrics

GET /api/observability/usage

GET /api/observability/cost

GET /api/observability/alerts

105. Health API

GET /health/live
GET /health/ready
GET /health/startup
GET /health

其中：

/live

用于 Kubernetes/container liveness。

/ready

用于 traffic readiness。

106. CLI

提供：

devos health
devos health check

devos logs
devos logs --follow
devos logs --level error

devos audit
devos audit search

devos trace <trace-id>

devos issues
devos incidents

devos diagnostics collect

devos usage
devos cost

devos alerts
devos alerts test

107. Diagnostic CLI

例如：

devos diagnostics collect

执行：

system information
health checks
worker status
queue status
plugin status
MCP status
database status
storage status
search status
recent errors
runtime versions

最后：

diagnostic artifact

通过 15_FILE_ARTIFACT_STORAGE_RUNTIME.md 保存。

108. Observability Context Middleware

HTTP：

app.use(observabilityMiddleware)

自动：

requestId
correlationId
traceId
actor
organization
project
environment

109. Worker Context

Worker 获取 Job：

const context = createObservabilityContext({
  taskId: job.taskId,
  jobId: job.id,
  executionId: execution.id,
})

然后：

logger.child(context)

110. Action Context

03_RESOURCE_ACTION_MODEL.md 中的：

ActionContext

必须包含：

observability: ObservabilityContext

例如：

export interface ActionContext {
  requestId: string
  executionId: string

  actor: Actor

  organizationId?: string
  projectId?: string
  environmentId?: string

  permissions: PermissionContext

  signal: AbortSignal

  observability: ObservabilityContext

  metadata?: Record<string, unknown>
}

111. Automatic Action Instrumentation

Action Runtime 统一包装：

async execute(actionId, input, context) {
  const span = tracing.startSpan(actionId)

  const start = Date.now()

  logger.info('action.started', {
    actionId
  })

  try {
    const result = await handler(input, context)

    metrics.counter('action_success_total', 1)

    auditIfRequired()

    return result
  } catch (error) {
    metrics.counter('action_failure_total', 1)

    handleError(error)

    throw error
  } finally {
    metrics.histogram(
      'action_duration_ms',
      Date.now() - start
    )

    span.end()
  }
}

业务 Action 不需要重复实现。

112. Automatic Task Instrumentation

Task Runtime 自动：

Task Created
Task Started
Task Progress
Task Paused
Task Resumed
Task Completed
Task Failed
Task Cancelled

并生成：

Log
Metric
Event
Trace where applicable

113. Automatic Job Instrumentation

Job Runtime 自动：

Job Queued
Job Leased
Job Started
Job Retrying
Job Completed
Job Failed
Job Cancelled

指标：

queue wait
processing duration
attempt count
retry count
failure rate

114. Database Instrumentation

Data Runtime 自动：

query count
query duration
rows
errors
transaction duration
connection pool status

慢查询自动进入：

Performance Runtime

115. Redis Instrumentation

Queue/Cache/Lock Runtime 自动统计：

redis commands
latency
errors
connection count
queue operations
lock contention

116. External Provider Instrumentation

统一记录：

provider
operation
latency
status
retry
rate limit
cost

例如：

OpenAI
GitHub
Google
AWS
Cloudflare
SSH
DNS
Browser

Plugin 不需要自己实现统计框架。

117. Cost Provider

定义：

export interface PricingProvider {
  getPrice(input: {
    provider: string
    model?: string
    metric: string
    quantity: number
  }): Promise<CostEstimate>
}

允许：

OpenAI pricing
Browser pricing
Cloud provider pricing
Plugin pricing

118. Cost Estimation

Agent 在执行高成本 Task 前可以：

estimate
 ↓
budget check
 ↓
approval if required
 ↓
execute

例如：

100 articles

Estimated:
Tokens: 8M
Cost: $17

Budget:
$10

Result:
REJECT / APPROVAL_REQUIRED

119. Alert Examples

系统默认提供：

High API Error Rate
Database Unhealthy
Redis Unhealthy
Queue Backlog
Worker Unhealthy
Action Failure Spike
Task Failure Spike
MCP Failure Spike
LLM Cost Spike
Storage Near Quota
Search Index Degraded
SLO Breached

120. Default Alert Policies

例如：

Action failure rate > 10%
for 5 minutes

或者：

queue depth > 1000
for 2 minutes

或者：

database health = unhealthy
for 30 seconds

121. Alert Permissions

只有拥有：

observability.alert.read
observability.alert.create
observability.alert.update
observability.alert.delete

的用户才能管理 Alert。

查看敏感 Audit 还需要：

observability.audit.read

122. Resource Definitions

DevOS Core 注册：

core.log
core.audit
core.issue
core.incident
core.alert
core.slo
core.usage
core.cost
core.diagnostic
core.health

其中部分可以设计为：

read-only resources

例如：

core.log
core.health

123. Resource Actions

例如：

core.audit.query
core.audit.export

core.issue.create
core.issue.update
core.issue.resolve
core.issue.ignore

core.incident.create
core.incident.update
core.incident.resolve

core.alert.create
core.alert.update
core.alert.enable
core.alert.disable
core.alert.test

core.diagnostic.collect

core.slo.create
core.slo.update

124. Export

Observability 数据可以导出：

CSV
JSON
NDJSON
Artifact

大规模导出：

Export Request
 ↓
Task
 ↓
Job
 ↓
Worker
 ↓
Artifact

不能：

HTTP request
 ↓
查询 10GB logs
 ↓
直接返回

125. Search Integration

Audit / Issues / Logs 可以接入：

Search Runtime

例如：

search:
"所有最近失败的 SEO scan"

Agent：

Semantic Search
      ↓
Search Runtime
      ↓
Issue / Execution / Audit

126. Agent Integration

Agent 可以使用：

observability.logs.search
observability.trace.get
observability.execution.inspect
observability.issue.search
observability.health.check
observability.usage.get
observability.cost.get

例如：

Agent Goal:

为什么昨天 SEO 扫描大量失败？

Agent：

Search Issues
 ↓
Inspect Errors
 ↓
Inspect Traces
 ↓
Inspect Task
 ↓
Inspect MCP
 ↓
Find root cause

127. Diagnostic Agent

可以提供：

System Diagnostic Agent

但它依然只能通过：

Capability
 ↓
Action Runtime

读取：

Logs
Metrics
Traces
Health
Issues
Tasks
Jobs
Executions

不能直接访问数据库。

128. Plugin Developer API

Plugin 开发者：

const plugin = definePlugin({
  id: 'seo',

  observability: {
    metrics: [
      'scan_total',
      'scan_duration',
      'scan_failed',
    ],

    healthChecks: [
      'google-api',
      'search-console',
    ],
  },
})

DevOS 自动添加 namespace：

plugin.seo.scan_total

129. Plugin Logging

Plugin：

const logger = ctx.logger

logger.info('scan started', {
  websiteId
})

而不是：

console.log()

生产环境禁止 Plugin 直接依赖：

console
pino
winston
bunyan

等具体 Logger 实现。

Plugin 依赖：

DevOS Logger interface

130. No Direct Infrastructure

禁止：

Plugin
 ├── new Redis()
 ├── new Logger()
 ├── new Queue()
 ├── new EventBus()
 └── new WebSocket()

应该：

Plugin
 ↓
DevOS Runtime
 ├── Logger
 ├── Metrics
 ├── Trace
 ├── Audit
 ├── Event
 ├── Queue
 └── Notification

131. Package Structure

建议：

packages/
├── observability/
│   ├── context/
│   ├── runtime/
│   └── types/
│
├── logging/
│   ├── runtime/
│   ├── sinks/
│   └── redaction/
│
├── metrics/
│   ├── runtime/
│   └── registry/
│
├── tracing/
│   ├── runtime/
│   └── propagation/
│
├── audit/
│   ├── runtime/
│   ├── repository/
│   └── policies/
│
├── error/
│   ├── runtime/
│   ├── codes/
│   └── normalization/
│
├── health/
│   ├── runtime/
│   └── checks/
│
├── diagnostics/
│   ├── runtime/
│   └── collectors/
│
├── usage/
│   ├── runtime/
│   └── providers/
│
├── cost/
│   ├── runtime/
│   └── pricing/
│
├── performance/
│   └── runtime/
│
├── alert/
│   ├── runtime/
│   ├── rules/
│   └── evaluation/
│
└── incident/
    └── runtime/

132. Recommended Backend Structure

NestJS：

src/
├── observability/
│   ├── observability.module.ts
│   ├── context/
│   ├── logging/
│   ├── metrics/
│   ├── tracing/
│   ├── audit/
│   ├── error/
│   ├── health/
│   ├── diagnostics/
│   ├── usage/
│   ├── cost/
│   ├── performance/
│   ├── alerts/
│   └── incidents/

133. Observability Module

@Module({
  providers: [
    ObservabilityRuntime,
    LoggingRuntime,
    MetricsRuntime,
    TracingRuntime,
    AuditRuntime,
    ErrorRuntime,
    HealthRuntime,
    DiagnosticRuntime,
    UsageRuntime,
    CostRuntime,
    PerformanceRuntime,
    AlertRuntime,
  ],

  exports: [
    ObservabilityRuntime,
    LoggingRuntime,
    MetricsRuntime,
    TracingRuntime,
    AuditRuntime,
    ErrorRuntime,
    HealthRuntime,
    DiagnosticRuntime,
    UsageRuntime,
    CostRuntime,
    PerformanceRuntime,
    AlertRuntime,
  ],
})
export class ObservabilityModule {}

134. Configuration

配置：

observability:
  logging:
    level: info
    retentionDays: 30

  tracing:
    enabled: true
    samplingRate: 0.1

  metrics:
    enabled: true

  audit:
    enabled: true
    retentionDays: 365

  diagnostics:
    enabled: true

  alerts:
    enabled: true

  cost:
    enabled: true

135. Sampling

Trace 可能数据量很大。

支持：

always
never
probabilistic
tail-based

例如生产：

normal traces: 10%
errors: 100%
slow traces: 100%

错误和慢请求应该提高采样率。

136. Backpressure

Observability 本身不能拖垮业务。

如果：

Log backend unavailable

不能导致：

Action failed

默认：

Business execution
    ↓
Observability asynchronously

Critical Audit 可以采用：

durable synchronous write

但必须明确策略。

137. Observability Failure Policy

Logging：

best effort

Metrics：

best effort

Tracing：

best effort

Audit：

durable

Security Audit：

durable

138. Audit Failure

如果某个高风险 Action：

必须写 Audit

而 Audit Storage 不可用：

默认：

Action blocked

例如：

delete production server

Audit 写入失败：

DENY

而普通：

read dashboard

不需要因此阻塞。

139. Security

Observability 数据本身也是敏感数据。

必须：

RBAC
Tenant isolation
Field-level security
Redaction
Retention
Encryption
Access Audit

尤其：

Audit
Logs
Trace attributes
Usage
Cost

140. Multi-Tenant Isolation

查询：

organizationId

必须自动注入。

例如：

User A
 ↓
Audit Query
 ↓
Scope Policy
 ↓
Organization A

不能查询：

Organization B

141. Field-Level Security

例如普通管理员：

可以看：

action
resource
status
time

但是：

不能看：

secret
before sensitive fields
private metadata

142. Audit Export Security

Audit Export 必须：

permission check
scope check
field masking
audit export itself

也就是说：

导出 Audit 本身也是 Audit。

143. Realtime Observability

与 16 集成：

Log Stream
Task Progress
Job Progress
Incident Timeline
Alert State
Worker Status
Health Status

通过：

SSE
WebSocket

推送。

144. Example: Task Detail Realtime

Task
 ↓
Event
 ↓
Realtime
 ↓
Browser

用户看到：

Task running 43%

Job #12 completed
Job #13 running
Job #14 retrying

Error:
MCP timeout

Cost:
$0.21

145. Observability Timeline

建议建立统一 Timeline API：

export interface TimelineEntry {
  timestamp: string

  type:
    | 'log'
    | 'event'
    | 'audit'
    | 'execution'
    | 'error'
    | 'alert'
    | 'health'

  title: string

  description?: string

  sourceId: string

  metadata?: Record<string, unknown>
}

用于：

Task
Job
Execution
Agent Run
Incident
Resource

146. Resource Timeline

任何 Resource 都可以拥有：

Timeline

例如：

Website
 ↓
Timeline

09:00 created
09:01 scan started
09:03 scan completed
09:05 keywords updated
09:07 backlink task started

底层数据来自：

Event
Audit
Execution

而不是单独再建一套 Timeline 数据系统。

147. Root Cause Analysis

Diagnostic Runtime 可以提供：

export interface RootCauseAnalysis {
  issueId: string

  suspectedCause?: string

  evidence: DiagnosticEvidence[]

  confidence?: number

  relatedResources: string[]

  relatedExecutions: string[]

  recommendations?: string[]
}

148. RCA Example

Issue:
SEO scan failure spike

Evidence:
1. MCP timeout increased
2. Google API latency increased
3. Action failure correlated with MCP calls
4. Database healthy
5. Worker healthy

Likely Cause:
Google Search Console API latency

Confidence:
0.87

149. Do Not Pretend RCA Is Certain

自动诊断必须区分：

observed
inferred
suspected
confirmed

例如：

suspectedCause

不能直接写：

rootCause = Google API

除非确实有人工确认或强证据。

150. Alert → Agent

未来可以：

Alert
 ↓
Agent Goal
 ↓
Diagnostic Skill
 ↓
Search Logs
 ↓
Inspect Trace
 ↓
Check Health
 ↓
Generate Diagnosis
 ↓
Ask Approval
 ↓
Remediation Task

例如：

Redis unhealthy
 ↓
DevOps Agent
 ↓
diagnose
 ↓
restart Redis

危险操作必须：

Approval

151. Self-Healing

允许：

Alert
 ↓
Agent
 ↓
Diagnostic
 ↓
Action

但必须满足：

Policy
Permission
Budget
Approval
Audit

例如：

restart worker

可以自动。

而：

delete production database

必须人工批准。

152. Observability Capability Registry

observability.logs.search
observability.logs.stream

observability.trace.inspect

observability.audit.search

observability.health.check

observability.issue.search
observability.issue.inspect

observability.diagnostic.collect

observability.usage.get
observability.cost.get

observability.performance.inspect

Agent 通过 Capability Registry 发现。

153. MCP Exposure

这些 Capability 可以选择暴露为：

MCP tools

例如：

search_logs
inspect_trace
inspect_execution
check_health
search_issues
get_usage
get_cost

但是：

MCP
 ↓
Capability
 ↓
Action Runtime

不能：

MCP
 ↓
direct database

154. CLI / API / UI / Agent Unified Runtime

必须保持：

UI
API
CLI
MCP
Agent
Workflow
Plugin
Scheduler
        ↓
       Action
        ↓
 Action Runtime
        ↓
Observability Runtime

因此同一个操作：

devos task cancel xxx

和：

UI → Cancel

以及：

Agent → task.cancel

得到完全一致的：

Audit
Log
Metric
Trace
Event

155. Testing Strategy

必须至少测试：

Context propagation
Logging
Redaction
Metrics
Tracing
Audit
Errors
Health
Diagnostics
Usage
Cost
Performance
Alerts
SLO
Multi-tenancy
Security
Realtime
Agent
MCP
Plugin

156. Context Tests

测试：

HTTP
 ↓
Action
 ↓
Task
 ↓
Job
 ↓
Worker
 ↓
Execution

最终：

correlationId
traceId
executionId
taskId
jobId

能够正确关联。

157. Redaction Tests

输入：

{
  "password": "secret",
  "token": "abc",
  "authorization": "Bearer xxx"
}

输出：

{
  "password": "[REDACTED]",
  "token": "[REDACTED]",
  "authorization": "[REDACTED]"
}

158. Audit Tests

验证：

create
update
delete
permission change
plugin install
plugin upgrade
agent execution
approval
export

全部生成正确 Audit。

159. Audit Immutability Test

尝试：

UPDATE audit_records
DELETE audit_records

必须失败。

160. Multi-Tenant Tests

验证：

Organization A

不能看到：

Organization B

的：

Logs
Audit
Usage
Cost
Issues
Incidents

161. Error Tests

测试：

retryable
non-retryable
timeout
network
permission
validation
internal

错误必须正确分类。

162. Alert Tests

测试：

threshold reached
cooldown
deduplication
resolve
re-fire
notification

例如：

100 errors

只能产生：

1 Alert

而不是：

100 Alerts

163. Health Tests

测试：

healthy
degraded
unhealthy

以及：

database down
redis down
queue down
plugin unhealthy
MCP unhealthy

164. Diagnostic Tests

执行：

devos diagnostics collect

验证：

bundle generated
secrets redacted
artifact created
audit created

165. Cost Tests

模拟：

LLM 10000 tokens

验证：

UsageRecord
CostRecord
Task attribution
Agent attribution
Project attribution

都正确。

166. Performance Tests

必须能够发现：

slow Action
slow Query
slow Search
slow MCP
slow LLM

167. Agent Tests

模拟：

Agent
 ↓
Capability
 ↓
Action
 ↓
Task
 ↓
Job
 ↓
Execution

验证：

Agent Run
Logs
Metrics
Trace
Audit
Usage
Cost
Events

完整关联。

168. Plugin Tests

Plugin 安装：

Plugin
 ↓
register metrics
 ↓
register health
 ↓
register audit
 ↓
register alerts

卸载后：

Plugin metrics removed
Plugin health removed
Plugin capabilities removed

历史 Observability 数据必须保留。

169. MCP Tests

测试：

MCP request
 ↓
Capability
 ↓
Action
 ↓
Execution

验证：

trace
log
metric
audit
usage
cost

170. Load Tests

至少测试：

10k logs/sec
1k metrics/sec
100 concurrent traces
1000 task executions
10000 jobs
100 workers

实际阈值根据部署规模调整。

171. Failure Tests

模拟：

Log backend down
Metric backend down
Trace backend down
Audit DB down
Redis down
PostgreSQL slow
Storage unavailable

验证业务系统不会因为普通 Observability 失败而整体崩溃。

172. Production Acceptance

Zcode 完成后必须满足：

Logging

[ ] structured logs
[ ] context propagation
[ ] redaction
[ ] query
[ ] stream
[ ] retention

Metrics

[ ] counter
[ ] gauge
[ ] histogram
[ ] cardinality protection

Tracing

[ ] trace
[ ] span
[ ] propagation
[ ] action instrumentation

Audit

[ ] immutable
[ ] actor
[ ] resource
[ ] before/after
[ ] approval
[ ] export

Error

[ ] error codes
[ ] categories
[ ] retryable
[ ] fingerprint

Health

[ ] liveness
[ ] readiness
[ ] startup
[ ] dependencies

Diagnostics

[ ] snapshot
[ ] diagnostic bundle
[ ] redaction
[ ] artifact

Usage / Cost

[ ] tokens
[ ] API calls
[ ] compute
[ ] storage
[ ] cost
[ ] attribution
[ ] budget

Performance

[ ] latency
[ ] slow operation
[ ] database
[ ] search
[ ] MCP
[ ] LLM

Alert

[ ] rules
[ ] threshold
[ ] cooldown
[ ] dedup
[ ] notification

SLO

[ ] SLI
[ ] SLO
[ ] breach detection

173. Zcode Implementation Rules

这是本文件最重要的部分之一。

Rule 1

禁止：

Plugin → console.log

必须：

Plugin → DevOS Logger

Rule 2

禁止：

Plugin → 自己创建 Metrics

必须：

Plugin → Metrics Runtime

Rule 3

禁止：

Plugin → 自己创建 Audit Table

必须：

Plugin → Audit Runtime

Rule 4

禁止：

Plugin → 自己创建 Error Framework

必须：

Plugin → Error Runtime

Rule 5

禁止：

MCP → direct DB

必须：

MCP
 ↓
Capability
 ↓
Action Runtime

Rule 6

禁止：

Agent → direct Logger
Agent → direct DB
Agent → direct Redis
Agent → direct Queue

必须：

Agent
 ↓
Capability
 ↓
Runtime

Rule 7

禁止：

每个 Plugin 自己建立：
Logs
Metrics
Audit
Health
Alerts

必须：

DevOS Observability Runtime
             ↑
          Plugins

174. Golden Integration

最终 DevOS 的一次操作：

User
 ↓
UI / API / CLI / MCP / Agent
 ↓
Goal / Task
 ↓
Job
 ↓
Action Runtime
 ↓
Execution
 ↓
Business Logic
 ↓
Event

同时：

             Execution
                 │
       ┌─────────┼─────────┐
       ↓         ↓         ↓
     Trace      Logs     Metrics
       │         │         │
       └─────────┼─────────┘
                 ↓
              Error
                 ↓
        ┌────────┴────────┐
        ↓                 ↓
      Issue              Alert
        │                 │
        └────────┬────────┘
                 ↓
             Incident

另外：

Execution
    ↓
Audit
    ↓
Usage
    ↓
Cost

最终：

Execution
├── Logs
├── Metrics
├── Trace
├── Error
├── Audit
├── Event
├── Usage
├── Cost
├── Performance
└── Artifact

175. Final Architecture

完成本文件以后，DevOS Core 应该形成：

                         DEVOS CORE
                             │
 ┌───────────────────────────┼────────────────────────────┐
 │                           │                            │
Resource                 Action Runtime             Data Runtime
 │                           │                            │
 │                           │                            │
 └───────────────┬───────────┴───────────────┬────────────┘
                 │                           │
              Task Runtime              Event Runtime
                 │                           │
        ┌────────┼────────┐          ┌───────┼────────┐
        │        │        │          │       │        │
       Job      Queue    Worker     Event   Webhook Notification
        │        │        │          │       │        │
        └────────┴────────┘          └───────┴────────┘
                 │
                 ↓
             Execution
                 │
       ┌─────────┼──────────────────────┐
       │         │                      │
       ↓         ↓                      ↓
 Observability  Storage              Search
       │
 ┌─────┼────────────┬──────────┬──────────┐
 │     │            │          │          │
Logs Metrics      Trace      Audit      Error
 │     │            │          │          │
 └─────┴────────────┴──────────┴──────────┘
                         │
                ┌────────┼─────────┐
                ↓        ↓         ↓
             Health   Diagnostic  Performance
                │        │         │
                └────────┼─────────┘
                         ↓
                    Usage / Cost
                         ↓
                       Alert
                         ↓
                    Incident

176. DevOS Unified Runtime Principle

到这里，DevOS 已经不能再被理解成：

NestJS + React + PostgreSQL

而应该理解为：

DevOS
│
├── Resource Runtime
├── Action Runtime
├── Data Runtime
├── Task Runtime
├── Event Runtime
├── Storage Runtime
├── Search Runtime
├── Observability Runtime
├── Plugin Runtime
├── MCP Runtime
├── Skill Runtime
├── Agent Runtime
├── Workflow Runtime
├── Security Runtime
└── UI Runtime

其中：

Observability Runtime

是整个系统的“神经系统”。

177. Final Golden Rule

任何进入 DevOS 的行为，都必须能够被定位、执行、记录、追踪和解释。

最终统一链路：

Actor
  ↓
Request
  ↓
Goal / Task
  ↓
Job
  ↓
Action
  ↓
Action Runtime
  ↓
Execution
  ↓
Event
  ↓
Observability Runtime
  │
  ├── Log
  ├── Metric
  ├── Trace
  ├── Audit
  ├── Error
  ├── Health
  ├── Performance
  ├── Usage
  ├── Cost
  ├── Issue
  ├── Alert
  └── Incident

Zcode 不允许任何 Plugin、MCP、Agent、Workflow、Worker 或业务模块绕过这条链路建立自己的日志、审计、指标、错误、健康检查、队列监控或告警体系。

64 KiB Raw Permalink Blame History Unescape Escape

17_OBSERVABILITY_AUDIT_LOGGING_RUNTIME.md

1. Document Purpose

2. Core Principles

2.1 Everything Is Correlated

3. Observability Context

4. Context Propagation

5. Request ID

6. Correlation ID

7. Trace ID

8. Span

9. Logging Runtime

10. Structured Logging

11. Log Entry

12. Log Levels

DEBUG

INFO

WARN

ERROR

FATAL

13. Sensitive Data Protection

14. Redaction Runtime

15. Logging Sink

16. Log Runtime

17. Log Query

18. Metrics Runtime

19. Metric Types

20. Counter

21. Gauge

22. Histogram

23. Metric API

24. Metric Cardinality

25. Tracing Runtime

26. Span API

27. Action Runtime Integration

28. Task / Job Integration

29. Worker Observability

30. Queue Observability

31. Audit Runtime

32. Audit Requirements

33. Audit Record

34. Audit Example

35. Audit Immutable

36. Audit Storage

37. Audit Query

38. Security Audit

39. Agent Audit

40. MCP Audit

41. Error Runtime

42. Error Code

43. Error Categories

44. Retryable Error

45. Error Fingerprint

46. Error Handling Pipeline

47. Health Runtime

48. Liveness

49. Readiness

50. Startup

51. Health Check

52. Health Result

53. Dependency Health

54. Plugin Health

55. MCP Health

56. Diagnostic Runtime

57. Diagnostic Snapshot

58. Diagnostic Bundle

59. Sensitive Diagnostic Data

60. Usage Runtime

61. Usage Record

62. LLM Usage

63. Cost Runtime

64. Cost Attribution

65. Budget Integration

66. Performance Runtime

67. Slow Operation

68. Performance Record

69. Database Performance

70. Search Performance

71. SLI

72. SLO

64 KiB

Raw Permalink Blame History

`17_OBSERVABILITY_AUDIT_LOGGING_RUNTIME.md`