# `17_OBSERVABILITY_AUDIT_LOGGING_RUNTIME.md` > Version: 1.0 > Status: Implementation Specification > Target: Zcode > Depends on: > > * `03_RESOURCE_ACTION_MODEL.md` > * `06_MCP_SKILL_AGENT_RUNTIME.md` > * `09_TASK_JOB_QUEUE_RUNTIME.md` > * `12_MIGRATION_VERSIONING_COMPATIBILITY.md` > * `14_DATA_QUERY_SEARCH_INDEX_RUNTIME.md` > * `15_FILE_ARTIFACT_STORAGE_RUNTIME.md` > * `16_EVENT_STREAMING_NOTIFICATION_REALTIME_RUNTIME.md` --- # 1. Document Purpose 本文件定义 DevOS 统一的： * Logging Runtime * Metrics Runtime * Tracing Runtime * Audit Runtime * Error Runtime * Health Runtime * Diagnostic Runtime * Usage / Cost Runtime * Performance Runtime * SLO / SLA Runtime * Alert Runtime 目标： > DevOS 中所有 Core、Plugin、MCP、Agent、Task、Job、Action、Worker、Scheduler、Storage、Search、Realtime 都必须使用统一的 Observability Runtime。 最终形成： ```text DevOS Observability Runtime │ ┌───────────────────┼───────────────────┐ │ │ │ Logging Metrics Tracing │ │ │ └───────────────────┼───────────────────┘ │ ┌───────────────────┼───────────────────┐ │ │ │ Audit Error Health │ │ │ └───────────────────┼───────────────────┘ │ ┌───────────────────┼───────────────────┐ │ │ │ Diagnostics Usage/Cost Performance │ ↓ Alert │ ↓ Notification / Event Runtime ``` 核心目标不是“增加日志”。 而是： > **让 DevOS 中任何一次行为都可以被定位、解释、审计、统计、诊断和追踪。** --- # 2. Core Principles DevOS Observability 必须遵循以下原则。 ## 2.1 Everything Is Correlated 所有 Runtime 行为必须能够关联： ```text requestId correlationId traceId spanId executionId taskId jobId agentRunId pluginId actionId resourceId actorId organizationId projectId environmentId ``` 最少要求： ```text request ↓ trace ↓ execution ↓ task/job ↓ action ↓ event ``` 可以完整还原。 --- # 3. Observability Context 定义统一： ```ts export interface ObservabilityContext { requestId?: string correlationId: string traceId?: string spanId?: string executionId?: string taskId?: string jobId?: string agentRunId?: string agentStepId?: string pluginId?: string pluginVersion?: string actionId?: string actionVersion?: string resourceType?: string resourceId?: string actorId?: string actorType?: 'user' | 'agent' | 'service' | 'system' organizationId?: string projectId?: string environmentId?: string metadata?: Record } ``` 任何 Runtime 都必须能够接收这个 Context。 --- # 4. Context Propagation Context 必须自动传播。 例如： ```text HTTP Request ↓ RequestContext ↓ Action Runtime ↓ Task ↓ Job ↓ Worker ↓ Execution ↓ External API ``` 每一层都不能重新生成一套互不关联的 ID。 --- # 5. Request ID `requestId` 表示： > 一次进入 DevOS 的请求。 例如： ```text HTTP CLI Webhook MCP Agent Scheduler Plugin ``` 都可以产生 request。 --- # 6. Correlation ID `correlationId` 表示： > 一组相关操作。 例如： ```text 用户创建 SEO Scan ↓ Task ↓ 10 Jobs ↓ 10 Executions ↓ 100 Events ``` 全部共享： ```text correlationId ``` --- # 7. Trace ID `traceId` 表示： > 一次完整分布式调用链。 推荐使用： ```text OpenTelemetry ``` 作为 Trace 标准。 DevOS 不应该自己重新发明分布式 tracing 协议。 --- # 8. Span 一个 Runtime 操作可以产生 Span。 例如： ```text Trace ├── HTTP POST /tasks │ ├── action.task.create │ ├── task.enqueue │ ├── redis.publish │ └── worker.execute ├── action.seo.scan ├── dns.lookup ├── http.request └── artifact.create ``` --- # 9. Logging Runtime 定义统一 Logger： ```ts export interface Logger { debug(message: string, data?: LogData): void info(message: string, data?: LogData): void warn(message: string, data?: LogData): void error(message: string, data?: LogData): void fatal(message: string, data?: LogData): void child(context: Partial): Logger } ``` --- # 10. Structured Logging 禁止： ```ts logger.info( `User ${userId} executed ${actionId} on ${resourceId}` ) ``` 必须使用结构化数据： ```ts logger.info('action executed', { actionId, resourceId, actorId, executionId, durationMs, }) ``` --- # 11. Log Entry 统一： ```ts export interface LogEntry { id: string timestamp: string level: | 'debug' | 'info' | 'warn' | 'error' | 'fatal' message: string context: ObservabilityContext category: | 'system' | 'request' | 'action' | 'task' | 'job' | 'agent' | 'plugin' | 'mcp' | 'database' | 'queue' | 'worker' | 'storage' | 'search' | 'security' | 'integration' data?: Record error?: LogError durationMs?: number source?: { service?: string module?: string file?: string line?: number } } ``` --- # 12. Log Levels 标准： ```text DEBUG INFO WARN ERROR FATAL ``` 使用规则： ### DEBUG 开发调试信息。 ### INFO 正常业务行为。 ### WARN 异常但系统仍然正常。 ### ERROR 操作失败。 ### FATAL 服务无法继续运行。 --- # 13. Sensitive Data Protection 日志绝对不能直接输出： ```text password token apiKey secret privateKey cookie authorization session creditCard ``` 例如： ```ts logger.info('plugin configured', { apiKey: 'xxx' }) ``` 必须自动变成： ```text apiKey: "[REDACTED]" ``` --- # 14. Redaction Runtime 定义： ```ts export interface RedactionRule { path: string action: | 'redact' | 'hash' | 'mask' | 'remove' } ``` 例如： ```ts { path: 'authorization', action: 'redact' } ``` 支持： ```text global rules tenant rules plugin rules resource rules ``` --- # 15. Logging Sink Logger 不直接决定日志存储位置。 定义： ```ts export interface LogSink { write(entry: LogEntry): Promise writeBatch(entries: LogEntry[]): Promise flush(): Promise } ``` 可以支持： ```text Console PostgreSQL Object Storage Loki OpenSearch Cloud Logging ``` DevOS Runtime 只定义抽象。 --- # 16. Log Runtime ```ts export interface LogRuntime { logger(context?: ObservabilityContext): Logger query(input: LogQuery): Promise stream(input: LogStreamInput): AsyncIterable } ``` --- # 17. Log Query ```ts export interface LogQuery { levels?: string[] categories?: string[] service?: string pluginId?: string actionId?: string executionId?: string taskId?: string jobId?: string traceId?: string actorId?: string organizationId?: string from?: string to?: string search?: string limit?: number } ``` --- # 18. Metrics Runtime Metrics 用于回答： > 系统运行得怎么样？ 例如： ```text Action execution count Action failure count Action latency Queue depth Worker utilization API request rate LLM token usage Storage usage Search latency Database latency ``` --- # 19. Metric Types 支持： ```text Counter Gauge Histogram Summary ``` --- # 20. Counter 例如： ```text devos_action_executions_total devos_action_failures_total devos_http_requests_total devos_agent_runs_total ``` --- # 21. Gauge 例如： ```text devos_queue_depth devos_worker_active devos_storage_bytes devos_active_websocket_connections ``` --- # 22. Histogram 用于： ```text HTTP latency Action latency Database query latency Queue wait time Job duration LLM latency ``` 例如： ```text devos_action_duration_ms ``` --- # 23. Metric API ```ts export interface MetricsRuntime { counter( name: string, value?: number, attributes?: Record ): void gauge( name: string, value: number, attributes?: Record ): void histogram( name: string, value: number, attributes?: Record ): void } ``` --- # 24. Metric Cardinality 禁止产生无限 Cardinality。 危险： ```text userId requestId executionId URL query ``` 不能直接作为 Metric label。 允许： ```text actionId pluginId status method provider environment ``` 必须提供 Cardinality Protection。 --- # 25. Tracing Runtime ```ts export interface TracingRuntime { startSpan( name: string, options?: SpanOptions ): Span } ``` --- # 26. Span API ```ts export interface Span { setAttribute( key: string, value: string | number | boolean ): void addEvent( name: string, attributes?: Record ): void recordException(error: unknown): void setStatus( status: 'ok' | 'error' ): void end(): void } ``` --- # 27. Action Runtime Integration 这是必须实现的。 所有： ```text Action ``` 执行自动生成： ```text Span Log Metric Audit Execution ``` 例如： ```text action.seo.scan ``` 自动： ```text Trace └── action.seo.scan ├── validation ├── authorization ├── database ├── external-http ├── storage └── result ``` 业务 Action 不需要自己实现这些。 --- # 28. Task / Job Integration Task： ```text task.created task.started task.completed task.failed ``` Job： ```text job.queued job.started job.retrying job.completed job.failed ``` 自动生成： ```text metrics logs traces ``` --- # 29. Worker Observability Worker 必须暴露： ```text worker.started worker.ready worker.busy worker.draining worker.unhealthy worker.stopped ``` Metrics： ```text worker_jobs_total worker_jobs_failed_total worker_active_jobs worker_utilization worker_heartbeat_age worker_queue_wait_ms worker_job_duration_ms ``` --- # 30. Queue Observability 必须记录： ```text queue depth enqueue rate dequeue rate wait time processing time retry count DLQ count ``` 例如： ```text devos_queue_depth{queue="default"} devos_queue_depth{queue="ai"} devos_queue_depth{queue="browser"} ``` --- # 31. Audit Runtime Audit 和 Log **不是一回事**。 Log： > 系统运行发生了什么？ Audit： > 谁对什么资源做了什么具有业务/安全意义的操作？ --- # 32. Audit Requirements Audit 必须： ```text durable immutable queryable tenant-isolated actor-aware resource-aware ``` 不能把普通日志当 Audit。 --- # 33. Audit Record ```ts export interface AuditRecord { id: string timestamp: string actor: { id?: string type: 'user' | 'agent' | 'service' | 'system' } organizationId?: string projectId?: string environmentId?: string actionId: string actionVersion?: string resourceType?: string resourceId?: string operation: | 'create' | 'update' | 'delete' | 'read' | 'execute' | 'approve' | 'reject' | 'login' | 'logout' | 'install' | 'upgrade' | 'configure' result: | 'success' | 'failure' | 'denied' reason?: string before?: unknown after?: unknown changes?: unknown approvalId?: string requestId?: string correlationId?: string executionId?: string metadata?: Record } ``` --- # 34. Audit Example 用户修改 Plugin 配置： ```text Actor: user_123 Action: plugin.configuration.update Target: plugin.seo Before: { "enabled": false } After: { "enabled": true } ``` Audit： ```text user_123 ↓ plugin.configuration.update ↓ plugin.seo ↓ enabled false → true ``` --- # 35. Audit Immutable Audit 创建以后： ```text UPDATE 禁止 DELETE 禁止 ``` 如果发现错误： ```text create correction audit ``` 不能修改原始记录。 --- # 36. Audit Storage 推荐： ```text PostgreSQL ``` 表： ```text audit_records ``` 核心字段： ```text id timestamp actor_id actor_type organization_id project_id action_id action_version resource_type resource_id operation result reason before_json after_json changes_json approval_id request_id correlation_id execution_id metadata_json ``` --- # 37. Audit Query 支持： ```text actor resource action operation result time range organization project execution ``` 例如： ```text 谁修改了 Plugin？ 谁删除了服务器？ 谁安装了 MCP？ 哪个 Agent 执行了危险操作？ 这个 Resource 最近发生了什么？ ``` --- # 38. Security Audit 以下操作必须 Audit： ```text login logout password change role change permission change API key create API key revoke secret access plugin install plugin uninstall plugin upgrade MCP install MCP configuration agent policy change approval deployment server operation data export file deletion ``` --- # 39. Agent Audit Agent 执行： ```text User ↓ Agent ↓ Action ``` Audit 必须能看出： ```text actorType = agent agentRunId = xxx agentStepId = xxx parentUserId = xxx ``` 例如： ```text User ↓ SEO Agent ↓ seo.website.scan ↓ Execution ``` 不能只记录： ```text system executed action ``` 必须保留责任链。 --- # 40. MCP Audit MCP 调用： ```text User ↓ Agent ↓ MCP Capability ↓ Action ``` 必须记录： ```text mcpServerId mcpToolId capabilityId actionId executionId ``` --- # 41. Error Runtime DevOS 必须统一错误模型。 ```ts export interface DevOSError { code: string category: | 'validation' | 'authentication' | 'authorization' | 'not_found' | 'conflict' | 'rate_limit' | 'timeout' | 'dependency' | 'network' | 'storage' | 'database' | 'internal' | 'unknown' message: string userMessage?: string retryable: boolean statusCode?: number cause?: unknown metadata?: Record fingerprint?: string } ``` --- # 42. Error Code 错误必须拥有稳定 Code。 例如： ```text CORE_AUTH_001 CORE_PERMISSION_001 TASK_NOT_FOUND TASK_CANCELLED JOB_LEASE_EXPIRED JOB_RETRY_EXHAUSTED ACTION_TIMEOUT ACTION_PERMISSION_DENIED PLUGIN_INCOMPATIBLE PLUGIN_MIGRATION_FAILED MCP_CONNECTION_FAILED STORAGE_OBJECT_NOT_FOUND STORAGE_QUOTA_EXCEEDED SEARCH_INDEX_FAILED ``` --- # 43. Error Categories 错误至少分： ```text Validation Authentication Authorization Not Found Conflict Rate Limit Timeout Dependency Network Storage Database Internal Unknown ``` --- # 44. Retryable Error 每个 Error 必须明确： ```ts retryable: boolean ``` 例如： ```text HTTP 500 network timeout temporary DNS failure rate limit ``` 可能： ```text retryable = true ``` 而： ```text invalid input permission denied resource not found ``` 通常： ```text retryable = false ``` --- # 45. Error Fingerprint 相同错误需要聚合。 例如： ```text DATABASE_TIMEOUT ``` 10000 次不能产生 10000 个独立 Incident。 通过： ```text fingerprint ``` 聚合： ```text error fingerprint ↓ Issue ↓ Occurrences ``` --- # 46. Error Handling Pipeline ```text Error ↓ Normalize ↓ Classify ↓ Fingerprint ↓ Log ↓ Metric ↓ Trace ↓ Audit if required ↓ Create/Update Issue ↓ Alert if threshold reached ``` --- # 47. Health Runtime Health 必须区分： ```text Liveness Readiness Startup Dependency Health ``` --- # 48. Liveness 回答： > 进程还活着吗？ 例如： ```http GET /health/live ``` 只检查： ```text process runtime ``` 不能检查数据库。 --- # 49. Readiness 回答： > 当前实例是否可以接收工作？ 例如： ```http GET /health/ready ``` 检查： ```text database redis queue critical providers migration state ``` --- # 50. Startup 启动检查： ```text configuration database migration plugin registry critical dependencies ``` --- # 51. Health Check ```ts export interface HealthCheck { id: string name: string type: | 'liveness' | 'readiness' | 'startup' | 'dependency' check(): Promise } ``` --- # 52. Health Result ```ts export interface HealthResult { status: | 'healthy' | 'degraded' | 'unhealthy' latencyMs: number message?: string metadata?: Record } ``` --- # 53. Dependency Health 例如： ```text DevOS ├── PostgreSQL healthy ├── Redis healthy ├── Storage healthy ├── Search degraded ├── OpenAI healthy └── GitHub healthy ``` --- # 54. Plugin Health 每个 Plugin 可以注册 Health Check。 例如： ```ts plugin.registerHealthCheck({ id: 'seo.google-search-console', check: async () => {} }) ``` 但 Plugin 不得创建自己的 Health Runtime。 必须注册到： ```text DevOS Health Runtime ``` --- # 55. MCP Health MCP Server 必须能够报告： ```text connected disconnected connecting degraded error ``` 指标： ```text mcp_connection_count mcp_request_count mcp_error_count mcp_latency ``` --- # 56. Diagnostic Runtime Health 只回答： > 有没有问题？ Diagnostic 回答： > 为什么有问题？ --- # 57. Diagnostic Snapshot 定义： ```ts export interface DiagnosticSnapshot { id: string createdAt: string system: { version: string runtimeVersion: string environment: string } health: unknown workers: unknown queues: unknown plugins: unknown mcp: unknown database: unknown storage: unknown search: unknown metrics?: unknown recentErrors?: unknown metadata?: Record } ``` --- # 58. Diagnostic Bundle 用户可以执行： ```bash devos diagnostics collect ``` 生成： ```text diagnostic bundle ``` 例如： ```text diagnostics/ ├── system.json ├── health.json ├── workers.json ├── queues.json ├── plugins.json ├── mcp.json ├── database.json ├── storage.json ├── recent-errors.json └── metrics.json ``` --- # 59. Sensitive Diagnostic Data Diagnostic Bundle 默认： ```text 不包含： password token apiKey secret cookie private key 完整 Authorization ``` 并进行： ```text redaction ``` --- # 60. Usage Runtime DevOS 必须统一统计资源使用量。 例如： ```text LLM Tokens LLM Requests API Calls Browser Minutes CPU Time Memory Storage Bandwidth Database Queries Search Queries Agent Iterations Action Executions ``` --- # 61. Usage Record ```ts export interface UsageRecord { id: string timestamp: string organizationId?: string projectId?: string actorId?: string actorType?: string agentRunId?: string taskId?: string jobId?: string executionId?: string resourceType: string metric: string quantity: number unit: string provider?: string model?: string metadata?: Record } ``` --- # 62. LLM Usage 例如： ```text provider = openai model = gpt-x inputTokens = 10000 outputTokens = 2000 totalTokens = 12000 ``` 记录： ```text agentRunId taskId executionId ``` 从而知道： > 哪个 Agent / Task / Action 消耗了多少 Token。 --- # 63. Cost Runtime Usage 不等于 Cost。 Usage： ```text 10000 tokens ``` Cost： ```text $0.03 ``` 定义： ```ts export interface CostRecord { usageId: string currency: string amount: number provider?: string model?: string pricingVersion?: string } ``` --- # 64. Cost Attribution 成本必须能够归属： ```text Organization Project User Agent Task Job Execution Plugin Action Provider Model ``` 例如： ```text Project A ├── Agent SEO │ ├── Task #1 │ ├── Task #2 │ └── Task #3 │ └── Total: 12,500,000 tokens $24.13 ``` --- # 65. Budget Integration 与 `06`、`09` 的 Budget Runtime 集成。 例如： ```text Agent Budget ├── maxTokens ├── maxExecutionTimeMs ├── maxTasks ├── maxMoney └── maxCredits ``` Usage Runtime 持续记录。 Budget Runtime 根据 Usage 判断： ```text continue warning pause stop ``` --- # 66. Performance Runtime Performance 负责回答： > DevOS 哪些地方慢？ 重点指标： ```text HTTP latency Action latency Task queue wait Job wait Job duration Database latency Search latency Storage latency MCP latency LLM latency Worker utilization ``` --- # 67. Slow Operation 系统必须自动识别慢操作。 例如： ```text Action > 3s DB Query > 1s HTTP > 2s Search > 1s MCP > 5s ``` 阈值必须配置化。 --- # 68. Performance Record ```ts export interface PerformanceRecord { id: string type: | 'http' | 'action' | 'task' | 'job' | 'database' | 'search' | 'storage' | 'mcp' | 'llm' name: string durationMs: number thresholdMs?: number slow: boolean context: ObservabilityContext metadata?: Record } ``` --- # 69. Database Performance Data Runtime 自动记录： ```text query duration rows query type resource provider cache hit ``` 不能默认记录完整 SQL 参数，因为可能包含敏感数据。 --- # 70. Search Performance Search Runtime 自动记录： ```text search mode resource provider query duration result count index organization ``` 禁止默认记录完整敏感搜索词。 --- # 71. SLI DevOS 可以定义： ```text Availability Latency Error Rate Queue Delay Task Success Rate Action Success Rate ``` 例如： ```text Action Success Rate = successful executions / total executions ``` --- # 72. SLO 定义： ```ts export interface SLO { id: string name: string target: number window: | 'hour' | 'day' | 'week' | 'month' indicator: string } ``` 例如： ```text API availability >= 99.9% Action success >= 99% Task completion <= 10 min >= 95% ``` --- # 73. Alert Runtime Alert 用于： > 系统出现需要关注的问题。 --- # 74. Alert Rule ```ts export interface AlertRule { id: string name: string enabled: boolean source: | 'metric' | 'log' | 'error' | 'health' | 'audit' | 'usage' | 'slo' condition: AlertCondition severity: | 'info' | 'warning' | 'critical' cooldownSeconds: number deduplicationKey?: string notificationChannels: string[] } ``` --- # 75. Alert Condition 例如： ```text queue depth > 1000 ``` 或者： ```text action failure rate > 10% ``` 或者： ```text database health = unhealthy ``` 或者： ```text LLM cost > $100/day ``` --- # 76. Alert State ```text inactive ↓ pending ↓ firing ↓ resolved ``` --- # 77. Alert Deduplication 例如： ```text 5000 次数据库 timeout ``` 不能发送： ```text 5000 notifications ``` 应该： ```text Issue #123 Occurrences: 5000 ``` 并根据： ```text deduplicationKey cooldown aggregation ``` 控制通知。 --- # 78. Alert → Notification Alert 不直接发送 Email。 使用： ```text Alert Runtime ↓ Event Runtime ↓ Notification Runtime ↓ Email / Webhook / Push / In-App ``` 这与 `16_EVENT_STREAMING_NOTIFICATION_REALTIME_RUNTIME.md` 保持一致。 --- # 79. Issue Runtime 建议建立： ```text core.issue ``` 用于聚合： ```text errors alerts incidents performance problems ``` --- # 80. Issue ```ts export interface Issue { id: string fingerprint: string title: string severity: | 'info' | 'warning' | 'error' | 'critical' status: | 'open' | 'acknowledged' | 'investigating' | 'resolved' | 'ignored' firstSeenAt: string lastSeenAt: string occurrenceCount: number service?: string pluginId?: string actionId?: string environmentId?: string metadata?: Record } ``` --- # 81. Issue Lifecycle ```text new ↓ open ↓ acknowledged ↓ investigating ↓ resolved ``` 也允许： ```text ignored ``` --- # 82. Incident Issue 与 Incident 不完全相同。 Issue： > 某类错误持续出现。 Incident： > 一个实际影响系统/业务的事件。 例如： ```text Issue: Database timeout Incident: Production database unavailable for 8 minutes ``` --- # 83. Incident Resource 建议： ```text core.incident ``` 包含： ```text severity status startedAt resolvedAt impact timeline relatedIssues relatedTasks relatedExecutions relatedAlerts relatedAuditRecords ``` --- # 84. Incident Timeline 可以自动聚合： ```text 09:01 Database latency increased 09:02 Error rate increased 09:03 Alert fired 09:04 Incident created 09:05 Worker degraded 09:08 Database recovered 09:09 Incident resolved ``` 来源： ```text Event Log Metric Trace Audit Health ``` --- # 85. Observability Dashboard DevOS UI 必须提供统一： ```text Observability ``` 导航： ```text Observability ├── Overview ├── Logs ├── Metrics ├── Traces ├── Audit ├── Errors ├── Issues ├── Incidents ├── Health ├── Diagnostics ├── Performance ├── Usage ├── Cost ├── SLO └── Alerts ``` --- # 86. Overview Dashboard 至少展示： ```text System Health Requests / min Error Rate Action Success Rate Queue Depth Worker Status Active Tasks Active Jobs Agent Runs LLM Usage LLM Cost Storage Usage Open Issues Active Incidents ``` --- # 87. Trace Explorer 用户输入： ```text traceId ``` 看到： ```text Trace ├── HTTP ├── Action ├── DB ├── Redis ├── MCP ├── LLM └── Storage ``` 支持： ```text duration attributes events errors logs ``` --- # 88. Log Explorer 支持过滤： ```text level service plugin action task job execution actor trace time search ``` 并支持： ```text Jump to Trace Jump to Execution Jump to Task Jump to Job Jump to Audit ``` --- # 89. Audit Explorer 支持： ```text Actor Action Resource Operation Result Organization Project Time ``` 点击 Audit 后： ```text Actor Action Target Before After Reason Approval Execution Trace ``` --- # 90. Execution Detail Integration 打开： ```text Execution #123 ``` 必须能够看到： ```text Execution ├── Status ├── Input ├── Output ├── Error ├── Logs ├── Trace ├── Metrics ├── Audit ├── Events ├── Artifacts ├── Usage └── Cost ``` 这是 DevOS 非常重要的统一观察入口。 --- # 91. Task Detail Integration Task 页面： ```text Task ├── Overview ├── Jobs ├── Progress ├── Timeline ├── Logs ├── Errors ├── Events ├── Usage ├── Cost └── Artifacts ``` --- # 92. Agent Run Detail Agent Run： ```text Agent Run ├── Goal ├── Plan ├── Steps ├── Capabilities ├── Actions ├── Tasks ├── Errors ├── Logs ├── Usage ├── Cost ├── Approvals └── Final Result ``` 必须能够追踪： ```text User ↓ Agent Goal ↓ Agent Plan ↓ Skill ↓ Capability ↓ Action ↓ Task ↓ Execution ``` --- # 93. Plugin Observability Plugin 可以注册： ```text logs metrics traces health checks audit definitions usage metrics alerts dashboards ``` 例如： ```ts export interface PluginObservabilityExtension { registerLogger?(): void registerMetrics?(): void registerHealthChecks?(): void registerAuditDefinitions?(): void registerUsageMetrics?(): void registerAlertRules?(): void } ``` 但是： > Plugin 只能扩展 DevOS Runtime，不能自己创建独立 Observability Runtime。 --- # 94. Plugin Metrics Namespace Plugin： ```text seo ``` 只能产生： ```text seo_scan_total seo_scan_duration seo_scan_errors ``` 或者统一命名： ```text devos_plugin_seo_scan_total ``` 不能污染 Core： ```text devos_database_xxx ``` --- # 95. MCP Observability MCP Gateway 自动产生： ```text mcp_request_total mcp_request_error_total mcp_request_duration mcp_connection_status ``` 每次 MCP 调用必须关联： ```text mcpServerId mcpToolId capabilityId actionId executionId agentRunId ``` --- # 96. Agent Observability Agent Runtime 自动记录： ```text agent.run.started agent.run.completed agent.run.failed agent.plan.created agent.plan.replanned agent.step.started agent.step.completed agent.capability.selected agent.action.requested agent.task.created agent.approval.requested ``` --- # 97. Do Not Store Chain of Thought Observability Runtime： > 不保存模型隐藏思维链。 可以保存： ```text decision summary reason selected capability selected action execution result replan reason error ``` 例如： ```json { "decision": "use seo.website.scan", "reason": "website health information is required", "result": "completed" } ``` 而不是保存： ```text 模型内部完整思维过程 ``` --- # 98. Event Integration 与： ```text 16_EVENT_STREAMING_NOTIFICATION_REALTIME_RUNTIME.md ``` 集成。 重要 Observability Event： ```text observability.error.created observability.issue.created observability.issue.resolved observability.alert.fired observability.alert.resolved observability.incident.created observability.incident.resolved observability.health.changed ``` --- # 99. Event vs Audit 必须明确： ```text Event = 系统发生了什么 Log = Runtime 运行过程中记录了什么 Audit = 谁对什么进行了什么重要操作 Metric = 系统运行数量/状态/趋势 Trace = 一次调用链怎么执行 Issue = 某类问题的聚合 Incident = 某个实际影响事件 ``` 禁止混用。 --- # 100. Database Schema 建议： ```text logs audit_records issues incidents alert_rules alerts usage_records cost_records slo_definitions diagnostic_snapshots ``` Metrics / Traces 如果使用 OpenTelemetry Collector / 外部存储，则不强制全部进入 PostgreSQL。 --- # 101. Recommended Storage 默认： ```text PostgreSQL ├── Audit ├── Issues ├── Incidents ├── Alert Rules ├── Usage └── Configuration Redis ├── Realtime metrics ├── Alert state ├── temporary aggregation └── distributed coordination Object Storage ├── Diagnostic Bundles └── archived logs OpenTelemetry-compatible backend ├── Traces └── Metrics Log backend └── Logs ``` DevOS 不应该把所有 Observability 数据强行存进 PostgreSQL。 --- # 102. Retention Policy 不同数据必须拥有不同 retention。 例如： ```text Debug Logs: 7 days Normal Logs: 30 days Audit: 1 year+ Metrics: 90 days Trace: 7-30 days Diagnostic Bundle: 30 days Usage: 1 year+ Cost: 1 year+ ``` 实际值必须配置化。 --- # 103. Archive 日志/审计等长期数据可以： ```text PostgreSQL ↓ Archive Job ↓ Object Storage ``` Archive 过程使用： ```text Task ↓ Job ↓ Worker ``` 而不是 HTTP Request 长时间执行。 --- # 104. Observability API 建议： ```text GET /api/observability/logs GET /api/observability/logs/stream GET /api/observability/traces/:traceId GET /api/observability/audit GET /api/observability/audit/:id GET /api/observability/errors GET /api/observability/issues GET /api/observability/issues/:id GET /api/observability/incidents GET /api/observability/incidents/:id GET /api/observability/health GET /api/observability/metrics GET /api/observability/usage GET /api/observability/cost GET /api/observability/alerts ``` --- # 105. Health API ```http GET /health/live GET /health/ready GET /health/startup GET /health ``` 其中： ```text /live ``` 用于 Kubernetes/container liveness。 ```text /ready ``` 用于 traffic readiness。 --- # 106. CLI 提供： ```bash devos health devos health check devos logs devos logs --follow devos logs --level error devos audit devos audit search devos trace devos issues devos incidents devos diagnostics collect devos usage devos cost devos alerts devos alerts test ``` --- # 107. Diagnostic CLI 例如： ```bash devos diagnostics collect ``` 执行： ```text system information health checks worker status queue status plugin status MCP status database status storage status search status recent errors runtime versions ``` 最后： ```text diagnostic artifact ``` 通过 `15_FILE_ARTIFACT_STORAGE_RUNTIME.md` 保存。 --- # 108. Observability Context Middleware HTTP： ```ts app.use(observabilityMiddleware) ``` 自动： ```text requestId correlationId traceId actor organization project environment ``` --- # 109. Worker Context Worker 获取 Job： ```ts const context = createObservabilityContext({ taskId: job.taskId, jobId: job.id, executionId: execution.id, }) ``` 然后： ```ts logger.child(context) ``` --- # 110. Action Context `03_RESOURCE_ACTION_MODEL.md` 中的： ```ts ActionContext ``` 必须包含： ```ts observability: ObservabilityContext ``` 例如： ```ts export interface ActionContext { requestId: string executionId: string actor: Actor organizationId?: string projectId?: string environmentId?: string permissions: PermissionContext signal: AbortSignal observability: ObservabilityContext metadata?: Record } ``` --- # 111. Automatic Action Instrumentation Action Runtime 统一包装： ```ts async execute(actionId, input, context) { const span = tracing.startSpan(actionId) const start = Date.now() logger.info('action.started', { actionId }) try { const result = await handler(input, context) metrics.counter('action_success_total', 1) auditIfRequired() return result } catch (error) { metrics.counter('action_failure_total', 1) handleError(error) throw error } finally { metrics.histogram( 'action_duration_ms', Date.now() - start ) span.end() } } ``` 业务 Action 不需要重复实现。 --- # 112. Automatic Task Instrumentation Task Runtime 自动： ```text Task Created Task Started Task Progress Task Paused Task Resumed Task Completed Task Failed Task Cancelled ``` 并生成： ```text Log Metric Event Trace where applicable ``` --- # 113. Automatic Job Instrumentation Job Runtime 自动： ```text Job Queued Job Leased Job Started Job Retrying Job Completed Job Failed Job Cancelled ``` 指标： ```text queue wait processing duration attempt count retry count failure rate ``` --- # 114. Database Instrumentation Data Runtime 自动： ```text query count query duration rows errors transaction duration connection pool status ``` 慢查询自动进入： ```text Performance Runtime ``` --- # 115. Redis Instrumentation Queue/Cache/Lock Runtime 自动统计： ```text redis commands latency errors connection count queue operations lock contention ``` --- # 116. External Provider Instrumentation 统一记录： ```text provider operation latency status retry rate limit cost ``` 例如： ```text OpenAI GitHub Google AWS Cloudflare SSH DNS Browser ``` Plugin 不需要自己实现统计框架。 --- # 117. Cost Provider 定义： ```ts export interface PricingProvider { getPrice(input: { provider: string model?: string metric: string quantity: number }): Promise } ``` 允许： ```text OpenAI pricing Browser pricing Cloud provider pricing Plugin pricing ``` --- # 118. Cost Estimation Agent 在执行高成本 Task 前可以： ```text estimate ↓ budget check ↓ approval if required ↓ execute ``` 例如： ```text 100 articles Estimated: Tokens: 8M Cost: $17 Budget: $10 Result: REJECT / APPROVAL_REQUIRED ``` --- # 119. Alert Examples 系统默认提供： ```text High API Error Rate Database Unhealthy Redis Unhealthy Queue Backlog Worker Unhealthy Action Failure Spike Task Failure Spike MCP Failure Spike LLM Cost Spike Storage Near Quota Search Index Degraded SLO Breached ``` --- # 120. Default Alert Policies 例如： ```text Action failure rate > 10% for 5 minutes ``` 或者： ```text queue depth > 1000 for 2 minutes ``` 或者： ```text database health = unhealthy for 30 seconds ``` --- # 121. Alert Permissions 只有拥有： ```text observability.alert.read observability.alert.create observability.alert.update observability.alert.delete ``` 的用户才能管理 Alert。 查看敏感 Audit 还需要： ```text observability.audit.read ``` --- # 122. Resource Definitions DevOS Core 注册： ```text core.log core.audit core.issue core.incident core.alert core.slo core.usage core.cost core.diagnostic core.health ``` 其中部分可以设计为： ```text read-only resources ``` 例如： ```text core.log core.health ``` --- # 123. Resource Actions 例如： ```text core.audit.query core.audit.export core.issue.create core.issue.update core.issue.resolve core.issue.ignore core.incident.create core.incident.update core.incident.resolve core.alert.create core.alert.update core.alert.enable core.alert.disable core.alert.test core.diagnostic.collect core.slo.create core.slo.update ``` --- # 124. Export Observability 数据可以导出： ```text CSV JSON NDJSON Artifact ``` 大规模导出： ```text Export Request ↓ Task ↓ Job ↓ Worker ↓ Artifact ``` 不能： ```text HTTP request ↓ 查询 10GB logs ↓ 直接返回 ``` --- # 125. Search Integration Audit / Issues / Logs 可以接入： ```text Search Runtime ``` 例如： ```text search: "所有最近失败的 SEO scan" ``` Agent： ```text Semantic Search ↓ Search Runtime ↓ Issue / Execution / Audit ``` --- # 126. Agent Integration Agent 可以使用： ```text observability.logs.search observability.trace.get observability.execution.inspect observability.issue.search observability.health.check observability.usage.get observability.cost.get ``` 例如： ```text Agent Goal: 为什么昨天 SEO 扫描大量失败？ ``` Agent： ```text Search Issues ↓ Inspect Errors ↓ Inspect Traces ↓ Inspect Task ↓ Inspect MCP ↓ Find root cause ``` --- # 127. Diagnostic Agent 可以提供： ```text System Diagnostic Agent ``` 但它依然只能通过： ```text Capability ↓ Action Runtime ``` 读取： ```text Logs Metrics Traces Health Issues Tasks Jobs Executions ``` 不能直接访问数据库。 --- # 128. Plugin Developer API Plugin 开发者： ```ts const plugin = definePlugin({ id: 'seo', observability: { metrics: [ 'scan_total', 'scan_duration', 'scan_failed', ], healthChecks: [ 'google-api', 'search-console', ], }, }) ``` DevOS 自动添加 namespace： ```text plugin.seo.scan_total ``` --- # 129. Plugin Logging Plugin： ```ts const logger = ctx.logger logger.info('scan started', { websiteId }) ``` 而不是： ```ts console.log() ``` 生产环境禁止 Plugin 直接依赖： ```text console pino winston bunyan ``` 等具体 Logger 实现。 Plugin 依赖： ```text DevOS Logger interface ``` --- # 130. No Direct Infrastructure 禁止： ```text Plugin ├── new Redis() ├── new Logger() ├── new Queue() ├── new EventBus() └── new WebSocket() ``` 应该： ```text Plugin ↓ DevOS Runtime ├── Logger ├── Metrics ├── Trace ├── Audit ├── Event ├── Queue └── Notification ``` --- # 131. Package Structure 建议： ```text packages/ ├── observability/ │ ├── context/ │ ├── runtime/ │ └── types/ │ ├── logging/ │ ├── runtime/ │ ├── sinks/ │ └── redaction/ │ ├── metrics/ │ ├── runtime/ │ └── registry/ │ ├── tracing/ │ ├── runtime/ │ └── propagation/ │ ├── audit/ │ ├── runtime/ │ ├── repository/ │ └── policies/ │ ├── error/ │ ├── runtime/ │ ├── codes/ │ └── normalization/ │ ├── health/ │ ├── runtime/ │ └── checks/ │ ├── diagnostics/ │ ├── runtime/ │ └── collectors/ │ ├── usage/ │ ├── runtime/ │ └── providers/ │ ├── cost/ │ ├── runtime/ │ └── pricing/ │ ├── performance/ │ └── runtime/ │ ├── alert/ │ ├── runtime/ │ ├── rules/ │ └── evaluation/ │ └── incident/ └── runtime/ ``` --- # 132. Recommended Backend Structure NestJS： ```text src/ ├── observability/ │ ├── observability.module.ts │ ├── context/ │ ├── logging/ │ ├── metrics/ │ ├── tracing/ │ ├── audit/ │ ├── error/ │ ├── health/ │ ├── diagnostics/ │ ├── usage/ │ ├── cost/ │ ├── performance/ │ ├── alerts/ │ └── incidents/ ``` --- # 133. Observability Module ```ts @Module({ providers: [ ObservabilityRuntime, LoggingRuntime, MetricsRuntime, TracingRuntime, AuditRuntime, ErrorRuntime, HealthRuntime, DiagnosticRuntime, UsageRuntime, CostRuntime, PerformanceRuntime, AlertRuntime, ], exports: [ ObservabilityRuntime, LoggingRuntime, MetricsRuntime, TracingRuntime, AuditRuntime, ErrorRuntime, HealthRuntime, DiagnosticRuntime, UsageRuntime, CostRuntime, PerformanceRuntime, AlertRuntime, ], }) export class ObservabilityModule {} ``` --- # 134. Configuration 配置： ```yaml observability: logging: level: info retentionDays: 30 tracing: enabled: true samplingRate: 0.1 metrics: enabled: true audit: enabled: true retentionDays: 365 diagnostics: enabled: true alerts: enabled: true cost: enabled: true ``` --- # 135. Sampling Trace 可能数据量很大。 支持： ```text always never probabilistic tail-based ``` 例如生产： ```text normal traces: 10% errors: 100% slow traces: 100% ``` 错误和慢请求应该提高采样率。 --- # 136. Backpressure Observability 本身不能拖垮业务。 如果： ```text Log backend unavailable ``` 不能导致： ```text Action failed ``` 默认： ```text Business execution ↓ Observability asynchronously ``` Critical Audit 可以采用： ```text durable synchronous write ``` 但必须明确策略。 --- # 137. Observability Failure Policy Logging： ```text best effort ``` Metrics： ```text best effort ``` Tracing： ```text best effort ``` Audit： ```text durable ``` Security Audit： ```text durable ``` --- # 138. Audit Failure 如果某个高风险 Action： ```text 必须写 Audit ``` 而 Audit Storage 不可用： 默认： ```text Action blocked ``` 例如： ```text delete production server ``` Audit 写入失败： ```text DENY ``` 而普通： ```text read dashboard ``` 不需要因此阻塞。 --- # 139. Security Observability 数据本身也是敏感数据。 必须： ```text RBAC Tenant isolation Field-level security Redaction Retention Encryption Access Audit ``` 尤其： ```text Audit Logs Trace attributes Usage Cost ``` --- # 140. Multi-Tenant Isolation 查询： ```text organizationId ``` 必须自动注入。 例如： ```text User A ↓ Audit Query ↓ Scope Policy ↓ Organization A ``` 不能查询： ```text Organization B ``` --- # 141. Field-Level Security 例如普通管理员： ```text 可以看： action resource status time ``` 但是： ```text 不能看： secret before sensitive fields private metadata ``` --- # 142. Audit Export Security Audit Export 必须： ```text permission check scope check field masking audit export itself ``` 也就是说： > 导出 Audit 本身也是 Audit。 --- # 143. Realtime Observability 与 `16` 集成： ```text Log Stream Task Progress Job Progress Incident Timeline Alert State Worker Status Health Status ``` 通过： ```text SSE WebSocket ``` 推送。 --- # 144. Example: Task Detail Realtime ```text Task ↓ Event ↓ Realtime ↓ Browser ``` 用户看到： ```text Task running 43% Job #12 completed Job #13 running Job #14 retrying Error: MCP timeout Cost: $0.21 ``` --- # 145. Observability Timeline 建议建立统一 Timeline API： ```ts export interface TimelineEntry { timestamp: string type: | 'log' | 'event' | 'audit' | 'execution' | 'error' | 'alert' | 'health' title: string description?: string sourceId: string metadata?: Record } ``` 用于： ```text Task Job Execution Agent Run Incident Resource ``` --- # 146. Resource Timeline 任何 Resource 都可以拥有： ```text Timeline ``` 例如： ```text Website ↓ Timeline 09:00 created 09:01 scan started 09:03 scan completed 09:05 keywords updated 09:07 backlink task started ``` 底层数据来自： ```text Event Audit Execution ``` 而不是单独再建一套 Timeline 数据系统。 --- # 147. Root Cause Analysis Diagnostic Runtime 可以提供： ```ts export interface RootCauseAnalysis { issueId: string suspectedCause?: string evidence: DiagnosticEvidence[] confidence?: number relatedResources: string[] relatedExecutions: string[] recommendations?: string[] } ``` --- # 148. RCA Example ```text Issue: SEO scan failure spike Evidence: 1. MCP timeout increased 2. Google API latency increased 3. Action failure correlated with MCP calls 4. Database healthy 5. Worker healthy Likely Cause: Google Search Console API latency Confidence: 0.87 ``` --- # 149. Do Not Pretend RCA Is Certain 自动诊断必须区分： ```text observed inferred suspected confirmed ``` 例如： ```text suspectedCause ``` 不能直接写： ```text rootCause = Google API ``` 除非确实有人工确认或强证据。 --- # 150. Alert → Agent 未来可以： ```text Alert ↓ Agent Goal ↓ Diagnostic Skill ↓ Search Logs ↓ Inspect Trace ↓ Check Health ↓ Generate Diagnosis ↓ Ask Approval ↓ Remediation Task ``` 例如： ```text Redis unhealthy ↓ DevOps Agent ↓ diagnose ↓ restart Redis ``` 危险操作必须： ```text Approval ``` --- # 151. Self-Healing 允许： ```text Alert ↓ Agent ↓ Diagnostic ↓ Action ``` 但必须满足： ```text Policy Permission Budget Approval Audit ``` 例如： ```text restart worker ``` 可以自动。 而： ```text delete production database ``` 必须人工批准。 --- # 152. Observability Capability Registry 注册： ```text observability.logs.search observability.logs.stream observability.trace.inspect observability.audit.search observability.health.check observability.issue.search observability.issue.inspect observability.diagnostic.collect observability.usage.get observability.cost.get observability.performance.inspect ``` Agent 通过 Capability Registry 发现。 --- # 153. MCP Exposure 这些 Capability 可以选择暴露为： ```text MCP tools ``` 例如： ```text search_logs inspect_trace inspect_execution check_health search_issues get_usage get_cost ``` 但是： ```text MCP ↓ Capability ↓ Action Runtime ``` 不能： ```text MCP ↓ direct database ``` --- # 154. CLI / API / UI / Agent Unified Runtime 必须保持： ```text UI API CLI MCP Agent Workflow Plugin Scheduler ↓ Action ↓ Action Runtime ↓ Observability Runtime ``` 因此同一个操作： ```bash devos task cancel xxx ``` 和： ```text UI → Cancel ``` 以及： ```text Agent → task.cancel ``` 得到完全一致的： ```text Audit Log Metric Trace Event ``` --- # 155. Testing Strategy 必须至少测试： ```text Context propagation Logging Redaction Metrics Tracing Audit Errors Health Diagnostics Usage Cost Performance Alerts SLO Multi-tenancy Security Realtime Agent MCP Plugin ``` --- # 156. Context Tests 测试： ```text HTTP ↓ Action ↓ Task ↓ Job ↓ Worker ↓ Execution ``` 最终： ```text correlationId traceId executionId taskId jobId ``` 能够正确关联。 --- # 157. Redaction Tests 输入： ```json { "password": "secret", "token": "abc", "authorization": "Bearer xxx" } ``` 输出： ```json { "password": "[REDACTED]", "token": "[REDACTED]", "authorization": "[REDACTED]" } ``` --- # 158. Audit Tests 验证： ```text create update delete permission change plugin install plugin upgrade agent execution approval export ``` 全部生成正确 Audit。 --- # 159. Audit Immutability Test 尝试： ```text UPDATE audit_records DELETE audit_records ``` 必须失败。 --- # 160. Multi-Tenant Tests 验证： ```text Organization A ``` 不能看到： ```text Organization B ``` 的： ```text Logs Audit Usage Cost Issues Incidents ``` --- # 161. Error Tests 测试： ```text retryable non-retryable timeout network permission validation internal ``` 错误必须正确分类。 --- # 162. Alert Tests 测试： ```text threshold reached cooldown deduplication resolve re-fire notification ``` 例如： ```text 100 errors ``` 只能产生： ```text 1 Alert ``` 而不是： ```text 100 Alerts ``` --- # 163. Health Tests 测试： ```text healthy degraded unhealthy ``` 以及： ```text database down redis down queue down plugin unhealthy MCP unhealthy ``` --- # 164. Diagnostic Tests 执行： ```bash devos diagnostics collect ``` 验证： ```text bundle generated secrets redacted artifact created audit created ``` --- # 165. Cost Tests 模拟： ```text LLM 10000 tokens ``` 验证： ```text UsageRecord CostRecord Task attribution Agent attribution Project attribution ``` 都正确。 --- # 166. Performance Tests 必须能够发现： ```text slow Action slow Query slow Search slow MCP slow LLM ``` --- # 167. Agent Tests 模拟： ```text Agent ↓ Capability ↓ Action ↓ Task ↓ Job ↓ Execution ``` 验证： ```text Agent Run Logs Metrics Trace Audit Usage Cost Events ``` 完整关联。 --- # 168. Plugin Tests Plugin 安装： ```text Plugin ↓ register metrics ↓ register health ↓ register audit ↓ register alerts ``` 卸载后： ```text Plugin metrics removed Plugin health removed Plugin capabilities removed ``` 历史 Observability 数据必须保留。 --- # 169. MCP Tests 测试： ```text MCP request ↓ Capability ↓ Action ↓ Execution ``` 验证： ```text trace log metric audit usage cost ``` --- # 170. Load Tests 至少测试： ```text 10k logs/sec 1k metrics/sec 100 concurrent traces 1000 task executions 10000 jobs 100 workers ``` 实际阈值根据部署规模调整。 --- # 171. Failure Tests 模拟： ```text Log backend down Metric backend down Trace backend down Audit DB down Redis down PostgreSQL slow Storage unavailable ``` 验证业务系统不会因为普通 Observability 失败而整体崩溃。 --- # 172. Production Acceptance Zcode 完成后必须满足： ### Logging ```text [ ] structured logs [ ] context propagation [ ] redaction [ ] query [ ] stream [ ] retention ``` ### Metrics ```text [ ] counter [ ] gauge [ ] histogram [ ] cardinality protection ``` ### Tracing ```text [ ] trace [ ] span [ ] propagation [ ] action instrumentation ``` ### Audit ```text [ ] immutable [ ] actor [ ] resource [ ] before/after [ ] approval [ ] export ``` ### Error ```text [ ] error codes [ ] categories [ ] retryable [ ] fingerprint ``` ### Health ```text [ ] liveness [ ] readiness [ ] startup [ ] dependencies ``` ### Diagnostics ```text [ ] snapshot [ ] diagnostic bundle [ ] redaction [ ] artifact ``` ### Usage / Cost ```text [ ] tokens [ ] API calls [ ] compute [ ] storage [ ] cost [ ] attribution [ ] budget ``` ### Performance ```text [ ] latency [ ] slow operation [ ] database [ ] search [ ] MCP [ ] LLM ``` ### Alert ```text [ ] rules [ ] threshold [ ] cooldown [ ] dedup [ ] notification ``` ### SLO ```text [ ] SLI [ ] SLO [ ] breach detection ``` --- # 173. Zcode Implementation Rules 这是本文件最重要的部分之一。 ## Rule 1 禁止： ```text Plugin → console.log ``` 必须： ```text Plugin → DevOS Logger ``` --- ## Rule 2 禁止： ```text Plugin → 自己创建 Metrics ``` 必须： ```text Plugin → Metrics Runtime ``` --- ## Rule 3 禁止： ```text Plugin → 自己创建 Audit Table ``` 必须： ```text Plugin → Audit Runtime ``` --- ## Rule 4 禁止： ```text Plugin → 自己创建 Error Framework ``` 必须： ```text Plugin → Error Runtime ``` --- ## Rule 5 禁止： ```text MCP → direct DB ``` 必须： ```text MCP ↓ Capability ↓ Action Runtime ``` --- ## Rule 6 禁止： ```text Agent → direct Logger Agent → direct DB Agent → direct Redis Agent → direct Queue ``` 必须： ```text Agent ↓ Capability ↓ Runtime ``` --- ## Rule 7 禁止： ```text 每个 Plugin 自己建立： Logs Metrics Audit Health Alerts ``` 必须： ```text DevOS Observability Runtime ↑ Plugins ``` --- # 174. Golden Integration 最终 DevOS 的一次操作： ```text User ↓ UI / API / CLI / MCP / Agent ↓ Goal / Task ↓ Job ↓ Action Runtime ↓ Execution ↓ Business Logic ↓ Event ``` 同时： ```text Execution │ ┌─────────┼─────────┐ ↓ ↓ ↓ Trace Logs Metrics │ │ │ └─────────┼─────────┘ ↓ Error ↓ ┌────────┴────────┐ ↓ ↓ Issue Alert │ │ └────────┬────────┘ ↓ Incident ``` 另外： ```text Execution ↓ Audit ↓ Usage ↓ Cost ``` 最终： ```text Execution ├── Logs ├── Metrics ├── Trace ├── Error ├── Audit ├── Event ├── Usage ├── Cost ├── Performance └── Artifact ``` --- # 175. Final Architecture 完成本文件以后，DevOS Core 应该形成： ```text DEVOS CORE │ ┌───────────────────────────┼────────────────────────────┐ │ │ │ Resource Action Runtime Data Runtime │ │ │ │ │ │ └───────────────┬───────────┴───────────────┬────────────┘ │ │ Task Runtime Event Runtime │ │ ┌────────┼────────┐ ┌───────┼────────┐ │ │ │ │ │ │ Job Queue Worker Event Webhook Notification │ │ │ │ │ │ └────────┴────────┘ └───────┴────────┘ │ ↓ Execution │ ┌─────────┼──────────────────────┐ │ │ │ ↓ ↓ ↓ Observability Storage Search │ ┌─────┼────────────┬──────────┬──────────┐ │ │ │ │ │ Logs Metrics Trace Audit Error │ │ │ │ │ └─────┴────────────┴──────────┴──────────┘ │ ┌────────┼─────────┐ ↓ ↓ ↓ Health Diagnostic Performance │ │ │ └────────┼─────────┘ ↓ Usage / Cost ↓ Alert ↓ Incident ``` --- # 176. DevOS Unified Runtime Principle 到这里，DevOS 已经不能再被理解成： ```text NestJS + React + PostgreSQL ``` 而应该理解为： ```text DevOS │ ├── Resource Runtime ├── Action Runtime ├── Data Runtime ├── Task Runtime ├── Event Runtime ├── Storage Runtime ├── Search Runtime ├── Observability Runtime ├── Plugin Runtime ├── MCP Runtime ├── Skill Runtime ├── Agent Runtime ├── Workflow Runtime ├── Security Runtime └── UI Runtime ``` 其中： ```text Observability Runtime ``` 是整个系统的“神经系统”。 --- # 177. Final Golden Rule > **任何进入 DevOS 的行为，都必须能够被定位、执行、记录、追踪和解释。** 最终统一链路： ```text Actor ↓ Request ↓ Goal / Task ↓ Job ↓ Action ↓ Action Runtime ↓ Execution ↓ Event ↓ Observability Runtime │ ├── Log ├── Metric ├── Trace ├── Audit ├── Error ├── Health ├── Performance ├── Usage ├── Cost ├── Issue ├── Alert └── Incident ``` **Zcode 不允许任何 Plugin、MCP、Agent、Workflow、Worker 或业务模块绕过这条链路建立自己的日志、审计、指标、错误、健康检查、队列监控或告警体系。**