Diagrama de flujo de observabilidad

Section titled “Diagrama de flujo de observabilidad”
flowchart TD
  U["OpenCode / OpenWebUI / API"] --> G["Gateway :8008"]
  G --> C["Client/Profile Guard"]
  C --> R["Route Classifier"]
  R --> P["Payload Builder"]
  P --> M["Memory Injector"]
  M --> T["Tool Policy"]
  T --> MR["Model Router"]
  MR --> LM["LM Studio RX9070"]
  LM --> RESP["Response"]
  RESP --> AUD["Governance Audit JSONL"]
  RESP --> MET["Prometheus Metrics"]
  RESP --> QD["Qdrant Memory"]
  MET --> PR["Prometheus :9090"]
  PR --> GF["Grafana :3000"]
  AUD --> REP["Request Replay API"]
  REP --> OPS["Ops Debug"]

Infraestructura

Section titled “Infraestructura”
ComponenteHostPuertoEstado
Prometheus192.168.1.409090Active
Grafana192.168.1.403000Active
Router métricas192.168.1.308083Active
Gateway métricas192.168.1.308008Active
LM Studio192.168.1.501234Active (RX9070)

Dashboards — TIER 1 (Operación diaria)

Section titled “Dashboards — TIER 1 (Operación diaria)”
#DashboardUIDDescripción
00Executive Overviewai-lab-overviewSalud general, burn-in mode, route dominance, contamination
01Routing & Modelsai-lab-runtimeRutas, latencia, tokens, modelos seleccionados
02Cognitive Profilesai-lab-profilesDistribución de perfiles, modelo por perfil
03Tool Governanceai-lab-toolsTools bloqueadas, 428 gate, question stripped, bash sanitizer
04Memory Runtimeai-lab-memoryRecall, skip reasons, contamination, hard caps, quality score
05Route Family Observabilityai-lab-route-familyLatencia p95/p99, errores, prompt tokens por familia
06Incidents & Auditai-lab-incidentsBloqueos, governance, fallback loops
07Request Traceabilityai-lab-traceabilityReplay API guide, drift signals, errors

Dashboards — TIER 2 (Troubleshooting)

Section titled “Dashboards — TIER 2 (Troubleshooting)”
DashboardUIDUso
AI-LAB Execution & Safetyai-lab-safetySeguridad y bloqueos detallados
AI-LAB GPUsai-lab-gpusVRAM, temperatura, uso GPU
AI-LAB Infrastructureai-lab-infraDocker, host, red
AI-LAB Production Burn-Inai-lab-burninBurn-in dashboard (temporal)

Métricas Prometheus (todas)

Section titled “Métricas Prometheus (todas)”

Perfiles cognitivos

Section titled “Perfiles cognitivos”
ailab_profile_total{profile, route_family, model}

Tool governance

Section titled “Tool governance”
ailab_tool_call_total{tool_name, result, policy, mode}
ailab_tool_fastpath_total
ailab_tool_fastpath_fallback_total
ailab_tool_calls_malformed_total
ailab_tool_empty_arguments_total
ailab_tool_question_stripped_total
ailab_confirmation_required_total

Memory runtime

Section titled “Memory runtime”
ailab_memory_recall_total{policy, hit}
ailab_memory_chars_injected{policy}           (histogram)
ailab_memory_items_total{policy, source}
ailab_memory_contamination_risk{policy}       (histogram)
ailab_memory_quality_score{policy}            (histogram)
ailab_context_cap_exceeded_total{policy}

Routing

Section titled “Routing”
ailab_route_family_total{family}
ailab_route_family_latency_ms{family}         (histogram)
ailab_route_family_prompt_tokens_total{family}
ailab_route_family_completion_tokens_total{family}
ailab_route_family_errors_total{family}
ailab_route_family_blocked_total{family}

Governance

Section titled “Governance”
ailab_governance_blocked_actions_total
ailab_governance_blocked_actions_by_reason_total{reason}

Alertas Prometheus (18 reglas)

Section titled “Alertas Prometheus (18 reglas)”
AlertaSeveridadDetecta
MinimalRouteRegressionwarningContexto pesado en ruta ligera
ToolFastpathLatencySpikecriticalFastpath roto
CognitiveRouteExplosionwarningRecall runaway
RouteFamilyErrorRatecriticalErrores recientes
GovernanceBlocksSpikewarningAbuso de prompts/tools
ProfileUnknownwarningPerfil no clasificado
ToolBudgetExceededwarningTools bloqueadas por politica
MemoryFallbackwarningMemory injector fallando
MemoryActivatedInMinimalcriticalFuga de memoria en minimal
MemoryCharsOverBudgetwarningLight policy excediendo budget
MemoryContaminationHighwarningContaminacion alta en recall
MemoryHitRatioLowwarningBajo ratio de hits
QuestionToolLeakagecriticalQuestion tool detectada
EmptyArgumentsSpikewarningBash tools sin argumentos
ConfirmationRequiredinfo428 gate activado
MinimalRouteDominancewarningMinimal perdiendo dominancia
ToolFastpathLeakagewarningTool fastpath inesperado
ContextCapExceededwarningHard cap excedido

Replay y Analytics API

Section titled “Replay y Analytics API”
Terminal window
# Full request replay
curl http://192.168.1.30:8083/api/request/replay/{request_id}


# Timeline estructurada
curl http://192.168.1.30:8083/api/request/flow/{request_id}


# Drift analytics
curl http://192.168.1.30:8083/api/analytics/drift


# Recall ROI
curl http://192.168.1.30:8083/api/analytics/recall-roi


# Memory replay
curl http://192.168.1.30:8083/api/memory/replay?limit=10

Provisioning

Section titled “Provisioning”

Los dashboards se cargan desde:

/home/albert/docker/monitorizacion/grafana/provisioning/dashboards/AI-LAB/

Reload sin reiniciar:

Terminal window
docker exec grafana kill -HUP 1

Alertas:

/home/albert/docker/monitorizacion/prometheus/config/rules/ai-lab-route-family-alerts.yml

Burn-In Mode (FASE 26.1)

Section titled “Burn-In Mode (FASE 26.1)”
  • Nodo único: RX9070 (192.168.1.50:1234)
  • Scheduler multi-GPU: OFF
  • Duración: 48-72h
  • Dashboards: Executive Overview + Burn-In + Route Family

URLs

Section titled “URLs”
  • Grafana: http://192.168.1.40:3000 (admin / 19682507)
  • Prometheus: http://192.168.1.40:9090
  • Router métricas: http://192.168.1.30:8083/metrics
  • Gateway métricas: http://192.168.1.30:8008/metrics