{"version":2,"width":120,"height":40,"timestamp":1790260387,"command":"bash recordings/demo.sh","title":"KCD Brasil: demo explicada, com pausas por etapa","env":{"SHELL":"/bin/bash"}}
[0.003137, "o", "\r\n\u001b[1;36m1. O caminho de uma chamada\u001b[0m\r\n\r\n"]
[0.003147, "o", "Cliente: POST /v1/chat/completions\r\n                |\r\n                v\r\n         Istio Gateway\r\n"]
[0.003149, "o", "                |\r\n       HTTPRoute: X-Demo-Pool\r\n"]
[0.003152, "o", "                |\r\n     +----------+-----------+"]
[0.003153, "o", "\r\n     |          |           |\r\n"]
[0.003163, "o", "   fast      quality     inválido\r\n     |          |           |\r\n     v          v           v\r\nInferencePool InferencePool  404\r\n fast-router  quality-router\r\n     |          |\r\n"]
[0.003165, "o", "     v          v\r\n  3 pods      1 pod\r\n"]
[0.003166, "o", "  100 ms      500 ms\r\n"]
[0.003169, "o", "\r\n"]
[0.003172, "o", "A chamada passa pelo Istio até chegar a um servidor de inferência.\r\nA rede e o roteamento são reais. Quem responde é o llm-d-inference-sim,\r\num simulador que permite executar a demo sem GPU.\r\n"]
[0.003173, "o", "\r\n"]
[0.003175, "o", "Usamos o formato de API da OpenAI, mas não chamamos OpenAI nem Claude.\r\nO simulador devolve o texto enviado e reproduz as latências configuradas.\r\n"]
[0.003176, "o", "\r\n"]
[0.003177, "o", "Cluster: kind-kcd-ai-networking-demo\r\n"]
[0.003202, "o", "\r\nEspaço para continuar.\r\n"]
[0.003202, "m", "Continuar"]
[1.004807, "o", "\r\n\u001b[1;36m2. Os servidores de inferência\u001b[0m\r\n\r\n"]
[1.022222, "o", "Modelo / header -> HTTPRoute -> InferencePool -> EPP -> pods (label app)\r\n\u001b[1;31mfast\u001b[0m -> \u001b[1;31mfast-route\u001b[0m -> \u001b[1;31mfast-router\u001b[0m -> \u001b[1;31mfast-router-epp\u001b[0m -> \u001b[1;31mfast-simulator\u001b[0m (3 pods)\r\n\u001b[1;31mquality\u001b[0m -> \u001b[1;31mquality-route\u001b[0m -> \u001b[1;31mquality-router\u001b[0m -> \u001b[1;31mquality-router-epp\u001b[0m -> \u001b[1;31mquality-simulator\u001b[0m (1 pod)\r\n\r\nGateway compartilhado: fast-inference-gateway (Istio).\r\n\r\n"]
[1.025003, "o", "O pool fast tem três réplicas; o quality tem uma.\r\nCada InferencePool agrupa os pods que podem atender aquele modelo.\r\n\r\nO header X-Demo-Pool escolhe o pool. Dentro dele, o roteador escolhe o pod.\r\nVamos conferir quais servidores estão disponíveis.\r\n"]
[1.025009, "o", "\r\nEspaço para continuar.\r\n"]
[1.025009, "m", "Continuar"]
[2.02648, "o", "$ kubectl --context kind-kcd-ai-networking-demo -n ai-networking-demo get pods -l 'app in (fast-simulator,quality-simulator)' -o yaml | python3 recordings/demo-yaml.py\r\n\r\n"]
[2.116714, "o", "---\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  name: \u001b[1;31mfast-simulator-5c995cb8c5-778vc\u001b[0m\r\n  namespace: ai-networking-demo\r\n  labels:\r\n    app: \u001b[1;31mfast-simulator\u001b[0m\r\nspec:\r\n  containers:\r\n  - name: simulator\r\n    image: ghcr.io/llm-d/llm-d-inference-sim:v0.9.0\r\n    args:\r\n    - --model\r\n    - \u001b[1;31mfast\u001b[0m\r\n"]
[2.116723, "o", "    - --port\r\n    - '8000'\r\n    - --mode\r\n    - echo\r\n    - --time-to-first-token\r\n    - 100ms\r\n    - --enable-request-id-headers\r\n"]
[2.116731, "o", "    ports:\r\n    - containerPort: 8000\r\n      name: http\r\n      protocol: TCP\r\n"]
[2.117052, "o", "---\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  name: \u001b[1;31mfast-simulator-5c995cb8c5-g4hjg\u001b[0m\r\n  namespace: ai-networking-demo\r\n  labels:\r\n"]
[2.117058, "o", "    app: \u001b[1;31mfast-simulator\u001b[0m\r\nspec:\r\n  containers:\r\n"]
[2.11706, "o", "  - name: simulator\r\n"]
[2.117068, "o", "    image: ghcr.io/llm-d/llm-d-inference-sim:v0.9.0\r\n    args:\r\n    - --model\r\n"]
[2.117069, "o", "    - \u001b[1;31mfast\u001b[0m\r\n    - --port\r\n"]
[2.117075, "o", "    - '8000'\r\n    - --mode\r\n    - echo\r\n"]
[2.11708, "o", "    - --time-to-first-token\r\n    - 100ms\r\n"]
[2.117083, "o", "    - --enable-request-id-headers\r\n    ports:\r\n    - containerPort: 8000\r\n"]
[2.11709, "o", "      name: http\r\n      protocol: TCP\r\n"]
[2.117538, "o", "---\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  name: \u001b[1;31mfast-simulator-5c995cb8c5-tshhl\u001b[0m\r\n  namespace: ai-networking-demo\r\n"]
[2.11754, "o", "  labels:\r\n"]
[2.117544, "o", "    app: \u001b[1;31mfast-simulator\u001b[0m\r\nspec:\r\n"]
[2.117545, "o", "  containers:\r\n"]
[2.117553, "o", "  - name: simulator\r\n    image: ghcr.io/llm-d/llm-d-inference-sim:v0.9.0\r\n    args:\r\n    - --model\r\n"]
[2.117555, "o", "    - \u001b[1;31mfast\u001b[0m\r\n"]
[2.117556, "o", "    - --port"]
[2.117557, "o", "\r\n"]
[2.117562, "o", "    - '8000'\r\n    - --mode\r\n    - echo\r\n"]
[2.117567, "o", "    - --time-to-first-token\r\n    - 100ms\r\n    - --enable-request-id-headers\r\n"]
[2.117568, "o", "    ports:\r\n"]
[2.117569, "o", "    - containerPort: 8000"]
[2.117571, "o", "\r\n"]
[2.117572, "o", "      name: http\r\n"]
[2.117577, "o", "      protocol: TCP\r\n"]
[2.117947, "o", "---\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n"]
[2.11795, "o", "  name: \u001b[1;31mquality-simulator-5b97946667-shn6q\u001b[0m\r\n"]
[2.117956, "o", "  namespace: ai-networking-demo\r\n  labels:\r\n"]
[2.117963, "o", "    app: \u001b[1;31mquality-simulator\u001b[0m\r\nspec:\r\n  containers:\r\n  - name: simulator\r\n    image: ghcr.io/llm-d/llm-d-inference-sim:v0.9.0\r\n    args:"]
[2.117964, "o", "\r\n"]
[2.117971, "o", "    - --model\r\n    - \u001b[1;31mquality\u001b[0m\r\n    - --port\r\n    - '8000'\r\n"]
[2.117977, "o", "    - --mode\r\n    - echo\r\n    - --time-to-first-token\r\n"]
[2.117979, "o", "    - 500ms\r\n"]
[2.11798, "o", "    - --enable-request-id-headers\r\n"]
[2.117982, "o", "    ports:\r\n"]
[2.117983, "o", "    - containerPort: 8000\r\n"]
[2.117987, "o", "      name: http\r\n      protocol: TCP\r\n"]
[2.125107, "o", "\r\nEspaço para continuar.\r\n"]
[2.125107, "m", "Continuar"]
[3.126774, "o", "\r\n\u001b[1;36m2. InferencePools\u001b[0m\r\n\r\n$ kubectl --context kind-kcd-ai-networking-demo -n ai-networking-demo get inferencepools -o yaml | python3 recordings/demo-yaml.py\r\n\r\n"]
[3.212843, "o", "---\r\napiVersion: inference.networking.k8s.io/v1\r\nkind: InferencePool\r\nmetadata:\r\n  name: \u001b[1;31mfast-router\u001b[0m\r\n  namespace: ai-networking-demo\r\nspec:\r\n  appProtocol: http\r\n  endpointPickerRef:\r\n    failureMode: FailClose\r\n    group: ''\r\n    kind: Service\r\n    name: \u001b[1;31mfast-router-epp\u001b[0m\r\n    port:"]
[3.212855, "o", "\r\n      number: 9002\r\n  selector:\r\n    matchLabels:\r\n      app: \u001b[1;31mfast-simulator\u001b[0m\r\n  targetPorts:\r\n  - number: 8000\r\n"]
[3.21318, "o", "---\r\napiVersion: inference.networking.k8s.io/v1\r\nkind: InferencePool\r\nmetadata:\r\n  name: \u001b[1;31mquality-router\u001b[0m\r\n  namespace: ai-networking-demo\r\n"]
[3.213186, "o", "spec:\r\n  appProtocol: http\r\n"]
[3.213188, "o", "  endpointPickerRef:\r\n"]
[3.213195, "o", "    failureMode: FailClose\r\n    group: ''\r\n"]
[3.213196, "o", "    kind: Service\r\n"]
[3.213203, "o", "    name: \u001b[1;31mquality-router-epp\u001b[0m\r\n    port:\r\n"]
[3.213209, "o", "      number: 9002\r\n  selector:\r\n"]
[3.21321, "o", "    matchLabels:\r\n"]
[3.213217, "o", "      app: \u001b[1;31mquality-simulator\u001b[0m\r\n  targetPorts:\r\n"]
[3.213228, "o", "  - number: 8000\r\n"]
[3.220659, "o", "\r\nEspaço para continuar.\r\n"]
[3.220659, "m", "Continuar"]
[4.222004, "o", "\r\n\u001b[1;36m2. HTTPRoutes\u001b[0m\r\n\r\n$ kubectl --context kind-kcd-ai-networking-demo -n ai-networking-demo get httproutes -o yaml | python3 recordings/demo-yaml.py\r\n\r\n"]
[4.288686, "o", "---\r\napiVersion: gateway.networking.k8s.io/v1\r\nkind: HTTPRoute\r\nmetadata:\r\n  name: \u001b[1;31mfast-route\u001b[0m\r\n  namespace: ai-networking-demo\r\nspec:\r\n  parentRefs:\r\n  - group: gateway.networking.k8s.io\r\n    kind: Gateway\r\n    name: fast-inference-gateway\r\n  rules:\r\n  - backendRefs:\r\n    - group: inference.networking.k8s.io\r\n      kind: InferencePool\r\n      name: \u001b[1;31mfast-router\u001b[0m\r\n      weight: 1\r\n    matches:\r\n"]
[4.288697, "o", "    - headers:\r\n      - name: x-demo-pool\r\n        type: Exact\r\n        value: \u001b[1;31mfast\u001b[0m\r\n      path:\r\n        type: PathPrefix\r\n"]
[4.288699, "o", "        value: /v1/chat/completions\r\n"]
[4.28911, "o", "---\r\napiVersion: gateway.networking.k8s.io/v1\r\nkind: HTTPRoute\r\nmetadata:\r\n"]
[4.289114, "o", "  name: \u001b[1;31mquality-route\u001b[0m\r\n"]
[4.28912, "o", "  namespace: ai-networking-demo\r\nspec:\r\n  parentRefs:\r\n"]
[4.289127, "o", "  - group: gateway.networking.k8s.io\r\n    kind: Gateway\r\n    name: fast-inference-gateway\r\n  rules:\r\n"]
[4.289129, "o", "  - backendRefs:\r\n"]
[4.289136, "o", "    - group: inference.networking.k8s.io\r\n      kind: InferencePool\r\n"]
[4.289139, "o", "      name: \u001b[1;31mquality-router\u001b[0m\r\n      weight: 1\r\n    matches:\r\n    - headers:"]
[4.289141, "o", "\r\n"]
[4.289147, "o", "      - name: x-demo-pool\r\n        type: Exact\r\n"]
[4.289153, "o", "        value: \u001b[1;31mquality\u001b[0m\r\n      path:\r\n        type: PathPrefix\r\n"]
[4.289161, "o", "        value: /v1/chat/completions\r\n"]
[4.296324, "o", "\r\nEspaço para continuar.\r\n"]
[4.296324, "m", "Continuar"]
[5.297607, "o", "\r\n\u001b[1;36m3. Uma chamada ao modelo rápido\u001b[0m\r\n\r\n[Cliente] -- X-Demo-Pool: fast --> [Istio / HTTPRoute]\r\n                                          |\r\n                                          v\r\n                             [InferencePool fast-router]\r\n                                          | escolhe 1 dos 3 pods\r\n                                          v\r\n                               [fast-simulator: 100 ms]\r\n\r\nCom X-Demo-Pool: fast, a chamada vai para o pool rápido.\r\nO header x-inference-pod revela qual réplica respondeu.\r\n\r\nConfiguramos 100 ms até o primeiro token no simulador.\r\nO curl mede o tempo até o primeiro byte, usado aqui como aproximação do TTFT.\r\nEsse tempo também inclui a passagem pela rede e pelo gateway.\r\n\r\nEspaço para continuar.\r\n"]
[5.297607, "m", "Continuar"]
[6.299385, "o", "\r\n$ kubectl --context kind-kcd-ai-networking-demo -n ai-networking-demo \\\r\n  exec deployment/fast-client -- \\\r\n  curl -sS -i --max-time 15 \\\r\n    'http://fast-inference-gateway-istio.ai-networking-demo.svc.cluster.local/v1/chat/completions' \\\r\n    -H 'Content-Type: application/json' \\\r\n    -H 'X-Demo-Pool: fast' \\\r\n    -d '{\"model\":\"fast\",\"messages\":[{\"role\":\"user\",\"content\":\"Ola KCD Brasil!\"}]}' \\\r\n    -w '\\nTTFT: %{time_starttransfer}s\\n'\r\n"]
[6.299394, "o", "\r\nEspaço para continuar.\r\n"]
[6.299394, "m", "Continuar"]
[7.501471, "o", "\r\n\u001b[1;36mResposta do pool fast | JSON formatado para leitura\u001b[0m\r\n\r\n"]
[7.517925, "o", "HTTP/1.1 200 OK\r\nx-inference-pod: \u001b[1;31mfast-simulator-5c995cb8c5-778vc\u001b[0m\r\nTTFT: 0.104800s\r\n"]
[7.547162, "o", "{\r\n  \"id\": \"chatcmpl-4648042b-10f0-4e93-9dc5-482ee45f3ed1\",\r\n  \"model\": \"\u001b[1;31mfast\u001b[0m\",\r\n  \"object\": \"chat.completion\",\r\n  \"kv_transfer_params\": null,\r\n  \"created\": 1790260394,\r\n  \"usage\": {\r\n    \"prompt_tokens\": 9,\r\n    \"completion_tokens\": 4,\r\n    \"total_tokens\": 13,\r\n    \"prompt_tokens_detail\": {\r\n"]
[7.54717, "o", "      \"cached_tokens\": 0\r\n    }\r\n  },\r\n  \"choices\": [\r\n    {\r\n      \"index\": 0,\r\n"]
[7.547176, "o", "      \"finish_reason\": \"stop\",\r\n      \"message\": {\r\n"]
[7.547178, "o", "        \"role\": \"assistant\",\r\n"]
[7.547186, "o", "        \"content\": \"Ola KCD Brasil!\"\r\n      }\r\n    }\r\n  ]\r\n"]
[7.547187, "o", "}\r\n"]
[7.554809, "o", "\r\n"]
[7.554846, "o", "\r\nEspaço para continuar.\r\n"]
[7.554846, "m", "Continuar"]
[8.556248, "o", "\r\n\u001b[1;36m4. Distribuição entre as réplicas\u001b[0m\r\n\r\nChamadas sequenciais podem chegar sempre ao mesmo pod.\r\nVamos enviar 12 chamadas concorrentes, com textos distintos, e contar\r\nquantas respostas vieram de cada réplica.\r\n\r\nA distribuição não precisa ser uniforme. A contagem mostra o que aconteceu\r\nnesta execução, sem presumir uma ordem fixa entre os pods.\r\n"]
[8.55626, "o", "\r\nEspaço para continuar.\r\n"]
[8.55626, "m", "Continuar"]
[9.557579, "o", "\r\n\u001b[1;36m4. As 12 chamadas concorrentes\u001b[0m\r\n\r\n$ kubectl --context kind-kcd-ai-networking-demo -n ai-networking-demo \\\r\n  exec deployment/fast-client -- sh -c '\r\n  for i in 1 2 3 4 5 6 7 8 9 10 11 12; do\r\n    curl -sS -i --max-time 15 \\\r\n      http://fast-inference-gateway-istio.ai-networking-demo.svc.cluster.local/v1/chat/completions \\\r\n      -H \"Content-Type: application/json\" -H \"X-Demo-Pool: fast\" \\\r\n      -d \"{\\\"model\\\":\\\"fast\\\",\\\"messages\\\":[{\\\"role\\\":\\\"user\\\",\\\"content\\\":\\\"Pedido $i\\\"}]}\" \\\r\n      | grep -i \"^x-inference-pod:\" &\r\n  done\r\n  wait\r\n'\r\n"]
[9.557588, "o", "\r\nEspaço para continuar.\r\n"]
[9.557588, "m", "Continuar"]
[10.875751, "o", "Chamadas por pod (distribuicao observada):\r\n"]
[10.893791, "o", "      1 \u001b[1;31mfast-simulator-5c995cb8c5-778vc\u001b[0m\r\n      7 \u001b[1;31mfast-simulator-5c995cb8c5-g4hjg\u001b[0m\r\n      4 \u001b[1;31mfast-simulator-5c995cb8c5-tshhl\u001b[0m\r\n"]
[10.897466, "o", "\r\nEspaço para continuar.\r\n"]
[10.897466, "m", "Continuar"]
[11.898563, "o", "\r\n\u001b[1;36m5. Uma chamada ao modelo de qualidade\u001b[0m\r\n\r\n[Cliente] -- X-Demo-Pool: quality --> [Istio / HTTPRoute]\r\n                                             |\r\n                                             v\r\n                               [InferencePool quality-router]\r\n                                             | 1 pod disponível\r\n                                             v\r\n                                 [quality-simulator: 500 ms]\r\n\r\n"]
[11.898573, "o", "Ao enviar X-Demo-Pool: quality, escolhemos outro pool.\r\nEle tem uma réplica e 500 ms configurados até o primeiro token.\r\n\r\nCompare o pod e a latência com as respostas de fast.\r\nO nome quality ilustra outro perfil de modelo. Como a resposta é simulada,\r\nesta demo não mede a qualidade de um modelo real.\r\n"]
[11.898588, "o", "\r\nEspaço para continuar.\r\n"]
[11.898588, "m", "Continuar"]
[12.90019, "o", "\r\n$ kubectl --context kind-kcd-ai-networking-demo -n ai-networking-demo \\\r\n  exec deployment/fast-client -- \\\r\n  curl -sS -i --max-time 15 \\\r\n    'http://fast-inference-gateway-istio.ai-networking-demo.svc.cluster.local/v1/chat/completions' \\\r\n    -H 'Content-Type: application/json' \\\r\n    -H 'X-Demo-Pool: quality' \\\r\n    -d '{\"model\":\"quality\",\"messages\":[{\"role\":\"user\",\"content\":\"Ola KCD Brasil!\"}]}' \\\r\n    -w '\\nTTFT: %{time_starttransfer}s\\n'\r\n"]
[12.900198, "o", "\r\nEspaço para continuar.\r\n"]
[12.900198, "m", "Continuar"]
[14.515288, "o", "\r\n\u001b[1;36mResposta do pool quality | JSON formatado para leitura\u001b[0m\r\n\r\n"]
[14.533462, "o", "HTTP/1.1 200 OK\r\nx-inference-pod: \u001b[1;31mquality-simulator-5b97946667-shn6q\u001b[0m\r\nTTFT: 0.504765s\r\n"]
[14.560298, "o", "{\r\n  \"id\": \"chatcmpl-267736ef-4352-44d7-8a09-1c22b8bd8bd2\",\r\n  \"model\": \"\u001b[1;31mquality\u001b[0m\",\r\n  \"object\": \"chat.completion\",\r\n  \"kv_transfer_params\": null,\r\n  \"created\": 1790260401,\r\n  \"usage\": {\r\n    \"prompt_tokens\": 9,\r\n    \"completion_tokens\": 4,\r\n    \"total_tokens\": 13,\r\n    \"prompt_tokens_detail\": {\r\n      \"cached_tokens\": 0\r\n    }\r\n  },\r\n  \"choices\": [\r\n    {\r\n      \"index\": 0,\r\n      \"finish_reason\": \"stop\",\r\n"]
[14.560309, "o", "      \"message\": {\r\n        \"role\": \"assistant\",\r\n        \"content\": \"Ola KCD Brasil!\"\r\n      }\r\n    }\r\n  ]\r\n}\r\n"]
[14.564643, "o", "\r\n"]
[14.564718, "o", "\r\nEspaço para continuar.\r\n"]
[14.564718, "m", "Continuar"]
[15.566599, "o", "\r\n\u001b[1;36m6. Um pool que não existe\u001b[0m\r\n\r\nAs rotas aceitam fast ou quality no header X-Demo-Pool.\r\nO valor invalid não corresponde a nenhuma delas, então o gateway retorna 404.\r\n\r\nA chamada não chega aos simuladores. Na resposta, esperamos ver o erro\r\nsem o header x-inference-pod.\r\n"]
[15.566632, "o", "\r\nEspaço para continuar.\r\n"]
[15.566632, "m", "Continuar"]
[16.568485, "o", "\r\n$ kubectl --context kind-kcd-ai-networking-demo -n ai-networking-demo \\\r\n  exec deployment/fast-client -- \\\r\n  curl -sS -i --max-time 15 \\\r\n    'http://fast-inference-gateway-istio.ai-networking-demo.svc.cluster.local/v1/chat/completions' \\\r\n    -H 'Content-Type: application/json' \\\r\n    -H 'X-Demo-Pool: invalid' \\\r\n"]
[16.568496, "o", "    -d '{\"model\":\"invalid\",\"messages\":[{\"role\":\"user\",\"content\":\"Ola KCD Brasil!\"}]}' \\\r\n    -w '\\nTTFT: %{time_starttransfer}s\\n'\r\n"]
[16.568528, "o", "\r\nEspaço para continuar.\r\n"]
[16.568528, "m", "Continuar"]
[17.663698, "o", "\r\n\u001b[1;36mResposta do pool invalid | JSON formatado para leitura\u001b[0m\r\n\r\n"]
[17.677924, "o", "HTTP/1.1 404 Not Found\r\nTTFT: 0.001743s\r\n"]
[17.685078, "o", "\r\n"]
[17.685121, "o", "\r\nEspaço para continuar.\r\n"]
[17.685121, "m", "Continuar"]
[18.686448, "o", "\r\n\u001b[1;36m7. Conferindo o resultado\u001b[0m\r\n\r\n"]
[18.686461, "o", "O script de validação faz uma chamada para cada pool.\r\nEle confere se os pods pertencem aos pools esperados e se fast respondeu\r\ncom latência menor que quality.\r\n\r\nEssas verificações confirmam o roteamento e a diferença de latência\r\nque acabamos de observar.\r\n"]
[18.686495, "o", "\r\nEspaço para continuar.\r\n"]
[18.686495, "m", "Continuar"]
[19.687863, "o", "$ bash scripts/validate-model-routing.sh\r\n\r\n"]
[20.511141, "o", "fast -> fast-simulator\r\nquality -> quality-simulator\r\nTTFT: fast=0.104679s quality=0.504469s\r\n"]
[20.511333, "o", "\r\nValidado: pods dos pools corretos e TTFT de fast menor que quality.\r\n\r\nClusters preservados. Nenhuma instalacao durante a gravacao.\r\n"]
[20.511361, "o", "\r\nEspaço para continuar.\r\n"]
[20.511361, "m", "Continuar"]
[21.512594, "x", "0"]