kvcached部署指南Docker、Kubernetes和云原生方案【免费下载链接】kvcachedVirtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond项目地址: https://gitcode.com/gh_mirrors/kv/kvcachedkvcached是一个革命性的虚拟化弹性KV缓存系统专为动态GPU共享和高效LLM服务优化而设计。通过引入操作系统风格的虚拟内存抽象kvcached实现了弹性按需KV缓存分配显著提升GPU在动态工作负载下的利用率。本指南将详细介绍kvcached的多种部署方案从最简单的Docker部署到完整的云原生Kubernetes架构帮助您快速上手这个强大的GPU内存管理工具。 环境准备与基础安装系统要求Python: 3.9-3.13版本GPU: NVIDIA GPU支持CUDA或AMD GPU支持ROCm推理引擎: SGLang≥v0.4.9或vLLM≥v0.8.4操作系统: LinuxUbuntu 20.04或CentOS 7快速安装方法kvcached支持多种安装方式您可以根据需求选择从PyPI安装推荐:pip install kvcached --no-build-isolation从源码安装:git clone https://gitcode.com/gh_mirrors/kv/kvcached cd kvcached pip install -e . --no-build-isolation --no-cache-dir python tools/dev_copy_pth.py验证安装: 安装完成后设置环境变量即可启用kvcachedexport ENABLE_KVCACHEDtrue export KVCACHED_AUTOPATCH1 Docker容器化部署kvcached提供了完整的Docker镜像支持与主流LLM推理引擎集成。预构建镜像kvcached团队维护了多个预构建的Docker镜像# vLLM引擎镜像 docker pull ghcr.io/ovg-project/vllm-v0.19.0-kvcached:latest # SGLang引擎镜像 docker pull ghcr.io/ovg-project/sglang-v0.5.10-kvcached:latest # 开发环境镜像包含vLLM和SGLang docker pull ghcr.io/ovg-project/kvcached-dev:latestDocker容器运行示例使用vLLM引擎运行容器docker run -itd \ --shm-size 32g \ --gpus all \ --env HF_TOKENyour_token \ -v /dev/shm:/shm \ --ipchost \ --networkhost \ --privileged \ --name kvcached-vllm \ ghcr.io/ovg-project/vllm-v0.19.0-kvcached \ bash容器内运行基准测试进入容器后您可以运行标准基准测试docker exec -it kvcached-vllm bash # 启用kvcached export ENABLE_KVCACHEDtrue export KVCACHED_AUTOPATCH1 # 启动vLLM服务器 vllm serve meta-llama/Llama-3.2-1B --no-enable-prefix-caching --port12346 # 运行基准测试 vllm bench serve --model meta-llama/Llama-3.2-1B \ --request-rate 10 \ --num-prompts 1000 \ --port 12346自定义镜像构建如果需要自定义CUDA版本或修改源码可以自行构建镜像# 构建vLLM镜像 docker build -f docker/Dockerfile.vllm -t vllm-custom-kvcached . # 构建SGLang镜像 docker build -f docker/Dockerfile.sglang -t sglang-custom-kvcached . # 构建开发镜像 docker build -f docker/Dockerfile.dev -t kvcached-dev-custom . Kubernetes云原生部署kvcached完全支持Kubernetes部署以下是完整的云原生部署方案。Kubernetes部署配置创建kvcached的Kubernetes部署文件kvcached-deployment.yamlapiVersion: apps/v1 kind: Deployment metadata: name: kvcached-vllm namespace: kvcached spec: replicas: 1 selector: matchLabels: app: kvcached-vllm template: metadata: labels: app: kvcached-vllm spec: nodeSelector: gpu-type: nvidia-a100 containers: - name: kvcached-vllm image: ghcr.io/ovg-project/vllm-v0.19.0-kvcached:latest env: - name: ENABLE_KVCACHED value: true - name: KVCACHED_AUTOPATCH value: 1 - name: VLLM_USE_V1 value: 1 - name: HF_TOKEN valueFrom: secretKeyRef: name: huggingface-secret key: token resources: limits: nvidia.com/gpu: 1 memory: 64Gi requests: nvidia.com/gpu: 1 memory: 32Gi command: [vllm] args: [serve, meta-llama/Llama-3.2-1B, --port8000] ports: - containerPort: 8000 volumeMounts: - name: shm-volume mountPath: /dev/shm volumes: - name: shm-volume emptyDir: medium: Memory sizeLimit: 32Gi多模型Kubernetes部署对于多模型场景使用StatefulSet配合kvcached控制器apiVersion: apps/v1 kind: StatefulSet metadata: name: kvcached-multi-model namespace: kvcached spec: serviceName: kvcached-service replicas: 3 selector: matchLabels: app: kvcached-multi-model template: metadata: labels: app: kvcached-multi-model spec: containers: - name: kvcached-controller image: ghcr.io/ovg-project/kvcached-dev:latest env: - name: KVCACHED_CONFIG_PATH value: /config/controller-config.yaml volumeMounts: - name: config-volume mountPath: /config - name: shm-volume mountPath: /dev/shm command: [python] args: [controller/launch.py, --config, /config/controller-config.yaml] volumes: - name: config-volume configMap: name: kvcached-config - name: shm-volume emptyDir: medium: Memory sizeLimit: 64Gi控制器配置文件创建控制器配置文件controller-config.yamlkvcached: kvcached_gpu_utilization: 0.95 kvcached_page_prealloc_enabled: true kvcached_min_reserved_pages: 5 kvcached_max_reserved_pages: 10 router: enable_router: true router_port: 8080 router_host: 0.0.0.0 sleep_manager: idle_threshold_seconds: 300 check_interval_seconds: 60 auto_sleep_enabled: true wakeup_on_request: true min_sleep_duration: 120 instances: - name: llama-3-2-1b model: meta-llama/Llama-3.2-1B engine: vllm kvcached_env: - ENABLE_KVCACHEDtrue - KVCACHED_AUTOPATCH1 engine_env: - VLLM_USE_V11 - VLLM_ATTENTION_BACKENDFLASH_ATTN engine_args: - --no-enable-prefix-caching - --host0.0.0.0 - --port12346 - --enable-sleep-mode - name: qwen-3-0-6b model: Qwen/Qwen3-0.6B engine: sglang kvcached_env: - ENABLE_KVCACHEDtrue - KVCACHED_AUTOPATCH1 engine_args: - --disable-radix-cache - --trust-remote-code - --host0.0.0.0 - --port30000 高级配置与优化内存管理配置kvcached提供了细粒度的内存管理配置选项# 高级内存配置示例 kvcached: kvcached_gpu_utilization: 0.90 # GPU利用率目标 kvcached_page_prealloc_enabled: true # 启用页面预分配 kvcached_min_reserved_pages: 10 # 最小保留页面数 kvcached_max_reserved_pages: 50 # 最大保留页面数 kvcached_sanity_check: false # 关闭完整性检查以提升性能 kvcached_log_level: INFO # 日志级别DEBUG/INFO/WARNING/ERROR性能优化参数根据工作负载调整性能参数# 针对高并发场景 export KVCACHED_PAGE_SIZE_MB64 export KVCACHED_MAX_CONCURRENT_ALLOCS32 export KVCACHED_PREFETCH_ENABLEDtrue # 针对大模型场景 export KVCACHED_CONTIGUOUS_LAYOUTtrue # NVIDIA GPU export KVCACHED_CONTIGUOUS_LAYOUTfalse # AMD ROCm GPU监控与日志kvcached提供丰富的监控指标# 查看kvcached状态 kvctl status # 监控内存使用情况 kvtop # 导出监控数据 kvctl export-metrics --formatjson --outputmetrics.json 生产环境部署最佳实践1. 高可用性部署对于生产环境建议采用以下高可用架构# Kubernetes高可用部署 apiVersion: v1 kind: Service metadata: name: kvcached-router namespace: kvcached spec: selector: app: kvcached-controller ports: - port: 8080 targetPort: 8080 name: http type: LoadBalancer --- apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: kvcached-hpa namespace: kvcached spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: kvcached-controller minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 702. 安全配置确保生产环境的安全性# 安全配置示例 securityContext: runAsNonRoot: true runAsUser: 1000 allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true seccompProfile: type: RuntimeDefault3. 监控与告警集成Prometheus和Grafana进行监控# Prometheus监控配置 apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: kvcached-monitor namespace: monitoring spec: selector: matchLabels: app: kvcached endpoints: - port: metrics interval: 30s path: /metrics 实际应用场景场景1多LLM模型共享GPU使用kvcached控制器部署多个LLM模型# 启动两个模型共享GPU cd examples/01_simple_two_models ./start_two_models.sh \ --engine-a vllm \ --engine-b vllm \ --model-a meta-llama/Llama-3.2-1B \ --model-b Qwen/Qwen3-0.6B \ --port-a 12346 \ --port-b 12347场景2推理与微调共存在单一GPU上同时运行推理和微调任务cd examples/04_inference_and_finetune ./start_inference_and_finetune.sh场景3多智能体系统部署LangChain多智能体系统cd examples/05_multi_agents ./start_multi_agent_models.sh️ 故障排除与调试常见问题解决kvcached未生效# 检查环境变量 echo $ENABLE_KVCACHED echo $KVCACHED_AUTOPATCH # 查看日志确认patch是否成功 grep Successfully patched logs/vllm.log内存分配问题# 检查GPU内存状态 nvidia-smi # 查看kvcached内存统计 kvctl memory-stats性能调优# 启用详细日志 export KVCACHED_LOG_LEVELDEBUG # 调整页面大小 export KVCACHED_PAGE_SIZE_MB128性能监控命令# 实时监控GPU使用情况 watch -n 1 nvidia-smi # 监控kvcached内存分配 kvctl monitor --interval1s # 查看请求统计 curl http://localhost:8080/traffic/stats 性能基准测试kvcached在多个实际场景中表现出色2-28倍TTFT降低相比传统静态内存分配GPU利用率提升30-50%通过动态内存共享支持多达5个模型并发在单张A100-80G GPU上多模型服务场景下的TTFT对比 总结kvcached为LLM服务提供了革命性的GPU内存管理方案。通过Docker容器化部署、Kubernetes云原生集成以及灵活的多模型控制器kvcached能够显著提升GPU利用率降低LLM服务成本。无论您是个人开发者还是企业用户kvcached都能为您提供弹性内存管理按需分配KV缓存成本优化减少GPU资源浪费易于部署支持多种部署方式完整监控提供丰富的性能指标生产就绪已被多家企业采用开始使用kvcached体验下一代GPU共享技术带来的效率提升吧【免费下载链接】kvcachedVirtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond项目地址: https://gitcode.com/gh_mirrors/kv/kvcached创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考