FastDeploy/docs/usage/code_overview.md

Code Overview

Below is an overview of the FastDeploy code structure and functionality organized by directory.

• custom_ops: Contains C++ operators used by FastDeploy for large model inference. Operators for different hardware are placed in corresponding subdirectories (e.g., cpu_ops, gpu_ops). The root-level setup_*.py files are used to compile these C++ operators.
• dockerfiles: Stores Dockerfiles for building FastDeploy runtime environment images.
• docs: Documentation related to the FastDeploy codebase.
• fastdeploy
  • agent: Scripts for launching large model services.
  • cache_manager: Cache management module for large models.
  • engine: Core engine classes for managing large model execution.
  • entrypoints: User-facing APIs for interaction.
  • input: Input processing module, including preprocessing, multimodal input handling, tokenization, etc.
  • model_executor
    • layers: Layer modules required for large model architecture.
    • model_runner: Model inference execution module.
    • models: Built-in large model classes in FastDeploy.
    • ops: Python-callable operator modules compiled from custom_ops, organized by hardware platform.
  • output: Post-processing for large model outputs.
  • platforms: Platform-specific modules for underlying hardware support.
  • scheduler: Request scheduling module for large models.
  • metrics: Core component for collecting, managing, and exporting Prometheus metrics, tracking key runtime performance data (e.g., request latency, resource utilization, successful request counts).
  • splitwise: Modules related to PD disaggragation deployment.
• scripts/tools: Utility scripts for FastDeploy operations (e.g., compilation, unit testing, code style fixes).
• test: Code for unit testing and validation.