serving

TensorFlow Serving is a serving system for machine learning models, designed for production environments. It focuses on the inference stage: taking trained models, managing their lifetimes, and providing clients with versioned access through a high-performance, reference-counted lookup table. While it integrates out of the box with TensorFlow models, its architecture allows extension to other model types and data. Key capabilities described in the README include: - Serving multiple models, or multiple versions of the same model, simultaneously. - Exposing both gRPC and HTTP inference endpoints. - Deploying new model versions without changing client code. - Supporting canarying of new versions and A/B testing of experimental models. - Adding minimal latency due to a low-overhead implementation. - A scheduler that groups individual inference requests into batches for joint execution on GPU, with configurable latency controls. - Support for many servables: TensorFlow models, embeddings, vocabularies, feature transformations, and even non-TensorFlow-based machine learning models. The README provides a quick-start example using Docker: pull the tensorflow/serving image, clone the repository, start a container with a demo model mounted, and query the REST API with a curl request. It also links to an end-to-end training and serving tutorial on the official TensorFlow documentation site. Documentation covers setup options, including Docker (recommended), installation without Docker, building from source with Docker, and deployment on Kubernetes. Usage guidance explains exporting a TensorFlow model as a SavedModel, then configuring and using TensorFlow Serving. Additional guides cover performance, TensorBoard profiling, the REST API, the gRPC API definition, SavedModel warmup, SignatureDefs, and serving models with custom ops. For extension, the README notes that the architecture is highly modular, allowing parts such as batch scheduling to be used individually or extended for new use cases. It points to architecture documentation, the C++ API reference, and guides for creating a new type of Servable or a custom source of Servable versions. Contribution guidelines are available in the repository, and the official TensorFlow website is referenced for further information.