panllm

panllm is an easy-to-use Python library designed to simplify local LLM inference by providing a consistent API across different backends. It allows developers to switch between inference engines without changing their core logic. Currently, it supports llama-cpp-python, with plans to integrate Transformers, ExLlamaV2, and ExLlamaV3. Key features include simple text completion, forced backend selection, and streamed chat generation with performance statistics like tokens per second. The library aims to abstract the complexities of individual inference implementations into a single, high-level interface for local model deployment.