hanlp

HanLP (Han Language Processing) is a production-ready multilingual natural language processing toolkit built on dual engines: PyTorch and TensorFlow 2.x. The project aims to bring cutting-edge NLP technologies into practical use, featuring comprehensive functionality, high accuracy, efficient performance, up-to-date corpora, clear architecture, and customizability. Functionality includes: - Word segmentation (coarse-grained and fine-grained) - Part-of-speech tagging (CTB, PKU, 863, and other standards) - Named entity recognition (PKU, MSRA, OntoNotes, and other standards) - Dependency parsing (SD, UD, PMT) - Constituency parsing (Chinese Tree Bank) - Semantic dependency parsing (CSDP) - Semantic role labeling (Chinese Proposition Bank) - Abstract meaning representation (CAMR) - Coreference resolution - Semantic textual similarity - Text style transfer - Keyphrase extraction - Extractive and abstractive summarization - Text grammar correction - Text classification and sentiment analysis - Language detection - Word vectors and cloze tasks - Traditional/simplified Chinese conversion, pinyin, new word discovery, text clustering (referencing the 1.x version) Multilingual support: Leveraging large-scale multilingual corpora, HanLP 2.1 supports 10 joint tasks and various single tasks across 130 languages, including traditional and simplified Chinese, English, Japanese, Russian, French, and German. Two API forms: 1. Lightweight RESTful API: Only a few KB in size, suitable for agile development, mobile applications, and other scenarios. No GPU environment required, quick installation. Provides Python, Golang, and Java clients. 2. Full-scale native API: Relies on deep learning technologies such as PyTorch and TensorFlow, suitable for professional NLP engineers, researchers, and local large-scale data scenarios. Requires Python 3.6 to 3.10, supports Windows, recommends Unix-like systems, can run on CPU, and recommends GPU/TPU. Model types: - Multi-task models: Fast and memory-efficient, capable of completing multiple tasks in a single call. - Single-task models: Higher accuracy and more flexible, can be assembled in a pipeline mode. Customization: Supports an efficient trie-based custom dictionary, as well as three rules: forced, merged, and corrected. The effects of these rules can be seamlessly applied to downstream statistical models, allowing rapid adaptation to new domains. Output format: Regardless of API, programming language, or natural language, the output is uniformly in JSON format, compatible with dict-like Document objects, containing fields such as tokenization, POS tags, named entities, semantic roles, dependency parsing, semantic dependency parsing, and constituency parsing. The Python RESTful and native APIs also support monospaced-font visualization, allowing linguistic structures to be displayed directly in the console. The project also provides a path for training custom domain models and includes an EMNLP paper citation.