chinese-llama-alpaca

## Project Overview Chinese-LLaMA-Alpaca is an open-source large language model project focused on Chinese Natural Language Processing, aiming to promote open research of large models in the Chinese community. Building upon the original LLaMA, the project expands the Chinese vocabulary and performs secondary pre-training using Chinese corpora to enhance fundamental Chinese semantic understanding. Furthermore, the Chinese Alpaca model is further fine-tuned with Chinese instruction data to improve its ability to understand and execute instructions. The project's technical report is titled "Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca" (arXiv:2304.08177). ## Key Capabilities - Expanded Chinese vocabulary for the original LLaMA to improve Chinese encoding and decoding efficiency. - Open-sourced Chinese LLaMA pre-trained with Chinese text, and Chinese Alpaca fine-tuned with instructions. - Provided pre-training and instruction fine-tuning scripts for users to continue training as needed. - Supports local quantization and deployment experience on personal computers (laptops) with CPU/GPU. - Compatible with ecosystem tools such as transformers, llama.cpp, text-generation-webui, LlamaChat, LangChain, and privateGPT. ## Model Versions Available open-sourced model sizes include 7B, 13B, and 33B, with each size divided into Base, Plus, and Pro versions. The Plus series uses more training data, while the Pro series addresses issues of overly short responses. Models are categorized into two types: - Chinese LLaMA: The base model, trained using traditional CLM, suitable for text continuation but not for instruction following or multi-turn chat. - Chinese Alpaca: An instruction-following model, fine-tuned with instructions, suitable for Q&A, writing, suggestions, and multi-turn context understanding. Due to the licensing restrictions of the original LLaMA, the project releases LoRA weights. These weights need to be merged with the original LLaMA to obtain the complete model and cannot be used independently. ## Usage Workflow 1. Download the corresponding LoRA weights from Hugging Face, ModelScope, or Baidu Netdisk. 2. Merge the LoRA weights with the original LLaMA. Refer to the online conversion (Colab notebook) or manual conversion tutorials. 3. Choose an inference and deployment method, including llama.cpp, Transformers, text-generation-webui, LlamaChat, LangChain, privateGPT, Colab Gradio Demo, and an OpenAI-like API call Demo. 4. Select the appropriate model and startup parameters based on the task type. For example, Alpaca requires input in a specific template format. The size of the merged model varies with scale. For instance, a 7B FP16 model is approximately 13GB, and after 4-bit quantization, it's about 3.9GB. A 33B FP16 model is about 60GB, and after 4-bit quantization, it's about 17.2GB. ## Training Details The training process consists of three parts: vocabulary expansion, pre-training, and instruction fine-tuning. Vocabulary expansion code can be found in merge_tokenizers.py. Pre-training and instruction fine-tuning code reference transformers' run_clm.py and the dataset processing part of Stanford Alpaca. Related training scripts have been open-sourced in the Wiki. ## Evaluation The project conducted objective evaluations of multiple models on the C-Eval dataset, which includes approximately 12.3K multiple-choice questions covering 52 subjects. The README lists the average zero-shot and 5-shot scores on the valid/test sets for each model. For example, Chinese-Alpaca-Plus-33B achieved a test 5-shot score of 43.5. The project also provides generation quality evaluation examples and an online battle platform. ## Limitations and Disclaimer The project explicitly states that models may generate unpredictable harmful content or content that does not align with human preferences. Due to limitations in computing power and data, the training is not exhaustive, and there is still room for improvement in Chinese understanding capabilities. The project itself does not provide an online interactive demo; users need to deploy it locally. All resources are for academic research purposes only and strictly prohibited for commercial use. The accuracy of model outputs is not guaranteed. ## Related Projects Subsequent official projects include Chinese-LLaMA-Alpaca-2 (Chinese LLaMA-2, Alpaca-2 large models) and Visual-Chinese-LLaMA-Alpaca (multimodal Chinese LLaMA & Alpaca large models).