Llama-2-7b-4bit推理
AI日日新
编辑于 2023年07月25日 08:21

可在线运行的notebook示例：

代码块
Python
自动换行
复制代码
https://www.kaggle.com/code/tiansztianszs/chinese-llama-2-7b-4bit/notebook
复制成功

首先需要安装关于量化的依赖：

代码块
Python
自动换行
复制代码
!pip install --upgrade accelerate
!pip install bitsandbytes transformers_stream_generator
复制成功

接着加载4比特量化后的模型，以及设置与Llama对应的提问模板：

代码块
Python
自动换行
复制代码
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer

model_path = "LinkSoul/Chinese-Llama-2-7b-4bit"

tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
        model_path,
        load_in_4bit=True,
        torch_dtype=torch.float16,
        device_map='auto'
    )
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
instruction = """[INST] <<SYS>>\nYou are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

            If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n<</SYS>>\n\n{} [/INST]"""
复制成功

最后，你可以向模型进行提问了。这里使用英文提问关于人生的意义的问题：

代码块
Python
自动换行
复制代码
prompt = instruction.format("What is the meaning of life")
generate_ids = model.generate(tokenizer(prompt, return_tensors='pt').input_ids.cuda(), max_new_tokens=4096, streamer=streamer)
复制成功

cv25258378