本地安全AI大模型攻防知识库搭建教程<\/h1>

1. 概述<\/h2>
本教程将详细介绍如何从零开始搭建一个本地大模型问答知识库，重点解决在搭建过程中遇到的各种问题及其解决方案。<\/p>

1.1 搭建方案选择<\/h3>

目前搭建大语言问答知识库主要有三种方案：<\/p>

微调模型(Fine-tuning)<\/strong>：在预训练模型基础上进行特定任务的调整<\/li>
再次训练模型(Retraining)<\/strong>：从零开始训练模型<\/li>

增强检索生成(RAG, Retrieval Augmented Generation)<\/strong>：结合检索和生成的技术<\/li> <\/ol>
推荐方案<\/strong>：考虑到成本和响应速度，建议使用开源本地大型语言模型结合RAG方案。经过测试，llama3:8b和qwen2:7b这类体量的模型响应速度快，适合搭建问答知识库。<\/p>
2. RAG原理详解<\/h2>
2.1 RAG基本工作原理<\/h3>
RAG(检索增强生成)结合了信息检索和文本生成技术，主要步骤如下：<\/p>

对知识库内容进行分片处理<\/li>
使用embedding模型将分片后的知识库向量化<\/li>
使用embedding模型将用户问题向量化<\/li>
计算问题向量与知识库向量的相似度，找出最相似的k个结果<\/li>
将这k个结果作为上下文放入prompt中，连同用户问题一起提交给大模型<\/li> <\/ol>
2.2 向量相似度计算示例<\/h3>
import<\/span> numpy as<\/span> np <\/span><\/span> <\/span><\/span>def<\/span> cosine_similarity<\/span>(a, b): <\/span><\/span> return<\/span> np.<\/span>dot(a, b) \/<\/span> (np.<\/span>linalg.<\/span>norm(a) *<\/span> np.<\/span>linalg.<\/span>norm(b)) <\/span><\/span> <\/span><\/span># 示例数据<\/span> <\/span><\/span>context =<\/span> ["北京,上海,杭州"<\/span>, "苹果,橘子,桃子"<\/span>, "太阳,月亮,星星"<\/span>] <\/span><\/span>questions =<\/span> ["城市"<\/span>, "水果"<\/span>, "天体"<\/span>] <\/span><\/span> <\/span><\/span># 计算相似度<\/span> <\/span><\/span>for<\/span> i in<\/span> range(3<\/span>): <\/span><\/span> for<\/span> j in<\/span> range(3<\/span>): <\/span><\/span> similar =<\/span> cosine_similarity(qVector[i], vector[j]) <\/span><\/span> print(f<\/span>"<\/span>{<\/span>questions[i]}<\/span>和<\/span>{<\/span>context[j]}<\/span>的相似度为:<\/span>{<\/span>similar}<\/span>"<\/span>) <\/span><\/span><\/code><\/pre>2.3 RAG的必要性<\/h3> 私有数据处理<\/strong>：当知识库包含私有数据(不在大模型训练数据中)时，RAG是比微调更经济的方案<\/li> 上下文长度限制<\/strong>：直接放入全部知识库内容会超过token限制(如qwen2:7b最大128k tokens)<\/li> 成本效益<\/strong>：相比微调或重新训练，RAG成本更低，实现更快<\/li> <\/ul> 3. 技术实现细节<\/h2> 3.1 向量存储与检索方案<\/h3> 对于大规模知识库，需要专门的向量数据库来存储和快速检索：<\/p> 推荐工具：<\/p> redis-search<\/li> chroma<\/li> elasticsearch<\/li> opensearch<\/li> lancedb<\/li> pinecone<\/li> qdrant<\/li> weaviate<\/li> zilliz<\/li> <\/ul> 3.1.1 使用redis-search实现<\/h4> import<\/span> redis <\/span><\/span>from<\/span> redis.commands.search.query import<\/span> Query <\/span><\/span>from<\/span> redis.commands.search.field import<\/span> TextField, VectorField <\/span><\/span> <\/span><\/span>class<\/span> RedisCache<\/span>: <\/span><\/span> def<\/span> __init__(self, host: str =<\/span> "localhost"<\/span>, port: int =<\/span> 6379<\/span>): <\/span><\/span> self.<\/span>cache =<\/span> redis.<\/span>Redis(host=<\/span>host, port=<\/span>port) <\/span><\/span> <\/span><\/span> # 其他方法省略...<\/span> <\/span><\/span> <\/span><\/span># 初始化<\/span> <\/span><\/span>redis =<\/span> RedisCache() <\/span><\/span> <\/span><\/span># 定义字段<\/span> <\/span><\/span>info =<\/span> TextField("info"<\/span>) <\/span><\/span>name =<\/span> "embedding"<\/span> <\/span><\/span>algorithm =<\/span> "HNSW"<\/span> <\/span><\/span>attributes =<\/span> { <\/span><\/span> "TYPE"<\/span>: "FLOAT32"<\/span>, <\/span><\/span> "DIM"<\/span>: DIM, # 向量维度<\/span> <\/span><\/span> "DISTANCE_METRIC"<\/span>: "COSINE"<\/span> <\/span><\/span>} <\/span><\/span>embed =<\/span> VectorField(name=<\/span>name, algorithm=<\/span>algorithm, attributes=<\/span>attributes) <\/span><\/span> <\/span><\/span># 创建索引<\/span> <\/span><\/span>scheme =<\/span> (info, embed) <\/span><\/span>index =<\/span> redis.<\/span>getSchema(self.<\/span>redisIndexName) <\/span><\/span>redis.<\/span>createIndex(index, scheme) <\/span><\/span> <\/span><\/span># 插入数据<\/span> <\/span><\/span>def<\/span> insertData<\/span>(self, model): <\/span><\/span> j =<\/span> 0<\/span> <\/span><\/span> for<\/span> file in<\/span> self.<\/span>filesPath: <\/span><\/span> for<\/span> i in<\/span> file.<\/span>content: <\/span><\/span> embed =<\/span> self.<\/span>engine.<\/span>embeddings(i, model=<\/span>model) <\/span><\/span> emb =<\/span> numpy.<\/span>array(embed, dtype=<\/span>numpy.<\/span>float32).<\/span>tobytes() <\/span><\/span> im =<\/span> { <\/span><\/span> "info"<\/span>: i, <\/span><\/span> "embedding"<\/span>: emb, <\/span><\/span> } <\/span><\/span> name =<\/span> f<\/span>"<\/span>{<\/span>self.<\/span>redisIndexName}<\/span>-<\/span>{<\/span>j}<\/span>"<\/span> <\/span><\/span> j +=<\/span> 1<\/span> <\/span><\/span> self.<\/span>redis.<\/span>hset(name, im) <\/span><\/span> <\/span><\/span># 查询<\/span> <\/span><\/span>k =<\/span> 10<\/span> <\/span><\/span>base_query =<\/span> f<\/span>"* => [KNN <\/span>{<\/span>k}<\/span> @embedding $query_embedding AS similarity]"<\/span> <\/span><\/span>return_fields =<\/span> ["info"<\/span>, "similarity"<\/span>] <\/span><\/span>qr =<\/span> self.<\/span>engine.<\/span>embeddings(question, model=<\/span>model) <\/span><\/span>params_dict =<\/span> {"query_embedding"<\/span>: np.<\/span>array(qr, dtype=<\/span>np.<\/span>float32).<\/span>tobytes()} <\/span><\/span>index =<\/span> self.<\/span>redis.<\/span>getSchema(self.<\/span>redisIndexName) <\/span><\/span>result =<\/span> self.<\/span>redis.<\/span>query(index, base_query, return_fields, params_dict, k) <\/span><\/span><\/code><\/pre>3.2 Prompt设计示例<\/h3> prompt =<\/span> [ <\/span><\/span> { <\/span><\/span> "role"<\/span>: "user"<\/span>, <\/span><\/span> "content"<\/span>: f<\/span>"""当你收到用户的问题时，请编写清晰、简洁、准确的回答。你会收到一组与问题相关的上下文，请使用这些上下文，请使用中文回答用户的提问。不允许在答案中添加编造成分，如果给定的上下文没有提供足够的信息，就回答"##no##"。不要提供与问题无关的信息，也不要重复。 <\/span><\/span><\/span>> 上下文: <\/span><\/span><\/span>>>> <\/span><\/span><\/span><\/span>{<\/span>context}<\/span> <\/span><\/span><\/span>>>> <\/span><\/span><\/span>> 问题: <\/span><\/span><\/span><\/span>{<\/span>question}<\/span>"""<\/span> <\/span><\/span> } <\/span><\/span>] <\/span><\/span><\/code><\/pre>4. 实际应用中的难点与解决方案<\/h2> 4.1 难点一：大语言模型能力不足<\/h3> 问题表现<\/strong>：<\/p> 本地小模型(如qwen2:7b)能力有限，回答质量不如GPT-4<\/li> Embedding模型准确率低(约50-60%)<\/li> <\/ul> 解决方案<\/strong>：<\/p> 优化提问方式和上下文顺序<\/li> 使用rerank模型提高准确率<\/li> <\/ol> 4.1.1 Rerank模型实现<\/h4> from<\/span> BCEmbedding import<\/span> RerankerModel <\/span><\/span> <\/span><\/span># 初始化rerank模型<\/span> <\/span><\/span>rerankModel =<\/span> RerankerModel(model_name_or_path=<\/span>"maidalun1020\/bce-reranker-base_v1"<\/span>, local_files_only=<\/span>True<\/span>) <\/span><\/span> <\/span><\/span># 搜索出前20相似内容<\/span> <\/span><\/span>k =<\/span> 20<\/span> <\/span><\/span>result =<\/span> self.<\/span>redis.<\/span>query(index, base_query, return_fields, params_dict, k) <\/span><\/span>passages =<\/span> [doc.<\/span>info for<\/span> doc in<\/span> result.<\/span>docs] <\/span><\/span> <\/span><\/span># 重打分<\/span> <\/span><\/span>rerank_results =<\/span> rerankModel.<\/span>rerank(question, passages) <\/span><\/span>info =<\/span> rerank_results["rerank_passages"<\/span>] <\/span><\/span>last_result =<\/span> info[:10<\/span>] # 取前10高分结果<\/span> <\/span><\/span><\/code><\/pre>4.2 难点二：提问的复杂性<\/h3> 4.2.1 范围搜索性提问<\/h4> 问题示例<\/strong>：<\/p> "2024年的文章有哪些？"<\/li> "CTF相关的文章有哪些？"<\/li> <\/ul> 挑战<\/strong>：<\/p> 难以确定合适的k值<\/li> 可能遗漏相关内容或浪费计算资源<\/li> <\/ul> 解决方案<\/strong>：<\/p> 设置较大的初始k值(如1000)<\/li> 根据相似度阈值过滤结果<\/li> 使用上下文压缩技术<\/li> <\/ol> 4.2.2 多轮对话处理<\/h4> 问题示例<\/strong>：用户: 2024年的文章有哪些？助手: (列出部分文章) 用户: 还有吗？<\/p> 解决方案<\/strong>：<\/p> 检测低相似度分数判断为多轮对话<\/li> 维护对话状态和已显示结果<\/li> <\/ul> 4.3 难点三：文本处理<\/h3> 4.3.1 图片处理方案<\/h4> 使用OCR识别图片文字(效果有限)<\/li> 使用llava或GPT-4等模型描述图片<\/li> 直接返回图片链接并在prompt中说明<\/li> <\/ol> 4.3.2 文本分片策略<\/h4> 问题<\/strong>：<\/p> 简单按长度分片会割裂相关上下文(如代码段)<\/li> 重叠分片效果提升有限<\/li> <\/ul> 推荐方案<\/strong>：<\/p> 根据文档结构分片(如按一级标题)<\/li> 每个chunk包含文章基础信息<\/li> 对长代码段特殊处理<\/li> 针对不同格式文档设计专门分片策略<\/li> <\/ul> 示例<\/strong>：<\/p> from<\/span> llama_index import<\/span> SimpleDirectoryReader <\/span><\/span>from<\/span> llama_index.node_parser import<\/span> SimpleNodeParser <\/span><\/span> <\/span><\/span>documents =<\/span> SimpleDirectoryReader(input_dir=<\/span>".\/Documents"<\/span>).<\/span>load_data() <\/span><\/span>node_parser =<\/span> SimpleNodeParser.<\/span>from_defaults(chunk_size=<\/span>514<\/span>, chunk_overlap=<\/span>80<\/span>) <\/span><\/span>nodes =<\/span> node_parser.<\/span>get_nodes_from_documents(documents) <\/span><\/span><\/code><\/pre>4.4 上下文压缩技术<\/h3> def<\/span> compress<\/span>(self, question: str, context: list[str], maxToken: int =<\/span> 1024<\/span>) -><\/span> list[str]: <\/span><\/span> template =<\/span> f<\/span>"下面将会提供问题和上下文，请判断上下文信息是否和问题相关，如果不相关，请回复##no##，如果相关，请提取出和上下文相关的内容。*注意*:请直接提取出上下文的关键内容，请*不要*自行发挥，*不要*进行任何修改或者压缩删减操作。<\/span>\n\n<\/span>> 问题:<\/span>{<\/span>question}<\/span>\n<\/span>> 上下文:<\/span>\n<\/span>>>><\/span>\n<\/span>%s<\/span>\n<\/span>>>>"<\/span> <\/span><\/span> <\/span><\/span> result =<\/span> [] <\/span><\/span> for<\/span> c in<\/span> context: <\/span><\/span> qs =<\/span> template %<\/span> c <\/span><\/span> answer =<\/span> self.<\/span>engine.<\/span>chat(qs) <\/span><\/span> if<\/span> "##no##"<\/span> not<\/span> in<\/span> answer: <\/span><\/span> result +=<\/span> [answer] <\/span><\/span> <\/span><\/span> newContent =<\/span> "<\/span>\n<\/span>"<\/span>.<\/span>join(result) <\/span><\/span> question =<\/span> f<\/span>"你是一个去重机器人，下面将会提供一组上下文，请你对上下文进行去重处理。*注意*，请*不要*自行发挥，*不要*进行任何添加修改，请直接在上下文内容中进行去重。<\/span>\n<\/span>上下文:>>><\/span>\n<\/span>{<\/span>newContent}<\/span>\n<\/span>>>>"<\/span> <\/span><\/span> answer =<\/span> self.<\/span>engine.<\/span>chat(question) <\/span><\/span> return<\/span> answer <\/span><\/span><\/code><\/pre>5. 优化策略总结<\/h2> 相似度过滤<\/strong>：设置相似度阈值(如>0.4)和rerank分数阈值(如>0.5)<\/li> 动态k值<\/strong>：根据查询类型动态调整k值大小<\/li> 上下文压缩<\/strong>：去除无关内容，增加有效上下文量<\/li> 分片优化<\/strong>：根据文档结构而非简单长度分片<\/li> 多模型协作<\/strong>：结合embedding模型、rerank模型和生成模型<\/li> <\/ol> 6. 参考资源<\/h2> QAnything项目<\/a><\/li> LLMLingua项目<\/a><\/li> llama_index文档<\/a><\/li> langchain框架<\/a><\/li> <\/ol> 7. 结论<\/h2> 当前本地问答知识库的搭建仍面临诸多挑战，主要受限于大语言模型的能力。最佳实践是构建灵活框架，便于替换不同组件和模型。随着大语言模型技术的发展，未来有望实现更快速、更精准的问答系统。<\/p>

本地安全AI大模型攻防知识库搭建教程<\/h1>

1. 概述<\/h2> 本教程将详细介绍如何从零开始搭建一个本地大模型问答知识库，重点解决在搭建过程中遇到的各种问题及其解决方案。<\/p>

3. 技术实现细节<\/h2>

4. 实际应用中的难点与解决方案<\/h2>

4.2 难点二：提问的复杂性<\/h3>

4.3 难点三：文本处理<\/h3>

7. 结论<\/h2> 当前本地问答知识库的搭建仍面临诸多挑战，主要受限于大语言模型的能力。最佳实践是构建灵活框架，便于替换不同组件和模型。随着大语言模型技术的发展，未来有望实现更快速、更精准的问答系统。<\/p>

1. 概述<\/h2>
本教程将详细介绍如何从零开始搭建一个本地大模型问答知识库，重点解决在搭建过程中遇到的各种问题及其解决方案。<\/p>

7. 结论<\/h2>
当前本地问答知识库的搭建仍面临诸多挑战，主要受限于大语言模型的能力。最佳实践是构建灵活框架，便于替换不同组件和模型。随着大语言模型技术的发展，未来有望实现更快速、更精准的问答系统。<\/p>