Feapder爬虫框架全面教学文档<\/h1>

一、Feapder框架概述<\/h2>

Feapder是一款功能强大且易于上手的Python爬虫框架，具有以下特点：<\/p>

支持四种爬虫类型应对不同场景需求<\/li>
内置断点续爬、监控报警、浏览器渲染、海量数据去重等功能<\/li>
提供强大的爬虫管理系统Feaplat<\/li>

相比Scrapy学习成本更低，比Pyspider维护更好<\/li> <\/ul>

二、Feapder框架架构设计<\/h2>

1. 核心模块<\/h3>

spider：爬虫调度器<\/li>
request_buffer：请求缓冲区<\/li>
collector：任务收集器<\/li>
parser_control：解析控制器<\/li>

item_buffer：数据缓冲区<\/li> <\/ul>

2. 工作流程<\/h3>

spider调度start_request生产任务<\/li>
start_request下发任务到request_buffer<\/li>
request_buffer批量存储任务到任务队列数据库<\/li>
collector从任务队列批量获取任务到内存队列<\/li>
parser_control从collector获取任务<\/li>
request请求和下载数据<\/li>
response封装返回给parser_control<\/li>
parser调度对应parser解析response<\/li>
解析结果分发到item_buffer和request_buffer<\/li>

数据批量入库<\/li> <\/ol>

三、环境准备与安装<\/h2>

1. 环境要求<\/h3>

Python 3.6.0+<\/li>

支持Linux\/Windows\/macOS<\/li> <\/ul>

2. 安装选项<\/h3>

# 精简版(不支持浏览器渲染、内存去重、MongoDB入库)<\/span>
<\/span><\/span>pip install feapder
<\/span><\/span>
<\/span><\/span># 浏览器渲染版<\/span>
<\/span><\/span>pip install "feapder[render]"<\/span>
<\/span><\/span>
<\/span><\/span># 完整版(支持所有功能)<\/span>
<\/span><\/span>pip install "feapder[all]"<\/span>
<\/span><\/span><\/code><\/pre>3. 安装验证<\/h3>
feapder
<\/span><\/span><\/code><\/pre>支持命令：create、retry、shell、zip<\/p>
四、爬虫类型详解<\/h2>
1. AirSpider（轻量级爬虫）<\/h3>
特点<\/strong>：<\/p>

学习成本低<\/li>
适合数据量少的简单场景<\/li>
无需断点续爬和分布式采集<\/li>
<\/ul>
创建项目<\/h4>
feapder create -p <project_name>
<\/span><\/span><\/code><\/pre>目录结构<\/h4>
project_name\/
├── items\/          # 数据库表映射的item
├── spiders\/        # 爬虫脚本
├── CHECK_DATA.md   # 数据审核建议
├── main.py         # 运行入口
└── setting.py      # 配置文件
<\/code><\/pre>
创建爬虫<\/h4>
feapder create -s <spider_name>
<\/span><\/span><\/code><\/pre>基础示例<\/h4>
import<\/span> feapder
<\/span><\/span>from<\/span> loguru import<\/span> logger
<\/span><\/span>
<\/span><\/span>class<\/span> FeapderSpiderDemo<\/span>(feapder.<\/span>AirSpider):
<\/span><\/span>    def<\/span> start_requests<\/span>(self):
<\/span><\/span>        yield<\/span> feapder.<\/span>Request("https:\/\/www.kuaidaili.com\/free"<\/span>)
<\/span><\/span>
<\/span><\/span>    def<\/span> parse<\/span>(self, request, response):
<\/span><\/span>        logger.<\/span>info(response.<\/span>xpath("\/\/title\/text()"<\/span>).<\/span>extract_first())
<\/span><\/span>        logger.<\/span>info(f<\/span>"网站地址: <\/span>{<\/span>response.<\/span>url}<\/span>"<\/span>)
<\/span><\/span>
<\/span><\/span>if<\/span> __name__ ==<\/span> "__main__"<\/span>:
<\/span><\/span>    FeapderSpiderDemo().<\/span>start()
<\/span><\/span><\/code><\/pre>进阶功能<\/h4>

自定义解析函数<\/li>
下载中间件<\/li>
失败重试机制<\/li>
线程数配置<\/li>
停止爬虫方法<\/li>
<\/ol>
2. Spider（分布式爬虫）<\/h3>
特点<\/strong>：<\/p>

基于Redis的分布式爬虫<\/li>
适用于海量数据采集<\/li>
支持断点续爬、爬虫报警、数据自动入库<\/li>
<\/ul>
创建示例<\/h4>
import<\/span> feapder
<\/span><\/span>
<\/span><\/span>class<\/span> FeapderSpiderDemo<\/span>(feapder.<\/span>Spider):
<\/span><\/span>    __custom_setting__ =<\/span> dict(
<\/span><\/span>        REDISDB_IP_PORTS=<\/span>"localhost:6379"<\/span>,
<\/span><\/span>        REDISDB_USER_PASS=<\/span>""<\/span>,
<\/span><\/span>        REDISDB_DB=<\/span>0<\/span>
<\/span><\/span>    )
<\/span><\/span>
<\/span><\/span>    def<\/span> start_requests<\/span>(self):
<\/span><\/span>        yield<\/span> feapder.<\/span>Request("https:\/\/www.kuaidaili.com\/free"<\/span>)
<\/span><\/span>
<\/span><\/span>    def<\/span> parse<\/span>(self, request, response):
<\/span><\/span>        print(response.<\/span>xpath("\/\/title\/text()"<\/span>).<\/span>extract_first())
<\/span><\/span>
<\/span><\/span>if<\/span> __name__ ==<\/span> "__main__"<\/span>:
<\/span><\/span>    FeapderSpiderDemo(redis_key=<\/span>"xxx:xxx"<\/span>).<\/span>start()
<\/span><\/span><\/code><\/pre>断点续爬原理<\/h4>

使用Redis有序集合存储任务<\/li>
任务分数=时间戳<\/li>
只取分数<当前时间的任务<\/li>
任务完成后主动删除<\/li>
<\/ul>
3. TaskSpider（任务型爬虫）<\/h3>
特点<\/strong>：<\/p>

分布式爬虫<\/li>
内置从Redis或MySQL获取种子任务的逻辑<\/li>
可自定义实现其他任务来源<\/li>
<\/ul>
4. BatchSpider（批次爬虫）<\/h3>
特点<\/strong>：<\/p>

分布式批次爬虫<\/li>
专为周期性采集设计<\/li>
自动维护批次信息表<\/li>
确保前一批次完成前不会开始新批次<\/li>
<\/ul>
五、核心功能详解<\/h2>
1. 数据库支持<\/h3>
MysqlDB<\/h4>

线程池管理连接<\/li>
断开自动重连<\/li>
最大连接数100<\/li>
封装增删改查方法<\/li>
<\/ul>
RedisDB<\/h4>

支持三种模式：

普通模式(单节点)<\/li>
哨兵模式<\/li>
集群模式<\/li>
<\/ul>
<\/li>
<\/ul>
2. 浏览器渲染<\/h3>
支持浏览器：<\/p>

Chrome<\/li>
Edge<\/li>
PhantomJS<\/li>
Firefox<\/li>
<\/ul>
配置示例：<\/p>
WEBDRIVER =<\/span> dict(
<\/span><\/span>    pool_size=<\/span>1<\/span>,          # 浏览器数量<\/span>
<\/span><\/span>    load_images=<\/span>True<\/span>,     # 是否加载图片<\/span>
<\/span><\/span>    headless=<\/span>False<\/span>,       # 无头模式<\/span>
<\/span><\/span>    driver_type=<\/span>"CHROME"<\/span>, # 浏览器类型<\/span>
<\/span><\/span>    timeout=<\/span>30<\/span>,          # 请求超时时间<\/span>
<\/span><\/span>    window_size=<\/span>(1024<\/span>, 800<\/span>),
<\/span><\/span>    render_time=<\/span>0<\/span>,        # 渲染等待时间<\/span>
<\/span><\/span>    auto_install_driver=<\/span>False<\/span> # 自动下载驱动<\/span>
<\/span><\/span>)
<\/span><\/span><\/code><\/pre>3. 多线程配置<\/h3>
三种配置方式：<\/p>

启动函数传递：<\/li>
<\/ol>
AirSpiderTest(thread_count=<\/span>10<\/span>).<\/span>start()
<\/span><\/span><\/code><\/pre>
setting.py配置：<\/li>
<\/ol>
SPIDER_THREAD_COUNT =<\/span> 32<\/span>
<\/span><\/span><\/code><\/pre>
类变量自定义：<\/li>
<\/ol>
__custom_setting__ =<\/span> dict(
<\/span><\/span>    SPIDER_THREAD_COUNT=<\/span>10<\/span>,
<\/span><\/span>)
<\/span><\/span><\/code><\/pre>4. 监控报警<\/h3>

内置监控打点(feapder>=1.6.6)<\/li>
可部署到Feaplat系统实现请求和数据监控<\/li>
<\/ul>
六、爬虫管理系统Feaplat<\/h2>
功能特点<\/h3>

方便部署和调度爬虫<\/li>
可视化监控<\/li>
任务管理<\/li>
报警系统<\/li>
<\/ul>
安装部署<\/h3>
# 通过Docker安装(暂不支持Apple芯片)<\/span>
<\/span><\/span>docker安装方式详见官方文档
<\/span><\/span><\/code><\/pre>七、最佳实践建议<\/h2>


根据数据量选择合适爬虫类型：<\/p>

少量数据：AirSpider<\/li>
海量数据：Spider<\/li>
周期性采集：BatchSpider<\/li>
任务驱动：TaskSpider<\/li>
<\/ul>
<\/li>

生产环境推荐使用完整版安装<\/p>
<\/li>

分布式环境确保Redis配置正确<\/p>
<\/li>

合理设置线程数以平衡性能与稳定性<\/p>
<\/li>

利用Feaplat系统进行生产环境管理<\/p>
<\/li>
<\/ol>
八、官方资源<\/h2>

GitHub: https:\/\/github.com\/Boris-code\/feapder<\/li>
官方文档: https:\/\/feapder.com\/<\/li>
官方公众号: feader爬虫教程<\/li>
<\/ul>

Feapder爬虫框架全面教学文档<\/h1>

二、Feapder框架架构设计<\/h2>

三、环境准备与安装<\/h2>

四、爬虫类型详解<\/h2>

五、核心功能详解<\/h2>

1. 数据库支持<\/h3>

六、爬虫管理系统Feaplat<\/h2>

八、官方资源<\/h2> GitHub: https:\/\/github.com\/Boris-code\/feapder<\/li> 官方文档: https:\/\/feapder.com\/<\/li> 官方公众号: feader爬虫教程<\/li> <\/ul>

八、官方资源<\/h2>

GitHub: https:\/\/github.com\/Boris-code\/feapder<\/li>
官方文档: https:\/\/feapder.com\/<\/li>
官方公众号: feader爬虫教程<\/li> <\/ul>