Scrapy爬虫框架全面教学文档<\/h1>

一、Scrapy简介<\/h2>

Scrapy是一个用于爬取网站并提取结构化数据的强大且灵活的开源框架。它具有以下特点：<\/p>

提供简单易用的工具和组件<\/li>
支持定义爬虫、调度请求、处理响应并存储提取的数据<\/li>
具有高效的异步处理能力<\/li>
支持分布式爬取<\/li>
通过中间件和扩展机制可以方便地定制和扩展功能<\/li>

广泛应用于数据挖掘、信息聚合和自动化测试等领域<\/li> <\/ul>

二、Scrapy工作流程<\/h2>

启动爬虫<\/strong>：Scrapy启动并激活爬虫，从初始URL开始爬取<\/li>
调度请求<\/strong>：爬虫生成初始请求，并将其发送给调度器<\/li>
下载页面<\/strong>：调度器将请求发送给下载器，下载器从互联网获取页面<\/li>
处理响应<\/strong>：下载器将响应返回给引擎，传递给爬虫<\/li>
提取数据<\/strong>：爬虫从响应中提取数据(items)和更多的URL(新的请求)<\/li>
处理数据<\/strong>：提取的数据通过项目管道进行处理，清洗并存储<\/li>

继续爬取<\/strong>：新的请求被调度器处理，继续下载和提取数据，直到所有请求处理完毕<\/li> <\/ol>
三、Scrapy安装与项目创建<\/h2>
1. 安装Scrapy<\/h3>
pip install scrapy <\/span><\/span><\/code><\/pre>2. 创建新项目<\/h3> scrapy startproject scrapy_demo <\/span><\/span><\/code><\/pre>3. 创建爬虫模板<\/h3> cd scrapy_demo <\/span><\/span>scrapy genspider example example.com <\/span><\/span><\/code><\/pre>4. 项目结构说明<\/h3> scrapy_demo\/ ├── spiders\/ # 存放爬虫文件 ├── items.py # 定义爬取的数据结构 ├── middlewares.py # 定义下载中间件和爬虫中间件 ├── pipelines.py # 定义管道，用于处理爬虫提取的数据 ├── settings.py # 项目的基本配置 <\/code><\/pre> 四、Scrapy基本使用<\/h2> 1. 编写基础爬虫<\/h3> import<\/span> scrapy <\/span><\/span> <\/span><\/span>class<\/span> ExampleSpider<\/span>(scrapy.<\/span>Spider): <\/span><\/span> name =<\/span> "example"<\/span> <\/span><\/span> start_urls =<\/span> ["https:\/\/movie.douban.com\/top250"<\/span>] <\/span><\/span> <\/span><\/span> def<\/span> parse<\/span>(self, response): <\/span><\/span> print(response.<\/span>text) <\/span><\/span><\/code><\/pre>2. 运行爬虫<\/h3> scrapy crawl example <\/span><\/span><\/code><\/pre>3. 解决403问题和robots协议<\/h3> 在settings.py中修改：<\/p> # 不遵守robots协议<\/span> <\/span><\/span>ROBOTSTXT_OBEY =<\/span> False<\/span> <\/span><\/span> <\/span><\/span># 设置默认请求头<\/span> <\/span><\/span>DEFAULT_REQUEST_HEADERS =<\/span> { <\/span><\/span> "Accept"<\/span>: "text\/html,application\/xhtml+xml,application\/xml;q=0.9,*\/*;q=0.8"<\/span>, <\/span><\/span> "Accept-Language"<\/span>: "en"<\/span>, <\/span><\/span> "User-Agent"<\/span>: "Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/125.0.0.0 Safari\/537.36 Edg\/125.0.0.0"<\/span> <\/span><\/span>} <\/span><\/span><\/code><\/pre>4. 两种运行方式<\/h3> 终端运行：<\/li> <\/ol> scrapy crawl example <\/span><\/span><\/code><\/pre> Python脚本运行：<\/li> <\/ol> from<\/span> scrapy import<\/span> cmdline <\/span><\/span> <\/span><\/span>if<\/span> __name__ ==<\/span> '__main__'<\/span>: <\/span><\/span> cmdline.<\/span>execute("scrapy crawl example"<\/span>.<\/span>split()) <\/span><\/span> # 不输出提示信息<\/span> <\/span><\/span> # cmdline.execute("scrapy crawl example --nolog".split())<\/span> <\/span><\/span><\/code><\/pre>五、数据提取与翻页<\/h2> 1. 数据提取示例<\/h3> def<\/span> parse<\/span>(self, response): <\/span><\/span> ol_list =<\/span> response.<\/span>xpath('\/\/ol[@class="grid_view"]\/li'<\/span>) <\/span><\/span> for<\/span> ol in<\/span> ol_list: <\/span><\/span> item =<\/span> {} <\/span><\/span> item['title'<\/span>] =<\/span> ol.<\/span>xpath('.\/\/div[@class="hd"]\/a\/span[1]\/text()'<\/span>).<\/span>extract_first() <\/span><\/span> item['rating'<\/span>] =<\/span> ol.<\/span>xpath('.\/\/div[@class="bd"]\/div\/span[2]\/text()'<\/span>).<\/span>extract_first() <\/span><\/span> item['quote'<\/span>] =<\/span> ol.<\/span>xpath('.\/\/div[@class="bd"]\/\/p[@class="quote"]\/span\/text()'<\/span>).<\/span>extract_first() <\/span><\/span> print(item) <\/span><\/span><\/code><\/pre>2. 翻页处理两种方式<\/h3> 方式一：使用callback回调<\/h4> def<\/span> parse<\/span>(self, response): <\/span><\/span> # ...数据提取代码...<\/span> <\/span><\/span> if<\/span> response.<\/span>xpath("\/\/a[text()='后页>']\/@href"<\/span>).<\/span>extract_first() is<\/span> not<\/span> None<\/span>: <\/span><\/span> next_url =<\/span> response.<\/span>urljoin(response.<\/span>xpath("\/\/a[text()='后页>']\/@href"<\/span>).<\/span>extract_first()) <\/span><\/span> yield<\/span> scrapy.<\/span>Request(url=<\/span>next_url, callback=<\/span>self.<\/span>parse) <\/span><\/span><\/code><\/pre>方式二：重写start_requests方法<\/h4> def<\/span> start_requests<\/span>(self): <\/span><\/span> for<\/span> i in<\/span> range(0<\/span>, 5<\/span>): <\/span><\/span> url =<\/span> 'https:\/\/movie.douban.com\/top250?start=<\/span>{}<\/span>&filter='<\/span>.<\/span>format(i *<\/span> 25<\/span>) <\/span><\/span> yield<\/span> scrapy.<\/span>Request(url) <\/span><\/span><\/code><\/pre>六、Scrapy Item使用<\/h2> 1. 定义Item<\/h3> import<\/span> scrapy <\/span><\/span> <\/span><\/span>class<\/span> ScrapyDemoItem<\/span>(scrapy.<\/span>Item): <\/span><\/span> title =<\/span> scrapy.<\/span>Field() <\/span><\/span> rating =<\/span> scrapy.<\/span>Field() <\/span><\/span> quote =<\/span> scrapy.<\/span>Field() <\/span><\/span><\/code><\/pre>2. 使用Item<\/h3> from<\/span> scrapy_demo.items import<\/span> ScrapyDemoItem <\/span><\/span> <\/span><\/span>def<\/span> parse<\/span>(self, response): <\/span><\/span> ol_list =<\/span> response.<\/span>xpath('\/\/ol[@class="grid_view"]\/li'<\/span>) <\/span><\/span> for<\/span> ol in<\/span> ol_list: <\/span><\/span> item =<\/span> ScrapyDemoItem() <\/span><\/span> item['title'<\/span>] =<\/span> ol.<\/span>xpath('.\/\/div[@class="hd"]\/a\/span[1]\/text()'<\/span>).<\/span>extract_first() <\/span><\/span> item['rating'<\/span>] =<\/span> ol.<\/span>xpath('.\/\/div[@class="bd"]\/div\/span[2]\/text()'<\/span>).<\/span>extract_first() <\/span><\/span> item['quote'<\/span>] =<\/span> ol.<\/span>xpath('.\/\/div[@class="bd"]\/\/p[@class="quote"]\/span\/text()'<\/span>).<\/span>extract_first() <\/span><\/span> yield<\/span> item <\/span><\/span><\/code><\/pre>七、数据存储Pipeline<\/h2> 1. MySQL存储示例<\/h3> import<\/span> pymysql <\/span><\/span>from<\/span> itemadapter import<\/span> ItemAdapter <\/span><\/span> <\/span><\/span>class<\/span> MysqlPipeline<\/span>: <\/span><\/span> def<\/span> __init__(self): <\/span><\/span> self.<\/span>connection =<\/span> pymysql.<\/span>connect( <\/span><\/span> user=<\/span>'root'<\/span>, <\/span><\/span> password=<\/span>'root'<\/span>, <\/span><\/span> db=<\/span>'scrapy_demo'<\/span>, <\/span><\/span> ) <\/span><\/span> self.<\/span>cursor =<\/span> self.<\/span>connection.<\/span>cursor() <\/span><\/span> self.<\/span>create_table() <\/span><\/span> <\/span><\/span> def<\/span> create_table<\/span>(self): <\/span><\/span> table =<\/span> """ <\/span><\/span><\/span> CREATE TABLE IF NOT EXISTS douban ( <\/span><\/span><\/span> id INT AUTO_INCREMENT PRIMARY KEY, <\/span><\/span><\/span> title VARCHAR(255) NOT NULL, <\/span><\/span><\/span> rating FLOAT NOT NULL, <\/span><\/span><\/span> quote TEXT <\/span><\/span><\/span> )CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; <\/span><\/span><\/span> """<\/span> <\/span><\/span> self.<\/span>cursor.<\/span>execute(table) <\/span><\/span> self.<\/span>connection.<\/span>commit() <\/span><\/span> <\/span><\/span> def<\/span> process_item<\/span>(self, item, spider): <\/span><\/span> try<\/span>: <\/span><\/span> self.<\/span>cursor.<\/span>execute("INSERT INTO douban(id,title, rating, quote) VALUES (<\/span>%s<\/span>,<\/span>%s<\/span>, <\/span>%s<\/span>, <\/span>%s<\/span>)"<\/span>, <\/span><\/span> (0<\/span>, item['title'<\/span>], item['rating'<\/span>], item['quote'<\/span>])) <\/span><\/span> self.<\/span>connection.<\/span>commit() <\/span><\/span> except<\/span> pymysql.<\/span>MySQLError as<\/span> e: <\/span><\/span> spider.<\/span>logger.<\/span>error(f<\/span>"Error saving item: <\/span>{<\/span>e}<\/span>"<\/span>) <\/span><\/span> print(e) <\/span><\/span> return<\/span> item <\/span><\/span> <\/span><\/span> def<\/span> close_spider<\/span>(self, spider): <\/span><\/span> self.<\/span>cursor.<\/span>close() <\/span><\/span> self.<\/span>connection.<\/span>close() <\/span><\/span><\/code><\/pre>2. 启用Pipeline<\/h3> 在settings.py中：<\/p> ITEM_PIPELINES =<\/span> { <\/span><\/span> "scrapy_demo.pipelines.MysqlPipeline"<\/span>: 300<\/span>, <\/span><\/span>} <\/span><\/span><\/code><\/pre>八、Scrapy中间件<\/h2> 1. 下载中间件基础<\/h3> from<\/span> feapder.network import<\/span> user_agent <\/span><\/span> <\/span><\/span>class<\/span> ScrapyDemoDownloaderMiddleware<\/span>: <\/span><\/span> def<\/span> process_request<\/span>(self, request, spider): <\/span><\/span> request.<\/span>headers['User-Agent'<\/span>] =<\/span> user_agent.<\/span>get() <\/span><\/span> return<\/span> None<\/span> <\/span><\/span><\/code><\/pre>2. 设置代理<\/h3> def<\/span> process_request<\/span>(self, request, spider): <\/span><\/span> request.<\/span>headers['User-Agent'<\/span>] =<\/span> user_agent.<\/span>get() <\/span><\/span> request.<\/span>meta['proxy'<\/span>] =<\/span> "http:\/\/127.0.0.1:7890"<\/span> <\/span><\/span> return<\/span> None<\/span> <\/span><\/span><\/code><\/pre>3. 中间件权重<\/h3> 在settings.py中：<\/p> DOWNLOADER_MIDDLEWARES =<\/span> { <\/span><\/span> "scrapy_demo.middlewares.OneMiddleware"<\/span>: 543<\/span>, <\/span><\/span> "scrapy_demo.middlewares.TwoMiddleware"<\/span>: 544<\/span> <\/span><\/span>} <\/span><\/span><\/code><\/pre>九、Scrapy-Redis分布式爬虫<\/h2> 1. 安装<\/h3> pip install scrapy-redis <\/span><\/span><\/code><\/pre>注意：Scrapy版本需要替换成2.9.0版本或者2.0.0以下<\/p> 2. 配置<\/h3> 在settings.py中：<\/p> # 设置Redis主机和端口<\/span> <\/span><\/span>REDIS_URL =<\/span> 'redis:\/\/127.0.0.1:6379\/0'<\/span> <\/span><\/span> <\/span><\/span># 使用Scrapy-Redis的调度器<\/span> <\/span><\/span>SCHEDULER =<\/span> "scrapy_redis.scheduler.Scheduler"<\/span> <\/span><\/span> <\/span><\/span># 使用Scrapy-Redis的去重器<\/span> <\/span><\/span>DUPEFILTER_CLASS =<\/span> "scrapy_redis.dupefilter.RFPDupeFilter"<\/span> <\/span><\/span> <\/span><\/span># 开启redis管道<\/span> <\/span><\/span>ITEM_PIPELINES =<\/span> { <\/span><\/span> "scrapy_redis.pipelines.RedisPipeline"<\/span>: 301<\/span> <\/span><\/span>} <\/span><\/span><\/code><\/pre>3. 编写Redis爬虫<\/h3> from<\/span> scrapy_redis.spiders import<\/span> RedisSpider <\/span><\/span> <\/span><\/span>class<\/span> RedisDemoSpider<\/span>(RedisSpider): <\/span><\/span> name =<\/span> "redis_demo"<\/span> <\/span><\/span> redis_key =<\/span> "redis_demo:start_urls"<\/span> # redis key<\/span> <\/span><\/span> <\/span><\/span> def<\/span> parse<\/span>(self, response): <\/span><\/span> ol_list =<\/span> response.<\/span>xpath('\/\/ol[@class="grid_view"]\/li'<\/span>) <\/span><\/span> for<\/span> ol in<\/span> ol_list: <\/span><\/span> item =<\/span> {} <\/span><\/span> item['title'<\/span>] =<\/span> ol.<\/span>xpath('.\/\/div[@class="hd"]\/a\/span[1]\/text()'<\/span>).<\/span>extract_first() <\/span><\/span> item['rating'<\/span>] =<\/span> ol.<\/span>xpath('.\/\/div[@class="bd"]\/div\/span[2]\/text()'<\/span>).<\/span>extract_first() <\/span><\/span> item['quote'<\/span>] =<\/span> ol.<\/span>xpath('.\/\/div[@class="bd"]\/\/p[@class="quote"]\/span\/text()'<\/span>).<\/span>extract_first() <\/span><\/span> print(item) <\/span><\/span> yield<\/span> item <\/span><\/span><\/code><\/pre>4. 添加URL到Redis队列<\/h3> import<\/span> redis <\/span><\/span> <\/span><\/span>r =<\/span> redis.<\/span>Redis(db=<\/span>0<\/span>) <\/span><\/span>r.<\/span>lpush('redis_demo:start_urls'<\/span>, "https:\/\/movie.douban.com\/top250"<\/span>) <\/span><\/span><\/code><\/pre>5. 持久化存储<\/h3> 在settings.py中：<\/p> SCHEDULER_PERSIST =<\/span> True<\/span> # 持久化爬取状态<\/span> <\/span><\/span><\/code><\/pre>6. 分布式配置<\/h3> 在settings.py中修改Redis连接：<\/p> REDIS_URL =<\/span> '远程电脑ip地址'<\/span> <\/span><\/span><\/code><\/pre>十、关键对象说明<\/h2> 1. Request对象参数<\/h3> 参数<\/th> 描述<\/th> <\/tr> <\/thead> url<\/td> 请求的URL<\/td> <\/tr> callback<\/td> 用于处理该请求的回调函数，默认是parse方法<\/td> <\/tr> method<\/td> HTTP请求方法，如'GET','POST'等，默认为'GET'<\/td> <\/tr> headers<\/td> 请求头信息<\/td> <\/tr> body<\/td> 请求体，通常在POST请求中使用<\/td> <\/tr> cookies<\/td> 请求携带的Cookies<\/td> <\/tr> meta<\/td> 该请求的元数据字典，用于在不同请求之间传递数据<\/td> <\/tr> encoding<\/td> 请求的编码格式，默认为'utf-8'<\/td> <\/tr> priority<\/td> 请求的优先级，默认值为0<\/td> <\/tr> <\/tbody> <\/table> 2. Response对象参数<\/h3> 参数<\/th> 描述<\/th> <\/tr> <\/thead> url<\/td> 响应的URL<\/td> <\/tr> status<\/td> HTTP响应状态码<\/td> <\/tr> headers<\/td> 响应头信息<\/td> <\/tr> body<\/td> 响应体内容，二进制格式<\/td> <\/tr> flags<\/td> 响应的标志列表<\/td> <\/tr> request<\/td> 生成此响应的请求对象<\/td> <\/tr> meta<\/td> 该请求的元数据字典<\/td> <\/tr> encoding<\/td> 响应的编码格式<\/td> <\/tr> text<\/td> 响应体内容，解码为字符串格式<\/td> <\/tr> css<\/td> CSS选择器<\/td> <\/tr> xpath<\/td> XPath选择器<\/td> <\/tr> json<\/td> 解析JSON响应体并返回字典或列表<\/td> <\/tr> <\/tbody> <\/table> 十一、最佳实践建议<\/h2> 遵守robots协议<\/strong>：除非必要，否则应遵守目标网站的robots.txt规则<\/li> 设置合理的请求间隔<\/strong>：避免对目标网站造成过大压力<\/li> 处理异常情况<\/strong>：网络错误、页面解析失败等情况应有相应处理<\/li> 日志记录<\/strong>：记录爬取过程中的关键信息，便于排查问题<\/li> 数据去重<\/strong>：使用Scrapy-Redis或其他方式实现数据去重<\/li> 分布式部署<\/strong>：对于大规模爬取任务，考虑使用分布式架构<\/li> 遵守法律法规<\/strong>：确保爬取行为符合相关法律法规和网站使用条款<\/li> <\/ol>