QtWebKit爬虫开发与去重规则详解<\/h1>

1. 基于QtWebKit的爬虫概述<\/h2>

传统爬虫(如urllib2、requests)的局限性：<\/p>

无法解析JavaScript<\/li>
难以抓取AJAX请求和动态生成的内容<\/li>

对交互式网页支持有限<\/li> <\/ul>

QtWebKit爬虫的优势：<\/p>

完整的浏览器内核，能解析JavaScript<\/li>
可捕获动态生成的内容和事件触发的请求<\/li>

适合漏洞扫描程序的需求<\/li> <\/ul>

2. 核心类解析<\/h2>

2.1 QNetworkAccessManager类<\/h3>

功能：网络请求管理，监控和拦截所有浏览器请求<\/p>

关键方法：<\/p>

createRequest<\/code>: 创建请求前记录URL、请求方式和数据<\/li>

_finished<\/code>: 请求完成后获取响应信息<\/li>
<\/ul>
重要代码实现：<\/p>
# 绑定网络访问管理器<\/span>
<\/span><\/span>page.<\/span>setNetworkAccessManager(manager)
<\/span><\/span>
<\/span><\/span># 获取返回状态<\/span>
<\/span><\/span>url =<\/span> str(reply.<\/span>url().<\/span>toString())
<\/span><\/span>status =<\/span> reply.<\/span>attribute(QNetworkRequest.<\/span>HttpStatusCodeAttribute)
<\/span><\/span>status, ok =<\/span> status.<\/span>toInt()
<\/span><\/span><\/code><\/pre>2.2 QWebView类<\/h3>
功能：浏览器控件，处理页面加载和JavaScript执行<\/p>
关键特性：<\/p>

需要自定义超时处理<\/li>
可修改浏览器环境(如禁用弹窗)<\/li>
可触发页面事件<\/li>
<\/ul>
重要实现：<\/p>
def<\/span> load<\/span>(self, url):
<\/span><\/span>    self.<\/span>loop =<\/span> QEventLoop()
<\/span><\/span>    timer =<\/span> QTimer()
<\/span><\/span>    timer.<\/span>setSingleShot(True<\/span>)
<\/span><\/span>    timer.<\/span>timeout.<\/span>connect(self.<\/span>_loadFinished)
<\/span><\/span>    timer.<\/span>start(self.<\/span>timeout *<\/span> 1000<\/span>)
<\/span><\/span>    super(Browser, self).<\/span>load(url)
<\/span><\/span>    self.<\/span>loop.<\/span>exec_()
<\/span><\/span>    if<\/span> timer.<\/span>isActive():
<\/span><\/span>        timer.<\/span>stop()
<\/span><\/span>    else<\/span>:
<\/span><\/span>        print "Request time out:"<\/span> +<\/span> str(url.<\/span>url().<\/span>toString())
<\/span><\/span><\/code><\/pre>3. 页面加载处理<\/h2>
3.1 加载前处理(_loadStarted)<\/h3>
禁用可能阻碍JavaScript执行的浏览器功能：<\/p>
def<\/span> _loadStarted<\/span>(self):
<\/span><\/span>    frame =<\/span> self.<\/span>page().<\/span>mainFrame()
<\/span><\/span>    frame.<\/span>evaluateJavaScript("window.alert=function()<\/span>{}<\/span>"<\/span>)
<\/span><\/span>    frame.<\/span>evaluateJavaScript("window.confirm=function(){return true}"<\/span>)
<\/span><\/span>    frame.<\/span>evaluateJavaScript("window.prompt=function(){return 0}"<\/span>)
<\/span><\/span>    frame.<\/span>evaluateJavaScript("window.open=function()<\/span>{}<\/span>"<\/span>)
<\/span><\/span><\/code><\/pre>3.2 加载后处理(_loadFinished)<\/h3>
触发各种DOM事件以捕获动态内容：<\/p>
def<\/span> _loadFinished<\/span>(self):
<\/span><\/span>    self.<\/span>loop.<\/span>quit()
<\/span><\/span>    frame =<\/span> self.<\/span>page().<\/span>mainFrame()
<\/span><\/span>    
<\/span><\/span>    # 转换a标签href为onclick并触发<\/span>
<\/span><\/span>    frame.<\/span>evaluateJavaScript('selectdom=document.querySelectorAll("a");for(var i=0;i<selectdom.length;i++){if(!selectdom[i].getAttribute("onclick")){selectdom[i].setAttribute("onclick",selectdom[i].getAttribute("href"))}}'<\/span>)
<\/span><\/span>    
<\/span><\/span>    # 触发各种事件处理器<\/span>
<\/span><\/span>    frame.<\/span>evaluateJavaScript('selectdom=document.querySelectorAll("[onerror]");for(var i=0;i<selectdom.length;i++){try{selectdom[i].onerror();}catch(err){continue;}}'<\/span>)
<\/span><\/span>    frame.<\/span>evaluateJavaScript('selectdom=document.querySelectorAll("[onchange]");for(var i=0;i<selectdom.length;i++){try{selectdom[i].onchange();}catch(err){continue;}}'<\/span>)
<\/span><\/span>    # 其他事件类型类似处理...<\/span>
<\/span><\/span><\/code><\/pre>4. 高效去重规则<\/h2>
4.1 URL规范化处理<\/h3>

去除netloc部分<\/li>
将所有数字替换为{{{int}}}<\/code>

示例：www.anquanke.com\/post\/id\/85596<\/code> → www.anquanke.com\/post\/id\/{{{int}}}<\/code><\/li>
<\/ul>
<\/li>
<\/ol>
4.2 参数分析去重<\/h3>
数据结构：<\/p>
{
<\/span><\/span>    netloc: [
<\/span><\/span>        {
<\/span><\/span>            参数1名称: [参数1内容1, 参数1内容2,...<\/span>],
<\/span><\/span>            参数2名称: [参数2内容1,...<\/span>]
<\/span><\/span>        }
<\/span><\/span>    ]
<\/span><\/span>}
<\/span><\/span><\/code><\/pre>去重规则：<\/p>

当某参数的值种类超过10个，则认为该参数无效<\/li>
后续比较时忽略该参数差异<\/li>
<\/ul>
复杂示例处理：

原始URL: http:\/\/tieba.baidu.com\/home\/main?un=_0d9y&fr=index&red_tag=p280448481<\/code><\/p>


数字替换后：

http:\/\/tieba.baidu.com\/home\/main?un=_{{{int}}}d{{{int}}}y&fr=index&red_tag=p{{{int}}}<\/code><\/p>
<\/li>

参数分析：<\/p>

如果un<\/code>参数出现超过10种不同值，后续忽略该参数差异<\/li>
<\/ul>
<\/li>
<\/ol>
5. 优化与实践建议<\/h2>


多进程架构<\/strong>：由于QtWebKit的page资源不能共享，建议使用多进程而非多线程<\/p>
<\/li>

数据记录<\/strong>：<\/p>

表单提交按钮信息(用于CSRF检测)<\/li>
请求方法、内容、状态码和Cookie<\/li>
响应头信息(用于检测重定向等漏洞)<\/li>
<\/ul>
<\/li>

反爬策略<\/strong>：<\/p>

在QNetworkAccessManager中随机生成User-Agent<\/li>
实现请求间隔和重试机制<\/li>
<\/ul>
<\/li>

漏洞检测准备<\/strong>：<\/p>

记录的数据可用于检测SQL注入、XSS、JSON劫持、404js等漏洞<\/li>
<\/ul>
<\/li>
<\/ol>
6. 替代方案<\/h2>
文中提到Google的Headless Chrome也是可行的替代方案，具有类似的功能但可能性能表现不同。<\/p>
7. 总结<\/h2>
QtWebKit爬虫通过完整的浏览器环境解决了传统爬虫无法处理JavaScript动态内容的问题，配合精心设计的去重规则，能够高效地抓取网页内容，特别适合漏洞扫描程序的需求。关键点在于合理利用QNetworkAccessManager和QWebView类，实现完整的请求监控和页面交互，并通过参数分析和URL规范化实现高效去重。<\/p>

QtWebKit爬虫开发与去重规则详解<\/h1>

2. 核心类解析<\/h2>

3. 页面加载处理<\/h2>

4. 高效去重规则<\/h2>

6. 替代方案<\/h2> 文中提到Google的Headless Chrome也是可行的替代方案，具有类似的功能但可能性能表现不同。<\/p>

6. 替代方案<\/h2>
文中提到Google的Headless Chrome也是可行的替代方案，具有类似的功能但可能性能表现不同。<\/p>