基于代理IP的挖掘与分析教学文档<\/h1>

1. 代理IP概述<\/h2>
代理IP是指介于客户端和目标服务器之间的中间服务器，用于转发客户端请求并隐藏真实IP地址。代理IP在网络安全、数据爬取、隐私保护等领域有广泛应用。<\/p>

2. 代理IP挖掘与分析流程<\/h2>

2.1 整体思路<\/h3>

获取代理地址<\/li>
验证代理地址可用性<\/li>
提取代理指纹信息<\/li>
自动化挖掘代理<\/li>

数据分析与利用<\/li> <\/ol>

2.2 代理IP的主要用途<\/h3>

网络爬虫<\/li>
隐藏真实IP<\/li>
访问受限内容<\/li>
薅羊毛活动<\/li>

安全研究<\/li> <\/ul>

3. 代理IP获取方法<\/h2>

3.1 公开代理IP平台<\/h3>

www.xicidaili.com（国内高匿\/普通\/HTTPS\/HTTP代理）<\/li>

cn-proxy.com（需要梯子）<\/li> <\/ul>

3.2 爬取代理IP示例代码<\/h3>

#coding:utf-8<\/span>
<\/span><\/span>from<\/span> requests import<\/span> *<\/span>
<\/span><\/span>import<\/span> re
<\/span><\/span>
<\/span><\/span>headers =<\/span> {
<\/span><\/span>    "accept"<\/span>:"text\/html,application\/xhtml+xml,application\/xml;"<\/span>,
<\/span><\/span>    "accept-encoding"<\/span>:"gzip"<\/span>,
<\/span><\/span>    "accept-language"<\/span>:"zh-cn,zh;q=0.8"<\/span>,
<\/span><\/span>    "referer"<\/span>:"Mozilla\/5.0(compatible;Baiduspider\/2.0;+http:\/\/www.baidu.com\/search\/spider.html)"<\/span>,
<\/span><\/span>    "connection"<\/span>:"keep-alive"<\/span>,
<\/span><\/span>    "user-agent"<\/span>:"mozilla\/5.0(windows NT 6.1;wow64) applewebkit\/537.36 (khtml,like gecko)chrome\/42.0.2311.90 safari\/537.36"<\/span>
<\/span><\/span>}
<\/span><\/span>
<\/span><\/span>for<\/span> i in<\/span> range(1<\/span>,835<\/span>):
<\/span><\/span>    url =<\/span> 'http:\/\/www.xicidaili.com\/wn\/'<\/span> +<\/span> str(i)
<\/span><\/span>    html =<\/span> get(url,timeout=<\/span>3<\/span>,headers=<\/span>headers)
<\/span><\/span>    html.<\/span>encoding =<\/span> html.<\/span>apparent_encoding
<\/span><\/span>    proxyip =<\/span> r<\/span>'(<td>.*<\/td>)'<\/span>
<\/span><\/span>    iplist =<\/span> re.<\/span>findall(proxyip,html.<\/span>text)
<\/span><\/span>    i =<\/span> 1<\/span>
<\/span><\/span>    for<\/span> ip in<\/span> iplist:
<\/span><\/span>        ip =<\/span> (ip.<\/span>split('<td>'<\/span>)[1<\/span>]).<\/span>split('<\/td>'<\/span>)[0<\/span>]
<\/span><\/span>        f =<\/span> open('.\/ip.txt'<\/span>,'a'<\/span>)
<\/span><\/span>        print(ip,file=<\/span>f)
<\/span><\/span>        if<\/span> i%<\/span>5<\/span>==<\/span>0<\/span>:
<\/span><\/span>            print('<\/span>\n<\/span>'<\/span>,file=<\/span>f)
<\/span><\/span>        i =<\/span> i +<\/span> 1<\/span>
<\/span><\/span><\/code><\/pre>4. 代理IP验证<\/h2>
4.1 验证方法<\/h3>
通过代理访问baidu.com，返回状态为200则说明代理可用<\/p>
4.2 验证代码示例<\/h3>
#coding:utf-8<\/span>
<\/span><\/span>from<\/span> requests import<\/span> *<\/span>
<\/span><\/span>import<\/span> re
<\/span><\/span>
<\/span><\/span>for<\/span> proxy in<\/span> open("https.txt"<\/span>):
<\/span><\/span>    proxy =<\/span> proxy.<\/span>replace('<\/span>\n<\/span>'<\/span>,''<\/span>)
<\/span><\/span>    proxies=<\/span>{"https"<\/span>:proxy}
<\/span><\/span>    headers =<\/span> {
<\/span><\/span>        "Host"<\/span>: "www.baidu.com"<\/span>,
<\/span><\/span>        "User-Agent"<\/span>: "Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko\/20100101 Firefox\/56.0"<\/span>,
<\/span><\/span>        "Accept"<\/span>: "*\/*"<\/span>,
<\/span><\/span>        "Accept-Language"<\/span>: "en-US,en;q=0.5"<\/span>,
<\/span><\/span>        "Accept-Encoding"<\/span>: "gzip, deflate"<\/span>,
<\/span><\/span>        "Referer"<\/span>: "https:\/\/www.baidu.com\/"<\/span>
<\/span><\/span>    }
<\/span><\/span>    url =<\/span> 'https:\/\/www.baidu.com'<\/span>
<\/span><\/span>    try<\/span>:
<\/span><\/span>        html =<\/span> get(url,timeout=<\/span>10<\/span>,headers=<\/span>headers,proxies=<\/span>proxies)
<\/span><\/span>        if<\/span> html.<\/span>status_code ==<\/span> 200<\/span>:
<\/span><\/span>            proxy =<\/span> proxy.<\/span>split('https:\/\/'<\/span>)[1<\/span>]
<\/span><\/span>            f =<\/span> open('.\/proxyip.txt'<\/span>,'a'<\/span>)
<\/span><\/span>            print(proxy,file=<\/span>f)
<\/span><\/span>    except<\/span> Exception<\/span> as<\/span> e:
<\/span><\/span>        print(e)
<\/span><\/span>        pass<\/span>
<\/span><\/span><\/code><\/pre>5. 代理指纹提取<\/h2>
5.1 指纹提取方法<\/h3>

使用Nmap扫描识别服务类型<\/li>
分析HTTP响应头部信息（特别是Server字段）<\/li>
<\/ol>
5.2 常见代理指纹<\/h3>

MikroTik http proxy<\/li>
Squid<\/li>
App-webs<\/li>
Tengine<\/li>
Proxy<\/li>
Microsoft-IIS<\/li>
PCSERVER<\/li>
Apache<\/li>
<\/ul>
5.3 指纹提取代码<\/h3>
#coding:utf-8<\/span>
<\/span><\/span>from<\/span> requests import<\/span> *<\/span>
<\/span><\/span>
<\/span><\/span>headers =<\/span> {
<\/span><\/span>    "User-Agent"<\/span>: "Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko\/20100101 Firefox\/56.0"<\/span>,
<\/span><\/span>    "Accept"<\/span>: "*\/*"<\/span>,
<\/span><\/span>    "Accept-Language"<\/span>: "en-US,en;q=0.5"<\/span>,
<\/span><\/span>    "Accept-Encoding"<\/span>: "gzip, deflate"<\/span>,
<\/span><\/span>}
<\/span><\/span>
<\/span><\/span>for<\/span> url in<\/span> open("proxytest.txt"<\/span>):
<\/span><\/span>    url =<\/span> url.<\/span>split('<\/span>\n<\/span>'<\/span>)[0<\/span>]
<\/span><\/span>    try<\/span>:
<\/span><\/span>        html =<\/span> get(url,timeout=<\/span>3<\/span>,headers=<\/span>headers)
<\/span><\/span>        html =<\/span> html.<\/span>headers['server'<\/span>]
<\/span><\/span>    except<\/span> Exception<\/span> as<\/span> e:
<\/span><\/span>        pass<\/span>
<\/span><\/span>    f =<\/span> open('.\/proxyanalysis.txt'<\/span>,'a'<\/span>)
<\/span><\/span>    print(url,html,file=<\/span>f)
<\/span><\/span><\/code><\/pre>6. 代理指纹实战应用<\/h2>
6.1 使用搜索引擎查找代理<\/h3>

Fofa搜索语法：server:"MikrotikHttpProxy"<\/code><\/li>
Shodan、Zoomeye等也可用于搜索<\/li>
<\/ul>
6.2 常见代理端口分布<\/h3>
对29700个HTTPS代理端口的分析显示，以下端口出现频率较高（>100次）：<\/p>

8080<\/li>
3128<\/li>
80<\/li>
8888<\/li>
8081<\/li>
8000<\/li>
1080<\/li>
9000<\/li>
443<\/li>
8082<\/li>
<\/ul>
7. 数据分析与安全应用<\/h2>
7.1 长期存活的代理IP分析<\/h3>
SELECT<\/span> *<\/span> FROM<\/span> httpsproxy 
<\/span><\/span>WHERE<\/span> LiveDays LIKE<\/span> '%天'<\/span> AND<\/span> LiveDays ><\/span> 100<\/span> 
<\/span><\/span>ORDER<\/span> BY<\/span> LiveDays DESC<\/span>;
<\/span><\/span><\/code><\/pre>7.2 安全分析<\/h3>

识别政企单位泄露的代理<\/li>
使用云悉平台查询IP关联域名：

http:\/\/yunsee_info.vuln.cn\/?type=webip&url=1.2.3.4<\/code><\/li>
<\/ol>
7.3 批量查询代码<\/h3>
#coding:utf-8<\/span>
<\/span><\/span>from<\/span> requests import<\/span> *<\/span>
<\/span><\/span>import<\/span> re
<\/span><\/span>
<\/span><\/span>for<\/span> ip in<\/span> open('livedays.txt'<\/span>):
<\/span><\/span>    url =<\/span> 'http:\/\/yunsee_info.vuln.cn\/?type=webip&url='<\/span> +<\/span> ip
<\/span><\/span>    html =<\/span> get(url)
<\/span><\/span>    html =<\/span> html.<\/span>text
<\/span><\/span>    if<\/span> '404'<\/span> not<\/span> in<\/span> html:
<\/span><\/span>        f =<\/span> open('.\/ipipip.txt'<\/span>,'a'<\/span>)
<\/span><\/span>        print(ip,html,file=<\/span>f)
<\/span><\/span><\/code><\/pre>8. IP情报应用<\/h2>
代理IP可作为IP信誉情报的一部分：<\/p>

识别恶意IP<\/li>
防范薅羊毛行为<\/li>
增强安全防护<\/li>
<\/ul>
9. 总结与建议<\/h2>
9.1 安全观<\/h3>

专注于细分领域深度研究<\/li>
采用简单有效的解决方案<\/li>
网络层防护优先于应用层防护<\/li>
<\/ul>
9.2 最佳实践<\/h3>

定期爬取公开代理IP<\/li>
建立代理IP指纹库<\/li>
分析长期存活的代理IP<\/li>
将代理IP纳入威胁情报系统<\/li>
关注政企单位泄露的代理配置<\/li>
<\/ol>
9.3 扩展方向<\/h3>

代理IP区域分布分析<\/li>
代理服务器业务组件指纹<\/li>
路由器等设备代理特征<\/li>
自动化代理IP挖掘系统开发<\/li>
<\/ul>
通过系统化的代理IP挖掘与分析，可以有效提升网络安全防护能力，并为威胁情报提供重要数据支持。<\/p>

基于代理IP的挖掘与分析教学文档<\/h1>

1. 代理IP概述<\/h2> 代理IP是指介于客户端和目标服务器之间的中间服务器，用于转发客户端请求并隐藏真实IP地址。代理IP在网络安全、数据爬取、隐私保护等领域有广泛应用。<\/p>

2. 代理IP挖掘与分析流程<\/h2>

3. 代理IP获取方法<\/h2>

4. 代理IP验证<\/h2>

4.1 验证方法<\/h3> 通过代理访问baidu.com，返回状态为200则说明代理可用<\/p>

5. 代理指纹提取<\/h2>

6. 代理指纹实战应用<\/h2>

7. 数据分析与安全应用<\/h2>

9. 总结与建议<\/h2>

1. 代理IP概述<\/h2>
代理IP是指介于客户端和目标服务器之间的中间服务器，用于转发客户端请求并隐藏真实IP地址。代理IP在网络安全、数据爬取、隐私保护等领域有广泛应用。<\/p>

4.1 验证方法<\/h3>
通过代理访问baidu.com，返回状态为200则说明代理可用<\/p>