URL采集器设计与实现教学文档<\/h1>

1. 项目背景与需求分析<\/h2>

1.1 项目起源<\/h3>

初始需求：检查公司网站首页是否被挂了暗链<\/li>
现有工具不足：网上可用脚本较少且功能有限<\/li>

需求扩展：随着要求不断增加，最终发展为一款功能完善的URL采集器<\/li> <\/ul>

1.2 传统URL采集工具的局限性<\/h3>

采集范围有限<\/strong>：仅能获取被搜索引擎收录的URL<\/li>
过滤机制粗糙<\/strong>：仅依靠去重+黑名单过滤<\/li>

结果同质化<\/strong>：使用相同关键词采集的结果相似，对安全人员不友好<\/li> <\/ol>
2. 系统设计与功能规划<\/h2>
2.1 核心功能<\/h3>

双入口设计<\/strong>：<\/p>

搜索引擎接口采集<\/li>
导入已采集网址进行二次处理<\/li> <\/ul> <\/li>

深度采集机制<\/strong>：<\/p>

基于初始URL进行友链爬行<\/li>
可自定义是否进行友链爬取<\/li> <\/ul> <\/li>

多维度过滤<\/strong>：<\/p>

URL黑白名单<\/li>
网站标题黑白名单<\/li>
网页内容黑白名单<\/li> <\/ul> <\/li> <\/ol>
2.2 系统流程图<\/h3>
[启动] → [选择采集方式] → [关键字采集\/文本导入] → [初步过滤] → [友链爬取(可选)] → [多级过滤] → [结果保存] <\/code><\/pre> 3. 关键技术实现<\/h2> 3.1 URL采集方式<\/h3> 3.1.1 正则表达式过滤方式<\/h4> def<\/span> GetLink<\/span>(url): <\/span><\/span> UA =<\/span> random.<\/span>choice(headerss) <\/span><\/span> headers =<\/span> {'User-Agent'<\/span>: UA, 'Connection'<\/span>: 'close'<\/span>} <\/span><\/span> link_urls =<\/span> [] <\/span><\/span> try<\/span>: <\/span><\/span> r =<\/span> requests.<\/span>get(url, headers=<\/span>headers, verify=<\/span>False<\/span>, timeout=<\/span>timeout) <\/span><\/span> encoing =<\/span> requests.<\/span>utils.<\/span>get_encodings_from_content(r.<\/span>text)[0<\/span>] <\/span><\/span> content =<\/span> r.<\/span>content.<\/span>decode(encoing) <\/span><\/span> urls =<\/span> [f<\/span>"<\/span>{<\/span>urlparse(url).<\/span>scheme}<\/span>:\/\/<\/span>{<\/span>urlparse(url).<\/span>netloc}<\/span>"<\/span> <\/span><\/span> for<\/span> url in<\/span> re.<\/span>findall(r<\/span>'http[s]?:\/\/(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*<\/span> <\/span><\/span>$$<\/span> <\/span><\/span>,]|<\/span>(?<\/span>:%<\/span>[0<\/span>-<\/span>9<\/span>a-<\/span>fA-<\/span>F][0<\/span>-<\/span>9<\/span>a-<\/span>fA-<\/span>F]))+<\/span>', content, re.I)]<\/span> <\/span><\/span> for<\/span> url in<\/span> list(set(urls)): <\/span><\/span> url =<\/span> url.<\/span>replace("'"<\/span>, ""<\/span>) <\/span><\/span> link_urls.<\/span>append(url) <\/span><\/span> except<\/span>: <\/span><\/span> pass<\/span> <\/span><\/span> return<\/span> list(set(link_urls)) <\/span><\/span><\/code><\/pre>3.1.2 BeautifulSoup过滤方式<\/h4> def<\/span> GetLink<\/span>(url): <\/span><\/span> UA =<\/span> random.<\/span>choice(headerss) <\/span><\/span> headers =<\/span> {'User-Agent'<\/span>: UA, 'Connection'<\/span>: 'close'<\/span>} <\/span><\/span> try<\/span>: <\/span><\/span> r =<\/span> requests.<\/span>get(url, headers=<\/span>headers, verify=<\/span>False<\/span>) <\/span><\/span> encoding =<\/span> requests.<\/span>utils.<\/span>get_encodings_from_content(r.<\/span>text)[0<\/span>] <\/span><\/span> content =<\/span> r.<\/span>content.<\/span>decode(encoding) <\/span><\/span> soup =<\/span> BeautifulSoup(content, 'html.parser'<\/span>) <\/span><\/span> bs4_urls =<\/span> set() <\/span><\/span> for<\/span> tag in<\/span> ['a'<\/span>, 'img'<\/span>, 'script'<\/span>, 'link'<\/span>]: <\/span><\/span> for<\/span> attr in<\/span> ['href'<\/span>, 'src'<\/span>]: <\/span><\/span> for<\/span> element in<\/span> soup.<\/span>find_all(tag): <\/span><\/span> if<\/span> attr in<\/span> element.<\/span>attrs: <\/span><\/span> href =<\/span> element.<\/span>get(attr) <\/span><\/span> if<\/span> href and<\/span> (href.<\/span>startswith('http:\/\/'<\/span>) or<\/span> href.<\/span>startswith('https:\/\/'<\/span>)): <\/span><\/span> parsed =<\/span> urlparse(href) <\/span><\/span> url =<\/span> f<\/span>"<\/span>{<\/span>parsed.<\/span>scheme}<\/span>:\/\/<\/span>{<\/span>parsed.<\/span>netloc}<\/span>"<\/span> <\/span><\/span> bs4_urls.<\/span>add(url) <\/span><\/span> except<\/span> Exception<\/span> as<\/span> e: <\/span><\/span> pass<\/span> <\/span><\/span> <\/span><\/span> link_urls =<\/span> [] <\/span><\/span> for<\/span> bs4_url in<\/span> bs4_urls: <\/span><\/span> try<\/span>: <\/span><\/span> r =<\/span> requests.<\/span>head(bs4_url, timeout=<\/span>5<\/span>, headers=<\/span>headers, verify=<\/span>False<\/span>) <\/span><\/span> if<\/span> r.<\/span>status_code in<\/span> [200<\/span>, 301<\/span>, 302<\/span>]: <\/span><\/span> link_urls.<\/span>append(bs4_url) <\/span><\/span> except<\/span>: <\/span><\/span> pass<\/span> <\/span><\/span> return<\/span> link_urls <\/span><\/span><\/code><\/pre>3.2 搜索引擎接口采集<\/h3> 百度搜索接口实现<\/h4> def<\/span> BDUrl<\/span>(key): <\/span><\/span> cookie =<\/span> input('请输入cookie:'<\/span>) <\/span><\/span> bd_headers =<\/span> { <\/span><\/span> "User-Agent"<\/span>: "Mozilla\/5.0 (Windows NT 10.0; Win64; x64)..."<\/span>, <\/span><\/span> "Cookie"<\/span>: cookie, <\/span><\/span> "Connection"<\/span>: "Keep-Alive"<\/span>, <\/span><\/span> } <\/span><\/span> bd_url =<\/span> [] <\/span><\/span> for<\/span> page in<\/span> range(0<\/span>, 8<\/span>): <\/span><\/span> url =<\/span> 'http:\/\/www.baidu.com\/s?wd=<\/span>{}<\/span>&pn=<\/span>{}<\/span>0'<\/span> <\/span><\/span> try<\/span>: <\/span><\/span> r =<\/span> requests.<\/span>get(url.<\/span>format(key, page), headers=<\/span>bd_headers, verify=<\/span>False<\/span>) <\/span><\/span> encoing =<\/span> requests.<\/span>utils.<\/span>get_encodings_from_content(r.<\/span>text)[0<\/span>] <\/span><\/span> content =<\/span> r.<\/span>content.<\/span>decode(encoing) <\/span><\/span> result =<\/span> [f<\/span>"<\/span>{<\/span>urlparse(url).<\/span>scheme}<\/span>:\/\/<\/span>{<\/span>urlparse(url).<\/span>netloc}<\/span>"<\/span> <\/span><\/span> for<\/span> url in<\/span> re.<\/span>findall('mu="http[s]?:\/\/(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*<\/span> <\/span><\/span>$$<\/span> <\/span><\/span>,]|<\/span>(?<\/span>:%<\/span>[0<\/span>-<\/span>9<\/span>a-<\/span>fA-<\/span>F][0<\/span>-<\/span>9<\/span>a-<\/span>fA-<\/span>F]))+<\/span>"', content)[1:]]<\/span> <\/span><\/span> for<\/span> res_url in<\/span> list(set(result)): <\/span><\/span> bd_url.<\/span>append(res_url) <\/span><\/span> except<\/span>: <\/span><\/span> pass<\/span> <\/span><\/span> return<\/span> list(set(bd_url)) <\/span><\/span><\/code><\/pre>3.3 配置文件设计<\/h3> config.ini 结构<\/h4> [User]<\/span> <\/span><\/span>whoami<\/span> =<\/span> JF<\/span> <\/span><\/span> <\/span><\/span>[Config]<\/span> <\/span><\/span># 友链爬行,0关闭,1开启<\/span> <\/span><\/span>state<\/span> =<\/span> 0<\/span> <\/span><\/span># 网址黑名单<\/span> <\/span><\/span>black_url<\/span> =<\/span> None<\/span> <\/span><\/span># 网址白名单<\/span> <\/span><\/span>white_url<\/span> =<\/span> None<\/span> <\/span><\/span># 标题黑名单<\/span> <\/span><\/span>black_title<\/span> =<\/span> None<\/span> <\/span><\/span># 标题白名单<\/span> <\/span><\/span>white_title<\/span> =<\/span> 安全狗<\/span> <\/span><\/span># 网页内容黑名单<\/span> <\/span><\/span>black_content<\/span> =<\/span> None<\/span> <\/span><\/span># 网页内容白名单<\/span> <\/span><\/span>white_content<\/span> =<\/span> None<\/span> <\/span><\/span># 连接超时5秒<\/span> <\/span><\/span>timeout<\/span> =<\/span> 5<\/span> <\/span><\/span><\/code><\/pre>3.4 过滤规则实现<\/h3> 3.4.1 多级过滤规则（单URL版）<\/h4> def<\/span> rule_url<\/span>(url): <\/span><\/span> # URL黑名单过滤<\/span> <\/span><\/span> if<\/span> black_url !=<\/span> 'None'<\/span> and<\/span> (any(key in<\/span> url for<\/span> key in<\/span> black_url.<\/span>split('|'<\/span>))): <\/span><\/span> return<\/span> False<\/span> <\/span><\/span> <\/span><\/span> # URL白名单过滤<\/span> <\/span><\/span> if<\/span> white_url !=<\/span> 'None'<\/span> and<\/span> (all(key not<\/span> in<\/span> url for<\/span> key in<\/span> white_url.<\/span>split('|'<\/span>))): <\/span><\/span> return<\/span> False<\/span> <\/span><\/span> <\/span><\/span> try<\/span>: <\/span><\/span> UA =<\/span> random.<\/span>choice(headerss) <\/span><\/span> header =<\/span> {'User-Agent'<\/span>: UA, 'Connection'<\/span>: 'close'<\/span>} <\/span><\/span> r =<\/span> requests.<\/span>get(url=<\/span>url, headers=<\/span>header, verify=<\/span>False<\/span>, timeout=<\/span>timeout) <\/span><\/span> if<\/span> r.<\/span>status_code in<\/span> [200<\/span>, 301<\/span>, 302<\/span>]: <\/span><\/span> encoing =<\/span> requests.<\/span>utils.<\/span>get_encodings_from_content(r.<\/span>text)[0<\/span>] <\/span><\/span> content =<\/span> r.<\/span>content.<\/span>decode(encoing) <\/span><\/span> title =<\/span> re.<\/span>findall('<title>(.*?)<\/title>'<\/span>, content, re.<\/span>S)[0<\/span>] <\/span><\/span> <\/span><\/span> # 标题黑名单过滤<\/span> <\/span><\/span> if<\/span> black_title !=<\/span> 'None'<\/span> and<\/span> (any(key in<\/span> title for<\/span> key in<\/span> black_title.<\/span>split('|'<\/span>))): <\/span><\/span> return<\/span> False<\/span> <\/span><\/span> <\/span><\/span> # 标题白名单过滤<\/span> <\/span><\/span> if<\/span> white_title !=<\/span> 'None'<\/span> and<\/span> (all(key not<\/span> in<\/span> title for<\/span> key in<\/span> white_title.<\/span>split('|'<\/span>))): <\/span><\/span> return<\/span> False<\/span> <\/span><\/span> <\/span><\/span> # 内容黑名单过滤<\/span> <\/span><\/span> if<\/span> black_content !=<\/span> 'None'<\/span> and<\/span> (any(key in<\/span> content for<\/span> key in<\/span> black_content.<\/span>split('|'<\/span>))): <\/span><\/span> return<\/span> False<\/span> <\/span><\/span> <\/span><\/span> # 内容白名单过滤<\/span> <\/span><\/span> if<\/span> white_content !=<\/span> 'None'<\/span> and<\/span> (all(key not<\/span> in<\/span> content for<\/span> key in<\/span> white_content.<\/span>split('|'<\/span>))): <\/span><\/span> return<\/span> False<\/span> <\/span><\/span> <\/span><\/span> return<\/span> url <\/span><\/span> else<\/span>: <\/span><\/span> return<\/span> False<\/span> <\/span><\/span> except<\/span>: <\/span><\/span> return<\/span> False<\/span> <\/span><\/span> return<\/span> False<\/span> <\/span><\/span><\/code><\/pre>3.4.2 多线程处理<\/h4> with<\/span> ThreadPoolExecutor(max_workers=<\/span>10<\/span>) as<\/span> executor: <\/span><\/span> results_bd =<\/span> executor.<\/span>map(rule_url, bd) <\/span><\/span> with<\/span> lock: <\/span><\/span> for<\/span> url in<\/span> results_bd: <\/span><\/span> if<\/span> url: <\/span><\/span> rule_bd_urls.<\/span>add(url) <\/span><\/span><\/code><\/pre>4. 系统使用指南<\/h2> 4.1 启动与配置<\/h3> 修改config.ini<\/code>文件：<\/p> state<\/code>: 友链爬行开关(0\/1)<\/li> 黑白名单设置：使用|<\/code>分隔多个关键词<\/li> None<\/code>表示不启用该过滤条件<\/li> <\/ul> <\/li> 优先级规则：<\/p> 网址黑名单 > 网址白名单 > 标题黑名单 > 标题白名单 > 网页内容黑名单 > 网页内容白名单 <\/code><\/pre> <\/li> <\/ol> 4.2 运行方式<\/h3> 方式一：关键字采集<\/h4> 0<\/span>:关键字扫描 <\/span><\/span>请输入关键字:教育 <\/span><\/span><\/code><\/pre>流程：<\/p> 通过百度搜索接口获取前7页结果<\/li> 对每个URL应用过滤规则<\/li> 可选进行友链爬取<\/li> 结果保存为{关键字}_url.txt<\/code><\/li> <\/ol> 方式二：文本导入采集<\/h4> 1<\/span>:导入文本扫描 <\/span><\/span>将需要扫描的url拖到此窗口:urls.<\/span>txt <\/span><\/span><\/code><\/pre>选项：<\/p> 不进行友链爬取(state=0)<\/li> 进行友链爬取(state=1)<\/li> <\/ul> 4.3 使用示例<\/h3> 示例1：搜索引擎采集教育类站点<\/h4> 选择方式0<\/li> 输入关键字"教育"<\/li> 系统自动采集并过滤<\/li> 结果保存为"教育_url.txt"<\/li> <\/ol> 示例2：文本导入+友链爬取<\/h4> 准备包含URL的文本文件<\/li> 选择方式1并拖入文件<\/li> 设置state=1启用友链爬取<\/li> 系统进行多级过滤和友链采集<\/li> 结果保存到指定文件<\/li> <\/ol> 5. 性能优化与注意事项<\/h2> 5.1 性能优化点<\/h3> 从批量处理改为单URL处理，便于并发<\/li> 使用线程池(ThreadPoolExecutor)提高效率<\/li> 减少不必要的网络请求<\/li> 使用HEAD方法检查URL存活状态<\/li> <\/ol> 5.2 注意事项<\/h3> 百度搜索需要提供有效的Cookie<\/li> 确保配置文件字段不为空<\/li> 文本中的URL需包含协议头(http\/https)<\/li> 合理设置超时时间(timeout)<\/li> 注意网络请求频率，避免被封禁<\/li> <\/ol> 6. 项目资源<\/h2> GitHub仓库: https:\/\/github.com\/JiangFengSec\/JF_URL<\/a><\/p> 7. 总结与扩展<\/h2> 本URL采集器通过多入口采集、深度爬取和多级过滤机制，解决了传统URL采集工具的局限性。系统设计考虑了安全人员的特殊需求，提供了高度可定制的过滤规则。开发者可以根据实际需求进一步扩展：<\/p> 增加更多搜索引擎接口<\/li> 实现分布式爬取<\/li> 添加代理支持<\/li> 完善异常处理和日志记录<\/li> 增加可视化界面<\/li> <\/ol>