ElastAlert监控日志告警Web攻击行为 - 详细教学文档<\/h1>

一、ELK环境搭建<\/h2>

1.1 ELK组件介绍<\/h3>

Elasticsearch<\/strong>：分布式、可扩展的实时搜索与数据分析引擎<\/li>
Logstash<\/strong>：轻量级日志搜集处理框架，可自定义处理日志并传输<\/li>

Kibana<\/strong>：开源分析与可视化平台，用于展示Elasticsearch数据<\/li> <\/ul>
三者关系：<\/p>

Logstash（Controller层）：数据收集和过滤<\/li>
Elasticsearch（Model层）：数据存储和索引<\/li>
Kibana（View层）：数据可视化和展示<\/li> <\/ul>
1.2 版本选择与安装<\/h3>
重要版本限制<\/strong>：<\/p>

ElastAlert不支持Elasticsearch 6.0+版本<\/li>
推荐版本组合：

Elasticsearch: 5.5.2<\/li>
Kibana: 5.5.2<\/li>
Logstash: 6.0.0<\/li>
Filebeat: 6.0.0<\/li> <\/ul> <\/li> <\/ul>
安装方法<\/strong>：<\/p>

Ubuntu系统使用deb包安装：
dpkg -i package.deb <\/span><\/span><\/code><\/pre><\/li> CentOS系统使用rpm包安装<\/li> <\/ul> 1.3 ELK配置<\/h3> Elasticsearch配置<\/h4> \/etc\/elasticsearch\/elasticsearch.yml<\/code>:<\/p> network.host<\/span>: 127.0.0.1<\/span> <\/span><\/span>http.port<\/span>: 9200<\/span> <\/span><\/span><\/code><\/pre>Kibana配置<\/h4> \/etc\/kibana\/kibana.yml<\/code>:<\/p> server.port<\/span>: 5601<\/span> <\/span><\/span>server.host<\/span>: "localhost"<\/span> <\/span><\/span>elasticsearch.url<\/span>: "http:\/\/localhost:9200"<\/span> <\/span><\/span><\/code><\/pre>Logstash配置<\/h4> 示例配置文件filebeat_log.conf<\/code>:<\/p> input { <\/span><\/span> beats { <\/span><\/span> port =><\/span> 5044<\/span> <\/span><\/span> client_inactivity_timeout =><\/span> 90<\/span> <\/span><\/span> codec =><\/span> json <\/span><\/span> } <\/span><\/span>} <\/span><\/span> <\/span><\/span>filter { <\/span><\/span> date { <\/span><\/span> match =><\/span> [<\/span>"timestamp"<\/span>, "dd\/MMM\/yyyy:HH:mm:ss Z"<\/span>]<\/span> <\/span><\/span> target =><\/span> [<\/span>"datetime"<\/span>]<\/span> <\/span><\/span> } <\/span><\/span> geoip { <\/span><\/span> source =><\/span> "remote_addr"<\/span> <\/span><\/span> } <\/span><\/span> mutate { <\/span><\/span> remove_field =><\/span> [<\/span>"tags"<\/span>, "beat"<\/span>]<\/span> <\/span><\/span> } <\/span><\/span>} <\/span><\/span> <\/span><\/span>output { <\/span><\/span> elasticsearch { <\/span><\/span> hosts =><\/span> "localhost:9200"<\/span> <\/span><\/span> index =><\/span> "logstash-%{+YYYY.MM.dd}"<\/span> <\/span><\/span> } <\/span><\/span> stdout {} <\/span><\/span>} <\/span><\/span><\/code><\/pre>1.4 ELK启动<\/h3> 启动Elasticsearch和Kibana：<\/p> systemctl daemon-reload <\/span><\/span>systemctl enable elasticsearch.service <\/span><\/span>systemctl enable kibana.service <\/span><\/span>systemctl start elasticsearch <\/span><\/span>systemctl start kibana <\/span><\/span><\/code><\/pre>启动Logstash：<\/p> nohup .\/logstash -f filebeat_log.conf > \/dev\/null 2>&1<\/span> & <\/span><\/span><\/code><\/pre>二、日志收集架构<\/h2> 2.1 架构设计<\/h3> [被监控服务器A\/B\/C...] → [主服务器(ELK+ElastAlert)] <\/code><\/pre> 使用Filebeat作为轻量级日志收集器，Logstash作为中心过滤器<\/p> 2.2 Filebeat配置<\/h3> 示例filebeat.yml<\/code>：<\/p> filebeat.prospectors<\/span>: <\/span><\/span>- type<\/span>: log<\/span> <\/span><\/span> enabled<\/span>: true<\/span> <\/span><\/span> paths<\/span>: <\/span><\/span> - \/path\/to\/logs\/*.txt<\/span> <\/span><\/span> document_type<\/span>: tomcat-log<\/span> <\/span><\/span> scan_frequency<\/span>: 15s<\/span> <\/span><\/span> ignore_older<\/span>: 20m<\/span> <\/span><\/span> close_inactive<\/span>: 12m<\/span> <\/span><\/span> clean_inactive<\/span>: 30m<\/span> <\/span><\/span> close_removed<\/span>: true<\/span> <\/span><\/span> clean_removed<\/span>: true<\/span> <\/span><\/span> <\/span><\/span>output.logstash<\/span>: <\/span><\/span> hosts<\/span>: ["logstash_server:5044"<\/span>] <\/span><\/span><\/code><\/pre>启动Filebeat：<\/p> nohup .\/filebeat -e -c filebeat.yml >\/dev\/null 2>&1<\/span> & <\/span><\/span><\/code><\/pre>三、日志格式转换<\/h2> 3.1 Tomcat JSON格式配置<\/h3> 修改server.xml<\/code>：<\/p> <Valve<\/span> className=<\/span>"org.apache.catalina.valves.AccessLogValve"<\/span> <\/span><\/span> directory=<\/span>"logs"<\/span> <\/span><\/span> prefix=<\/span>"localhost_access_log"<\/span> <\/span><\/span> suffix=<\/span>".txt"<\/span> <\/span><\/span> pattern=<\/span>'{"time":"%t","remote_addr":"%h","remote_user":"%l","request":"%r","status":"%s","body_bytes_sent":"%b","http_referer":"%{Referer}i","http_user_agent":"%{User-Agent}i","http_x_forwarded_for":"%{X-Forwarded-For}i","request_time":"%T","host":"%v","port":"%p"}'<\/span> \/><\/span> <\/span><\/span><\/code><\/pre>3.2 Nginx JSON格式配置<\/h3> 在nginx.conf<\/code>中添加：<\/p> log_format<\/span> logstash_json<\/span> '<\/span>{ "time"<\/span> : "<\/span>$time_local", '<\/span> <\/span><\/span> '"remote_addr"<\/span> : "<\/span>$remote_addr", '<\/span> <\/span><\/span> '"remote_user"<\/span> : "<\/span>$remote_user", '<\/span> <\/span><\/span> '"request"<\/span> : "<\/span>$request", '<\/span> <\/span><\/span> '"status"<\/span> : "<\/span>$status", '<\/span> <\/span><\/span> '"body_bytes_sent"<\/span> : "<\/span>$body_bytes_sent", '<\/span> <\/span><\/span> '"http_referer"<\/span> : "<\/span>$http_referer", '<\/span> <\/span><\/span> '"http_user_agent"<\/span> : "<\/span>$http_user_agent", '<\/span> <\/span><\/span> '"http_x_forwarded_for"<\/span> : "<\/span>$http_x_forwarded_for", '<\/span> <\/span><\/span> '"request_time"<\/span> : "<\/span>$request_time", '<\/span> <\/span><\/span> '"request_length"<\/span> : "<\/span>$request_length", '<\/span> <\/span><\/span> '"host"<\/span> : "<\/span>$http_host" }<\/span>'<\/span>; <\/span><\/span><\/code><\/pre>四、ElastAlert配置与使用<\/h2> 4.1 安装ElastAlert<\/h3> git clone https:\/\/github.com\/Yelp\/elastalert.git <\/span><\/span>cd elastalert <\/span><\/span>python setup.py install <\/span><\/span>pip install -r requirements.txt <\/span><\/span>cp config.yaml.example config.yaml <\/span><\/span><\/code><\/pre>4.2 创建索引<\/h3> elastalert-create-index <\/span><\/span><\/code><\/pre>4.3 主配置文件config.yaml<\/h3> rules_folder<\/span>: example_rules<\/span> <\/span><\/span>run_every<\/span>: <\/span><\/span> seconds<\/span>: 3<\/span> <\/span><\/span>buffer_time<\/span>: <\/span><\/span> minutes<\/span>: 15<\/span> <\/span><\/span>es_host<\/span>: 188.88.88.88<\/span> <\/span><\/span>es_port<\/span>: 9200<\/span> <\/span><\/span>writeback_index<\/span>: elastalert_status<\/span> <\/span><\/span>alert_time_limit<\/span>: <\/span><\/span> days<\/span>: 1<\/span> <\/span><\/span><\/code><\/pre>4.4 告警规则配置<\/h3> 示例webattack_frequency.yaml<\/code>：<\/p> name<\/span>: web attack<\/span> <\/span><\/span>realert<\/span>: <\/span><\/span> minutes<\/span>: 5<\/span> <\/span><\/span>type<\/span>: frequency<\/span> <\/span><\/span>index<\/span>: logstash-*<\/span> <\/span><\/span>num_events<\/span>: 10<\/span> <\/span><\/span>timeframe<\/span>: <\/span><\/span> minutes<\/span>: 1<\/span> <\/span><\/span> <\/span><\/span>filter<\/span>: <\/span><\/span>- query_string<\/span>: <\/span><\/span> query<\/span>: "request: select.+(from|limit) OR request: union(.*?)select OR request: into.+(dump|out)file OR request: (base64_decode|sleep|benchmark|and.+1=1|and.+1=2|or %20|exec|information_schema|where %20|union %20|%2ctable_name %20|cmdshell|table_schema) OR request: (iframe|script|body|img|layer|div|meta|style|base|object|input|onmouseover|onerror|onload) OR request: .+etc.+passwd OR http_user_agent:(HTTrack|harvest|audit|dirbuster|pangolin|nmap|sqln|-scan|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|PycURL|zmeu|BabyKrokodil|netsparker|httperf|bench) OR status: (400|404|500|501) NOT (request:_health.html OR remote_addr:222.222.222.222)"<\/span> <\/span><\/span> <\/span><\/span>smtp_host<\/span>: smtp.qiye.163.com<\/span> <\/span><\/span>smtp_port<\/span>: 25<\/span> <\/span><\/span>smtp_auth_file<\/span>: \/path\/to\/smtp_auth_file.yaml<\/span> <\/span><\/span>email_reply_to<\/span>: xxx@163.com<\/span> <\/span><\/span>from_addr<\/span>: xxx@163.com<\/span> <\/span><\/span>alert<\/span>: <\/span><\/span>- "email"<\/span> <\/span><\/span>email<\/span>: <\/span><\/span>- "recipient@example.com"<\/span> <\/span><\/span>alert_subject<\/span>: "web attack may be by {} at @{}"<\/span> <\/span><\/span>alert_subject_args<\/span>: <\/span><\/span>- remote_addr<\/span> <\/span><\/span>- time<\/span> <\/span><\/span>alert_text_type<\/span>: alert_text_only<\/span> <\/span><\/span>alert_text<\/span>: | <\/span><\/span><\/span> 你好,服务器({})可能正在受到web攻击,请采取手段阻止!!!! <\/span><\/span><\/span> ### 截止发邮件前匹配到的请求数:{} <\/span><\/span><\/span> > 发生时间: {} <\/span><\/span><\/span> > timestamp:{} <\/span><\/span><\/span> > attacker's ip: {} <\/span><\/span><\/span> > request: {} <\/span><\/span><\/span> > status:{} <\/span><\/span><\/span> > UA头:{} <\/span><\/span><\/span> >>> 参考来源:{}<\/span> <\/span><\/span>alert_text_args<\/span>: <\/span><\/span>- host<\/span> <\/span><\/span>- num_hits<\/span> <\/span><\/span>- time<\/span> <\/span><\/span>- "@timestamp"<\/span> <\/span><\/span>- remote_addr<\/span> <\/span><\/span>- request<\/span> <\/span><\/span>- status<\/span> <\/span><\/span>- http_user_agent<\/span> <\/span><\/span>- source<\/span> <\/span><\/span><\/code><\/pre>SMTP认证文件smtp_auth_file.yaml<\/code>：<\/p> user<\/span>: xxx@163.com<\/span> <\/span><\/span>password<\/span>: yourpassword<\/span> <\/span><\/span><\/code><\/pre>4.5 启动ElastAlert<\/h3> nohup python -m elastalert.elastalert --verbose --rule webattack_frequency.yaml >\/dev\/null 2>&1<\/span> & <\/span><\/span><\/code><\/pre>五、告警效果示例<\/h2> 收到的告警邮件示例：<\/p> 主题: web attack may be by 104.38.13.21 at @[13\/Jan\/2018:16:06:58 +0800] 内容: 你好,服务器(199.222.36.31)可能正在受到web攻击,请采取手段阻止!!!! ### 截止发邮件前匹配到的请求数:20 > 发生时间: [13\/Jan\/2018:16:06:58 +0800] > timestamp: 2018-01-13T08:07:04.930Z > attacker's ip: 184.233.9.121 > request: GET \/dbadmin\/scripts\/setup.php HTTP\/1.0 > status: 200 > UA头: ZmEu >>> 参考来源: \/log\/localhost_access_log.2018-01-13.txt <\/code><\/pre> 六、最佳实践与注意事项<\/h2> 版本兼容性<\/strong>：确保ElastAlert与Elasticsearch版本兼容<\/li> 性能考虑<\/strong>：使用Filebeat代替直接使用Logstash收集日志<\/li> 日志格式<\/strong>：统一使用JSON格式便于分析<\/li> 规则优化<\/strong>：根据实际攻击模式调整检测规则<\/li> 告警频率<\/strong>：合理设置realert时间避免告警风暴<\/li> 测试验证<\/strong>：使用elastalert-test-rule<\/code>测试规则有效性<\/li> 错误处理<\/strong>：遇到索引问题时可以删除并重建索引<\/li> <\/ol> 七、扩展功能<\/h2> 其他告警方式<\/strong>：可集成微信、钉钉等告警渠道<\/li> 其他规则类型<\/strong>：ElastAlert支持11种告警规则，可根据需求选择<\/li> 可视化增强<\/strong>：结合Kibana Dashboard进行攻击可视化分析<\/li> 守护进程<\/strong>：使用Supervisor管理ElastAlert进程<\/li> <\/ol> 通过以上配置，可以构建一个完整的Web攻击监控和告警系统，及时发现并响应潜在的安全威胁。<\/p>