WAF自学习模式开发实战教学文档<\/h1>

一、自学习模式概述<\/h2>

1.1 传统WAF白名单功能的问题<\/h3>

基于概率统计学的传统方法存在两个主要问题：

学习时间过长，业务频繁变更时模型难以收敛<\/li>

误报率较高，导致用户下线功能<\/li> <\/ul> <\/li> <\/ul>

1.2 自学习模式的目标<\/h3>

无需或只需少量用户干预<\/li>
降低WAF漏报率<\/li>

提升检测速度<\/li> <\/ol>

1.3 jxwaf自学习模式的创新点<\/h3>

不采用概率统计学方法<\/li>

实现三个维度的分析：

正则自适应匹配<\/li>
长度分析<\/li>

词法分析<\/li> <\/ul> <\/li> <\/ul>

二、核心实现原理<\/h2>

2.1 基本思路<\/h3>

通过分析用户输入中的特殊字符来区分正常数据和攻击语句<\/li>

防护关键：识别用户输入中的特殊字符<\/li> <\/ul>

2.2 正则自适应匹配<\/h3>

2.2.1 学习过程<\/h4>

local<\/span> function<\/span> white_learn_rx<\/span>(value)
<\/span><\/span>    local<\/span> learn_value =<\/span> value
<\/span><\/span>    local<\/span> level
<\/span><\/span>    local<\/span> level_one =<\/span> _white_config.level_one or<\/span> [=[\W]=]<\/span>
<\/span><\/span>    local<\/span> level_two =<\/span> _white_config.level_two or<\/span> [==[[^\w ]==]<\/span>
<\/span><\/span>    local<\/span> level_three =<\/span> _white_config.level_three or<\/span> [==[[^\w ]==]<\/span>
<\/span><\/span>    local<\/span> level_four =<\/span> _white_config.level_four or<\/span> [==<\/span>[<>=<\/span>]=<\/span>]
<\/span><\/span>    
<\/span><\/span>    if<\/span> (not<\/span> ngx.re.find(learn_value, level_one,"oij"<\/span>)) then<\/span>
<\/span><\/span>        level =<\/span> 1<\/span>
<\/span><\/span>    elseif<\/span> (not<\/span> ngx.re.find(learn_value, level_two,"oij"<\/span>)) then<\/span>
<\/span><\/span>        level =<\/span> 2<\/span>
<\/span><\/span>    elseif<\/span> (not<\/span> ngx.re.find(learn_value, level_three,"oij"<\/span>)) then<\/span>
<\/span><\/span>        level =<\/span> 3<\/span>
<\/span><\/span>    elseif<\/span> (not<\/span> ngx.re.find(learn_value, level_four,"oij"<\/span>)) then<\/span>
<\/span><\/span>        level =<\/span> 4<\/span>
<\/span><\/span>    else<\/span>
<\/span><\/span>        level =<\/span> 6<\/span>
<\/span><\/span>    end<\/span>
<\/span><\/span>    return<\/span> level
<\/span><\/span>end<\/span>
<\/span><\/span><\/code><\/pre>2.2.2 五层匹配规则<\/h4>

第一层：参数值为数字、字母和_组成<\/li>
第二层：参数值为数字、字母和_、"、()、.组成<\/li>
第三层：参数值为数字、字母和_组成（含常见特殊字符）<\/li>
第四层：参数值不包含<、>、=等危险特殊字符<\/li>
第五层：含任意字符<\/li>
<\/ol>
2.2.3 检测过程<\/h4>
local<\/span> function<\/span> white_check_rx<\/span>(value,level)
<\/span><\/span>    local<\/span> check_value =<\/span> value
<\/span><\/span>    local<\/span> check_level =<\/span> level
<\/span><\/span>    local<\/span> level_one =<\/span> _white_config.level_one or<\/span> [=[\W]=]<\/span>
<\/span><\/span>    local<\/span> level_two =<\/span> _white_config.level_two or<\/span> [==[[^\w ]==]<\/span>
<\/span><\/span>    local<\/span> level_three =<\/span> _white_config.level_three or<\/span> [==[[^\w ]==]<\/span>
<\/span><\/span>    local<\/span> level_four =<\/span> _white_config.level_four or<\/span> [==<\/span>[<>=<\/span>]=<\/span>]
<\/span><\/span>    
<\/span><\/span>    local<\/span> result =<\/span> nil<\/span>
<\/span><\/span>    if<\/span> check_level ==<\/span> 1<\/span> and<\/span> ngx.re.find(check_value, level_one,"oij"<\/span>) then<\/span>
<\/span><\/span>        result =<\/span> 1<\/span>
<\/span><\/span>    elseif<\/span> check_level ==<\/span> 2<\/span> and<\/span> ngx.re.find(check_value, level_two,"oij"<\/span>) then<\/span>
<\/span><\/span>        result =<\/span> 2<\/span>
<\/span><\/span>    elseif<\/span> check_level ==<\/span> 3<\/span> and<\/span> ngx.re.find(check_value, level_three,"oij"<\/span>) then<\/span>
<\/span><\/span>        result =<\/span> 3<\/span>
<\/span><\/span>    elseif<\/span> check_level ==<\/span> 4<\/span> and<\/span> ngx.re.find(check_value, level_four,"oij"<\/span>) then<\/span>
<\/span><\/span>        result =<\/span> 4<\/span>
<\/span><\/span>    elseif<\/span> type(check_level) ==<\/span> string and<\/span> ngx.re.find(check_value,check_level,"oij"<\/span>) then<\/span>
<\/span><\/span>        result =<\/span> 5<\/span>
<\/span><\/span>    else<\/span>
<\/span><\/span>    end<\/span>
<\/span><\/span>    return<\/span> result
<\/span><\/span>end<\/span>
<\/span><\/span><\/code><\/pre>2.3 长度分析<\/h3>
2.3.1 相关配置<\/h4>

Learn length bypass<\/code>: true\/false（检测到异常只告警不拦截）<\/li>
Learn length bypass length<\/code>: 数字（默认30，小于该值不处理）<\/li>
Learn length limit<\/code>: 数字（默认1500，参数值最大长度）<\/li>
Learn length update count<\/code>: 数字（默认5，长度参数迭代次数）<\/li>
<\/ul>
2.3.2 实现特点<\/h4>

新参数值长度比旧值大时会覆盖旧长度<\/li>
迭代次数超过限制时，长度值将被设置为Learn length limit<\/code><\/li>
<\/ul>
2.4 词法分析<\/h3>
2.4.1 核心处理函数<\/h4>
local<\/span> function<\/span> _process_string<\/span>(arg)
<\/span><\/span>    local<\/span> result =<\/span> {}
<\/span><\/span>    local<\/span> _arg =<\/span> arg
<\/span><\/span>    local<\/span> char =<\/span> nil<\/span>
<\/span><\/span>    for<\/span> count=<\/span>1<\/span>,#<\/span>_arg,1<\/span> do<\/span>
<\/span><\/span>        local<\/span> tmp =<\/span> string.byte(_arg,count)
<\/span><\/span>        if<\/span> (tmp >=<\/span> 0<\/span> and<\/span> tmp <=<\/span> 31<\/span>) or<\/span> tmp ==<\/span> 127<\/span> then<\/span>
<\/span><\/span>            -- control<\/span>
<\/span><\/span>            if<\/span> char ~=<\/span> 'c'<\/span> then<\/span>
<\/span><\/span>                table.insert(result,'c'<\/span>)
<\/span><\/span>                char =<\/span> 'c'<\/span>
<\/span><\/span>            end<\/span>
<\/span><\/span>        elseif<\/span> (tmp >=<\/span> 48<\/span> and<\/span> tmp <=<\/span> 57<\/span>) or<\/span> (tmp >=<\/span> 65<\/span> and<\/span> tmp <=<\/span> 90<\/span>) or<\/span> (tmp >=<\/span> 97<\/span> and<\/span> tmp <=<\/span> 122<\/span>) or<\/span> tmp ><\/span> 127<\/span> then<\/span>
<\/span><\/span>            -- normal<\/span>
<\/span><\/span>            if<\/span> char ~=<\/span> 'n'<\/span> then<\/span>
<\/span><\/span>                table.insert(result,'n'<\/span>)
<\/span><\/span>                char =<\/span> 'n'<\/span>
<\/span><\/span>            end<\/span>
<\/span><\/span>        elseif<\/span> (tmp >=<\/span> 32<\/span> and<\/span> tmp <=<\/span>34<\/span>) or<\/span> tmp ==<\/span> 40<\/span> or<\/span> tmp ==<\/span> 41<\/span> or<\/span> tmp ==<\/span> 44<\/span> or<\/span> tmp ==<\/span> 46<\/span> or<\/span> tmp ==<\/span> 64<\/span> or<\/span> tmp ==<\/span> 95<\/span> or<\/span> tmp ==<\/span> 63<\/span> then<\/span>
<\/span><\/span>            -- usual char (space)<\/span>
<\/span><\/span>            if<\/span> char ~=<\/span> 'u'<\/span> then<\/span>
<\/span><\/span>                table.insert(result,'u'<\/span>)
<\/span><\/span>                char =<\/span> 'u'<\/span>
<\/span><\/span>            end<\/span>
<\/span><\/span>        elseif<\/span> (tmp >=<\/span> 35<\/span> and<\/span> tmp <=<\/span> 39<\/span> ) or<\/span> tmp ==<\/span> 42<\/span> or<\/span> tmp ==<\/span> 43<\/span> or<\/span> tmp==<\/span> 45<\/span> or<\/span> tmp ==<\/span> 47<\/span> or<\/span> (tmp >=<\/span> 58<\/span> and<\/span> tmp <=<\/span> 62<\/span>) or<\/span> (tmp >=<\/span> 91<\/span> and<\/span> tmp <=<\/span>96<\/span>) or<\/span> (tmp >=<\/span> 123<\/span> and<\/span> tmp <=<\/span> 126<\/span>) then<\/span>
<\/span><\/span>            -- anomaly char<\/span>
<\/span><\/span>            if<\/span> char ~=<\/span> 'a'<\/span> then<\/span>
<\/span><\/span>                table.insert(result,'a'<\/span>)
<\/span><\/span>                char =<\/span> 'a'<\/span>
<\/span><\/span>            end<\/span>
<\/span><\/span>        else<\/span>
<\/span><\/span>            ngx.log(ngx.ERR,"process string error "<\/span>,_arg)
<\/span><\/span>        end<\/span>
<\/span><\/span>    end<\/span>
<\/span><\/span>    return<\/span> table.concat(result)
<\/span><\/span>end<\/span>
<\/span><\/span><\/code><\/pre>2.4.2 字符分类规则<\/h4>

控制字符（0~31及127）：标记为'c'<\/li>
数字、字母、中文等：标记为'n'<\/li>
空格等常见字符：标记为'u'<\/li>
其他特殊字符：标记为'a'<\/li>
<\/ol>
2.4.3 特征合并<\/h4>

相同特征合并为一个，如"aaaaaabbbbb哇哇哇哇"→"n"<\/li>
特征数超过5个时，检测进入bypass模式<\/li>
<\/ul>
三、全局配置<\/h2>
3.1 主要配置项<\/h3>

Force_reject<\/code>: true\/false（默认为false，未被学习到的参数和URL请求是否拒绝）<\/li>
Igonre uri<\/code>: 正则（Force_reject为true时，匹配该正则的URL请求放行）<\/li>
check_reject<\/code>: true\/false（默认为true，未能通过检测的请求是否拒绝）<\/li>
learn_count<\/code>: 数字（默认为1000，同一URL请求的学习迭代次数）<\/li>
<\/ul>
四、案例分析（以DVWA登录接口为例）<\/h2>
4.1 学习过程<\/h3>

初始学习数据示例：
"Login":[1,5,["n"],"false"]
"user_token":[1,32,["n"],"false"]
"username":[1,5,["n"],"false"]
"password":[1,8,["n"],"false"]
<\/code><\/pre>
<\/li>
<\/ul>
4.2 学习后数据变化<\/h3>

Login参数：固定值"Login"，学习数据不变<\/li>
user_token：32位十六进制数据，学习数据不变<\/li>
username参数：

正则自适应匹配值由1→3<\/li>
长度达到21<\/li>
词法特征增加"nunun"<\/li>
<\/ul>
<\/li>
password参数：

正则自适应匹配值由1→6（任意字符）<\/li>
长度维度失效<\/li>
词法分析维度失效<\/li>
<\/ul>
<\/li>
<\/ol>
4.3 攻击检测示例<\/h3>

SQL注入：' and 1=(select @@VERSION)<\/code>

正则匹配值：6<\/li>
长度：24<\/li>
词法特征："aununununu"<\/li>
结果：拒绝<\/li>
<\/ul>
<\/li>
命令执行：ls -al<\/code>

正则匹配值：3<\/li>
长度：6<\/li>
词法特征："nun"<\/li>
结果：拒绝<\/li>
<\/ul>
<\/li>
代码执行：phpinfo()<\/code>

正则匹配值：2<\/li>
长度：9<\/li>
词法特征："nu"<\/li>
结果：拒绝<\/li>
<\/ul>
<\/li>
<\/ol>
五、总结<\/h2>
5.1 解决的问题<\/h3>

模型收敛问题：通过增量更新（边检测边学习）<\/li>
误报问题：采用"宁放过不错杀"原则<\/li>
<\/ol>
5.2 引入的新问题<\/h3>

提高了用户介入度<\/li>
提高了漏报率<\/li>
<\/ol>
5.3 整体防护策略<\/h3>

黑名单+全局白名单+自学习+机器学习多层防护<\/li>
性能消耗低（相当于多1-2条规则）<\/li>
支持管理界面审计功能优化<\/li>
<\/ul>

WAF自学习模式开发实战教学文档<\/h1>

一、自学习模式概述<\/h2>

二、核心实现原理<\/h2>

2.2 正则自适应匹配<\/h3>

2.3 长度分析<\/h3>

2.4 词法分析<\/h3>

三、全局配置<\/h2>

四、案例分析（以DVWA登录接口为例）<\/h2>

五、总结<\/h2>

5.2 引入的新问题<\/h3> 提高了用户介入度<\/li> 提高了漏报率<\/li> <\/ol> 5.3 整体防护策略<\/h3> 黑名单+全局白名单+自学习+机器学习多层防护<\/li> 性能消耗低（相当于多1-2条规则）<\/li> 支持管理界面审计功能优化<\/li> <\/ul>

5.3 整体防护策略<\/h3> 黑名单+全局白名单+自学习+机器学习多层防护<\/li> 性能消耗低（相当于多1-2条规则）<\/li> 支持管理界面审计功能优化<\/li> <\/ul>

5.2 引入的新问题<\/h3>

提高了用户介入度<\/li>
提高了漏报率<\/li> <\/ol>
5.3 整体防护策略<\/h3>

黑名单+全局白名单+自学习+机器学习多层防护<\/li>
性能消耗低（相当于多1-2条规则）<\/li>
支持管理界面审计功能优化<\/li> <\/ul>

5.3 整体防护策略<\/h3>

黑名单+全局白名单+自学习+机器学习多层防护<\/li>
性能消耗低（相当于多1-2条规则）<\/li>
支持管理界面审计功能优化<\/li> <\/ul>