YisouSpider 完全不遵守 robots 啊,一次来一组 IP,一组 IP 有 5 个,轮着来! KAO,太无耻了! 加了一组 IP 到 Deny,改天又来新的一组 IP !
1
mydns 2017-12-04 17:54:35 +08:00
防火墙列表里全是 yisou 和神马搜索 这两个的 IP 爬起来和 cc 一样~~~
|
2
Had 2017-12-04 17:55:38 +08:00
直接按 UA 过滤呗...
|
3
tumbzzc 2017-12-04 18:00:42 +08:00 2
你没见过更无耻的 MJ12bot 吧
|
6
huaxing0211 OP @tumbzzc 88.198.16.12 - - [04/Dec/2017:04:06:01 +0800] "GET /robots.txt HTTP/1.1" 200 1485 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)"
还守规矩啊,爬了 robots 就悄悄的走了,没其它痕迹…… |
7
flyingghost 2017-12-04 18:20:09 +08:00
@huaxing0211 #6 于是换个 UA 再来。/机智
|
8
qdcanyun 2017-12-04 18:29:53 +08:00
是的。。。他们的爬虫就像 cc 攻击一样,超级烦
|
9
JoeyChan 2017-12-04 18:36:25 +08:00
吓得我赶紧查了一下,还好已经在黑名单了,return 444。
|
10
mydns 2017-12-04 18:48:58 +08:00 1
YisouSpider
MJ12bot AhrefsBot DotBot SemrushBot 以上就是这几年屏蔽的搜索引擎列表 robots 都无效 用防火墙封 ip 段解决的 每次一上站 cpu 就 100% 而且持续一整天 |
11
mydns 2017-12-04 18:52:04 +08:00
补充 IP 段:
42.156.254.37-42.156.254.42 103.47.210.1-103.47.210.254 42.156.136.103 42.120.161.103 42.120.160.103 42.156.139.103 42.156.138.103 42.156.137.103 180.97.106.162 180.97.106.161 180.97.106.37 42.120.161.72 42.156.138.72 42.120.160.72 42.156.137.72 42.156.136.72 42.156.139.72 5.9.6.51 69.30.198.178 193.70.39.162 163.172.68.136 72.90.76.89 173.234.159.250 144.76.29.66 93.219.71.27 163.172.32.175 136.243.152.18 144.76.7.106 178.203.146.96 193.70.37.50 42.156.137.107 42.120.160.107 42.156.138.107 42.156.139.107 213.251.184.38 42.156.139.46 42.120.161.107 91.209.51.22 42.156.138.46 42.156.136.46 42.156.137.46 62.138.8.181 42.156.139.106 42.120.161.106 42.120.160.106 42.156.137.106 42.156.136.106 42.120.160.83 42.156.138.83 42.156.136.83 42.156.254.47 42.156.136.108 42.156.137.108 42.156.138.108 42.120.160.108 42.120.161.108 42.156.139.108 216.244.66.247 51.255.65.88 137.74.201.99 164.132.161.7 164.132.161.8 51.255.71.118 164.132.161.59 182.118.33.6 175.2.133.236 175.15.110.180 175.15.118.111 |
12
huaxing0211 OP @mydns
42.156.139.62 42.156.138.62 42.156.137.96 42.156.137.62 42.156.136.62 42.120.160.62 42.156.136.33 42.156.139.33 42.120.161.33 42.156.137.33 42.156.138.33 106.11.152.186 106.11.156.169 106.11.157.173 106.11.155.161 106.11.154.169 106.11.153.181 106.11.159.197 106.11.158.214 106.11.152.198 106.11.153.204 106.11.157.220 106.11.156.193 106.11.157.148 106.11.154.153 106.11.158.139 106.11.155.158 106.11.152.134 106.11.159.138 |
13
ryd994 2017-12-05 00:29:24 +08:00 via Android 1
屏蔽 UA
说实话,换我可能直接屏蔽所有阿里云 IP 段了 其次优化性能+限制频率,这么爬两下就挂了,那傻逼用户按住 F5 也能挂。 |
14
opengps 2017-12-05 10:37:23 +08:00
完全同意 13 楼说法,如果这点压力你都排斥,那么你的业务得多不稳定
|
15
mytsing520 2017-12-06 00:46:18 +08:00
基本上干掉了。不是不稳定,而是他读起来无底线
|
16
ty5yidc 2020-02-22 04:06:50 +08:00
竟然 POST 我的数据,见鬼了,ajax 的数据直接给 POST 了,还好数据都是加密的
|