V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
xatest
V2EX  ›  Amazon Web Services

AWS的验尸报告~

  •  
  •   xatest ·
    aoxu · 2011-05-02 00:16:05 +08:00 · 5598 次点击
    这是一个创建于 4939 天前的主题,其中的信息可能已经有所发展或是发生改变。
    http://aws.amazon.com/message/65648/

    AWS这次严重的故障,影响到了Quora和Foursquare等多家客户,不知道有看了官方验尸报告的吗?
    我花了2个小时看完,整个事故的过程分析起来惊心动魄啊!
    6 条回复    1970-01-01 08:00:00 +08:00
    MarkFull
        1
    MarkFull  
       2011-05-02 06:11:40 +08:00
    ry_wang
        2
    ry_wang  
       2011-05-02 11:24:24 +08:00
    发现异常为啥不能及时回滚呢?其实我好奇这个
    xatest
        3
    xatest  
    OP
       2011-05-02 11:48:30 +08:00
    @ry_wang 他们错误的回滚策略造成了雪崩,引起更大范围的错误~
    ry_wang
        4
    ry_wang  
       2011-05-02 12:10:05 +08:00
    @xatest 哦?有详细点的说明么?官方文档看不懂呀
    xatest
        5
    xatest  
    OP
       2011-05-02 14:52:17 +08:00
    @ry_wang 这还不够详细?5700字了~
    Sidney
        6
    Sidney  
       2011-05-02 22:02:25 +08:00
    看着有种心惊肉跳的感觉

    两点感受
    1. 人为的操作容易出现错误。之前碰到过hosting的人员在配置Firewall的时候误操作,Firewall Hang住,导致网站无法访问。
    The traffic shift was executed incorrectly and rather than routing the traffic to the other router on the primary network, the traffic was routed onto the lower capacity redundant EBS network.

    2. 对回滚操作的后果估计不足,带来连锁雪崩效应。除非对系统各个模块和它们之间的交互有足够的理解,说不定就会踩中地雷。
    Incorrect traffic shift was rolled back and network connectivity was restored. This quickly led to a “re-mirroring storm,” where a large number of volumes were effectively “stuck” while the nodes searched the cluster for the storage space it needed for its new replica.
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   1057 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 23ms · UTC 19:36 · PVG 03:36 · LAX 11:36 · JFK 14:36
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.