V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
Distributions
Ubuntu
Fedora
CentOS
中文资源站
网易开源镜像站
firejoke
V2EX  ›  Linux

Linux 的 ReusePort 特性

  •  
  •   firejoke · 2020-11-26 10:31:10 +08:00 · 2872 次点击
    这是一个创建于 1462 天前的主题,其中的信息可能已经有所发展或是发生改变。

    最近在看 Django-channels, 因为自带的 daphne asgi 服务器本身没有提供多 worker 的工作方式,
    而文档内推荐的部署方案是用 supervisor 做进程管理,
    但考虑到本来系统上就有 systemd, 所以尝试用 systmed 来做进程管理,
    用 systemd.socket 监听指定端口, 在收到连接后, 启用对应的服务, 并把 socket 传递给服务,
    先贴一下单个进程的 dc.socket 和 dc.service 文件

    # dc.socket
    [Unit]
    Description= Django Channels socket
    
    [Socket]
    ListenStream=0.0.0.0:9001
    
    [Install]
    WantedBy=sockets.target
    
    # dc.service
    [Unit]
    Description=Django Channels
    Requires=veops1.socket
    
    [Service]
    Type=simple
    WorkingDirectory=/project_path/
    NonBlocking=true
    ExecStart=/project_path/virtenv/bin/daphne -e systemd:domain=INET:index=0 \
    -s DC --access-log /var/log/dc/daphne_access1.log dc.asgi:application
    ExecReload=/bin/kill -HUP $MAINPID
    KillSignal=SIGQUIT
    KillMode = control-group
    Restart=on-failure
    RestartSec=10s
    User=root
    Group=root
    
    
    [Install]
    WantedBy=multi-user.target
    

    这样只能启用一个进程, 如果要像 supervisor 那样多个进程监听一个端口, 需要用到从 Linux kernel 3.9 开始增加的 SO_REUSEPORT 特性, 把之前的实例配置变成模板配置 [email protected][email protected] 文件转变成模板, 像这样

    # [email protected]
    [Unit]
    Description= Django Channels %i socket
    
    [Socket]
    ListenStream=0.0.0.0:9001
    ReusePort=true
    Service=dc@%i.service
    
    [Install]
    WantedBy=sockets.target
    
    # [email protected]
    [Unit]
    Description= Django Channels %i
    After=dc@%i.socket
    
    [Service]
    Type=simple
    WorkingDirectory=/project_path/
    ExecStart=/project_path/virtenv/bin/daphne -e systemd:domain=INET:index=0 \
    -s DC --access-log /var/log/dc/daphne%i.log dc.asgi:application
    NonBlocking=yes
    ExecReload=/bin/kill -HUP $MAINPID
    KillMode = control-group
    Restart=on-failure
    RestartSec=10s
    User=root
    
    [Install]
    WantedBy=multi-user.target
    


    systemctl start dc@0
    systemctl start dc@1
    systemctl start dc@2
    ...
    可以生成多个 service 实例并监听同一个端口
    所有访问这个端口的连接都会被分发给各个进程

    测试环境:
    system-release: CentOS Linux release 7.9.2009 (Core)
    uname: 3.10.0-1160.6.1.el7.x86_64
    启用第二个 socket 的时候报错了

    failed to listen on sockets: Address already in use
    

    看起来 ReusePort 并没有生效
    但放在 Centos8, kernel 4.19 上却可以生效,
    怀疑是系统有这个特性, 但 systemd 没有启用, 搜了一番也没搜到
    有在 3.10 上用 systemd 使用过这个特性的吗?
    希望能告诉我一下是不是要改什么配置

    11 条回复    2020-12-02 21:21:06 +08:00
    warcraft1236
        1
    warcraft1236  
       2020-11-26 10:40:28 +08:00
    升级吧,现在内核都 5 开头了
    50infivedays
        2
    50infivedays  
       2020-11-26 11:11:54 +08:00
    3.10 可以用的 应该是配置错了 要设置 socket option 的
    boboliu
        3
    boboliu  
       2020-11-26 11:17:42 +08:00
    systemd 对 reuseport 引入还挺早了,v206 就有,centos7 的包还没这么老

    是不是没关干净不 reuse 的
    firejoke
        4
    firejoke  
    OP
       2020-11-26 11:37:38 +08:00
    @boboliu #3 没有, 这是个测试环境, 也检查了端口占用
    firejoke
        5
    firejoke  
    OP
       2020-11-26 11:39:16 +08:00
    @50infivedays #2 你说的 socket option 是指 service 文件内的吗,
    我单独启 dc@0. socket [email protected] 也不行
    julyclyde
        6
    julyclyde  
       2020-11-26 16:34:39 +08:00
    systemd 传递的 socket,是 listen socket 还是 accepted FD 呢?
    firejoke
        7
    firejoke  
    OP
       2020-11-26 20:43:34 +08:00
    @julyclyde #6 传递的应该是 fd
    https://www.freedesktop.org/software/systemd/man/systemd.socket.html#
    ```
    Note that the daemon software configured for socket activation with socket units needs to be able to accept sockets from systemd, either via systemd's native socket passing interface (see sd_listen_fds(3) for details about the precise protocol used and the order in which the file descriptors are passed) or via traditional inetd(8)-style socket passing (i.e. sockets passed in via standard input and output, using StandardInput=socket in the service file).
    ```
    julyclyde
        8
    julyclyde  
       2020-11-27 17:43:44 +08:00
    原来两种都行啊?
    前者似乎挺高级的,看起来是需要特地编程才能支持的功能
    见过 docker.service 和 docker.socket 就是这种关系

    后者 inetd-style 的话其实就是 stdio 模式了,应用程序自己并不负责多进程管理的,每个 accept 就给一个单独的进程处理
    firejoke
        9
    firejoke  
    OP
       2020-11-27 22:53:28 +08:00
    @julyclyde #8 但我的问题还是没找到原因, 同样的 sokcet 配置, 在 centos8 kernel 4.x 上可以, centos7 kernel 3.10 就报错了...
    目前唯一发现的区别就是
    /usr/include/asm-generic/socket.h 里面和 REUSEPORT 有关的, 在 centos8 多了几个
    tomychen
        10
    tomychen  
       2020-12-02 17:57:51 +08:00
    SO_REUSEPORT (since Linux 3.9)
    Permits multiple AF_INET or AF_INET6 sockets to be bound to an identical socket address. This option must be set on each socket (including the first socket) prior to calling bind(2) on the
    socket. To prevent port hijacking, all of the processes binding to the same address must have the same effective UID. This option can be employed with both TCP and UDP sockets.

    For TCP sockets, this option allows accept(2) load distribution in a multi-threaded server to be improved by using a distinct listener socket for each thread. This provides improved load dis‐
    tribution as compared to traditional techniques such using a single accept(2)ing thread that distributes connections, or having multiple threads that compete to accept(2) from the same socket.

    For UDP sockets, the use of this option can provide better distribution of incoming datagrams to multiple processes (or threads) as compared to the traditional technique of having multiple
    processes compete to receive datagrams on the same socket.


    ```c
    #include <stdio.h>
    #include <sys/socket.h>
    #include <sys/types.h>
    #include <string.h>
    #include <netinet/in.h>
    #include <unistd.h>

    int main(void)
    {
    int sock = -1;
    int flags = 1;
    int ret;
    struct sockaddr_in sa;

    sock = socket(AF_INET, SOCK_STREAM, 0);
    if (sock == -1) {
    perror("socket");
    return 1;
    }
    if (setsockopt(sock, SOL_SOCKET, SO_REUSEPORT, &flags, sizeof(int)) <
    0) {
    perror("setsockopt");
    return 1;
    }

    memset(&sa, 0, sizeof(sa));

    sa.sin_addr.s_addr = htonl(INADDR_ANY);
    sa.sin_family = AF_INET;
    sa.sin_port = htons(10086);

    ret = bind(sock, (struct sockaddr *)&sa, sizeof(sa));
    if (ret == -1) {
    perror("bind()");
    return 1;
    }

    ret = listen(sock, 10);
    if (ret == -1) {
    perror("listen()");
    return 1;
    }
    while (1) {
    printf("listen\n");
    sleep(2);
    }
    }
    ```
    uname -a
    Linux localhost.localdomain 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

    代码工作正常,所以和系统无关
    firejoke
        11
    firejoke  
    OP
       2020-12-02 21:21:06 +08:00
    @tomychen #10 嗯, 我也用程序试了的, 系统支持, 现在怀疑是 systemd 版本太老了
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   5992 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 29ms · UTC 06:22 · PVG 14:22 · LAX 22:22 · JFK 01:22
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.