一起bvar中死锁的分析

Posted 2022-08-25 存储之厨

背景

fork的系统调用后，子进程会继承父进程的锁、内存、打开文件列表等资源，特别地对锁还会继承同样的状态。这样在多线程以及设计全局初始化顺序未定的场景，就可能造成死锁。

现象

在大名鼎鼎的bthread中，我们就碰到一起bvar中的死锁， 其现象如下：

dead lock anaylsis

很多bvar::Variable::dump_exposed 线程等待锁 0x234c1d0, 而 0x234c1d0被thread 5 (LWP 13756) 拥有

(gdb) p *(pthread_mutex_t *) 0x234c1d0
$2 = __data = __lock = 2, __count = 1, __owner = 13756, __nusers = 1, __kind = 1, __spins = 0, __list = __prev = 0x0, __next = 0x0,
  __size = "\\002\\000\\000\\000\\001\\000\\000\\000\\274\\065\\000\\000\\001\\000\\000\\000\\001", \\000 <repeats 22 times>, __align = 4294967298
(gdb) quit

而thread 5 (LWP 13756) 在等一把锁：

Thread 5 (Thread 0x7fbf2ffff700 (LWP 13756)):
#0  0x00007fbf8a5a051d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fbf8a59be1b in _L_lock_812 () from /lib64/libpthread.so.0
#2  0x00007fbf8a59bce8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000004d9d00 in pthread_mutex_lock (__mutex=0x2454d50) at /home/chenlei6/essd-demo/third-party/brpc/src/bthread/mutex.cpp:554
#4  0x00000000004ef0f9 in lock (this=<optimized out>) at /home/chenlei6/essd-demo/third-party/brpc/src/butil/synchronization/lock.h:69
#5  lock_guard (__m=..., this=<synthetic pointer>) at /usr/include/c++/4.8.2/mutex:414
#6  get_value (result=<synthetic pointer>, window_size=10, this=<optimized out>) at /home/chenlei6/essd-demo/third-party/brpc/src/bvar/detail/sampler.h:148
#7  get_span (this=<optimized out>, result=<synthetic pointer>, window_size=10) at /home/chenlei6/essd-demo/third-party/brpc/src/bvar/window.h:96
#8  bvar::PerSecond<bvar::PassiveStatus<unsigned long> >::get_value (this=<optimized out>, window_size=10) at /home/chenlei6/essd-demo/third-party/brpc/src/bvar/window.h:225
#9  0x00000000004ee5e6 in get_value (this=<optimized out>) at /home/chenlei6/essd-demo/third-party/brpc/src/bvar/window.h:111
#10 bvar::detail::WindowBase<bvar::PassiveStatus<unsigned long>, (bvar::SeriesFrequency)1>::describe (this=<optimized out>, os=..., quote_string=<optimized out>) at /home/chenlei6/essd-demo/third-party/brpc/src/bvar/window.h:117
#11 0x00000000004fd0d5 in bvar::Variable::describe_exposed (name="process_faults_minor_second", os=..., quote_string=<optimized out>, display_filter=bvar::DISPLAY_ON_html) at /home/chenlei6/essd-demo/third-party/brpc/src/bvar/variable.cpp:258

(__mutex=0x2454d50) 的owner是： 12793

$1 = (pthread_mutex_t *) 0x2454d50
(gdb) p *(pthread_mutex_t *) 0x2454d50
$2 = __data = __lock = 2, __count = 0, __owner = 12793, __nusers = 1, __kind = 0, __spins = 0, __list = __prev = 0x0, __next = 0x0, __size = "\\002\\000\\000\\000\\000\\000\\000\\000\\371\\061\\000\\000\\001", \\000 <repeats 26 times>,
  __align = 2

可是12793 已经结束了：info threads 看不到， 指看到一个12794的线程。

(gdb) info threads
  Id   Target Id         Frame
  23   Thread 0x7fbf8766a700 (LWP 12795) "spdk_vhost_essd" 0x00007fbf891fc56d in nanosleep () from /lib64/libc.so.6
  22   Thread 0x7fbf86e69700 (LWP 12796) "spdk_vhost_essd" 0x00007fbf891fc56d in nanosleep () from /lib64/libc.so.6
  21   Thread 0x7fbf86668700 (LWP 12797) "spdk_vhost_essd" 0x00007fbf8a59dd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  20   Thread 0x7fbf85e67700 (LWP 12927) "spdk_vhost_essd" 0x00007fbf8a59dd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  19   Thread 0x7fbf85666700 (LWP 12930) "eal-intr-thread" 0x00007fbf89236183 in epoll_wait () from /lib64/libc.so.6
  18   Thread 0x7fbf84e65700 (LWP 12931) "rte_mp_handle" 0x00007fbf8a5a0c4d in recvmsg () from /lib64/libpthread.so.0
  17   Thread 0x7fbf667fe700 (LWP 13442) "reactor_1" thread_poll (now=121104563516851692, max_msgs=0, thread=0x247d510) at thread.c:694
  16   Thread 0x7fbf65ffd700 (LWP 13443) "reactor_2" 0x00007fbf8de811a2 in rte_rdtsc () at /usr/src/debug/essd-20.04/dpdk/build/include/rte_cycles.h:46
  15   Thread 0x7fbf657fc700 (LWP 13444) "reactor_3" reactor_post_process_lw_thread (reactor=reactor@entry=0x247cf80, lw_thread=lw_thread@entry=0x2953e38) at reactor.c:645
  14   Thread 0x7fbf64ffb700 (LWP 13445) "spdk_vhost_essd" 0x00007fbf8a5a0a7d in accept () from /lib64/libpthread.so.0
  13   Thread 0x7fbf59343700 (LWP 13618) "async" 0x00007fbf89236183 in epoll_wait () from /lib64/libc.so.6
  12   Thread 0x7fbf4bfff700 (LWP 13748) "spdk_vhost_essd" 0x00007fbf8922fec9 in syscall () from /lib64/libc.so.6
  11   Thread 0x7fbf4b7fe700 (LWP 13749) "spdk_vhost_essd" 0x00007fbf8a5a051d in __lll_lock_wait () from /lib64/libpthread.so.0
  10   Thread 0x7fbf4affd700 (LWP 13750) "spdk_vhost_essd" 0x00007fbf8a5a051d in __lll_lock_wait () from /lib64/libpthread.so.0
  9    Thread 0x7fbf4a7fc700 (LWP 13752) "spdk_vhost_essd" 0x00007fbf8a5a051d in __lll_lock_wait () from /lib64/libpthread.so.0
  8    Thread 0x7fbf49ffb700 (LWP 13753) "spdk_vhost_essd" 0x00007fbf8a5a051d in __lll_lock_wait () from /lib64/libpthread.so.0
  7    Thread 0x7fbf497fa700 (LWP 13754) "spdk_vhost_essd" 0x00007fbf8a5a051d in __lll_lock_wait () from /lib64/libpthread.so.0
  6    Thread 0x7fbf48ff9700 (LWP 13755) "spdk_vhost_essd" 0x00007fbf8a5a051d in __lll_lock_wait () from /lib64/libpthread.so.0
* 5    Thread 0x7fbf2ffff700 (LWP 13756) "spdk_vhost_essd" 0x00007fbf8a5a051d in __lll_lock_wait () from /lib64/libpthread.so.0
  4    Thread 0x7fbf2f7fe700 (LWP 13757) "spdk_vhost_essd" 0x00007fbf8a5a051d in __lll_lock_wait () from /lib64/libpthread.so.0
  3    Thread 0x7fbf2effd700 (LWP 13758) "spdk_vhost_essd" 0x00007fbf8a5a051d in __lll_lock_wait () from /lib64/libpthread.so.0
  2    Thread 0x7fbf2e7fc700 (LWP 13761) "vhost-events" 0x00007fbf8a59fafb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  1    Thread 0x7fbfa9db0f80 (LWP 12794) "spdk_vhost_essd" 0x00007fbf8a5a051d in __lll_lock_wait () from /lib64/libpthread.so.0

为什么会出现这种情况呢？

解决办法

通过上面线程ID分析，可以看到：是有进程启动后创建了上面的线程5然后又退出了，但是整个业务进程还在，这个看起来是一个daemon形式的进程。和部署进程的业务同学确认了，果然是这样。那么该怎么解决呢？

非daemon形式执行

daemon进程中父进程显然已经退出了，而子进程还在，显然用到了fork. 既然父进程因为退出了导致没有释放锁，那么让它不退出，也即不运行daemon模式，就可以避免这个问题了。验证了一下，果然没有再出现。

调整父子进程执行顺序

但是业务方不同意这种模式，为此需要彻底解决这种问题。分析当前流程是：当前父子进程都开启了bvar的sampling thread, 开启sampling thread中会涉及到锁。如果子进程申请锁的时候，其从父进程继承的那把锁已经处于上锁状态就可能出现上面的现象。为此，可以先停一遍sampling thread, 再启动业务进程，fork完了后再开启sampling thread. 那么这就需要用到：

int pthread_atfork(void (*prepare)(void), void (*parent)(void),
                          void (*child)(void));

可以同用atfork注册一个钩子，在prepare里stop sampler，然后后两个钩子里再启动起来。