如何在 pthread_crate.c 文件中找到我的哪些代码导致 SIGABRT，但 coredump 中没有我的任何代码信息

Posted 2023-02-14

【中文标题】如何在 pthread_crate.c 文件中找到我的哪些代码导致 SIGABRT，但 coredump 中没有我的任何代码信息

【英文标题】：How can I find which of my code cause the SIGABRT in pthread_crate.c file, but without any my code information in the coredump

【发布时间】：2022-01-11 08:57:17

【问题描述】：

我的程序导致了一个核心转储，如下所示：

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `./remote_speaker plug:SLAVE='dmix:tlv320aic3106au' default rtmp://pili-publish.'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0xffff802791d0 (LWP 1511))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000ffff812b9f54 in __GI_abort () at abort.c:79
#2  0x0000ffff81304d3c in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0xffff813bf638 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x0000ffff8130c32c in malloc_printerr (str=str@entry=0xffff813bafb0 "free(): invalid pointer") at malloc.c:5332
#4  0x0000ffff8130db04 in _int_free (av=0xffff813fb9f8 <main_arena>, p=0xffff7d783ff0, have_lock=<optimized out>) at malloc.c:4173
#5  0x0000ffff81310b50 in tcache_thread_shutdown () at malloc.c:2964
#6  __malloc_arena_thread_freeres () at arena.c:949
#7  0x0000ffff81313e8c in __libc_thread_freeres () at thread-freeres.c:38
#8  0x0000ffff81614844 in start_thread (arg=0xffffc2f50ff6) at pthread_create.c:493
#9  0x0000ffff81365a7c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

总线程信息如下所示：

(gdb) info threads 
  Id   Target Id                        Frame 
* 1    Thread 0xffff802791d0 (LWP 1511) __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
  2    Thread 0xffff80a7a1d0 (LWP 1510) 0x0000ffff8135d084 in __GI___poll (fds=0xffff80a79740, nfds=1, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  3    Thread 0xffff8127c010 (LWP 1492) 0x0000ffff81339d88 in __GI___nanosleep (requested_time=requested_time@entry=0xffffc2f510a0, remaining=remaining@entry=0x0)
    at ../sysdeps/unix/sysv/linux/nanosleep.c:28
  4    Thread 0xffff7e2751d0 (LWP 1515) 0x0000ffff81365bcc in __GI_epoll_pwait (epfd=6, events=0x594160 <self+128>, maxevents=32, timeout=100, set=0x0)
    at ../sysdeps/unix/sysv/linux/epoll_pwait.c:42
  5    Thread 0xffff8127b1d0 (LWP 1509) futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x596538 <output_notice>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:208
  6    Thread 0xffff7fa781d0 (LWP 1512) 0x0000ffff81339d84 in __GI___nanosleep (requested_time=requested_time@entry=0xffff7fa77870, remaining=remaining@entry=0xffff7fa77870)
    at ../sysdeps/unix/sysv/linux/nanosleep.c:28

线程 1-6

(gdb) thread 1
[Switching to thread 1 (Thread 0xffff802791d0 (LWP 1511))]
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      in ../sysdeps/unix/sysv/linux/raise.c
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000ffff812b9f54 in __GI_abort () at abort.c:79
#2  0x0000ffff81304d3c in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0xffff813bf638 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x0000ffff8130c32c in malloc_printerr (str=str@entry=0xffff813bafb0 "free(): invalid pointer") at malloc.c:5332
#4  0x0000ffff8130db04 in _int_free (av=0xffff813fb9f8 <main_arena>, p=0xffff7d783ff0, have_lock=<optimized out>) at malloc.c:4173
#5  0x0000ffff81310b50 in tcache_thread_shutdown () at malloc.c:2964
#6  __malloc_arena_thread_freeres () at arena.c:949
#7  0x0000ffff81313e8c in __libc_thread_freeres () at thread-freeres.c:38
#8  0x0000ffff81614844 in start_thread (arg=0xffffc2f50ff6) at pthread_create.c:493
#9  0x0000ffff81365a7c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb) thread 2
[Switching to thread 2 (Thread 0xffff80a7a1d0 (LWP 1510))]
#0  0x0000ffff8135d084 in __GI___poll (fds=0xffff80a79740, nfds=1, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
41      ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.
(gdb) bt
#0  0x0000ffff8135d084 in __GI___poll (fds=0xffff80a79740, nfds=1, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x00000000004e8714 in nn_efd_wait ()
#2  0x00000000004e46a0 in nn_sock_recv ()
#3  0x00000000004e24b0 in nn_recvmsg ()
#4  0x00000000004e1ef4 in nn_recv ()
#5  0x0000000000445c2c in nanomsg_recv ()
#6  0x0000ffff816148f8 in start_thread (arg=0xffffc2f50ff6) at pthread_create.c:479
#7  0x0000ffff81365a7c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb) thread 3
[Switching to thread 3 (Thread 0xffff8127c010 (LWP 1492))]
#0  0x0000ffff81339d88 in __GI___nanosleep (requested_time=requested_time@entry=0xffffc2f510a0, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
28      ../sysdeps/unix/sysv/linux/nanosleep.c: No such file or directory.
(gdb) bt
#0  0x0000ffff81339d88 in __GI___nanosleep (requested_time=requested_time@entry=0xffffc2f510a0, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x0000ffff8135fb54 in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:32
#2  0x0000000000434278 in main ()
(gdb) thread 4
[Switching to thread 4 (Thread 0xffff7e2751d0 (LWP 1515))]
#0  0x0000ffff81365bcc in __GI_epoll_pwait (epfd=6, events=0x594160 <self+128>, maxevents=32, timeout=100, set=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:42
42      ../sysdeps/unix/sysv/linux/epoll_pwait.c: No such file or directory.
(gdb) bt
#0  0x0000ffff81365bcc in __GI_epoll_pwait (epfd=6, events=0x594160 <self+128>, maxevents=32, timeout=100, set=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:42
#1  0x00000000004f2a14 in nn_poller_wait ()
#2  0x00000000004e712c in nn_worker_routine ()
#3  0x00000000004e9eb8 in nn_thread_main_routine ()
#4  0x0000ffff816148f8 in start_thread (arg=0xffff80a796c6) at pthread_create.c:479
#5  0x0000ffff81365a7c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb) thread 5
[Switching to thread 5 (Thread 0xffff8127b1d0 (LWP 1509))]
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x596538 <output_notice>) at ../sysdeps/unix/sysv/linux/futex-internal.h:208
208     ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
(gdb) bt
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x596538 <output_notice>) at ../sysdeps/unix/sysv/linux/futex-internal.h:208
#1  do_futex_wait (sem=sem@entry=0x596538 <output_notice>, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x0000ffff8161dd10 in __new_sem_wait_slow (sem=0x596538 <output_notice>, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x0000000000529aa8 in async_output ()
#4  0x0000ffff816148f8 in start_thread (arg=0xffffc2f50fa6) at pthread_create.c:479
#5  0x0000ffff81365a7c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb) thread 6
[Switching to thread 6 (Thread 0xffff7fa781d0 (LWP 1512))]
#0  0x0000ffff81339d84 in __GI___nanosleep (requested_time=requested_time@entry=0xffff7fa77870, remaining=remaining@entry=0xffff7fa77870) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
28      ../sysdeps/unix/sysv/linux/nanosleep.c: No such file or directory.
(gdb) bt
#0  0x0000ffff81339d84 in __GI___nanosleep (requested_time=requested_time@entry=0xffff7fa77870, remaining=remaining@entry=0xffff7fa77870) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x0000ffff81339c14 in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55
#2  0x0000000000434ee0 in play_sound ()
#3  0x0000ffff816148f8 in start_thread (arg=0xffffc2f50ff6) at pthread_create.c:479
#4  0x0000ffff81365a7c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

很抱歉我不能发布源代码，因为代码对我们来说有点重要。

问题是我找不到我的哪个代码导致了这个转储。 在 pthread_create 函数之后的线程 1 中没有任何我的代码。正常情况下，线程中肯定有我的一些代码，是吗？

非常感谢您的帮助。

【问题讨论】：

我应该附上这个现象不是 Appearing-Certain。

【参考方案1】：

问题是我找不到我的哪个代码导致了这个转储

malloc 或 free 内的任何崩溃都是 99.9% 的堆损坏迹象（freeing 两次，freeing 未分配内存，写入超过分配缓冲区的末尾等）。

这里有free() 告诉你free正在处理未分配的东西。地址为freed:0xffff7d783ff0 看起来像一个堆栈地址。 很可能你之前有freed 一些堆栈地址。

不幸的是，通过事后调试来调试堆损坏几乎是不可能的，因为损坏的根本原因可能发生在 1000 条指令之前，可能在完全不相关的代码中。

好消息：使用地址清理程序 (gcc -fsanitize=address ...) 检测您的程序并通过您的测试运行此类程序（您确实有测试，对吗？）通常会直接导致问题。

【讨论】：

是的，很多人和网上论坛都和你一样。可能是其他错误导致了此崩溃。而且我已经在我的代码上测试了asan工具，但是运行时没有任何错误。这让我秃了。