[OpenSIPS-Users] B2BUA Segfault

Bogdan-Andrei Iancu bogdan at opensips.org
Mon May 27 15:27:17 CEST 2013


Hello Tolga,

Please test the attached patch.

Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com


On 05/24/2013 08:33 PM, Tolga Tarhan wrote:
> Sure, here you go:
>
> version: opensips 1.9.0-tls (x86_64/linux)
> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, DISABLE_NAGLE,
> USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC,
> FAST_LOCK-ADAPTIVE_WAIT
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> svnrevision: unknown
> @(#) $Id: main.c 9790 2013-02-15 10:14:34Z bogdan_iancu $
> main.c compiled on 12:39:26 May 22 2013 with gcc 4.4.7
>
> It also has the couple patches that I've previously submitted (and
> you've since merged) in the build (the three related to GRUU bugs).
> They aren't a factor, as this particular instance is not acting as a
> registrar at all.
>
> Thanks,
> Tolga
>
>
>
> On Fri, May 24, 2013 at 9:33 AM, Bogdan-Andrei Iancu
> <bogdan at opensips.org <mailto:bogdan at opensips.org>> wrote:
>
>     Hi Tolga,
>
>     Thanks for the info.
>
>     What exact OpenSIPs version/revision are you using ? I need to
>     correlate logs with sources.
>
>     Regards,
>
>     Bogdan-Andrei Iancu
>     OpenSIPS Founder and Developer
>     http://www.opensips-solutions.com
>
>
>     On 05/22/2013 11:02 PM, Tolga Tarhan wrote:
>>     Sorry for the self-reply -- here's the (same) stacktraces with
>>     line numbers and params:
>>
>>     #0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
>>     #1  0x0000003564c34085 in abort () from /lib64/libc.so.6
>>     #2  0x000000000049d370 in qm_free (qm=<value optimized out>,
>>     p=<value optimized out>, file=0x7ffccb25df64 "logic.c",
>>     func=<value optimized out>, line=755) at mem/q_malloc.c:450
>>     #3  0x00007ffccb253804 in process_bridge_200OK
>>     (msg=0x7ffcccdd2600, extra_headers=0xd5, body=<value optimized
>>     out>, tuple=0x7ffcc9518de0, entity=<value optimized out>) at
>>     logic.c:755
>>     #4  0x00007ffccb254ba2 in b2b_logic_notify_reply (src=<value
>>     optimized out>, msg=0x7ffcccdd2600, key=<value optimized out>,
>>     body=0x7fffc2ce15d0, extra_headers=0x7fffc2ce15c0,
>>     b2bl_key=0x7fffc2ce23f0, hash_index=649, local_index=1)
>>         at logic.c:1133
>>     #5  0x00007ffccb2565e1 in b2b_logic_notify (src=1,
>>     msg=0x7ffcccdd2600, key=0x7ffcc9525e40, type=1,
>>     param=0x7fffc2ce23f0) at logic.c:2040
>>     #6  0x00007ffccb47a7a7 in b2b_tm_cback (t=0x7ffcc951c110,
>>     htable=0x7ffcc94f38d0, ps=<value optimized out>) at dlg.c:2678
>>     #7  0x00007ffccc744b71 in run_trans_callbacks (type=256,
>>     trans=0x7ffcc951c110, req=<value optimized out>, rpl=<value
>>     optimized out>, code=<value optimized out>) at t_hooks.c:212
>>     #8  0x00007ffccc74fa12 in local_reply (t=0x7ffcc951c110,
>>     p_msg=<value optimized out>, branch=<value optimized out>,
>>     msg_status=<value optimized out>, cancel_bitmap=<value optimized
>>     out>) at t_reply.c:1391
>>     #9  0x00007ffccc750cc5 in reply_received (p_msg=0x7ffcccdd2600)
>>     at t_reply.c:1540
>>     #10 0x00000000004266da in forward_reply (msg=0x7ffcccdd2600) at
>>     forward.c:574
>>     #11 0x0000000000452ad8 in receive_msg (buf=<value optimized out>,
>>     len=<value optimized out>, rcv_info=0x7fffc2ce28c0) at receive.c:207
>>     #12 0x0000000000496c95 in udp_rcv_loop () at udp_server.c:424
>>     #13 0x000000000042d763 in main_loop (argc=<value optimized out>,
>>     argv=<value optimized out>) at main.c:884
>>     #14 main (argc=<value optimized out>, argv=<value optimized out>)
>>     at main.c:1557
>>
>>     #0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
>>     #1  0x0000003564c34085 in abort () from /lib64/libc.so.6
>>     #2  0x000000000049d370 in qm_free (qm=<value optimized out>,
>>     p=<value optimized out>, file=0x7ffccb262d75 "records.c",
>>     func=<value optimized out>, line=595) at mem/q_malloc.c:450
>>     #3  0x00007ffccb25a003 in b2bl_delete (tuple=0x7ffcc9518de0,
>>     hash_index=<value optimized out>, not_del_b2be=1) at records.c:595
>>     #4  0x00007ffccb25a3d5 in destroy_b2bl_htable () at records.c:705
>>     #5  0x000000000046dbf2 in destroy_modules () at sr_module.c:371
>>     #6  0x00000000004298a1 in cleanup (show_status=1) at main.c:348
>>     #7  0x000000000042a360 in handle_sigs () at main.c:549
>>     #8  0x000000000042db66 in main_loop (argc=<value optimized out>,
>>     argv=<value optimized out>) at main.c:1024
>>     #9  main (argc=<value optimized out>, argv=<value optimized out>)
>>     at main.c:1557
>>
>>     Thanks,
>>     Tolga
>>
>>
>>     On Wed, May 22, 2013 at 1:00 PM, Tolga Tarhan
>>     <tolga at netbrains.com <mailto:tolga at netbrains.com>> wrote:
>>
>>         Thank you -- I've recompiled and enabled the memory debug. I
>>         have the log file from the whole experience available here:
>>
>>         http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log
>>
>>         (note - real phone numbers and domain names in the log have
>>         been replaced with placeholders)
>>
>>         The key item seems to be:
>>
>>         CRITICAL:core:qm_free: freeing already freed pointer, first
>>         free: logic.c: process_bridge_200OK(740) - aborting
>>
>>         Although this appears to be after we've already decided we're
>>         going to crash, as I see "INFO:core:cleanup: cleanup" and
>>         "NFO:core:handle_sigs: child process 24788 exited by a signal
>>         6" above this part of the log.
>>
>>         Also worth noting is the existance of
>>         "CRITICAL:b2b_logic:b2bl_drop_entity: we should never end up
>>         here" throughout the log.
>>
>>         Also, here's the stack trace at crash time. Note that there
>>         were two core files generated for the same crash, so this is
>>         the backtrace from each:
>>
>>         #0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
>>         #1  0x0000003564c34085 in abort () from /lib64/libc.so.6
>>         #2  0x000000000049d370 in qm_free ()
>>         #3  0x00007ffccb253804 in process_bridge_200OK () from
>>         /usr/lib64/opensips/modules/b2b_logic.so
>>         #4  0x00007ffccb254ba2 in b2b_logic_notify_reply () from
>>         /usr/lib64/opensips/modules/b2b_logic.so
>>         #5  0x00007ffccb2565e1 in b2b_logic_notify () from
>>         /usr/lib64/opensips/modules/b2b_logic.so
>>         #6  0x00007ffccb47a7a7 in b2b_tm_cback () from
>>         /usr/lib64/opensips/modules/b2b_entities.so
>>         #7  0x00007ffccc744b71 in run_trans_callbacks () from
>>         /usr/lib64/opensips/modules/tm.so
>>         #8  0x00007ffccc74fa12 in local_reply () from
>>         /usr/lib64/opensips/modules/tm.so
>>         #9  0x00007ffccc750cc5 in reply_received () from
>>         /usr/lib64/opensips/modules/tm.so
>>         #10 0x00000000004266da in forward_reply ()
>>         #11 0x0000000000452ad8 in receive_msg ()
>>         #12 0x0000000000496c95 in udp_rcv_loop ()
>>         #13 0x000000000042d763 in main ()
>>
>>         #0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
>>         #1  0x0000003564c34085 in abort () from /lib64/libc.so.6
>>         #2  0x000000000049d370 in qm_free ()
>>         #3  0x00007ffccb25a003 in b2bl_delete () from
>>         /usr/lib64/opensips/modules/b2b_logic.so
>>         #4  0x00007ffccb25a3d5 in destroy_b2bl_htable () from
>>         /usr/lib64/opensips/modules/b2b_logic.so
>>         #5  0x000000000046dbf2 in destroy_modules ()
>>         #6  0x00000000004298a1 in cleanup ()
>>         #7  0x000000000042a360 in handle_sigs ()
>>         #8  0x000000000042db66 in main ()
>>
>>         I am unfamiliar with this codebase. Can anyone garner
>>         anything useful from the logs?
>>
>>         Thanks,
>>         Tolga
>>
>>
>>
>>         On Wed, May 22, 2013 at 9:02 AM, Bogdan-Andrei Iancu
>>         <bogdan at opensips.org <mailto:bogdan at opensips.org>> wrote:
>>
>>             Hello Tolga,
>>
>>             The crash seems to be in the memory manager, most
>>             probably because of memory corruption. To troubleshoot
>>             such issues you need to compile-in the memory debugger -
>>             see
>>             http://www.opensips.org/Documentation/TroubleShooting-OutOfMem
>>             .
>>
>>             Using memlog=1 + memdump=10 you should get a lot of logs
>>             related to mem ops, including a final report + abort()
>>             when the corruption is detected.
>>
>>             Regards,
>>
>>             Bogdan-Andrei Iancu
>>             OpenSIPS Founder and Developer
>>             http://www.opensips-solutions.com
>>
>>
>>             On 05/22/2013 01:29 AM, Tolga Tarhan wrote:
>>>             Hello,
>>>
>>>             While using the B2BUA module in OpenSIPS 1.9.0,
>>>             we've encountered a consistent segfault. We are using a
>>>             refer scenario just like the one in the B2BUA sample
>>>             docs, and after several REFERs for the same call (to
>>>             different destinations), OpenSIPS crashes with a
>>>             segfault. The core file indicates the following backtrace:
>>>
>>>             #0  0x000000000049a334 in fm_malloc ()
>>>             #1  0x00007fdaecd96230 in shm_malloc_unsafe
>>>             (type=B2B_CLIENT, entity_id=0x7fdaee8ec750,
>>>             to_uri=0x7fff2d346360, from_uri=0x7fff2d346320,
>>>             from_dname=0x0, ssid=<value optimized out>, msg=0x0) at
>>>             ../../mem/shm_mem.h:248
>>>             #2  shm_malloc (type=B2B_CLIENT,
>>>             entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360,
>>>             from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value
>>>             optimized out>, msg=0x0) at ../../mem/shm_mem.h:258
>>>             #3  b2bl_create_new_entity (type=B2B_CLIENT,
>>>             entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360,
>>>             from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value
>>>             optimized out>, msg=0x0) at logic.c:293
>>>             #4  0x00007fdaecd96882 in b2bl_new_client (to_uri=<value
>>>             optimized out>, from_uri=<value optimized out>,
>>>             tuple=<value optimized out>, ssid=0x7fdaeb026c00,
>>>             msg=<value optimized out>) at logic.c:607
>>>             #5  0x00007fdaecda3579 in process_bridge_200OK
>>>             (msg=0x7fdaee8e8b30, extra_headers=0x7fdaeb03d578,
>>>             body=<value optimized out>, tuple=0x7fdaeb01ada8,
>>>             entity=<value optimized out>) at logic.c:816
>>>             #6  0x00007fdaecda46c2 in b2b_logic_notify_reply
>>>             (src=<value optimized out>, msg=0x7fdaee8e8b30,
>>>             key=<value optimized out>, body=0x7fff2d3468b0,
>>>             extra_headers=0x7fff2d3468a0, b2bl_key=0x7fff2d3476d0,
>>>             hash_index=649, local_index=0)
>>>                 at logic.c:1133
>>>             #7  0x00007fdaecda6081 in b2b_logic_notify (src=1,
>>>             msg=0x7fdaee8e8b30, key=0x7fdaeb03d500, type=1,
>>>             param=0x7fff2d3476d0) at logic.c:2040
>>>             #8  0x00007fdaecfca7ad in b2b_tm_cback
>>>             (t=0x7fdaeb054118, htable=0x7fdaeb014630, ps=<value
>>>             optimized out>) at dlg.c:2678
>>>             #9  0x00007fdaee291441 in run_trans_callbacks (type=256,
>>>             trans=0x7fdaeb054118, req=<value optimized out>,
>>>             rpl=<value optimized out>, code=<value optimized out>)
>>>             at t_hooks.c:212
>>>             #10 0x00007fdaee29c0e2 in local_reply (t=0x7fdaeb054118,
>>>             p_msg=<value optimized out>, branch=<value optimized
>>>             out>, msg_status=<value optimized out>,
>>>             cancel_bitmap=<value optimized out>) at t_reply.c:1391
>>>             #11 0x00007fdaee29d31d in reply_received
>>>             (p_msg=0x7fdaee8e8b30) at t_reply.c:1540
>>>             #12 0x000000000042625a in forward_reply ()
>>>             #13 0x0000000000451c28 in receive_msg ()
>>>             #14 0x0000000000494e45 in udp_rcv_loop ()
>>>             #15 0x000000000042d1a3 in main ()
>>>
>>>             I'm not really sure how to diagnose this one. Any
>>>             hints/fixes/suggestions would be very appreciated.
>>>
>>>             Thanks,
>>>             Tolga
>>>
>>>
>>>             _______________________________________________
>>>             Users mailing list
>>>             Users at lists.opensips.org <mailto:Users at lists.opensips.org>
>>>             http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20130527/6383f46d/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: b2b_logic_2free.patch
Type: text/x-patch
Size: 422 bytes
Desc: not available
URL: <http://lists.opensips.org/pipermail/users/attachments/20130527/6383f46d/attachment-0001.bin>


More information about the Users mailing list