[OpenSIPS-Users] Opensips(1.11.5) Crash - " #0 0x00007fcd7223ae2c in vfprintf () from /lib64/libc.so.6"

Ravitez Ravi ravitez.dondeti at gmail.com
Fri Jan 27 15:14:11 EST 2017


Hi All,
        Good Day,
        OpenSips(1.11.5) is constantly crashing at " vfprintf ()"
        Logs are filled up with the following error message
              2017-01-24T18:14:02.000-05:00 [local2] [err] E01Z01.dev.com
/usr/sbin/opensips[4473]: ERROR:core:io_watch_del: trying to delete already
erased entry 46 in the hash(0, 0, (nil)) )
              2017-01-24T18:14:02.000-05:00 [local2] [err] E01Z01.dev.com
/usr/sbin/opensips[4473]: ERROR:core:tcp_read_req: bad request, state=0,
error=2 buf:

        We see the following in tcp_connection list :
                Connection::  ID=4029 Type=*tcp*
*State=18446744073709551615* Source=172.29.106.95:5060 Destination=
172.29.106.10:5060 Timeout=2017-01-25 19:35:35 Pending lifetime=0
        not sure if its a valid state.
        Here's the debug log :
        [20758]: DBG:core:tcp_read_req: - received from: port 5060
        [20758]: DBG:core:print_ip: - received from: ip *172.29.106.95*
        [20758]: DBG:core:io_watch_del: io_watch_del op on index -1 52
(0x8100e0, 52, -1, 0x10,0x3) fd_no=2 called
        [20758]: ERROR:core:io_watch_del: trying to delete already erased
entry 52 in the hash(0, 0, (nil)) )
        [20758]: DBG:core:release_tcpconn:  releasing con 0x7f543179d700,
state -2, fd=52, id=4029
        [20758]: DBG:core:release_tcpconn:  extra_data (nil)
        [20758]: DBG:core:tcp_read_req: Using the global ( per process )
buff
        [20758]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf

         Please find the attached Stack Trace and logs generated.
         Not sure if this is already address in a different version of
OpenSips or a know issue,please share your thoughts.
         Thank you for your time.

Attached Log File Contains Following Sections :
        Line 1: ###### Core 1  Time : 2017-01-24T18:04 ######
Line 174: ###### Log Information ######
Line 175: ###### Before Crash ######
Line 188: 2017-01-24T18:04:29.000-05:00 [local2] [err] E01Z01.dev.com
/usr/sbin/opensips[31607]: ERROR:core:tls_shutdown: something wrong in SSL:
       #### Crash ############
Line 190: ###### After Crash ######
Line 215: ###### Continuous Errors ######
Line 235: ######   Second Crash ######
Line 249: ######  Core 2  Time : 2017-01-25T09:08 ######
Line 421: ##### Core 3 Time : 2017-01-25T09:08 #####
Line 619: ##### Core 4 Time : 2017-01-25T09:08 #####
Line 820: ##### Core 5 Time : 2017-01-25T19:35 #####
Line 1011: ##### Additional Logs ##### --> This section contains TCP
Connection list and Debug information from Opensips


Regards,
Ravitez.D
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170127/f10ae2de/attachment-0001.html>
-------------- next part --------------
###### Core 1  Time : 2017-01-24T18:04 ######
gdb /usr/sbin/opensips core.31643
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/opensips...done.

warning: core file may not match specified executable file.
[New Thread 31643]
Missing separate debuginfo for /usr/lib64/opensips/modules/signaling.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a5/6fe5f3fb5f5a5945cd0fd8384403a1610ac723
Missing separate debuginfo for /usr/lib64/opensips/modules/sl.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a0/3aa3c43c41ef6b5a3443edee6b57557f99e95f
Missing separate debuginfo for /usr/lib64/opensips/modules/tm.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/86/ec031a2c632fc2c1c2ebc807d134588c879e8f
Missing separate debuginfo for /usr/lib64/opensips/modules/rr.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/f9/5a55b1e8ec652178bbf56f1de12a130f2d3fc2
Missing separate debuginfo for /usr/lib64/opensips/modules/maxfwd.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/3c/bcedbe5e2cb56ce0232c168ce10d01549154e9
Missing separate debuginfo for /usr/lib64/opensips/modules/sipmsgops.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/68/3d8e3c5d0e47bb4e69df2e47f62b8a6c2b02e1
Missing separate debuginfo for /usr/lib64/opensips/modules/mi_fifo.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/25/0c6e832f8702623763334c66d80701abd2f757
Missing separate debuginfo for /usr/lib64/opensips/modules/uri.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6c/8c7108c432cb039e548473b9c190265cb21c3c
Missing separate debuginfo for /usr/lib64/opensips/modules/usrloc.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7a/3f6b2b6c1c5e4d9143db3610667472c5b1f1fb
Missing separate debuginfo for /usr/lib64/opensips/modules/nathelper.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/fe/eb381245a35d7cc8504f30496cbce200bc2a32
Missing separate debuginfo for /usr/lib64/opensips/modules/registrar.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e1/e152c5eb09765db719d2de687674d4b38bec29
Missing separate debuginfo for /usr/lib64/opensips/modules/acc.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/0d/53590fb94de20cbeb1aee103e38332cdb6783c
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7f/719af91ee951b4fcb6647e7868f95f766a616b
Reading symbols from /lib64/snoopy.so...done.
Loaded symbols for /lib64/snoopy.so
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /usr/lib64/opensips/modules/signaling.so...done.
Loaded symbols for /usr/lib64/opensips/modules/signaling.so
Reading symbols from /usr/lib64/opensips/modules/sl.so...done.
Loaded symbols for /usr/lib64/opensips/modules/sl.so
Reading symbols from /usr/lib64/opensips/modules/tm.so...done.
Loaded symbols for /usr/lib64/opensips/modules/tm.so
Reading symbols from /usr/lib64/opensips/modules/rr.so...done.
Loaded symbols for /usr/lib64/opensips/modules/rr.so
Reading symbols from /usr/lib64/opensips/modules/maxfwd.so...done.
Loaded symbols for /usr/lib64/opensips/modules/maxfwd.so
Reading symbols from /usr/lib64/opensips/modules/sipmsgops.so...done.
Loaded symbols for /usr/lib64/opensips/modules/sipmsgops.so
Reading symbols from /usr/lib64/opensips/modules/mi_fifo.so...done.
Loaded symbols for /usr/lib64/opensips/modules/mi_fifo.so
Reading symbols from /usr/lib64/opensips/modules/uri.so...done.
Loaded symbols for /usr/lib64/opensips/modules/uri.so
Reading symbols from /usr/lib64/opensips/modules/usrloc.so...done.
Loaded symbols for /usr/lib64/opensips/modules/usrloc.so
Reading symbols from /usr/lib64/opensips/modules/nathelper.so...done.
Loaded symbols for /usr/lib64/opensips/modules/nathelper.so
Reading symbols from /usr/lib64/opensips/modules/registrar.so...done.
Loaded symbols for /usr/lib64/opensips/modules/registrar.so
Reading symbols from /usr/lib64/opensips/modules/acc.so...done.
Loaded symbols for /usr/lib64/opensips/modules/acc.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/usr/sbin/opensips -u root -g root -P /var/run/opensips.pid -f'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fcd7223ae2c in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6_7.7.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-42z1.el6_7.x86_64 libcom_err-1.41.12-18.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 openssl-1.0.1e-48.el6_8.1.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt full
#0  0x00007fcd7223ae2c in vfprintf () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007fcd722d7ea0 in __vsyslog_chk () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007fcd722d80d0 in syslog () from /lib64/libc.so.6
No symbol table info available.
#3  0x00000000004d1ab8 in tcp_read_req (con=0x7fcd4e9e9ec8, bytes_read=0x7ffe3a05943c) at tcp_read.c:597
        bytes = <value optimized out>
        total_bytes = <value optimized out>
        resp = 0
        size = <value optimized out>
        req = 0x7fcd6fefeef0
        c = <value optimized out>
        local_rcv = {src_ip = {af = 973443984, len = 32766, u = {addrl = {1, 140729871864688}, addr32 = {1, 0, 973443952, 32766}, addr16 = {1, 0, 0, 0, 37744, 14853, 32766, 0},
              addr = "\001\000\000\000\000\000\000\000p\223\005:\376\177\000"}}, dst_ip = {af = 24, len = 0, u = {addrl = {0, 0}, addr32 = {0, 0, 0, 0}, addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, addr = '\000' <repeats 15 times>}},
          src_port = 20, dst_port = 0, proto = 0, proto_reserved1 = 1, proto_reserved2 = 1, src_su = {s = {sa_family = 45, sa_data = "\000\000\254\035j_\000\000\000\000\000\000\000"}, sin = {sin_family = 45, sin_port = 0, sin_addr = {
                s_addr = 1600789932}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 45, sin6_port = 0, sin6_flowinfo = 1600789932, sin6_addr = {__in6_u = {
                  __u6_addr8 = "\000\000\000\000\000\000\000\000\001\000\000\000X\376\344o", __u6_addr16 = {0, 0, 0, 0, 1, 0, 65112, 28644}, __u6_addr32 = {0, 0, 1, 1877278296}}}, sin6_scope_id = 32717}}, bind_address = 0x7fcd6fe39398}
        msg_buf = <value optimized out>
        msg_len = <value optimized out>
        __FUNCTION__ = "tcp_read_req"
#4  0x00000000004d31e8 in handle_io (fm=<value optimized out>, idx=-1, event_type=<value optimized out>) at tcp_read.c:1033
        ret = 0
        n = <value optimized out>
        con = 0x7fcd4e9e9ec8
        s = <value optimized out>
        rw = <value optimized out>
        resp = <value optimized out>
        response = {140519764041416, 1}
        __FUNCTION__ = "handle_io"
#5  0x00000000004d5985 in io_wait_loop_epoll (unix_sock=<value optimized out>) at io_wait.h:845
        n = 1
        r = <value optimized out>
#6  tcp_receive_loop (unix_sock=<value optimized out>) at tcp_read.c:1141
        __FUNCTION__ = "tcp_receive_loop"
#7  0x00000000004bb82c in tcp_init_children (chd_rank=<value optimized out>, startup_done=0x0) at tcp_main.c:2390
        r = <value optimized out>
        reader_fd = {43, 44}
        pid = <value optimized out>
        si = <value optimized out>
        load_p = 0x7fcd4e565bb0
        __FUNCTION__ = "tcp_init_children"
#8  0x000000000043d6b8 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:1011
        i = <value optimized out>
        pid = <value optimized out>
        si = 0x0
        startup_done = 0x0
        chd_rank = 33
        rc = <value optimized out>
        load_p = 0x7fcd4e572ed0
#9  main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1612
        cfg_log_stderr = <value optimized out>
        cfg_stream = <value optimized out>
        c = <value optimized out>
        r = <value optimized out>
        tmp = 0x7ffe3a05af40 "172.29.106.10"
        tmp_len = <value optimized out>
        port = <value optimized out>
        proto = <value optimized out>
        options = 0x595880 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
        ret = -1
        seed = 2270273504
        rfd = <value optimized out>
        __FUNCTION__ = "main"
(gdb)



###### Log Information ######
###### Before Crash ######
2017-01-24T17:45:32.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31644]: ERROR:core:tcpconn_async_connect: Internal error encountered when connecting
2017-01-24T17:45:32.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31644]: ERROR:core:tcp_send: async TCP connect failed
2017-01-24T17:45:32.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31644]: ERROR:core:msg_send: tcp_send failed
2017-01-24T17:45:32.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31644]: ERROR:core:async_connect_or_pass: poll error: flags 1c
2017-01-24T17:45:32.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31644]: ERROR:core:async_connect_or_pass: failed to retrieve SO_ERROR [server=172.29.106.10:56606] (111) Connection refused
2017-01-24T17:45:32.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31644]: ERROR:core:tcpconn_async_connect: Internal error encountered when connecting
2017-01-24T17:45:32.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31644]: ERROR:core:tcp_send: async TCP connect failed
2017-01-24T17:45:32.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31644]: ERROR:core:msg_send: tcp_send failed
2017-01-24T18:04:29.000-05:00 [local2] [crit] E01Z01.dev.com /usr/sbin/opensips[31675]: CRITICAL:core:receive_fd: EOF on 45         
2017-01-24T18:04:29.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31607]: ERROR:core:tls_shutdown: something wrong in SSL:
2017-01-24T18:04:29.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31607]: ERROR:core:tls_shutdown: something wrong in SSL:
2017-01-24T18:04:29.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31607]: ERROR:core:tls_shutdown: something wrong in SSL:
2017-01-24T18:04:29.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[31607]: ERROR:core:tls_shutdown: something wrong in SSL:        #### Crash ############

###### After Crash ###### 
2017-01-24T18:04:30.000-05:00 [local2] [warning] E01Z01.dev.com opensips[4415]: WARNING:core:init_tls: disabling compression due ZLIB problems
2017-01-24T18:04:30.000-05:00 [local2] [notice] E01Z01.dev.com opensips[4415]: NOTICE:core:init_ssl_ctx_behavior: No EC curve defined
2017-01-24T18:04:30.000-05:00 [local2] [notice] E01Z01.dev.com opensips[4415]: NOTICE:core:init_ssl_ctx_behavior: cipher list set to ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:AES128-SHA256:EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:!ECDHE-RSA-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:AES128-SHA:!DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4
2017-01-24T18:04:30.000-05:00 [local2] [warning] E01Z01.dev.com opensips[4415]: WARNING:core:init_ssl_ctx_behavior: client verification NOT activated. Weaker security.
2017-01-24T18:04:30.000-05:00 [local2] [notice] E01Z01.dev.com opensips[4415]: NOTICE:core:init_tls_domains: no CA for tls[0.0.0.0:0] defined, using default '/etc/pki/CA/'
2017-01-24T18:04:30.000-05:00 [local2] [notice] E01Z01.dev.com opensips[4415]: NOTICE:core:init_ssl_ctx_behavior: No EC curve defined
2017-01-24T18:04:30.000-05:00 [local2] [notice] E01Z01.dev.com opensips[4415]: NOTICE:core:init_ssl_ctx_behavior: cipher list set to ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:AES128-SHA256:EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:!ECDHE-RSA-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:AES128-SHA:!DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4
2017-01-24T18:04:30.000-05:00 [local2] [warning] E01Z01.dev.com opensips[4415]: WARNING:core:init_ssl_ctx_behavior: server verification activated.
2017-01-24T18:04:30.000-05:00 [local2] [notice] E01Z01.dev.com opensips[4415]: NOTICE:core:init_tls_domains: no CA for tls[0.0.0.0:0] defined, using default '/etc/pki/CA/'
2017-01-24T18:04:30.000-05:00 [local2] [notice] E01Z01.dev.com /usr/sbin/opensips[4432]: NOTICE:core:main: version: opensips 1.11.5-tls (x86_64/linux)
2017-01-24T18:04:30.000-05:00 [local2] [notice] E01Z01.dev.com /usr/sbin/opensips[4432]: NOTICE:signaling:mod_init: initializing module ...
2017-01-24T18:04:32.000-05:00 [local2] [notice] E01Z01.dev.com /usr/sbin/opensips[4445]: NOTICE:core:verify_callback: depth = 1
2017-01-24T18:04:32.000-05:00 [local2] [notice] E01Z01.dev.com /usr/sbin/opensips[4445]: NOTICE:core:verify_callback: preverify is good: verify return: 1
2017-01-24T18:04:32.000-05:00 [local2] [notice] E01Z01.dev.com /usr/sbin/opensips[4445]: NOTICE:core:verify_callback: depth = 0
2017-01-24T18:04:32.000-05:00 [local2] [notice] E01Z01.dev.com /usr/sbin/opensips[4445]: NOTICE:core:verify_callback: preverify is good: verify return: 1
2017-01-24T18:05:16.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:add_write_chunk: We have reached the limit of max async postponed chunks
2017-01-24T18:05:16.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:async_tsend_stream: Failed to add write chunk to connection
2017-01-24T18:05:16.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:tcp_send: failed to send
2017-01-24T18:05:16.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:msg_send: tcp_send failed
2017-01-24T18:05:54.000-05:00 [local2] [notice] E01Z01.dev.com /usr/sbin/opensips[4465]: NOTICE:core:verify_callback: depth = 1
2017-01-24T18:05:54.000-05:00 [local2] [notice] E01Z01.dev.com /usr/sbin/opensips[4465]: NOTICE:core:verify_callback: preverify is good: verify return: 1
2017-01-24T18:05:54.000-05:00 [local2] [notice] E01Z01.dev.com /usr/sbin/opensips[4465]: NOTICE:core:verify_callback: depth = 0
2017-01-24T18:05:54.000-05:00 [local2] [notice] E01Z01.dev.com /usr/sbin/opensips[4465]: NOTICE:core:verify_callback: preverify is good: verify return: 1

###### Continuous Errors ######
2017-01-24T18:14:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:io_watch_del: trying to delete already erased entry 46 in the hash(0, 0, (nil)) )
2017-01-24T18:14:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf:

parsed:
2017-01-24T18:14:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:io_watch_del: trying to delete already erased entry 46 in the hash(0, 0, (nil)) )
2017-01-24T18:14:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf:

parsed:
2017-01-24T18:14:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:io_watch_del: trying to delete already erased entry 46 in the hash(0, 0, (nil)) )
2017-01-24T18:14:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf:

parsed:
2017-01-24T23:52:03.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:io_watch_del: trying to delete already erased entry 46 in the hash(0, 0, (nil)) )
2017-01-24T23:52:03.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf:

parsed:
2017-01-24T23:52:03.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:io_watch_del: trying to delete already erased entry 46 in the hash(0, 0, (nil)) )
2017-01-24T23:52:03.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf:

######   Second Crash ######
parsed:
2017-01-25T09:06:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:io_watch_del: trying to delete already erased entry 46 in the hash(0, 0, (nil)) )
2017-01-25T09:06:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf:

parsed:
2017-01-25T09:06:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:io_watch_del: trying to delete already erased entry 46 in the hash(0, 0, (nil)) )
2017-01-25T09:06:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4473]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf:

parsed:
2017-01-25T09:08:28.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4432]: ERROR:core:tls_shutdown: something wrong in SSL:
2017-01-25T09:08:30.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4432]: ERROR:core:tls_shutdown: something wrong in SSL:
2017-01-25T09:08:30.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[4432]: ERROR:core:tls_shutdown: something wrong in SSL:

######  Core 2  Time : 2017-01-25T09:08 ######
gdb /usr/sbin/opensips core.4475
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/opensips...done.

warning: core file may not match specified executable file.
[New Thread 4475]
Missing separate debuginfo for /usr/lib64/opensips/modules/signaling.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a5/6fe5f3fb5f5a5945cd0fd8384403a1610ac723
Missing separate debuginfo for /usr/lib64/opensips/modules/sl.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a0/3aa3c43c41ef6b5a3443edee6b57557f99e95f
Missing separate debuginfo for /usr/lib64/opensips/modules/tm.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/86/ec031a2c632fc2c1c2ebc807d134588c879e8f
Missing separate debuginfo for /usr/lib64/opensips/modules/rr.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/f9/5a55b1e8ec652178bbf56f1de12a130f2d3fc2
Missing separate debuginfo for /usr/lib64/opensips/modules/maxfwd.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/3c/bcedbe5e2cb56ce0232c168ce10d01549154e9
Missing separate debuginfo for /usr/lib64/opensips/modules/sipmsgops.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/68/3d8e3c5d0e47bb4e69df2e47f62b8a6c2b02e1
Missing separate debuginfo for /usr/lib64/opensips/modules/mi_fifo.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/25/0c6e832f8702623763334c66d80701abd2f757
Missing separate debuginfo for /usr/lib64/opensips/modules/uri.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6c/8c7108c432cb039e548473b9c190265cb21c3c
Missing separate debuginfo for /usr/lib64/opensips/modules/usrloc.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7a/3f6b2b6c1c5e4d9143db3610667472c5b1f1fb
Missing separate debuginfo for /usr/lib64/opensips/modules/nathelper.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/fe/eb381245a35d7cc8504f30496cbce200bc2a32
Missing separate debuginfo for /usr/lib64/opensips/modules/registrar.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e1/e152c5eb09765db719d2de687674d4b38bec29
Missing separate debuginfo for /usr/lib64/opensips/modules/acc.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/0d/53590fb94de20cbeb1aee103e38332cdb6783c
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7f/719af91ee951b4fcb6647e7868f95f766a616b
Reading symbols from /lib64/snoopy.so...done.
Loaded symbols for /lib64/snoopy.so
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /usr/lib64/opensips/modules/signaling.so...done.
Loaded symbols for /usr/lib64/opensips/modules/signaling.so
Reading symbols from /usr/lib64/opensips/modules/sl.so...done.
Loaded symbols for /usr/lib64/opensips/modules/sl.so
Reading symbols from /usr/lib64/opensips/modules/tm.so...done.
Loaded symbols for /usr/lib64/opensips/modules/tm.so
Reading symbols from /usr/lib64/opensips/modules/rr.so...done.
Loaded symbols for /usr/lib64/opensips/modules/rr.so
Reading symbols from /usr/lib64/opensips/modules/maxfwd.so...done.
Loaded symbols for /usr/lib64/opensips/modules/maxfwd.so
Reading symbols from /usr/lib64/opensips/modules/sipmsgops.so...done.
Loaded symbols for /usr/lib64/opensips/modules/sipmsgops.so
Reading symbols from /usr/lib64/opensips/modules/mi_fifo.so...done.
Loaded symbols for /usr/lib64/opensips/modules/mi_fifo.so
Reading symbols from /usr/lib64/opensips/modules/uri.so...done.
Loaded symbols for /usr/lib64/opensips/modules/uri.so
Reading symbols from /usr/lib64/opensips/modules/usrloc.so...done.
Loaded symbols for /usr/lib64/opensips/modules/usrloc.so
Reading symbols from /usr/lib64/opensips/modules/nathelper.so...done.
Loaded symbols for /usr/lib64/opensips/modules/nathelper.so
Reading symbols from /usr/lib64/opensips/modules/registrar.so...done.
Loaded symbols for /usr/lib64/opensips/modules/registrar.so
Reading symbols from /usr/lib64/opensips/modules/acc.so...done.
Loaded symbols for /usr/lib64/opensips/modules/acc.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/usr/sbin/opensips -u root -g root -P /var/run/opensips.pid -f'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f555a002e2c in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6_7.7.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-42z1.el6_7.x86_64 libcom_err-1.41.12-18.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 openssl-1.0.1e-48.el6_8.1.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt full
#0  0x00007f555a002e2c in vfprintf () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f555a09fea0 in __vsyslog_chk () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007f555a0a00d0 in syslog () from /lib64/libc.so.6
No symbol table info available.
#3  0x00000000004d1ab8 in tcp_read_req (con=0x7f55363848a8, bytes_read=0x7ffc1ba7eedc) at tcp_read.c:597
        bytes = <value optimized out>
        total_bytes = <value optimized out>
        resp = 0
        size = <value optimized out>
        req = 0x7f5557cb9278
        c = <value optimized out>
        local_rcv = {src_ip = {af = 463990320, len = 32764, u = {addrl = {1, 140720772476432}, addr32 = {1, 0, 463990288, 32764}, addr16 = {1, 0, 0, 0, 60944, 7079, 32764, 0},
              addr = "\001\000\000\000\000\000\000\000\020\356\247\033\374\177\000"}}, dst_ip = {af = 24, len = 0, u = {addrl = {0, 0}, addr32 = {0, 0, 0, 0}, addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, addr = '\000' <repeats 15 times>}},
          src_port = 20, dst_port = 0, proto = 0, proto_reserved1 = 1, proto_reserved2 = 1, src_su = {s = {sa_family = 52, sa_data = "\000\000\254\035j_\000\000\000\000\000\000\000"}, sin = {sin_family = 52, sin_port = 0, sin_addr = {
                s_addr = 1600789932}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 52, sin6_port = 0, sin6_flowinfo = 1600789932, sin6_addr = {__in6_u = {
                  __u6_addr8 = "\000\000\000\000\000\000\000\000\001\000\000\000\000\177\301W", __u6_addr16 = {0, 0, 0, 0, 1, 0, 32512, 22465}, __u6_addr32 = {0, 0, 1, 1472298752}}}, sin6_scope_id = 32597}},
          bind_address = 0x7f5557c01398}
        msg_buf = <value optimized out>
        msg_len = <value optimized out>
        __FUNCTION__ = "tcp_read_req"
#4  0x00000000004d31e8 in handle_io (fm=<value optimized out>, idx=-1, event_type=<value optimized out>) at tcp_read.c:1033
        ret = 0
        n = <value optimized out>
        con = 0x7f55363848a8
        s = <value optimized out>
        rw = <value optimized out>
        resp = <value optimized out>
        response = {140003958605992, 1}
        __FUNCTION__ = "handle_io"
#5  0x00000000004d5985 in io_wait_loop_epoll (unix_sock=<value optimized out>) at io_wait.h:845
        n = 1
        r = <value optimized out>
#6  tcp_receive_loop (unix_sock=<value optimized out>) at tcp_read.c:1141
        __FUNCTION__ = "tcp_receive_loop"
#7  0x00000000004bb82c in tcp_init_children (chd_rank=<value optimized out>, startup_done=0x0) at tcp_main.c:2390
        r = <value optimized out>
        reader_fd = {49, 51}
        pid = <value optimized out>
        si = <value optimized out>
        load_p = 0x7f553632dbb0
        __FUNCTION__ = "tcp_init_children"
#8  0x000000000043d6b8 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:1011
        i = <value optimized out>
        pid = <value optimized out>
        si = 0x0
        startup_done = 0x0
        chd_rank = 36
        rc = <value optimized out>
        load_p = 0x7f553633aed0
#9  main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1612
        cfg_log_stderr = <value optimized out>
        cfg_stream = <value optimized out>
        c = <value optimized out>
        r = <value optimized out>
        tmp = 0x7ffc1ba80f40 "172.29.106.10"
        tmp_len = <value optimized out>
        port = <value optimized out>
        proto = <value optimized out>
        options = 0x595880 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
        ret = -1
        seed = 3019838257
        rfd = <value optimized out>
        __FUNCTION__ = "main"
(gdb)

##### Core 3 Time : 2017-01-25T09:08 #####
gdb /usr/sbin/opensips core.4474
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/opensips...done.

warning: core file may not match specified executable file.
[New Thread 4474]
Missing separate debuginfo for /usr/lib64/opensips/modules/signaling.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a5/6fe5f3fb5f5a5945cd0fd8384403a1610ac723
Missing separate debuginfo for /usr/lib64/opensips/modules/sl.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a0/3aa3c43c41ef6b5a3443edee6b57557f99e95f
Missing separate debuginfo for /usr/lib64/opensips/modules/tm.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/86/ec031a2c632fc2c1c2ebc807d134588c879e8f
Missing separate debuginfo for /usr/lib64/opensips/modules/rr.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/f9/5a55b1e8ec652178bbf56f1de12a130f2d3fc2
Missing separate debuginfo for /usr/lib64/opensips/modules/maxfwd.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/3c/bcedbe5e2cb56ce0232c168ce10d01549154e9
Missing separate debuginfo for /usr/lib64/opensips/modules/sipmsgops.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/68/3d8e3c5d0e47bb4e69df2e47f62b8a6c2b02e1
Missing separate debuginfo for /usr/lib64/opensips/modules/mi_fifo.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/25/0c6e832f8702623763334c66d80701abd2f757
Missing separate debuginfo for /usr/lib64/opensips/modules/uri.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6c/8c7108c432cb039e548473b9c190265cb21c3c
Missing separate debuginfo for /usr/lib64/opensips/modules/usrloc.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7a/3f6b2b6c1c5e4d9143db3610667472c5b1f1fb
Missing separate debuginfo for /usr/lib64/opensips/modules/nathelper.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/fe/eb381245a35d7cc8504f30496cbce200bc2a32
Missing separate debuginfo for /usr/lib64/opensips/modules/registrar.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e1/e152c5eb09765db719d2de687674d4b38bec29
Missing separate debuginfo for /usr/lib64/opensips/modules/acc.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/0d/53590fb94de20cbeb1aee103e38332cdb6783c
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7f/719af91ee951b4fcb6647e7868f95f766a616b
Reading symbols from /lib64/snoopy.so...done.
Loaded symbols for /lib64/snoopy.so
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /usr/lib64/opensips/modules/signaling.so...done.
Loaded symbols for /usr/lib64/opensips/modules/signaling.so
Reading symbols from /usr/lib64/opensips/modules/sl.so...done.
Loaded symbols for /usr/lib64/opensips/modules/sl.so
Reading symbols from /usr/lib64/opensips/modules/tm.so...done.
Loaded symbols for /usr/lib64/opensips/modules/tm.so
Reading symbols from /usr/lib64/opensips/modules/rr.so...done.
Loaded symbols for /usr/lib64/opensips/modules/rr.so
Reading symbols from /usr/lib64/opensips/modules/maxfwd.so...done.
Loaded symbols for /usr/lib64/opensips/modules/maxfwd.so
Reading symbols from /usr/lib64/opensips/modules/sipmsgops.so...done.
Loaded symbols for /usr/lib64/opensips/modules/sipmsgops.so
Reading symbols from /usr/lib64/opensips/modules/mi_fifo.so...done.
Loaded symbols for /usr/lib64/opensips/modules/mi_fifo.so
Reading symbols from /usr/lib64/opensips/modules/uri.so...done.
Loaded symbols for /usr/lib64/opensips/modules/uri.so
Reading symbols from /usr/lib64/opensips/modules/usrloc.so...done.
Loaded symbols for /usr/lib64/opensips/modules/usrloc.so
Reading symbols from /usr/lib64/opensips/modules/nathelper.so...done.
Loaded symbols for /usr/lib64/opensips/modules/nathelper.so
Reading symbols from /usr/lib64/opensips/modules/registrar.so...done.
Loaded symbols for /usr/lib64/opensips/modules/registrar.so
Reading symbols from /usr/lib64/opensips/modules/acc.so...done.
Loaded symbols for /usr/lib64/opensips/modules/acc.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/usr/sbin/opensips -u root -g root -P /var/run/opensips.pid -f'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f555a02fcea in _IO_cleanup () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6_7.7.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-42z1.el6_7.x86_64 libcom_err-1.41.12-18.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 openssl-1.0.1e-48.el6_8.1.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt full
#0  0x00007f555a02fcea in _IO_cleanup () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f5559ff0ac2 in exit () from /lib64/libc.so.6
No symbol table info available.
#2  0x000000000043df19 in sig_usr (signo=<value optimized out>) at main.c:631
        __FUNCTION__ = "sig_usr"
#3  <signal handler called>
No symbol table info available.
#4  0x00007f555a096560 in __write_nocancel () from /lib64/libc.so.6
No symbol table info available.
#5  0x00007f555a02cad3 in _IO_new_file_write () from /lib64/libc.so.6
No symbol table info available.
#6  0x00007f555a02e085 in _IO_new_do_write () from /lib64/libc.so.6
No symbol table info available.
#7  0x00007f555a02d3df in _IO_new_file_overflow () from /lib64/libc.so.6
No symbol table info available.
#8  0x00007f555a02f639 in _IO_default_xsputn_internal () from /lib64/libc.so.6
No symbol table info available.
#9  0x00007f555a02c792 in _IO_new_file_xsputn () from /lib64/libc.so.6
No symbol table info available.
#10 0x00007f555a003190 in vfprintf () from /lib64/libc.so.6
No symbol table info available.
#11 0x00007f555a02a550 in vdprintf () from /lib64/libc.so.6
No symbol table info available.
#12 0x00007f555a00a3d8 in dprintf () from /lib64/libc.so.6
No symbol table info available.
#13 0x00007f555a09fe1b in __vsyslog_chk () from /lib64/libc.so.6
No symbol table info available.
#14 0x00007f555a0a00d0 in syslog () from /lib64/libc.so.6
No symbol table info available.
#15 0x00000000004d1ab8 in tcp_read_req (con=0x7f55363848a8, bytes_read=0x7ffc1ba7eedc) at tcp_read.c:597
        bytes = <value optimized out>
        total_bytes = <value optimized out>
        resp = 0
        size = <value optimized out>
        req = 0x8449a0
        c = <value optimized out>
        local_rcv = {src_ip = {af = 909658280, len = 32597, u = {addrl = {18446744073709551614, 49}, addr32 = {4294967294, 4294967295, 49, 0}, addr16 = {65534, 65535, 65535, 65535, 49, 0, 0, 0},
              addr = "\376\377\377\377\377\377\377\377\061\000\000\000\000\000\000"}}, dst_ip = {af = 1, len = 0, u = {addrl = {1, 5048363}, addr32 = {1, 0, 5048363, 0}, addr16 = {1, 0, 0, 0, 2091, 77, 0, 0},
              addr = "\001\000\000\000\000\000\000\000+\bM\000\000\000\000"}}, src_port = 5060, dst_port = 5060, proto = 2, proto_reserved1 = 5046786, proto_reserved2 = 0, src_su = {s = {sa_family = 56296,
              sa_data = "36U\177\000\000\000\000\000\000\000\000\000"}, sin = {sin_family = 56296, sin_port = 13875, sin_addr = {s_addr = 32597}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 56296,
              sin6_port = 13875, sin6_flowinfo = 32597, sin6_addr = {__in6_u = {__u6_addr8 = "\000\000\000\000\000\000\000\000\250H86U\177\000", __u6_addr16 = {0, 0, 0, 0, 18600, 13880, 32597, 0}, __u6_addr32 = {0, 0, 909658280,
                    32597}}}, sin6_scope_id = 4294967294}}, bind_address = 0x7f5557c01398}
        msg_buf = <value optimized out>
        msg_len = <value optimized out>
        __FUNCTION__ = "tcp_read_req"
#16 0x00000000004d31e8 in handle_io (fm=<value optimized out>, idx=-1, event_type=<value optimized out>) at tcp_read.c:1033
        ret = 0
        n = <value optimized out>
        con = 0x7f55363848a8
        s = <value optimized out>
        rw = <value optimized out>
        resp = <value optimized out>
        response = {140003958605992, 1}
        __FUNCTION__ = "handle_io"
#17 0x00000000004d5985 in io_wait_loop_epoll (unix_sock=<value optimized out>) at io_wait.h:845
        n = 1
        r = <value optimized out>
#18 tcp_receive_loop (unix_sock=<value optimized out>) at tcp_read.c:1141
        __FUNCTION__ = "tcp_receive_loop"
#19 0x00000000004bb82c in tcp_init_children (chd_rank=<value optimized out>, startup_done=0x0) at tcp_main.c:2390
        r = <value optimized out>
        reader_fd = {47, 49}
        pid = <value optimized out>
        si = <value optimized out>
        load_p = 0x7f553632dbb0
        __FUNCTION__ = "tcp_init_children"
#20 0x000000000043d6b8 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:1011
        i = <value optimized out>
        pid = <value optimized out>
---Type <return> to continue, or q <return> to quit---
        si = 0x0
        startup_done = 0x0
        chd_rank = 35
        rc = <value optimized out>

        load_p = 0x7f553633aed0
#21 main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1612
        cfg_log_stderr = <value optimized out>
        cfg_stream = <value optimized out>
        c = <value optimized out>
        r = <value optimized out>
        tmp = 0x7ffc1ba80f40 "172.29.106.10"
        tmp_len = <value optimized out>
        port = <value optimized out>
        proto = <value optimized out>
        options = 0x595880 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
        ret = -1
        seed = 3019838257
        rfd = <value optimized out>
        __FUNCTION__ = "main"
(gdb)

##### Core 4 Time : 2017-01-25T09:08 #####
gdb /usr/sbin/opensips core.4473
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/opensips...done.

warning: core file may not match specified executable file.
[New Thread 4473]
Missing separate debuginfo for /usr/lib64/opensips/modules/signaling.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a5/6fe5f3fb5f5a5945cd0fd8384403a1610ac723
Missing separate debuginfo for /usr/lib64/opensips/modules/sl.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a0/3aa3c43c41ef6b5a3443edee6b57557f99e95f
Missing separate debuginfo for /usr/lib64/opensips/modules/tm.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/86/ec031a2c632fc2c1c2ebc807d134588c879e8f
Missing separate debuginfo for /usr/lib64/opensips/modules/rr.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/f9/5a55b1e8ec652178bbf56f1de12a130f2d3fc2
Missing separate debuginfo for /usr/lib64/opensips/modules/maxfwd.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/3c/bcedbe5e2cb56ce0232c168ce10d01549154e9
Missing separate debuginfo for /usr/lib64/opensips/modules/sipmsgops.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/68/3d8e3c5d0e47bb4e69df2e47f62b8a6c2b02e1
Missing separate debuginfo for /usr/lib64/opensips/modules/mi_fifo.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/25/0c6e832f8702623763334c66d80701abd2f757
Missing separate debuginfo for /usr/lib64/opensips/modules/uri.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6c/8c7108c432cb039e548473b9c190265cb21c3c
Missing separate debuginfo for /usr/lib64/opensips/modules/usrloc.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7a/3f6b2b6c1c5e4d9143db3610667472c5b1f1fb
Missing separate debuginfo for /usr/lib64/opensips/modules/nathelper.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/fe/eb381245a35d7cc8504f30496cbce200bc2a32
Missing separate debuginfo for /usr/lib64/opensips/modules/registrar.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e1/e152c5eb09765db719d2de687674d4b38bec29
Missing separate debuginfo for /usr/lib64/opensips/modules/acc.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/0d/53590fb94de20cbeb1aee103e38332cdb6783c
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7f/719af91ee951b4fcb6647e7868f95f766a616b
Reading symbols from /lib64/snoopy.so...done.
Loaded symbols for /lib64/snoopy.so
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /usr/lib64/opensips/modules/signaling.so...done.
Loaded symbols for /usr/lib64/opensips/modules/signaling.so
Reading symbols from /usr/lib64/opensips/modules/sl.so...done.
Loaded symbols for /usr/lib64/opensips/modules/sl.so
Reading symbols from /usr/lib64/opensips/modules/tm.so...done.
Loaded symbols for /usr/lib64/opensips/modules/tm.so
Reading symbols from /usr/lib64/opensips/modules/rr.so...done.
Loaded symbols for /usr/lib64/opensips/modules/rr.so
Reading symbols from /usr/lib64/opensips/modules/maxfwd.so...done.
Loaded symbols for /usr/lib64/opensips/modules/maxfwd.so
Reading symbols from /usr/lib64/opensips/modules/sipmsgops.so...done.
Loaded symbols for /usr/lib64/opensips/modules/sipmsgops.so
Reading symbols from /usr/lib64/opensips/modules/mi_fifo.so...done.
Loaded symbols for /usr/lib64/opensips/modules/mi_fifo.so
Reading symbols from /usr/lib64/opensips/modules/uri.so...done.
Loaded symbols for /usr/lib64/opensips/modules/uri.so
Reading symbols from /usr/lib64/opensips/modules/usrloc.so...done.
Loaded symbols for /usr/lib64/opensips/modules/usrloc.so
Reading symbols from /usr/lib64/opensips/modules/nathelper.so...done.
Loaded symbols for /usr/lib64/opensips/modules/nathelper.so
Reading symbols from /usr/lib64/opensips/modules/registrar.so...done.
Loaded symbols for /usr/lib64/opensips/modules/registrar.so
Reading symbols from /usr/lib64/opensips/modules/acc.so...done.
Loaded symbols for /usr/lib64/opensips/modules/acc.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/usr/sbin/opensips -u root -g root -P /var/run/opensips.pid -f'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f555a02fcea in _IO_cleanup () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6_7.7.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-42z1.el6_7.x86_64 libcom_err-1.41.12-18.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 openssl-1.0.1e-48.el6_8.1.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt full
#0  0x00007f555a02fcea in _IO_cleanup () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f5559ff0ac2 in exit () from /lib64/libc.so.6
No symbol table info available.
#2  0x000000000043df19 in sig_usr (signo=<value optimized out>) at main.c:631
        __FUNCTION__ = "sig_usr"
#3  <signal handler called>
No symbol table info available.
#4  0x00007f555a09655e in __write_nocancel () from /lib64/libc.so.6
No symbol table info available.
#5  0x00007f555a02cad3 in _IO_new_file_write () from /lib64/libc.so.6
No symbol table info available.
#6  0x00007f555a02e085 in _IO_new_do_write () from /lib64/libc.so.6
No symbol table info available.
#7  0x00007f555a02d3df in _IO_new_file_overflow () from /lib64/libc.so.6
No symbol table info available.
#8  0x00007f555a02f639 in _IO_default_xsputn_internal () from /lib64/libc.so.6
No symbol table info available.
#9  0x00007f555a02c792 in _IO_new_file_xsputn () from /lib64/libc.so.6
No symbol table info available.
#10 0x00007f555a003190 in vfprintf () from /lib64/libc.so.6
No symbol table info available.
#11 0x00007f555a02a550 in vdprintf () from /lib64/libc.so.6
No symbol table info available.
#12 0x00007f555a00a3d8 in dprintf () from /lib64/libc.so.6
No symbol table info available.
#13 0x00007f555a09fe1b in __vsyslog_chk () from /lib64/libc.so.6
No symbol table info available.
#14 0x00007f555a0a00d0 in syslog () from /lib64/libc.so.6
No symbol table info available.
#15 0x00000000004d1ab8 in tcp_read_req (con=0x7f553636dfb8, bytes_read=0x7ffc1ba7eedc) at tcp_read.c:597
        bytes = <value optimized out>
        total_bytes = <value optimized out>
        resp = 0
        size = <value optimized out>
        req = 0x8449a0
        c = <value optimized out>
        local_rcv = {src_ip = {af = 909565880, len = 32597, u = {addrl = {18446744073709551614, 47}, addr32 = {4294967294, 4294967295, 47, 0}, addr16 = {65534, 65535, 65535, 65535, 47, 0, 0, 0},
              addr = "\376\377\377\377\377\377\377\377/\000\000\000\000\000\000"}}, dst_ip = {af = 1, len = 0, u = {addrl = {1, 5048363}, addr32 = {1, 0, 5048363, 0}, addr16 = {1, 0, 0, 0, 2091, 77, 0, 0},
              addr = "\001\000\000\000\000\000\000\000+\bM\000\000\000\000"}}, src_port = 0, dst_port = 0, proto = 0, proto_reserved1 = 5046838, proto_reserved2 = 0, src_su = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>},
            sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {
                  __u6_addr8 = "\000\000\000\000\000\000\000\000\270\337\066\066U\177\000", __u6_addr16 = {0, 0, 0, 0, 57272, 13878, 32597, 0}, __u6_addr32 = {0, 0, 909565880, 32597}}}, sin6_scope_id = 4294967294}},
          bind_address = 0x7f5557c01398}
        msg_buf = <value optimized out>
        msg_len = <value optimized out>
        __FUNCTION__ = "tcp_read_req"
#16 0x00000000004d31e8 in handle_io (fm=<value optimized out>, idx=-1, event_type=<value optimized out>) at tcp_read.c:1033
        ret = 0
        n = <value optimized out>
        con = 0x7f553636dfb8
        s = <value optimized out>
        rw = <value optimized out>
        resp = <value optimized out>
        response = {140003958513592, -2}
        __FUNCTION__ = "handle_io"
#17 0x00000000004d5985 in io_wait_loop_epoll (unix_sock=<value optimized out>) at io_wait.h:845
        n = 1
        r = <value optimized out>
#18 tcp_receive_loop (unix_sock=<value optimized out>) at tcp_read.c:1141
        __FUNCTION__ = "tcp_receive_loop"
#19 0x00000000004bb82c in tcp_init_children (chd_rank=<value optimized out>, startup_done=0x0) at tcp_main.c:2390
        r = <value optimized out>
        reader_fd = {45, 47}
        pid = <value optimized out>
        si = <value optimized out>
        load_p = 0x7f553632dbb0
        __FUNCTION__ = "tcp_init_children"
#20 0x000000000043d6b8 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:1011
        i = <value optimized out>
        pid = <value optimized out>
---Type <return> to continue, or q <return> to quit---
        si = 0x0
        startup_done = 0x0
        chd_rank = 34
        rc = <value optimized out>
        load_p = 0x7f553633aed0
#21 main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1612
        cfg_log_stderr = <value optimized out>
        cfg_stream = <value optimized out>
        c = <value optimized out>
        r = <value optimized out>
        tmp = 0x7ffc1ba80f40 "172.29.106.10"
        tmp_len = <value optimized out>
        port = <value optimized out>
        proto = <value optimized out>
        options = 0x595880 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
        ret = -1
        seed = 3019838257
        rfd = <value optimized out>
        __FUNCTION__ = "main"
(gdb)

##### Logs #####
Logs were not captured(logged during Core 2,3,4 failures) may be syslogs were being rotated and blocked further logging momentarily.


##### Core 5 Time : 2017-01-25T19:35 #####
gdb /usr/sbin/opensips core.12396
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/opensips...done.

warning: core file may not match specified executable file.
[New Thread 12396]
Missing separate debuginfo for /usr/lib64/opensips/modules/signaling.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a5/6fe5f3fb5f5a5945cd0fd8384403a1610ac723
Missing separate debuginfo for /usr/lib64/opensips/modules/sl.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a0/3aa3c43c41ef6b5a3443edee6b57557f99e95f
Missing separate debuginfo for /usr/lib64/opensips/modules/tm.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/86/ec031a2c632fc2c1c2ebc807d134588c879e8f
Missing separate debuginfo for /usr/lib64/opensips/modules/rr.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/f9/5a55b1e8ec652178bbf56f1de12a130f2d3fc2
Missing separate debuginfo for /usr/lib64/opensips/modules/maxfwd.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/3c/bcedbe5e2cb56ce0232c168ce10d01549154e9
Missing separate debuginfo for /usr/lib64/opensips/modules/sipmsgops.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/68/3d8e3c5d0e47bb4e69df2e47f62b8a6c2b02e1
Missing separate debuginfo for /usr/lib64/opensips/modules/mi_fifo.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/25/0c6e832f8702623763334c66d80701abd2f757
Missing separate debuginfo for /usr/lib64/opensips/modules/uri.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6c/8c7108c432cb039e548473b9c190265cb21c3c
Missing separate debuginfo for /usr/lib64/opensips/modules/usrloc.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7a/3f6b2b6c1c5e4d9143db3610667472c5b1f1fb
Missing separate debuginfo for /usr/lib64/opensips/modules/nathelper.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/fe/eb381245a35d7cc8504f30496cbce200bc2a32
Missing separate debuginfo for /usr/lib64/opensips/modules/registrar.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e1/e152c5eb09765db719d2de687674d4b38bec29
Missing separate debuginfo for /usr/lib64/opensips/modules/acc.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/0d/53590fb94de20cbeb1aee103e38332cdb6783c
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7f/719af91ee951b4fcb6647e7868f95f766a616b
Reading symbols from /lib64/snoopy.so...done.
Loaded symbols for /lib64/snoopy.so
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /usr/lib64/opensips/modules/signaling.so...done.
Loaded symbols for /usr/lib64/opensips/modules/signaling.so
Reading symbols from /usr/lib64/opensips/modules/sl.so...done.
Loaded symbols for /usr/lib64/opensips/modules/sl.so
Reading symbols from /usr/lib64/opensips/modules/tm.so...done.
Loaded symbols for /usr/lib64/opensips/modules/tm.so
Reading symbols from /usr/lib64/opensips/modules/rr.so...done.
Loaded symbols for /usr/lib64/opensips/modules/rr.so
Reading symbols from /usr/lib64/opensips/modules/maxfwd.so...done.
Loaded symbols for /usr/lib64/opensips/modules/maxfwd.so
Reading symbols from /usr/lib64/opensips/modules/sipmsgops.so...done.
Loaded symbols for /usr/lib64/opensips/modules/sipmsgops.so
Reading symbols from /usr/lib64/opensips/modules/mi_fifo.so...done.
Loaded symbols for /usr/lib64/opensips/modules/mi_fifo.so
Reading symbols from /usr/lib64/opensips/modules/uri.so...done.
Loaded symbols for /usr/lib64/opensips/modules/uri.so
Reading symbols from /usr/lib64/opensips/modules/usrloc.so...done.
Loaded symbols for /usr/lib64/opensips/modules/usrloc.so
Reading symbols from /usr/lib64/opensips/modules/nathelper.so...done.
Loaded symbols for /usr/lib64/opensips/modules/nathelper.so
Reading symbols from /usr/lib64/opensips/modules/registrar.so...done.
Loaded symbols for /usr/lib64/opensips/modules/registrar.so
Reading symbols from /usr/lib64/opensips/modules/acc.so...done.
Loaded symbols for /usr/lib64/opensips/modules/acc.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/usr/sbin/opensips -u root -g root -P /var/run/opensips.pid -f'.
Program terminated with signal 11, Segmentation fault.
#0  fm_insert_free (qm=0x7f3e64fcf010, p=0x7f3e6508f260) at mem/f_malloc.c:147
147     mem/f_malloc.c: No such file or directory.
        in mem/f_malloc.c
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6_7.7.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-42z1.el6_7.x86_64 libcom_err-1.41.12-18.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 openssl-1.0.1e-48.el6_8.1.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt full
#0  fm_insert_free (qm=0x7f3e64fcf010, p=0x7f3e6508f260) at mem/f_malloc.c:147
        f = 0x7f3e6508f0c8
        hash = 2097
#1  fm_free (qm=0x7f3e64fcf010, p=0x7f3e6508f260) at mem/f_malloc.c:470
        f = 0x7f3e6508f248
        n = <value optimized out>
        __FUNCTION__ = "fm_free"
#2  0x00000000004d07f7 in release_tcpconn (c=0x7f3e43767ce8, state=-2, unix_sock=48) at tcp_read.c:443
        response = {7277499135891079169, 32574}
        __FUNCTION__ = "release_tcpconn"
#3  0x00000000004d3552 in handle_io (fm=<value optimized out>, idx=<value optimized out>, event_type=<value optimized out>) at tcp_read.c:1040
        ret = -1
        n = <value optimized out>
        con = 0x7f3e43767ce8
        s = <value optimized out>
        rw = <value optimized out>
        resp = -2
        response = {139905396538600, 1}
        __FUNCTION__ = "handle_io"
#4  0x00000000004d5985 in io_wait_loop_epoll (unix_sock=<value optimized out>) at io_wait.h:845
        n = 1
        r = <value optimized out>
#5  tcp_receive_loop (unix_sock=<value optimized out>) at tcp_read.c:1141
        __FUNCTION__ = "tcp_receive_loop"
#6  0x00000000004bb82c in tcp_init_children (chd_rank=<value optimized out>, startup_done=0x0) at tcp_main.c:2390
        r = <value optimized out>
        reader_fd = {46, 48}
        pid = <value optimized out>
        si = <value optimized out>
        load_p = 0x7f3e43703bb0
        __FUNCTION__ = "tcp_init_children"
#7  0x000000000043d6b8 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:1011
        i = <value optimized out>
        pid = <value optimized out>
        si = 0x0
        startup_done = 0x0
        chd_rank = 35
        rc = <value optimized out>
        load_p = 0x7f3e43710ed0
#8  main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1612
        cfg_log_stderr = <value optimized out>
        cfg_stream = <value optimized out>
        c = <value optimized out>
        r = <value optimized out>
        tmp = 0x7ffc16782f40 "172.29.106.10"
        tmp_len = <value optimized out>
        port = <value optimized out>
        proto = <value optimized out>
        options = 0x595880 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
        ret = -1
        seed = 4155673706
        rfd = <value optimized out>
        __FUNCTION__ = "main"
(gdb)

##### Logs #####
2017-01-25T12:27:53.000-05:00 [local2] [debug] E01Z01.dev.com /usr/sbin/opensips[12395]: DBG:core:forward_request: orig. len=633, new_len=808, proto=1
2017-01-25T12:27:53.000-05:00 [local2] [debug] E01Z01.dev.com /usr/sbin/opensips[12395]: DBG:core:destroy_avp_list: destroying list (nil)
2017-01-25T12:27:53.000-05:00 [local2] [debug] E01Z01.dev.com /usr/sbin/opensips[12395]: DBG:core:receive_msg: cleaning up
2017-01-25T12:27:53.000-05:00 [local2] [debug] E01Z01.dev.com /usr/sbin/opensips[12395]: DBG:core:tcp_read_req: preparing for new request, kept 0 bytes
2017-01-25T12:27:53.000-05:00 [local2] [debug] E01Z01.dev.com /usr/sbin/opensips[12395]: DBG:core:tcp_read_req: tcp_read_req end
2017-01-25T13:58:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12395]: ERROR:core:io_watch_del: trying to delete already erased entry 49 in the hash(-1, 0, 0x7f3e437111d0) )
2017-01-25T13:58:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12395]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf:

parsed:
2017-01-25T13:58:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12395]: ERROR:core:io_watch_del: trying to delete already erased entry 49 in the hash(-1, 0, 0x7f3e437111d0) )
2017-01-25T13:58:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12395]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf:

parsed:
2017-01-25T13:58:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12395]: ERROR:core:io_watch_del: trying to delete already erased entry 49 in the hash(-1, 0, 0x7f3e437111d0) )
2017-01-25T13:58:02.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12395]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf:

parsed:
2017-01-25T19:35:33.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12395]: ERROR:core:io_watch_del: trying to delete already erased entry 49 in the hash(-1, 0, 0x7f3e437111d0) )
2017-01-25T19:35:33.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12395]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf:

parsed:
2017-01-25T19:35:33.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12395]: ERROR:core:io_watch_del: trying to delete already erased entry 49 in the hash(-1, 0, 0x7f3e437111d0) )
2017-01-25T19:35:33.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12303]: ERROR:core:tls_shutdown: something wrong in SSL:
2017-01-25T19:35:33.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12303]: ERROR:core:tls_shutdown: something wrong in SSL:
2017-01-25T19:35:33.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12303]: ERROR:core:tls_shutdown: something wrong in SSL:
2017-01-25T19:35:33.000-05:00 [local2] [err] E01Z01.dev.com /usr/sbin/opensips[12303]: ERROR:core:tls_shutdown: something wrong in SSL:


##### Additional Logs #####
[root at E01Z01 opensips]# opensipsCtrl fifo list_tcp_conns
Connection::  ID=1 Type=tcp State=3 Source=172.29.106.10:35384 Destination=172.29.106.10:5060 Timeout=2017-01-27 02:18:17 Pending lifetime=0
Connection::  ID=4 Type=tls State=0 Source=172.29.106.22:5071 Destination=172.29.106.10:5062 Timeout=2017-01-27 02:05:07 Pending lifetime=0
Connection::  ID=26 Type=tcp State=4 Source=172.29.106.22:5070 Destination=172.29.106.10:5060 Timeout=2017-01-27 02:15:09 Pending lifetime=0
Connection::  ID=27 Type=tcp State=4 Source=172.29.106.13:5070 Destination=172.29.106.10:5060 Timeout=2017-01-27 02:15:10 Pending lifetime=0
Connection::  ID=3241 Type=tcp State=3 Source=172.29.106.13:63523 Destination=172.29.106.10:5060 Timeout=2017-01-26 20:06:06 Pending lifetime=0
Connection::  ID=3242 Type=tcp State=3 Source=172.29.106.22:44879 Destination=172.29.106.10:5060 Timeout=2017-01-26 20:06:06 Pending lifetime=0
Connection::  ID=3243 Type=tcp State=3 Source=172.29.106.22:52354 Destination=172.29.106.10:5060 Timeout=2017-01-27 02:14:34 Pending lifetime=0
Connection::  ID=4791 Type=tcp State=3 Source=172.29.106.11:44876 Destination=172.29.106.10:5060 Timeout=2017-01-27 02:18:16 Pending lifetime=0
Connection::  ID=4830 Type=tcp State=3 Source=172.29.106.95:37611 Destination=172.29.106.10:5060 Timeout=2017-01-27 02:18:19 Pending lifetime=0
Connection::  ID=4871 Type=tcp State=3 Source=172.29.106.10:42402 Destination=172.29.106.10:5060 Timeout=2017-01-27 02:18:20 Pending lifetime=0
Connection::  ID=4872 Type=tcp State=3 Source=172.29.106.20:53258 Destination=172.29.106.10:5060 Timeout=2017-01-26 14:18:21 Pending lifetime=0
Connection::  ID=4029 Type=tcp State=18446744073709551615 Source=172.29.106.95:5060 Destination=172.29.106.10:5060 Timeout=2017-01-25 19:35:35 Pending lifetime=0
Connection::  ID=4030 Type=tcp State=4 Source=172.29.106.95:5060 Destination=172.29.106.10:5060 Timeout=2017-01-27 02:18:20 Pending lifetime=0
[root at E01Z01 opensips]#

Debug Logs : 
[20758]: DBG:core:tcp_read_req: - received from: port 5060
[20758]: DBG:core:print_ip: - received from: ip 172.29.106.95
[20758]: DBG:core:io_watch_del: io_watch_del op on index -1 52 (0x8100e0, 52, -1, 0x10,0x3) fd_no=2 called
[20758]: ERROR:core:io_watch_del: trying to delete already erased entry 52 in the hash(0, 0, (nil)) )
[20758]: DBG:core:release_tcpconn:  releasing con 0x7f543179d700, state -2, fd=52, id=4029
[20758]: DBG:core:release_tcpconn:  extra_data (nil)
[20758]: DBG:core:tcp_read_req: Using the global ( per process ) buff
[20758]: ERROR:core:tcp_read_req: bad request, state=0, error=2 buf





More information about the Users mailing list