[OpenSIPS-Users] running sip tls on 443
Tito Cumpen
tito at xsvoce.com
Tue Jul 5 14:02:17 CEST 2016
Bogdan,
Apologies for not providing a full bt. This is a development server with
very little load.
Reading symbols from /usr/sbin/opensips...done.
[New LWP 2685]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/sbin/opensips -P /var/run/opensips.pid -u opensips
-g opensips -M 1024 -f /etc'.
Program terminated with signal 6, Aborted.
#0 0x00007fc95ab1d5f7 in __GI_raise (sig=sig at entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt full
#0 0x00007fc95ab1d5f7 in __GI_raise (sig=sig at entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
resultvar = 0
pid = 2685
selftid = 2685
#1 0x00007fc95ab1ece8 in __GI_abort () at abort.c:90
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x7fc90eb056c0
<wss_current_req>, sa_sigaction = 0x7fc90eb056c0 <wss_current_req>},
sa_mask = {__val = {140501681180296, 140501713793640,
140502794949216, 140501683341128, 140501670335743,
140501683341128, 16668947009245032192, 58, 140501683442448,
140501713793689, 65534, 125184, 42698496, 125008, 0, 140502790453088}},
sa_flags = 1, sa_restorer = 0x0}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x000000000050d48c in qm_free (qm=<optimized out>,
p=p at entry=0x7fc91abdb298,
file=file at entry=0x7fc90e900108 "../proto_ws/ws_common.h",
func=func at entry=0x7fc90e903d89 <__FUNCTION__.22006> "ws_process",
line=line at entry=560) at mem/q_malloc.c:469
f = <optimized out>
prev = <optimized out>
next = <optimized out>
size = <optimized out>
__FUNCTION__ = "qm_free"
#3 0x00007fc90e8fcd6c in ws_process (con=<optimized out>) at
../proto_ws/ws_common.h:560
size = 0
req = <optimized out>
ret_code = WS_ERR_NONE
bk = <optimized out>
msg_len = <optimized out>
local_rcv = {src_ip = {af = 2, len = 4, u = {addrl = {3870954855,
0}, addr32 = {3870954855, 0, 0, 0}, addr16 = {5479, 59066, 0, 0, 0, 0, 0,
0},
addr = "g\025\272\346", '\000' <repeats 11 times>}}, dst_ip =
{af = 2, len = 4, u = {addrl = {1252388288, 0}, addr32 = {1252388288, 0, 0,
0}, addr16 = {60864, 19109, 0, 0, 0, 0, 0, 0},
addr = "\300\355\245J", '\000' <repeats 11 times>}}, src_port
= 57327, dst_port = 10443, proto = 6, proto_reserved1 = 702,
proto_reserved2 = 0, src_su = {s = {sa_family = 2,
sa_data =
"\337\357g\025\272\346\000\000\000\000\000\000\000"}, sin = {sin_family =
2, sin_port = 61407, sin_addr = {s_addr = 3870954855}, sin_zero =
"\000\000\000\000\000\000\000"},
sin6 = {sin6_family = 2, sin6_port = 61407, sin6_flowinfo =
3870954855, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {
0, 0, 0, 0}}}, sin6_scope_id = 1024}}, bind_address =
0x7fc91ab99738}
newreq = <optimized out>
msg_buf = <optimized out>
#4 wss_read_req (con=<optimized out>, bytes_read=<optimized out>) at
proto_wss.c:428
size = <optimized out>
__FUNCTION__ = "wss_read_req"
#5 0x000000000059049c in handle_io (fm=<optimized out>, idx=idx at entry=4,
event_type=event_type at entry=1) at net/net_tcp_proc.c:205
ret = 0
n = <optimized out>
con = 0x7fc918ee9148
s = 58
rw = <optimized out>
resp = <optimized out>
response = {140501683441992, 1}
__FUNCTION__ = "handle_io"
#6 0x0000000000591a44 in io_wait_loop_epoll (h=<optimized out>,
t=<optimized out>, repeat=<optimized out>) at net/../io_wait_loop.h:221
ret = <optimized out>
e = <optimized out>
n = 1
r = 4
#7 tcp_worker_proc (unix_sock=<optimized out>) at net/net_tcp_proc.c:312
__FUNCTION__ = "tcp_worker_proc"
#8 0x000000000059bf09 in tcp_start_processes (chd_rank=chd_rank at entry=0x834800
<chd_rank.11049>, startup_done=startup_done at entry=0x7fc918ce2c60) at
net/net_tcp.c:1761
r = 4
reader_fd = {48, 50}
pid = 0
load_p = 0x7fc918ce4390
__FUNCTION__ = "tcp_start_processes"
#9 0x0000000000419417 in main_loop () at main.c:677
startup_done = 0x7fc918ce2c60
chd_rank = 17
#10 main (argc=<optimized out>, argv=<optimized out>) at main.c:1249
cfg_stream = <optimized out>
---Type <return> to continue, or q <return> to quit---
c = <optimized out>
r = <optimized out>
tmp = 0x7ffe75313f65 ""
tmp_len = <optimized out>
port = <optimized out>
proto = <optimized out>
protos_no = <optimized out>
options = 0x5ce330 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
ret = -1
seed = 4282461619
rfd = <optimized out>
__FUNCTION__ = "main"
On Tue, Jul 5, 2016 at 7:58 AM, Bogdan-Andrei Iancu <bogdan at opensips.org>
wrote:
> Hi Tito,
>
> Could you print the whole backtrace ?
>
> Thanks,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>
> On 05.07.2016 14:37, Tito Cumpen wrote:
>
> Bogdan,
>
> Here is the backtrace with those flags enabled
>
> gdb /usr/sbin/opensips core.2685
>
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
>
> Copyright (C) 2013 Free Software Foundation, Inc.
>
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
>
> This is free software: you are free to change and redistribute it.
>
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>
> and "show warranty" for details.
>
> This GDB was configured as "x86_64-redhat-linux-gnu".
>
> For bug reporting instructions, please see:
>
> <http://www.gnu.org/software/gdb/bugs/>...
>
> Reading symbols from /usr/sbin/opensips...done.
>
> [New LWP 2685]
>
> [Thread debugging using libthread_db enabled]
>
> Using host libthread_db library "/lib64/libthread_db.so.1".
>
> Core was generated by `/sbin/opensips -P /var/run/opensips.pid -u opensips
> -g opensips -M 1024 -f /etc'.
>
> Program terminated with signal 6, Aborted.
>
> #0 0x00007fc95ab1d5f7 in __GI_raise (sig=sig at entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>
> 56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
>
> (gdb)
>
>
> Let me know if you need anything else
>
>
>
> On Mon, Jul 4, 2016 at 5:54 AM, Bogdan-Andrei Iancu <
> <bogdan at opensips.org>bogdan at opensips.org> wrote:
>
>> Thank you Tito,
>>
>> It looks like a crash in the memory manager, while trying to allocate a
>> new structure. Do debug such problems there is no other way than enabling
>> debugging for the memory manager (QM_MALLOC + DBG_MALLOC flags) - is this a
>> production system with considerable load on it ?
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>
>> On 01.07.2016 18:44, Tito Cumpen wrote:
>>
>> Bogdan,
>>
>>
>> Correction I was using :
>>
>> version: opensips 2.2.0 (x86_64/linux)
>>
>> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC,
>> F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>
>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>
>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>
>> git revision: d835721
>>
>> main.c compiled on 15:24:26 Jun 28 2016 with gcc 4.8.5
>>
>>
>>
>>
>>
>> On Fri, Jul 1, 2016 at 9:23 AM, Tito Cumpen < <tito at xsvoce.com>
>> tito at xsvoce.com> wrote:
>>
>>> Bogdan,
>>>
>>> Here is the backtrace:
>>>
>>>
>>> Reading symbols from /usr/sbin/opensips...done.
>>>
>>> [New LWP 2648]
>>>
>>> [Thread debugging using libthread_db enabled]
>>>
>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>>
>>> Core was generated by `/sbin/opensips -P /var/run/opensips.pid -u
>>> opensips -g opensips -M 1024 -f /etc'.
>>>
>>> Program terminated with signal 11, Segmentation fault.
>>>
>>> #0 0x0000000000512782 in fm_remove_free (n=0x7fadc0898210,
>>> qm=0x7fadc0809010) at mem/f_malloc.c:187
>>>
>>> 187 *pf=n->u.nxt_free;
>>>
>>> Missing separate debuginfos, use: debuginfo-install
>>> cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 glibc-2.17-106.el7_2.4.x86_64
>>> gmp-6.0.0-12.el7_1.x86_64 gnutls-3.3.8-14.el7_2.x86_64
>>> keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.13.2-12.el7_2.x86_64
>>> libcom_err-1.42.9-7.el7.x86_64 libcurl-7.29.0-25.el7.centos.x86_64
>>> libffi-3.0.13-16.el7.x86_64 libgcc-4.8.5-4.el7.x86_64
>>> libgcrypt-1.5.3-12.el7_1.1.x86_64 libgpg-error-1.12-3.el7.x86_64
>>> libidn-1.28-4.el7.x86_64 libmicrohttpd-0.9.33-2.el7.x86_64
>>> librabbitmq-0.5.2-1.el7.x86_64 libselinux-2.2.2-6.el7.x86_64
>>> libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-4.el7.x86_64
>>> libtasn1-3.8-2.el7.x86_64 nettle-2.7.1-4.el7.x86_64
>>> nspr-4.11.0-1.el7_2.x86_64 nss-3.21.0-9.el7_2.x86_64
>>> nss-softokn-freebl-3.16.2.3-14.2.el7_2.x86_64
>>> nss-util-3.21.0-2.2.el7_2.x86_64 openldap-2.4.40-9.el7_2.x86_64
>>> openssl-libs-1.0.1e-51.el7_2.4.x86_64 p11-kit-0.20.7-3.el7.x86_64
>>> pcre-8.32-15.el7.x86_64 trousers-0.3.13-1.el7.x86_64
>>> xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
>>>
>>> (gdb) bt full
>>>
>>> #0 0x0000000000512782 in fm_remove_free (n=0x7fadc0898210,
>>> qm=0x7fadc0809010) at mem/f_malloc.c:187
>>>
>>> pf = 0x0
>>>
>>> hash = 2051
>>>
>>> #1 fm_malloc (qm=0x7fadc0809010, size=size at entry=65608) at
>>> mem/f_malloc.c:415
>>>
>>> frag = 0x7fadc0898210
>>>
>>> n = <optimized out>
>>>
>>> hash = 2051
>>>
>>> __FUNCTION__ = "fm_malloc"
>>>
>>> #2 0x00007fadb4615091 in ws_process (con=0x7fadbeb3f718) at
>>> ../proto_ws/ws_common.h:576
>>>
>>> size = <optimized out>
>>>
>>> req = <optimized out>
>>>
>>> ret_code = WS_ERR_NONE
>>>
>>> bk = <optimized out>
>>>
>>> msg_len = <optimized out>
>>>
>>> local_rcv = {src_ip = {af = 3082146304, len = 32685, u = {addrl
>>> = {34359801655, 140720703251544}, addr32 = {63287, 8, 394765400, 32764},
>>> addr16 = {63287, 0, 8, 0, 42072, 6023, 32764, 0},
>>>
>>> addr =
>>> "7\367\000\000\b\000\000\000X\244\207\027\374\177\000"}}, dst_ip = {af =
>>> 3080002780, len = 32685, u = {addrl = {4, 63119970000}, addr32 = {4, 0,
>>> 2990427856, 14}, addr16 = {4, 0, 0,
>>>
>>> 0, 20176, 45630, 14, 0}, addr =
>>> "\004\000\000\000\000\000\000\000\320N>\262\016\000\000"}}, src_port =
>>> 52976, dst_port = 49287, proto = 32685, proto_reserved1 = -1214962633,
>>>
>>> proto_reserved2 = 32685, src_su = {s = {sa_family = 2, sa_data
>>> = ")\350n\343\276C\360#%\275\016\000\000"}, sin = {sin_family = 2, sin_port
>>> = 59433, sin_addr = {s_addr = 1136583534},
>>>
>>> sin_zero = "\360#%\275\016\000\000"}, sin6 = {sin6_family
>>> = 2, sin6_port = 59433, sin6_flowinfo = 1136583534, sin6_addr = {__in6_u = {
>>>
>>> __u6_addr8 =
>>> "\360#%\275\016\000\000\000\360·\300\255\177\000", __u6_addr16 = {9200,
>>> 48421, 14, 0, 52976, 49287, 32685, 0}, __u6_addr32 = {3173327856, 14,
>>> 3230125808, 32685}}},
>>>
>>> sin6_scope_id = 1}}, bind_address = 0x7fadc0882ea8}
>>>
>>> newreq = <optimized out>
>>>
>>> msg_buf = <optimized out>
>>>
>>> #3 wss_read_req (con=0x7fadbeb3f718, bytes_read=<optimized out>) at
>>> proto_wss.c:428
>>>
>>> size = <optimized out>
>>>
>>> __FUNCTION__ = "wss_read_req"
>>>
>>> #4 0x000000000059c374 in handle_io (fm=<optimized out>, idx=idx at entry=0,
>>> event_type=event_type at entry=1) at net/net_tcp_proc.c:205
>>>
>>> ret = 0
>>>
>>> n = <optimized out>
>>>
>>> con = 0x7fadbeb3f718
>>>
>>> s = 57
>>>
>>> rw = <optimized out>
>>>
>>> resp = <optimized out>
>>>
>>> response = {140384203757736, 1}
>>>
>>> __FUNCTION__ = "handle_io"
>>>
>>> #5 0x000000000059d914 in io_wait_loop_epoll (h=<optimized out>,
>>> t=<optimized out>, repeat=<optimized out>) at net/../io_wait_loop.h:221
>>>
>>> ret = <optimized out>
>>>
>>> e = <optimized out>
>>>
>>> n = 1
>>>
>>> r = 0
>>>
>>> #6 tcp_worker_proc (unix_sock=<optimized out>) at net/net_tcp_proc.c:312
>>>
>>> __FUNCTION__ = "tcp_worker_proc"
>>>
>>> #7 0x00000000005a7fc9 in tcp_start_processes (chd_rank=chd_rank at entry=0x8417e0
>>> <chd_rank.11018>, startup_done=startup_done at entry=0x7fadbe978738) at
>>> net/net_tcp.c:1761
>>>
>>> r = 5
>>>
>>> reader_fd = {50, 52}
>>>
>>> pid = 0
>>>
>>> load_p = 0x7fadbe979398
>>>
>>> __FUNCTION__ = "tcp_start_processes"
>>>
>>> #8 0x0000000000419ef5 in main_loop () at main.c:677
>>>
>>> startup_done = 0x7fadbe978738
>>>
>>> chd_rank = 18
>>>
>>> #9 main (argc=<optimized out>, argv=<optimized out>) at main.c:1249
>>>
>>> cfg_stream = <optimized out>
>>>
>>> c = <optimized out>
>>>
>>> r = <optimized out>
>>>
>>> tmp = 0x7ffc1787bf65 ""
>>>
>>> tmp_len = <optimized out>
>>>
>>> port = <optimized out>
>>>
>>> proto = <optimized out>
>>>
>>> protos_no = <optimized out>
>>>
>>> options = 0x5dae60 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
>>>
>>> ---Type <return> to continue, or q <return> to quit---
>>>
>>> ret = -1
>>>
>>> seed = 995413301
>>>
>>> rfd = <optimized out>
>>>
>>> __FUNCTION__ = "main"
>>>
>>>
>>>
>>> currently using version: opensips 2.3.0-dev (x86_64/linux)
>>>
>>> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC,
>>> QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>
>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>
>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>
>>> git revision: 92434ef
>>>
>>> main.c compiled on 15:05:06 Jun 28 2016 with gcc 4.8.5
>>>
>>> On Tue, Jun 28, 2016 at 8:26 AM, Bogdan-Andrei Iancu <
>>> <bogdan at opensips.org>bogdan at opensips.org> wrote:
>>>
>>>> Hi Tito,
>>>>
>>>> If opensips crashes, were you able to extract a backtrace from the core
>>>> file(s) ?
>>>>
>>>> Thanks and regards,
>>>>
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>
>>>> On 27.06.2016 21:20, Tito Cumpen wrote:
>>>>
>>>> Group,
>>>>
>>>>
>>>> I am experiencing strange behavior when configuring sip tls on port
>>>> 443. At time opensips crashes or stops accepting new connections. Here are
>>>> the tcp configs I am using:
>>>>
>>>> #disable_tcp=no
>>>>
>>>> tcp_connection_lifetime=3600
>>>>
>>>> tcp_connect_timeout=3
>>>>
>>>> tcp_keepidle = 30
>>>>
>>>> tcp_keepinterval = 5
>>>>
>>>> tcp_keepalive = 1
>>>>
>>>> tcp_keepcount = 5
>>>>
>>>> tcp_max_msg_time = 8
>>>> tcp_children=10
>>>>
>>>>
>>>> Any idea what would case this? I am assuming there are probes out in
>>>> the internet that eventually make opensips crash?
>>>>
>>>> Thanks,
>>>> Tito
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>>
>>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20160705/d69f748d/attachment-0001.htm>
More information about the Users
mailing list