[OpenSIPS-Users] Opensips 2.2.2 memory issue?

Vasilios Tzanoudakis vasilios.tzanoudakis at voiceland.gr
Wed Mar 29 08:03:15 EDT 2017


Dear Liviu,

Regarding other UDP processes, I tried 1850 but wasn't showing backtraces
with memory information on it or any other info It seemed to be normal but
i have't kept them. 1851,1852 gave me almost the same results.

Also Cpu load started increasing by the time the problem started happening.
Here is the screenshot to see. No other reason for High CPU load.
CPU Usage came back to normal after restarting processes.
here is the cpu graph on this server : http://oi68.tinypic.com/2m79irs.jpg

I will try to enable DBG_MALLOC and see if this happens again , so as I can
understand you don't see any obvious thing around....

Thank you


*Vasilios Tzanoudakis* | Technology Director
t. +30-212-222-8003  |  f. +30-212-222-8001
2, Klisthenous Str., Metamorfosi, 144 52, Attica, Greece
vasilios.tzanoudakis at voiceland.gr  |  www.voiceland.gr


On Wed, Mar 29, 2017 at 12:25 PM, Vasilios Tzanoudakis <
vasilios.tzanoudakis at voiceland.gr> wrote:

> Dear all,
>
> After 11 days os usage Opensips process was not responding at all.
>
> This is the second time I am facing this kind of issue.
> I am sending you all the information below:
>
> System has 4 CPU Cores / 4 GB Ram
>
> Opensips version 2.2.2
>
> *Run parameters : *
> S_MEMORY=2048
> P_MEMORY=1024
>
> *uname -a*
> Linux 4.4.0-62-generic #83~14.04.1-Ubuntu SMP Wed Jan 18 18:10:30 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
>
> *Compiled with the following flags:*
>
> PKG_MALLOC
> SHM_MMAP
> USE_MCAST
> DISABLE_NAGLE
> STATISTICS
> HAVE_RESOLV_RES
> F_MALLOC
> F_MALLOC_OPTIMIZATIONS
> NO_DEBUG
>
> *and with the following extra modules : *
>
> db_mysql
> dialplan
> regex
> snmpstats
>
> *Loaded modules in opensips.cfg*
>
> loadmodule "proto_udp.so"
> loadmodule "mi_datagram.so"
> loadmodule "signaling.so"
> loadmodule "sl.so"
> loadmodule "tm.so"
> loadmodule "rr.so"
> loadmodule "maxfwd.so"
> loadmodule "sipmsgops.so"
> loadmodule "mi_fifo.so"
> loadmodule "uri.so"
> loadmodule "db_mysql.so"
> loadmodule "usrloc.so"
> loadmodule "registrar.so"
> loadmodule "auth.so"
> loadmodule "auth_db.so"
> loadmodule "dialog.so"
> loadmodule "nathelper.so"
> loadmodule "rtpproxy.so"
> loadmodule "drouting.so"
> loadmodule "uac.so"
>
> */usr/local/opensips2/sbin/opensipsctl fifo ps*
>
> Process::  ID=0 PID=1843 Type=attendant
> Process::  ID=1 PID=1845 Type=MI FIFO
> Process::  ID=2 PID=1846 Type=MI Datagram
> Process::  ID=3 PID=1847 Type=time_keeper
> Process::  ID=4 PID=1849 Type=timer
> Process::  ID=5 PID=1850 Type=SIP receiver udp:X.X.X.X:65100
> Process::  ID=6 PID=1851 Type=SIP receiver udp:X.X.X.X:65100
> Process::  ID=7 PID=1852 Type=SIP receiver udp:X.X.X.X:65100
> Process::  ID=8 PID=1853 Type=SIP receiver udp:X.X.X.X:65100
> Process::  ID=9 PID=1854 Type=SIP receiver udp:X.X.X.X:65100
> Process::  ID=10 PID=1855 Type=SIP receiver udp:X.X.X.X:65100
> Process::  ID=11 PID=1856 Type=SIP receiver udp:X.X.X.X:65100
> Process::  ID=12 PID=1857 Type=SIP receiver udp:X.X.X.X:65100
> Process::  ID=13 PID=1859 Type=Timer handler
>
> *gdb /usr/local/opensips2/sbin/opensips 1851*
>
> Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging
> symbols found)...done.
> Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
> Reading symbols from /lib/x86_64-linux-gnu/libnss_dns.so.2...Reading
> symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_dns-2.19.so..
> .done.
> done.
> Loaded symbols for /lib/x86_64-linux-gnu/libnss_dns.so.2
> 0x00007faee52c36cd in parse_backslash (offset=0x7ffc00000000,
> words=<optimized out>, max_length=0x7ffc32c06040, word_length=0xd,
> word=0x636f732e3379786f) at wordexp.c:222
> 222 wordexp.c: No such file or directory.
>
>
> *(gdb) bt*
> #0  0x00007faee52c36cd in parse_backslash (offset=0x7ffc00000000,
> words=<optimized out>, max_length=0x7ffc32c06040, word_length=0xd,
> word=0x636f732e3379786f) at wordexp.c:222
> #1  parse_backtick (ifs_white=0x722f7261762f0001 <error: *Cannot access
> memory at address* 0x722f7261762f0001>, ifs=0x7fae24ccc960 "",
> pwordexp=0x838fd4 <buf+916>, flags=0, offset=0x7ffc00000000,
>     words=<optimized out>, max_length=0x1, word_length=0x7ffc32c06060,
> word=0x7ffc32c060c0) at wordexp.c:2147
> #2  wordexp (words=<optimized out>, pwordexp=<optimized out>, flags=0) at
> wordexp.c:2359
> #3  0x00007ffc32c06220 in ?? ()
> #4  0x0000000000000000 in ?? ()
>
>
> *(gdb) bt full*
> #0  0x00007faee52c36cd in parse_backslash (offset=0x7ffc00000000,
> words=<optimized out>, max_length=0x7ffc32c06040, word_length=0xd,
> word=0x636f732e3379786f) at wordexp.c:222
> No locals.
> #1  parse_backtick (ifs_white=0x722f7261762f0001 <error: *Cannot access
> memory at address *0x722f7261762f0001>, ifs=0x7fae24ccc960 "",
> pwordexp=0x838fd4 <buf+916>, flags=0, offset=0x7ffc00000000,
>     words=<optimized out>, max_length=0x1, word_length=0x7ffc32c06060,
> word=0x7ffc32c060c0) at wordexp.c:2147
>         squoting = <optimized out>
>         comm_length = 8622050
>         *comm = 0x6 <error: Cannot access memory at address 0x6>*
>         error = <optimized out>
>         comm_maxlen = 8700952
> #2  wordexp (words=<optimized out>, pwordexp=<optimized out>, flags=0) at
> wordexp.c:2359
>         words_offset = 0
>         word_length = 0
>         max_length = 140388071391744
>         word = 0x7ffc32c065c8 ""
>         error = <optimized out>
>         ifs = 0x7fae24ccc960 ""
>         ifs_white = "\000\000\000"
>         old_word = {we_wordc = 0, we_wordv = 0x0, we_offs = 0}
> #3  0x00007ffc32c06220 in ?? ()
> No symbol table info available.
> #4  0x0000000000000000 in ?? ()
> No symbol table info available.
>
> *ldd -v /usr/local/opensips2/sbin/opensips*
> linux-vdso.so.1 =>  (0x00007fff96f49000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f75c45de000)
> libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
> (0x00007f75c43c3000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f75c3ffa000)
> /lib64/ld-linux-x86-64.so.2 (0x000055e9cad86000)
>
> Version information:
> /usr/local/opensips2/sbin/opensips:
> libresolv.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libresolv.so.2
> libdl.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libdl.so.2
> libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.15) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
> /lib/x86_64-linux-gnu/libdl.so.2:
> ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
> libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
> /lib/x86_64-linux-gnu/libresolv.so.2:
> libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
> /lib/x86_64-linux-gnu/libc.so.6:
> ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
> ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
>
> *# free -m*
>              total       used       free     shared    buffers     cached
> Mem:          3951       2842       1109         28         66       2156
> -/+ buffers/cache:        619       3332
> Swap:         4091         90       4001
>
> *# vmstat *
> procs -----------memory---------- ---swap-- -----io---- -system--
> ------cpu-----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
> wa st
>  2  0  92396 1136328  68576 2207768    0    0     5    14   10    2  1  2
> 97  0  0
>
> *# w*
>  11:43:18 up 11 days, 23:16,  1 user,  load average: 2.46, 2.39, 2.36
>
> *# netstat -ulnp*
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address           Foreign Address         State
>       PID/Program name
> udp   426496      0 X.X.X.X:65100     0.0.0.0:*
> 1843/opensips
>
> also after restarting opensips process system started to receive traffic
> again BUT I got the following errors on syslog regarding rtpproxy
>
> Mar 29 09:57:32 rtp1 /usr/local/opensips2/sbin/opensips[1850]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:09 rtp1 /usr/local/opensips2/sbin/opensips[1850]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:10 rtp1 /usr/local/opensips2/sbin/opensips[1857]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:12 rtp1 /usr/local/opensips2/sbin/opensips[1850]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:26 rtp1 /usr/local/opensips2/sbin/opensips[1850]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:26 rtp1 /usr/local/opensips2/sbin/opensips[1851]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:26 rtp1 /usr/local/opensips2/sbin/opensips[1853]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
>
> and I had to reboot rtpproxy processes too for traffic to pass through
> without problems.
> I don't know if this is another problem with rtpproxy but I am just
> mentioning it.
>
> thank you in advance
>
>
> *Vasilios Tzanoudakis* | Technology Director
> t. +30-212-222-8003 <+30%2021%202222%208003>  |  f. +30-212-222-8001
> <+30%2021%202222%208001>
> 2, Klisthenous Str., Metamorfosi, 144 52, Attica, Greece
> vasilios.tzanoudakis at voiceland.gr  |  www.voiceland.gr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170329/ba44fbf2/attachment-0001.html>


More information about the Users mailing list