[OpenSIPS-Users] OpenSIPS crashed after "out of pkg memory"

Bogdan-Andrei Iancu bogdan at opensips.org
Fri Jun 27 10:38:14 CEST 2014


Hi Kevin,

There is no need to wait for a crash. From time to time, you can send a 
SIGUSR1 to a worker process (or a process you suspect as running out of 
pkg mem) -> the process will do a pkg dump to the log.

Also, I would strongly advice upgrading to 1.11 (latest LTS) - 1.9 is no 
longer maintained and there were some fixes in the memory manager since 
then.

Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

On 27.06.2014 10:36, Kevin Mathy wrote:
> Hi Bogdan,
>
> I've set given options, and now I'm waiting for a new crash of the 
> service... Where the memdump will be located ? In another logfile than 
> opensips.log, or in the same ?
>
> Thanks
>
> *
> Bien cordialement,
> Best Regards,
>
> **Kevin MATHY* |**Ingénieur VoIP
> *
> *
>
>
> 2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org 
> <mailto:bogdan at opensips.org>>:
>
>     Kevin,
>
>     Restarting should not make you loose ongoing calls (even if you
>     use the dialog module), do do not worry on that.
>
>     Regards,
>
>     Bogdan-Andrei Iancu
>     OpenSIPS Founder and Developer
>     http://www.opensips-solutions.com
>
>     On 26.06.2014 18 <tel:26.06.2014%2018>:31, Kevin Mathy wrote:
>>     Yes, I'll set those options, and wait 'till the trafic on the
>>     server reduces a lot (tonight I think) to restart opensips.
>>     Unless if you have another method to reload the config without
>>     losing calls ?
>>
>>     Thanks a lot,
>>
>>
>>     *
>>     Bien cordialement,
>>     Best Regards,
>>
>>     **Kevin MATHY* |**Ingénieur VoIP
>>     *
>>     *
>>
>>
>>     2014-06-26 17:20 GMT+02:00 Bogdan-Andrei Iancu
>>     <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>>
>>         Kevin,
>>
>>         The trick is to set (in this order):
>>             memlog = 4
>>             memdump = 1
>>
>>         This will do no logging at runtime, but it will log the mem
>>         dump (at shutdown or at signal).
>>
>>         Could you try that ?
>>
>>         Regards,
>>
>>         Bogdan-Andrei Iancu
>>         OpenSIPS Founder and Developer
>>         http://www.opensips-solutions.com
>>
>>         On 26.06.2014 18 <tel:26.06.2014%2018>:09, Kevin Mathy wrote:
>>>         Hi Bogdan,
>>>
>>>         We already have compiled opensips as described, and set
>>>         memlog=1 in the configuration file.
>>>         But as we had a lot of debug messages about memory usage, we
>>>         unset memlog=1 in the config, so we don't have any report now.
>>>         Is there a way to get those messages in a different log
>>>         file, instead of inside the "classical" opensips.log ? I've
>>>         found an option like "memdump" that can be set in
>>>         opensips.cfg, but what about it ?
>>>
>>>         And as cacti doesn't seems to be relevant to check
>>>         opensips's mem usage, do you have any other solution to
>>>         check it ?
>>>
>>>         Thanks for your help,
>>>
>>>         *
>>>         Bien cordialement,
>>>         Best Regards,
>>>
>>>         **Kevin MATHY* |**Ingénieur VoIP
>>>         *
>>>         *
>>>
>>>
>>>         2014-06-26 16:46 GMT+02:00 Bogdan-Andrei Iancu
>>>         <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>>>
>>>             Hi Kevin,
>>>
>>>             For debugging potential mem leaks, please look into:
>>>             http://www.opensips.org/Documentation/TroubleShooting-OutOfMem
>>>             If you need help with that, let me know.
>>>
>>>             Also, cacti is monitoring the system memory which is
>>>             irrelevant for OpenSIPS (as OpenSIPS is pre-allocating -
>>>             from OS - all the needed memory at startup).
>>>
>>>             Regards,
>>>
>>>             Bogdan-Andrei Iancu
>>>             OpenSIPS Founder and Developer
>>>             http://www.opensips-solutions.com
>>>
>>>             On 26.06.2014 15 <tel:26.06.2014%2015>:04, Kevin Mathy
>>>             wrote:
>>>>             Hi List,
>>>>
>>>>             I'm facing a memory usage issue with OpenSIPS 1.9.2 on
>>>>             a Debian 7 server
>>>>
>>>>                 root at asbc2:/home/kemathy# uname -a
>>>>                 Linux asbc2 3.2.0-4-amd64 #1 SMP Debian
>>>>                 3.2.57-3+deb7u2 x86_64 GNU/Linux
>>>>
>>>>                 root at asbc2:/home/kemathy# opensips -V
>>>>
>>>>                 version: opensips 1.9.2-notls (x86_64/linux)
>>>>                 flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE,
>>>>                 USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC,
>>>>                 DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>>                 ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE
>>>>                 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE
>>>>                 65535
>>>>                 poll method support: poll, epoll_lt, epoll_et,
>>>>                 sigio_rt, select.
>>>>                 svnrevision: unknown
>>>>                 @(#) $Id$
>>>>                 main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7
>>>>
>>>>
>>>>             The server handles something like 70000 to 90000 calls
>>>>             per day, with around 20 to 40 Calls Per Second
>>>>
>>>>
>>>>             As you can see on the following Cacti graphs, system's
>>>>             free memory decreases constantly to finally reach the
>>>>             lower value of 160M :
>>>>
>>>>             https://www.dropbox.com/s/np3fnc6ni4vuoet/ASBC2_Memory_Usage_Evolution.jpg
>>>>
>>>>
>>>>             The system can stay 2-3 days working in this state, but
>>>>             for example, yesterday evening (2014-06-25 18:50:00),
>>>>             opensips crashed, after showing those log messages :
>>>>
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11390]:
>>>>                 ERROR:db_mysql:db_mysql_convert_rows: no private
>>>>                 memory left
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11390]:
>>>>                 ERROR:core:build_res_buf_from_sip_req: out of pkg
>>>>                 memory  ; needs 412
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11380]:
>>>>                 ERROR:core:build_res_buf_from_sip_req: out of pkg
>>>>                 memory  ; needs 355
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11380]:
>>>>                 ERROR:core:received_builder: out of pkg memory
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11364]:
>>>>                 ERROR:core:build_res_buf_from_sip_req: out of pkg
>>>>                 memory  ; needs 355
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11364]:
>>>>                 ERROR:core:add_lump_rpl: no free pkg memory !
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11362]:
>>>>                 ERROR:core:build_res_buf_from_sip_req: out of pkg
>>>>                 memory  ; needs 355
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11362]:
>>>>                 ERROR:core:db_allocate_rows: no memory left
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11362]:
>>>>                 ERROR:db_mysql:db_mysql_convert_rows: no private
>>>>                 memory left
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11362]:
>>>>                 ERROR:core:build_res_buf_from_sip_req: out of pkg
>>>>                 memory  ; needs 412
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11356]:
>>>>                 ERROR:core:build_res_buf_from_sip_req: out of pkg
>>>>                 memory  ; needs 355
>>>>                 Jun 25 18:51:18 asbc2
>>>>                 /usr/local/sbin/opensips[11356]:
>>>>                 ERROR:core:add_lump_rpl: no free pkg memory !
>>>>
>>>>
>>>>             And after the service's restart, everything was OK :
>>>>
>>>>                 Jun 25 19:12:39 asbc2
>>>>                 /usr/local/sbin/opensips[29351]: INFO:core:main:
>>>>                 using 64 Mb shared memory
>>>>                 Jun 25 19:12:39 asbc2
>>>>                 /usr/local/sbin/opensips[29351]: INFO:core:main:
>>>>                 using 4 Mb private memory per process
>>>>
>>>>
>>>>
>>>>             For information, here are the memory values at this
>>>>             moment; opensips has handled 5000 calls since his bootup :
>>>>
>>>>                 root at asbc2:/home/kemathy# free -m
>>>>                  total used free shared  buffers cached
>>>>                 Mem: 16082  12887 3195  0        339      11393
>>>>                 -/+ buffers/cache:       1154  14928
>>>>                 Swap:  32755  0      32755
>>>>
>>>>
>>>>             For now, OpenSIPS is launched with S_MEMORY = 64 and
>>>>             P_MEMORY = 4
>>>>             My first idea would be to increase the P_MEMORY value
>>>>             to 8 for example, as it seems to be a lack of PKG
>>>>             Memory, but in case it's a memory-freeing problem,
>>>>             it'll not solve our issue (I think).
>>>>
>>>>             My questions are : How can we calculate the appropriate
>>>>             value for S_MEMORY and P_MEMORY ? And how can we solve
>>>>             our "out of pkg memory" problem ?
>>>>
>>>>             If you need further informations, or anything else,
>>>>             feel free to ask me !
>>>>
>>>>             Thanks a lot for your help,
>>>>
>>>>             *
>>>>             Bien cordialement,
>>>>             Best Regards,
>>>>
>>>>             **Kevin MATHY* |**Ingénieur VoIP
>>>>             *
>>>>             *
>>>>
>>>>
>>>>
>>>>             _______________________________________________
>>>>             Users mailing list
>>>>             Users at lists.opensips.org  <mailto:Users at lists.opensips.org>
>>>>             http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>>
>>
>>
>>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20140627/34de1994/attachment-0001.htm>


More information about the Users mailing list