[OpenSIPS-Users] OpenSIPS crashed after "out of pkg memory"

Bogdan-Andrei Iancu bogdan at opensips.org
Mon Jul 7 18:40:10 CEST 2014


Hi Kevin,

OK, found the problem. As expected, this was fixed in 1.11, but not 
ported to 1.9 as this version is not maintained any more. Here is the 
link to the fix for your crash:

https://github.com/OpenSIPS/opensips/commit/eef59c1d7b47ecc35438c01e7dcbdfeea903115a#diff-30710a8823b167e0aae5a14bb4878b75

Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

On 07.07.2014 18:15, Kevin Mathy wrote:
> Hi Bogdan,
>
> Opensips has crashed once again; I hope this time the memlog will be 
> right to be analysed !
> Here is the link to download the file : ..................
>
> If you need further informations, feel free to ask me !
>
> Thanks for your help,
>
> Kevin
>
>
>
> *
> Bien cordialement,
> Best Regards,
>
> **Kevin MATHY* |**Ingénieur VoIP
> *
> *
>
>
> 2014-07-04 11:54 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org 
> <mailto:bogdan at opensips.org>>:
>
>     Hi Kevin,
>
>     Indeed, it looks compiled ok now (with the memory debugger). Keep
>     in mind to set (in this order):
>         mem_log=4
>         mem_dump=1
>     to avoid runtime logs, but to get the mem dumps.
>
>     Regards,
>
>     Bogdan-Andrei Iancu
>     OpenSIPS Founder and Developer
>     http://www.opensips-solutions.com
>
>     On 04.07.2014 12:41, Kevin Mathy wrote:
>>     Hi Bogdan,
>>
>>     We've recompiled OpenSIPS with requested modules (I think) :
>>
>>         root at asbc2:/home/kemathy# opensips -V
>>         version: opensips 1.9.2-notls (x86_64/linux)
>>         flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE,
>>         USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, *DBG_QM_MALLOC*,
>>         FAST_LOCK-ADAPTIVE_WAIT
>>         ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
>>         MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
>>         poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>         svnrevision: unknown
>>         @(#) $Id$
>>         main.c compiled on 22:12:10 Jul  3 2014 with gcc 4.7
>>
>>
>>     Now we'll keep an eye on our server to check if everything is OK,
>>     and if the memory error still occur; as we upgraded from 1.9.1 to
>>     1.9.2...
>>
>>     I'll get back to you with some logs if needed ;-)
>>
>>     Kevin
>>
>>
>>
>>     *
>>     Bien cordialement,
>>     Best Regards,
>>
>>     **Kevin MATHY* |**Ingénieur VoIP
>>     *
>>     *
>>
>>
>>     2014-07-02 9:37 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr
>>     <mailto:k.mathy at hexanet.fr>>:
>>
>>         Hi Bogdan,
>>
>>         Hummm, right, opensips doesn't seem to have been compiled
>>         with the requested modules for memory debugging...
>>
>>             root at asbc2:/home/kemathy# opensips -V
>>             version: opensips 1.9.2-notls (x86_64/linux)
>>             flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE,
>>             USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC,
>>             FAST_LOCK-ADAPTIVE_WAIT
>>             ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
>>             MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
>>             poll method support: poll, epoll_lt, epoll_et, sigio_rt,
>>             select.
>>             svnrevision: unknown
>>             @(#) $Id$
>>             main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7
>>
>>
>>          So I think I'll have to re-compile opensips
>>         with QM_DBG_MALLOC, and try again to export the memdump log...
>>
>>         I'll get back to you when done.
>>
>>         Thanks a lot for your help !
>>
>>         Kevin
>>
>>
>>         *
>>         Bien cordialement,
>>         Best Regards,
>>
>>         **Kevin MATHY* |**Ingénieur VoIP
>>         *
>>         *
>>
>>
>>         2014-07-01 17:31 GMT+02:00 Bogdan-Andrei Iancu
>>         <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>>
>>             Hi Kevin,
>>
>>             Unfortunately the logs are not correct - are you sure you
>>             properly compiled the mem debug ? like adding the
>>             QM_DBG_MALLOC and removing the FM_MALLOC flags ? As the
>>             logs show the standard memory manager (without debugging).
>>             Check it with "opensips -V" to see the list of compiled
>>             flags.
>>
>>             I tried to get some ideas by only looking at the
>>             available memory and how many fragments were allocated in
>>             each process - indeed, there are some processes using
>>             maybe like 2 or 3 times more PKG, but not sure if a leak.
>>             Getting the proper logs (which will be huge) will tell us
>>             more.
>>
>>
>>             Regards,
>>
>>             Bogdan-Andrei Iancu
>>             OpenSIPS Founder and Developer
>>             http://www.opensips-solutions.com
>>
>>             On 01.07.2014 18:11, Kevin Mathy wrote:
>>>             Hi Bogdan,
>>>
>>>             I have now a memdump log, as we restarted opensips this
>>>             afternoon for a configuration maintenance... But the
>>>             file is too big, even if I try to put it to pastebin.com
>>>             <http://pastebin.com> ... So, here is the file; I don't
>>>             want to give the link on the mailing-list :-)
>>
>>             [removed]
>>
>>>             I hope this will help understanding our problem's cause
>>>             :-) ...
>>>
>>>             Thanks for your help,
>>>
>>>             Kevin
>>>
>>>
>>>
>>>
>>>
>>>
>>>             *
>>>             Bien cordialement,
>>>             Best Regards,
>>>
>>>             **Kevin MATHY* |**Ingénieur VoIP
>>>             *
>>>             *
>>>
>>>
>>>             2014-06-30 16:30 GMT+02:00 Kevin Mathy
>>>             <k.mathy at hexanet.fr <mailto:k.mathy at hexanet.fr>>:
>>>
>>>                 Hi Bogdan,
>>>
>>>                 Ooops, I thought my two first mails have been
>>>                 cancelled :-)
>>>                 I prefer waiting till there's no traffic, so I'll
>>>                 send a SIGUSR1 comme this evening, and reply to this
>>>                 topic with the log.
>>>
>>>                 I'll try working with MI statistics to make some
>>>                 memory usage graphs better than with Cacti...
>>>
>>>                 I'll come back to you with logs; thanks for all !
>>>
>>>                 Kevin
>>>
>>>
>>>
>>>                 *
>>>                 Bien cordialement,
>>>                 Best Regards,
>>>
>>>                 **Kevin MATHY* |**Ingénieur VoIP
>>>                 *
>>>                 *
>>>
>>>
>>>                 2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu
>>>                 <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>>>
>>>                     Hi Kevin,
>>>
>>>                     There is no need to send your email three times
>>>                     ;). One time is enough.
>>>
>>>                     Waiting and taking the dump when there is not
>>>                     traffic is good (but not a must) - the idea is
>>>                     to be sure that all temporary memory (used for
>>>                     processing the current traffic) was freed - so
>>>                     what you still have in memory is configuration
>>>                     data or possible leaks.
>>>                     If you do not have the luxury of waiting, you
>>>                     can do it whenever you can.
>>>
>>>                     Once again, do not look at the memory usage
>>>                     reported by OS - it is irrelevant as OpenSIPS is
>>>                     doing its own internal memory management.
>>>
>>>                     Check the memory usage via MI, see the mem
>>>                     related statistics:
>>>                     http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11
>>>
>>>                     Regards,
>>>
>>>                     Bogdan-Andrei Iancu
>>>                     OpenSIPS Founder and Developer
>>>                     http://www.opensips-solutions.com
>>>
>>>                     On 30.06.2014 12:01, Kevin Mathy wrote:
>>>>                     Hi Bogdan,
>>>>
>>>>                     If I want to send a SIGUSR1, may I have to wait
>>>>                     20 minutes after the last call ? 20 minutes
>>>>                     without any call ? I don't understand well this
>>>>                     sentence :
>>>>
>>>>                         It is highly recommended to do this after
>>>>                         waiting about 20 minutes to be sure that as
>>>>                         much as possile memory is freed - all
>>>>                         temporary memory used during processing is
>>>>                         freed by lack of load on the proxy
>>>>
>>>>
>>>>                     Also, last week-end, the traffic reduced a lot,
>>>>                     and between last friday, when the free system's
>>>>                     memory was around 170M, and this morning, the
>>>>                     free memory seems to have increased : this
>>>>                     morning, it was around 700M, before the traffic
>>>>                     comes back.
>>>>
>>>>                     So, opensips seems to well free the memory,
>>>>                     isn't it ?
>>>>
>>>>                     Thanks for your help,
>>>>
>>>>                     Kevin
>>>>
>>>>
>>>>                     *
>>>>                     Bien cordialement,
>>>>                     Best Regards,
>>>>
>>>>                     **Kevin MATHY* |**Ingénieur VoIP
>>>>                     *
>>>>                     *
>>>>
>>>>
>>>>                     2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu
>>>>                     <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>>>>
>>>>                         Hi Kevin,
>>>>
>>>>                         There is no need to wait for a crash. From
>>>>                         time to time, you can send a SIGUSR1 to a
>>>>                         worker process (or a process you suspect as
>>>>                         running out of pkg mem) -> the process will
>>>>                         do a pkg dump to the log.
>>>>
>>>>                         Also, I would strongly advice upgrading to
>>>>                         1.11 (latest LTS) - 1.9 is no longer
>>>>                         maintained and there were some fixes in the
>>>>                         memory manager since then.
>>>>
>>>>                         Regards,
>>>>
>>>>                         Bogdan-Andrei Iancu
>>>>                         OpenSIPS Founder and Developer
>>>>                         http://www.opensips-solutions.com
>>>>
>>>>                         On 27.06.2014 10 <tel:27.06.2014%2010>:36,
>>>>                         Kevin Mathy wrote:
>>>>>                         Hi Bogdan,
>>>>>
>>>>>                         I've set given options, and now I'm
>>>>>                         waiting for a new crash of the service...
>>>>>                         Where the memdump will be located ? In
>>>>>                         another logfile than opensips.log, or in
>>>>>                         the same ?
>>>>>
>>>>>                         Thanks
>>>>>
>>>>>                         *
>>>>>                         Bien cordialement,
>>>>>                         Best Regards,
>>>>>
>>>>>                         **Kevin MATHY* |**Ingénieur VoIP
>>>>>                         *
>>>>>                         *
>>>>>
>>>>>
>>>>>                         2014-06-26 18:32 GMT+02:00 Bogdan-Andrei
>>>>>                         Iancu <bogdan at opensips.org
>>>>>                         <mailto:bogdan at opensips.org>>:
>>>>>
>>>>>                             Kevin,
>>>>>
>>>>>                             Restarting should not make you loose
>>>>>                             ongoing calls (even if you use the
>>>>>                             dialog module), do do not worry on that.
>>>>>
>>>>>                             Regards,
>>>>>
>>>>>                             Bogdan-Andrei Iancu
>>>>>                             OpenSIPS Founder and Developer
>>>>>                             http://www.opensips-solutions.com
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20140707/97fd6fcc/attachment-0001.htm>


More information about the Users mailing list