[OpenSIPS-Users] OpenSIPS crashed after "out of pkg memory"

Bogdan-Andrei Iancu bogdan at opensips.org
Fri Jul 4 11:54:26 CEST 2014


Hi Kevin,

Indeed, it looks compiled ok now (with the memory debugger). Keep in 
mind to set (in this order):
     mem_log=4
     mem_dump=1
to avoid runtime logs, but to get the mem dumps.

Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

On 04.07.2014 12:41, Kevin Mathy wrote:
> Hi Bogdan,
>
> We've recompiled OpenSIPS with requested modules (I think) :
>
>     root at asbc2:/home/kemathy# opensips -V
>     version: opensips 1.9.2-notls (x86_64/linux)
>     flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST,
>     SHM_MEM, SHM_MMAP, PKG_MALLOC, *DBG_QM_MALLOC*,
>     FAST_LOCK-ADAPTIVE_WAIT
>     ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
>     16, MAX_URI_SIZE 1024, BUF_SIZE 65535
>     poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>     svnrevision: unknown
>     @(#) $Id$
>     main.c compiled on 22:12:10 Jul  3 2014 with gcc 4.7
>
>
> Now we'll keep an eye on our server to check if everything is OK, and 
> if the memory error still occur; as we upgraded from 1.9.1 to 1.9.2...
>
> I'll get back to you with some logs if needed ;-)
>
> Kevin
>
>
>
> *
> Bien cordialement,
> Best Regards,
>
> **Kevin MATHY* |**Ingénieur VoIP
> *
> *
>
>
> 2014-07-02 9:37 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr 
> <mailto:k.mathy at hexanet.fr>>:
>
>     Hi Bogdan,
>
>     Hummm, right, opensips doesn't seem to have been compiled with the
>     requested modules for memory debugging...
>
>         root at asbc2:/home/kemathy# opensips -V
>         version: opensips 1.9.2-notls (x86_64/linux)
>         flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST,
>         SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC,
>         FAST_LOCK-ADAPTIVE_WAIT
>         ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
>         MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
>         poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>         svnrevision: unknown
>         @(#) $Id$
>         main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7
>
>
>      So I think I'll have to re-compile opensips with QM_DBG_MALLOC,
>     and try again to export the memdump log...
>
>     I'll get back to you when done.
>
>     Thanks a lot for your help !
>
>     Kevin
>
>
>     *
>     Bien cordialement,
>     Best Regards,
>
>     **Kevin MATHY* |**Ingénieur VoIP
>     *
>     *
>
>
>     2014-07-01 17:31 GMT+02:00 Bogdan-Andrei Iancu
>     <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>
>         Hi Kevin,
>
>         Unfortunately the logs are not correct - are you sure you
>         properly compiled the mem debug ? like adding the
>         QM_DBG_MALLOC and removing the FM_MALLOC flags ? As the logs
>         show the standard memory manager (without debugging).
>         Check it with "opensips -V" to see the list of compiled flags.
>
>         I tried to get some ideas by only looking at the available
>         memory and how many fragments were allocated in each process -
>         indeed, there are some processes using maybe like 2 or 3 times
>         more PKG, but not sure if a leak.
>         Getting the proper logs (which will be huge) will tell us more.
>
>
>         Regards,
>
>         Bogdan-Andrei Iancu
>         OpenSIPS Founder and Developer
>         http://www.opensips-solutions.com
>
>         On 01.07.2014 18:11, Kevin Mathy wrote:
>>         Hi Bogdan,
>>
>>         I have now a memdump log, as we restarted opensips this
>>         afternoon for a configuration maintenance... But the file is
>>         too big, even if I try to put it to pastebin.com
>>         <http://pastebin.com> ... So, here is the file; I don't want
>>         to give the link on the mailing-list :-)
>
>         [removed]
>
>>         I hope this will help understanding our problem's cause :-) ...
>>
>>         Thanks for your help,
>>
>>         Kevin
>>
>>
>>
>>
>>
>>
>>         *
>>         Bien cordialement,
>>         Best Regards,
>>
>>         **Kevin MATHY* |**Ingénieur VoIP
>>         *
>>         *
>>
>>
>>         2014-06-30 16:30 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr
>>         <mailto:k.mathy at hexanet.fr>>:
>>
>>             Hi Bogdan,
>>
>>             Ooops, I thought my two first mails have been cancelled :-)
>>             I prefer waiting till there's no traffic, so I'll send a
>>             SIGUSR1 comme this evening, and reply to this topic with
>>             the log.
>>
>>             I'll try working with MI statistics to make some memory
>>             usage graphs better than with Cacti...
>>
>>             I'll come back to you with logs; thanks for all !
>>
>>             Kevin
>>
>>
>>
>>             *
>>             Bien cordialement,
>>             Best Regards,
>>
>>             **Kevin MATHY* |**Ingénieur VoIP
>>             *
>>             *
>>
>>
>>             2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu
>>             <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>>
>>                 Hi Kevin,
>>
>>                 There is no need to send your email three times ;).
>>                 One time is enough.
>>
>>                 Waiting and taking the dump when there is not traffic
>>                 is good (but not a must) - the idea is to be sure
>>                 that all temporary memory (used for processing the
>>                 current traffic) was freed - so what you still have
>>                 in memory is configuration data or possible leaks.
>>                 If you do not have the luxury of waiting, you can do
>>                 it whenever you can.
>>
>>                 Once again, do not look at the memory usage reported
>>                 by OS - it is irrelevant as OpenSIPS is doing its own
>>                 internal memory management.
>>
>>                 Check the memory usage via MI, see the mem related
>>                 statistics:
>>                 http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11
>>
>>                 Regards,
>>
>>                 Bogdan-Andrei Iancu
>>                 OpenSIPS Founder and Developer
>>                 http://www.opensips-solutions.com
>>
>>                 On 30.06.2014 12:01, Kevin Mathy wrote:
>>>                 Hi Bogdan,
>>>
>>>                 If I want to send a SIGUSR1, may I have to wait 20
>>>                 minutes after the last call ? 20 minutes without any
>>>                 call ? I don't understand well this sentence :
>>>
>>>                     It is highly recommended to do this after
>>>                     waiting about 20 minutes to be sure that as much
>>>                     as possile memory is freed - all temporary
>>>                     memory used during processing is freed by lack
>>>                     of load on the proxy
>>>
>>>
>>>                 Also, last week-end, the traffic reduced a lot, and
>>>                 between last friday, when the free system's memory
>>>                 was around 170M, and this morning, the free memory
>>>                 seems to have increased : this morning, it was
>>>                 around 700M, before the traffic comes back.
>>>
>>>                 So, opensips seems to well free the memory, isn't it ?
>>>
>>>                 Thanks for your help,
>>>
>>>                 Kevin
>>>
>>>
>>>                 *
>>>                 Bien cordialement,
>>>                 Best Regards,
>>>
>>>                 **Kevin MATHY* |**Ingénieur VoIP
>>>                 *
>>>                 *
>>>
>>>
>>>                 2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu
>>>                 <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>>>
>>>                     Hi Kevin,
>>>
>>>                     There is no need to wait for a crash. From time
>>>                     to time, you can send a SIGUSR1 to a worker
>>>                     process (or a process you suspect as running out
>>>                     of pkg mem) -> the process will do a pkg dump to
>>>                     the log.
>>>
>>>                     Also, I would strongly advice upgrading to 1.11
>>>                     (latest LTS) - 1.9 is no longer maintained and
>>>                     there were some fixes in the memory manager
>>>                     since then.
>>>
>>>                     Regards,
>>>
>>>                     Bogdan-Andrei Iancu
>>>                     OpenSIPS Founder and Developer
>>>                     http://www.opensips-solutions.com
>>>
>>>                     On 27.06.2014 10 <tel:27.06.2014%2010>:36, Kevin
>>>                     Mathy wrote:
>>>>                     Hi Bogdan,
>>>>
>>>>                     I've set given options, and now I'm waiting for
>>>>                     a new crash of the service... Where the memdump
>>>>                     will be located ? In another logfile than
>>>>                     opensips.log, or in the same ?
>>>>
>>>>                     Thanks
>>>>
>>>>                     *
>>>>                     Bien cordialement,
>>>>                     Best Regards,
>>>>
>>>>                     **Kevin MATHY* |**Ingénieur VoIP
>>>>                     *
>>>>                     *
>>>>
>>>>
>>>>                     2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu
>>>>                     <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>>>>
>>>>                         Kevin,
>>>>
>>>>                         Restarting should not make you loose
>>>>                         ongoing calls (even if you use the dialog
>>>>                         module), do do not worry on that.
>>>>
>>>>                         Regards,
>>>>
>>>>                         Bogdan-Andrei Iancu
>>>>                         OpenSIPS Founder and Developer
>>>>                         http://www.opensips-solutions.com
>>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20140704/600976dc/attachment-0001.htm>


More information about the Users mailing list