[OpenSIPS-Users] OpenSIPS crashed after "out of pkg memory"

Kevin Mathy k.mathy at hexanet.fr
Fri Jul 4 11:41:20 CEST 2014 

Hi Bogdan,

We've recompiled OpenSIPS with requested modules (I think) :

> root at asbc2:/home/kemathy# opensips -V
> version: opensips 1.9.2-notls (x86_64/linux)
> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM,
> SHM_MMAP, PKG_MALLOC, *DBG_QM_MALLOC*, FAST_LOCK-ADAPTIVE_WAIT
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> svnrevision: unknown
> @(#) $Id$
> main.c compiled on 22:12:10 Jul  3 2014 with gcc 4.7


Now we'll keep an eye on our server to check if everything is OK, and if
the memory error still occur; as we upgraded from 1.9.1 to 1.9.2...

I'll get back to you with some logs if needed ;-)

Kevin




*Bien cordialement, Best Regards,  **Kevin MATHY* | Ingénieur VoIP



2014-07-02 9:37 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr>:

> Hi Bogdan,
>
> Hummm, right, opensips doesn't seem to have been compiled with the
> requested modules for memory debugging...
>
> root at asbc2:/home/kemathy# opensips -V
>> version: opensips 1.9.2-notls (x86_64/linux)
>> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM,
>> SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>> svnrevision: unknown
>> @(#) $Id$
>> main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7
>
>
>  So I think I'll have to re-compile opensips with QM_DBG_MALLOC, and try
> again to export the memdump log...
>
> I'll get back to you when done.
>
>  Thanks a lot for your help !
>
> Kevin
>
>
>
> *Bien cordialement,  Best Regards, **Kevin MATHY* | Ingénieur VoIP
>
>
>
> 2014-07-01 17:31 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>
>  Hi Kevin,
>>
>> Unfortunately the logs are not correct - are you sure you properly
>> compiled the mem debug ? like adding the QM_DBG_MALLOC and removing the
>> FM_MALLOC flags ? As the logs show the standard memory manager (without
>> debugging).
>> Check it with "opensips -V" to see the list of compiled flags.
>>
>> I tried to get some ideas by only looking at the available memory and how
>> many fragments were allocated in each process - indeed, there are some
>> processes using maybe like 2 or 3 times more PKG, but not sure if a leak.
>> Getting the proper logs (which will be huge) will tell us more.
>>
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>
>> On 01.07.2014 18:11, Kevin Mathy wrote:
>>
>> Hi Bogdan,
>>
>>  I have now a memdump log, as we restarted opensips this afternoon for a
>> configuration maintenance... But the file is too big, even if I try to put
>> it to pastebin.com ... So, here is the file; I don't want to give the
>> link on the mailing-list :-)
>>
>>
>> [removed]
>>
>>  I hope this will help understanding our problem's cause :-) ...
>>
>>  Thanks for your help,
>>
>>  Kevin
>>
>>
>>
>>
>>
>>
>>
>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>
>>
>>
>> 2014-06-30 16:30 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr>:
>>
>>> Hi Bogdan,
>>>
>>>  Ooops, I thought my two first mails have been cancelled :-)
>>> I prefer waiting till there's no traffic, so I'll send a SIGUSR1 comme
>>> this evening, and reply to this topic with the log.
>>>
>>>  I'll try working with MI statistics to make some memory usage graphs
>>> better than with Cacti...
>>>
>>>  I'll come back to you with logs; thanks for all !
>>>
>>>  Kevin
>>>
>>>
>>>
>>>
>>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>>
>>>
>>>
>>>  2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>>
>>>  Hi Kevin,
>>>>
>>>> There is no need to send your email three times ;). One time is enough.
>>>>
>>>> Waiting and taking the dump when there is not traffic is good (but not
>>>> a must) - the idea is to be sure that all temporary memory (used for
>>>> processing the current traffic) was freed - so what you still have in
>>>> memory is configuration data or possible leaks.
>>>> If you do not have the luxury of waiting, you can do it whenever you
>>>> can.
>>>>
>>>> Once again, do not look at the memory usage reported by OS - it is
>>>> irrelevant as OpenSIPS is doing its own internal memory management.
>>>>
>>>> Check the memory usage via MI, see the mem related statistics:
>>>>
>>>> http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11
>>>>
>>>> Regards,
>>>>
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>
>>>>   On 30.06.2014 12:01, Kevin Mathy wrote:
>>>>
>>>>   Hi Bogdan,
>>>>
>>>>  If I want to send a SIGUSR1, may I have to wait 20 minutes after the
>>>> last call ? 20 minutes without any call ? I don't understand well this
>>>> sentence :
>>>>
>>>>>  It is highly recommended to do this after waiting about 20 minutes
>>>>> to be sure that as much as possile memory is freed - all temporary memory
>>>>> used during processing is freed by lack of load on the proxy
>>>>
>>>>
>>>>  Also, last week-end, the traffic reduced a lot, and between last
>>>> friday, when the free system's memory was around 170M, and this morning,
>>>> the free memory seems to have increased : this morning, it was around 700M,
>>>> before the traffic comes back.
>>>>
>>>>  So, opensips seems to well free the memory, isn't it ?
>>>>
>>>>  Thanks for your help,
>>>>
>>>>  Kevin
>>>>
>>>>
>>>>
>>>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>>>
>>>>
>>>>
>>>> 2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>>>
>>>>>  Hi Kevin,
>>>>>
>>>>> There is no need to wait for a crash. From time to time, you can send
>>>>> a SIGUSR1 to a worker process (or a process you suspect as running out of
>>>>> pkg mem) -> the process will do a pkg dump to the log.
>>>>>
>>>>> Also, I would strongly advice upgrading to 1.11 (latest LTS) - 1.9 is
>>>>> no longer maintained and there were some fixes in the memory manager since
>>>>> then.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Bogdan-Andrei Iancu
>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>>
>>>>>   On 27.06.2014 10 <27.06.2014%2010>:36, Kevin Mathy wrote:
>>>>>
>>>>>  Hi Bogdan,
>>>>>
>>>>>  I've set given options, and now I'm waiting for a new crash of the
>>>>> service... Where the memdump will be located ? In another logfile than
>>>>> opensips.log, or in the same ?
>>>>>
>>>>>  Thanks
>>>>>
>>>>>
>>>>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>>>>
>>>>>
>>>>>
>>>>> 2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>>>>
>>>>>>  Kevin,
>>>>>>
>>>>>> Restarting should not make you loose ongoing calls (even if you use
>>>>>> the dialog module), do do not worry on that.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Bogdan-Andrei Iancu
>>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>

-- 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20140704/c7d3a46d/attachment-0001.htm>

More information about the Users mailing list