[OpenSIPS-Users] OpenSIPS crashed after "out of pkg memory"

Wed Jul 2 09:37:16 CEST 2014

Hi Bogdan,

Hummm, right, opensips doesn't seem to have been compiled with the
requested modules for memory debugging...

root at asbc2:/home/kemathy# opensips -V
> version: opensips 1.9.2-notls (x86_64/linux)
> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM,
> SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> svnrevision: unknown
> @(#) $Id$
> main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7

 So I think I'll have to re-compile opensips with QM_DBG_MALLOC, and try
again to export the memdump log...

I'll get back to you when done.

Thanks a lot for your help !

Kevin

*Bien cordialement,  Best Regards, **Kevin MATHY* | Ingénieur VoIP

2014-07-01 17:31 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:

>  Hi Kevin,
>
> Unfortunately the logs are not correct - are you sure you properly
> compiled the mem debug ? like adding the QM_DBG_MALLOC and removing the
> FM_MALLOC flags ? As the logs show the standard memory manager (without
> debugging).
> Check it with "opensips -V" to see the list of compiled flags.
>
> I tried to get some ideas by only looking at the available memory and how
> many fragments were allocated in each process - indeed, there are some
> processes using maybe like 2 or 3 times more PKG, but not sure if a leak.
> Getting the proper logs (which will be huge) will tell us more.
>
>
> Regards,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>
> On 01.07.2014 18:11, Kevin Mathy wrote:
>
> Hi Bogdan,
>
>  I have now a memdump log, as we restarted opensips this afternoon for a
> configuration maintenance... But the file is too big, even if I try to put
> it to pastebin.com ... So, here is the file; I don't want to give the
> link on the mailing-list :-)
>
>
> [removed]
>
>  I hope this will help understanding our problem's cause :-) ...
>
>  Thanks for your help,
>
>  Kevin
>
>
>
>
>
>
>
> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>
>
>
> 2014-06-30 16:30 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr>:
>
>> Hi Bogdan,
>>
>>  Ooops, I thought my two first mails have been cancelled :-)
>> I prefer waiting till there's no traffic, so I'll send a SIGUSR1 comme
>> this evening, and reply to this topic with the log.
>>
>>  I'll try working with MI statistics to make some memory usage graphs
>> better than with Cacti...
>>
>>  I'll come back to you with logs; thanks for all !
>>
>>  Kevin
>>
>>
>>
>>
>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>
>>
>>
>>  2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>
>>  Hi Kevin,
>>>
>>> There is no need to send your email three times ;). One time is enough.
>>>
>>> Waiting and taking the dump when there is not traffic is good (but not a
>>> must) - the idea is to be sure that all temporary memory (used for
>>> processing the current traffic) was freed - so what you still have in
>>> memory is configuration data or possible leaks.
>>> If you do not have the luxury of waiting, you can do it whenever you can.
>>>
>>> Once again, do not look at the memory usage reported by OS - it is
>>> irrelevant as OpenSIPS is doing its own internal memory management.
>>>
>>> Check the memory usage via MI, see the mem related statistics:
>>>      http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11
>>>
>>> Regards,
>>>
>>> Bogdan-Andrei Iancu
>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>
>>>   On 30.06.2014 12:01, Kevin Mathy wrote:
>>>
>>>   Hi Bogdan,
>>>
>>>  If I want to send a SIGUSR1, may I have to wait 20 minutes after the
>>> last call ? 20 minutes without any call ? I don't understand well this
>>> sentence :
>>>
>>>>  It is highly recommended to do this after waiting about 20 minutes to
>>>> be sure that as much as possile memory is freed - all temporary memory used
>>>> during processing is freed by lack of load on the proxy
>>>
>>>
>>>  Also, last week-end, the traffic reduced a lot, and between last
>>> friday, when the free system's memory was around 170M, and this morning,
>>> the free memory seems to have increased : this morning, it was around 700M,
>>> before the traffic comes back.
>>>
>>>  So, opensips seems to well free the memory, isn't it ?
>>>
>>>  Thanks for your help,
>>>
>>>  Kevin
>>>
>>>
>>>
>>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>>
>>>
>>>
>>> 2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>>
>>>>  Hi Kevin,
>>>>
>>>> There is no need to wait for a crash. From time to time, you can send a
>>>> SIGUSR1 to a worker process (or a process you suspect as running out of pkg
>>>> mem) -> the process will do a pkg dump to the log.
>>>>
>>>> Also, I would strongly advice upgrading to 1.11 (latest LTS) - 1.9 is
>>>> no longer maintained and there were some fixes in the memory manager since
>>>> then.
>>>>
>>>> Regards,
>>>>
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>
>>>>   On 27.06.2014 10 <27.06.2014%2010>:36, Kevin Mathy wrote:
>>>>
>>>>  Hi Bogdan,
>>>>
>>>>  I've set given options, and now I'm waiting for a new crash of the
>>>> service... Where the memdump will be located ? In another logfile than
>>>> opensips.log, or in the same ?
>>>>
>>>>  Thanks
>>>>
>>>>
>>>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>>>
>>>>
>>>>
>>>> 2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>>>
>>>>>  Kevin,
>>>>>
>>>>> Restarting should not make you loose ongoing calls (even if you use
>>>>> the dialog module), do do not worry on that.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Bogdan-Andrei Iancu
>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>

-- 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20140702/aa9a488d/attachment-0001.htm>