[OpenSIPS-Users] Quest to find memory leak

Mon Mar 6 08:27:07 EST 2017

Ok will try that. Is it possible that wrong usage of drouting may cause
this to happen instead of actual leak?... What are the things private
memory is used for?

On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <razvan at opensips.org> wrote:

> Hi, John!
>
> From the dump you sent, I don't see any leaks. Perhaps some of those
> fragments increase over time. Can you make a memory dump after the server
> runs some time, like after it gets 100 messages?
>
> Best regards,
>
> Răzvan Crainea
> OpenSIPS Solutionswww.opensips-solutions.com
>
> On 03/06/2017 03:02 PM, John Nash wrote:
>
> Here is the dump
> http://pastebin.com/DTEHF5Vc
>
> On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <razvan at opensips.org>
> wrote:
>
>> None of the "actions" you are talking about have big impact on private
>> memory, but the shared one. Better do the dump and send it over to point
>> out what is "eating" memory.
>>
>> Best regards,
>>
>> Răzvan Crainea
>> OpenSIPS Solutionswww.opensips-solutions.com
>>
>> On 03/06/2017 02:39 PM, John Nash wrote:
>>
>> with every call attempt it decreases. I tried some changes by rejecting
>> invite before drouting call (That means after auth , dispatcher) and found
>> memory is stable but when drouting sends Invite to external gateway and
>> external gateway rejects it. Then this issue happens.
>>
>> Inuse transactions and active dialogs also 0. Somthing wrong happening in
>> handling of failure replies. But apart from use_next_gw and setting some
>> avps for CDR not much going on there.
>>
>> On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <razvan at opensips.org>
>> wrote:
>>
>>> Ok, so it is the first listener for the private IP that leaks. Next, is
>>> the memory stabilizing in time? Or it is continously decreasing?
>>> Yes, that's how you should make the dump.
>>>
>>> Best regards,
>>>
>>> Răzvan Crainea
>>> OpenSIPS Solutionswww.opensips-solutions.com
>>>
>>> On 03/06/2017 10:57 AM, John Nash wrote:
>>>
>>> Dear Razvan,
>>>
>>> Below is the info on my processes
>>> Process::  ID=0 PID=17351 Type=attendant
>>> Process::  ID=1 PID=17352 Type=MI FIFO
>>> Process::  ID=2 PID=17353 Type=MI Datagram
>>> Process::  ID=3 PID=17354 Type=time_keeper
>>> Process::  ID=4 PID=17355 Type=timer
>>> Process::  ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094
>>> Process::  ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060
>>> Process::  ID=7 PID=17358 Type=SIP receiver udp:192.168.45.5:5064
>>> Process::  ID=8 PID=17359 Type=Timer handler
>>>
>>> 1.1.1.1 is public IP (I changed). The decrease in memory I see is for
>>> Process::  ID=7 PID=17358 mainly. My call flow is as following
>>>
>>> - New Invite hits the opensips on 1.1.1.1:9094
>>> - Apart from message validity checks I query DB to check if its a valid
>>> user (Using local cache also there)
>>> - Create dialog, Topology_hiding functions are called along with some
>>> avp population
>>> - Using dispatcher ds_select_domain Call sent to udp:192.168.45.2:7060
>>> (using force socket). This 192.168.45.2:7060 is actually freeswitch
>>> - Call again comes back to opensips on udp:192.168.45.5:5064
>>> - New dialog is created and topology_hiding is called
>>> - Drouting function do_routing is called which tries one gateway and
>>> fails
>>>
>>>
>>> Dump i need to create with memlog=4 memdump=1 right?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea <razvan at opensips.org>
>>> wrote:
>>>
>>>> Hi, John!
>>>>
>>>> Transactions are stored in shared memory, not in the private one. So
>>>> the possible leak you are facing its not related to transactions.
>>>> During runtime, OpenSIPS might resize some internal structures, which
>>>> may lead to increase memory usage. However, after a while, these
>>>> allocations should stabilize .
>>>> Can you post the output of the kill -SIGUSR1 on pastebin so we can take
>>>> a look? Also, what type of process is the one you are seeing the leak into?
>>>> You can find out using the 'opensipsctl ps' command.
>>>>
>>>> Best regards,
>>>>
>>>> Răzvan Crainea
>>>> OpenSIPS Solutionswww.opensips-solutions.com
>>>>
>>>> On 03/06/2017 09:55 AM, John Nash wrote:
>>>>
>>>> I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed
>>>> private memory is decreasing constantly for one process mainly and
>>>> ultimately leading to memory errors and crash.
>>>>
>>>> To debug this issue I prepared a test server and compiled opensips as
>>>> per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem
>>>>
>>>> I made only one single call (which was rejected by opensips as it was
>>>> not authorized user) and I saw private free memory decreased. I was hoping
>>>> since transaction is done ideally it should release memory and should show
>>>> me same memory as startup but it did not. I verified this with many call
>>>> attempts and i see free memory is always decreasing slowly.
>>>>
>>>> I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am
>>>> unable to make sense of it. It shows log like ...
>>>>
>>>> r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
>>>> Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
>>>> Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
>>>> Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768,
>>>> used+overhead=848792, free=3345512
>>>> Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)=
>>>> 931920
>>>> Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed.
>>>> fragments:
>>>> Mar  6 07:29:19 Server3021 opensips[13276]:       0. N
>>>>  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
>>>> Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from
>>>> script_cb.c: add_callback(60)
>>>> Mar  6 07:29:19 Server3021 opensips[13276]:         start
>>>> check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
>>>> Mar  6 07:29:19 Server3021 opensips[13276]:       1. N
>>>>  address=0x7f5b8ebef5b0
>>>>
>>>> I pasted only few lines in this mail. What should be my next
>>>> step?...How can i really trace what is wrong in my script or any other
>>>> memory leak?
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>> _______________________________________________ Users mailing list
>>>> Users at lists.opensips.org http://lists.opensips.org/cgi-
>>>> bin/mailman/listinfo/users
>>>
>>> _______________________________________________
>>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>> _______________________________________________ Users mailing list
>>> Users at lists.opensips.org http://lists.opensips.org/cgi-
>>> bin/mailman/listinfo/users
>>
>> _______________________________________________
>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>> _______________________________________________ Users mailing list
>> Users at lists.opensips.org http://lists.opensips.org/cgi-
>> bin/mailman/listinfo/users
>
> _______________________________________________
> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170306/087204f4/attachment-0001.html>