[OpenSIPS-Users] Quest to find memory leak
Răzvan Crainea
razvan at opensips.org
Mon Mar 6 08:18:04 EST 2017
Hi, John!
From the dump you sent, I don't see any leaks. Perhaps some of those
fragments increase over time. Can you make a memory dump after the
server runs some time, like after it gets 100 messages?
Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:
> Here is the dump
> http://pastebin.com/DTEHF5Vc
>
> On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <razvan at opensips.org
> <mailto:razvan at opensips.org>> wrote:
>
> None of the "actions" you are talking about have big impact on
> private memory, but the shared one. Better do the dump and send it
> over to point out what is "eating" memory.
>
> Best regards,
>
> Răzvan Crainea
> OpenSIPS Solutions
> www.opensips-solutions.com <http://www.opensips-solutions.com>
>
> On 03/06/2017 02:39 PM, John Nash wrote:
>> with every call attempt it decreases. I tried some changes by
>> rejecting invite before drouting call (That means after auth ,
>> dispatcher) and found memory is stable but when drouting sends
>> Invite to external gateway and external gateway rejects it. Then
>> this issue happens.
>>
>> Inuse transactions and active dialogs also 0. Somthing wrong
>> happening in handling of failure replies. But apart from
>> use_next_gw and setting some avps for CDR not much going on there.
>>
>> On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea
>> <razvan at opensips.org <mailto:razvan at opensips.org>> wrote:
>>
>> Ok, so it is the first listener for the private IP that
>> leaks. Next, is the memory stabilizing in time? Or it is
>> continously decreasing?
>> Yes, that's how you should make the dump.
>>
>> Best regards,
>>
>> Răzvan Crainea
>> OpenSIPS Solutions
>> www.opensips-solutions.com <http://www.opensips-solutions.com>
>>
>> On 03/06/2017 10:57 AM, John Nash wrote:
>>> Dear Razvan,
>>>
>>> Below is the info on my processes
>>> Process:: ID=0 PID=17351 Type=attendant
>>> Process:: ID=1 PID=17352 Type=MI FIFO
>>> Process:: ID=2 PID=17353 Type=MI Datagram
>>> Process:: ID=3 PID=17354 Type=time_keeper
>>> Process:: ID=4 PID=17355 Type=timer
>>> Process:: ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094
>>> <http://1.1.1.1:9094>
>>> Process:: ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060
>>> <http://1.1.1.1:5060>
>>> Process:: ID=7 PID=17358 Type=SIP receiver
>>> udp:192.168.45.5:5064 <http://192.168.45.5:5064>
>>> Process:: ID=8 PID=17359 Type=Timer handler
>>>
>>> 1.1.1.1 is public IP (I changed). The decrease in memory I
>>> see is for Process:: ID=7 PID=17358 mainly. My call flow is
>>> as following
>>>
>>> - New Invite hits the opensips on 1.1.1.1:9094
>>> <http://1.1.1.1:9094>
>>> - Apart from message validity checks I query DB to check if
>>> its a valid user (Using local cache also there)
>>> - Create dialog, Topology_hiding functions are called along
>>> with some avp population
>>> - Using dispatcher ds_select_domain Call sent to
>>> udp:192.168.45.2:7060 <http://192.168.45.2:7060> (using
>>> force socket). This 192.168.45.2:7060
>>> <http://192.168.45.2:7060> is actually freeswitch
>>> - Call again comes back to opensips on udp:192.168.45.5:5064
>>> <http://192.168.45.5:5064>
>>> - New dialog is created and topology_hiding is called
>>> - Drouting function do_routing is called which tries one
>>> gateway and fails
>>>
>>>
>>> Dump i need to create with memlog=4 memdump=1 right?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea
>>> <razvan at opensips.org <mailto:razvan at opensips.org>> wrote:
>>>
>>> Hi, John!
>>>
>>> Transactions are stored in shared memory, not in the
>>> private one. So the possible leak you are facing its not
>>> related to transactions.
>>> During runtime, OpenSIPS might resize some internal
>>> structures, which may lead to increase memory usage.
>>> However, after a while, these allocations should stabilize.
>>> Can you post the output of the kill -SIGUSR1 on pastebin
>>> so we can take a look? Also, what type of process is the
>>> one you are seeing the leak into? You can find out using
>>> the 'opensipsctl ps' command.
>>>
>>> Best regards,
>>>
>>> Răzvan Crainea
>>> OpenSIPS Solutions
>>> www.opensips-solutions.com
>>> <http://www.opensips-solutions.com>
>>>
>>> On 03/06/2017 09:55 AM, John Nash wrote:
>>>> I am using OpenSIPS (2.1.5 (x86_64/linux)) in
>>>> production. I observed private memory is decreasing
>>>> constantly for one process mainly and ultimately
>>>> leading to memory errors and crash.
>>>>
>>>> To debug this issue I prepared a test server and
>>>> compiled opensips as per
>>>> https://www.opensips.org/Documentation/TroubleShooting-OutOfMem
>>>> <https://www.opensips.org/Documentation/TroubleShooting-OutOfMem>
>>>>
>>>> I made only one single call (which was rejected by
>>>> opensips as it was not authorized user) and I saw
>>>> private free memory decreased. I was hoping since
>>>> transaction is done ideally it should release memory
>>>> and should show me same memory as startup but it did
>>>> not. I verified this with many call attempts and i see
>>>> free memory is always decreasing slowly.
>>>>
>>>> I used kill -SIGUSR1 <lowest pid> to create memory
>>>> dump. But i am unable to make sense of it. It shows log
>>>> like ...
>>>>
>>>> r 6 07:29:19 Server3021 opensips[13276]: Memory status
>>>> (pkg):
>>>> Mar 6 07:29:19 Server3021 opensips[13276]: qm_status
>>>> (0x7f5b8ebba010):
>>>> Mar 6 07:29:19 Server3021 opensips[13276]: heap size=
>>>> 4194304
>>>> Mar 6 07:29:19 Server3021 opensips[13276]: used=
>>>> 346768, used+overhead=848792, free=3345512
>>>> Mar 6 07:29:19 Server3021 opensips[13276]: max used
>>>> (+overhead)= 931920
>>>> Mar 6 07:29:19 Server3021 opensips[13276]: dumping all
>>>> alloc'ed. fragments:
>>>> Mar 6 07:29:19 Server3021 opensips[13276]: 0. N
>>>> address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
>>>> Mar 6 07:29:19 Server3021 opensips[13276]:
>>>> alloc'd from script_cb.c: add_callback(60)
>>>> Mar 6 07:29:19 Server3021 opensips[13276]: start
>>>> check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0,
>>>> abcdefedabcdefed
>>>> Mar 6 07:29:19 Server3021 opensips[13276]: 1. N
>>>> address=0x7f5b8ebef5b0
>>>>
>>>> I pasted only few lines in this mail. What should be my
>>>> next step?...How can i really trace what is wrong in my
>>>> script or any other memory leak?
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>> <http://lists.opensips.org/cgi-bin/mailman/listinfo/users>
>>> _______________________________________________ Users
>>> mailing list Users at lists.opensips.org
>>> <mailto:Users at lists.opensips.org>
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>> <http://lists.opensips.org/cgi-bin/mailman/listinfo/users>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>> <http://lists.opensips.org/cgi-bin/mailman/listinfo/users>
>> _______________________________________________ Users mailing
>> list Users at lists.opensips.org
>> <mailto:Users at lists.opensips.org>
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>> <http://lists.opensips.org/cgi-bin/mailman/listinfo/users>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>> <http://lists.opensips.org/cgi-bin/mailman/listinfo/users>
> _______________________________________________ Users mailing list
> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> <http://lists.opensips.org/cgi-bin/mailman/listinfo/users>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170306/8c57edb4/attachment-0001.html>
More information about the Users
mailing list