[OpenSIPS-Users] Memory leak
Bogdan-Andrei Iancu
bogdan at opensips.org
Wed Jan 19 07:54:59 UTC 2022
Better don't :), as we never explored all the possible side effects of
such combination of versions....
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
OpenSIPS eBootcamp 2021
https://opensips.org/training/OpenSIPS_eBootcamp_2021/
On 1/19/22 9:45 AM, Schneur Rosenberg wrote:
> That's a given the question is can it mess up the active server if the
> passive one was updated already? I'm not planning on leaving it like
> that I just like to do the upgrades slowly, one at a time and test it
> and only then to upgrade the second one.
>
> Scott
>
> On Wed, Jan 19, 2022, 09:26 Bogdan-Andrei Iancu <bogdan at opensips.org
> <mailto:bogdan at opensips.org>> wrote:
>
> Hi Schneur,
>
> It is strongly recommend that all OpenSIPS nodes in a cluster to have
> the same version.
>
> Best regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
> https://www.opensips-solutions.com
> <https://www.opensips-solutions.com>
> OpenSIPS eBootcamp 2021
> https://opensips.org/training/OpenSIPS_eBootcamp_2021/
> <https://opensips.org/training/OpenSIPS_eBootcamp_2021/>
>
> On 1/18/22 6:08 PM, Schneur Rosenberg wrote:
> > Hi, it seems like it was fixed in 3.2, I will have to migrate all my
> > servers, I use binary replication will it break if one server is
> > running 2.4 and the other 3.2? its a active/passive setup so I will
> > take down one at a time and upgrade it, I'm just worried what will
> > happen while one is 3.2 and the second one is still 2.4, in the
> past I
> > disabled the replication while I was doing the updates and I'm
> > wondering if its necessary.
> >
> > thanks
> > Scott (Schneur)
> >
> > On Fri, Dec 17, 2021 at 6:21 PM Bogdan-Andrei Iancu
> <bogdan at opensips.org <mailto:bogdan at opensips.org>> wrote:
> >> While trying to reproduce (as I failed to do so), I noticed you
> mentioned this is on version 2.4.11, right ? As I was testing on
> 3.2 without getting the leak.
> >>
> >> Could you try on 3.2/3.1 ? Keep in mind 2.4 is not maintained
> anymore :(
> >>
> >> Regards,
> >>
> >> Bogdan-Andrei Iancu
> >>
> >> OpenSIPS Founder and Developer
> >> https://www.opensips-solutions.com
> <https://www.opensips-solutions.com>
> >> OpenSIPS eBootcamp 2021
> >> https://opensips.org/training/OpenSIPS_eBootcamp_2021/
> <https://opensips.org/training/OpenSIPS_eBootcamp_2021/>
> >>
> >> On 12/17/21 1:40 PM, Schneur Rosenberg wrote:
> >>
> >> Thanks Bogdan!, this is my entire local_route, all my dst_uri's
> are IP only.
> >>
> >> On Fri, Dec 17, 2021, 12:36 Bogdan-Andrei Iancu
> <bogdan at opensips.org <mailto:bogdan at opensips.org>> wrote:
> >>> Hi Schneur,
> >>>
> >>> I suspect that the leaking mk_proxy is related to the changing
> of the
> >>> RURI in local route. Let me test your snippet. BTW, is that
> the whole
> >>> processing you do in local route? is the $rd (from LB) a FQDN or
> >>> straight IP ?
> >>>
> >>> Regards,
> >>>
> >>> Bogdan-Andrei Iancu
> >>>
> >>> OpenSIPS Founder and Developer
> >>> https://www.opensips-solutions.com
> <https://www.opensips-solutions.com>
> >>> OpenSIPS eBootcamp 2021
> >>> https://opensips.org/training/OpenSIPS_eBootcamp_2021/
> <https://opensips.org/training/OpenSIPS_eBootcamp_2021/>
> >>>
> >>> On 12/16/21 9:43 AM, Schneur Rosenberg wrote:
> >>>> Hi Bogdan
> >>>>
> >>>> I think I found the issue, I recently added these lines of code,
> >>>> because of a probing issue I was having, I just searched from my
> >>>> previous tickets and I see that you have warned me about the
> >>>> implications but for some reason I never read the message.
> >>>>
> >>>> Here is the old ticket
> >>>>
> https://www.mail-archive.com/users@lists.opensips.org/msg43301.html
> <https://www.mail-archive.com/users@lists.opensips.org/msg43301.html>
> >>>> the reason I'm using INVITE to probe is because I want the
> servers
> >>>> that were probed not only to respond but also check if the
> database is
> >>>> working, I did it this way because I had cases where mysql
> crashed but
> >>>> my asterisk servers were still responding to the probe but
> all of the
> >>>> calls just hung, so I do a invite and it does a DB lookup and
> it will
> >>>> only return a positive message if it was able to query the
> DB, do you
> >>>> have a better solution? at the time I set it up I couldn't
> run a query
> >>>> on receipt of a OPTIONS but perhaps I didn't look good enough
> :-),
> >>>> either way can I do anything to make sure this code doesn't leak
> >>>> memory? this probing has worked for years until I needed the
> Contact
> >>>> header.
> >>>>
> >>>> local_route {
> >>>> if (is_method("INVITE")&& $fU=="pingTest"){
> >>>> $ru="sip:s@"+$rd ;
> >>>> append_hf("Contact: <sip:pingTest@$fd:5060>\r\n");
> >>>> exit;
> >>>> }
> >>>> }
> >>>>
> >>>> On Fri, Dec 10, 2021 at 2:16 PM Schneur Rosenberg
> >>>> <rosenberg11219 at gmail.com <mailto:rosenberg11219 at gmail.com>>
> wrote:
> >>>>> Hi Bogdan,
> >>>>>
> >>>>> I did it on a backup server, its also leaking memory but at
> a slower
> >>>>> pace, I'm attaching the logs when running kill -SIGUSR1 on
> the pid
> >>>>> that's growing in size, it still has available memory, I hop
> this will
> >>>>> give you a clue.
> >>>>>
> >>>>> Here is a pastbin to the loggs https://pastebin.com/KJVb9Y75
> <https://pastebin.com/KJVb9Y75>
> >>>>>
> >>>>> On Fri, Dec 10, 2021 at 11:00 AM Schneur Rosenberg
> >>>>> <rosenberg11219 at gmail.com <mailto:rosenberg11219 at gmail.com>>
> wrote:
> >>>>>> Thank you, does this reduce performance? can I leave it
> enabled on a
> >>>>>> production machine? I will wait for the memory leak to be
> apparent and
> >>>>>> I'll post the result.
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Dec 9, 2021 at 12:31 PM Bogdan-Andrei Iancu
> <bogdan at opensips.org <mailto:bogdan at opensips.org>> wrote:
> >>>>>>> Hi Schneur,
> >>>>>>>
> >>>>>>> Just follow the
> >>>>>>>
> https://www.opensips.org/Documentation/TroubleShooting-OutOfMem
> <https://www.opensips.org/Documentation/TroubleShooting-OutOfMem> and
> >>>>>>> provide the dump. This is the only way to investigate this.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Bogdan-Andrei Iancu
> >>>>>>>
> >>>>>>> OpenSIPS Founder and Developer
> >>>>>>> https://www.opensips-solutions.com
> <https://www.opensips-solutions.com>
> >>>>>>> OpenSIPS eBootcamp 2021
> >>>>>>> https://opensips.org/training/OpenSIPS_eBootcamp_2021/
> <https://opensips.org/training/OpenSIPS_eBootcamp_2021/>
> >>>>>>>
> >>>>>>> On 12/8/21 12:14 PM, Schneur Rosenberg wrote:
> >>>>>>>> I just noticed that process 88 runs the timer handler,
> perhaps this
> >>>>>>>> might shed light on whats going on.
> >>>>>>>>
> >>>>>>>> opensipsctl fifo ps
> >>>>>>>> Process:: ID=88 PID=5327 Type=Timer handler
> >>>>>>>>
> >>>>>>>> On Wed, Dec 8, 2021 at 10:55 AM Schneur Rosenberg
> >>>>>>>> <rosenberg11219 at gmail.com
> <mailto:rosenberg11219 at gmail.com>> wrote:
> >>>>>>>>> Now a few hours later this is what I'm getting
> >>>>>>>>> Dec 8 09:50:13 /sbin/opensips[21699]:
> ERROR:nathelper:nh_timer: out
> >>>>>>>>> of pkg memory
> >>>>>>>>> Dec 8 09:50:16 /sbin/opensips[21699]:
> WARNING:core:fm_malloc: not
> >>>>>>>>> enough continuous free pkg memory (3024 bytes left, need
> 5128),
> >>>>>>>>> attempting defragmentation... please increase the "-M"
> command line
> >>>>>>>>> parameter!
> >>>>>>>>> Dec 8 09:50:16 /sbin/opensips[21699]:
> ERROR:core:fm_malloc: not
> >>>>>>>>> enough free pkg memory (3024 bytes left, need 5128),
> please increase
> >>>>>>>>> the "-M" command line parameter!
> >>>>>>>>>
> >>>>>>>>> Here is the last 20 package memory max_used_size
> >>>>>>>>> pkmem:70-max_used_size:: 1009584
> >>>>>>>>> pkmem:71-max_used_size:: 1009584
> >>>>>>>>> pkmem:72-max_used_size:: 1009584
> >>>>>>>>> pkmem:73-max_used_size:: 1009584
> >>>>>>>>> pkmem:74-max_used_size:: 1009584
> >>>>>>>>> pkmem:75-max_used_size:: 1009584
> >>>>>>>>> pkmem:76-max_used_size:: 1009584
> >>>>>>>>> pkmem:77-max_used_size:: 1009584
> >>>>>>>>> pkmem:78-max_used_size:: 1009584
> >>>>>>>>> pkmem:79-max_used_size:: 1009584
> >>>>>>>>> pkmem:80-max_used_size:: 1044752
> >>>>>>>>> pkmem:81-max_used_size:: 1075552
> >>>>>>>>> pkmem:82-max_used_size:: 1116848
> >>>>>>>>> pkmem:83-max_used_size:: 1117456
> >>>>>>>>> pkmem:84-max_used_size:: 1102640
> >>>>>>>>> pkmem:85-max_used_size:: 1306992
> >>>>>>>>> pkmem:86-max_used_size:: 1706304
> >>>>>>>>> pkmem:87-max_used_size:: 2507000
> >>>>>>>>> pkmem:88-max_used_size:: 4194264
> >>>>>>>>> pkmem:89-max_used_size:: 1009584
> >>>>>>>>>
> >>>>>>>>> And here is the real used size, you can see that process
> 88 maxed out
> >>>>>>>>> pkmem:69-real_used_size:: 975528
> >>>>>>>>> pkmem:70-real_used_size:: 978016
> >>>>>>>>> pkmem:71-real_used_size:: 989592
> >>>>>>>>> pkmem:72-real_used_size:: 951416
> >>>>>>>>> pkmem:73-real_used_size:: 982496
> >>>>>>>>> pkmem:74-real_used_size:: 965744
> >>>>>>>>> pkmem:75-real_used_size:: 959424
> >>>>>>>>> pkmem:76-real_used_size:: 949472
> >>>>>>>>> pkmem:77-real_used_size:: 983080
> >>>>>>>>> pkmem:78-real_used_size:: 961400
> >>>>>>>>> pkmem:79-real_used_size:: 977808
> >>>>>>>>> pkmem:80-real_used_size:: 978928
> >>>>>>>>> pkmem:81-real_used_size:: 1009936
> >>>>>>>>> pkmem:82-real_used_size:: 1110760
> >>>>>>>>> pkmem:83-real_used_size:: 1116720
> >>>>>>>>> pkmem:84-real_used_size:: 1096568
> >>>>>>>>> pkmem:85-real_used_size:: 1300592
> >>>>>>>>> pkmem:86-real_used_size:: 1699648
> >>>>>>>>> pkmem:87-real_used_size:: 2501096
> >>>>>>>>> pkmem:88-real_used_size:: 4191280
> >>>>>>>>> pkmem:89-real_used_size:: 882528
> >>>>>>>>>
> >>>>>>>>> On Tue, Dec 7, 2021 at 7:53 PM Schneur Rosenberg
> >>>>>>>>> <rosenberg11219 at gmail.com
> <mailto:rosenberg11219 at gmail.com>> wrote:
> >>>>>>>>>> Hi, lately I'm getting these errors in my logs.
> >>>>>>>>>>
> >>>>>>>>>> ERROR:core:fm_malloc: not enough free pkg memory (1792
> bytes left,
> >>>>>>>>>> need 2184), please increase the "-M" command line para
> >>>>>>>>>> meter!
> >>>>>>>>>>
> >>>>>>>>>> CRITICAL:core:hostent_cpy: pkg memory allocation failure
> >>>>>>>>>>
> >>>>>>>>>> ERROR:nathelper:nh_timer: out of pkg memory
> >>>>>>>>>>
> >>>>>>>>>> ERROR:core:fm_malloc: not enough free pkg memory (5952
> bytes left,
> >>>>>>>>>> need 5408), please increase the "-M" command line para
> >>>>>>>>>> meter!
> >>>>>>>>>>
> >>>>>>>>>> I was on version 2.4.8 and I upgraded to 2.4.11 and I'm
> monitoring the
> >>>>>>>>>> max_used_size of the package memory, a few hours later
> I see that 2
> >>>>>>>>>> processes keep on getting bigger, so far the rest are
> pretty stable, I
> >>>>>>>>>> have 90 processes and 87 and 88 are growing.
> >>>>>>>>>>
> >>>>>>>>>> here you can see the last few processes, OpenSIPS set
> aside 4 mb per process.
> >>>>>>>>>>
> >>>>>>>>>> pkmem:80-max_used_size:: 1009584
> >>>>>>>>>> pkmem:81-max_used_size:: 1009584
> >>>>>>>>>> pkmem:82-max_used_size:: 1009584
> >>>>>>>>>> pkmem:83-max_used_size:: 1009584
> >>>>>>>>>> pkmem:84-max_used_size:: 1009584
> >>>>>>>>>> pkmem:85-max_used_size:: 1009584
> >>>>>>>>>> pkmem:86-max_used_size:: 1143608
> >>>>>>>>>> pkmem:87-max_used_size:: 1323256
> >>>>>>>>>> pkmem:88-max_used_size:: 1831928
> >>>>>>>>>> pkmem:89-max_used_size:: 1009584
> >>>>>>>>>>
> >>>>>>>>>> Any hints where to start looking besides the solutions
> fund here.
> >>>>>>>>>>
> >>>>>>>>>>
> https://www.opensips.org/Documentation/TroubleShooting-OutOfMem
> <https://www.opensips.org/Documentation/TroubleShooting-OutOfMem>
> >>>>>>>>>>
> >>>>>>>>>> thank you
> >>>>>>>>>> Scott
> >>>>>>>> _______________________________________________
> >>>>>>>> Users mailing list
> >>>>>>>> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
> >>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> <http://lists.opensips.org/cgi-bin/mailman/listinfo/users>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20220119/56392f19/attachment-0001.html>
More information about the Users
mailing list