[OpenSIPS-Users] Troubleshooting Memory Leak 1.7 R9225+

Fri Oct 26 17:06:05 CEST 2012

Greetings list,

We're trying to track down a memory leak with 1.7 R9225+ (It's actually
been happening since we've upgraded to 1.7 in general, but we've updated
svn a few times and recompiled to see if the problem goes away).

Here's a diagram of memory usage over a period of time:

http://i45.tinypic.com/ngpqwp.png

Basically we're seeing a package memory leak where package memory is never
freed under negligible load.  The memory used graph looks like a step
graph, where flat correlates to a period where we're off peak with traffic.
 This will continue for 2-3 days before we see opensips processes start to
hang trying to free package memory.  At this point we force an HA restart
and the memory dips.

##########################

2012-10-12T18:05:35.569400+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27800]: ERROR:core:parse_headers: pkg
memory allocation failed
2012-10-12T18:05:35.569439+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27800]: ERROR:core:receive_msg: parse_msg
failed
2012-10-12T18:05:35.569871+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27795]: ERROR:core:receive_msg: no pkg
mem left for sip_msg
2012-10-12T18:05:35.587197+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27805]: [REQUEST
d9c7633662d475f58263856ca5b91efe at 10.2.1.173] :: Beginning processing
OPTIONS for d9c7633662d475f58263856ca5b91efe at 10.2.1.173 from 10.2.1.173
2012-10-12T18:05:35.590072+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27798]: ERROR:core:receive_msg: no pkg
mem left for sip_msg
2012-10-12T18:05:35.590094+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27798]: ERROR:core:receive_msg: no pkg
mem left for sip_msg
2012-10-12T18:05:35.590579+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: ERROR:core:get_hdr_field: out of
pkg memory
2012-10-12T18:05:35.590600+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: INFO:core:parse_headers: bad
header field
2012-10-12T18:05:35.590618+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: ERROR:core:pv_get_callid: cannot
parse Call-Id header

2012-10-12T18:05:35.791241+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]:
ERROR:core:build_res_buf_from_sip_req: parse_headers failed
2012-10-12T18:05:35.791258+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: ERROR:maxfwd:is_maxfwd_present:
parsing MAX_FORWARD header failed!
2012-10-12T18:05:35.791280+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]: ERROR:sl:sl_send_reply_helper:
response building failed
2012-10-12T18:05:35.791295+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]: ERROR:core:get_hdr_field: bad via
2012-10-12T18:05:35.791309+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: ERROR:core:get_hdr_field: out of
pkg memory
2012-10-12T18:05:35.791324+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]: INFO:core:parse_headers: bad
header field
2012-10-12T18:05:35.791338+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: INFO:core:parse_headers: bad
header field
2012-10-12T18:05:35.791354+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: ERROR:tm:t_newtran: parse_headers
failed
2012-10-12T18:05:35.791366+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]: ERROR:core:parse_via: no pkg
memory left
2012-10-12T18:05:35.791376+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: ERROR:core:get_hdr_field: out of
pkg memory
2012-10-12T18:05:35.791384+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: INFO:core:parse_headers: bad
header field
2012-10-12T18:05:35.791392+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]: ERROR:core:get_hdr_field: bad via
2012-10-12T18:05:35.791401+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: ERROR:core:get_hdr_field: out of
pkg memory
2012-10-12T18:05:35.791409+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]: INFO:core:parse_headers: bad
header field
2012-10-12T18:05:35.791418+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: INFO:core:parse_headers: bad
header field
2012-10-12T18:05:35.791426+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]:
ERROR:core:build_res_buf_from_sip_req: parse_headers failed
2012-10-12T18:05:35.791436+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
ERROR:core:build_res_buf_from_sip_req: parse_headers failed
2012-10-12T18:05:35.791443+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]: ERROR:sl:sl_send_reply_helper:
response building failed
2012-10-12T18:05:35.791452+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]: ERROR:sl:sl_send_reply_helper:
response building failed
2012-10-12T18:05:35.791470+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27797]: ERROR:core:receive_msg: no pkg
mem left for sip_msg
2012-10-12T18:05:35.796399+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27808]: [REQUEST
a1db91ba-bfb8788d at 192.168.1.66] :: Beginning processing BYE for
a1db91ba-bfb8788d at 192.168.1.66 from 10.2.1.178
2012-10-12T18:05:35.796427+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27807]: [REQUEST
a5b2db42-50208983 at 192.168.1.12] :: Beginning processing BYE for
a5b2db42-50208983 at 192.168.1.12 from 10.2.1.175
2012-10-12T18:05:35.796550+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27804]: [REQUEST
149befeef9bb0c032d5ec2d34b0cf287 at 10.2.1.195] :: Beginning processing BYE
for 149befeef9bb0c032d5ec2d34b0cf287 at 10.2.1.195 from 10.2.1.195
2012-10-12T18:05:35.796749+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27803]: CRITICAL:core:qm_free: freeing
already freed pointer, first free: dlg_hash.c: free_dlg_dlg(177) - aborting
2012-10-12T18:05:35.813460+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]: ERROR:core:get_hdr_field: out of
pkg memory
2012-10-12T18:05:35.813489+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]: INFO:core:parse_headers: bad
header field
2012-10-12T18:05:35.813508+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]: ERROR:core:pv_get_callid: cannot
parse Call-Id header
2012-10-12T18:05:35.813542+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]: ERROR:core:get_hdr_field: out of
pkg memory
2012-10-12T18:05:35.813578+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]: INFO:core:parse_headers: bad
header field
2012-10-12T18:05:35.813601+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]: ERROR:core:pv_get_callid: cannot
parse Call-Id header
2012-10-12T18:05:35.813638+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]: [REQUEST <null>] :: Beginning
processing INVITE for <null> from X.X.X.X

###############

We've tried increasing the pkg mem from 32 to 64 MB per process.  However,
we are also using a set of 1.6 proxies with the same configuration, and
these regularly approach 32 MB of memory but free memory during our "off"
periods, so we never encounter a leak there.

We have some memory logs with mem debugging support compiled and enabled,
is there a place we can upload these and have someone take a look to try
and figure out where the memory leak might be coming from?

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20121026/815aeab3/attachment.htm>