[OpenSIPS-Users] Pending OpenSIPS minor releases: Last minute bug fixes!

Liviu Chircu liviu at opensips.org
Wed Oct 12 11:22:18 CEST 2016 

Hi, Agalya!

As you have already noticed, the timeout fix has made its way to 2.2, so 
from my perspective, all should be good now regarding the previous issues.

Your tests are very interesting, but I need more info in order to 
isolate the cause of the crash. Here are some things to consider:

- first and foremost: keep in mind that it's difficult to help debug 
code that I cannot see (REST PUT). If you were to make it crash while 
doing 10,000 REST POSTs, I would be able to jump in and attempt to 
replicate the crash.

- do your workers have enough PKG memory? (use "opensipsctl fifo 
get_statistics pkmem:" during runtime to find out)

- 10,000 calls is a nice number, but what was your CPS rate? Also, how 
many OpenSIPS processes did you run with?

- I noticed you enabled QM_MALLOC, which is good, but did you also 
enable DBG_MALLOC for it? This would help catch any memory corruption 
earliest possible.

Best regards,

Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com

On 10.10.2016 20:09, Ramachandran, Agalya (Contractor) wrote:
> Hi Liviu/team,
>
> There is one item pending from my mail thread - opensips crash when doing load test.
> Please find the details of that message below.
>
> Hi Bogdan,
>
> When I try to apply patch, I see all the code are already available in 2.2. So I did not add any patch.
> But did a code addition for 'REST_PUT' request in rest_client.c, rest_methods.c, rest_methohs.h
> Tried to perform load test. Observed 2 issues.
>
> 1)	Tried to load 10,000 calls - But route[resume_http] is called only for 9985 calls.
> Every time approximately 10-20 calls, route[resume_http]  is not called. But if I see the tcpdump, I am seeing 10,000 HTTP request and 10,000 HTTP 200 OK responses.
> When printing the response in resume_http for every call-id, 10-20 calls response is not printed - which means resume is not called for these calls.
> Am not filtering any response code.
> route[resume_http] {
>          xlog("L_INFO","Resp $rc in HTTP PUT!\n");
>          xlog("L_INFO","route[relay] The content received from SM for $rm: [callId=$ci] : $var(body) in HTTP PUT\n");
>          .....................
> }
>     For first few calls, $rc is 1(ASYNC_DONE), for others $rc is -5(ASYNC_SYNC)
>
> 2)	Tried to load more than 10,000 calls. When it reaches 25,000 calls after that some point opensips crashes.
> I have two core dump generated as of now.
> Coredump1:
> --------------
> (gdb) bt
> #0  0x00007f8defacc840 in Curl_num_addresses () from /lib64/libcurl.so.4
> #1  0x00007f8defaf09ee in Curl_connecthost () from /lib64/libcurl.so.4
> #2  0x00007f8defae34ff in Curl_setup_conn () from /lib64/libcurl.so.4
> #3  0x00007f8defae370c in Curl_connect () from /lib64/libcurl.so.4
> #4  0x00007f8defaf32c0 in multi_runsingle () from /lib64/libcurl.so.4
> #5  0x00007f8defaf4121 in curl_multi_perform () from /lib64/libcurl.so.4
> #6  0x00007f8defd2b65b in start_async_http_req (msg=msg at entry=0x7f8df3740fc0, method=method at entry=REST_CLIENT_PUT,
>      url=0x7f8df36ceb88 "http://test.comcast.net/RTCGSessionManager/rest/tel/session/createroom?", req_body=<optimized out>,
>      req_ctype=<optimized out>, out_handle=out_handle at entry=0x7f8df3743300, body=body at entry=0x7f8df3743308, ctype=0x7f8df3743318)
>      at rest_methods.c:265
> #7  0x00007f8defd31b56 in w_async_rest_put (msg=0x7f8df3740fc0, resume_f=0x7ffd4afd8c90, resume_param=0x7ffd4afd8c88,
>      gp_url=<optimized out>, gp_body=<optimized out>, gp_ctype=<optimized out>, body_pv=0x7f8df36f85a8 "N",
>      ctype_pv=0x7f8df36f8700 "N", code_pv=0x7f8df36f88c0 "N") at rest_client.c:544
> #8  0x00007f8df1031841 in t_handle_async (msg=0x7f8df3740fc0, a=0x7f8df36cf0c8, resume_route=2) at async.c:240
> #9  0x0000000000424056 in do_action (a=0x7f8df36cf2d0, msg=0x7f8df3740fc0) at action.c:1863
> #10 0x000000000041c330 in run_action_list (a=0x7f8df36cf2d0, msg=0x7f8df3740fc0) at action.c:172
> #11 0x0000000000420637 in do_action (a=0x7f8df36cf3f8, msg=0x7f8df3740fc0) at action.c:1108
> #12 0x000000000041c330 in run_action_list (a=0x7f8df36ce568, msg=0x7f8df3740fc0) at action.c:172
> #13 0x000000000041c1fd in run_actions (a=0x7f8df36ce568, msg=0x7f8df3740fc0) at action.c:137
> #14 0x000000000041ed55 in do_action (a=0x7f8df36ca080, msg=0x7f8df3740fc0) at action.c:745
> #15 0x000000000041c330 in run_action_list (a=0x7f8df36c9e78, msg=0x7f8df3740fc0) at action.c:172
> #16 0x0000000000420637 in do_action (a=0x7f8df36ca1a8, msg=0x7f8df3740fc0) at action.c:1108
> #17 0x000000000041c330 in run_action_list (a=0x7f8df36c2560, msg=0x7f8df3740fc0) at action.c:172
> #18 0x000000000041c1fd in run_actions (a=0x7f8df36c2560, msg=0x7f8df3740fc0) at action.c:137
> #19 0x000000000041c3fd in run_top_route (a=0x7f8df36c2560, msg=0x7f8df3740fc0) at action.c:204
> #20 0x000000000042bb64 in receive_msg (
>      buf=0x7eb340 <buf.8031> "INVITE sip:++12001000004 at 10.0.0.1:5060 SIP/2.0\r\nTo: sut <sip:+19086774567 at 10.0.0.0.1:5060;user=phone>\r\nFrom: sipp <sip:sipp at 10.0.0.2:5060;user=phone>;tag=26939SIPpTag0014404\r\nCall-ID: 14"..., len=837,
>      rcv_info=0x7ffd4afdae00, existing_context=0x0) at receive.c:208
> #21 0x0000000000521838 in udp_read_req (si=0x7f8df36bda50, bytes_read=0x7ffd4afdaec8) at net/proto_udp/proto_udp.c:192
> #22 0x000000000050d05b in handle_io (fm=0x7f8df371e128, idx=0, event_type=1) at net/net_udp.c:259
> #23 0x000000000050ba2b in io_wait_loop_epoll (h=0x80be40 <_worker_io>, t=1, repeat=0) at net/../io_wait_loop.h:221
> #24 0x000000000050d3db in udp_rcv_loop (si=0x7f8df36bda50) at net/net_udp.c:311
> #25 0x000000000050d973 in udp_start_processes (chd_rank=0x7d8128 <chd_rank.10725>, startup_done=0x0) at net/net_udp.c:375
> #26 0x0000000000495241 in main_loop () at main.c:671
> #27 0x0000000000497cec in main (argc=3, argv=0x7ffd4afdb1c8) at main.c:1271
>
> Coredump2:
> --------------
> (gdb) bt
> #0  0x00000000004affeb in qm_detach_free (qm=0x7f5aa3ff4010, frag=0x7f5aa40c50f0) at mem/q_malloc.c:281
> #1  0x00000000004b1233 in qm_free (qm=0x7f5aa3ff4010, p=0x7f5aa40c50b0, file=0x7f5aa069fbf5 "rest_client.c",
>      func=0x7f5aa06a01fd <__FUNCTION__.8581> "osips_free", line=205) at mem/q_malloc.c:494
> #2  0x00007f5aa04774f6 in curl_thread_create_thunk () from /lib64/libcurl.so.4
> #3  0x00007f5a9edb9dc5 in start_thread () from /lib64/libpthread.so.0
> #4  0x00007f5aa42eb28d in clone () from /lib64/libc.so.6
>
> Regards,
> Agalya
>
>
>
> -----Original Message-----
> From: users-bounces at lists.opensips.org [mailto:users-bounces at lists.opensips.org] On Behalf Of Liviu Chircu
> Sent: Monday, October 10, 2016 4:53 AM
> To: OpenSIPS devel mailling list <devel at lists.opensips.org>; OpenSIPS users mailling list <users at lists.opensips.org>
> Subject: [OpenSIPS-Users] Pending OpenSIPS minor releases: Last minute bug fixes!
>
> Hi, everyone!
>
> A new series of OpenSIPS minor releases is planned for next week, October 19th.
>
> If you have any pending GitHub issues / mailing list bug-threads in OpenSIPS 1.11, 2.1 or 2.2 which are still not resolved, this would be a good time to bump them!
>
> Best regards,
>
> --
> Liviu Chircu
> OpenSIPS Developer
> http://www.opensips-solutions.com
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>

More information about the Users mailing list