[OpenSIPS-Users] sched_yield()
Alex Massover
alex at jajah.com
Fri Jan 22 14:04:40 CET 2010
Thanks!
I'll try it.
Some update meanwhile, children=1 definitely behaves better for me than 8, on 4-way SMP.
8 - hang, 1 - doesn't hang.
Interesting...
--
Best Regards,
Alex Massover
VoIP R&D TL
Jajah Inc.
> -----Original Message-----
> From: users-bounces at lists.opensips.org [mailto:users-
> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
> Sent: Friday, January 22, 2010 2:22 PM
> To: OpenSIPS users mailling list
> Subject: Re: [OpenSIPS-Users] sched_yield()
>
>
> To determine if running on vmware is a problem you could try to test it
> on a real machine and see.
>
> And just to make sure that f_malloc is not the issue you could try
> getting an older version of the mem folder ( either one from revision
> r6274 or one from 1.5 branch)
>
>
> Alex Massover wrote:
> > Hi,
> >
> > Unfortunately 'fifo get_statistics' crashes opensips, I opened a bug.
> > But no chance that 1G is not enough, only about 400M is used for all
> linux processes:
> > Mem: 3115120k total, 398360k used, 2716760k free, 536k
> buffers
> >
> > Maybe sched_yield() just cause problems on 2.3.62 or on vmware or on
> SMP?
> >
> > I'm trying now with fork=yes and children=1.
> > If I have only one working child, does it suppose to lock and
> shed_yeild() itself from any reason?
> >
> > Meanwhile with single child OpenSIPS easily handles 4K of concurrent
> calls at 15cps, load average is 0.00 (!) and CPU is about 96% idle.
> >
> > I wonder if single working child also hangs.
> >
> > --
> > Best Regards,
> > Alex Massover
> > VoIP R&D TL
> > Jajah Inc.
> >
> >> -----Original Message-----
> >> From: users-bounces at lists.opensips.org [mailto:users-
> >> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
> >> Sent: Friday, January 22, 2010 1:17 PM
> >> To: OpenSIPS users mailling list
> >> Subject: Re: [OpenSIPS-Users] sched_yield()
> >>
> >> Hi,
> >>
> >> The new f_malloc will not do anything extra when compared to the old
> >> one
> >> until memory usage goes way up.
> >> I've added a warning in mem/f_malloc.c so you can see when defrag
> >> starts. If you get this warning then it is clear that the problem is
> >> from high memory usage.
> >>
> >> 1 GB for 4k calls seems a lot ( 250k per call). You can try to use
> >> "opensipsctl fifo get_statistics shmem:" and see what the memory
> usage
> >> is for diferent number of concurrent calls ( 1k,2k,3k,4k), and if
> >> indeed
> >> the memory usage is that high we should investigate the cause.
> >>
> >>
> >> Alex Massover wrote:
> >>
> >>> Hi,
> >>>
> >>> Now shared memory is 1G (-m 1024), and all memory is dedicated to
> >>>
> >> the virtual machine (it was shared till now).
> >>
> >>> But it still happens, just not so often.
> >>>
> >>> I originate the calls for this stress test in Asterisk with the
> same
> >>>
> >> resources and looks like Asterisk performs much better than
> OpenSIPS.
> >> How can it be?
> >>
> >>> In my stress OpenSIPS does no blocking/slow requests. And it's just
> >>>
> >> 4K concurrent calls, each one is 2-3 min.
> >>
> >>> Maybe OpenSIPS does too much low level memory management and
> virtual
> >>>
> >> machine is not suitable for it (despite that Asterisk runs well over
> >> VMware)?
> >>
> >>> I'm not sure but I have a feeling that 1.4 performed better. What
> can
> >>>
> >> cause performance degradation in 1.6? Storing vars on dialog, new
> >> malloc()?
> >>
> >>> gdb) bt
> >>> #0 0xb78ad424 in __kernel_vsyscall ()
> >>> #1 0xb77e841c in sched_yield () from /lib/i686/cmov/libc.so.6
> >>> #2 0x080bf23d in new_avp ()
> >>> #3 0x080bf53f in add_avp ()
> >>> #4 0x08080e6e in pv_set_avp ()
> >>> #5 0x0808229c in pv_set_value ()
> >>> #6 0x08053c9d in do_assign ()
> >>> #7 0x0805447a in do_action ()
> >>> #8 0x08053ebf in run_action_list ()
> >>> #9 0x08056e7a in do_action ()
> >>> #10 0x08053ebf in run_action_list ()
> >>> #11 0x08056e7a in do_action ()
> >>> #12 0x08053ebf in run_action_list ()
> >>> #13 0x080569d8 in do_action ()
> >>> #14 0x08053ebf in run_action_list ()
> >>> #15 0x08056e7a in do_action ()
> >>> #16 0x08053ebf in run_action_list ()
> >>> #17 0x08057d99 in run_top_route ()
> >>> #18 0x0808ad6c in receive_msg ()
> >>> #19 0x080bd2f2 in udp_rcv_loop ()
> >>> #20 0x08069339 in main ()
> >>> (gdb) quit
> >>>
> >>>
> >>> (gdb) bt
> >>> #0 0xb78ad424 in __kernel_vsyscall ()
> >>> #1 0xb77e841c in sched_yield () from /lib/i686/cmov/libc.so.6
> >>> #2 0xb76f52cd in build_cell () from
> /usr/lib/opensips/modules/tm.so
> >>> #3 0xb770ac4a in t_newtran () from /usr/lib/opensips/modules/tm.so
> >>> #4 0xb76ff7b8 in t_relay_to () from
> /usr/lib/opensips/modules/tm.so
> >>> #5 0xb770c501 in ?? () from /usr/lib/opensips/modules/tm.so
> >>> #6 0x08055030 in do_action ()
> >>> #7 0x08053ebf in run_action_list ()
> >>> #8 0x08095cf2 in eval_expr ()
> >>> #9 0x080958d9 in eval_expr ()
> >>> #10 0x08095919 in eval_expr ()
> >>> #11 0x080554e2 in do_action ()
> >>> #12 0x08053ebf in run_action_list ()
> >>> #13 0x080569d8 in do_action ()
> >>> #14 0x08053ebf in run_action_list ()
> >>> #15 0x08056e7a in do_action ()
> >>> #16 0x08053ebf in run_action_list ()
> >>> #17 0x08057d99 in run_top_route ()
> >>> #18 0x0808ad6c in receive_msg ()
> >>> #19 0x080bd2f2 in udp_rcv_loop ()
> >>> #20 0x08069339 in main ()
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Alex Massover
> >>> VoIP R&D TL
> >>> Jajah Inc.
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: users-bounces at lists.opensips.org [mailto:users-
> >>>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
> >>>> Sent: Thursday, January 21, 2010 3:43 PM
> >>>> To: OpenSIPS users mailling list
> >>>> Subject: Re: [OpenSIPS-Users] sched_yield()
> >>>>
> >>>> My guess is that there is not enough shared memory. When an
> >>>>
> >> allocation
> >>
> >>>> failes OpenSIPS tries to defragment memory to make room which
> takes
> >>>>
> >> a
> >>
> >>>> lot of time and must be done under lock.
> >>>>
> >>>> Please try to increase the shared memory size and tell me if it
> >>>> persists.
> >>>>
> >>>>
> >>>> Alex Massover wrote:
> >>>>
> >>>>
> >>>>> Hi!
> >>>>>
> >>>>> Yes, with -DF_MALLOC.
> >>>>>
> >>>>> 1.6.1 from sources, I build deb package.
> >>>>> I use 128M of shared and 10*1024*1024 private memory (can
> increase
> >>>>>
> >> -
> >>
> >>>> no problem).
> >>>>
> >>>>
> >>>>> Hmmmm, "opensipsctl fifo get_statistics all" crashes/stops the
> >>>>>
> >>>>>
> >>>> opensips.
> >>>>
> >>>>
> >>>>> 'fifo uptime' or 'fifo debug' are OK.
> >>>>>
> >>>>> strace while 'fifo get_statistics all':
> >>>>> Process 9509 attached - interrupt to quit
> >>>>> pause() = ? ERESTARTNOHAND (To be
> >>>>>
> >>>>>
> >>>> restarted)
> >>>>
> >>>>
> >>>>> --- SIGUSR2 (User defined signal 2) @ 0 (0) ---
> >>>>> sigreturn() = ? (mask now [])
> >>>>> pause() = ? ERESTARTNOHAND (To be
> >>>>>
> >>>>>
> >>>> restarted)
> >>>>
> >>>>
> >>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
> >>>>> sigreturn() = ? (mask now [])
> >>>>> waitpid(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGUSR2}],
> WNOHANG)
> >>>>>
> >> =
> >>
> >>>> 9520
> >>>>
> >>>>
> >>>>> waitpid(-1, 0xbf84b4c8, WNOHANG) = 0
> >>>>> kill(0, SIGTERM) = 0
> >>>>> --- SIGTERM (Terminated) @ 0 (0) ---
> >>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
> >>>>> sigreturn() = ? (mask now [TERM])
> >>>>> sigreturn() = ? (mask now [])
> >>>>> rt_sigaction(SIGALRM, {0x8065920, [ALRM], SA_RESTART}, {SIG_DFL},
> >>>>>
> >> 8)
> >>
> >>>> = 0
> >>>>
> >>>>
> >>>>> alarm(60) = 0
> >>>>> wait4(-1, NULL, 0, NULL) = 9514
> >>>>> wait4(-1, NULL, 0, NULL) = 9519
> >>>>> wait4(-1, NULL, 0, NULL) = 9521
> >>>>> wait4(-1, NULL, 0, NULL) = 9522
> >>>>> wait4(-1, NULL, 0, NULL) = 9512
> >>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
> >>>>> sigreturn() = ? (mask now [])
> >>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
> >>>>> sigreturn() = ? (mask now [])
> >>>>> wait4(-1, NULL, 0, NULL) = 9510
> >>>>> wait4(-1, NULL, 0, NULL) = 9516
> >>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
> >>>>> sigreturn() = ? (mask now [])
> >>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
> >>>>> sigreturn() = ? (mask now [])
> >>>>> wait4(-1, NULL, 0, NULL) = 9515
> >>>>> wait4(-1, NULL, 0, NULL) = 9517
> >>>>> wait4(-1, NULL, 0, NULL) = 9524
> >>>>> wait4(-1, NULL, 0, NULL) = 9525
> >>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
> >>>>> sigreturn() = ? (mask now [])
> >>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
> >>>>> sigreturn() = ? (mask now [])
> >>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
> >>>>> sigreturn() = ? (mask now [])
> >>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
> >>>>> sigreturn() = ? (mask now [])
> >>>>> wait4(-1, NULL, 0, NULL) = 9511
> >>>>> wait4(-1, NULL, 0, NULL) = 9513
> >>>>> wait4(-1, NULL, 0, NULL) = 9518
> >>>>> wait4(-1, NULL, 0, NULL) = 9523
> >>>>> wait4(-1, NULL, 0, NULL) = -1 ECHILD (No child
> >>>>>
> >>>>>
> >>>> processes)
> >>>>
> >>>>
> >>>>> rt_sigaction(SIGALRM, {0x8066080, [ALRM], SA_RESTART},
> {0x8065920,
> >>>>>
> >>>>>
> >>>> [ALRM], SA_RESTART}, 8) = 0
> >>>>
> >>>>
> >>>>> stat64("/tmp/opensips_fifo", {st_mode=S_IFIFO|0660, st_size=0,
> >>>>>
> >> ...})
> >>
> >>>> = 0
> >>>>
> >>>>
> >>>>> unlink("/tmp/opensips_fifo") = 0
> >>>>> munmap(0xaed25000, 134217728) = 0
> >>>>> unlink("/var/run/opensips/opensips.pid") = 0
> >>>>> alarm(0) = 60
> >>>>> rt_sigaction(SIGALRM, {SIG_IGN}, {0x8066080, [ALRM], SA_RESTART},
> >>>>>
> >> 8)
> >>
> >>>> = 0
> >>>>
> >>>>
> >>>>> exit_group(0) = ?
> >>>>> Process 9509 detached
> >>>>>
> >>>>> --
> >>>>> Best Regards,
> >>>>> Alex Massover
> >>>>> VoIP R&D TL
> >>>>> Jajah Inc.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: users-bounces at lists.opensips.org [mailto:users-
> >>>>>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
> >>>>>> Sent: Thursday, January 21, 2010 3:09 PM
> >>>>>> To: OpenSIPS users mailling list
> >>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
> >>>>>>
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> Since all the backtraces are in allocation routines my guess is
> >>>>>>
> >> that
> >>
> >>>>>> the
> >>>>>> shared memory lock might be causing a problem.
> >>>>>>
> >>>>>> Are you compiling with -DF_MALLOC?
> >>>>>> What version of OpenSIPS are you using?
> >>>>>> What is the total shared memory pool you are allocating?
> >>>>>> What amount of memory are you using? ( Use : opensipsctl fifo
> >>>>>> get_statistics all )
> >>>>>>
> >>>>>> Alex Massover wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Some more,
> >>>>>>>
> >>>>>>> (gdb) bt
> >>>>>>> #0 0xb78dc424 in __kernel_vsyscall ()
> >>>>>>> #1 0xb781741c in sched_yield () from /lib/i686/cmov/libc.so.6
> >>>>>>> #2 0xb73d77fd in build_new_dlg () from
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> /usr/lib/opensips/modules/dialog.so
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> #3 0xb73d4b81 in dlg_create_dialog () from
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> /usr/lib/opensips/modules/dialog.so
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> #4 0xb73c9c9e in ?? () from
> /usr/lib/opensips/modules/dialog.so
> >>>>>>> #5 0x08055030 in do_action ()
> >>>>>>> #6 0x08053ebf in run_action_list ()
> >>>>>>> #7 0x08056e7a in do_action ()
> >>>>>>> #8 0x08053ebf in run_action_list ()
> >>>>>>> #9 0x08057d99 in run_top_route ()
> >>>>>>> #10 0x0808ad6c in receive_msg ()
> >>>>>>> #11 0x080bd2f2 in udp_rcv_loop ()
> >>>>>>> #12 0x08069339 in main ()
> >>>>>>>
> >>>>>>>
> >>>>>>> (gdb) bt
> >>>>>>> #0 0xb78dc424 in __kernel_vsyscall ()
> >>>>>>> #1 0xb781741c in sched_yield () from /lib/i686/cmov/libc.so.6
> >>>>>>> #2 0xb77242cd in build_cell () from
> >>>>>>>
> >>>>>>>
> >>>> /usr/lib/opensips/modules/tm.so
> >>>>
> >>>>
> >>>>>>> #3 0xb7739c4a in t_newtran () from
> >>>>>>>
> >> /usr/lib/opensips/modules/tm.so
> >>
> >>>>>>> #4 0xb772e7b8 in t_relay_to () from
> >>>>>>>
> >>>>>>>
> >>>> /usr/lib/opensips/modules/tm.so
> >>>>
> >>>>
> >>>>>>> #5 0xb773b501 in ?? () from /usr/lib/opensips/modules/tm.so
> >>>>>>> #6 0x08055030 in do_action ()
> >>>>>>> #7 0x08053ebf in run_action_list ()
> >>>>>>> #8 0x08095cf2 in eval_expr ()
> >>>>>>> #9 0x080958d9 in eval_expr ()
> >>>>>>> #10 0x08095919 in eval_expr ()
> >>>>>>> #11 0x080554e2 in do_action ()
> >>>>>>> #12 0x08053ebf in run_action_list ()
> >>>>>>> #13 0x080569d8 in do_action ()
> >>>>>>> #14 0x08053ebf in run_action_list ()
> >>>>>>> #15 0x08056e7a in do_action ()
> >>>>>>> #16 0x08053ebf in run_action_list ()
> >>>>>>> #17 0x08057d99 in run_top_route ()
> >>>>>>> #18 0x0808ad6c in receive_msg ()
> >>>>>>> #19 0x080bd2f2 in udp_rcv_loop ()
> >>>>>>> #20 0x08069339 in main ()
> >>>>>>>
> >>>>>>> --
> >>>>>>> Best Regards,
> >>>>>>> Alex Massover
> >>>>>>> VoIP R&D TL
> >>>>>>> Jajah Inc.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
> >>>>>>>> bounces at lists.opensips.org] On Behalf Of Alex Massover
> >>>>>>>> Sent: Thursday, January 21, 2010 2:24 PM
> >>>>>>>> To: OpenSIPS users mailling list
> >>>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> Another one.. It hangs for a number of seconds (but it's
> enough
> >>>>>>>>
> >> to
> >>
> >>>>>>>> cause to SIP timeouts - MSG queue jumps to 260K), it's hard to
> >>>>>>>>
> >>>>>>>>
> >>>> make
> >>>>
> >>>>
> >>>>>> a
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>> bt at the right moment.
> >>>>>>>> This one looks better because there's sched_yield() there :)
> >>>>>>>>
> >>>>>>>> (gdb) bt
> >>>>>>>> #0 0xb77d5424 in __kernel_vsyscall ()
> >>>>>>>> #1 0xb771041c in sched_yield () from /lib/i686/cmov/libc.so.6
> >>>>>>>> #2 0x080bf23d in new_avp ()
> >>>>>>>> #3 0x080bf53f in add_avp ()
> >>>>>>>> #4 0xb72c1c9c in ?? () from
> /usr/lib/opensips/modules/dialog.so
> >>>>>>>> #5 0x08055030 in do_action ()
> >>>>>>>> #6 0x08053ebf in run_action_list ()
> >>>>>>>> #7 0x08056e7a in do_action ()
> >>>>>>>> #8 0x08053ebf in run_action_list ()
> >>>>>>>> #9 0x08056e7a in do_action ()
> >>>>>>>> #10 0x08053ebf in run_action_list ()
> >>>>>>>> #11 0x08056e7a in do_action ()
> >>>>>>>> #12 0x08053ebf in run_action_list ()
> >>>>>>>> #13 0x08057d99 in run_top_route ()
> >>>>>>>> #14 0x0808ad6c in receive_msg ()
> >>>>>>>> #15 0x080bd2f2 in udp_rcv_loop ()
> >>>>>>>> #16 0x08069339 in main ()
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Best Regards,
> >>>>>>>> Alex Massover
> >>>>>>>> VoIP R&D TL
> >>>>>>>> Jajah Inc.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
> >>>>>>>>> bounces at lists.opensips.org] On Behalf Of Alex Massover
> >>>>>>>>> Sent: Thursday, January 21, 2010 2:05 PM
> >>>>>>>>> To: OpenSIPS users mailling list
> >>>>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
> >>>>>>>>>
> >>>>>>>>> Hi Andrei,
> >>>>>>>>> Hopefully this is it (with FASTLOCK)
> >>>>>>>>>
> >>>>>>>>> #0 0xb77d5424 in __kernel_vsyscall ()
> >>>>>>>>> #1 0xb772babb in poll () from /lib/i686/cmov/libc.so.6
> >>>>>>>>> #2 0xb77ba83a in ?? () from /lib/i686/cmov/libresolv.so.2
> >>>>>>>>> #3 0xb77b8946 in __libc_res_nquery () from
> >>>>>>>>> /lib/i686/cmov/libresolv.so.2
> >>>>>>>>> #4 0xb77b8fdb in ?? () from /lib/i686/cmov/libresolv.so.2
> >>>>>>>>> #5 0xb77b92ae in __libc_res_nsearch () from
> >>>>>>>>> /lib/i686/cmov/libresolv.so.2
> >>>>>>>>> #6 0xb77b96d4 in __res_nsearch () from
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> /lib/i686/cmov/libresolv.so.2
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>> #7 0xb77b808a in res_search () from
> >>>>>>>>>
> >>>>>>>>>
> >>>> /lib/i686/cmov/libresolv.so.2
> >>>>
> >>>>
> >>>>>>>>> #8 0x0808c613 in get_record ()
> >>>>>>>>> #9 0x0808cf05 in ?? ()
> >>>>>>>>> #10 0x0808e385 in sip_resolvehost ()
> >>>>>>>>> #11 0x0807a26c in mk_proxy ()
> >>>>>>>>> #12 0xb7627d39 in t_relay_to () from
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> /usr/lib/opensips/modules/tm.so
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>> #13 0xb7634501 in ?? () from /usr/lib/opensips/modules/tm.so
> >>>>>>>>> #14 0x08055030 in do_action ()
> >>>>>>>>> #15 0x08053ebf in run_action_list ()
> >>>>>>>>> #16 0x08095cf2 in eval_expr ()
> >>>>>>>>> #17 0x080958d9 in eval_expr ()
> >>>>>>>>> #18 0x08095919 in eval_expr ()
> >>>>>>>>> #19 0x080554e2 in do_action ()
> >>>>>>>>> #20 0x08053ebf in run_action_list ()
> >>>>>>>>> #21 0x08056e7a in do_action ()
> >>>>>>>>> #22 0x08053ebf in run_action_list ()
> >>>>>>>>> ---Type <return> to continue, or q <return> to quit---
> >>>>>>>>> #23 0x080569d8 in do_action ()
> >>>>>>>>> #24 0x08053ebf in run_action_list ()
> >>>>>>>>> #25 0x08056e7a in do_action ()
> >>>>>>>>> #26 0x08053ebf in run_action_list ()
> >>>>>>>>> #27 0x08057d99 in run_top_route ()
> >>>>>>>>> #28 0x0808ad6c in receive_msg ()
> >>>>>>>>> #29 0x080bd2f2 in udp_rcv_loop ()
> >>>>>>>>> #30 0x08069339 in main ()
> >>>>>>>>> (gdb)
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Best Regards,
> >>>>>>>>> Alex Massover
> >>>>>>>>> VoIP R&D TL
> >>>>>>>>> Jajah Inc.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
> >>>>>>>>>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
> >>>>>>>>>> Sent: Wednesday, January 20, 2010 2:58 PM
> >>>>>>>>>> To: OpenSIPS users mailling list
> >>>>>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
> >>>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> I think that there is a lock that is being held more than it
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>> should
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>> be
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> and that's what causes starvation. It would help us if you
> >>>>>>>>>>
> >> could
> >>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> attach
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> to a process using gdb and give us a full backtrace.
> >>>>>>>>>>
> >>>>>>>>>> Temporary solutions which should work would be to reduce the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>> number
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>> of
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> processes to 4-6 or to recompile replacing -DFAST_LOCK with
> >>>>>>>>>>
> >> one
> >>
> >>>> of
> >>>>
> >>>>
> >>>>>>>>> the
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> other options (-DUSE_POSIX_SEM or -DUSE_PTHREAD_MUTEX) but
> we
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>> should
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> see
> >>>>>>>>>> where this is from to fix it.
> >>>>>>>>>>
> >>>>>>>>>> Alex Massover wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Hi!
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, from the source on debian, I build deb package. (I did
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>> some
> >>>>
> >>>>
> >>>>>>>>>> minor changes to the source, but the problem happens also
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> without
> >>>>
> >>>>
> >>>>>>>> my
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> changes)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> 16 children on 4 cores.
> >>>>>>>>>>>
> >>>>>>>>>>> What do you suggest to reduce it to 4? It runs on 2.6.32 on
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>> VMware
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> ESX.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> I'm also trying now sleep(0) instead of sched_yield().
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Alex Massover
> >>>>>>>>>>> VoIP R&D TL
> >>>>>>>>>>> Jajah Inc.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
> >>>>>>>>>>>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
> >>>>>>>>>>>> Sent: Wednesday, January 20, 2010 1:05 PM
> >>>>>>>>>>>> To: OpenSIPS users mailling list
> >>>>>>>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Alex,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Are you building OpenSIPS from source?
> >>>>>>>>>>>> How many processes do you have and on how many cores?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Alex Massover wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hello!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'm facing a strange problem, sometimes under a stress
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>> OpenSIPS
> >>>>
> >>>>
> >>>>>>>>>>>>> "locks" - load average jumps, SIP processing delays,
> >>>>>>>>>>>>>
> >> opensips
> >>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> msg
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>>>>> queue fills with a lot of sip messages, opensips
> processes
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> start
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> to
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>> comsume a lot of CPU.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> And strace shows:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> sched_yield()
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> sched_yield()
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> sched_yield()
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> sched_yield()
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ....
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> for all processes.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If I stop the stress - after a while (not immediately) -
> it
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> unlocks,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>> also suddenly, I can see in top that all opensips
> processes
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> stop
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> to
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>> consume CPU.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> What can it be? Some kind of starvation?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Alex Massover
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> VoIP R&D TL
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jajah Inc.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This mail was sent via Mail-SeCure System.
> >>>>>>>>>>>>> ---------------------------------------------------------
> --
> >>>>>>>>>>>>>
> >> --
> >>
> >>>> --
> >>>>
> >>>>
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>> Users mailing list
> >>>>>>>>>>>>> Users at lists.opensips.org
> >>>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Andrei Dragus
> >>>>>>>>>>>> www.voice-system.ro
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> Users mailing list
> >>>>>>>>>>>> Users at lists.opensips.org
> >>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>>>>>>>>>>>
> >>>>>>>>>>>> This mail was received via Mail-SeCure System.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> This mail was sent via Mail-SeCure System.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>> Users mailing list
> >>>>>>>>>>> Users at lists.opensips.org
> >>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Andrei Dragus
> >>>>>>>>>> www.voice-system.ro
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> Users mailing list
> >>>>>>>>>> Users at lists.opensips.org
> >>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>>>>>>>>>
> >>>>>>>>>> This mail was received via Mail-SeCure System.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> This mail was sent via Mail-SeCure System.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> Users mailing list
> >>>>>>>>> Users at lists.opensips.org
> >>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>>>>>>>>
> >>>>>>>>> This mail was received via Mail-SeCure System.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> This mail was sent via Mail-SeCure System.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> Users mailing list
> >>>>>>>> Users at lists.opensips.org
> >>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>>>>>>>
> >>>>>>>> This mail was received via Mail-SeCure System.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> This mail was sent via Mail-SeCure System.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> Users mailing list
> >>>>>>> Users at lists.opensips.org
> >>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> --
> >>>>>> Andrei Dragus
> >>>>>> www.voice-system.ro
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Users mailing list
> >>>>>> Users at lists.opensips.org
> >>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>>>>>
> >>>>>> This mail was received via Mail-SeCure System.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> This mail was sent via Mail-SeCure System.
> >>>>>
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> Users mailing list
> >>>>> Users at lists.opensips.org
> >>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>>>>
> >>>>>
> >>>>>
> >>>> --
> >>>> Andrei Dragus
> >>>> www.voice-system.ro
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Users mailing list
> >>>> Users at lists.opensips.org
> >>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>>>
> >>>> This mail was received via Mail-SeCure System.
> >>>>
> >>>>
> >>>>
> >>> This mail was sent via Mail-SeCure System.
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Users mailing list
> >>> Users at lists.opensips.org
> >>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>>
> >>>
> >> --
> >> Andrei Dragus
> >> www.voice-system.ro
> >>
> >>
> >> _______________________________________________
> >> Users mailing list
> >> Users at lists.opensips.org
> >> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >>
> >> This mail was received via Mail-SeCure System.
> >>
> >>
> >
> >
> > This mail was sent via Mail-SeCure System.
> >
> >
> >
> > _______________________________________________
> > Users mailing list
> > Users at lists.opensips.org
> > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >
>
> --
> Andrei Dragus
> www.voice-system.ro
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
> This mail was received via Mail-SeCure System.
>
This mail was sent via Mail-SeCure System.
More information about the Users
mailing list