[OpenSIPS-Users] sched_yield()

Alex Massover alex at jajah.com
Thu Jan 21 14:07:31 CET 2010 

And one more

(gdb) bt
#0  0xb78dc424 in __kernel_vsyscall ()
#1  0xb781741c in sched_yield () from /lib/i686/cmov/libc.so.6
#2  0x080c3705 in _shm_resize ()
#3  0xb7746069 in relay_reply () from /usr/lib/opensips/modules/tm.so
#4  0xb7746d74 in reply_received () from /usr/lib/opensips/modules/tm.so
#5  0x08063408 in forward_reply ()
#6  0x0808ae2b in receive_msg ()
#7  0x080bd2f2 in udp_rcv_loop ()
#8  0x08069339 in main ()

--
Best Regards,
Alex Massover
VoIP R&D TL
Jajah Inc.

> -----Original Message-----
> From: users-bounces at lists.opensips.org [mailto:users-
> bounces at lists.opensips.org] On Behalf Of Alex Massover
> Sent: Thursday, January 21, 2010 2:51 PM
> To: OpenSIPS users mailling list
> Subject: Re: [OpenSIPS-Users] sched_yield()
> 
> Some more,
> 
> (gdb) bt
> #0  0xb78dc424 in __kernel_vsyscall ()
> #1  0xb781741c in sched_yield () from /lib/i686/cmov/libc.so.6
> #2  0xb73d77fd in build_new_dlg () from
> /usr/lib/opensips/modules/dialog.so
> #3  0xb73d4b81 in dlg_create_dialog () from
> /usr/lib/opensips/modules/dialog.so
> #4  0xb73c9c9e in ?? () from /usr/lib/opensips/modules/dialog.so
> #5  0x08055030 in do_action ()
> #6  0x08053ebf in run_action_list ()
> #7  0x08056e7a in do_action ()
> #8  0x08053ebf in run_action_list ()
> #9  0x08057d99 in run_top_route ()
> #10 0x0808ad6c in receive_msg ()
> #11 0x080bd2f2 in udp_rcv_loop ()
> #12 0x08069339 in main ()
> 
> 
>    (gdb) bt
> #0  0xb78dc424 in __kernel_vsyscall ()
> #1  0xb781741c in sched_yield () from /lib/i686/cmov/libc.so.6
> #2  0xb77242cd in build_cell () from /usr/lib/opensips/modules/tm.so
> #3  0xb7739c4a in t_newtran () from /usr/lib/opensips/modules/tm.so
> #4  0xb772e7b8 in t_relay_to () from /usr/lib/opensips/modules/tm.so
> #5  0xb773b501 in ?? () from /usr/lib/opensips/modules/tm.so
> #6  0x08055030 in do_action ()
> #7  0x08053ebf in run_action_list ()
> #8  0x08095cf2 in eval_expr ()
> #9  0x080958d9 in eval_expr ()
> #10 0x08095919 in eval_expr ()
> #11 0x080554e2 in do_action ()
> #12 0x08053ebf in run_action_list ()
> #13 0x080569d8 in do_action ()
> #14 0x08053ebf in run_action_list ()
> #15 0x08056e7a in do_action ()
> #16 0x08053ebf in run_action_list ()
> #17 0x08057d99 in run_top_route ()
> #18 0x0808ad6c in receive_msg ()
> #19 0x080bd2f2 in udp_rcv_loop ()
> #20 0x08069339 in main ()
> 
> --
> Best Regards,
> Alex Massover
> VoIP R&D TL
> Jajah Inc.
> 
> > -----Original Message-----
> > From: users-bounces at lists.opensips.org [mailto:users-
> > bounces at lists.opensips.org] On Behalf Of Alex Massover
> > Sent: Thursday, January 21, 2010 2:24 PM
> > To: OpenSIPS users mailling list
> > Subject: Re: [OpenSIPS-Users] sched_yield()
> >
> > Hi,
> >
> > Another one.. It hangs for a number of seconds (but it's enough to
> > cause to SIP timeouts - MSG queue jumps to 260K), it's hard to make a
> > bt at the right moment.
> > This one looks better because there's sched_yield() there :)
> >
> > (gdb) bt
> > #0  0xb77d5424 in __kernel_vsyscall ()
> > #1  0xb771041c in sched_yield () from /lib/i686/cmov/libc.so.6
> > #2  0x080bf23d in new_avp ()
> > #3  0x080bf53f in add_avp ()
> > #4  0xb72c1c9c in ?? () from /usr/lib/opensips/modules/dialog.so
> > #5  0x08055030 in do_action ()
> > #6  0x08053ebf in run_action_list ()
> > #7  0x08056e7a in do_action ()
> > #8  0x08053ebf in run_action_list ()
> > #9  0x08056e7a in do_action ()
> > #10 0x08053ebf in run_action_list ()
> > #11 0x08056e7a in do_action ()
> > #12 0x08053ebf in run_action_list ()
> > #13 0x08057d99 in run_top_route ()
> > #14 0x0808ad6c in receive_msg ()
> > #15 0x080bd2f2 in udp_rcv_loop ()
> > #16 0x08069339 in main ()
> >
> > --
> > Best Regards,
> > Alex Massover
> > VoIP R&D TL
> > Jajah Inc.
> >
> > > -----Original Message-----
> > > From: users-bounces at lists.opensips.org [mailto:users-
> > > bounces at lists.opensips.org] On Behalf Of Alex Massover
> > > Sent: Thursday, January 21, 2010 2:05 PM
> > > To: OpenSIPS users mailling list
> > > Subject: Re: [OpenSIPS-Users] sched_yield()
> > >
> > > Hi Andrei,
> > > Hopefully this is it (with FASTLOCK)
> > >
> > > #0  0xb77d5424 in __kernel_vsyscall ()
> > > #1  0xb772babb in poll () from /lib/i686/cmov/libc.so.6
> > > #2  0xb77ba83a in ?? () from /lib/i686/cmov/libresolv.so.2
> > > #3  0xb77b8946 in __libc_res_nquery () from
> > > /lib/i686/cmov/libresolv.so.2
> > > #4  0xb77b8fdb in ?? () from /lib/i686/cmov/libresolv.so.2
> > > #5  0xb77b92ae in __libc_res_nsearch () from
> > > /lib/i686/cmov/libresolv.so.2
> > > #6  0xb77b96d4 in __res_nsearch () from
> /lib/i686/cmov/libresolv.so.2
> > > #7  0xb77b808a in res_search () from /lib/i686/cmov/libresolv.so.2
> > > #8  0x0808c613 in get_record ()
> > > #9  0x0808cf05 in ?? ()
> > > #10 0x0808e385 in sip_resolvehost ()
> > > #11 0x0807a26c in mk_proxy ()
> > > #12 0xb7627d39 in t_relay_to () from
> /usr/lib/opensips/modules/tm.so
> > > #13 0xb7634501 in ?? () from /usr/lib/opensips/modules/tm.so
> > > #14 0x08055030 in do_action ()
> > > #15 0x08053ebf in run_action_list ()
> > > #16 0x08095cf2 in eval_expr ()
> > > #17 0x080958d9 in eval_expr ()
> > > #18 0x08095919 in eval_expr ()
> > > #19 0x080554e2 in do_action ()
> > > #20 0x08053ebf in run_action_list ()
> > > #21 0x08056e7a in do_action ()
> > > #22 0x08053ebf in run_action_list ()
> > > ---Type <return> to continue, or q <return> to quit---
> > > #23 0x080569d8 in do_action ()
> > > #24 0x08053ebf in run_action_list ()
> > > #25 0x08056e7a in do_action ()
> > > #26 0x08053ebf in run_action_list ()
> > > #27 0x08057d99 in run_top_route ()
> > > #28 0x0808ad6c in receive_msg ()
> > > #29 0x080bd2f2 in udp_rcv_loop ()
> > > #30 0x08069339 in main ()
> > > (gdb)
> > >
> > > --
> > > Best Regards,
> > > Alex Massover
> > > VoIP R&D TL
> > > Jajah Inc.
> > > > -----Original Message-----
> > > > From: users-bounces at lists.opensips.org [mailto:users-
> > > > bounces at lists.opensips.org] On Behalf Of Andrei Dragus
> > > > Sent: Wednesday, January 20, 2010 2:58 PM
> > > > To: OpenSIPS users mailling list
> > > > Subject: Re: [OpenSIPS-Users] sched_yield()
> > > >
> > > > Hi,
> > > >
> > > > I think that there is a lock that is being held more than it
> should
> > > be
> > > > and that's what causes starvation. It would help us if you could
> > > attach
> > > > to a process using gdb and give us a full backtrace.
> > > >
> > > > Temporary solutions which should work would be to reduce the
> number
> > > of
> > > > processes to 4-6 or to recompile replacing -DFAST_LOCK with one
> of
> > > the
> > > > other options (-DUSE_POSIX_SEM or -DUSE_PTHREAD_MUTEX) but we
> > should
> > > > see
> > > > where this is from to fix it.
> > > >
> > > > Alex Massover wrote:
> > > > > Hi!
> > > > >
> > > > > Yes, from the source on debian, I build deb package. (I did
> some
> > > > minor changes to the source, but the problem happens also without
> > my
> > > > changes)
> > > > > 16 children on 4 cores.
> > > > >
> > > > > What do you suggest to reduce it to 4? It runs on 2.6.32 on
> > VMware
> > > > ESX.
> > > > >
> > > > > I'm also trying now sleep(0) instead of sched_yield().
> > > > >
> > > > > --
> > > > > Best Regards,
> > > > > Alex Massover
> > > > > VoIP R&D TL
> > > > > Jajah Inc.
> > > > >
> > > > >> -----Original Message-----
> > > > >> From: users-bounces at lists.opensips.org [mailto:users-
> > > > >> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
> > > > >> Sent: Wednesday, January 20, 2010 1:05 PM
> > > > >> To: OpenSIPS users mailling list
> > > > >> Subject: Re: [OpenSIPS-Users] sched_yield()
> > > > >>
> > > > >> Hi Alex,
> > > > >>
> > > > >> Are you building OpenSIPS from source?
> > > > >> How many processes do you have and on how many cores?
> > > > >>
> > > > >>
> > > > >> Alex Massover wrote:
> > > > >>
> > > > >>> Hello!
> > > > >>>
> > > > >>> I'm facing a strange problem, sometimes under a stress
> OpenSIPS
> > > > >>> "locks" - load average jumps, SIP processing delays, opensips
> > msg
> > > > >>> queue fills with a lot of sip messages, opensips processes
> > start
> > > to
> > > > >>> comsume a lot of CPU.
> > > > >>>
> > > > >>> And strace shows:
> > > > >>>
> > > > >>> sched_yield()
> > > > >>>
> > > > >>> sched_yield()
> > > > >>>
> > > > >>> sched_yield()
> > > > >>>
> > > > >>> sched_yield()
> > > > >>>
> > > > >>> ....
> > > > >>>
> > > > >>> for all processes.
> > > > >>>
> > > > >>> If I stop the stress - after a while (not immediately) - it
> > > > unlocks,
> > > > >>> also suddenly, I can see in top that all opensips processes
> > stop
> > > to
> > > > >>> consume CPU.
> > > > >>>
> > > > >>> What can it be? Some kind of starvation?
> > > > >>>
> > > > >>> --
> > > > >>>
> > > > >>> Best Regards,
> > > > >>>
> > > > >>> Alex Massover
> > > > >>>
> > > > >>> VoIP R&D TL
> > > > >>>
> > > > >>> Jajah Inc.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> This mail was sent via Mail-SeCure System.
> > > > >>> -------------------------------------------------------------
> --
> > --
> > > --
> > > > --
> > > > >>>
> > > > >> ---
> > > > >>
> > > > >>> _______________________________________________
> > > > >>> Users mailing list
> > > > >>> Users at lists.opensips.org
> > > > >>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> > > > >>>
> > > > >>>
> > > > >> --
> > > > >> Andrei Dragus
> > > > >> www.voice-system.ro
> > > > >>
> > > > >>
> > > > >> _______________________________________________
> > > > >> Users mailing list
> > > > >> Users at lists.opensips.org
> > > > >> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> > > > >>
> > > > >> This mail was received via Mail-SeCure System.
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > > This mail was sent via Mail-SeCure System.
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Users mailing list
> > > > > Users at lists.opensips.org
> > > > > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> > > > >
> > > >
> > > > --
> > > > Andrei Dragus
> > > > www.voice-system.ro
> > > >
> > > >
> > > > _______________________________________________
> > > > Users mailing list
> > > > Users at lists.opensips.org
> > > > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> > > >
> > > > This mail was received via Mail-SeCure System.
> > > >
> > >
> > >
> > > This mail was sent via Mail-SeCure System.
> > >
> > >
> > >
> > > _______________________________________________
> > > Users mailing list
> > > Users at lists.opensips.org
> > > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> > >
> > > This mail was received via Mail-SeCure System.
> > >
> >
> >
> > This mail was sent via Mail-SeCure System.
> >
> >
> >
> > _______________________________________________
> > Users mailing list
> > Users at lists.opensips.org
> > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >
> > This mail was received via Mail-SeCure System.
> >
> 
> 
> This mail was sent via Mail-SeCure System.
> 
> 
> 
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> 
> This mail was received via Mail-SeCure System.
> 


This mail was sent via Mail-SeCure System.

More information about the Users mailing list