[OpenSIPS-Users] OpenSIPS timers

[Bogdan-Andrei Iancu]

Hi Ovidiu,

Originally you mentioned tm-utimer, now tm-timer....which one is ? As it 
is very important.

When increasing the timer_partitions, what you mean by "instability" of 
the system?

Yes, in the reactor, the UDP workers may handle timer jobs also beside 
the UDP traffic. While the timer procs are 100% dedicated to the timer 
jobs only. So yes, if the workers are idle, they can act as any timer 
procs also.

Increasing the TM_TABLE_ENTRIES should not impact too much, at the 
performance over the timer lists (in TM) has nothing to do with the size 
of the hash table.

I will check the mentioned ticket, but if what you are saying is true on 
the HP malloc, it means the bottle neck is actually in the ops on the 
shared memory.

Best regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS eBootcamp 23rd May - 3rd June 2022
   https://opensips.org/training/OpenSIPS_eBootcamp_2022/

On 4/1/22 12:31 AM, Ovidiu Sas wrote:
> Hello Bogdan,
>
> Thank you for looking into this!
>
> I get warnings mostly from tm-timer. I've seen warnings from
> blcore-expire, dlg-options-pinger, dlg-reinvite-pinger, dlg-timer (in
> the logs, but not during my testing).
> While testing, I saw only the tm-timer warnings.
>
> I took a superficial look at the "timer_partitions" and your
> explanation matches my findings. However, increasing the
> "timer_partitions" makes the system unstable (doesn't matter how many
> timer procs we have).
> I found that I can get the most out of the system if one
> "timer_partiton" is used along with one timer_proc.
>
> With the reactor scheme, a UDP receiver can handle timer jobs, is that
> right? If yes, if the UDP workers are idle, there are enough resources
> to handle timer jobs, correct?
>
> I was also increasing the TM_TABLE_ENTRIES to (1<<18) and there was a
> little bit of performance increase, but I will need to test more to
> come up with a valid conclusion.
>
> On the other hand, I noticed a strange behavior on timer handling.
> Take a look at:
> https://github.com/OpenSIPS/opensips/issues/2797
> Not sure if this is related to the warnings that I'm seeing.
>
> The biggest performance improvement was switching to HP_MALLOC for
> both pkg and shm memory.
>
> I will keep you posted with my findings,
> Ovidiu
>
> On Thu, Mar 31, 2022 at 10:28 AM Bogdan-Andrei Iancu
> <bogdan at opensips.org> wrote:
>> Hi Ovidiu,
>>
>> As warnings from the timer_ticker, do you get only for the tm-utimer
>> task ? I'm asking as the key question here is where the bottleneck is :
>> in the whole "timer" subsystem, or in the tm-utimer task only?
>>
>> The TM "timer_partitions" creates multiple parallel timer lists, to
>> avoid having large "amounts" of transactions handled at a moment in a
>> single tm-utimer task (but rather split/partition the whole of amount of
>> handled transactions into smaller chunks, to be handled one at a time in
>> the timer task.
>>
>> The "timer_workers" creates  more than one dedicated processes for
>> handling the timer tasks (so scales up the timer sub-system).
>>
>> If you get warnings only on tm-utimer, I suspect the bottleneck is TM
>> related, mainly on performing re-transmissions (that's what that task is
>> doing). So the increasing the timer-partitions should be the way to help.
>>
>> Best regards,
>>
>> Bogdan-Andrei Iancu
>>
>> OpenSIPS Founder and Developer
>>     https://www.opensips-solutions.com
>> OpenSIPS eBootcamp 23rd May - 3rd June 2022
>>     https://opensips.org/training/OpenSIPS_eBootcamp_2022/
>>
>> On 3/24/22 12:54 AM, Ovidiu Sas wrote:
>>> Hello all,
>>>
>>> I'm working on tuning an opensips server. I get this pesky:
>>> WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled
>>> I was trying to get rid of them by playing with the tm
>>> timer_partitions parameter and the timer_workers core param.
>>> By increasing any of them doesn't increase performance.
>>> By increasing both of them, it actually decreases performance.
>>> The server is not at limit, the load on the UDP workers is around
>>> 50-60 with some spikes.
>>> I have around 3500+ cps sipp traffic.
>>>
>>> My understanding is that by increasing the number of timer_partitions,
>>> we will have more procs walking in parallel over the timer structures.
>>> If we have on timer structure, we have one proc walking over it.
>>> How is this working for two timer structures? What is the difference
>>> between the first and the second timer structure? Should we expect
>>> less work for each proc?
>>>
>>> For now, to reduce the occurrence of the warning log, I increased the
>>> timer interval for tm-utimer from 100ms to 200ms. This should be ok as
>>> the timer has the TIMER_FLAG_DELAY_ON_DELAY flag set.
>>>
>>> Thanks,
>>> Ovidiu
>>>
>