[OpenSIPS-Users] opensips will not start while loading huge blacklist

Tue Oct 2 09:04:16 EDT 2018

OpenSIPs is loading the global blacklist into a digit trie structure into
shared memory on startup (or reload_blacklist command). That causes the
startup delay.

- Jon

On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit <basit.engg at gmail.com> wrote:

> Hi Bogdan,
>
> I don't think so its translating to DB query on run time, instead its
> caching the blacklist.
> I am saying so because opensips response is quick, where as when I query
> manually from DB, it take some time even prefix col. is indexed.
> Moreover, while opensips restart i get following warnings in syslog:
>
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task <nh-timer> already scheduled for 990 ms (now 17920 ms), it may
> overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task <dlg-reinvite-pinger> already scheduled for 990 ms (now 17920
> ms), it may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task <dlg-options-pinger> already scheduled for 990 ms (now 17920
> ms), it may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task <dlg-timer> already scheduled for 990 ms (now 17920 ms), it
> may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task <tm-timer> already scheduled for 990 ms (now 17920 ms), it may
> overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task <blcore-expire> already scheduled for 990 ms (now 17920 ms),
> it may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker:
> utimer task <tm-utimer> already scheduled for 100 ms (now 18020 ms), it
> may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker:
> utimer task <tm-utimer> already scheduled for 100 ms (now 18120 ms), it
> may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker:
> utimer task <tm-utimer> already scheduled for 100 ms (now 18220 ms), it
> may overlap..
> *Oct  2 12:35:28 srouter-215 SIPSrv[20882]:
> INFO:userblacklist:reload_sources: got 1386901 entries from
> 'globalblacklist' *
> Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job:
> utimer job <tm-utimer> has a 18170000 us delay in execution
> Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job:
> timer job <nh-timer> has a 17280000 us delay in execution
> Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job:
> timer job <dlg-reinvite-pinger> has a 17280000 us delay in execution
>
> Once globalblacklist loaded *1386901 entries* , opensips behave normally.
> When I will add more records to globalblacklist, opensips wouldn't start
> due to service timeout.
> One way might be to increase service timeout to allow opensips keep
> loading the entries but it may take ages.
>
> --
> regards,
>
> abdul basit
>
> On Tue, 2 Oct 2018 at 13:19, Bogdan-Andrei Iancu <bogdan at opensips.org>
> wrote:
>
>> Hi Abdul,
>>
>> OK, but what is the blacklist related operation you do in startup route
>> ?? As far as I know, the blacklist module is not doing any kind of caching,
>> so any check you do against the blacklists will translate into a DB query,
>> right ?
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>>
>> OpenSIPS Founder and Developer
>>   http://www.opensips-solutions.com
>> OpenSIPS Bootcamp 2018
>>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>
>> On 10/01/2018 06:58 PM, Abdul Basit wrote:
>>
>> Yes. That's right.
>> The code I am using is as below. globalblacklist table has complete
>> number as prefix with and whitelist status.
>>
>> route {
>> .....
>> if ( !check_blacklist("globalblacklist") ) {
>> xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not Allowed.
>> Dropping call.\n");
>> sl_send_reply("403", "Forbidden");
>> exit;
>> }
>> else {
>> xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
>> route(10);
>> exit;
>> }
>> .....
>> }
>>
>>
>> --
>> regards,
>>
>> abdul basit
>>
>>
>> On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu <bogdan at opensips.org>
>> wrote:
>>
>>> Hi Abdul,
>>>
>>> IF I understand correctly, you use startup route to load (from DB?) some
>>> blacklists into localcache ?
>>>
>>> Regards,
>>>
>>> Bogdan-Andrei Iancu
>>>
>>> OpenSIPS Founder and Developer
>>>   http://www.opensips-solutions.com
>>> OpenSIPS Bootcamp 2018
>>>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>>
>>> On 09/28/2018 04:10 AM, Abdul Basit wrote:
>>>
>>> Hi team,
>>>
>>> I have a opensips 2.4 deployed having 4 children and with mysql
>>> configured for loading user blacklist to opensips local cache on opensips
>>> startup.
>>> Opensips work fine on filtering the call based on user blacklist records
>>> loaded into local cache.
>>>
>>> While blacklist was growing upto 1.3m records, opensips startup was
>>> taking ~60 sec on VM of 16GB RAM and 6 CPU.
>>> I increased shared memory to *6144* and package memory to *256.*
>>> Filter works fine here yet. Call load is not that much only blacklist
>>> entries are growing.
>>>
>>> But now blacklist has grown to very huge number, up to 10m records in
>>> mysql. Opensips is not starting successfully and service startup requests
>>> timeout after huge CPU spikes while trying to populate local cache from
>>> mysql.
>>> I increased shared memory to *12288* and package memory to *1024* but
>>> no success yet.
>>>
>>> I indexed mysql blacklist table but no luck.
>>>
>>> Please advise what settings I should add to opensips for successful
>>> startup.
>>>
>>> --
>>> regards,
>>>
>>> abdul basit
>>>
>>>
>>> _______________________________________________
>>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>>
>> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20181002/3816a8ae/attachment-0001.html>