[OpenSIPS-Users] opensips will not start while loading huge blacklist

[Abdul Basit]

Thanks a lot.
Earlier status
Opensips service was started in *(real 1m6.474s)*

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'

real    0m3.948s
user    0m0.039s
sys     0m0.013s

time systemctl stop opensips

real    0m1.424s
user    0m0.004s
sys     0m0.000s

*I pulled source from master.*

# time systemctl start opensips
Oct  2 15:57:41 srouter-215 systemd[1]: Starting OpenSIPS is a very fast
and flexible SIP (RFC3261) server...
Oct  2 15:57:41 srouter-215 opensips[15906]: Listening on
Oct  2 15:57:41 srouter-215 opensips[15906]: udp: x.x.x.x [x.x.x.x]:9020
Oct  2 15:57:41 srouter-215 opensips[15906]: udp:
x6:x6:x6:x6:x6:x6 [[x6:x6:0:0:x6:x6:x6:x6]]:9020
Oct  2 15:57:41 srouter-215 opensips[15906]: Aliases:
Oct  2 15:57:41 srouter-215 opensips[15906]: *: x.x.x.x:*

real    0m0.066s
user    0m0.003s
sys     0m0.002s
root at srouter-215:~# Oct  2 15:57:41 srouter-215 systemd[1]: Started
OpenSIPS is a very fast and flexible SIP (RFC3261) server.

/var/log/opensips.log is here ... https://pastebin.com/8md6sBiP

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'
Oct  2 16:17:24 srouter-215 SIPSrv[16185]:
INFO:userblacklist:reload_sources: got 1386901 entries from
'globalblacklist'

real    0m41.551s
user    0m0.037s
sys     0m0.015s

# time systemctl stop opensips
Oct  2 16:06:38 srouter-215 systemd[1]: Stopping OpenSIPS is a very fast and
flexible SIP (RFC3261) server...
Oct  2 16:06:50 srouter-215 systemd[1]: Stopped OpenSIPS is a very fast and
flexible SIP (RFC3261) server.

real    0m12.134s
user    0m0.003s
sys     0m0.002s


Looks like issue has been fixed.. :)

--
regards,

abdul basit


On Tue, 2 Oct 2018 at 18:55, Bogdan-Andrei Iancu <bogdan at opensips.org>
wrote:

> OK, I pushed on master a patch to move the initial data load after the
> startup sequence. The patch should apply to 2.4 also.
>
>
> https://github.com/OpenSIPS/opensips/commit/32cba1d751ee5a5b705da97ac4333fcad3d67728
>
> While checking the module, I found some issues with it:
>
>     * before my patch, the load (for same data) was independently done in
> each opensips process (even if the data is shared) :)
>
>     * the "reload" is no error pron - if the new data load fails, the old
> existing data is lost
>
>     * during the data reload the module has no data, so it cannot operate
> (it will actually block) - normally it should continue with the old data
> until the new data is fully loaded and validated.
>
> Let me know if the patch makes a different for you and I will backport it
> to 2.4 too.
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
>   http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/02/2018 04:18 PM, Bogdan-Andrei Iancu wrote:
>
> Hi,
>
> Taking a closer look at the module (to be honest I do not user it, as
> drouting does a similar job) I see the module is loading the data into
> memory. It is doing this during the init stage, holding back the entire
> OpenSIPS startup.
> Module doing similar heavy db load were changed to do the load
> post-startup. Let patch the module to decouple the loading from the
> startup, so that OpenSIPS will start without delays.
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
>   http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/02/2018 04:04 PM, Jon Abrams wrote:
>
> OpenSIPs is loading the global blacklist into a digit trie structure into
> shared memory on startup (or reload_blacklist command). That causes the
> startup delay.
>
> - Jon
>
> On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit <basit.engg at gmail.com> wrote:
>
>> Hi Bogdan,
>>
>> I don't think so its translating to DB query on run time, instead its
>> caching the blacklist.
>> I am saying so because opensips response is quick, where as when I query
>> manually from DB, it take some time even prefix col. is indexed.
>> Moreover, while opensips restart i get following warnings in syslog:
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20181002/185c6ae2/attachment.html>