[OpenSIPS-Users] opensips will not start while loading huge blacklist

Tue Oct 2 16:20:00 EDT 2018

Yes. I understand and I check that point specifically.
Thank you for your quick solution.

--
regards,

abdul basit

On Wed, 3 Oct 2018 at 00:37, Bogdan-Andrei Iancu <bogdan at opensips.org>
wrote:

> Hi Abdul,
>
> Thanks for the feedback. I just want to be sure you understand that the
> change I did does not speed up the loading process, but it simply decouples
> the startup sequence from the data loading process. Shortly OpenSIPS does
> not have to wait for the DB data to be loaded in order to start up.
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
>   http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/02/2018 07:22 PM, Abdul Basit wrote:
>
> Thanks a lot.
> Earlier status
> Opensips service was started in *(real 1m6.474s)*
>
> # time opensipsctl fifo reload_blacklist
> WARNING: database engine not found - tried 'MYSQL'
>
> real    0m3.948s
> user    0m0.039s
> sys     0m0.013s
>
> time systemctl stop opensips
>
> real    0m1.424s
> user    0m0.004s
> sys     0m0.000s
>
> *I pulled source from master.*
>
> # time systemctl start opensips
> Oct  2 15:57:41 srouter-215 systemd[1]: Starting OpenSIPS is a very fast
> and flexible SIP (RFC3261) server...
> Oct  2 15:57:41 srouter-215 opensips[15906]: Listening on
> Oct  2 15:57:41 srouter-215 opensips[15906]: udp: x.x.x.x [x.x.x.x]:9020
> Oct  2 15:57:41 srouter-215 opensips[15906]: udp:
> x6:x6:x6:x6:x6:x6 [[x6:x6:0:0:x6:x6:x6:x6]]:9020
> Oct  2 15:57:41 srouter-215 opensips[15906]: Aliases:
> Oct  2 15:57:41 srouter-215 opensips[15906]: *: x.x.x.x:*
>
> real    0m0.066s
> user    0m0.003s
> sys     0m0.002s
> root at srouter-215:~# Oct  2 15:57:41 srouter-215 systemd[1]: Started
> OpenSIPS is a very fast and flexible SIP (RFC3261) server.
>
> /var/log/opensips.log is here ... https://pastebin.com/8md6sBiP
>
> # time opensipsctl fifo reload_blacklist
> WARNING: database engine not found - tried 'MYSQL'
> Oct  2 16:17:24 srouter-215 SIPSrv[16185]:
> INFO:userblacklist:reload_sources: got 1386901 entries from
> 'globalblacklist'
>
> real    0m41.551s
> user    0m0.037s
> sys     0m0.015s
>
> # time systemctl stop opensips
> Oct  2 16:06:38 srouter-215 systemd[1]: Stopping OpenSIPS is a very fast
> and flexible SIP (RFC3261) server...
> Oct  2 16:06:50 srouter-215 systemd[1]: Stopped OpenSIPS is a very fast
> and flexible SIP (RFC3261) server.
>
> real    0m12.134s
> user    0m0.003s
> sys     0m0.002s
>
>
> Looks like issue has been fixed.. :)
>
> --
> regards,
>
> abdul basit
>
>
> On Tue, 2 Oct 2018 at 18:55, Bogdan-Andrei Iancu <bogdan at opensips.org>
> wrote:
>
>> OK, I pushed on master a patch to move the initial data load after the
>> startup sequence. The patch should apply to 2.4 also.
>>
>>
>> https://github.com/OpenSIPS/opensips/commit/32cba1d751ee5a5b705da97ac4333fcad3d67728
>>
>> While checking the module, I found some issues with it:
>>
>>     * before my patch, the load (for same data) was independently done in
>> each opensips process (even if the data is shared) :)
>>
>>     * the "reload" is no error pron - if the new data load fails, the old
>> existing data is lost
>>
>>     * during the data reload the module has no data, so it cannot operate
>> (it will actually block) - normally it should continue with the old data
>> until the new data is fully loaded and validated.
>>
>> Let me know if the patch makes a different for you and I will backport it
>> to 2.4 too.
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>>
>> OpenSIPS Founder and Developer
>>   http://www.opensips-solutions.com
>> OpenSIPS Bootcamp 2018
>>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>
>> On 10/02/2018 04:18 PM, Bogdan-Andrei Iancu wrote:
>>
>> Hi,
>>
>> Taking a closer look at the module (to be honest I do not user it, as
>> drouting does a similar job) I see the module is loading the data into
>> memory. It is doing this during the init stage, holding back the entire
>> OpenSIPS startup.
>> Module doing similar heavy db load were changed to do the load
>> post-startup. Let patch the module to decouple the loading from the
>> startup, so that OpenSIPS will start without delays.
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>>
>> OpenSIPS Founder and Developer
>>   http://www.opensips-solutions.com
>> OpenSIPS Bootcamp 2018
>>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>
>> On 10/02/2018 04:04 PM, Jon Abrams wrote:
>>
>> OpenSIPs is loading the global blacklist into a digit trie structure into
>> shared memory on startup (or reload_blacklist command). That causes the
>> startup delay.
>>
>> - Jon
>>
>> On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit <basit.engg at gmail.com> wrote:
>>
>>> Hi Bogdan,
>>>
>>> I don't think so its translating to DB query on run time, instead its
>>> caching the blacklist.
>>> I am saying so because opensips response is quick, where as when I query
>>> manually from DB, it take some time even prefix col. is indexed.
>>> Moreover, while opensips restart i get following warnings in syslog:
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20181003/b23ecb3c/attachment-0001.html>