[OpenSIPS-Users] opensips will not start while loading huge blacklist

Tue Oct 2 15:36:59 EDT 2018

Hi Abdul,

Thanks for the feedback. I just want to be sure you understand that the 
change I did does not speed up the loading process, but it simply 
decouples the startup sequence from the data loading process. Shortly 
OpenSIPS does not have to wait for the DB data to be loaded in order to 
start up.

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/02/2018 07:22 PM, Abdul Basit wrote:
> Thanks a lot.
> Earlier status
> Opensips service was started in *(real 1m6.474s)*
>
> # time opensipsctl fifo reload_blacklist
> WARNING: database engine not found - tried 'MYSQL'
>
> real   0m3.948s
> user   0m0.039s
> sys    0m0.013s
>
> time systemctl stop opensips
>
> real    0m1.424s
> user   0m0.004s
> sys    0m0.000s
>
> *I pulled source from master.*
>
> # time systemctl start opensips
> Oct 2 15:57:41 srouter-215 systemd[1]: Starting OpenSIPS is a very 
> fast and flexible SIP (RFC3261) server...
> Oct 2 15:57:41 srouter-215 opensips[15906]: Listening on
> Oct 2 15:57:41 srouter-215 opensips[15906]: udp: x.x.x.x [x.x.x.x]:9020
> Oct 2 15:57:41 srouter-215 opensips[15906]: udp: 
> x6:x6:x6:x6:x6:x6 [[x6:x6:0:0:x6:x6:x6:x6]]:9020
> Oct 2 15:57:41 srouter-215 opensips[15906]: Aliases:
> Oct 2 15:57:41 srouter-215 opensips[15906]: *: x.x.x.x:*
>
> real   0m0.066s
> user   0m0.003s
> sys    0m0.002s
> root at srouter-215:~# Oct  2 15:57:41 srouter-215 systemd[1]: Started 
> OpenSIPS is a very fast and flexible SIP (RFC3261) server.
>
> /var/log/opensips.log is here ... https://pastebin.com/8md6sBiP
>
> # time opensipsctl fifo reload_blacklist
> WARNING: database engine not found - tried 'MYSQL'
> Oct 2 16:17:24 srouter-215 SIPSrv[16185]: 
> INFO:userblacklist:reload_sources: got 1386901 entries from 
> 'globalblacklist'
>
> real   0m41.551s
> user   0m0.037s
> sys    0m0.015s
>
> # time systemctl stop opensips
> Oct  2 16:06:38 srouter-215 systemd[1]: Stopping OpenSIPS is a very 
> fast and flexible SIP (RFC3261) server...
> Oct  2 16:06:50 srouter-215 systemd[1]: Stopped OpenSIPS is a very 
> fast and flexible SIP (RFC3261) server.
>
> real    0m12.134s
> user   0m0.003s
> sys    0m0.002s
>
>
> Looks like issue has been fixed.. :)
>
> --
> regards,
>
> abdul basit
>
>
> On Tue, 2 Oct 2018 at 18:55, Bogdan-Andrei Iancu <bogdan at opensips.org 
> <mailto:bogdan at opensips.org>> wrote:
>
>     OK, I pushed on master a patch to move the initial data load after
>     the startup sequence. The patch should apply to 2.4 also.
>
>     https://github.com/OpenSIPS/opensips/commit/32cba1d751ee5a5b705da97ac4333fcad3d67728
>
>     While checking the module, I found some issues with it:
>
>         * before my patch, the load (for same data) was independently
>     done in each opensips process (even if the data is shared) :)
>
>         * the "reload" is no error pron - if the new data load fails,
>     the old existing data is lost
>
>         * during the data reload the module has no data, so it cannot
>     operate (it will actually block) - normally it should continue
>     with the old data until the new data is fully loaded and validated.
>
>     Let me know if the patch makes a different for you and I will
>     backport it to 2.4 too.
>
>     Regards,
>
>     Bogdan-Andrei Iancu
>
>     OpenSIPS Founder and Developer
>        http://www.opensips-solutions.com
>     OpenSIPS Bootcamp 2018
>        http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
>     On 10/02/2018 04:18 PM, Bogdan-Andrei Iancu wrote:
>>     Hi,
>>
>>     Taking a closer look at the module (to be honest I do not user
>>     it, as drouting does a similar job) I see the module is loading
>>     the data into memory. It is doing this during the init stage,
>>     holding back the entire OpenSIPS startup.
>>     Module doing similar heavy db load were changed to do the load
>>     post-startup. Let patch the module to decouple the loading from
>>     the startup, so that OpenSIPS will start without delays.
>>
>>     Regards,
>>     Bogdan-Andrei Iancu
>>
>>     OpenSIPS Founder and Developer
>>        http://www.opensips-solutions.com
>>     OpenSIPS Bootcamp 2018
>>        http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>     On 10/02/2018 04:04 PM, Jon Abrams wrote:
>>>     OpenSIPs is loading the global blacklist into a digit trie
>>>     structure into shared memory on startup (or reload_blacklist
>>>     command). That causes the startup delay.
>>>
>>>     - Jon
>>>
>>>     On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit <basit.engg at gmail.com
>>>     <mailto:basit.engg at gmail.com>> wrote:
>>>
>>>         Hi Bogdan,
>>>
>>>         I don't think so its translating to DB query on run time,
>>>         instead its caching the blacklist.
>>>         I am saying so because opensips response is quick, where as
>>>         when I query manually from DB, it take some time even prefix
>>>         col. is indexed.
>>>         Moreover, while opensips restart i get following warnings in
>>>         syslog:
>>>
>>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20181002/131bd404/attachment-0001.html>