[OpenSIPS-Users] Opensips hang for 30 seconds from time to time

[Xi Jin]

Hi Team

Our opensips production server has run for almost 5 years with no issue
until recently we notice little hiccups from the server graph.

The symptom is:
The opensips stops responding to any packets for 30 seconds, and then back
up online and continue process requests.
We noticed during that 30 seconds, the CPU of opensips processors hit 100
and opensipsctl trap complaints "Can't access memory address"

The server hosts about 35k subscribers and 1500 concurrent calls during
peak hours. Opensips only handles signalling, media is handled by rtpengine
servers and upstream SBC gateways.
This happens almost every day in the production environment Mon-Fri, and
less frequently during weekends. There is no big boost of subscriber
numbers or calls in past couple of months.

We are running opensips 2.4.8 on Debian 9, installed from opensips
APT(which is not available now), It's an anycast deployment
opensips -V
version: opensips 2.4.8 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC,
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on  with gcc 6.3.0

opensipsctl trap output has been attached (replaced some sensitive data)

Just wondering if anyone can shed some lights on this?

Let me know if extra info needs to be provided.

Regards,
Will J
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20230731/26891e0b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gdb_opensips_20230727_123835
Type: application/octet-stream
Size: 555622 bytes
Desc: not available
URL: <http://lists.opensips.org/pipermail/users/attachments/20230731/26891e0b/attachment-0001.obj>