[OpenSIPS-Users] Opensips crash
Bogdan-Andrei Iancu
bogdan at opensips.org
Thu Apr 6 04:08:10 EDT 2017
Hi Denis,
Please get some more info from the corefile :
f 10
p p_cell->method.s
p p_cell->uas.request
p p_cell->uas.status
Thanks,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Summit May 2017 Amsterdam
http://www.opensips.org/events/Summit-2017Amsterdam.html
On 04/05/2017 11:56 AM, Denis wrote:
> Ok, Bogdan, I'll be waiting any result.
> Thank you.
> --
> С уважением, Денис.
> Best regards, Denis
> 05.04.2017, 11:53, "Bogdan-Andrei Iancu" <bogdan at opensips.org>:
>> Hi Denis,
>>
>> Many thanks for the output - this confirm a suspicious of mine in
>> regards to the cause of the crash - let me try to see if we can
>> reproduce it on our side or if we need more info from your side.
>> Please do not delete the corefile for the moment :)
>>
>> Best regards,
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developer
>> http://www.opensips-solutions.com <http://www.opensips-solutions.com/>
>>
>> OpenSIPS Summit May 2017 Amsterdam
>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>> On 04/05/2017 11:03 AM, Denis wrote:
>>> Hello, Bogdan!
>>> f 5
>>> #5 0x00007fdf6844c4f2 in free_dlg_dlg (dlg=0x7fdf6c4ac158) at
>>> dlg_hash.c:183
>>> 183 destroy_dlg_callbacks_list(dlg->cbs.first);
>>> p *dlg->cbs.first
>>> $1 = {types = 96, callback = 0x7fdf67531300 <acc_dlg_callback>,
>>> param = 0x0, callback_param_free = 0x7fdf675318fe
>>> <dlg_free_acc_mask>, next = 0x7fdf84191080}
>>> p *dlg->cbs.first->next
>>> $2 = {types = 96, callback = 0x7fdf67531300 <acc_dlg_callback>,
>>> param = 0x7fdf83c9f340, callback_param_free = 0x7fdf675318fe
>>> <dlg_free_acc_mask>, next = 0x0}
>>> p *dlg->cbs.first->next->next
>>> Cannot access memory at address 0x0
>>> p *dlg->cbs.first->next->next->next
>>> Cannot access memory at address 0x20
>>> and so on with ....at address 0x20
>>> --
>>> С уважением, Денис.
>>> Best regards, Denis
>>> 04.04.2017, 18:43, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>> <mailto:bogdan at opensips.org>:
>>>> Super :)....we got to something useful.
>>>>
>>>> Now, again in GDB please do:
>>>>
>>>> f 5
>>>> p *dlg->cbs.first
>>>> p *dlg->cbs.first->next
>>>> p *dlg->cbs.first->next->next
>>>>
>>>> and keep adding "->next" until you get error from gdb.
>>>>
>>>> Thanks,
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developer
>>>> http://www.opensips-solutions.com <http://www.opensips-solutions.com/>
>>>>
>>>> OpenSIPS Summit May 2017 Amsterdam
>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>> On 04/04/2017 06:07 PM, Денис Путято wrote:
>>>>> Sorry, Bogdan, i missed that
>>>>> I found such CRITICAL message
>>>>> "CRITICAL:core:qm_free: freeing already freed pointer, first free:
>>>>> acc_logic.c: dlg_free_acc_mask(211) - aborting"
>>>>> And after that and till crash only
>>>>> "WARNING:core:utimer_ticker:...." reports in syslog.
>>>>> (gdb)
>>>>> f 2
>>>>> #2 0x0000000000515592 in qm_free (qm=<optimized out>,
>>>>> p=p at entry=0x7fdf83c9f340, file=file at entry=0x7fdf6753ad62
>>>>> "acc_logic.c",
>>>>> func=func at entry=0x7fdf6753af90 <__FUNCTION__.12529>
>>>>> "dlg_free_acc_mask", line=line at entry=211) at mem/q_malloc.c:483
>>>>> 483 abort();
>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>> $1 = {size = 8, u = {nxt_free = 0x7fdf71a8c6b0, is_free =
>>>>> 140597661320880}, file = 0x7fdf6753ad62 "acc_logic.c", func =
>>>>> 0x7fdf6753af90 <__FUNCTION__.12529> "dlg_free_acc_mask", line = 211,
>>>>> check = 17361641481138401520}
>>>>> --
>>>>> С уважением, Денис.
>>>>> Best regards, Denis
>>>>> 04.04.2017, 17:58, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>> <mailto:bogdan at opensips.org>:
>>>>>> The RADIUS part is not relevant.
>>>>>>
>>>>>> Still you didn;t answer to my question on the CRITICAL error -
>>>>>> according to the backtrace, you should have a CRITICAL message in
>>>>>> the opensips log, just before the crash.
>>>>>>
>>>>>> In GDB, do:
>>>>>> f 2
>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>
>>>>>> Regards,
>>>>>> Bogdan-Andrei Iancu
>>>>>> OpenSIPS Founder and Developer
>>>>>> http://www.opensips-solutions.com
>>>>>> <http://www.opensips-solutions.com/>
>>>>>>
>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>> On 04/04/2017 05:55 PM, Denis wrote:
>>>>>>> During crash there was another problem with LAN network.
>>>>>>> Because of this problem Opensips could not connect to radius server.
>>>>>>> rc_ip_hostname: couldn't look up host by addr: xxxxxx
>>>>>>> rc_send_server: no reply from RADIUS server unknown:1812
>>>>>>> No more critical messages.
>>>>>>> Bogdan, i am sorry, but
>>>>>>> "in frame 2". What does it mean?
>>>>>>> --
>>>>>>> С уважением, Денис.
>>>>>>> Best regards, Denis
>>>>>>> 04.04.2017, 16:35, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>>>> <mailto:bogdan at opensips.org>:
>>>>>>>> Any CRITICAL message into the log before the crash ?
>>>>>>>>
>>>>>>>> also, in gdb in frame 2, please print:
>>>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>> http://www.opensips-solutions.com
>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>
>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>> On 04/04/2017 03:19 PM, Denis wrote:
>>>>>>>>> p shm_block->first_frag
>>>>>>>>> $1 = (struct qm_frag *) 0x7fdf6b9d44f8
>>>>>>>>> p shm_block->last_frag_end
>>>>>>>>> $2 = (struct qm_frag_end *) 0x7fe02899efd0
>>>>>>>>> p mem_block->first_frag
>>>>>>>>> $3 = (struct qm_frag *) 0x7fe029214508
>>>>>>>>> p mem_block->last_frag_end
>>>>>>>>> $4 = (struct qm_frag_end *) 0x7fe0691defe0
>>>>>>>>> --
>>>>>>>>> С уважением, Денис.
>>>>>>>>> Best regards, Denis
>>>>>>>>> 04.04.2017, 14:51, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>>>>>> <mailto:bogdan at opensips.org>:
>>>>>>>>>> OK,please enter gdb and print:
>>>>>>>>>> p shm_block->first_frag
>>>>>>>>>> p shm_block->last_frag_end
>>>>>>>>>>
>>>>>>>>>> p mem_block->first_frag
>>>>>>>>>> p mem_block->last_frag_end
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>
>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>> On 04/04/2017 01:42 PM, Denis wrote:
>>>>>>>>>>> Hello, Bogdan!
>>>>>>>>>>> No, i cannot see in the log any messages you mentioned it
>>>>>>>>>>> your letter.
>>>>>>>>>>> The situation dealing with crash had such stage.
>>>>>>>>>>> 1. I prepared to migrate to 2.2.3 from 2.2.2. But do not
>>>>>>>>>>> have a time to restart Opensips
>>>>>>>>>>> 2. As a result "i waited" when Opensips have crashed. During
>>>>>>>>>>> crash of 2.2.2 no core file has been generated, but Opensips
>>>>>>>>>>> has been restarted using cron.
>>>>>>>>>>> 3. After 2) Opensips became 2.2.3. It was working about 1
>>>>>>>>>>> minute and crashed again, but in that time, core file has
>>>>>>>>>>> been generated and i attached information from it.
>>>>>>>>>>> So, between 2) and 3) , as i mentioned above, there were no
>>>>>>>>>>> messages about memory, but i see many messages like
>>>>>>>>>>> 'WARNING:core:utimer_ticker: utimer task <tm-utimer> already
>>>>>>>>>>> scheduled for 71500 ms
>>>>>>>>>>> WARNING:core:timer_ticker: timer task <pike-clean> already
>>>>>>>>>>> scheduled for 72400 ms
>>>>>>>>>>> WARNING:core:timer_ticker: timer task <nh-timer> already
>>>>>>>>>>> scheduled for 72400 ms
>>>>>>>>>>> WARNING:core:timer_ticker: timer task <dlg-timer> already
>>>>>>>>>>> scheduled for 72400 ms
>>>>>>>>>>> WARNING:core:timer_ticker: timer task <blcore-expire>
>>>>>>>>>>> already scheduled for 72400 ms'
>>>>>>>>>>> and so on
>>>>>>>>>>> As for about version
>>>>>>>>>>> version: opensips 2.2.3 (x86_64/linux)
>>>>>>>>>>> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
>>>>>>>>>>> PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>>>>>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
>>>>>>>>>>> MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>>>>>>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>>>>>>>>> git revision: 7922417
>>>>>>>>>>> main.c compiled on 06:29:28 Mar 23 2017 with gcc 5.4.0
>>>>>>>>>>> --
>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>> 04.04.2017, 12:43, "Bogdan-Andrei Iancu"
>>>>>>>>>>> <bogdan at opensips.org> <mailto:bogdan at opensips.org>:
>>>>>>>>>>>> Hello Denis,
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for the report. Do you see any CRITICAL log just
>>>>>>>>>>>> before the crash, something like:
>>>>>>>>>>>> "bad pointer 0xxxxxxxx (out of memory block!) - aborting"
>>>>>>>>>>>> ?
>>>>>>>>>>>> If yes, please post here the exact line.
>>>>>>>>>>>>
>>>>>>>>>>>> Also, do you use the latest OpenSIPS 2.2 from GIT repo ?
>>>>>>>>>>>> (run "opensips -v")
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>>>
>>>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>>>> On 04/04/2017 10:58 AM, Denis via Users wrote:
>>>>>>>>>>>>> Hello!
>>>>>>>>>>>>> Server:: OpenSIPS (2.2.3 (x86_64/linux))
>>>>>>>>>>>>> Thank you for any help.
>>>>>>>>>>>>> P.S. Opensips has been compiled with QM_MALLOC and
>>>>>>>>>>>>> DBG_MALLOC flags
>>>>>>>>>>>>> --
>>>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
>>>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170406/83596709/attachment-0001.html>
More information about the Users
mailing list