[OpenSIPS-Users] Opensips crash
Bogdan-Andrei Iancu
bogdan at opensips.org
Thu Apr 6 07:37:39 EDT 2017
Oh, that is good !
When starting gdb, point to the old opensips binary. To solve the
problem with the modules (which are libraries), see:
http://stackoverflow.com/questions/84341/how-do-i-prepend-a-directory-the-library-path-when-loading-a-core-file-in-gdb-on
(the first answer)
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Summit May 2017 Amsterdam
http://www.opensips.org/events/Summit-2017Amsterdam.html
On 04/06/2017 12:31 PM, Denis wrote:
> Before making upgrade i made tar archive of the directory where
> opensips has been installed. Can this help us?
> It was crash on time only. I do not think, that o can reproduce it.
> --
> С уважением, Денис.
> Best regards, Denis
> 06.04.2017, 12:27, "Bogdan-Andrei Iancu" <bogdan at opensips.org>:
>> yeah, the opensips binary and module libs do not match the corefile
>> anymore :(.....Do you have the old build ? Also, was this crash on
>> time only ? or can you reproduce it ?
>>
>> Regards,
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developer
>> http://www.opensips-solutions.com <http://www.opensips-solutions.com/>
>>
>> OpenSIPS Summit May 2017 Amsterdam
>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>> On 04/06/2017 12:17 PM, Denis wrote:
>>> Hello, Bogdan.
>>> Probably i made a big mistake.
>>> I have already updated Opensips from latest git (to solve another
>>> problem)
>>> So, when i enter in gdb i get such warning message
>>> [New LWP 14909]
>>> warning: .dynamic section for
>>> "/usr/local/opensips2.2/lib64/opensips/modules/dialog.so" is not at
>>> the expected address (wrong library or version mismatch?)
>>> Anyway,
>>> f 10
>>> #10 0x00007fdf6c4ac158 in ?? ()
>>> p p_cell->method.s
>>> No symbol "p_cell" in current context.
>>> p p_cell->uas.request
>>> No symbol "p_cell" in current context.
>>> p p_cell->uas.status
>>> No symbol "p_cell" in current context.
>>> --
>>> С уважением, Денис.
>>> Best regards, Denis
>>> 06.04.2017, 11:08, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>> <mailto:bogdan at opensips.org>:
>>>> Hi Denis,
>>>>
>>>> Please get some more info from the corefile :
>>>>
>>>> f 10
>>>> p p_cell->method.s
>>>> p p_cell->uas.request
>>>> p p_cell->uas.status
>>>>
>>>> Thanks,
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developer
>>>> http://www.opensips-solutions.com <http://www.opensips-solutions.com/>
>>>>
>>>> OpenSIPS Summit May 2017 Amsterdam
>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>> On 04/05/2017 11:56 AM, Denis wrote:
>>>>> Ok, Bogdan, I'll be waiting any result.
>>>>> Thank you.
>>>>> --
>>>>> С уважением, Денис.
>>>>> Best regards, Denis
>>>>> 05.04.2017, 11:53, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>> <mailto:bogdan at opensips.org>:
>>>>>> Hi Denis,
>>>>>>
>>>>>> Many thanks for the output - this confirm a suspicious of mine in
>>>>>> regards to the cause of the crash - let me try to see if we can
>>>>>> reproduce it on our side or if we need more info from your side.
>>>>>> Please do not delete the corefile for the moment :)
>>>>>>
>>>>>> Best regards,
>>>>>> Bogdan-Andrei Iancu
>>>>>> OpenSIPS Founder and Developer
>>>>>> http://www.opensips-solutions.com
>>>>>> <http://www.opensips-solutions.com/>
>>>>>>
>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>> On 04/05/2017 11:03 AM, Denis wrote:
>>>>>>> Hello, Bogdan!
>>>>>>> f 5
>>>>>>> #5 0x00007fdf6844c4f2 in free_dlg_dlg (dlg=0x7fdf6c4ac158) at
>>>>>>> dlg_hash.c:183
>>>>>>> 183 destroy_dlg_callbacks_list(dlg->cbs.first);
>>>>>>> p *dlg->cbs.first
>>>>>>> $1 = {types = 96, callback = 0x7fdf67531300 <acc_dlg_callback>,
>>>>>>> param = 0x0, callback_param_free = 0x7fdf675318fe
>>>>>>> <dlg_free_acc_mask>, next = 0x7fdf84191080}
>>>>>>> p *dlg->cbs.first->next
>>>>>>> $2 = {types = 96, callback = 0x7fdf67531300 <acc_dlg_callback>,
>>>>>>> param = 0x7fdf83c9f340, callback_param_free = 0x7fdf675318fe
>>>>>>> <dlg_free_acc_mask>, next = 0x0}
>>>>>>> p *dlg->cbs.first->next->next
>>>>>>> Cannot access memory at address 0x0
>>>>>>> p *dlg->cbs.first->next->next->next
>>>>>>> Cannot access memory at address 0x20
>>>>>>> and so on with ....at address 0x20
>>>>>>> --
>>>>>>> С уважением, Денис.
>>>>>>> Best regards, Denis
>>>>>>> 04.04.2017, 18:43, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>>>> <mailto:bogdan at opensips.org>:
>>>>>>>> Super :)....we got to something useful.
>>>>>>>>
>>>>>>>> Now, again in GDB please do:
>>>>>>>>
>>>>>>>> f 5
>>>>>>>> p *dlg->cbs.first
>>>>>>>> p *dlg->cbs.first->next
>>>>>>>> p *dlg->cbs.first->next->next
>>>>>>>>
>>>>>>>> and keep adding "->next" until you get error from gdb.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>> http://www.opensips-solutions.com
>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>
>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>> On 04/04/2017 06:07 PM, Денис Путято wrote:
>>>>>>>>> Sorry, Bogdan, i missed that
>>>>>>>>> I found such CRITICAL message
>>>>>>>>> "CRITICAL:core:qm_free: freeing already freed pointer, first
>>>>>>>>> free: acc_logic.c: dlg_free_acc_mask(211) - aborting"
>>>>>>>>> And after that and till crash only
>>>>>>>>> "WARNING:core:utimer_ticker:...." reports in syslog.
>>>>>>>>> (gdb)
>>>>>>>>> f 2
>>>>>>>>> #2 0x0000000000515592 in qm_free (qm=<optimized out>,
>>>>>>>>> p=p at entry=0x7fdf83c9f340, file=file at entry=0x7fdf6753ad62
>>>>>>>>> "acc_logic.c",
>>>>>>>>> func=func at entry=0x7fdf6753af90 <__FUNCTION__.12529>
>>>>>>>>> "dlg_free_acc_mask", line=line at entry=211) at mem/q_malloc.c:483
>>>>>>>>> 483 abort();
>>>>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>>>> $1 = {size = 8, u = {nxt_free = 0x7fdf71a8c6b0, is_free =
>>>>>>>>> 140597661320880}, file = 0x7fdf6753ad62 "acc_logic.c", func =
>>>>>>>>> 0x7fdf6753af90 <__FUNCTION__.12529> "dlg_free_acc_mask", line
>>>>>>>>> = 211,
>>>>>>>>> check = 17361641481138401520}
>>>>>>>>> --
>>>>>>>>> С уважением, Денис.
>>>>>>>>> Best regards, Denis
>>>>>>>>> 04.04.2017, 17:58, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>>>>>> <mailto:bogdan at opensips.org>:
>>>>>>>>>> The RADIUS part is not relevant.
>>>>>>>>>>
>>>>>>>>>> Still you didn;t answer to my question on the CRITICAL error
>>>>>>>>>> - according to the backtrace, you should have a CRITICAL
>>>>>>>>>> message in the opensips log, just before the crash.
>>>>>>>>>>
>>>>>>>>>> In GDB, do:
>>>>>>>>>> f 2
>>>>>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>
>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>> On 04/04/2017 05:55 PM, Denis wrote:
>>>>>>>>>>> During crash there was another problem with LAN network.
>>>>>>>>>>> Because of this problem Opensips could not connect to radius
>>>>>>>>>>> server.
>>>>>>>>>>> rc_ip_hostname: couldn't look up host by addr: xxxxxx
>>>>>>>>>>> rc_send_server: no reply from RADIUS server unknown:1812
>>>>>>>>>>> No more critical messages.
>>>>>>>>>>> Bogdan, i am sorry, but
>>>>>>>>>>> "in frame 2". What does it mean?
>>>>>>>>>>> --
>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>> 04.04.2017, 16:35, "Bogdan-Andrei Iancu"
>>>>>>>>>>> <bogdan at opensips.org> <mailto:bogdan at opensips.org>:
>>>>>>>>>>>> Any CRITICAL message into the log before the crash ?
>>>>>>>>>>>>
>>>>>>>>>>>> also, in gdb in frame 2, please print:
>>>>>>>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>>>
>>>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>>>> On 04/04/2017 03:19 PM, Denis wrote:
>>>>>>>>>>>>> p shm_block->first_frag
>>>>>>>>>>>>> $1 = (struct qm_frag *) 0x7fdf6b9d44f8
>>>>>>>>>>>>> p shm_block->last_frag_end
>>>>>>>>>>>>> $2 = (struct qm_frag_end *) 0x7fe02899efd0
>>>>>>>>>>>>> p mem_block->first_frag
>>>>>>>>>>>>> $3 = (struct qm_frag *) 0x7fe029214508
>>>>>>>>>>>>> p mem_block->last_frag_end
>>>>>>>>>>>>> $4 = (struct qm_frag_end *) 0x7fe0691defe0
>>>>>>>>>>>>> --
>>>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>>>> 04.04.2017, 14:51, "Bogdan-Andrei Iancu"
>>>>>>>>>>>>> <bogdan at opensips.org> <mailto:bogdan at opensips.org>:
>>>>>>>>>>>>>> OK,please enter gdb and print:
>>>>>>>>>>>>>> p shm_block->first_frag
>>>>>>>>>>>>>> p shm_block->last_frag_end
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> p mem_block->first_frag
>>>>>>>>>>>>>> p mem_block->last_frag_end
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>>>>>> On 04/04/2017 01:42 PM, Denis wrote:
>>>>>>>>>>>>>>> Hello, Bogdan!
>>>>>>>>>>>>>>> No, i cannot see in the log any messages you mentioned
>>>>>>>>>>>>>>> it your letter.
>>>>>>>>>>>>>>> The situation dealing with crash had such stage.
>>>>>>>>>>>>>>> 1. I prepared to migrate to 2.2.3 from 2.2.2. But do not
>>>>>>>>>>>>>>> have a time to restart Opensips
>>>>>>>>>>>>>>> 2. As a result "i waited" when Opensips have crashed.
>>>>>>>>>>>>>>> During crash of 2.2.2 no core file has been generated,
>>>>>>>>>>>>>>> but Opensips has been restarted using cron.
>>>>>>>>>>>>>>> 3. After 2) Opensips became 2.2.3. It was working about
>>>>>>>>>>>>>>> 1 minute and crashed again, but in that time, core file
>>>>>>>>>>>>>>> has been generated and i attached information from it.
>>>>>>>>>>>>>>> So, between 2) and 3) , as i mentioned above, there were
>>>>>>>>>>>>>>> no messages about memory, but i see many messages like
>>>>>>>>>>>>>>> 'WARNING:core:utimer_ticker: utimer task <tm-utimer>
>>>>>>>>>>>>>>> already scheduled for 71500 ms
>>>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <pike-clean>
>>>>>>>>>>>>>>> already scheduled for 72400 ms
>>>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <nh-timer> already
>>>>>>>>>>>>>>> scheduled for 72400 ms
>>>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <dlg-timer>
>>>>>>>>>>>>>>> already scheduled for 72400 ms
>>>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <blcore-expire>
>>>>>>>>>>>>>>> already scheduled for 72400 ms'
>>>>>>>>>>>>>>> and so on
>>>>>>>>>>>>>>> As for about version
>>>>>>>>>>>>>>> version: opensips 2.2.3 (x86_64/linux)
>>>>>>>>>>>>>>> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
>>>>>>>>>>>>>>> PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>>>>>>>>>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
>>>>>>>>>>>>>>> MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>>>>>>>>>>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt,
>>>>>>>>>>>>>>> select.
>>>>>>>>>>>>>>> git revision: 7922417
>>>>>>>>>>>>>>> main.c compiled on 06:29:28 Mar 23 2017 with gcc 5.4.0
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>>>>>> 04.04.2017, 12:43, "Bogdan-Andrei Iancu"
>>>>>>>>>>>>>>> <bogdan at opensips.org> <mailto:bogdan at opensips.org>:
>>>>>>>>>>>>>>>> Hello Denis,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you for the report. Do you see any CRITICAL log
>>>>>>>>>>>>>>>> just before the crash, something like:
>>>>>>>>>>>>>>>> "bad pointer 0xxxxxxxx (out of memory block!) -
>>>>>>>>>>>>>>>> aborting"
>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>> If yes, please post here the exact line.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, do you use the latest OpenSIPS 2.2 from GIT repo
>>>>>>>>>>>>>>>> ? (run "opensips -v")
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>>>>>>>> On 04/04/2017 10:58 AM, Denis via Users wrote:
>>>>>>>>>>>>>>>>> Hello!
>>>>>>>>>>>>>>>>> Server:: OpenSIPS (2.2.3 (x86_64/linux))
>>>>>>>>>>>>>>>>> Thank you for any help.
>>>>>>>>>>>>>>>>> P.S. Opensips has been compiled with QM_MALLOC and
>>>>>>>>>>>>>>>>> DBG_MALLOC flags
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>>>>> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
>>>>>>>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170406/32aed9ea/attachment-0001.html>
More information about the Users
mailing list