[OpenSIPS-Users] Opensips crash
Bogdan-Andrei Iancu
bogdan at opensips.org
Wed Apr 12 10:22:15 EDT 2017
Denis,
Could you put in an archive the core file and corresponding opensips
binary and modules - and have them available for download somewhere ?
and I try to look into on my side.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Summit May 2017 Amsterdam
http://www.opensips.org/events/Summit-2017Amsterdam.html
On 04/12/2017 05:11 PM, Denis wrote:
> Hello, Bogdan!
> Is there any additional information that i should gather about problem?
> Or everything is clear?
> Thank you.
> --
> С уважением, Денис.
> Best regards, Denis
> 06.04.2017, 15:18, "Denis via Users" <users at lists.opensips.org>:
>> Bogdan, I copied tar to the backup server and restore opensips from
>> tar there (on the same path as in main was).
>> Now, when entering to the gdb on backup, it does not show any warning
>> messages, but the result is the same
>> f 10
>> #0 0x0000000000000000 in ?? ()
>> (gdb) p p_cell->method.s
>> No symbol "p_cell" in current context.
>> (gdb) p p_cell->uas.request
>> No symbol "p_cell" in current context.
>> (gdb) p p_cell->uas.status
>> No symbol "p_cell" in current context.
>> --
>> С уважением, Денис.
>> Best regards, Denis
>> 06.04.2017, 14:38, "Bogdan-Andrei Iancu" <bogdan at opensips.org
>> <mailto:bogdan at opensips.org>>:
>>> Oh, that is good !
>>>
>>> When starting gdb, point to the old opensips binary. To solve the
>>> problem with the modules (which are libraries), see:
>>> http://stackoverflow.com/questions/84341/how-do-i-prepend-a-directory-the-library-path-when-loading-a-core-file-in-gdb-on
>>> (the first answer)
>>>
>>> Regards,
>>> Bogdan-Andrei Iancu
>>> OpenSIPS Founder and Developer
>>> http://www.opensips-solutions.com <http://www.opensips-solutions.com/>
>>>
>>> OpenSIPS Summit May 2017 Amsterdam
>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>> On 04/06/2017 12:31 PM, Denis wrote:
>>>> Before making upgrade i made tar archive of the directory where
>>>> opensips has been installed. Can this help us?
>>>> It was crash on time only. I do not think, that o can reproduce it.
>>>> --
>>>> С уважением, Денис.
>>>> Best regards, Denis
>>>> 06.04.2017, 12:27, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>> <mailto:bogdan at opensips.org>:
>>>>> yeah, the opensips binary and module libs do not match the
>>>>> corefile anymore :(.....Do you have the old build ? Also, was
>>>>> this crash on time only ? or can you reproduce it ?
>>>>>
>>>>> Regards,
>>>>> Bogdan-Andrei Iancu
>>>>> OpenSIPS Founder and Developer
>>>>> http://www.opensips-solutions.com <http://www.opensips-solutions.com/>
>>>>>
>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>> On 04/06/2017 12:17 PM, Denis wrote:
>>>>>> Hello, Bogdan.
>>>>>> Probably i made a big mistake.
>>>>>> I have already updated Opensips from latest git (to solve another
>>>>>> problem)
>>>>>> So, when i enter in gdb i get such warning message
>>>>>> [New LWP 14909]
>>>>>> warning: .dynamic section for
>>>>>> "/usr/local/opensips2.2/lib64/opensips/modules/dialog.so" is not
>>>>>> at the expected address (wrong library or version mismatch?)
>>>>>> Anyway,
>>>>>> f 10
>>>>>> #10 0x00007fdf6c4ac158 in ?? ()
>>>>>> p p_cell->method.s
>>>>>> No symbol "p_cell" in current context.
>>>>>> p p_cell->uas.request
>>>>>> No symbol "p_cell" in current context.
>>>>>> p p_cell->uas.status
>>>>>> No symbol "p_cell" in current context.
>>>>>> --
>>>>>> С уважением, Денис.
>>>>>> Best regards, Denis
>>>>>> 06.04.2017, 11:08, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>>> <mailto:bogdan at opensips.org>:
>>>>>>> Hi Denis,
>>>>>>>
>>>>>>> Please get some more info from the corefile :
>>>>>>>
>>>>>>> f 10
>>>>>>> p p_cell->method.s
>>>>>>> p p_cell->uas.request
>>>>>>> p p_cell->uas.status
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bogdan-Andrei Iancu
>>>>>>> OpenSIPS Founder and Developer
>>>>>>> http://www.opensips-solutions.com
>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>
>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>> On 04/05/2017 11:56 AM, Denis wrote:
>>>>>>>> Ok, Bogdan, I'll be waiting any result.
>>>>>>>> Thank you.
>>>>>>>> --
>>>>>>>> С уважением, Денис.
>>>>>>>> Best regards, Denis
>>>>>>>> 05.04.2017, 11:53, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>>>>> <mailto:bogdan at opensips.org>:
>>>>>>>>> Hi Denis,
>>>>>>>>>
>>>>>>>>> Many thanks for the output - this confirm a suspicious of mine
>>>>>>>>> in regards to the cause of the crash - let me try to see if we
>>>>>>>>> can reproduce it on our side or if we need more info from your
>>>>>>>>> side.
>>>>>>>>> Please do not delete the corefile for the moment :)
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>
>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>> On 04/05/2017 11:03 AM, Denis wrote:
>>>>>>>>>> Hello, Bogdan!
>>>>>>>>>> f 5
>>>>>>>>>> #5 0x00007fdf6844c4f2 in free_dlg_dlg (dlg=0x7fdf6c4ac158)
>>>>>>>>>> at dlg_hash.c:183
>>>>>>>>>> 183 destroy_dlg_callbacks_list(dlg->cbs.first);
>>>>>>>>>> p *dlg->cbs.first
>>>>>>>>>> $1 = {types = 96, callback = 0x7fdf67531300
>>>>>>>>>> <acc_dlg_callback>, param = 0x0, callback_param_free =
>>>>>>>>>> 0x7fdf675318fe <dlg_free_acc_mask>, next = 0x7fdf84191080}
>>>>>>>>>> p *dlg->cbs.first->next
>>>>>>>>>> $2 = {types = 96, callback = 0x7fdf67531300
>>>>>>>>>> <acc_dlg_callback>, param = 0x7fdf83c9f340,
>>>>>>>>>> callback_param_free = 0x7fdf675318fe <dlg_free_acc_mask>,
>>>>>>>>>> next = 0x0}
>>>>>>>>>> p *dlg->cbs.first->next->next
>>>>>>>>>> Cannot access memory at address 0x0
>>>>>>>>>> p *dlg->cbs.first->next->next->next
>>>>>>>>>> Cannot access memory at address 0x20
>>>>>>>>>> and so on with ....at address 0x20
>>>>>>>>>> --
>>>>>>>>>> С уважением, Денис.
>>>>>>>>>> Best regards, Denis
>>>>>>>>>> 04.04.2017, 18:43, "Bogdan-Andrei Iancu"
>>>>>>>>>> <bogdan at opensips.org> <mailto:bogdan at opensips.org>:
>>>>>>>>>>> Super :)....we got to something useful.
>>>>>>>>>>>
>>>>>>>>>>> Now, again in GDB please do:
>>>>>>>>>>>
>>>>>>>>>>> f 5
>>>>>>>>>>> p *dlg->cbs.first
>>>>>>>>>>> p *dlg->cbs.first->next
>>>>>>>>>>> p *dlg->cbs.first->next->next
>>>>>>>>>>>
>>>>>>>>>>> and keep adding "->next" until you get error from gdb.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>>
>>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>>> On 04/04/2017 06:07 PM, Денис Путято wrote:
>>>>>>>>>>>> Sorry, Bogdan, i missed that
>>>>>>>>>>>> I found such CRITICAL message
>>>>>>>>>>>> "CRITICAL:core:qm_free: freeing already freed pointer,
>>>>>>>>>>>> first free: acc_logic.c: dlg_free_acc_mask(211) - aborting"
>>>>>>>>>>>> And after that and till crash only
>>>>>>>>>>>> "WARNING:core:utimer_ticker:...." reports in syslog.
>>>>>>>>>>>> (gdb)
>>>>>>>>>>>> f 2
>>>>>>>>>>>> #2 0x0000000000515592 in qm_free (qm=<optimized out>,
>>>>>>>>>>>> p=p at entry=0x7fdf83c9f340, file=file at entry=0x7fdf6753ad62
>>>>>>>>>>>> "acc_logic.c",
>>>>>>>>>>>> func=func at entry=0x7fdf6753af90 <__FUNCTION__.12529>
>>>>>>>>>>>> "dlg_free_acc_mask", line=line at entry=211) at mem/q_malloc.c:483
>>>>>>>>>>>> 483 abort();
>>>>>>>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>>>>>>> $1 = {size = 8, u = {nxt_free = 0x7fdf71a8c6b0, is_free =
>>>>>>>>>>>> 140597661320880}, file = 0x7fdf6753ad62 "acc_logic.c", func
>>>>>>>>>>>> = 0x7fdf6753af90 <__FUNCTION__.12529> "dlg_free_acc_mask",
>>>>>>>>>>>> line = 211,
>>>>>>>>>>>> check = 17361641481138401520}
>>>>>>>>>>>> --
>>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>>> 04.04.2017, 17:58, "Bogdan-Andrei Iancu"
>>>>>>>>>>>> <bogdan at opensips.org> <mailto:bogdan at opensips.org>:
>>>>>>>>>>>>> The RADIUS part is not relevant.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Still you didn;t answer to my question on the CRITICAL
>>>>>>>>>>>>> error - according to the backtrace, you should have a
>>>>>>>>>>>>> CRITICAL message in the opensips log, just before the crash.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In GDB, do:
>>>>>>>>>>>>> f 2
>>>>>>>>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>>>>
>>>>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>>>>> On 04/04/2017 05:55 PM, Denis wrote:
>>>>>>>>>>>>>> During crash there was another problem with LAN network.
>>>>>>>>>>>>>> Because of this problem Opensips could not connect to
>>>>>>>>>>>>>> radius server.
>>>>>>>>>>>>>> rc_ip_hostname: couldn't look up host by addr: xxxxxx
>>>>>>>>>>>>>> rc_send_server: no reply from RADIUS server unknown:1812
>>>>>>>>>>>>>> No more critical messages.
>>>>>>>>>>>>>> Bogdan, i am sorry, but
>>>>>>>>>>>>>> "in frame 2". What does it mean?
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>>>>> 04.04.2017, 16:35, "Bogdan-Andrei Iancu"
>>>>>>>>>>>>>> <bogdan at opensips.org> <mailto:bogdan at opensips.org>:
>>>>>>>>>>>>>>> Any CRITICAL message into the log before the crash ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> also, in gdb in frame 2, please print:
>>>>>>>>>>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>>>>>>> On 04/04/2017 03:19 PM, Denis wrote:
>>>>>>>>>>>>>>>> p shm_block->first_frag
>>>>>>>>>>>>>>>> $1 = (struct qm_frag *) 0x7fdf6b9d44f8
>>>>>>>>>>>>>>>> p shm_block->last_frag_end
>>>>>>>>>>>>>>>> $2 = (struct qm_frag_end *) 0x7fe02899efd0
>>>>>>>>>>>>>>>> p mem_block->first_frag
>>>>>>>>>>>>>>>> $3 = (struct qm_frag *) 0x7fe029214508
>>>>>>>>>>>>>>>> p mem_block->last_frag_end
>>>>>>>>>>>>>>>> $4 = (struct qm_frag_end *) 0x7fe0691defe0
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>>>>>>> 04.04.2017, 14:51, "Bogdan-Andrei Iancu"
>>>>>>>>>>>>>>>> <bogdan at opensips.org> <mailto:bogdan at opensips.org>:
>>>>>>>>>>>>>>>>> OK,please enter gdb and print:
>>>>>>>>>>>>>>>>> p shm_block->first_frag
>>>>>>>>>>>>>>>>> p shm_block->last_frag_end
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> p mem_block->first_frag
>>>>>>>>>>>>>>>>> p mem_block->last_frag_end
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>>>>>>>>> On 04/04/2017 01:42 PM, Denis wrote:
>>>>>>>>>>>>>>>>>> Hello, Bogdan!
>>>>>>>>>>>>>>>>>> No, i cannot see in the log any messages you
>>>>>>>>>>>>>>>>>> mentioned it your letter.
>>>>>>>>>>>>>>>>>> The situation dealing with crash had such stage.
>>>>>>>>>>>>>>>>>> 1. I prepared to migrate to 2.2.3 from 2.2.2. But do
>>>>>>>>>>>>>>>>>> not have a time to restart Opensips
>>>>>>>>>>>>>>>>>> 2. As a result "i waited" when Opensips have crashed.
>>>>>>>>>>>>>>>>>> During crash of 2.2.2 no core file has been
>>>>>>>>>>>>>>>>>> generated, but Opensips has been restarted using cron.
>>>>>>>>>>>>>>>>>> 3. After 2) Opensips became 2.2.3. It was working
>>>>>>>>>>>>>>>>>> about 1 minute and crashed again, but in that time,
>>>>>>>>>>>>>>>>>> core file has been generated and i attached
>>>>>>>>>>>>>>>>>> information from it.
>>>>>>>>>>>>>>>>>> So, between 2) and 3) , as i mentioned above, there
>>>>>>>>>>>>>>>>>> were no messages about memory, but i see many
>>>>>>>>>>>>>>>>>> messages like
>>>>>>>>>>>>>>>>>> 'WARNING:core:utimer_ticker: utimer task <tm-utimer>
>>>>>>>>>>>>>>>>>> already scheduled for 71500 ms
>>>>>>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <pike-clean>
>>>>>>>>>>>>>>>>>> already scheduled for 72400 ms
>>>>>>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <nh-timer>
>>>>>>>>>>>>>>>>>> already scheduled for 72400 ms
>>>>>>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <dlg-timer>
>>>>>>>>>>>>>>>>>> already scheduled for 72400 ms
>>>>>>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <blcore-expire>
>>>>>>>>>>>>>>>>>> already scheduled for 72400 ms'
>>>>>>>>>>>>>>>>>> and so on
>>>>>>>>>>>>>>>>>> As for about version
>>>>>>>>>>>>>>>>>> version: opensips 2.2.3 (x86_64/linux)
>>>>>>>>>>>>>>>>>> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
>>>>>>>>>>>>>>>>>> PKG_MALLOC, QM_MALLOC, DBG_MALLOC,
>>>>>>>>>>>>>>>>>> FAST_LOCK-ADAPTIVE_WAIT
>>>>>>>>>>>>>>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE
>>>>>>>>>>>>>>>>>> 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>>>>>>>>>>>>>>>> poll method support: poll, epoll_lt, epoll_et,
>>>>>>>>>>>>>>>>>> sigio_rt, select.
>>>>>>>>>>>>>>>>>> git revision: 7922417
>>>>>>>>>>>>>>>>>> main.c compiled on 06:29:28 Mar 23 2017 with gcc 5.4.0
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>>>>>>>>> 04.04.2017, 12:43, "Bogdan-Andrei Iancu"
>>>>>>>>>>>>>>>>>> <bogdan at opensips.org> <mailto:bogdan at opensips.org>:
>>>>>>>>>>>>>>>>>>> Hello Denis,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you for the report. Do you see any CRITICAL
>>>>>>>>>>>>>>>>>>> log just before the crash, something like:
>>>>>>>>>>>>>>>>>>> "bad pointer 0xxxxxxxx (out of memory block!) -
>>>>>>>>>>>>>>>>>>> aborting"
>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>> If yes, please post here the exact line.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Also, do you use the latest OpenSIPS 2.2 from GIT
>>>>>>>>>>>>>>>>>>> repo ? (run "opensips -v")
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>>>>>>>>>>> On 04/04/2017 10:58 AM, Denis via Users wrote:
>>>>>>>>>>>>>>>>>>>> Hello!
>>>>>>>>>>>>>>>>>>>> Server:: OpenSIPS (2.2.3 (x86_64/linux))
>>>>>>>>>>>>>>>>>>>> Thank you for any help.
>>>>>>>>>>>>>>>>>>>> P.S. Opensips has been compiled with QM_MALLOC and
>>>>>>>>>>>>>>>>>>>> DBG_MALLOC flags
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>>>>>>>> Users at lists.opensips.org
>>>>>>>>>>>>>>>>>>>> <mailto:Users at lists.opensips.org>
>>>>>>>>>>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>> ,
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170412/fb5813cc/attachment-0001.html>
More information about the Users
mailing list