[Users] memory leak in presence module?

Daniel-Constantin Mierla daniel at voice-system.ro
Thu Apr 26 16:04:43 CEST 2007


Hello,

can we get the sipp scenario from you, looking at the stop dump 
(http://pernau.at/kd/openser/debug-stop3.txt):

Apr 26 12:55:26 ds3000 /usr/sbin/openser[25443]: Memory status (shm): 
Apr 26 12:55:26 ds3000 /usr/sbin/openser[25443]: qm_status (0xb38d3000): 
Apr 26 12:55:26 ds3000 /usr/sbin/openser[25443]:  heap size= 67108864 
Apr 26 12:55:26 ds3000 /usr/sbin/openser[25443]:  used= 780, used+overhead=1240976, free=65867888 
Apr 26 12:55:26 ds3000 /usr/sbin/openser[25443]:  max used (+overhead)= 66953980


Whis says kind of shm free size is 65867888. We have to reproduce for a 
better hunting.

Cheers,
Daniel




On 04/26/07 16:59, Klaus Darilion wrote:
> Hi!
>
> I also tried now with QM_JOIN_FREE but the same result:
> The shared mem (64M) is full after ~8000 PUBLISH (=8000 NOTIFY) and 
> does not get freed:
>
> darilion at ds3000:~/software/openser-svn-1.2/openser$ sudo openserctl 
> fifo get_statistics all
> core:rcv_requests = 8743
> core:rcv_replies = 8198
> core:fwd_requests = 0
> core:fwd_replies = 0
> core:drop_requests = 0
> core:drop_replies = 0
> core:err_requests = 0
> core:err_replies = 0
> core:bad_URIs_rcvd = 0
> core:unsupported_methods = 0
> core:bad_msg_hdr = 0
> shmem:total_size = 67108864
> shmem:used_size = 65865712
> shmem:real_used_size = 67004532
> shmem:max_used_size = 67108376
> shmem:free_size = 104332
> shmem:fragments = 239
> sl:1xx_replies = 0
> sl:2xx_replies = 8221
> sl:3xx_replies = 0
> sl:4xx_replies = 196
> sl:5xx_replies = 186
> sl:6xx_replies = 0
> sl:sent_replies = 8603
> sl:sent_err_replies = 186
> sl:received_ACKs = 0
> tm:received_replies = 8198
> tm:relayed_replies = 8198
> tm:local_replies = 0
> tm:UAS_transactions = 8218
> tm:UAC_transactions = 8200
> tm:2xx_transactions = 8198
> tm:3xx_transactions = 0
> tm:4xx_transactions = 0
> tm:5xx_transactions = 0
> tm:6xx_transactions = 0
> tm:inuse_transactions = 8217
> usrloc:registered_users = 0
> usrloc:location-users = 0
> usrloc:location-contacts = 0
> usrloc:location-expires = 0
> registrar:max_expires = 0
> registrar:max_contacts = 0
> registrar:default_expire = 3600
> registrar:accepted_regs = 1
> registrar:rejected_regs = 0
> darilion at ds3000:~/software/openser-svn-1.2/openser$
>
>
> regards
> klaus
>
>
>
> Klaus Darilion wrote:
>> memory logs:
>>
>> kill -SIGUSR1:
>> http://pernau.at/kd/openser/debug-kill-usr1.txt
>>
>> openser stop:
>> http://pernau.at/kd/openser/debug-stop3.txt
>>
>> regards
>> klaus
>>
>> Klaus Darilion wrote:
>>>
>>> Openser runs out of mem:
>>> $ sudo openserctl fifo get_statistics all
>>> core:rcv_requests = 13893
>>> core:rcv_replies = 18007
>>> core:fwd_requests = 0
>>> core:fwd_replies = 0
>>> core:drop_requests = 0
>>> core:drop_replies = 0
>>> core:err_requests = 0
>>> core:err_replies = 0
>>> core:bad_URIs_rcvd = 0
>>> core:unsupported_methods = 0
>>> core:bad_msg_hdr = 0
>>> shmem:total_size = 67108864
>>> shmem:used_size = 65351684
>>> shmem:real_used_size = 66591880
>>> shmem:max_used_size = 66953980
>>> shmem:free_size = 516984
>>> shmem:fragments = 1052
>>> sl:1xx_replies = 0
>>> sl:2xx_replies = 8878
>>> sl:3xx_replies = 0
>>> sl:4xx_replies = 2322
>>> sl:5xx_replies = 2308
>>> sl:6xx_replies = 0
>>> sl:sent_replies = 13508
>>> sl:sent_err_replies = 2308
>>> sl:received_ACKs = 0
>>> tm:received_replies = 18007
>>> tm:relayed_replies = 17652
>>> tm:local_replies = 1
>>> tm:UAS_transactions = 8861
>>> tm:UAC_transactions = 17653
>>> tm:2xx_transactions = 17652
>>> tm:3xx_transactions = 0
>>> tm:4xx_transactions = 1
>>> tm:5xx_transactions = 0
>>> tm:6xx_transactions = 0
>>> tm:inuse_transactions = 8861
>>> usrloc:registered_users = 2
>>> usrloc:location-users = 2
>>> usrloc:location-contacts = 2
>>> usrloc:location-expires = 0
>>> registrar:max_expires = 0
>>> registrar:max_contacts = 0
>>> registrar:default_expire = 3600
>>> registrar:accepted_regs = 12
>>> registrar:rejected_regs = 0
>>>
>>>
>>> after 1h of inactivity:
>>> $ sudo openserctl fifo get_statistics all
>>> Password:
>>> core:rcv_requests = 14285
>>> core:rcv_replies = 18007
>>> core:fwd_requests = 0
>>> core:fwd_replies = 0
>>> core:drop_requests = 0
>>> core:drop_replies = 0
>>> core:err_requests = 0
>>> core:err_replies = 0
>>> core:bad_URIs_rcvd = 0
>>> core:unsupported_methods = 0
>>> core:bad_msg_hdr = 0
>>> shmem:total_size = 67108864
>>> shmem:used_size = 65351684
>>> shmem:real_used_size = 66591880
>>> shmem:max_used_size = 66953980
>>> shmem:free_size = 516984
>>> shmem:fragments = 1052
>>> sl:1xx_replies = 0
>>> sl:2xx_replies = 8922
>>> sl:3xx_replies = 0
>>> sl:4xx_replies = 2481
>>> sl:5xx_replies = 2497
>>> sl:6xx_replies = 0
>>> sl:sent_replies = 13900
>>> sl:sent_err_replies = 2497
>>> sl:received_ACKs = 0
>>> tm:received_replies = 18007
>>> tm:relayed_replies = 17652
>>> tm:local_replies = 1
>>> tm:UAS_transactions = 8861
>>> tm:UAC_transactions = 17653
>>> tm:2xx_transactions = 17652
>>> tm:3xx_transactions = 0
>>> tm:4xx_transactions = 1
>>> tm:5xx_transactions = 0
>>> tm:6xx_transactions = 0
>>> tm:inuse_transactions = 8861
>>> usrloc:registered_users = 2
>>> usrloc:location-users = 2
>>> usrloc:location-contacts = 2
>>> usrloc:location-expires = 0
>>> registrar:max_expires = 0
>>> registrar:max_contacts = 0
>>> registrar:default_expire = 3600
>>> registrar:accepted_regs = 51
>>> registrar:rejected_regs = 0
>>>
>>>
>>>
>>> regards
>>> klaus
>>>
>>> Daniel-Constantin Mierla wrote:
>>>> my mistake:
>>>>
>>>> # openserctl fifo get_statistics all
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 04/26/07 12:36, Klaus Darilion wrote:
>>>>> # openserctl fifo get_statistics
>>>>> 400 Too few or too many arguments
>>>>>
>>>>> :-(
>>>>>
>>>>> I'm using 1.2
>>>>>
>>>>>
>>>>> regards
>>>>> klaus
>>>>>
>>>>>
>>>>>
>>>>> Klaus Darilion wrote:
>>>>>>
>>>>>>
>>>>>> Daniel-Constantin Mierla wrote:
>>>>>>> Hello Klaus,
>>>>>>>
>>>>>>> if that is the full log, seems no mem leak in shm. What is the 
>>>>>>> shm memory size you use?
>>>>>>
>>>>>> I don't know - I use the default values. But the size should be 
>>>>>> irrelevant as the "no memory left" is still there after 20 
>>>>>> minutes of inactivity.
>>>>>>
>>>>>>> Once you get out of memory can you issue" openserctl fifo 
>>>>>>> get_statistics" and send output?
>>>>>> ok
>>>>>>>
>>>>>>> I don't think is fragmentation, but you can try to compile with 
>>>>>>> -DQM_JOIN_FREE
>>>>>>>
>>>>>> ok
>>>>>>> Do you do the tests over TLS?
>>>>>> ok
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Daniel
>>>>>>>
>>>>>>>
>>>>>>> On 04/26/07 10:17, Klaus Darilion wrote:
>>>>>>>> btw: there is no
>>>>>>>> "PRESENCE:notify: ERROR in function tmb.t_request_within"
>>>>>>>> in my log file, thus I guess the problem is somewhere else.
>>>>>>>>
>>>>>>>> regards
>>>>>>>> klaus
>>>>>>>>
>>>>>>>> Klaus Darilion wrote:
>>>>>>>>> Hi Anca!
>>>>>>>>>
>>>>>>>>> There is still a leak:
>>>>>>>>> http://pernau.at/kd/openser/debug-stop2.txt
>>>>>>>>>
>>>>>>>>> regards
>>>>>>>>> klaus
>>>>>>>>>
>>>>>>>>> Anca-Maria Vamanu wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I have made a fix. Please test again with sipp and say if it 
>>>>>>>>>> works now.
>>>>>>>>>>
>>>>>>>>>> regards,
>>>>>>>>>>
>>>>>>>>>> Anca
>>>>>>>>>>
>>>>>>>>>> Klaus Darilion wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi!
>>>>>>>>>>>
>>>>>>>>>>> Openser 1.2 runs out of memory during presence load tests 
>>>>>>>>>>> with sipp.
>>>>>>>>>>>
>>>>>>>>>>> Config snippet:
>>>>>>>>>>>
>>>>>>>>>>> if (is_method("PUBLISH")) {
>>>>>>>>>>>   xlog("L_INFO","$ci PUBLISH detected ... outside 
>>>>>>>>>>> t_newtran\n");
>>>>>>>>>>>   if (t_newtran()) {
>>>>>>>>>>>     xlog("L_INFO","$ci PUBLISH detected ... inside 
>>>>>>>>>>> t_newtran\n");
>>>>>>>>>>>     handle_publish();
>>>>>>>>>>>     xlog("L_INFO","$ci PUBLISH detected, handle_publish ... 
>>>>>>>>>>> done\n");
>>>>>>>>>>>     exit;
>>>>>>>>>>>   } else {
>>>>>>>>>>>     sl_reply_error();
>>>>>>>>>>>     exit;
>>>>>>>>>>>   }
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> Memory log after openser stop (after 20 minutes of inactivity):
>>>>>>>>>>> http://pernau.at/kd/openser/debug-stop.txt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> regards
>>>>>>>>>>> klaus Klaus Darilion wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Cesc!
>>>>>>>>>>>>
>>>>>>>>>>>> Where do you have t_newtran() - in ser or openser?
>>>>>>>>>>>>
>>>>>>>>>>>> I have it here in openser just before handle_publish(). It 
>>>>>>>>>>>> works but now openser has run out of mem during my sipp 
>>>>>>>>>>>> performance tests :-(
>>>>>>>>>>>>
>>>>>>>>>>>> ...more to come....
>>>>>>>>>>>>
>>>>>>>>>>>> Cesc wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I think I can answer that one ... stateless?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have a presence openser proxied by a SER, both on same 
>>>>>>>>>>>>> machine ...
>>>>>>>>>>>>> For easier configuration purposes, openser and ser 
>>>>>>>>>>>>> communicate via
>>>>>>>>>>>>> 127.0.0.1 ...
>>>>>>>>>>>>>
>>>>>>>>>>>>> The result is that OK from the PUAs do not get to openser 
>>>>>>>>>>>>> (i guess SER
>>>>>>>>>>>>> gets confussed by the vias of openser being 0.0.0.0:5060 
>>>>>>>>>>>>> and the
>>>>>>>>>>>>> 127.0.0.1 stuff) ... So, openser gets no OK, but it does 
>>>>>>>>>>>>> not matter
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> BTW, i have a t_newtran() in there ...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Oh, for sure my configuration is messy and improvable ... 
>>>>>>>>>>>>> but for now
>>>>>>>>>>>>> it works steady :) But the OKs not going through is 
>>>>>>>>>>>>> worrying me a bit
>>>>>>>>>>>>> ... and also openser not "timing out" or giving an error.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cesc
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 4/25/07, Klaus Darilion <klaus.mailinglists at pernau.at> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Does presence module handles PUBLISH/SUBSCRIBE stateful 
>>>>>>>>>>>>>> or stateless?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thus, how should retransmission be handled. Will they be 
>>>>>>>>>>>>>> detected by
>>>>>>>>>>>>>> presence module or is it possible to wrap the 
>>>>>>>>>>>>>> handle_publish inside
>>>>>>>>>>>>>> t_newtran()?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>> klaus
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>> Users at openser.org
>>>>>>>>>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>> Users at openser.org
>>>>>>>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Users mailing list
>>>>>>>>>>> Users at openser.org
>>>>>>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list
>>>>>>>>> Users at openser.org
>>>>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at openser.org
>>>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users at openser.org
>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at openser.org
>>> http://openser.org/cgi-bin/mailman/listinfo/users
>




More information about the Users mailing list