Fwd: [Users] Openser fails when compiled on Solaris 64bit

Jeffrey Magder jmagder at somanetworks.com
Thu Mar 29 22:28:19 CEST 2007


Thanks!  It appears that cb.container's 'find' function-pointer is not 
initialized.  I'm not sure why that is happening on your system, 
especially since it looks like the other function pointers are set up.  
The strange thing is that this isn't happening on Linux (as far as I 
know!).  I'll have to do some searching and see if the NetSNMP libraries 
handle setting up cb.container differently on Solaris and Linux.

Has anybody on Solaris managed to get the snmpstats module running?  Has 
anyone on Linux had a similar crash?

Sergio Gutierrez wrote:
> Hi Jeffrey.
>
> These are the printings you asked:
>
> $1 = {row_copy = 0, row_compare = 0, get_value = 0xfe98a0c0 
> <openserSIPPortTable_get_value>, can_activate = 0, activated = 0, 
> can_deactivate = 0,
>   deactivated = 0, can_delete = 0, create_row = 0, duplicate_row = 0, 
> delete_row = 0, set_reserve1 = 0, set_reserve2 = 0, set_action = 0, 
> set_commit = 0,
>   set_free = 0, set_undo = 0, container = 0x2e8510, can_set = 0 '\0'}
>
> (gdb) print *cb.container
> $3 = {private = 0x2e6b48, get_size = 0xfe76c238 <_ba_size>, init = 0, 
> cfree = 0xfe76c224 <_ba_free>, insert = 0xfe76c1f0 <_ba_insert>,
>   remove = 0xfe76c208 <_ba_remove>, find = 0, find_next = 0xfe76c1b8 
> <_ba_find>, get_subset = 0xfe76c1d4 <_ba_find_next>, get_iterator = 
> 0xfe76c288,
>   for_each = 0xfe76c9ec <_ba_iterator_get>, compare = 0xfe76c24c 
> <_ba_for_each>, ncompare = 0xfe76c26c <_ba_clear>, next = 0x0, prev = 
> 0xfe76b254}
>
> (gdb) print cb.container->find
> $5 = (netsnmp_container_rtn *) 0
>
>
> Thanks!
>
> On 3/29/07, *Jeffrey Magder* <jmagder at somanetworks.com 
> <mailto:jmagder at somanetworks.com>> wrote:
>
>     Sergio, could you set a breakpoint at line 125 in
>     openserSIPPortTable.c,
>     and display the values of: cb, cb.container, and cb.container->find?
>
>     Jeffrey Magder wrote:
>     > You are correct, the crash is happening in the SNMPStats
>     module.  I'll
>     > take a look!
>     >
>     > - JM
>     >
>     > Daniel-Constantin Mierla wrote:
>     >> Hello,
>     >>
>     >> this seems to be in snmpstats module ... maybe the developer
>     can give
>     >> some hints.
>     >>
>     >> Cheers,
>     >> Daniel
>     >>
>     >>
>     >> On 03/29/07 19:29, Sergio Gutierrez wrote:
>     >>>
>     >>> Hi Daniel.
>     >>>
>     >>> In the meantime, I have begun to test using more standard
>     >>> conditions; I am not using the optimized compiler and I am running
>     >>> openser compiled on 32 bits.
>     >>>
>     >>> I have discovered an even nastier issue. When Openser is
>     configured
>     >>> in forking mode, and with or without log_stderror, it presents the
>     >>> same symptom I reported when compiled on 64 bits (it exhausts the
>     >>> shared memory reservation, and crashes with core dumping).
>     >>>
>     >>> This is the backtrace:
>     >>>
>     >>> (gdb) bt
>     >>> #0  0x00000000 in ?? ()
>     >>> #1  0xfe989b3c in getRow (ipType=1, ipAddress=0x141888) at
>     >>> openserSIPPortTable.c:125
>     >>> #2  0xfe989c5c in createRowsFromIPList (theList=0x141a50,
>     >>> listSize=1, protocol=0, snmpIndex=0xffbff870) at
>     >>> openserSIPPortTable.c:201
>     >>> #3  0xfe98a07c in init_openserSIPPortTable () at
>     >>> openserSIPPortTable.c:241
>     >>> #4  0xfe98d708 in spawn_agentx_child () at sub_agent.c:74
>     >>> #5  0xfe98cf04 in mod_child_init (rank=0) at snmpstats.c:271
>     >>> #6  0x00059ca8 in init_mod_child (m=0x133840, rank=0,
>     type=0xc3c08
>     >>> "PROC_MAIN") at sr_module.c:406
>     >>> #7  0x00059c24 in init_mod_child (m=0x1338e0, rank=0, type=0xc3c08
>     >>> "PROC_MAIN") at sr_module.c:395
>     >>> #8  0x000336c4 in main_loop () at main.c:952
>     >>> #9  0x00035be4 in main (argc=9, argv=0xe7800) at main.c:1399
>     >>>
>     >>> When I configure fork=no, openser does not crash, and works right.
>     >>>
>     >>> I apologize if I should better create a new thread. If you
>     suggest
>     >>> so I will do it.
>     >>>
>     >>> Thanks in advance for your attention.
>     >>>
>     >>> Kind regards
>     >>>
>     >>> Sergio Gutierrez.
>     >>> EPM Telecomunicaciones
>     >>> Medellín, Colombia
>     >>> Tel: 574 - 3950022
>     >>>
>     >>>
>     >>>
>     >>>
>     >>>         > ---------- Forwarded message ----------
>     >>>         > From: *Sergio Gutierrez* < saguti at gmail.com
>     <mailto:saguti at gmail.com>
>     >>>         <mailto:saguti at gmail.com <mailto:saguti at gmail.com>>
>     <mailto: saguti at gmail.com <mailto:saguti at gmail.com>
>     >>>         <mailto: saguti at gmail.com <mailto:saguti at gmail.com>>>>
>     >>>         > Date: Mar 27, 2007 9:52 AM
>     >>>         > Subject: Re: [Users] Openser fails when compiled on
>     >>> Solaris 64bit
>     >>>         > To: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro> <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>>
>     >>>         <mailto: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>> <mailto:daniel at voice-system.ro <mailto:daniel at voice-system.ro>>>
>     >>>         > Cc: users at openser.org <mailto:users at openser.org>
>     <mailto: users at openser.org <mailto:users at openser.org>>
>     >>>         <mailto:users at openser.org <mailto:users at openser.org>
>     <mailto:users at openser.org <mailto:users at openser.org>>>
>     >>>         >
>     >>>         > Hi Daniel.
>     >>>         >
>     >>>         > I am using mysql version 5.0.37, compiled on 64 bits
>     too.
>     >>>         >
>     >>>         > These are the printing of the symbols you asked:
>     >>>         >
>     >>>         > (gdb) print row
>     >>>         > $1 = (db_row_t *) 0x10026bf68
>     >>>         > (gdb) print ROW_VALUES(row)
>     >>>         > No symbol "ROW_VALUES" in current context.
>     >>>         > (gdb) print VAL_NULL(ROW_VALUES(row)
>     >>>         > No symbol "VAL_NULL" in current context.
>     >>>         > (gdb)
>     >>>         >
>     >>>         >
>     >>>         > Thanks.
>     >>>         >
>     >>>         >
>     >>>         > On 3/27/07, *Daniel-Constantin Mierla* <
>     >>>         daniel at voice-system.ro <mailto:daniel at voice-system.ro>
>     <mailto:daniel at voice-system.ro <mailto:daniel at voice-system.ro>>
>     >>>         > <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>>         <mailto: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>>>> wrote:
>     >>>         >
>     >>>         >     Hello Sergio,
>     >>>         >
>     >>>         >     seems to be some data corrupted from what database
>     >>>         returned. What
>     >>>         >     is the
>     >>>         >     database type you use, mysql?
>     >>>         >
>     >>>         >     Could you print the the content of row and
>     >>>         ROW_VALUES(row) (via print
>     >>>         >     command in gdb)?
>     >>>         >
>     >>>         >     Thanks,
>     >>>         >     Daniel
>     >>>         >
>     >>>         >
>     >>>         >     On 03/26/07 23:46, Sergio Gutierrez wrote:
>     >>>         >     > Hi again Daniel.
>     >>>         >     >
>     >>>         >     > Testing other things trying to solve the problem I
>     >>>         found that the
>     >>>         >     > segmentation fault occurs when Openser is
>     restarted and
>     >>>         there are
>     >>>         >     > records within location database table.
>     >>>         >     > If the table is empty initially, openser
>     starts OK, and
>     >>>         users can
>     >>>         >     > register without problem.
>     >>>         >     >
>     >>>         >     > Below is the backtrace of a error when the
>     location
>     >>>         table is not
>     >>>         >     empty:
>     >>>         >     >
>     >>>         >     > #0  0xffffffff7d90e2a0 in preload_udomain
>     >>> (_c=0x10026ad90,
>     >>>         >     > _d=0xffffffff5028e9f0) at udomain.c:404
>     >>>         >     > 404                             if
>     >>>         (VAL_NULL(ROW_VALUES(row))
>     >>>         >     > (gdb) bt
>     >>>         >     > #0  0xffffffff7d90e2a0 in preload_udomain
>     >>> (_c=0x10026ad90,
>     >>>         >     > _d=0xffffffff5028e9f0) at udomain.c:404
>     >>>         >     > #1  0xffffffff7d915844 in child_init (_rank=1) at
>     >>>         ul_mod.c:344
>     >>>         >     > #2  0x0000000100082500 in init_mod_child (m=0x1,
>     >>> rank=1,
>     >>>         >     > type=0x1000f41b0 "CHILD") at
>     >>>         >     >
>     /home/operador/openser-1.2.0-notls//sr_module.c:400
>     >>>         >     > #3  0x0000000100082460 in init_mod_child (m=0x1,
>     >>> rank=1,
>     >>>         >     > type=0x1000f41b0 "CHILD") at
>     >>>         >     >
>     /home/operador/openser-1.2.0-notls//sr_module.c:394
>     >>>         >     > #4  0x0000000100082460 in init_mod_child (m=0x1,
>     >>> rank=1,
>     >>>         >     > type=0x1000f41b0 "CHILD") at
>     >>>         >     >
>     /home/operador/openser-1.2.0-notls//sr_module.c:394
>     >>>         >     > #5  0x0000000100082770 in init_child (rank=1) at
>     >>>         >     >
>     /home/operador/openser-1.2.0-notls//sr_module.c:394
>     >>>         >     > #6  0x0000000100032414 in main_loop () at
>     >>>         >     > /home/operador/openser-1.2.0-notls//main.c:724
>     >>>         >     > #7  0x00000001000345a4 in main (argc=2,
>     >>>         argv=0xffffff7eaeeec4ff) at
>     >>>         >     > /home/operador/openser-1.2.0-notls//main.c:1399
>     >>>         >     >
>     >>>         >     >
>     >>>         >     > Thanks.
>     >>>         >     >
>     >>>         >     >
>     >>>         >     >
>     >>>         >     >
>     >>>         >     >
>     >>>         >     > On 3/26/07, *Sergio Gutierrez* <
>     saguti at gmail.com <mailto:saguti at gmail.com>
>     >>>         <mailto:saguti at gmail.com <mailto:saguti at gmail.com>>
>     >>>         >     <mailto:saguti at gmail.com
>     <mailto:saguti at gmail.com> <mailto:saguti at gmail.com
>     <mailto:saguti at gmail.com>>>
>     >>>         >     > <mailto: saguti at gmail.com
>     <mailto:saguti at gmail.com> <mailto:saguti at gmail.com
>     <mailto:saguti at gmail.com>>
>     >>>         <mailto:saguti at gmail.com <mailto:saguti at gmail.com>
>     <mailto:saguti at gmail.com <mailto:saguti at gmail.com>>>>> wrote:
>     >>>         >     >
>     >>>         >     >     Hi Daniel.
>     >>>         >     >
>     >>>         >     >     This is the backtrace of the core.
>     >>>         >     >     Thanks
>     >>>         >     >
>     >>>         >     >     #0  0xffffffff7d90ee9c in preload_udomain
>     >>>         (_c=0x10026b7d800,
>     >>>         >     >     _d=0x0) at
>     >>>         >     >     /home/operador/openser-
>     >>>         1.2.0-notls/modules/usrloc//udomain.c:404
>     >>>         >     >     #1  0xffffffff7d9131c4 in child_init
>     >>>         (_rank=1344858528) at
>     >>>         >     >     /home/operador/openser-
>     >>>         1.2.0-notls/modules/usrloc//ul_mod.c:344
>     >>>         >     >     #2  0x0000000100082500 in init_mod_child
>     (m=0x1,
>     >>>         rank=1,
>     >>>         >     >     type=0x1000f41b0 "CHILD") at
>     >>>         >     >     /home/operador/openser-
>     >>> 1.2.0-notls//sr_module.c:400
>     >>>         >     >     #3  0x0000000100082460 in init_mod_child
>     (m=0x1,
>     >>>         rank=1,
>     >>>         >     >     type=0x1000f41b0 "CHILD") at
>     >>>         >     >     /home/operador/openser-1.2.0-notls
>     >>> //sr_module.c:394
>     >>>         >     >     #4  0x0000000100082460 in init_mod_child
>     (m=0x1,
>     >>>         rank=1,
>     >>>         >     >     type=0x1000f41b0 "CHILD") at
>     >>>         >     >     /home/operador/openser- 1.2.0-notls
>     >>> //sr_module.c:394
>     >>>         >     >     #5  0x0000000100082770 in init_child
>     (rank=1) at
>     >>>         >     >    
>     /home/operador/openser-1.2.0-notls//sr_module.c:394
>     >>>         >     >     #6  0x0000000100032414 in main_loop () at
>     >>>         >     >     /home/operador/openser-
>     1.2.0-notls//main.c:724
>     >>>         >     >     #7  0x00000001000345a4 in main (argc=2,
>     >>>         argv=0xffffff7e7007a000)
>     >>>         >     >     at
>     /home/operador/openser-1.2.0-notls//main.c:1399
>     >>>         >     >
>     >>>         >     >
>     >>>         >     >
>     >>>         >     >
>     >>>         >     >     On 3/26/07, *Daniel-Constantin Mierla* <
>     >>>         >     daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro> <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>>
>     >>>         <mailto: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>> <mailto: daniel at voice-system.ro <mailto:daniel at voice-system.ro>>>
>     >>>         >     >     <mailto: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>>         <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>>
>     >>>         >     <mailto: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>>         <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>>>>> wrote:
>     >>>         >     >
>     >>>         >     >         Hello,
>     >>>         >     >
>     >>>         >     >         can you send a gdb backtrace :-D -- I
>     cannot
>     >>>         read adb
>     >>>         >     backtrace.
>     >>>         >     >
>     >>>         >     >         Cheers,
>     >>>         >     >         Daniel
>     >>>         >     >
>     >>>         >     >
>     >>>         >     >         On 03/26/07 19:42, Sergio Gutierrez wrote:
>     >>>         >     >         > Hi Daniel.
>     >>>         >     >         >
>     >>>         >     >         > This is the backtrace, obtained with
>     adb:
>     >>>         >     >         >
>     >>>         >     >         > usrloc.so`preload_udomain+0x4cc (1,
>     >>>         ffffffff7228e9f0,
>     >>>         >     100215a00,
>     >>>         >     >         > ffffffff7d919e58, ffffffffffefe3f8, 0)
>     >>>         >     >         > 0xffffffff7d9131bc(2, ffffffff7da1c9b0,
>     >>>         ffffffff7d91a170,
>     >>>         >     >         > ffffffff7228e9a0, ffffffff7da1b0a8,
>     >>>         ffffffffffeff0c8)
>     >>>         >     >         > init_mod_child+0xd8(100269930, 1,
>     1000f41b0,
>     >>>         1000f40f0, 1,
>     >>>         >     >         100269870)
>     >>>         >     >         > init_mod_child+0x38(100269bb0, 1,
>     1000f41b0,
>     >>>         1000f40f0, 1,
>     >>>         >     >         1002699f0)
>     >>>         >     >         > init_mod_child+0x38(100269d30, 1,
>     1000f41b0,
>     >>>         1000f40f0, 1,
>     >>>         >     >         100269c70)
>     >>>         >     >         > init_child+0xa8(1, 100269df0, 1,
>     1000f4168,
>     >>>         10020d000,
>     >>>         >     100269eb0)
>     >>>         >     >         > main_loop+0xf34(0, 8c, 1000eefd8,
>     >>>         ffffffff720083c4,
>     >>>         >     1002694b0, 0)
>     >>>         >     >         > main+0x1e10(100215, 9, 0, 10020d000,
>     >>>         1000ef5b8, 10020d000)
>     >>>         >     >         > _start+0x7c(0, 0, 0, 0, 0, 0)
>     >>>         >     >         >
>     >>>         >     >         > On 3/26/07, *Sergio Gutierrez*
>     >>>         <saguti at gmail.com <mailto:saguti at gmail.com>
>     <mailto:saguti at gmail.com <mailto:saguti at gmail.com>>
>     >>>         >     <mailto: saguti at gmail.com
>     <mailto:saguti at gmail.com> <mailto:saguti at gmail.com
>     <mailto:saguti at gmail.com>>>
>     >>>         >     >         <mailto: saguti at gmail.com
>     <mailto:saguti at gmail.com>
>     >>>         <mailto:saguti at gmail.com <mailto:saguti at gmail.com>>
>     <mailto: saguti at gmail.com <mailto:saguti at gmail.com>
>     >>>         <mailto: saguti at gmail.com <mailto:saguti at gmail.com>>>>
>     >>>         >     >         > <mailto: saguti at gmail.com
>     <mailto:saguti at gmail.com>
>     >>>         <mailto:saguti at gmail.com <mailto:saguti at gmail.com>>
>     <mailto: saguti at gmail.com <mailto:saguti at gmail.com>
>     >>>         <mailto: saguti at gmail.com <mailto:saguti at gmail.com>>>
>     >>>         >     <mailto:saguti at gmail.com
>     <mailto:saguti at gmail.com> <mailto:saguti at gmail.com
>     <mailto:saguti at gmail.com>>
>     >>>         <mailto: saguti at gmail.com <mailto:saguti at gmail.com>
>     <mailto:saguti at gmail.com <mailto:saguti at gmail.com>>>>>> wrote:
>     >>>         >     >         >
>     >>>         >     >         >     Hi Daniel.
>     >>>         >     >         >
>     >>>         >     >         >     I used 512 for -m parameter.
>     >>>         >     >         >
>     >>>         >     >         >     Would you like the backtrace of
>     the core
>     >>>         file?
>     >>>         >     >         >
>     >>>         >     >         >     Thanks.
>     >>>         >     >         >
>     >>>         >     >         >
>     >>>         >     >         >     On 3/26/07, *Daniel-Constantin
>     Mierla* <
>     >>>         >     >         daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>>         <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>> <mailto:
>     >>>         daniel at voice-system.ro <mailto:daniel at voice-system.ro>
>     <mailto:daniel at voice-system.ro <mailto:daniel at voice-system.ro>>>
>     >>>         >     <mailto: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>>         <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>> <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>>         <mailto: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>>>>
>     >>>         >     >         >     <mailto: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>>         <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>>
>     >>>         >     <mailto: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>>         <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>>>
>     >>>         >     >         <mailto: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>>         <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>>
>     >>>         >     <mailto: daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>
>     >>>         <mailto:daniel at voice-system.ro
>     <mailto:daniel at voice-system.ro>>>>>> wrote:
>     >>>         >     >         >
>     >>>         >     >         >         Hello,
>     >>>         >     >         >
>     >>>         >     >         >         could you send a backtrace?
>     What was
>     >>>         the value for
>     >>>         >     >         parameter -m?
>     >>>         >     >         >
>     >>>         >     >         >         Cheers,
>     >>>         >     >         >         Daniel
>     >>>         >     >         >
>     >>>         >     >         >
>     >>>         >     >         >         On 03/26/07 19:29, Sergio
>     Gutierrez
>     >>>         wrote:
>     >>>         >     >         >         > Hi
>     >>>         >     >         >         >
>     >>>         >     >         >         > I am trying to run Openser
>     compiled
>     >>>         on 64
>     >>>         >     bits on a
>     >>>         >     >         SPARC
>     >>>         >     >         >         Machine
>     >>>         >     >         >         > running Solaris 10.
>     >>>         >     >         >         >
>     >>>         >     >         >         > When I create a simple
>     >>>         configuration file for
>     >>>         >     >         testing radius
>     >>>         >     >         >         > integration, Openser starts to
>     >>>         consume the whole
>     >>>         >     >         memory
>     >>>         >     >         >         reservation
>     >>>         >     >         >         > (-m parameter) and fails with
>     >>>         segmentation
>     >>>         >     fault error.
>     >>>         >     >         >         >
>     >>>         >     >         >         > after several tests, I
>     have found
>     >>>         that the
>     >>>         >     error is
>     >>>         >     >         caused by
>     >>>         >     >         >         save()
>     >>>         >     >         >         > function (registrar module).
>     >>>         >     >         >         >
>     >>>         >     >         >         > This is the main route my
>     >>>         configuration file:
>     >>>         >     >         >         >
>     >>>         >     >         >         > route {
>     >>>         >     >         >         >         if(method=="REGISTER")
>     >>>         >     >         >         >         {
>     >>>         >     >         >                >
>     >>> if(!radius_www_authorize(""))
>     >>>         >     >         >         >                {
>     >>>         >     >         >                >
>     >>> www_challenge("", "0");
>     >>>         >     >         >         >                        return;
>     >>>         >     >         >         >                };
>     >>>         >     >         >         >
>     >>>         >     >         >         >
>     >>> if(!save("location"))
>     >>>         >     >         >         >                {
>     >>>         >     >         >                >
>     >>> sl_reply_error();
>     >>>         >     >         >         >                };
>     >>>         >     >         >         >                return;
>     >>>         >     >         >         >
>     >>>         >     >         >         >         }
>     >>>         >     >         >         >         else
>     >>>         >     >         >         >         {
>     >>>         >     >         >         >         }
>     >>>         >     >         >         > }
>     >>>         >     >         >         >
>     >>>         >     >         >         > Thanks in advance for your
>     help.
>     >>>         >     >         >         >
>     >>>         >     >         >         > Sergio G.
>     >>>         >     >         >         >
>     >>>         >     >         >         >
>     >>>         >     >
>     >>>         >
>     >>>
>     ------------------------------------------------------------------------
>     >>>
>     >>>         >
>     >>>         >     >         >         >
>     >>>         >     >         >         >
>     >>>         _______________________________________________
>     >>>         >     >         >         > Users mailing list
>     >>>         >     >         >         > Users at openser.org
>     <mailto:Users at openser.org>
>     >>>         <mailto:Users at openser.org <mailto:Users at openser.org>>
>     <mailto: Users at openser.org <mailto:Users at openser.org>
>     >>>         <mailto:Users at openser.org <mailto:Users at openser.org>>>
>     >>>         >     <mailto: Users at openser.org
>     <mailto:Users at openser.org> <mailto:Users at openser.org
>     <mailto:Users at openser.org>>
>     >>>         <mailto: Users at openser.org <mailto:Users at openser.org>
>     <mailto:Users at openser.org <mailto:Users at openser.org>>>>
>     >>>         >     >         <mailto: Users at openser.org
>     <mailto:Users at openser.org>
>     >>>         <mailto:Users at openser.org <mailto:Users at openser.org>>
>     <mailto: Users at openser.org <mailto:Users at openser.org>
>     >>>         <mailto:Users at openser.org <mailto:Users at openser.org>>>
>     >>>         >     <mailto: Users at openser.org
>     <mailto:Users at openser.org> <mailto:Users at openser.org
>     <mailto:Users at openser.org>>
>     >>>         <mailto: Users at openser.org <mailto:Users at openser.org>
>     <mailto:Users at openser.org <mailto:Users at openser.org>>>>>
>     >>>         >     >         >         >
>     >>>         http://openser.org/cgi-bin/mailman/listinfo/users
>     >>>         >     >         <
>     >>>         http://openser.org/cgi-bin/mailman/listinfo/users
>     >>>         < http://openser.org/cgi-bin/mailman/listinfo/users>
>     >>>         >     <http://openser.org/cgi-bin/mailman/listinfo/users>>
>     >>>         >     >         >         >
>     >>>         >     >         >
>     >>>         >     >         >
>     >>>         >     >         >
>     >>>         >     >
>     >>>         >     >
>     >>>         >     >
>     >>>         >
>     >>>         >
>     >>>
>     >>>
>     >>>
>     >>
>     >> _______________________________________________
>     >> Users mailing list
>     >> Users at openser.org <mailto:Users at openser.org>
>     >> http://openser.org/cgi-bin/mailman/listinfo/users
>     >
>     >
>     > _______________________________________________
>     > Users mailing list
>     > Users at openser.org <mailto:Users at openser.org>
>     > http://openser.org/cgi-bin/mailman/listinfo/users
>
>
>     _______________________________________________
>     Users mailing list
>     Users at openser.org <mailto:Users at openser.org>
>     http://openser.org/cgi-bin/mailman/listinfo/users
>
>





More information about the Users mailing list