[Chilli] Segfault Chilli 1.2.9
David Bird
david at coova.com
Tue Mar 6 19:22:52 UTC 2012
Ouch, that isn't a good length. Can you do a backtrace in gdb ('bt'
command)?
On Tue, 2012-03-06 at 15:57 -0300, Fernando Macedo wrote:
> GDB output of segfault:
>
> root# gdb /usr/local/sbin/chilli
>
> GNU gdb (GDB) 7.1-ubuntu
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/local/sbin/chilli...done.
> (gdb) run -c /usr/local/etc/chilli.conf --maxclients=1024
> --dhcpopt=0104FFFFFC00 --fg
> Starting program: /usr/local/sbin/chilli -c /usr/local/etc/chilli.conf
> --maxclients=1024 --dhcpopt=0104FFFFFC00 --fg
> [Thread debugging using libthread_db enabled]
>
> Program received signal SIGSEGV, Segmentation fault.
> dhcp_gettag (pack=0x7ffffffedc00, length=18446744073709551608,
> tag=0x7ffffffecdc8, tagtype=55 '7') at dhcp.c:2914
> 2914 if (t->t == tagtype) {
> (gdb)
>
>
> Em 02/03/2012 14:13, Fernando Macedo escreveu:
> > Again:
> >
> > recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> > pkttype=PACKET_HOST, addr(6)={1, 0022fbce29a0},
> > msg_iov(1)=[{"\232M\"\336X\16\0\"\373\316)\240\10\0E\0\5\2146\273@\0\200\6\317J\2176\250\20\270H"...,
> > 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> > cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 1434
> > write(4,
> > "E\0\5\2146\273@\0\200\6\317J\2176\250\20\270H\377\326\302\354\1\273c\224\3\350~5\365\241"...,
> > 1420) = 1420
> > select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) = 1
> > (in [5], left {0, 999997})
> > clock_gettime(CLOCK_MONOTONIC, {170230, 11114377}) = 0
> > recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> > pkttype=PACKET_HOST, addr(6)={1, 0022fbce29a0},
> > msg_iov(1)=[{"\232M\"\336X\16\0\"\373\316)\240\10\0E\0\5\2146\274@\0\200\6\317I\2176\250\20\270H"...,
> > 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> > cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 1434
> > write(4,
> > "E\0\5\2146\274@\0\200\6\317I\2176\250\20\270H\377\326\302\354\1\273c\224\tL~5\365\241"...,
> > 1420) = 1420
> > select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) = 1
> > (in [5], left {0, 999997})
> > clock_gettime(CLOCK_MONOTONIC, {170230, 11421407}) = 0
> > recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> > pkttype=PACKET_HOST, addr(6)={1, 0022fbce29a0},
> > msg_iov(1)=[{"\232M\"\336X\16\0\"\373\316)\240\10\0E\0\5\2146\275@\0\200\6\317H\2176\250\20\270H"...,
> > 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> > cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 1434
> > write(4,
> > "E\0\5\2146\275@\0\200\6\317H\2176\250\20\270H\377\326\302\354\1\273c\224\16\260~5\365\241"...,
> > 1420) = 1420
> > select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) = 1
> > (in [5], left {0, 999997})
> > clock_gettime(CLOCK_MONOTONIC, {170230, 11759495}) = 0
> > recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> > pkttype=PACKET_HOST, addr(6)={1, 0022fbce29a0},
> > msg_iov(1)=[{"\232M\"\336X\16\0\"\373\316)\240\10\0E\0\00146\276@\0\200\6\323\237\2176\250\20\270H"...,
> > 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> > cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 322
> > write(4,
> > "E\0\00146\276@\0\200\6\323\237\2176\250\20\270H\377\326\302\354\1\273c\224\24\24~5\365\241"...,
> > 308) = 308
> > select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) = 1
> > (in [5], left {0, 999997})
> > clock_gettime(CLOCK_MONOTONIC, {170230, 12066338}) = 0
> > recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> > pkttype=PACKET_HOST, addr(6)={1, 0022fbce29a0},
> > msg_iov(1)=[{"\232M\"\336X\16\0\"\373\316)\240\10\0E\0\5\2146\277@\0\200\6\317F\2176\250\20\270H"...,
> > 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> > cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 1434
> > write(4,
> > "E\0\5\2146\277@\0\200\6\317F\2176\250\20\270H\377\326\302\354\1\273c\224\25
> > ~5\365\241"..., 1420) = 1420
> > select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) = 1
> > (in [5], left {0, 999997})
> > clock_gettime(CLOCK_MONOTONIC, {170230, 12373554}) = 0
> > recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> > pkttype=PACKET_HOST, addr(6)={1, 0022fbce29a0},
> > msg_iov(1)=[{"\232M\"\336X\16\0\"\373\316)\240\10\0E\0\2\0226\300@\0\200\6\322\277\2176\250\20\270H"...,
> > 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> > cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 544
> > write(4,
> > "E\0\2\0226\300@\0\200\6\322\277\2176\250\20\270H\377\326\302\354\1\273c\224\32\204~5\365\241"...,
> > 530) = 530
> > select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) = 1
> > (in [5], left {0, 999997})
> > clock_gettime(CLOCK_MONOTONIC, {170230, 12705270}) = 0
> > recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> > pkttype=PACKET_HOST, addr(6)={1, 0022fbce29a0},
> > msg_iov(1)=[{"\232M\"\336X\16\0\"\373\316)\240\10\0E\0\5 at 6\301@\0\200\6\317\220\2176\250\20\270H"...,
> > 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> > cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 1358
> > write(4,
> > "E\0\5 at 6\301@\0\200\6\317\220\2176\250\20\270H\377\326\302\354\1\273c\224\34n~5\365\241"...,
> > 1344) = 1344
> > select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) = 1
> > (in [5], left {0, 999997})
> > clock_gettime(CLOCK_MONOTONIC, {170230, 13015073}) = 0
> > recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> > pkttype=PACKET_HOST, addr(6)={1, 0022fbce29a0},
> > msg_iov(1)=[{"\232M\"\336X\16\0\"\373\316)\240\10\0E\0\5\2146\302@\0\200\6\317C\2176\250\20\270H"...,
> > 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> > cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 1434
> > write(4,
> > "E\0\5\2146\302@\0\200\6\317C\2176\250\20\270H\377\326\302\354\1\273c\224!\206~5\365\241"...,
> > 1420) = 1420
> > select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) = 1
> > (in [5], left {0, 999997})
> > clock_gettime(CLOCK_MONOTONIC, {170230, 13360233}) = 0
> > recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> > pkttype=PACKET_HOST, addr(6)={1, 0022fbce29a0},
> > msg_iov(1)=[{"\232M\"\336X\16\0\"\373\316)\240\10\0E\0\5\2146\303@\0\200\6\317B\2176\250\20\270H"...,
> > 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> > cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 1434
> > write(4,
> > "E\0\5\2146\303@\0\200\6\317B\2176\250\20\270H\377\326\302\354\1\273c\224&\352~5\365\241"...,
> > 1420) = 1420
> > select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) = 1
> > (in [5], left {0, 999997})
> > clock_gettime(CLOCK_MONOTONIC, {170230, 13667642}) = 0
> > recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> > pkttype=PACKET_HOST, addr(6)={1, b407f94804a5},
> > msg_iov(1)=[{"\232M\"\336X\16\264\7\371H\4\245\10\0E\0\1P\10\26@\0\377\21\2\325\2176\251C\2176"...,
> > 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> > cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 350
> > --- SIGSEGV (Segmentation fault) @ 0 (0) ---
> > Process 20463 detached
> >
> > Em 02/03/2012 13:51, Fernando Macedo escreveu:
> >> Strace of my last segfault:
> >>
> >> select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) =
> >> 1 (in [4], left {0, 999997})
> >> clock_gettime(CLOCK_MONOTONIC, {156370, 941588185}) = 0
> >> read(4,
> >> "E\0\5\2147y\0\0005\6\361\347\10\376\24\376\2176\251\331\0P\300\230jN\25\223:\26a\316"...,
> >> 65497) = 1420
> >> sendto(5,
> >> "h\243\3041\267\320\232M\"\336X\16\10\0E\0\5\2147y\0\0005\6\361\347\10\376\24\376\2176"...,
> >> 1434, 0, {sa_family=AF_PACKET, proto=0x03, if3, pkttype=PACKET_HOST,
> >> addr(6)={0, 68a3c431b7d0}, 20) = 1434
> >> select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) =
> >> 1 (in [4], left {0, 999997})
> >> clock_gettime(CLOCK_MONOTONIC, {156370, 941877024}) = 0
> >> read(4,
> >> "E\0\5\2147z\0\0005\6\361\346\10\376\24\376\2176\251\331\0P\300\230jN\32\367:\26a\316"...,
> >> 65497) = 1420
> >> sendto(5,
> >> "h\243\3041\267\320\232M\"\336X\16\10\0E\0\5\2147z\0\0005\6\361\346\10\376\24\376\2176"...,
> >> 1434, 0, {sa_family=AF_PACKET, proto=0x03, if3, pkttype=PACKET_HOST,
> >> addr(6)={0, 68a3c431b7d0}, 20) = 1434
> >> select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) =
> >> 1 (in [4], left {0, 996370})
> >> clock_gettime(CLOCK_MONOTONIC, {156370, 945791076}) = 0
> >> read(4,
> >> "E\0\5\2147{\0\0005\6\361\345\10\376\24\376\2176\251\331\0P\300\230jN
> >> [:\26a\316"..., 65497) = 1420
> >> sendto(5,
> >> "h\243\3041\267\320\232M\"\336X\16\10\0E\0\5\2147{\0\0005\6\361\345\10\376\24\376\2176"...,
> >> 1434, 0, {sa_family=AF_PACKET, proto=0x03, if3, pkttype=PACKET_HOST,
> >> addr(6)={0, 68a3c431b7d0}, 20) = 1434
> >> select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) =
> >> 1 (in [4], left {0, 999997})
> >> clock_gettime(CLOCK_MONOTONIC, {156370, 946070102}) = 0
> >> read(4,
> >> "E\0\5\2147|\0\0005\6\361\344\10\376\24\376\2176\251\331\0P\300\230jN%\277:\26a\316"...,
> >> 65497) = 1420
> >> sendto(5,
> >> "h\243\3041\267\320\232M\"\336X\16\10\0E\0\5\2147|\0\0005\6\361\344\10\376\24\376\2176"...,
> >> 1434, 0, {sa_family=AF_PACKET, proto=0x03, if3, pkttype=PACKET_HOST,
> >> addr(6)={0, 68a3c431b7d0}, 20) = 1434
> >> select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) =
> >> 2 (in [4 5], left {0, 999997})
> >> clock_gettime(CLOCK_MONOTONIC, {156370, 946381872}) = 0
> >> read(4,
> >> "E\0\5\2147}\0\0005\6\361\343\10\376\24\376\2176\251\331\0P\300\230jN+#:\26a\316"...,
> >> 65497) = 1420
> >> sendto(5,
> >> "h\243\3041\267\320\232M\"\336X\16\10\0E\0\5\2147}\0\0005\6\361\343\10\376\24\376\2176"...,
> >> 1434, 0, {sa_family=AF_PACKET, proto=0x03, if3, pkttype=PACKET_HOST,
> >> addr(6)={0, 68a3c431b7d0}, 20) = 1434
> >> recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> >> pkttype=PACKET_HOST, addr(6)={1, 68a3c431b7d0},
> >> msg_iov(1)=[{"\232M\"\336X\16h\243\3041\267\320\10\0E\0\0(\20M@\0\200\6\223w\2176\251\331\10\376"...,
> >> 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> >> cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 60
> >> write(4,
> >> "E\0\0(\20M@\0\200\6\223w\2176\251\331\10\376\24\376\300\230\0P:\26a\316jN\n\313"...,
> >> 46) = 46
> >> select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) =
> >> 1 (in [4], left {0, 999997})
> >> clock_gettime(CLOCK_MONOTONIC, {156370, 946868309}) = 0
> >> read(4,
> >> "E\0\5\2147~\0\0005\6\361\342\10\376\24\376\2176\251\331\0P\300\230jN0\207:\26a\316"...,
> >> 65497) = 1420
> >> sendto(5,
> >> "h\243\3041\267\320\232M\"\336X\16\10\0E\0\5\2147~\0\0005\6\361\342\10\376\24\376\2176"...,
> >> 1434, 0, {sa_family=AF_PACKET, proto=0x03, if3, pkttype=PACKET_HOST,
> >> addr(6)={0, 68a3c431b7d0}, 20) = 1434
> >> select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) =
> >> 1 (in [5], left {0, 997678})
> >> clock_gettime(CLOCK_MONOTONIC, {156370, 949470668}) = 0
> >> recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> >> pkttype=PACKET_HOST, addr(6)={1, 68a3c431b7d0},
> >> msg_iov(1)=[{"\232M\"\336X\16h\243\3041\267\320\10\0E\0\0(\20N@\0\200\6\223v\2176\251\331\10\376"...,
> >> 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> >> cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 60
> >> write(4,
> >> "E\0\0(\20N@\0\200\6\223v\2176\251\331\10\376\24\376\300\230\0P:\26a\316jN\25\223"...,
> >> 46) = 46
> >> select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) =
> >> 1 (in [5], left {0, 999332})
> >> clock_gettime(CLOCK_MONOTONIC, {156370, 950476429}) = 0
> >> recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> >> pkttype=PACKET_HOST, addr(6)={1, 74f06d058e5a},
> >> msg_iov(1)=[{"\232M\"\336X\16t\360m\5\216Z\10\0E\0\0)e0@\0\200\6~\362\2176\251\22\310["...,
> >> 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> >> cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 60
> >> write(4,
> >> "E\0\0)e0@\0\200\6~\362\2176\251\22\310[\26\10\312H\0P~\314HR\377\5\r\313"...,
> >> 46) = 46
> >> select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) =
> >> 1 (in [5], left {0, 999285})
> >> clock_gettime(CLOCK_MONOTONIC, {156370, 951522574}) = 0
> >> recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> >> pkttype=PACKET_HOST, addr(6)={1, 68a3c431b7d0},
> >> msg_iov(1)=[{"\232M\"\336X\16h\243\3041\267\320\10\0E\0\0(\20O@\0\200\6\223u\2176\251\331\10\376"...,
> >> 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> >> cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 60
> >> write(4,
> >> "E\0\0(\20O@\0\200\6\223u\2176\251\331\10\376\24\376\300\230\0P:\26a\316jN
> >> ["..., 46) = 46
> >> select(13, [4 5 6 8 9 10 11 12], [], [4 5 6 8 9 10 11 12], {1, 0}) =
> >> 1 (in [5], left {0, 998912})
> >> clock_gettime(CLOCK_MONOTONIC, {156370, 952930646}) = 0
> >> recvmsg(5, {msg_name(18)={sa_family=AF_PACKET, proto=0x800, if3,
> >> pkttype=PACKET_HOST, addr(6)={1, b407f94804a5},
> >> msg_iov(1)=[{"\232M\"\336X\16\264\7\371H\4\245\10\0E\0\1P\7y@\0\377\21\3\34\2176\251\231\2176"...,
> >> 65497}], msg_controllen=40, {cmsg_len=36, cmsg_level=SOL_PACKET,
> >> cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 350
> >> --- SIGSEGV (Segmentation fault) @ 0 (0) ---
> >> Process 25821 detached
> >>
> >> Em 14/02/2012 11:36, Fernando Macedo escreveu:
> >>> Any idea? I'm using chilli_query login toauthenticate the users. It
> >>> can be related to this bug: http://www.coova.org/node/2653.
> >>>
> >>> Em 13/02/2012 17:33, Fernando Macedo escreveu:
> >>>> More crashes:
> >>>>
> >>>> Feb 13 14:16:07 x kernel: [2788498.078399] chilli[893]: segfault at
> >>>> 7ffff6a8b058 ip 00007f5621e96d64 sp 00007ffff6a78088 error 4 in
> >>>> libchilli.so.0.0.0[7f5621e6d000+4a000]
> >>>> Feb 13 16:35:56 x kernel: [2796886.973424] chilli[1317]: segfault
> >>>> at 7fffd306d02d ip 00007f5f14cb4d64 sp 00007fffd305acf8 error 4 in
> >>>> libchilli.so.0.0.0[7f5f14c8b000+4a000]
> >>>> Feb 13 16:36:08 x kernel: [2796899.050379] chilli[31138]: segfault
> >>>> at 7fffeb998019 ip 00007ff00cbdbd64 sp 00007fffeb984cd8 error 4 in
> >>>> libchilli.so.0.0.0[7ff00cbb2000+4a000]
> >>>> Feb 13 16:40:58 x kernel: [2797189.473169] chilli[31371]: segfault
> >>>> at 7fff5258202d ip 00007fd99487fd64 sp 00007fff5256f948 error 4 in
> >>>> libchilli.so.0.0.0[7fd994856000+4a000]
> >>>> Feb 13 16:41:11 x kernel: [2797201.999800] chilli[5316]: segfault
> >>>> at 7ffff7f38019 ip 00007f27c4ae2d64 sp 00007ffff7f24c08 error 4 in
> >>>> libchilli.so.0.0.0[7f27c4ab9000+4a000]
> >>>> Feb 13 16:41:27 x kernel: [2797218.491727] chilli[5601]: segfault
> >>>> at 7fff10237009 ip 00007fbc046d2d64 sp 00007fff10223818 error 4 in
> >>>> libchilli.so.0.0.0[7fbc046a9000+4a000]
> >>>>
> >>>>
> >>>> Em 13/02/2012 14:25, Fernando Macedo escreveu:
> >>>>> Hi, one month after installing chilli we experienced a segfault in
> >>>>> the last version.
> >>>>>
> >>>>> kernel: [2788498.078399] chilli[893]: segfault at 7ffff6a8b058 ip
> >>>>> 00007f5621e96d64 sp 00007ffff6a78088 error 4 in
> >>>>> libchilli.so.0.0.0[7f5621e6d000+4a000]
> >>>>>
> >>>>> Has anyone the same problem?
> >>>>>
> >>>>> Fernando Macedo
> >>>>> _______________________________________________
> >>>>> Chilli mailing list
> >>>>> Chilli at coova.org
> >>>>> http://lists.coova.org/cgi-bin/mailman/listinfo/chilli
> >>>>
> >>>> _______________________________________________
> >>>> Chilli mailing list
> >>>> Chilli at coova.org
> >>>> http://lists.coova.org/cgi-bin/mailman/listinfo/chilli
> >>>
> >>> _______________________________________________
> >>> Chilli mailing list
> >>> Chilli at coova.org
> >>> http://lists.coova.org/cgi-bin/mailman/listinfo/chilli
> >>
> >> _______________________________________________
> >> Chilli mailing list
> >> Chilli at coova.org
> >> http://lists.coova.org/cgi-bin/mailman/listinfo/chilli
> >
> > _______________________________________________
> > Chilli mailing list
> > Chilli at coova.org
> > http://lists.coova.org/cgi-bin/mailman/listinfo/chilli
>
> _______________________________________________
> Chilli mailing list
> Chilli at coova.org
> http://lists.coova.org/cgi-bin/mailman/listinfo/chilli
More information about the Chilli
mailing list