[Chilli] 100% cpu problem

Thu Oct 7 09:19:10 UTC 2010

In getnextattr offset should always increment, and both offset and len
are unsigned. Though, to be safe (to protect against a bad radius
packet), we could change:

-    if (t->t == 0)
+    if (t->t == 0 || t->l < 2)
       return -1;

On Thu, 2010-10-07 at 10:54 +0200, Marco Simioni wrote:
> Thank you alberto for your answer.
> 
> I tried to debug another time with gdb, and i saw it was into the
> "rad_getnextattr" again.
> 
> I'm now trying to look into the "rad_getnextattr()" function.
> 
> I see there is a while loop, that could turn into potential endless
> loop: "while (offset < len)".
> 
> I inserted some debug messages, let's see the next time it happens
> 
> Then, i'll try your suggestions.
> 
> Thank you.
> 
> Regards,
> 
> Marco
> 
> 2010/10/6 Alberto Bellettato <albesvs at yahoo.it>:
> > Have you tried to temporarily remove the "/etc/rc2.d/S20chilli radconfig"
> > script by your cron table? It could help isolating the problem.
> > Then have you enabled ssl or redir in your chilli config?
> >
> > ----- Original Message ----- From: "Marco Simioni" <m.simioni at gmail.com>
> > To: <chilli at coova.org>
> > Sent: Wednesday, October 06, 2010 11:26 AM
> > Subject: Re: [Chilli] 100% cpu problem
> >
> >
> > No news.
> >
> > Stil happened.
> >
> > I'm using latest SVN version.
> >
> > How can i investigate the problem ?
> >
> > 2010/9/23 Marco Simioni <m.simioni at gmail.com>:
> >>
> >> It happened again: i had 100% cpu after days of uptime.
> >>
> >> This time, i was able to identify the following:
> >>
> >> 1) Thanks to "sar" utility, i was able to localize the CPU usage
> >> between 09:45 and 10:05.
> >>
> >> 09:05:02 CPU %user %nice %system %iowait %steal %idle
> >> 09:45:02 all 0,00 0,00 0,04 4,26 0,00 95,70
> >> 09:55:02 all 2,51 0,00 0,08 3,47 0,00 93,94
> >> 10:05:02 all 98,32 0,00 1,68 0,00 0,00 0,00
> >>
> >> 2) In syslog, the messages i see are:
> >>
> >> Sep 23 09:51:10 izc coova-chilli[939]: chilli.c: 3402: DHCP addr
> >> released by MAC=90-84-0D-D2-00-2D IP=0.0.0.0
> >> Sep 23 09:51:11 izc coova-chilli[939]: chilli.c: 3248: New DHCP
> >> request from MAC=90-84-0D-D2-00-2D
> >> Sep 23 09:52:02 izc coova-chilli[939]: chilli.c: 3248: New DHCP
> >> request from MAC=00-0E-6A-7A-AB-9C
> >> Sep 23 09:52:02 izc coova-chilli[939]: chilli.c: 3209: Client
> >> MAC=00-0E-6A-7A-AB-9C assigned IP 10.1.0.73
> >> Sep 23 09:55:02 izc CRON[9452]: (root) CMD (command -v debian-sa1 >
> >> /dev/null && debian-sa1 1 1)
> >> Sep 23 10:00:03 izc CRON[9455]: (root) CMD (/etc/rc2.d/S20chilli
> >> radconfig)
> >> Sep 23 10:05:01 izc CRON[9461]: (root) CMD (command -v debian-sa1 >
> >> /dev/null && debian-sa1 1 1)
> >>
> >> 3) I tried to attach with "gdb" to the chilli process. Nothing
> >> happened. I didn't know what to do, so tried to make a step and i got
> >> the following:
> >>
> >> (gdb) step
> >> Single stepping until exit from function radius_getnextattr, wich has
> >> no line number information.
> >>
> >> than nothing else.
> >>
> >> 4) Tried to attach with "strace"
> >>
> >> root at izc:# strace -p 939
> >> Process 939 attached - interrupt to quit
> >>
> >> and nothing happened.
> >>
> >> Now i had to reboot to let customers surf.
> >>
> >> What can i do next time?
> >>
> >> The syslog "radconf" and the gdb message "radius_getnextattr" could
> >> point to something ?
> >>
> >> Keep in mind that the radius server is a proprietary one, it is not
> >> freeradius or something else.
> >>
> >> Is there something i can to the next time with gdb and/or strace ?
> >>
> >> Thank you again.
> >>
> >> 2010/7/27 David Bird <david at coova.com>:
> >>>
> >>> Wichert asked a good question; are you using SSL features of
> >>> CoovaChilli?
> >>>
> >>> For info on gdb, you can google for it, of course. Here is a quick howto
> >>> page:
> >>> http://www.freebsd.org/doc/en/books/developers-handbook/debugging.html
> >>>
> >>> For strace, use "strace -p <pid>" and you will see the system calls
> >>> being executed - if it is using 100%, there must be a runaway loop
> >>> occurring.
> >>>
> >>> David
> >>>
> >>> On Tue, 2010-07-27 at 09:20 +0200, Marco Simioni wrote:
> >>>>
> >>>> It happened 3 times in a month,
> >>>>
> >>>> not immediately but after some day of regular work.
> >>>>
> >>>> "top" command sayd chilli was consuming 100% CPU.
> >>>>
> >>>> customers reported that they could not get dhcp and then login page.
> >>>>
> >>>> i will try to use chilli_query when will happen again.
> >>>>
> >>>> how can i attach with gdb or strace? can you point me some
> >>>> documentation?
> >>>>
> >>>> i can run it in debug mode only in console mode, with -fd, or also as
> >>>> a service ?
> >>>>
> >>>> thank you i.a.
> >>>>
> >>>> regards,
> >>>>
> >>>> Marco
> >>>>
> >>>> 2010/7/27 David Bird <david at coova.com>:
> >>>> > How quickly does this start to happen? Immediately? After how long?
> >>>> >
> >>>> > Is chilli also not working during this time? Does chilli_query hang?
> >>>> >
> >>>> > Are you able to attach gdb or use strace to get more info?
> >>>> >
> >>>> > If able, you can try running in debug mode for additional log
> >>>> > information?
> >>>> >
> >>>> > Thanks,
> >>>> > David
> >>>> >
> >>>> >
> >>>> > On Tue, 2010-07-27 at 08:55 +0200, Marco Simioni wrote:
> >>>> >> Hi all, my customer is reporting a cpu problem.
> >>>> >>
> >>>> >> Chilli goes to consume all the processor, going to 100%.
> >>>> >>
> >>>> >> It is a brand new setup, bult on a VMWare ESXi Virtual Machine on HP
> >>>> >> >> ML115,
> >>>> >> 1.8GHz CPU allocated,
> >>>> >> 512MB RAM allocated,
> >>>> >> coova-chilli 1.2.2,
> >>>> >> Ubuntu 9.10 ( 2.6.31-14-server ).
> >>>> >>
> >>>> >> It's about three times it happens, solved it with a reboot.
> >>>> >>
> >>>> >> Very few clients, < 10.
> >>>> >>
> >>>> >> Suggestions ?
> >>>> >>
> >>>> >> How can i investigate and understand when it happens ?
> >>>> >>
> >>>> >> Thank i.a.
> >>>> >>
> >>>> >> Best regards,
> >>>> >>
> >>>> >> Marco
> >>>> >> _______________________________________________
> >>>> >> Chilli mailing list
> >>>> >> Chilli at coova.org
> >>>> >> http://lists.coova.org/cgi-bin/mailman/listinfo/chilli
> >>>> >
> >>>> >
> >>>> >
> >>>
> >>>
> >>>
> >>
> > _______________________________________________
> > Chilli mailing list
> > Chilli at coova.org
> > http://lists.coova.org/cgi-bin/mailman/listinfo/chilli
> > _______________________________________________
> > Chilli mailing list
> > Chilli at coova.org
> > http://lists.coova.org/cgi-bin/mailman/listinfo/chilli
> >
> _______________________________________________
> Chilli mailing list
> Chilli at coova.org
> http://lists.coova.org/cgi-bin/mailman/listinfo/chilli