[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [leafnode-list] leafnode very slow on first XOVER



On Fri, 8 Mar 2002, Matthias Andree wrote:

> "Michael O'Quinn" <michael@xxxxxxxxxxx> writes:
> 
> > I've seen similar behavior.  In my case what is happening is that
> > fetchnews dies with the message to the console "Broken pipe" and never
> > spawns the process that updates the .overview files.  As a result the
> 
> Can anyone of you figure from his logs when or why fetchnews gets the
> "broken pipe"? It may be necessary to run fetchnews after placing
> debugmode=1 in /etc/leafnode/config to get useful logs.

I already have debugmode=1.

news.notice has this each time fetchnews starts AFTER a crash...

   Mar  7 21:23:56 shire fetchnews[23087]: erased stale pid 23086 host 
   shire.oquinn.info lockfile /var/lock/news/fetchnews.lck

..all on one line of course.  Nothing very mysterious here.

news.crit and news.err have nothing.  News.debug looks like it just dies
right while fetching an article.  I'm not sure if it's changing from one
group to another or if it's just at some random post.  I can look for that
the next time it happens.

> 
> > first attempt to access new articles in a group causes leafnode to update
> > the .overview file.  This requires opening and reading the headers of each
> > and every new message, and that can take quite a long time on very active
> > groups.
> 
> I can have leafnode 1.9.20's fetchnews catch the SIGPIPE and still run
> the XOVER updater, to work around this problem, 

That might be good.

> but I'd rather go figure
> why that happens, and maybe fix the bug that triggers it. 

Agreed.

FWIW, I have YET to see fetchnews complete a run without dieing.  
Admittedly it's sometimes after a day and half and about 5 gigs...  (And
no, it's not disk space or inodes.)

Other times, it's happens after 10 or 20 minutes.

Spewing some state information into the logs when EPIPE/SIPGPPE happens
might also help.  I'd recommend a CRIT or at least ERR level to make it
easy to find.  I'll certainly be glad to provide you the outputs.

> I might also
> do other changes like adding a switch to fetchnews, but I'm reluctant to
> do that before 1.9.20.

What kind of switch?  Or more to the point, what would the switch do?

> 
> > I've often wished for either a daemon or a program that could be
> > invoked from cron whose only purpose is to scan the entire news tree
> > on a regular basis and make sure the .overview files are up to date.
> 
> You can run texpire for now, it does more than just fixing overviews
> though.

Yes, that's the other way I've been dealing with this, but it means going
into the config file and diddling the expire times, then un-diddling
afterwords.

I don't care if it's a separate utility or just a switch on one of the
existing ones, but having the ability to force an overview update and ONLY
an overview update would be immensely useful.  An added bonus would be the
ability to force either the entire spool, or just one group.

With a news spool there is ALWAYS something that crashes this or messes up
that.  As an administrator being able to get closer to the "atomic" tools
is always a Good Thing.

> 
> > This issue may not be technically considered a "Bug", but it does
> 
> How do you know triggering EPIPE/SIGPIPE is not a "bug"? :-)

Well, you're right of course.  

I was referring to the overall sense that I have to keep doing something
untoward just to make it work, not the specific SIGPIPE exit condition.  
THAT is most certainly a bug.


Michael O'Quinn


-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list