Re: Huge numbers of slow, spurious discards ...

[Lloyd Zusman <ljz@xxxxxxxxxx>]

krasel@xxxxxxxxxxxxxxxxxxxxxxxxxxxx (Cornelius Krasel) writes:

> Greg Weeks wrote:
> 
> > Lloyd Zusman wrote:
> > 
> > > I'm using leafnode-1.9.2, and now I'm getting huge quantities of slow,
> > > spurious discards when `fetch' is running.
> 
> [log snipped]

[ previously snipped explanation restored so that I can comment ... ]

> > I think I can explain what happened. When you start accessing a new
> > group fetch tries to download the entire group from your upstream
> > server. There is a default that it won't download anything over 10
> > days old. It has to fetch the header for every message to find out how
> > old it is though. At least this looks like what's happening. Once it
> > goes through this for a given group it shouldn't need to do it
> > again. If you add a new server it goes through this for every group on
> > the new server.  I'm not saying this is definitely what's going on. I
> > haven't dug through the code enough. I've seen behavior like this
> > though. The article numbers are for the upstream server it's fetching
> > from by the way.

But none of these are new newsgroups.  I have been running fetch
against the groups in question several times a day, every day since
maybe 5 days ago, from all the upstream servers that I'm using, and I
have been getting news from all of the newsgroups regularly.

Furthermore, even if something got out of kilter which caused leafnode
to once again think that these are new newsgroups. I have the
`initialfetch' variable set to 1 in my configuration file (a copy of
which was enclosed in my earlier message).  Shouldn't this
`initialfetch' setting prevent the entire contents of the upstream
newsgroups (which could be thousands of articles) to be accessed in
this extremely slow manner?

> > I'm not saying this is definitely what's going on.
> 
> You are entirely correct. To avoid leafnode checking the age for each
> message, use the "initialfetch" parameter in the config file. Unfortunately,
> this remedies the problem only partially, since a group will not be
> recognized as new if you have fetched it already from another server.
> You might consider setting "maxfetch" to an appropriate value only for
> the first run, but this is not entirely satisfying either :-(

Aha!  Well, then, should the operation of `initialfetch' be changed to
handle this case concerning multiple upstream servers?  Or should there
perhaps be another variable which might control this behavior?

Either way, I'm happy to patch the source code to add this
functionality, but I'd like to discuss this here and establish a
consensus as to what the proper behavior should be before jumping in
and coding a solution.

-- 
 Lloyd Zusman
 ljz@xxxxxxxxxx

-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list