Re: [leafnode-list] Fetchnews: Falling back and regrouping.

[Matthias Andree <ma@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>]

Michael O'Quinn schrieb am Montag, den 25. März 2002:

> I am of course referring to the /var/spool/news/leaf.node/SERVERNAME file.
> 
> I can trace most of my heartache and headache with leafnode since I first
> started using it to this single file.
> 
> To wit:
> 
> (1) When fetchnews for some unknown reason terminates before the end of
> it's run, the SERVERNAME file is not updated.

True.

> (2) When fetchnews manages to complete it's run, but for some reason skips
> 10,000 or so articles in one group (that I've confirmed ARE there), the
> SERVERNAME file is not updated.

I have yet to see why, because on my machine, this does not show.

> (3) When I terminate fetchnews early via <CTRL-C>, the SERVERNAME file is
> not updated.

Not easily fixed. In this case, however, the SERVERNAME~ file should be
there and contain some useful information, can you verify that on your
machine?

> (4) When a group has been fetched in the past, but is not being fetched in 
> this run, that group is DELETED ENTIRELY from the SERVERNAME file.

True. The reasoning behind this is: presume you don't read that group
for another four weeks. Say, in these four weeks, 28,000 new articles
arrive. If we kept this state information, we would fetch all 28,000 new
articles (or however many of them remain on the server) on the next run,
because no initiallimit would apply. artlimit is just a protection
against runaway fetchmail.

> (5) No history file ala INN is produced.  Producing a history file
> containing the message-id's of all retrieved messages would help a lot
> with the broken SERVERNAME file, but this isn't done.

True. However, history files reek of file locking, and avoiding file
locking as far as possible is not a bad thing. Admittedly, if nntpd was
to NOT use this file, it would not matter. However, I'd not use a plain
text file for this, but something like gdbm or Berkeley db
(www.sleepycat.com), and I'm not introducing anything like this into 1.9
any more. Effectively, 1.9.20.rel is ready and will be released soon,
without any further changes.

> (6) Since the SERVERNAME file is not getting updated and no history file
> is produced, the only state information about a group is contained within
> the articles themselves.  Once the articles are expired, all state
> information from the upstream server about that group is lost.

That's true. It'd be useful to set maxage from the expire time
automatically, and I can envision that feature for 1.9.21.

> (7) When the news spool fills up, the SERVERNAME file falls off the
> data-bus.  [This already been fixed with a patch against 1.9.20.rc9.  I
> haven't applied this patch, nor have I had to opportunity to test it, so
> I'm not sure how it currently acts when the spool fills up.]

The patch makes sure that the old SERVERNAME file remains in place.

I have thought if I should read both the (older) SERVERNAME and the
complete lines from the (newer) SERVERNAME~ file and take whichever has
the higher number. This will not be implemented in 1.9.20, it may be for
1.9.21, I have yet to decide this.

> I have verified (1), (2) and (3) under 1.9.20.rc9.  I haven't seen a
> complete enough run under 1.9.20.rc9 to cause (4) to kick in, so I can't
> say if it still does it.  I think you said it wasn't going to be fixed any
> time soon.

Ad 4: If at all, for reasons given above.

> I don't know.  Maybe I'm trying to make leafnode do more than I should.  
> But I'm only downloading a few groups -- a couple dozen at the most, and
> often only one or two.  Some of them are just very active groups with a
> lot of articles, so the download can run all night.
> 
> Is it reasonable for a program to run for hours and hours, and NOT
> periodically save it's state, so that it can gracefully pick up where it
> left off should the need arise?

Certainly, checkpointing the state would be helpful, but then again,
leafnode assumes that you do NOT expire faster than you download. As
written above, I can imagine deriving a per-group maxage setting from
the (per-group) expire time.

> I am more than willing to help hammer this out and then help debug the
> changes.  But, I need to see the the problem of (extremely) sloppy state
> saving being dealt with seriously to be willing to do that.

Ok, here's the plan.

1.9.20 will try hard to keep at least the old state. 1.9.19 can, on
write errors ("disk full" is one such condition) kill the SERVERINFO
file without notice. 1.9.20 will at least leave the old SERVERINFO where
it is. On a spontaneous abort that is not handled, like SIGSEGV, the new
SERVERINFO~ will be shattered, probably empty, because there is no
chance to flush the stdio buffers. However, since that file is not
written too often, I'll make it unbuffered or line buffered, and
together with the "SERVERNAME~" merge-in, this will save most of the
state that is important to you. The patch against 1.9.20 is mostly
trivial and will appear alongside the 1.9.20 release.

-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list