[leafnode-list] Fetchnews: Falling back and regrouping.

["Michael O'Quinn" <michael@xxxxxxxxxxx>]

Matthew,

I have run 1.9.20.rc9 fetchnews various times with differing levels of
success, and produced some log outputs that may or may not be useful.

But I'm not going to pursue the issue of fetchnews terminating early or
skipping groups any further.  The reason for this is that there is a MUCH
more serious issue that I don't believe is being addressed seriously.  
Continuing to chase after the the sudden termination and skipped
retrievals is like trying to fix a broken leg with a band-aid.

The issue of how the program reacts and copes when it something DOES go
wrong is much more important than preventing it from crashing in the first
place.  Programs will always crash.  Sometimes they will even be crashed
(stopped before they are done) intentionally, as when the system
administrator needs stop it for some reason.

Think seat-belts, air-bags, and crumple zones.  They add nothing the
comfort or utility of a vehicle, you hope to God you never have to use
them for their intended purpose, but you're awfully glad to have them
when the time comes.

When fetchnews fails to complete its run in the orderly fashion it
prefers, it totally and completely fails to update it's state information.  
In addition, sometimes it INTENTIONALLY deletes state information from
previous runs that HAVE NOTHING TO DO WITH THE CURRENT RUN.

I am of course referring to the /var/spool/news/leaf.node/SERVERNAME file.

I can trace most of my heartache and headache with leafnode since I first
started using it to this single file.

To wit:

(1) When fetchnews for some unknown reason terminates before the end of
it's run, the SERVERNAME file is not updated.

(2) When fetchnews manages to complete it's run, but for some reason skips
10,000 or so articles in one group (that I've confirmed ARE there), the
SERVERNAME file is not updated.

(3) When I terminate fetchnews early via <CTRL-C>, the SERVERNAME file is
not updated.

(4) When a group has been fetched in the past, but is not being fetched in 
this run, that group is DELETED ENTIRELY from the SERVERNAME file.

(5) No history file ala INN is produced.  Producing a history file
containing the message-id's of all retrieved messages would help a lot
with the broken SERVERNAME file, but this isn't done.

(6) Since the SERVERNAME file is not getting updated and no history file
is produced, the only state information about a group is contained within
the articles themselves.  Once the articles are expired, all state
information from the upstream server about that group is lost.

(7) When the news spool fills up, the SERVERNAME file falls off the
data-bus.  [This already been fixed with a patch against 1.9.20.rc9.  I
haven't applied this patch, nor have I had to opportunity to test it, so
I'm not sure how it currently acts when the spool fills up.]

I have verified (1), (2) and (3) under 1.9.20.rc9.  I haven't seen a
complete enough run under 1.9.20.rc9 to cause (4) to kick in, so I can't
say if it still does it.  I think you said it wasn't going to be fixed any
time soon.

I don't know.  Maybe I'm trying to make leafnode do more than I should.  
But I'm only downloading a few groups -- a couple dozen at the most, and
often only one or two.  Some of them are just very active groups with a
lot of articles, so the download can run all night.

Is it reasonable for a program to run for hours and hours, and NOT
periodically save it's state, so that it can gracefully pick up where it
left off should the need arise?

I know that I have carefully monitored my SERVERNAME file since I became
aware of this problem (about 6 weeks ago) and I have only TWICE seen it
updated correctly and entirely.

Only TWICE in 6 weeks, with several runs a day.

I am more than willing to help hammer this out and then help debug the
changes.  But, I need to see the the problem of (extremely) sloppy state
saving being dealt with seriously to be willing to do that.

Regards,
Michael O'Quinn


-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list