Re: [leafnode-list] leafnode is not pulling in any news

[Matthias Andree <ma@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>]

Ross Boylan <RossBoylan@xxxxxxxxxxxxxxxxxx> writes:

> I had trouble parsing parts of your reply, and have some more comments
> and questions.
>
> On Thu, Jul 03, 2003 at 11:25:45AM +0200, Matthias Andree wrote:
>> Ross Boylan schrieb am 2003-07-02:
>> 
>> > > Then that's the one you're looking for. If you "cat" it, you should be
>> > > given the PID and the host name of the process holding the lock.
>> > 
>> > Now I'm confused.  leafnode is running all the time.  Are you saying
>> > leafnode and fetchnews can't run at the same time?
>> 
>> Well, fetchnews and texpire (and some of the minor utilities as
>> applyfilter) are mutually exclusive, leafnode itself (the nntpd process)
>> and run at the same time as fetchnews and texpire.

> Is that "fetchnews and texpire ... are mutually exclusive, but
> leafnode can run at the same time as them.."?

Yes, it is. You can run at most one of { fetchnews, texpire, applyfilter
} at the same time, and additionally any number of leafnode processes at
any (including the same) time.

> I assume two leafnode's can't or shouldn't run at the same time, but
> leafnode doesn't seem to put its process id in

You can safely run 40 leafnodes at the same time if your computer has
sufficient RAM or swap space.

> /var/lock/news/leafnode.  So is it true that /var/lock/news/leafnode
> is just for the use of the auxiliary processes (fetchnews, texpire,
> .), and only one of them can run at once?

Correct. The name of the lockfile isn't too indicative here.

>> The problem is that leafnode-1 sends commands and expects responses in a
>> lock-step way, which means that your connection is idle for a full
>> round-trip time without filtering, or for two with filtering that cannot
>> happen at the XOVER level.
>
> What's filtering?

Defining a filter file for the main part (or using maxage, maxbytes,
.. options, see below).

> And what does "or for two with filtering that cannot happen at the
> XOVER level"?  Does that mean it takes two round trips if filtering
> can't be done with XOVER (whatever that is)?

Leafnode queries overview data, usually comprising from:, subject:,
date:, size in bytes and in lines, message-id:, references:, before
downloading any articles from a group. Leafnode has options to filter
after some of these headers, say minlines, maxlines, maxbytes and all
that. Filters that apply to these headers are "cheap", filters that
apply to other headers (Summary) are "expensive".

> Also, another thought struck me: does nntp set speed early on and not
> adjust later?

NNTP doesn't have a notion of speed. It's just as fast as the underlying
transport, TCP across your modem or DSL or leased line will do. However,
NNTP is also a request-response scheme. The client (fetchnews) sends a
command, the server waits for the command, and sends a response. The
client (at least as of leafnode-1) doesn't send the next command before
it's received the previous response. Given that it takes a while for the
command to travel to the server (I recall around 40 ms for ISDN, 100 ms
for DSL, less than 1 ms for LAN, satellite links are likely to be much
"slower" than even DSL, for the full round trip, forth AND back
together), the line will be idle while the command is in transit. This
affects the average bandwidth use adversely.

leafnode-2's fetchnews sends some "ARTICLE" commands in a batch in order
to maximize bandwidth utilization, personally, I've been using
windowsize=90 in the configuration for a while now without
difficulties. However, this is not guaranteed to work yet - but
practically, it does, I haven't received any reports to the contrary.

OTOH, leafnode-2 is in alpha state, i. e. it may break compatibility
with previous leafnode-2 versions without advance warning, and
leafnode-2 and leafnode-1 spools are sort-of incompatible, IOW: there's
no way back without losing your spool currently.

> When fetchnews is first running it typically is competing with lots of
> other activities for bandwidth, so it may have relatively little.  If
> it somehow locks into that rate, that might account for its slowness.

fetchnews should only really suffer if at least one direction of your
network connection is overloaded, other than that, it'll fit in between.

One should also know that TCP itself doesn't support a notion of
"fairness", continuous data streams tend to dominate over interactive
connections.

To alleviate that, traffic shapers and sophisticated schedulers can be
used, see the Linux Advanced Routing and Traffic Control HOWTO -- but it
is advanced material, as it claims. (Linux isn't the only system to have
traffic shapers though, but it's the one I know best in this area.)

> In view of the other comments about this behavior, it seems odd that I
> had a fetchnews process that seemed to last for hours after the
> connection was down.

I'd be interested to get a stack backtrace (see the FAQ for
instructions) in case that happens again (you may need to recompile and
reinstall, using "env CFLAGS=-g ./configure" to configure it, to obtain
useful backtraces). Please state the exact version along with the
backtrace, because I need to match line numbers.

> There is some indication this problem started after I upgraded to
> 1.9.41 on Debian, but it was only several days later that it
> occurred.  That fact, and the remark that fetchnews should figure out
> when the connection drops, suggest that some somewhat unusual
> circumstances were necessary to get the problem started.  I'm not sure
> what, though; it seemed pretty repeatable when I tested it.  However,
> I typically waited much less than 10 minutes after bringing the
> connectionn down to test the offline behavior or dial back in.

I seem to recall something about TCP taking nearly 800 s to time out.

> There were some references to this in the documentation, but the
> meaning of "accidentally touched" was not clear to me until this
> explanation.  It might be nice to say a little more in the
> documentation on this concept.
>
> Having two times is certainly useful; it just seems more conventional
> for ctime (creation time, right?) to be fixed at creation, and mtime
> (modification time) to be the one that is changed later.

Well, ctime is often mistaken for creation time, but it's really "change
time", where change is a change of inode attributes such as owner, mode,
(hard) link count and all that. On Linux, man 2 stat tells you something
about these times.

mtime is the modification time, this one applies to the file contents
(i. e. write to a file, truncate it).

There is no notion of "mtime should be no older than ctime" attached (in
fact, you can use the touch command to set mtime to be older than ctime,
because using touch will update the ctime to the current time); the
relations I mentioned are based on leafnode's native behaviour of
marking the interesting.groups/* files, and this behaviour has revealed
more than one kernel bug or invalid short-cut in the past (reiserfs for
example didn't implement ctime updates properly some time ago).

-- 
Matthias Andree

-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list