[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [leafnode-list] speed [was leafnode is not pulling in any news]



Your answer got my thinking about how fetchnews could operate faster,
particularly since we were talking about multiple processes (in
deleted parts of the message).

If individual dialogs with the new server are slow, why not run lots
of them in parallel?  They could be split by groups or by blocks of
messages in groups.

A master process could watch what was going on to determine the
optimal degree of parallelism, which might change depending congestion
and other dynamic factors.

I think cvsup uses a strategy like this in its work.  However, it's
written in some funky language that makes such programming relatively
easy.  I'm sure actually implementing it would be quite a job.

But could this strategy work?  Or are there characteristics of nntp or
leafnode's data structures that would make parallel fetching of news
problematic?


On Fri, Jul 04, 2003 at 01:31:38AM +0200, Matthias Andree wrote:
> Ross Boylan <RossBoylan@xxxxxxxxxxxxxxxxxx> writes:
[snip]
> 
> >> The problem is that leafnode-1 sends commands and expects responses in a
> >> lock-step way, which means that your connection is idle for a full
> >> round-trip time without filtering, or for two with filtering that cannot
> >> happen at the XOVER level.
> >
> > What's filtering?
> 
> Defining a filter file for the main part (or using maxage, maxbytes,
> .. options, see below).
> 
> > And what does "or for two with filtering that cannot happen at the
> > XOVER level"?  Does that mean it takes two round trips if filtering
> > can't be done with XOVER (whatever that is)?
> 
> Leafnode queries overview data, usually comprising from:, subject:,
> date:, size in bytes and in lines, message-id:, references:, before
> downloading any articles from a group. Leafnode has options to filter
> after some of these headers, say minlines, maxlines, maxbytes and all
> that. Filters that apply to these headers are "cheap", filters that
> apply to other headers (Summary) are "expensive".
> 
> > Also, another thought struck me: does nntp set speed early on and not
> > adjust later?
> 
> NNTP doesn't have a notion of speed. It's just as fast as the underlying
> transport, TCP across your modem or DSL or leased line will do. However,
> NNTP is also a request-response scheme. The client (fetchnews) sends a
> command, the server waits for the command, and sends a response. The
> client (at least as of leafnode-1) doesn't send the next command before
> it's received the previous response. Given that it takes a while for the
> command to travel to the server (I recall around 40 ms for ISDN, 100 ms
> for DSL, less than 1 ms for LAN, satellite links are likely to be much
> "slower" than even DSL, for the full round trip, forth AND back
> together), the line will be idle while the command is in transit. This
> affects the average bandwidth use adversely.
> 
> leafnode-2's fetchnews sends some "ARTICLE" commands in a batch in order
> to maximize bandwidth utilization, personally, I've been using
> windowsize=90 in the configuration for a while now without
> difficulties. However, this is not guaranteed to work yet - but
> practically, it does, I haven't received any reports to the contrary.
> 
> OTOH, leafnode-2 is in alpha state, i. e. it may break compatibility
> with previous leafnode-2 versions without advance warning, and
> leafnode-2 and leafnode-1 spools are sort-of incompatible, IOW: there's
> no way back without losing your spool currently.
> 
> > When fetchnews is first running it typically is competing with lots of
> > other activities for bandwidth, so it may have relatively little.  If
> > it somehow locks into that rate, that might account for its slowness.
> 
> fetchnews should only really suffer if at least one direction of your
> network connection is overloaded, other than that, it'll fit in between.
> 
> One should also know that TCP itself doesn't support a notion of
> "fairness", continuous data streams tend to dominate over interactive
> connections.
> 
> To alleviate that, traffic shapers and sophisticated schedulers can be
> used, see the Linux Advanced Routing and Traffic Control HOWTO -- but it
> is advanced material, as it claims. (Linux isn't the only system to have
> traffic shapers though, but it's the one I know best in this area.)
> 

-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list