Re: [leafnode-list] speed

[Matthias Andree <ma@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>]

Ross Boylan <RossBoylan@xxxxxxxxxxxxxxxxxx> writes:

> Your answer got my thinking about how fetchnews could operate faster,
> particularly since we were talking about multiple processes (in
> deleted parts of the message).
>
> If individual dialogs with the new server are slow, why not run lots
> of them in parallel?  They could be split by groups or by blocks of
> messages in groups.

To thrash the server's drives or risk being thrown out for "abuse"?
Leafnode's structure doesn't yield to simultaneous connections to
several servers. I believe this would be leafnode-2.1 or 3.0 stuff...

leafnode-2 pipelines the ARTICLE commands, it's _much_ faster than
leafnode-1.

> A master process could watch what was going on to determine the
> optimal degree of parallelism, which might change depending congestion
> and other dynamic factors.

..or I might investigate if it's possible to enable IHAVE for
configurable hosts so you could use newsstar to feed leafnode. I don't
know if that's possible, just a brain wave.

> I think cvsup uses a strategy like this in its work.  However, it's
> written in some funky language that makes such programming relatively
> easy.  I'm sure actually implementing it would be quite a job.

I don't mind mixing other languages. Many people use leafnode in the
version that comes with their distro.

> But could this strategy work?  Or are there characteristics of nntp or
> leafnode's data structures that would make parallel fetching of news
> problematic?

leafnode stands in its own way here. Too much ad-hoc code, too many
global variables, and a global semaphore (lockfile actually).

I think parallel fetching won't happen without a complete fetchnews
rewrite, and the latter is something that is unlikely to happen in
C. Some parts of it perhaps, but not the whole thing. But that rewrite
is also unlikely to happen for 2.0. leafnode-2, although "alpha", is
working, and the "local groups" functionality reason enough to polish
and release it. Once I can afford to take the time...

-- 
Matthias Andree

-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list