Re: [leafnode-list] fetchnews in parallel?

[Jeff Wasilko <jw-leafnode@xxxxxxxx>]

On Tue, Jul 09, 2002 at 12:24:39AM +0200, Matthias Andree wrote:
> Witold Wladyslaw Wojciech Wilk <maniack@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> writes:
> 
> > and I was wondering - will fetchnews someday will be able to go into
> > parallel with this? almost al of them carry all the groups, so the fetch
> > could go faster, way faster when going parallel instead of serial.
> >
> > is it planned, is it made in leafnode 2.x, or will it never be? :)
> 
> That would indeed be nice to have, but currently, servers must still be
> queried sequentially, even in leafnode 2.x. There are two and a half
> places in the leafnode spool that require this:
> 
> 1) the groupinfo file updates. parallel fetch would currently corrupt
>    data or require very inefficient locking.
> 
> 2) the .overview file updates. parallel fetch would not currently update
>    the .overview file properly.

One possibility for a 'quick' fix to allow multiple writers is to
use a database library that provides the locking and support for
a coherent view of the database.

I've had good luck with BerkeleyDB's Concurrent Data Store in a
similar environment. You can read about it at:

http://www.sleepycat.com/docs/ref/cam/intro.html


Plus, using an indexed database might improve performance...

-j

-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list