Re: [leafnode-list] fetchnews in parallel?

[Witold Wladyslaw Wojciech Wilk <maniack@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>]

Dnia Tue, 09 Jul 2002 00:24:39 +0200 niejaki(a) Matthias Andree
<ma@xxxxxxxxxxxxxxxxxxxxxxxxxxxx> napisal(a):

> 1) the groupinfo file updates. parallel fetch would currently corrupt
>    data or require very inefficient locking.
> 
> 2) the .overview file updates. parallel fetch would not currently update
>    the .overview file properly.
> 
> 2.5) the store into the newsgroup itself. It should cope with races, but
>      inefficiently.
> 
> I'm not sure if multi-threading could help us here or bring us there
> faster than multi-process can (fork()) -- only that I'm not really
> acquainted with threads.

the remedium would be changing groupinfo and .overview formats into
something that would allow it, but the only problem would be
incompatibility...

.. no, bad thinking, downgrading capability. if upgrade we'd from lets
say 2.0.0.rel -> 3.0.0.rel (whereas 2.0.0.rel ain't parallel, and
3.0.0.rel is) then we'd still be able to use the the old system ->
backward compatiblity.

The problem propably would be those file formats - more memory, isn't it?

The spooler would cope, it depends on the filesystem & disk & memory
(buffers) - but I think that is the least of the problems... if anything
the disk would go into a seek loop for some time... but data would go
through (high latency, transfer acceptable - depends on the number of
servers). but looking from the other side - it would take shorter for the
same amount of data). 

(just popped into my mind)
also - prebuffering in leafnode - if data comes in faster than we can
write it - leafnode does prebuffering inside of itself? if it does the
buffer must be 8kB... so if it would be set higher or there could be a
config option - this one could help. Right now I'm having a maxtor d540x
in the server, and the time fetchnews takes to write articles in SERIAL is
twice the time that would take to get the articles (I've got a
squid/samba/ftp on a 40gig 5400rpm - and squid is making the pain).

the only problem that I see right now is "what happens if they cut the
power...". 

> Therefore, I cannot promise leafnode 2.0.0.rel will have this feature as
> it is released (talk about the future...), because I cannot defer
> leafnode 2.0.0.rel ad infinitum. Sorry.

then maybe 3.0.0.rel? :) though its propably a major code reworking, isn't
it? I'm not a programmer (unless you count simple hello worlds/etc ;) ) so
I'm guessing. but maybe someday? :)

-- 
|_Witold_Wilk____<maniack@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>__|
|_____________________________________________(+48(0)322270471)_|
|GIT d- s+:- a--- C++ UL++++ P+ L+++ E- W N++ o? K? w-- !O !M !V|
|_PS+ PE+++ Y+ PGP !t !5 !X R+ !tv b++++ !DI D+++ G e- h! r- y++|


-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list