Re: [leafnode-list] fetchnews in parallel?

[Witold Wladyslaw Wojciech Wilk <maniack@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>]

Dnia Wed, 10 Jul 2002 11:21:09 +0200 niejaki(a) Matthias Andree
<matthias.andree@xxxxxx> napisal(a):

> Yes, but you can't boost the performance with a larger buffer. You will
> have to wait for the directory updates (synchronous on most systems) in
> any case.

I'm not talking about the disk i/o - I'm talking about the network i/o. 

> > second of all - not everybody makes a small lan/home server based on a
> > 4 raided 15krpm cheetahs :)
> 
> Neither do I. It's my home "workstation", I was annoyed of my previous
> 5400/min ATA drive.

It depends. I prefer a computer, which is rigged for ultra-quiet. Right
now the most noisy element of my computer is the monitor... I don't have
any fans. I can hear the disk only when its in seek (Seagate U10, lying on
a special pad, that takes the vibrations).

> > > When you pipeline writes, how do you propagate errors back to know
> > > where to pick up?
> >=20
> > hmmmm....again I'm going into theory mode ;) lets make some steps:
> >=20
> > 1) fetchnews connects to every news.server.com
> > 2) fetchnews downloads all the article headers for every groups that
> > is marked as interesting
> 
> It won't. It will look at overview data only.

yeah, right, my mistake.

> > 3) for every server it disconnects after finishing that task
> 
> No way will there be unnecessary disconnects, that would be a
> regression. They cost 4 round trips for TCP breakdown, 3 round trips for
> TCP establishment.

mmm... then maybe maintain connections until we get all the concurrent
overview downloads? or even start the selection and downloading for the
groups that have been downloaded already (the server thats last should
have those ready). but implementing such would be a nightmare perhaps? I
get the shivvers, so I'm almost sure ;)

> > 6) fetchnews starts downloading, buffering it in 4 pipes with
> > simultanous disk access (the number of pipes could be set via vconfig,
> > just like the size of every pipe)
> 
> Increasing the concurrency on the disk will backfire, especially with
> slowly-seeking disk.

this could be configurable. some may have a good scsi disk, or even a
small raid. for them, the option could speed up things.

> Thrashing contention is considerable.

I don't get the meaning here :( You mean data corruption? or loss of
articles?

> > 8b) we lose the power. system reboots. fetchnews starts and sees that
> > the file with the files IS in the direcotory. then check which
> > articles are present and downloaded. when it sees that in a place the
> > download ends, than it discards th last downloaded downloading it
> > again, just in case, and then download the rest. go back to 8a. or 8b
> > ;)
> 
> Ok, your approach is to keep a TODO list. Effectively, leafnode already
> keeps a pointer of the last article read, which is just as good, but
> consumes less space.

yes, but the last article read != not necessarily the last article
downloaded. we would end having missing some downloads.

> Only passing back write errors through a pipe is a
> problem.

hmmm... on this one I'm green so its better for me not to say anythig,
less I make a fool of myself :)

-- 
|_Witold_Wilk____<maniack@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>__|
|_____________________________________________(+48(0)322270471)_|
|GIT d- s+:- a--- C++ UL++++ P+ L+++ E- W N++ o? K? w-- !O !M !V|
|_PS+ PE+++ Y+ PGP !t !5 !X R+ !tv b++++ !DI D+++ G e- h! r- y++|


-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list