[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [leafnode-list] fetchnews in parallel?

* Witold Wladyslaw Wojciech Wilk wrote on Mon, Jul 08, 2002 at 08:47:05PM CEST:
> I got stuck with downloading many articles (7 days worth of 20 groups, 3
> of them > 200/day, 9 > 100/day). but I have the servers in the config like
> this:
> and I was wondering - will fetchnews someday will be able to go into
> parallel with this? almost al of them carry all the groups, so the fetch
> could go faster, way faster when going parallel instead of serial.
> is it planned, is it made in leafnode 2.x, or will it never be? :)

Well, since that is not implemented, if you still have not started to
catch up on those servers, here's a possible suggestion to reduce the
load:  Wait for the next leafnode-2 alpha.  Then you have a few
possibilities (some apply now already):

- If you do not use delaybody (which so far did not work anyway) _and_
  you do not use header filtering _and_ you know your servers can answer
  XHDR pretty quickly:  set 'usexhdr = 1' for all but the first server.
  This will at least lower the necessary traffic.
- If the first does not apply:  use two separate fetchnews invocations,
  the first choosing only one server with the '-S server' commandline
  option (or all pretty reliable servers).  Then, lower maxfetch in the
  config file and specify all other servers on the command line (this
  will work in the next alpha).
- There will be a new config option 'only_fetch_once' which will query
  all servers sequentially but stop fetching a group again, once it was
  successfully fetched.  Note the downside of this is that the "last
  article pointer" is not updated for the (server,group) pairs which are
  not fetched, so in case of breakage of the first server many old
  overviews of the next (server,group) will be scanned.  Use maxfetch
  and initialfetch to relieve this somewhat.
  Updating the "last article pointer" for not queried (server,group)
  pairs is not easily possible without races, you will miss articles


leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list