Re: [leafnode-list] 2.0b8_ma3 leafnode very slow

[Stefan Wiens <s.wi@xxxxxxx>]

Matthias Andree <ma@xxxxxxxxxxxxxxxxxxxxxxxxxxxx> writes:

> Stefan Wiens <s.wi@xxxxxxx> writes:
>
>> When connected via inetd(8), it chooses full buffering (4096 chars).

More precisely, the buffer size (but /not/ its mode) depends on
st_blksize, which seems to be influenced by the initial TCP window size.

> It seems relying on libc to set fully-buffered mode is not a good idea.

OK, 16kB fixed buffer size should do no harm, although I'm unsure if
it's always advisable to override the kernel's suggestion.

At least here, adjusting buffer size yields only little performance
improvement. 20MB of overview data need <3s for transmission with
an initial delay of at least 4s (for reading and parsing;
P166 / 64MB RAM).

Because the initial delay does not depend on the number of requested
lines, it's much more annoying to me.

> What is the highest memory usage you see in fixxover that makes you
> optimize this function which is sort of a one-shot use, runs for a
> couple of seconds?

25MB (no memory leaks ;-). 2 minutes. Reading a large group
simultaneously causes noticeable swap activity. (xoverutils.c usage
patterns cause lots of pagefaults.)

The same work can be done using 1MB, so IMHO fixxover() is bloated.

> My current version calls mergegroups less often than _ma4, which helps
> already. I believe the readlocalgroups() in rereadactive() is the
> culprit here. I'll see what I can do to cut the qsort invocations down.

I've noticed that you removed the implicit call to readactive() in
findgroup(). That change considerably speeds up leafnode's startup.
Next readlocalgroups() will reread the local.groups file but yield an
empty newgroups list. local.groups might of course be examined
for changes before rereading, avoiding loss of local groups not
yet in groupinfo.

I think rereadactive() and readlocalgroups() should be moved into
activutil.c; rereadactive() being the only exported function.
All Leafnode programs need the same knowledge about local groups.

> I'd rather not see what happens when a newly-fetched active is sorted
> with a "weaker" sort. You cannot rely on your servers sending sorted
> lists. (t-online, for instance, doesn't).

One could sort the list of new groups before merging...

Keeping track of ascending order when parsing groupinfo (saves 0.3s
here...) and aborting mergegroups() if the list is empty should catch
the most common situation.
Using qsort() if necessary won't hurt that much; it can only happen
in fetchnews or when a local group is not yet in groupinfo.

Stefan


-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list