Re: [leafnode-list] 2.0b8_ma3 leafnode very slow

[Matthias Andree <ma@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>]

Stefan Wiens <s.wi@xxxxxxx> writes:

> Matthias Andree <ma@xxxxxxxxxxxxxxxxxxxxxxxxxxxx> writes:
> 
> > Stefan Wiens <s.wi@xxxxxxx> writes:
> >
> >> When connected via inetd(8), it chooses full buffering (4096 chars).
> 
> More precisely, the buffer size (but /not/ its mode) depends on
> st_blksize, which seems to be influenced by the initial TCP window size.

Wow. The games people play...

> > It seems relying on libc to set fully-buffered mode is not a good idea.
> 
> OK, 16kB fixed buffer size should do no harm, although I'm unsure if
> it's always advisable to override the kernel's suggestion.

What should happen? Do you want larger buffers? No problem, change
leafnode.h. :-)

> > What is the highest memory usage you see in fixxover that makes you
> > optimize this function which is sort of a one-shot use, runs for a
> > couple of seconds?
> 
> 25MB (no memory leaks ;-). 2 minutes. Reading a large group
> simultaneously causes noticeable swap activity. (xoverutils.c usage
> patterns cause lots of pagefaults.)
> 
> The same work can be done using 1MB, so IMHO fixxover() is bloated.

Agreed.

> I've noticed that you removed the implicit call to readactive() in
> findgroup(). That change considerably speeds up leafnode's startup.
> Next readlocalgroups() will reread the local.groups file but yield an
> empty newgroups list. 

> local.groups might of course be examined for changes before rereading,
> avoiding loss of local groups not yet in groupinfo.

I don't get that. 
 
> I think rereadactive() and readlocalgroups() should be moved into
> activutil.c; rereadactive() being the only exported function.
> All Leafnode programs need the same knowledge about local groups.

rereadactive is already in activutil.c :^)

I'm currently thinking if it's useful to split the entire thing into
separate files.

Cornelius, would you find it acceptable if we made the leafnode compile
process require GNU make rather than any make?

> > I'd rather not see what happens when a newly-fetched active is sorted
> > with a "weaker" sort. You cannot rely on your servers sending sorted
> > lists. (t-online, for instance, doesn't).
> 
> One could sort the list of new groups before merging...
> 
> Keeping track of ascending order when parsing groupinfo (saves 0.3s
> here...) and aborting mergegroups() if the list is empty should catch
> the most common situation.
> Using qsort() if necessary won't hurt that much; it can only happen
> in fetchnews or when a local group is not yet in groupinfo.

Is that "check for ascending order, if violated, call qsort"? If so,
natural mergesort will implicitly keep track of that.

I think I'll implement that with qsort-like arguments, unless I find it
on the web under BSD/MIT license.

-- 
Matthias Andree

-- 
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list