[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[leafnode-list] Leafnode slow since 1.9.33
Hello,
I am using leafnode on a Debian Sarge (testing) system. Sometime ago I
noticed that after an upgrade of Leafnode to 1.9.33 the system became
terribly slow, perhaps even stopped making progress.
At the time I did not have the time to look into it, downgraded to
1.9.27 and left it at that.
Now I got some unrelated problems, losing some messages, so I upgraded
again and found the slowness still exists. Here is the result of my
investigation.
The slowness is when fetchnews is gone to the background updating the
xover database. In 1.9.27 this part would take 0.5-1 minute perhaps. In
1.9.42 this may take up to 45 minutes on my machine. Retrieving headers
from Leafnode seems to be slow as well, but I did not look to much at
that.
Here is part of the debug log:
| Aug 23 19:29:33 bali fetchnews[12301]: Process forked.
| Aug 23 19:34:41 bali fetchnews[12301]: reading XOVER info from /var/spool/news/comp/os/linux/setup/35220
| Aug 23 19:34:41 bali fetchnews[12301]: updated 1 lines in /var/spool/news/comp/os/linux/setup/.overview
| Aug 23 19:34:42 bali fetchnews[12301]: wrote /var/spool/news/comp/os/linux/setup/.overview
| Aug 23 19:35:01 bali fetchnews[12301]: reading XOVER info from /var/spool/news/nl/internet/providers/30489
| Aug 23 19:35:01 bali fetchnews[12301]: updated 1 lines in /var/spool/news/nl/internet/providers/.overview
| Aug 23 19:35:03 bali fetchnews[12301]: wrote /var/spool/news/nl/internet/providers/.overview
| Aug 23 19:36:50 bali fetchnews[12301]: reading XOVER info from /var/spool/news/nl/juridisch/55582
| Aug 23 19:36:50 bali fetchnews[12301]: reading XOVER info from /var/spool/news/nl/juridisch/55583
| Aug 23 19:36:50 bali fetchnews[12301]: reading XOVER info from /var/spool/news/nl/juridisch/55584
| Aug 23 19:36:50 bali fetchnews[12301]: reading XOVER info from /var/spool/news/nl/juridisch/55585
| Aug 23 19:36:50 bali fetchnews[12301]: reading XOVER info from /var/spool/news/nl/juridisch/55586
| Aug 23 19:36:50 bali fetchnews[12301]: updated 5 lines in /var/spool/news/nl/juridisch/.overview
| Aug 23 19:36:53 bali fetchnews[12301]: wrote /var/spool/news/nl/juridisch/.overview
| Aug 23 19:36:57 bali fetchnews[12301]: reading XOVER info from /var/spool/news/sci/engr/heat-vent-ac/2794
| Aug 23 19:36:57 bali fetchnews[12301]: reading XOVER info from /var/spool/news/sci/engr/heat-vent-ac/2795
| Aug 23 19:36:57 bali fetchnews[12301]: updated 2 lines in /var/spool/news/sci/engr/heat-vent-ac/.overview
| Aug 23 19:36:58 bali fetchnews[12301]: wrote /var/spool/news/sci/engr/heat-vent-ac/.overview
| Aug 23 19:42:55 bali fetchnews[12301]: reading XOVER info from /var/spool/news/gmane/linux/kernel/54218
[cut similar lines]
| Aug 23 19:59:38 bali fetchnews[12301]: wrote /var/spool/news/uk/telecom/broadband/.overview
| Aug 23 19:59:57 bali fetchnews[12301]: reading XOVER info from /var/spool/news/nl/comp/hardware/23973
| Aug 23 19:59:57 bali fetchnews[12301]: reading XOVER info from /var/spool/news/nl/comp/hardware/23974
| Aug 23 19:59:57 bali fetchnews[12301]: reading XOVER info from /var/spool/news/nl/comp/hardware/23975
| Aug 23 19:59:57 bali fetchnews[12301]: updated 3 lines in /var/spool/news/nl/comp/hardware/.overview
| Aug 23 19:59:58 bali fetchnews[12301]: wrote /var/spool/news/nl/comp/hardware/.overview
| Aug 23 20:00:20 bali fetchnews[12301]: reading XOVER info from /var/spool/news/sci/engr/chem/2973
| Aug 23 20:00:20 bali fetchnews[12301]: reading XOVER info from /var/spool/news/sci/engr/chem/2974
| Aug 23 20:00:20 bali fetchnews[12301]: updated 2 lines in /var/spool/news/sci/engr/chem/.overview
| Aug 23 20:00:20 bali fetchnews[12301]: wrote /var/spool/news/sci/engr/chem/.overview
| Aug 23 20:00:24 bali fetchnews[12301]: reading XOVER info from /var/spool/news/sci/engr/mech/5712
| Aug 23 20:00:24 bali fetchnews[12301]: reading XOVER info from /var/spool/news/sci/engr/mech/5713
| Aug 23 20:00:24 bali fetchnews[12301]: updated 2 lines in /var/spool/news/sci/engr/mech/.overview
| Aug 23 20:00:24 bali fetchnews[12301]: wrote /var/spool/news/sci/engr/mech/.overview
| Aug 23 20:00:59 bali fetchnews[12301]: Process done.
To me it looks that there is no obvious errors and the only thing that
stands out is the rather long time between the last message about a
newsgroup and the first of the next one.
A log under 1.9.27 (not shown) does not have these time gaps.
Now working from 1.9.27 up to 1.9.33 I found what is causing the
slowness. It is this part of the 1.9.33 diff:
| diff -Nur leafnode-1.9.32.rel/xoverutil.c leafnode-1.9.33.rel/xoverutil.c
| --- leafnode-1.9.32.rel/xoverutil.c 2003-01-03 03:35:09.000000000 +0100
| +++ leafnode-1.9.33.rel/xoverutil.c 2003-01-27 04:18:46.000000000 +0100
| @@ -265,7 +265,10 @@
| xfirst = ULONG_MAX;
| xlast = 0;
| while ((de = readdir(d))) {
| + struct stat st;
| if (de->d_name[0] == '.') continue; /* skip dot files */
| + if (stat(de->d_name, &st) == 0 && !S_ISREG(st.st_mode))
| + continue; /* skip directories, e. g. alt.games.simcity.3000 */
| art = strtoul(de->d_name, &p, 10);
| if (art && p && !*p) {
| if (art < xfirst)
| @@ -341,8 +344,11 @@
|
| /* so, what was missing? */
| while ((de = readdir(d))) {
| + struct stat st;
| if (de->d_name[0] == '.')
| continue;
| + if (stat(de->d_name, &st) == 0 && !S_ISREG(st.st_mode))
| + continue;
| art = strtoul(de->d_name, &p, 10);
| if (p && !*p && art >= xfirst && art <= xlast) {
| if (!xoverinfo[art - xfirst].text) {
Reverting it also makes 1.9.42 run fine as before. I'm not subscribed to
such stupidly named newsgroups, but I realise this brings a bug back
again.
It looks that a stat() for every article is just too expensive here, soo
perhaps a list of known subdirs should be maintained?
Details of my system:
Debian testing
kernel 2.4.18
Pentium I 166 MHz with 64 MB memory
News spool is about 2.5 GB on ReiserFS
Disk is a moderately fast ATE with "hdparm -t" throughput of 10-11 MB/s
Rein.
--
Rein Klazes
rklazes@xxxxxxxxx
--
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- mailing list for leafnode
To unsubscribe, send mail with "unsubscribe" in the subject to the list