Re: [leafnode-list] fetchnews and logrotate?

["Mike Vanecek" <leaf_list@xxxxxxxxxxxxx>]

On Mon, 19 Jan 2004 18:18:14 +0100, Matthias Andree wrote
> "Mike Vanecek" <leaf_list@xxxxxxxxxxxxx> writes:
> 
> > I received the following cron message. From what I can tell the news log file
> > reached a size maximum during an active fetchnews. As a result, a large
> > portion of the log file to be rotated was lost and, from the message below,
> > the leafnode pointers were messed up.
> 
> That mustn't happen. If logrotate loses entries, it's misconfigured.

My error, sorry.

> > Date: Fri, 16 Jan 2004 18:14:41 -0600 (CST) 
> > Subject: Cron <news@www> /usr/sbin/fetchnews 	All headers
> >  
> > Active has not been fetched completely in previous run
> > or has never been fetched, forcing active fetch.
> > Warning: skipping group "alt.", invalid name (NULL component).
> > Newsgroup name conflict: 3dfx.oem.products.diamond.monster3d vs.
> > 3dfx.oem.products.diamond.monster3D
> 
> Complain to the maintainers of your news servers for not agreeing on
> group names. Leafnode prefers the version with least upper-case
> characters.

OK, not a problem except when active is refreshed. I can safely ignore the
name conflict messages.

Why this message:  "skipping group "alt.", invalid name (NULL component)?"

Alt is being retrieved successfully.

> > # Newslog
> > /var/log/newslog  {
> >     missingok
> >     compress
> >     size 2M
> >     rotate 3
> >     create 0660 root news
> > }
> 
> Broken. Add these lines:
> 
> sharedscripts
> postrotate
>     kill -1 $(cat /var/run/syslogd.pid)
> endscript
> 
> before the closing curly brace.
> 

Argg, I should have caught that ... I have moved it to /etc/logrotate.d/newslog as

/var/log/newslog  {
    missingok
    sharedscripts
    compress
    rotate 4
    size 4M
    create 0660 root news
    postrotate
      /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

> > Obviously, if the rotate occurs when fetchnews is writing to the log,
> > confusion will occur.
> 
> No. Leafnode does not write to the log, but syslogd does.

The error has been there for some time but was never triggered because the
monthly rotate occurred rather than a size. Since newslog had been rotated
before syslog items, syslogd was restarted by the syslog items and never
showed up in the log. Nothing is ever bug free .... drat ! Restarted syslogd
and fetchnews, as you said it would, is logging again.

> No, the group list is just fetchnews' memory where it needs to pick 
> up at the next run. What groups are fetched is determined by 
> interesting.groups and can optionally be filtered trough the server-
> specific configuration parameter only_groups_pcre.

OK, thanks, that works

server = news.spamcop.net
only_groups_pcre = spamcop\.

However, this all started because my news log file was growing faster than I
remember from the past.

> > I am getting tons of messages such as
> >
> > news.software.readers: killed 170187
> > (<rk38d1-tl5.ln1@xxxxxxxxxxxxxxxxxxxxxxxxxx>), already fetched before
> 
> These are fine. You're using multiple upstreams and fetchnews is telling
> you that a server queried previously already had this article downloaded.

I am confused on this one. The servers do not carry the same groups. Spamcop
only carries spamcop.* news groups. Newscene carries the rest. If I run with
-vvv, the log/screen has 5000-6000 lines of these "already fetched before"
messages. Should not it remember what was fetched previously and not ask for
them again? I do not remember seeing this type of activity with previous
versions of fetchmail?

I am su - news then /usr/sbin/fetchnews ... I did this twice in a row. My news
log file is now 1.1M (from 0). 

For example,

Jan 19 13:02:19 www fetchnews[1054]: microsoft.public.pocketpc: last seen
article was 350319, server now has 342874-350319

Jan 19 13:02:19 www fetchnews[1054]: skipping articles 342874-348309 inclusive
(article limit)

** Why is it even considering 342874-348309 since that is clearly before
350319? **

Jan 19 13:02:19 www fetchnews[1054]: microsoft.public.pocketpc: considering
articles 348310 - 350310

** Why is it even considering 348310 - 350310 since that is clearly before
350319? **

Jan 19 13:02:20 www fetchnews[1054]: microsoft.public.pocketpc: killed 348310
(<xPcLb.1206$Wa.1169@xxxxxxxxxxxxxxxxxxxxxxxxxx>), too old (11 > 10) days
Jan 19 13:02:20 www fetchnews[1054]: microsoft.public.pocketpc: killed 348311
(<btjmju$7gr4l$1@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>), too old (11 > 10) days
Jan 19 13:02:20 www fetchnews[1054]: microsoft.public.pocketpc: killed 348312
(<btjmmc$848n6$1@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>), too old (11 > 10) days
Jan 19 13:02:20 www fetchnews[1054]: microsoft.public.pocketpc: killed 348313
(<btjmpj$7vmrk$1@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>), too old (11 > 10) days

** If the last seen article was 350319, why does it need to skip 348310? **

and 

Jan 19 13:02:20 www fetchnews[1054]: microsoft.public.pocketpc: killed 348517
(<eQ1KrLq1DHA.2544@xxxxxxxxxxxxxxxxxxxx>), already fetched before
Jan 19 13:02:20 www fetchnews[1054]: microsoft.public.pocketpc: killed 348518
(<#eEarOq1DHA.3116@xxxxxxxxxxxxxxxxxxxx>), already fetched before
Jan 19 13:02:20 www fetchnews[1054]: microsoft.public.pocketpc: killed 348519
(<uhUk5Pq1DHA.2060@xxxxxxxxxxxxxxxxxxxx>), already fetched before

** Why is it looking at 348517 since it knows that the last article was 350319? **

I do not remember seeing this before nor do I remember seeing the newslog fill
up so fast.

What am I missing?

Thank you for your help. If I need to read and/or test something else, please
point me in the correct direction.

-- 
_______________________________________________
leafnode-list mailing list
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx
http://www.dt.e-technik.uni-dortmund.de/mailman/listinfo/leafnode-list
http://leafnode.sourceforge.net/