[leafnode-list] Re: Filtering articals

[Paul Brooks <paulbrooks59@xxxxxxxxxxx>]

----------------------------------------
> To: leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> From: ino-news@xxxxxxxxxxxxxxxxxxxxxxxxx
> Date: Sun, 27 Mar 2011 23:24:51 +0200
> Subject: [leafnode-list] Re: Filtering articals
>
> Paul Brooks wrote:
>
> > I see that I can remove articles using applyfilter by filtering on the
> > contents of the article headers before and/or after fetching the
> > article. But is there a way of filtering the articles with reference
> > to the contents of the article body itself. Even if it means using
> > grep and then deleting, if possible, the offending article in some
> > way.
>
> The Lua version has this feature amongst others, but it is (still)
> incompatible with any old-school filtering in leafnode.
>
> If you don't want Lua, you could issue a "find" command to get a list of
> recently pulled articles, "grep" them and remove according to "grep"
> return status.
>
> you could make a small shell script like (untested):
>
> #+v
>
> #!/bin/sh
> #
> # filter-and-delete.sh
> #
> # mind your step and all pathnames and patterns here!
> #
> patterns="${1:?need patterns to check against!}"
> bad_articles="${2:?need directory to put bad articles into!}"
> check_this="${3:?need a file to check!}"
> grep="/bin/egrep"
> copy="/bin/cp"
> sed="/bin/sed -r -n"
> sed_mid="s/^Message-ID:[[:space:]]+(.*)$/\1/p"
> delete="/usr/local/sbin/texpire -C"
> bad_mid=""
> ${grep} "${patterns}" "${check_this}" &&
> bad_mid=$(${sed} -e "${sed_mid}" < "${check_this}")
> [ -n "${bad_mid}" ] && {
> ${copy} "${check_this}" "${bad_articles}/${bad_mid}" &&
> ${delete} "${bad_mid}"
> ex=$?
> }
> exit ${ex}
>
> #-v
>
> use like (equally well tested :)
>
> - "true > /tmp/bad-articles/some-file" before the start of "fetchnews",
> - "fetchnews ..."
> - "find -type f -newer /tmp/bad-articles/some-file \
> /var/spool/news/ -print0 |
> xargs -0 -n1 filter-and-delete.sh /tmp/bad-articles 'patterns'"
>
> Afterwards and with luck you'll find bad articles in the directory
> mentioned. They will also have been expired from any overview files.
>
> You'd clean out the directory with bad articles from time to time, of
> course.
>
>
> clemens

Clemens thanks for the ideas, with reference to what you say about the Lua version,
which is the reason I havnt looked at that route.

re the script, it will take me a while to work my way through it as im not up
on scripting, but thats what learning is about. Thanks

I have since had another thought, I dont know how feasable it is but,
if another switch (say -d(elete)) was added to applyfilter and it was called with
applyfilter -d /path/to/news/artical/10001 to remove the offending article.
Since all the filtering part can be missed and the call to the functions to remove
the article can be made after validity checks that the article is there ect. It would
look after all the overview files and other house keeping itself.
Could that do the trick?


Thanks

Paul.







 		 	   		  
-- 
_______________________________________________
leafnode-list mailing list
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx
https://www.dt.e-technik.uni-dortmund.de/mailman/listinfo/leafnode-list
http://leafnode.sourceforge.net/