[leafnode-list] Re: add lua-scripting to leafnode2

[Cory Albrecht <cory@xxxxxxxxxxxxxx>]

clemens fischer wrote:
> On Thu, 19 Jun 2008 21:43:33 +0200 clemens fischer wrote:
>
>   
>> C. Albrecht (in private mail):
>>     
My bad, sorry. I mean to reply to the entire list.


>
>   
>> i'm planning to add lua-scripting support to leafnode2.
>>     
>
>   
>>> How about a scripting language other than lua, one that's more well
>>> known, like perl?
>>>
>>> Or, better yet, how about a sufficiently generic interface such that
>>> if you add lua, I could add perl for myself and Joe could ad python
>>> for himself?
>>>       
>
> well, all the code comes in three files: script-lua.c, script-lua.h and
> script.h. the first two contain lua specific code. script.h contains
> a very lean magic implementing just that: depending on which type of
> scripting is requested, the necessary header files are included or
> macros defined. for example, if no scripting is desired, script.h
> contains dummy definitions for the stubs in fetchnews. the compilers
> preprocessor substitutes them with nothing, which effectively compiles
> fetchnews "proper", ie. without any scripting.
>
> currently only the original ln2 file called "store.c" is instrumented
> with calls like: "script_err = script_init_article()", "script_err =
> script_add_headers(head)" etc. thus anybody wanting to implement python-
> or perl-scripting just needs to add, eg. script-{perl,python}.{c,h} and
> implement "script_init_article()" et.al.  the instrumentation calls only
> have to be changed for new features, but not (necessarily) for certain
> script-language interfaces.
>
>   
>>> I would like perl, because then one could conceivably integrate
>>> SpamAssassin processing into leafnode2.
>>>       
>
> then you have the following alternatives:  model the interface to SA in
> lua or add perl-scripting.
>   

Since SpamAssassin is written completely in perl, I can't imagine that a
lua interface to SA woudl be a small undertaking. However, I know zero
about lua, other than it exists.

> the reason i chose lua were:
>
>   
>> i picked lua as the language for the extensions because (a) matthias
>> mentioned it in a private mail, (b) it is quite easy to implement, (c)
>> users can easily add a lua-sockets module and use it for persistent
>> connections to their favourite filter-engine, and (d) it is nice and
>> easy to program in. you might want to check <url:http://www.lua.org>
>> for further information on the language.
>>     
>
> the scripting addon will be as lightweight and generic as possible.
> lua comes in the form of a library. the entire compiler + interpreter
> + builtin-libs + utility-libs is 155648 bytes of size. these numbers
> are from the mappings of a live elinks-browser on freebsd-current, they
> may vary for other platforms. lua compiles, embeds and acts the same on
> every supported platform.
>
>   
Since perl is already there on 99.9% of un*x systems by default, one
would only link with the appropriate embedding *.so file. I've done it under
 Windows, but not a un*x, so I can't say I am 100% on the exact
mechanics in un*x.

Also, I was assuming that one would use autoconf (or whatever)
--with-<feature>/--enable-<feature> arguments when setting up the make
environment. That would negate extra size because an admin could build
with it or without based on personal preferences.

> i cannot imagine perl or python to sum up to less then 200kB, but
> nonetheless i don't want to be in the way.  if you want another language
> and can embed it, please help yourself.  i would strongly suggest
> waiting for the completion of the lua-scripting though.
>
>   
>From my perspective, preferring perl, all I would ask that the scripting
interface be language agnostic.

What I'm thinking is sort of like this:

1. the main leafnode2 config file has a way of saying dlopen <blank>.so
for a scripting language (allows for multiple modules), i.e. :

    load_script_lib = perl /usr/lib/leafnode2/leafnode-perl.so
    load_script_lib = lua /usr/lib/leafnode2/leafnode-lua.so"

2. in the filters file you have a way specify  the .so file from the
load_script_lib and the name of the filter, e.g.

    language = perl
    name = CheckWithSpamAssassin
    action = select

3. When processing filters on a received message, leafnode2 determines
whether the filter is a old-style filter or an external filter. it's
it's an external filter, send it off to routines in hypothetical
script.c part of leafnode2 which take care of calling the external .so
and passing it the entire NNTP article and then reporting back to the
filter processor the possibly altered article and a true or false value
indicating whether action happens or not. In my hypothetical perl
CheckWithSpamAssassin filter, it always returns true but it adds various
X-Spam-* headers.

Of course there are many of way this could be done. Maybe the external
.so files only only need to export the function int
DoLeafnodeScript(char *lpScriptName, struct _article *pArticle).

Or maybe the command load_script_lib in config only needs one param, the
name of the .so to load and not the language.Leafnode2, when parsing
config calls char **ListAllFilters(void) in the .so which returns an
array of strings which are the symbols in the .so as well as what goes
in the "name =" line in filters. The filters processor sees the name
parameter and hands it off to script.c  which takes care of calling int
CheckWithSpamAssassin(struct _article *) from the appropriate .so and
again returning the possibly altered article along with the pass/fail
result.

Like I said, I don't know lua from a hole in the ground, but I do know
that by using the embedding magic for perl that one can take values in a
C/C++ program and link them with perl values so that the perl script can
read and maybe modify variables internal to the program. Thus the
calling of the external flters could get very complex and possible
modify the state of the running fetchnews rathe rthan merely returning a
pass/fail value on a filter.

> as interfacing to lua and from lua to C isn't very difficult, it might
> well turn out to be an easier task once the interface has stabilized.
>
> lua has lexical scoping, provides closures, does automatic memory
> management, handles multiple return values etc. it might suit most
> peoples programming styles, there are readymade modules to connect
> sockets and it is supported by SWIG[1]. also, lua is considered to be a
> fast scripting language. i think one gets the most out of using lua for
> this effort.
>
> [1] <url:http://www.swig.org/>
>
>
> regards, clemens
>
>   

-- 
_______________________________________________
leafnode-list mailing list
leafnode-list@xxxxxxxxxxxxxxxxxxxxxxxxxxxx
https://www.dt.e-technik.uni-dortmund.de/mailman/listinfo/leafnode-list
http://leafnode.sourceforge.net/