Passing python functions to I3Tray.AddModule()

I3Tray.AddModule() has typically accepted only strings in its first argument. These strings are used to look up C++ classes that inherit from I3Module, like so:

tray.AddModule('Dump', 'dump')

where the C++ module Dump has been registered via the C++ I3_MODULE() macro.

I3Tray.AddModule() now additionally accepts python functions in its first argument, for instance:

tray = I3Tray()

def frame_printer(frame):
    print("Frame is:\n", frame)

tray.AddModule(frame_printer, 'printer')

When icetray receives a python function as the first argument to AddModule(), it constructs a special I3Module of type PythonFunction which forwards the frames that it receives to the python function passed. By default it does this for the Physics stream only.

Warning

it is not AddModule('frame_printer', ... it is AddModule(frame_printer, ...

If you see a message like

RuntimeError: Module/service "frame_printer" not registered with I3_MODULE() or I3_SERVICE_FACTORY()

it is because you’ve put the python function into quotes, making it a string, and icetray is failing to find that string in its registry of available C++ I3Modules.

If you put that function between a BottomlessSource, (which just pushed empty physics frames), the output should look like this:

Frame is:
[ I3Frame :
]

Frame is:
[ I3Frame :
]

Let’s add a second function that puts something into the frame, and modify the frame_printer function to get and print it.

def int_putter(frame):
    frame['some_int'] = icetray.I3Int(777)

tray.AddModule(int_putter, 'putter')


def frame_printer(frame):
    print "Frame is:\n", frame
    value = frame['some_int'].value
    print "Value of int at some_int is", value

tray.AddModule(frame_printer, 'printer')

Here the function int_putter puts I3Int with the value 777 in into the frames as they go by. This is reflected in the table of contents printed by the frame_printer function.

Output:

Frame is:
[ I3Frame :
  'some_int' ==> I3Int
]
Value of int at some_int is 777

Frame is:
[ I3Frame :
  'some_int' ==> I3Int
]
Value of int at some_int is 777

Functions with parameters

To be useful, reusable and modular, such functions need to take parameters such as the location in the frame of useful frame objects, values, thresholds, etc. The hardcoded values 777 and some_int just make our code brittle.

Functions passed to AddModule() may take more than one parameter (the first parameter is always the I3Frame that is flowing through the framework). The parameter values passed to AddModule() will be delivered (along with the current I3Frame, of course) to the keyword parameters of the associated python function passed each time the function is executed.

We modify the function int_putter to accept parameters that specify what value to put inside the I3Int, and where in the frame to put them:

def int_putter(frame, where = 'someplace', value = -1):
    frame[where] = icetray.I3Int(value)

tray.AddModule(int_putter, 'putter',
               where = 'some_int',
               value = 777)

def frame_printer(frame, whatvalue):
    print "Frame is:\n", frame
    value = frame[whatvalue].value
    print "Value of int at", whatvalue, "is", value

tray.AddModule(frame_printer, 'printer',
               whatvalue = 'some_int')

Note the default parameter values for the function int_putter.

Direct Usage of Lambda Functions

Here we use a lambda, (nameless inline) function. Lambda functions are also called lambda expressions because they can only contain simple expressions. Note that functions created with lambda expressions cannot contain statements (if, while, for, try, with, …). Check google for more information on this standard python construct.

This makes writing very short modules possible. A simple function:

def int_putter(frame):
    frame['some_int'] = icetray.I3Int(777)

tray.AddModule(int_putter, 'putter')

can become the single line:

tray.AddModule(lambda fr: fr['a_int'] = icetray.I3Int(777), 'putter')

Choosing streams the functions should run on

The underlying PythonFunction module also takes a parameter Streams, which is a list of stream types that the function should run on. By default this list is [icetray.I3Frame.Physics]. To e.g. cause a python function foo to run on Calibration and Geometry streams, configure as follows:

from icecube import icetray

def foo(frame):
    ...  # do something physicsy here

tray.AddModule(foo, 'foofunc',
               Streams = [icetray.I3Frame.Geometry,
                          icetray.I3Frame.Calibration])

Functions as filters

The functions passed to AddModule() may return None (i.e. never call return at all), or a boolean. The PythonFunction module examines the return values of these functions and if the value is None or True, the module will call PushFrame(): modules further down the chain will see the frame. If the function returns False, the module will drop the frame.

Note

The rationale for having None and True correspond to the same action (typically None is taken to be False), is so that the ‘default’ behavior (when nothing is returned) is reasonable. Otherwise one- or two-line functions that just check or print data would need to have lines return True added. The thinking is that this extra work to provoke behavior that should be default isn’t so elegant. So the rule of thumb is, if you want to drop the frame, return False, otherwise don’t bother returning anything (or return True if it is clearer to do so).

For instance, the following code would cause frames that contain an I3Int with value less than 80 to be dropped:

def ints_are_greater_than(frame,  key,  threshold):
    frameval = frame[key].value
    return frameval > threshold

tray.AddModule(ints_are_greater_than,
               key = 'intlocation',
               threshold = 80)

Passing python functions to I3ConditionalModules

The old way

Recall that an I3ConditionalModule looks for an I3IcePick in its I3Context, indexed by string. So the user must configure an I3IcePickInstaller<T> (where T is the class containing the desired pick logic) and the name given by the user to the instance of this pick logic must match the name that the using module accesses it by.:

tray.AddService('I3IcePickInstaller<I3FrameObjectFilter>', 'fofilter')(
    ("FrameObjectKey", 'some_int')
    )

tray.AddModule('AddNulls', 'adder')(
    ('IcePickServiceKey', 'fofilter'),
    ('where', ['x1', 'x2', 'x3'])
    )

Here the module AddNulls, being an I3ConditionalModule, will add nulls named ‘x1’, ‘x2’, and ‘x3’ to the frame when its icepick, located in its context via the string ‘fofilter’, returns true.

This has several disadvantages:

  • The logic that triggers the AddNulls module is separated from the configuration of the module itself

  • There is the possibility for name collisions in the various I3Context.

If the condition is complicated, for instance the disjunction of two other conditions, the syntax gets yet more verbose.

The new way

As of icetray v3, one can pass a python function to the parameter If of I3ConditionalModule. Identical to the above is the following:

tray.AddModule('AddNulls', 'adder',
               Where = ['x1', 'x2', 'x3'],
               If    = lambda frame: 'some_int' in frame)

Another example: run the reconstruction LineFit if the I3Int at ‘where’ is greater than 80:

def ints_are_greater_than_80(frame):
    frameval = frame['where'].value
    return frameval > 80

tray.AddModule('LineFit', 'linefit',
               HitSeries = 'WhereTheIntIs',
               If = ints_are_greater_than_80)

Note that in this case the key in the frame and the value ‘80’ are hardcoded inside the python function we pass. Not so good: we want to reuse the functions we wrote in previous sections. To do so we use a small python forwarding function:

def fwd(fn, **kwargs):
    def wrap(frame):
        return fn(frame, **kwargs)
    return wrap

Which captures the values of parameters passed to it and passes them on to the function fn. You would use this like this:

def ints_are_greater_than(frame,  key,  value):
    frameval = frame[key].value
    return frameval > value

tray.AddModule('LineFit', 'linefit',
               If = fwd(ints_are_greater_than,
                        key = 'WhereTheIntIs',
                        value = 80))

A forwarding function is necessary here, but not when passing a python function directly to AddModule(). This asymmetry is unfortunate but presently unavoidable.

Functions as I3ConditionalModules

Python functions now support the I3ConditionalModule argument syntax, with optional arguments IcePickServiceKey or If. Use them exactly as described above, or for another example, like this:

def int_putter(frame):
    frame['other_int'] = icetray.I3Int(frame['some_int']*10)

tray.AddModule(int_putter, 'putter',
               If = lambda frame: 'some_int' in frame)

Source code organization

You may want to store your useful functions in their own file, say my_utils.py:

#
#   my_utils.py
#
#   My useful stuff
#

def ints_are_greater_than(frame,  key,  value):
    frameval = frame[key].value
    return frameval > value

Which should be located somewhere along your PYTHONPATH or in the current working directory. To use them from your python scripts simply:

#!/usr/bin/env python3

from my_utils import ints_are_greater_than
from icecube.icetray import I3Tray

tray = I3Tray()

...

tray.AddModule(ints_are_greater_than, 'igt',
               key = 'where',
               value = 30)