Writing Gamera Plugins

Last modified: May 11, 2016

Contents

Introduction

The functionality of Gamera can be extended by writing plugins in either C++ or Python. A plugin is simply a set of methods (which are automagically added to the Image class) or free-standing functions. Plugins are technially just Python modules, but with more information that allows for easier wrapping and compilation of C++ methods and to support all kinds of automatic things in the graphical user interface.

Plugins can also be grouped together, with other tools, into toolkits. Toolkits provide higher-level workflow framework for end-to-end document recognition by joining together a number of steps from various plugins. Toolkits are a whole other discussion, so see the writing Gamera toolkits for more information.

Before writing any plugin, you should make sure there isn't already a plugin included that does what you want. Look at the list of plugins included with Gamera.

The files involved

Each plugin is made up of two files:

  1. A Python file that describes each method in the plugin and the plugin itself. If any methods are "pure Python", they can also be defined here, or they can just delegate to functions in other Python modules.
  2. Optionally, a C++ header file containing implementations of any C++ methods of the plugin. This is a header file (.hpp) and not an implementation file (.cpp) because the code you write will be templatized and the concrete methods and the glue code connecting to Python to C++ will be generated automatically at compile time by the Gamera build system. (What templates are is beyond the scope of this document, but it's covered very well in [Stroustrup1997].)

Plugins in the Gamera source tree

The Python metadata files are stored in ./gamera/plugins/ and the C++ source files are stored in ./include/plugins.

If you keep these files in the proper directories, they will be automatically picked up by the build system and compiled. When Gamera is started up, it will search the ./gamera/plugins directory and load all plugins.

Plugins in a toolkit

In a toolkit, the Python metadata files are stored in ./gamera/toolkits/my_toolkit/plugins/ and the C++ source files are stored in ./include/plugins, both rooted at the top of your toolkit directory.

Plugin modules included in toolkits will have to be explicitly imported before they are available.

A simple example

Plugin metadata

Let's look at a simple metadata file, example.py. Each method is described by creating a class that inherits from gamera.plugin.PluginFunction and defining a number of special members. The whole plugin is described by a class that inherits from gamera.plugin.PluginModule:

from gamera.plugin import *
import _example

# C++ method
class volume(PluginFunction):
    """
    Returns the ratio of black pixels to white pixels within the
    bounding box of the image.
    """
    self_type = ImageType([ONEBIT])
    return_type = Float("volume")
    doc_examples = [(ONEBIT,)]

class ExampleModule(PluginModule):
    category = "Example"
    cpp_headers=["example.hpp"]
    functions = [volume]
    author = "Michael Droettboom and Karl MacMillan"
    url = "http://gamera.informatik.hsnr.de/"
module = ExampleModule()

Okay, now let's break it down.

The gamera.plugin module contains all of the utilities necessary to create Gamera plugins, so the first thing we do is import it:

from gamera.plugin import *

Next, we import the C++ (compiled object file) side of the plugin, (described in the next section) which always has the same name as the Python metadata module, except with a leading underscore:

import _example

Let's start by describing a minimal C++ method. All methods and functions in a plugin are described using a class that inherits from gamera.plugin.PluginFunction:

# C++ method
class volume(PluginFunction):

Each plugin can (and should) be documented in the usual Python docstring way:

"""
Returns the ratio of black pixels to white pixels within the
bounding box of the image.
"""

On a related note, you can also have the documentation system (doc/gendoc.py) generate an example automatically. See documenting and unit-testing Plugin functions.

Next, we define self_type, which is the type of object this method can be called on. If self_type is an ImageType, the method will automatically be added to all Image objects in Gamera whenever the plugin is imported. Within the ImageType specifier, you can choose which types of pixels are supported using a list of pixel type names. Valid options are ONEBIT, GREYSCALE, GREY16, FLOAT and RGB (these are all constants imported from the gamera.plugin module):

self_type = ImageType([ONEBIT])

You can also optionally define return_type. This specifier is used to generate a variable name for the result in the GUI, and so the C++ wrapping machanism knows how to return the result to Python. If you don't specify a return type, Gamera assumes there is no return result:

return_type = Float("volume")

Obviously, this is a very simple plugin method with no arguments. Some more involved examples are given below. In the meantime, let's look at how this method is contained in a plugin module.

For each plugin module, you also need a class to describe the entire plugin. There may be only one of these classes per plugin. This is done in a similar manner to how the methods are described.

There is a class that inherits from gamera.plugin.PluginModule:

class ExampleModule(PluginModule):

You can specify a category for the plugin's methods on the context (right-click) menu in the GUI:

category = "Example"

If you have any C++ methods (which we do in this case), you must specify the C++ header files to include which contain the corresponding method's source code:

cpp_headers=["example.hpp"]

You must also list all of the plugins and methods in the file so they can be generated and loaded:

functions = [volume]

Optionally, the author names and a URL for more information can be specified:

author = "Michael Droettboom and Karl MacMillan"
url = "http://gamera.informatik.hsnr.de/"

Lastly, we create an instance of this class so the module loader can do its work:

module = ExampleModule()

C++ code

Since the volume method needs to look at individual pixels, it is likely going to be much faster written in C++ than in Python. Below is the corresponding example.hpp that contains the C++ implementation:

#include "gamera.hpp"

using namespace Gamera;

template<class T>
float volume(const T &m) {
  unsigned int count = 0;
  typename T::const_vec_iterator i = m.vec_begin();
  for (; i != m.vec_end(); i++)
    if (is_black(*i))
      count++;
  return (feature_t(count) / (m.nrows() * m.ncols()));
}

Most of the declarations needed for Gamera are in gamera.hpp, and all of that stuff is in the Gamera namespace, to prevent name collisions. You may find it most convenient to just put using namespace Gamera at the top of your plugin file, rather than specifying Gamera::... everywhere:

#include "gamera.hpp"

using namespace Gamera;

Next we get to the function itself. Note that it is templatized. Since it is our goal to write a single algorithm that may work on multiple image types, all plugin methods are templatized, and the instantiations of these templates are generated by the Gamera build system at compile-time based on the self_type specifier in the method metadata class (that we specified in example.py). See how the first argument self is templatized as T so that any (image) type can be passed in. The body of the function used the Gamera C++ Image API to access and examine the individual pixels:

template<class T>
float volume(const T &m) {
  unsigned int count = 0;
  typename T::const_vec_iterator i = m.vec_begin();
  for (; i != m.vec_end(); i++)
    if (is_black(*i))
      count++;
  return (feature_t(count) / (m.nrows() * m.ncols()));
}

Building the plugin

Okay, so now we're done with the minimal plugin, but obviously something more has to happen in order to access the C++ code from Python. Fortunately, that is all done automatically by the Gamera build system. If the example.py is placed in the ./gamera/plugins directory, the build system will automatically find it, use the metadata to generate a wrapper to access example.hpp, and compile everything. The next time Gamera is run, the plugin will automatically be loaded. The plugin author does not have to learn about the intricacies of the Python/C API.

But, for the sake of some sick curiosity, the generated code looks something like:

#include <string>
#include <stdexcept>
#include "Python.h"
#include <list>
#include "gameramodule.hpp"

init_features (void)
{
  Py_InitModule ("_features", _features_methods);
}

#include "features.hpp"

using namespace Gamera;

extern "C"
{
  void init_example (void);
  static PyObject *call_volume (PyObject * self, PyObject * args);
  static PyMethodDef _features_methods[] =
    { {"volume", call_volume, METH_VARARGS} };

  static PyObject *call_volume(PyObject * self, PyObject * args)
  {
    PyObject *real_self;
    Image *real_self_image;

    FloatVector *return_value = 0;

    if (PyArg_ParseTuple(args, "O", &real_self) <= 0)
      return 0;
    if (!is_ImageObject(real_self)) {
      PyErr_SetString(PyExc_TypeError, "Object is not an image as expected!");
      return 0;
    }
    real_self_image = ((Image *) ((RectObject *) real_self)->m_x);
    image_get_fv(real_self, &real_self_image->features, &real_self_image->features_len);
    try {
    switch (get_image_combination (real_self)) {
      case ONEBITRLEIMAGEVIEW:
        return_value = volume(*((OneBitRleImageView *) real_self_image));
        break;
      case RLECC:
        return_value = volume(*((RleCc *) real_self_image));
        break;
      case CC:
        return_value = volume(*((Cc *) real_self_image));
        break;
      case ONEBITIMAGEVIEW:
        return_value = volume (*((OneBitImageView *) real_self_image));
        break;
      default:
        PyErr_SetString (PyExc_TypeError,
                         "Image types do not match function signature.");
        return 0;
      }
    }
    catch (std::exception & e)
    {
      PyErr_SetString (PyExc_RuntimeError, e.what());
      return 0;
    }
    PyObject *array_init = get_ArrayInit();
    if (array_init == 0)
      return 0;
    PyObject *str = PyString_FromStringAndSize((char *) (&((*return_value)[0])),
                                return_value->size () * sizeof (double));
    PyObject *array = PyObject_CallFunction(array_init, "sO", "d", str);
    delete return_value;
    return array;
  }
  DL_EXPORT (void) init_example (void)
  {
    Py_InitModule ("_example", _example_methods);
  }
}

Aren't you glad you don't have to write something like that every time!

Advanced features

Specifying arguments

Of course, many plugin methods will need to have arguments. See this resize_copy method, for example:

# C++ image method with some arguments
class resize_copy(PluginFunction):
    """
    Copies and resizes an image. In addition to size the type of
    interpolation can be specified to allow tradeoffs between speed
    and quality.
    """
    category = "Utility/Copy"
    self_type = ImageType([ONEBIT, GREYSCALE, GREY16, FLOAT, RGB])
    args = Args([Int("nrows"), Int("ncols"),
                Choice("Interpolation Type", ["None", "Linear", "Spline"])])
    return_type = ImageType([ONEBIT, GREYSCALE, GREY16, FLOAT, RGB])

And the corresponding C++ declaration:

template<class T>
Image* resize_copy(T& image, int nrows, int ncols, int resize_quality);

The args member variable specifies a list of the arguments that are passable to the method. Note that this does not include the first "argument" to the C++ function, which always corresponds to self_type. This specification is used to generate wrapper code, and also to generate dialog boxes in the GUI. The format of these argument lists are documented in Specifying arguments.

The args parameter in the plugin prototype allows the specification of default values for arguments. These are however used only in the GUI for the argument dialog box. If you need an actual default argument for your plugin function, you must define the __call__ method in your plugin, e.g.

  # wrapper for passing a default argument
  def __call__(self, nrows, ncols, interpolation="Linear"):
      return _example.resize_copy(self, nrows, ncols, interpolation)
__call__ = staticmethod(__call__)

_example must be replaced by the actual name of your source file plus a leading underscore.

Free functions

It doesn't always make sense to have everything be a method of images. For example, you may want to create a function that requires a list of images as input. Fortunately, you can still use the plugin system to automate the wrapping/building process, while foregoing the automatic inclusion of the method in the Image class and on the right-click context menu. It's as simple as setting self_type to None in the metadata object:

# C++ free function
class union_images(PluginFunction):
    """
    Creates a new image by overlaying all the images in the given list.
    """
    self_type = None
    args = Args([ImageList('list_of_images')])
    return_type = ImageType([ONEBIT])

As these functions are not image methods, but standalone callable classes, you additionally must create an instance of this class in the same python metadata file:

union_images = union_images()

Pure Python methods

Sometimes there is not much efficiency to be gained by writing the plugin method in C++, or you want the flexibility of Python for experimentation. In that case you can implement the method in pure Python. Everything else is the same, except you add a __call__ method with the Python implementation. It is important that this method is a staticmethod, since the first self argument is going to be an Image object and not a PluginFunction object:

# Python image method
class area(PluginFunction):
    """
    Returns the aspect ratio of the bounding box of the image.
    """
    self_type = ImageType([ONEBIT, GREYSCALE, GREY16, FLOAT, RGB])
    return_type = Float("area")
    pure_python = True
    def __call__(self):
        return float(self.ncols) / float(self.nrows)
    __call__ = staticmethod(__call__)

Free pure Python functions

Since the plugin modules are also just regular Python modules underneath, it is of course possible to just add ordinary Python functions as well:

# Python free function
def filter_small_images(l):
    return [x for x in l if x.ncols > 2 and x.nrows > 2]

Raising exceptions

The convention in Gamera is to use exceptions for error conditions rather than by using error codes.

C++ exceptions are automatically propagated to Python. (All C++ exception types will be converted to Python RuntimeError.)

throw std::runtime_error("Input is out of range");

From pure Python functions, the standard Python exception mechanism can be used:

raise RuntimeError("Input is out of range")

Progress bars

This section describes how to display a progress bar dialog from a long-running plugin. When the GUI is running, the progress will be displayed in a window:

images/progress_bar.png

It is also possible to display a progress bar made from text characters in the console, when the GUI is not running. To make it appear, set the config option progress_bar to True as follows:

from gamera.config import config
config.set("progress_bar",True)

Progress bars will add some overhead when displayed, so they only make sense for plugins that take a long time to complete. Supporting progress bars adds very minimal overhead when they are not displayed.

Progress Bars in Python

To create a progress bar that is a message box in the GUI and a text line in a non GUI script, you can use the ProgressFactory:

from gamera.util import ProgressFactory
progress = ProgressFactory("Title", length, numsteps=0)

where length is the total number of which the progress fraction is to be shown. The optional argument numsteps can be useful to reduce the overhead by only updating the progress bar in numsteps discrete steps; when zero, every update call will result in an update of the progress bar.

To update the progress bar, there are two alternative methods:

  • .step() increases the progress counter by one. Whether the displayed progress bar actually is updated depends on numsteps.
  • .update(steps, length) sets the progress bar to the steps/length fraction.

Progress Bars in C++ Plugins

To create a plugin method with a progress bar, simply set the progress_bar member to the message that will be displayed in the progress box.

class cc_analysis(Segmenter):
  ...
  progress_bar = "Generating connected components"
  ...

Add an extra argument to the C++ function that takes an object of type ProgressBar. This can be a default argument, to make it easier to call the plugin function code without requiring a ProgressBar instance. Creating a ProgressBar with no constructor arguments creates a dummy ProgressBar object where all methods are ignored.

template<class T>
ImageList* cc_analysis(T& image, ProgressBar progress_bar = ProgressBar()) {
  ...
}

The progress bar window will automatically disappear when the function returns.

There are essentially two ways to update the progress bar:

  • Call .set_length(*length*) to set the number of steps that will be performed, and then call .step() to increase the step.
  • Call .update(*num*, *den*) to say that num of den steps have completed. .update is useful when the number of steps can not be pre-determined.

Documenting and unit-testing Plugin functions

The docstring of each PluginFunction class is used like a regular Python docstring, but also has the following advantages in Gamera:

  • It will be included in the automatically generated HTML documentation.
  • It is displayed in the Documentation pane in the Gamera shell window.
  • It is displayed in the automatically-generated dialog box for the plugin.

The docstrings should be formatted in reStructuredText, which is becoming a de-facto Python standard for documentation, as well as being rather easy to read and use.

The Gamera documentation can be regenerated by going to the doc directory (in the source distribution) and running the gendoc.py script:

python gendoc.py

In addition to text, image examples can be generated on-the-fly and included in the documentation using the doc_examples member. The doc_examples mechanism is also used to write rudimentary unit-tests for Gamera's unit-testing framework.

The doc_examples member is a list of tuples or functions:

  • If the element is a tuple, it is a list of arguments that will be passed into the plugin method to create an example. Where image arguments are expected, image type identifiers can be used, which will load a standard image from disk and use it. For example:

    (ONEBIT, 52, 32)
    

    will use the standard OneBit image and the arguments of 52, 32.

  • If the element is a function, that function will be called to create the example. The function will be passed one argument, images, which is a dictionary of the standard Gamera example images. Any images or values returned will be included in the documentation. Any exceptions raised by the function will be logged by the unit-testing framework when it is run. For example the following loads the standard RGB and GreyScale images, clips them appropriately, adds them together, and then returns all of them for inclusion in the documentation.

def __doc_example1__(images):
    rgb = images[RGB]
    greyscale = images[GREYSCALE]
    clipped = rgb.clip_image(greyscale)
    return [clipped, greyscale,
            clipped.add_images(greyscale.to_rgb(), False)]
doc_examples = [__doc_example1__]

Examples

The following doc_examples specifier (from simple_sharpen) produces two examples, one on the standard GREYSCALE image, and another on the standard RGB image, using different values for the sharpening_factor.

doc_examples = [(GREYSCALE, 1.0), (RGB, 3.0)]

For draw_bezier, a custom example function was used, which does not load a standard image.

def __doc_example1__(images):
    from random import randint
    from gamera.core import Image, Dim
    image = Image((0, 0), Dim(100, 100), RGB, DENSE)
    for i in range(10):
        image.draw_bezier((randint(0, 100), randint(0, 100)),
                          (randint(0, 100), randint(0, 100)),
                          (randint(0, 100), randint(0, 100)),
                          (randint(0, 100), randint(0, 100)),
                          RGBPixel(randint(0, 255),
                          randint(0,255),
                          randint(0, 255)))
    return image
doc_examples = [__doc_example1__]

Feature generators: A special kind of plugin

Plugin methods that take an image as input and generate some floating point features from it are called "feature generators". The resulting floating point features are used by the classifier to classify images.

For efficiency reasons, feature generator functions are implemented slightly differently from a regular PluginFunction. Rather than returning the features as a return value, which would require a memory copy into the image's feature vector, feature generators write directly to a buffer that is passed in as an argument.

As an example, let's look at the nholes feature generator. The Python metadata is the same as you would expect, except the member feature_function is set to True. This will tell the build system to treat this plugin method as a feature generator with the different return value behavior. The return_type must be a FloatVector, where length indicates the number of feature values that are generated.

class nholes(PluginFunction):
    """
    Returns the average number of transitions from white to black
    in each row or column.

    The elements of the returned ``FloatVector`` are:

    0. vertical
    1. horizontal

    These features are scale invariant.
    """
    self_type = ImageType([ONEBIT])
    return_type = FloatVector(length=2)
    feature_function = True
    doc_examples = [(ONEBIT,)]

On the C++ side, the function takes two arguments: the image, and a pointer to a floating point buffer. Note that the return type is void.

template<class T>
void nholes(T &m, feature_t* buf) {
  int vert, horiz;

  vert = nholes_1d(m.col_begin(), m.col_end());
  horiz = nholes_1d(m.row_begin(), m.row_end());

  *(buf++) = (feature_t)vert / m.ncols();
  *buf = (feature_t)horiz / m.nrows();
}

(The C++ function nholes_1d is where all the real work gets done, and is not important for this illustration.) Note how the result of the function is copied directly into the buffer.

Note

It is extrememly important not to write more values to the buffer than is defined in the metadata return_value. Doing so could cause Python/Gamera to behave erratically or segfault.

It is also possible to write a feature generator in pure Python.

class nholes(PluginFunction):
    self_type = ImageType([ONEBIT])
    return_type = FloatVector(length=2)
    feature_function = True
    doc_examples = [(ONEBIT,)]

    def __call__(self, index):
        buffer = self.features

        # Do some processing to get values...

        buffer[index] = result1
        buffer[index + 1] = result2
    __call__ = staticmethod(__call__)

Further reading

References

[Stroustrup1997]Stroustrup, B. 1997. The C++ Programming Language: Third Edition. Reading, MA: Addison-Wesley.