Making a new table writer service

There are writer services that produce HDF5 tables, ROOT files, and plain text (CSV). To make a new writer service for your pet analysis format, you must implement subclasses of I3Table and I3TableService.

I3Table

type I3Table

represents a table on disk

void I3Table::WriteRows(I3TableRowConstPtr rows)

Write the data in rows to disk

void I3TableRow::Flush(size_t nrows)

Flush nrows to disk.

Your subclass must implement WriteRows(). If it implements any kind of caching, you should also implement Flush(). Here, for example, is an implementation that writes comma-separated text (CSV).

WriteRows()

void I3CSVTable::WriteRows(I3TableRowConstPtr row) {
    unsigned int i,j;
    std::vector<NativeType>::iterator t_it;
    I3TableRowPtr rows = boost::const_pointer_cast<I3TableRow>(row);

    const std::vector<size_t>& arrayLengths = description_->GetFieldArrayLengths();
    std::vector<size_t>::const_iterator size_it;
    rows->SetEnumsAreInts(true);
    for (i = 0; i < rows->GetNumberOfRows(); i++) {
        rows->SetCurrentRow(i);
        for (t_it = fieldTypes_.begin(), size_it = arrayLengths.begin(), j = 0;
             t_it != fieldTypes_.end();
             t_it++, size_it++, j++) {
            switch (*t_it) {
                case NATIVE_SCHAR:
                    WriteField(rows->GetPointer<signed char>(j),*size_it);
                    break;
                case NATIVE_UCHAR:
                    WriteField(rows->GetPointer<unsigned char>(j),*size_it);
                    break;
                case NATIVE_SHORT:
                    WriteField(rows->GetPointer<short>(j),*size_it);
                    break;
                case NATIVE_USHORT:
                    WriteField(rows->GetPointer<unsigned short>(j),*size_it);
                    break;
                case NATIVE_INT:
                    WriteField(rows->GetPointer<int>(j),*size_it);
                    break;
                case NATIVE_UINT:
                    WriteField(rows->GetPointer<unsigned int>(j),*size_it);
                    break;
                case NATIVE_LONG:
                    WriteField(rows->GetPointer<long>(j),*size_it);
                    break;
                case NATIVE_ULONG:
                    WriteField(rows->GetPointer<unsigned long>(j),*size_it);
                    break;
                case NATIVE_LLONG:
                    WriteField(rows->GetPointer<long long>(j),*size_it);
                    break;
                case NATIVE_ULLONG:
                    WriteField(rows->GetPointer<unsigned long long>(j),*size_it);
                    break;
                case NATIVE_FLOAT:
                    WriteField(rows->GetPointer<float>(j),*size_it);
                    break;
                case NATIVE_DOUBLE:
                    WriteField(rows->GetPointer<double>(j),*size_it);
                    break;
                case NATIVE_BOOL:
                    WriteField(rows->GetPointer<bool>(j),*size_it);
                    break;
                default:
                    log_warn("Don't know how to write type '%s'",description_->GetFieldTypes().at(j).description.c_str());
            }
            if (t_it == fieldTypes_.end()-1) {
                output_ << std::endl;
            } else {
                output_ << ",";
            }
        }
    }
    rows->SetEnumsAreInts(false);
}

For each field, a pointer to the j th field is cast to the appropriate native type and passed to WriteField(), which is just a templated function that dumps the argument to iostream output_:

template <typename T>
void WriteField(T* value,size_t arraySize) {
    for (size_t i = 0; i < arraySize; i++) {
        output_ << value[i];
        if ((arraySize > 1) && (i != arraySize-1)) output_ << ",";
    }
}

Type-switching

Since you don’t have compile-time information about the types of the fields, some type-switching is unavoidable. I3CSVTable resolves and caches an equivalent native type for each field:

I3CSVTable::NativeType I3CSVTable::GetNativeType(const I3Datatype& dtype) {
    I3CSVTable::NativeType type = UNKNOWN;
    switch (dtype.kind) {
        case I3Datatype::Int:
            if (dtype.size == sizeof(char)) {
                if (dtype.is_signed) type = NATIVE_SCHAR;
                else                 type = NATIVE_UCHAR;
            } else if (dtype.size == sizeof(short)) {
                if (dtype.is_signed) type = NATIVE_SHORT;
                else                 type = NATIVE_USHORT;
            } else if (dtype.size == sizeof(int)) {
                if (dtype.is_signed) type = NATIVE_INT;
                else                 type = NATIVE_UINT;
            } else if (dtype.size == sizeof(long)) {
                if (dtype.is_signed) type = NATIVE_LONG;
                else                 type = NATIVE_ULONG;
            } else if (dtype.size == sizeof(long long)) {
                if (dtype.is_signed) type = NATIVE_LLONG;
                else                 type = NATIVE_ULLONG;
            }
            break;
        case I3Datatype::Float:
            if (dtype.size == sizeof(float)) {
                type = NATIVE_FLOAT;
            } else if (dtype.size == sizeof(double)) {
                type = NATIVE_DOUBLE;
            }
            break;
        case I3Datatype::Bool:
            type = NATIVE_BOOL;
            break;
        case I3Datatype::Enum:
            type = NATIVE_INT;
            break;
    }
    return type;
}

Constructor

You can set up any header information in your constructor. In I3CSVTable, CreateTable() is called from the constructor:

void I3CSVTable::CreateTable() {
    std::vector<std::string>::const_iterator name_it,unit_it,doc_it;
    std::vector<size_t>::const_iterator size_it;
    std::vector<I3Datatype>::const_iterator dtype_it;
    const std::vector<std::string>& names = description_->GetFieldNames();
    const std::vector<std::string>& units = description_->GetFieldUnits();
    const std::vector<std::string>& doc = description_->GetFieldDocStrings();
    const std::vector<size_t>& arrayLengths = description_->GetFieldArrayLengths();
    const std::vector<I3Datatype>& dtypes = description_->GetFieldTypes();

    for (name_it = names.begin(), unit_it = units.begin(), size_it = arrayLengths.begin();
         name_it != names.end();
         name_it++, unit_it++, size_it++) {
        output_ << "\"" << *name_it;
        if (unit_it->size() > 0) output_ << " [" << *unit_it << "]";
        output_ << "\"";
        if (*size_it > 1) {
            for (size_t i = 1; i < *size_it; i++) output_ << ",";
        }
        if (name_it == names.end()-1) {
            output_ << std::endl;
        } else {
            output_ << ",";
        }
    }
    for (doc_it = doc.begin(), size_it = arrayLengths.begin(), dtype_it = dtypes.begin();
         doc_it != doc.end();
         doc_it++, size_it++, dtype_it++) {
        output_ << "\"" << *doc_it;
        // if this is an enum type, write out the enum members
        if (dtype_it->kind == I3Datatype::Enum) {
            output_ << "{ ";
            std::vector<std::pair<std::string,long> >::const_iterator member_it;
            for (member_it = dtype_it->enum_members.begin();
                 member_it != dtype_it->enum_members.end();
                 member_it++) {
                output_ << member_it->first << " = " << member_it->second;
                if (member_it != dtype_it->enum_members.end()-1) output_ << ", ";
            }
            output_ << " }";
        }
        output_ << "\"";
        // if this is an array type, add enough columns for the array size
        if (*size_it > 1) {
            for (size_t i = 1; i < *size_it; i++) output_ << ",";
        }
        // terminate or continue the line
        if (doc_it == doc.end()-1) {
            output_ << std::endl;
        } else {
            output_ << ",";
        }
    }

    output_.precision(12);
    output_ << std::scientific;

    for (dtype_it = dtypes.begin(); dtype_it != dtypes.end(); dtype_it++) {
        fieldTypes_.push_back(GetNativeType(*dtype_it));
    }
}

I3TableService

To manage the collection of tables, you must implement a subclass of I3TableService.

type I3TableService

represents a bundle of tables on disk, possibly belonging to a single file

virtual I3TablePtr I3TableService::CreateTable(const std::string &tableName, I3TableRowDescriptionConstPtr description)

Create a new table called tableName based on description.

virtual void I3TableService::CloseFile()

Called at the end of operations.

CreateTable()

You must implement CreateTable(), which is called whenever the writer module requests a table that does not yet exist. The I3CSVTable implementation is fairly simple:

I3TablePtr I3CSVTableService::CreateTable(const std::string& tableName,
                               I3TableRowDescriptionConstPtr description) {

        I3TableRowDescriptionConstPtr index_desc = GetIndexDescription();
        std::string indexpath = folderName_ + "/__I3Index__";
        I3TablePtr index_table(new I3CSVTable(*this,tableName,index_desc,indexpath));
        return I3TablePtr(new I3CSVTable(*this,tableName,description,folderName_,index_table));
}

Two tables are created: one “data” table, and an auxiliary “index” table that contains the starting and stopping positions of individual events. This is nontrivial for compound objects like DOMLaunches that must be flattened across multiple rows. The function returns a shared pointer to the data table.

CloseFile()

CloseFile() should perform any required clean-up, including flushing any internal caches.

Constructor

You can implement any required setup in the constructor. I3CSVTable, for example, writes tables to individual files in a directory, so it re-creates that directory in its constructor:

I3CSVTableService::I3CSVTableService(const std::string& foldername) : folderName_(foldername) {
    fs::remove_all( folderName_ );
    fs::create_directory( folderName_ );
    fs::create_directory( folderName_ + "/__I3Index__" );
}

Python bindings

You must also provide trivial Python bindings for your I3TableService:

#include <icetray/load_project.h>
#include "textwriter/I3CSVTableService.h"

namespace bp = boost::python;

I3_PYTHON_MODULE(textwriter)
{
  load_project("textwriter", false);

  bp::class_<I3CSVTableService,
             boost::shared_ptr<I3CSVTableService>,
             bp::bases<I3TableService> >
             ("I3CSVTableService",
              bp::init<const std::string>(bp::args("folder_name")))
             ;
}