IceProd Server Details

The server handles many different tasks and has several independent modules to take care of them. It uses asynchronous programming to avoid blocking on I/O. The server daemon handles startup of each module. It can also reload, restart, stop, or kill the modules.

IceProd can be reloaded to update a configuration variable. This will only reload the affected parts of IceProd, while letting the rest continue running.

Configuration

The configuration is in Detailed Configuration, which is stored as a dictionary and dumped to a json file on every modification.

Queueing

This is where IceProd actually submits tasks to a grid. Various grid architectures are supported through plugins:

• HTCondor (primary)

• PBS / Torque

• Cream

The queueing module can submit to multiple grids at once since multiple plugins can be active at the same time.

• From Submission to Queue
  • Dataset Submission
  • Buffering
    • Algorithm
  • Queueing
• Task Relationships
  • Submission
  • Within task_rel
  • Within task
• Lifecycles
  • Datasets
  • Jobs
  • Tasks
• Global Queueing
  • Protocol

Database

IceProd stores most of its information in a database. By default, it uses SQLite and a few local database files. For a master site that must handle the information of a large number of sites, MySQL can be used as a database (though this will take some external setup).

The database structure is given in dbtables.

Website

The website is the main way to communicate with IceProd, both from jobs and as a user.

• Asynchronous Programming
  • Some Background
    • Brief Summary
    • Notes
  • Practical Usage
    • Callbacks
    • Callback Details
  • Futures
  • AsyncIO
• Website
  • Task Communication
  • Site-to-Site Communication
  • Human Interaction
  • Nginx
  • Security
    • CSRF
• REST Authorization
  • Authorization Implementation
• User Accounts
  • Roles
  • Groups

Proxying

Proxying is taken care of by Squid Cache, if configured. IceProd can start its own squid or use an external caching server.

Other Utilities

OpenSSL

OpenSSL can be used to make a local CA certificate, make a regular certificate signed by a CA certificate, or verify a certificate. This is mostly used by sites to create self-signed certificates.

Scheduler

The scheduler can be used like cron, to run assigned tasks at specific intervals. It will mostly be used to update graphs and run other timed interactions.