Grid Queueing

Interface for configuring and submitting jobs on a computing cluster. Do not use this class directly. Instead use one of the implementations that inherit from this class.

iceprod.server.grid.get_host()

Cache the host fqdn for 1 hour

class iceprod.server.grid.BaseGrid(gridspec, queue_cfg, cfg, modules, executor, statsd, rest_client, cred_client)

Interface for a generic job distribution system. Do not use this class directly. Use one of the plugins.

GRID_STATES = ('queued', 'processing', 'completed', 'error', 'unknown')

async check_and_clean()

Check and clean the grid

async queue()

Queue tasks to the grid

get_queue_num(available=100000)

Determine how many pilots to queue.

async setup_pilots(tasks)

Setup pilots for the task reqs

get_token()

async get_user_credentials(username)

async get_group_credentials(group)

async customize_task_config(task_cfg, job_cfg=None, dataset=None)

Transforms for the task config

async setup_submit_directory(task)

Set up submit directory

create_config(task)

write_cfg(task)

Write the config file for a task-like object

get_submit_args(task, cfg=None, passkey=None)

Get the submit arguments to start the loader script.

async get_grid_status()

Get all tasks running on the queue system. Returns {grid_queue_id:{status,submit_dir}}

async generate_submit_file(task, cfg=None, passkey=None, filelist=None)

Generate queueing system submit file for task in dir.

async submit(task)

Submit task to queueing system.

async remove(tasks)

Remove tasks from queueing system.