Python in Radcube

There are several tools, modules, segments, etc, that are written in python for the radcube project. This is an overview of what is there. The docmumentation here is useful, but the files also have extensive docstrings and can be found here: icecube.radcube.

The python-based classes for this project can be found at Basic Classes. The tools for using machine learning within radcube can be found at: Machine Learning Tools. Pure python modules can be found at :ref`RadcubePythonModules`.

Python Tools

There are two tools to help do very common functions when analyzing radio data. The first converts all types of I3AntennaWaveforms (time series and spectra) into/from numpy arrays, PythonListToRadTrace and RadTraceToPythonList.

There is another function, GetChunkSNR, which is used to calculate SNR using the “chunk method”. In this algorithm, the RMS amplitude of a time series is calculated in N non-overlapping windows and the SNR is calculated as the maximum-absolute amplitude in the whole waveform divided by the median RMS of the chunks, all squared.

See also the docstrings at icecube.radcube.python_tools.

Basic Classes

Spike Filter

Applies a filter to remove frequency-domain RFI spikes. This class requires a properly formatted file with the weight for each frequency bin. Spike filters are applied per-antenna-channel to account for the different noise observed at different locations.

For more details on the theory of the method, see Sec 3 of this ICRC proceeding.

See also the docstrings at: icecube.radcube.SpikeFilter.

Star Interpolator

This class interpolates the simulated electric fields using CoREAS output. A detailed explaination of the algorithm is given in Star Interpolation Tool. See also the “radcube paper”.

See also the docstrings at: icecube.radcube.StarInterpolator.

TAXI Artifact Remover

General tool for removing the chip/bin artifacts from the TAXI waveforms. All functions are expected to be given numpy arrays corresponding to the amplitudes of the waveform bins. It is assumed that all amplitudes being given are in units of ADC bins. In general the functions require the waveforms to be NxM in shape where N is the number of copies of that waveform, depending on the cascading mode being used and M is the length of the waveform. To ensure that this class is properly used, use the GetCascadeReshaped, apply any corrections, and then use FlattenWaveforms.

There are three main corrections that are applied:

  1. ReplaceMissingBins - Fixes the missing bins and/or corrupted bins during readout

  2. RemoveChipOffsets - Removes the offsets of individual chips based on the median amplitude of each chip

  3. BinSpikeRemover - Identifies individual bins that are broken (e.g. by bit flips) and corrects them depending on how many copies of the waveform are recorded (AKA the “cascading mode”)

See also the docstrings at: icecube.radcube.TAXIArtifactRemover.

TAXI Background Reader

Singleton object that can be pinged for background waveforms from measured radio waveforms. After supplying an/some I3File(s), the “soft triggered” waveforms from the files are read out and saved until either all the waveforms in the passed in frames have been read or until the specified number of waveforms (nTraces) has been read in.

Since these waveforms are read in in units of ADC, several operations can be done to them such that they will be in a sufficient state to add to, e.g. simulated waveforms. These include converting them into a voltage (convertToVoltage) and removing TAXI artifacts.

Internally, the waveforms are stored in a python dict, indexed by antenna and channel key. The radio data can then be index by requesting a spectrum or a time series. Alternatively, the FFTDataContainer can simply be requested to get both.

For an example, see $I3_BUILD/radcube/resources/examples/ExampleScript_AddingMeasuredNoise.py

See also the docstrings at: icecube.radcube.TAXIBackgroundReader.

Machine Learning Tools

Machine learning is used in radcube to waveform indification and cleaning. Classes have been included to handle the reading in of networks and their application to the IceCube data structures.

Please note that these will only be available if TensorFlow is installed on the machine that you are running. Additionally, the speed can largely vary depending on if GPUs are available or not, but the identification of this is left to the user.

All classes can be accessed through icecube.radcube.ml_tools.

SingleChannelClassifier

Loads in trained model and has functions to return a classification score for a passed in waveform. The user is required to perform any preprocessing of the waveforms (e.g. upsampling, bandpassing) outside of a normalization. The models are trained on 4000 bins long waveform, so the input waveform sould also be 4000 bins long.

The user is expected to supply a .h5 file with the trained network for use.

See docs

TimeSeriesDenoiser

Applies a trained machine learning algorithm to the passed in waveforms in the frame and outputs a “denoised” version of that waveform. The models are trained on 4000 bins long waveform, so the input waveform sould also be 4000 bins long.

The user is expected to supply a .h5 file with the trained network for use.

Python Modules

There are several classes that are pure python modules. These are almost exclusively wrappers for the classes specified above and simply run the relevant functions on waveforms in the frames.

ApplySpikeFilter

Runs the Spike Filter class on obejects in the frame.

ApplySpikeFilter includes the following parameters:

  • InputName: Name of the I3AntennaDataMap in the frame to apply the RFI filtering to

  • OutputName: Name of the I3AntennaDataMap to output to the frame after filtering

  • FilterFile: Name of the file containing the filter weights.

  • SpikePower: Weights are applied in the frequency domain \(A_{i}' = A_i * w_i^{\rm SpikePower}\)

  • ApplyInDAQ: If true, module will run in the Q Frame, otherwise on the P Frame

MeasuredNoiseAdder

Uses background waveforms measured in the real antennas to add to simulated radio waveforms to generate realistic simulations. This module is essentially just an I3Module that runs the TAXI Background Reader and adds these waveforms in the time domain.

This class also can run the icecube.radcube.SpikeFilter on the read in data as well.

This algorithm considers two cases:

  1. You have noise from antennas that you want to add to simulations of those exact same antennas. i.e. you are simulating existing antennas

  2. You have noise from a set of antennas that are not in your simulation, e.g. you are simulating a possible layout for an array on the moon

Set MatchExactNoise to True for case 1 and False for case 2. In case 1, the measured noise will only be added to antennas with matching keys. In case 2, a background waveform will be added and an object will be added to the frame which know the linkage between the AntennaKey for the measured/simulated antennas

MeasuredNoiseAdder includes the following parameters:

  • InputName: Name of the input I3AntennaDataMap in the frame

  • OutputName: Name of the output I3AntennaDataMap to output to put in the frame

  • TaxiFile: List of the I3File(s) with the background TAXI waveforms to be read in and added later

  • NTimeBins: The number of time bins that are needed from each waveform.

  • NTraces: The number of waveforms to be read in from the I3File(s). If less are available, it will read in as many as it can.

  • ConvertToVoltage: If False, will keep waveforms in ADC, otherwise they will be converted to voltage.

  • InsertNoiseOnly: If True, the pure noise will also be added to the frame.

  • Overuse: If True, individual background waveforms will be used more than once. Otherwise, the tray will be terminated when the background waveforms are exhausted.

  • FrequencyBand: If set with a list of low/high frequencies. All waveforms will first be filtered on that band.

  • RemoveBinSpikes: If True the TAXI Artifact Remover class will be run on the waveforms before they are internally stored (since some artifacts may not be correctable after adding to simulated pulses).

  • SpikeFilter: If given an instance of a Spike Filter class, the spike filter will be applied to the background waveforms before being stored internally.

RadcubePlotter

Module which can be used to quickly makes plots of a single antenna/E-Field. The user gives a list of lists where each entry includes the following five things:

  1. Name of the object in the frame to plot, e.g., “FilteredWaveform”

  2. The fraction of the bins to plot. Plots bins 0 to specified value

  3. Descriptive label to put on the plots for this entry, e.g., “After Bandpass Filter”

  4. True if quantity is in ADC units, False if not

  5. Method to plot the frequency spectrum “lin” for linear y-axis, “dB” to put it in dBm/Hz, “raw” for electric field

RadcubePlotter includes the following parameters:

  • DataToPlot: List which includes the points listed above for each plot to be made

  • StationID: Id number of the station to plot

  • AntennaID: Id number of the antenna to plot

  • OutputDir: Directory where the output plots will be saved

  • ZoomToPulse: If non-zero, will zoom in on the pulse in the waveforms using the specified width

RemoveTAXIArtifacts

This class is a wrapper to run the TAXI Artifact Remover on the measured digitized waveforms in the frame. See the documentation for that class for more information on the underlying algorithms

Note that this code is only meant to be run on the raw ADC amplitudes. i.e. do not run it on waveforms that have already been filtered or otherwise modified.

RemoveTAXIArtifacts includes the following parameters:

  • InputName: Name of the input I3AntennaDataMap in the frame

  • OutputName: Name of the output I3AntennaDataMap to output to put in the frame

  • BaselineValue: Value to set the baselines to in ADC bins

  • MedianOverCascades: Apply the median baselines over non-cascaded/semi-cascaded chunks

  • RemoveBinSpikes: Remove the single bin artifacts using an algorithm depending on the cascading mode

  • BinSpikeDeviance: Parameter which defines what constitutes a bin spike

  • RemoveNegativeBins: Removes the -1 bins (corrupted data) and replace it with average

SimulateFromStar

Simulates shower on a specified array using simulations on a star shaped pattern. This class does three things:

  1. Loads up a star pattern simulation into an instance of the icecube.radcube.StarInterpolator class

  2. Puts the other meta-data into the frame (e.g. I3Particle)

  3. Can randomize the core location or match the core location to an existing particle in the frame

The intended use cases for this module are:

  • Make a radio-only simulation, creating all frames Q frames, as needed

  • Add radio data to an existing simulation in the QFrame

In the first case set, PiggybackMode = False and choose a randomization scheme for the cores. You can either use the RandomCoreCenter and a CoreRadius about that center, sampled uniformly on the disk. The number of times you want to choose a random core is set by NThrows. This needs it’s own RNG and due to limitations in IceTray, C++ implemented services cannot be used by python-implemented modules. So this class has it’s own RNG instance, provide it a RNGSeed.

Or, you can supply a list of I3Positions though which the axis will go.

In the second case, set PiggybackModel = True and set the PiggybackPrimary to the name of the I3Particle in the frame that will be used as the core location, directly.

SimulateFromStar includes the following parameters:

  • DirectoryList: List of paths to CORSIKA/CoREAS sims

  • NThrows: How many random cores per shower

  • CoreRadius: The radius from RandomCoreCenter within which the cores will be placed

  • RandomCoreCenter: The core will be randomized about this point

  • RNGSeed: Seed for the randomization of the core location

  • PiggybackMode: Set to True if another module is also reading the CORSIKA file (will use the same core as the other module’s primary)

  • PiggybackPrimary: The particle in the frame that will set the seed for the particle