.. _python_radcube: ================= Python in Radcube ================= There are several tools, modules, segments, etc, that are written in python for the radcube project. This is an overview of what is there. The docmumentation here is useful, but the files also have extensive docstrings and can be found here: :py:class:`icecube.radcube`. The python-based classes for this project can be found at :ref:`baseClassesPythonRadcube`. The tools for using machine learning within radcube can be found at: :ref:`RadcubeMachineLearning`. Pure python modules can be found at :ref`RadcubePythonModules`. ------------- Python Tools ------------- There are two tools to help do very common functions when analyzing radio data. The first converts all types of I3AntennaWaveforms (time series and spectra) into/from numpy arrays, ``PythonListToRadTrace`` and ``RadTraceToPythonList``. There is another function, ``GetChunkSNR``, which is used to calculate SNR using the "chunk method". In this algorithm, the RMS amplitude of a time series is calculated in N non-overlapping windows and the SNR is calculated as the maximum-absolute amplitude in the whole waveform divided by the median RMS of the chunks, all squared. See also the docstrings at :py:class:`icecube.radcube.python_tools`. .. _baseClassesPythonRadcube: ------------- Basic Classes ------------- .. _SpikeFilter: ^^^^^^^^^^^^ Spike Filter ^^^^^^^^^^^^ Applies a filter to remove frequency-domain RFI spikes. This class requires a properly formatted file with the weight for each frequency bin. Spike filters are applied per-antenna-channel to account for the different noise observed at different locations. For more details on the theory of the method, see Sec 3 of `this ICRC proceeding `_. See also the docstrings at: :py:class:`icecube.radcube.SpikeFilter`. .. _StarInterpolator: ^^^^^^^^^^^^^^^^^ Star Interpolator ^^^^^^^^^^^^^^^^^ This class interpolates the simulated electric fields using CoREAS output. A detailed explaination of the algorithm is given in :ref:`star_interpolation`. See also the "`radcube paper `_". See also the docstrings at: :py:class:`icecube.radcube.StarInterpolator`. .. _TAXIArtifactRemover: ^^^^^^^^^^^^^^^^^^^^^ TAXI Artifact Remover ^^^^^^^^^^^^^^^^^^^^^ General tool for removing the chip/bin artifacts from the TAXI waveforms. All functions are expected to be given numpy arrays corresponding to the amplitudes of the waveform bins. It is assumed that all amplitudes being given are in units of ADC bins. In general the functions require the waveforms to be NxM in shape where N is the number of copies of that waveform, depending on the cascading mode being used and M is the length of the waveform. To ensure that this class is properly used, use the ``GetCascadeReshaped``, apply any corrections, and then use ``FlattenWaveforms``. There are three main corrections that are applied: 1. ``ReplaceMissingBins`` - Fixes the missing bins and/or corrupted bins during readout 2. ``RemoveChipOffsets`` - Removes the offsets of individual chips based on the median amplitude of each chip 3. ``BinSpikeRemover`` - Identifies individual bins that are broken (e.g. by bit flips) and corrects them depending on how many copies of the waveform are recorded (AKA the "cascading mode") See also the docstrings at: :py:class:`icecube.radcube.TAXIArtifactRemover`. .. _TAXIBackgroundReader: ^^^^^^^^^^^^^^^^^^^^^^ TAXI Background Reader ^^^^^^^^^^^^^^^^^^^^^^ Singleton object that can be pinged for background waveforms from measured radio waveforms. After supplying an/some I3File(s), the "soft triggered" waveforms from the files are read out and saved until either all the waveforms in the passed in frames have been read or until the specified number of waveforms (`nTraces`) has been read in. Since these waveforms are read in in units of ADC, several operations can be done to them such that they will be in a sufficient state to add to, e.g. simulated waveforms. These include converting them into a voltage (convertToVoltage) and removing TAXI artifacts. Internally, the waveforms are stored in a python dict, indexed by antenna and channel key. The radio data can then be index by requesting a spectrum or a time series. Alternatively, the ``FFTDataContainer`` can simply be requested to get both. For an example, see ``$I3_BUILD/radcube/resources/examples/ExampleScript_AddingMeasuredNoise.py`` See also the docstrings at: :py:class:`icecube.radcube.TAXIBackgroundReader`. .. _RadcubeMachineLearning: ---------------------- Machine Learning Tools ---------------------- Machine learning is used in radcube to waveform indification and cleaning. Classes have been included to handle the reading in of networks and their application to the IceCube data structures. Please note that these will only be available if ``TensorFlow`` is installed on the machine that you are running. Additionally, the speed can largely vary depending on if GPUs are available or not, but the identification of this is left to the user. All classes can be accessed through :py:class:`icecube.radcube.ml_tools`. ^^^^^^^^^^^^^^^^^^^^^^^ SingleChannelClassifier ^^^^^^^^^^^^^^^^^^^^^^^ Loads in trained model and has functions to return a classification score for a passed in waveform. The user is required to perform any preprocessing of the waveforms (e.g. upsampling, bandpassing) outside of a normalization. The models are trained on 4000 bins long waveform, so the input waveform sould also be 4000 bins long. The user is expected to supply a .h5 file with the trained network for use. See docs ^^^^^^^^^^^^^^^^^^^^^^^ TimeSeriesDenoiser ^^^^^^^^^^^^^^^^^^^^^^^ Applies a trained machine learning algorithm to the passed in waveforms in the frame and outputs a "denoised" version of that waveform. The models are trained on 4000 bins long waveform, so the input waveform sould also be 4000 bins long. The user is expected to supply a .h5 file with the trained network for use. .. _RadcubePythonModules: -------------- Python Modules -------------- There are several classes that are pure python modules. These are almost exclusively wrappers for the classes specified above and simply run the relevant functions on waveforms in the frames. ^^^^^^^^^^^^^^^^ ApplySpikeFilter ^^^^^^^^^^^^^^^^ Runs the :ref:`SpikeFilter` class on obejects in the frame. ``ApplySpikeFilter`` includes the following parameters: - **InputName**: Name of the ``I3AntennaDataMap`` in the frame to apply the RFI filtering to - **OutputName**: Name of the ``I3AntennaDataMap`` to output to the frame after filtering - **FilterFile**: Name of the file containing the filter weights. - **SpikePower**: Weights are applied in the frequency domain :math:`A_{i}' = A_i * w_i^{\rm SpikePower}` - **ApplyInDAQ**: If true, module will run in the Q Frame, otherwise on the P Frame ^^^^^^^^^^^^^^^^^^ MeasuredNoiseAdder ^^^^^^^^^^^^^^^^^^ Uses background waveforms measured in the real antennas to add to simulated radio waveforms to generate realistic simulations. This module is essentially just an ``I3Module`` that runs the :ref:`TAXIBackgroundReader` and adds these waveforms in the time domain. This class also can run the :py:class:`icecube.radcube.SpikeFilter` on the read in data as well. This algorithm considers two cases: 1. You have noise from antennas that you want to add to simulations of those exact same antennas. i.e. you are simulating existing antennas 2. You have noise from a set of antennas that are not in your simulation, e.g. you are simulating a possible layout for an array on the moon Set ``MatchExactNoise`` to ``True`` for case 1 and ``False`` for case 2. In case 1, the measured noise will only be added to antennas with matching keys. In case 2, a background waveform will be added and an object will be added to the frame which know the linkage between the AntennaKey for the measured/simulated antennas ``MeasuredNoiseAdder`` includes the following parameters: - **InputName**: Name of the input ``I3AntennaDataMap`` in the frame - **OutputName**: Name of the output ``I3AntennaDataMap`` to output to put in the frame - **TaxiFile**: List of the I3File(s) with the background TAXI waveforms to be read in and added later - **NTimeBins**: The number of time bins that are needed from each waveform. - **NTraces**: The number of waveforms to be read in from the I3File(s). If less are available, it will read in as many as it can. - **ConvertToVoltage**: If ``False``, will keep waveforms in ADC, otherwise they will be converted to voltage. - **InsertNoiseOnly**: If ``True``, the pure noise will also be added to the frame. - **Overuse**: If ``True``, individual background waveforms will be used more than once. Otherwise, the tray will be terminated when the background waveforms are exhausted. - **FrequencyBand**: If set with a list of low/high frequencies. All waveforms will first be filtered on that band. - **RemoveBinSpikes**: If ``True`` the :ref:`TAXIArtifactRemover` class will be run on the waveforms before they are internally stored (since some artifacts may not be correctable after adding to simulated pulses). - **SpikeFilter**: If given an instance of a :ref:`SpikeFilter` class, the spike filter will be applied to the background waveforms before being stored internally. ^^^^^^^^^^^^^^^^ RadcubePlotter ^^^^^^^^^^^^^^^^ Module which can be used to quickly makes plots of a single antenna/E-Field. The user gives a list of lists where each entry includes the following five things: 1. Name of the object in the frame to plot, e.g., "FilteredWaveform" 2. The fraction of the bins to plot. Plots bins 0 to specified value 3. Descriptive label to put on the plots for this entry, e.g., "After Bandpass Filter" 4. True if quantity is in ADC units, False if not 5. Method to plot the frequency spectrum "lin" for linear y-axis, "dB" to put it in dBm/Hz, "raw" for electric field ``RadcubePlotter`` includes the following parameters: - **DataToPlot**: List which includes the points listed above for each plot to be made - **StationID**: Id number of the station to plot - **AntennaID**: Id number of the antenna to plot - **OutputDir**: Directory where the output plots will be saved - **ZoomToPulse**: If non-zero, will zoom in on the pulse in the waveforms using the specified width ^^^^^^^^^^^^^^^^^^^ RemoveTAXIArtifacts ^^^^^^^^^^^^^^^^^^^ This class is a wrapper to run the :ref:`TAXIArtifactRemover` on the measured digitized waveforms in the frame. See the documentation for that class for more information on the underlying algorithms Note that this code is only meant to be run on the raw ADC amplitudes. i.e. do not run it on waveforms that have already been filtered or otherwise modified. ``RemoveTAXIArtifacts`` includes the following parameters: - **InputName**: Name of the input ``I3AntennaDataMap`` in the frame - **OutputName**: Name of the output ``I3AntennaDataMap`` to output to put in the frame - **BaselineValue**: Value to set the baselines to in ADC bins - **MedianOverCascades**: Apply the median baselines over non-cascaded/semi-cascaded chunks - **RemoveBinSpikes**: Remove the single bin artifacts using an algorithm depending on the cascading mode - **BinSpikeDeviance**: Parameter which defines what constitutes a bin spike - **RemoveNegativeBins**: Removes the -1 bins (corrupted data) and replace it with average ^^^^^^^^^^^^^^^^^^^ SimulateFromStar ^^^^^^^^^^^^^^^^^^^ Simulates shower on a specified array using simulations on a star shaped pattern. This class does three things: 1. Loads up a star pattern simulation into an instance of the :py:class:`icecube.radcube.StarInterpolator` class 2. Puts the other meta-data into the frame (e.g. I3Particle) 3. Can randomize the core location or match the core location to an existing particle in the frame The intended use cases for this module are: - Make a radio-only simulation, creating all frames Q frames, as needed - Add radio data to an existing simulation in the QFrame In the first case set, ``PiggybackMode = False`` and choose a randomization scheme for the cores. You can either use the ``RandomCoreCenter`` and a ``CoreRadius`` about that center, sampled uniformly on the disk. The number of times you want to choose a random core is set by ``NThrows``. This needs it's own RNG and due to limitations in IceTray, C++ implemented services cannot be used by python-implemented modules. So this class has it's own RNG instance, provide it a ``RNGSeed``. Or, you can supply a list of I3Positions though which the axis will go. In the second case, set ``PiggybackModel = True`` and set the ``PiggybackPrimary`` to the name of the I3Particle in the frame that will be used as the core location, directly. ``SimulateFromStar`` includes the following parameters: - **DirectoryList**: List of paths to CORSIKA/CoREAS sims - **NThrows**: How many random cores per shower - **CoreRadius**: The radius from ``RandomCoreCenter`` within which the cores will be placed - **RandomCoreCenter**: The core will be randomized about this point - **RNGSeed**: Seed for the randomization of the core location - **PiggybackMode**: Set to ``True`` if another module is also reading the CORSIKA file (will use the same core as the other module's primary) - **PiggybackPrimary**: The particle in the frame that will set the seed for the particle