DataStore

class gammapy.data.DataStore(hdu_table=None, obs_table=None, name=None)[source]

Bases: object

IACT data store.

The data selection and access happens using an observation and an HDU index file as described at IACT data storage.

See Data Store or data_iact.html for usage examples.

Parameters:

hdu_table : HDUIndexTable

HDU index table

obs_table : ObservationTable

Observation index table

name : str

Data store name

Examples

Here’s an example how to create a DataStore to access H.E.S.S. data:

>>> from gammapy.data import DataStore
>>> dir = '$GAMMAPY_EXTRA/datasets/hess-crab4-hd-hap-prod2'
>>> data_store = DataStore.from_dir(dir)
>>> data_store.info()

Attributes Summary

DEFAULT_HDU_TABLE Default HDU table filename.
DEFAULT_NAME Default data store name.
DEFAULT_OBS_TABLE Default observation table filename.

Methods Summary

check_available_event_lists([logger]) Check if all event lists are available.
check_integrity([logger]) Check integrity, i.e.
check_observations() Perform some sanity checks for all observations.
copy_obs(obs_id, outdir[, hdu_class, …]) Create a new DataStore containing a subset of observations.
data_summary([obs_id, summed]) Create a summary Table with HDU size information.
from_all(val) Try different DataStore constructors.
from_config(config) Create from a config dict.
from_dir(base_dir[, name]) Create from a directory.
from_files(base_dir[, hdu_table_filename, …]) Construct from HDU and observation index table files.
from_name(name) Convenience method to look up DataStore from DataManager.
info([file]) Print some info.
load_all([hdu_type, hdu_class]) Load a given file type for all observations.
load_many(obs_ids[, hdu_type, hdu_class]) Load a given file type for certain observations in an observation table.
make_table_of_files([observation_table, …]) Make list of files in the datastore directory.
obs(obs_id) Access a given DataStoreObservation.
obs_list(obs_id[, skip_missing]) Generate a ObservationList.

Attributes Documentation

DEFAULT_HDU_TABLE = 'hdu-index.fits.gz'

Default HDU table filename.

DEFAULT_NAME = 'noname'

Default data store name.

DEFAULT_OBS_TABLE = 'obs-index.fits.gz'

Default observation table filename.

Methods Documentation

check_available_event_lists(logger=None)[source]

Check if all event lists are available.

TODO: extend this function, or combine e.g. with make_table_of_files.

Returns:

file_available : ndarray

Boolean mask which files are available.

check_integrity(logger=None)[source]

Check integrity, i.e. whether index and observation table match.

check_observations()[source]

Perform some sanity checks for all observations.

Returns:

results : OrderedDict

dictionary containing failure messages for all runs that fail a check.

copy_obs(obs_id, outdir, hdu_class=None, verbose=False, overwrite=False)[source]

Create a new DataStore containing a subset of observations.

Parameters:

obs_id : array-like, ObservationTable

List of observations to copy

outdir : str, Path

Directory for the new store

hdu_class : list of str

verbose : bool

Print copied files

overwrite : bool

Overwrite

data_summary(obs_id=None, summed=False)[source]

Create a summary Table with HDU size information.

Parameters:

obs_id : array-like

Observation IDs to include in the summary

summed : bool

Sum up file size?

classmethod from_all(val)[source]

Try different DataStore constructors.

Currently tried (in this order) - from_dir() - from_name()

Parameters:

val : str

Key to construct DataStore from

classmethod from_config(config)[source]

Create from a config dict.

classmethod from_dir(base_dir, name=None)[source]

Create from a directory.

This assumes that the HDU and observations index tables have the default filename.

classmethod from_files(base_dir, hdu_table_filename=None, obs_table_filename=None, name=None)[source]

Construct from HDU and observation index table files.

classmethod from_name(name)[source]

Convenience method to look up DataStore from DataManager.

info(file=None)[source]

Print some info.

load_all(hdu_type=None, hdu_class=None)[source]

Load a given file type for all observations.

Parameters:

hdu_type : str

HDU type (see VALID_HDU_TYPE)

hdu_class : str

HDU class (see VALID_HDU_CLASS)

Returns:

list : python list of object

Object depends on type, e.g. for events it is a list of EventList.

load_many(obs_ids, hdu_type=None, hdu_class=None)[source]

Load a given file type for certain observations in an observation table.

Parameters:

obs_ids : list

List of observation IDs

hdu_type : str

HDU type (see VALID_HDU_TYPE)

hdu_class : str

HDU class (see VALID_HDU_CLASS)

Returns:

list : list of object

Object depends on type, e.g. for events it is a list of EventList.

make_table_of_files(observation_table=None, filetypes=['events'])[source]

Make list of files in the datastore directory.

Parameters:

observation_table : ObservationTable or None

Observation table (None means select all observations).

filetypes : list of str

File types (TODO: document in a central location and reference from here).

Returns:

table : Table

Table summarising info about files.

obs(obs_id)[source]

Access a given DataStoreObservation.

Parameters:

obs_id : int

Observation ID.

Returns:

obs : DataStoreObservation

Observation container

obs_list(obs_id, skip_missing=False)[source]

Generate a ObservationList.

Parameters:

obs_id : list

Observation IDs.

skip_missing : bool, optional

Skip missing observations, default: False

Returns:

obs : ObservationList