Serveur documentation

New in 1.8.5

Added 3 function to facilitate access to the various metadata, in a more python way.

These new functions return a python dictionnary that you can manipulate with any usual dict() methods.

metadata()

Return a python dictionnary [name, value(s)] for all metadata sections

>>> import ceflib as cef
>>> cef.read("../DATA/C1_CP_WHI_ELECTRON_DENSITY__20020105_000000_20020106_000000_V071001.cef")
>>> m = cef.metadata()

>>> m.keys()
dict_keys(['LOGICAL_FILE_ID', 'VERSION_NUMBER', 'DATASET_VERSION', 'FILE_CAVEATS', 'FILE_TIME_SPAN', 'GENERATION_DATE', 'OBSERVATORY', 'OBSERVATORY_CAVEATS', 'OBSERVATORY_DESCRIPTION', 'OBSERVATORY_TIME_SPAN', 'OBSERVATORY_REGION', 'MISSION', 'MISSION_TIME_SPAN', 'MISSION_AGENCY', 'MISSION_DESCRIPTION', 'MISSION_KEY_PERSONNEL', 'MISSION_REFERENCES', 'MISSION_REGION', 'MISSION_CAVEATS', 'EXPERIMENT', 'INVESTIGATOR_COORDINATES', 'EXPERIMENT_KEY_PERSONNEL', 'EXPERIMENT_CAVEATS', 'EXPERIMENT_REFERENCES', 'EXPERIMENT_DESCRIPTION', 'INSTRUMENT_NAME', 'INSTRUMENT_CAVEATS', 'INSTRUMENT_DESCRIPTION', 'INSTRUMENT_TYPE', 'MEASUREMENT_TYPE', 'DATASET_ID', 'DATASET_TITLE', 'DATA_TYPE', 'DATASET_DESCRIPTION', 'TIME_RESOLUTION', 'MIN_TIME_RESOLUTION', 'MAX_TIME_RESOLUTION', 'PROCESSING_LEVEL', 'CONTACT_COORDINATES', 'ACKNOWLEDGEMENT', 'FILE_TYPE', 'METADATA_TYPE'])

>>> m["DATASET_TITLE"]
'Electron Density'

>>> for line in m["MISSION_DESCRIPTION"]: print(line)
The aim of the Cluster mission is to study small-scale structures of the magnetosphere
and its environment in three dimensions. To achieve this, Cluster is constituted of four
identical spacecraft that will flight in a tetrahedral configuration. The separation distances
between the spacecraft will be varied between ~40 km and 10 000 km, according to the
key scientific regions.

Deprecated functions

The previous functions are still available:

metanames() → returning the names all the metadata sections
meta(section-name) → the value of a given metadata section

But it may be easier to manipulate a single dictionnary :

Supposing there is a variable :

m = cef.metadata():

There should be replaced with:

m.keys()
m["DATASET_TITLE"]

gattrs()

Return a python dictionnary [name, value] with all global attributes.

>>> g = cef.gattrs()

>>> g.keys()
dict_keys(['FILE_NAME', 'FILE_FORMAT_VERSION', 'END_OF_RECORD_MARKER', 'DATA_UNTIL'])

>>> g["FILE_NAME"]
'C1_CP_WHI_ELECTRON_DENSITY__20020105_000000_20020106_000000_V071001.cef'

Deprecated functions

Here also, the previous functions remain available:

gattributes() → return the list of global attribute names
gattr(name) → return the value of a given atrtibute

Could be replaced with:

g.keys()
g[name]

vattrs (varname)

Return a python dictionnary [name, value] for the attributes of a CEF variable

>>> va = cef.vattrs("Electron_density")
>>> for k,v in va.items(): print (k, ":", v)
...
SIZES : 1
VALUE_TYPE : FLOAT
ENTITY : Electron
PROPERTY : Number_Density
LABLAXIS : E-D
FIELDNAM : Electron Density
SI_CONVERSION : 1.0e6>m^-3
UNITS : cm^-3
FILLVAL : -1
ERROR_PLUS : Accuracy
ERROR_MINUS : Accuracy
CATDESC : Electron Density
DEPEND_0 : time_tags
SIGNIFICANT_DIGITS : 9
PARAMETER_TYPE : Data
QUALITY : 3

Deprecated functions

Here also, the previous functions remain available:

vattributes(variable) → return list of attribute names for a given variable
vattr(variable, attr) → return value of a given attribute

Should be replaced with:

va.keys()
va[attr]

However, the varnames() function is necessary to get the full list of the CEF variables in a file

New in 1.8.4

Previously, when accessing to CEF time variables, the values were expressed in internal timestamps.

These values were corresponding to milli-seconds since 1958-01-01T00:00:00, as a numpy array of double.

This is still the case if you are using the celib.var() function.

You will have then to convert these values to Unix timestamps or ISO 8601 strings, using ceflib.milli_to_timestamp() or ceflib.milli_to_isotime()

>>> ceflib.verbosity(0)
>>> ceflib.read("../DATA/C1_CP_WHI_ELECTRON_DENSITY__20020105_000000_20020106_000000_V071001.cef")
>>> t = ceflib.var("time_tags")
>>> t[0]
1388880021529.0
>>> ceflib.milli_to_timestamp (t[0])
1010188821.529
>>> ceflib.milli_to_isotime (t[0],6)
'2002-01-05T00:00:21.529000Z

We have added 3 functions to manipulate time variables

timestamp (time-variable)

This function returns a numpy array of Unix timestamps.

The values are expressed in fractional number of seconds since 1970-01-01T00:00:00.

>>> t = ceflib.timestamp ("time_tags")
>>> t [0]
1010188821.529
>>> from datetime import datetime
>>> datetime.utcfromtimestamp (t[0])
datetime.datetime(2002, 1, 5, 0, 0, 21, 529000)

datetime (time-variable)

This function returns a numpy array of python datetime.datetime objects

You can then use the usual methods defined for datetime.datetime objects

>>> dt = ceflib.datetime ("time_tags")
>>> dt [0]
datetime.datetime(2002, 1, 5, 0, 0, 21, 529000)
>>> dt [-1].isoformat()
'2002-01-05T19:45:49.848000'

datetime64 (time-variable)

This function returns a numpy array of numpy.datetime64[ms]

>>> dt = ceflib.datetime64 ("time_tags")
>>> dt [0]
numpy.datetime64('2002-01-05T00:00:21.529')
>>> print (dt[-1])
2002-01-05T19:45:49.848

There are internally stored as npy_int64 integers, number of milli-seconds.

ISO_TIME_RANGE variables

When accessing to CEF variables with VALUE_TYPE = ISO_TIME_RANGE, you will get a numpy array of number-of-records x 2 values, corresponding to the start and the end of each time interval.

>>> ceflib.read ("C1_CT_AUX_GRMB__20010101_000000_20020101_000000_V240620.cef")
>>> ceflib.vattr("time_tags", "VALUE_TYPE")
'ISO_TIME_RANGE'
>>> dt = ceflib.datetime64("time_tags")
>>> dt.shape
(2383,2)
>>> dt [0]
array(['2001-01-01T00:02:56.000', '2001-01-01T00:19:16.000'],
      dtype='datetime64[ms]')

>>> str(dt[0])
"['2001-01-01T00:02:56.000' '2001-01-01T00:19:16.000']"

How to start

Import the library with cef alias and call its methods with cef.method-name()

$ python
>>> import ceflib as cef
>>> cef.read ("C1_CP_CIS-HIA_ONBOARD_MOMENTS__20030927_V01.cef")
>>> cef.varnames ()
>>> help(cef)

Or, import all the methods of the library in current environment:

$ python
>>> from ceflib import *
>>> read ("C1_CP_CIS-HIA_ONBOARD_MOMENTS__20030927_V01.cef")
>>> varnames ()

Librairie Reference

verbosity (level)

Set CEFLIB verbosity level (0..5) that can be useful to debug in case of problem.

You can set level to 0 to suppress unexpected CEFLIB messages.

>>> import ceflib as cef
>>> cef.verbosity (0)

read (filename) → integer

Read CEF file content and load data in memory.

Return error code (0 if OK)

Example

>>> status = cef.read ("C1_CP_CIS-HIA_ONBOARD_MOMENTS__20030927_V01.cef")
1 : CEF.Clear_descriptor           : Release memory
1 : CEF.Open_CEF_file              : C1_CP_CIS-HIA_ONBOARD_MOMENTS__20030907_V09.cef.gz
1 : CEF.Read_CEF_header            : 20206 records found in 0.03 seconds
1 : CEF.Read_CEF_header            : 24 values per record
1 : CEF.Display_variables          : 12 variables + 0 constants
1 : CEF.Read_CEF_file              : 20206 records physically read in 0.06 seconds
1 : CEF.Read_CEF_file              : STATUS = 0 : OK : Terminaison correcte
>>> print status == 0 and "OK" or "Error"
OK

You can open a regular .cef file or a compressed .cef.gz file

read_metadata (filename) → integer

New method (added in v1.8.0) that parse the metadata content of a given CEF header, without reading data records to improve performance.

Any metadata queries will be available, but the content of variables is undefined.

>>> status = cef.read_metadata ("C1_CP_CIS-HIA_ONBOARD_MOMENTS__20030927_V01.cef")
1 : CEF.Clear_descriptor           : Release memory
1 : CEF.Open_CEF_file              : C1_CP_CIS-HIA_ONBOARD_MOMENTS__20030907_V09.cef.gz
1 : CEF.Read_CEF_header            : 24 values per record
1 : CEF.Display_variables          : 12 variables + 0 constants
1 : CEF.Read_CEF_metadata          : STATUS = 0 : OK : Terminaison correcte

close ()

Release allocated ressources.

records () → integer

Return the number of records in the current CEF file.

>>> print (cef.records ())
20206

Is the file has been opened with read_metadata(), the records() function will return 1, instead of the number of records in the file.

gattributes () → list of strings

Return the list of CEF global attributes

>>> cef.gattributes ()
['FILE_NAME', 'FILE_FORMAT_VERSION', 'END_OF_RECORD_MARKER', 'DATA_UNTIL']

gattr (attribute-name) → string

Return value of a global attribute.

>>> print (cef.gattr ("FILE_NAME"))
C1_CP_CIS-HIA_ONBOARD_MOMENTS__20030907_V09.cef

metanames () → list of strings

Return list of metadata sections names.

>>> print (cef.metanames ()[:5])
['Logical_file_ID', 'Generation_date', 'File_time_span', 'Version_number', 'MISSION']

meta (metadata) → list of string

Return values for a given metadata section

>> for m in cef.metanames (): print (m, meta (m))
Logical_file_ID ['C1_CP_CIS-HIA_ONBOARD_MOMENTS__20030907_V09']
Generation_date ['2013-09-09T18:01:52Z']
File_time_span ['2003-09-07T00:00:00.000Z/2003-09-07T23:59:59.999Z']
Version_number ['09']
MISSION ['Cluster']
MISSION_TIME_SPAN ['2000-08-16T12:39:00Z/2030-12-31T23:59:59Z']
MISSION_AGENCY ['ESA']
...

this function returns allways a list of strings, even there is only one value.

varnames () → list of strings

Return the list of CEF variables defined in the current file

>>> print "\n".join (cef.varnames ())
time_tags
delta_time
sensitivity
cis_mode
density
velocity_isr2
velocity_gse
temperature
temp_par
temp_perp
pressure
pressure_tensor

var (varname) → numpy array of variable data type

Return data content for the specified variable name

>>> print (cef.var ("density"))
[ 0.01923148  0.03846296  0.01035187 ...,  0.01831131  0.01923148
  0.02668483]

>>> print var ("time_tags")
[  1.44158400e+12   1.44158401e+12   1.44158401e+12 ...,   1.44167035e+12
   1.44167036e+12   1.44167036e+12]

vattributes (varname) → list of strings

Return list of attibutes for a given variable

>>> cef.vattributes ("density")
['CATDESC', 'FIELDNAM', 'PARAMETER_TYPE', 'VALUE_TYPE', 'ENTITY', 'PROPERTY',
'UNITS', 'SI_CONVERSION', 'DEPEND_0', 'QUALITY', 'SIGNIFICANT_DIGITS',
'FILLVAL']

vattr (varname, attribute) → string

Get attribute value for a given variable

>>> cef.vattr ("density", "UNITS")
'particles cm^-3'

>>> for a in cef.vattributes ("density"):
       print ("%s = %s" % (a, vattr("density", a)))
CATDESC = Density (in cm^-3)
FIELDNAM = Density
PARAMETER_TYPE = Data
VALUE_TYPE = FLOAT
ENTITY = Ion
PROPERTY = Mass_Density
UNITS = particles cm^-3
SI_CONVERSION = 1e6>(particles) m^-3
DEPEND_0 = time_tags
QUALITY = 3
SIGNIFICANT_DIGITS = 6
FILLVAL = -1e31

isotime_to_milli (string) → double

Converts ISOTIME string to internal time-tags

>>> t = cef.isotime _to_milli ("2003-09-07T00:00:03Z")

milli_to_isotime (double, integer) → string

Converts internal time-tags to ISOTIME strings.

The first arguments must be the content of an ISOTIME CEF variable.

The precision represents the number of digits after the dot:

0 for plain seconds
3 for milli-seconds
6 for micro-seconds

>>> t = cef.var("time_tags") [0]
>>> t
1.4415840031e+12
>>> print (cef.Lmilli_to_isotime (t, 0))
2003-09-07T00:00:03Z
>>> print (cef.milli_to_isotime (t, 3))
2003-09-07T00:00:03.103Z

Python user manual

New in 1.8.5

metadata()

Deprecated functions

gattrs()

Deprecated functions

vattrs (varname)

Deprecated functions

New in 1.8.4

timestamp (time-variable)

datetime (time-variable)

datetime64 (time-variable)

ISO_TIME_RANGE variables

How to start

Librairie Reference

verbosity (level)

read (filename) → integer

read_metadata (filename) → integer

close ()

records () → integer

gattributes () → list of strings

gattr (attribute-name) → string

metanames () → list of strings

meta (metadata) → list of string

varnames () → list of strings

var (varname) → numpy array of variable data type

vattributes (varname) → list of strings

vattr (varname, attribute) → string

isotime_to_milli (string) → double

milli_to_isotime (double, integer) → string