Data and time¶

schema ¶

Input data model: a long table of measurements, a wide table with a YAML column mapping, canonical signals.

Long format (the native one), one row per time stamp, device and signal:

column	meaning
timestamp	start (or end, or centre: see `label`) of the averaging interval
device_id	any string: an inverter, a turbine, a meter, `PLANT` for site signals
signal	a canonical signal name (`SIGNALS`) or a name mapped to one
value	number in the canonical unit of the signal, or in the unit the mapping states

Wide format: one row per time stamp and one column per device and signal, e.g. a logger export. A mapping names the time stamp column and says, for every column used, which device and signal it holds and in which unit. Mappings are plain dictionaries or YAML files:

format: wide
timestamp_column: Timestamp
timezone: Europe/Warsaw     # naive local time with DST; ISO strings with offsets are read exactly
label: end                  # the logger stamps the end of each average
step: 15min                 # output grid
columns:
  "Irradiance POA [W/m2]": {signal: poa, unit: W/m2}
  "INV01 Pac [W]":         {device: INV01, signal: p_ac, unit: W}

For a long table the mapping renames signals and devices instead:

format: long
timezone: Etc/GMT+8
signals:
  ambient_temp_f: {signal: t_amb, unit: degF}
  P_POI_kW:       {signal: p_meter, unit: kW, sign: -1}
devices: {MET: PLANT}

read turns either form into a tidy long table with UTC time stamps and canonical units; to_wide puts it on a regular grid with one column per (device, signal).

LONG_COLUMNS `module-attribute` ¶

LONG_COLUMNS = ('timestamp', 'device_id', 'signal', 'value')

Columns of the long format, in order.

SIGNALS `module-attribute` ¶

SIGNALS: dict[str, SignalInfo] = {'poa': SignalInfo('W/m2', 'mean', 'plane-of-array irradiance'), 'ghi': SignalInfo('W/m2', 'mean', 'global horizontal irradiance'), 't_amb': SignalInfo('degC', 'mean', 'ambient air temperature'), 't_mod': SignalInfo('degC', 'mean', 'back-of-module temperature'), 't_cell': SignalInfo('degC', 'mean', 'PV cell temperature'), 'wind_speed': SignalInfo('m/s', 'mean', 'wind speed (nacelle anemometer or met mast)'), 'wind_dir': SignalInfo('deg', 'circular', 'wind direction'), 'pressure': SignalInfo('Pa', 'mean', 'air pressure'), 'precip': SignalInfo('mm', 'sum', 'precipitation per interval'), 'p_ac': SignalInfo('kW', 'mean', 'AC active power, positive = export'), 'p_dc': SignalInfo('kW', 'mean', 'DC power, positive = discharge or generation'), 'e_export': SignalInfo('kWh', 'sum', 'exported energy per interval'), 'e_import': SignalInfo('kWh', 'sum', 'imported energy per interval'), 'setpoint': SignalInfo('fraction', 'mean', 'active power limit as a share of the rating'), 'rotor_speed': SignalInfo('rpm', 'mean', 'rotor speed'), 'pitch': SignalInfo('deg', 'mean', 'blade pitch angle'), 'nacelle_dir': SignalInfo('deg', 'circular', 'nacelle position (yaw)'), 'p_meter': SignalInfo('kW', 'mean', 'active power at the revenue meter, positive = export'), 'p_aux': SignalInfo('kW', 'mean', 'auxiliary consumption, positive = consumption'), 'soc': SignalInfo('fraction', 'instant', 'state of charge'), 'e_stored': SignalInfo('kWh', 'instant', 'stored energy'), 'p_avail_dis': SignalInfo('kW', 'mean', 'available discharge power'), 'p_avail_ch': SignalInfo('kW', 'mean', 'available charge power (magnitude)'), 'e_avail': SignalInfo('kWh', 'mean', 'available (dischargeable) energy'), 't_cell_avg': SignalInfo('degC', 'mean', 'mean battery cell temperature'), 't_cell_max': SignalInfo('degC', 'mean', 'highest battery cell temperature')}

Canonical signals: unit, aggregation rule (how intervals are combined) and meaning.

SchemaError ¶

Bases: ValueError

Input data or a mapping does not follow the data model.

SignalInfo `dataclass` ¶

SignalInfo(unit: str, aggregation: Aggregation, description: str)

Canonical unit, aggregation rule and meaning of a signal.

ColumnSpec `dataclass` ¶

ColumnSpec(signal: str, device: str | None = None, unit: str | None = None, scale: float = 1.0)

What one column of a wide table holds.

SignalSpec `dataclass` ¶

SignalSpec(signal: str, unit: str | None = None, scale: float = 1.0)

How a signal name of a long table maps to a canonical signal.

Mapping `dataclass` ¶

Mapping(format: Literal['long', 'wide'] = 'long', timestamp_column: str = 'timestamp', timezone: str = 'Europe/Warsaw', label: Label = 'start', step: str | None = None, plant_device: str = 'PLANT', columns: dict[str, ColumnSpec] = dict(), signals: dict[str, SignalSpec] = dict(), devices: dict[str, str] = dict())

Column and signal mapping of an input table (see the module description).

Attributes:

Name	Type	Description
`format`	`{'long', 'wide'}`
`timestamp_column`	`str`	Name of the time stamp column.
`timezone`	`str`	Time zone of naive time stamps.
`label`	`{'start', 'end', 'center'}`	Which instant of its interval a time stamp marks.
`step`	`str or None`	Output interval length; `None` takes the coarsest interval found.
`plant_device`	`str`	Device id given to wide columns without a `device`.
`columns`	`dict of str to ColumnSpec`	Wide format: the columns used.
`signals`	`dict of str to SignalSpec`	Long format: renamed signals (others must already be canonical).
`devices`	`dict of str to str`	Long format: renamed devices.

load_mapping ¶

load_mapping(source: str | Path | dict[str, Any]) -> Mapping

Read a mapping from a YAML file or a dictionary and check it.

Parameters:

Name	Type	Description	Default
`source`	`(str, Path or dict)`	Path of a YAML file, or the parsed content.	required

Returns:

Type	Description
`Mapping`

Raises:

Type	Description
`SchemaError`	Unknown keys, unknown signals, or units of the wrong dimension.

Examples:

>>> m = load_mapping({"format": "wide", "timestamp_column": "Time",
...                   "columns": {"Pac [W]": {"device": "INV1", "signal": "p_ac", "unit": "W"}}})
>>> m.columns["Pac [W]"].unit
'W'

validate_long ¶

validate_long(df: DataFrame) -> pd.DataFrame

Check a long table and return a clean copy.

The copy has exactly the columns timestamp, device_id, signal (strings) and value (float; non-numeric entries become NaN).

Raises:

Type	Description
`SchemaError`	Missing columns, or signals that are not canonical.

Examples:

>>> raw = pd.DataFrame({"timestamp": ["2024-01-01 00:00"], "device_id": ["INV1"], "signal": ["p_ac"],
...                     "value": ["12.5"]})
>>> float(validate_long(raw)["value"].iloc[0])
12.5

read ¶

read(raw: DataFrame, mapping: Mapping | dict[str, Any] | str | Path | None = None) -> pd.DataFrame

Turn a long or wide input table into a tidy long table with UTC time stamps and canonical units.

Parameters:

Name	Type	Description	Default
`raw`	`DataFrame`	Input table.	required
`mapping`	`(Mapping, dict, str or Path)`	The mapping (see `load_mapping`). Without one the table must already be a long table with canonical signals and naive `Europe/Warsaw` or offset-carrying time stamps.	`None`

Returns:

Type	Description
`DataFrame`	Columns `timestamp` (UTC, as given by `label`), `device_id`, `signal`, `value`. Rows whose time stamp cannot be placed (DST) are dropped with a warning in the log.

Examples:

>>> wide = pd.DataFrame({"Time": ["2024-07-01 12:00", "2024-07-01 12:15"], "Pac [W]": [1000.0, 2000.0]})
>>> m = {"format": "wide", "timestamp_column": "Time",
...      "columns": {"Pac [W]": {"device": "INV1", "signal": "p_ac", "unit": "W"}}}
>>> read(wide, m)
                  timestamp device_id signal  value
0 2024-07-01 10:00:00+00:00      INV1   p_ac    1.0
1 2024-07-01 10:15:00+00:00      INV1   p_ac    2.0

to_wide ¶

to_wide(long: DataFrame, step: str | Timedelta | None = None, label: Label | None = None) -> pd.DataFrame

Put a tidy long table on one regular UTC grid, one column per (device, signal).

Each device and signal is regularised on its own (see regularise) with the aggregation rule of the signal, then all are aligned on the union of their spans.

Parameters:

Name	Type	Description	Default
`long`	`DataFrame`	Output of `read` (or any long table with UTC time stamps and canonical signals).	required
`step`	`str or Timedelta`	Output interval; default: `long.attrs["step"]` or the coarsest interval found.	`None`
`label`	`('start', 'end', 'center')`	Default: `long.attrs["label"]` or `"start"`.	`"start"`

Returns:

Type	Description
`DataFrame`	Index: interval starts (UTC); columns: `MultiIndex` (device, signal). `attrs["step"]` holds the interval length.

Examples:

>>> long = pd.DataFrame({"timestamp": pd.date_range("2024-07-01 10:00", periods=4, freq="5min", tz="UTC"),
...                      "device_id": "INV1", "signal": "p_ac", "value": [1.0, 2.0, 3.0, 4.0]})
>>> wide = to_wide(long, "15min")
>>> wide[("INV1", "p_ac")].tolist()
[2.0, 4.0]
>>> wide.index[0]
Timestamp('2024-07-01 10:00:00+0000', tz='UTC')

to_long ¶

to_long(wide: DataFrame) -> pd.DataFrame

Inverse of to_wide: back to the long format (missing values dropped).

Examples:

>>> idx = pd.date_range("2024-01-01", periods=2, freq="1h", tz="UTC")
>>> wide = pd.DataFrame({("INV1", "p_ac"): [1.0, float("nan")]}, index=idx)
>>> to_long(wide)
                  timestamp device_id signal  value
0 2024-01-01 00:00:00+00:00      INV1   p_ac    1.0

timeseries ¶

Time stamps, time zones, regular grids and reporting periods.

Conventions used throughout the package:

Every series is indexed by UTC time stamps that mark the start of the averaging interval.
Reporting periods (days, months, years, contract years) are taken in local time, by default Europe/Warsaw. A local day then has 23 or 25 hours on the days of the daylight saving time (DST) changes, March has 743 hours and October 745.
Supported interval lengths are 5, 10, 15 and 30 minutes and 1 hour. A series can be put on a coarser grid (averaged or summed) or a finer one (spread) when one length divides the other.

Naive local time stamps are ambiguous once a year: in autumn the hour 02:00-03:00 occurs twice. Data loggers export it twice, in time order, so within each device and signal the first occurrence is read as summer time and the second as winter time. A repeated hour seen only once, a third occurrence and the skipped spring hour cannot be placed; they become NaT and are reported in the log.

SUPPORTED_STEPS `module-attribute` ¶

SUPPORTED_STEPS: tuple[Timedelta, ...] = tuple(pd.Timedelta(s) for s in ('5min', '10min', '15min', '30min', '1h'))

Supported interval lengths: 5, 10, 15 and 30 minutes and 1 hour.

check_step ¶

check_step(step: str | Timedelta) -> pd.Timedelta

Validate an interval length.

Parameters:

Name	Type	Description	Default
`step`	`str or Timedelta`	For example `"15min"` or `"1h"`.	required

Returns:

Type	Description
`Timedelta`

Raises:

Type	Description
`ValueError`	If the length is not one of `SUPPORTED_STEPS`.

Examples:

>>> check_step("15min")
Timedelta('0 days 00:15:00')

infer_step ¶

infer_step(timestamps: DatetimeIndex | Series | Sequence[Timestamp]) -> pd.Timedelta

Most common positive spacing of the time stamps, rounded to the minute (median of the differences).

Examples:

>>> idx = pd.date_range("2024-01-01", periods=5, freq="10min", tz="UTC")
>>> infer_step(idx)
Timedelta('0 days 00:10:00')

interval_hours ¶

interval_hours(index: DatetimeIndex) -> float

Length of one interval of a regular index, in hours.

Examples:

>>> interval_hours(pd.date_range("2024-01-01", periods=3, freq="15min", tz="UTC"))
0.25

to_utc ¶

to_utc(timestamps: Series | DatetimeIndex | Sequence[str], tz: str = 'Europe/Warsaw', groups: Series | DataFrame | None = None, fmt: str | None = None) -> pd.Series

Parse time stamps and convert them to UTC.

Parameters:

Name	Type	Description	Default
`timestamps`	`pandas.Series, pandas.DatetimeIndex or sequence of str`	Strings or datetimes. Strings that all carry a UTC offset (`2024-10-27T02:30:00+02:00`) and time-zone-aware datetimes are exact and `tz` is ignored.	required
`tz`	`str`	IANA time zone of naive time stamps; `"UTC"` for naive UTC.	`"Europe/Warsaw"`
`groups`	`Series or DataFrame`	Keys (e.g. device and signal) within which the repeated autumn hour is resolved by order of appearance. Without it the whole input is one group.	`None`
`fmt`	`str`	`strftime` pattern of the strings, e.g. `"%d.%m.%Y %H:%M"`; ISO 8601 by default.	`None`

Returns:

Type	Description
`Series`	`datetime64[ns, UTC]` with the index of the input; time stamps that cannot be placed are `NaT`.

Raises:

Type	Description
`ValueError`	If only some strings carry a UTC offset.

Examples:

The autumn change in Poland: 02:30 local time occurs twice.

>>> t = to_utc(["2024-10-27 01:30", "2024-10-27 02:30", "2024-10-27 02:30", "2024-10-27 03:30"])
>>> [x.strftime("%H:%M") for x in t]
['23:30', '00:30', '01:30', '02:30']

regularise ¶

regularise(values: Series, step: str | Timedelta, *, source_step: str | Timedelta | None = None, label: Label = 'start', how: Aggregation = 'mean') -> pd.Series

Put one signal on a regular UTC grid whose time stamps mark the start of each interval.

Parameters:

Name	Type	Description	Default
`values`	`Series`	Values indexed by time-zone-aware time stamps (any order; duplicates are averaged or summed).	required
`step`	`str or Timedelta`	Interval length of the output grid (see `SUPPORTED_STEPS`).	required
`source_step`	`str or Timedelta`	Interval length of the input; inferred from the time stamps by default.	`None`
`label`	`('start', 'end', 'center')`	Which instant of its averaging interval an input time stamp marks.	`"start"`
`how`	`('mean', 'sum', 'instant', 'circular')`	`mean` for interval averages (power, irradiance, temperature), `sum` for amounts per interval (energy, precipitation; spread evenly when the grid is finer), `instant` for states read at an instant (state of charge, stored energy; not shifted by `label`, interpolated linearly onto a finer grid), `circular` for directions in degrees.	`"mean"`

Returns:

Type	Description
`Series`	Values on the complete grid from the first to the last interval; missing intervals are NaN.

Raises:

Type	Description
`ValueError`	If the index is naive, a step is not supported, or neither step divides the other.

Examples:

Three 5-minute means labelled by the end of their interval become one 15-minute mean:

>>> idx = pd.to_datetime(["2024-06-01 10:05", "2024-06-01 10:10", "2024-06-01 10:15"]).tz_localize("UTC")
>>> regularise(pd.Series([1.0, 2.0, 6.0], index=idx), "15min", label="end")
2024-06-01 10:00:00+00:00    3.0
Freq: 15min, dtype: float64

period_labels ¶

period_labels(index: DatetimeIndex, freq: Freq = 'M', tz: str = 'Europe/Warsaw') -> pd.Index

Label of the local-time period each time stamp falls in.

Parameters:

Name	Type	Description	Default
`index`	`DatetimeIndex`	Time-zone-aware time stamps (interval starts).	required
`freq`	`('D', 'M', 'Q', 'Y')`	Local day (`2024-03-31`), month (`2024-03`), quarter (`2024-Q1`) or year (`2024`).	`"D"`
`tz`	`str`	Time zone in which the periods are counted.	`"Europe/Warsaw"`

Returns:

Type	Description
`Index`	One string label per time stamp.

Examples:

23:30 UTC on 31 March is already 1 April in Warsaw:

>>> idx = pd.DatetimeIndex(["2024-03-31 21:00", "2024-03-31 23:30"], tz="UTC")
>>> list(period_labels(idx, "M"))
['2024-03', '2024-04']

period_bounds ¶

period_bounds(start: Timestamp | str, end: Timestamp | str, freq: Freq = 'M', tz: str = 'Europe/Warsaw') -> pd.DataFrame

Local-time periods covering [start, end) with their UTC bounds and length in hours.

Parameters:

Name	Type	Description	Default
`start`	`Timestamp or str`	Time-zone-aware bounds (strings with an offset, or naive strings read as UTC).	required
`end`	`Timestamp or str`	Time-zone-aware bounds (strings with an offset, or naive strings read as UTC).	required
`freq`	`('D', 'M', 'Q', 'Y')`		`"D"`
`tz`	`str`		`"Europe/Warsaw"`

Returns:

Type	Description
`DataFrame`	Columns `period`, `start` and `end` (UTC) and `hours`.

Examples:

>>> p = period_bounds("2024-03-01 00:00+01:00", "2024-04-01 00:00+02:00")
>>> p[["period", "hours"]]
    period  hours
0  2024-03  743.0

contract_years ¶

contract_years(index: DatetimeIndex, start: Timestamp | str, tz: str = 'Europe/Warsaw') -> np.ndarray

1-based contract year of each time stamp; year 1 starts at local midnight of start.

Years run from anniversary to anniversary of the start date (calendar years, 365 or 366 days). Time stamps before the start get 0.

Examples:

>>> idx = pd.DatetimeIndex(["2024-06-30 21:59", "2024-06-30 22:00", "2026-01-01 00:00"], tz="UTC")
>>> contract_years(idx, "2023-07-01").tolist()
[1, 2, 3]

units ¶

Units of measurement: a small registry and conversion between units of the same dimension.

Every unit belongs to one dimension and is stored as (dimension, scale, offset) against the canonical unit of that dimension, so that canonical = value * scale + offset. Unit names are case-insensitive and tolerate the usual spellings (degC, °C, W/m², W/m2).

dimension	canonical unit	other units
power	kW	W, MW, GW
energy	kWh	Wh, MWh, GWh, J, MJ, GJ
irradiance	W/m2	kW/m2
irradiation	kWh/m2	Wh/m2, MJ/m2
temperature	degC	degF, K
speed	m/s	km/h, mph, kn
rotation	rpm	rad/s
fraction	fraction	%, 1, pu
angle	deg	rad
pressure	Pa	hPa, kPa, mbar, bar
density	kg/m3
length	mm	cm, m
time	h	s, min, d

UNITS `module-attribute` ¶

UNITS: dict[str, tuple[str, float, float]] = {'kw': ('power', 1.0, 0.0), 'w': ('power', 0.001, 0.0), 'mw': ('power', 1000.0, 0.0), 'gw': ('power', 1000000.0, 0.0), 'kwh': ('energy', 1.0, 0.0), 'wh': ('energy', 0.001, 0.0), 'mwh': ('energy', 1000.0, 0.0), 'gwh': ('energy', 1000000.0, 0.0), 'j': ('energy', 1.0 / 3600000.0, 0.0), 'mj': ('energy', 1.0 / 3.6, 0.0), 'gj': ('energy', 1000.0 / 3.6, 0.0), 'w/m2': ('irradiance', 1.0, 0.0), 'kw/m2': ('irradiance', 1000.0, 0.0), 'kwh/m2': ('irradiation', 1.0, 0.0), 'wh/m2': ('irradiation', 0.001, 0.0), 'mj/m2': ('irradiation', 1.0 / 3.6, 0.0), 'degc': ('temperature', 1.0, 0.0), 'c': ('temperature', 1.0, 0.0), 'degf': ('temperature', 5.0 / 9.0, -32.0 * 5.0 / 9.0), 'f': ('temperature', 5.0 / 9.0, -32.0 * 5.0 / 9.0), 'k': ('temperature', 1.0, -273.15), 'm/s': ('speed', 1.0, 0.0), 'km/h': ('speed', 1.0 / 3.6, 0.0), 'mph': ('speed', 0.44704, 0.0), 'kn': ('speed', 1852.0 / 3600.0, 0.0), 'rpm': ('rotation', 1.0, 0.0), 'rad/s': ('rotation', 60.0 / (2.0 * math.pi), 0.0), 'fraction': ('fraction', 1.0, 0.0), '1': ('fraction', 1.0, 0.0), 'pu': ('fraction', 1.0, 0.0), '%': ('fraction', 0.01, 0.0), 'deg': ('angle', 1.0, 0.0), 'rad': ('angle', 180.0 / math.pi, 0.0), 'pa': ('pressure', 1.0, 0.0), 'hpa': ('pressure', 100.0, 0.0), 'mbar': ('pressure', 100.0, 0.0), 'kpa': ('pressure', 1000.0, 0.0), 'bar': ('pressure', 100000.0, 0.0), 'kg/m3': ('density', 1.0, 0.0), 'mm': ('length', 1.0, 0.0), 'cm': ('length', 10.0, 0.0), 'm': ('length', 1000.0, 0.0), 'h': ('time', 1.0, 0.0), 's': ('time', 1.0 / 3600.0, 0.0), 'min': ('time', 1.0 / 60.0, 0.0), 'd': ('time', 24.0, 0.0)}

Unit registry: name -> (dimension, scale, offset), canonical = value * scale + offset.

normalise_unit ¶

normalise_unit(unit: str) -> str

Registry key of a unit name.

Parameters:

Name	Type	Description	Default
`unit`	`str`	Unit as written in a file header or a mapping, e.g. `"°C"`, `"W/m²"`, `"MWh"`.	required

Returns:

Type	Description
`str`	The lower-case key used in `UNITS`.

Raises:

Type	Description
`ValueError`	If the unit is not in the registry.

Examples:

>>> normalise_unit("W/m²")
'w/m2'
>>> normalise_unit("°C")
'degc'

dimension ¶

dimension(unit: str) -> str

Dimension of a unit ("power", "energy", ...).

Examples:

>>> dimension("MWh")
'energy'

canonical_unit ¶

canonical_unit(unit: str) -> str

Canonical unit of the dimension unit belongs to.

Examples:

>>> canonical_unit("MW")
'kw'
>>> canonical_unit("degF")
'degc'

convert ¶

convert(values: T, from_unit: str, to_unit: str) -> T

Convert values between two units of the same dimension.

Parameters:

Name	Type	Description	Default
`values`	`(float, ndarray, Series or DataFrame)`	Values in `from_unit`; NaN stays NaN.	required
`from_unit`	`str`	Unit names (see the table in the module description).	required
`to_unit`	`str`	Unit names (see the table in the module description).	required

Returns:

Type	Description
Same type as ``values``	Values in `to_unit`.

Raises:

Type	Description
`ValueError`	If a unit is unknown or the two units measure different dimensions.

Examples:

>>> convert(1500.0, "W", "kW")
1.5
>>> round(convert(32.0, "degF", "degC"), 6)
0.0
>>> convert(0.5, "fraction", "%")
50.0

quality ¶

Data quality flags: missing, stuck and out-of-range values, as bit flags per interval.

Flags are integers combined with |, so one interval can carry several. Checks never change the data; apply_flags sets the rejected values to NaN when the analyst decides so, and coverage reports the share of usable intervals per period.

Default plausible ranges (RANGES) are wide physical limits meant to catch unit and scaling errors, not to judge a sensor; tighten them per site.

Examples:

>>> s = pd.Series([20.0, 21.0, 21.0, 21.0, 21.0, None, 400.0])
>>> f = quality_flags(s, "t_amb", stuck_run=4)
>>> [str(Flag(int(x))) for x in f]
['Flag.OK', 'Flag.STUCK', 'Flag.STUCK', 'Flag.STUCK', 'Flag.STUCK', 'Flag.MISSING', 'Flag.OUT_OF_RANGE']

REJECT `module-attribute` ¶

REJECT = Flag.MISSING | Flag.STUCK | Flag.OUT_OF_RANGE

Flags that make a value unusable by default.

RANGES `module-attribute` ¶

RANGES: dict[str, tuple[float, float]] = {'poa': (-20.0, 1600.0), 'ghi': (-20.0, 1500.0), 't_amb': (-50.0, 60.0), 't_mod': (-50.0, 100.0), 't_cell': (-50.0, 110.0), 'wind_speed': (0.0, 60.0), 'wind_dir': (0.0, 360.0), 'nacelle_dir': (0.0, 360.0), 'pressure': (50000.0, 110000.0), 'setpoint': (0.0, 1.0), 'soc': (0.0, 1.0), 'pitch': (-10.0, 120.0)}

Default plausible range per canonical signal, in canonical units (inclusive).

Flag ¶

Bases: IntFlag

Data quality flags of one interval (combine with |).

MISSING `class-attribute` `instance-attribute` ¶

MISSING = 1

No value recorded.

STUCK `class-attribute` `instance-attribute` ¶

STUCK = 2

The same value repeated over a run of intervals (frozen sensor or logger).

OUT_OF_RANGE `class-attribute` `instance-attribute` ¶

OUT_OF_RANGE = 4

Outside the plausible range of the signal.

FILLED `class-attribute` `instance-attribute` ¶

FILLED = 8

Value filled or corrected by the analyst (set by the user, never by the checks here).

flag_missing ¶

flag_missing(values: Series | ndarray | Iterable[float]) -> pd.Series

Flag.MISSING where the value is NaN.

Examples:

>>> flag_missing([1.0, float("nan")]).tolist()
[0, 1]

flag_stuck ¶

flag_stuck(values: Series | ndarray | Iterable[float], min_run: int, ignore: Iterable[float] = ()) -> pd.Series

Flag.STUCK on every value of a run of at least min_run equal values.

Parameters:

Name	Type	Description	Default
`values`	`array - like`	One signal in time order.	required
`min_run`	`int`	Shortest run that counts as stuck (e.g. six hours of intervals for an ambient temperature, three 10-minute records for an anemometer).	required
`ignore`	`iterable of float`	Values whose runs are normal (e.g. `0` for irradiance at night or power of a stopped turbine).	`()`

Examples:

>>> flag_stuck([0.0, 0.0, 0.0, 5.0, 5.0, 5.0, 6.0], min_run=3, ignore=[0.0]).tolist()
[0, 0, 0, 2, 2, 2, 0]

flag_out_of_range ¶

flag_out_of_range(values: Series | ndarray | Iterable[float], lower: float | None = None, upper: float | None = None) -> pd.Series

Flag.OUT_OF_RANGE below lower or above upper.

Bounds are inclusive; None leaves that side open.

Examples:

>>> flag_out_of_range([-1.0, 0.5, 2.0], 0.0, 1.0).tolist()
[4, 0, 4]

quality_flags ¶

quality_flags(values: Series | ndarray | Iterable[float], signal: str | None = None, *, stuck_run: int | None = None, stuck_ignore: Iterable[float] = (), lower: float | None = None, upper: float | None = None) -> pd.Series

All checks of one signal combined.

Parameters:

Name	Type	Description	Default
`values`	`array - like`	One signal in time order, canonical units.	required
`signal`	`str`	Canonical signal name; selects the default range from `RANGES`.	`None`
`stuck_run`	`int`	Run length for `flag_stuck`; no stuck check without it.	`None`
`stuck_ignore`	`iterable of float`	Passed to `flag_stuck`.	`()`
`lower`	`float`	Override the default range.	`None`
`upper`	`float`	Override the default range.	`None`

Returns:

Type	Description
`Series`	`uint8` flags.

apply_flags ¶

apply_flags(values: Series, flags: Series | ndarray, reject: Flag = REJECT) -> pd.Series

Values with every interval carrying a rejected flag set to NaN.

Examples:

>>> apply_flags(pd.Series([1.0, 2.0]), np.array([0, Flag.STUCK])).tolist()
[1.0, nan]

coverage ¶

coverage(flags: Series | DataFrame, periods: Index | ndarray | None = None, reject: Flag = REJECT) -> pd.Series | pd.DataFrame

Share of intervals without a rejected flag, overall or per period label.

Parameters:

Name	Type	Description	Default
`flags`	`Series or DataFrame`	Flags of one or several signals.	required
`periods`	`array - like`	One label per interval, e.g. from `period_labels`.	`None`
`reject`	`Flag`		[`REJECT`][res_asset_kpi.quality.REJECT]

Examples:

>>> coverage(pd.Series([0, 1, 0, 0]), periods=np.array(["a", "a", "b", "b"])).tolist()
[0.5, 1.0]

summary ¶

summary(flags: Series) -> pd.Series

Number of intervals carrying each flag.

Examples:

>>> summary(pd.Series([0, 1, 3, 2])).to_dict()
{'MISSING': 2, 'STUCK': 2, 'OUT_OF_RANGE': 0, 'FILLED': 0}

intervals ¶

Operations on half-open time intervals [start, end): union, difference, overlap and first-up attribution.

Event logs (stops, alarms, maintenance) overlap: a turbine stopped by a fault may also show a grid event and a maintenance mode. Counting each event's own duration would count the same hour several times. The functions here work on the time line instead. Times are passed as anything pandas.to_datetime accepts and are handled internally as UTC nanoseconds.

to_ns ¶

to_ns(times: ArrayLike | Series | DatetimeIndex) -> np.ndarray

UTC datetimes (naive ones read as UTC) to int64 nanoseconds.

Examples:

>>> to_ns(["1970-01-01 01:00"]).tolist()
[3600000000000]

union ¶

union(starts: ArrayLike, ends: ArrayLike) -> tuple[np.ndarray, np.ndarray]

Union of intervals as sorted, disjoint (starts, ends) arrays (int64 nanoseconds in and out).

Examples:

>>> s, e = union([0, 5, 20], [10, 12, 30])
>>> s.tolist(), e.tolist()
([0, 20], [12, 30])

subtract ¶

subtract(a_starts: ArrayLike, a_ends: ArrayLike, b_starts: ArrayLike, b_ends: ArrayLike) -> tuple[np.ndarray, np.ndarray]

Intervals A minus the union of intervals B (int64 nanoseconds in and out).

Examples:

>>> s, e = subtract([0], [100], [10, 50], [20, 60])
>>> s.tolist(), e.tolist()
([0, 20, 60], [10, 50, 100])

overlap_hours ¶

overlap_hours(starts: ArrayLike, ends: ArrayLike, window_start: object, window_end: object) -> np.ndarray

Hours of each interval that fall inside [window_start, window_end).

Examples:

>>> overlap_hours(["2024-01-01 22:00"], ["2024-01-02 03:00"], "2024-01-02", "2024-01-03").tolist()
[3.0]

first_up ¶

first_up(starts: ArrayLike, ends: ArrayLike) -> pd.DataFrame

Split the union of overlapping intervals into segments owned by the earliest-starting active interval.

Each moment covered by at least one interval belongs to the interval that started first among those still active (ties: the one listed first). An interval entirely inside an earlier one owns nothing; summing the segment hours per owner then counts every hour once.

Parameters:

Name	Type	Description	Default
`starts`	`array - like`	Interval bounds (datetimes).	required
`ends`	`array - like`	Interval bounds (datetimes).	required

Returns:

Type	Description
`DataFrame`	Columns `start`, `end` (UTC), `owner` (position of the owning interval in the input) and `hours`.

Examples:

A fault from 10:00 to 14:00 and a grid event from 12:00 to 16:00:

>>> seg = first_up(["2024-01-01 10:00", "2024-01-01 12:00"], ["2024-01-01 14:00", "2024-01-01 16:00"])
>>> seg[["owner", "hours"]].to_dict("list")
{'owner': [0, 1], 'hours': [4.0, 2.0]}

Data and time¶

schema ¶

LONG_COLUMNS module-attribute ¶

SIGNALS module-attribute ¶

SchemaError ¶

SignalInfo dataclass ¶

ColumnSpec dataclass ¶

SignalSpec dataclass ¶

Mapping dataclass ¶

load_mapping ¶

validate_long ¶

read ¶

to_wide ¶

to_long ¶

timeseries ¶

SUPPORTED_STEPS module-attribute ¶

check_step ¶

infer_step ¶

interval_hours ¶

to_utc ¶

regularise ¶

period_labels ¶

period_bounds ¶

contract_years ¶

units ¶

UNITS module-attribute ¶

normalise_unit ¶

dimension ¶

canonical_unit ¶

convert ¶

quality ¶

REJECT module-attribute ¶

RANGES module-attribute ¶

Flag ¶

MISSING class-attribute instance-attribute ¶

STUCK class-attribute instance-attribute ¶

OUT_OF_RANGE class-attribute instance-attribute ¶

FILLED class-attribute instance-attribute ¶

flag_missing ¶

flag_stuck ¶

flag_out_of_range ¶

quality_flags ¶

apply_flags ¶

coverage ¶

summary ¶

intervals ¶

to_ns ¶

union ¶

subtract ¶

overlap_hours ¶

first_up ¶

LONG_COLUMNS `module-attribute` ¶

SIGNALS `module-attribute` ¶

SignalInfo `dataclass` ¶

ColumnSpec `dataclass` ¶

SignalSpec `dataclass` ¶

Mapping `dataclass` ¶

SUPPORTED_STEPS `module-attribute` ¶

UNITS `module-attribute` ¶

REJECT `module-attribute` ¶

RANGES `module-attribute` ¶

MISSING `class-attribute` `instance-attribute` ¶

STUCK `class-attribute` `instance-attribute` ¶

OUT_OF_RANGE `class-attribute` `instance-attribute` ¶

FILLED `class-attribute` `instance-attribute` ¶