IntenseWebs/grafana
3
0
Fork 0
mirror of https://github.com/grafana/grafana.git synced 2025-02-25 18:55:37 -06:00
Code Issues Packages Projects Releases Wiki Activity

Transformations: Cumulative and window modes for Add field from calculation (#77029)

Browse Source 
* cumulative sum

* refactor and create new mode

* refactor - use reduceOptions for new mode also

* revert naming

* Add window function, rename statistical to cumulative (#77066)

* Add window function, rename statistical to cumulative

* Fix merge errors

* fix more merge errors

* refactor + add window funcs

Co-authored-by: Oscar Kilhed <oscar.kilhed@grafana.com>

* add ff + tests + centered moving avg

Co-authored-by: Oscar Kilhed <oscar.kilhed@grafana.com>

* make sum and mean cumulative more efficient (#77173)

* make sum and mean cumulative more efficient

* remove cumulative variance, add window stddev

* refactor window to not use reducer for mean. wip variance stdDev

* fix tests after optimization

---------

Co-authored-by: Victor Marin <victor.marin@grafana.com>

* optimize window func (#77266)

* make sum and mean cumulative more efficient

* remove cumulative variance, add window stddev

* refactor window to not use reducer for mean. wip variance stdDev

* fix tests after optimization

* fix test lint

* optimize window

* tests are passing

* fix nulls

* fix all nulls

---------

Co-authored-by: Victor Marin <victor.marin@grafana.com>

* change window size to be percentage

* fix tests to use percentage

* fixed/percentage window size (#77369)

* Add docs for cumulative and window functions of the add field from calculation transform. (#77352)

add docs

* splling

* change WindowType -> WindowAlignment

* update betterer

* refactor getWindowCreator

* add docs to content.ts

* add feature toggle message

---------

Co-authored-by: Oscar Kilhed <oscar.kilhed@grafana.com>
This commit is contained in:
Victor Marin
2023-11-03 16:39:58 +02:00
committed by GitHub
parent 67b2972052
commit 61d63d3034
11 changed files with 1071 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
.betterer.results
View File

@@ -5089,7 +5089,12 @@ exports[`better eslint`] = {
[0, 0, 0, "Styles should be written using objects.", "0"]
],
"public/app/features/transformers/editors/CalculateFieldTransformerEditor.tsx:5381": [
[0, 0, 0, "Do not use any type assertions.", "0"]
[0, 0, 0, "Do not use any type assertions.", "0"],
[0, 0, 0, "Do not use any type assertions.", "1"],
[0, 0, 0, "Do not use any type assertions.", "2"],
[0, 0, 0, "Do not use any type assertions.", "3"],
[0, 0, 0, "Do not use any type assertions.", "4"],
[0, 0, 0, "Do not use any type assertions.", "5"]
],
"public/app/features/transformers/editors/ConvertFieldTypeTransformerEditor.tsx:5381": [
[0, 0, 0, "Do not use any type assertions.", "0"]

26
docs/sources/panels-visualizations/query-transform-data/transform-data/index.md
View File

@@ -127,6 +127,7 @@ You can perform the following transformations on your data.
Use this transformation to add a new field calculated from two other fields. Each transformation allows you to add one new field.
- **Mode -** Select a mode:
- **Reduce row -** Apply selected calculation on each row of selected fields independently.
- **Binary operation -** Apply basic binary operations (for example, sum or multiply) on values in a single row from two selected fields.
- **Unary operation -** Apply basic unary operations on values in a single row from a selected field. The available operations are:
@@ -135,7 +136,32 @@ Use this transformation to add a new field calculated from two other fields. Eac
- **Natural logarithm (ln)** - Returns the natural logarithm of a given expression.
- **Floor (floor)** - Returns the largest integer less than or equal to a given expression.
- **Ceiling (ceil)** - Returns the smallest integer greater than or equal to a given expression.
- **Cumulative functions** - Apply functions on the current row and all preceding rows.
**Note:** This mode is an experimental feature. Engineering and on-call support is not available.
Documentation is either limited or not provided outside of code comments. No SLA is provided.
Enable the 'addFieldFromCalculationStatFunctions' in Grafana to use this feature.
Contact Grafana Support to enable this feature in Grafana Cloud.
- **Total** - Calculates the cumulative total up to and including the current row.
- **Mean** - Calculates the mean up to and including the current row.
- **Window functions** - Apply window functions. The window can either be **trailing** or **centered**.
With a trailing window the current row will be the last row in the window.
With a centered window the window will be centered on the current row.
For even window sizes, the window will be centered between the current row, and the previous row.
**Note:** This mode is an experimental feature. Engineering and on-call support is not available.
Documentation is either limited or not provided outside of code comments. No SLA is provided.
Enable the 'addFieldFromCalculationStatFunctions' in Grafana to use this feature.
Contact Grafana Support to enable this feature in Grafana Cloud.
- **Mean** - Calculates the moving mean or running average.
- **Stddev** - Calculates the moving standard deviation.
- **Variance** - Calculates the moving variance.
- **Row index -** Insert a field with the row index.
- **Field name -** Select the names of fields you want to use in the calculation for the new field.
- **Calculation -** If you select **Reduce row** mode, then the **Calculation** field appears. Click in the field to see a list of calculation choices you can use to create the new field. For information about available calculations, refer to [Calculation types][].
- **Operation -** If you select **Binary operation** or **Unary operation** mode, then the **Operation** fields appear. These fields allow you to apply basic math operations on values in a single row from selected fields. You can also use numerical values for binary operations.

1
docs/sources/setup-grafana/configure-grafana/feature-toggles/index.md
View File

@@ -157,6 +157,7 @@ Experimental features might be changed or removed without prior notice.
| `costManagementUi` | Toggles the display of the cost management ui plugin |
| `managedPluginsInstall` | Install managed plugins directly from plugins catalog |
| `prometheusPromQAIL` | Prometheus and AI/ML to assist users in creating a query |
| `addFieldFromCalculationStatFunctions` | Add cumulative and window functions to the add field from calculation transformation |
| `alertmanagerRemoteSecondary` | Enable Grafana to sync configuration and state with a remote Alertmanager. |
| `alertmanagerRemotePrimary` | Enable Grafana to have a remote Alertmanager instance as the primary Alertmanager. |
| `alertmanagerRemoteOnly` | Disable the internal Alertmanager and only use the external one defined. |

496
packages/grafana-data/src/transformations/transformers/calculateField.test.ts
View File

@@ -7,7 +7,13 @@ import { mockTransformationsRegistry } from '../../utils/tests/mockTransformatio
import { ReducerID } from '../fieldReducer';
import { transformDataFrame } from '../transformDataFrame';
import { CalculateFieldMode, calculateFieldTransformer, ReduceOptions } from './calculateField';
import {
CalculateFieldMode,
calculateFieldTransformer,
ReduceOptions,
WindowSizeMode,
WindowAlignment,
} from './calculateField';
import { DataTransformerID } from './ids';
const seriesA = toDataFrame({
@@ -398,4 +404,492 @@ describe('calculateField transformer w/ timeseries', () => {
`);
});
});
it('calculates centered moving average on odd window size', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Centered,
field: 'x',
windowSize: 1,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.mean,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(1.5);
expect(data.fields[1].values[1]).toEqual(2);
expect(data.fields[1].values[2]).toEqual(2.5);
});
});
it('calculates centered moving average on even window size', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Centered,
field: 'x',
windowSize: 0.5,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.mean,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(1);
expect(data.fields[1].values[1]).toEqual(1.5);
expect(data.fields[1].values[2]).toEqual(2.5);
});
});
it('calculates centered moving average when window size larger than dataset', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Centered,
field: 'x',
windowSize: 5,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.mean,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(2);
expect(data.fields[1].values[1]).toEqual(2);
expect(data.fields[1].values[2]).toEqual(2);
});
});
it('calculates trailing moving average', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Trailing,
field: 'x',
windowSize: 1,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.mean,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(1);
expect(data.fields[1].values[1]).toEqual(1.5);
expect(data.fields[1].values[2]).toEqual(2);
});
});
it('throws error when calculating moving average if window size < 1', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Trailing,
field: 'x',
windowSize: 0,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.mean,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const err = new Error('Add field from calculation transformation - Window size must be larger than 0');
expect(received[0]).toEqual(err);
});
});
it('calculates cumulative total', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.CumulativeFunctions,
cumulative: {
field: 'x',
reducer: ReducerID.sum,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(1);
expect(data.fields[1].values[1]).toEqual(3);
expect(data.fields[1].values[2]).toEqual(6);
});
});
it('calculates cumulative mean', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.CumulativeFunctions,
cumulative: {
field: 'x',
reducer: ReducerID.mean,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(1);
expect(data.fields[1].values[1]).toEqual(1.5);
expect(data.fields[1].values[2]).toEqual(2);
});
});
it('calculates cumulative total with nulls', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.CumulativeFunctions,
cumulative: {
field: 'x',
reducer: ReducerID.sum,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, null, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(1);
expect(data.fields[1].values[1]).toEqual(1);
expect(data.fields[1].values[2]).toEqual(3);
expect(data.fields[1].values[3]).toEqual(6);
});
});
it('calculates trailing moving average with nulls', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Trailing,
field: 'x',
windowSize: 0.75,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.mean,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, null, 2, 7] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(1);
expect(data.fields[1].values[1]).toEqual(1);
expect(data.fields[1].values[2]).toEqual(1.5);
expect(data.fields[1].values[3]).toEqual(4.5);
});
});
it('calculates trailing moving variance', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Trailing,
field: 'x',
windowSize: 1,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.variance,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(0);
expect(data.fields[1].values[1]).toEqual(0.25);
expect(data.fields[1].values[2]).toBeCloseTo(0.6666666, 4);
});
});
it('calculates centered moving stddev', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Centered,
field: 'x',
windowSize: 1,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.stdDev,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(0.5);
expect(data.fields[1].values[1]).toBeCloseTo(0.8164, 2);
expect(data.fields[1].values[2]).toEqual(0.5);
});
});
it('calculates centered moving stddev with null', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Centered,
field: 'x',
windowSize: 0.75,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.stdDev,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, null, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(0);
expect(data.fields[1].values[1]).toEqual(0.5);
expect(data.fields[1].values[2]).toEqual(0.5);
expect(data.fields[1].values[3]).toEqual(0.5);
});
});
it('calculates centered moving average with nulls', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Centered,
field: 'x',
windowSize: 0.75,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.mean,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, null, 2, 7] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(1);
expect(data.fields[1].values[1]).toEqual(1.5);
expect(data.fields[1].values[2]).toEqual(4.5);
expect(data.fields[1].values[3]).toEqual(4.5);
});
});
it('calculates centered moving average with only nulls', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Centered,
field: 'x',
windowSize: 0.75,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.mean,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [null, null, null, null] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(0);
expect(data.fields[1].values[1]).toEqual(0);
expect(data.fields[1].values[2]).toEqual(0);
expect(data.fields[1].values[3]).toEqual(0);
});
});
it('calculates centered moving average with 4 values', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Centered,
field: 'x',
windowSize: 0.75,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.mean,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, 2, 3, 4] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(1.5);
expect(data.fields[1].values[1]).toEqual(2);
expect(data.fields[1].values[2]).toEqual(3);
expect(data.fields[1].values[3]).toEqual(3.5);
});
});
it('calculates trailing moving variance with null in the middle', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Trailing,
field: 'x',
windowSize: 0.75,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.variance,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [1, null, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(0);
expect(data.fields[1].values[1]).toEqual(0);
expect(data.fields[1].values[2]).toEqual(0.25);
expect(data.fields[1].values[3]).toEqual(0.25);
});
});
it('calculates trailing moving variance with null in position 0', async () => {
const cfg = {
id: DataTransformerID.calculateField,
options: {
mode: CalculateFieldMode.WindowFunctions,
window: {
windowAlignment: WindowAlignment.Trailing,
field: 'x',
windowSize: 0.75,
windowSizeMode: WindowSizeMode.Percentage,
reducer: ReducerID.variance,
},
},
};
const series = toDataFrame({
fields: [{ name: 'x', type: FieldType.number, values: [null, 1, 2, 3] }],
});
await expect(transformDataFrame([cfg], [series])).toEmitValuesWith((received) => {
const data = received[0][0];
expect(data.fields.length).toEqual(2);
expect(data.fields[1].values[0]).toEqual(0);
expect(data.fields[1].values[1]).toEqual(0);
expect(data.fields[1].values[2]).toEqual(0.25);
expect(data.fields[1].values[3]).toBeCloseTo(0.6666666, 4);
});
});
});

318
packages/grafana-data/src/transformations/transformers/calculateField.ts
View File

@@ -16,17 +16,40 @@ import { noopTransformer } from './noop';
export enum CalculateFieldMode {
ReduceRow = 'reduceRow',
CumulativeFunctions = 'cumulativeFunctions',
WindowFunctions = 'windowFunctions',
BinaryOperation = 'binary',
UnaryOperation = 'unary',
Index = 'index',
}
export enum WindowSizeMode {
Percentage = 'percentage',
Fixed = 'fixed',
}
export enum WindowAlignment {
Trailing = 'trailing',
Centered = 'centered',
}
export interface ReduceOptions {
include?: string[]; // Assume all fields
reducer: ReducerID;
nullValueMode?: NullValueMode;
}
export interface CumulativeOptions {
field?: string;
reducer: ReducerID;
}
export interface WindowOptions extends CumulativeOptions {
windowSize?: number;
windowSizeMode?: WindowSizeMode;
windowAlignment?: WindowAlignment;
}
export interface UnaryOptions {
operator: UnaryOperationID;
fieldName: string;
@@ -46,6 +69,13 @@ const defaultReduceOptions: ReduceOptions = {
reducer: ReducerID.sum,
};
export const defaultWindowOptions: WindowOptions = {
reducer: ReducerID.mean,
windowAlignment: WindowAlignment.Trailing,
windowSizeMode: WindowSizeMode.Percentage,
windowSize: 0.1,
};
const defaultBinaryOptions: BinaryOptions = {
left: '',
operator: BinaryOperationID.Add,
@@ -64,6 +94,8 @@ export interface CalculateFieldTransformerOptions {
// Only one should be filled
reduce?: ReduceOptions;
window?: WindowOptions;
cumulative?: CumulativeOptions;
binary?: BinaryOptions;
unary?: UnaryOptions;
index?: IndexOptions;
@@ -104,39 +136,49 @@ export const calculateFieldTransformer: DataTransformerInfo<CalculateFieldTransf
const mode = options.mode ?? CalculateFieldMode.ReduceRow;
let creator: ValuesCreator | undefined = undefined;
if (mode === CalculateFieldMode.ReduceRow) {
creator = getReduceRowCreator(defaults(options.reduce, defaultReduceOptions), data);
} else if (mode === CalculateFieldMode.UnaryOperation) {
creator = getUnaryCreator(defaults(options.unary, defaultUnaryOptions), data);
} else if (mode === CalculateFieldMode.BinaryOperation) {
const binaryOptions = {
...options.binary,
left: ctx.interpolate(options.binary?.left!),
right: ctx.interpolate(options.binary?.right!),
};
switch (mode) {
case CalculateFieldMode.ReduceRow:
creator = getReduceRowCreator(defaults(options.reduce, defaultReduceOptions), data);
break;
case CalculateFieldMode.CumulativeFunctions:
creator = getCumulativeCreator(defaults(options.cumulative, defaultReduceOptions), data);
break;
case CalculateFieldMode.WindowFunctions:
creator = getWindowCreator(defaults(options.window, defaultWindowOptions), data);
break;
case CalculateFieldMode.UnaryOperation:
creator = getUnaryCreator(defaults(options.unary, defaultUnaryOptions), data);
break;
case CalculateFieldMode.BinaryOperation:
const binaryOptions = {
...options.binary,
left: ctx.interpolate(options.binary?.left!),
right: ctx.interpolate(options.binary?.right!),
};
creator = getBinaryCreator(defaults(binaryOptions, defaultBinaryOptions), data);
} else if (mode === CalculateFieldMode.Index) {
return data.map((frame) => {
const indexArr = [...Array(frame.length).keys()];
creator = getBinaryCreator(defaults(binaryOptions, defaultBinaryOptions), data);
break;
case CalculateFieldMode.Index:
return data.map((frame) => {
const indexArr = [...Array(frame.length).keys()];
if (options.index?.asPercentile) {
for (let i = 0; i < indexArr.length; i++) {
indexArr[i] = indexArr[i] / indexArr.length;
if (options.index?.asPercentile) {
for (let i = 0; i < indexArr.length; i++) {
indexArr[i] = indexArr[i] / indexArr.length;
}
}
}
const f = {
name: options.alias ?? 'Row',
type: FieldType.number,
values: indexArr,
config: options.index?.asPercentile ? { unit: 'percentunit' } : {},
};
return {
...frame,
fields: options.replaceFields ? [f] : [...frame.fields, f],
};
});
const f = {
name: options.alias ?? 'Row',
type: FieldType.number,
values: indexArr,
config: options.index?.asPercentile ? { unit: 'percentunit' } : {},
};
return {
...frame,
fields: options.replaceFields ? [f] : [...frame.fields, f],
};
});
}
// Nothing configured
@@ -180,6 +222,213 @@ export const calculateFieldTransformer: DataTransformerInfo<CalculateFieldTransf
},
};
function getWindowCreator(options: WindowOptions, allFrames: DataFrame[]): ValuesCreator {
if (options.windowSize! <= 0) {
throw new Error('Add field from calculation transformation - Window size must be larger than 0');
}
let matcher = getFieldMatcher({
id: FieldMatcherID.numeric,
});
if (options.field) {
matcher = getFieldMatcher({
id: FieldMatcherID.byNames,
options: {
names: [options.field],
},
});
}
return (frame: DataFrame) => {
const window = Math.ceil(
options.windowSize! * (options.windowSizeMode === WindowSizeMode.Percentage ? frame.length : 1)
);
// Find the columns that should be examined
let selectedField: Field | null = null;
for (const field of frame.fields) {
if (matcher(field, frame, allFrames)) {
selectedField = field;
break;
}
}
if (!selectedField) {
return;
}
if (![ReducerID.mean, ReducerID.stdDev, ReducerID.variance].includes(options.reducer)) {
throw new Error(`Add field from calculation transformation - Unsupported reducer: ${options.reducer}`);
}
if (options.windowAlignment === WindowAlignment.Centered) {
return getCenteredWindowValues(frame, options.reducer, selectedField, window);
} else {
return getTrailingWindowValues(frame, options.reducer, selectedField, window);
}
};
}
function getTrailingWindowValues(frame: DataFrame, reducer: ReducerID, selectedField: Field, window: number) {
const vals: number[] = [];
let sum = 0;
let count = 0;
for (let i = 0; i < frame.length; i++) {
if (reducer === ReducerID.mean) {
const currentValue = selectedField.values[i];
if (currentValue !== null) {
count++;
sum += currentValue;
if (i > window - 1) {
sum -= selectedField.values[i - window];
count--;
}
}
vals.push(count === 0 ? 0 : sum / count);
} else if (reducer === ReducerID.variance) {
const start = Math.max(0, i - window + 1);
const end = i + 1;
vals.push(calculateVariance(selectedField.values.slice(start, end)));
} else if (reducer === ReducerID.stdDev) {
const start = Math.max(0, i - window + 1);
const end = i + 1;
vals.push(calculateStdDev(selectedField.values.slice(start, end)));
}
}
return vals;
}
function getCenteredWindowValues(frame: DataFrame, reducer: ReducerID, selectedField: Field, window: number) {
const vals: number[] = [];
let sum = 0;
let count = 0;
// Current value (i) is included in the leading part of the window. Which means if the window size is odd,
// the leading part of the window will be larger than the trailing part.
const leadingPartOfWindow = Math.ceil(window / 2) - 1;
const trailingPartOfWindow = Math.floor(window / 2);
for (let i = 0; i < frame.length; i++) {
const first = i - trailingPartOfWindow;
const last = i + leadingPartOfWindow;
if (reducer === ReducerID.mean) {
if (i === 0) {
// We're at the start and need to prime the leading part of the window
for (let x = 0; x < leadingPartOfWindow + 1 && x < selectedField.values.length; x++) {
if (selectedField.values[x] !== null) {
sum += selectedField.values[x];
count++;
}
}
} else {
if (last < selectedField.values.length) {
// Last is inside the data and should be added.
if (selectedField.values[last] !== null) {
sum += selectedField.values[last];
count++;
}
}
if (first > 0) {
// Remove values that have fallen outside of the window, if the start of the window isn't outside of the data.
if (selectedField.values[first - 1] !== null) {
sum -= selectedField.values[first - 1];
count--;
}
}
}
vals.push(count === 0 ? 0 : sum / count);
} else if (reducer === ReducerID.variance) {
const windowVals = selectedField.values.slice(
Math.max(0, first),
Math.min(last + 1, selectedField.values.length)
);
vals.push(calculateVariance(windowVals));
} else if (reducer === ReducerID.stdDev) {
const windowVals = selectedField.values.slice(
Math.max(0, first),
Math.min(last + 1, selectedField.values.length)
);
vals.push(calculateStdDev(windowVals));
}
}
return vals;
}
function calculateVariance(vals: number[]): number {
if (vals.length < 1) {
return 0;
}
let squareSum = 0;
let runningMean = 0;
let nonNullCount = 0;
for (let i = 0; i < vals.length; i++) {
const currentValue = vals[i];
if (currentValue !== null) {
nonNullCount++;
let _oldMean = runningMean;
runningMean += (currentValue - _oldMean) / nonNullCount;
squareSum += (currentValue - _oldMean) * (currentValue - runningMean);
}
}
if (nonNullCount === 0) {
return 0;
}
const variance = squareSum / nonNullCount;
return variance;
}
function calculateStdDev(vals: number[]): number {
return Math.sqrt(calculateVariance(vals));
}
function getCumulativeCreator(options: CumulativeOptions, allFrames: DataFrame[]): ValuesCreator {
let matcher = getFieldMatcher({
id: FieldMatcherID.numeric,
});
if (options.field) {
matcher = getFieldMatcher({
id: FieldMatcherID.byNames,
options: {
names: [options.field],
},
});
}
if (![ReducerID.mean, ReducerID.sum].includes(options.reducer)) {
throw new Error(`Add field from calculation transformation - Unsupported reducer: ${options.reducer}`);
}
return (frame: DataFrame) => {
// Find the columns that should be examined
let selectedField: Field | null = null;
for (const field of frame.fields) {
if (matcher(field, frame, allFrames)) {
selectedField = field;
break;
}
}
if (!selectedField) {
return;
}
const vals: number[] = [];
let total = 0;
for (let i = 0; i < frame.length; i++) {
total += selectedField.values[i];
if (options.reducer === ReducerID.sum) {
vals.push(total);
} else if (options.reducer === ReducerID.mean) {
vals.push(total / (i + 1));
}
}
return vals;
};
}
function getReduceRowCreator(options: ReduceOptions, allFrames: DataFrame[]): ValuesCreator {
let matcher = getFieldMatcher({
id: FieldMatcherID.numeric,
@@ -227,6 +476,7 @@ function getReduceRowCreator(options: ReduceOptions, allFrames: DataFrame[]): Va
for (let j = 0; j < size; j++) {
row.values[j] = columns[j][i];
}
vals.push(reducer(row, ignoreNulls, nullAsZero)[options.reducer]);
}
@@ -309,6 +559,16 @@ export function getNameFromOptions(options: CalculateFieldTransformerOptions) {
}
switch (options.mode) {
case CalculateFieldMode.CumulativeFunctions: {
const { cumulative } = options;
return `cumulative ${cumulative?.reducer ?? ''}${cumulative?.field ? `(${cumulative.field})` : ''}`;
}
case CalculateFieldMode.WindowFunctions: {
const { window } = options;
return `${window?.windowAlignment ?? ''} moving ${window?.reducer ?? ''}${
window?.field ? `(${window.field})` : ''
}`;
}
case CalculateFieldMode.UnaryOperation: {
const { unary } = options;
return `${unary?.operator ?? ''}${unary?.fieldName ? `(${unary.fieldName})` : ''}`;

1
packages/grafana-data/src/types/featureToggles.gen.ts
View File

@@ -151,6 +151,7 @@ export interface FeatureToggles {
costManagementUi?: boolean;
managedPluginsInstall?: boolean;
prometheusPromQAIL?: boolean;
addFieldFromCalculationStatFunctions?: boolean;
alertmanagerRemoteSecondary?: boolean;
alertmanagerRemotePrimary?: boolean;
alertmanagerRemoteOnly?: boolean;

7
pkg/services/featuremgmt/registry.go
View File

@@ -935,6 +935,13 @@ var (
FrontendOnly: true,
Owner: grafanaObservabilityMetricsSquad,
},
{
Name: "addFieldFromCalculationStatFunctions",
Description: "Add cumulative and window functions to the add field from calculation transformation",
Stage: FeatureStageExperimental,
FrontendOnly: true,
Owner: grafanaBiSquad,
},
{
Name: "alertmanagerRemoteSecondary",
Description: "Enable Grafana to sync configuration and state with a remote Alertmanager.",

1
pkg/services/featuremgmt/toggles_gen.csv
View File

@@ -132,6 +132,7 @@ pluginsInstrumentationStatusSource,experimental,@grafana/plugins-platform-backen
costManagementUi,experimental,@grafana/databases-frontend,false,false,false,false
managedPluginsInstall,experimental,@grafana/plugins-platform-backend,false,false,false,false
prometheusPromQAIL,experimental,@grafana/observability-metrics,false,false,false,true
addFieldFromCalculationStatFunctions,experimental,@grafana/grafana-bi-squad,false,false,false,true
alertmanagerRemoteSecondary,experimental,@grafana/alerting-squad,false,false,false,false
alertmanagerRemotePrimary,experimental,@grafana/alerting-squad,false,false,false,false
alertmanagerRemoteOnly,experimental,@grafana/alerting-squad,false,false,false,false
1 Name Stage Owner requiresDevMode RequiresLicense RequiresRestart FrontendOnly
132 costManagementUi experimental @grafana/databases-frontend false false false false
133 managedPluginsInstall experimental @grafana/plugins-platform-backend false false false false
134 prometheusPromQAIL experimental @grafana/observability-metrics false false false true
135 addFieldFromCalculationStatFunctions experimental @grafana/grafana-bi-squad false false false true
136 alertmanagerRemoteSecondary experimental @grafana/alerting-squad false false false false
137 alertmanagerRemotePrimary experimental @grafana/alerting-squad false false false false
138 alertmanagerRemoteOnly experimental @grafana/alerting-squad false false false false

4
pkg/services/featuremgmt/toggles_gen.go
View File

@@ -539,6 +539,10 @@ const (
// Prometheus and AI/ML to assist users in creating a query
FlagPrometheusPromQAIL = "prometheusPromQAIL"
// FlagAddFieldFromCalculationStatFunctions
// Add cumulative and window functions to the add field from calculation transformation
FlagAddFieldFromCalculationStatFunctions = "addFieldFromCalculationStatFunctions"
// FlagAlertmanagerRemoteSecondary
// Enable Grafana to sync configuration and state with a remote Alertmanager.
FlagAlertmanagerRemoteSecondary = "alertmanagerRemoteSecondary"

20
public/app/features/transformers/docs/content.ts
View File

@@ -50,6 +50,26 @@ export const transformationDocsContent: TransformationDocsContentType = {
- **Natural logarithm (ln)** - Returns the natural logarithm of a given expression.
- **Floor (floor)** - Returns the largest integer less than or equal to a given expression.
- **Ceiling (ceil)** - Returns the smallest integer greater than or equal to a given expression.
- **Cumulative functions** - Apply functions on the current row and all preceding rows.
**Note:** This mode is an experimental feature. Engineering and on-call support is not available.
Documentation is either limited or not provided outside of code comments. No SLA is provided.
Enable the 'addFieldFromCalculationStatFunctions' in Grafana to use this feature.
Contact Grafana Support to enable this feature in Grafana Cloud.
- **Total** - Calculates the cumulative total up to and including the current row.
- **Mean** - Calculates the mean up to and including the current row.
- **Window functions** - Apply window functions. The window can either be **trailing** or **centered**.
With a trailing window the current row will be the last row in the window.
With a centered window the window will be centered on the current row.
For even window sizes, the window will be centered between the current row, and the previous row.
**Note:** This mode is an experimental feature. Engineering and on-call support is not available.
Documentation is either limited or not provided outside of code comments. No SLA is provided.
Enable the 'addFieldFromCalculationStatFunctions' in Grafana to use this feature.
Contact Grafana Support to enable this feature in Grafana Cloud.
- **Mean** - Calculates the moving mean or running average.
- **Stddev** - Calculates the moving standard deviation.
- **Variance** - Calculates the moving variance.
- **Row index -** Insert a field with the row index.
- **Field name -** Select the names of fields you want to use in the calculation for the new field.
- **Calculation -** If you select **Reduce row** mode, then the **Calculation** field appears. Click in the field to see a list of calculation choices you can use to create the new field. For information about available calculations, refer to [Calculation types][].

223
public/app/features/transformers/editors/CalculateFieldTransformerEditor.tsx
View File

@@ -24,10 +24,15 @@ import {
BinaryOptions,
UnaryOptions,
CalculateFieldMode,
WindowAlignment,
CalculateFieldTransformerOptions,
getNameFromOptions,
IndexOptions,
ReduceOptions,
CumulativeOptions,
WindowOptions,
WindowSizeMode,
defaultWindowOptions,
} from '@grafana/data/src/transformations/transformers/calculateField';
import { getTemplateSrv, config as cfg } from '@grafana/runtime';
import {
@@ -38,9 +43,11 @@ import {
InlineLabel,
InlineSwitch,
Input,
RadioButtonGroup,
Select,
StatsPicker,
} from '@grafana/ui';
import { NumberInput } from 'app/core/components/OptionsUI/NumberInput';
interface CalculateFieldTransformerEditorProps extends TransformerUIProps<CalculateFieldTransformerOptions> {}
@@ -57,6 +64,13 @@ const calculationModes = [
{ value: CalculateFieldMode.Index, label: 'Row index' },
];
if (cfg.featureToggles.addFieldFromCalculationStatFunctions) {
calculationModes.push(
{ value: CalculateFieldMode.CumulativeFunctions, label: 'Cumulative functions' },
{ value: CalculateFieldMode.WindowFunctions, label: 'Window functions' }
);
}
const okTypes = new Set<FieldType>([FieldType.time, FieldType.number, FieldType.string]);
const labelWidth = 16;
@@ -188,6 +202,9 @@ export class CalculateFieldTransformerEditor extends React.PureComponent<
onModeChanged = (value: SelectableValue<CalculateFieldMode>) => {
const { options, onChange } = this.props;
const mode = value.value ?? CalculateFieldMode.BinaryOperation;
if (mode === CalculateFieldMode.WindowFunctions) {
options.window = options.window ?? defaultWindowOptions;
}
onChange({
...options,
mode,
@@ -203,14 +220,13 @@ export class CalculateFieldTransformerEditor extends React.PureComponent<
};
//---------------------------------------------------------
// Reduce by Row
// Cumulative functions
//---------------------------------------------------------
updateReduceOptions = (v: ReduceOptions) => {
const { options, onChange } = this.props;
onChange({
...options,
mode: CalculateFieldMode.ReduceRow,
reduce: v,
});
};
@@ -250,6 +266,149 @@ export class CalculateFieldTransformerEditor extends React.PureComponent<
);
}
//---------------------------------------------------------
// Window functions
//---------------------------------------------------------
updateWindowOptions = (v: WindowOptions) => {
const { options, onChange } = this.props;
onChange({
...options,
mode: CalculateFieldMode.WindowFunctions,
window: v,
});
};
onWindowFieldChange = (v: SelectableValue<string>) => {
const { window } = this.props.options;
this.updateWindowOptions({
...window!,
field: v.value!,
});
};
onWindowSizeChange = (v?: number) => {
const { window } = this.props.options;
this.updateWindowOptions({
...window!,
windowSize: v && window?.windowSizeMode === WindowSizeMode.Percentage ? v / 100 : v,
});
};
onWindowSizeModeChange = (val: string) => {
const { window } = this.props.options;
const mode = val as WindowSizeMode;
this.updateWindowOptions({
...window!,
windowSize: window?.windowSize
? mode === WindowSizeMode.Percentage
? window!.windowSize! / 100
: window!.windowSize! * 100
: undefined,
windowSizeMode: mode,
});
};
onWindowStatsChange = (stats: string[]) => {
const reducer = stats.length ? (stats[0] as ReducerID) : ReducerID.sum;
const { window } = this.props.options;
this.updateWindowOptions({ ...window, reducer });
};
onTypeChange = (val: string) => {
const { window } = this.props.options;
this.updateWindowOptions({
...window!,
windowAlignment: val as WindowAlignment,
});
};
renderWindowFunctions(options?: WindowOptions) {
const { names } = this.state;
options = defaults(options, { reducer: ReducerID.sum });
const selectOptions = names.map((v) => ({ label: v, value: v }));
const typeOptions = [
{ label: 'Trailing', value: WindowAlignment.Trailing },
{ label: 'Centered', value: WindowAlignment.Centered },
];
const windowSizeModeOptions = [
{ label: 'Percentage', value: WindowSizeMode.Percentage },
{ label: 'Fixed', value: WindowSizeMode.Fixed },
];
return (
<>
<InlineField label="Field" labelWidth={labelWidth}>
<Select
placeholder="Field"
options={selectOptions}
className="min-width-18"
value={options?.field}
onChange={this.onWindowFieldChange}
/>
</InlineField>
<InlineField label="Calculation" labelWidth={labelWidth}>
<StatsPicker
allowMultiple={false}
className="width-18"
stats={[options.reducer]}
onChange={this.onWindowStatsChange}
defaultStat={ReducerID.mean}
filterOptions={(ext) =>
ext.id === ReducerID.mean || ext.id === ReducerID.variance || ext.id === ReducerID.stdDev
}
/>
</InlineField>
<InlineField label="Type" labelWidth={labelWidth}>
<RadioButtonGroup
value={options.windowAlignment ?? WindowAlignment.Trailing}
options={typeOptions}
onChange={this.onTypeChange}
/>
</InlineField>
<InlineField label="Window size mode">
<RadioButtonGroup
value={options.windowSizeMode ?? WindowSizeMode.Percentage}
options={windowSizeModeOptions}
onChange={this.onWindowSizeModeChange}
></RadioButtonGroup>
</InlineField>
<InlineField
label={options.windowSizeMode === WindowSizeMode.Percentage ? 'Window size %' : 'Window size'}
labelWidth={labelWidth}
tooltip={
options.windowSizeMode === WindowSizeMode.Percentage
? 'Set the window size as a percentage of the total data'
: 'Window size'
}
>
<NumberInput
placeholder="Auto"
min={0.1}
value={
options.windowSize && options.windowSizeMode === WindowSizeMode.Percentage
? options.windowSize * 100
: options.windowSize
}
onChange={this.onWindowSizeChange}
></NumberInput>
</InlineField>
</>
);
}
//---------------------------------------------------------
// Reduce by Row
//---------------------------------------------------------
onReducerStatsChange = (stats: string[]) => {
const reducer = stats.length ? (stats[0] as ReducerID) : ReducerID.sum;
const { reduce } = this.props.options;
this.updateReduceOptions({ ...reduce, reducer });
};
renderReduceRow(options?: ReduceOptions) {
const { names, selected } = this.state;
options = defaults(options, { reducer: ReducerID.sum });
@@ -285,6 +444,64 @@ export class CalculateFieldTransformerEditor extends React.PureComponent<
);
}
//---------------------------------------------------------
// Cumulative Operator
//---------------------------------------------------------
onCumulativeStatsChange = (stats: string[]) => {
const reducer = stats.length ? (stats[0] as ReducerID) : ReducerID.sum;
const { reduce } = this.props.options;
this.updateCumulativeOptions({ ...reduce, reducer });
};
updateCumulativeOptions = (v: CumulativeOptions) => {
const { options, onChange } = this.props;
onChange({
...options,
mode: CalculateFieldMode.CumulativeFunctions,
cumulative: v,
});
};
onCumulativeFieldChange = (v: SelectableValue<string>) => {
const { cumulative } = this.props.options;
this.updateCumulativeOptions({
...cumulative!,
field: v.value!,
});
};
renderCumulativeFunctions(options?: CumulativeOptions) {
const { names } = this.state;
options = defaults(options, { reducer: ReducerID.sum });
const selectOptions = names.map((v) => ({ label: v, value: v }));
return (
<>
<InlineField label="Field" labelWidth={labelWidth}>
<Select
placeholder="Field"
options={selectOptions}
className="min-width-18"
value={options?.field}
onChange={this.onCumulativeFieldChange}
/>
</InlineField>
<InlineField label="Calculation" labelWidth={labelWidth}>
<StatsPicker
allowMultiple={false}
className="width-18"
stats={[options.reducer]}
onChange={this.onCumulativeStatsChange}
defaultStat={ReducerID.sum}
filterOptions={(ext) => ext.id === ReducerID.sum || ext.id === ReducerID.mean}
/>
</InlineField>
</>
);
}
//---------------------------------------------------------
// Binary Operator
//---------------------------------------------------------
@@ -468,6 +685,8 @@ export class CalculateFieldTransformerEditor extends React.PureComponent<
{mode === CalculateFieldMode.BinaryOperation && this.renderBinaryOperation(options.binary)}
{mode === CalculateFieldMode.UnaryOperation && this.renderUnaryOperation(options.unary)}
{mode === CalculateFieldMode.ReduceRow && this.renderReduceRow(options.reduce)}
{mode === CalculateFieldMode.CumulativeFunctions && this.renderCumulativeFunctions(options.cumulative)}
{mode === CalculateFieldMode.WindowFunctions && this.renderWindowFunctions(options.window)}
{mode === CalculateFieldMode.Index && this.renderRowIndex(options.index)}
<InlineField labelWidth={labelWidth} label="Alias">
<Input