Transformations: Fix true inner join in joinByField transformation (#87409)

* baldm0mma/inner_join_fix/ init commit

* baldm0mma/inner_join_fix/ first attempt at an tabular inner join

* baldm0mma/inner_join_fix/ add innerTabular

* baldm0mma/inner_join_fix/ add innerJoin option to editor

* baldm0mma/inner_join_fix/ add editor option

* baldm0mma/inner_join_fix/ update joinInnerTabular function behavior

* baldm0mma/inner_join_fix/ add js docs

* baldm0mma/inner_join_fix/ update jsdocs

* baldm0mma/inner_join_fix/ update docs

* baldm0mma/inner_join_fix/ remove unused console.logs

* baldm0mma/inner_join_fix/ update tests

* baldm0mma/inner_join_fix/ simplify getValue

* baldm0mma/inner_join_fix/ update tests

* baldm0mma/inner_join_fix/ update docs builder

* baldm0mma/inner_join_fix/ add tables to Outer join (for tabular data)

* baldm0mma/inner_join_fix/ update docs

* baldm0mma/inner_join_fix/ build docs

* baldm0mma/inner_join_fix/ remove innertab for inner

* baldm0mma/inner_join_fix/ rename innertab

* baldm0mma/inner_join_fix/ update tests

* baldm0mma/inner_join_fix/ rem con logs

* baldm0mma/inner_join_fix/ rem comment

* baldm0mma/inner_join_fix/ rem sample data

* baldm0mma/inner_join_fix/ remove irrelevant test

* baldm0mma/inner_join_fix/ update docs

* Update packages/grafana-data/src/transformations/transformers/joinDataFrames.test.ts

Co-authored-by: Nathan Marrs  <nathanielmarrs@gmail.com>

* baldm0mma/inner_join_fix/ update test description

---------

Co-authored-by: nmarrs <nathanielmarrs@gmail.com>

docs/sources/panels-visualizations/query-transform-data/transform-data/index.md

@@ -39,7 +39,7 @@ labels:
     - enterprise
     - oss
 title: Transform data
-description: Use transformations to rename fields, join series data, apply mathematical operations, and more
+description: Use transformations to rename fields, join time series/SQL-like data, apply mathematical operations, and more
 weight: 100
 ---
 
@@ -48,7 +48,7 @@ weight: 100
 Transformations are a powerful way to manipulate data returned by a query before the system applies a visualization. Using transformations, you can:
 
 - Rename fields
-- Join time series data
+- Join time series/SQL-like data
 - Perform mathematical operations across queries
 - Use the output of one transformation as the input to another transformation
 
@@ -761,13 +761,13 @@ Use this transformation to merge multiple results into a single table, enabling
 
 This is especially useful for converting multiple time series results into a single wide table with a shared time field.
 
-#### Inner join
+#### Inner join (for Time Series or SQL-like data)
 
 An inner join merges data from multiple tables where all tables share the same value from the selected field. This type of join excludes data where values do not match in every result.
 
-Use this transformation to combine the results from multiple queries (combining on a passed join field or the first time column) into one result, and drop rows where a successful join cannot occur.
+Use this transformation to combine the results from multiple queries (combining on a passed join field or the first time column) into one result, and drop rows where a successful join cannot occur. This is not optimized for large Time Series datasets.
 
-In the following example, two queries return table data. It is visualized as two separate tables before applying the inner join transformation.
+In the following example, two queries return Time Series data. It is visualized as two separate tables before applying the inner join transformation.
 
 **Query A:**
 
@@ -792,7 +792,39 @@ The result after applying the inner join transformation looks like the following
 | 2020-07-07 11:34:20 | node    | 25260122  | server 1 | 15     |
 | 2020-07-07 11:24:20 | postgre | 123001233 | server 2 | 5      |
 
-#### Outer join
+This works in the same way for non-Time Series tabular data as well.
+
+**Students**
+
+| StudentID | Name     | Major            |
+| --------- | -------- | ---------------- |
+| 1         | John     | Computer Science |
+| 2         | Emily    | Mathematics      |
+| 3         | Michael  | Physics          |
+| 4         | Jennifer | Chemistry        |
+
+**Enrollments**
+
+| StudentID | CourseID | Grade |
+| --------- | -------- | ----- |
+| 1         | CS101    | A     |
+| 1         | CS102    | B     |
+| 2         | MATH201  | A     |
+| 3         | PHYS101  | B     |
+| 5         | HIST101  | B     |
+
+The result after applying the inner join transformation looks like the following:
+
+| StudentID | Name    | Major            | CourseID | Grade |
+| --------- | ------- | ---------------- | -------- | ----- |
+| 1         | John    | Computer Science | CS101    | A     |
+| 1         | John    | Computer Science | CS102    | B     |
+| 2         | Emily   | Mathematics      | MATH201  | A     |
+| 3         | Michael | Physics          | PHYS101  | B     |
+
+The inner join only includes rows where there is a match between the "StudentID" in both tables. In this case, the result does not include "Jennifer" from the "Students" table because there are no matching enrollments for her in the "Enrollments" table.
+
+#### Outer join (for Time Series data)
 
 An outer join includes all data from an inner join and rows where values do not match in every input. While the inner join joins Query A and Query B on the time field, the outer join includes all rows that don't match on the time field.
 
@@ -831,6 +863,38 @@ I applied a transformation to join the query results using the time field. Now I
 
 {{< figure src="/static/img/docs/transformations/join-fields-after-7-0.png" class="docs-image--no-shadow" max-width= "1100px" alt="A table visualization showing results for multiple servers" >}}
 
+#### Outer join (for SQL-like data)
+
+A tabular outer join combining tables so that the result includes matched and unmatched rows from either or both tables.
+
+| StudentID | Name     | Major            |
+| --------- | -------- | ---------------- |
+| 1         | John     | Computer Science |
+| 2         | Emily    | Mathematics      |
+| 3         | Michael  | Physics          |
+| 4         | Jennifer | Chemistry        |
+
+Can now be joined with:
+
+| StudentID | CourseID | Grade |
+| --------- | -------- | ----- |
+| 1         | CS101    | A     |
+| 1         | CS102    | B     |
+| 2         | MATH201  | A     |
+| 3         | PHYS101  | B     |
+| 5         | HIST101  | B     |
+
+The result after applying the outer join transformation looks like the following:
+
+| StudentID | Name     | Major            | CourseID | Grade |
+| --------- | -------- | ---------------- | -------- | ----- |
+| 1         | John     | Computer Science | CS101    | A     |
+| 1         | John     | Computer Science | CS102    | B     |
+| 2         | Emily    | Mathematics      | MATH201  | A     |
+| 3         | Michael  | Physics          | PHYS101  | B     |
+| 4         | Jennifer | Chemistry        | NULL     | NULL  |
+| 5         | NULL     | NULL             | HIST101  | B     |
+
 Combine and analyze data from various queries with table joining for a comprehensive view of your information.
 
 ### Join by labels

packages/grafana-data/src/transformations/transformers/joinByField.test.ts

@@ -761,93 +761,6 @@ describe('JOIN Transformer', () => {
       });
     });
 
-    it('inner joins by time field in reverse order', async () => {
-      const cfg: DataTransformerConfig<JoinByFieldOptions> = {
-        id: DataTransformerID.seriesToColumns,
-        options: {
-          byField: 'time',
-          mode: JoinMode.inner,
-        },
-      };
-
-      seriesA.fields[0].values = seriesA.fields[0].values.reverse();
-      seriesA.fields[1].values = seriesA.fields[1].values.reverse();
-      seriesA.fields[2].values = seriesA.fields[2].values.reverse();
-
-      await expect(transformDataFrame([cfg], [seriesA, seriesB])).toEmitValuesWith((received) => {
-        const data = received[0];
-        const filtered = data[0];
-        expect(filtered.fields).toMatchInlineSnapshot(`
-          [
-            {
-              "config": {},
-              "name": "time",
-              "state": {
-                "multipleFrames": true,
-              },
-              "type": "time",
-              "values": [
-                3000,
-                5000,
-              ],
-            },
-            {
-              "config": {},
-              "labels": {
-                "name": "A",
-              },
-              "name": "temperature",
-              "state": {},
-              "type": "number",
-              "values": [
-                10.3,
-                10.5,
-              ],
-            },
-            {
-              "config": {},
-              "labels": {
-                "name": "A",
-              },
-              "name": "humidity",
-              "state": {},
-              "type": "number",
-              "values": [
-                10000.3,
-                10000.5,
-              ],
-            },
-            {
-              "config": {},
-              "labels": {
-                "name": "B",
-              },
-              "name": "temperature",
-              "state": {},
-              "type": "number",
-              "values": [
-                10.3,
-                10.5,
-              ],
-            },
-            {
-              "config": {},
-              "labels": {
-                "name": "B",
-              },
-              "name": "humidity",
-              "state": {},
-              "type": "number",
-              "values": [
-                10000.3,
-                10000.5,
-              ],
-            },
-          ]
-        `);
-      });
-    });
-
     describe('Field names', () => {
       const seriesWithSameFieldAndDataFrameName = toDataFrame({
         name: 'temperature',

packages/grafana-data/src/transformations/transformers/joinDataFrames.test.ts

@@ -32,7 +32,7 @@ describe('align frames', () => {
 
     // the following does not work for tabular joins where the joined on field value is duplicated
     // the time will never have a duplicated time which is joined on
-    it('should perform an outer join', () => {
+    it('should perform an outer join - as expected on time series data', () => {
       const out = joinDataFrames({ frames: [series1, series2] })!;
       expect(
         out.fields.map((f) => ({
@@ -85,7 +85,7 @@ describe('align frames', () => {
       `);
     });
 
-    it('should perform an inner join', () => {
+    it('should perform an inner join - as expected on time series data', () => {
       const out = joinDataFrames({ frames: [series1, series2], mode: JoinMode.inner })!;
       expect(
         out.fields.map((f) => ({
@@ -139,7 +139,11 @@ describe('align frames', () => {
 
     const tableData1 = toDataFrame({
       fields: [
-        { name: 'gender', type: FieldType.string, values: ['MALE', 'MALE', 'MALE', 'FEMALE', 'FEMALE', 'FEMALE'] },
+        {
+          name: 'gender',
+          type: FieldType.string,
+          values: ['NON-BINARY', 'MALE', 'MALE', 'FEMALE', 'FEMALE', 'NON-BINARY'],
+        },
         {
           name: 'day',
           type: FieldType.string,
@@ -150,12 +154,12 @@ describe('align frames', () => {
     });
     const tableData2 = toDataFrame({
       fields: [
-        { name: 'gender', type: FieldType.string, values: ['MALE', 'FEMALE'] },
+        { name: 'gender', type: FieldType.string, values: ['MALE', 'NON-BINARY', 'FEMALE'] },
-        { name: 'count', type: FieldType.number, values: [103, 95] },
+        { name: 'count', type: FieldType.number, values: [103, 95, 201] },
       ],
     });
 
-    it('should perform an outer join with duplicated values to join on', () => {
+    it('should perform an outer join with duplicated values to join on - as expected for tabular data', () => {
       const out = joinDataFrames({
         frames: [tableData1, tableData2],
         joinBy: fieldMatchers.get(FieldMatcherID.byName).get('gender'),
@@ -171,12 +175,12 @@ describe('align frames', () => {
           {
             "name": "gender",
             "values": [
-              "MALE",
+              "NON-BINARY",
               "MALE",
               "MALE",
               "FEMALE",
               "FEMALE",
-              "FEMALE",
+              "NON-BINARY",
             ],
           },
           {
@@ -204,11 +208,72 @@ describe('align frames', () => {
           {
             "name": "count",
             "values": [
-              103,
-              103,
-              103,
               95,
+              103,
+              103,
+              201,
+              201,
               95,
+            ],
+          },
+        ]
+      `);
+    });
+
+    it('should perform an inner join with duplicated values to join on - as expected for tabular data', () => {
+      const out = joinDataFrames({
+        frames: [tableData1, tableData2],
+        joinBy: fieldMatchers.get(FieldMatcherID.byName).get('gender'),
+        mode: JoinMode.inner,
+      })!;
+      expect(
+        out.fields.map((f) => ({
+          name: f.name,
+          values: f.values,
+        }))
+      ).toMatchInlineSnapshot(`
+        [
+          {
+            "name": "gender",
+            "values": [
+              "NON-BINARY",
+              "MALE",
+              "MALE",
+              "FEMALE",
+              "FEMALE",
+              "NON-BINARY",
+            ],
+          },
+          {
+            "name": "day",
+            "values": [
+              "Wednesday",
+              "Tuesday",
+              "Monday",
+              "Wednesday",
+              "Tuesday",
+              "Monday",
+            ],
+          },
+          {
+            "name": "count",
+            "values": [
+              18,
+              72,
+              13,
+              17,
+              71,
+              7,
+            ],
+          },
+          {
+            "name": "count",
+            "values": [
+              95,
+              103,
+              103,
+              201,
+              201,
               95,
             ],
           },

packages/grafana-data/src/transformations/transformers/joinDataFrames.ts

@@ -1,5 +1,3 @@
-import intersect from 'fast_array_intersect';
-
 import { getTimeField, sortDataFrame } from '../../dataframe';
 import { DataFrame, Field, FieldMatcher, FieldType, TIME_SERIES_VALUE_FIELD_NAME } from '../../types';
 import { fieldMatchers } from '../matchers';
@@ -256,6 +254,8 @@ export function joinDataFrames(options: JoinOptions): DataFrame | undefined {
 
   if (options.mode === JoinMode.outerTabular) {
     joined = joinOuterTabular(allData, originalFieldsOrderByFrame, originalFields.length, nullModes);
+  } else if (options.mode === JoinMode.inner) {
+    joined = joinInner(allData);
   } else {
     joined = join(allData, nullModes, options.mode);
   }
@@ -362,6 +362,74 @@ function joinOuterTabular(
   return data;
 }
 
+/**
+ * This function performs a sql-style inner join on tabular data;
+ * it will combine records from two tables whenever there are matching
+ * values in a field common to both tables.
+ *
+ * NOTE: This function implicitly assumes that the first array in each AlignedData
+ * contains the values to join on. It doesn't explicitly specify a column field to join on,
+ * but rather uses the 0th position of the arrays (AlignedData[0]) to determine the joining keys.
+ * Then, when processing the tables, the function iterates over the values in the `xValues`
+ * (the joining keys) array and checks if the current row `currentRow` already includes the value.
+ * If a matching value is found, it joins the corresponding values from the remaining arrays `yValues`
+ * (all other non-joining key arrays) to create a new row in the joined table.
+ *
+ * @param {AlignedData[]} tables - The tables to join.
+ *
+ * @returns {Array<Array<string | number | null | undefined>>} The joined tables as an array of arrays, where each array represents a row in the joined table.
+ */
+function joinInner(tables: AlignedData[]): Array<Array<string | number | null | undefined>> {
+  const joinedTables: Array<Array<string | number | null | undefined>> = [];
+
+  // Recursive function to perform the inner join.
+  const joinTables = (
+    currentTables: AlignedData[],
+    currentIndex: number,
+    currentRow: Array<string | number | null | undefined>
+  ) => {
+    if (currentIndex === currentTables.length) {
+      // Base case: all tables have been joined, add the current row to the final result.
+      joinedTables.push(currentRow);
+      return;
+    }
+
+    const currentTable = currentTables[currentIndex];
+    const [xValues, ...yValues] = currentTable;
+
+    for (let i = 0; i < xValues.length; i++) {
+      const value = xValues[i];
+
+      if (currentIndex === 0 || currentRow.includes(value)) {
+        const newRow = [...currentRow];
+
+        if (currentIndex === 0) {
+          newRow.push(value);
+        }
+
+        for (let j = 0; j < yValues.length; j++) {
+          newRow.push(yValues[j][i]);
+        }
+
+        // Recursive call for the next table
+        joinTables(currentTables, currentIndex + 1, newRow);
+      }
+    }
+  };
+
+  // Start the recursive join process.
+  joinTables(tables, 0, []);
+
+  // Check if joinedTables is empty before transposing. No need to transpose if there are no joined tables.
+  if (joinedTables.length === 0) {
+    return [];
+  }
+
+  // Transpose the joined tables to get the desired output format.
+  // This essentially flips the rows and columns back to the stucture of the original `tables`.
+  return joinedTables[0].map((_, colIndex) => joinedTables.map((row) => row[colIndex]));
+}
+
 //--------------------------------------------------------------------------------
 // Below here is copied from uplot (MIT License)
 // https://github.com/leeoniya/uPlot/blob/master/src/utils.js#L325
@@ -412,13 +480,7 @@ function nullExpand(yVals: Array<number | null>, nullIdxs: number[], alignedLen:
 
 // nullModes is a tables-matched array indicating how to treat nulls in each series
 export function join(tables: AlignedData[], nullModes?: number[][], mode: JoinMode = JoinMode.outer) {
-  let xVals: Set<number>;
+  let xVals: Set<number> = new Set();
-
-  if (mode === JoinMode.inner) {
-    // @ts-ignore
-    xVals = new Set(intersect(tables.map((t) => t[0])));
-  } else {
-    xVals = new Set();
 
   for (let ti = 0; ti < tables.length; ti++) {
     let t = tables[ti];
@@ -429,7 +491,6 @@ export function join(tables: AlignedData[], nullModes?: number[][], mode: JoinMo
       xVals.add(xs[i]);
     }
   }
-  }
 
   let data = [Array.from(xVals).sort((a, b) => a - b)];
 

public/app/features/transformers/docs/content.ts

@@ -788,13 +788,13 @@ Use this transformation to merge multiple results into a single table, enabling
 
 This is especially useful for converting multiple time series results into a single wide table with a shared time field.
 
-#### Inner join
+#### Inner join (for Time Series or SQL-like data)
 
 An inner join merges data from multiple tables where all tables share the same value from the selected field. This type of join excludes data where values do not match in every result.
 
-Use this transformation to combine the results from multiple queries (combining on a passed join field or the first time column) into one result, and drop rows where a successful join cannot occur.
+Use this transformation to combine the results from multiple queries (combining on a passed join field or the first time column) into one result, and drop rows where a successful join cannot occur. This is not optimized for large Time Series datasets.
 
-In the following example, two queries return table data. It is visualized as two separate tables before applying the inner join transformation.
+In the following example, two queries return Time Series data. It is visualized as two separate tables before applying the inner join transformation.
 
 **Query A:**
 
@@ -819,7 +819,39 @@ The result after applying the inner join transformation looks like the following
 | 2020-07-07 11:34:20 | node    | 25260122  | server 1 | 15     |
 | 2020-07-07 11:24:20 | postgre | 123001233 | server 2 | 5      |
 
-#### Outer join
+This works in the same way for non-Time Series tabular data as well.
+
+**Students**
+
+| StudentID | Name     | Major            |
+| --------- | -------- | ---------------- |
+| 1         | John     | Computer Science |
+| 2         | Emily    | Mathematics      |
+| 3         | Michael  | Physics          |
+| 4         | Jennifer | Chemistry        |
+
+**Enrollments**
+
+| StudentID | CourseID | Grade |
+|-----------|----------|-------|
+| 1         | CS101    | A     |
+| 1         | CS102    | B     |
+| 2         | MATH201  | A     |
+| 3         | PHYS101  | B     |
+| 5         | HIST101  | B     |
+
+The result after applying the inner join transformation looks like the following:
+
+| StudentID | Name    | Major            | CourseID | Grade |
+| --------- | ------- | ---------------- | -------  | ----- |
+| 1         | John    | Computer Science | CS101    | A     |
+| 1         | John    | Computer Science | CS102    | B     |
+| 2         | Emily   | Mathematics      | MATH201  | A     |
+| 3         | Michael | Physics          | PHYS101  | B     |
+
+The inner join only includes rows where there is a match between the "StudentID" in both tables. In this case, the result does not include "Jennifer" from the "Students" table because there are no matching enrollments for her in the "Enrollments" table.
+
+#### Outer join (for Time Series data)
 
 An outer join includes all data from an inner join and rows where values do not match in every input. While the inner join joins Query A and Query B on the time field, the outer join includes all rows that don't match on the time field.
 
@@ -866,6 +898,38 @@ ${buildImageContent(
   'A table visualization showing results for multiple servers'
 )}
 
+#### Outer join (for SQL-like data)
+
+A tabular outer join combining tables so that the result includes matched and unmatched rows from either or both tables.
+
+| StudentID | Name      | Major            |
+| --------- | --------- | ---------------- |
+| 1         | John      | Computer Science |
+| 2         | Emily     | Mathematics      |
+| 3         | Michael   | Physics          |
+| 4         | Jennifer  | Chemistry        |
+
+Can now be joined with:
+
+| StudentID | CourseID | Grade |
+| --------- | -------- | ----- |
+| 1         | CS101    | A     |
+| 1         | CS102    | B     |
+| 2         | MATH201  | A     |
+| 3         | PHYS101  | B     |
+| 5         | HIST101  | B     |
+
+The result after applying the outer join transformation looks like the following:
+
+| StudentID | Name     | Major            | CourseID | Grade |
+| --------- | -------- | ---------------- | -------- | ----- |
+| 1         | John     | Computer Science | CS101    | A     |
+| 1         | John     | Computer Science | CS102    | B     |
+| 2         | Emily    | Mathematics      | MATH201  | A     |
+| 3         | Michael  | Physics          | PHYS101  | B     |
+| 4         | Jennifer | Chemistry        | NULL     | NULL  |
+| 5         | NULL     | NULL             | HIST101  | B     |
+
 Combine and analyze data from various queries with table joining for a comprehensive view of your information.
   `;
     },

public/app/features/transformers/editors/JoinByFieldTransformerEditor.tsx

@@ -28,7 +28,11 @@ const modes = [
     description:
       'Join on a field value with duplicated values. Non performant outer join best used for tabular(SQL like) data.',
   },
-  { value: JoinMode.inner, label: 'INNER', description: 'Drop rows that do not match a value in all tables.' },
+  {
+    value: JoinMode.inner,
+    label: 'INNER',
+    description: 'Combine data from two tables whenever there are matching values in a fields common to both tables.',
+  },
 ];
 
 export function SeriesToFieldsTransformerEditor({ input, options, onChange }: TransformerUIProps<JoinByFieldOptions>) {

scripts/docs/generate-transformations.ts

@@ -53,7 +53,7 @@ labels:
     - enterprise
     - oss
 title: Transform data
-description: Use transformations to rename fields, join series data, apply mathematical operations, and more
+description: Use transformations to rename fields, join time series/SQL-like data, apply mathematical operations, and more
 weight: 100
 ---`;
 
@@ -62,7 +62,7 @@ const templateIntroContent = `# Transform data
 Transformations are a powerful way to manipulate data returned by a query before the system applies a visualization. Using transformations, you can:
 
 - Rename fields
-- Join time series data
+- Join time series/SQL-like data
 - Perform mathematical operations across queries
 - Use the output of one transformation as the input to another transformation