QueryProcessing: Observable query interface and RxJS for query & stream processing (#18899)

* I needed to learn some rxjs and understand this more, so just playing around

* Updated

* Removed all the complete calls

* Refactoring

* StreamHandler -> observable start

* progress

* simple singal works

* Handle update time range

* added error handling

* wrap old function

* minor changes

* handle data format in the subscribe function

* Use replay subject to return last value to subscribers

* Set loading state after no response in 50ms

* added missing file

* updated comment

* Added cancelation of network requests

* runRequest: Added unit test scenario framework

* Progress on tests

* minor refactor of unit tests

* updated test

* removed some old code

* Shared queries work again, and also became so much simplier

* unified query and observe methods

* implict any fix

* Fixed closed subject issue

* removed comment

* Use last returned data for loading state

* WIP: Explore to runRequest makover step1

* Minor progress

* Minor progress on explore and runRequest

* minor progress

* Things are starting to work in explore

* Updated prometheus to use new observable query response, greatly simplified code

* Revert refId change

* Found better solution for key/refId/requestId problem

* use observable with loki

* tests compile

* fix loki query prep

* Explore: correct first response handling

* Refactorings

* Refactoring

* Explore: Fixes LoadingState and GraphResults between runs (#18986)

* Refactor: Adds state to DataQueryResponse

* Fix: Fixes so we do not empty results before new data arrives
Fixes: #17409

* Transformations work

* observable test data

* remove single() from loki promise

* Fixed comment

* Explore: Fixes failing Loki and Prometheus unit tests (#18995)

* Tests: Makes datasource tests work again

* Fix: Fixes loki datasource so highligthing works

* Chore: Runs Prettier

* Fixed query runner tests

* Delay loading state indication to 200ms

* Fixed test

* fixed unit tests

* Clear cached calcs

* Fixed bug getProcesedDataFrames

* Fix the correct test is a better idea

* Fix: Fixes so queries in Explore are only run if Graph/Table is shown (#19000)

* Fix: Fixes so queries in Explore are only run if Graph/Table is shown
Fixes: #18618

* Refactor: Removes unnecessary condition

* PanelData: provide legacy data only when needed  (#19018)

* no legacy

* invert logic... now compiles

* merge getQueryResponseData and getDataRaw

* update comment about query editor

* use single getData() function

* only send legacy when it is used in explore

* pre process rather than post process

* pre process rather than post process

* Minor refactoring

* Add missing tags to test datasource response

* MixedDatasource: Adds query observable pattern to MixedDatasource (#19037)

* start mixed datasource

* Refactor: Refactors into observable parttern

* Tests: Fixes tests

* Tests: Removes console.log

* Refactor: Adds unique requestId

packages/grafana-data/src/utils/dataFrameHelper.ts

@@ -393,8 +393,8 @@ export class CircularDataFrame<T = any> extends MutableDataFrame<T> {
   constructor(options: CircularOptions) {
     super(undefined, (buffer?: any[]) => {
       return new CircularVector({
-        buffer,
         ...options,
+        buffer,
       });
     });
   }

packages/grafana-data/src/utils/fieldReducer.ts

@@ -101,6 +101,7 @@ export function reduceField(options: ReduceFieldOptions): FieldCalcs {
 
   // For now everything can use the standard stats
   let values = doStandardCalcs(field, ignoreNulls, nullAsZero);
+
   for (const reducer of queue) {
     if (!values.hasOwnProperty(reducer.id) && reducer.reduce) {
       values = {
@@ -109,10 +110,12 @@ export function reduceField(options: ReduceFieldOptions): FieldCalcs {
       };
     }
   }
+
   field.calcs = {
     ...field.calcs,
     ...values,
   };
+
   return values;
 }
 

packages/grafana-data/src/utils/processDataFrame.ts

@@ -238,16 +238,19 @@ export const toDataFrame = (data: any): DataFrame => {
     // This will convert the array values into Vectors
     return new MutableDataFrame(data as DataFrameDTO);
   }
+
   if (data.hasOwnProperty('datapoints')) {
     return convertTimeSeriesToDataFrame(data);
   }
+
   if (data.hasOwnProperty('data')) {
     return convertGraphSeriesToDataFrame(data);
   }
+
   if (data.hasOwnProperty('columns')) {
     return convertTableToDataFrame(data);
   }
-  // TODO, try to convert JSON/Array to table?
+
   console.warn('Can not convert', data);
   throw new Error('Unsupported data format');
 };