* Elasticsearch: Fix processing of percentiles on backend
* Remove redundant logic as we do this in addMetricValue
* Refactor for less cyclomatic complexity
* Remove changes in createFieldsFromPropKeys
* Elasticsearch: Add highlight for logs
* Fix running of queries trough backend only when shouldRunTroughBackend
* Fix incorrect json in test string
* Address feedback
* Keep order of words same
* Elasticsearch: Use displayName field for naming
* Change solution to frame.Name to be backward compatible
* Fix snapshot tests
* Use Time and Value for time and value fields
* Use variables from grafana-plugin-sdk-go for name
* Elasticsearch: Add processing of logs query to backend
* Add and fix tests
* Add snapshot tests
* Fix test in ES client
* Small updates, remove redundant logic
* Refactor setPreferredVisType to improve readability
* WIP
* WIP
* Refactor
* Add tests
* Cleanup
* Fix whitespace
* Fix test and lint
* In snapshot tests update counter to be number
* Add boolean value for snapshot testing
* Update pkg/tsdb/elasticsearch/response_parser.go
Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
* Update pkg/tsdb/elasticsearch/response_parser.go
Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
* Use generic to reuse logic when creating fields
* Use nullable fields
* Fix lint
* WIP (#63272)
wip
* Fix snapshot test after we changed field types to nullable
---------
Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
Use latest github.com/grafana/grafana-plugin-sdk-go which includes changes to the TypeVersion property (always present in JSON).
Also included is squtil changes: SQL util - allow using the database scan type for converters
* Update oapi library and thema
* Use fork commit to fix elasticsearch and cloudwatch generators
* Update thema
* Fixes
* Update thema with last fixes
* Sync
* Fix test
* Update thema and schemas
* Update thema
* Elasticsearch: Implement schema for query
* Comment out types I am not sure how to do
* Manually fix typing for PipelineMetricAggregationWithMultipleBucketPaths and BasePipelineMetricAggregation
* Import types to types.ts to have single source of truth
* Cleanup, reorder
* Remove unnecesary Schema.
* Fix test
* Refactor
Look for 'caused_by.reason' in ES error response
When the ES response does not contain `reason` or `root_cause[0].reason`
is empty, there is no information for the user to know what is going
wrong.
An example of the error message after this change:
```
Failed to evaluate queries and expressions: failed to execute query A: Trying to create too many buckets. Must be less than or equal to: [65536] but this number of buckets was exceeded. This limit can be set by changing the [search.max_buckets] cluster level setting.
```
Related to https://github.com/grafana/grafana/issues/61246
* Refactor parse query to functions
* Move parsing to new file
* Create empty result variable and use it when returning early
* Fix linting
* Revert "Create empty result variable and use it when returning early"
This reverts commit 36a503f66e.
* Elasticsearch: Fix ordering in raw_document and add logic for raw_data
* Add comments
* Fix raw data request to use correct timefield
* Fix linting
* Add raw data as metric type
* Fix linting
* Elasticsearch: Add defaults for log query
* Add higlight
* Fix lint
* Add snapshot test
* Implement correct query for logs
* Update
* Adjust naming and comments
* Fix lint
* Remove ifs
* Elasticsearch: Fix ordering in raw_document and add logic for raw_data
* Add comments
* Fix raw data request to use correct timefield
* Fix linting
* Add raw data as metric type
* Fix linting
* Hopefully fix lint
* Elasticsearch: Fix removing of empty settings from query in backend implementation
* Update
* Update
* Update pkg/tsdb/elasticsearch/time_series_query.go
Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
* Chore: Update grafana-plugin-sdk-go to v0.142.0
* Update tests and golden files for 207 status code
* Chore: Move update flag definition at the top in response_parser_test.go
* retrigger
Co-authored-by: Will Browne <will.browne@grafana.com>
* make sql engine use pick log context for logs
* update tempo to get log context
* update opentsdb to use log context
* update es client to use log context
* Elasticsearch: Fix calculation of trimEdges
When a value of trimEdges is set greater than 1 we need to drop both the
first and last sample of the data from the response.
* Elasticsearch: Fix reading trimEdges from the query settings
Currently the trimEdges property in the JSON panel is stored as a string
and not directly as a number.
This caused that the reading of the value failed in the go backend
because the simplejson.Int() method doesn't properly handle this case.
This failure when decoding the value goes unnoticed because of the early
return causing the trimEdges configuration to be ignored.
* Refactor castToInt to also return an error
Add a new test case that sets the `trimEdges` property as a quoted
number.
* Elasticsearch: Remove support for versions after their EOL
* Update docs
* Remove old versions from config
* Update pkg/tsdb/elasticsearch/elasticsearch.go
Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
* Fix tests
* Fix typecheck errors
Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
* chore: remove all remaining uses of golang.org/x/net/context
This PR finishes the work started in #47532, replacing all calls to that package with the stdlib context and using http.NewRequestWithContext to include the context where necessary.
Bonus: small formatting fixes to goimports in these files.
closes#44178
* tweak: use context.Background in favor of TODO for tests
* Use integers for time range filter
Previously it was passed as a string which is automatically converted by Elastic to a number only if the field type is "date". For other types (e.g. "long") such conversion doesn't work. In theory "date" could be passed as a formatted string but we don't use it this way and always pass it as a number so it is safe to always pass numbers, not strings.
* Fix time_series_query_test
* Retrigger build
* use fixed_interval in date_histogram
* Add 8.0 to available versions in datasource settings
* Remove moving_avg from available metric aggregations
* Add ES8 devenv
* Update public/app/plugins/datasource/elasticsearch/components/QueryEditor/MetricAggregationsEditor/utils.ts
Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
* Add FE tests
* Add BE test
* fix FE test
* fix BE test
Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
Refactor usage of legacy data contracts. Moves legacy data contracts
to pkg/tsdb/legacydata package.
Refactor pkg/expr to be a proper service/dependency that can be provided
to wire to remove some unneeded dependencies to SSE in ngalert and other places.
Refactor pkg/expr to not use the legacydata,RequestHandler and use
backend.QueryDataHandler instead.
* TimeZonePicker: Allow specifying internal timezones to display
* Elasticsearch: Add time zone setting to Date Histogram aggregation
* rename time_zone to timeZone
* Add tests
* fix typo
* Update packages/grafana-data/src/datetime/timezones.ts
Co-authored-by: Chris Cowan <chris@chriscowan.us>
Co-authored-by: Chris Cowan <chris@chriscowan.us>
* Revert "Prometheus: add functionality to specify desired step interval in dashboards panels (#36422)"
This reverts commit ddf5b65c51.
Co-authored-by: Ivana Huckova <ivana.huckova@gmail.com>
* Revert "Explore: add functionality for supporting different step modes in prometheus (#37829)"
This reverts commit f433cfd8d9.
Co-authored-by: Ivana Huckova <ivana.huckova@gmail.com>
* Revert stepMode BE implementation from #36796
Co-authored-by: "Ivana Huckova" <ivana.huckova@gmail.com>
* convert SQLs to use sdk contracts
* make draft
* postgres
* intermedia
* get datasourceinfo filled at the beginning of the service
* move the interval into package because of cyclict import and fix all postgres tests
* fix mysql test
* fix mssql
* fix the test for pr https://github.com/grafana/grafana/issues/35839
* fix some issue about intervalv2 package
* update sql test
* wire migration for SQLs
* add sqls to the background process
* make it register instead of register and start
* revert formatting
* fix tests
* fix linter
* remove integration test
* Postgres test fix
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
Fixes#30144
Co-authored-by: dsotirakis <sotirakis.dim@gmail.com>
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
Co-authored-by: Ida Furjesova <ida.furjesova@grafana.com>
Co-authored-by: Jack Westbrook <jack.westbrook@gmail.com>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Leon Sorokin <leeoniya@gmail.com>
Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
Co-authored-by: spinillos <selenepinillos@gmail.com>
Co-authored-by: Karl Persson <kalle.persson@grafana.com>
Co-authored-by: Leonard Gram <leo@xlson.com>
* Add select component for choosing step option
* Add onStepChange
* Add functionality for max step
* Rename minInterval to stepInterval to describe min, max and exact step interval
* Change select option from standard to exact
* Add new type StepType for better type safety
* Add tests for adjustInterval
* Add functionality and tests for exact step option
* Prometheus: Spell out min and max in select component
* Prometheus: Change width of step select component and add placeholder
* Prometheus: Adjust for the factor in exact step
* Prometheus: Update tooltip of step lable to include max and exact options and add padding to select component to give it some breathing room from other components
* Update snapshot for step tooltip
* Prometheus: make tooltip more informative
* Prometheus: add tooltip to interval input element
* Prometheus: extract default step option
* Prometheus: update snapshot for PromQueryEditor
* Prometheus: change step labels to uppercase
* Prometheus: define a default step option
* Prometheus: use default step option in both ui component and logic
* Prometheus: update snapshot for PromQueryEditor
* Prometheus: refactor datasource.ts for better readability
* Prometheus: change tool tip for step
* Prometheus: update snapshots
* Prometheus: add correct styling
* Prometheus: update snapshots
* Prometheus change variable name to something less superfluous
* Prometheus: refactor
* Prometheus: add new test for adjustInterval
* Docs: Update docummentation on the step parameter for prometheus
* Prometheus: make step input field smaller and change placeholder text to 15s
* Prometheus: update snapshots
* Prometheus: Make stepMode uniform in all places in the code
* Adjust step based on stepMode
* Adjust comment
* Check if we have queryInterval
* Refactor, add safe interval
* Fix merge resolutions
* Fix tests and add tests
* Update snapshot
* Update docs/sources/datasources/prometheus.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/datasources/prometheus.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/datasources/prometheus.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/datasources/prometheus.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/datasources/prometheus.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/datasources/prometheus.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/datasources/prometheus.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Implement calculation with intervalMode in calculator.go
* Update tests, add calculate safe interval method
* Replace panic with error
* Update pkg/tsdb/interval/interval_test.go
Co-authored-by: idafurjes <36131195+idafurjes@users.noreply.github.com>
* Update pkg/tsdb/calculator_test.go
Co-authored-by: idafurjes <36131195+idafurjes@users.noreply.github.com>
* Impotrt require
* Remove lint errors
Co-authored-by: Olof Bourghardt <ob655088@gmail.com>
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
Co-authored-by: idafurjes <36131195+idafurjes@users.noreply.github.com>
* Migrate Elasticsearch to backend plugin SDK
* Fix linting
* Move away from Convey!
* Rebase commit
* Small logger fix
* Fixes according to reviewer's comments
* Fixes according to reviewer's comments
* Fixes according to reviewer's comments
* More cleanup
* Move things around - small refactoring
* Fix typo
* Update calculator - add tests
* Fixes according to reviewer's comments
* Revert "Revert "Elasticsearch: add frozen indices search support (#27472)" (#27726)"
This reverts commit 4c7131425b.
* Make label width a bit more consistent
* Add documentation for X-Pack & Frozen Indices support in Elasticsearch
* Change UI & docs casing
* create default empty dataframe
* Remove backticks and simplify regex
* small doc improvement
* Elasticsearch: add first version of rate aggregation
* Add test for rate aggreation settings editor
* Fix unit
* Add default value for mode for rate agg
* Add todo
* Revert "Add default value for mode for rate agg"
This reverts commit a44c985d8b.
* Fix width when select is empty
* Add rate aggregation to elasticsearch models
* Remove rate aggregation from extended stats
* Move elastic2 docker block & add elastic (latest)
* Add versionRange to rate aggregation
* add 7.10 elasticsearch version option
* Set supportsInlineScript to true for rate aggregation
* Remove fixed width from select
* Change seconds label for rate aggregation unit options dropdown
Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
Co-authored-by: Elfo404 <me@giordanoricci.com>
Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
* Rebase (broken tests)
* Removed tables - refactored processAggregationDocs func
* Tests cleanup
* Nit - add space in percentile legend titles
* Fix labels naming - use metricAggType map
* Fix bug which appended same fields over and over again
* Replace test with dataframes
* Fix tests
* Add nolint:gocyclo - will need refactoring
* Move frames tags assignment
* Add failing test fo when ES server is booting up
* Revert "Add failing test fo when ES server is booting up"
This reverts commit fd14a1fd5e.
Co-authored-by: Elfo404 <me@giordanoricci.com>
Uses new httpclient package from grafana-plugin-sdk-go introduced
via grafana/grafana-plugin-sdk-go#328.
Replaces the GetHTTPClient, GetTransport, GetTLSConfig methods defined
on DataSource model.
Longer-term the goal is to migrate core HTTP backend data sources to use the
SDK contracts and using httpclient.Provider for creating HTTP clients and such.
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
* Devenv: add block for es 5.0, provisioned datasource & dashboard
* Trasnsform script property based on running ES version
* Handle different scripts format in BE
* Elasticsearch: use proper semver strings to identify ES version
* Update BE & tests
* refactor BE tests
* refactor isValidOption check
* update test
* Update pkg/tsdb/elasticsearch/client/client.go
Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
* Update pkg/tsdb/elasticsearch/client/search_request_test.go
Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
* Remove leftover FIXME comment
* Add new test cases for new version format
* Docs: add documentation about version dropdown
* Update docs/sources/datasources/elasticsearch.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/datasources/elasticsearch.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/datasources/elasticsearch.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update provisioning documentation
Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Elasticsearch: Add generic support for template variables
* format MovingAverage settings as numbers
* Move formatting logic to query builder & forma serial_diff settings as numbers
* modify presence check
* add todo
* minor fixes
* transform string values to numbers
* Move casting logic
* Slightly cleaner implementation
* Add BE tests
* Leverage elastic validation when string doesn't resolve to a numeric value
* move newly introduced test to testify
* add FE query_builder tests
* check error
* Parse values to float instead of int
* Fix tests & ParseFloat bit size
* Elasticsearch: Allow omitting field when metric supports inline script
* Add tests for MetricEditor to show a None option
* Add tests for useFields hook
* Alerting: allow elasticsearch metrics without field
Adds support to the terms aggregation for ordering by percentiles and extended stats.
Closes#5148
Co-authored-by: Giordano Ricci <grdnricci@gmail.com>
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
