Azure: Basic Logs support (#88025)

* Azure monitor: Basic Logs frontend (#85905)

* adds datasource level config for enabling basic logs

* add basiclogsquery type to query json

* add toggle between basic and analytics

* adds basic logs toggle from UI, blocks time picker to only dashboard if basic logs is selected

* add check to remove UI if alerting

* tests for logsmanagement component

* tests for logs query editor

* tests for time mangement control

* remove unused imports

* clears query whenever toggle changes from basic <-> analytics

* add test to account for clearning query

* Update public/app/plugins/datasource/azuremonitor/components/ConfigEditor/BasicLogsToggle.tsx

wording

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx

spelling

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx

spelling

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* update dependency list

* clear basic logs if resources change

* fix tests

---------

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* Azure Monitor: Basic Logs modal acknowledgement (#86244)

* adds datasource level config for enabling basic logs

* add basiclogsquery type to query json

* add toggle between basic and analytics

* adds basic logs toggle from UI, blocks time picker to only dashboard if basic logs is selected

* add check to remove UI if alerting

* tests for logsmanagement component

* tests for logs query editor

* tests for time mangement control

* remove unused imports

* add confirm modal

* clears query whenever toggle changes from basic <-> analytics

* add test to account for clearning query

* adds modal acknowledgement for basic logs query

* tests for handling modal logic

* basic logs ack type

* Update public/app/plugins/datasource/azuremonitor/components/ConfigEditor/BasicLogsToggle.tsx

wording

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx

spelling

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx

spelling

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* update dependency list

* clear basic logs if resources change

* remove modal from config page

* remove basic logs query ack type

* add modal acknowledgement to toggle between basic and analytics

* clear query if resources change

* fix tests

* fix tests

* Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsManagement.tsx

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* fix tests

---------

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* Azure Monitor: Basic Logs Backend (#87653)

* fix logic for showingBasicLogsToggle

* move to utils function and add basiclogsquery in apply template variable

* add backend safeguards for basiclogsqueries

* adds support for calling search or query apis based on whether it is basic logs or not

* add tests for utils

* initial test for basic logs query in the backend

* tests for basic logs

* remve comment

* simplify checks for basic logs

* adds fromAlert prop for azure monitor backend services

* adds fromAlert check fo basic logs

* fix working and empty tags

* add telemetry for basic logs

* remove import from grafana core package

* change fromAlert true in tests

* change the way tests catch errors

* Update pkg/tsdb/azuremonitor/loganalytics/utils.go

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* Update pkg/tsdb/azuremonitor/loganalytics/utils.go

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* Update pkg/tsdb/azuremonitor/loganalytics/utils.go

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* Update pkg/tsdb/azuremonitor/loganalytics/azure-log-analytics-datasource.go

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* restructure code to only run basic logs checks if basiclogsflag is true

* data retention warning

* tests for calculate time range

* Simplify determining if request is from alerting

* Fix lint and bool check

* Fix tests

* clarify data retention

---------

Co-authored-by: Jocelyn <jcolladokuri@microsoft.com>
Co-authored-by: jcolladokuri <jocelyncollado52@gmail.com>

* Azure Monitor: Basic Logs data volume notification (#88009)

* frontend changes for data ingested warning

* initial logic for getResource

* payload processing

* create basicLogs usage function

* add utils for converting time and getting the data volume query for basic logs

* frontend updates for showing the data ingested for the given query

* frontend tests

* add check for when no dataIngested is returned

* remove backend.logger prints

* comment on what function does

* fix merge

* make resource URI regex case insensitive

* add support for workspace variables in basic logs flow

* add undefined check

* structure and add tests for variable support

* Update pkg/tsdb/azuremonitor/loganalytics/azure-log-analytics-datasource.go

Co-authored-by: Andreas Christou <andreas.christou@grafana.com>

* add tracing for basic logs usage request

* clean up data volume query struct

* use async/await instead of callback

* fix parameters for getApiURL

* restrict time on usage query to 8 days max

* add time to dependency array to refetch basic logs usage

* move time check implementation to backend

* fix utils tests

---------

Co-authored-by: Jocelyn <jcolladokuri@microsoft.com>
Co-authored-by: jcolladokuri <jocelyncollado52@gmail.com>

---------

Co-authored-by: jcolladokuri <jcolladokuri@microsoft.com>
Co-authored-by: jcolladokuri <jocelyncollado52@gmail.com>
This commit is contained in:
Andreas Christou 2024-05-28 18:06:27 +01:00 committed by GitHub
parent 0b2fab9967
commit 3eea71cc6b
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
35 changed files with 1188 additions and 55 deletions

View File

@ -163,6 +163,10 @@ export const defaultAzureMetricQuery: Partial<AzureMetricQuery> = {
* Azure Monitor Logs sub-query properties * Azure Monitor Logs sub-query properties
*/ */
export interface AzureLogsQuery { export interface AzureLogsQuery {
/**
* If set to true the query will be run as a basic logs query
*/
basicLogsQuery?: boolean;
/** /**
* If set to true the dashboard time range will be used as a filter for the query. Otherwise the query time ranges will be used. Defaults to false. * If set to true the dashboard time range will be used as a filter for the query. Otherwise the query time ranges will be used. Defaults to false.
*/ */

View File

@ -137,7 +137,7 @@ func NewInstanceSettings(clientProvider *httpclient.Provider, executors map[stri
} }
type azDatasourceExecutor interface { type azDatasourceExecutor interface {
ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string) (*backend.QueryDataResponse, error) ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string, fromAlert bool) (*backend.QueryDataResponse, error)
ResourceRequest(rw http.ResponseWriter, req *http.Request, cli *http.Client) (http.ResponseWriter, error) ResourceRequest(rw http.ResponseWriter, req *http.Request, cli *http.Client) (http.ResponseWriter, error)
} }
@ -172,7 +172,9 @@ func (s *Service) newQueryMux() *datasource.QueryTypeMux {
if !ok { if !ok {
return nil, fmt.Errorf("missing service for %s", dst) return nil, fmt.Errorf("missing service for %s", dst)
} }
return executor.ExecuteTimeSeriesQuery(ctx, req.Queries, dsInfo, service.HTTPClient, service.URL) // FromAlert header is defined in pkg/services/ngalert/models/constants.go
fromAlert := req.Headers["FromAlert"] == "true"
return executor.ExecuteTimeSeriesQuery(ctx, req.Queries, dsInfo, service.HTTPClient, service.URL, fromAlert)
}) })
} }
return mux return mux

View File

@ -149,7 +149,7 @@ func (f *fakeExecutor) ResourceRequest(rw http.ResponseWriter, req *http.Request
return nil, nil return nil, nil
} }
func (f *fakeExecutor) ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string) (*backend.QueryDataResponse, error) { func (f *fakeExecutor) ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string, fromAlert bool) (*backend.QueryDataResponse, error) {
if client == nil { if client == nil {
f.t.Errorf("The HTTP client for %s is missing", f.queryType) f.t.Errorf("The HTTP client for %s is missing", f.queryType)
} else { } else {

View File

@ -117,6 +117,9 @@ type AppInsightsMetricNameQueryKind string
// Azure Monitor Logs sub-query properties // Azure Monitor Logs sub-query properties
type AzureLogsQuery struct { type AzureLogsQuery struct {
// If set to true the query will be run as a basic logs query
BasicLogsQuery *bool `json:"basicLogsQuery,omitempty"`
// If set to true the dashboard time range will be used as a filter for the query. Otherwise the query time ranges will be used. Defaults to false. // If set to true the dashboard time range will be used as a filter for the query. Otherwise the query time ranges will be used. Defaults to false.
DashboardTime *bool `json:"dashboardTime,omitempty"` DashboardTime *bool `json:"dashboardTime,omitempty"`

View File

@ -27,16 +27,116 @@ import (
) )
func (e *AzureLogAnalyticsDatasource) ResourceRequest(rw http.ResponseWriter, req *http.Request, cli *http.Client) (http.ResponseWriter, error) { func (e *AzureLogAnalyticsDatasource) ResourceRequest(rw http.ResponseWriter, req *http.Request, cli *http.Client) (http.ResponseWriter, error) {
if req.URL.Path == "/usage/basiclogs" {
newUrl := &url.URL{
Scheme: req.URL.Scheme,
Host: req.URL.Host,
Path: "/v1/query",
}
return e.GetBasicLogsUsage(req.Context(), newUrl.String(), cli, rw, req.Body)
}
return e.Proxy.Do(rw, req, cli) return e.Proxy.Do(rw, req, cli)
} }
// builds and executes a new query request that will get the data ingeted for the given table in the basic logs query
func (e *AzureLogAnalyticsDatasource) GetBasicLogsUsage(ctx context.Context, url string, client *http.Client, rw http.ResponseWriter, reqBody io.ReadCloser) (http.ResponseWriter, error) {
// read the full body
originalPayload, readErr := io.ReadAll(reqBody)
if readErr != nil {
return rw, fmt.Errorf("failed to read request body %w", readErr)
}
var payload BasicLogsUsagePayload
jsonErr := json.Unmarshal(originalPayload, &payload)
if jsonErr != nil {
return rw, fmt.Errorf("error decoding basic logs table usage payload: %w", jsonErr)
}
table := payload.Table
from, fromErr := ConvertTime(payload.From)
if fromErr != nil {
return rw, fmt.Errorf("failed to convert from time: %w", fromErr)
}
to, toErr := ConvertTime(payload.To)
if toErr != nil {
return rw, fmt.Errorf("failed to convert to time: %w", toErr)
}
// basic logs queries only show data for last 8 days or less
// data volume query should also only calculate volume for last 8 days if time range exceeds that.
diff := to.Sub(from).Hours()
if diff > float64(MaxHoursBasicLogs) {
from = to.Add(-time.Duration(MaxHoursBasicLogs) * time.Hour)
}
dataVolumeQueryRaw := GetDataVolumeRawQuery(table)
dataVolumeQuery := &AzureLogAnalyticsQuery{
Query: dataVolumeQueryRaw,
DashboardTime: true, // necessary to ensure TimeRange property is used since query will not have an in-query time filter
TimeRange: backend.TimeRange{
From: from,
To: to,
},
TimeColumn: "TimeGenerated",
Resources: []string{payload.Resource},
QueryType: dataquery.AzureQueryTypeAzureLogAnalytics,
URL: getApiURL(payload.Resource, false, false),
}
req, err := e.createRequest(ctx, url, dataVolumeQuery)
if err != nil {
return rw, err
}
_, span := tracing.DefaultTracer().Start(ctx, "azure basic logs usage query", trace.WithAttributes(
attribute.String("target", dataVolumeQuery.Query),
attribute.String("table", table),
attribute.Int64("from", dataVolumeQuery.TimeRange.From.UnixNano()/int64(time.Millisecond)),
attribute.Int64("until", dataVolumeQuery.TimeRange.To.UnixNano()/int64(time.Millisecond)),
))
defer span.End()
resp, err := client.Do(req)
if err != nil {
return rw, err
}
defer func() {
if err := resp.Body.Close(); err != nil {
e.Logger.Warn("Failed to close response body for data volume request", "err", err)
}
}()
logResponse, err := e.unmarshalResponse(resp)
if err != nil {
return rw, err
}
t, err := logResponse.GetPrimaryResultTable()
if err != nil {
return rw, err
}
num := t.Rows[0][0].(json.Number)
value, err := num.Float64()
if err != nil {
return rw, err
}
_, err = rw.Write([]byte(fmt.Sprintf("%f", value)))
if err != nil {
return rw, err
}
return rw, err
}
// executeTimeSeriesQuery does the following: // executeTimeSeriesQuery does the following:
// 1. build the AzureMonitor url and querystring for each query // 1. build the AzureMonitor url and querystring for each query
// 2. executes each query by calling the Azure Monitor API // 2. executes each query by calling the Azure Monitor API
// 3. parses the responses for each query into data frames // 3. parses the responses for each query into data frames
func (e *AzureLogAnalyticsDatasource) ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string) (*backend.QueryDataResponse, error) { func (e *AzureLogAnalyticsDatasource) ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string, fromAlert bool) (*backend.QueryDataResponse, error) {
result := backend.NewQueryDataResponse() result := backend.NewQueryDataResponse()
queries, err := e.buildQueries(ctx, originalQueries, dsInfo) queries, err := e.buildQueries(ctx, originalQueries, dsInfo, fromAlert)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@ -53,23 +153,38 @@ func (e *AzureLogAnalyticsDatasource) ExecuteTimeSeriesQuery(ctx context.Context
return result, nil return result, nil
} }
func buildLogAnalyticsQuery(query backend.DataQuery, dsInfo types.DatasourceInfo, appInsightsRegExp *regexp.Regexp) (*AzureLogAnalyticsQuery, error) { func buildLogAnalyticsQuery(query backend.DataQuery, dsInfo types.DatasourceInfo, appInsightsRegExp *regexp.Regexp, fromAlert bool) (*AzureLogAnalyticsQuery, error) {
queryJSONModel := types.LogJSONQuery{} queryJSONModel := types.LogJSONQuery{}
err := json.Unmarshal(query.JSON, &queryJSONModel) err := json.Unmarshal(query.JSON, &queryJSONModel)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to decode the Azure Log Analytics query object from JSON: %w", err) return nil, fmt.Errorf("failed to decode the Azure Log Analytics query object from JSON: %w", err)
} }
var queryString string var queryString string
appInsightsQuery := false appInsightsQuery := false
dashboardTime := false dashboardTime := false
timeColumn := "" timeColumn := ""
azureLogAnalyticsTarget := queryJSONModel.AzureLogAnalytics azureLogAnalyticsTarget := queryJSONModel.AzureLogAnalytics
basicLogsQuery := false
resultFormat := ParseResultFormat(azureLogAnalyticsTarget.ResultFormat, dataquery.AzureQueryTypeAzureLogAnalytics) resultFormat := ParseResultFormat(azureLogAnalyticsTarget.ResultFormat, dataquery.AzureQueryTypeAzureLogAnalytics)
basicLogsQueryFlag := false
if azureLogAnalyticsTarget.BasicLogsQuery != nil {
basicLogsQueryFlag = *azureLogAnalyticsTarget.BasicLogsQuery
}
resources, resourceOrWorkspace := retrieveResources(azureLogAnalyticsTarget) resources, resourceOrWorkspace := retrieveResources(azureLogAnalyticsTarget)
appInsightsQuery = appInsightsRegExp.Match([]byte(resourceOrWorkspace)) appInsightsQuery = appInsightsRegExp.Match([]byte(resourceOrWorkspace))
if basicLogsQueryFlag {
if meetsBasicLogsCriteria, meetsBasicLogsCriteriaErr := meetsBasicLogsCriteria(resources, fromAlert); meetsBasicLogsCriteriaErr != nil {
return nil, meetsBasicLogsCriteriaErr
} else {
basicLogsQuery = meetsBasicLogsCriteria
}
}
if azureLogAnalyticsTarget.Query != nil { if azureLogAnalyticsTarget.Query != nil {
queryString = *azureLogAnalyticsTarget.Query queryString = *azureLogAnalyticsTarget.Query
} }
@ -86,7 +201,7 @@ func buildLogAnalyticsQuery(query backend.DataQuery, dsInfo types.DatasourceInfo
} }
} }
apiURL := getApiURL(resourceOrWorkspace, appInsightsQuery) apiURL := getApiURL(resourceOrWorkspace, appInsightsQuery, basicLogsQuery)
rawQuery, err := macros.KqlInterpolate(query, dsInfo, queryString, "TimeGenerated") rawQuery, err := macros.KqlInterpolate(query, dsInfo, queryString, "TimeGenerated")
if err != nil { if err != nil {
@ -105,10 +220,11 @@ func buildLogAnalyticsQuery(query backend.DataQuery, dsInfo types.DatasourceInfo
AppInsightsQuery: appInsightsQuery, AppInsightsQuery: appInsightsQuery,
DashboardTime: dashboardTime, DashboardTime: dashboardTime,
TimeColumn: timeColumn, TimeColumn: timeColumn,
BasicLogs: basicLogsQuery,
}, nil }, nil
} }
func (e *AzureLogAnalyticsDatasource) buildQueries(ctx context.Context, queries []backend.DataQuery, dsInfo types.DatasourceInfo) ([]*AzureLogAnalyticsQuery, error) { func (e *AzureLogAnalyticsDatasource) buildQueries(ctx context.Context, queries []backend.DataQuery, dsInfo types.DatasourceInfo, fromAlert bool) ([]*AzureLogAnalyticsQuery, error) {
azureLogAnalyticsQueries := []*AzureLogAnalyticsQuery{} azureLogAnalyticsQueries := []*AzureLogAnalyticsQuery{}
appInsightsRegExp, err := regexp.Compile("providers/Microsoft.Insights/components") appInsightsRegExp, err := regexp.Compile("providers/Microsoft.Insights/components")
if err != nil { if err != nil {
@ -117,7 +233,7 @@ func (e *AzureLogAnalyticsDatasource) buildQueries(ctx context.Context, queries
for _, query := range queries { for _, query := range queries {
if query.QueryType == string(dataquery.AzureQueryTypeAzureLogAnalytics) { if query.QueryType == string(dataquery.AzureQueryTypeAzureLogAnalytics) {
azureLogAnalyticsQuery, err := buildLogAnalyticsQuery(query, dsInfo, appInsightsRegExp) azureLogAnalyticsQuery, err := buildLogAnalyticsQuery(query, dsInfo, appInsightsRegExp, fromAlert)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to build azure log analytics query: %w", err) return nil, fmt.Errorf("failed to build azure log analytics query: %w", err)
} }
@ -161,6 +277,7 @@ func (e *AzureLogAnalyticsDatasource) executeQuery(ctx context.Context, query *A
_, span := tracing.DefaultTracer().Start(ctx, "azure log analytics query", trace.WithAttributes( _, span := tracing.DefaultTracer().Start(ctx, "azure log analytics query", trace.WithAttributes(
attribute.String("target", query.Query), attribute.String("target", query.Query),
attribute.Bool("basic_logs", query.BasicLogs),
attribute.Int64("from", query.TimeRange.From.UnixNano()/int64(time.Millisecond)), attribute.Int64("from", query.TimeRange.From.UnixNano()/int64(time.Millisecond)),
attribute.Int64("until", query.TimeRange.To.UnixNano()/int64(time.Millisecond)), attribute.Int64("until", query.TimeRange.To.UnixNano()/int64(time.Millisecond)),
attribute.Int64("datasource_id", dsInfo.DatasourceID), attribute.Int64("datasource_id", dsInfo.DatasourceID),

View File

@ -21,6 +21,10 @@ import (
"github.com/grafana/grafana/pkg/tsdb/azuremonitor/types" "github.com/grafana/grafana/pkg/tsdb/azuremonitor/types"
) )
func makeQueryPointer(q AzureLogAnalyticsQuery) *AzureLogAnalyticsQuery {
return &q
}
func TestBuildLogAnalyticsQuery(t *testing.T) { func TestBuildLogAnalyticsQuery(t *testing.T) {
fromStart := time.Date(2018, 3, 15, 13, 0, 0, 0, time.UTC).In(time.Local) fromStart := time.Date(2018, 3, 15, 13, 0, 0, 0, time.UTC).In(time.Local)
timeRange := backend.TimeRange{From: fromStart, To: fromStart.Add(34 * time.Minute)} timeRange := backend.TimeRange{From: fromStart, To: fromStart.Add(34 * time.Minute)}
@ -95,12 +99,14 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
fromAlert bool
queryModel backend.DataQuery queryModel backend.DataQuery
azureLogAnalyticsQuery AzureLogAnalyticsQuery azureLogAnalyticsQuery *AzureLogAnalyticsQuery
Err require.ErrorAssertionFunc Err require.ErrorAssertionFunc
}{ }{
{ {
name: "Query with macros should be interpolated", name: "Query with macros should be interpolated",
fromAlert: false,
queryModel: backend.DataQuery{ queryModel: backend.DataQuery{
JSON: []byte(fmt.Sprintf(`{ JSON: []byte(fmt.Sprintf(`{
"queryType": "Azure Log Analytics", "queryType": "Azure Log Analytics",
@ -115,7 +121,7 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
TimeRange: timeRange, TimeRange: timeRange,
QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics), QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics),
}, },
azureLogAnalyticsQuery: AzureLogAnalyticsQuery{ azureLogAnalyticsQuery: makeQueryPointer(AzureLogAnalyticsQuery{
RefID: "A", RefID: "A",
ResultFormat: dataquery.ResultFormatTimeSeries, ResultFormat: dataquery.ResultFormatTimeSeries,
URL: "v1/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/AppInsightsTestDataWorkspace/query", URL: "v1/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/AppInsightsTestDataWorkspace/query",
@ -134,11 +140,12 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
QueryType: dataquery.AzureQueryTypeAzureLogAnalytics, QueryType: dataquery.AzureQueryTypeAzureLogAnalytics,
AppInsightsQuery: false, AppInsightsQuery: false,
DashboardTime: false, DashboardTime: false,
}, }),
Err: require.NoError, Err: require.NoError,
}, },
{ {
name: "Legacy queries with a workspace GUID should use workspace-centric url", name: "Legacy queries with a workspace GUID should use workspace-centric url",
fromAlert: false,
queryModel: backend.DataQuery{ queryModel: backend.DataQuery{
JSON: []byte(fmt.Sprintf(`{ JSON: []byte(fmt.Sprintf(`{
"queryType": "Azure Log Analytics", "queryType": "Azure Log Analytics",
@ -151,7 +158,7 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
RefID: "A", RefID: "A",
QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics), QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics),
}, },
azureLogAnalyticsQuery: AzureLogAnalyticsQuery{ azureLogAnalyticsQuery: makeQueryPointer(AzureLogAnalyticsQuery{
RefID: "A", RefID: "A",
ResultFormat: dataquery.ResultFormatTimeSeries, ResultFormat: dataquery.ResultFormatTimeSeries,
URL: "v1/workspaces/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/query", URL: "v1/workspaces/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/query",
@ -168,11 +175,12 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
QueryType: dataquery.AzureQueryTypeAzureLogAnalytics, QueryType: dataquery.AzureQueryTypeAzureLogAnalytics,
AppInsightsQuery: false, AppInsightsQuery: false,
DashboardTime: false, DashboardTime: false,
}, }),
Err: require.NoError, Err: require.NoError,
}, },
{ {
name: "Legacy workspace queries with a resource URI (from a template variable) should use resource-centric url", name: "Legacy workspace queries with a resource URI (from a template variable) should use resource-centric url",
fromAlert: false,
queryModel: backend.DataQuery{ queryModel: backend.DataQuery{
JSON: []byte(fmt.Sprintf(`{ JSON: []byte(fmt.Sprintf(`{
"queryType": "Azure Log Analytics", "queryType": "Azure Log Analytics",
@ -185,7 +193,7 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
RefID: "A", RefID: "A",
QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics), QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics),
}, },
azureLogAnalyticsQuery: AzureLogAnalyticsQuery{ azureLogAnalyticsQuery: makeQueryPointer(AzureLogAnalyticsQuery{
RefID: "A", RefID: "A",
ResultFormat: dataquery.ResultFormatTimeSeries, ResultFormat: dataquery.ResultFormatTimeSeries,
URL: "v1/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/AppInsightsTestDataWorkspace/query", URL: "v1/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/AppInsightsTestDataWorkspace/query",
@ -202,11 +210,12 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
QueryType: dataquery.AzureQueryTypeAzureLogAnalytics, QueryType: dataquery.AzureQueryTypeAzureLogAnalytics,
AppInsightsQuery: false, AppInsightsQuery: false,
DashboardTime: false, DashboardTime: false,
}, }),
Err: require.NoError, Err: require.NoError,
}, },
{ {
name: "Queries with multiple resources", name: "Queries with multiple resources",
fromAlert: false,
queryModel: backend.DataQuery{ queryModel: backend.DataQuery{
JSON: []byte(fmt.Sprintf(`{ JSON: []byte(fmt.Sprintf(`{
"queryType": "Azure Log Analytics", "queryType": "Azure Log Analytics",
@ -220,7 +229,7 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
RefID: "A", RefID: "A",
QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics), QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics),
}, },
azureLogAnalyticsQuery: AzureLogAnalyticsQuery{ azureLogAnalyticsQuery: makeQueryPointer(AzureLogAnalyticsQuery{
RefID: "A", RefID: "A",
ResultFormat: dataquery.ResultFormatTimeSeries, ResultFormat: dataquery.ResultFormatTimeSeries,
URL: "v1/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/AppInsightsTestDataWorkspace/query", URL: "v1/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/AppInsightsTestDataWorkspace/query",
@ -238,11 +247,12 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
QueryType: dataquery.AzureQueryTypeAzureLogAnalytics, QueryType: dataquery.AzureQueryTypeAzureLogAnalytics,
AppInsightsQuery: false, AppInsightsQuery: false,
DashboardTime: false, DashboardTime: false,
}, }),
Err: require.NoError, Err: require.NoError,
}, },
{ {
name: "Query with multiple resources", name: "Query with multiple resources",
fromAlert: false,
queryModel: backend.DataQuery{ queryModel: backend.DataQuery{
JSON: []byte(fmt.Sprintf(`{ JSON: []byte(fmt.Sprintf(`{
"queryType": "Azure Log Analytics", "queryType": "Azure Log Analytics",
@ -257,7 +267,7 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
TimeRange: timeRange, TimeRange: timeRange,
QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics), QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics),
}, },
azureLogAnalyticsQuery: AzureLogAnalyticsQuery{ azureLogAnalyticsQuery: makeQueryPointer(AzureLogAnalyticsQuery{
RefID: "A", RefID: "A",
ResultFormat: dataquery.ResultFormatTimeSeries, ResultFormat: dataquery.ResultFormatTimeSeries,
URL: "v1/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/AppInsightsTestDataWorkspace/query", URL: "v1/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/AppInsightsTestDataWorkspace/query",
@ -276,11 +286,12 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
QueryType: dataquery.AzureQueryTypeAzureLogAnalytics, QueryType: dataquery.AzureQueryTypeAzureLogAnalytics,
AppInsightsQuery: false, AppInsightsQuery: false,
DashboardTime: false, DashboardTime: false,
}, }),
Err: require.NoError, Err: require.NoError,
}, },
{ {
name: "Query that uses dashboard time", name: "Query that uses dashboard time",
fromAlert: false,
queryModel: backend.DataQuery{ queryModel: backend.DataQuery{
JSON: []byte(fmt.Sprintf(`{ JSON: []byte(fmt.Sprintf(`{
"queryType": "Azure Log Analytics", "queryType": "Azure Log Analytics",
@ -296,7 +307,7 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
TimeRange: timeRange, TimeRange: timeRange,
QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics), QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics),
}, },
azureLogAnalyticsQuery: AzureLogAnalyticsQuery{ azureLogAnalyticsQuery: makeQueryPointer(AzureLogAnalyticsQuery{
RefID: "A", RefID: "A",
ResultFormat: dataquery.ResultFormatTimeSeries, ResultFormat: dataquery.ResultFormatTimeSeries,
URL: "v1/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/AppInsightsTestDataWorkspace/query", URL: "v1/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/AppInsightsTestDataWorkspace/query",
@ -317,16 +328,127 @@ func TestBuildLogAnalyticsQuery(t *testing.T) {
AppInsightsQuery: false, AppInsightsQuery: false,
DashboardTime: true, DashboardTime: true,
TimeColumn: "TimeGenerated", TimeColumn: "TimeGenerated",
}, }),
Err: require.NoError, Err: require.NoError,
}, },
{
name: "Basic Logs query",
fromAlert: false,
queryModel: backend.DataQuery{
JSON: []byte(fmt.Sprintf(`{
"queryType": "Azure Log Analytics",
"azureLogAnalytics": {
"resources": ["/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/TestDataWorkspace"],
"query": "Perf",
"resultFormat": "%s",
"dashboardTime": true,
"timeColumn": "TimeGenerated",
"basicLogsQuery": true
}
}`, dataquery.ResultFormatTimeSeries)),
RefID: "A",
TimeRange: timeRange,
QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics),
},
azureLogAnalyticsQuery: makeQueryPointer(AzureLogAnalyticsQuery{
RefID: "A",
ResultFormat: dataquery.ResultFormatTimeSeries,
URL: "v1/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/TestDataWorkspace/search",
JSON: []byte(fmt.Sprintf(`{
"queryType": "Azure Log Analytics",
"azureLogAnalytics": {
"resources": ["/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/TestDataWorkspace"],
"query": "Perf",
"resultFormat": "%s",
"dashboardTime": true,
"timeColumn": "TimeGenerated",
"basicLogsQuery": true
}
}`, dataquery.ResultFormatTimeSeries)),
Query: "Perf",
Resources: []string{"/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/TestDataWorkspace"},
TimeRange: timeRange,
QueryType: dataquery.AzureQueryTypeAzureLogAnalytics,
AppInsightsQuery: false,
DashboardTime: true,
BasicLogs: true,
TimeColumn: "TimeGenerated",
}),
Err: require.NoError,
},
{
name: "Basic Logs query with multiple resources",
fromAlert: false,
queryModel: backend.DataQuery{
JSON: []byte(fmt.Sprintf(`{
"queryType": "Azure Log Analytics",
"azureLogAnalytics": {
"resources": ["/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/TestDataWorkspace1", "/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.OperationalInsights/workspaces/TestDataWorkspace2"],
"query": "Perf",
"resultFormat": "%s",
"dashboardTime": true,
"timeColumn": "TimeGenerated",
"basicLogsQuery": true
}
}`, dataquery.ResultFormatTimeSeries)),
RefID: "A",
TimeRange: timeRange,
QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics),
},
azureLogAnalyticsQuery: nil,
Err: require.Error,
},
{
name: "Basic Logs query with non LA workspace resources",
fromAlert: false,
queryModel: backend.DataQuery{
JSON: []byte(fmt.Sprintf(`{
"queryType": "Azure Log Analytics",
"azureLogAnalytics": {
"resources": ["/subscriptions/test-sub/resourceGroups/test-rg/providers/Microsoft.Insights/components/r1"],
"query": "Perf",
"resultFormat": "%s",
"dashboardTime": true,
"timeColumn": "TimeGenerated",
"basicLogsQuery": true
}
}`, dataquery.ResultFormatTimeSeries)),
RefID: "A",
TimeRange: timeRange,
QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics),
},
azureLogAnalyticsQuery: nil,
Err: require.Error,
},
{
name: "Basic Logs query from alerts",
fromAlert: true,
queryModel: backend.DataQuery{
JSON: []byte(fmt.Sprintf(`{
"queryType": "Azure Log Analytics",
"azureLogAnalytics": {
"resources": ["/subscriptions/test-sub/resourceGroups/test-rg/providers/Microsoft.Insights/components/r1"],
"query": "Perf",
"resultFormat": "%s",
"dashboardTime": true,
"timeColumn": "TimeGenerated",
"basicLogsQuery": true
}
}`, dataquery.ResultFormatTimeSeries)),
RefID: "A",
TimeRange: timeRange,
QueryType: string(dataquery.AzureQueryTypeAzureLogAnalytics),
},
azureLogAnalyticsQuery: nil,
Err: require.Error,
},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
query, err := buildLogAnalyticsQuery(tt.queryModel, dsInfo, appInsightsRegExp) query, err := buildLogAnalyticsQuery(tt.queryModel, dsInfo, appInsightsRegExp, tt.fromAlert)
tt.Err(t, err) tt.Err(t, err)
if diff := cmp.Diff(&tt.azureLogAnalyticsQuery, query); diff != "" { if diff := cmp.Diff(tt.azureLogAnalyticsQuery, query); diff != "" {
t.Errorf("Result mismatch (-want +got): \n%s", diff) t.Errorf("Result mismatch (-want +got): \n%s", diff)
} }
}) })

View File

@ -2,6 +2,8 @@ package loganalytics
var Tables = []string{"availabilityResults", "dependencies", "customEvents", "exceptions", "pageViews", "requests", "traces"} var Tables = []string{"availabilityResults", "dependencies", "customEvents", "exceptions", "pageViews", "requests", "traces"}
var MaxHoursBasicLogs = 192 // 8 days in hours
// AttributesOmit - Properties to omit when generating the attributes bag // AttributesOmit - Properties to omit when generating the attributes bag
var AttributesOmit = map[string]string{"operationId": "operationId", "duration": "duration", "id": "id", "name": "name", "problemId": "problemId", "operation_ParentId": "operation_ParentId", "timestamp": "timestamp", "customDimensions": "customDimensions", "operation_Name": "operation_Name"} var AttributesOmit = map[string]string{"operationId": "operationId", "duration": "duration", "id": "id", "name": "name", "problemId": "problemId", "operation_ParentId": "operation_ParentId", "timestamp": "timestamp", "customDimensions": "customDimensions", "operation_Name": "operation_Name"}

View File

@ -227,7 +227,7 @@ func buildAppInsightsQuery(ctx context.Context, query backend.DataQuery, dsInfo
return nil, err return nil, err
} }
apiURL := getApiURL(resourceOrWorkspace, appInsightsQuery) apiURL := getApiURL(resourceOrWorkspace, appInsightsQuery, false)
rawQuery, err := macros.KqlInterpolate(query, dsInfo, queryString, "TimeGenerated") rawQuery, err := macros.KqlInterpolate(query, dsInfo, queryString, "TimeGenerated")
if err != nil { if err != nil {

View File

@ -32,6 +32,7 @@ type AzureLogAnalyticsQuery struct {
AppInsightsQuery bool AppInsightsQuery bool
DashboardTime bool DashboardTime bool
TimeColumn string TimeColumn string
BasicLogs bool
} }
// Error definition has been inferred from real data and other model definitions like // Error definition has been inferred from real data and other model definitions like
@ -73,3 +74,12 @@ type AzureCorrelationAPIResponseProperties struct {
Resources []string `json:"resources"` Resources []string `json:"resources"`
NextLink *string `json:"nextLink,omitempty"` NextLink *string `json:"nextLink,omitempty"`
} }
// BasicLogsUsagePayload is the payload that the frontend resourcerequest will send to the backend to calculate the basic logs query usage
type BasicLogsUsagePayload struct {
Table string `json:"table"`
Resource string `json:"resource"`
QueryType string `json:"queryType"`
From string `json:"from"`
To string `json:"to"`
}

View File

@ -3,7 +3,9 @@ package loganalytics
import ( import (
"fmt" "fmt"
"regexp" "regexp"
"strconv"
"strings" "strings"
"time"
"github.com/grafana/grafana-plugin-sdk-go/data" "github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana/pkg/tsdb/azuremonitor/kinds/dataquery" "github.com/grafana/grafana/pkg/tsdb/azuremonitor/kinds/dataquery"
@ -37,6 +39,25 @@ func AddConfigLinks(frame data.Frame, dl string, title *string) data.Frame {
return frame return frame
} }
// Check whether a query should be handled as basic logs query
// 2. resource selected is a workspace
// 3. query is not an alerts query
// 4. number of selected resources is exactly one
func meetsBasicLogsCriteria(resources []string, fromAlert bool) (bool, error) {
if fromAlert {
return false, fmt.Errorf("basic Logs queries cannot be used for alerts")
}
if len(resources) != 1 {
return false, fmt.Errorf("basic logs queries cannot be run against multiple resources")
}
if !strings.Contains(strings.ToLower(resources[0]), "microsoft.operationalinsights/workspaces") {
return false, fmt.Errorf("basic Logs queries may only be run against Log Analytics workspaces")
}
return true, nil
}
func ParseResultFormat(queryResultFormat *dataquery.ResultFormat, queryType dataquery.AzureQueryType) dataquery.ResultFormat { func ParseResultFormat(queryResultFormat *dataquery.ResultFormat, queryType dataquery.AzureQueryType) dataquery.ResultFormat {
var resultFormat dataquery.ResultFormat var resultFormat dataquery.ResultFormat
if queryResultFormat != nil { if queryResultFormat != nil {
@ -55,17 +76,22 @@ func ParseResultFormat(queryResultFormat *dataquery.ResultFormat, queryType data
return resultFormat return resultFormat
} }
func getApiURL(resourceOrWorkspace string, isAppInsightsQuery bool) string { func getApiURL(resourceOrWorkspace string, isAppInsightsQuery bool, basicLogsQuery bool) string {
matchesResourceURI, _ := regexp.MatchString("^/subscriptions/", resourceOrWorkspace) matchesResourceURI, _ := regexp.MatchString("^/subscriptions/", resourceOrWorkspace)
queryOrSearch := "query"
if basicLogsQuery {
queryOrSearch = "search"
}
if matchesResourceURI { if matchesResourceURI {
if isAppInsightsQuery { if isAppInsightsQuery {
componentName := resourceOrWorkspace[strings.LastIndex(resourceOrWorkspace, "/")+1:] componentName := resourceOrWorkspace[strings.LastIndex(resourceOrWorkspace, "/")+1:]
return fmt.Sprintf("v1/apps/%s/query", componentName) return fmt.Sprintf("v1/apps/%s/query", componentName)
} }
return fmt.Sprintf("v1%s/query", resourceOrWorkspace) return fmt.Sprintf("v1%s/%s", resourceOrWorkspace, queryOrSearch)
} else { } else {
return fmt.Sprintf("v1/workspaces/%s/query", resourceOrWorkspace) return fmt.Sprintf("v1/workspaces/%s/%s", resourceOrWorkspace, queryOrSearch)
} }
} }
@ -88,3 +114,21 @@ func retrieveResources(query dataquery.AzureLogsQuery) ([]string, string) {
return resources, resourceOrWorkspace return resources, resourceOrWorkspace
} }
func ConvertTime(timeStamp string) (time.Time, error) {
// Convert the timestamp string to an int64
timestampInt, err := strconv.ParseInt(timeStamp, 10, 64)
if err != nil {
// Handle error
return time.Time{}, err
}
// Convert the Unix timestamp (in milliseconds) to a time.Time
convTimeStamp := time.Unix(0, timestampInt*int64(time.Millisecond))
return convTimeStamp, nil
}
func GetDataVolumeRawQuery(table string) string {
return fmt.Sprintf("Usage \n| where DataType == \"%s\"\n| where IsBillable == true\n| summarize BillableDataGB = round(sum(Quantity) / 1000, 3)", table)
}

View File

@ -48,7 +48,7 @@ func (e *AzureMonitorDatasource) ResourceRequest(rw http.ResponseWriter, req *ht
// 1. build the AzureMonitor url and querystring for each query // 1. build the AzureMonitor url and querystring for each query
// 2. executes each query by calling the Azure Monitor API // 2. executes each query by calling the Azure Monitor API
// 3. parses the responses for each query into data frames // 3. parses the responses for each query into data frames
func (e *AzureMonitorDatasource) ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string) (*backend.QueryDataResponse, error) { func (e *AzureMonitorDatasource) ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string, fromAlert bool) (*backend.QueryDataResponse, error) {
result := backend.NewQueryDataResponse() result := backend.NewQueryDataResponse()
queries, err := e.buildQueries(originalQueries, dsInfo) queries, err := e.buildQueries(originalQueries, dsInfo)

View File

@ -58,7 +58,7 @@ func (e *AzureResourceGraphDatasource) ResourceRequest(rw http.ResponseWriter, r
// 1. builds the AzureMonitor url and querystring for each query // 1. builds the AzureMonitor url and querystring for each query
// 2. executes each query by calling the Azure Monitor API // 2. executes each query by calling the Azure Monitor API
// 3. parses the responses for each query into data frames // 3. parses the responses for each query into data frames
func (e *AzureResourceGraphDatasource) ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string) (*backend.QueryDataResponse, error) { func (e *AzureResourceGraphDatasource) ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string, fromAlert bool) (*backend.QueryDataResponse, error) {
result := &backend.QueryDataResponse{ result := &backend.QueryDataResponse{
Responses: map[string]backend.DataResponse{}, Responses: map[string]backend.DataResponse{},
} }

View File

@ -66,6 +66,7 @@ export default function createMockDatasource(overrides?: DeepPartial<Datasource>
azureLogAnalyticsDatasource: { azureLogAnalyticsDatasource: {
getKustoSchema: () => Promise.resolve(), getKustoSchema: () => Promise.resolve(),
getDeprecatedDefaultWorkSpace: () => 'defaultWorkspaceId', getDeprecatedDefaultWorkSpace: () => 'defaultWorkspaceId',
getBasicLogsQueryUsage: jest.fn(),
}, },
resourcePickerData: { resourcePickerData: {
getSubscriptions: () => jest.fn().mockResolvedValue([]), getSubscriptions: () => jest.fn().mockResolvedValue([]),

View File

@ -130,6 +130,7 @@ export default class AzureLogAnalyticsDatasource extends DataSourceWithBackend<
// Workspace was removed in Grafana 8, but remains for backwards compat // Workspace was removed in Grafana 8, but remains for backwards compat
workspace, workspace,
dashboardTime: item.dashboardTime, dashboardTime: item.dashboardTime,
basicLogsQuery: item.basicLogsQuery,
timeColumn: this.templateSrv.replace(item.timeColumn, scopedVars), timeColumn: this.templateSrv.replace(item.timeColumn, scopedVars),
}, },
}; };
@ -252,4 +253,17 @@ export default class AzureLogAnalyticsDatasource extends DataSourceWithBackend<
async getAzureLogAnalyticsCheatsheetQueries() { async getAzureLogAnalyticsCheatsheetQueries() {
return await this.getResource(`${this.resourcePath}/v1/metadata`); return await this.getResource(`${this.resourcePath}/v1/metadata`);
} }
async getBasicLogsQueryUsage(query: AzureMonitorQuery, table: string): Promise<number> {
const templateSrv = getTemplateSrv();
const data = {
table: table,
resource: templateSrv.replace(query.azureLogAnalytics?.resources?.[0]),
queryType: query.queryType,
from: templateSrv.replace('$__from'),
to: templateSrv.replace('$__to'),
};
return await this.postResource(`${this.resourcePath}/usage/basiclogs`, data);
}
} }

View File

@ -44,6 +44,7 @@ export default class AzureMonitorDatasource extends DataSourceWithBackend<AzureM
providerApiVersion = '2021-04-01'; providerApiVersion = '2021-04-01';
locationsApiVersion = '2020-01-01'; locationsApiVersion = '2020-01-01';
defaultSubscriptionId?: string; defaultSubscriptionId?: string;
basicLogsEnabled?: boolean;
resourcePath: string; resourcePath: string;
azurePortalUrl: string; azurePortalUrl: string;
declare resourceGroup: string; declare resourceGroup: string;
@ -56,6 +57,7 @@ export default class AzureMonitorDatasource extends DataSourceWithBackend<AzureM
super(instanceSettings); super(instanceSettings);
this.defaultSubscriptionId = instanceSettings.jsonData.subscriptionId; this.defaultSubscriptionId = instanceSettings.jsonData.subscriptionId;
this.basicLogsEnabled = instanceSettings.jsonData.basicLogsEnabled;
const cloud = getAzureCloud(instanceSettings); const cloud = getAzureCloud(instanceSettings);
this.resourcePath = routeNames.azureMonitor; this.resourcePath = routeNames.azureMonitor;

View File

@ -0,0 +1,21 @@
import { render, screen } from '@testing-library/react';
import React from 'react';
import { createMockInstanceSetttings } from '../../__mocks__/instanceSettings';
import { BasicLogsToggle, Props } from './BasicLogsToggle';
const mockInstanceSettings = createMockInstanceSetttings();
const defaultProps: Props = {
options: mockInstanceSettings.jsonData,
onBasicLogsEnabledChange: jest.fn(),
};
describe('BasicLogsToggle', () => {
it('should render component', () => {
render(<BasicLogsToggle {...defaultProps} />);
expect(screen.getByText('Enable Basic Logs')).toBeInTheDocument();
});
});

View File

@ -0,0 +1,52 @@
import { css } from '@emotion/css';
import React from 'react';
import { Field, Switch, useTheme2 } from '@grafana/ui';
import { AzureDataSourceJsonData } from '../../types';
export interface Props {
options: AzureDataSourceJsonData;
onBasicLogsEnabledChange: (basicLogsEnabled: boolean) => void;
}
export const BasicLogsToggle = (props: Props) => {
const { options, onBasicLogsEnabledChange } = props;
const theme = useTheme2();
const styles = {
text: css({
...theme.typography.body,
color: theme.colors.text.secondary,
fontSize: '11px',
a: css({
color: theme.colors.text.link,
textDecoration: 'underline',
'&:hover': {
textDecoration: 'none',
},
}),
}),
};
const onChange = (e: React.ChangeEvent<HTMLInputElement>) => onBasicLogsEnabledChange(e.target.checked);
const description = (
<p className={styles.text}>
Enabling this feature incurs Azure Monitor per-query costs on dashboard panels that query tables configured for{' '}
<a
href="https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-configure?tabs=portal-1"
target="__blank"
rel="noreferrer"
>
Basic Logs
</a>
.
</p>
);
return (
<Field description={description} label="Enable Basic Logs">
<div>
<Switch aria-label="Basic Logs" onChange={onChange} value={options.basicLogsEnabled ?? false} />
</div>
</Field>
);
};

View File

@ -8,6 +8,7 @@ import { getCredentials, updateCredentials } from '../../credentials';
import { AzureDataSourceSettings, AzureCredentials } from '../../types'; import { AzureDataSourceSettings, AzureCredentials } from '../../types';
import { AzureCredentialsForm } from './AzureCredentialsForm'; import { AzureCredentialsForm } from './AzureCredentialsForm';
import { BasicLogsToggle } from './BasicLogsToggle';
import { DefaultSubscription } from './DefaultSubscription'; import { DefaultSubscription } from './DefaultSubscription';
const legacyAzureClouds: SelectableValue[] = [ const legacyAzureClouds: SelectableValue[] = [
@ -49,6 +50,9 @@ export const MonitorConfig = (props: Props) => {
const onSubscriptionChange = (subscriptionId?: string) => const onSubscriptionChange = (subscriptionId?: string) =>
updateOptions((options) => ({ ...options, jsonData: { ...options.jsonData, subscriptionId } })); updateOptions((options) => ({ ...options, jsonData: { ...options.jsonData, subscriptionId } }));
const onBasicLogsEnabledChange = (enableBasicLogs: boolean) =>
updateOptions((options) => ({ ...options, jsonData: { ...options.jsonData, basicLogsEnabled: enableBasicLogs } }));
// The auth type needs to be set on the first load of the data source // The auth type needs to be set on the first load of the data source
useEffectOnce(() => { useEffectOnce(() => {
if (!options.jsonData.authType) { if (!options.jsonData.authType) {
@ -68,15 +72,18 @@ export const MonitorConfig = (props: Props) => {
onCredentialsChange={onCredentialsChange} onCredentialsChange={onCredentialsChange}
disabled={props.options.readOnly} disabled={props.options.readOnly}
> >
<DefaultSubscription <>
subscriptions={subscriptions} <DefaultSubscription
credentials={credentials} subscriptions={subscriptions}
getSubscriptions={getSubscriptions} credentials={credentials}
disabled={props.options.readOnly} getSubscriptions={getSubscriptions}
onSubscriptionsChange={onSubscriptionsChange} disabled={props.options.readOnly}
onSubscriptionChange={onSubscriptionChange} onSubscriptionsChange={onSubscriptionsChange}
options={options.jsonData} onSubscriptionChange={onSubscriptionChange}
/> options={options.jsonData}
/>
<BasicLogsToggle options={options.jsonData} onBasicLogsEnabledChange={onBasicLogsEnabledChange} />
</>
</AzureCredentialsForm> </AzureCredentialsForm>
</> </>
); );

View File

@ -0,0 +1,142 @@
import { render, screen } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import React from 'react';
import createMockDatasource from '../../__mocks__/datasource';
import createMockQuery from '../../__mocks__/query';
import { LogsManagement } from './LogsManagement';
const variableOptionGroup = {
label: 'Template variables',
options: [],
};
describe('LogsQueryEditor.LogsManagement', () => {
it('should set Basic Logs to true if Basic is clicked and acknowledged', async () => {
const mockDatasource = createMockDatasource();
const query = createMockQuery({ azureLogAnalytics: { basicLogsQuery: undefined } });
const onChange = jest.fn();
render(
<LogsManagement
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onQueryChange={onChange}
setError={() => {}}
/>
);
const logsManagementOption = await screen.findByLabelText('Basic');
await userEvent.click(logsManagementOption);
// ensures that modal shows
expect(await screen.findByText('Basic Logs Queries')).toBeInTheDocument();
const acknowledgedAction = await screen.findByText('Confirm');
await userEvent.click(acknowledgedAction);
expect(onChange).toBeCalledWith(
expect.objectContaining({
azureLogAnalytics: expect.objectContaining({
basicLogsQuery: true,
query: '',
dashboardTime: true,
}),
})
);
});
it('should set Basic Logs to false if Analytics is clicked', async () => {
const mockDatasource = createMockDatasource();
const query = createMockQuery({ azureLogAnalytics: { basicLogsQuery: true } });
const onChange = jest.fn();
render(
<LogsManagement
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onQueryChange={onChange}
setError={() => {}}
/>
);
const logsManagementOption = await screen.findByLabelText('Analytics');
await userEvent.click(logsManagementOption);
expect(onChange).toBeCalledWith(
expect.objectContaining({
azureLogAnalytics: expect.objectContaining({
basicLogsQuery: false,
query: '',
}),
})
);
});
it('should set Basic Logs to true if Basic is clicked and clear query', async () => {
const mockDatasource = createMockDatasource();
const query = createMockQuery({ azureLogAnalytics: { basicLogsQuery: undefined, query: 'table | my test query' } });
const onChange = jest.fn();
render(
<LogsManagement
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onQueryChange={onChange}
setError={() => {}}
/>
);
const logsManagementOption = await screen.findByLabelText('Basic');
await userEvent.click(logsManagementOption);
const acknowledgedAction = await screen.findByText('Confirm');
await userEvent.click(acknowledgedAction);
expect(onChange).toBeCalledWith(
expect.objectContaining({
azureLogAnalytics: expect.objectContaining({
basicLogsQuery: true,
query: '',
dashboardTime: true,
}),
})
);
});
it('should handle modal acknowledgements - cancel', async () => {
const mockDatasource = createMockDatasource();
const query = createMockQuery({ azureLogAnalytics: { basicLogsQuery: undefined } });
const onChange = jest.fn();
render(
<LogsManagement
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onQueryChange={onChange}
setError={() => {}}
/>
);
const logsManagementOption = await screen.findByLabelText('Basic');
await userEvent.click(logsManagementOption);
// ensures that modal shows
expect(await screen.findByText('Basic Logs Queries')).toBeInTheDocument();
const cancelAcknowledgement = await screen.findByText('Cancel');
await userEvent.click(cancelAcknowledgement);
//ensures that if cancel is clicked, Logs is set back to analytics
expect(onChange).toBeCalledWith(
expect.objectContaining({
azureLogAnalytics: expect.objectContaining({
basicLogsQuery: false,
}),
})
);
});
});

View File

@ -0,0 +1,51 @@
import React, { useState } from 'react';
import { ConfirmModal, InlineField, RadioButtonGroup } from '@grafana/ui';
import { AzureQueryEditorFieldProps } from '../../types';
import { setBasicLogsQuery, setDashboardTime, setKustoQuery } from './setQueryValue';
export function LogsManagement({ query, onQueryChange: onChange }: AzureQueryEditorFieldProps) {
const [basicLogsAckOpen, setBasicLogsAckOpen] = useState<boolean>(false);
return (
<>
<ConfirmModal
isOpen={basicLogsAckOpen}
title="Basic Logs Queries"
body="Are you sure you want to switch to Basic Logs?"
description="Basic Logs queries incur cost based on the amount of data scanned."
confirmText="Confirm"
onConfirm={() => {
setBasicLogsAckOpen(false);
let updatedBasicLogsQuery = setBasicLogsQuery(query, true);
// if basic logs selected, set dashboard time
updatedBasicLogsQuery = setDashboardTime(updatedBasicLogsQuery, 'dashboard');
onChange(setKustoQuery(updatedBasicLogsQuery, ''));
}}
onDismiss={() => {
setBasicLogsAckOpen(false);
onChange(setBasicLogsQuery(query, false));
}}
confirmButtonVariant="primary"
/>
<InlineField label="Logs" tooltip={<span>Specifies whether to run a Basic or Analytics Logs query.</span>}>
<RadioButtonGroup
options={[
{ label: 'Analytics', value: false },
{ label: 'Basic', value: true },
]}
value={query.azureLogAnalytics?.basicLogsQuery ?? false}
size={'md'}
onChange={(val) => {
setBasicLogsAckOpen(val);
if (!val) {
const updatedBasicLogsQuery = setBasicLogsQuery(query, val);
onChange(setKustoQuery(updatedBasicLogsQuery, ''));
}
}}
/>
</InlineField>
</>
);
}

View File

@ -1,4 +1,4 @@
import { render, screen } from '@testing-library/react'; import { act, render, screen, waitFor } from '@testing-library/react';
import userEvent from '@testing-library/user-event'; import userEvent from '@testing-library/user-event';
import React from 'react'; import React from 'react';
@ -14,6 +14,9 @@ jest.mock('@grafana/runtime', () => ({
...jest.requireActual('@grafana/runtime'), ...jest.requireActual('@grafana/runtime'),
getTemplateSrv: () => ({ getTemplateSrv: () => ({
replace: (val: string) => { replace: (val: string) => {
if (val === '$ws') {
return '/subscriptions/def-456/resourceGroups/dev-3/providers/microsoft.operationalinsights/workspaces/la-workspace';
}
return val; return val;
}, },
}), }),
@ -40,6 +43,7 @@ describe('LogsQueryEditor', () => {
delete query?.subscription; delete query?.subscription;
delete query?.azureLogAnalytics?.resources; delete query?.azureLogAnalytics?.resources;
const onChange = jest.fn(); const onChange = jest.fn();
const basicLogsEnabled = false;
render( render(
<LogsQueryEditor <LogsQueryEditor
@ -48,6 +52,7 @@ describe('LogsQueryEditor', () => {
variableOptionGroup={variableOptionGroup} variableOptionGroup={variableOptionGroup}
onChange={onChange} onChange={onChange}
setError={() => {}} setError={() => {}}
basicLogsEnabled={basicLogsEnabled}
/> />
); );
@ -87,6 +92,7 @@ describe('LogsQueryEditor', () => {
const query = createMockQuery(); const query = createMockQuery();
delete query?.subscription; delete query?.subscription;
delete query?.azureLogAnalytics?.resources; delete query?.azureLogAnalytics?.resources;
const basicLogsEnabled = false;
const onChange = jest.fn(); const onChange = jest.fn();
render( render(
@ -96,6 +102,7 @@ describe('LogsQueryEditor', () => {
variableOptionGroup={variableOptionGroup} variableOptionGroup={variableOptionGroup}
onChange={onChange} onChange={onChange}
setError={() => {}} setError={() => {}}
basicLogsEnabled={basicLogsEnabled}
/> />
); );
@ -120,6 +127,7 @@ describe('LogsQueryEditor', () => {
const query = createMockQuery(); const query = createMockQuery();
delete query?.subscription; delete query?.subscription;
delete query?.azureLogAnalytics?.resources; delete query?.azureLogAnalytics?.resources;
const basicLogsEnabled = false;
const onChange = jest.fn(); const onChange = jest.fn();
render( render(
@ -129,6 +137,7 @@ describe('LogsQueryEditor', () => {
variableOptionGroup={variableOptionGroup} variableOptionGroup={variableOptionGroup}
onChange={onChange} onChange={onChange}
setError={() => {}} setError={() => {}}
basicLogsEnabled={basicLogsEnabled}
/> />
); );
@ -153,6 +162,7 @@ describe('LogsQueryEditor', () => {
const query = createMockQuery(); const query = createMockQuery();
delete query?.subscription; delete query?.subscription;
delete query?.azureLogAnalytics?.resources; delete query?.azureLogAnalytics?.resources;
const basicLogsEnabled = false;
const onChange = jest.fn(); const onChange = jest.fn();
render( render(
@ -162,6 +172,7 @@ describe('LogsQueryEditor', () => {
variableOptionGroup={variableOptionGroup} variableOptionGroup={variableOptionGroup}
onChange={onChange} onChange={onChange}
setError={() => {}} setError={() => {}}
basicLogsEnabled={basicLogsEnabled}
/> />
); );
@ -190,6 +201,7 @@ describe('LogsQueryEditor', () => {
it('should update the dashboardTime prop', async () => { it('should update the dashboardTime prop', async () => {
const mockDatasource = createMockDatasource({ resourcePickerData: createMockResourcePickerData() }); const mockDatasource = createMockDatasource({ resourcePickerData: createMockResourcePickerData() });
const query = createMockQuery(); const query = createMockQuery();
const basicLogsEnabled = false;
const onChange = jest.fn(); const onChange = jest.fn();
render( render(
@ -199,6 +211,7 @@ describe('LogsQueryEditor', () => {
variableOptionGroup={variableOptionGroup} variableOptionGroup={variableOptionGroup}
onChange={onChange} onChange={onChange}
setError={() => {}} setError={() => {}}
basicLogsEnabled={basicLogsEnabled}
/> />
); );
@ -218,6 +231,7 @@ describe('LogsQueryEditor', () => {
it('should show the link button', async () => { it('should show the link button', async () => {
const mockDatasource = createMockDatasource({ resourcePickerData: createMockResourcePickerData() }); const mockDatasource = createMockDatasource({ resourcePickerData: createMockResourcePickerData() });
const query = createMockQuery(); const query = createMockQuery();
const basicLogsEnabled = false;
const onChange = jest.fn(); const onChange = jest.fn();
const date = dateTime(new Date()); const date = dateTime(new Date());
@ -228,6 +242,7 @@ describe('LogsQueryEditor', () => {
variableOptionGroup={variableOptionGroup} variableOptionGroup={variableOptionGroup}
onChange={onChange} onChange={onChange}
setError={() => {}} setError={() => {}}
basicLogsEnabled={basicLogsEnabled}
data={{ data={{
state: LoadingState.Done, state: LoadingState.Done,
timeRange: { timeRange: {
@ -246,4 +261,242 @@ describe('LogsQueryEditor', () => {
expect(await screen.findByText('View query in Azure Portal')).toBeInTheDocument(); expect(await screen.findByText('View query in Azure Portal')).toBeInTheDocument();
}); });
}); });
describe('basic logs toggle', () => {
it('should show basic logs toggle', async () => {
const mockDatasource = createMockDatasource({ resourcePickerData: createMockResourcePickerData() });
const query = createMockQuery({
azureLogAnalytics: {
resources: [
'/subscriptions/def-456/resourceGroups/dev-3/providers/microsoft.operationalinsights/workspaces/la-workspace',
],
},
});
const basicLogsEnabled = true;
const onChange = jest.fn();
await act(async () => {
render(
<LogsQueryEditor
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onChange={onChange}
setError={() => {}}
basicLogsEnabled={basicLogsEnabled}
/>
);
});
expect(await screen.findByLabelText('Basic')).toBeInTheDocument();
});
it('should show basic logs toggle for workspace variables', async () => {
const mockDatasource = createMockDatasource({ resourcePickerData: createMockResourcePickerData() });
const query = createMockQuery({
azureLogAnalytics: {
resources: ['$ws'],
},
});
const basicLogsEnabled = true;
const onChange = jest.fn();
await act(async () => {
render(
<LogsQueryEditor
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onChange={onChange}
setError={() => {}}
basicLogsEnabled={basicLogsEnabled}
/>
);
});
expect(await screen.findByLabelText('Basic')).toBeInTheDocument();
});
it('should not show basic logs toggle - basic logs not enabled', async () => {
const mockDatasource = createMockDatasource({ resourcePickerData: createMockResourcePickerData() });
const query = createMockQuery({
azureLogAnalytics: {
resources: [
'/subscriptions/def-456/resourceGroups/dev-3/providers/microsoft.operationalinsights/workspaces/la-workspace',
],
},
});
const basicLogsEnabled = false;
const onChange = jest.fn();
await act(async () => {
render(
<LogsQueryEditor
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onChange={onChange}
setError={() => {}}
basicLogsEnabled={basicLogsEnabled}
/>
);
});
expect(await screen.queryByLabelText('Basic')).not.toBeInTheDocument();
});
it('should not show basic logs toggle for non workspace variables', async () => {
const mockDatasource = createMockDatasource({ resourcePickerData: createMockResourcePickerData() });
const query = createMockQuery({
azureLogAnalytics: {
resources: ['$non_ws_var'],
},
});
const basicLogsEnabled = true;
const onChange = jest.fn();
await act(async () => {
render(
<LogsQueryEditor
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onChange={onChange}
setError={() => {}}
basicLogsEnabled={basicLogsEnabled}
/>
);
});
expect(await screen.queryByLabelText('Basic')).not.toBeInTheDocument();
});
it('should not show basic logs toggle - selected resource is not LA workspace', async () => {
const mockDatasource = createMockDatasource({ resourcePickerData: createMockResourcePickerData() });
const query = createMockQuery({
azureLogAnalytics: {
resources: [
'/subscriptions/def-456/resourceGroups/dev-3/providers/Microsoft.Compute/virtualMachines/web-server',
],
},
});
const basicLogsEnabled = true;
const onChange = jest.fn();
await act(async () => {
render(
<LogsQueryEditor
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onChange={onChange}
setError={() => {}}
basicLogsEnabled={basicLogsEnabled}
/>
);
});
expect(await screen.queryByLabelText('Basic')).not.toBeInTheDocument();
});
});
describe('data ingestion warning', () => {
it('should show generic data ingested warning when running basic logs queries', async () => {
const mockDatasource = createMockDatasource();
const onChange = jest.fn();
const query = createMockQuery({
azureLogAnalytics: {
resources: [
'/subscriptions/def-456/resourceGroups/dev-3/providers/microsoft.operationalinsights/workspaces/la-workspace',
],
basicLogsQuery: true,
},
});
mockDatasource.azureLogAnalyticsDatasource.getBasicLogsQueryUsage.mockResolvedValue(0);
await act(async () => {
render(
<LogsQueryEditor
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onChange={onChange}
setError={() => {}}
basicLogsEnabled={true}
/>
);
});
await act(async () => {
await waitFor(() =>
expect(
screen.findByText(/This is a Basic Logs query and incurs cost per GiB scanned./)
).resolves.toBeInTheDocument()
);
});
});
it('should show data ingested warning when running basic logs queries', async () => {
const mockDatasource = createMockDatasource();
const onChange = jest.fn();
const query = createMockQuery({
azureLogAnalytics: {
resources: [
'/subscriptions/def-456/resourceGroups/dev-3/providers/microsoft.operationalinsights/workspaces/la-workspace',
],
basicLogsQuery: true,
},
});
mockDatasource.azureLogAnalyticsDatasource.getBasicLogsQueryUsage.mockResolvedValue(0.45);
await act(async () => {
render(
<LogsQueryEditor
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onChange={onChange}
setError={() => {}}
basicLogsEnabled={true}
/>
);
});
await act(async () => {
await waitFor(() =>
expect(screen.findByText(/This query is processing 0.45 GiB when run./)).resolves.toBeInTheDocument()
);
});
});
it('should not show data ingested warning when running basic logs queries', async () => {
const mockDatasource = createMockDatasource();
const onChange = jest.fn();
const query = createMockQuery({
azureLogAnalytics: {
resources: [
'/subscriptions/def-456/resourceGroups/dev-3/providers/microsoft.operationalinsights/workspaces/la-workspace',
],
basicLogsQuery: true,
query: '',
},
});
mockDatasource.azureLogAnalyticsDatasource.getBasicLogsQueryUsage.mockResolvedValue(0.5);
await act(async () => {
render(
<LogsQueryEditor
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onChange={onChange}
setError={() => {}}
basicLogsEnabled={true}
/>
);
});
expect(await screen.queryByLabelText(/This query is processing 0.50 GiB when run./)).not.toBeInTheDocument();
});
});
}); });

View File

@ -2,7 +2,8 @@ import React, { useEffect, useState } from 'react';
import { PanelData, TimeRange } from '@grafana/data'; import { PanelData, TimeRange } from '@grafana/data';
import { EditorFieldGroup, EditorRow, EditorRows } from '@grafana/experimental'; import { EditorFieldGroup, EditorRow, EditorRows } from '@grafana/experimental';
import { Alert, LinkButton } from '@grafana/ui'; import { getTemplateSrv } from '@grafana/runtime';
import { Alert, LinkButton, Text, TextLink } from '@grafana/ui';
import Datasource from '../../datasource'; import Datasource from '../../datasource';
import { selectors } from '../../e2e/selectors'; import { selectors } from '../../e2e/selectors';
@ -13,14 +14,19 @@ import { parseResourceDetails } from '../ResourcePicker/utils';
import FormatAsField from '../shared/FormatAsField'; import FormatAsField from '../shared/FormatAsField';
import AdvancedResourcePicker from './AdvancedResourcePicker'; import AdvancedResourcePicker from './AdvancedResourcePicker';
import { LogsManagement } from './LogsManagement';
import QueryField from './QueryField'; import QueryField from './QueryField';
import { TimeManagement } from './TimeManagement'; import { TimeManagement } from './TimeManagement';
import { setFormatAs } from './setQueryValue'; import { setBasicLogsQuery, setFormatAs, setKustoQuery } from './setQueryValue';
import useMigrations from './useMigrations'; import useMigrations from './useMigrations';
import { calculateTimeRange, shouldShowBasicLogsToggle } from './utils';
const MAX_DATA_RETENTION_DAYS = 8; // limit is only for basic logs
interface LogsQueryEditorProps { interface LogsQueryEditorProps {
query: AzureMonitorQuery; query: AzureMonitorQuery;
datasource: Datasource; datasource: Datasource;
basicLogsEnabled: boolean;
subscriptionId?: string; subscriptionId?: string;
onChange: (newQuery: AzureMonitorQuery) => void; onChange: (newQuery: AzureMonitorQuery) => void;
variableOptionGroup: { label: string; options: AzureMonitorOption[] }; variableOptionGroup: { label: string; options: AzureMonitorOption[] };
@ -33,6 +39,7 @@ interface LogsQueryEditorProps {
const LogsQueryEditor = ({ const LogsQueryEditor = ({
query, query,
datasource, datasource,
basicLogsEnabled,
subscriptionId, subscriptionId,
variableOptionGroup, variableOptionGroup,
onChange, onChange,
@ -42,6 +49,15 @@ const LogsQueryEditor = ({
data, data,
}: LogsQueryEditorProps) => { }: LogsQueryEditorProps) => {
const migrationError = useMigrations(datasource, query, onChange); const migrationError = useMigrations(datasource, query, onChange);
const [showBasicLogsToggle, setShowBasicLogsToggle] = useState<boolean>(
shouldShowBasicLogsToggle(query.azureLogAnalytics?.resources || [], basicLogsEnabled)
);
const [showDataRetentionWarning, setShowDataRetentionWarning] = useState<boolean>(false);
const [dataIngestedWarning, setDataIngestedWarning] = useState<React.ReactNode | null>(null);
const templateSrv = getTemplateSrv();
const from = templateSrv?.replace('$__from');
const to = templateSrv?.replace('$__to');
const disableRow = (row: ResourceRow, selectedRows: ResourceRowGroup) => { const disableRow = (row: ResourceRow, selectedRows: ResourceRowGroup) => {
if (selectedRows.length === 0) { if (selectedRows.length === 0) {
// Only if there is some resource(s) selected we should disable rows // Only if there is some resource(s) selected we should disable rows
@ -65,6 +81,66 @@ const LogsQueryEditor = ({
} }
}, [query.azureLogAnalytics?.resources, datasource.azureLogAnalyticsDatasource]); }, [query.azureLogAnalytics?.resources, datasource.azureLogAnalyticsDatasource]);
useEffect(() => {
if (shouldShowBasicLogsToggle(query.azureLogAnalytics?.resources || [], basicLogsEnabled)) {
setShowBasicLogsToggle(true);
} else {
setShowBasicLogsToggle(false);
}
}, [basicLogsEnabled, query.azureLogAnalytics?.resources, templateSrv]);
useEffect(() => {
if ((!basicLogsEnabled || !showBasicLogsToggle) && query.azureLogAnalytics?.basicLogsQuery) {
const updatedBasicLogsQuery = setBasicLogsQuery(query, false);
onChange(setKustoQuery(updatedBasicLogsQuery, ''));
}
}, [basicLogsEnabled, onChange, query, showBasicLogsToggle]);
useEffect(() => {
const timeRange = calculateTimeRange(parseInt(from, 10), parseInt(to, 10));
// Basic logs data retention is fixed at 8 days
// need to add this check to make user aware of this limitation in case they have selected a longer time range
if (showBasicLogsToggle && query.azureLogAnalytics?.basicLogsQuery && timeRange > MAX_DATA_RETENTION_DAYS) {
setShowDataRetentionWarning(true);
} else {
setShowDataRetentionWarning(false);
}
}, [query.azureLogAnalytics?.basicLogsQuery, showBasicLogsToggle, from, to]);
useEffect(() => {
const getBasicLogsUsage = async (query: AzureMonitorQuery) => {
try {
if (showBasicLogsToggle && query.azureLogAnalytics?.basicLogsQuery && !!query.azureLogAnalytics.query) {
const querySplit = query.azureLogAnalytics.query.split('|');
// Basic Logs queries are required to start the query with a table
const table = querySplit[0].trim();
const dataIngested = await datasource.azureLogAnalyticsDatasource.getBasicLogsQueryUsage(query, table);
const textToShow = !!dataIngested
? `This query is processing ${dataIngested} GiB when run. `
: 'This is a Basic Logs query and incurs cost per GiB scanned. ';
setDataIngestedWarning(
<>
<Text color="primary">
{textToShow}{' '}
<TextLink
href="https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-configure?tabs=portal-1"
external
>
Learn More
</TextLink>
</Text>
</>
);
} else {
setDataIngestedWarning(null);
}
} catch (err) {
console.error(err);
}
};
getBasicLogsUsage(query).catch((err) => console.error(err));
}, [datasource.azureLogAnalyticsDatasource, query, showBasicLogsToggle, from, to]);
let portalLinkButton = null; let portalLinkButton = null;
if (data?.series) { if (data?.series) {
@ -116,6 +192,15 @@ const LogsQueryEditor = ({
)} )}
selectionNotice={() => 'You may only choose items of the same resource type.'} selectionNotice={() => 'You may only choose items of the same resource type.'}
/> />
{showBasicLogsToggle && (
<LogsManagement
query={query}
datasource={datasource}
variableOptionGroup={variableOptionGroup}
onQueryChange={onChange}
setError={setError}
/>
)}
<TimeManagement <TimeManagement
query={query} query={query}
datasource={datasource} datasource={datasource}
@ -135,6 +220,7 @@ const LogsQueryEditor = ({
setError={setError} setError={setError}
schema={schema} schema={schema}
/> />
{dataIngestedWarning}
<EditorRow> <EditorRow>
<EditorFieldGroup> <EditorFieldGroup>
{!hideFormatAs && ( {!hideFormatAs && (
@ -161,6 +247,13 @@ const LogsQueryEditor = ({
</EditorFieldGroup> </EditorFieldGroup>
</EditorRow> </EditorRow>
</EditorRows> </EditorRows>
{showDataRetentionWarning && (
<Alert severity="warning" title="Basic Logs data retention">
<Text>
Data retention for Basic Logs is fixed at eight days. You will only see data within this timeframe.
</Text>
</Alert>
)}
</span> </span>
); );
}; };

View File

@ -169,4 +169,24 @@ describe('LogsQueryEditor.TimeManagement', () => {
expect(onChange).not.toBeCalled(); expect(onChange).not.toBeCalled();
expect(screen.getByText('Alert > TestTimeColumn')).toBeInTheDocument(); expect(screen.getByText('Alert > TestTimeColumn')).toBeInTheDocument();
}); });
it('should set time to dashboard and query disabled if basic logs is selected', async () => {
const mockDatasource = createMockDatasource();
const query = createMockQuery({ azureLogAnalytics: { basicLogsQuery: true, dashboardTime: true } });
const onChange = jest.fn();
render(
<TimeManagement
query={query}
datasource={mockDatasource}
variableOptionGroup={variableOptionGroup}
onQueryChange={onChange}
setError={() => {}}
schema={FakeSchemaData.getLogAnalyticsFakeEngineSchema()}
/>
);
expect(screen.getByLabelText('Query')).toBeDisabled();
expect(screen.getByLabelText('Dashboard')).toBeChecked();
});
}); });

View File

@ -10,6 +10,7 @@ import { setDashboardTime, setTimeColumn } from './setQueryValue';
export function TimeManagement({ query, onQueryChange: onChange, schema }: AzureQueryEditorFieldProps) { export function TimeManagement({ query, onQueryChange: onChange, schema }: AzureQueryEditorFieldProps) {
const [defaultTimeColumns, setDefaultTimeColumns] = useState<SelectableValue[] | undefined>(); const [defaultTimeColumns, setDefaultTimeColumns] = useState<SelectableValue[] | undefined>();
const [timeColumns, setTimeColumns] = useState<SelectableValue[] | undefined>(); const [timeColumns, setTimeColumns] = useState<SelectableValue[] | undefined>();
const [disabledTimePicker, setDisabledTimePicker] = useState<boolean>(false);
const setDefaultColumn = useCallback((column: string) => onChange(setTimeColumn(query, column)), [query, onChange]); const setDefaultColumn = useCallback((column: string) => onChange(setTimeColumn(query, column)), [query, onChange]);
@ -76,6 +77,14 @@ export function TimeManagement({ query, onQueryChange: onChange, schema }: Azure
}, },
[onChange, query] [onChange, query]
); );
useEffect(() => {
if (query.azureLogAnalytics?.basicLogsQuery) {
setDisabledTimePicker(true);
} else {
setDisabledTimePicker(false);
}
}, [query.azureLogAnalytics?.basicLogsQuery]);
return ( return (
<> <>
<InlineField <InlineField
@ -89,12 +98,14 @@ export function TimeManagement({ query, onQueryChange: onChange, schema }: Azure
> >
<RadioButtonGroup <RadioButtonGroup
options={[ options={[
{ label: 'Query', value: false }, { label: 'Query', value: 'query' },
{ label: 'Dashboard', value: true }, { label: 'Dashboard', value: 'dashboard' },
]} ]}
value={query.azureLogAnalytics?.dashboardTime ?? false} value={query.azureLogAnalytics?.dashboardTime ? 'dashboard' : 'query'}
size={'md'} size={'md'}
onChange={(val) => onChange(setDashboardTime(query, val))} onChange={(val) => onChange(setDashboardTime(query, val))}
disabled={disabledTimePicker}
disabledOptions={disabledTimePicker ? ['query'] : []}
/> />
</InlineField> </InlineField>
{query.azureLogAnalytics?.dashboardTime && ( {query.azureLogAnalytics?.dashboardTime && (

View File

@ -20,12 +20,12 @@ export function setFormatAs(query: AzureMonitorQuery, formatAs: ResultFormat): A
}; };
} }
export function setDashboardTime(query: AzureMonitorQuery, dashboardTime: boolean): AzureMonitorQuery { export function setDashboardTime(query: AzureMonitorQuery, dashboardTime: string): AzureMonitorQuery {
return { return {
...query, ...query,
azureLogAnalytics: { azureLogAnalytics: {
...query.azureLogAnalytics, ...query.azureLogAnalytics,
dashboardTime, dashboardTime: dashboardTime === 'dashboard' ? true : false,
}, },
}; };
} }
@ -39,3 +39,13 @@ export function setTimeColumn(query: AzureMonitorQuery, timeColumn: string): Azu
}, },
}; };
} }
export function setBasicLogsQuery(query: AzureMonitorQuery, basicLogsQuery: boolean): AzureMonitorQuery {
return {
...query,
azureLogAnalytics: {
...query.azureLogAnalytics,
basicLogsQuery,
},
};
}

View File

@ -0,0 +1,101 @@
import { calculateTimeRange, shouldShowBasicLogsToggle } from './utils';
jest.mock('@grafana/runtime', () => ({
...jest.requireActual('@grafana/runtime'),
getTemplateSrv: () => ({
replace: (val: string) => {
if (val === '$ws') {
return '/subscriptions/def-456/resourceGroups/dev-3/providers/microsoft.operationalinsights/workspaces/la-workspace';
}
return val;
},
}),
}));
describe('LogsQueryEditor utils', () => {
describe('shouldShowBasicLogsToggle', () => {
it('should return false if basic logs are not enabled', () => {
expect(
shouldShowBasicLogsToggle(
[
'/subscriptions/def-456/resourceGroups/dev-3/providers/microsoft.operationalinsights/workspaces/la-workspace',
],
false
)
).toBe(false);
});
it('should return false if selected resource is not an LA workspace', () => {
expect(
shouldShowBasicLogsToggle(
[
'/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/resourceGroups/cloud-datasources/providers/Microsoft.Storage/storageAccounts/csb100320016c43d2d0/fileServices/default',
],
true
)
).toBe(false);
});
it('should return false if no resources are selected', () => {
expect(shouldShowBasicLogsToggle([], true)).toBe(false);
});
it('should return false if more than one resource is selected', () => {
expect(
shouldShowBasicLogsToggle(
[
'/subscriptions/def-456/resourceGroups/dev-3/providers/microsoft.operationalinsights/workspaces/la-workspace',
'/subscriptions/def-456/resourceGroups/dev-3/providers/microsoft.OperationalInsights/workspaces/la-workspace2',
],
true
)
).toBe(false);
});
it('should return true if basic logs are enabled and selected single resource is an LA workspace', () => {
expect(
shouldShowBasicLogsToggle(
[
'/subscriptions/def-456/resourceGroups/dev-3/providers/microsoft.operationalinsights/workspaces/la-workspace',
],
true
)
).toBe(true);
});
it('should return true if basic logs are enabled and selected single resource is an LA workspace variable', () => {
expect(shouldShowBasicLogsToggle(['$ws'], true)).toBe(true);
});
});
describe('calculateTimeRange', () => {
it('should correctly calculate the time range in days', () => {
const from = Date.now() - 1000 * 60 * 60 * 24 * 3; // 3 days ago
const to = Date.now();
const result = calculateTimeRange(from, to);
// The result should be approximately 3
expect(result).toBeCloseTo(3, 0);
});
it('should return 0 when from and to are the same', () => {
const from = Date.now();
const to = from;
const result = calculateTimeRange(from, to);
expect(result).toBe(0);
});
it('should return a negative number when from is later than to', () => {
const from = Date.now();
const to = from - 1000 * 60 * 60 * 24; // 1 day ago
const result = calculateTimeRange(from, to);
// The result should be approximately -1
expect(result).toBeCloseTo(-1, 0);
});
});
});

View File

@ -0,0 +1,22 @@
import { getTemplateSrv } from '@grafana/runtime';
import { parseResourceURI } from '../ResourcePicker/utils';
export function shouldShowBasicLogsToggle(resources: string[], basicLogsEnabled: boolean) {
const selectedResource = getTemplateSrv()?.replace(resources[0]);
return (
basicLogsEnabled &&
resources.length === 1 &&
parseResourceURI(selectedResource).metricNamespace?.toLowerCase() === 'microsoft.operationalinsights/workspaces'
);
}
export function calculateTimeRange(from: number, to: number): number {
const second = 1000;
const minute = second * 60;
const hour = minute * 60;
const day = hour * 24;
const timeRange = (to - from) / day;
return timeRange;
}

View File

@ -25,6 +25,11 @@ jest.mock('@grafana/ui', () => ({
jest.mock('@grafana/runtime', () => ({ jest.mock('@grafana/runtime', () => ({
___esModule: true, ___esModule: true,
...jest.requireActual('@grafana/runtime'), ...jest.requireActual('@grafana/runtime'),
getTemplateSrv: () => ({
replace: (val: string) => {
return val;
},
}),
})); }));
describe('Azure Monitor QueryEditor', () => { describe('Azure Monitor QueryEditor', () => {

View File

@ -55,13 +55,16 @@ const QueryEditor = ({
const query = usePreparedQuery(baseQuery, onQueryChange); const query = usePreparedQuery(baseQuery, onQueryChange);
const subscriptionId = query.subscription || datasource.azureMonitorDatasource.defaultSubscriptionId; const subscriptionId = query.subscription || datasource.azureMonitorDatasource.defaultSubscriptionId;
const basicLogsEnabled =
datasource.azureMonitorDatasource.basicLogsEnabled &&
app !== CoreApp.UnifiedAlerting &&
app !== CoreApp.CloudAlerting;
const variableOptionGroup = { const variableOptionGroup = {
label: 'Template Variables', label: 'Template Variables',
options: datasource.getVariables().map((v) => ({ label: v, value: v })), options: datasource.getVariables().map((v) => ({ label: v, value: v })),
}; };
const isAzureAuthenticated = config.bootData.user.authenticatedBy === 'oauth_azuread'; const isAzureAuthenticated = config.bootData.user.authenticatedBy === 'oauth_azuread';
if (datasource.currentUserAuth) { if (datasource.currentUserAuth) {
if ( if (
app === CoreApp.UnifiedAlerting && app === CoreApp.UnifiedAlerting &&
@ -105,6 +108,7 @@ const QueryEditor = ({
<EditorForQueryType <EditorForQueryType
data={data} data={data}
subscriptionId={subscriptionId} subscriptionId={subscriptionId}
basicLogsEnabled={basicLogsEnabled ?? false}
query={query} query={query}
datasource={datasource} datasource={datasource}
onChange={onQueryChange} onChange={onQueryChange}
@ -127,6 +131,7 @@ const QueryEditor = ({
interface EditorForQueryTypeProps extends Omit<AzureMonitorQueryEditorProps, 'onRunQuery'> { interface EditorForQueryTypeProps extends Omit<AzureMonitorQueryEditorProps, 'onRunQuery'> {
subscriptionId?: string; subscriptionId?: string;
basicLogsEnabled: boolean;
variableOptionGroup: { label: string; options: AzureMonitorOption[] }; variableOptionGroup: { label: string; options: AzureMonitorOption[] };
setError: (source: string, error: AzureMonitorErrorish | undefined) => void; setError: (source: string, error: AzureMonitorErrorish | undefined) => void;
} }
@ -134,6 +139,7 @@ interface EditorForQueryTypeProps extends Omit<AzureMonitorQueryEditorProps, 'on
const EditorForQueryType = ({ const EditorForQueryType = ({
data, data,
subscriptionId, subscriptionId,
basicLogsEnabled,
query, query,
datasource, datasource,
variableOptionGroup, variableOptionGroup,
@ -159,6 +165,7 @@ const EditorForQueryType = ({
<LogsQueryEditor <LogsQueryEditor
data={data} data={data}
subscriptionId={subscriptionId} subscriptionId={subscriptionId}
basicLogsEnabled={basicLogsEnabled}
query={query} query={query}
datasource={datasource} datasource={datasource}
onChange={onChange} onChange={onChange}

View File

@ -13,7 +13,7 @@ import { ResourceRow, ResourceRowGroup } from './types';
// - resource groups: /subscriptions/44693801-6ee6-49de-9b2d-9106972f9572/resourceGroups/cloud-datasources // - resource groups: /subscriptions/44693801-6ee6-49de-9b2d-9106972f9572/resourceGroups/cloud-datasources
// - resources: /subscriptions/44693801-6ee6-49de-9b2d-9106972f9572/resourceGroups/cloud-datasources/providers/Microsoft.Compute/virtualMachines/GithubTestDataVM // - resources: /subscriptions/44693801-6ee6-49de-9b2d-9106972f9572/resourceGroups/cloud-datasources/providers/Microsoft.Compute/virtualMachines/GithubTestDataVM
const RESOURCE_URI_REGEX = const RESOURCE_URI_REGEX =
/\/subscriptions\/(?<subscription>[^/]+)(?:\/resourceGroups\/(?<resourceGroup>[^/]+)(?:\/providers\/(?<metricNamespaceAndResource>.+))?)?/; /\/subscriptions\/(?<subscription>[^/]+)(?:\/resourceGroups\/(?<resourceGroup>[^/]+)(?:\/providers\/(?<metricNamespaceAndResource>.+))?)?/i;
type RegexGroups = Record<string, string | undefined>; type RegexGroups = Record<string, string | undefined>;

View File

@ -18,6 +18,15 @@ jest.mock('@grafana/ui', () => ({
}, },
})); }));
jest.mock('@grafana/runtime', () => ({
...jest.requireActual('@grafana/runtime'),
getTemplateSrv: () => ({
replace: (val: string) => {
return val;
},
}),
}));
const defaultProps = { const defaultProps = {
query: { query: {
refId: 'A', refId: 'A',

View File

@ -246,6 +246,7 @@ const VariableEditor = (props: Props) => {
variableOptionGroup={variableOptionGroup} variableOptionGroup={variableOptionGroup}
setError={setError} setError={setError}
hideFormatAs={true} hideFormatAs={true}
basicLogsEnabled={datasource.azureMonitorDatasource.basicLogsEnabled ?? false}
/> />
{errorMessage && ( {errorMessage && (
<> <>

View File

@ -115,6 +115,8 @@ composableKinds: DataQuery: {
dashboardTime?: bool dashboardTime?: bool
// If dashboardTime is set to true this value dictates which column the time filter will be applied to. Defaults to the first tables timeSpan column, the first datetime column found, or TimeGenerated // If dashboardTime is set to true this value dictates which column the time filter will be applied to. Defaults to the first tables timeSpan column, the first datetime column found, or TimeGenerated
timeColumn?: string timeColumn?: string
// If set to true the query will be run as a basic logs query
basicLogsQuery?: bool
// Workspace ID. This was removed in Grafana 8, but remains for backwards compat. // Workspace ID. This was removed in Grafana 8, but remains for backwards compat.
workspace?: string workspace?: string

View File

@ -161,6 +161,10 @@ export const defaultAzureMetricQuery: Partial<AzureMetricQuery> = {
* Azure Monitor Logs sub-query properties * Azure Monitor Logs sub-query properties
*/ */
export interface AzureLogsQuery { export interface AzureLogsQuery {
/**
* If set to true the query will be run as a basic logs query
*/
basicLogsQuery?: boolean;
/** /**
* If set to true the dashboard time range will be used as a filter for the query. Otherwise the query time ranges will be used. Defaults to false. * If set to true the dashboard time range will be used as a filter for the query. Otherwise the query time ranges will be used. Defaults to false.
*/ */

View File

@ -81,6 +81,7 @@ export interface AzureDataSourceJsonData extends DataSourceJsonData {
subscriptionId?: string; subscriptionId?: string;
oauthPassThru?: boolean; oauthPassThru?: boolean;
azureCredentials?: AzureCredentials; azureCredentials?: AzureCredentials;
basicLogsEnabled?: boolean;
// logs // logs
/** @deprecated Azure Logs credentials */ /** @deprecated Azure Logs credentials */