IntenseWebs/grafana
3
0
Fork 0
mirror of https://github.com/grafana/grafana.git synced 2025-02-25 18:55:37 -06:00
Code Issues Packages Projects Releases Wiki Activity

Docs: Loki data sources updates (#72041)

Browse Source 
* added new configure loki doc

* finished first draft of config doc

* changed file name

* started updates to query editor

* updated config and qury editor

* updated index doc config doc and query editor

* updated query editor

* updates to query editor doc

* more updates to query editor and index doc

* one small update

* updates to query editor doc and index

* Update docs/sources/datasources/loki/_index.md

Co-authored-by: Matias Chomicki <matyax@gmail.com>

* Update docs/sources/datasources/loki/query-editor/index.md

Co-authored-by: Matias Chomicki <matyax@gmail.com>

* Update docs/sources/datasources/loki/query-editor/index.md

Co-authored-by: Matias Chomicki <matyax@gmail.com>

* Update docs/sources/datasources/loki/query-editor/index.md

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

* Update docs/sources/datasources/loki/_index.md

Co-authored-by: Jack Baldry <jack.baldry@grafana.com>

* made changes suggested in PR

---------

Co-authored-by: Matias Chomicki <matyax@gmail.com>
Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>
Co-authored-by: Jack Baldry <jack.baldry@grafana.com>
This commit is contained in:
lwandz13 2023-07-25 11:10:28 -05:00 committed by GitHub
parent e4649e7099
commit 443b4b0327
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 228 additions and 125 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

77
docs/sources/datasources/loki/_index.md
View File

@ -14,14 +14,25 @@ labels:
- enterprise
- oss
menuTitle: Loki
title: Loki data source
title: Configure the Loki data source
weight: 800
---
# Loki data source
Grafana ships with built-in support for [Loki](/docs/loki/latest/), an open-source log aggregation system by Grafana Labs.
This topic explains configuring and querying specific to the Loki data source.
Grafana Loki is a set of components that can be combined into a fully featured logging stack.
Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on a filesystem.
The following guides will help you get started with Loki:
- [Getting started with Loki](/docs/loki/latest/getting-started/)
- [Install Loki](/docs/loki/latest/installation/)
- [Loki best practices](/docs/loki/latest/best-practices/#best-practices)
- [Configure the Loki data source](/docs/grafana/latest/datasources/loki/configure-loki-data-source/)
- [LogQL](/docs/loki/latest/logql/)
- [Loki query editor]({{< relref "./query-editor" >}})
## Adding a data source
For instructions on how to add a data source to Grafana, refer to the [administration documentation]({{< relref "../../administration/data-source-management/" >}}).
Only users with the organization administrator role can add data sources.
@ -29,72 +40,16 @@ Administrators can also [configure the data source via YAML]({{< relref "#provis
Once you've added the Loki data source, you can [configure it]({{< relref "#configure-the-data-source" >}}) so that your Grafana instance's users can create queries in its [query editor]({{< relref "./query-editor/" >}}) when they [build dashboards]({{< relref "../../dashboards/build-dashboards/" >}}), use [Explore]({{< relref "../../explore/" >}}), and [annotate visualizations]({{< relref "./query-editor/#apply-annotations" >}}).
## Configure the data source
To configure basic settings for the data source, complete the following steps:
1. Click **Connections** in the left-side menu.
1. Under Your connections, click **Data sources**.
1. Enter `Loki` in the search bar.
1. Select **Loki**.
The **Settings** tab of the data source is displayed.
1. Set the data source's basic configuration options:
| Name | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Name** | Sets the name you use to refer to the data source in panels and queries. |
| **Default** | Sets the data source that's pre-selected for new panels. |
| **URL** | Sets the HTTP protocol, IP, and port of your Loki instance, such as `http://localhost:3100`. |
| **Allowed cookies** | Defines which cookies are forwarded to the data source. Grafana Proxy deletes all other cookies. |
| **Maximum lines** | Sets the upper limit for the number of log lines returned by Loki. Defaults to 1,000. Lower this limit if your browser is sluggish when displaying logs in Explore. |
{{% admonition type="note" %}}
To troubleshoot configuration and other issues, check the log file located at `/var/log/grafana/grafana.log` on Unix systems, or in `<grafana_install_dir>/data/log` on other platforms and manual installations.
{{% /admonition %}}
### Configure derived fields
The **Derived Fields** configuration helps you:
- Add fields parsed from the log message
- Add a link that uses the value of the field
For example, you can link to your tracing backend directly from your logs, or link to a user profile page if the log line contains a corresponding userId.
These links appear in the [log details]({{< relref "../../explore/logs-integration/#labels-and-detected-fields" >}}).
{{% admonition type="note" %}}
If you use Grafana Cloud, you can request modifications to this feature by [opening a support ticket in the Cloud Portal](/profile/org#support).
{{% /admonition %}}
Each derived field consists of:
| Field name | Description |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Name** | Sets the field name. Displayed as a label in the log details. |
| **Regex** | Defines a regular expression to evaluate on the log message and capture part of it as the value of the new field. Can contain only one capture group. |
| **URL/query** | Sets the full link URL if the link is external, or a query for the target data source if the link is internal. You can interpolate the value from the field with the `${__value.raw}` macro. |
| **URL Label** | _(Optional)_ Sets a custom display label for the link. This setting overrides the link label, which defaults to the full external URL or name of the linked internal data source. |
| **Internal link** | Defines whether the link is internal or external. For internal links, you can select the target data source from a selector. This supports only tracing data sources. |
#### Troubleshoot interpolation
You can use a debug section to see what your fields extract and how the URL is interpolated.
Select **Show example log message** to display a text area where you can enter a log message.
{{< figure src="/static/img/docs/v75/loki_derived_fields_settings.png" class="docs-image--no-shadow" max-width="800px" caption="Screenshot of the derived fields debugging" >}}
The new field with the link shown in log details:
{{< figure src="/static/img/docs/explore/data-link-9-4.png" max-width="800px" caption="Data link in Explore" >}}
### Provision the data source
## Provision the Loki data source
You can define and configure the data source in YAML files as part of Grafana's provisioning system.
For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana]({{< relref "../../administration/provisioning/#data-sources" >}}).
#### Provisioning examples
### Provisioning examples
```yaml
apiVersion: 1

130
docs/sources/datasources/loki/configure-loki-data-source.md Normal file
View File

@ -0,0 +1,130 @@
---
aliases:
- ../data-sources/loki/
- ../features/datasources/loki/
description: Configure the Loki data source
keywords:
- grafana
- loki
- logging
- guide
- data source
menuTitle: Configure Loki
title: Configure the Loki data source
weight: 200
---
# Loki data source
Grafana ships with built-in support for [Loki](/docs/loki/latest/), an open-source log aggregation system by Grafana Labs. If you are new to Loki the following documentation will help you get started:
- [Getting started](/docs/loki/latest/getting-started/)
- [Best practices](/docs/loki/latest/best-practices/#best-practices)
## Configure the Loki data source
To add the Loki data source, complete the following steps:
1. Click **Connections** in the left-side menu.
1. Under **Connections**, click **Add new connection**.
1. Enter `Loki` in the search bar.
1. Select **Loki data source**.
1. Click **Create a Loki data source** in the upper right.
You will be taken to the **Settings** tab where you will set up your Loki configuration.
## Configuration options
The following is a list of configuration options for Loki.
The first option to configure is the name of your connection:
- **Name** - The data source name. This is how you refer to the data source in panels and queries. Examples: loki-1, loki_logs.
- **Default** - Toggle to select as the default name in dashboard panels. When you go to a dashboard panel this will be the default selected data source.
### HTTP section
- **URL** - The URL of your Loki server. Loki uses port 3100. If your Loki server is local, use `http://localhost:3100`. If it is on a server within a network, this is the URL with port where you are running Loki. Example: `http://loki.example.orgname:3100`.
- **Allowed cookies** - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default.
- **Timeout** - The HTTP request timeout. This must be in seconds. There is no default, so this setting is up to you.
### Auth section
There are several authentication methods you can choose in the Authentication section.
{{% admonition type="note" %}}
Use TLS (Transport Layer Security) for an additional layer of security when working with Loki. For information on setting up TLS encryption with Loki see [Grafana Loki configuration parameters](/docs/loki/latest/configuration/).
{{% /admonition %}}
- **Basic authentication** - The most common authentication method. Use your `data source` user name and `data source` password to connect.
- **With credentials** - Toggle on to enable credentials such as cookies or auth headers to be sent with cross-site requests.
- **TLS client authentication** - Toggle on to use client authentication. When enabled, add the `Server name`, `Client cert` and `Client key`. The client provides a certificate that is validated by the server to establish the client's trusted identity. The client key encrypts the data between client and server.
- **With CA cert** - Authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file.
- **Skip TLS verify** - Toggle on to bypass TLS certificate validation.
- **Forward OAuth identity** - Forward the OAuth access token (and also the OIDC ID token if available) of the user querying the data source.
### Custom HTTP headers
- **Header** - Add a custom header. This allows custom headers to be passed based on the needs of your Loki instance.
- **Value** - The value of the header.
### Alerting
- **Manage alert rules in Alerting UI** - Toggle on to manage alert rules for the Loki data source. To manage other alerting resources add an `Alertmanager` data source.
### Queries
- **Maximum lines** - Sets the maximum number of log lines returned by Loki. Increase the limit to have a bigger results set for ad-hoc analysis. Decrease the limit if your browser is sluggish when displaying log results. The default is `1000`.
<!-- {{% admonition type="note" %}}
To troubleshoot configuration and other issues, check the log file located at `/var/log/grafana/grafana.log` on Unix systems, or in `<grafana_install_dir>/data/log` on other platforms and manual installations.
{{% /admonition %}} -->
### Derived fields
Derived Fields are used to extract new fields from a log message and create a link from the value of the field.
For example, you can link to your tracing backend directly from your logs, or link to a user profile page if the log line contains a corresponding userId.
These links appear in the [log details]({{< relref "../../explore/logs-integration/#labels-and-detected-fields" >}}).
You can add multiple derived fields.
{{% admonition type="note" %}}
If you use Grafana Cloud, you can request modifications to this feature by clicking **Open a Support Ticket** from the Grafana Cloud Portal.
{{% /admonition %}}
Each derived field consists of the following:
- **Name** - Sets the field name. Displayed as a label in the log details.
- **Regex** - Defines a regular expression to parse a part of the log message and capture it as the value of the new field. Can contain only one capture group.
- **URL/query** Sets the full link URL if the link is external, or a query for the target data source if the link is internal. You can interpolate the value from the field with the `${__value.raw}` macro.
- **URL Label** - Sets a custom display label for the link. This setting overrides the link label, which defaults to the full external URL or name of the linked internal data source.
- **Internal link** - Toggle on to define an internal link. For internal links, you can select the target data source from a selector. This supports only tracing data sources.
- **Show example log message** - Click to paste an example log line to test the regular expression of your derived fields.
Click **Save & test** to test your connection.
#### Troubleshoot interpolation
You can use a debug section to see what your fields extract and how the URL is interpolated.
Select **Show example log message** to display a text area where you can enter a log message.
{{< figure src="/static/img/docs/v75/loki_derived_fields_settings.png" class="docs-image--no-shadow" max-width="800px" caption="Screenshot of the derived fields debugging" >}}
The new field with the link shown in log details:
{{< figure src="/static/img/docs/explore/data-link-9-4.png" max-width="800px" caption="Data link in Explore" >}}

146
docs/sources/datasources/loki/query-editor/index.md
View File

@ -21,15 +21,14 @@ weight: 300
The Loki data source's query editor helps you create [log]({{< relref "#create-a-log-query" >}}) and [metric]({{< relref "#create-a-metric-query" >}}) queries that use Loki's query language, [LogQL](/docs/loki/latest/logql/).
This topic explains querying specific to the Loki data source.
For general documentation on querying data sources in Grafana, see [Query and transform data]({{< relref "../../../panels-visualizations/query-transform-data" >}}).
## Choose a query editing mode
You can switch the Loki query editor between two modes:
The Loki query editor has two modes:
- [Code mode]({{< relref "#code-mode" >}}), which provides a feature-rich editor for writing queries
- [Builder mode]({{< relref "#builder-mode" >}}), which provides a visual query designer
- [Builder mode]({{< relref "#builder-mode" >}}), which provides a visual query designer.
- [Code mode]({{< relref "#code-mode" >}}), which provides a feature-rich editor for writing queries.
To switch between the editor modes, select the corresponding **Builder** and **Code** tabs.
@ -39,38 +38,63 @@ To run a query, select **Run queries** located at the top of the editor.
To run Loki queries in [Explore]({{< relref "../../../explore/" >}}), select **Run query**.
{{% /admonition %}}
Each mode is synchronized with the other modes, so you can switch between them without losing your work, although there are some limitations.
Builder mode doesn't yet support some complex queries.
When you switch from Code mode to Builder mode with such a query, the editor displays a popup that explains how you might lose parts of the query if you continue.
Each mode is synchronized, so you can switch between them without losing your work, although there are some limitations. Builder mode doesn't support some complex queries.
When you switch from Code mode to Builder mode with such a query, the editor displays a warning message that explains how you might lose parts of the query if you continue.
You can then decide whether you still want to switch to Builder mode.
You can also augment queries by using [template variables]({{< relref "./template-variables/" >}}).
## Code mode
## Toolbar elements
In **Code mode**, you can write complex queries using a text editor with autocompletion features and syntax highlighting.
It also contains a [label browser]({{< relref "#label-browser" >}}) to further help you write queries.
The query editor toolbar contains the following elements:
For more information about Loki's query language, refer to the [Loki documentation](/docs/loki/latest/logql/).
- **Kick start your query** - Click to see a list of queries that help you quickly get started creating LogQL queries. You can then continue to complete your query.
### Use autocompletion
These include:
Code mode's autocompletion feature works automatically while typing.
- Log query starters
- Metric query starters
The query editor can autocomplete static functions, aggregations, and keywords, and also dynamic items like labels.
The autocompletion dropdown includes documentation for the suggested items where available.
Click the arrow next to each to see available query options.
- **Label browser** - Use the Loki label browser to navigate through your labels and values, and build queries.
To navigate Loki and build a query:
1. Choose labels to locate.
1. Search for the values of your selected labels.
The search field supports fuzzy search, and the label browser also supports faceting to list only possible label combinations.
1. Select the **Show logs** button to display log lines based on the selected labels, or select the **Show logs rate** button to show the rate based on metrics such as requests per second. Additionally, you can validate the selector by clicking the **Validate selector** button. Click **Clear** to start from the beginning.
{{< figure src="/static/img/docs/explore/Loki_label_browser.png" class="docs-image--no-shadow" max-width="800px" caption="The Loki label browser" >}}
- **Explain query** - Toggle to display a step-by-step explanation of all query components and operations.
{{< figure src="/static/img/docs/prometheus/explain-results.png" max-width="500px" class="docs-image--no-shadow" caption="Explain results" >}}
- **Builder/Code** - Click the corresponding **Builder** or **Code** tab on the toolbar to select an editor mode.
## Builder mode
Use Builder mode to visually construct queries, without needing to manually enter LogQL.
Builder mode helps you build queries using a visual interface without needing to manually enter LogQL. This option is best for users who have limited or no previous experience working with Loki and LogQL.
### Use the Labels selector
### Label filters
Select labels and their values from the dropdown list.
When you select a label, Grafana retrieves available values from the server.
Use the `+` button to add a label and the `x` button to remove a label.
Use the `+` button to add a label and the `x` button to remove a label. You can add multiple labels.
Select comparison operators from the following options:
- `=` - equal to
- `!=` - is not equal
- `=~` - matches regex
- `!~` - does not match regex
Select values by using the dropdown, which displays all possible values based on the label selected.
### Operations
@ -86,17 +110,50 @@ Each operation's header displays its name, and additional action buttons appear
| {{< figure src="/static/img/docs/v95/loki_operation_description.png" class="docs-image--no-shadow" max-width="30px" >}} | Opens the operation's description tooltip. |
| {{< figure src="/static/img/docs/v95/loki_operation_remove.png" class="docs-image--no-shadow" max-width="30px" >}} | Removes the operation. |
Some operations have additional parameters under the operation header.
For details about each operation, use the `info` button to view the operation's description, or refer to the [Loki documentation](/docs/loki/latest/operations/).
The query editor groups operations into the following sections:
Some operations make sense only when used in a specific order.
If adding an operation would result in nonsensical query, the query editor adds the operation to the correct place.
To re-order operations manually, drag the operation box by its name and drop it into the desired place.
- Aggregations - see [Built-in aggregation operators](/docs/loki/latest/logql/metric_queries/#built-in-aggregation-operators)
- Range functions - see [Range Vector aggregation](/docs/loki/latest/logql/metric_queries/#range-vector-aggregation)
- Formats - see [Log queries](/docs/loki/latest/logql/log_queries/#log-queries)
- Binary operations - see [Binary operators](/docs/loki/latest/logql/#binary-operators)
- Label filters - see [Label filter expression](/docs/loki/latest/logql/log_queries/#label-filter-expression)
- Line filters - see [Line filter expression](/docs/loki/latest/logql/log_queries/#label-filter-expression)
#### Hints
Some operations make sense only when used in a specific order. If adding an operation would result in nonsensical query, the query editor adds the operation to the correct place.
To re-order operations manually, drag the operation box by its name and drop it into the desired place. For additional information see [Order of operations](/docs/loki/latest/logql/#order-of-operations).
### Hints
In same cases the query editor can detect which operations would be most appropriate for a selected log stream. In such cases it will show a hint next to the `+ Operations` button. Click on the hint to add the operations to your query.
## Code mode
In **Code mode**, you can write complex queries using a text editor with autocompletion feature, syntax highlighting, and query validation.
It also contains a [label browser]({{< relref "#label-browser" >}}) to further help you write queries.
For more information about Loki's query language, refer to the [Loki documentation](/docs/loki/latest/logql/).
### Use autocompletion
Code mode's autocompletion feature works automatically while typing.
The query editor can autocomplete static functions, aggregations, and keywords, and also dynamic items like labels.
The autocompletion dropdown includes documentation for the suggested items where available.
## Options
The following options are the same for both **Builder** and **Code** mode:
- **Legend** - Controls the time series name, using a name or pattern. For example, `{{hostname}}` is replaced with the label value for the label `hostname`.
- **Type** - Selects the query type to run. The `instant` type queries against a single point in time. We use the "To" time from the time range. The `range` type queries over the selected range of time.
- **Line limit** -Defines the upper limit for the number of log lines returned by a query. The default is `1000`
- **Step** Sets the step parameter of Loki metrics queries. The default value equals to the value of `$__interval` variable, which is calculated using the time range and the width of the graph (the number of pixels).
- **Resolution** Deprecated. Sets the step parameter of Loki metrics range queries. With a resolution of `1/1`, each pixel corresponds to one data point. `1/2` retrieves one data point for every other pixel, `1/10` retrieves one data point per 10 pixels, and so on. Lower resolutions perform better.
## Create a log query
Loki log queries return the contents of the log lines.
@ -182,45 +239,6 @@ You can use LogQL to wrap a log query with functions that create metrics from yo
For more information about metric queries, refer to the [Loki metric queries documentation](/docs/loki/latest/logql/metric_queries/).
## Review toolbar features
In addition to the **Run query** button and mode switcher, Builder mode provides additional elements:
| Name | Description |
| ------------------------- | ----------------------------------------------------------------------------------------- |
| **Kick start your query** | A list of useful operation patterns you can use to add multiple operations to your query. |
| **Label browser** | Used to navigate through your labels and values, and also build queries. |
| **Explain query** | Displays a step-by-step explanation of all query components and operations. |
### Label browser
You can use the Loki label browser to navigate through your labels and values, and build queries.
**To navigate Loki and build a query:**
1. Choose labels to locate.
1. Search for the values of your selected labels.
The search field supports fuzzy search, and the label browser also supports faceting to list only possible label combinations.
1. Select the **Show logs** button to display log lines based on the selected labels, or select the **Show logs rate** button to show the rate based on metrics such as requests per second. Additionally, you can validate the selector by clicking the **Validate selector** button.
{{< figure src="/static/img/docs/v75/loki_label_browser.png" class="docs-image--no-shadow" max-width="800px" caption="The Loki label browser" >}}
### Explain query
This section is only shown if the `Explain query` switch from the query editor top toolbar is set to `on`. It shows a step by step explanation of all query parts and the operations.
## Configure query settings
| Name | Description |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Type** | Selects the query type to run. The `instant` type queries against a single point in time. We use the "To" time from the time range. The `range` type queries over the selected range of time. |
| **Line limit** | Defines the upper limit for the number of log lines returned by a query. The default is Loki's configured maximum lines limit. |
| **Legend** | _(Available only in a dashboard)_ Controls the time series name, using a name or pattern. For example, `{{hostname}}` is replaced with the label value for the label `hostname`. |
| **Step** | Sets the step parameter of Loki metrics queries. The default value equals to the value of `$__interval` variable, which is calculated using the time range and the width of the graph (the number of pixels). |
| **Resolution** | Deprecated. Sets the step parameter of Loki metrics range queries. With a resolution of `1/1`, each pixel corresponds to one data point. `1/2` retrieves one data point for every other pixel, `1/10` retrieves one data point per 10 pixels, and so on. Lower resolutions perform better. |
## Apply annotations
[Annotations]({{< relref "../../../dashboards/build-dashboards/annotate-visualizations" >}}) overlay rich event information on top of graphs.