Plugins: Make backend plugin metrics endpoints available with optional authentication (#46467)

* add new endpoint without auth+config

* add cfg check

* fit lint issue

* Add basic auth support

Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>

* WIP docs

* Update docs/sources/administration/view-server/internal-metrics.md

Co-authored-by: Dave Henderson <dhenderson@gmail.com>

* update instructions

Co-authored-by: Will Browne <will.browne@grafana.com>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Dave Henderson <dhenderson@gmail.com>

docs/sources/administration/view-server/internal-metrics.md

@@ -30,7 +30,7 @@ These instructions assume you have already added Prometheus as a data source in
 1. Enable Prometheus to scrape metrics from Grafana. In your configuration file (`grafana.ini` or `custom.ini` depending on your operating system) remove the semicolon to enable the following configuration options:
 
    ```
-   # Metrics available at HTTP API Url /metrics
+   # Metrics available at HTTP URL /metrics and /metrics/plugins/:pluginId
    [metrics]
    # Disable / Enable internal metrics
    enabled           = true
@@ -39,7 +39,7 @@ These instructions assume you have already added Prometheus as a data source in
    disable_total_stats = false
    ```
 
-1. (optional) If you want to require authorization to view the metrics endpoint, then uncomment and set the following options:
+1. (optional) If you want to require authorization to view the metrics endpoints, then uncomment and set the following options:
 
    ```
    basic_auth_username =
@@ -92,3 +92,48 @@ These instructions assume you have already added Graphite as a data source in Gr
    ```
 
 1. Restart Grafana. Grafana now exposes metrics at http://localhost:3000/metrics and sends them to the Graphite location you specified.
+
+## Pull metrics from Grafana backend plugin into Prometheus
+
+Any installed [backend plugin]({{< relref "../../developers/plugins/backend/_index.md" >}}) exposes a metrics endpoint through Grafana that you can configure Prometheus to scrape.
+
+These instructions assume you have already added Prometheus as a data source in Grafana.
+
+1. Enable Prometheus to scrape backend plugin metrics from Grafana. In your configuration file (`grafana.ini` or `custom.ini` depending on your operating system) remove the semicolon to enable the following configuration options:
+
+   ```
+   # Metrics available at HTTP URL /metrics and /metrics/plugins/:pluginId
+   [metrics]
+   # Disable / Enable internal metrics
+   enabled           = true
+
+   # Disable total stats (stat_totals_*) metrics to be generated
+   disable_total_stats = false
+   ```
+
+1. (optional) If you want to require authorization to view the metrics endpoints, then uncomment and set the following options:
+
+   ```
+   basic_auth_username =
+   basic_auth_password =
+   ```
+
+1. Restart Grafana. Grafana now exposes metrics at `http://localhost:3000/metrics/plugins/<plugin id>`, e.g. http://localhost:3000/metrics/plugins/grafana-github-datasource if you have the [Grafana GitHub datasource](https://grafana.com/grafana/plugins/grafana-github-datasource/) installed.
+1. Add the job to your prometheus.yml file.
+   Example:
+
+   ```
+   - job_name: 'grafana_github_datasource'
+
+      scrape_interval: 15s
+      scrape_timeout: 5s
+      metrics_path: /metrics/plugins/grafana-test-datasource
+
+      static_configs:
+        - targets: ['localhost:3000']
+   ```
+
+1. Restart Prometheus. Your new job should appear on the Targets tab.
+1. In Grafana, hover your mouse over the **Configuration** (gear) icon on the left sidebar and then click **Data Sources**.
+1. Select the **Prometheus** data source.
+1. Import a Golang application metrics dashboard - for example [Go Processes](https://grafana.com/grafana/dashboards/6671).