grafana/pkg
Ryan McKinley 2aafa39879
Live: support real time measurements (alpha) (#28022)
* improve reduce transformer

* add measurment classes

* sync with new grafana measure format

* use address for live

* use plural in URL

* set the field name

* fix build

* find changes

* POST http to channel

* Yarn: Update lock file (#28014)

* Loki: Run instant query only in Explore (#27974)

* Run instant query only in Explore

* Replace forEach with for loop

* don't cast

* Docs: Fixed row display in table (#28031)

* Plugins: Let descendant plugins inherit their root's signature (#27970)

* plugins: Let descendant plugins inherit their root's signature

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Registry: Fix service shutdown mode trigger location (#28025)

* Add Alex Khomenko as member (#28032)

* show history

* fix confirm

* fix confirm

* add tests

* fix lint

* add more errors

* set values

* remove unrelated changes

* unrelated changes

* Update pkg/models/live.go

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Update pkg/models/live.go

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Update pkg/services/live/live.go

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Update pkg/services/live/pluginHandler.go

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Update pkg/services/live/pluginHandler.go

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Update pkg/services/live/pluginHandler.go

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* use measurments for testdata endpoints

* add live to testdata

* add live to testdata

* Update pkg/services/live/channel.go

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Apply suggestions from code review

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* update comment formats

* uprevert testdata

* Apply suggestions from code review

Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Hugo Häggmark <hugo.haggmark@grafana.com>

* Apply suggestions from code review

* CloudWatch: Add EC2CapacityReservations Namespace (#28309)

* API: Fix short URLs (#28300)

* API: Fix short URLs

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Chore: Add cloud-middleware as code owners (#28310)

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* SQLStore: Run tests as integration tests (#28265)

* sqlstore: Run tests as integration tests

* Truncate database instead of re-creating it on each test

* Fix test description

See https://github.com/grafana/grafana/pull/12129

* Fix lint issues

* Fix postgres dialect after review suggestion

* Rename and document functions after review suggestion

* Add periods

* Fix auto-increment value for mysql dialect

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

* Drone: Fix grafana-mixin linting (#28308)

* Drone: Fix Starlark script

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* grafana-mixin: Move build logic to scripts

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Drone: Use mixin scripts

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* CI build image: Install jsonnetfmt and mixtool

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Makefile: Print commands

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* should only ignore the file in the grafana mixin root folder (#28306)

Signed-off-by: bergquist <carl.bergquist@gmail.com>

* fix: for graph size not taking up full height or width

* Graph NG: fix toggling queries and extract Graph component from graph3 panel (#28290)

* Fix issue when data and config is not in sync

* Extract GraphNG component from graph panel and add some tests coverage

* Update packages/grafana-ui/src/components/uPlot/hooks.test.ts

* Update packages/grafana-ui/src/components/uPlot/hooks.test.ts

* Update packages/grafana-ui/src/components/uPlot/hooks.test.ts

* Fix grid color and annotations refresh

* Drone: Use ${DRONE_TAG} in release pipelines, since it should work (#28299)

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Explore: respect min_refresh_interval (#27988)

* Explore: respect min_refresh_interval

Fixes #27494

* fixup! Explore: respect min_refresh_interval

* fixup! Explore: respect min_refresh_interval

* UI: export defaultIntervals from refresh picker

* fixup! Explore: respect min_refresh_interval

Co-authored-by: Zoltán Bedi <zoltan.bedi@gmail.com>

* Loki: Base maxDataPoints limits on query type (#28298)

* Base maxLines and maxDataPoints based on query type

* Allow overriding the limit to higher value

* Bump tree-kill from 1.2.1 to 1.2.2 (#27405)

Bumps [tree-kill](https://github.com/pkrumins/node-tree-kill) from 1.2.1 to 1.2.2.
- [Release notes](https://github.com/pkrumins/node-tree-kill/releases)
- [Commits](https://github.com/pkrumins/node-tree-kill/compare/v1.2.1...v1.2.2)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump handlebars from 4.4.3 to 4.7.6 (#27416)

Bumps [handlebars](https://github.com/wycats/handlebars.js) from 4.4.3 to 4.7.6.
- [Release notes](https://github.com/wycats/handlebars.js/releases)
- [Changelog](https://github.com/handlebars-lang/handlebars.js/blob/master/release-notes.md)
- [Commits](https://github.com/wycats/handlebars.js/compare/v4.4.3...v4.7.6)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Build(deps): Bump http-proxy from 1.18.0 to 1.18.1 (#27507)

Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases)
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/http-party/node-http-proxy/compare/1.18.0...1.18.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Automation: Add backport github action (#28318)

* BackendSrv: Fixes queue countdown when unsubscribe is before response (#28323)

* GraphNG: Use AxisSide enum (#28320)

* IssueTriage: Needs more info automation and messages (#28137)

* IssueTriage: Needs more info automation and messages

* Updated

* Updated

* Updated wording

* SAML: IdP-initiated SSO docs (#28280)

* SAML: IdP-initiated SSO docs

* Update docs/sources/enterprise/saml.md

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

* Apply suggestions from code review

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

* Loki: Run instant query only when doing metric query (#28325)

* Run instant query only when doing metric query

* Update public/app/plugins/datasource/loki/datasource.ts

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

* Automation: Tweaks to more info message (#28332)

* AlertingNG: remove warn/crit from eval prototype (#28334)

and misc cleanup

* area/grafana/toolkit: update e2e docker image (#28335)

* add xvfb to image

* comment out toolkit inclusion

* add latest tag

* update packages for cypress

* cleanup script

* Update auth-proxy.md (#28339)

Fix a minor grammar mistake: 'handling' to 'handle'.

* Git: Create .gitattributes for windows line endings (#28340)

With this set, Windows users will have text files converted from Windows style line endings (\r\n) to Unix style line endings (\n) when they’re added to the repository.
https://www.edwardthomson.com/blog/git_for_windows_line_endings.html

* Docs: Add docs for valuepicker (#28327)

* Templating: Replace all '$tag' in tag values query (#28343)

* Docs: Add missing records from grafana-ui 7.2.1 CHANGELOG (#28302)

* Dashboard links: Places drop down list so it's always visible (#28330)

* calculating whether to place the list on the right or left edge of the parent

* change naming and add import of createRef

* Automation: Update backport github action trigger (#28352)

It seems like GitHub has solved the problem of running github actions on PRs from forks with access to secrets. 

https://github.blog/2020-08-03-github-actions-improvements-for-fork-and-pull-request-workflows/#improvements-for-public-repository-forks

If I change the event that triggers it to pull_request_target the action is run in the context of the base instead of the merged PR branch

* ColorSchemes: Adds more color schemes and text colors that depend on the background (#28305)

* Adding more color modes and text colors that depend on the background color

* Updates

* Updated

* Another big value fix

* Fixing unit tests

* Updated

* Updated test

* Update

* Updated

* Updated

* Updated

* Updated

* Added new demo dashboard

* Updated

* updated

* Updated

* Updateed

* added beta notice

* Fixed e2e test

* Fix typos

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* revert pseduo code

* apply feedback

* remove HTTP for now

* fix backend test

* change to datasource

* clear input for streams

* fix docs?

* consistent measure vs measurements

* better jsdocs

* fix a few jsdoc errors

* fix comment style

* Remove commented out code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clean up code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clean up code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clean up code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clean up code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clean up code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clean up code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clean up code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Update pkg/models/live.go

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix build

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* set the stringField

Co-authored-by: Torkel Ödegaard <torkel@grafana.org>
Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>
Co-authored-by: ozhuang <ozhuang.95@gmail.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Amos Law <ahlaw.dev@gmail.com>
Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Hugo Häggmark <hugo.haggmark@grafana.com>
Co-authored-by: The Rock Guy <fabian.bracco@gvcgroup.com.au>
Co-authored-by: Sofia Papagiannaki <papagian@users.noreply.github.com>
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
Co-authored-by: Carl Bergquist <carl@grafana.com>
Co-authored-by: Jack Westbrook <jack.westbrook@gmail.com>
Co-authored-by: Dominik Prokop <dominik.prokop@grafana.com>
Co-authored-by: Elliot Pryde <elliot.pryde@elliotpryde.com>
Co-authored-by: Zoltán Bedi <zoltan.bedi@gmail.com>
Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alexander Zobnin <alexanderzobnin@gmail.com>
Co-authored-by: Kyle Brandt <kyle@grafana.com>
Co-authored-by: Brian Gann <briangann@users.noreply.github.com>
Co-authored-by: J-F-Far <joel.f.farthing@gmail.com>
Co-authored-by: acoder77 <73009264+acoder77@users.noreply.github.com>
Co-authored-by: Peter Holmberg <peterholmberg@users.noreply.github.com>
Co-authored-by: Krzysztof Dąbrowski <krzysdabro@live.com>
Co-authored-by: maknik <mooniczkam@gmail.com>
2020-10-22 09:10:26 +02:00
..
api Chore: Use net.JoinHostPort (#28421) 2020-10-22 07:34:26 +02:00
bus Auth proxy: Ignore stale cache entries (#23979) 2020-06-17 18:43:16 +02:00
cmd Chore: Fix conversion of a 64-bit integer to a lower bit size type uint (#28425) 2020-10-21 11:25:56 +03:00
components Snapshots: Store dashboard data encrypted in the database (#28129) 2020-10-13 10:19:42 +02:00
events Tests: Change util GoConvey tests to use testify (#26724) 2020-08-13 11:10:48 +02:00
extensions SAML Single Logout (#27995) 2020-10-08 17:42:55 +03:00
infra Chore: Rewrite some tests to use testify (#28420) 2020-10-21 10:09:42 +02:00
login OAuth: configurable user name attribute (#28286) 2020-10-20 09:56:48 +03:00
middleware Instrumentation: Add histogram for request duration (#28364) 2020-10-20 09:44:38 +02:00
models Live: support real time measurements (alpha) (#28022) 2020-10-22 09:10:26 +02:00
plugins plugins: Don't exit on duplicate plugin (#28390) 2020-10-21 12:39:41 +02:00
registry Provisioning: Fix bug when provision app plugins using Enterprise edition (#26340) 2020-07-16 10:36:08 +02:00
server Instrumentation: Add histogram for request duration (#28364) 2020-10-20 09:44:38 +02:00
services Live: support real time measurements (alpha) (#28022) 2020-10-22 09:10:26 +02:00
setting Instrumentation: Add histogram for request duration (#28364) 2020-10-20 09:44:38 +02:00
tsdb Live: support real time measurements (alpha) (#28022) 2020-10-22 09:10:26 +02:00
util Test: Create unit test to SplitEmails (#27985) 2020-10-02 08:42:10 +02:00
README.md Chore: Fix various spelling errors in back-end code (#25241) 2020-06-01 17:11:25 +02:00

Backend

This directory contains the code for the Grafana backend. This document gives an overview of the directory structure, and ongoing refactorings.

For more information on developing for the backend:

Central folders of Grafana's backend

folder description
/pkg/api HTTP handlers and routing. Almost all handler funcs are global which is something we would like to improve in the future. Handlers should be associated with a struct that refers to all dependencies.
/pkg/cmd The binaries that we build: grafana-server and grafana-cli.
/pkg/components A mix of third-party packages and packages we have implemented ourselves. Includes our packages that have out-grown the util package and don't naturally belong somewhere else.
/pkg/infra Packages in infra should be packages that are used in multiple places in Grafana without knowing anything about the Grafana domain.
/pkg/services Packages in services are responsible for persisting domain objects and manage the relationship between domain objects. Services should communicate with each other using DI when possible. Most of Grafana's codebase still relies on global state for this. Any new features going forward should use DI.
/pkg/tsdb All backend implementations of the data sources in Grafana. Used by both Grafana's frontend and alerting.
/pkg/util Small helper functions that are used in multiple parts of the codebase. Many functions are placed directly in the util folders which is something we want to avoid. Its better to give the util function a more descriptive package name. Ex errutil.

Central components of Grafana's backend

package description
/pkg/bus The bus is described in more details under Communication
/pkg/models This is where we keep our domain model. This package should not depend on any package outside standard library. It does contain some references within Grafana but that is something we should avoid going forward.
/pkg/registry Package for managing services.
/pkg/services/alerting Grafana's alerting services. The alerting engine runs in a separate goroutine and shouldn't depend on anything else within Grafana.
/pkg/services/sqlstore Currently where the database logic resides.
/pkg/setting Anything related to Grafana global configuration should be dealt with in this package.

Dependency management

Refer to UPGRADING_DEPENDENCIES.md.

Ongoing refactoring

These issues are not something we want to address all at once but something we will improve incrementally. Since Grafana is released at a regular schedule the preferred approach is to do this in batches. Not only is it easier to review, but it also reduces the risk of conflicts when cherry-picking fixes from master to release branches. Please try to submit changes that span multiple locations at the end of the release cycle. We prefer to wait until the end because we make fewer patch releases at the end of the release cycle, so there are fewer opportunities for complications.

Global state

Global state makes testing and debugging software harder and it's something we want to avoid when possible. Unfortunately, there is quite a lot of global state in Grafana.

We want to migrate away from this by using the inject package to wire up all dependencies either in pkg/cmd/grafana-server/main.go or self-registering using registry.RegisterService ex https://github.com/grafana/grafana/blob/master/pkg/services/cleanup/cleanup.go#L25.

Limit the use of the init() function

Only use the init() function to register services/implementations.

Settings refactoring

The plan is to move all settings to from package level vars in settings package to the setting.Cfg struct. To access the settings, services and components can inject this setting.Cfg struct:

Cfg struct Injection example

Reduce the use of GoConvey

We want to migrate away from using GoConvey. Instead, we want to use stdlib testing, because it's the most common approach in the Go community and we think it will be easier for new contributors. Read more about how we want to write tests in the style guide.

Refactor SqlStore

The sqlstore handlers all use a global xorm engine variable. Refactor them to use the SqlStore instance.

Avoid global HTTP handler functions

Refactor HTTP handlers so that the handler methods are on the HttpServer instance or a more detailed handler struct. E.g (AuthHandler). This ensures they get access to HttpServer service dependencies (and Cfg object) and can avoid global state.

Date comparison

Store newly introduced date columns in the database as epochs if they require date comparison. This permits a unified approach for comparing dates against all the supported databases instead of handling dates differently for each database. Also, by comparing epochs, we no longer need error pruning transformations to and from other time zones.

Avoid use of the simplejson package

Use of the simplejson package (pkg/components/simplejson) in place of types (Go structs) results in code that is difficult to maintain. Instead, create types for objects and use the Go standard library's encoding/json package.

Provisionable*

All new features that require state should be possible to configure using config files. For example:

Today its only possible to provision data sources and dashboards but this is something we want to support all over Grafana.