nosqlbench/devdocs/docstructure/bundled_docs.md

NBDocs - NoSQLBench Docs

THIS IS A WORK IN PROGRESS

In order to keep the structure of NoSQLBench modular enough to allow for easy extension by contributors, yet cohesive in how it presents documentation and features to users, it is necessary to provide internal services which aggregate content by subject matter into a consumable whole that can be used by the documentation system.

MarkdownDocs Service

The primary markdown service that is meant to be consumed by the documentation system is known simply as

MarkdownDocs

Static methods on this class will provide all of the markdown content in pre-baked and organized form. The markdown service is responsible for reading all the raw markdown sources and organizing their content into a single cohesive structure. MardownDocs finds all content that is provided by individual MarkdownProvider services, as described below.

All of the rules for how raw markdown content is to be combined are owned by the MarkdownDocs service.

The MarkdownDocs service relies on SPI published services which provide raw markdown sources as described below.

RawMarkdownSource Services

The RawMarkdownSource service is responsible for bundling the raw markdown for a path within a NoSQLBench module. Each module that wishes to publish markdown docs to users must provide one or more RawMarkdownSource services via SPI. This is most easily done with a @Service(RawMarkdownSource.class) annotation.

RawMarkdownSource endpoints provide Content

Each instance of a RawMarkdownSource service provides all of the individual markdown files it finds indirectly as io.nosqlbench.nb.api.content.Content, which allows the internal file content to be read appropriately regardless of whether it comes from a classpath resource stream, a file on disk, or even a dynamic source like function metadata.

RawMarkdownSources Aggregator

A service aggregator called RawMarkdownSources provides easy access to all raw markdown sources provided by all published instances of the service.

Front Matter Interpretation

There is a set of rules observed by MarkdownDocs for repacking markdown for structured display. These rules are largely driven by front matter.

Doc Scope

The scope front matter property determines where content should be visible. This is useful for providing documentation about a topic or concept that can be pulled into multiple places. There are three doc scopes that can be added to source markdown via the scopes front matter. This is a multi-valued property

• cli - The source content should be included for command-line searching and viewing.
• web - The source content should be included for static web documentation.
• app - The source content should be included in accompanying documentation for web applications.
• all - This is the default scope which includes all of the above.

If no scopes are provided, then a special scope any is assigned to source content.

Examples

---
scopes: all
---

---
scopes: cli,web
---

Topic Names

The topic front-matter property determines the assocation between a documentation fragment and the ways that a user might name or search for it. All content within NBDocs falls within one or more nested topics. That is, raw content could be homed under a simple topic name, or it could be homed under a topic which has a another topic above.

1. All front matter that is considered is required to have at least one topic value.
2. Topic values which contain , or ; patterns are auto-split into multiple topics.
3. Topics can be hierarchical. Topics in the form of cat1/cat2/topicfoo are considered nested topics. Topics which contain other topics are called topic categories, but they are also topics.
4. Topics can be literal values or they can be patterns which match other topics.

examples

---
topics: cli, parameters
---

Topic Aggregation

Topics can be placeholders for matching other topics. When a topic name starts with a caret ^, contains either of .*, or .+, or ends with a $, it is called a topic pattern.

The patterns are regular expressions, even though the patterns above are used explicitly to enable pattern detection.

This allows for content to be included in other places. When source content is marked with a topic pattern, any other source content that is matched by that pattern is included in views of its raw content. Further, the topics which are matched are added to an included property in the matching content's front matter. The matched content is not affected in any other way and is still visible under the matched topic names.

It is considered an error to create a circular reference between different topics. This condition is checked explicitly by the topic mapping logic.

examples

---
title: owning topic
topics: ^cli$, foobarbaz
---

# Main Content

main content

---
title: included topic
topics: cli, parameters
---

## Included content

included content

Taking the two source documents above, the markdown loading system would present the following document which globs the second one onto the first:

---
title: owning topic
topics: cli, foobarbaz, parameters
---

# Main Content

main content

## Included Content

included content

Within an aggregate source like the above, all included sections will be ordered according to the weight front-matter property first, and then by the title, and then by the name of the interior top-level heading of source item.

Logical Structure

According to the rules and mechanisms above, it is possible to organize all the provided content into a clean and consistent strucure for search and presentation.

The example below represents a schematic of what content sources will be provided in every document after loading and processing:

---
title: <String>
scope: <Set<String>>
topics: <Set<String>>
included: <Set<String>>
weight: <Number>
---
<optional content>

In specific:

title will be provided as a string, even if it is empty. scope will be provided as a set of strings, possibly an empty set. topics will be provided as a set of strings, possibly an empty set. included will be provided as a set of strings, possibly an empty set. weight will be provided as a number, possibly 0.

Headings and content may or may not be provided, depending on how the content is aggregated and what topics are matched for content aggregation.

Repackaged Artifacts

The documentation aggregator system may be asked to store repackaged forms. These are suggested:

1. A metadata.json which includes the basic metadata of all the documentation entries, including:
   1. module path of logical document
   2. title
   3. topics
   4. included
   5. weight
2. A manifest.json which includes:
   1. module path of logical document
   2. title of document
   3. heading structure of document
3. A topics.json which includes the topic structure of all topics and headings
4. A full-text search index.

With these artifacts provides as services, clients can be efficient in discovering content and making it searchable for users.

Topic Mapping Logic

Topics may be transitive. There is no clear rationale for avoiding this, and at least one good
reason to allow it. The method for validating and resolving topic aggregations, where there may be layers of dependencies, is explained here:

BEING REWRITTEN

1. All content sources are put into a linked list in no particular order.
2. The list is traversed from head to tail repeatedly.
3. When the head of the list is an aggregating source, and all of the matching elements are non-aggregating, then the element converted to a non-aggregating element.
4. If the head of the list is a non-aggregating source, it is moved to the tail.
5. When all of the elements of the list are non-aggregating, the mapping is complete.
6. When the