diff options
author | Joel Hans <joel@netdata.cloud> | 2020-09-29 10:57:52 -0700 |
---|---|---|
committer | GitHub <noreply@github.com> | 2020-09-29 10:57:52 -0700 |
commit | 61d7e23eed0503bf591274df70713970213b5c7f (patch) | |
tree | 125879ff5f780c2937ee206fc510d89e9a91a868 /docs/store | |
parent | e3b04fb39a06991d9a2deed0488044dd7d340e3f (diff) |
Add docsv2 project to master branch (#10000)
* Add overview docs to docsv2 project
* Add quickstart docs to docsv2 project (#9865)
* Init quickstart docs
* Begin work on quickstart guides
* Finish quickstart drafts
* Tweaks to both quickstarts
* Add titles
* Copyedit pass to both docs
* Fixes for Amy and Jen
* Add Get doc to docsv2 project (#9854)
* Init get file
* Add some links
* Change h2 to h1
* Rephrase
* Add configure docs to docsv2 project (#9878)
* Add overview docs to docsv2 project (#9849)
* Init files
* Add drafts of what and why
* Finish initial drafts
* Fix edit URL
* Copyedit pass
* Finish initial drafts of configure docs
* Copyedit all docs
* Fixes for Amy
* Fixes for Jen
* Add collect docs to the docsv2 project (#9932)
* Init files
* Finish first two collect docs
* Finish drafts of collect docs
* Copyedit pass
* Fixes for Amy
* Fix for Jen
* Add visualize docs to the docsv2 project (#9943)
* Add visualize docs
* Copyedits and cleanup
* New images and features
* Copyedit pass and cleanup
* Missing word
* Fixes for Jen
* Add monitor docs to docsv2 project (#9949)
* Finish drafts of monitor docs
* Copyedit pass
* Remove whitespace
* Fixes for Jen
* Add store docs to docsv2 project (#9969)
* Finalize store documents
* Fix import path
* Finishing edit section
* Copyedit pass
* Add export docs to docsv2 project (#9986)
* Add install and claim videos to Get doc
* Finish drafts of exporting docs plus other tweaks
* Init new exporting READMEs
* Copyedit pass and new links
* Fixes for Amy, Vlad, Jen
* Fix links in docsv2 project (#9993)
* Fix links
* Fix a bunch of links ahead of export merge
* Fix additional links
* Fix links, nuke what-is-netdata
* Fixing a few last links
* Improve product images in overview
* Remove extra paren
* Quick tweaks for Jen
* Fixes for Jen
* Access fix
* Remove extra word
Diffstat (limited to 'docs/store')
-rw-r--r-- | docs/store/change-metrics-storage.md | 72 | ||||
-rw-r--r-- | docs/store/distributed-data-architecture.md | 71 |
2 files changed, 143 insertions, 0 deletions
diff --git a/docs/store/change-metrics-storage.md b/docs/store/change-metrics-storage.md new file mode 100644 index 0000000000..cf3b5bb816 --- /dev/null +++ b/docs/store/change-metrics-storage.md @@ -0,0 +1,72 @@ +<!-- +title: "Change how long Netdata stores metrics" +description: "With a single configuration change, the Netdata Agent can store days, weeks, or months of metrics at its famous per-second granularity." +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/store/change-metrics-storage.md +--> + +# Change how long Netdata stores metrics + +import { Calculator } from '../../src/components/agent/dbCalc/' + +The [database engine](/database/engine/README.md) uses RAM to store recent metrics. When metrics reach a certain age, +and based on how much system RAM you allocate toward storing metrics in memory, they are compressed and "spilled" to +disk for long-term storage. + +The default settings retain about two day's worth of metrics on a system collecting 2,000 metrics every second, but the +Netdata Agent is highly configurable if you want your nodes to store days, weeks, or months worth of per-second data. + +The Netdata Agent uses two settings in `netdata.conf` to change the behavior of the database engine: + +```conf +[global] + page cache size = 32 + dbengine multihost disk space = 256 +``` + +`page cache size` sets the maximum amount of RAM (in MiB) the database engine uses to cache and index recent metrics. +`dbengine multihost disk space` sets the maximum disk space (again, in MiB) the database engine uses to store +historical, compressed metrics. When the size of stored metrics exceeds the allocated disk space, the database engine +removes the oldest metrics on a rolling basis. + +## Calculate the system resources (RAM, disk space) needed to store metrics + +You can store more or less metrics using the database engine by changing the allocated disk space. Use the calculator +below to find an appropriate value for `dbengine multihost disk space` based on how many metrics your node(s) collect, +whether you are streaming metrics to a parent node, and more. + +You do not need to edit the `page cache size` setting to store more metrics using the database engine. However, if you +want to store more metrics _specifically in memory_, you can increase the cache size. + +> ⚠️ This calculator provides an estimate of disk and RAM usage for **metrics storage**, along with its best +> recommendation for the `dbengine multihost disk space` setting. Real-life usage may vary based on the accuracy of the +> values you enter below, changes in the compression ratio, and the types of metrics stored. + +<Calculator /> + +## Edit `netdata.conf` with recommended database engine settings + +Now that you have a recommended setting for `dbengine multihost disk space`, open `netdata.conf` with +[`edit-config`](/docs/configure/nodes.md#use-edit-config-to-edit-netdataconf) and look for the `dbengine multihost disk +space` setting. Change it to the value recommended above. For example: + +```conf +[global] + dbengine multihost disk space = 1024 +``` + +Save the file and restart the Agent with `service netdata restart` to change the database engine's size. + +## What's next? + +For more information about the database engine, see our [database reference doc](/database/engine/README.md). + +Storing metrics with the database engine is completely interoperable with [exporting to other time-series +databases](/docs/export/external-databases.md). With exporting, you can use the node's resources to surface metrics +when [viewing dashboards](/docs/visualize/interact-dashboards-charts.md), while also archiving metrics elsewhere for +further analysis, visualization, or correlation with other tools. + +If you don't want to always store metrics on the node that collects them or run ephemeral nodes without dedicated +storage, you can use [streaming](/streaming/README.md). Streaming allows you to centralize your data, run Agents as +headless collectors, replicate data, and more. + +[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdocs%2Fstore%2Fchange-metrics-storage&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>) diff --git a/docs/store/distributed-data-architecture.md b/docs/store/distributed-data-architecture.md new file mode 100644 index 0000000000..354b3ce8e0 --- /dev/null +++ b/docs/store/distributed-data-architecture.md @@ -0,0 +1,71 @@ +<!-- +title: "Distributed data architecture" +description: "Netdata's distributed data architecture stores metrics on individual nodes for high performance and scalability using all your granular metrics." +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/store/distributed-data.md +--> + +# Distributed data architecture + +Netdata uses a distributed data architecture to help you collect and store per-second metrics from any number of nodes. +Every node in your infrastructure, whether it's one or a thousand, stores the metrics it collects. + +Netdata Cloud bridges the gap between many distributed databases by _centralizing the interface you use_ to query and +visualize your nodes' metrics. When you [look at charts in Netdata +Cloud](/docs/visualize/interact-dashboards-charts.md), the metrics values are queried directly from that node's database +and securely streamed to Netdata Cloud, which proxies them to your browser. + +Netdata's distributed data architecture has a number of benefits: + +- **Performance**: Every query to a node's database takes only a few milliseconds to complete for responsiveness when + viewing dashboards or using features like [Metric + Correlations](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations). +- **Scalability**: As your infrastructure scales, install the Netdata Agent on every new node to immediately add it to + your monitoring solution without adding cost or complexity. +- **1-second granularity**: Without an expensive centralized data lake, you can store all of your nodes' per-second + metrics, for any period of time, while keeping costs down. +- **No filtering or selecting of metrics**: Because Netdata's distributed data architecture allows you to store all + metrics, you don't have to configure which metrics you retain. Keep everything for full visibility during + troubleshooting and root cause analysis. +- **Easy maintenance**: There is no centralized data lake to purchase, allocate, monitor, and update, removing + complexity from your monitoring infrastructure. + +## Does Netdata Cloud store my metrics? + +Netdata Cloud does not store metric values. + +To enable certain features, such as [viewing active alarms](/docs/monitor/view-active-alarms.md) or [filtering by +service](/docs/visualize/view-all-nodes.md#filter-and-group-your-infrastructure), Netdata Cloud does store configured +alarms, their status, and a list of active collectors. + +Netdata does not and never will sell your personal data or data about your deployment. + +## Long-term metrics storage with Netdata + +Any node running the Netdata Agent can store long-term metrics for any retention period, given you allocate the +appropriate amount of RAM and disk space. + +Read our document on changing [how long Netdata stores metrics](/docs/store/change-metrics-storage.md) on your nodes for +details. + +## Other options for your metrics data + +While a distributed data architecture is the default when monitoring infrastructure with Netdata, you can also configure +its behavior based on your needs or the type of infrastructure you manage. + +To archive metrics to an external time-series database, such as InfluxDB, Graphite, OpenTSDB, Elasticsearch, +TimescaleDB, and many others, see details on [integrating Netdata via exporting](/docs/export/external-databases.md). + +You can also stream between nodes using [streaming](/streaming/README.md), allowing to replicate databases and create +your own centralized data lake of metrics, if you choose to do so. + +When you use the database engine to store your metrics, you can always perform a quick backup of a node's +`/var/cache/netdata/dbengine/` folder using the tool of your choice. + +## What's next? + +You can configure the Netdata Agent to store days, weeks, or months worth of distributed, per-second data by +[configuring the database engine](/docs/store/change-metrics-storage.md). Use our calculator to determine the system +resources required to retain your desired amount of metrics, and expand or contract the database by editing a single +setting. + +[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdocs%2Fstore%2Fdistributed-data&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>) |