Anonymous statistics (#5113)

* Added shell and dashboard anonymous statistics

* Check for environment var NETDATA_REGISTRY_UNIQUE_ID

* Fix indentation

* Removed health-cmdapi-test

* docs/anonymous-statistics.md

5 files changed, 119 insertions, 25 deletions

diff --git a/docs/anonymous-statistics.md b/docs/anonymous-statistics.md
new file mode 100644
index 0000000000..1e426e2c58
--- /dev/null
+++ b/docs/anonymous-statistics.md
@@ -0,0 +1,62 @@
+# Anonymous Statistics
+
+From Netdata v1.12 and above, anonymous usage information is collected by default and send to Google Analytics. 
+The statistics calculated from this information will be used for:
+
+1. **Quality assurance**, to help us understand if netdata behaves as expected and help us identify repeating issues for certain distributions or environment.
+
+2. **Usage statistics**, to help us focus on the parts of netdata that are used the most, or help us identify the extend our development decisions influence the community.
+
+Information is sent to Netdata via two different channels:
+- Google Tag Manager is used when an agent's dashboard is accessed.
+- The script `anonymous-statistics.sh` is executed by the Netdata daemon, when Netdata starts, stops cleanly, or fails.
+
+Both methods are controlled via the same [opt-out mechanism](#opt-out)
+
+## Google tag manager
+
+Google tag manager (GTM) is the recommended way of collecting statistics for new implementations using GA. Unlike the older API, the logic of when to send information to GA and what information to send is controlled centrally.
+
+We have configured GTM to trigger the tag only when the variable `anonymous_statistics` is true. The value of this variable is controlled via the [opt-out mechanism](#opt-out).
+
+To ensure anonymity of the stored information, we have configured GTM's GA variable "Fields to set" as follows: 
+
+|Field Name|Value
+|---|---
+|page|netdata-dashboard
+|hostname|dashboard.my-netdata.io
+|anonymizeIp|true
+|title|netdata dashboard
+|campaignSource|{{machine_guid}}
+|campaignMedium|web
+|referrer|http://dashboard.my-netdata.io
+|Page URL|http://dashboard.my-netdata.io/netdata-dashboard
+|Page Hostname|http://dashboard.my-netdata.io
+|Page Path|/netdata-dashboard
+|location|http://dashboard.my-netdata.io
+
+In addition, the netdata-generated unique machine guid is sent to GA via a custom dimension.
+You can verify the effect of these settings by examining the GA `collect` request parameters.
+
+The only thing that's impossible for us to prevent from being **sent** is the URL in the "Referrer" Header of the browser request to GA. However, the settings above ensure that all **stored** URLs and host names are anonymized.
+
+## Anonymous Statistics Script
+
+Every time the daemon is started or stopped and every time a fatal condition is encountered, netdata uses the anonymous statistics script to collect system information and send it to GA via an http call. The information collected for all events is:
+ - Netdata version
+ - OS name, version, id, id_like
+ - Kernel name, version, architecture
+ - Virtualization technology 
+ - Containerization technology 
+
+Furthermore, the FATAL event sends the Netdata process & thread name, along with the source code function, source code filename and source code line number of the fatal error.
+ 
+To see exactly what and how is collected, you can review the script template `daemon/anonymous-statistics.sh.in`. The template is converted to a bash script called `anonymous-statistics.sh`, installed under the Netdata `plugins directory`, which is usually `/usr/libexec/netdata/plugins.d`. 
+
+## Opt-Out
+
+To opt-out from sending anonymous statistics, you can create a file called `.opt-out-from-anonymous-statistics` under the user configuration directory (usually `/etc/netdata`). The effect of creating the file is the following:
+- The daemon will never execute the anonymous statistics script
+- The anonymous statistics script will exit immediately if called via any other way (e.g. shell)
+- The Google Tag Manager Javascript snippet will remain in the page, but the linked tag will not be fired. The effect is that no data will ever be sent to GA. 
+
diff --git a/docs/generator/buildyaml.sh b/docs/generator/buildyaml.sh
index 3a96643ac8..1e70f8ec54 100755
--- a/docs/generator/buildyaml.sh
+++ b/docs/generator/buildyaml.sh
@@ -116,6 +116,7 @@ navpart 1 . README "About"
 
 echo -ne "    - 'docs/Demo-Sites.md'
     - 'docs/netdata-security.md'
+    - 'docs/anonymous-statistics.md'
     - 'docs/Donations-netdata-has-received.md'
     - 'docs/a-github-star-is-important.md'
     - REDISTRIBUTED.md
@@ -229,7 +230,7 @@ echo -ne "- Hacking netdata:
     - 'docs/Netdata-Security-and-Disclosure-Information.md'
     - CONTRIBUTORS.md
 "
-navpart 2 makeself "" "" 4
+navpart 2 packaging/makeself "" "" 4
 navpart 2 libnetdata "" "libnetdata" 4
 navpart 2 contrib
 navpart 2 tests "" "" 2
diff --git a/docs/netdata-security.md b/docs/netdata-security.md
index 09631dc724..642881067e 100644
--- a/docs/netdata-security.md
+++ b/docs/netdata-security.md
@@ -149,7 +149,9 @@ Of course, there are many more methods you could use to protect Netdata:
 
 - install all your Netdata in **headless data collector** mode, forwarding all metrics in real-time to a master Netdata server, which will be protected with authentication using an nginx server running locally at the master Netdata server. This requires more resources (you will need a bigger master Netdata server), but does not require any firewall changes, since all the slave Netdata servers will not be listening for incoming connections.
 
-## Registry or how to not send any information to a third party server
+## Anonymous Statistics
+
+### Registry or how to not send any information to a third party server
 
 The default configuration uses a public registry under registry.my-netdata.io (more information about the registry here: [mynetdata-menu-item](../registry/) ). Please be aware that if you use that public registry, you submit the following information to a third party server: 
 - The url where you open the web-ui in the browser (via http request referer)
@@ -157,6 +159,16 @@ The default configuration uses a public registry under registry.my-netdata.io (m
 
 If sending this information to the central Netdata registry violates your security policies, you can configure Netdat to [run your own registry](../registry/#run-your-own-registry).
 
+### Opt out of anonymous statistics
+
+Starting with v1.12 Netdata also collects [anonymous statistics](anonymous-statistics.md) on certain events for: 
+
+1. **Quality assurance**, to help us understand if netdata behaves as expected and help us identify repeating issues for certain distributions or environments.
+
+2. **Usage statistics**, to help us focus on the parts of Netdata that are used the most, or help us identify the extent our development decisions influence the community.
+
+To opt-out from sending anonymous statistics, you can create a file called `.opt-out-from-anonymous-statistics` under the user configuration directory (usually `/etc/netdata`). 
+
 ## Netdata directories
 
 path|owner|permissions| netdata |comments|
diff --git a/docs/privacy-policy.md b/docs/privacy-policy.md
index d8f1973fc8..af50b88510 100644
--- a/docs/privacy-policy.md
+++ b/docs/privacy-policy.md
@@ -1,12 +1,12 @@
-# Privacy policy
+# Privacy Policy
 
 ## 1. Preamble
 
-This Privacy Policy explains the collection, use, processing, transferring and disclosure of personal information by Netdata, Inc (“ND” or “Netdata”), a Delaware corporation.
+This Privacy Policy explains the collection, use, processing, transferring and disclosure of personal information by Netdata, Inc (“ND” or “Netdata”), a Delaware Corporation.
 
-This Privacy Policy is incorporated into and made part of Netdata Master Terms of Use (“Master Terms”) located [here](terms-of-use.md).
+This Privacy Policy is incorporated into and made part of the Netdata Master Terms of Use (“Master Terms”) located [here](terms-of-use.md).
 
-Unless otherwise noted on a particular website or service hosted by Netdata, this Privacy Policy applies to your use of all websites that Netdata operates. These include https://netdata.io and https://netdata.cloud, together with all other subdomains thereof, (collectively, the “Websites”). This Privacy Policy also applies to all products, information, and services provided through the Websites, including without limitation the license chooser, legal tools, the ND Login Services (defined below), and the ND Global Network community website (together with the Websites, the “Services”).
+Unless otherwise noted on a particular website or service hosted by Netdata, this Privacy Policy applies to your use of all websites that Netdata operates. These include https://my-netdata.io and https://netdata.cloud, together with all other subdomains thereof, (collectively, the “Websites”). This Privacy Policy also applies to all products, information, and services provided through the Websites, including without limitation the ND agent, the ND registry, the ND hub and the ND cloud website (together with the Websites, the “Services”).
 
 In addition, supplemental Privacy Policy terms (“Supplemental Privacy Policy Terms”) may apply to a particular Service. All such Supplemental Privacy Policy Terms will be accessible for you to read either within, or through your use of, that particular Service.
 
@@ -26,28 +26,52 @@ As used in this policy, “personal information” means information that would
 
 ND collects and uses personal information in the following ways:
 
-Website and Fundraising Analytics: When you visit our Websites and use our Services, ND collects some information about your activities through tools such as Google Analytics. The type of information that we collect focuses on general information such as country or city where you are located, pages visited, time spent on pages, heat-map of visitors’ activity on the site, information about the browser you are using, etc. ND collects and uses this information pursuant to our legitimate interest in enhancing the security and utility of our Services. The information we gather and process is used in the aggregate to spot trends without deliberately identifying individuals, except in cases where you engage in a transaction with Netdata by donating money, or purchasing services. In those cases, ND retains certain information about your visit to the Services pursuant to its legitimate interest in understanding its community of supporters for fundraising purposes, and this information is stored in connection with other personal information you provide to ND.
+Website and Analytics: When you visit our Websites and use our Services, ND collects some information about your activities through tools such as Google Analytics. The type of information that we collect focuses on general information such as country or city where you are located, pages visited, time spent on pages, heat-map of visitors’ activity on the site, information about the browser you are using, etc. ND collects and uses this information pursuant to our legitimate interest in enhancing the security and utility of our Services. The information we gather and process is used in the aggregate to spot trends without deliberately identifying individuals.
 
-Note that you can learn about Google’s practices in connection with its analytics services and how to opt out of it by downloading the Google Analytics opt-out browser add-on, available at https://tools.google.com/dlpage/gaoptout. If you would like to opt out of ND’s fundraising analytics, please contact legal@netdata.cloud with your request.
+Note that you can learn about Google’s practices in connection with its analytics services and how to opt out of it by downloading the Google Analytics opt-out browser add-on, available at https://tools.google.com/dlpage/gaoptout. 
 
 Information from Cookies: We and our service providers (for example, Google Analytics as described above) may collect information using cookies or similar technologies for the purposes described above and below. Cookies are pieces of information that are stored by your browser on the hard drive or memory of your computer or other Internet access device.  Cookies may enable us to personalize your experience on the Services, maintain a persistent session, passively collect demographic information about your computer, and monitor advertisements and other activities. The Websites may use different kinds of cookies and other types of local storage (such as browser-based or plugin-based local storage).
 
-Log-In Services: When you register to obtain a user account on any of the Services (any such person, a “Registered User”), including but not limited to the NDID service, you will be asked to provide personal information to create your account and establish a password and profile. ND collects and uses this personal information pursuant to its legitimate interest in establishing and maintaining your account providing you with the features we provide Registered Users. We may use your email address to contact you regarding changes to this policy or other applicable policies. The name or nickname you provide in connection with your account may be used to attribute you in connection with any content you submit to any Service. In addition, whenever you use a ND Login Service to log into a Website or use the Services, our servers keep a plain text log of the Websites you visit and when you visit them.
+
+ND Registry: The global registry, together with certain browser features, allow netdata to provide unified cross-server dashboards, via the `my-netdata` menu. The menu lists the netdata servers you have visited. For example, when you jump from server to server using the `my-netdata` menu, several session settings (like the currently viewed charts, the current zoom and pan operations on the charts, etc.) are propagated to the new server, so that the new dashboard will come with exactly the same view. The global registry keeps track of 3 entities:
+
+1. **machines**: i.e. the netdata installations (a random GUID generated by each netdata the first time it starts; we call this **machine_guid**). For each netdata installation (each `machine_guid`) the registry keeps track of the different URLs it is accessed.
+
+2. **persons**: i.e. the web browsers accessing the netdata installations (a random GUID generated by the registry the first time it sees a new web browser; we call this **person_guid**). For each person, the registry keeps track of the netdata installations it has accessed and their URLs.
+
+3. **URLs** of netdata installations (as seen by the web browsers). For each URL, the registry keeps the URL and nothing more. Each URL is linked to *persons* and *machines*. The only way to find a URL is to know its **machine_guid** or have a **person_guid** it is linked to it.
+
+If sending this information is against your policies, you can [run your own registry](../registry/#run-your-own-registry). 
+Note that ND versions with the 'Sign in' feature of the ND Cloud do not use the global registry. 
+
+ND Cloud: When you sign up to obtain a user account via the 'Sign in' link on the ND agent user interface, ND is granted access to personal information in the user profile of the authentication provider you choose (e.g. GitHub or Google). ND collects and uses this personal information pursuant to its legitimate interest in establishing and maintaining your account providing you with the features we provide Registered Users. We may use your email address to contact you regarding changes to this policy or other applicable policies. The login name or email address of your profile may be used to attribute you in connection with any content you submit to any Service. 
+
+Anonymous Usage Statistics: From Netdata v1.12 and above, anonymous usage information is collected by default on certain events of the ND daemon and send to Google Analytics. Every time the daemon is started or stopped and every time a fatal condition is encountered, netdata collects system information and sends it to GA via an http call. The information collected for all events is:
+ - Netdata version
+ - OS name, version, id, id_like
+ - Kernel name, version, architecture
+ - Virtualization technology 
+ - Containerization technology 
+Furthermore, the FATAL event sends the Netdata process & thread info, along with the file, function and line of the fatal error.
+
+The statistics calculated from this information are used for:
+
+1. **Quality assurance**, to help us understand if netdata behaves as expected and help us identify repeating issues for certain distributions or environment.
+
+2. **Usage statistics**, to help us focus on the parts of netdata that are used the most, or help us identify the extend our development decisions influence the community.
+
+To opt-out from sending anonymous statistics, you can create reate a file called `.opt-out-from-anonymous-statistics` under the user configuration directory (usually `/etc/netdata`).
 
 Emails and Newsletters: When you sign up to receive updates from Netdata or otherwise subscribe to one of our mailing lists, you will be asked to provide some personal information. ND collects and uses this personal information pursuant to its legitimate interest in providing news and updates to, and collaborating with, its supporters and volunteers.
 
 Email Analytics: When you receive communications from ND after signing up for the ND newsletter, campaign updates, or other ongoing email communications from ND, we may use analytics to track whether you open the mail, click on the links, and otherwise interact with what we send. You may opt out of this tracking by choosing to get plain-text emails from ND. ND collects and uses this personal information pursuant to its legitimate interest in understanding the interests of its community of supporters and volunteers in order to provide more relevant news and updates.
 
-Donations: When you donate money to Netdata or purchase Services, you will be asked to provide personal information, including payment information. ND collects and uses this personal information pursuant to its legitimate interest in raising funds to ensure the sustainability of our nonprofit organization and, where applicable, to provide you with the merchandise, event, or program you purchased.
-
 Other Voluntarily Provided Information: When you provide feedback to Netdata, sign a petition distributed by ND, or otherwise submit personal information to Netdata, ND collects and uses this personal information pursuant to its legitimate interest in better understanding our community of supporters and volunteers and in furtherance of the particular program or activity to which you provided feedback or other input.
 
 ## 4. Retention of Personal Information
 
 The majority of the personal information collected and used as explained in Section 3 above is aggregated and stored in a central database provided by a third party service provider. ND aggregates this data pursuant to its legitimate interest in having information stored in a single location to minimize complexity, increase consistency in internal practices, better understand its community of supporters and volunteers, and enhance the security of the data.  
 
-ND erases the web browser logs described above on a regular, rolling basis.
-
 ## 5. Access to Your Personal Information
 
 You are generally entitled to access personal information that Netdata holds and to have inaccurate data corrected or removed to the extent ND still maintains it. In certain circumstances, you also may have the right to object for legitimate reasons to the processing or transfer of personal information. If you wish to exercise any of these rights, please write to legal@netdata.cloud explaining your request.
@@ -56,15 +80,13 @@ You are generally entitled to access personal information that Netdata holds and
 
 ND does not disclose personal information to third parties except as specified elsewhere in this policy and in the following instances:
 
-Netdata may share personal information with our contractors and service providers in order to undertake the activities described in Section 3. 
-
 We may disclose your personal information to third parties in a good faith belief that such disclosure is reasonably necessary to (a) take action regarding suspected illegal activities; (b) enforce or apply our Master Terms and this Privacy Policy; (c) enforce our Charter, including the Code of Conduct and policies contained and incorporated therein, or (d) comply with legal process, such as a search warrant, subpoena, statute, or court order.
 
 ## 7. Security of Your Personal Information
 
-Netdata has implemented reasonable physical, technical, and organizational security measures for personal information that Netdata processes against accidental or unlawful destruction, or accidental loss, alteration, unauthorized disclosure or access, in compliance with applicable law. However, no website can fully eliminate security risks. Third parties may circumvent our security measures to unlawfully intercept or access transmissions or private communications. If any data breach occurs, we will post a reasonably prominent notice to the Websites and comply with all other applicable data privacy requirements including, when required, personal notice to you if you have provided and we have maintained an email address for you.
+Netdata has implemented reasonable physical, technical, and organizational security measures for personal information that Netdata processes against accidental or unlawful destruction, or accidental loss, alteration, unauthorized disclosure or access, in compliance with applicable law. However, no website can fully eliminate security risks. If any data breach occurs, we will post a reasonably prominent notice to the Websites and comply with all other applicable data privacy requirements including, when required, personal notice to you if you have provided and we have maintained an email address for you.
 
-The ND Login Services account systems have security risks in addition to those described above. Among other things, they are vulnerable to DNS attacks, and using any ND Login Service may increase the risk of phishing.
+The ND Cloud Services have security risks in addition to those described above. Among other things, they are vulnerable to DNS attacks, and using any ND Cloud Service may increase the risk of phishing.
 
 ## 8. Children
 
@@ -82,14 +104,12 @@ The Services may provide links to a wide variety of third party websites. You sh
 
 If you are accessing or using the Services in regions with laws governing data collection, processing, transfer and use, please note that when we use and share your data as specified in this policy, we may transfer your information to recipients in countries other than the country in which the information was originally collected. Those countries may not have the same data protection laws as the country in which you initially provided the information.
 
-The Services are currently hosted in the United States and the United Kingdom, which means your personal information may be located on servers in the United States and/or the United Kingdom. The majority of contractors that Netdata is using as of the effective date of this Privacy Policy are located in the United States and in Canada, but this may change from time to time.
-
 Data transferred from the European Union to the United States or outside the European Union will be made on the grounds of a certification to the E.U./U.S. Privacy Shield regime and/or a data transfer agreement based on the Standard Contractual Clauses approved of by the European Commission respectively, consistent with applicable data privacy requirements.
 
 ## 12. Changes to this Privacy Policy
 
 We may occasionally update this Privacy Policy. When we do, we will provide you with notice of such update through (at a minimum) a reasonably prominent notice on the Websites and Services, and will revise the Effective Date below. We encourage you to periodically review this Privacy Policy to stay informed about how we are protecting, using, processing and transferring the personal information we collect.
 
-***Effective Date: 10 November 2018.***
+Effective Date: 8 January 2019.
 
 [![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdocs%2Fprivacy-policy&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)]()
diff --git a/docs/terms-of-use.md b/docs/terms-of-use.md
index 378ba3e85d..5565f60565 100644
--- a/docs/terms-of-use.md
+++ b/docs/terms-of-use.md
@@ -1,12 +1,11 @@
-# Terms of use
-
-Netdata Terms of Use for the sites https://my-netdata.io and https://netdata.cloud and their subdomains. For the Netdata agent and the included redistributed software terms of use,  refer to [Redistributed software](../REDISTRIBUTED.md)
+# Terms of Use
 
+Netdata Master Terms of Use
 Effective as of 25 May 2018
 
 ## 1. General Information Regarding These Terms of Use
 
-Master terms: Welcome, and thank you for your interest in Netdata (“Netdata,” “ND,” “we,” “our,” or “us”). Unless otherwise noted on a particular site or service, these master terms of use (“Master Terms”) apply to your use of all of the websites that Netdata Corporation operates. These include https://my-netdata.io and https://netdata.cloud, together with all other subdomains thereof, (collectively, the “Websites”). The Master Terms also apply to all products, information, and services provided through the Websites, such as the NDID Login Service.
+Master terms: Welcome, and thank you for your interest in Netdata (“Netdata, Inc.” “ND,” “we,” “our,” or “us”). Unless otherwise noted on a particular site or service, these master terms of use (“Master Terms”) apply to your use of all of the websites that Netdata Corporation operates. These include https://my-netdata.io and https://netdata.cloud, together with all other subdomains thereof, (collectively, the “Websites”). The Master Terms also apply to all products, information, and services provided through the Websites, such as the NDID Login Service.
 
 Additional terms: In addition to the Master Terms, your use of any Services may also be subject to specific terms applicable to a particular Service (“Additional Terms”). If there is any conflict between the Additional Terms and the Master Terms, then the Additional Terms apply in relation to the relevant Service.
 
@@ -38,7 +37,7 @@ Provided as-is: You acknowledge that Netdata does not make any representations o
 
 You agree that you are solely responsible for your reuse of Content made available through the Services, including providing proper attribution. You should review the terms of the applicable license before you use the Content so that you know what you can and cannot do.
 
-Licensing: ND-Owned Content: Other than the text of Netdata licenses, CC licenses, and other legal tools and the text of the deeds for all legal tools (all of which are made available under the CC Public Domain Dedication), Netdata trademarks (subject to the Trademark Policy), and the software code, all Content on the Websites is licensed under the Creative Commons Attribution 4.0 license, unless otherwise marked. See the ND Policies page for more information.
+Licensing: ND-Owned Content: Other than the text of Netdata licenses, ND licenses, and other legal tools and the text of the deeds for all legal tools, Netdata trademarks (subject to the Trademark Policy), and the software code, all Content on the Websites is licensed under the Creative Commons Attribution 4.0 license, unless otherwise marked. See the ND Policies page for more information.
 
 ND-Owned Code: All of CC’s software code is free software; please check our code repository for the specific license on software you want to reuse.