Overhaul handling of auto-updates in the installer code. (#12076)

* Bundle updater script in native packages.

* Move code for enabling/disabling auto-updates to netdata-updater.sh

This lets us handle the logic sanely from the kickstart script
regardless of the install method, and allows users to more reliably
toggle auto-updates themselves without having to understand what is
being done.

* Add proper case-agnosticism to auto-update type selection.

* Move auto-updater handling code to kickstart script.

* Properly handle running against an older source tree.

* First part of updater docmentation updates.

* Fixed handling of updater in DEB packages.

* Further documentation updates.

* Minor typo fixes.

6 files changed, 252 insertions, 100 deletions

diff --git a/packaging/installer/UPDATE.md b/packaging/installer/UPDATE.md
index fbe7f57579..3b75e8223e 100644
--- a/packaging/installer/UPDATE.md
+++ b/packaging/installer/UPDATE.md
@@ -6,8 +6,9 @@ custom_edit_url: https://github.com/netdata/netdata/edit/master/packaging/instal
 
 # Update the Netdata Agent
 
-By default, the Netdata Agent automatically updates with the latest nightly version. If you opted out of automatic
-updates, you need to update your Netdata Agent to the latest nightly or stable version.
+By default, the Netdata Agent automatically updates with the latest nightly or stable version depending on which
+you installed. If you opted out of automatic updates, you need to update your Netdata Agent to the latest nightly
+or stable version. You can also [enable or disable automatic updates on an existing install](#control-automatic-updates).
 
 > 💡 Looking to reinstall the Netdata Agent to enable a feature, update an Agent that cannot update automatically, or
 > troubleshoot an error during the installation process? See our [reinstallation doc](/packaging/installer/REINSTALL.md)
@@ -20,28 +21,35 @@ icon in the local Agent dashboard's top navigation. This modal informs you wheth
 
 ## Determine which installation method you used
 
+Starting with netdata v1.33.0, you can use Netdata itself to determine the installation type by running:
+
+```bash
+netdata -W buildinfo | grep 'Install type:'
+```
+
+If this produces no output, you have an older install and will have to manually look in the Netdata config directory.
 If you are not sure where your Netdata config directory is, see the [configuration doc](/docs/configure/nodes.md). In
 most installations, this is `/etc/netdata`.
 
 Use `cd` to navigate to the Netdata config directory, then use `ls -a` to look for a file called `.install-type`.
 
 -   If the `.install-type` file doex not exist, look for a file in the same directory called `.environment`.
-    -   If the `.environment` file does not exist, you probably installed Netdata using your system package manager
-        and should update it the same way you would run updates on the system itself.
-    -   If the `.environment` file does exist, then our [regular update method](#updates-for-most-systems) should
-        work correctly.
+    -   If the `.environment` file does not exist, then you have a ‘custom’ install.
+    -   If the `.environment` file does exist, then you have either a ‘static’ or ‘build’ install.
 -   If the `.install-type` file does exist, check it’s contents with `cat .install-type`.
-    -   If the `INSTALL_TYPE` key has a value of `custom`, you probably installed Netdata using your system
-        package manager and should update it the same way you would run updates on the system itself.
-    -   If the `INSTALL_TYPE` key has a value of `oci`, the install is from a Docker image.
-    -   Otherwise, the install should work with our [regular update method](#updates-for-most-systems).
+    -   The value of the `INSTALL_TYPE` key indicates what type of install you have.
 
-Next, use the appropriate method to update the Netdata Agent:
+The exact update method to use depends on the install type:
 
--   [Updates for most systems](#updates-for-most-systems)
--   [Docker](#docker)
--   [macOS](#macos)
--   [Manual installation from Git](#manual-installation-from-git)
+-   Installs with an install type of 'custom' usually indicate installing a third-party package through the system
+    package manager. To update these installs, you should update the package just like you would any other package
+    on your system.
+-   Installs with an install type starting with `binpkg` or ending with `build` or `static` can be updated using
+    our [regular update method](#updates-for-most-systems).
+-   Installs with an install type of 'oci' were created from our official Docker images, and should be updated
+    using our [Docker](#docker) update procedure.
+-   macOS users should check [our update instructions for macOS](#macos).
+-   Manually built installs should check [our update instructions for manual builds](#manual-installation-from-git).
 
 ## Updates for most systems
 
@@ -95,6 +103,9 @@ Homebrew downloads the latest Netdata via the
 [formulae](https://github.com/Homebrew/homebrew-core/blob/master/Formula/netdata.rb), ensures all dependencies are met,
 and updates Netdata via reinstallation.
 
+If you instead installed Netdata using our one-line installation script, you can use our [regular update
+instructions](#updates-for-most-systems) to update Netdata.
+
 ## Manual installation from Git
 
 If you installed [Netdata manually from Git](/packaging/installer/methods/manual.md), you can run that installer again
@@ -117,4 +128,31 @@ sudo ./netdata-installer.sh
 > ⚠️ If you installed Netdata with any optional parameters, such as `--no-updates` to disable automatic updates, and
 > want to retain those settings, you need to set them again during this process.
 
+## Control automatic updates
+
+Starting with Netdata v1.34.0, you can easily enable or disable automatic updates on an existing installation
+using the updater script.
+
+For most installs on Linux, you can enable auto-updates with:
+
+```bash
+/usr/libexec/netdata/netdata-updater.sh --enable-auto-updates
+```
+
+and disable them with:
 
+```bash
+/usr/libexec/netdata/netdata-updater.sh --disable-auto-updates
+```
+
+For static installs, instead use:
+
+```bash
+/opt/netdata/usr/libexec/netdata/netdata-updater.sh --enable-auto-updates
+```
+
+and:
+
+```bash
+/opt/netdata/usr/libexec/netdata/netdata-updater.sh --disable-auto-updates
+```
diff --git a/packaging/installer/functions.sh b/packaging/installer/functions.sh
index 56f1a56585..83e21f3c29 100644
--- a/packaging/installer/functions.sh
+++ b/packaging/installer/functions.sh
@@ -968,76 +968,6 @@ cleanup_old_netdata_updater() {
   return 0
 }
 
-enable_netdata_updater() {
-
-  if [ -n "${1}" ] ; then
-    updater_type="${1}"
-  else
-    updater_type="$(_get_scheduler_type)"
-  fi
-
-  case "${updater_type}" in
-    "systemd")
-      systemctl enable netdata-updater.timer
-
-      echo >&2 "Auto-updating has been enabled using a systemd timer unit."
-      echo >&2
-      echo >&2 "If the update process fails, the failure will be logged to the systemd journal just like a regular service failure."
-      echo >&2 "Successful updates should produce empty logs."
-      echo >&2
-      ;;
-    "interval")
-      ln -sf "${NETDATA_PREFIX}/usr/libexec/netdata/netdata-updater.sh" "$(_get_intervaldir)/netdata-updater"
-
-      echo >&2 "Auto-updating has been enabled through cron, updater script linked to ${TPUT_RED}${TPUT_BOLD}$(_get_intervaldir)/netdata-updater${TPUT_RESET}"
-      echo >&2
-      echo >&2 "If the update process fails and you have email notifications set up correctly for cron on this system, you should receive an email notification of the failure."
-      echo >&2 "Successful updates will not send an email."
-      echo >&2
-      ;;
-    "crontab")
-      cat "${NETDATA_SOURCE_DIR}/system/netdata.crontab" > "/etc/cron.d/netdata-updater"
-
-      echo >&2 "Auto-updating has been enabled through cron, using a crontab at ${TPUT_RED}${TPUT_BOLD}/etc/cron.d/netdata-updater${TPUT_RESET}"
-      echo >&2
-      echo >&2 "If the update process fails and you have email notifications set up correctly for cron on this system, you should receive an email notification of the failure."
-      echo >&2 "Successful updates will not send an email."
-      echo >&2
-      ;;
-    *)
-      echo >&2 "Unable to determine what type of auto-update scheduling to use, not enabling auto-updates."
-      echo >&2
-      return 1
-  esac
-
-  return 0
-}
-
-disable_netdata_updater() {
-  echo >&2 "You chose *NOT* to enable auto-update, removing any links to the updater from cron (it may have happened if you are reinstalling)"
-  echo >&2
-
-  if issystemd && [ -n "$(get_systemd_service_dir)" ] ; then
-    systemctl disable netdata-updater.timer
-  fi
-
-  if [ -d /etc/cron.daily ]; then
-    rm -f /etc/cron.daily/netdata-updater.sh
-    rm -f /etc/cron.daily/netdata-updater
-  fi
-
-  if [ -d /etc/periodic/daily ]; then
-    rm -f /etc/periodic/daily/netdata-updater.sh
-    rm -f /etc/periodic/daily/netdata-updater
-  fi
-
-  if [ -d /etc/cron.d ]; then
-    rm -f /etc/cron.d/netdata-updater
-  fi
-
-  return 0
-}
-
 set_netdata_updater_channel() {
   sed -i -e "s/^RELEASE_CHANNEL=.*/RELEASE_CHANNEL=\"${RELEASE_CHANNEL}\"/" "${NETDATA_USER_CONFIG_DIR}/.environment"
 }
diff --git a/packaging/installer/kickstart.sh b/packaging/installer/kickstart.sh
index b91ef24197..da087272a1 100755
--- a/packaging/installer/kickstart.sh
+++ b/packaging/installer/kickstart.sh
@@ -64,8 +64,9 @@ USAGE: kickstart.sh [options]
   --dont-start-it            Do not start the agent by default (only for static installs or local builds)
   --stable-channel           Install a stable version instead of a nightly build (default: install a nightly build)
   --nightly-channel          Install a nightly build instead of a stable version
-  --no-updates               Do not enable automatic updates (default: enable automatic updates)
+  --no-updates               Do not enable automatic updates (default: enable automatic updates using the best supported scheduling method)
   --auto-update              Enable automatic updates.
+  --auto-update-type         Specify a particular scheduling type for auto-updates (valid types: systemd, interval, crontab)
   --disable-telemetry        Opt-out of anonymous statistics.
   --native-only              Only install if native binary packages are available.
   --static-only              Only install if a static build is available.
@@ -785,6 +786,30 @@ claim() {
 }
 
 # ======================================================================
+# Auto-update handling code.
+set_auto_updates() {
+  if [ -x "${INSTALL_PREFIX}/usr/libexec/netdata/netdata-updater.sh" ]; then
+    updater="${INSTALL_PREFIX}/usr/libexec/netdata/netdata-updater.sh"
+  elif [ -x "${INSTALL_PREFIX}/netdata/usr/libexec/netdata/netdata-updater.sh" ]; then
+    updater="${INSTALL_PREFIX}/netdata/usr/libexec/netdata/netdata-updater.sh"
+  else
+    warning "Could not find netdata-updater.sh."
+    return 0
+  fi
+
+  if [ "${NETDATA_AUTO_UPDATES}" = "1" ]; then
+    # This first case is for catching using a new kickstart script with an old build. It can be safely removed after v1.34.0 is released.
+    if grep -qv '--enable-auto-updates' ${updater}; then
+      echo
+    elif ! ${updater} --enable-auto-updates "${NETDATA_AUTO_UPDATE_TYPE}"; then
+      error "Failed to enable auto updates. Netdata will still work, but you will need to update manually."
+    fi
+  else
+    ${updater} --disable-auto-updates
+  fi
+}
+
+# ======================================================================
 # Native package install code.
 
 # Check for an already installed package with a given name.
@@ -1228,6 +1253,7 @@ install_on_linux() {
     case "$?" in
       0)
         NETDATA_INSTALL_SUCCESSFUL=1
+        INSTALL_PREFIX="/"
         ;;
       1)
         fatal "Unable to install on this system." F0300
@@ -1342,6 +1368,18 @@ while [ -n "${1}" ]; do
     "--stable-channel") RELEASE_CHANNEL="stable" ;;
     "--no-updates") NETDATA_AUTO_UPDATES=0 ;;
     "--auto-update") NETDATA_AUTO_UPDATES="1" ;;
+    "--auto-update-method")
+      NETDATA_AUTO_UPDATE_TYPE="$(echo "${2}" | tr '[:upper:]' '[:lower:]')"
+      case "${NETDATA_AUTO_UPDATE_TYPE}" in
+        systemd|interval|crontab)
+          shift 1
+          ;;
+        *)
+          echo "Unrecognized value for --auto-update-type. Valid values are: systemd, interval, crontab"
+          exit 1
+          ;;
+      esac
+      ;;
     "--reinstall") NETDATA_REINSTALL=1 ;;
     "--reinstall-even-if-unsafe") NETDATA_UNSAFE_REINSTALL=1 ;;
     "--claim-only") NETDATA_CLAIM_ONLY=1 ;;
@@ -1441,6 +1479,8 @@ elif [ "${NETDATA_DISABLE_CLOUD}" -eq 1 ]; then
   soft_disable_cloud
 fi
 
+set_auto_updates
+
 telemetry_event INSTALL_SUCCESS "" ""
 cleanup
 trap - EXIT
diff --git a/packaging/installer/methods/kickstart.md b/packaging/installer/methods/kickstart.md
index 822da22363..82050017a2 100644
--- a/packaging/installer/methods/kickstart.md
+++ b/packaging/installer/methods/kickstart.md
@@ -140,7 +140,7 @@ To use `md5sum` to verify the integrity of the `kickstart.sh` script you will do
 run the following:
 
 ```bash
-[ "373053d7b715af0d9c30e479a6c53809" = "$(curl -Ss https://my-netdata.io/kickstart.sh | md5sum | cut -d ' ' -f 1)" ] && echo "OK, VALID" || echo "FAILED, INVALID"
+[ "5d558c5a86050ec267f126fbe40df0ec" = "$(curl -Ss https://my-netdata.io/kickstart.sh | md5sum | cut -d ' ' -f 1)" ] && echo "OK, VALID" || echo "FAILED, INVALID"
 ```
 
 If the script is valid, this command will return `OK, VALID`.
diff --git a/packaging/installer/netdata-updater.sh b/packaging/installer/netdata-updater.sh
index bfb5503ca4..f6ae05bff2 100755
--- a/packaging/installer/netdata-updater.sh
+++ b/packaging/installer/netdata-updater.sh
@@ -73,6 +73,142 @@ if [ "${ENVIRONMENT_FILE}" = "THIS_SHOULD_BE_REPLACED_BY_INSTALLER_SCRIPT" ]; th
   fi
 fi
 
+issystemd() {
+  # if the directory /lib/systemd/system OR /usr/lib/systemd/system (SLES 12.x) does not exit, it is not systemd
+  if [ ! -d /lib/systemd/system ] && [ ! -d /usr/lib/systemd/system ]; then
+    return 1
+  fi
+
+  # if there is no systemctl command, it is not systemd
+  systemctl=$(command -v systemctl 2> /dev/null)
+  if [ -z "${systemctl}" ] || [ ! -x "${systemctl}" ]; then
+    return 1
+  fi
+
+  # if pid 1 is systemd, it is systemd
+  [ "$(basename "$(readlink /proc/1/exe)" 2> /dev/null)" = "systemd" ] && return 0
+
+  # if systemd is not running, it is not systemd
+  pids=$(safe_pidof systemd 2> /dev/null)
+  [ -z "${pids}" ] && return 1
+
+  # check if the running systemd processes are not in our namespace
+  myns="$(readlink /proc/self/ns/pid 2> /dev/null)"
+  for p in ${pids}; do
+    ns="$(readlink "/proc/${p}/ns/pid" 2> /dev/null)"
+
+    # if pid of systemd is in our namespace, it is systemd
+    [ -n "${myns}" ] && [ "${myns}" = "${ns}" ] && return 0
+  done
+
+  # else, it is not systemd
+  return 1
+}
+
+_get_scheduler_type() {
+  if _get_intervaldir > /dev/null ; then
+    echo 'interval'
+  elif issystemd ; then
+    echo 'systemd'
+  elif [ -d /etc/cron.d ] ; then
+    echo 'crontab'
+  else
+    echo 'none'
+  fi
+}
+
+_get_intervaldir() {
+  if [ -d /etc/cron.daily ]; then
+    echo /etc/cron.daily
+  elif [ -d /etc/periodic/daily ]; then
+    echo /etc/periodic/daily
+  else
+    return 1
+  fi
+
+  return 0
+}
+
+enable_netdata_updater() {
+  if [ -n "${1}" ] ; then
+    updater_type="${1}"
+  else
+    updater_type="$(_get_scheduler_type)"
+  fi
+
+  case "${updater_type}" in
+    "systemd")
+      if issystemd; then
+        systemctl enable netdata-updater.timer
+
+        info "Auto-updating has been ENABLED using a systemd timer unit.\n"
+        info "If the update process fails, the failure will be logged to the systemd journal just like a regular service failure."
+        info "Successful updates should produce empty logs."
+      else
+        error "Systemd-based auto-update scheduling requested, but this does not appear to be a systemd system."
+        error "Auto-updates have NOT been enabled."
+        return 1
+      fi
+      ;;
+    "interval")
+      if _get_intervaldir > /dev/null; then
+        ln -sf "${NETDATA_PREFIX}/usr/libexec/netdata/netdata-updater.sh" "$(_get_intervaldir)/netdata-updater"
+
+        info "Auto-updating has been ENABLED through cron, updater script linked to $(_get_intervaldir)/netdata-updater\n"
+        info "If the update process fails and you have email notifications set up correctly for cron on this system, you should receive an email notification of the failure."
+        info "Successful updates will not send an email."
+      else
+        error "Interval-based auto-update scheduling requested, but I could not find an interval scheduling directory."
+        error "Auto-updates have NOT been enabled."
+        return 1
+      fi
+      ;;
+    "crontab")
+      if [ -d "/etc/cron.d" ]; then
+        cat "${NETDATA_SOURCE_DIR}/system/netdata.crontab" > "/etc/cron.d/netdata-updater"
+
+        info "Auto-updating has been ENABLED through cron, using a crontab at /etc/cron.d/netdata-updater\n"
+        info "If the update process fails and you have email notifications set up correctly for cron on this system, you should receive an email notification of the failure."
+        info "Successful updates will not send an email."
+      else
+        error "Crontab-based auto-update scheduling requested, but there is no '/etc/cron.d'."
+        error "Auto-updates have NOT been enabled."
+        return 1
+      fi
+      ;;
+    *)
+      error "Unable to determine what type of auto-update scheduling to use."
+      error "Auto-updates have NOT been enabled."
+      return 1
+  esac
+
+  return 0
+}
+
+disable_netdata_updater() {
+  if issystemd && ( systemctl list-units --full -all | grep -Fq "netdata-updater.timer" ) ; then
+    systemctl disable netdata-updater.timer
+  fi
+
+  if [ -d /etc/cron.daily ]; then
+    rm -f /etc/cron.daily/netdata-updater.sh
+    rm -f /etc/cron.daily/netdata-updater
+  fi
+
+  if [ -d /etc/periodic/daily ]; then
+    rm -f /etc/periodic/daily/netdata-updater.sh
+    rm -f /etc/periodic/daily/netdata-updater
+  fi
+
+  if [ -d /etc/cron.d ]; then
+    rm -f /etc/cron.d/netdata-updater
+  fi
+
+  info "Auto-updates have been DISABLED."
+
+  return 0
+}
+
 str_in_list() {
   printf "%s\n" "${2}" | tr ' ' "\n" | grep -qE "^${1}\$"
   return $?
@@ -584,6 +720,25 @@ while [ -n "${1}" ]; do
   elif [ "${1}" = "--tmpdir-path" ]; then
     NETDATA_TMPDIR_PATH="${2}"
     shift 2
+  elif [ "${1}" = "--enable-auto-updates" ]; then
+    AUTO_UPDATE_TYPE="$(echo "${2}" | tr '[:upper:]' '[:lower:]')"
+    case "${AUTO_UPDATE_TYPE}" in
+      systemd|interval|crontab)
+        enable_netdata_updater "${AUTO_UPDATE_TYPE}"
+        exit $?
+        ;;
+      "")
+        enable_netdata_updater ""
+        exit $?
+        ;;
+      *)
+        error "Unrecognized updater type ${2} requested. Supported types are 'systemd', 'interval', and 'crontab'."
+        exit 1
+        ;;
+    esac
+  elif [ "${1}" = "--disable-auto-updates" ]; then
+    disable_netdata_updater
+    exit $?
   else
     break
   fi
@@ -653,7 +808,7 @@ case "${INSTALL_TYPE}" in
       update_legacy
       ;;
     custom)
-      # At this point, we _should_ have a valid `.environment` file, but it0s best to just check.
+      # At this point, we _should_ have a valid `.environment` file, but it's best to just check.
       # If we do, then behave like the legacy updater.
       if [ -n "${RELEASE_CHANNEL}" ] && [ -n "${NETDATA_PREFIX}" ] && [ -n "${REINSTALL_OPTIONS}" ]; then
         update_legacy
diff --git a/packaging/makeself/install-or-update.sh b/packaging/makeself/install-or-update.sh
index 2fd7629b20..6a4cd5b971 100755
--- a/packaging/makeself/install-or-update.sh
+++ b/packaging/makeself/install-or-update.sh
@@ -26,7 +26,6 @@ if [ -d /opt/netdata/etc/netdata.old ]; then
 fi
 
 STARTIT=1
-AUTOUPDATE=0
 REINSTALL_OPTIONS=""
 RELEASE_CHANNEL="nightly" # check .travis/create_artifacts.sh before modifying
 
@@ -36,10 +35,7 @@ while [ "${1}" ]; do
       STARTIT=0
       REINSTALL_OPTIONS="${REINSTALL_OPTIONS} ${1}"
       ;;
-    "--auto-update" | "-u")
-      AUTOUPDATE=1
-      REINSTALL_OPTIONS="${REINSTALL_OPTIONS} ${1}"
-      ;;
+    "--auto-update" | "-u") ;;
     "--stable-channel")
       RELEASE_CHANNEL="stable"
       REINSTALL_OPTIONS="${REINSTALL_OPTIONS} ${1}"
@@ -161,13 +157,6 @@ progress "Install (but not enable) netdata updater tool"
 cleanup_old_netdata_updater || run_failed "Cannot cleanup old netdata updater tool."
 install_netdata_updater || run_failed "Cannot install netdata updater tool."
 
-progress "Check if we must enable/disable the netdata updater tool"
-if [ "${AUTOUPDATE}" = "1" ]; then
-  enable_netdata_updater || run_failed "Cannot enable netdata updater tool"
-else
-  disable_netdata_updater || run_failed "Cannot disable netdata updater tool"
-fi
-
 # -----------------------------------------------------------------------------
 progress "creating quick links"