Fixing DNS-lookup performance issue on FreeBSD. (#7132)

Our default configuration includes:
   allow connections from = localhost *
   allow management from = localhost

The problem occurs when a connection is received that passes the `allow connections` pattern
match, but fails the ACL check for `allow management`. During the failure processing path the
DNS lookup is triggered to allow the FQDN to be checked against the pattern. On a FreeBSD
system this lookup fails more slowly than linux and causes a visible performance problem
during stress-testing.

The fix adds a heuristic to analyse the patterns and determine if it is possible to match a DNS name,
or only match a numeric IP address (either IPv4 or IPv6), or only match a constant value. This
heuristic is used to disable the DNS checks when they cannot produce anything that may match
the pattern. Each heuristic is evaluated once, when the configuration is loaded, not per-connection to the agent.

Because the heuristic is not exact it can be overridden using the new config options for each of the ACL connection filters to set it to "yes", "no" or "heuristic". The default for everything *except* the netdata.conf ACL is "heuristic". Because of the numeric-patterns in the netdata.conf ACL the default is set to "no".

4 files changed, 42 insertions, 7 deletions

diff --git a/web/server/README.md b/web/server/README.md
index 9f47cb8d93..4866173545 100644
--- a/web/server/README.md
+++ b/web/server/README.md
@@ -149,7 +149,7 @@ Netdata supports access lists in `netdata.conf`:
 	allow management from = localhost
 ```
 
-`*` does string matches on the IPs of the clients.
+`*` does string matches on the IPs or FQDNs of the clients.
 
 -   `allow connections from` matches anyone that connects on the Netdata port(s).
      So, if someone is not allowed, it will be connected and disconnected immediately, without reading even
@@ -169,6 +169,26 @@ Netdata supports access lists in `netdata.conf`:
 
 -   `allow management from` checks the IPs to allow API management calls. Management via the API is currently supported for [health](../api/health/#health-management-api)
 
+In order to check the FQDN of the connection without opening the Netdata agent to DNS-spoofing, a reverse-dns record
+must be setup for the connecting host. At connection time the reverse-dns of the peer IP address is resolved, and
+a forward DNS resolution is made to validate the IP address against the name-pattern.
+
+Please note that this process can be expensive on a machine that is serving many connections. Each access list has an
+associated configuration option to turn off DNS-based patterns completely to avoid incurring this cost at run-time:
+
+```
+	allow connections by dns = heuristic
+	allow dashboard by dns = heuristic
+	allow badges by dns = heuristic
+	allow streaming by dns = heuristic
+	allow netdata.conf by dns = no
+	allow management by dns = heuristic
+```
+
+The three possible values for each of these options are `yes`, `no` and `heuristic`. The `heuristic` option disables
+the check when the pattern only contains IPv4/IPv6 addresses or `localhost`, and enables it when wildcards are
+present that may match DNS FQDNs.
+
 ### Other netdata.conf [web] section options
 
 |setting|default|info|
diff --git a/web/server/static/static-threaded.c b/web/server/static/static-threaded.c
index c432b9bf30..b9c91ef147 100644
--- a/web/server/static/static-threaded.c
+++ b/web/server/static/static-threaded.c
@@ -396,6 +396,7 @@ void *socket_listen_main_static_threaded_worker(void *ptr) {
                         , web_server_snd_callback
                         , web_server_tmr_callback
                         , web_allow_connections_from
+                        , web_allow_connections_dns
                         , NULL
                         , web_client_first_request_timeout
                         , web_client_timeout
diff --git a/web/server/web_server.c b/web/server/web_server.c
index b8b84bc36e..4da08d4319 100644
--- a/web/server/web_server.c
+++ b/web/server/web_server.c
@@ -74,46 +74,53 @@ void api_listen_sockets_setup(void) {
 // access lists
 
 SIMPLE_PATTERN *web_allow_connections_from = NULL;
+int web_allow_connections_dns;
 
 // WEB_CLIENT_ACL
 SIMPLE_PATTERN *web_allow_dashboard_from = NULL;
+int             web_allow_dashboard_dns;
 SIMPLE_PATTERN *web_allow_registry_from = NULL;
+int             web_allow_registry_dns;
 SIMPLE_PATTERN *web_allow_badges_from = NULL;
+int             web_allow_badges_dns;
 SIMPLE_PATTERN *web_allow_mgmt_from = NULL;
+int             web_allow_mgmt_dns;
 SIMPLE_PATTERN *web_allow_streaming_from = NULL;
+int             web_allow_streaming_dns;
 SIMPLE_PATTERN *web_allow_netdataconf_from = NULL;
+int             web_allow_netdataconf_dns;
 
 void web_client_update_acl_matches(struct web_client *w) {
     w->acl = WEB_CLIENT_ACL_NONE;
 
     if (!web_allow_dashboard_from ||
         connection_allowed(w->ifd, w->client_ip, w->client_host, sizeof(w->client_host),
-                           web_allow_dashboard_from, "dashboard"))
+                           web_allow_dashboard_from, "dashboard", web_allow_dashboard_dns))
         w->acl |= WEB_CLIENT_ACL_DASHBOARD;
 
     if (!web_allow_registry_from ||
         connection_allowed(w->ifd, w->client_ip, w->client_host, sizeof(w->client_host),
-                           web_allow_registry_from, "registry"))
+                           web_allow_registry_from, "registry", web_allow_registry_dns))
         w->acl |= WEB_CLIENT_ACL_REGISTRY;
 
     if (!web_allow_badges_from ||
         connection_allowed(w->ifd, w->client_ip, w->client_host, sizeof(w->client_host),
-                           web_allow_badges_from, "badges"))
+                           web_allow_badges_from, "badges", web_allow_badges_dns))
         w->acl |= WEB_CLIENT_ACL_BADGE;
 
     if (!web_allow_mgmt_from ||
         connection_allowed(w->ifd, w->client_ip, w->client_host, sizeof(w->client_host),
-                           web_allow_mgmt_from, "management"))
+                           web_allow_mgmt_from, "management", web_allow_mgmt_dns))
         w->acl |= WEB_CLIENT_ACL_MGMT;
 
     if (!web_allow_streaming_from ||
         connection_allowed(w->ifd, w->client_ip, w->client_host, sizeof(w->client_host),
-                           web_allow_streaming_from, "streaming"))
+                           web_allow_streaming_from, "streaming", web_allow_streaming_dns))
         w->acl |= WEB_CLIENT_ACL_STREAMING;
 
     if (!web_allow_netdataconf_from ||
        connection_allowed(w->ifd, w->client_ip, w->client_host, sizeof(w->client_host),
-                          web_allow_netdataconf_from, "netdata.conf"))
+                          web_allow_netdataconf_from, "netdata.conf", web_allow_netdataconf_dns))
         w->acl |= WEB_CLIENT_ACL_NETDATACONF;
 
     w->acl &= w->port_acl;
diff --git a/web/server/web_server.h b/web/server/web_server.h
index e7c2dd4482..7bfc81c9df 100644
--- a/web/server/web_server.h
+++ b/web/server/web_server.h
@@ -20,12 +20,19 @@ typedef enum web_server_mode {
 } WEB_SERVER_MODE;
 
 extern SIMPLE_PATTERN *web_allow_connections_from;
+extern int             web_allow_connections_dns;
 extern SIMPLE_PATTERN *web_allow_dashboard_from;
+extern int             web_allow_dashboard_dns;
 extern SIMPLE_PATTERN *web_allow_registry_from;
+extern int             web_allow_registry_dns;
 extern SIMPLE_PATTERN *web_allow_badges_from;
+extern int             web_allow_badges_dns;
 extern SIMPLE_PATTERN *web_allow_streaming_from;
+extern int             web_allow_streaming_dns;
 extern SIMPLE_PATTERN *web_allow_netdataconf_from;
+extern int             web_allow_netdataconf_dns;
 extern SIMPLE_PATTERN *web_allow_mgmt_from;
+extern int             web_allow_mgmt_dns;
 
 extern WEB_SERVER_MODE web_server_mode;