Windows Support Phase 1 (#17497)

* abstraction layer for O/S

* updates

* updates

* updates

* temp fix for protobuf

* emulated waitid()

* fix

* fix

* compatibility layer

* fix for idtype

* fix for missing includes

* fix for missing includes

* added missing includes

* added missing includes

* added missing includes

* added missing includes

* added missing includes

* added missing includes

* UUID renamed to ND_UUID to avoid conflict with windows.h

* include libnetdata.h always - no conflicts

* simplify abstraction headers

* fix missing functions

* fix missing functions

* fix missing functions

* fix missing functions

* rename MSYS to WINDOWS

* moved byteorder.h

* structure for an internal windows plugin

* 1st windows plugin

* working plugin

* fix printf

* Special case windows for protobuf

* remove cygwin, compile both as windows

* log windows libraries used

* fix cmake

* fix protobuf

* compilation

* updated compilation script

* added system.ram

* windows uptime

* perflib

* working perflibdump

* minify dump

* updates to windows plugins, enable ML

* minor compatibility fixes for cygwin and msys

* perflib-dump to its own file

* perflib now indexes names

* improvements to the library; disks module WIP

* API for selectively traversing the metrics

* first working perflib chart: disk.space

* working chart on logical and physical disks

* added windows protocols

* fix datatypes for loops

* tinysleep for native smallest sleep support

* remove libuuid dependency on windows

* fix uuid functions for macos compilation

* fix uuid comparison function

* do not overwrite uuid library functions, define them as aliases to our own

* fixed uuid_unparse functions

* fixed typo

* added perflib processor charts

* updates for compiling without posix emulation

* gather common contexts together

* fix includes on linux

* perflib-memory

* windows mem.available

* Update variable names for protobuf

* network traffic

* add network adapters that have traffic as virtual interfaces

* add -pipe to windows compilation

* reset or overflow flag is now per dimension

* dpc is now counted separately

* verified all perflib fields are processed and no text fields are present in the data

* more common contexts

* fix crash

* do not add system.net multiple times

* install deps update and shortcut

* all threads are now joinable behind the scenes

* fix threads cleanup

* prepare for abstracting threads API

* netdata threads full abstraction from pthreads

* more threads abstraction and cleanup

* more compatibility changes

* fix compiler warnings

* add base-devel to packages

* removed duplicate base-devel

* check for strndup

* check headers in quotes

* fix linux compilation

* fix attribute gnu_printf on macos

* fix for threads on macos

* mingw64 compatibility

* enable compilation on windows clion

* added instructions

* enable cloud

* compatibility fixes

* compatibility fixes

* compatibility fixes

* clion works on windows

* support both MSYSTEM=MSYS and MSYSTEM=MINGW64 for configure

* cleanup and docs

* rename uuid_t to nd_uuid_t to avoid conflict with windows uuid_t

* leftovers uuid_t

* do not include uuid.h on macos

* threads signaled cancellations

* do not install v0 dashboard on windows

* script to install openssh server on windows

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* use cleanup variable instead of pthreads push and pop

* replace all calls to netdata_thread_cleanup_push() and netdata_thread_cleanup_pop() with __attribute__((cleanup(...)))

* remove left-over freez

* make sure there are no locks acquired at thread exit

* add missing parameter

* stream receivers and senders are now voluntarily cancelled

* plugins.d now voluntarily exits its threads

* uuid_t may not be aligned to word boundaries - fix the uuid_t functions to work on unaligned objects too.

* collectors evloop is now using the new threading cancellation; ml is now not using pthread_cancel; more fixes

* eliminate threads cancellability from the code base

* fix exit timings and logs; fix uv_threads tags

* use SSL_has_pending() only when it is available

* do not use SSL_has_pending()

* dyncfg files on windows escape collon and pipe characters

* fix compilation on older systems

* fix compilation on older systems

* Create windows installer.

The installer will install everything
under C:\netdata by default.

It will:
  - Install msys2 at C:\netdata
  - Install netdata dependencies with pacman
  - Install the agent itself under C:\netdata\opt

You can start the agent by running an MSYS
shell with C:\netdata\msys2_shell.cmd and
then start the agent normally with:

/opt/netdata/usr/sbin/netdata -D

There are a more couple things to work on:

  - Verify publisher.
  - Install all deps not just libuv & protobuf.
  - Figure out how we want to auto-start the agent as a service.
  - Check how to uninstall things.

* fixed typo

* code cleanup

* Create uninstaller

---------

Co-authored-by: vkalintiris <vasilis@netdata.cloud>

1 files changed, 134 insertions, 18 deletions

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 2f9774c232..d918d3aedd 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -225,23 +225,51 @@ endif()
 # detect OS
 #
 
-set(LINUX   False)
-set(FREEBSD False)
-set(MACOS   False)
+set(LINUX       False)
+set(FREEBSD     False)
+set(MACOS       False)
+set(WINDOWS     False)
+set(FOREIGN_OS  False)
 
-if(${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
+if("${CMAKE_SYSTEM_NAME}" STREQUAL "Darwin")
         set(MACOS True)
         set(COMPILED_FOR_MACOS True)
-
         find_library(IOKIT IOKit)
         find_library(FOUNDATION Foundation)
-elseif(${CMAKE_SYSTEM_NAME} MATCHES "FreeBSD")
+        message(INFO " Compiling for MacOS... ")
+elseif("${CMAKE_SYSTEM_NAME}" STREQUAL "FreeBSD")
         set(FREEBSD True)
         set(COMPILED_FOR_FREEBSD True)
-else()
+        message(INFO " Compiling for FreeBSD... ")
+elseif("${CMAKE_SYSTEM_NAME}" STREQUAL "Linux")
         set(LINUX True)
         set(COMPILED_FOR_LINUX True)
         add_definitions(-D_GNU_SOURCE)
+        message(INFO " Compiling for Linux... ")
+elseif("${CMAKE_SYSTEM_NAME}" STREQUAL "CYGWIN" OR "${CMAKE_SYSTEM_NAME}" STREQUAL "MSYS" OR "${CMAKE_SYSTEM_NAME}" STREQUAL "Windows")
+        set(WINDOWS True)
+        set(COMPILED_FOR_WINDOWS True)
+        add_definitions(-D_GNU_SOURCE)
+
+        if($ENV{CLION_IDE})
+                # clion needs these to find the includes
+                if("${CMAKE_SYSTEM_NAME}" STREQUAL "MSYS" OR "${CMAKE_SYSTEM_NAME}" STREQUAL "Windows")
+                        if("$ENV{MSYSTEM}" STREQUAL "MSYS")
+                                include_directories(c:/msys64/usr/include)
+                                include_directories(c:/msys64/usr/include/w32api)
+                        elseif("$ENV{MSYSTEM}" STREQUAL "MINGW64")
+                                include_directories(c:/msys64/mingw64/include)
+                        elseif("$ENV{MSYSTEM}" STREQUAL "UCRT64")
+                                include_directories(c:/msys64/ucrt64/include)
+                        endif()
+                endif()
+        endif()
+
+        message(INFO " Compiling for Windows (${CMAKE_SYSTEM_NAME}, MSYSTEM=$ENV{MSYSTEM})... ")
+else()
+        set(FOREIGN_OS True)
+        set(COMPILED_FOR_FOREIGN_OS True)
+        message(WARNING " Compiling for Unknown O/S... (${CMAKE_SYSTEM_NAME})")
 endif()
 
 if(ENABLE_PLUGIN_EBPF)
@@ -326,6 +354,20 @@ check_include_file("sys/statvfs.h" HAVE_SYS_STATVFS_H)
 check_include_file("inttypes.h" HAVE_INTTYPES_H)
 check_include_file("stdint.h" HAVE_STDINT_H)
 check_include_file("sys/capability.h" HAVE_SYS_CAPABILITY_H)
+check_include_file("arpa/inet.h" HAVE_ARPA_INET_H)
+check_include_file("netinet/tcp.h" HAVE_NETINET_TCP_H)
+check_include_file("sys/ioctl.h" HAVE_SYS_IOCTL_H)
+check_include_file("grp.h" HAVE_GRP_H)
+check_include_file("pwd.h" HAVE_PWD_H)
+check_include_file("net/if.h" HAVE_NET_IF_H)
+check_include_file("poll.h" HAVE_POLL_H)
+check_include_file("syslog.h" HAVE_SYSLOG_H)
+check_include_file("sys/mman.h" HAVE_SYS_MMAN_H)
+check_include_file("sys/resource.h" HAVE_SYS_RESOURCE_H)
+check_include_file("sys/socket.h" HAVE_SYS_SOCKET_H)
+check_include_file("sys/wait.h" HAVE_SYS_WAIT_H)
+check_include_file("sys/un.h" HAVE_SYS_UN_H)
+check_include_file("spawn.h" HAVE_SPAWN_H)
 
 #
 # check symbols
@@ -340,9 +382,15 @@ check_symbol_exists(finite "math.h" HAVE_FINITE)
 check_symbol_exists(isfinite "math.h" HAVE_ISFINITE)
 check_symbol_exists(dlsym "dlfcn.h" HAVE_DLSYM)
 
+check_function_exists(pthread_getthreadid_np HAVE_PTHREAD_GETTHREADID_NP)
+check_function_exists(pthread_threadid_np HAVE_PTHREAD_THREADID_NP)
+check_function_exists(gettid HAVE_GETTID)
+check_function_exists(waitid HAVE_WAITID)
 check_function_exists(nice HAVE_NICE)
 check_function_exists(recvmmsg HAVE_RECVMMSG)
 check_function_exists(getpriority HAVE_GETPRIORITY)
+check_function_exists(setenv HAVE_SETENV)
+check_function_exists(strndup HAVE_STRNDUP)
 
 check_function_exists(sched_getscheduler HAVE_SCHED_GETSCHEDULER)
 check_function_exists(sched_setscheduler HAVE_SCHED_SETSCHEDULER)
@@ -624,10 +672,10 @@ set(LIBNETDATA_FILES
         src/libnetdata/log/journal.h
         src/libnetdata/log/log.c
         src/libnetdata/log/log.h
-        src/libnetdata/os.c
-        src/libnetdata/os.h
+        src/libnetdata/os/os.c
+        src/libnetdata/os/os.h
         src/libnetdata/simple_hashtable.h
-        src/libnetdata/byteorder.h
+        src/libnetdata/os/byteorder.h
         src/libnetdata/onewayalloc/onewayalloc.c
         src/libnetdata/onewayalloc/onewayalloc.h
         src/libnetdata/popen/popen.c
@@ -682,6 +730,34 @@ set(LIBNETDATA_FILES
         src/libnetdata/linked-lists.h
         src/libnetdata/storage-point.h
         src/libnetdata/bitmap64.h
+        src/libnetdata/os/waitid.c
+        src/libnetdata/os/waitid.h
+        src/libnetdata/os/gettid.c
+        src/libnetdata/os/gettid.h
+        src/libnetdata/os/adjtimex.c
+        src/libnetdata/os/adjtimex.h
+        src/libnetdata/os/setresuid.c
+        src/libnetdata/os/setresuid.h
+        src/libnetdata/os/setresgid.c
+        src/libnetdata/os/setresgid.h
+        src/libnetdata/os/getgrouplist.c
+        src/libnetdata/os/getgrouplist.h
+        src/libnetdata/os/get_pid_max.c
+        src/libnetdata/os/get_pid_max.h
+        src/libnetdata/os/os-freebsd-wrappers.c
+        src/libnetdata/os/os-freebsd-wrappers.h
+        src/libnetdata/os/os-macos-wrappers.c
+        src/libnetdata/os/os-macos-wrappers.h
+        src/libnetdata/os/get_system_cpus.c
+        src/libnetdata/os/get_system_cpus.h
+        src/libnetdata/os/tinysleep.c
+        src/libnetdata/os/tinysleep.h
+        src/libnetdata/os/uuid_generate.c
+        src/libnetdata/os/uuid_generate.h
+        src/libnetdata/os/setenv.c
+        src/libnetdata/os/setenv.h
+        src/libnetdata/os/strndup.c
+        src/libnetdata/os/strndup.h
 )
 
 if(ENABLE_PLUGIN_EBPF)
@@ -956,6 +1032,16 @@ else()
         )
 endif()
 
+set(INTERNAL_COLLECTORS_FILES
+        src/collectors/common-contexts/common-contexts.h
+        src/collectors/common-contexts/disk.io.h
+        src/collectors/common-contexts/system.io.h
+        src/collectors/common-contexts/system.ram.h
+        src/collectors/common-contexts/mem.swap.h
+        src/collectors/common-contexts/mem.pgfaults.h
+        src/collectors/common-contexts/mem.available.h
+)
+
 set(PLUGINSD_PLUGIN_FILES
         src/collectors/plugins.d/plugins_d.c
         src/collectors/plugins.d/plugins_d.h
@@ -1197,6 +1283,24 @@ set(FREEBSD_PLUGIN_FILES
         src/collectors/proc.plugin/zfs_common.h
 )
 
+set(WINDOWS_PLUGIN_FILES
+        src/collectors/windows.plugin/windows_plugin.c
+        src/collectors/windows.plugin/windows_plugin.h
+        src/collectors/windows.plugin/GetSystemUptime.c
+        src/collectors/windows.plugin/GetSystemRAM.c
+        src/collectors/windows.plugin/GetSystemCPU.c
+        src/collectors/windows.plugin/perflib.c
+        src/collectors/windows.plugin/perflib.h
+        src/collectors/windows.plugin/perflib-rrd.c
+        src/collectors/windows.plugin/perflib-rrd.h
+        src/collectors/windows.plugin/perflib-names.c
+        src/collectors/windows.plugin/perflib-dump.c
+        src/collectors/windows.plugin/perflib-storage.c
+        src/collectors/windows.plugin/perflib-processor.c
+        src/collectors/windows.plugin/perflib-network.c
+        src/collectors/windows.plugin/perflib-memory.c
+)
+
 set(PROC_PLUGIN_FILES
         src/collectors/proc.plugin/ipc.c
         src/collectors/proc.plugin/plugin_proc.c
@@ -1315,6 +1419,7 @@ if(LINUX)
                 ${PROC_PLUGIN_FILES}
                 ${TC_PLUGIN_FILES}
                 ${TIMEX_PLUGIN_FILES}
+                ${INTERNAL_COLLECTORS_FILES}
         )
 
         if(ENABLE_SENTRY)
@@ -1327,12 +1432,20 @@ elseif(MACOS)
                 src/daemon/static_threads_macos.c
                 ${MACOS_PLUGIN_FILES}
                 ${TIMEX_PLUGIN_FILES}
+                ${INTERNAL_COLLECTORS_FILES}
         )
 elseif(FREEBSD)
         list(APPEND NETDATA_FILES
                 src/daemon/static_threads_freebsd.c
                 ${FREEBSD_PLUGIN_FILES}
                 ${TIMEX_PLUGIN_FILES}
+                ${INTERNAL_COLLECTORS_FILES}
+        )
+elseif(WINDOWS)
+        list(APPEND NETDATA_FILES
+                src/daemon/static_threads_windows.c
+                ${WINDOWS_PLUGIN_FILES}
+                ${INTERNAL_COLLECTORS_FILES}
         )
 endif()
 
@@ -1518,6 +1631,7 @@ target_include_directories(libnetdata BEFORE PUBLIC ${CONFIG_H_DIR} ${CMAKE_SOUR
 target_link_libraries(libnetdata PUBLIC
         "$<$<NOT:$<BOOL:${HAVE_BUILTIN_ATOMICS}>>:atomic>"
         "$<$<OR:$<BOOL:${LINUX}>,$<BOOL:${FREEBSD}>>:pthread;rt>"
+        "$<$<BOOL:${WINDOWS}>:kernel32;advapi32;winmm;rpcrt4>"
         "$<$<BOOL:${LINK_LIBM}>:m>"
         "${SYSTEMD_LDFLAGS}")
 
@@ -1578,7 +1692,7 @@ if(LIBBROTLI_FOUND)
 endif()
 
 # uuid
-if(MACOS)
+if(MACOS OR WINDOWS)
         # UUID functionality is part of the system libraries here, so no extra
         # stuff needed.
 else()
@@ -1879,9 +1993,9 @@ endif()
 
 if(ENABLE_PLUGIN_CUPS)
     pkg_check_modules(CUPS libcups)
-    if(NOT CUPS_LIBRARIES)
+    if(NOT CUPS_FOUND)
         pkg_check_modules(CUPS cups)
-        if(NOT CUPS_LIBRARIES)
+        if(NOT CUPS_FOUND)
                 find_program(CUPS_CONFIG cups-config)
                 if(CUPS_CONFIG)
                         execute_process(COMMAND ${CUPS_CONFIG} --api-version OUTPUT_VARIABLE CUPS_API_VERSION OUTPUT_STRIP_TRAILING_WHITESPACE)
@@ -2914,10 +3028,12 @@ install(FILES
         COMPONENT netdata
         DESTINATION ${WEB_DEST}/.well-known/dnt)
 
-# v0 dashboard
-install(FILES
-        src/web/gui/v0/index.html
-        COMPONENT netdata
-        DESTINATION ${WEB_DEST}/v0)
+if(NOT WINDOWS)
+        # v0 dashboard
+        install(FILES
+                src/web/gui/v0/index.html
+                COMPONENT netdata
+                DESTINATION ${WEB_DEST}/v0)
+endif()
 
 include(Packaging)