summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
4 daysbelow: cgroupfs: Provide buffer for CgroupReader lifetimeHEADmainChris Down
Summary: This saves a lot of CPU when the cgroup hierarchy is dense. In some cases with large cgroup hierarchies I have seen up to 3-4% CPU win. Reviewed By: lnyng Differential Revision: D58238969 fbshipit-source-id: d7a5c2979de5df798854560277b8d9de9c0b4808
4 daysbelow: procfs: Extract internal buffer handling into a functionChris Down
Summary: This will also be used for CgroupReader in the next commit. Reviewed By: lnyng Differential Revision: D58238968 fbshipit-source-id: a8859cadcaa4d0547a67d07b5be0afdbc3c53a45
4 daysbelow: procfs: Use RefCell for internal file bufferChris Down
Summary: This avoids proliferating the need for a mutable reader everywhere. Performance looks on par with the non-RefCell version. Reviewed By: lnyng, brianc118 Differential Revision: D58280899 fbshipit-source-id: 59d83a6ef6e357ec3c985fd8b030c9aeed1521c3
6 daysbelow: read_all_pids: Use byte iteration for pid walkingChris Down
Summary: core::str::pattern::CharSearcher::next_match is very expensive (~2.5% CPU instructions per iteration). This can be made much more efficient by using byte iteration, and by avoiding having to check whether this is a pid dir twice. This is more efficient for a few reasons: 1. We do not require a check that the filename is valid UTF-8 2. We do not require allocations (as .to_string_lossy() may) 3. We now avoid doing the work twice Rudimentary testing with a small pid hierarchy shows around a 0.5% to 1% reduction in CPU instructions per iteration. Reviewed By: lnyng Differential Revision: D58193111 fbshipit-source-id: cba2b6f72995fe38e0ba4e055e884c069392c01e
6 daysbelow: ioctl: Ensure ioctl(SIOCETHTOOL) compat across libc implementationsChris Down
Summary: Different libc implementations (e.g., musl and glibc) define the `ioctl` request argument with different types (`c_int` for musl and `c_ulong` for glibc). In C, this difference is masked by typeless #defines, but in Rust, it causes a type mismatch that must be explicitly handled: error[E0308]: mismatched types --> below/ethtool/src/reader.rs:36:58 | 36 | let exit_code = unsafe { libc::ioctl(fd.as_raw_fd(), nix::libc::SIOCETHTOOL, &ifr) }; | ----------- ^^^^^^^^^^^^^^^^^^^^^^ expected `i32`, found `u64` | | | arguments to this function are incorrect To ensure portability and prevent type errors, we now convert `libc::SIOCETHTOOL` to the appropriate type. This conversion ensures that our code remains compatible with both musl and glibc. Reviewed By: dschatzberg Differential Revision: D58168265 fbshipit-source-id: d19789150d73dbe8fb0351cdfd4a02d79ea01400
7 daysbelow: uptime: Reduce syscall overhead with clock_gettime(CLOCK_BOOTTIME)Chris Down
Summary: Reading /proc/uptime is very expensive in terms of syscalls and procfs synchronisation (to be honest I was surprised at how high it is). By using clock_gettime(), we can directly access the system clock with a single syscall, which is significantly cheaper in terms of execution cost. This saves about 3-4% CPU(!) each iteration in my tests. Reviewed By: dschatzberg Differential Revision: D58083973 fbshipit-source-id: f97b70cc9bb788dc31672daca42d850291855277
7 daysbelow: pid_cgroup: Avoid intermediate vector allocChris Down
Summary: This is a small improvement, around 0.1% or so of CPU, but it's also just cleaner. Reviewed By: dschatzberg Differential Revision: D58084032 fbshipit-source-id: 298baf1e37402bc4ddedfb23d0384f40b0f49afc
7 daysbelow: stat: Avoid intermediate split vector allocationChris Down
Summary: /proc/pid/stat is one of the more complicated files to parse, and using the iterator directly saves about 1% CPU per iteration in my tests. Reviewed By: dschatzberg Differential Revision: D58083975 fbshipit-source-id: 649ae79b5893b5010b961fa5da8aa71f927705ba
7 daysbelow: procfs: Reinterpret internal buffer as &str without reallocChris Down
Summary: Almost all files go through this path, so a little gain here goes a long way. In terms of safety, none of the files we read using read_file_to_string from should have invalid UTF-8. The two that can (cmdline/exe) already have special handling outside of this. This saves about 1-2% of CPU per iteration. Reviewed By: dschatzberg Differential Revision: D58083974 fbshipit-source-id: 9a4fc1a7c0bd7b7bdfb442ef33cfefd21ed4c49d
7 daysbelow: procfs: Amortise file buffer allocationChris Down
Summary: procfs is one of the hottest parts of the below codebase, accounting for roughly a quarter of its overall work. It currently uses BufReader for reading the majority of files, but this is highly inefficient for small files: 1. BufReader allocates a buffer in memory, which is typically larger than the size of such small procfs files. For small files, this buffer allocation can be more costly in terms of time and memory than simply reading the file directly without buffering. 2. BufReader is designed to reduce the number of syscalls, but in this case we basically only do one (a single read()). We do the vast majority of reads synchronously (except for cmdline due to priority inversion avoidance). This allows us to use a shared buffer for all reads, avoiding paying the cost each time for buffer instantiation. In some cases where we don't use it as a string at all, it avoids any allocation at all. As an extra bonus, this even reduces the amount of code required. We do not use std::fs::read_to_string (or the Read trait) because that unfortunately does this, which is extremely unhelpfyl on procfs with no inode st_size: let size = file.metadata().map(|m| m.len() as usize).ok(); let mut string = String::new(); string.try_reserve_exact(size.unwrap_or(0))?; This saves about 2% on-CPU time per iteration in my rudimentary laptop tests with a few pids. With more pids, as it is in production, it will be even more. Reviewed By: brianc118 Differential Revision: D58083972 fbshipit-source-id: 7ae233e83232f866734230d6b4029cce71a501e8
7 daysbelow: pidwalk: Avoid statx() and use DT_* where possibleChris Down
Summary: procfs already gives us reliable information about whether this is a directory or not when internally calling readdir(), so use that instead of calling statx(). This saves about 1% on-CPU time per iteration in my tests. Reviewed By: dschatzberg Differential Revision: D58083971 fbshipit-source-id: ff0ad587c9092cf7e6e1b23b3b02f671566eebe9
7 daysbelow: cgroupfs: Parse KV format without vector allocChris Down
Summary: This is about a 0.5-1% CPU improvement per iteration. Reviewed By: dschatzberg Differential Revision: D58083969 fbshipit-source-id: b9f178273b46798da0276e39d839c51cb38a41aa
7 daysbelow: cgroupfs: Parse eq-based format without vector allocChris Down
Summary: This is about a 0.5-1% CPU improvement per iteration. Reviewed By: dschatzberg Differential Revision: D58083970 fbshipit-source-id: d368ed5b982aaef2c3ff83014f7be9ee0eb5a406
7 daysbelow: cmdline: Avoid perverse std::io::Read::read_to_end heuristicsChris Down
Summary: Reading a pid's command line is one of the most common operations we perform, and it shows up as a decent chunk of our on-CPU time for each collection interval. As a virtual filesystem, /proc does not set st_size on most files, and /proc/pid/cmdline is no exception: % stat --printf=%s /proc/self/cmdline 0 std::io::Read's `read_to_end` does not interact particularly well with this, and procfs in general. In particular, it does a whole bunch of probing, checks, heuristics, and reallocations that are unhelpful on procfs. We can drastically reduce this for the nominal case by instead reading directly into a &[u8] buffer. This saves about 0.5-1% on-CPU time per iteration in my tests. Reviewed By: dschatzberg Differential Revision: D58083967 fbshipit-source-id: 6be6588372a32305e62752ef63fccd79a6ba30d4
8 daysFix system model cpu irqLeon Yang (Containers)
Summary: Copy pasta from 4 years ago causing irq to always get the same value as iowait and no one realize. Reviewed By: dschatzberg Differential Revision: D58104586 fbshipit-source-id: ad5e9165150264ff1e634c9a4cc2709b1da3924b
8 daysksm viewJP Kobryn
Summary: adds ksm tab to system view Reviewed By: lnyng Differential Revision: D56961743 fbshipit-source-id: f4b13ccea93236cf49afced64ab270add2b6642e
2024-05-24ksm render model and configJP Kobryn
Summary: adds the necessary objects to be used within a view Reviewed By: lnyng Differential Revision: D56961811 fbshipit-source-id: 805eda9bb201847e6cdcc8c2efa13328e33f7c22
2024-05-24ksm reading capabilityJP Kobryn
Summary: adds reader for parsing the ksm files in `/sys/kernel/mm/ksm` Reviewed By: lnyng Differential Revision: D56961774 fbshipit-source-id: efc201d74b872c249d25dbe46bf6fd6762a9e10c
2024-05-20Update `libbpf-rs` & `libbpf-cargo` to `0.23.1`Daniel Mueller
Summary: Update the libbpf-rs and libbpf-cargo crates to version 0.23.1 to get the latest and greatest. Reviewed By: anakryiko Differential Revision: D57113437 fbshipit-source-id: 50597e92ad645abfa64b2fe16e491bdb6e2a6b17
2024-05-17set timezone in test before dumping timestampMichel Lind
Summary: The tests that check the serialized result fail when they contain timestamps when run on any system that is not set to Pacific time (e.g. Fedora build systems are set to UTC). Set the timezone we expect explicitly for the tests Reviewed By: davide125 Differential Revision: D57490840 fbshipit-source-id: c65f2ee919772776c3436cd614877db77635f58b
2024-05-15update authorsMichel Lind
Summary: - drop Daniel Xu, no longer sole author - s|Facebook|Meta Platforms, Inc. and affiliates|g' Reviewed By: danobi Differential Revision: D57342331 fbshipit-source-id: 4d23e2780385c46910d958c189e5dcc6653409b2
2024-05-14fix descriptionMichel Lind
Summary: This was copied from the `model` crate and needs to be edited to say `ethtool` instead. Reviewed By: danobi Differential Revision: D57342033 fbshipit-source-id: 86617700e51339f3919a9f110b7cc28b107717a1
2024-05-13Record memory.events.localLeon Yang (Containers)
Summary: As title Reviewed By: antonis-m Differential Revision: D57226252 fbshipit-source-id: 09d071360a886f56a75b27447d3b110aa27e56ab
2024-04-26Fix open source buildLeon Yang (Containers)
Summary: as title Reviewed By: brianc118 Differential Revision: D56634065 fbshipit-source-id: 8c46ca84b38aae8b7974fb76ae1e1c0f008e8230
2024-04-26Support summary view extra rowsLeon Yang (Containers)
Summary: This diff adds the feature to add extra rows to the bottom of the summary view. Each row has an optional title and each field has an optional alias. Invalid field IDs are skipped. Supported field IDs can be found in common_field_ids.rs or shown in the status bar at the bottom of below when a column is in focused. An example viewrc looks like: ``` [[view.summary_view_extra_rows]] title = "My Extra Row" [[view.summary_view_extra_rows.items]] alias = "Kernel" field_id = "system.kernel_version" [[view.summary_view_extra_rows.items]] field_id = "system.vm.oom_kill" [[view.summary_view_extra_rows.items]] alias = "Sys Slice CPU" field_id = "cgroup.path:/system.slice/.cpu.usage_pct" [[view.summary_view_extra_rows]] [[view.summary_view_extra_rows.items]] field_id = "system.stat.blocked_processes" [[view.summary_view_extra_rows.items]] field_id = "system.stat.running_processes" [[view.summary_view_extra_rows.items]] field_id = "invalid.field.id" [[view.summary_view_extra_rows.items]] field_id = "system.mem.swap_free" ``` Reviewed By: brianc118 Differential Revision: D56504773 fbshipit-source-id: 22874aec922e8a4f3e6699094ec63dc6c66ad5d1
2024-04-26Make viewrc part of view stateLeon Yang (Containers)
Summary: As title. This makes it easy to access viewrc from any view. Reviewed By: brianc118 Differential Revision: D56504776 fbshipit-source-id: 69ed566e17d8dbc04f2678ca4c07bf5cd0a28ee6
2024-04-26Rename core_view to system_viewLeon Yang (Containers)
Summary: Similar to previous diff. Update core_view to system_view since it contains basic system info and has shortcut 's' which comes from "system". Reviewed By: brianc118 Differential Revision: D56504775 fbshipit-source-id: 718f5bafbcddb22af8012d5e51528a616f1efe46
2024-04-26Rename system_view to summary_viewLeon Yang (Containers)
Summary: There is a misnomer of system_view since it can contain basic system info as well as cgroup, network, gpu metrics. It's more of a summary view. More importantly, core_view should actually be called system_view, which has a short cut of 's' that comes from "system". Reviewed By: brianc118 Differential Revision: D56504774 fbshipit-source-id: 13ac89113d26a2a7ea34c04c4fcd9c5c88c07497
2024-04-25Add sub-module for reading `tc` stats (#8210)mmynk
Summary: - Adds the sub-crate for reading `tc` stats - Uses `netlink-packet-route` library which reads `qdisc`s via rtnetlink Result: ```sh $ below dump tc -b '5s ago' Datetime Interface Kind Queue Length Bps Pps Bytes Packets Backlog Drops Requeues Overlimits MaxPacket EcnMark NewFlowsLen OldFlowsLen CeMark DropOverlimit NewFlowCount MemoryUsage DropOvermemory Target Limit Interval Ecn Quantum CeThreshold DropBatchSize MemoryLimit Flows Timestamp 2024-01-30 20:24:31 lo noqueue 0 0.0 B/s 0/s 0.0 B/s 0/s 0/s 0/s 0/s 0/s ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1706646271 2024-01-30 20:24:31 ens5 mq 0 0.0 B/s 0/s 167 B/s 2/s 0/s 0/s 0/s 0/s ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1706646271 2024-01-30 20:24:31 ens5 fq_codel 0 0.0 B/s 0/s 0.0 B/s 0/s 0/s 0/s 0/s 0/s 110 0 0 0 0 0/s 0/s 0/s 0/s 4999 10240 99999 1 1514 0 64 33554432 0/s 1706646271 2024-01-30 20:24:31 ens5 fq_codel 0 0.0 B/s 0/s 167 B/s 2/s 0/s 0/s 0/s 0/s 182 0 0 0 0 0/s 0/s 0/s 0/s 4999 10240 99999 1 1514 0 64 33554432 0/s 1706646271 ``` Pull Request resolved: https://github.com/facebookincubator/below/pull/8210 Reviewed By: lnyng Differential Revision: D56263764 Pulled By: brianc118 fbshipit-source-id: bc225ddd29ddd81fa34d0f0f106177f4a1e2d4b9
2024-04-25update 'clap-4' crate to 4.5.4Mark Feng
Summary: version bump follow https://www.internalfb.com/intern/wiki/Rust/Third_Party_Libraries/Adding_or_Updating_Libraries/ Also updated `clap_complete` together allow-large-files Reviewed By: capickett Differential Revision: D56519704 fbshipit-source-id: deb160a682bd156c69e3f0509a54b1f6a5a85f27
2024-04-24Add local mm_rss_stat definitionAndrii Nakryiko
Summary: Add local definition of struct mm_rss_stat available pre-6.2. Once we switch to 6.4-based vmlinux.h in fbcode this stop compiling because `struct mm_rss_stat` is not available anymore. Reviewed By: lnyng Differential Revision: D56500799 fbshipit-source-id: e5371bae807615011d6f615ea78ea5e7bd83b9ab
2024-04-24Update tokio: 1.36.0 -> 1.37.0Henry Swanson
Summary: Followed the wiki page here: https://www.internalfb.com/intern/wiki/Rust/Third_Party_Libraries/Adding_or_Updating_Libraries/ Reviewed By: capickett Differential Revision: D56484070 fbshipit-source-id: e1ed52d58f7db8ad32ce67b67df2e83a567db123
2024-04-11QueriableContainerLeon Yang (Containers)
Summary: Some rust type gymnastics to unify the FieldId type for Vec, BTreeMap, and CgroupModel. Reduce some code. We can then add feature to this trait to get X field of an item from a container whose Y field equals Z, is min/max among other items, etc. Reviewed By: brianc118 Differential Revision: D55968833 fbshipit-source-id: bab97d65bb72234fd67b723f23043327bac87234
2024-04-11[below[ Add below_derive::queriable_derivesLeon Yang (Containers)
Summary: Added a short hand proc macro to add #[derive] for necessary trait derives for Queriable structs. Unify the model's derives. Reviewed By: boyuni Differential Revision: D55968832 fbshipit-source-id: 0f27566687584b53d4d54b02102eae0038906481
2024-04-08Fix integration testLeon Yang (Containers)
Summary: The host may not have io controller enabled and thus may not have IO stats. Let's use memory pressure since exists even without any controller. Reviewed By: brianc118 Differential Revision: D55880562 fbshipit-source-id: b416a90eb5324f2f526d26f652cd4f1f6c40cf96
2024-04-05third-party/rust: Bump bytes 1.1 -> 1.6.0Astrid Yu
Summary: As it says on the tin. There have been many new features added since the old version. [Staring at the changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md), I don't believe they added breaking changes. Changes to other libraries: - *common/rust/folly/iobuf* - the panic message changed so I changed the should_panic rule to match Reviewed By: diliop Differential Revision: D55807882 fbshipit-source-id: 8d70a8bae57509fdca82ab18770275111a1f1902
2024-04-05Bump to 0.8.1 (#8228)v0.8.1Brian Chen
Summary: Pull Request resolved: https://github.com/facebookincubator/below/pull/8228 0.8.0 depended on an unreleased version of libbpf-rs preventing us from releasing to crates.io. Reviewed By: antonis-m Differential Revision: D55769523 fbshipit-source-id: 796e5e9412717fad0293338ceacc76f96c2933b2
2024-04-04Add LICENSE and README for ethtool/resctrlfs crates and update crates.io ↵Brian Chen
publish script to publish ethtool/resctrlfs (#8227) Summary: Pull Request resolved: https://github.com/facebookincubator/below/pull/8227 Reviewed By: antonis-m, lnyng Differential Revision: D55695037 Pulled By: brianc118 fbshipit-source-id: 56c72ff057557c8dbded8f7051d37c41384b35fc
2024-04-02Update `libbpf-rs` & `libbpf-cargo` to `0.23.0`Daniel Mueller
Summary: Update the `libbpf-rs` and `libbpf-cargo` crates to version `0.23.0` to get the latest and greatest. Reviewed By: brianc118 Differential Revision: D55609212 fbshipit-source-id: 59a8221387a07e337be9c2b9d79919de57faad70
2024-03-29Bump to 0.8.0 (#8226)v0.8.0Brian Chen
Summary: Notable changes since 0.7.1: * Support for ethtool stats. * New cgroup stats including per-cgroup zswap memory usage, memory.min and kernel field from memory.stat. * /sys/fs/resctrl data collection. * /proc/slabinfo collection and display. * viewrc support for `cgroup_name_width` to configure cgroup view name column width. Fixes: * Exitstat bpf program is fixed. Pull Request resolved: https://github.com/facebookincubator/below/pull/8226 Reviewed By: antonis-m Differential Revision: D55491440 Pulled By: brianc118 fbshipit-source-id: 989004d5e6a3618e598a7efc8e59643dba19cfd3
2024-03-28Update Cargo.lock (#8225)Brian Chen
Summary: Pull Request resolved: https://github.com/facebookincubator/below/pull/8225 Reviewed By: dschatzberg Differential Revision: D55492007 Pulled By: brianc118 fbshipit-source-id: 696ffd2d6fd2f862f8dcb9c8c7808b92cb67ff83
2024-03-27rust/third-party: update to futures-0.3.30J.T. Conklin
Summary: Changelog since last import: ``` # 0.3.30 - 2023-12-24 * Add `{BiLock,SplitStream,SplitSink,ReadHalf,WriteHalf}::is_pair_of` (#2797) * Fix panic in `FuturesUnordered::clear` (#2809) * Fix panic in `AsyncBufReadExt::fill_buf` (#2801, #2812) * Improve support for targets without atomic CAS (#2811) * Remove build scripts (#2811) # 0.3.29 - 2023-10-26 * Add `TryStreamExt::try_ready_chunks` (#2757) * Add `TryStreamExt::{try_all,try_any}` (#2783) * Add `UnboundedSender::{len,is_empty}` (#2750) * Fix `Sync` impl of `FuturesUnordered` (#2788) * Fix infinite loop caused by invalid UTF-8 bytes (#2785) * Fix build error with -Z minimal-versions (#2761) ``` I'm updating this as I need `TryStreamExt::try_all()` for a Sandcastle Worker change. Reviewed By: krallin Differential Revision: D55318002 fbshipit-source-id: 0f64b13570cd1644000c3639bf16c0746266cb8e
2024-03-18Upgrade tokioStiopa Koltsov
Summary: Need to add some third-party, which is pulling newer tokio. Land upgrade separately for visibility and for easier bisect and revert. Reviewed By: Imxset21, JakobDegen Differential Revision: D55030643 fbshipit-source-id: 47b509ce2103d1dd64c66aa250133eb25758a750
2024-03-12Upgrade clap 4.4.12 -> 4.5.2Zoltán Nagy
Summary: Also `clap_complete` Reviewed By: zertosh Differential Revision: D54802104 fbshipit-source-id: 9c00f60eb04fe82aed90d24c8cf79c42528f2f55
2024-03-11Re-enable tests (#8224)Brian Chen
Summary: Pull Request resolved: https://github.com/facebookincubator/below/pull/8224 Reviewed By: lnyng Differential Revision: D54754136 fbshipit-source-id: 64eb834de2b8a05e3a496aea226dfccbc9f98611
2024-03-11Update README to point to correct Grafana directoryBrian Chen
Reviewed By: lnyng Differential Revision: D54754806 fbshipit-source-id: 248d2a796e980333ec4a1891346ba19fb940fab3
2024-03-11Remove URL popupBrian Chen
Summary: URL popup is no longer used internally. Let's remove it. Reviewed By: dschatzberg Differential Revision: D54697749 fbshipit-source-id: 479ae6915267e6f8cb1984e1b3a79ee03a19bae6
2024-03-07Add exitstat vmtestBrian Chen
Summary: Add exitstat vmtest to make sure we keep the bpf program working on different kernel versions. Reviewed By: dschatzberg Differential Revision: D54585856 fbshipit-source-id: 1b63bb2ad88a1ddefe2697a3d30377696707f80b
2024-03-06Fix open source buildBrian Chen
Reviewed By: boyuni Differential Revision: D54599673 fbshipit-source-id: c439c381e084b2ae3ce310c7a33ce098eefa8f63
2024-03-06fix exitstat bpf progBrian Chen
Summary: Prod below is failing to load the exitstat bpf prog. This should fix it. Reviewed By: lnyng Differential Revision: D54558876 fbshipit-source-id: ef61aff5614d5999b8bbd2d0e06b3a61e154c7bf
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-12 00:38:02 +0000