Which of the following commands will display CPU load data along with information concerning users who are currently logged into the system?

2.1. The purpose of TuneD

TuneD is a service that monitors your system and optimizes the performance under certain workloads. The core of TuneD are profiles, which tune your system for different use cases.

TuneD is distributed with a number of predefined profiles for use cases such as:

• High throughput
• Low latency
• Saving power

It is possible to modify the rules defined for each profile and customize how to tune a particular device. When you switch to another profile or deactivate TuneD, all changes made to the system settings by the previous profile revert back to their original state.

You can also configure TuneD to react to changes in device usage and adjusts settings to improve performance of active devices and reduce power consumption of inactive devices.

2.2. TuneD profiles

A detailed analysis of a system can be very time-consuming. TuneD provides a number of predefined profiles for typical use cases. You can also create, modify, and delete profiles.

The profiles provided with TuneD are divided into the following categories:

• Power-saving profiles
• Performance-boosting profiles

The performance-boosting profiles include profiles that focus on the following aspects:

• Low latency for storage and network
• High throughput for storage and network
• Virtual machine performance
• Virtualization host performance

Syntax of profile configuration

The tuned.conf file can contain one [main] section and other sections for configuring plug-in instances. However, all sections are optional.

Lines starting with the hash sign (#) are comments.

Additional resources

• tuned.conf(5) man page.

2.3. The default TuneD profile

During the installation, the best profile for your system is selected automatically. Currently, the default profile is selected according to the following customizable rules:

Additional resources

• tuned.conf(5) man page.

2.4. Merged TuneD profiles

As an experimental feature, it is possible to select more profiles at once. TuneD will try to merge them during the load.

If there are conflicts, the settings from the last specified profile takes precedence.

Example 2.1. Low power consumption in a virtual guest

The following example optimizes the system to run in a virtual machine for the best performance and concurrently tunes it for low power consumption, while the low power consumption is the priority:

Merging is done automatically without checking whether the resulting combination of parameters makes sense. Consequently, the feature might tune some parameters the opposite way, which might be counterproductive: for example, setting the disk for high throughput by using the throughput-performance profile and concurrently setting the disk spindown to the low value by the spindown-disk profile.

Additional resources

*tuned-adm man page. * tuned.conf(5) man page.

2.5. The location of TuneD profiles

TuneD stores profiles in the following directories:

/usr/lib/tuned/ Distribution-specific profiles are stored in the directory. Each profile has its own directory. The profile consists of the main configuration file called tuned.conf, and optionally other files, for example helper scripts. /etc/tuned/ If you need to customize a profile, copy the profile directory into the directory, which is used for custom profiles. If there are two profiles of the same name, the custom profile located in /etc/tuned/ is used.

Additional resources

• tuned.conf(5) man page.

2.6. TuneD profiles distributed with RHEL

The following is a list of profiles that are installed with TuneD on Red Hat Enterprise Linux.

There might be more product-specific or third-party TuneD profiles available. Such profiles are usually provided by separate RPM packages.

balanced

The default power-saving profile. It is intended to be a compromise between performance and power consumption. It uses auto-scaling and auto-tuning whenever possible. The only drawback is the increased latency. In the current TuneD release, it enables the CPU, disk, audio, and video plugins, and activates the conservative CPU governor. The radeon_powersave option uses the dpm-balanced value if it is supported, otherwise it is set to auto.

It changes the energy_performance_preference attribute to the normal energy setting. It also changes the scaling_governor policy attribute to either the conservative or powersave CPU governor.

A profile for maximum power saving performance. It can throttle the performance in order to minimize the actual power consumption. In the current TuneD release it enables USB autosuspend, WiFi power saving, and Aggressive Link Power Management (ALPM) power savings for SATA host adapters. It also schedules multi-core power savings for systems with a low wakeup rate and activates the ondemand governor. It enables AC97 audio power saving or, depending on your system, HDA-Intel power savings with a 10 seconds timeout. If your system contains a supported Radeon graphics card with enabled KMS, the profile configures it to automatic power saving. On ASUS Eee PCs, a dynamic Super Hybrid Engine is enabled.

It changes the energy_performance_preference attribute to the powersave or power energy setting. It also changes the scaling_governor policy attribute to either the ondemand or powersave CPU governor.

In certain cases, the balanced profile is more efficient compared to the powersave profile.

Consider there is a defined amount of work that needs to be done, for example a video file that needs to be transcoded. Your machine might consume less energy if the transcoding is done on the full power, because the task is finished quickly, the machine starts to idle, and it can automatically step-down to very efficient power save modes. On the other hand, if you transcode the file with a throttled machine, the machine consumes less power during the transcoding, but the process takes longer and the overall consumed energy can be higher.

That is why the balanced profile can be generally a better option.

A server profile optimized for high throughput. It disables power savings mechanisms and enables sysctl settings that improve the throughput performance of the disk and network IO. CPU governor is set to performance.

It changes the energy_performance_preference and scaling_governor attribute to the performance profile.

A server profile optimized for low latency. It disables power savings mechanisms and enables sysctl settings that improve latency. CPU governor is set to performance and the CPU is locked to the low C states (by PM QoS).

It changes the energy_performance_preference and scaling_governor attribute to the performance profile.

A profile for low latency network tuning. It is based on the latency-performance profile. It additionally disables transparent huge pages and NUMA balancing, and tunes several other network-related sysctl parameters.

It inherits the latency-performance profile which changes the energy_performance_preference and scaling_governor attribute to the performance profile.

A profile for throughput network tuning. It is based on the throughput-performance profile. It additionally increases kernel network buffers.

It inherits either the latency-performance or throughput-performance profile, and changes the energy_performance_preference and scaling_governor attribute to the performance profile.

A profile designed for Red Hat Enterprise Linux 8 virtual machines and VMWare guests based on the throughput-performance profile that, among other tasks, decreases virtual memory swappiness and increases disk readahead values. It does not disable disk barriers.

It inherits the throughput-performance profile and changes the energy_performance_preference and scaling_governor attribute to the performance profile.

A profile designed for virtual hosts based on the throughput-performance profile that, among other tasks, decreases virtual memory swappiness, increases disk readahead values, and enables a more aggressive value of dirty pages writeback.

It inherits the throughput-performance profile and changes the energy_performance_preference and scaling_governor attribute to the performance profile.

A profile that tunes down I/O activity to the serial console by reducing the printk value. This should make the serial console more responsive. This profile is intended to be used as an overlay on other profiles. For example:

A profile optimized for systems with user-defined Intel Speed Select Technology configurations. This profile is intended to be used as an overlay on other profiles. For example:

2.7. TuneD cpu-partitioning profile

For tuning Red Hat Enterprise Linux 8 for latency-sensitive workloads, Red Hat recommends to use the cpu-partitioning TuneD profile.

Prior to Red Hat Enterprise Linux 8, the low-latency Red Hat documentation described the numerous low-level steps needed to achieve low-latency tuning. In Red Hat Enterprise Linux 8, you can perform low-latency tuning more efficiently by using the cpu-partitioning TuneD profile. This profile is easily customizable according to the requirements for individual low-latency applications.

The following figure is an example to demonstrate how to use the cpu-partitioning profile. This example uses the CPU and node layout.

Figure 2.1. Figure cpu-partitioning

You can configure the cpu-partitioning profile in the /etc/tuned/cpu-partitioning-variables.conf file using the following configuration options:

Isolated CPUs with load balancing

In the cpu-partitioning figure, the blocks numbered from 4 to 23, are the default isolated CPUs. The kernel scheduler’s process load balancing is enabled on these CPUs. It is designed for low-latency processes with multiple threads that need the kernel scheduler load balancing.

You can configure the cpu-partitioning profile in the /etc/tuned/cpu-partitioning-variables.conf file using the isolated_cores=cpu-list option, which lists CPUs to isolate that will use the kernel scheduler load balancing.

The list of isolated CPUs is comma-separated or you can specify a range using a dash, such as 3-5. This option is mandatory. Any CPU missing from this list is automatically considered a housekeeping CPU.

In the cpu-partitioning figure, the blocks numbered 2 and 3, are the isolated CPUs that do not provide any additional kernel scheduler process load balancing.

You can configure the cpu-partitioning profile in the /etc/tuned/cpu-partitioning-variables.conf file using the no_balance_cores=cpu-list option, which lists CPUs to isolate that will not use the kernel scheduler load balancing.

Specifying the no_balance_cores option is optional, however any CPUs in this list must be a subset of the CPUs listed in the isolated_cores list.

Application threads using these CPUs need to be pinned individually to each CPU.

Additional resources

• tuned-profiles-cpu-partitioning(7) man page

2.8. Using the TuneD cpu-partitioning profile for low-latency tuning

This procedure describes how to tune a system for low-latency using the TuneD’s cpu-partitioning profile. It uses the example of a low-latency application that can use cpu-partitioning and the CPU layout as mentioned in the cpu-partitioning figure.

The application in this case uses:

• One dedicated reader thread that reads data from the network will be pinned to CPU 2.
• A large number of threads that process this network data will be pinned to CPUs 4-23.
• A dedicated writer thread that writes the processed data to the network will be pinned to CPU 3.

Prerequisites

• You have installed the cpu-partitioning TuneD profile by using the yum install tuned-profiles-cpu-partitioning command as root.

Procedure

1. Edit /etc/tuned/cpu-partitioning-variables.conf file and add the following information:

   # Isolated CPUs with the kernel’s scheduler load balancing: isolated_cores=2-23 # Isolated CPUs without the kernel’s scheduler load balancing: no_balance_cores=2,3

2. Set the cpu-partitioning TuneD profile:

   # tuned-adm profile cpu-partitioning

3. Reboot

   After rebooting, the system is tuned for low-latency, according to the isolation in the cpu-partitioning figure. The application can use taskset to pin the reader and writer threads to CPUs 2 and 3, and the remaining application threads on CPUs 4-23.

Additional resources

• tuned-profiles-cpu-partitioning(7) man page

2.9. Customizing the cpu-partitioning TuneD profile

You can extend the TuneD profile to make additional tuning changes.

For example, the cpu-partitioning profile sets the CPUs to use cstate=1. In order to use the cpu-partitioning profile but to additionally change the CPU cstate from cstate1 to cstate0, the following procedure describes a new TuneD profile named my_profile, which inherits the cpu-partitioning profile and then sets C state-0.

Procedure

1. Create the /etc/tuned/my_profile directory:

   # mkdir /etc/tuned/my_profile

2. Create a tuned.conf file in this directory, and add the following content:

   # vi /etc/tuned/my_profile/tuned.conf [main] summary=Customized tuning on top of cpu-partitioning include=cpu-partitioning [cpu] force_latency=cstate.id:0|1

3. Use the new profile:

   # tuned-adm profile my_profile

In the shared example, a reboot is not required. However, if the changes in the my_profile profile require a reboot to take effect, then reboot your machine.

Additional resources

• tuned-profiles-cpu-partitioning(7) man page

2.10. Real-time TuneD profiles distributed with RHEL

Real-time profiles are intended for systems running the real-time kernel. Without a special kernel build, they do not configure the system to be real-time. On RHEL, the profiles are available from additional repositories.

The following real-time profiles are available:

realtime

Use on bare-metal real-time systems.

Provided by the tuned-profiles-realtime package, which is available from the RT or NFV repositories.

Use in a virtualization host configured for real-time.

Provided by the tuned-profiles-nfv-host package, which is available from the NFV repository.

Use in a virtualization guest configured for real-time.

Provided by the tuned-profiles-nfv-guest package, which is available from the NFV repository.

2.11. Static and dynamic tuning in TuneD

This section explains the difference between the two categories of system tuning that TuneD applies: static and dynamic.

Static tuning Mainly consists of the application of predefined sysctl and sysfs settings and one-shot activation of several configuration tools such as ethtool. Dynamic tuning

Watches how various system components are used throughout the uptime of your system. TuneD adjusts system settings dynamically based on that monitoring information.

For example, the hard drive is used heavily during startup and login, but is barely used later when the user might mainly work with applications such as web browsers or email clients. Similarly, the CPU and network devices are used differently at different times. TuneD monitors the activity of these components and reacts to the changes in their use.

By default, dynamic tuning is disabled. To enable it, edit the /etc/tuned/tuned-main.conf file and change the dynamic_tuning option to 1. TuneD then periodically analyzes system statistics and uses them to update your system tuning settings. To configure the time interval in seconds between these updates, use the update_interval option.

Currently implemented dynamic tuning algorithms try to balance the performance and powersave, and are therefore disabled in the performance profiles. Dynamic tuning for individual plug-ins can be enabled or disabled in the TuneD profiles.

Example 2.2. Static and dynamic tuning on a workstation

On a typical office workstation, the Ethernet network interface is inactive most of the time. Only a few emails go in and out or some web pages might be loaded.

For those kinds of loads, the network interface does not have to run at full speed all the time, as it does by default. TuneD has a monitoring and tuning plug-in for network devices that can detect this low activity and then automatically lower the speed of that interface, typically resulting in a lower power usage.

If the activity on the interface increases for a longer period of time, for example because a DVD image is being downloaded or an email with a large attachment is opened, TuneD detects this and sets the interface speed to maximum to offer the best performance while the activity level is high.

This principle is used for other plug-ins for CPU and disks as well.

2.12. TuneD no-daemon mode

You can run TuneD in no-daemon mode, which does not require any resident memory. In this mode, TuneD applies the settings and exits.

By default, no-daemon mode is disabled because a lot of TuneD functionality is missing in this mode, including:

• D-Bus support
• Hot-plug support
• Rollback support for settings

To enable no-daemon mode, include the following line in the /etc/tuned/tuned-main.conf file:

2.13. Installing and enabling TuneD

This procedure installs and enables the TuneD application, installs TuneD profiles, and presets a default TuneD profile for your system.

Procedure

1. Install the tuned package:

   # yum install tuned

2. Enable and start the tuned service:

   # systemctl enable --now tuned

3. Optionally, install TuneD profiles for real-time systems:

   # yum install tuned-profiles-realtime tuned-profiles-nfv

4. Verify that a TuneD profile is active and applied:

   $ tuned-adm active Current active profile: balanced$ tuned-adm verify Verfication succeeded, current system settings match the preset profile. See tuned log file ('/var/log/tuned/tuned.log') for details.

2.14. Listing available TuneD profiles

This procedure lists all TuneD profiles that are currently available on your system.

Procedure

• To list all available TuneD profiles on your system, use:

  $ tuned-adm list Available profiles: - accelerator-performance - Throughput performance based tuning with disabled higher latency STOP states - balanced - General non-specialized tuned profile - desktop - Optimize for the desktop use-case - latency-performance - Optimize for deterministic performance at the cost of increased power consumption - network-latency - Optimize for deterministic performance at the cost of increased power consumption, focused on low latency network performance - network-throughput - Optimize for streaming network throughput, generally only necessary on older CPUs or 40G+ networks - powersave - Optimize for low power consumption - throughput-performance - Broadly applicable tuning that provides excellent performance across a variety of common server workloads - virtual-guest - Optimize for running inside a virtual guest - virtual-host - Optimize for running KVM guests Current active profile: balanced

• To display only the currently active profile, use:

  $ tuned-adm active Current active profile: balanced

Additional resources

• tuned-adm(8) man page.

2.15. Setting a TuneD profile

This procedure activates a selected TuneD profile on your system.

Procedure

1. Optionally, you can let TuneD recommend the most suitable profile for your system:

   # tuned-adm recommend balanced

2. Activate a profile:

   # tuned-adm profile selected-profile

   Alternatively, you can activate a combination of multiple profiles:

   # tuned-adm profile profile1 profile2

   Example 2.3. A virtual machine optimized for low power consumption

   The following example optimizes the system to run in a virtual machine with the best performance and concurrently tunes it for low power consumption, while the low power consumption is the priority:

   # tuned-adm profile virtual-guest powersave

3. View the current active TuneD profile on your system:

   # tuned-adm active Current active profile: selected-profile

4. Reboot the system:

   # reboot

Verification steps

• Verify that the TuneD profile is active and applied:

  $ tuned-adm verify Verfication succeeded, current system settings match the preset profile. See tuned log file ('/var/log/tuned/tuned.log') for details.

Additional resources

• tuned-adm(8) man page

2.16. Disabling TuneD

This procedure disables TuneD and resets all affected system settings to their original state before TuneD modified them.

Procedure

• To disable all tunings temporarily:

  # tuned-adm off

  The tunings are applied again after the tuned service restarts.

• Alternatively, to stop and disable the tuned service permanently:

  # systemctl disable --now tuned

Additional resources

• tuned-adm(8) man page

3.1. TuneD profiles

A detailed analysis of a system can be very time-consuming. TuneD provides a number of predefined profiles for typical use cases. You can also create, modify, and delete profiles.

The profiles provided with TuneD are divided into the following categories:

• Power-saving profiles
• Performance-boosting profiles

The performance-boosting profiles include profiles that focus on the following aspects:

• Low latency for storage and network
• High throughput for storage and network
• Virtual machine performance
• Virtualization host performance

Syntax of profile configuration

The tuned.conf file can contain one [main] section and other sections for configuring plug-in instances. However, all sections are optional.

Lines starting with the hash sign (#) are comments.

Additional resources

• tuned.conf(5) man page.

3.2. The default TuneD profile

During the installation, the best profile for your system is selected automatically. Currently, the default profile is selected according to the following customizable rules:

Additional resources

• tuned.conf(5) man page.

3.3. Merged TuneD profiles

As an experimental feature, it is possible to select more profiles at once. TuneD will try to merge them during the load.

If there are conflicts, the settings from the last specified profile takes precedence.

Example 3.1. Low power consumption in a virtual guest

The following example optimizes the system to run in a virtual machine for the best performance and concurrently tunes it for low power consumption, while the low power consumption is the priority:

Merging is done automatically without checking whether the resulting combination of parameters makes sense. Consequently, the feature might tune some parameters the opposite way, which might be counterproductive: for example, setting the disk for high throughput by using the throughput-performance profile and concurrently setting the disk spindown to the low value by the spindown-disk profile.

Additional resources

*tuned-adm man page. * tuned.conf(5) man page.

3.4. The location of TuneD profiles

TuneD stores profiles in the following directories:

/usr/lib/tuned/ Distribution-specific profiles are stored in the directory. Each profile has its own directory. The profile consists of the main configuration file called tuned.conf, and optionally other files, for example helper scripts. /etc/tuned/ If you need to customize a profile, copy the profile directory into the directory, which is used for custom profiles. If there are two profiles of the same name, the custom profile located in /etc/tuned/ is used.

Additional resources

• tuned.conf(5) man page.

3.5. Inheritance between TuneD profiles

TuneD profiles can be based on other profiles and modify only certain aspects of their parent profile.

The [main] section of TuneD profiles recognizes the include option:

All settings from the parent profile are loaded in this child profile. In the following sections, the child profile can override certain settings inherited from the parent profile or add new settings not present in the parent profile.

You can create your own child profile in the /etc/tuned/ directory based on a pre-installed profile in /usr/lib/tuned/ with only some parameters adjusted.

If the parent profile is updated, such as after a TuneD upgrade, the changes are reflected in the child profile.

Example 3.2. A power-saving profile based on balanced

The following is an example of a custom profile that extends the balanced profile and sets Aggressive Link Power Management (ALPM) for all devices to the maximum powersaving.

Additional resources

• tuned.conf(5) man page

3.6. Static and dynamic tuning in TuneD

This section explains the difference between the two categories of system tuning that TuneD applies: static and dynamic.

Static tuning Mainly consists of the application of predefined sysctl and sysfs settings and one-shot activation of several configuration tools such as ethtool. Dynamic tuning

Watches how various system components are used throughout the uptime of your system. TuneD adjusts system settings dynamically based on that monitoring information.

For example, the hard drive is used heavily during startup and login, but is barely used later when the user might mainly work with applications such as web browsers or email clients. Similarly, the CPU and network devices are used differently at different times. TuneD monitors the activity of these components and reacts to the changes in their use.

By default, dynamic tuning is disabled. To enable it, edit the /etc/tuned/tuned-main.conf file and change the dynamic_tuning option to 1. TuneD then periodically analyzes system statistics and uses them to update your system tuning settings. To configure the time interval in seconds between these updates, use the update_interval option.

Currently implemented dynamic tuning algorithms try to balance the performance and powersave, and are therefore disabled in the performance profiles. Dynamic tuning for individual plug-ins can be enabled or disabled in the TuneD profiles.

Example 3.3. Static and dynamic tuning on a workstation

On a typical office workstation, the Ethernet network interface is inactive most of the time. Only a few emails go in and out or some web pages might be loaded.

For those kinds of loads, the network interface does not have to run at full speed all the time, as it does by default. TuneD has a monitoring and tuning plug-in for network devices that can detect this low activity and then automatically lower the speed of that interface, typically resulting in a lower power usage.

If the activity on the interface increases for a longer period of time, for example because a DVD image is being downloaded or an email with a large attachment is opened, TuneD detects this and sets the interface speed to maximum to offer the best performance while the activity level is high.

This principle is used for other plug-ins for CPU and disks as well.

3.7. TuneD plug-ins

Plug-ins are modules in TuneD profiles that TuneD uses to monitor or optimize different devices on the system.

TuneD uses two types of plug-ins:

Monitoring plug-ins

Monitoring plug-ins are used to get information from a running system. The output of the monitoring plug-ins can be used by tuning plug-ins for dynamic tuning.

Monitoring plug-ins are automatically instantiated whenever their metrics are needed by any of the enabled tuning plug-ins. If two tuning plug-ins require the same data, only one instance of the monitoring plug-in is created and the data is shared.

Syntax for plug-ins in TuneD profiles

Sections describing plug-in instances are formatted in the following way:

NAME is the name of the plug-in instance as it is used in the logs. It can be an arbitrary string. TYPE is the type of the tuning plug-in. DEVICES

is the list of devices that this plug-in instance handles.

The devices line can contain a list, a wildcard (*), and negation (!). If there is no devices line, all devices present or later attached on the system of the TYPE are handled by the plug-in instance. This is same as using the devices=* option.

Example 3.4. Matching block devices with a plug-in

The following example matches all block devices starting with sd, such as sda or sdb, and does not disable barriers on them:

The following example matches all block devices except sda1 and sda2:

If no instance of a plug-in is specified, the plug-in is not enabled.

If the plug-in supports more options, they can be also specified in the plug-in section. If the option is not specified and it was not previously specified in the included plug-in, the default value is used.

Short plug-in syntax

If you do not need custom names for the plug-in instance and there is only one definition of the instance in your configuration file, TuneD supports the following short syntax:

In this case, it is possible to omit the type line. The instance is then referred to with a name, same as the type. The previous example could be then rewritten into:

Example 3.5. Matching block devices using the short syntax

[disk] devices=sdb* disable_barriers=false

Conflicting plug-in definitions in a profile

If the same section is specified more than once using the include option, the settings are merged. If they cannot be merged due to a conflict, the last conflicting definition overrides the previous settings. If you do not know what was previously defined, you can use the replace Boolean option and set it to true. This causes all the previous definitions with the same name to be overwritten and the merge does not happen.

You can also disable the plug-in by specifying the enabled=false option. This has the same effect as if the instance was never defined. Disabling the plug-in is useful if you are redefining the previous definition from the include option and do not want the plug-in to be active in your custom profile.

NOTE

TuneD includes the ability to run any shell command as part of enabling or disabling a tuning profile. This enables you to extend TuneD profiles with functionality that has not been integrated into TuneD yet.

You can specify arbitrary shell commands using the script plug-in.

Additional resources

• tuned.conf(5) man page

3.8. Available TuneD plug-ins

This section lists all monitoring and tuning plug-ins currently available in TuneD.

Monitoring plug-ins

Currently, the following monitoring plug-ins are implemented:

disk Gets disk load (number of IO operations) per device and measurement interval. net Gets network load (number of transferred packets) per network card and measurement interval. load Gets CPU load per CPU and measurement interval.

Tuning plug-ins

Currently, the following tuning plug-ins are implemented. Only some of these plug-ins implement dynamic tuning. Options supported by plug-ins are also listed:

cpu

Sets the CPU governor to the value specified by the governor option and dynamically changes the Power Management Quality of Service (PM QoS) CPU Direct Memory Access (DMA) latency according to the CPU load.

If the CPU load is lower than the value specified by the load_threshold option, the latency is set to the value specified by the latency_high option, otherwise it is set to the value specified by latency_low.

You can also force the latency to a specific value and prevent it from dynamically changing further. To do so, set the force_latency option to the required latency value.

Dynamically sets the front-side bus (FSB) speed according to the CPU load.

This feature can be found on some netbooks and is also known as the ASUS Super Hybrid Engine (SHE).

If the CPU load is lower or equal to the value specified by the load_threshold_powersave option, the plug-in sets the FSB speed to the value specified by the she_powersave option. If the CPU load is higher or equal to the value specified by the load_threshold_normal option, it sets the FSB speed to the value specified by the she_normal option.

Static tuning is not supported and the plug-in is transparently disabled if TuneD does not detect the hardware support for this feature.

Sets various sysctl settings specified by the plug-in options.

The syntax is name=value, where name is the same as the name provided by the sysctl utility.

Use the sysctl plug-in if you need to change system settings that are not covered by other plug-ins available in TuneD. If the settings are covered by some specific plug-ins, prefer these plug-ins.

Sets autosuspend timeout of USB devices to the value specified by the autosuspend parameter.

The value 0 means that autosuspend is disabled.

Enables or disables transparent huge pages depending on the value of the transparent_hugepages option.

Valid values of the transparent_hugepages option are:

• "always"
• "never"
• "madvise"

Sets the autosuspend timeout for audio codecs to the value specified by the timeout option.

Currently, the snd_hda_intel and snd_ac97_codec codecs are supported. The value 0 means that the autosuspend is disabled. You can also enforce the controller reset by setting the Boolean option reset_controller to true.

Sets the disk elevator to the value specified by the elevator option.

It also sets:

• APM to the value specified by the apm option
• Scheduler quantum to the value specified by the scheduler_quantum option
• Disk spindown timeout to the value specified by the spindown option
• Disk readahead to the value specified by the readahead parameter
• The current disk readahead to a value multiplied by the constant specified by the readahead_multiply option

In addition, this plug-in dynamically changes the advanced power management and spindown timeout setting for the drive according to the current drive utilization. The dynamic tuning can be controlled by the Boolean option dynamic and is enabled by default.

Tunes options for SCSI hosts.

It sets Aggressive Link Power Management (ALPM) to the value specified by the alpm option.

Executes an external script or binary when the profile is loaded or unloaded. You can choose an arbitrary executable.

The script plug-in is provided mainly for compatibility with earlier releases. Prefer other TuneD plug-ins if they cover the required functionality.

TuneD calls the executable with one of the following arguments:

• start when loading the profile
• stop when unloading the profile

You need to correctly implement the stop action in your executable and revert all settings that you changed during the start action. Otherwise, the roll-back step after changing your TuneD profile will not work.

Bash scripts can import the /usr/lib/tuned/functions Bash library and use the functions defined there. Use these functions only for functionality that is not natively provided by TuneD. If a function name starts with an underscore, such as _wifi_set_power_level, consider the function private and do not use it in your scripts, because it might change in the future.

Specify the path to the executable using the script parameter in the plug-in configuration.

Example 3.6. Running a Bash script from a profile

To run a Bash script named script.sh that is located in the profile directory, use:

Sets various sysfs settings specified by the plug-in options.

The syntax is name=value, where name is the sysfs path to use.

Use this plugin in case you need to change some settings that are not covered by other plug-ins. Prefer specific plug-ins if they cover the required settings.

Sets various powersave levels on video cards. Currently, only the Radeon cards are supported.

The powersave level can be specified by using the radeon_powersave option. Supported values are:

• default
• auto
• low
• mid
• high
• dynpm
• dpm-battery
• dpm-balanced
• dpm-perfomance

For details, see www.x.org. Note that this plug-in is experimental and the option might change in future releases.

Adds options to the kernel command line. This plug-in supports only the GRUB 2 boot loader.

Customized non-standard location of the GRUB 2 configuration file can be specified by the grub2_cfg_file option.

The kernel options are added to the current GRUB configuration and its templates. The system needs to be rebooted for the kernel options to take effect.

Switching to another profile or manually stopping the tuned service removes the additional options. If you shut down or reboot the system, the kernel options persist in the grub.cfg file.

The kernel options can be specified by the following syntax:

Example 3.7. Modifying the kernel command line

For example, to add the quiet kernel option to a TuneD profile, include the following lines in the tuned.conf file:

The following is an example of a custom profile that adds the isolcpus=2 option to the kernel command line:

3.9. Variables in TuneD profiles

Variables expand at run time when a TuneD profile is activated.

Using TuneD variables reduces the amount of necessary typing in TuneD profiles.

There are no predefined variables in TuneD profiles. You can define your own variables by creating the [variables] section in a profile and using the following syntax:

To expand the value of a variable in a profile, use the following syntax:

Example 3.8. Isolating CPU cores using variables

In the following example, the ${isolated_cores} variable expands to 1,2; hence the kernel boots with the isolcpus=1,2 option:

The variables can be specified in a separate file. For example, you can add the following lines to tuned.conf:

If you add the isolated_cores=1,2 option to the /etc/tuned/my-variables.conf file, the kernel boots with the isolcpus=1,2 option.

Additional resources

• tuned.conf(5) man page

3.10. Built-in functions in TuneD profiles

Built-in functions expand at run time when a TuneD profile is activated.

You can:

• Use various built-in functions together with TuneD variables
• Create custom functions in Python and add them to TuneD in the form of plug-ins

To call a function, use the following syntax:

To expand the directory path where the profile and the tuned.conf file are located, use the PROFILE_DIR function, which requires special syntax:

Example 3.9. Isolating CPU cores using variables and built-in functions

In the following example, the ${non_isolated_cores} variable expands to 0,3-5, and the cpulist_invert built-in function is called with the 0,3-5 argument:

The cpulist_invert function inverts the list of CPUs. For a 6-CPU machine, the inversion is 1,2, and the kernel boots with the isolcpus=1,2 command-line option.

Additional resources

• tuned.conf(5) man page

3.11. Built-in functions available in TuneD profiles

The following built-in functions are available in all TuneD profiles:

PROFILE_DIR Returns the directory path where the profile and the tuned.conf file are located. exec Executes a process and returns its output. assertion Compares two arguments. If they do not match, the function logs text from the first argument and aborts profile loading. assertion_non_equal Compares two arguments. If they match, the function logs text from the first argument and aborts profile loading. kb2s Converts kilobytes to disk sectors. s2kb Converts disk sectors to kilobytes. strip Creates a string from all passed arguments and deletes both leading and trailing white space. virt_check

Checks whether TuneD is running inside a virtual machine (VM) or on bare metal:

• Inside a VM, the function returns the first argument.
• On bare metal, the function returns the second argument, even in case of an error.

3.12. Creating new TuneD profiles

This procedure creates a new TuneD profile with custom performance rules.

Procedure

1. In the /etc/tuned/ directory, create a new directory named the same as the profile that you want to create:

   # mkdir /etc/tuned/my-profile

2. In the new directory, create a file named tuned.conf. Add a [main] section and plug-in definitions in it, according to your requirements.

   For example, see the configuration of the balanced profile:

   [main] summary=General non-specialized tuned profile [cpu] governor=conservative energy_perf_bias=normal [audio] timeout=10 [video] radeon_powersave=dpm-balanced, auto [scsi_host] alpm=medium_power

3. To activate the profile, use:

   # tuned-adm profile my-profile

4. Verify that the TuneD profile is active and the system settings are applied:

   $ tuned-adm active Current active profile: my-profile$ tuned-adm verify Verfication succeeded, current system settings match the preset profile. See tuned log file ('/var/log/tuned/tuned.log') for details.

Additional resources

• tuned.conf(5) man page

3.13. Modifying existing TuneD profiles

This procedure creates a modified child profile based on an existing TuneD profile.

Procedure

1. In the /etc/tuned/ directory, create a new directory named the same as the profile that you want to create:

   # mkdir /etc/tuned/modified-profile

2. In the new directory, create a file named tuned.conf, and set the [main] section as follows:

   [main] include=parent-profile

   Replace parent-profile with the name of the profile you are modifying.

3. Include your profile modifications.

   Example 3.10. Lowering swappiness in the throughput-performance profile

   To use the settings from the throughput-performance profile and change the value of vm.swappiness to 5, instead of the default 10, use:

   [main] include=throughput-performance [sysctl] vm.swappiness=5

4. To activate the profile, use:

   # tuned-adm profile modified-profile

5. Verify that the TuneD profile is active and the system settings are applied:

   $ tuned-adm active Current active profile: my-profile$ tuned-adm verify Verfication succeeded, current system settings match the preset profile. See tuned log file ('/var/log/tuned/tuned.log') for details.

Additional resources

• tuned.conf(5) man page

3.14. Setting the disk scheduler using TuneD

This procedure creates and enables a TuneD profile that sets a given disk scheduler for selected block devices. The setting persists across system reboots.

In the following commands and configuration, replace:

• device with the name of the block device, for example sdf
• selected-scheduler with the disk scheduler that you want to set for the device, for example bfq

Procedure

1. Optional: Select an existing TuneD profile on which your profile will be based. For a list of available profiles, see TuneD profiles distributed with RHEL.

   To see which profile is currently active, use:

   $ tuned-adm active

2. Create a new directory to hold your TuneD profile:

   # mkdir /etc/tuned/my-profile

3. Find the system unique identifier of the selected block device:

   $ udevadm info --query=property --name=/dev/device | grep -E '(WWN|SERIAL)' ID_WWN=0x5002538d00000000_ ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0 ID_SERIAL_SHORT=20120501030900000

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

4. Create the /etc/tuned/my-profile/tuned.conf configuration file. In the file, set the following options:

   1. Optional: Include an existing profile:

      [main] include=existing-profile

   2. Set the selected disk scheduler for the device that matches the WWN identifier:

      [disk] devices_udev_regex=IDNAME=device system unique id elevator=selected-scheduler

      Here:

      • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).

      • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

        To match multiple devices in the devices_udev_regex option, enclose the identifiers in parentheses and separate them with vertical bars:

        devices_udev_regex=(ID_WWN=0x5002538d00000000)|(ID_WWN=0x1234567800000000)

5. Enable your profile:

   # tuned-adm profile my-profile

Verification steps

1. Verify that the TuneD profile is active and applied:

   $ tuned-adm active Current active profile: my-profile$ tuned-adm verify Verification succeeded, current system settings match the preset profile. See tuned log file ('/var/log/tuned/tuned.log') for details.

2. Read the contents of the /sys/block/device/queue/scheduler file:

   # cat /sys/block/device/queue/scheduler [mq-deadline] kyber bfq none

   In the file name, replace device with the block device name, for example sdc.

   The active scheduler is listed in square brackets ([]).

4.1. Installing tuna tool

The tuna tool is designed to be used on a running system. This allows application-specific measurement tools to see and analyze system performance immediately after changes have been made.

This procedure describes how to install the tuna tool.

Procedure

• Install the tuna tool:

  # yum install tuna

Verification steps

• View the available tuna CLI options:

  # tuna -h

Additional resources

• tuna(8) man page

4.2. Viewing the system status using tuna tool

This procedure describes how to view the system status using the tuna command-line interface (CLI) tool.

Procedure

• To view the current policies and priorities:

  # tuna --show_threads thread pid SCHED_ rtpri affinity cmd 1 OTHER 0 0,1 init 2 FIFO 99 0 migration/0 3 OTHER 0 0 ksoftirqd/0 4 FIFO 99 0 watchdog/0

• To view a specific thread corresponding to a PID or matching a command name:

  # tuna --threads=pid_or_cmd_list --show_threads

  The pid_or_cmd_list argument is a list of comma-separated PIDs or command-name patterns.

• To tune CPUs using the tuna CLI, see Tuning CPUs using tuna tool.
• To tune the IRQs using the tuna tool, see Tuning IRQs using tuna tool.

• To save the changed configuration:

  # tuna --save=filename

  This command saves only currently running kernel threads. Processes that are not running are not saved.

Additional resources

• tuna(8) man page

4.3. Tuning CPUs using tuna tool

The tuna tool commands can target individual CPUs.

Using the tuna tool, you can:

Isolate CPUs All tasks running on the specified CPU move to the next available CPU. Isolating a CPU makes it unavailable by removing it from the affinity mask of all threads. Include CPUs Allows tasks to run on the specified CPU Restore CPUs Restores the specified CPU to its previous configuration.

This procedure describes how to tune CPUs using the tuna CLI.

Procedure

• To specify the list of CPUs to be affected by a command:

  # tuna --cpus=cpu_list [command]

  The cpu_list argument is a list of comma-separated CPU numbers. For example, --cpus=0,2. CPU lists can also be specified in a range, for example --cpus=”1-3”, which would select CPUs 1, 2, and 3.

  To add a specific CPU to the current cpu_list, for example, use --cpus=+0.

  Replace [command] with, for example, --isolate.

• To isolate a CPU:

  # tuna --cpus=cpu_list --isolate

• To include a CPU:

  # tuna --cpus=cpu_list --include

• To use a system with four or more processors, display how to make all the ssh threads run on CPU 0 and 1, and all the http threads on CPU 2 and 3:

  # tuna --cpus=0,1 --threads=ssh\* \ --move --cpus=2,3 --threads=http\* --move

  This command performs the following operations sequentially:

  1. Selects CPUs 0 and 1.
  2. Selects all threads that begin with ssh.
  3. Moves the selected threads to the selected CPUs. Tuna sets the affinity mask of threads starting with ssh to the appropriate CPUs. The CPUs can be expressed numerically as 0 and 1, in hex mask as 0x3, or in binary as 11.
  4. Resets the CPU list to 2 and 3.
  5. Selects all threads that begin with http.
  6. Moves the selected threads to the specified CPUs. Tuna sets the affinity mask of threads starting with http to the specified CPUs. The CPUs can be expressed numerically as 2 and 3, in hex mask as 0xC, or in binary as 1100.

Verification steps

• Display the current configuration and verify that the changes were performed as expected:

  # tuna --threads=gnome-sc\* --show_threads \ --cpus=0 --move --show_threads --cpus=1 \ --move --show_threads --cpus=+0 --move --show_threads thread ctxt_switches pid SCHED_ rtpri affinity voluntary nonvoluntary cmd 3861 OTHER 0 0,1 33997 58 gnome-screensav thread ctxt_switches pid SCHED_ rtpri affinity voluntary nonvoluntary cmd 3861 OTHER 0 0 33997 58 gnome-screensav thread ctxt_switches pid SCHED_ rtpri affinity voluntary nonvoluntary cmd 3861 OTHER 0 1 33997 58 gnome-screensav thread ctxt_switches pid SCHED_ rtpri affinity voluntary nonvoluntary cmd 3861 OTHER 0 0,1 33997 58 gnome-screensav

  This command performs the following operations sequentially:

  1. Selects all threads that begin with the gnome-sc threads.
  2. Displays the selected threads to enable the user to verify their affinity mask and RT priority.
  3. Selects CPU 0.
  4. Moves the gnome-sc threads to the specified CPU, CPU 0.
  5. Shows the result of the move.
  6. Resets the CPU list to CPU 1.
  7. Moves the gnome-sc threads to the specified CPU, CPU 1.
  8. Displays the result of the move.
  9. Adds CPU 0 to the CPU list.
  10. Moves the gnome-sc threads to the specified CPUs, CPUs 0 and 1.
  11. Displays the result of the move.

Additional resources

• /proc/cpuinfo file
• tuna(8) man page

4.4. Tuning IRQs using tuna tool

The /proc/interrupts file records the number of interrupts per IRQ, the type of interrupt, and the name of the device that is located at that IRQ.

This procedure describes how to tune the IRQs using the tuna tool.

Procedure

• To view the current IRQs and their affinity:

  # tuna --show_irqs # users affinity 0 timer 0 1 i8042 0 7 parport0 0

• To specify the list of IRQs to be affected by a command:

  # tuna --irqs=irq_list [command]

  The irq_list argument is a list of comma-separated IRQ numbers or user-name patterns.

  Replace [command] with, for example, --spread.

• To move an interrupt to a specified CPU:

  # tuna --irqs=128 --show_irqs # users affinity 128 iwlwifi 0,1,2,3 # tuna --irqs=128 --cpus=3 --move

  Replace 128 with the irq_list argument and 3 with the cpu_list argument.

  The cpu_list argument is a list of comma-separated CPU numbers, for example, --cpus=0,2. For more information, see Tuning CPUs using tuna tool.

Verification steps

• Compare the state of the selected IRQs before and after moving any interrupt to a specified CPU:

  # tuna --irqs=128 --show_irqs # users affinity 128 iwlwifi 3

Additional resources

• /procs/interrupts file
• tuna(8) man page

5.1. Introduction to RHEL System Roles

RHEL System Roles is a collection of Ansible roles and modules. RHEL System Roles provide a configuration interface to remotely manage multiple RHEL systems. The interface enables managing system configurations across multiple versions of RHEL, as well as adopting new major releases.

On Red Hat Enterprise Linux 8, the interface currently consists of the following roles:

• Certificate Issuance and Renewal
• Cockpit
• Firewalld
• HA Cluster
• Kernel Dumps
• Kernel Settings
• Logging
• Metrics (PCP)
• Microsoft SQL Server
• Networking
• Network Bound Disk Encryption client and Network Bound Disk Encryption server
• Postfix
• SELinux
• SSH client
• SSH server
• Storage
• Terminal Session Recording
• Time Synchronization
• VPN

All these roles are provided by the rhel-system-roles package available in the AppStream repository.

5.2. RHEL System Roles terminology

You can find the following terms across this documentation:

Ansible playbook Playbooks are Ansible’s configuration, deployment, and orchestration language. They can describe a policy you want your remote systems to enforce, or a set of steps in a general IT process. Control node Any machine with Ansible installed. You can run commands and playbooks, invoking /usr/bin/ansible or /usr/bin/ansible-playbook, from any control node. You can use any computer that has Python installed on it as a control node - laptops, shared desktops, and servers can all run Ansible. However, you cannot use a Windows machine as a control node. You can have multiple control nodes. Inventory A list of managed nodes. An inventory file is also sometimes called a “hostfile”. Your inventory can specify information like IP address for each managed node. An inventory can also organize managed nodes, creating and nesting groups for easier scaling. To learn more about inventory, see the Working with Inventory section. Managed nodes The network devices, servers, or both that you manage with Ansible. Managed nodes are also sometimes called “hosts”. Ansible is not installed on managed nodes.

5.3. Installing RHEL System Roles in your system

To use the RHEL System Roles, install the required packages in your system.

Prerequisites

• The Ansible Core package is installed on the control machine.
• You have Ansible packages installed in the system you want to use as a control node.

Procedure

1. Install the rhel-system-roles package on the system that you want to use as a control node:

   # yum install rhel-system-roles

2. Install the Ansible Core package:

   # yum install ansible-core

The Ansible Core package provides the ansible-playbook CLI, the Ansible Vault functionality, and the basic modules and filters required by RHEL Ansible content.

As a result, you are able to create an Ansible playbook.

5.4. Applying a role

The following procedure describes how to apply a particular role.

Prerequisites

• Ensure that the rhel-system-roles package is installed on the system that you want to use as a control node:

  # yum install rhel-system-roles

  1. Install the Ansible Core package:

     # yum install ansible-core

     The Ansible Core package provides the ansible-playbook CLI, the Ansible Vault functionality, and the basic modules and filters required by RHEL Ansible content.

• Ensure that you are able to create an Ansible inventory.

  Inventories represent the hosts, host groups, and some of the configuration parameters used by the Ansible playbooks.

  Playbooks are typically human-readable, and are defined in ini, yaml, json, and other file formats.

• Ensure that you are able to create an Ansible playbook.

  Playbooks represent Ansible’s configuration, deployment, and orchestration language. By using playbooks, you can declare and manage configurations of remote machines, deploy multiple remote machines or orchestrate steps of any manual ordered process.

  A playbook is a list of one or more plays. Every play can include Ansible variables, tasks, or roles.

  Playbooks are human-readable, and are defined in the yaml format.

Procedure

1. Create the required Ansible inventory containing the hosts and groups that you want to manage. Here is an example using a file called inventory.ini of a group of hosts called webservers:

   [webservers] host1 host2 host3

2. Create an Ansible playbook including the required role. The following example shows how to use roles through the roles: option for a playbook:

   The following example shows how to use roles through the roles: option for a given play:

   --- - hosts: webservers roles: - rhel-system-roles.network - rhel-system-roles.timesync

   Every role includes a README file, which documents how to use the role and supported parameter values. You can also find an example playbook for a particular role under the documentation directory of the role. Such documentation directory is provided by default with the rhel-system-roles package, and can be found in the following location:

   /usr/share/doc/rhel-system-roles/SUBSYSTEM/

   Replace SUBSYSTEM with the name of the required role, such as selinux, kdump, network, timesync, or storage.

3. To execute the playbook on specific hosts, you must perform one of the following:

   • Edit the playbook to use hosts: host1[,host2,…​], or hosts: all, and execute the command:

     # ansible-playbook name.of.the.playbook

   • Edit the inventory to ensure that the hosts you want to use are defined in a group, and execute the command:

     # ansible-playbook -i name.of.the.inventory name.of.the.playbook

   • Specify all hosts when executing the ansible-playbook command:

     # ansible-playbook -i host1,host2,... name.of.the.playbook

     Be aware that the -i flag specifies the inventory of all hosts that are available. If you have multiple targeted hosts, but want to select a host against which you want to run the playbook, you can add a variable in the playbook to be able to select a host. For example:

     Ansible Playbook | example-playbook.yml: - hosts: "{{ target_host }}" roles: - rhel-system-roles.network - rhel-system-roles.timesync

     Playbook execution command:

     # ansible-playbook -i host1,..hostn -e target_host=host5 example-playbook.yml

5.5. Introduction to the Metrics System Role

RHEL System Roles is a collection of Ansible roles and modules that provide a consistent configuration interface to remotely manage multiple RHEL systems. The Metrics System Role configures performance analysis services for the local system and, optionally, includes a list of remote systems to be monitored by the local system. The Metrics System Role enables you to use pcp to monitor your systems performance without having to configure pcp separately, as the set-up and deployment of pcp is handled by the playbook.

Table 5.1. Metrics system role variables

For details about the parameters used in metrics_connections and additional information about the Metrics System Role, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file.

5.6. Using the Metrics System Role to monitor your local system with visualization

This procedure describes how to use the Metrics RHEL System Role to monitor your local system while simultaneously provisioning data visualization via Grafana.

Prerequisites

• The Ansible Core package is installed on the control machine.
• You have the rhel-system-roles package installed on the machine you want to monitor.

Procedure

1. Configure localhost in the the /etc/ansible/hosts Ansible inventory by adding the following content to the inventory:

   localhost ansible_connection=local

2. Create an Ansible playbook with the following content:

   --- - hosts: localhost vars: metrics_graph_service: yes roles: - rhel-system-roles.metrics

3. Run the Ansible playbook:

   # ansible-playbook name_of_your_playbook.yml

   Since the metrics_graph_service boolean is set to value="yes", Grafana is automatically installed and provisioned with pcp added as a data source.

4. To view visualization of the metrics being collected on your machine, access the grafana web interface as described in Accessing the Grafana web UI.

5.7. Using the Metrics System Role to setup a fleet of individual systems to monitor themselves

This procedure describes how to use the Metrics System Role to set up a fleet of machines to monitor themselves.

Prerequisites

• The Ansible Core package is installed on the control machine.
• You have the rhel-system-roles package installed on the machine you want to use to run the playbook.
• You have the SSH connection established.

Procedure

1. Add the name or IP of the machines you wish to monitor via the playbook to the /etc/ansible/hosts Ansible inventory file under an identifying group name enclosed in brackets:

   [remotes] webserver.example.com database.example.com

2. Create an Ansible playbook with the following content:

   --- - hosts: remotes vars: metrics_retention_days: 0 roles: - rhel-system-roles.metrics

3. Run the Ansible playbook:

   # ansible-playbook name_of_your_playbook.yml -k

Where the -k prompt for password to connect to remote system.

5.8. Using the Metrics System Role to monitor a fleet of machines centrally via your local machine

This procedure describes how to use the Metrics System Role to set up your local machine to centrally monitor a fleet of machines while also provisioning visualization of the data via grafana and querying of the data via redis.

Prerequisites

• The Ansible Core package is installed on the control machine.
• You have the rhel-system-roles package installed on the machine you want to use to run the playbook.

Procedure

1. Create an Ansible playbook with the following content:

   --- - hosts: localhost vars: metrics_graph_service: yes metrics_query_service: yes metrics_retention_days: 10 metrics_monitored_hosts: ["database.example.com", "webserver.example.com"] roles: - rhel-system-roles.metrics

2. Run the Ansible playbook:

   # ansible-playbook name_of_your_playbook.yml

   Since the metrics_graph_service and metrics_query_service booleans are set to value="yes", grafana is automatically installed and provisioned with pcp added as a data source with the pcp data recording indexed into redis, allowing the pcp querying language to be used for complex querying of the data.

3. To view graphical representation of the metrics being collected centrally by your machine and to query the data, access the grafana web interface as described in Accessing the Grafana web UI.

5.9. Setting up authentication while monitoring a system using the Metrics System Role

PCP supports the scram-sha-256 authentication mechanism through the Simple Authentication Security Layer (SASL) framework. The Metrics RHEL System Role automates the steps to setup authentication using the scram-sha-256 authentication mechanism. This procedure describes how to setup authentication using the Metrics RHEL System Role.

Prerequisites

• The Ansible Core package is installed on the control machine.
• You have the rhel-system-roles package installed on the machine you want to use to run the playbook.

Procedure

1. Include the following variables in the Ansible playbook you want to setup authentication for:

   --- vars: metrics_username: your_username metrics_password: your_password

2. Run the Ansible playbook:

   # ansible-playbook name_of_your_playbook.yml

Verification steps

• Verify the sasl configuration:

  # pminfo -f -h "pcp://ip_adress?username=your_username" disk.dev.read Password: disk.dev.read inst [0 or "sda"] value 19540

  ip_adress should be replaced by the IP address of the host.

5.10. Using the Metrics System Role to configure and enable metrics collection for SQL Server

This procedure describes how to use the Metrics RHEL System Role to automate the configuration and enabling of metrics collection for Microsoft SQL Server via pcp on your local system.

Prerequisites

• The Ansible Core package is installed on the control machine.
• You have the rhel-system-roles package installed on the machine you want to monitor.
• You have installed Microsoft SQL Server for Red Hat Enterprise Linux and established a 'trusted' connection to an SQL server. See Install SQL Server and create a database on Red Hat.
• You have installed the Microsoft ODBC driver for SQL Server for Red Hat Enterprise Linux. See Red Hat Enterprise Server and Oracle Linux.

Procedure

1. Configure localhost in the the /etc/ansible/hosts Ansible inventory by adding the following content to the inventory:

   localhost ansible_connection=local

2. Create an Ansible playbook that contains the following content:

   --- - hosts: localhost roles: - role: rhel-system-roles.metrics vars: metrics_from_mssql: yes

3. Run the Ansible playbook:

   # ansible-playbook name_of_your_playbook.yml

Verification steps

• Use the pcp command to verify that SQL Server PMDA agent (mssql) is loaded and running:

  # pcp platform: Linux rhel82-2.local 4.18.0-167.el8.x86_64 #1 SMP Sun Dec 15 01:24:23 UTC 2019 x86_64 hardware: 2 cpus, 1 disk, 1 node, 2770MB RAM timezone: PDT+7 services: pmcd pmproxy pmcd: Version 5.0.2-1, 12 agents, 4 clients pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm mssql jbd2 dm pmlogger: primary logger: /var/log/pcp/pmlogger/rhel82-2.local/20200326.16.31 pmie: primary engine: /var/log/pcp/pmie/rhel82-2.local/pmie.log

6.1. Overview of PCP

You can add performance metrics using Python, Perl, C++, and C interfaces. Analysis tools can use the Python, C++, C client APIs directly, and rich web applications can explore all available performance data using a JSON interface.

You can analyze data patterns by comparing live results with archived data.

Features of PCP:

• Light-weight distributed architecture, which is useful during the centralized analysis of complex systems.
• It allows the monitoring and management of real-time data.
• It allows logging and retrieval of historical data.

PCP has the following components:

• The Performance Metric Collector Daemon (pmcd) collects performance data from the installed Performance Metric Domain Agents (pmda). PMDAs can be individually loaded or unloaded on the system and are controlled by the PMCD on the same host.
• Various client tools, such as pminfo or pmstat, can retrieve, display, archive, and process this data on the same host or over the network.
• The pcp package provides the command-line tools and underlying functionality.
• The pcp-gui package provides the graphical application. Install the pcp-gui package by executing the yum install pcp-gui command. For more information, see Visually tracing PCP log archives with the PCP Charts application.

Additional resources

• pcp(1) man page
• /usr/share/doc/pcp-doc/ directory
• Tools distributed with PCP
• Index of Performance Co-Pilot (PCP) articles, solutions, tutorials, and white papers fromon Red Hat Customer Portal
• Side-by-side comparison of PCP tools with legacy tools Red Hat Knowledgebase article
• PCP upstream documentation

6.2. Installing and enabling PCP

To begin using PCP, install all the required packages and enable the PCP monitoring services.

This procedure describes how to install PCP using the pcp package. If you want to automate the PCP installation, install it using the pcp-zeroconf package. For more information on installing PCP by using pcp-zeroconf, see Setting up PCP with pcp-zeroconf.

Procedure

1. Install the pcp package:

   # yum install pcp

2. Enable and start the pmcd service on the host machine:

   # systemctl enable pmcd # systemctl start pmcd

Verification steps

• Verify if the pmcd process is running on the host:

  # pcp Performance Co-Pilot configuration on workstation: platform: Linux workstation 4.18.0-80.el8.x86_64 #1 SMP Wed Mar 13 12:02:46 UTC 2019 x86_64 hardware: 12 cpus, 2 disks, 1 node, 36023MB RAM timezone: CEST-2 services: pmcd pmcd: Version 4.3.0-1, 8 agents pmda: root pmcd proc xfs linux mmv kvm jbd2

6.3. Deploying a minimal PCP setup

The minimal PCP setup collects performance statistics on Red Hat Enterprise Linux. The setup involves adding the minimum number of packages on a production system needed to gather data for further analysis.

You can analyze the resulting tar.gz file and the archive of the pmlogger output using various PCP tools and compare them with other sources of performance information.

Procedure

1. Update the pmlogger configuration:

   # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default

2. Start the pmcd and pmlogger services:

   # systemctl start pmcd.service # systemctl start pmlogger.service
3. Execute the required operations to record the performance data.

4. Stop the pmcd and pmlogger services:

   # systemctl stop pmcd.service # systemctl stop pmlogger.service

5. Save the output and save it to a tar.gz file named based on the host name and the current date and time:

   # cd /var/log/pcp/pmlogger/ # tar -czf $(hostname).$(date +%F-%Hh%M).pcp.tar.gz $(hostname)

   Extract this file and analyze the data using PCP tools.

6.4. System services distributed with PCP

The following table describes roles of various system services, which are distributed with PCP.

Table 6.1. Roles of system services distributed with PCP

6.5. Tools distributed with PCP

The following table describes usage of various tools, which are distributed with PCP.

Table 6.2. Usage of tools distributed with PCP

6.6. PCP deployment architectures

Performance Co-Pilot (PCP) offers many options to accomplish advanced setups. From the huge variety of possible architectures, this section describes how to scale your PCP deployment based on the recommended deployment set up by Red Hat, sizing factors, and configuration options.

PCP supports multiple deployment architectures, based on the scale of the PCP deployment.

Available scaling deployment setup variants:

Since the PCP version 5.3.0 is unavailable in Red Hat Enterprise Linux 8.4 and the prior minor versions of Red Hat Enterprise Linux 8, Red Hat recommends localhost and pmlogger farm architectures.

For more information about known memory leaks in pmproxy in PCP versions before 5.3.0, see Memory leaks in pmproxy in PCP.

Localhost

Each service runs locally on the monitored machine. When you start a service without any configuration changes, this is the default deployment. Scaling beyond the individual node is not possible in this case.

By default, the deployment setup for Redis is standalone, localhost. However, Redis can optionally perform in a highly-available and highly scalable clustered fashion, where data is shared across multiple hosts. Another viable option is to deploy a Redis cluster in the cloud, or to utilize a managed Redis cluster from a cloud vendor.

The only difference between localhost and decentralized setup is the centralized Redis service. In this model, the host executes pmlogger service on each monitored host and retrieves metrics from a local pmcd instance. A local pmproxy service then exports the performance metrics to a central Redis instance.

Figure 6.1. Decentralized logging

When the resource usage on the monitored hosts is constrained, another deployment option is a pmlogger farm, which is also known as centralized logging. In this setup, a single logger host executes multiple pmlogger processes, and each is configured to retrieve performance metrics from a different remote pmcd host. The centralized logger host is also configured to execute the pmproxy service, which discovers the resulting PCP archives logs and loads the metric data into a Redis instance.

Figure 6.2. Centralized logging - pmlogger farm

For large scale deployments, Red Hat recommends to deploy multiple pmlogger farms in a federated fashion. For example, one pmlogger farm per rack or data center. Each pmlogger farm loads the metrics into a central Redis instance.

Figure 6.3. Federated - multiple pmlogger farms

By default, the deployment setup for Redis is standalone, localhost. However, Redis can optionally perform in a highly-available and highly scalable clustered fashion, where data is shared across multiple hosts. Another viable option is to deploy a Redis cluster in the cloud, or to utilize a managed Redis cluster from a cloud vendor.

6.7. Recommended deployment architecture

The following table describes the recommended deployment architectures based on the number of monitored hosts.

Table 6.3. Recommended deployment architecture

6.8. Sizing factors

The following are the sizing factors required for scaling:

Remote system size The number of CPUs, disks, network interfaces, and other hardware resources affects the amount of data collected by each pmlogger on the centralized logging host. Logged Metrics The number and types of logged metrics play an important role. In particular, the per-process proc.* metrics require a large amount of disk space, for example, with the standard pcp-zeroconf setup, 10s logging interval, 11 MB without proc metrics versus 155 MB with proc metrics - a factor of 10 times more. Additionally, the number of instances for each metric, for example the number of CPUs, block devices, and network interfaces also impacts the required storage capacity. Logging Interval The interval how often metrics are logged, affects the storage requirements. The expected daily PCP archive file sizes are written to the pmlogger.log file for each pmlogger instance. These values are uncompressed estimates. Since PCP archives compress very well, approximately 10:1, the actual long term disk space requirements can be determined for a particular site. pmlogrewrite After every PCP upgrade, the pmlogrewrite tool is executed and rewrites old archives if there were changes in the metric metadata from the previous version and the new version of PCP. This process duration scales linear with the number of archives stored.

Additional resources

• pmlogrewrite(1) and pmlogger(1) man pages

6.9. Configuration options for PCP scaling

The following are the configuration options, which are required for scaling:

sysctl and rlimit settings When archive discovery is enabled, pmproxy requires four descriptors for every pmlogger that it is monitoring or log-tailing, along with the additional file descriptors for the service logs and pmproxy client sockets, if any. Each pmlogger process uses about 20 file descriptors for the remote pmcd socket, archive files, service logs, and others. In total, this can exceed the default 1024 soft limit on a system running around 200 pmlogger processes. The pmproxy service in pcp-5.3.0 and later automatically increases the soft limit to the hard limit. On earlier versions of PCP, tuning is required if a high number of pmlogger processes are to be deployed, and this can be accomplished by increasing the soft or hard limits for pmlogger. For more information, see How to set limits (ulimit) for services run by systemd. Local Archives The pmlogger service stores metrics of local and remote pmcds in the /var/log/pcp/pmlogger/ directory. To control the logging interval of the local system, update the /etc/pcp/pmlogger/control.d/configfile file and add -t X in the arguments, where X is the logging interval in seconds. To configure which metrics should be logged, execute pmlogconf /var/lib/pcp/config/pmlogger/config.clienthostname. This command deploys a configuration file with a default set of metrics, which can optionally be further customized. To specify retention settings, that is when to purge old PCP archives, update the /etc/sysconfig/pmlogger_timers file and specify PMLOGGER_DAILY_PARAMS="-E -k X", where X is the amount of days to keep PCP archives. Redis

The pmproxy service sends logged metrics from pmlogger to a Redis instance. The following are the available two options to specify the retention settings in the /etc/pcp/pmproxy/pmproxy.conf configuration file:

• stream.expire specifies the duration when stale metrics should be removed, that is metrics which were not updated in a specified amount of time in seconds.
• stream.maxlen specifies the maximum number of metric values for one metric per host. This setting should be the retention time divided by the logging interval, for example 20160 for 14 days of retention and 60s logging interval (60*60*24*14/60)

Additional resources

• pmproxy(1), pmlogger(1), and sysctl(8) man pages

6.10. Example: Analyzing the centralized logging deployment

The following results were gathered on a centralized logging setup, also known as pmlogger farm deployment, with a default pcp-zeroconf 5.3.0 installation, where each remote host is an identical container instance running pmcd on a server with 64 CPU cores, 376 GB RAM, and one disk attached.

The logging interval is 10s, proc metrics of remote nodes are not included, and the memory values refer to the Resident Set Size (RSS) value.

Table 6.4. Detailed utilization statistics for 10s logging interval

Table 6.5. Used resources depending on monitored hosts for 60s logging interval

The pmproxy queues Redis requests and employs Redis pipelining to speed up Redis queries. This can result in high memory usage. For troubleshooting this issue, see Troubleshooting high memory usage.

6.11. Example: Analyzing the federated setup deployment

The following results were observed on a federated setup, also known as multiple pmlogger farms, consisting of three centralized logging (pmlogger farm) setups, where each pmlogger farm was monitoring 100 remote hosts, that is 300 hosts in total.

This setup of the pmlogger farms is identical to the configuration mentioned in the Example: Analyzing the centralized logging deployment for 60s logging interval, except that the Redis servers were operating in cluster mode.

Table 6.6. Used resources depending on federated hosts for 60s logging interval

Here, all values are per host. The network bandwidth is higher due to the inter-node communication of the Redis cluster.

6.12. Troubleshooting high memory usage

The following scenarios can result in high memory usage:

• The pmproxy process is busy processing new PCP archives and does not have spare CPU cycles to process Redis requests and responses.
• The Redis node or cluster is overloaded and cannot process incoming requests on time.

The pmproxy service daemon uses Redis streams and supports the configuration parameters, which are PCP tuning parameters and affects Redis memory usage and key retention. The /etc/pcp/pmproxy/pmproxy.conf file lists the available configuration options for pmproxy and the associated APIs.

This section describes how to troubleshoot high memory usage issue.

Prerequisites

1. Install the pcp-pmda-redis package:

   # yum install pcp-pmda-redis

2. Install the redis PMDA:

   # cd /var/lib/pcp/pmdas/redis && ./Install

Procedure

• To troubleshoot high memory usage, execute the following command and observe the inflight column:

  $ pmrep :pmproxy backlog inflight reqs/s resp/s wait req err resp err changed throttled byte count count/s count/s s/s count/s count/s count/s count/s 14:59:08 0 0 N/A N/A N/A N/A N/A N/A N/A 14:59:09 0 0 2268.9 2268.9 28 0 0 2.0 4.0 14:59:10 0 0 0.0 0.0 0 0 0 0.0 0.0 14:59:11 0 0 0.0 0.0 0 0 0 0.0 0.0

  This column shows how many Redis requests are in-flight, which means they are queued or sent, and no reply was received so far.

  A high number indicates one of the following conditions:

  • The pmproxy process is busy processing new PCP archives and does not have spare CPU cycles to process Redis requests and responses.
  • The Redis node or cluster is overloaded and cannot process incoming requests on time.

• To troubleshoot the high memory usage issue, reduce the number of pmlogger processes for this farm, and add another pmlogger farm. Use the federated - multiple pmlogger farms setup.

  If the Redis node is using 100% CPU for an extended amount of time, move it to a host with better performance or use a clustered Redis setup instead.

• To view the pmproxy.redis.* metrics, use the following command:

  $ pminfo -ftd pmproxy.redis pmproxy.redis.responses.wait [wait time for responses] Data Type: 64-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff Semantics: counter Units: microsec value 546028367374 pmproxy.redis.responses.error [number of error responses] Data Type: 64-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff Semantics: counter Units: count value 1164 [...] pmproxy.redis.requests.inflight.bytes [bytes allocated for inflight requests] Data Type: 64-bit int InDom: PM_INDOM_NULL 0xffffffff Semantics: discrete Units: byte value 0 pmproxy.redis.requests.inflight.total [inflight requests] Data Type: 64-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff Semantics: discrete Units: count value 0 [...]

  To view how many Redis requests are inflight, see the pmproxy.redis.requests.inflight.total metric and pmproxy.redis.requests.inflight.bytes metric to view how many bytes are occupied by all current inflight Redis requests.

  In general, the redis request queue would be zero but can build up based on the usage of large pmlogger farms, which limits scalability and can cause high latency for pmproxy clients.

• Use the pminfo command to view information about performance metrics. For example, to view the redis.* metrics, use the following command:

  $ pminfo -ftd redis redis.redis_build_id [Build ID] Data Type: string InDom: 24.0 0x6000000 Semantics: discrete Units: count inst [0 or "localhost:6379"] value "87e335e57cffa755" redis.total_commands_processed [Total number of commands processed by the server] Data Type: 64-bit unsigned int InDom: 24.0 0x6000000 Semantics: counter Units: count inst [0 or "localhost:6379"] value 595627069 [...] redis.used_memory_peak [Peak memory consumed by Redis (in bytes)] Data Type: 32-bit unsigned int InDom: 24.0 0x6000000 Semantics: instant Units: count inst [0 or "localhost:6379"] value 572234920 [...]

  To view the peak memory usage, see the redis.used_memory_peak metric.

7.1. Modifying the pmlogger configuration file with pmlogconf

When the pmlogger service is running, PCP logs a default set of metrics on the host.

Use the pmlogconf utility to check the default configuration. If the pmlogger configuration file does not exist, pmlogconf creates it with a default metric values.

Procedure

1. Create or modify the pmlogger configuration file:

   # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default
2. Follow pmlogconf prompts to enable or disable groups of related performance metrics and to control the logging interval for each enabled group.

7.2. Editing the pmlogger configuration file manually

To create a tailored logging configuration with specific metrics and given intervals, edit the pmlogger configuration file manually. The default pmlogger configuration file is /var/lib/pcp/config/pmlogger/config.default. The configuration file specifies which metrics are logged by the primary logging instance.

In manual configuration, you can:

• Record metrics which are not listed in the automatic configuration.
• Choose custom logging frequencies.
• Add PMDA with the application metrics.

Procedure

• Open and edit the /var/lib/pcp/config/pmlogger/config.default file to add specific metrics:

  # It is safe to make additions from here on ... # log mandatory on every 5 seconds { xfs.write xfs.write_bytes xfs.read xfs.read_bytes } log mandatory on every 10 seconds { xfs.allocs xfs.block_map xfs.transactions xfs.log } [access] disallow * : all; allow localhost : enquire;

7.3. Enabling the pmlogger service

The pmlogger service must be started and enabled to log the metric values on the local machine.

This procedure describes how to enable the pmlogger service.

Procedure

• Start and enable the pmlogger service:

  # systemctl start pmlogger # systemctl enable pmlogger

Verification steps

• Verify if the pmlogger service is enabled:

  # pcp Performance Co-Pilot configuration on workstation: platform: Linux workstation 4.18.0-80.el8.x86_64 #1 SMP Wed Mar 13 12:02:46 UTC 2019 x86_64 hardware: 12 cpus, 2 disks, 1 node, 36023MB RAM timezone: CEST-2 services: pmcd pmcd: Version 4.3.0-1, 8 agents, 1 client pmda: root pmcd proc xfs linux mmv kvm jbd2 pmlogger: primary logger: /var/log/pcp/pmlogger/workstation/20190827.15.54

7.4. Setting up a client system for metrics collection

This procedure describes how to set up a client system so that a central server can collect metrics from clients running PCP.

Procedure

1. Install the pcp-system-tools package:

   # yum install pcp-system-tools

2. Configure an IP address for pmcd:

   # echo "-i 192.168.4.62" >>/etc/pcp/pmcd/pmcd.options

   Replace 192.168.4.62 with the IP address, the client should listen on.

   By default, pmcd is listening on the localhost.

3. Configure the firewall to add the public zone permanently:

   # firewall-cmd --permanent --zone=public --add-port=44321/tcp success # firewall-cmd --reload success

4. Set an SELinux boolean:

   # setsebool -P pcp_bind_all_unreserved_ports on

5. Enable the pmcd and pmlogger services:

   # systemctl enable pmcd pmlogger # systemctl restart pmcd pmlogger

Verification steps

• Verify if the pmcd is correctly listening on the configured IP address:

  # ss -tlp | grep 44321 LISTEN 0 5 127.0.0.1:44321 0.0.0.0:* users:(("pmcd",pid=151595,fd=6)) LISTEN 0 5 192.168.4.62:44321 0.0.0.0:* users:(("pmcd",pid=151595,fd=0)) LISTEN 0 5 [::1]:44321 [::]:* users:(("pmcd",pid=151595,fd=7))

7.5. Setting up a central server to collect data

This procedure describes how to create a central server to collect metrics from clients running PCP.

Prerequisites

• PCP is installed. For more information, see Installing and enabling PCP.
• Client is configured for metrics collection. For more information, see Setting up a client system for metrics collection.

Procedure

1. Install the pcp-system-tools package:

   # yum install pcp-system-tools

2. Create the /etc/pcp/pmlogger/control.d/remote file with the following content:

   # DO NOT REMOVE OR EDIT THE FOLLOWING LINE $version=1.1 192.168.4.13 n n PCP_ARCHIVE_DIR/rhel7u4a -r -T24h20m -c config.rhel7u4a 192.168.4.14 n n PCP_ARCHIVE_DIR/rhel6u10a -r -T24h20m -c config.rhel6u10a 192.168.4.62 n n PCP_ARCHIVE_DIR/rhel8u1a -r -T24h20m -c config.rhel8u1a

   Replace 192.168.4.13, 192.168.4.14 and 192.168.4.62 with the client IP addresses.

   In Red Hat Enterpirse Linux 8.0, 8.1 and 8.2 use the following format for remote hosts in the control file: PCP_LOG_DIR/pmlogger/host_name.

3. Enable the pmcd and pmlogger services:

   # systemctl enable pmcd pmlogger # systemctl restart pmcd pmlogger

Verification steps

• Ensure that you can access the latest archive file from each directory:

  # for i in /var/log/pcp/pmlogger/rhel*/*.0; do pmdumplog -L $i; done Log Label (Log Format Version 2) Performance metrics from host rhel6u10a.local commencing Mon Nov 25 21:55:04.851 2019 ending Mon Nov 25 22:06:04.874 2019 Archive timezone: JST-9 PID for pmlogger: 24002 Log Label (Log Format Version 2) Performance metrics from host rhel7u4a commencing Tue Nov 26 06:49:24.954 2019 ending Tue Nov 26 07:06:24.979 2019 Archive timezone: CET-1 PID for pmlogger: 10941 [..]

  The archive files from the /var/log/pcp/pmlogger/ directory can be used for further analysis and graphing.

7.6. Replaying the PCP log archives with pmrep

After recording the metric data, you can replay the PCP log archives. To export the logs to text files and import them into spreadsheets, use PCP utilities such as pcp2csv, pcp2xml, pmrep or pmlogsummary.

Using the pmrep tool, you can:

• View the log files
• Parse the selected PCP log archive and export the values into an ASCII table
• Extract the entire archive log or only select metric values from the log by specifying individual metrics on the command line

Prerequisites

• PCP is installed. For more information, see Installing and enabling PCP.
• The pmlogger service is enabled. For more information, see Enabling the pmlogger service.

• Install the pcp-system-tools package:

  # yum install pcp-gui

Procedure

• Display the data on the metric:

  $ pmrep --start @3:00am --archive 20211128 --interval 5seconds --samples 10 --output csv disk.dev.write Time,"disk.dev.write-sda","disk.dev.write-sdb" 2021-11-28 03:00:00,, 2021-11-28 03:00:05,4.000,5.200 2021-11-28 03:00:10,1.600,7.600 2021-11-28 03:00:15,0.800,7.100 2021-11-28 03:00:20,16.600,8.400 2021-11-28 03:00:25,21.400,7.200 2021-11-28 03:00:30,21.200,6.800 2021-11-28 03:00:35,21.000,27.600 2021-11-28 03:00:40,12.400,33.800 2021-11-28 03:00:45,9.800,20.600

  The mentioned example displays the data on the disk.dev.write metric collected in an archive at a 5 second interval in comma-separated-value format.

  Replace 20211128 in this example with a filename containing the pmlogger archive you want to display data for.

8.1. Monitoring postfix with pmda-postfix

This procedure describes how to monitor performance metrics of the postfix mail server with pmda-postfix. It helps to check how many emails are received per second.

Prerequisites

• PCP is installed. For more information, see Installing and enabling PCP.
• The pmlogger service is enabled. For more information, see Enabling the pmlogger service.

Procedure

1. Install the following packages:

   1. Install the pcp-system-tools:

      # yum install pcp-system-tools

   2. Install the pmda-postfix package to monitor postfix:

      # yum install pcp-pmda-postfix postfix

   3. Install the logging daemon:

      # yum install rsyslog

   4. Install the mail client for testing:

      # yum install mutt

2. Enable the postfix and rsyslog services:

   # systemctl enable postfix rsyslog # systemctl restart postfix rsyslog

3. Enable the SELinux boolean, so that pmda-postfix can access the required log files:

   # setsebool -P pcp_read_generic_logs=on

4. Install the PMDA:

   # cd /var/lib/pcp/pmdas/postfix/ # ./Install Updating the Performance Metrics Name Space (PMNS) ... Terminate PMDA if already installed ... Updating the PMCD control file, and notifying PMCD ... Waiting for pmcd to terminate ... Starting pmcd ... Check postfix metrics have appeared ... 7 metrics and 58 values

Verification steps

• Verify the pmda-postfix operation:

  echo testmail | mutt root

• Verify the available metrics:

  # pminfo postfix postfix.received postfix.sent postfix.queues.incoming postfix.queues.maildrop postfix.queues.hold postfix.queues.deferred postfix.queues.active

8.2. Visually tracing PCP log archives with the PCP Charts application

After recording metric data, you can replay the PCP log archives as graphs. The metrics are sourced from one or more live hosts with alternative options to use metric data from PCP log archives as a source of historical data. To customize the PCP Charts application interface to display the data from the performance metrics, you can use line plot, bar graphs, or utilization graphs.

Using the PCP Charts application, you can:

• Replay the data in the PCP Charts application application and use graphs to visualize the retrospective data alongside live data of the system.
• Plot performance metric values into graphs.
• Display multiple charts simultaneously.

Prerequisites

• PCP is installed. For more information, see Installing and enabling PCP.
• Logged performance data with the pmlogger. For more information, see Logging performance data with pmlogger.

• Install the pcp-gui package:

  # yum install pcp-gui

Procedure

1. Launch the PCP Charts application from the command line:

   # pmchart

   Figure 8.1. PCP Charts application

   The pmtime server settings are located at the bottom. The start and pause button allows you to control:

   • The interval in which PCP polls the metric data
   • The date and time for the metrics of historical data

2. Click File and then New Chart to select metric from both the local machine and remote machines by specifying their host name or address. Advanced configuration options include the ability to manually set the axis values for the chart, and to manually choose the color of the plots.

3. Record the views created in the PCP Charts application:

   Following are the options to take images or record the views created in the PCP Charts application:

   • Click File and then Export to save an image of the current view.
   • Click Record and then Start to start a recording. Click Record and then Stop to stop the recording. After stopping the recording, the recorded metrics are archived to be viewed later.

4. Optional: In the PCP Charts application, the main configuration file, known as the view, allows the metadata associated with one or more charts to be saved. This metadata describes all chart aspects, including the metrics used and the chart columns. Save the custom view configuration by clicking File and then Save View, and load the view configuration later.

   The following example of the PCP Charts application view configuration file describes a stacking chart graph showing the total number of bytes read and written to the given XFS file system loop1:

   #kmchart version 1 chart title "Filesystem Throughput /loop1" style stacking antialiasing off plot legend "Read rate" metric xfs.read_bytes instance "loop1" plot legend "Write rate" metric xfs.write_bytes instance "loop1"

8.3. Collecting data from SQL server using PCP

With Red Hat Enterprise Linux 8.2 or later, the SQL Server agent is available in Performance Co-Pilot (PCP), which helps you to monitor and analyze database performance issues.

This procedure describes how to collect data for Microsoft SQL Server via pcp on your system.

Prerequisites

• You have installed Microsoft SQL Server for Red Hat Enterprise Linux and established a 'trusted' connection to an SQL server.
• You have installed the Microsoft ODBC driver for SQL Server for Red Hat Enterprise Linux.

Procedure

1. Install PCP:

   # yum install pcp-zeroconf

2. Install packages required for the pyodbc driver:

   # yum install gcc-c++ python3-devel unixODBC-devel # yum install python3-pyodbc

3. Install the mssql agent:

   1. Install the Microsoft SQL Server domain agent for PCP:

      # yum install pcp-pmda-mssql

   2. Edit the /etc/pcp/mssql/mssql.conf file to configure the SQL server account’s username and password for the mssql agent. Ensure that the account you configure has access rights to performance data.

      username: user_name password: user_password

      Replace user_name with the SQL Server account and user_password with the SQL Server user password for this account.

4. Install the agent:

   # cd /var/lib/pcp/pmdas/mssql # ./Install Updating the Performance Metrics Name Space (PMNS) ... Terminate PMDA if already installed ... Updating the PMCD control file, and notifying PMCD ... Check mssql metrics have appeared ... 168 metrics and 598 values [...]

Verification steps

• Using the pcp command, verify if the SQL Server PMDA (mssql) is loaded and running:

  $ pcp Performance Co-Pilot configuration on rhel.local: platform: Linux rhel.local 4.18.0-167.el8.x86_64 #1 SMP Sun Dec 15 01:24:23 UTC 2019 x86_64 hardware: 2 cpus, 1 disk, 1 node, 2770MB RAM timezone: PDT+7 services: pmcd pmproxy pmcd: Version 5.0.2-1, 12 agents, 4 clients pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm mssql jbd2 dm pmlogger: primary logger: /var/log/pcp/pmlogger/rhel.local/20200326.16.31 pmie: primary engine: /var/log/pcp/pmie/rhel.local/pmie.log

• View the complete list of metrics that PCP can collect from the SQL Server:

  # pminfo mssql

• After viewing the list of metrics, you can report the rate of transactions. For example, to report on the overall transaction count per second, over a five second time window:

  # pmval -t 1 -T 5 mssql.databases.transactions
• View the graphical chart of these metrics on your system by using the pmchart command. For more information, see Visually tracing PCP log archives with the PCP Charts application.

9.1. Installing XFS PMDA manually

If the XFS PMDA is not listed in the pcp configuration output, install the PMDA agent manually.

This procedure describes how to manually install the PMDA agent.

Procedure

1. Navigate to the xfs directory:

   # cd /var/lib/pcp/pmdas/xfs/

Verification steps

• Verify that the pmcd process is running on the host and the XFS PMDA is listed as enabled in the configuration:

  # pcp Performance Co-Pilot configuration on workstation: platform: Linux workstation 4.18.0-80.el8.x86_64 #1 SMP Wed Mar 13 12:02:46 UTC 2019 x86_64 hardware: 12 cpus, 2 disks, 1 node, 36023MB RAM timezone: CEST-2 services: pmcd pmcd: Version 4.3.0-1, 8 agents pmda: root pmcd proc xfs linux mmv kvm jbd2

9.2. Examining XFS performance metrics with pminfo

PCP enables XFS PMDA to allow the reporting of certain XFS metrics per each of the mounted XFS file systems. This makes it easier to pinpoint specific mounted file system issues and evaluate performance.

The pminfo command provides per-device XFS metrics for each mounted XFS file system.

This procedure displays a list of all available metrics provided by the XFS PMDA.

Procedure

• Display the list of all available metrics provided by the XFS PMDA:

  # pminfo xfs

• Display information for the individual metrics. The following examples examine specific XFS read and write metrics using the pminfo tool:

  • Display a short description of the xfs.write_bytes metric:

    # pminfo --oneline xfs.write_bytes xfs.write_bytes [number of bytes written in XFS file system write operations]

  • Display a long description of the xfs.read_bytes metric:

    # pminfo --helptext xfs.read_bytes xfs.read_bytes Help: This is the number of bytes read via read(2) system calls to files in XFS file systems. It can be used in conjunction with the read_calls count to calculate the average size of the read operations to file in XFS file systems.

  • Obtain the current performance value of the xfs.read_bytes metric:

    # pminfo --fetch xfs.read_bytes xfs.read_bytes value 4891346238

  • Obtain per-device XFS metrics with pminfo:

    # pminfo --fetch --oneline xfs.perdev.read xfs.perdev.write xfs.perdev.read [number of XFS file system read operations] inst [0 or "loop1"] value 0 inst [0 or "loop2"] value 0 xfs.perdev.write [number of XFS file system write operations] inst [0 or "loop1"] value 86 inst [0 or "loop2"] value 0

9.3. Resetting XFS performance metrics with pmstore

With PCP, you can modify the values of certain metrics, especially if the metric acts as a control variable, such as the xfs.control.reset metric. To modify a metric value, use the pmstore tool.

This procedure describes how to reset XFS metrics using the pmstore tool.

Procedure

1. Display the value of a metric:

   $ pminfo -f xfs.write xfs.write value 325262

2. Reset all the XFS metrics:

   # pmstore xfs.control.reset 1 xfs.control.reset old value=0 new value=1

Verification steps

• View the information after resetting the metric:

  $ pminfo --fetch xfs.write xfs.write value 0

9.4. PCP metric groups for XFS

The following table describes the available PCP metric groups for XFS.

Table 9.1. Metric groups for XFS

9.5. Per-device PCP metric groups for XFS

The following table describes the available per-device PCP metric group for XFS.

Table 9.2. Per-device PCP metric groups for XFS

10.1. Setting up PCP with pcp-zeroconf

This procedure describes how to set up PCP on a system with the pcp-zeroconf package. Once the pcp-zeroconf package is installed, the system records the default set of metrics into archived files.

Procedure

• Install the pcp-zeroconf package:

  # yum install pcp-zeroconf

Verification steps

• Ensure that the pmlogger service is active, and starts archiving the metrics:

  # pcp | grep pmlogger pmlogger: primary logger: /var/log/pcp/pmlogger/localhost.localdomain/20200401.00.12

10.2. Setting up a grafana-server

Grafana generates graphs that are accessible from a browser. The grafana-server is a back-end server for the Grafana dashboard. It listens, by default, on all interfaces, and provides web services accessed through the web browser. The grafana-pcp plugin interacts with the pmproxy protocol in the backend.

This procedure describes how to set up a grafana-server.

Procedure

1. Install the following packages:

   # yum install grafana grafana-pcp

2. Restart and enable the following service:

   # systemctl restart grafana-server # systemctl enable grafana-server

3. Open the server’s firewall for network traffic to the Grafana service.

   # firewall-cmd --permanent --add-service=grafana success # firewall-cmd --reload success

Verification steps

• Ensure that the grafana-server is listening and responding to requests:

  # ss -ntlp | grep 3000 LISTEN 0 128 *:3000 *:* users:(("grafana-server",pid=19522,fd=7))

• Ensure that the grafana-pcp plugin is installed:

  # grafana-cli plugins ls | grep performancecopilot-pcp-app performancecopilot-pcp-app @ 3.1.0

Additional resources

• pmproxy(1) and grafana-server man pages

10.3. Accessing the Grafana web UI

This procedure describes how to access the Grafana web interface.

Using the Grafana web interface, you can:

• add PCP Redis, PCP bpftrace, and PCP Vector data sources
• create dashboard
• view an overview of any useful metrics
• create alerts in PCP Redis

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

1. On the client system, open a browser and access the grafana-server on port 3000, using http://192.0.2.0:3000 link.

   Replace 192.0.2.0 with your machine IP.

2. For the first login, enter admin in both the Email or username and Password field.

   Grafana prompts to set a New password to create a secured account. If you want to set it later, click Skip.

3. From the menu, hover over the   
      Configuration icon and then click Plugins.
4. In the Plugins tab, type performance co-pilot in the Search by name or type text box and then click Performance Co-Pilot (PCP) plugin.
5. In the Plugins / Performance Co-Pilot pane, click Enable.

6. Click Grafana   

      icon. The Grafana Home page is displayed.

   Figure 10.1. Home Dashboard

   The top corner of the screen has a similar   

      icon, but it controls the general Dashboard settings.

7. In the Grafana Home page, click Add your first data source to add PCP Redis, PCP bpftrace, and PCP Vector data sources. For more information on adding data source, see:

   • To add pcp redis data source, view default dashboard, create a panel, and an alert rule, see Creating panels and alert in PCP Redis data source.
   • To add pcp bpftrace data source and view the default dashboard, see Viewing the PCP bpftrace System Analysis dashboard.
   • To add pcp vector data source, view the default dashboard, and to view the vector checklist, see Viewing the PCP Vector Checklist.

8. Optional: From the menu, hover over the admin profile   
      icon to change the Preferences including Edit Profile, Change Password, or to Sign out.

Additional resources

• grafana-cli and grafana-server man pages

10.4. Configuring PCP Redis

This section provides information for configuring PCP Redis data source.

Use the PCP Redis data source to:

• View data archives
• Query time series using pmseries language
• Analyze data across multiple hosts

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

1. Install the redis package:

   # yum install redis

2. Start and enable the following services:

   # systemctl start pmproxy redis # systemctl enable pmproxy redis
3. Mail transfer agent, for example, sendmail or postfix is installed and configured.

4. Ensure that the allow_loading_unsigned_plugins parameter is set to PCP Redis database in the grafana.ini file:

   # vi /etc/grafana/grafana.ini allow_loading_unsigned_plugins = pcp-redis-datasource

5. Restart the grafana-server:

   # systemctl restart grafana-server

Verification steps

• Ensure that the pmproxy and redis are working:

  # pmseries disk.dev.read 2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df

  This command does not return any data if the redis package is not installed.

Additional resources

• pmseries(1) man page

10.5. Creating panels and alert in PCP Redis data source

After adding the PCP Redis data source, you can view the dashboard with an overview of useful metrics, add a query to visualize the load graph, and create alerts that help you to view the system issues after they occur.

Prerequisites

1. The PCP Redis is configured. For more information, see Configuring PCP Redis.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type redis in the Filter by name or type text box and then click PCP Redis.

4. In the Data Sources / PCP Redis pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click → → to see a dashboard with an overview of any useful metrics.

      Figure 10.2. PCP Redis: Host Overview

5. Add a new panel:

   1. From the menu, hover over the   
         → → to add a panel.
   2. In the Query tab, select the PCP Redis from the query list instead of the selected default option and in the text field of A, enter metric, for example, kernel.all.load to visualize the kernel load graph.
   3. Optional: Add Panel title and Description, and update other options from the Settings.
   4. Click Save to apply changes and save the dashboard. Add Dashboard name.

   5. Click Apply to apply changes and go back to the dashboard.

      Figure 10.3. PCP Redis query panel

6. Create an alert rule:

   1. In the PCP Redis query panel, click   
         Alert and then click Create Alert.
   2. Edit the Name, Evaluate query, and For fields from the Rule, and specify the Conditions for your alert.

   3. Click Save to apply changes and save the dashboard. Click Apply to apply changes and go back to the dashboard.

      Figure 10.4. Creating alerts in the PCP Redis panel

   4. Optional: In the same panel, scroll down and click Delete icon to delete the created rule.

   5. Optional: From the menu, click   

         Alerting icon to view the created alert rules with different alert statuses, to edit the alert rule, or to pause the existing rule from the Alert Rules tab.

      To add a notification channel for the created alert rule to receive an alert notification from Grafana, see Adding notification channels for alerts.

10.6. Adding notification channels for alerts

By adding notification channels, you can receive an alert notification from Grafana whenever the alert rule conditions are met and the system needs further monitoring.

You can receive these alerts after selecting any one type from the supported list of notifiers, which includes DingDing, Discord, Email, Google Hangouts Chat, HipChat, Kafka REST Proxy, LINE, Microsoft Teams, OpsGenie, PagerDuty, Prometheus Alertmanager, Pushover, Sensu, Slack, Telegram, Threema Gateway, VictorOps, and webhook.

Prerequisites

1. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.
2. An alert rule is created. For more information, see Creating panels and alert in PCP Redis data source.

3. Configure SMTP and add a valid sender’s email address in the grafana/grafana.ini file:

   # vi /etc/grafana/grafana.ini [smtp] enabled = true from_address =

   Replace by a valid email address.

Procedure

1. From the menu, hover over the   
      → → .

2. In the Add notification channel details pane, perform the following:

   1. Enter your name in the Name text box
   2. Select the communication Type, for example, Email and enter the email address. You can add multiple email addresses using the ; separator.
   3. Optional: Configure Optional Email settings and Notification settings.

3. Click Save.

4. Select a notification channel in the alert rule:

   1. From the menu, hover over the   
         Alerting icon and then click Alert rules.
   2. From the Alert Rules tab, click the created alert rule.
   3. On the Notifications tab, select your notification channel name from the Send to option, and then add an alert message.
   4. Click Apply.

10.7. Setting up authentication between PCP components

You can setup authentication using the scram-sha-256 authentication mechanism, which is supported by PCP through the Simple Authentication Security Layer (SASL) framework.

From Red Hat Enterprise Linux 8.3, PCP supports the scram-sha-256 authentication mechanism.

Procedure

1. Install the sasl framework for the scram-sha-256 authentication mechanism:

   # yum install cyrus-sasl-scram cyrus-sasl-lib

2. Specify the supported authentication mechanism and the user database path in the pmcd.conf file:

   # vi /etc/sasl2/pmcd.conf mech_list: scram-sha-256 sasldb_path: /etc/pcp/passwd.db

3. Create a new user:

   # useradd -r metrics

   Replace metrics by your user name.

4. Add the created user in the user database:

   # saslpasswd2 -a pmcd metrics Password: Again (for verification):

   To add the created user, you are required to enter the metrics account password.

5. Set the permissions of the user database:

   # chown root:pcp /etc/pcp/passwd.db # chmod 640 /etc/pcp/passwd.db

6. Restart the pmcd service:

   # systemctl restart pmcd

Verification steps

• Verify the sasl configuration:

  # pminfo -f -h "pcp://127.0.0.1?username=metrics" disk.dev.read Password: disk.dev.read inst [0 or "sda"] value 19540

10.8. Installing PCP bpftrace

Install the PCP bpftrace agent to introspect a system and to gather metrics from the kernel and user-space tracepoints.

The bpftrace agent uses bpftrace scripts to gather the metrics. The bpftrace scripts use the enhanced Berkeley Packet Filter (eBPF).

This procedure describes how to install a pcp bpftrace.

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.
3. The scram-sha-256 authentication mechanism is configured. For more information, see Setting up authentication between PCP components.

Procedure

1. Install the pcp-pmda-bpftrace package:

   # yum install pcp-pmda-bpftrace

2. Edit the bpftrace.conf file and add the user that you have created in the {setting-up-authentication-between-pcp-components}:

   # vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf [dynamic_scripts] enabled = true auth_enabled = true allowed_users = root,metrics

   Replace metrics by your user name.

3. Install bpftrace PMDA:

   # cd /var/lib/pcp/pmdas/bpftrace/ # ./Install Updating the Performance Metrics Name Space (PMNS) ... Terminate PMDA if already installed ... Updating the PMCD control file, and notifying PMCD ... Check bpftrace metrics have appeared ... 7 metrics and 6 values

   The pmda-bpftrace is now installed, and can only be used after authenticating your user. For more information, see Viewing the PCP bpftrace System Analysis dashboard.

Additional resources

• pmdabpftrace(1) and bpftrace man pages

10.9. Viewing the PCP bpftrace System Analysis dashboard

Using the PCP bpftrace data source, you can access the live data from sources which are not available as normal data from the pmlogger or archives

In the PCP bpftrace data source, you can view the dashboard with an overview of useful metrics.

Prerequisites

1. The PCP bpftrace is installed. For more information, see Installing PCP bpftrace.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type bpftrace in the Filter by name or type text box and then click PCP bpftrace.

4. In the Data Sources / PCP bpftrace pane, perform the following:

   1. Add http://localhost:44322 in the URL field.
   2. Toggle the Basic Auth option and add the created user credentials in the User and Password field.

   3. Click Save & Test.

      Figure 10.5. Adding PCP bpftrace in the data source

   4. Click → → to see a dashboard with an overview of any useful metrics.

      Figure 10.6. PCP bpftrace: System Analysis

10.10. Installing PCP Vector

This procedure describes how to install a pcp vector.

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

1. Install the pcp-pmda-bcc package:

   # yum install pcp-pmda-bcc

2. Install the bcc PMDA:

   # cd /var/lib/pcp/pmdas/bcc # ./Install [Wed Apr 1 00:27:48] pmdabcc(22341) Info: Initializing, currently in 'notready' state. [Wed Apr 1 00:27:48] pmdabcc(22341) Info: Enabled modules: [Wed Apr 1 00:27:48] pmdabcc(22341) Info: ['biolatency', 'sysfork', [...] Updating the Performance Metrics Name Space (PMNS) ... Terminate PMDA if already installed ... Updating the PMCD control file, and notifying PMCD ... Check bcc metrics have appeared ... 1 warnings, 1 metrics and 0 values

Additional resources

• pmdabcc(1) man page

10.11. Viewing the PCP Vector Checklist

The PCP Vector data source displays live metrics and uses the pcp metrics. It analyzes data for individual hosts.

After adding the PCP Vector data source, you can view the dashboard with an overview of useful metrics and view the related troubleshooting or reference links in the checklist.

Prerequisites

1. The PCP Vector is installed. For more information, see Installing PCP Vector.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type vector in the Filter by name or type text box and then click PCP Vector.

4. In the Data Sources / PCP Vector pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click → → to see a dashboard with an overview of any useful metrics.

      Figure 10.7. PCP Vector: Host Overview

5. From the menu, hover over the   

      Performance Co-Pilot plugin and then click PCP Vector Checklist.

   In the PCP checklist, click   

      help or   
      warning icon to view the related troubleshooting or reference links.

   Figure 10.8. Performance Co-Pilot / PCP Vector Checklist

10.12. Troubleshooting Grafana issues

This section describes how to troubleshoot Grafana issues, such as, Grafana does not display any data, the dashboard is black, or similar issues.

Procedure

• Verify that the pmlogger service is up and running by executing the following command:

  $ systemctl status pmlogger

• Verify if files were created or modified to the disk by executing the following command:

  $ ls /var/log/pcp/pmlogger/$(hostname)/ -rlt total 4024 -rw-r--r--. 1 pcp pcp 45996 Oct 13 2019 20191013.20.07.meta.xz -rw-r--r--. 1 pcp pcp 412 Oct 13 2019 20191013.20.07.index -rw-r--r--. 1 pcp pcp 32188 Oct 13 2019 20191013.20.07.0.xz -rw-r--r--. 1 pcp pcp 44756 Oct 13 2019 20191013.20.30-00.meta.xz [..]

• Verify that the pmproxy service is running by executing the following command:

  $ systemctl status pmproxy

• Verify that pmproxy is running, time series support is enabled, and a connection to Redis is established by viewing the /var/log/pcp/pmproxy/pmproxy.log file and ensure that it contains the following text:

  pmproxy(1716) Info: Redis slots, command keys, schema version setup

  Here, 1716 is the PID of pmproxy, which will be different for every invocation of pmproxy.

• Verify if the Redis database contains any keys by executing the following command:

  $ redis-cli dbsize (integer) 34837

• Verify if any PCP metrics are in the Redis database and pmproxy is able to access them by executing the following commands:

  $ pmseries disk.dev.read 2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df $ pmseries "disk.dev.read[count:10]" 2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df [Mon Jul 26 12:21:10.085468000 2021] 117971 70e83e88d4e1857a3a31605c6d1333755f2dd17c [Mon Jul 26 12:21:00.087401000 2021] 117758 70e83e88d4e1857a3a31605c6d1333755f2dd17c [Mon Jul 26 12:20:50.085738000 2021] 116688 70e83e88d4e1857a3a31605c6d1333755f2dd17c [...]$ redis-cli --scan --pattern "*$(pmseries 'disk.dev.read')" pcp:metric.name:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df pcp:values:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df pcp:desc:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df pcp:labelvalue:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df pcp:instances:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df pcp:labelflags:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df

• Verify if there are any errors in the Grafana logs by executing the following command:

  $ journalctl -e -u grafana-server -- Logs begin at Mon 2021-07-26 11:55:10 IST, end at Mon 2021-07-26 12:30:15 IST. -- Jul 26 11:55:17 localhost.localdomain systemd[1]: Starting Grafana instance... Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Starting Grafana" logger=server version=7.3.6 c> Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Config loaded from" logger=settings file=/usr/s> Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Config loaded from" logger=settings file=/etc/g> [...]

11.1. Performance tuning options in the web console

Red Hat Enterprise Linux 8 provides several performance profiles that optimize the system for the following tasks:

• Systems using the desktop
• Throughput performance
• Latency performance
• Network performance
• Low power consumption
• Virtual machines

The tuned service optimizes system options to match the selected profile.

In the web console, you can set which performance profile your system uses.

11.2. Setting a performance profile in the web console

This procedure uses the web console to optimize the system performance for a selected task.

Procedure

1. Log into the RHEL web console. For details, see Logging in to the web console.
2. Click Overview.

3. In the Performance Profile field, click the current performance profile.

4. In the Change Performance Profile dialog box, change the profile if necessary.

5. Click Change Profile.

Verification steps

• The Overview tab now shows the selected performance profile.

11.3. Monitoring performance using the web console

Red Hat’s web console uses the Utilization Saturation and Errors (USE) Method for troubleshooting. The new performance metrics page has a historical view of your data organized chronologically with the newest data at the top.

Here, you can view the events, errors, and graphical representation for resource utilization and saturation.

Prerequisites

1. Make sure the web console is installed and accessible. For details, see Installing the web console.

2. Install the cockpit-pcp package, which enables collecting the performance metrics:

   # yum install cockpit-pcp

Procedure

1. Log into the RHEL 8 web console. For details, see Logging in to the web console.

2. Click Overview.

3. Click View details and history to view the Performance Metrics.

12.1. Available disk schedulers

The following multi-queue disk schedulers are supported in Red Hat Enterprise Linux 8:

none Implements a first-in first-out (FIFO) scheduling algorithm. It merges requests at the generic block layer through a simple last-hit cache. mq-deadline

Attempts to provide a guaranteed latency for requests from the point at which requests reach the scheduler.

The mq-deadline scheduler sorts queued I/O requests into a read or write batch and then schedules them for execution in increasing logical block addressing (LBA) order. By default, read batches take precedence over write batches, because applications are more likely to block on read I/O operations. After mq-deadline processes a batch, it checks how long write operations have been starved of processor time and schedules the next read or write batch as appropriate.

This scheduler is suitable for most use cases, but particularly those in which the write operations are mostly asynchronous.

Targets desktop systems and interactive tasks.

The bfq scheduler ensures that a single application is never using all of the bandwidth. In effect, the storage device is always as responsive as if it was idle. In its default configuration, bfq focuses on delivering the lowest latency rather than achieving the maximum throughput.

bfq is based on cfq code. It does not grant the disk to each process for a fixed time slice but assigns a budget measured in number of sectors to the process.

This scheduler is suitable while copying large files and the system does not become unresponsive in this case.

The scheduler tunes itself to achieve a latency goal by calculating the latencies of every I/O request submitted to the block I/O layer. You can configure the target latencies for read, in the case of cache-misses, and synchronous write requests.

This scheduler is suitable for fast devices, for example NVMe, SSD, or other low latency devices.

12.2. Different disk schedulers for different use cases

Depending on the task that your system performs, the following disk schedulers are recommended as a baseline prior to any analysis and tuning tasks:

Table 12.1. Disk schedulers for different use cases

12.3. The default disk scheduler

Block devices use the default disk scheduler unless you specify another scheduler.

For non-volatile Memory Express (NVMe) block devices specifically, the default scheduler is none and Red Hat recommends not changing this.

The kernel selects a default disk scheduler based on the type of device. The automatically selected scheduler is typically the optimal setting. If you require a different scheduler, Red Hat recommends to use udev rules or the TuneD application to configure it. Match the selected devices and switch the scheduler only for those devices.

12.4. Determining the active disk scheduler

This procedure determines which disk scheduler is currently active on a given block device.

Procedure

• Read the content of the /sys/block/device/queue/scheduler file:

  # cat /sys/block/device/queue/scheduler [mq-deadline] kyber bfq none

  In the file name, replace device with the block device name, for example sdc.

  The active scheduler is listed in square brackets ([ ]).

12.5. Setting the disk scheduler using TuneD

This procedure creates and enables a TuneD profile that sets a given disk scheduler for selected block devices. The setting persists across system reboots.

In the following commands and configuration, replace:

• device with the name of the block device, for example sdf
• selected-scheduler with the disk scheduler that you want to set for the device, for example bfq

Procedure

1. Optional: Select an existing TuneD profile on which your profile will be based. For a list of available profiles, see TuneD profiles distributed with RHEL.

   To see which profile is currently active, use:

   $ tuned-adm active

2. Create a new directory to hold your TuneD profile:

   # mkdir /etc/tuned/my-profile

3. Find the system unique identifier of the selected block device:

   $ udevadm info --query=property --name=/dev/device | grep -E '(WWN|SERIAL)' ID_WWN=0x5002538d00000000_ ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0 ID_SERIAL_SHORT=20120501030900000

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

4. Create the /etc/tuned/my-profile/tuned.conf configuration file. In the file, set the following options:

   1. Optional: Include an existing profile:

      [main] include=existing-profile

   2. Set the selected disk scheduler for the device that matches the WWN identifier:

      [disk] devices_udev_regex=IDNAME=device system unique id elevator=selected-scheduler

      Here:

      • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).

      • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

        To match multiple devices in the devices_udev_regex option, enclose the identifiers in parentheses and separate them with vertical bars:

        devices_udev_regex=(ID_WWN=0x5002538d00000000)|(ID_WWN=0x1234567800000000)

5. Enable your profile:

   # tuned-adm profile my-profile

Verification steps

1. Verify that the TuneD profile is active and applied:

   $ tuned-adm active Current active profile: my-profile$ tuned-adm verify Verification succeeded, current system settings match the preset profile. See tuned log file ('/var/log/tuned/tuned.log') for details.

2. Read the contents of the /sys/block/device/queue/scheduler file:

   # cat /sys/block/device/queue/scheduler [mq-deadline] kyber bfq none

   In the file name, replace device with the block device name, for example sdc.

   The active scheduler is listed in square brackets ([]).

12.6. Setting the disk scheduler using udev rules

This procedure sets a given disk scheduler for specific block devices using udev rules. The setting persists across system reboots.

In the following commands and configuration, replace:

• device with the name of the block device, for example sdf
• selected-scheduler with the disk scheduler that you want to set for the device, for example bfq

Procedure

1. Find the system unique identifier of the block device:

   $ udevadm info --name=/dev/device | grep -E '(WWN|SERIAL)' E: ID_WWN=0x5002538d00000000 E: ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0 E: ID_SERIAL_SHORT=20120501030900000

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

2. Configure the udev rule. Create the /etc/udev/rules.d/99-scheduler.rules file with the following content:

   ACTION=="add|change", SUBSYSTEM=="block", ENV{IDNAME}=="device system unique id", ATTR{queue/scheduler}="selected-scheduler"

   Here:

   • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).
   • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

3. Reload udev rules:

   # udevadm control --reload-rules

4. Apply the scheduler configuration:

   # udevadm trigger --type=devices --action=change

Verification steps

• Verify the active scheduler:

  # cat /sys/block/device/queue/scheduler

12.7. Temporarily setting a scheduler for a specific disk

This procedure sets a given disk scheduler for specific block devices. The setting does not persist across system reboots.

Procedure

• Write the name of the selected scheduler to the /sys/block/device/queue/scheduler file:

  # echo selected-scheduler > /sys/block/device/queue/scheduler

  In the file name, replace device with the block device name, for example sdc.

Verification steps

• Verify that the scheduler is active on the device:

  # cat /sys/block/device/queue/scheduler

13.1. Setting the SMB protocol version

Each new SMB version adds features and improves the performance of the protocol. The recent Windows and Windows Server operating systems always supports the latest protocol version. If Samba also uses the latest protocol version, Windows clients connecting to Samba benefit from the performance improvements. In Samba, the default value of the server max protocol is set to the latest supported stable SMB protocol version.

To always have the latest stable SMB protocol version enabled, do not set the server max protocol parameter. If you set the parameter manually, you will need to modify the setting with each new version of the SMB protocol, to have the latest protocol version enabled.

The following procedure explains how to use the default value in the server max protocol parameter.

Procedure

1. Remove the server max protocol parameter from the [global] section in the /etc/samba/smb.conf file.

2. Reload the Samba configuration

   # smbcontrol all reload-config

13.2. Tuning shares with directories that contain a large number of files

Linux supports case-sensitive file names. For this reason, Samba needs to scan directories for uppercase and lowercase file names when searching or accessing a file. You can configure a share to create new files only in lowercase or uppercase, which improves the performance.

Prerequisites

• Samba is configured as a file server

Procedure

1. Rename all files on the share to lowercase.

   Using the settings in this procedure, files with names other than in lowercase will no longer be displayed.

2. Set the following parameters in the share’s section:

   case sensitive = true default case = lower preserve case = no short preserve case = no

   For details about the parameters, see their descriptions in the smb.conf(5) man page.

3. Verify the /etc/samba/smb.conf file:

   # testparm

4. Reload the Samba configuration:

   # smbcontrol all reload-config

After you applied these settings, the names of all newly created files on this share use lowercase. Because of these settings, Samba no longer needs to scan the directory for uppercase and lowercase, which improves the performance.

13.3. Settings that can have a negative performance impact

By default, the kernel in Red Hat Enterprise Linux is tuned for high network performance. For example, the kernel uses an auto-tuning mechanism for buffer sizes. Setting the socket options parameter in the /etc/samba/smb.conf file overrides these kernel settings. As a result, setting this parameter decreases the Samba network performance in most cases.

To use the optimized settings from the Kernel, remove the socket options parameter from the [global] section in the /etc/samba/smb.conf.

14.1. What influences virtual machine performance

VMs are run as user-space processes on the host. The hypervisor therefore needs to convert the host’s system resources so that the VMs can use them. As a consequence, a portion of the resources is consumed by the conversion, and the VM therefore cannot achieve the same performance efficiency as the host.

The impact of virtualization on system performance

More specific reasons for VM performance loss include:

• Virtual CPUs (vCPUs) are implemented as threads on the host, handled by the Linux scheduler.
• VMs do not automatically inherit optimization features, such as NUMA or huge pages, from the host kernel.
• Disk and network I/O settings of the host might have a significant performance impact on the VM.
• Network traffic typically travels to a VM through a software-based bridge.
• Depending on the host devices and their models, there might be significant overhead due to emulation of particular hardware.

The severity of the virtualization impact on the VM performance is influenced by a variety factors, which include:

• The number of concurrently running VMs.
• The amount of virtual devices used by each VM.
• The device types used by the VMs.

Reducing VM performance loss

RHEL 8 provides a number of features you can use to reduce the negative performance effects of virtualization. Notably:

• The tuned service can automatically optimize the resource distribution and performance of your VMs.
• Block I/O tuning can improve the performances of the VM’s block devices, such as disks.
• NUMA tuning can increase vCPU performance.
• Virtual networking can be optimized in various ways.

Tuning VM performance can have adverse effects on other virtualization functions. For example, it can make migrating the modified VM more difficult.

14.2. Optimizing virtual machine performance using tuned

The tuned utility is a tuning profile delivery mechanism that adapts RHEL for certain workload characteristics, such as requirements for CPU-intensive tasks or storage-network throughput responsiveness. It provides a number of tuning profiles that are pre-configured to enhance performance and reduce power consumption in a number of specific use cases. You can edit these profiles or create new profiles to create performance solutions tailored to your environment, including virtualized environments.

To optimize RHEL 8 for virtualization, use the following profiles:

• For RHEL 8 virtual machines, use the virtual-guest profile. It is based on the generally applicable throughput-performance profile, but also decreases the swappiness of virtual memory.
• For RHEL 8 virtualization hosts, use the virtual-host profile. This enables more aggressive writeback of dirty memory pages, which benefits the host performance.

Procedure

To enable a specific tuned profile:

1. List the available tuned profiles.

   # tuned-adm list Available profiles: - balanced - General non-specialized tuned profile - desktop - Optimize for the desktop use-case [...] - virtual-guest - Optimize for running inside a virtual guest - virtual-host - Optimize for running KVM guests Current active profile: balanced

2. Optional: Create a new tuned profile or edit an existing tuned profile.

   For more information, see Customizing tuned profiles.

3. Activate a tuned profile.

   # tuned-adm profile selected-profile

   • To optimize a virtualization host, use the virtual-host profile.

     # tuned-adm profile virtual-host

   • On a RHEL guest operating system, use the virtual-guest profile.

     # tuned-adm profile virtual-guest

14.3. Configuring virtual machine memory

To improve the performance of a virtual machine (VM), you can assign additional host RAM to the VM. Similarly, you can decrease the amount of memory allocated to a VM so the host memory can be allocated to other VMs or tasks.

To perform these actions, you can use the web console or the command-line interface.

14.4. Optimizing virtual machine I/O performance

The input and output (I/O) capabilities of a virtual machine (VM) can significantly limit the VM’s overall efficiency. To address this, you can optimize a VM’s I/O by configuring block I/O parameters.

14.5. Optimizing virtual machine CPU performance

Much like physical CPUs in host machines, vCPUs are critical to virtual machine (VM) performance. As a result, optimizing vCPUs can have a significant impact on the resource efficiency of your VMs. To optimize your vCPU:

1. Adjust how many host CPUs are assigned to the VM. You can do this using the CLI or the web console.

2. Ensure that the vCPU model is aligned with the CPU model of the host. For example, to set the testguest1 VM to use the CPU model of the host:

   # virt-xml testguest1 --edit --cpu host-model
3. Deactivate kernel same-page merging (KSM).

4. If your host machine uses Non-Uniform Memory Access (NUMA), you can also configure NUMA for its VMs. This maps the host’s CPU and memory processes onto the CPU and memory processes of the VM as closely as possible. In effect, NUMA tuning provides the vCPU with a more streamlined access to the system memory allocated to the VM, which can improve the vCPU processing effectiveness.

   For details, see Configuring NUMA in a virtual machine and Sample vCPU performance tuning scenario.

14.6. Optimizing virtual machine network performance

Due to the virtual nature of a VM’s network interface card (NIC), the VM loses a portion of its allocated host network bandwidth, which can reduce the overall workload efficiency of the VM. The following tips can minimize the negative impact of virtualization on the virtual NIC (vNIC) throughput.

Procedure

Use any of the following methods and observe if it has a beneficial effect on your VM network performance:

Enable the vhost_net module

On the host, ensure the vhost_net kernel feature is enabled:

If the output of this command is blank, enable the vhost_net kernel module:

To set up the multi-queue virtio-net feature for a VM, use the virsh edit command to edit to the XML configuration of the VM. In the XML, add the following to the section, and replace N with the number of vCPUs in the VM, up to 16:

If the VM is running, restart it for the changes to take effect.

Batching network packets

In Linux VM configurations with a long transmission path, batching packets before submitting them to the kernel may improve cache utilization. To set up packet batching, use the following command on the host, and replace tap0 with the name of the network interface that the VMs use:

14.7. Virtual machine performance monitoring tools

To identify what consumes the most VM resources and which aspect of VM performance needs optimization, performance diagnostic tools, both general and VM-specific, can be used.

Default OS performance monitoring tools

For standard performance evaluation, you can use the utilities provided by default by your host and guest operating systems:

• On your RHEL 8 host, as root, use the top utility or the system monitor application, and look for qemu and virt in the output. This shows how much host system resources your VMs are consuming.

  • If the monitoring tool displays that any of the qemu or virt processes consume a large portion of the host CPU or memory capacity, use the perf utility to investigate. For details, see below.
  • In addition, if a vhost_net thread process, named for example vhost_net-1234, is displayed as consuming an excessive amount of host CPU capacity, consider using virtual network optimization features, such as multi-queue virtio-net.

• On the guest operating system, use performance utilities and applications available on the system to evaluate which processes consume the most system resources.

  • On Linux systems, you can use the top utility.
  • On Windows systems, you can use the Task Manager application.

perf kvm

You can use the perf utility to collect and analyze virtualization-specific statistics about the performance of your RHEL 8 host. To do so:

1. On the host, install the perf package:

   # yum install perf

2. Use one of the perf kvm stat commands to display perf statistics for your virtualization host:

   • For real-time monitoring of your hypervisor, use the perf kvm stat live command.
   • To log the perf data of your hypervisor over a period of time, activate the logging using the perf kvm stat record command. After the command is canceled or interrupted, the data is saved in the perf.data.guest file, which can be analyzed using the perf kvm stat report command.

3. Analyze the perf output for types of VM-EXIT events and their distribution. For example, the PAUSE_INSTRUCTION events should be infrequent, but in the following output, the high occurrence of this event suggests that the host CPUs are not handling the running vCPUs well. In such a scenario, consider shutting down some of your active VMs, removing vCPUs from these VMs, or tuning the performance of the vCPUs.

   # perf kvm stat report Analyze events for all VMs, all VCPUs: VM-EXIT Samples Samples% Time% Min Time Max Time Avg time EXTERNAL_INTERRUPT 365634 31.59% 18.04% 0.42us 58780.59us 204.08us ( +- 0.99% ) MSR_WRITE 293428 25.35% 0.13% 0.59us 17873.02us 1.80us ( +- 4.63% ) PREEMPTION_TIMER 276162 23.86% 0.23% 0.51us 21396.03us 3.38us ( +- 5.19% ) PAUSE_INSTRUCTION 189375 16.36% 11.75% 0.72us 29655.25us 256.77us ( +- 0.70% ) HLT 20440 1.77% 69.83% 0.62us 79319.41us 14134.56us ( +- 0.79% ) VMCALL 12426 1.07% 0.03% 1.02us 5416.25us 8.77us ( +- 7.36% ) EXCEPTION_NMI 27 0.00% 0.00% 0.69us 1.34us 0.98us ( +- 3.50% ) EPT_MISCONFIG 5 0.00% 0.00% 5.15us 10.85us 7.88us ( +- 11.67% ) Total Samples:1157497, Total events handled time:413728274.66us.

   Other event types that can signal problems in the output of perf kvm stat include:

   • INSN_EMULATION - suggests suboptimal VM I/O configuration.

For more information on using perf to monitor virtualization performance, see the perf-kvm man page.

numastat

To see the current NUMA configuration of your system, you can use the numastat utility, which is provided by installing the numactl package.

The following shows a host with 4 running VMs, each obtaining memory from multiple NUMA nodes. This is not optimal for vCPU performance, and warrants adjusting:

In contrast, the following shows memory being provided to each VM by a single node, which is significantly more efficient.

15.1. Power management basics

Effective power management is built on the following principles:

An idle CPU should only wake up when needed

Since Red Hat Enterprise Linux 6, the kernel runs tickless, which means the previous periodic timer interrupts have been replaced with on-demand interrupts. Therefore, idle CPUs are allowed to remain idle until a new task is queued for processing, and CPUs that have entered lower power states can remain in these states longer. However, benefits from this feature can be offset if your system has applications that create unnecessary timer events. Polling events, such as checks for volume changes or mouse movement, are examples of such events.

Red Hat Enterprise Linux includes tools using which you can identify and audit applications on the basis of their CPU usage. For more information see, Audit and analysis overview and Tools for auditing.

In many cases, however, this depends on modern hardware and correct BIOS configuration or UEFI on modern systems, including non-x86 architectures. Make sure that you are using the latest official firmware for your systems and that in the power management or device configuration sections of the BIOS the power management features are enabled. Some features to look for include:

• Collaborative Processor Performance Controls (CPPC) support for ARM64
• PowerNV support for IBM Power Systems
• SpeedStep
• PowerNow!
• Cool’n’Quiet
• ACPI (C-state)

• Smart

  If your hardware has support for these features and they are enabled in the BIOS, Red Hat Enterprise Linux uses them by default.

Modern CPUs together with Advanced Configuration and Power Interface (ACPI) provide different power states. The three different states are:

• Sleep (C-states)
• Frequency and voltage (P-states)

• Heat output (T-states or thermal states)

  A CPU running on the lowest sleep state, consumes the least amount of watts, but it also takes considerably more time to wake it up from that state when needed. In very rare cases this can lead to the CPU having to wake up immediately every time it just went to sleep. This situation results in an effectively permanently busy CPU and loses some of the potential power saving if another state had been used.

15.2. Audit and analysis overview

The detailed manual audit, analysis, and tuning of a single system is usually the exception because the time and cost spent to do so typically outweighs the benefits gained from these last pieces of system tuning.

However, performing these tasks once for a large number of nearly identical systems where you can reuse the same settings for all systems can be very useful. For example, consider the deployment of thousands of desktop systems, or an HPC cluster where the machines are nearly identical. Another reason to do auditing and analysis is to provide a basis for comparison against which you can identify regressions or changes in system behavior in the future. The results of this analysis can be very helpful in cases where hardware, BIOS, or software updates happen regularly and you want to avoid any surprises with regard to power consumption. Generally, a thorough audit and analysis gives you a much better idea of what is really happening on a particular system.

Auditing and analyzing a system with regard to power consumption is relatively hard, even with the most modern systems available. Most systems do not provide the necessary means to measure power use via software. Exceptions exist though:

• iLO management console of Hewlett Packard server systems has a power management module that you can access through the web.
• IBM provides a similar solution in their BladeCenter power management module.
• On some Dell systems, the IT Assistant offers power monitoring capabilities as well.

Other vendors are likely to offer similar capabilities for their server platforms, but as can be seen there is no single solution available that is supported by all vendors. Direct measurements of power consumption are often only necessary to maximize savings as far as possible.

15.3. Tools for auditing

Red Hat Enterprise Linux 8 offers tools using which you can perform system auditing and analysis. Most of them can be used as supplementary sources of information in case you want to verify what you have discovered already or in case you need more in-depth information on certain parts.

Many of these tools are used for performance tuning as well, which include:

PowerTOP It identifies specific components of kernel and user-space applications that frequently wake up the CPU. Use the powertop command as root to start the PowerTop tool and powertop --calibrate to calibrate the power estimation engine. For more information on PowerTop, see Managing power consumption with PowerTOP. Diskdevstat and netdevstat

They are SystemTap tools that collect detailed information about the disk activity and network activity of all applications running on a system. Using the collected statistics by these tools, you can identify applications that waste power with many small I/O operations rather than fewer, larger operations. Using the yum install tuned-utils-systemtap kernel-debuginfo command as root, install the diskdevstat and netdevstat tool.

To view the detailed information about the disk and network activity, use:

With these commands, you can specify three parameters: update_interval, total_duration, and display_histogram.

It is provided by the procps-ng package. Using this tool, you can view the detailed information about processes, memory, paging, block I/O, traps, and CPU activity.

To view this information, use:

Using the vmstat -a command, you can display active and inactive memory. For more information on other vmstat options, see the vmstat man page.

It is provided by the sysstat package. This tool is similar to vmstat, but only for monitoring I/O on block devices. It also provides more verbose output and statistics.

To monitor the system I/O, use:

It provides detailed information about how time is spent in the I/O subsystem.

To view this information in human readable format, use:

Here, The first column, 253,0 is the device major and minor tuple. The second column, 1, gives information about the CPU, followed by columns for timestamps and PID of the process issuing the IO process.

The sixth column, Q, shows the event type, the 7th column, W for write operation, the 8th column, 76423384, is the block number, and the + 8 is the number of requested blocks.

The last field, [kworker/u16:1], is the process name.

By default, the blktrace command runs forever until the process is explicitly killed. Use the -w option to specify the run-time duration.

It is provided by the kernel-tools package. It reports on processor topology, frequency, idle power-state statistics, temperature, and power usage on x86-64 processors.

To view this summary, use:

By default, turbostat prints a summary of counter results for the entire screen, followed by counter results every 5 seconds. Specify a different period between counter results with the -i option, for example, execute turbostat -i 10 to print results every 10 seconds instead.

Turbostat is also useful for identifying servers that are inefficient in terms of power usage or idle time. It also helps to identify the rate of system management interrupts (SMIs) occurring on the system. It can also be used to verify the effects of power management tuning.

IT is a collection of tools to examine and tune power saving related features of processors. Use the cpupower command with the frequency-info, frequency-set, idle-info, idle-set, set, info, and monitor options to display and set processor related values.

For example, to view available cpufreq governors, use:

For more information about cpupower, see Viewing CPU related information.

Additional resources

• powertop(1), diskdevstat(8), netdevstat(8), tuned(8), vmstat(8), iostat(1), blktrace(8), blkparse(8), and turbostat(8) man pages
• cpupower(1), cpupower-set(1), cpupower-info(1), cpupower-idle(1), cpupower-frequency-set(1), cpupower-frequency-info(1), and cpupower-monitor(1) man pages

16.1. The purpose of PowerTOP

PowerTOP is a program that diagnoses issues related to power consumption and provides suggestions on how to extend battery lifetime.

The PowerTOP tool can provide an estimate of the total power usage of the system and also individual power usage for each process, device, kernel worker, timer, and interrupt handler. The tool can also identify specific components of kernel and user-space applications that frequently wake up the CPU.

Red Hat Enterprise Linux 8 uses version 2.x of PowerTOP.

16.2. Using PowerTOP

Prerequisites

• To be able to use PowerTOP, make sure that the powertop package has been installed on your system:

  # yum install powertop

16.3. PowerTOP statistics

While it runs, PowerTOP gathers statistics from the system.

PowerTOP's output provides multiple tabs:

• Overview
• Idle stats
• Frequency stats
• Device stats
• Tunables
• WakeUp

You can use the Tab and Shift+Tab keys to cycle through these tabs.

16.3.5. The WakeUp tab

The WakeUp tab displays the device wakeup settings available for users to change as and when required.

Use the up and down keys to move through the available settings, and the enter key to enable or disable a setting.

Figure 16.1. PowerTOP output

16.4. Why Powertop does not display Frequency stats values in some instances

While using the Intel P-State driver, PowerTOP only displays values in the Frequency Stats tab if the driver is in passive mode. But, even in this case, the values may be incomplete.

In total, there are three possible modes of the Intel P-State driver:

• Active mode with Hardware P-States (HWP)
• Active mode without HWP
• Passive mode

Switching to the ACPI CPUfreq driver results in complete information being displayed by PowerTOP. However, it is recommended to keep your system on the default settings.

To see what driver is loaded and in what mode, run:

• intel_pstate is returned if the Intel P-State driver is loaded and in active mode.
• intel_cpufreq is returned if the Intel P-State driver is loaded and in passive mode.
• acpi-cpufreq is returned if the ACPI CPUfreq driver is loaded.

While using the Intel P-State driver, add the following argument to the kernel boot command line to force the driver to run in passive mode:

To disable the Intel P-State driver and use, instead, the ACPI CPUfreq driver, add the following argument to the kernel boot command line:

16.5. Generating an HTML output

Apart from the powertop’s output in terminal, you can also generate an HTML report.

Procedure

• Run the powertop command with the --html option:

  # powertop --html=htmlfile.html

  Replace the htmlfile.html parameter with the required name for the output file.

16.6. Optimizing power consumption

To optimize power consumption, you can use either the powertop service or the powertop2tuned utility.

17.1. Supported cpupower tool commands

The cpupower tool is a collection of tools to examine and tune power saving related features of processors.

The cpupower tool supports the following commands:

idle-info Displays the available idle states and other statistics for the CPU idle driver using the cpupower idle-info command. For more information, see CPU Idle States. idle-set Enables or disables specific CPU idle state using the cpupower idle-set command as root. Use -d to disable and -e to enable a specific CPU idle state. frequency-info Displays the current cpufreq driver and available cpufreq governors using the cpupower frequency-info command. For more information, see CPUfreq drivers, Core CPUfreq Governors, and Intel P-state CPUfreq governors. frequency-set Sets the cpufreq and governors using the cpupower frequency-set command as root. For more information, see Setting up CPUfreq governor. set

Sets processor power saving policies using the cpupower set command as root.

Using the --perf-bias option, you can enable software on supported Intel processors to determine the balance between optimum performance and saving power. Assigned values range from 0 to 15, where 0 is optimum performance and 15 is optimum power efficiency. By default, the --perf-bias option applies to all cores. To apply it only to individual cores, add the --cpu cpulist option.

Displays processor power related and hardware configurations, which you have enabled using the cpupower set command. For example, if you assign the --perf-bias value as 5:

Displays the idle statistics and CPU demands using the cpupower monitor command.

Using the -l option, you can list all available monitors on your system and the -m option to display information related to specific monitors. For example, to monitor information related to the Mperf monitor, use the cpupower monitor -m Mperf command as root.

Additional resources

• cpupower(1), cpupower-idle-info(1), cpupower-idle-set(1), cpupower-frequency-set(1), cpupower-frequency-info(1), cpupower-set(1), cpupower-info(1), and cpupower-monitor(1) man pages

17.2. CPU Idle States

CPUs with the x86 architecture support various states, such as, few parts of the CPU are deactivated or using lower performance settings, known as C-states.

With this state, you can save power by partially deactivating CPUs that are not in use. There is no need to configure the C-state, unlike P-states that require a governor and potentially some set up to avoid undesirable power or performance issues. C-states are numbered from C0 upwards, with higher numbers representing decreased CPU functionality and greater power saving. C-states of a given number are broadly similar across processors, although the exact details of the specific feature sets of the state may vary between processor families. C-states 0–3 are defined as follows:

C0 In this state, the CPU is working and not idle at all. C1, Halt In this state, the processor is not executing any instructions but is typically not in a lower power state. The CPU can continue processing with practically no delay. All processors offering C-states need to support this state. Pentium 4 processors support an enhanced C1 state called C1E that actually is a state for lower power consumption. C2, Stop-Clock In this state, the clock is frozen for this processor but it keeps the complete state for its registers and caches, so after starting the clock again it can immediately start processing again. This is an optional state. C3, Sleep In this state, the processor goes to sleep and does not need to keep its cache up to date. Due to this reason, waking up from this state needs considerably more time than from the C2 state. This is an optional state.

You can view the available idle states and other statistics for the CPUidle driver using the following command:

Intel CPUs with the "Nehalem" microarchitecture features a C6 state, which can reduce the voltage supply of a CPU to zero, but typically reduces power consumption by between 80% and 90%. The kernel in Red Hat Enterprise Linux 8 includes optimizations for this new C-state.

Additional resources

• cpupower(1) and cpupower-idle(1) man pages

17.3. Overview of CPUfreq

One of the most effective ways to reduce power consumption and heat output on your system is CPUfreq, which is supported by x86 and ARM64 architectures in Red Hat Enterprise Linux 8. CPUfreq, also referred to as CPU speed scaling, is the infrastructure in the Linux kernel that enables it to scale the CPU frequency in order to save power.

CPU scaling can be done automatically depending on the system load, in response to Advanced Configuration and Power Interface (ACPI) events, or manually by user-space programs, and it allows the clock speed of the processor to be adjusted on the fly. This enables the system to run at a reduced clock speed to save power. The rules for shifting frequencies, whether to a faster or slower clock speed and when to shift frequencies, are defined by the CPUfreq governor.

You can view the cpufreq information using the cpupower frequency-info command as root.

18.1. Introduction to perf

The perf user-space tool interfaces with the kernel-based subsystem Performance Counters for Linux (PCL). perf is a powerful tool that uses the Performance Monitoring Unit (PMU) to measure, record, and monitor a variety of hardware and software events. perf also supports tracepoints, kprobes, and uprobes.

18.2. Installing perf

This procedure installs the perf user-space tool.

Procedure

• Install the perf tool:

  # yum install perf

18.3. Common perf commands

This section provides an overview of commonly used perf commands.

Commonly used perf commands

Additional resources

• Add the --help option to a subcommand to open the man page.

19.1. The purpose of perf top

The perf top command is used for real time system profiling and functions similarly to the top utility. However, where the top utility generally shows you how much CPU time a given process or thread is using, perf top shows you how much CPU time each specific function uses. In its default state, perf top tells you about functions being used across all CPUs in both the user-space and the kernel-space. To use perf top you need root access.

19.2. Profiling CPU usage with perf top

This procedure activates perf top and profiles CPU usage in real time.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.
• You have root access

Procedure

• Start the perf top monitoring interface:

  # perf top

  The monitoring interface looks similar to the following:

  Samples: 8K of event 'cycles', 2000 Hz, Event count (approx.): 4579432780 lost: 0/0 drop: 0/0 Overhead Shared Object Symbol 2.20% [kernel] [k] do_syscall_64 2.17% [kernel] [k] module_get_kallsym 1.49% [kernel] [k] copy_user_enhanced_fast_string 1.37% libpthread-2.29.so [.] pthread_mutex_lock 1.31% [unknown] [.] 0000000000000000 1.07% [kernel] [k] psi_task_change 1.04% [kernel] [k] switch_mm_irqs_off 0.94% [kernel] [k] fget 0.74% [kernel] [k] entry_SYSCALL_64 0.69% [kernel] [k] syscall_return_via_sysret 0.69% libxul.so [.] 0x000000000113f9b0 0.67% [kernel] [k] kallsyms_expand_symbol.constprop.0 0.65% firefox [.] moz_xmalloc 0.65% libpthread-2.29.so [.] __pthread_mutex_unlock_usercnt 0.60% firefox [.] free 0.60% libxul.so [.] 0x000000000241d1cd 0.60% [kernel] [k] do_sys_poll 0.58% [kernel] [k] menu_select 0.56% [kernel] [k] _raw_spin_lock_irqsave 0.55% perf [.] 0x00000000002ae0f3

  In this example, the kernel function do_syscall_64 is using the most CPU time.

Additional resources

• perf-top(1) man page

19.3. Interpretation of perf top output

The perf top monitoring interface displays the data in several columns:

The "Overhead" column Displays the percent of CPU a given function is using. The "Shared Object" column Displays name of the program or library which is using the function. The "Symbol" column Displays the function name or symbol. Functions executed in the kernel-space are identified by [k] and functions executed in the user-space are identified by [.].

19.4. Why perf displays some function names as raw function addresses

For kernel functions, perf uses the information from the /proc/kallsyms file to map the samples to their respective function names or symbols. For functions executed in the user space, however, you might see raw function addresses because the binary is stripped.

The debuginfo package of the executable must be installed or, if the executable is a locally developed application, the application must be compiled with debugging information turned on (the -g option in GCC) to display the function names or symbols in such a situation.

It is not necessary to re-run the perf record command after installing the debuginfo associated with an executable. Simply re-run the perf report command.

19.5. Enabling debug and source repositories

A standard installation of Red Hat Enterprise Linux does not enable the debug and source repositories. These repositories contain information needed to debug the system components and measure their performance.

Procedure

• Enable the source and debug information package channels:

  # subscription-manager repos --enable rhel-8-for-$(uname -i)-baseos-debug-rpms # subscription-manager repos --enable rhel-8-for-$(uname -i)-baseos-source-rpms # subscription-manager repos --enable rhel-8-for-$(uname -i)-appstream-debug-rpms # subscription-manager repos --enable rhel-8-for-$(uname -i)-appstream-source-rpms

  The $(uname -i) part is automatically replaced with a matching value for architecture of your system:

  Architecture name	Value
  64-bit Intel and AMD	x86_64
  64-bit ARM	aarch64
  IBM POWER	ppc64le
  64-bit IBM Z	s390x

19.6. Getting debuginfo packages for an application or library using GDB

Debugging information is required to debug code. For code that is installed from a package, the GNU Debugger (GDB) automatically recognizes missing debug information, resolves the package name and provides concrete advice on how to get the package.

Prerequisites

• The application or library you want to debug must be installed on the system.
• GDB and the debuginfo-install tool must be installed on the system. For details, see Setting up to debug applications.
• Channels providing debuginfo and debugsource packages must be configured and enabled on the system.

Procedure

1. Start GDB attached to the application or library you want to debug. GDB automatically recognizes missing debugging information and suggests a command to run.

   $ gdb -q /bin/ls Reading symbols from /bin/ls...Reading symbols from .gnu_debugdata for /usr/bin/ls...(no debugging symbols found)...done. (no debugging symbols found)...done. Missing separate debuginfos, use: dnf debuginfo-install coreutils-8.30-6.el8.x86_64 (gdb)

2. Exit GDB: type q and confirm with Enter.

   (gdb) q

3. Run the command suggested by GDB to install the required debuginfo packages:

   # dnf debuginfo-install coreutils-8.30-6.el8.x86_64

   The dnf package management tool provides a summary of the changes, asks for confirmation and once you confirm, downloads and installs all the necessary files.

4. In case GDB is not able to suggest the debuginfo package, follow the procedure described in Getting debuginfo packages for an application or library manually.

20.1. The purpose of perf stat

The perf stat command executes a specified command, keeps a running count of hardware and software event occurrences during the commands execution, and generates statistics of these counts. If you do not specify any events, then perf stat counts a set of common hardware and software events.

20.2. Counting events with perf stat

You can use perf stat to count hardware and software event occurrences during command execution and generate statistics of these counts. By default, perf stat operates in per-thread mode.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

• Count the events.

  • Running the perf stat command without root access will only count events occurring in the user space:

    $ perf stat ls

    Example 20.1. Output of perf stat ran without root access

    Desktop Documents Downloads Music Pictures Public Templates Videos Performance counter stats for 'ls': 1.28 msec task-clock:u # 0.165 CPUs utilized 0 context-switches:u # 0.000 M/sec 0 cpu-migrations:u # 0.000 K/sec 104 page-faults:u # 0.081 M/sec 1,054,302 cycles:u # 0.823 GHz 1,136,989 instructions:u # 1.08 insn per cycle 228,531 branches:u # 178.447 M/sec 11,331 branch-misses:u # 4.96% of all branches 0.007754312 seconds time elapsed 0.000000000 seconds user 0.007717000 seconds sys

    As you can see in the previous example, when perf stat runs without root access the event names are followed by :u, indicating that these events were counted only in the user-space.

  • To count both user-space and kernel-space events, you must have root access when running perf stat:

    # perf stat ls

    Example 20.2. Output of perf stat ran with root access

    Desktop Documents Downloads Music Pictures Public Templates Videos Performance counter stats for 'ls': 3.09 msec task-clock # 0.119 CPUs utilized 18 context-switches # 0.006 M/sec 3 cpu-migrations # 0.969 K/sec 108 page-faults # 0.035 M/sec 6,576,004 cycles # 2.125 GHz 5,694,223 instructions # 0.87 insn per cycle 1,092,372 branches # 352.960 M/sec 31,515 branch-misses # 2.89% of all branches 0.026020043 seconds time elapsed 0.000000000 seconds user 0.014061000 seconds sys

    • By default, perf stat operates in per-thread mode. To change to CPU-wide event counting, pass the -a option to perf stat. To count CPU-wide events, you need root access:

      # perf stat -a ls

Additional resources

• perf-stat(1) man page

20.3. Interpretation of perf stat output

perf stat executes a specified command and counts event occurrences during the commands execution and displays statistics of these counts in three columns:

1. The number of occurrences counted for a given event
2. The name of the event that was counted

3. When related metrics are available, a ratio or percentage is displayed after the hash sign (#) in the right-most column.

   For example, when running in default mode, perf stat counts both cycles and instructions and, therefore, calculates and displays instructions per cycle in the right-most column. You can see similar behavior with regard to branch-misses as a percent of all branches since both events are counted by default.

20.4. Attaching perf stat to a running process

You can attach perf stat to a running process. This will instruct perf stat to count event occurrences only in the specified processes during the execution of a command.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

• Attach perf stat to a running process:

  $ perf stat -p ID1,ID2 sleep seconds

  The previous example counts events in the processes with the IDs of ID1 and ID2 for a time period of seconds seconds as dictated by using the sleep command.

Additional resources

• perf-stat(1) man page

21.1. The purpose of perf record

The perf record command samples performance data and stores it in a file, perf.data, which can be read and visualized with other perf commands. perf.data is generated in the current directory and can be accessed at a later time, possibly on a different machine.

If you do not specify a command for perf record to record during, it will record until you manually stop the process by pressing Ctrl+C. You can attach perf record to specific processes by passing the -p option followed by one or more process IDs. You can run perf record without root access, however, doing so will only sample performance data in the user space. In the default mode, perf record uses CPU cycles as the sampling event and operates in per-thread mode with inherit mode enabled.

21.2. Recording a performance profile without root access

You can use perf record without root access to sample and record performance data in the user-space only.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

• Sample and record the performance data:

  $ perf record command

  Replace command with the command you want to sample data during. If you do not specify a command, then perf record will sample data until you manually stop it by pressing Ctrl+C.

Additional resources

• perf-record(1) man page

21.3. Recording a performance profile with root access

You can use perf record with root access to sample and record performance data in both the user-space and the kernel-space simultaneously.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.
• You have root access.

Procedure

• Sample and record the performance data:

  # perf record command

  Replace command with the command you want to sample data during. If you do not specify a command, then perf record will sample data until you manually stop it by pressing Ctrl+C.

Additional resources

• perf-record(1) man page

21.4. Recording a performance profile in per-CPU mode

You can use perf record in per-CPU mode to sample and record performance data in both and user-space and the kernel-space simultaneously across all threads on a monitored CPU. By default, per-CPU mode monitors all online CPUs.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

• Sample and record the performance data:

  # perf record -a command

  Replace command with the command you want to sample data during. If you do not specify a command, then perf record will sample data until you manually stop it by pressing Ctrl+C.

Additional resources

• perf-record(1) man page

21.5. Capturing call graph data with perf record

You can configure the perf record tool so that it records which function is calling other functions in the performance profile. This helps to identify a bottleneck if several processes are calling the same function.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

• Sample and record performance data with the --call-graph option:

  $ perf record --call-graph method command
  • Replace command with the command you want to sample data during. If you do not specify a command, then perf record will sample data until you manually stop it by pressing Ctrl+C.

  • Replace method with one of the following unwinding methods:

    fp Uses the frame pointer method. Depending on compiler optimization, such as with binaries built with the GCC option --fomit-frame-pointer, this may not be able to unwind the stack. dwarf Uses DWARF Call Frame Information to unwind the stack. lbr Uses the last branch record hardware on Intel processors.

Additional resources

• perf-record(1) man page

21.6. Analyzing perf.data with perf report

You can use perf report to display and analyze a perf.data file.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.
• There is a perf.data file in the current directory.
• If the perf.data file was created with root access, you need to run perf report with root access too.

Procedure

• Display the contents of the perf.data file for further analysis:

  # perf report

  This command displays output similar to the following:

  Samples: 2K of event 'cycles', Event count (approx.): 235462960 Overhead Command Shared Object Symbol 2.36% kswapd0 [kernel.kallsyms] [k] page_vma_mapped_walk 2.13% sssd_kcm libc-2.28.so [.] memset_avx2_erms 2.13% perf [kernel.kallsyms] [k] smp_call_function_single 1.53% gnome-shell libc-2.28.so [.] strcmp_avx2 1.17% gnome-shell libglib-2.0.so.0.5600.4 [.] g_hash_table_lookup 0.93% Xorg libc-2.28.so [.] memmove_avx_unaligned_erms 0.89% gnome-shell libgobject-2.0.so.0.5600.4 [.] g_object_unref 0.87% kswapd0 [kernel.kallsyms] [k] page_referenced_one 0.86% gnome-shell libc-2.28.so [.] memmove_avx_unaligned_erms 0.83% Xorg [kernel.kallsyms] [k] alloc_vmap_area 0.63% gnome-shell libglib-2.0.so.0.5600.4 [.] g_slice_alloc 0.53% gnome-shell libgirepository-1.0.so.1.0.0 [.] g_base_info_unref 0.53% gnome-shell ld-2.28.so [.] _dl_find_dso_for_object 0.49% kswapd0 [kernel.kallsyms] [k] vma_interval_tree_iter_next 0.48% gnome-shell libpthread-2.28.so [.] pthread_getspecific 0.47% gnome-shell libgirepository-1.0.so.1.0.0 [.] 0x0000000000013b1d 0.45% gnome-shell libglib-2.0.so.0.5600.4 [.] g_slice_free1 0.45% gnome-shell libgobject-2.0.so.0.5600.4 [.] g_type_check_instance_is_fundamentally_a 0.44% gnome-shell libc-2.28.so [.] malloc 0.41% swapper [kernel.kallsyms] [k] apic_timer_interrupt 0.40% gnome-shell ld-2.28.so [.] _dl_lookup_symbol_x 0.39% kswapd0 [kernel.kallsyms] [k] raw_callee_save___pv_queued_spin_unlock

Additional resources

• perf-report(1) man page

21.7. Interpretation of perf report output

The table displayed by running the perf report command sorts the data into several columns:

The 'Overhead' column Indicates what percentage of overall samples were collected in that particular function. The 'Command' column Tells you which process the samples were collected from. The 'Shared Object' column Displays the name of the ELF image where the samples come from (the name [kernel.kallsyms] is used when the samples come from the kernel). The 'Symbol' column Displays the function name or symbol.

In default mode, the functions are sorted in descending order with those with the highest overhead displayed first.

21.8. Generating a perf.data file that is readable on a different device

You can use the perf tool to record performance data into a perf.data file to be analyzed on a different device.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.
• The kernel debuginfo package is installed. For more information, see Getting debuginfo packages for an application or library using GDB.

Procedure

1. Capture performance data you are interested in investigating further:

   # perf record -a --call-graph fp sleep seconds

   This example would generate a perf.data over the entire system for a period of seconds seconds as dictated by the use of the sleep command. It would also capture call graph data using the frame pointer method.

2. Generate an archive file containing debug symbols of the recorded data:

   # perf archive

Verification steps

• Verify that the archive file has been generated in your current active directory:

  # ls perf.data*

  The output will display every file in your current directory that begins with perf.data. The archive file will be named either:

  perf.data.tar.gz

  or

  perf.data.tar.bz2

21.9. Analyzing a perf.data file that was created on a different device

You can use the perf tool to analyze a perf.data file that was generated on a different device.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.
• A perf.data file and associated archive file generated on a different device are present on the current device being used.

Procedure

1. Copy both the perf.data file and the archive file into your current active directory.

2. Extract the archive file into ~/.debug:

   # mkdir -p ~/.debug # tar xf perf.data.tar.bz2 -C ~/.debug

   The archive file might also be named perf.data.tar.gz.

3. Open the perf.data file for further analysis:

   # perf report

21.10. Why perf displays some function names as raw function addresses

For kernel functions, perf uses the information from the /proc/kallsyms file to map the samples to their respective function names or symbols. For functions executed in the user space, however, you might see raw function addresses because the binary is stripped.

The debuginfo package of the executable must be installed or, if the executable is a locally developed application, the application must be compiled with debugging information turned on (the -g option in GCC) to display the function names or symbols in such a situation.

It is not necessary to re-run the perf record command after installing the debuginfo associated with an executable. Simply re-run the perf report command.

21.11. Enabling debug and source repositories

A standard installation of Red Hat Enterprise Linux does not enable the debug and source repositories. These repositories contain information needed to debug the system components and measure their performance.

Procedure

• Enable the source and debug information package channels:

  # subscription-manager repos --enable rhel-8-for-$(uname -i)-baseos-debug-rpms # subscription-manager repos --enable rhel-8-for-$(uname -i)-baseos-source-rpms # subscription-manager repos --enable rhel-8-for-$(uname -i)-appstream-debug-rpms # subscription-manager repos --enable rhel-8-for-$(uname -i)-appstream-source-rpms

  The $(uname -i) part is automatically replaced with a matching value for architecture of your system:

  Architecture name	Value
  64-bit Intel and AMD	x86_64
  64-bit ARM	aarch64
  IBM POWER	ppc64le
  64-bit IBM Z	s390x

21.12. Getting debuginfo packages for an application or library using GDB

Debugging information is required to debug code. For code that is installed from a package, the GNU Debugger (GDB) automatically recognizes missing debug information, resolves the package name and provides concrete advice on how to get the package.

Prerequisites

• The application or library you want to debug must be installed on the system.
• GDB and the debuginfo-install tool must be installed on the system. For details, see Setting up to debug applications.
• Channels providing debuginfo and debugsource packages must be configured and enabled on the system.

Procedure

1. Start GDB attached to the application or library you want to debug. GDB automatically recognizes missing debugging information and suggests a command to run.

   $ gdb -q /bin/ls Reading symbols from /bin/ls...Reading symbols from .gnu_debugdata for /usr/bin/ls...(no debugging symbols found)...done. (no debugging symbols found)...done. Missing separate debuginfos, use: dnf debuginfo-install coreutils-8.30-6.el8.x86_64 (gdb)

2. Exit GDB: type q and confirm with Enter.

   (gdb) q

3. Run the command suggested by GDB to install the required debuginfo packages:

   # dnf debuginfo-install coreutils-8.30-6.el8.x86_64

   The dnf package management tool provides a summary of the changes, asks for confirmation and once you confirm, downloads and installs all the necessary files.

4. In case GDB is not able to suggest the debuginfo package, follow the procedure described in Getting debuginfo packages for an application or library manually.

22.1. Displaying which CPU events were counted on with perf stat

You can use perf stat to display which CPU events were counted on by disabling CPU count aggregation. You must count events in system-wide mode by using the -a flag in order to use this functionality.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

• Count the events with CPU count aggregation disabled:

  # perf stat -a -A sleep seconds

  The previous example displays counts of a default set of common hardware and software events recorded over a time period of seconds seconds, as dictated by using the sleep command, over each individual CPU in ascending order, starting with CPU0. As such, it may be useful to specify an event such as cycles:

  # perf stat -a -A -e cycles sleep seconds

22.2. Displaying which CPU samples were taken on with perf report

The perf record command samples performance data and stores this data in a perf.data file which can be read with the perf report command. The perf record command always records which CPU samples were taken on. You can configure perf report to display this information.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.
• There is a perf.data file created with perf record in the current directory. If the perf.data file was created with root access, you need to run perf report with root access too.

Procedure

• Display the contents of the perf.data file for further analysis while sorting by CPU:

  # perf report --sort cpu

  • You can sort by CPU and command to display more detailed information about where CPU time is being spent:

    # perf report --sort cpu,comm

    This example will list commands from all monitored CPUs by total overhead in descending order of overhead usage and identify the CPU the command was executed on.

22.3. Displaying specific CPUs during profiling with perf top

You can configure perf top to display specific CPUs and their relative usage while profiling your system in real time.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

• Start the perf top interface while sorting by CPU:

  # perf top --sort cpu

  This example will list CPUs and their respective overhead in descending order of overhead usage in real time.

  • You can sort by CPU and command for more detailed information of where CPU time is being spent:

    # perf top --sort cpu,comm

    This example will list commands by total overhead in descending order of overhead usage and identify the CPU the command was executed on in real time.

22.4. Monitoring specific CPUs with perf record and perf report

You can configure perf record to only sample specific CPUs of interest and analyze the generated perf.data file with perf report for further analysis.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

1. Sample and record the performance data in the specific CPU’s, generating a perf.data file:

   • Using a comma separated list of CPUs:

     # perf record -C 0,1 sleep seconds

     The previous example samples and records data in CPUs 0 and 1 for a period of seconds seconds as dictated by the use of the sleep command.

   • Using a range of CPUs:

     # perf record -C 0-2 sleep seconds

     The previous example samples and records data in all CPUs from CPU 0 to 2 for a period of seconds seconds as dictated by the use of the sleep command.

2. Display the contents of the perf.data file for further analysis:

   # perf report

   This example will display the contents of perf.data. If you are monitoring several CPUs and want to know which CPU data was sampled on, see Displaying which CPU samples were taken on with perf report.

23.1. Attaching perf record to a running process

You can attach perf record to a running process. This will instruct perf record to only sample and record performance data in the specified processes.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

• Attach perf record to a running process:

  $ perf record -p ID1,ID2 sleep seconds

  The previous example samples and records performance data of the processes with the process ID’s ID1 and ID2 for a time period of seconds seconds as dictated by using the sleep command. You can also configure perf to record events in specific threads:

  $ perf record -t ID1,ID2 sleep seconds

  When using the -t flag and stipulating thread ID’s, perf disables inheritance by default. You can enable inheritance by adding the --inherit option.

23.2. Capturing call graph data with perf record

You can configure the perf record tool so that it records which function is calling other functions in the performance profile. This helps to identify a bottleneck if several processes are calling the same function.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

• Sample and record performance data with the --call-graph option:

  $ perf record --call-graph method command
  • Replace command with the command you want to sample data during. If you do not specify a command, then perf record will sample data until you manually stop it by pressing Ctrl+C.

  • Replace method with one of the following unwinding methods:

    fp Uses the frame pointer method. Depending on compiler optimization, such as with binaries built with the GCC option --fomit-frame-pointer, this may not be able to unwind the stack. dwarf Uses DWARF Call Frame Information to unwind the stack. lbr Uses the last branch record hardware on Intel processors.

Additional resources

• perf-record(1) man page

23.3. Analyzing perf.data with perf report

You can use perf report to display and analyze a perf.data file.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.
• There is a perf.data file in the current directory.
• If the perf.data file was created with root access, you need to run perf report with root access too.

Procedure

• Display the contents of the perf.data file for further analysis:

  # perf report

  This command displays output similar to the following:

  Samples: 2K of event 'cycles', Event count (approx.): 235462960 Overhead Command Shared Object Symbol 2.36% kswapd0 [kernel.kallsyms] [k] page_vma_mapped_walk 2.13% sssd_kcm libc-2.28.so [.] memset_avx2_erms 2.13% perf [kernel.kallsyms] [k] smp_call_function_single 1.53% gnome-shell libc-2.28.so [.] strcmp_avx2 1.17% gnome-shell libglib-2.0.so.0.5600.4 [.] g_hash_table_lookup 0.93% Xorg libc-2.28.so [.] memmove_avx_unaligned_erms 0.89% gnome-shell libgobject-2.0.so.0.5600.4 [.] g_object_unref 0.87% kswapd0 [kernel.kallsyms] [k] page_referenced_one 0.86% gnome-shell libc-2.28.so [.] memmove_avx_unaligned_erms 0.83% Xorg [kernel.kallsyms] [k] alloc_vmap_area 0.63% gnome-shell libglib-2.0.so.0.5600.4 [.] g_slice_alloc 0.53% gnome-shell libgirepository-1.0.so.1.0.0 [.] g_base_info_unref 0.53% gnome-shell ld-2.28.so [.] _dl_find_dso_for_object 0.49% kswapd0 [kernel.kallsyms] [k] vma_interval_tree_iter_next 0.48% gnome-shell libpthread-2.28.so [.] pthread_getspecific 0.47% gnome-shell libgirepository-1.0.so.1.0.0 [.] 0x0000000000013b1d 0.45% gnome-shell libglib-2.0.so.0.5600.4 [.] g_slice_free1 0.45% gnome-shell libgobject-2.0.so.0.5600.4 [.] g_type_check_instance_is_fundamentally_a 0.44% gnome-shell libc-2.28.so [.] malloc 0.41% swapper [kernel.kallsyms] [k] apic_timer_interrupt 0.40% gnome-shell ld-2.28.so [.] _dl_lookup_symbol_x 0.39% kswapd0 [kernel.kallsyms] [k] raw_callee_save___pv_queued_spin_unlock

Additional resources

• perf-report(1) man page

24.1. Creating uprobes at the function level with perf

You can use the perf tool to create dynamic tracepoints at arbitrary points in a process or application. These tracepoints can then be used in conjunction with other perf tools such as perf stat and perf record to better understand the process or applications behavior.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

1. Create the uprobe in the process or application you are interested in monitoring at a location of interest within the process or application:

   # perf probe -x /path/to/executable -a function Added new event: probe_executable:function (on function in /path/to/executable) You can now use it in all perf tools, such as: perf record -e probe_executable:function -aR sleep 1

24.2. Creating uprobes on lines within a function with perf

These tracepoints can then be used in conjunction with other perf tools such as perf stat and perf record to better understand the process or applications behavior.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

• You have gotten the debugging symbols for your executable:

  # objdump -t ./your_executable | head

  To do this, the debuginfo package of the executable must be installed or, if the executable is a locally developed application, the application must be compiled with debugging information, the -g option in GCC.

Procedure

1. View the function lines where you can place a uprobe:

   $ perf probe -x ./your_executable -L main

   Output of this command looks similar to:

   user/my_executable:0> 0 int main(int argc, const char **argv) 1 { int err; const char *cmd; char sbuf[STRERR_BUFSIZE]; /* libsubcmd init */ 7 exec_cmd_init("perf", PREFIX, PERF_EXEC_PATH, EXEC_PATH_ENVIRONMENT); 8 pager_init(PERF_PAGER_ENVIRONMENT);

2. Create the uprobe for the desired function line:

   # perf probe -x ./my_executable main:8 Added new event: probe_my_executable:main_L8 (on main:8 in /home/user/my_executable) You can now use it in all perf tools, such as: perf record -e probe_my_executable:main_L8 -aR sleep 1

24.3. Perf script output of data recorded over uprobes

A common method to analyze data collected using uprobes is using the perf script command to read a perf.data file and display a detailed trace of the recorded workload.

In the perf script example output: * A uprobe is added to the function isprime() in a program called my_prog * a is a function argument added to the uprobe. Alternatively, a could be an arbitrary variable visible in the code scope of where you add your uprobe:

25.1. The purpose of perf mem

The mem subcommand of the perf tool enables the sampling of memory accesses (loads and stores). The perf mem command provides information about memory latency, types of memory accesses, functions causing cache hits and misses, and, by recording the data symbol, the memory locations where these hits and misses occur.

25.2. Sampling memory access with perf mem

This procedure describes how to use the perf mem command to sample memory accesses on your system. The command takes the same options as perf record and perf report as well as some options exclusive to the mem subcommand. The recorded data is stored in a perf.data file in the current directory for later analysis.

Prerequisites

• You have the perf user space tool installed as described in Installing perf.

Procedure

1. Sample the memory accesses:

   # perf mem record -a sleep seconds

   This example samples memory accesses across all CPUs for a period of seconds seconds as dictated by the sleep command. You can replace the sleep command for any command during which you want to sample memory access data. By default, perf mem samples both memory loads and stores. You can select only one memory operation by using the -t option and specifying either "load" or "store" between perf mem and record. For loads, information over the memory hierarchy level, TLB memory accesses, bus snoops, and memory locks is captured.

2. Open the perf.data file for analysis:

   # perf mem report

   If you have used the example commands, the output is:

   Available samples 35k cpu/mem-loads,ldlat=30/P 54k cpu/mem-stores/P

   The cpu/mem-loads,ldlat=30/P line denotes data collected over memory loads and the cpu/mem-stores/P line denotes data collected over memory stores. Highlight the category of interest and press Enter to view the data:

   Samples: 35K of event 'cpu/mem-loads,ldlat=30/P', Event count (approx.): 4067062 Overhead Samples Local Weight Memory access Symbol Shared Object Data Symbol Data Object Snoop TLB access Locked 0.07% 29 98 L1 or L1 hit [.] 0x000000000000a255 libspeexdsp.so.1.5.0 [.] 0x00007f697a3cd0f0 anon None L1 or L2 hit No 0.06% 26 97 L1 or L1 hit [.] 0x000000000000a255 libspeexdsp.so.1.5.0 [.] 0x00007f697a3cd0f0 anon None L1 or L2 hit No 0.06% 25 96 L1 or L1 hit [.] 0x000000000000a255 libspeexdsp.so.1.5.0 [.] 0x00007f697a3cd0f0 anon None L1 or L2 hit No 0.06% 1 2325 Uncached or N/A hit [k] pci_azx_readl [kernel.kallsyms] [k] 0xffffb092c06e9084 [kernel.kallsyms] None L1 or L2 hit No 0.06% 1 2247 Uncached or N/A hit [k] pci_azx_readl [kernel.kallsyms] [k] 0xffffb092c06e8164 [kernel.kallsyms] None L1 or L2 hit No 0.05% 1 2166 L1 or L1 hit [.] 0x00000000038140d6 libxul.so [.] 0x00007ffd7b84b4a8 [stack] None L1 or L2 hit No 0.05% 1 2117 Uncached or N/A hit [k] check_for_unclaimed_mmio [kernel.kallsyms] [k] 0xffffb092c1842300 [kernel.kallsyms] None L1 or L2 hit No 0.05% 22 95 L1 or L1 hit [.] 0x000000000000a255 libspeexdsp.so.1.5.0 [.] 0x00007f697a3cd0f0 anon None L1 or L2 hit No 0.05% 1 1898 L1 or L1 hit [.] 0x0000000002a30e07 libxul.so [.] 0x00007f610422e0e0 anon None L1 or L2 hit No 0.05% 1 1878 Uncached or N/A hit [k] pci_azx_readl [kernel.kallsyms] [k] 0xffffb092c06e8164 [kernel.kallsyms] None L2 miss No 0.04% 18 94 L1 or L1 hit [.] 0x000000000000a255 libspeexdsp.so.1.5.0 [.] 0x00007f697a3cd0f0 anon None L1 or L2 hit No 0.04% 1 1593 Local RAM or RAM hit [.] 0x00000000026f907d libxul.so [.] 0x00007f3336d50a80 anon Hit L2 miss No 0.03% 1 1399 L1 or L1 hit [.] 0x00000000037cb5f1 libxul.so [.] 0x00007fbe81ef5d78 libxul.so None L1 or L2 hit No 0.03% 1 1229 LFB or LFB hit [.] 0x0000000002962aad libxul.so [.] 0x00007fb6f1be2b28 anon None L2 miss No 0.03% 1 1202 LFB or LFB hit [.] __pthread_mutex_lock libpthread-2.29.so [.] 0x00007fb75583ef20 anon None L1 or L2 hit No 0.03% 1 1193 Uncached or N/A hit [k] pci_azx_readl [kernel.kallsyms] [k] 0xffffb092c06e9164 [kernel.kallsyms] None L2 miss No 0.03% 1 1191 L1 or L1 hit [k] azx_get_delay_from_lpib [kernel.kallsyms] [k] 0xffffb092ca7efcf0 [kernel.kallsyms] None L1 or L2 hit No

   Alternatively, you can sort your results to investigate different aspects of interest when displaying the data. For example, to sort data over memory loads by type of memory accesses occurring during the sampling period in descending order of overhead they account for:

   # perf mem -t load report --sort=mem

   For example, the output can be:

   Samples: 35K of event 'cpu/mem-loads,ldlat=30/P', Event count (approx.): 40670 Overhead Samples Memory access 31.53% 9725 LFB or LFB hit 29.70% 12201 L1 or L1 hit 23.03% 9725 L3 or L3 hit 12.91% 2316 Local RAM or RAM hit 2.37% 743 L2 or L2 hit 0.34% 9 Uncached or N/A hit 0.10% 69 I/O or N/A hit 0.02% 825 L3 miss

Additional resources

• perf-mem(1) man page.

25.3. Interpretation of perf mem report output

The table displayed by running the perf mem report command without any modifiers sorts the data into several columns:

The 'Overhead' column Indicates percentage of overall samples collected in that particular function. The 'Samples' column Displays the number of samples accounted for by that row. The 'Local Weight' column Displays the access latency in processor core cycles. The 'Memory Access' column Displays the type of memory access that occurred. The 'Symbol' column Displays the function name or symbol. The 'Shared Object' column Displays the name of the ELF image where the samples come from (the name [kernel.kallsyms] is used when the samples come from the kernel). The 'Data Symbol' column Displays the address of the memory location that row was targeting.

Oftentimes, due to dynamic allocation of memory or stack memory being accessed, the 'Data Symbol' column will display a raw address.

The "Snoop" column Displays bus transactions. The 'TLB Access' column Displays TLB memory accesses. The 'Locked' column Indicates if a function was or was not memory locked.

In default mode, the functions are sorted in descending order with those with the highest overhead displayed first.

26.1. The purpose of perf c2c

The c2c subcommand of the perf tool enables Shared Data Cache-to-Cache (C2C) analysis. You can use the perf c2c command to inspect cache-line contention to detect both true and false sharing.

Cache-line contention occurs when a processor core on a Symmetric Multi Processing (SMP) system modifies data items on the same cache line that is in use by other processors. All other processors using this cache-line must then invalidate their copy and request an updated one. This can lead to degraded performance.

The perf c2c command provides the following information:

• Cache lines where contention has been detected
• Processes reading and writing the data
• Instructions causing the contention
• The Non-Uniform Memory Access (NUMA) nodes involved in the contention

26.2. Detecting cache-line contention with perf c2c

Use the perf c2c command to detect cache-line contention in a system.

The perf c2c command supports the same options as perf record as well as some options exclusive to the c2c subcommand. The recorded data is stored in a perf.data file in the current directory for later analysis.

Prerequisites

• The perf user space tool is installed. For more information, see installing perf.

Procedure

• Use perf c2c to detect cache-line contention:

  # perf c2c record -a sleep seconds

  This example samples and records cache-line contention data across all CPU’s for a period of seconds as dictated by the sleep command. You can replace the sleep command with any command you want to collect cache-line contention data over.

Additional resources

• perf-c2c(1) man page

26.3. Visualizing a perf.data file recorded with perf c2c record

This procedure describes how to visualize the perf.data file, which is recorded using the perf c2c command.

Prerequisites

• The perf user space tool is installed. For more information, see Installing perf.
• A perf.data file recorded using the perf c2c command is available in the current directory. For more information, see Detecting cache-line contention with perf c2c.

Procedure

1. Open the perf.data file for further analysis:

   # perf c2c report --stdio

   This command visualizes the perf.data file into several graphs within the terminal:

   ================================================= Trace Event Information ================================================= Total records : 329219 Locked Load/Store Operations : 14654 Load Operations : 69679 Loads - uncacheable : 0 Loads - IO : 0 Loads - Miss : 3972 Loads - no mapping : 0 Load Fill Buffer Hit : 11958 Load L1D hit : 17235 Load L2D hit : 21 Load LLC hit : 14219 Load Local HITM : 3402 Load Remote HITM : 12757 Load Remote HIT : 5295 Load Local DRAM : 976 Load Remote DRAM : 3246 Load MESI State Exclusive : 4222 Load MESI State Shared : 0 Load LLC Misses : 22274 LLC Misses to Local DRAM : 4.4% LLC Misses to Remote DRAM : 14.6% LLC Misses to Remote cache (HIT) : 23.8% LLC Misses to Remote cache (HITM) : 57.3% Store Operations : 259539 Store - uncacheable : 0 Store - no mapping : 11 Store L1D Hit : 256696 Store L1D Miss : 2832 No Page Map Rejects : 2376 Unable to parse data source : 1 ================================================= Global Shared Cache Line Event Information ================================================= Total Shared Cache Lines : 55 Load HITs on shared lines : 55454 Fill Buffer Hits on shared lines : 10635 L1D hits on shared lines : 16415 L2D hits on shared lines : 0 LLC hits on shared lines : 8501 Locked Access on shared lines : 14351 Store HITs on shared lines : 109953 Store L1D hits on shared lines : 109449 Total Merged records : 126112 ================================================= c2c details ================================================= Events : cpu/mem-loads,ldlat=30/P : cpu/mem-stores/P Cachelines sort on : Remote HITMs Cacheline data groupping : offset,pid,iaddr ================================================= Shared Data Cache Line Table ================================================= # # Total Rmt ----- LLC Load Hitm ----- ---- Store Reference ---- --- Load Dram ---- LLC Total ----- Core Load Hit ----- -- LLC Load Hit -- # Index Cacheline records Hitm Total Lcl Rmt Total L1Hit L1Miss Lcl Rmt Ld Miss Loads FB L1 L2 Llc Rmt # ..... .................. ....... ....... ....... ....... ....... ....... ....... ....... ........ ........ ....... ....... ....... ....... ....... ........ ........ # 0 0x602180 149904 77.09% 12103 2269 9834 109504 109036 468 727 2657 13747 40400 5355 16154 0 2875 529 1 0x602100 12128 22.20% 3951 1119 2832 0 0 0 65 200 3749 12128 5096 108 0 2056 652 2 0xffff883ffb6a7e80 260 0.09% 15 3 12 161 161 0 1 1 15 99 25 50 0 6 1 3 0xffffffff81aec000 157 0.07% 9 0 9 1 0 1 0 7 20 156 50 59 0 27 4 4 0xffffffff81e3f540 179 0.06% 9 1 8 117 97 20 0 10 25 62 11 1 0 24 7 ================================================= Shared Cache Line Distribution Pareto ================================================= # # ----- HITM ----- -- Store Refs -- Data address ---------- cycles ---------- cpu Shared # Num Rmt Lcl L1 Hit L1 Miss Offset Pid Code address rmt hitm lcl hitm load cnt Symbol Object Source:Line Node{cpu list} # ..... ....... ....... ....... ....... .................. ....... .................. ........ ........ ........ ........ ................... .................... ........................... .... # ------------------------------------------------------------- 0 9834 2269 109036 468 0x602180 ------------------------------------------------------------- 65.51% 55.88% 75.20% 0.00% 0x0 14604 0x400b4f 27161 26039 26017 9 [.] read_write_func no_false_sharing.exe false_sharing_example.c:144 0{0-1,4} 1{24-25,120} 2{48,54} 3{169} 0.41% 0.35% 0.00% 0.00% 0x0 14604 0x400b56 18088 12601 26671 9 [.] read_write_func no_false_sharing.exe false_sharing_example.c:145 0{0-1,4} 1{24-25,120} 2{48,54} 3{169} 0.00% 0.00% 24.80% 100.00% 0x0 14604 0x400b61 0 0 0 9 [.] read_write_func no_false_sharing.exe false_sharing_example.c:145 0{0-1,4} 1{24-25,120} 2{48,54} 3{169} 7.50% 9.92% 0.00% 0.00% 0x20 14604 0x400ba7 2470 1729 1897 2 [.] read_write_func no_false_sharing.exe false_sharing_example.c:154 1{122} 2{144} 17.61% 20.89% 0.00% 0.00% 0x28 14604 0x400bc1 2294 1575 1649 2 [.] read_write_func no_false_sharing.exe false_sharing_example.c:158 2{53} 3{170} 8.97% 12.96% 0.00% 0.00% 0x30 14604 0x400bdb 2325 1897 1828 2 [.] read_write_func no_false_sharing.exe false_sharing_example.c:162 0{96} 3{171} ------------------------------------------------------------- 1 2832 1119 0 0 0x602100 ------------------------------------------------------------- 29.13% 36.19% 0.00% 0.00% 0x20 14604 0x400bb3 1964 1230 1788 2 [.] read_write_func no_false_sharing.exe false_sharing_example.c:155 1{122} 2{144} 43.68% 34.41% 0.00% 0.00% 0x28 14604 0x400bcd 2274 1566 1793 2 [.] read_write_func no_false_sharing.exe false_sharing_example.c:159 2{53} 3{170} 27.19% 29.40% 0.00% 0.00% 0x30 14604 0x400be7 2045 1247 2011 2 [.] read_write_func no_false_sharing.exe false_sharing_example.c:163 0{96} 3{171}