kernel-hardening.lists.openwall.com archive mirror
 help / color / mirror / Atom feed
* [PATCH v15 0/3] Add trusted_for(2) (was O_MAYEXEC)
@ 2021-10-12 19:24 Mickaël Salaün
  2021-10-12 19:24 ` [PATCH v15 1/3] fs: Add trusted_for(2) syscall implementation and related sysctl Mickaël Salaün
                   ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: Mickaël Salaün @ 2021-10-12 19:24 UTC (permalink / raw)
  To: Al Viro, Andrew Morton
  Cc: Mickaël Salaün, Aleksa Sarai, Andy Lutomirski,
	Arnd Bergmann, Casey Schaufler, Christian Brauner,
	Christian Heimes, Deven Bowers, Dmitry Vyukov, Eric Biggers,
	Eric Chiang, Florian Weimer, Geert Uytterhoeven, James Morris,
	Jan Kara, Jann Horn, Jonathan Corbet, Kees Cook,
	Lakshmi Ramasubramanian, Madhavan T . Venkataraman,
	Matthew Garrett, Matthew Wilcox, Miklos Szeredi, Mimi Zohar,
	Paul Moore, Philippe Trébuchet, Scott Shell, Shuah Khan,
	Steve Dower, Steve Grubb, Thibaut Sautereau, Vincent Strubel,
	kernel-hardening, linux-api, linux-fsdevel, linux-integrity,
	linux-kernel, linux-security-module

Hi,

This new patch series, rebase on v5.15-rc4, adds a full syscall
documentation as requested by Andrew Morton.  This cover letter also
briefly describes the syscall.  Andrew, can you please consider to merge
this into your tree?

Overview
========

The final goal of this patch series is to enable the kernel to be a
global policy manager by entrusting processes with access control at
their level.  To reach this goal, two complementary parts are required:
* user space needs to be able to know if it can trust some file
  descriptor content for a specific usage;
* and the kernel needs to make available some part of the policy
  configured by the system administrator.

Primary goal of trusted_for(2)
==============================

This new syscall enables user space to ask the kernel: is this file
descriptor's content trusted to be used for this purpose?  The set of
usage currently only contains execution, but other may follow (e.g.
configuration, sensitive data).  If the kernel identifies the file
descriptor as trustworthy for this usage, user space should then take
this information into account.  The "execution" usage means that the
content of the file descriptor is trusted according to the system policy
to be executed by user space, which means that it interprets the content
or (try to) maps it as executable memory.

A simple system-wide security policy can be set by the system
administrator through a sysctl configuration consistent with the mount
points or the file access rights.  The documentation explains the
prerequisites.

It is important to note that this can only enable to extend access
control managed by the kernel.  Hence it enables current access control
mechanism to be extended and become a superset of what they can
currently control.  Indeed, the security policy could also be delegated
to an LSM, either a MAC system or an integrity system.  For instance,
this is required to close a major IMA measurement/appraisal interpreter
integrity gap by bringing the ability to check the use of scripts [1].
Other uses are expected, such as for magic-links [2], SGX integration
[3], bpffs [4].

Complementary W^X protections can be brought by SELinux, IPE [5] and
trampfd [6].

System call description
=======================

trusted_for(int fd, enum trusted_for_usage usage, u32 flags);

@fd is the file descriptor to check.

@usage identifies the user space usage intended for @fd: only
TRUSTED_FOR_EXECUTION for now, but trusted_for_usage could be extended
to identify other usages (e.g. configuration, sensitive data).

@flags must be 0 for now but it could be used in the future to do
complementary checks (e.g. signature or integrity requirements, origin
of the file).

This system call returns 0 on success, or -EACCES if the kernel policy
denies the specified usage (which should be enforced by the caller).

The first patch contains the full syscall and sysctl documentation.

Prerequisite of its use
=======================

User space needs to adapt to take advantage of this new feature.  For
example, the PEP 578 [7] (Runtime Audit Hooks) enables Python 3.8 to be
extended with policy enforcement points related to code interpretation,
which can be used to align with the PowerShell audit features.
Additional Python security improvements (e.g. a limited interpreter
without -c, stdin piping of code) are on their way [8].

Examples
========

The initial idea comes from CLIP OS 4 and the original implementation
has been used for more than 13 years:
https://github.com/clipos-archive/clipos4_doc
Chrome OS has a similar approach:
https://chromium.googlesource.com/chromiumos/docs/+/master/security/noexec_shell_scripts.md

Userland patches can be found here:
https://github.com/clipos-archive/clipos4_portage-overlay/search?q=O_MAYEXEC
Actually, there is more than the O_MAYEXEC changes (which matches this search)
e.g., to prevent Python interactive execution. There are patches for
Bash, Wine, Java (Icedtea), Busybox's ash, Perl and Python. There are
also some related patches which do not directly rely on O_MAYEXEC but
which restrict the use of browser plugins and extensions, which may be
seen as scripts too:
https://github.com/clipos-archive/clipos4_portage-overlay/tree/master/www-client

An introduction to O_MAYEXEC was given at the Linux Security Summit
Europe 2018 - Linux Kernel Security Contributions by ANSSI:
https://www.youtube.com/watch?v=chNjCRtPKQY&t=17m15s
The "write xor execute" principle was explained at Kernel Recipes 2018 -
CLIP OS: a defense-in-depth OS:
https://www.youtube.com/watch?v=PjRE0uBtkHU&t=11m14s
See also a first LWN article about O_MAYEXEC and a new one about
trusted_for(2) and its background:
* https://lwn.net/Articles/820000/
* https://lwn.net/Articles/832959/

This patch series can be applied on top of v5.10-rc6 .  This can be
tested with CONFIG_SYSCTL.  I would really appreciate constructive
comments on this patch series.

Previous series:
https://lore.kernel.org/r/20211008104840.1733385-1-mic@digikod.net/

[1] https://lore.kernel.org/lkml/1544647356.4028.105.camel@linux.ibm.com/
[2] https://lore.kernel.org/lkml/20190904201933.10736-6-cyphar@cyphar.com/
[3] https://lore.kernel.org/lkml/CALCETrVovr8XNZSroey7pHF46O=kj_c5D9K8h=z2T_cNrpvMig@mail.gmail.com/
[4] https://lore.kernel.org/lkml/CALCETrVeZ0eufFXwfhtaG_j+AdvbzEWE0M3wjXMWVEO7pj+xkw@mail.gmail.com/
[5] https://lore.kernel.org/lkml/20200406221439.1469862-12-deven.desai@linux.microsoft.com/
[6] https://lore.kernel.org/lkml/20200922215326.4603-1-madvenka@linux.microsoft.com/
[7] https://www.python.org/dev/peps/pep-0578/
[8] https://lore.kernel.org/lkml/0c70debd-e79e-d514-06c6-4cd1e021fa8b@python.org/

Regards,

Mickaël Salaün (3):
  fs: Add trusted_for(2) syscall implementation and related sysctl
  arch: Wire up trusted_for(2)
  selftest/interpreter: Add tests for trusted_for(2) policies

 Documentation/admin-guide/sysctl/fs.rst       |  50 +++
 arch/alpha/kernel/syscalls/syscall.tbl        |   1 +
 arch/arm/tools/syscall.tbl                    |   1 +
 arch/arm64/include/asm/unistd.h               |   2 +-
 arch/arm64/include/asm/unistd32.h             |   2 +
 arch/ia64/kernel/syscalls/syscall.tbl         |   1 +
 arch/m68k/kernel/syscalls/syscall.tbl         |   1 +
 arch/microblaze/kernel/syscalls/syscall.tbl   |   1 +
 arch/mips/kernel/syscalls/syscall_n32.tbl     |   1 +
 arch/mips/kernel/syscalls/syscall_n64.tbl     |   1 +
 arch/mips/kernel/syscalls/syscall_o32.tbl     |   1 +
 arch/parisc/kernel/syscalls/syscall.tbl       |   1 +
 arch/powerpc/kernel/syscalls/syscall.tbl      |   1 +
 arch/s390/kernel/syscalls/syscall.tbl         |   1 +
 arch/sh/kernel/syscalls/syscall.tbl           |   1 +
 arch/sparc/kernel/syscalls/syscall.tbl        |   1 +
 arch/x86/entry/syscalls/syscall_32.tbl        |   1 +
 arch/x86/entry/syscalls/syscall_64.tbl        |   1 +
 arch/xtensa/kernel/syscalls/syscall.tbl       |   1 +
 fs/open.c                                     | 110 ++++++
 include/linux/fs.h                            |   1 +
 include/linux/syscalls.h                      |   2 +
 include/uapi/asm-generic/unistd.h             |   4 +-
 include/uapi/linux/trusted-for.h              |  18 +
 kernel/sysctl.c                               |  12 +-
 tools/testing/selftests/Makefile              |   1 +
 .../testing/selftests/interpreter/.gitignore  |   2 +
 tools/testing/selftests/interpreter/Makefile  |  21 +
 tools/testing/selftests/interpreter/config    |   1 +
 .../selftests/interpreter/trust_policy_test.c | 362 ++++++++++++++++++
 30 files changed, 600 insertions(+), 4 deletions(-)
 create mode 100644 include/uapi/linux/trusted-for.h
 create mode 100644 tools/testing/selftests/interpreter/.gitignore
 create mode 100644 tools/testing/selftests/interpreter/Makefile
 create mode 100644 tools/testing/selftests/interpreter/config
 create mode 100644 tools/testing/selftests/interpreter/trust_policy_test.c


base-commit: 9e1ff307c779ce1f0f810c7ecce3d95bbae40896
-- 
2.32.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v15 1/3] fs: Add trusted_for(2) syscall implementation and related sysctl
  2021-10-12 19:24 [PATCH v15 0/3] Add trusted_for(2) (was O_MAYEXEC) Mickaël Salaün
@ 2021-10-12 19:24 ` Mickaël Salaün
  2021-11-05  6:41   ` [fs] a0918006f9: netperf.Throughput_tps -11.6% regression kernel test robot
  2021-10-12 19:24 ` [PATCH v15 2/3] arch: Wire up trusted_for(2) Mickaël Salaün
  2021-10-12 19:24 ` [PATCH v15 3/3] selftest/interpreter: Add tests for trusted_for(2) policies Mickaël Salaün
  2 siblings, 1 reply; 10+ messages in thread
From: Mickaël Salaün @ 2021-10-12 19:24 UTC (permalink / raw)
  To: Al Viro, Andrew Morton
  Cc: Mickaël Salaün, Aleksa Sarai, Andy Lutomirski,
	Arnd Bergmann, Casey Schaufler, Christian Brauner,
	Christian Heimes, Deven Bowers, Dmitry Vyukov, Eric Biggers,
	Eric Chiang, Florian Weimer, Geert Uytterhoeven, James Morris,
	Jan Kara, Jann Horn, Jonathan Corbet, Kees Cook,
	Lakshmi Ramasubramanian, Madhavan T . Venkataraman,
	Matthew Garrett, Matthew Wilcox, Miklos Szeredi, Mimi Zohar,
	Paul Moore, Philippe Trébuchet, Scott Shell, Shuah Khan,
	Steve Dower, Steve Grubb, Thibaut Sautereau, Vincent Strubel,
	kernel-hardening, linux-api, linux-fsdevel, linux-integrity,
	linux-kernel, linux-security-module, Mickaël Salaün

From: Mickaël Salaün <mic@linux.microsoft.com>

The trusted_for() syscall enables user space tasks to check that files
are trusted to be executed or interpreted by user space.  This may allow
script interpreters to check execution permission before reading
commands from a file, or dynamic linkers to allow shared object loading.
This may be seen as a way for a trusted task (e.g. interpreter) to check
the trustworthiness of files (e.g. scripts) before extending its control
flow graph with new ones originating from these files.

The security policy is consistently managed by the kernel through the
new sysctl: fs.trusted_for_policy .  This enables system administrators
to enforce two complementary security policies according to the
installed system: enforce the noexec mount option, and enforce
executable file permission.  Indeed, because of compatibility with
installed systems, only system administrators are able to check that
this new enforcement is in line with the system mount points and file
permissions.

For this to be possible, script interpreters must use trusted_for(2)
with the TRUSTED_FOR_EXECUTION usage.  To be fully effective, these
interpreters also need to handle the other ways to execute code: command
line parameters (e.g., option -e for Perl), module loading (e.g., option
-m for Python), stdin, file sourcing, environment variables,
configuration files, etc.  According to the threat model, it may be
acceptable to allow some script interpreters (e.g. Bash) to interpret
commands from stdin, may it be a TTY or a pipe, because it may not be
enough to (directly) perform syscalls.

Even without enforced security policy, user space interpreters can use
this syscall to try as much as possible to enforce the system policy at
their level, knowing that it will not break anything on running systems
which do not care about this feature.  However, on systems which want
this feature enforced, there will be knowledgeable people (i.e. system
administrator who configured fs.trusted_for_policy deliberately) to
manage it.

Because trusted_for(2) is a mean to enforce a system-wide security
policy (but not application-centric policies), it does not make sense
for user space to check the sysctl value.  Indeed, this new syscall only
enables to extend the system ability to enforce a policy thanks to (some
trusted) user space collaboration.  Moreover, additional security
policies could be managed by LSMs.  This is a best-effort approach from
the application developer point of view:
https://lore.kernel.org/lkml/1477d3d7-4b36-afad-7077-a38f42322238@digikod.net/

trusted_for(2) with TRUSTED_FOR_EXECUTION should not be confused with
the O_EXEC flag (for open) which is intended for execute-only, which
obviously doesn't work for scripts.  However, a similar behavior could
be implemented in user space with O_PATH:
https://lore.kernel.org/lkml/1e2f6913-42f2-3578-28ed-567f6a4bdda1@digikod.net/

Being able to restrict execution also enables to protect the kernel by
restricting arbitrary syscalls that an attacker could perform with a
crafted binary or certain script languages.  It also improves multilevel
isolation by reducing the ability of an attacker to use side channels
with specific code.  These restrictions can natively be enforced for ELF
binaries (with the noexec mount option) but require this kernel
extension to properly handle scripts (e.g. Python, Perl).  To get a
consistent execution policy, additional memory restrictions should also
be enforced (e.g. thanks to SELinux).

This is a new implementation of a patch initially written by
Vincent Strubel for CLIP OS 4:
https://github.com/clipos-archive/src_platform_clip-patches/blob/f5cb330d6b684752e403b4e41b39f7004d88e561/1901_open_mayexec.patch
This patch has been used for more than 13 years with customized script
interpreters.  Some examples (with the original O_MAYEXEC) can be found
here:
https://github.com/clipos-archive/clipos4_portage-overlay/search?q=O_MAYEXEC

Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Jonathan Corbet <corbet@lwn.net>
Co-developed-by: Thibaut Sautereau <thibaut.sautereau@ssi.gouv.fr>
Signed-off-by: Thibaut Sautereau <thibaut.sautereau@ssi.gouv.fr>
Signed-off-by: Mickaël Salaün <mic@linux.microsoft.com>
Acked-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20211012192410.2356090-2-mic@digikod.net
---

Changes since v14:
* Add full syscall documentation (requested by Andrew Morton).

Changes since v13:
* Rename sysctl from "trust_policy" to "trusted_for_policy" (suggested
  by Kees Cook).
* Add Acked-by Kees Cook.

Changes since v12:
* Update inode_permission() call to allign with commit 47291baa8ddf
  ("namei: make permission helpers idmapped mount aware").
* Switch from d_backing_inode(f.file->f_path.dentry) to
  file_inode(f.file).

Changes since v10:
* Add enum definition to syscalls.h .

Changes since v9:
* Rename the syscall to trusted_for(2) and the sysctl to fs.trust_policy
* Add a dedicated enum trusted_for_usage with include/uapi/linux/trusted-for.h
* Remove the extra MAY_INTROSPECTION_EXEC bit.  LSMs can still implement
  this feature themselves.

Changes since v8:
* Add a dedicated syscall introspect_access() (requested by Al Viro).
* Rename MAY_INTERPRETED_EXEC to MAY_INTROSPECTION_EXEC .
* Rename the sysctl fs.interpreted_access to fs.introspection_policy .
* Update documentation.

Changes since v7:
* Replaces openat2/O_MAYEXEC with faccessat2/X_OK/AT_INTERPRETED .
  Switching to an FD-based syscall was suggested by Al Viro and Jann
  Horn.
* Handle special file descriptors.
* Add a compatibility mode for execute/read check.
* Move the sysctl policy from fs/namei.c to fs/open.c for the new
  faccessat2/AT_INTERPRETED.
* Rename the sysctl from fs.open_mayexec_enforce to
  fs.interpreted_access .
* Update documentation accordingly.

Changes since v6:
* Allow opening pipes, block devices and character devices with
  O_MAYEXEC when there is no enforced policy, but forbid any non-regular
  file opened with O_MAYEXEC otherwise (i.e. for any enforced policy).
* Add a paragraph about the non-regular files policy.
* Move path_noexec() calls out of the fast-path (suggested by Kees
  Cook).
* Do not set __FMODE_EXEC for now because of inconsistent behavior:
  https://lore.kernel.org/lkml/202007160822.CCDB5478@keescook/
* Returns EISDIR when opening a directory with O_MAYEXEC.
* Removed Deven Bowers and Kees Cook Reviewed-by tags because of the
  current update.

Changes since v5:
* Remove the static enforcement configuration through Kconfig because it
  makes the code more simple like this, and because the current sysctl
  configuration can only be set with CAP_SYS_ADMIN, the same way mount
  options (i.e. noexec) can be set.  If an harden distro wants to
  enforce a configuration, it should restrict capabilities or sysctl
  configuration.  Furthermore, an LSM can easily leverage O_MAYEXEC to
  fit its need.
* Move checks from inode_permission() to may_open() and make the error
  codes more consistent according to file types (in line with a previous
  commit): opening a directory with O_MAYEXEC returns EISDIR and other
  non-regular file types may return EACCES.
* In may_open(), when OMAYEXEC_ENFORCE_FILE is set, replace explicit
  call to generic_permission() with an artificial MAY_EXEC to avoid
  double calls.  This makes sense especially when an LSM policy forbids
  execution of a file.
* Replace the custom proc_omayexec() with
  proc_dointvec_minmax_sysadmin(), and then replace the CAP_MAC_ADMIN
  check with a CAP_SYS_ADMIN one (suggested by Kees Cook and Stephen
  Smalley).
* Use BIT() (suggested by Kees Cook).
* Rename variables (suggested by Kees Cook).
* Reword the kconfig help.
* Import the documentation patch (suggested by Kees Cook):
  https://lore.kernel.org/lkml/20200505153156.925111-6-mic@digikod.net/
* Update documentation and add LWN.net article.

Changes since v4:
* Add kernel configuration options to enforce O_MAYEXEC at build time,
  and disable the sysctl in such case (requested by James Morris).
* Reword commit message.

Changes since v3:
* Switch back to O_MAYEXEC, but only handle it with openat2(2) which
  checks unknown flags (suggested by Aleksa Sarai). Cf.
  https://lore.kernel.org/lkml/20200430015429.wuob7m5ofdewubui@yavin.dot.cyphar.com/

Changes since v2:
* Replace O_MAYEXEC with RESOLVE_MAYEXEC from openat2(2).  This change
  enables to not break existing application using bogus O_* flags that
  may be ignored by current kernels by using a new dedicated flag, only
  usable through openat2(2) (suggested by Jeff Layton).  Using this flag
  will results in an error if the running kernel does not support it.
  User space needs to manage this case, as with other RESOLVE_* flags.
  The best effort approach to security (for most common distros) will
  simply consists of ignoring such an error and retry without
  RESOLVE_MAYEXEC.  However, a fully controlled system may which to
  error out if such an inconsistency is detected.
* Cosmetic changes.

Changes since v1:
* Set __FMODE_EXEC when using O_MAYEXEC to make this information
  available through the new fanotify/FAN_OPEN_EXEC event (suggested by
  Jan Kara and Matthew Bobrowski):
  https://lore.kernel.org/lkml/20181213094658.GA996@lithium.mbobrowski.org/
* Move code from Yama to the FS subsystem (suggested by Kees Cook).
* Make omayexec_inode_permission() static (suggested by Jann Horn).
* Use mode 0600 for the sysctl.
* Only match regular files (not directories nor other types), which
  follows the same semantic as commit 73601ea5b7b1 ("fs/open.c: allow
  opening only regular files during execve()").
---
 Documentation/admin-guide/sysctl/fs.rst |  50 +++++++++++
 fs/open.c                               | 110 ++++++++++++++++++++++++
 include/linux/fs.h                      |   1 +
 include/linux/syscalls.h                |   2 +
 include/uapi/linux/trusted-for.h        |  18 ++++
 kernel/sysctl.c                         |  12 ++-
 6 files changed, 191 insertions(+), 2 deletions(-)
 create mode 100644 include/uapi/linux/trusted-for.h

diff --git a/Documentation/admin-guide/sysctl/fs.rst b/Documentation/admin-guide/sysctl/fs.rst
index 2a501c9ddc55..e364d6c45790 100644
--- a/Documentation/admin-guide/sysctl/fs.rst
+++ b/Documentation/admin-guide/sysctl/fs.rst
@@ -48,6 +48,7 @@ Currently, these files are in /proc/sys/fs:
 - suid_dumpable
 - super-max
 - super-nr
+- trusted_for_policy
 
 
 aio-nr & aio-max-nr
@@ -382,3 +383,52 @@ Each "watch" costs roughly 90 bytes on a 32bit kernel, and roughly 160 bytes
 on a 64bit one.
 The current default value for  max_user_watches  is the 1/25 (4%) of the
 available low memory, divided for the "watch" cost in bytes.
+
+
+trusted_for_policy
+------------------
+
+An interpreter can call :manpage:`trusted_for(2)` with a
+``TRUSTED_FOR_EXECUTION`` usage to check that opened regular files are expected
+to be executable.  If the file is not identified as executable, then the
+syscall returns -EACCES.  This may allow a script interpreter to check
+executable permission before reading commands from a file, or a dynamic linker
+to only load executable shared objects.  One interesting use case is to enforce
+a "write xor execute" policy through interpreters.
+
+The ability to restrict code execution must be thought as a system-wide policy,
+which first starts by restricting mount points with the ``noexec`` option.
+This option is also automatically applied to special filesystems such as /proc .
+This prevents files on such mount points to be directly executed by the kernel
+or mapped as executable memory (e.g. libraries).  With script interpreters
+using :manpage:`trusted_for(2)`, the executable permission can then be checked
+before reading commands from files.  This makes it possible to enforce the
+``noexec`` at the interpreter level, and thus propagates this security policy
+to scripts.  To be fully effective, these interpreters also need to handle the
+other ways to execute code: command line parameters (e.g., option ``-e`` for
+Perl), module loading (e.g., option ``-m`` for Python), stdin, file sourcing,
+environment variables, configuration files, etc.  According to the threat
+model, it may be acceptable to allow some script interpreters (e.g.  Bash) to
+interpret commands from stdin, may it be a TTY or a pipe, because it may not be
+enough to (directly) perform syscalls.
+
+There are two complementary security policies: enforce the ``noexec`` mount
+option, and enforce executable file permission.  These policies are handled by
+the ``fs.trusted_for_policy`` sysctl (writable only with ``CAP_SYS_ADMIN``) as
+a bitmask:
+
+1 - Mount restriction: checks that the mount options for the underlying VFS
+    mount do not prevent execution.
+
+2 - File permission restriction: checks that the file is marked as
+    executable for the current process (e.g., POSIX permissions, ACLs).
+
+Note that as long as a policy is enforced, checking any non-regular file with
+:manpage:`trusted_for(2)` returns -EACCES (e.g. TTYs, pipe), even when such a
+file is marked as executable or is on an executable mount point.
+
+Code samples can be found in
+tools/testing/selftests/interpreter/trust_policy_test.c and interpreter patches
+(for the original O_MAYEXEC) are available at
+https://github.com/clipos-archive/clipos4_portage-overlay/search?q=O_MAYEXEC .
+See also an overview article: https://lwn.net/Articles/820000/ .
diff --git a/fs/open.c b/fs/open.c
index daa324606a41..c79c138a638c 100644
--- a/fs/open.c
+++ b/fs/open.c
@@ -32,6 +32,8 @@
 #include <linux/ima.h>
 #include <linux/dnotify.h>
 #include <linux/compat.h>
+#include <linux/sysctl.h>
+#include <uapi/linux/trusted-for.h>
 
 #include "internal.h"
 
@@ -480,6 +482,114 @@ SYSCALL_DEFINE2(access, const char __user *, filename, int, mode)
 	return do_faccessat(AT_FDCWD, filename, mode, 0);
 }
 
+#define TRUST_POLICY_EXEC_MOUNT			BIT(0)
+#define TRUST_POLICY_EXEC_FILE			BIT(1)
+
+int sysctl_trusted_for_policy __read_mostly;
+
+/**
+ * sys_trusted_for - Check that a FD is trusted for a specific usage
+ *
+ * @fd: File descriptor to check.
+ * @usage: Identify the user space usage intended for the file descriptor (only
+ *         TRUSTED_FOR_EXECUTION for now).
+ * @flags: Must be 0.
+ *
+ * This system call enables user space to ask the kernel: is this file
+ * descriptor's content trusted to be used for this purpose?  The set of @usage
+ * currently only contains TRUSTED_FOR_EXECUTION, but other may follow (e.g.
+ * configuration, sensitive data).  If the kernel identifies the file
+ * descriptor as trustworthy for this usage, this call returns 0 and the caller
+ * should then take this information into account.
+ *
+ * The execution usage means that the content of the file descriptor is trusted
+ * according to the system policy to be executed by user space, which means
+ * that it interprets the content or (try to) maps it as executable memory.
+ *
+ * A simple system-wide security policy can be set by the system administrator
+ * through a sysctl configuration consistent with the mount points or the file
+ * access rights: Documentation/admin-guide/sysctl/fs.rst
+ *
+ * @flags could be used in the future to do complementary checks (e.g.
+ * signature or integrity requirements, origin of the file).
+ *
+ * Possible returned errors are:
+ *
+ * - EINVAL: unknown @usage or unknown @flags;
+ * - EBADF: @fd is not a file descriptor for the calling thread;
+ * - EACCES: the requested usage is denied (and user space should enforce it).
+ */
+SYSCALL_DEFINE3(trusted_for, const int, fd, const enum trusted_for_usage, usage,
+		const u32, flags)
+{
+	int mask, err = -EACCES;
+	struct fd f;
+	struct inode *inode;
+
+	if (flags)
+		return -EINVAL;
+
+	/* Only handles execution for now. */
+	if (usage != TRUSTED_FOR_EXECUTION)
+		return -EINVAL;
+	mask = MAY_EXEC;
+
+	f = fdget(fd);
+	if (!f.file)
+		return -EBADF;
+	inode = file_inode(f.file);
+
+	/*
+	 * For compatibility reasons, without a defined security policy, we
+	 * must map the execute permission to the read permission.  Indeed,
+	 * from user space point of view, being able to execute data (e.g.
+	 * scripts) implies to be able to read this data.
+	 */
+	if ((mask & MAY_EXEC)) {
+		/*
+		 * If there is a system-wide execute policy enforced, then
+		 * forbids access to non-regular files and special superblocks.
+		 */
+		if ((sysctl_trusted_for_policy & (TRUST_POLICY_EXEC_MOUNT |
+						TRUST_POLICY_EXEC_FILE))) {
+			if (!S_ISREG(inode->i_mode))
+				goto out_fd;
+			/*
+			 * Denies access to pseudo filesystems that will never
+			 * be mountable (e.g. sockfs, pipefs) but can still be
+			 * reachable through /proc/self/fd, or memfd-like file
+			 * descriptors, or nsfs-like files.
+			 *
+			 * According to the selftests, SB_NOEXEC seems to be
+			 * only used by proc and nsfs filesystems.
+			 */
+			if ((f.file->f_path.dentry->d_sb->s_flags &
+						(SB_NOUSER | SB_KERNMOUNT | SB_NOEXEC)))
+				goto out_fd;
+		}
+
+		if ((sysctl_trusted_for_policy & TRUST_POLICY_EXEC_MOUNT) &&
+				path_noexec(&f.file->f_path))
+			goto out_fd;
+		/*
+		 * For compatibility reasons, if the system-wide policy doesn't
+		 * enforce file permission checks, then replaces the execute
+		 * permission request with a read permission request.
+		 */
+		if (!(sysctl_trusted_for_policy & TRUST_POLICY_EXEC_FILE))
+			mask &= ~MAY_EXEC;
+		/* To be executed *by* user space, files must be readable. */
+		mask |= MAY_READ;
+	}
+
+	err = inode_permission(file_mnt_user_ns(f.file), inode,
+			mask | MAY_ACCESS);
+
+out_fd:
+	fdput(f);
+	return err;
+}
+
 SYSCALL_DEFINE1(chdir, const char __user *, filename)
 {
 	struct path path;
diff --git a/include/linux/fs.h b/include/linux/fs.h
index e7a633353fd2..9689b8a22ec5 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -86,6 +86,7 @@ extern int sysctl_protected_symlinks;
 extern int sysctl_protected_hardlinks;
 extern int sysctl_protected_fifos;
 extern int sysctl_protected_regular;
+extern int sysctl_trusted_for_policy;
 
 typedef __kernel_rwf_t rwf_t;
 
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 252243c7783d..8a69a6b1c1ef 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -71,6 +71,7 @@ struct open_how;
 struct mount_attr;
 struct landlock_ruleset_attr;
 enum landlock_rule_type;
+enum trusted_for_usage;
 
 #include <linux/types.h>
 #include <linux/aio_abi.h>
@@ -461,6 +462,7 @@ asmlinkage long sys_fallocate(int fd, int mode, loff_t offset, loff_t len);
 asmlinkage long sys_faccessat(int dfd, const char __user *filename, int mode);
 asmlinkage long sys_faccessat2(int dfd, const char __user *filename, int mode,
 			       int flags);
+asmlinkage long sys_trusted_for(int fd, enum trusted_for_usage usage, u32 flags);
 asmlinkage long sys_chdir(const char __user *filename);
 asmlinkage long sys_fchdir(unsigned int fd);
 asmlinkage long sys_chroot(const char __user *filename);
diff --git a/include/uapi/linux/trusted-for.h b/include/uapi/linux/trusted-for.h
new file mode 100644
index 000000000000..cc4f030c5103
--- /dev/null
+++ b/include/uapi/linux/trusted-for.h
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _UAPI_LINUX_TRUSTED_FOR_H
+#define _UAPI_LINUX_TRUSTED_FOR_H
+
+/**
+ * enum trusted_for_usage - Usage for which a file descriptor is trusted
+ *
+ * Argument of trusted_for(2).
+ */
+enum trusted_for_usage {
+	/**
+	 * @TRUSTED_FOR_EXECUTION: Check that the data read from a file
+	 * descriptor is trusted to be executed or interpreted (e.g. scripts).
+	 */
+	TRUSTED_FOR_EXECUTION = 1,
+};
+
+#endif /* _UAPI_LINUX_TRUSTED_FOR_H */
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index 083be6af29d7..002dc830c165 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -115,6 +115,7 @@ static int sixty = 60;
 
 static int __maybe_unused neg_one = -1;
 static int __maybe_unused two = 2;
+static int __maybe_unused three = 3;
 static int __maybe_unused four = 4;
 static unsigned long zero_ul;
 static unsigned long one_ul = 1;
@@ -936,7 +937,6 @@ static int proc_taint(struct ctl_table *table, int write,
 	return err;
 }
 
-#ifdef CONFIG_PRINTK
 static int proc_dointvec_minmax_sysadmin(struct ctl_table *table, int write,
 				void *buffer, size_t *lenp, loff_t *ppos)
 {
@@ -945,7 +945,6 @@ static int proc_dointvec_minmax_sysadmin(struct ctl_table *table, int write,
 
 	return proc_dointvec_minmax(table, write, buffer, lenp, ppos);
 }
-#endif
 
 /**
  * struct do_proc_dointvec_minmax_conv_param - proc_dointvec_minmax() range checking structure
@@ -3357,6 +3356,15 @@ static struct ctl_table fs_table[] = {
 		.extra1		= SYSCTL_ZERO,
 		.extra2		= &two,
 	},
+	{
+		.procname       = "trusted_for_policy",
+		.data           = &sysctl_trusted_for_policy,
+		.maxlen         = sizeof(int),
+		.mode           = 0600,
+		.proc_handler	= proc_dointvec_minmax_sysadmin,
+		.extra1		= SYSCTL_ZERO,
+		.extra2		= &three,
+	},
 #if defined(CONFIG_BINFMT_MISC) || defined(CONFIG_BINFMT_MISC_MODULE)
 	{
 		.procname	= "binfmt_misc",
-- 
2.32.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v15 2/3] arch: Wire up trusted_for(2)
  2021-10-12 19:24 [PATCH v15 0/3] Add trusted_for(2) (was O_MAYEXEC) Mickaël Salaün
  2021-10-12 19:24 ` [PATCH v15 1/3] fs: Add trusted_for(2) syscall implementation and related sysctl Mickaël Salaün
@ 2021-10-12 19:24 ` Mickaël Salaün
  2021-10-12 19:24 ` [PATCH v15 3/3] selftest/interpreter: Add tests for trusted_for(2) policies Mickaël Salaün
  2 siblings, 0 replies; 10+ messages in thread
From: Mickaël Salaün @ 2021-10-12 19:24 UTC (permalink / raw)
  To: Al Viro, Andrew Morton
  Cc: Mickaël Salaün, Aleksa Sarai, Andy Lutomirski,
	Arnd Bergmann, Casey Schaufler, Christian Brauner,
	Christian Heimes, Deven Bowers, Dmitry Vyukov, Eric Biggers,
	Eric Chiang, Florian Weimer, Geert Uytterhoeven, James Morris,
	Jan Kara, Jann Horn, Jonathan Corbet, Kees Cook,
	Lakshmi Ramasubramanian, Madhavan T . Venkataraman,
	Matthew Garrett, Matthew Wilcox, Miklos Szeredi, Mimi Zohar,
	Paul Moore, Philippe Trébuchet, Scott Shell, Shuah Khan,
	Steve Dower, Steve Grubb, Thibaut Sautereau, Vincent Strubel,
	kernel-hardening, linux-api, linux-fsdevel, linux-integrity,
	linux-kernel, linux-security-module, Mickaël Salaün

From: Mickaël Salaün <mic@linux.microsoft.com>

Wire up trusted_for(2) for all architectures.

Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Kees Cook <keescook@chromium.org>
Signed-off-by: Mickaël Salaün <mic@linux.microsoft.com>
Reviewed-by: Thibaut Sautereau <thibaut.sautereau@ssi.gouv.fr>
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20211012192410.2356090-3-mic@digikod.net
---

Changes since v13:
* Add Reviewed-by Kees Cook.

Changes since v12:
* Update syscall IDs to align with the new ones.

Changes since v11:
* Add Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
* Rebase and leave space for watch_mount(2) and epoll_pwait2(2) from
  -next.

Changes since v9:
* Rename introspect_access(2) to trusted_for(2).
* Increase syscall number to leave space for memfd_secret(2) in -next.

Changes since v7:
* New patch for the new syscall.
* Increase syscall numbers by 2 to leave space for new ones (in
  linux-next): watch_mount(2) and process_madvise(2).
---
 arch/alpha/kernel/syscalls/syscall.tbl      | 1 +
 arch/arm/tools/syscall.tbl                  | 1 +
 arch/arm64/include/asm/unistd.h             | 2 +-
 arch/arm64/include/asm/unistd32.h           | 2 ++
 arch/ia64/kernel/syscalls/syscall.tbl       | 1 +
 arch/m68k/kernel/syscalls/syscall.tbl       | 1 +
 arch/microblaze/kernel/syscalls/syscall.tbl | 1 +
 arch/mips/kernel/syscalls/syscall_n32.tbl   | 1 +
 arch/mips/kernel/syscalls/syscall_n64.tbl   | 1 +
 arch/mips/kernel/syscalls/syscall_o32.tbl   | 1 +
 arch/parisc/kernel/syscalls/syscall.tbl     | 1 +
 arch/powerpc/kernel/syscalls/syscall.tbl    | 1 +
 arch/s390/kernel/syscalls/syscall.tbl       | 1 +
 arch/sh/kernel/syscalls/syscall.tbl         | 1 +
 arch/sparc/kernel/syscalls/syscall.tbl      | 1 +
 arch/x86/entry/syscalls/syscall_32.tbl      | 1 +
 arch/x86/entry/syscalls/syscall_64.tbl      | 1 +
 arch/xtensa/kernel/syscalls/syscall.tbl     | 1 +
 include/uapi/asm-generic/unistd.h           | 4 +++-
 19 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/arch/alpha/kernel/syscalls/syscall.tbl b/arch/alpha/kernel/syscalls/syscall.tbl
index e4a041cd5715..ee33f1631f6d 100644
--- a/arch/alpha/kernel/syscalls/syscall.tbl
+++ b/arch/alpha/kernel/syscalls/syscall.tbl
@@ -488,3 +488,4 @@
 556	common	landlock_restrict_self		sys_landlock_restrict_self
 # 557 reserved for memfd_secret
 558	common	process_mrelease		sys_process_mrelease
+559	common	trusted_for			sys_trusted_for
diff --git a/arch/arm/tools/syscall.tbl b/arch/arm/tools/syscall.tbl
index e842209e135d..93edcdeda698 100644
--- a/arch/arm/tools/syscall.tbl
+++ b/arch/arm/tools/syscall.tbl
@@ -462,3 +462,4 @@
 446	common	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	common	process_mrelease		sys_process_mrelease
+449	common	trusted_for			sys_trusted_for
diff --git a/arch/arm64/include/asm/unistd.h b/arch/arm64/include/asm/unistd.h
index 3cb206aea3db..6bdb5f5db438 100644
--- a/arch/arm64/include/asm/unistd.h
+++ b/arch/arm64/include/asm/unistd.h
@@ -38,7 +38,7 @@
 #define __ARM_NR_compat_set_tls		(__ARM_NR_COMPAT_BASE + 5)
 #define __ARM_NR_COMPAT_END		(__ARM_NR_COMPAT_BASE + 0x800)
 
-#define __NR_compat_syscalls		449
+#define __NR_compat_syscalls		450
 #endif
 
 #define __ARCH_WANT_SYS_CLONE
diff --git a/arch/arm64/include/asm/unistd32.h b/arch/arm64/include/asm/unistd32.h
index 844f6ae58662..3cb7df3a441c 100644
--- a/arch/arm64/include/asm/unistd32.h
+++ b/arch/arm64/include/asm/unistd32.h
@@ -903,6 +903,8 @@ __SYSCALL(__NR_landlock_add_rule, sys_landlock_add_rule)
 __SYSCALL(__NR_landlock_restrict_self, sys_landlock_restrict_self)
 #define __NR_process_mrelease 448
 __SYSCALL(__NR_process_mrelease, sys_process_mrelease)
+#define __NR_trusted_for 449
+__SYSCALL(__NR_trusted_for, sys_trusted_for)
 
 /*
  * Please add new compat syscalls above this comment and update
diff --git a/arch/ia64/kernel/syscalls/syscall.tbl b/arch/ia64/kernel/syscalls/syscall.tbl
index 6fea1844fb95..50970d778005 100644
--- a/arch/ia64/kernel/syscalls/syscall.tbl
+++ b/arch/ia64/kernel/syscalls/syscall.tbl
@@ -369,3 +369,4 @@
 446	common	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	common	process_mrelease		sys_process_mrelease
+449	common	trusted_for			sys_trusted_for
diff --git a/arch/m68k/kernel/syscalls/syscall.tbl b/arch/m68k/kernel/syscalls/syscall.tbl
index 7976dff8f879..caee4759d9f4 100644
--- a/arch/m68k/kernel/syscalls/syscall.tbl
+++ b/arch/m68k/kernel/syscalls/syscall.tbl
@@ -448,3 +448,4 @@
 446	common	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	common	process_mrelease		sys_process_mrelease
+449	common	trusted_for			sys_trusted_for
diff --git a/arch/microblaze/kernel/syscalls/syscall.tbl b/arch/microblaze/kernel/syscalls/syscall.tbl
index 6b0e11362bd2..25761d27aa9e 100644
--- a/arch/microblaze/kernel/syscalls/syscall.tbl
+++ b/arch/microblaze/kernel/syscalls/syscall.tbl
@@ -454,3 +454,4 @@
 446	common	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	common	process_mrelease		sys_process_mrelease
+449	common	trusted_for			sys_trusted_for
diff --git a/arch/mips/kernel/syscalls/syscall_n32.tbl b/arch/mips/kernel/syscalls/syscall_n32.tbl
index 70e32de2bcaa..5e61c18fa3df 100644
--- a/arch/mips/kernel/syscalls/syscall_n32.tbl
+++ b/arch/mips/kernel/syscalls/syscall_n32.tbl
@@ -387,3 +387,4 @@
 446	n32	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	n32	process_mrelease		sys_process_mrelease
+449	n32	trusted_for			sys_trusted_for
diff --git a/arch/mips/kernel/syscalls/syscall_n64.tbl b/arch/mips/kernel/syscalls/syscall_n64.tbl
index 1ca7bc337932..9a4928032039 100644
--- a/arch/mips/kernel/syscalls/syscall_n64.tbl
+++ b/arch/mips/kernel/syscalls/syscall_n64.tbl
@@ -363,3 +363,4 @@
 446	n64	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	n64	process_mrelease		sys_process_mrelease
+449	n64	trusted_for			sys_trusted_for
diff --git a/arch/mips/kernel/syscalls/syscall_o32.tbl b/arch/mips/kernel/syscalls/syscall_o32.tbl
index a61c35edaa74..7fd966e720c2 100644
--- a/arch/mips/kernel/syscalls/syscall_o32.tbl
+++ b/arch/mips/kernel/syscalls/syscall_o32.tbl
@@ -436,3 +436,4 @@
 446	o32	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	o32	process_mrelease		sys_process_mrelease
+449	o32	trusted_for			sys_trusted_for
diff --git a/arch/parisc/kernel/syscalls/syscall.tbl b/arch/parisc/kernel/syscalls/syscall.tbl
index bf751e0732b7..b43fe303922c 100644
--- a/arch/parisc/kernel/syscalls/syscall.tbl
+++ b/arch/parisc/kernel/syscalls/syscall.tbl
@@ -446,3 +446,4 @@
 446	common	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	common	process_mrelease		sys_process_mrelease
+449	common	trusted_for			sys_trusted_for
diff --git a/arch/powerpc/kernel/syscalls/syscall.tbl b/arch/powerpc/kernel/syscalls/syscall.tbl
index 7bef917cc84e..0de4795a4f85 100644
--- a/arch/powerpc/kernel/syscalls/syscall.tbl
+++ b/arch/powerpc/kernel/syscalls/syscall.tbl
@@ -528,3 +528,4 @@
 446	common	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	common	process_mrelease		sys_process_mrelease
+449	common	trusted_for			sys_trusted_for
diff --git a/arch/s390/kernel/syscalls/syscall.tbl b/arch/s390/kernel/syscalls/syscall.tbl
index df5261e5cfe1..b524343bb67d 100644
--- a/arch/s390/kernel/syscalls/syscall.tbl
+++ b/arch/s390/kernel/syscalls/syscall.tbl
@@ -451,3 +451,4 @@
 446  common	landlock_restrict_self	sys_landlock_restrict_self	sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448  common	process_mrelease	sys_process_mrelease		sys_process_mrelease
+449  common	trusted_for		sys_trusted_for			sys_trusted_for
diff --git a/arch/sh/kernel/syscalls/syscall.tbl b/arch/sh/kernel/syscalls/syscall.tbl
index 208f131659c5..d4c925d83129 100644
--- a/arch/sh/kernel/syscalls/syscall.tbl
+++ b/arch/sh/kernel/syscalls/syscall.tbl
@@ -451,3 +451,4 @@
 446	common	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	common	process_mrelease		sys_process_mrelease
+449	common	trusted_for			sys_trusted_for
diff --git a/arch/sparc/kernel/syscalls/syscall.tbl b/arch/sparc/kernel/syscalls/syscall.tbl
index c37764dc764d..158bf1b028ac 100644
--- a/arch/sparc/kernel/syscalls/syscall.tbl
+++ b/arch/sparc/kernel/syscalls/syscall.tbl
@@ -494,3 +494,4 @@
 446	common	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	common	process_mrelease		sys_process_mrelease
+449	common	trusted_for			sys_trusted_for
diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index 960a021d543e..a9eda13c86cd 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -453,3 +453,4 @@
 446	i386	landlock_restrict_self	sys_landlock_restrict_self
 447	i386	memfd_secret		sys_memfd_secret
 448	i386	process_mrelease	sys_process_mrelease
+449	i386	trusted_for		sys_trusted_for
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index 18b5500ea8bf..c352f1b200ee 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -370,6 +370,7 @@
 446	common	landlock_restrict_self	sys_landlock_restrict_self
 447	common	memfd_secret		sys_memfd_secret
 448	common	process_mrelease	sys_process_mrelease
+449	common	trusted_for		sys_trusted_for
 
 #
 # Due to a historical design error, certain syscalls are numbered differently
diff --git a/arch/xtensa/kernel/syscalls/syscall.tbl b/arch/xtensa/kernel/syscalls/syscall.tbl
index 104b327f8ac9..f26a1313a055 100644
--- a/arch/xtensa/kernel/syscalls/syscall.tbl
+++ b/arch/xtensa/kernel/syscalls/syscall.tbl
@@ -419,3 +419,4 @@
 446	common	landlock_restrict_self		sys_landlock_restrict_self
 # 447 reserved for memfd_secret
 448	common	process_mrelease		sys_process_mrelease
+449	common	trusted_for			sys_trusted_for
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 1c5fb86d455a..67b4bebe7c87 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -879,9 +879,11 @@ __SYSCALL(__NR_memfd_secret, sys_memfd_secret)
 #endif
 #define __NR_process_mrelease 448
 __SYSCALL(__NR_process_mrelease, sys_process_mrelease)
+#define __NR_trusted_for 449
+__SYSCALL(__NR_trusted_for, sys_trusted_for)
 
 #undef __NR_syscalls
-#define __NR_syscalls 449
+#define __NR_syscalls 450
 
 /*
  * 32 bit systems traditionally used different
-- 
2.32.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v15 3/3] selftest/interpreter: Add tests for trusted_for(2) policies
  2021-10-12 19:24 [PATCH v15 0/3] Add trusted_for(2) (was O_MAYEXEC) Mickaël Salaün
  2021-10-12 19:24 ` [PATCH v15 1/3] fs: Add trusted_for(2) syscall implementation and related sysctl Mickaël Salaün
  2021-10-12 19:24 ` [PATCH v15 2/3] arch: Wire up trusted_for(2) Mickaël Salaün
@ 2021-10-12 19:24 ` Mickaël Salaün
  2 siblings, 0 replies; 10+ messages in thread
From: Mickaël Salaün @ 2021-10-12 19:24 UTC (permalink / raw)
  To: Al Viro, Andrew Morton
  Cc: Mickaël Salaün, Aleksa Sarai, Andy Lutomirski,
	Arnd Bergmann, Casey Schaufler, Christian Brauner,
	Christian Heimes, Deven Bowers, Dmitry Vyukov, Eric Biggers,
	Eric Chiang, Florian Weimer, Geert Uytterhoeven, James Morris,
	Jan Kara, Jann Horn, Jonathan Corbet, Kees Cook,
	Lakshmi Ramasubramanian, Madhavan T . Venkataraman,
	Matthew Garrett, Matthew Wilcox, Miklos Szeredi, Mimi Zohar,
	Paul Moore, Philippe Trébuchet, Scott Shell, Shuah Khan,
	Steve Dower, Steve Grubb, Thibaut Sautereau, Vincent Strubel,
	kernel-hardening, linux-api, linux-fsdevel, linux-integrity,
	linux-kernel, linux-security-module, Mickaël Salaün

From: Mickaël Salaün <mic@linux.microsoft.com>

Test that checks performed by trusted_for(2) on file descriptors are
consistent with noexec mount points and file execute permissions,
according to the policy configured with the fs.trust_policy sysctl.

Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Mickaël Salaün <mic@linux.microsoft.com>
Reviewed-by: Thibaut Sautereau <thibaut.sautereau@ssi.gouv.fr>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20211012192410.2356090-4-mic@digikod.net
---

Changes since v14:
* Add Reviewed-by Kees Cook.

Changes since v13:
* Move -I to CFLAGS (suggested by Kees Cook).
* Update sysctl name.

Changes since v12:
* Fix Makefile's license.

Changes since v10:
* Update selftest Makefile.

Changes since v9:
* Rename the syscall and the sysctl.
* Update tests for enum trusted_for_usage

Changes since v8:
* Update with the dedicated syscall introspect_access(2) and the renamed
  fs.introspection_policy sysctl.
* Remove check symlink which can't be use as is anymore.
* Use socketpair(2) to test UNIX socket.

Changes since v7:
* Update tests with faccessat2/AT_INTERPRETED, including new ones to
  check that setting R_OK or W_OK returns EINVAL.
* Add tests for memfd, pipefs and nsfs.
* Rename and move back tests to a standalone directory.

Changes since v6:
* Add full combination tests for all file types, including block
  devices, character devices, fifos, sockets and symlinks.
* Properly save and restore initial sysctl value for all tests.

Changes since v5:
* Refactor with FIXTURE_VARIANT, which make the tests much more easy to
  read and maintain.
* Save and restore initial sysctl value (suggested by Kees Cook).
* Test with a sysctl value of 0.
* Check errno in sysctl_access_write test.
* Update tests for the CAP_SYS_ADMIN switch.
* Update tests to check -EISDIR (replacing -EACCES).
* Replace FIXTURE_DATA() with FIXTURE() (spotted by Kees Cook).
* Use global const strings.

Changes since v3:
* Replace RESOLVE_MAYEXEC with O_MAYEXEC.
* Add tests to check that O_MAYEXEC is ignored by open(2) and openat(2).

Changes since v2:
* Move tests from exec/ to openat2/ .
* Replace O_MAYEXEC with RESOLVE_MAYEXEC from openat2(2).
* Cleanup tests.

Changes since v1:
* Move tests from yama/ to exec/ .
* Fix _GNU_SOURCE in kselftest_harness.h .
* Add a new test sysctl_access_write to check if CAP_MAC_ADMIN is taken
  into account.
* Test directory execution which is always forbidden since commit
  73601ea5b7b1 ("fs/open.c: allow opening only regular files during
  execve()"), and also check that even the root user can not bypass file
  execution checks.
* Make sure delete_workspace() always as enough right to succeed.
* Cosmetic cleanup.
---
 tools/testing/selftests/Makefile              |   1 +
 .../testing/selftests/interpreter/.gitignore  |   2 +
 tools/testing/selftests/interpreter/Makefile  |  21 +
 tools/testing/selftests/interpreter/config    |   1 +
 .../selftests/interpreter/trust_policy_test.c | 362 ++++++++++++++++++
 5 files changed, 387 insertions(+)
 create mode 100644 tools/testing/selftests/interpreter/.gitignore
 create mode 100644 tools/testing/selftests/interpreter/Makefile
 create mode 100644 tools/testing/selftests/interpreter/config
 create mode 100644 tools/testing/selftests/interpreter/trust_policy_test.c

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index c852eb40c4f7..3a032a545f74 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -20,6 +20,7 @@ TARGETS += ftrace
 TARGETS += futex
 TARGETS += gpio
 TARGETS += intel_pstate
+TARGETS += interpreter
 TARGETS += ipc
 TARGETS += ir
 TARGETS += kcmp
diff --git a/tools/testing/selftests/interpreter/.gitignore b/tools/testing/selftests/interpreter/.gitignore
new file mode 100644
index 000000000000..82a4846cbc4b
--- /dev/null
+++ b/tools/testing/selftests/interpreter/.gitignore
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+/*_test
diff --git a/tools/testing/selftests/interpreter/Makefile b/tools/testing/selftests/interpreter/Makefile
new file mode 100644
index 000000000000..7402fdb6533f
--- /dev/null
+++ b/tools/testing/selftests/interpreter/Makefile
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: GPL-2.0
+
+CFLAGS += -Wall -O2 -I$(khdr_dir)
+LDLIBS += -lcap
+
+src_test := $(wildcard *_test.c)
+TEST_GEN_PROGS := $(src_test:.c=)
+
+KSFT_KHDR_INSTALL := 1
+include ../lib.mk
+
+khdr_dir = $(top_srcdir)/usr/include
+
+$(khdr_dir)/asm-generic/unistd.h: khdr
+	@:
+
+$(khdr_dir)/linux/trusted-for.h: khdr
+	@:
+
+$(OUTPUT)/%_test: %_test.c $(khdr_dir)/asm-generic/unistd.h $(khdr_dir)/linux/trusted-for.h ../kselftest_harness.h
+	$(LINK.c) $< $(LDLIBS) -o $@
diff --git a/tools/testing/selftests/interpreter/config b/tools/testing/selftests/interpreter/config
new file mode 100644
index 000000000000..dd53c266bf52
--- /dev/null
+++ b/tools/testing/selftests/interpreter/config
@@ -0,0 +1 @@
+CONFIG_SYSCTL=y
diff --git a/tools/testing/selftests/interpreter/trust_policy_test.c b/tools/testing/selftests/interpreter/trust_policy_test.c
new file mode 100644
index 000000000000..b59f07f537ad
--- /dev/null
+++ b/tools/testing/selftests/interpreter/trust_policy_test.c
@@ -0,0 +1,362 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Test trusted_for(2) with fs.trusted_for_policy sysctl
+ *
+ * Copyright © 2018-2020 ANSSI
+ *
+ * Author: Mickaël Salaün <mic@digikod.net>
+ */
+
+#define _GNU_SOURCE
+#include <asm-generic/unistd.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <linux/trusted-for.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/capability.h>
+#include <sys/mman.h>
+#include <sys/mount.h>
+#include <sys/socket.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/sysmacros.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "../kselftest_harness.h"
+
+#ifndef trusted_for
+static int trusted_for(const int fd, const enum trusted_for_usage usage,
+		const __u32 flags)
+{
+	errno = 0;
+	return syscall(__NR_trusted_for, fd, usage, flags);
+}
+#endif
+
+static const char sysctl_path[] = "/proc/sys/fs/trusted_for_policy";
+
+static const char workdir_path[] = "./test-mount";
+static const char reg_file_path[] = "./test-mount/regular_file";
+static const char dir_path[] = "./test-mount/directory";
+static const char block_dev_path[] = "./test-mount/block_device";
+static const char char_dev_path[] = "./test-mount/character_device";
+static const char fifo_path[] = "./test-mount/fifo";
+
+static void ignore_dac(struct __test_metadata *_metadata, int override)
+{
+	cap_t caps;
+	const cap_value_t cap_val[2] = {
+		CAP_DAC_OVERRIDE,
+		CAP_DAC_READ_SEARCH,
+	};
+
+	caps = cap_get_proc();
+	ASSERT_NE(NULL, caps);
+	ASSERT_EQ(0, cap_set_flag(caps, CAP_EFFECTIVE, 2, cap_val,
+				override ? CAP_SET : CAP_CLEAR));
+	ASSERT_EQ(0, cap_set_proc(caps));
+	EXPECT_EQ(0, cap_free(caps));
+}
+
+static void ignore_sys_admin(struct __test_metadata *_metadata, int override)
+{
+	cap_t caps;
+	const cap_value_t cap_val[1] = {
+		CAP_SYS_ADMIN,
+	};
+
+	caps = cap_get_proc();
+	ASSERT_NE(NULL, caps);
+	ASSERT_EQ(0, cap_set_flag(caps, CAP_EFFECTIVE, 1, cap_val,
+				override ? CAP_SET : CAP_CLEAR));
+	ASSERT_EQ(0, cap_set_proc(caps));
+	EXPECT_EQ(0, cap_free(caps));
+}
+
+static void test_omx(struct __test_metadata *_metadata,
+		const char *const path, const int err_access)
+{
+	int flags = O_RDONLY | O_CLOEXEC;
+	int fd, access_ret, access_errno;
+
+	/* Do not block on pipes. */
+	if (path == fifo_path)
+		flags |= O_NONBLOCK;
+
+	fd = open(path, flags);
+	ASSERT_LE(0, fd) {
+		TH_LOG("Failed to open %s: %s", path, strerror(errno));
+	}
+	access_ret = trusted_for(fd, TRUSTED_FOR_EXECUTION, 0);
+	access_errno = errno;
+	if (err_access) {
+		ASSERT_EQ(err_access, access_errno) {
+			TH_LOG("Wrong error for trusted_for(2) with %s: %s",
+					path, strerror(access_errno));
+		}
+		ASSERT_EQ(-1, access_ret);
+	} else {
+		ASSERT_EQ(0, access_ret) {
+			TH_LOG("Access denied for %s: %s", path, strerror(access_errno));
+		}
+	}
+
+	/* Tests unsupported trusted usage. */
+	access_ret = trusted_for(fd, 0, 0);
+	ASSERT_EQ(-1, access_ret);
+	ASSERT_EQ(EINVAL, errno);
+
+	access_ret = trusted_for(fd, 2, 0);
+	ASSERT_EQ(-1, access_ret);
+	ASSERT_EQ(EINVAL, errno);
+
+	EXPECT_EQ(0, close(fd));
+}
+
+static void test_policy_fd(struct __test_metadata *_metadata, const int fd,
+		const bool has_policy)
+{
+	const int ret = trusted_for(fd, TRUSTED_FOR_EXECUTION, 0);
+
+	if (has_policy) {
+		ASSERT_EQ(-1, ret);
+		ASSERT_EQ(EACCES, errno) {
+			TH_LOG("Wrong error for trusted_for(2) with FD: %s", strerror(errno));
+		}
+	} else {
+		ASSERT_EQ(0, ret) {
+			TH_LOG("Access denied for FD: %s", strerror(errno));
+		}
+	}
+}
+
+FIXTURE(access) {
+	char initial_sysctl_value;
+	int memfd, pipefd;
+	int pipe_fds[2], socket_fds[2];
+};
+
+static void test_file_types(struct __test_metadata *_metadata, FIXTURE_DATA(access) *self,
+		const int err_code, const bool has_policy)
+{
+	/* Tests are performed on a tmpfs mount point. */
+	test_omx(_metadata, reg_file_path, err_code);
+	test_omx(_metadata, dir_path, has_policy ? EACCES : 0);
+	test_omx(_metadata, block_dev_path, has_policy ? EACCES : 0);
+	test_omx(_metadata, char_dev_path, has_policy ? EACCES : 0);
+	test_omx(_metadata, fifo_path, has_policy ? EACCES : 0);
+
+	/* Checks that exec is denied for any socket FD. */
+	test_policy_fd(_metadata, self->socket_fds[0], has_policy);
+
+	/* Checks that exec is denied for any memfd. */
+	test_policy_fd(_metadata, self->memfd, has_policy);
+
+	/* Checks that exec is denied for any pipefs FD. */
+	test_policy_fd(_metadata, self->pipefd, has_policy);
+}
+
+static void test_files(struct __test_metadata *_metadata, FIXTURE_DATA(access) *self,
+		const int err_code, const bool has_policy)
+{
+	/* Tests as root. */
+	ignore_dac(_metadata, 1);
+	test_file_types(_metadata, self, err_code, has_policy);
+
+	/* Tests without bypass. */
+	ignore_dac(_metadata, 0);
+	test_file_types(_metadata, self, err_code, has_policy);
+}
+
+static void sysctl_write_char(struct __test_metadata *_metadata, const char value)
+{
+	int fd;
+
+	fd = open(sysctl_path, O_WRONLY | O_CLOEXEC);
+	ASSERT_LE(0, fd);
+	ASSERT_EQ(1, write(fd, &value, 1));
+	EXPECT_EQ(0, close(fd));
+}
+
+static char sysctl_read_char(struct __test_metadata *_metadata)
+{
+	int fd;
+	char sysctl_value;
+
+	fd = open(sysctl_path, O_RDONLY | O_CLOEXEC);
+	ASSERT_LE(0, fd);
+	ASSERT_EQ(1, read(fd, &sysctl_value, 1));
+	EXPECT_EQ(0, close(fd));
+	return sysctl_value;
+}
+
+FIXTURE_VARIANT(access) {
+	const bool mount_exec;
+	const bool file_exec;
+	const int sysctl_err_code[3];
+};
+
+FIXTURE_VARIANT_ADD(access, mount_exec_file_exec) {
+	.mount_exec = true,
+	.file_exec = true,
+	.sysctl_err_code = {0, 0, 0},
+};
+
+FIXTURE_VARIANT_ADD(access, mount_exec_file_noexec)
+{
+	.mount_exec = true,
+	.file_exec = false,
+	.sysctl_err_code = {0, EACCES, EACCES},
+};
+
+FIXTURE_VARIANT_ADD(access, mount_noexec_file_exec)
+{
+	.mount_exec = false,
+	.file_exec = true,
+	.sysctl_err_code = {EACCES, 0, EACCES},
+};
+
+FIXTURE_VARIANT_ADD(access, mount_noexec_file_noexec)
+{
+	.mount_exec = false,
+	.file_exec = false,
+	.sysctl_err_code = {EACCES, EACCES, EACCES},
+};
+
+FIXTURE_SETUP(access)
+{
+	int procfd_path_size;
+	static const char path_template[] = "/proc/self/fd/%d";
+	char procfd_path[sizeof(path_template) + 10];
+
+	/*
+	 * Cleans previous workspace if any error previously happened (don't
+	 * check errors).
+	 */
+	umount(workdir_path);
+	rmdir(workdir_path);
+
+	/* Creates a clean mount point. */
+	ASSERT_EQ(0, mkdir(workdir_path, 00700));
+	ASSERT_EQ(0, mount("test", workdir_path, "tmpfs", MS_MGC_VAL |
+				(variant->mount_exec ? 0 : MS_NOEXEC),
+				"mode=0700,size=4k"));
+
+	/* Creates a regular file. */
+	ASSERT_EQ(0, mknod(reg_file_path, S_IFREG | (variant->file_exec ? 0500 : 0400), 0));
+	/* Creates a directory. */
+	ASSERT_EQ(0, mkdir(dir_path, variant->file_exec ? 0500 : 0400));
+	/* Creates a character device: /dev/null. */
+	ASSERT_EQ(0, mknod(char_dev_path, S_IFCHR | 0400, makedev(1, 3)));
+	/* Creates a block device: /dev/loop0 */
+	ASSERT_EQ(0, mknod(block_dev_path, S_IFBLK | 0400, makedev(7, 0)));
+	/* Creates a fifo. */
+	ASSERT_EQ(0, mknod(fifo_path, S_IFIFO | 0400, 0));
+
+	/* Creates a regular file without user mount point. */
+	self->memfd = memfd_create("test-interpreted", MFD_CLOEXEC);
+	ASSERT_LE(0, self->memfd);
+	/* Sets mode, which must be ignored by the exec check. */
+	ASSERT_EQ(0, fchmod(self->memfd, variant->file_exec ? 0500 : 0400));
+
+	/* Creates a pipefs file descriptor. */
+	ASSERT_EQ(0, pipe(self->pipe_fds));
+	procfd_path_size = snprintf(procfd_path, sizeof(procfd_path),
+			path_template, self->pipe_fds[0]);
+	ASSERT_LT(procfd_path_size, sizeof(procfd_path));
+	self->pipefd = open(procfd_path, O_RDONLY | O_CLOEXEC);
+	ASSERT_LE(0, self->pipefd);
+	ASSERT_EQ(0, fchmod(self->pipefd, variant->file_exec ? 0500 : 0400));
+
+	/* Creates a socket file descriptor. */
+	ASSERT_EQ(0, socketpair(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC, 0, self->socket_fds));
+
+	/* Saves initial sysctl value. */
+	self->initial_sysctl_value = sysctl_read_char(_metadata);
+
+	/* Prepares for sysctl writes. */
+	ignore_sys_admin(_metadata, 1);
+}
+
+FIXTURE_TEARDOWN(access)
+{
+	EXPECT_EQ(0, close(self->memfd));
+	EXPECT_EQ(0, close(self->pipefd));
+	EXPECT_EQ(0, close(self->pipe_fds[0]));
+	EXPECT_EQ(0, close(self->pipe_fds[1]));
+	EXPECT_EQ(0, close(self->socket_fds[0]));
+	EXPECT_EQ(0, close(self->socket_fds[1]));
+
+	/* Restores initial sysctl value. */
+	sysctl_write_char(_metadata, self->initial_sysctl_value);
+
+	/* There is no need to unlink the test files. */
+	ASSERT_EQ(0, umount(workdir_path));
+	ASSERT_EQ(0, rmdir(workdir_path));
+}
+
+TEST_F(access, sysctl_0)
+{
+	/* Do not enforce anything. */
+	sysctl_write_char(_metadata, '0');
+	test_files(_metadata, self, 0, false);
+}
+
+TEST_F(access, sysctl_1)
+{
+	/* Enforces mount exec check. */
+	sysctl_write_char(_metadata, '1');
+	test_files(_metadata, self, variant->sysctl_err_code[0], true);
+}
+
+TEST_F(access, sysctl_2)
+{
+	/* Enforces file exec check. */
+	sysctl_write_char(_metadata, '2');
+	test_files(_metadata, self, variant->sysctl_err_code[1], true);
+}
+
+TEST_F(access, sysctl_3)
+{
+	/* Enforces mount and file exec check. */
+	sysctl_write_char(_metadata, '3');
+	test_files(_metadata, self, variant->sysctl_err_code[2], true);
+}
+
+FIXTURE(cleanup) {
+	char initial_sysctl_value;
+};
+
+FIXTURE_SETUP(cleanup)
+{
+	/* Saves initial sysctl value. */
+	self->initial_sysctl_value = sysctl_read_char(_metadata);
+}
+
+FIXTURE_TEARDOWN(cleanup)
+{
+	/* Restores initial sysctl value. */
+	ignore_sys_admin(_metadata, 1);
+	sysctl_write_char(_metadata, self->initial_sysctl_value);
+}
+
+TEST_F(cleanup, sysctl_access_write)
+{
+	int fd;
+	ssize_t ret;
+
+	ignore_sys_admin(_metadata, 1);
+	sysctl_write_char(_metadata, '0');
+
+	ignore_sys_admin(_metadata, 0);
+	fd = open(sysctl_path, O_WRONLY | O_CLOEXEC);
+	ASSERT_LE(0, fd);
+	ret = write(fd, "0", 1);
+	ASSERT_EQ(-1, ret);
+	ASSERT_EQ(EPERM, errno);
+	EXPECT_EQ(0, close(fd));
+}
+
+TEST_HARNESS_MAIN
-- 
2.32.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [fs]  a0918006f9:  netperf.Throughput_tps -11.6% regression
  2021-10-12 19:24 ` [PATCH v15 1/3] fs: Add trusted_for(2) syscall implementation and related sysctl Mickaël Salaün
@ 2021-11-05  6:41   ` kernel test robot
  2021-11-09 17:21     ` Kees Cook
  0 siblings, 1 reply; 10+ messages in thread
From: kernel test robot @ 2021-11-05  6:41 UTC (permalink / raw)
  To: Mickaël Salaün
  Cc: lkp, lkp, ying.huang, feng.tang, zhengjun.xing, fengwei.yin,
	Al Viro, Andrew Morton, Mickaël Salaün, Aleksa Sarai,
	Andy Lutomirski, Arnd Bergmann, Casey Schaufler,
	Christian Brauner, Christian Heimes, Deven Bowers, Dmitry Vyukov,
	Eric Biggers, Eric Chiang, Florian Weimer, Geert Uytterhoeven,
	James Morris, Jan Kara, Jann Horn, Jonathan Corbet, Kees Cook,
	Lakshmi Ramasubramanian, Madhavan T . Venkataraman,
	Matthew Garrett, Matthew Wilcox, Miklos Szeredi, Mimi Zohar,
	Paul Moore, Philippe Trébuchet, Scott Shell, Shuah Khan,
	Steve Dower, Steve Grubb, Thibaut Sautereau, Vincent Strubel,
	kernel-hardening, linux-api, linux-fsdevel, linux-integrity,
	linux-kernel, linux-security-module, Mickaël Salaün

[-- Attachment #1: Type: text/plain, Size: 32857 bytes --]



Greeting,

FYI, we noticed a -11.6% regression of netperf.Throughput_tps due to commit:


commit: a0918006f9284b77397ae4f163f055c3e0f987b2 ("[PATCH v15 1/3] fs: Add trusted_for(2) syscall implementation and related sysctl")
url: https://github.com/0day-ci/linux/commits/Micka-l-Sala-n/Add-trusted_for-2-was-O_MAYEXEC/20211013-032533
patch link: https://lore.kernel.org/kernel-hardening/20211012192410.2356090-2-mic@digikod.net

in testcase: netperf
on test machine: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 16
	cluster: cs-localhost
	test: TCP_CRR
	cpufreq_governor: performance
	ucode: 0x5003006

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/


please be noted we made out some further analysis/tests, as Fengwei mentioned:
==============================================================================
Here is my investigation result of this regression:

If I add patch to make sure the kernel function address and data address is
almost same even with this patch, there is almost no performance delta(0.1%)
w/o the patch.

And if I only make sure function address same w/o the patch, the performance
delta is about 5.1%.

So suppose this regression is triggered by different function and data address.
We don't know why the different address could bring such kind of regression yet
===============================================================================


we also tested on other platforms.
on a Cooper Lake (Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz with 128G memory),
we also observed regression but the gap is smaller:
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-cpl-4sp1/TCP_CRR/netperf/0x700001e

commit:
  v5.15-rc4
  a0918006f9284b77397ae4f163f055c3e0f987b2

       v5.15-rc4 a0918006f9284b77397ae4f163f
---------------- ---------------------------
         %stddev     %change         %stddev
             \          |                \
    333492            -5.7%     314346 ±  2%  netperf.Throughput_total_tps
     20843            -4.5%      19896        netperf.Throughput_tps


but no regression on a 96 threads 2 sockets Ice Lake with 256G memory:
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-icl-2sp1/TCP_CRR/netperf/0xb000280

commit:
  v5.15-rc4
  a0918006f9284b77397ae4f163f055c3e0f987b2

       v5.15-rc4 a0918006f9284b77397ae4f163f
---------------- ---------------------------
         %stddev     %change         %stddev
             \          |                \
    555600            -0.1%     555305        netperf.Throughput_total_tps
     34725            -0.1%      34706        netperf.Throughput_tps


Fengwei also helped review these results and commented:
I suppose these three CPUs have different cache policy. It also could be
related with netperf throughput testing.


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@intel.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2ap3/TCP_CRR/netperf/0x5003006

commit: 
  v5.15-rc4
  a0918006f9 ("fs: Add trusted_for(2) syscall implementation and related sysctl")

       v5.15-rc4 a0918006f9284b77397ae4f163f 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    354692           -11.6%     313620        netperf.Throughput_total_tps
     22168           -11.6%      19601        netperf.Throughput_tps
 2.075e+08           -11.6%  1.834e+08        netperf.time.voluntary_context_switches
 1.064e+08           -11.6%   94086163        netperf.workload
      0.27 ± 35%      -0.1        0.22 ±  2%  mpstat.cpu.all.usr%
   2207583            -6.3%    2068413        vmstat.system.cs
   3029480 ±  6%     -23.3%    2324079 ±  7%  interrupts.CAL:Function_call_interrupts
     13768 ± 25%     -35.6%       8872 ± 23%  interrupts.CPU30.CAL:Function_call_interrupts
   2014617 ± 16%     -26.3%    1485200 ± 24%  softirqs.CPU180.NET_RX
 3.268e+08           -12.1%  2.874e+08        softirqs.NET_RX
    287881 ±  2%     +24.6%     358692        softirqs.TIMER
   3207001            -9.6%    2899010        perf-sched.wait_and_delay.count.schedule_timeout.inet_csk_accept.inet_accept.do_accept
      0.01 ± 15%     +67.1%       0.01 ±  9%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__release_sock.release_sock.sk_wait_data
      0.02 ±  2%     +23.3%       0.03 ± 21%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_accept.do_accept
      0.01           +20.0%       0.01        perf-sched.wait_time.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
     63320 ±  2%     -10.6%      56615 ±  2%  slabinfo.sock_inode_cache.active_objs
      1626 ±  2%     -10.6%       1454 ±  2%  slabinfo.sock_inode_cache.active_slabs
     63445 ±  2%     -10.6%      56722 ±  2%  slabinfo.sock_inode_cache.num_objs
      1626 ±  2%     -10.6%       1454 ±  2%  slabinfo.sock_inode_cache.num_slabs
     49195            -3.2%      47624        proc-vmstat.nr_slab_reclaimable
   4278441            -6.6%    3996109        proc-vmstat.numa_hit
   4052317 ±  2%      -7.4%    3751341        proc-vmstat.numa_local
   4285136            -6.5%    4006356        proc-vmstat.pgalloc_normal
   1704913           -11.4%    1511123        proc-vmstat.pgfree
 9.382e+09           -10.1%  8.438e+09        perf-stat.i.branch-instructions
 1.391e+08           -10.0%  1.252e+08        perf-stat.i.branch-misses
     13.98            +2.2       16.20        perf-stat.i.cache-miss-rate%
  87082775           +14.0%   99273064        perf-stat.i.cache-misses
   2231661            -6.4%    2088571        perf-stat.i.context-switches
      1.65            +8.6%       1.79        perf-stat.i.cpi
 7.603e+10            -2.1%  7.441e+10        perf-stat.i.cpu-cycles
    907.53 ±  2%     -13.0%     789.92 ±  2%  perf-stat.i.cycles-between-cache-misses
    920324 ± 19%     -20.3%     733572 ±  5%  perf-stat.i.dTLB-load-misses
 1.417e+10           -10.3%  1.271e+10        perf-stat.i.dTLB-loads
    182445 ± 16%     -57.6%      77419 ±  9%  perf-stat.i.dTLB-store-misses
 8.254e+09           -10.3%  7.403e+09        perf-stat.i.dTLB-stores
     88.23            -1.7       86.52        perf-stat.i.iTLB-load-miss-rate%
  96633753           -11.0%   85983323        perf-stat.i.iTLB-load-misses
  12277057            +4.0%   12766535        perf-stat.i.iTLB-loads
 4.741e+10           -10.2%  4.259e+10        perf-stat.i.instructions
      0.62            -8.2%       0.57        perf-stat.i.ipc
      0.40            -2.1%       0.39        perf-stat.i.metric.GHz
    168.88           -10.1%     151.87        perf-stat.i.metric.M/sec
  16134360 ±  2%     +15.0%   18550862        perf-stat.i.node-load-misses
   1576525 ±  2%     +10.0%    1734370 ±  2%  perf-stat.i.node-loads
  10027868           -11.5%    8871598        perf-stat.i.node-store-misses
    386034 ±  3%     -16.0%     324290 ±  7%  perf-stat.i.node-stores
     13.15            +9.2%      14.36        perf-stat.overall.MPKI
     13.97            +2.3       16.23        perf-stat.overall.cache-miss-rate%
      1.60            +8.9%       1.75        perf-stat.overall.cpi
    873.29           -14.2%     749.60        perf-stat.overall.cycles-between-cache-misses
      0.00 ± 15%      -0.0        0.00 ±  9%  perf-stat.overall.dTLB-store-miss-rate%
     88.73            -1.7       87.07        perf-stat.overall.iTLB-load-miss-rate%
      0.62            -8.2%       0.57        perf-stat.overall.ipc
    135778            +1.7%     138069        perf-stat.overall.path-length
 9.351e+09           -10.1%   8.41e+09        perf-stat.ps.branch-instructions
 1.387e+08           -10.0%  1.248e+08        perf-stat.ps.branch-misses
  86797490           +14.0%   98949207        perf-stat.ps.cache-misses
   2224197            -6.4%    2081616        perf-stat.ps.context-switches
 7.578e+10            -2.1%  7.416e+10        perf-stat.ps.cpu-cycles
    917495 ± 19%     -20.3%     731365 ±  5%  perf-stat.ps.dTLB-load-misses
 1.412e+10           -10.3%  1.267e+10        perf-stat.ps.dTLB-loads
    181859 ± 16%     -57.6%      77179 ±  9%  perf-stat.ps.dTLB-store-misses
 8.227e+09           -10.3%  7.379e+09        perf-stat.ps.dTLB-stores
  96313891           -11.0%   85700283        perf-stat.ps.iTLB-load-misses
  12236194            +4.0%   12724086        perf-stat.ps.iTLB-loads
 4.726e+10           -10.2%  4.245e+10        perf-stat.ps.instructions
  16081690 ±  2%     +15.0%   18490522        perf-stat.ps.node-load-misses
   1571411 ±  2%     +10.0%    1728755 ±  2%  perf-stat.ps.node-loads
   9995103           -11.5%    8842824        perf-stat.ps.node-store-misses
    385193 ±  3%     -16.0%     323588 ±  7%  perf-stat.ps.node-stores
 1.445e+13           -10.1%  1.299e+13        perf-stat.total.instructions
      1.51 ±  7%      -0.2        1.29 ±  7%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
      1.53 ±  7%      -0.2        1.31 ±  7%  perf-profile.calltrace.cycles-pp.ret_from_fork
      1.53 ±  7%      -0.2        1.31 ±  7%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      1.48 ±  7%      -0.2        1.26 ±  7%  perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread
      1.49 ±  7%      -0.2        1.27 ±  7%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
      1.50 ±  7%      -0.2        1.27 ±  7%  perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
      1.47 ±  7%      -0.2        1.25 ±  7%  perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn
      1.41 ±  7%      -0.2        1.19 ±  7%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd
      1.25 ±  7%      -0.2        1.06 ±  7%  perf-profile.calltrace.cycles-pp.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start
      1.21 ±  7%      -0.2        1.03 ±  7%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core
      0.94 ±  7%      -0.1        0.80 ±  7%  perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch
      0.62 ±  7%      +0.2        0.80 ±  9%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      1.51 ±  7%      -0.2        1.29 ±  7%  perf-profile.children.cycles-pp.smpboot_thread_fn
      1.53 ±  7%      -0.2        1.31 ±  7%  perf-profile.children.cycles-pp.ret_from_fork
      1.53 ±  7%      -0.2        1.31 ±  7%  perf-profile.children.cycles-pp.kthread
      1.50 ±  7%      -0.2        1.27 ±  7%  perf-profile.children.cycles-pp.run_ksoftirqd
      1.73 ±  6%      -0.2        1.51 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      1.25 ±  5%      -0.2        1.07 ±  6%  perf-profile.children.cycles-pp.lock_sock_nested
      1.03 ±  7%      -0.1        0.88 ±  6%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.83 ±  6%      -0.1        0.72 ±  6%  perf-profile.children.cycles-pp.sk_clone_lock
      0.84 ±  6%      -0.1        0.73 ±  6%  perf-profile.children.cycles-pp.inet_csk_clone_lock
      0.45 ±  8%      -0.1        0.34 ±  6%  perf-profile.children.cycles-pp.__tcp_get_metrics
      0.70 ±  6%      -0.1        0.60 ±  6%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.52 ±  8%      -0.1        0.42 ±  6%  perf-profile.children.cycles-pp.tcp_get_metrics
      0.72 ±  5%      -0.1        0.62 ±  6%  perf-profile.children.cycles-pp.sk_forced_mem_schedule
      0.32 ±  7%      -0.1        0.24 ±  7%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.49 ±  7%      -0.1        0.41 ±  8%  perf-profile.children.cycles-pp.tcp_v4_destroy_sock
      0.26 ±  7%      -0.0        0.22 ±  8%  perf-profile.children.cycles-pp.ip_finish_output
      0.29 ±  6%      -0.0        0.25 ±  9%  perf-profile.children.cycles-pp.tcp_write_queue_purge
      0.16 ± 10%      -0.0        0.12 ±  8%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.10 ±  8%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__destroy_inode
      0.10 ±  8%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.destroy_inode
      0.10 ±  9%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.sock_put
      0.10 ± 10%      -0.0        0.07 ±  8%  perf-profile.children.cycles-pp.d_instantiate
      0.08 ± 11%      -0.0        0.06 ±  9%  perf-profile.children.cycles-pp.kmem_cache_alloc_trace
      0.11 ±  8%      +0.0        0.15 ±  6%  perf-profile.children.cycles-pp.__inet_lookup_listener
      0.08 ±  9%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.inet_lhash2_lookup
      0.10 ±  7%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.tcp_ca_openreq_child
      0.08 ±  9%      +0.0        0.13 ±  9%  perf-profile.children.cycles-pp.tcp_newly_delivered
      0.08 ±  6%      +0.0        0.12 ±  9%  perf-profile.children.cycles-pp.tcp_mtup_init
      0.09 ±  8%      +0.1        0.15 ±  6%  perf-profile.children.cycles-pp.tcp_stream_memory_free
      0.24 ±  6%      +0.1        0.30 ±  8%  perf-profile.children.cycles-pp.ip_rcv_core
      0.06 ±  9%      +0.1        0.12 ±  7%  perf-profile.children.cycles-pp.tcp_push
      0.11 ±  9%      +0.1        0.17 ±  7%  perf-profile.children.cycles-pp.tcp_synack_rtt_meas
      0.00 ±412%      +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.tcp_rack_update_reo_wnd
      0.20 ±  8%      +0.1        0.28 ±  6%  perf-profile.children.cycles-pp.tcp_assign_congestion_control
      0.34 ±  8%      +0.1        0.42 ±  6%  perf-profile.children.cycles-pp.tcp_init_metrics
      0.14 ±  6%      +0.1        0.22 ±  8%  perf-profile.children.cycles-pp.tcp_sync_mss
      0.33 ±  5%      +0.1        0.41 ±  8%  perf-profile.children.cycles-pp.inet_csk_route_req
      0.31 ±  6%      +0.1        0.40 ±  6%  perf-profile.children.cycles-pp.inet_csk_route_child_sock
      0.13 ±  8%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.skb_entail
      0.21 ±  6%      +0.1        0.32 ±  7%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.24 ±  5%      +0.1        0.35 ±  7%  perf-profile.children.cycles-pp.ip_rcv_finish
      0.20 ±  7%      +0.1        0.32 ±  5%  perf-profile.children.cycles-pp.tcp_select_initial_window
      0.14 ±  5%      +0.1        0.26 ±  8%  perf-profile.children.cycles-pp.secure_tcp_ts_off
      0.45 ±  6%      +0.1        0.58 ±  6%  perf-profile.children.cycles-pp.tcp_finish_connect
      0.23 ±  5%      +0.1        0.35 ±  5%  perf-profile.children.cycles-pp.tcp_parse_options
      0.17 ±  7%      +0.1        0.31 ±  6%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
      0.20 ±  7%      +0.1        0.35 ±  6%  perf-profile.children.cycles-pp.tcp_openreq_init_rwin
      0.27 ±  9%      +0.1        0.42 ±  7%  perf-profile.children.cycles-pp.tcp_connect_init
      0.45 ±  7%      +0.2        0.60 ±  5%  perf-profile.children.cycles-pp.tcp_v4_init_sock
      0.44 ±  7%      +0.2        0.60 ±  6%  perf-profile.children.cycles-pp.tcp_init_sock
      0.23 ±  7%      +0.2        0.39 ±  6%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.35 ±  6%      +0.2        0.57 ±  7%  perf-profile.children.cycles-pp.inet_sk_rebuild_header
      0.25 ±  9%      +0.2        0.49 ±  7%  perf-profile.children.cycles-pp.__tcp_select_window
      0.35 ±  6%      +0.3        0.61 ±  6%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      0.76 ±  5%      +0.3        1.04 ±  6%  perf-profile.children.cycles-pp.ip_route_output_flow
      0.78 ±  6%      +0.3        1.08 ±  6%  perf-profile.children.cycles-pp.tcp_init_transfer
      1.78 ±  6%      +0.3        2.11 ±  6%  perf-profile.children.cycles-pp.tcp_conn_request
      1.07 ±  4%      +0.4        1.44 ±  5%  perf-profile.children.cycles-pp.ip_route_output_key_hash
      1.02 ±  5%      +0.4        1.40 ±  5%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      2.02 ±  5%      +0.5        2.50 ±  6%  perf-profile.children.cycles-pp.tcp_ack
      1.04 ±  7%      +0.6        1.63 ±  7%  perf-profile.children.cycles-pp.__sk_dst_check
      1.18 ±  7%      +0.7        1.86 ±  7%  perf-profile.children.cycles-pp.ipv4_dst_check
      5.95 ±  5%      +0.9        6.87 ±  6%  perf-profile.children.cycles-pp.tcp_v4_connect
      1.02 ±  7%      -0.2        0.87 ±  5%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.44 ±  8%      -0.1        0.34 ±  6%  perf-profile.self.cycles-pp.__tcp_get_metrics
      0.69 ±  6%      -0.1        0.59 ±  6%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.71 ±  5%      -0.1        0.61 ±  6%  perf-profile.self.cycles-pp.sk_forced_mem_schedule
      0.32 ±  6%      -0.1        0.26 ±  8%  perf-profile.self.cycles-pp.ip_finish_output2
      0.35 ±  7%      -0.1        0.29 ±  5%  perf-profile.self.cycles-pp.tcp_recvmsg_locked
      0.15 ±  7%      -0.0        0.12 ±  8%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.17 ±  6%      -0.0        0.14 ± 10%  perf-profile.self.cycles-pp.__skb_clone
      0.07 ±  5%      -0.0        0.04 ± 43%  perf-profile.self.cycles-pp.sk_filter_trim_cap
      0.09 ±  9%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.08 ±  7%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.release_sock
      0.07 ± 10%      +0.0        0.09 ±  9%  perf-profile.self.cycles-pp.tcp_create_openreq_child
      0.11 ±  7%      +0.0        0.15 ±  5%  perf-profile.self.cycles-pp.tcp_connect
      0.08 ±  9%      +0.0        0.12 ±  8%  perf-profile.self.cycles-pp.inet_lhash2_lookup
      0.09 ±  9%      +0.0        0.13 ±  6%  perf-profile.self.cycles-pp.inet_csk_get_port
      0.08 ± 10%      +0.0        0.12 ±  8%  perf-profile.self.cycles-pp.tcp_init_transfer
      0.08 ±  9%      +0.0        0.13 ±  8%  perf-profile.self.cycles-pp.tcp_newly_delivered
      0.07 ±  7%      +0.0        0.12 ±  9%  perf-profile.self.cycles-pp.tcp_mtup_init
      0.35 ±  5%      +0.1        0.40 ±  5%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.16 ±  7%      +0.1        0.22 ±  6%  perf-profile.self.cycles-pp.__inet_bind
      0.09 ±  8%      +0.1        0.15 ±  6%  perf-profile.self.cycles-pp.tcp_stream_memory_free
      0.24 ±  6%      +0.1        0.30 ±  8%  perf-profile.self.cycles-pp.ip_rcv_core
      0.06 ±  9%      +0.1        0.12 ±  6%  perf-profile.self.cycles-pp.tcp_push
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.tcp_rack_update_reo_wnd
      0.23 ±  8%      +0.1        0.30 ±  6%  perf-profile.self.cycles-pp.ip_output
      0.20 ±  8%      +0.1        0.28 ±  5%  perf-profile.self.cycles-pp.tcp_assign_congestion_control
      0.10 ±  8%      +0.1        0.18 ±  7%  perf-profile.self.cycles-pp.tcp_v4_syn_recv_sock
      0.09 ±  7%      +0.1        0.17 ±  7%  perf-profile.self.cycles-pp.tcp_openreq_init_rwin
      0.07 ± 10%      +0.1        0.16 ±  6%  perf-profile.self.cycles-pp.tcp_v4_send_synack
      0.13 ±  7%      +0.1        0.22 ±  7%  perf-profile.self.cycles-pp.tcp_sync_mss
      0.12 ±  8%      +0.1        0.20 ±  7%  perf-profile.self.cycles-pp.skb_entail
      0.18 ±  8%      +0.1        0.27 ±  6%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.21 ±  5%      +0.1        0.31 ±  6%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.15 ±  9%      +0.1        0.26 ±  6%  perf-profile.self.cycles-pp.tcp_update_metrics
      0.20 ±  8%      +0.1        0.31 ±  5%  perf-profile.self.cycles-pp.tcp_select_initial_window
      0.12 ±  9%      +0.1        0.25 ±  8%  perf-profile.self.cycles-pp.tcp_connect_init
      0.11 ±  8%      +0.1        0.24 ±  8%  perf-profile.self.cycles-pp.secure_tcp_ts_off
      0.22 ±  5%      +0.1        0.35 ±  5%  perf-profile.self.cycles-pp.tcp_parse_options
      0.13 ± 12%      +0.1        0.27 ±  7%  perf-profile.self.cycles-pp.tcp_init_metrics
      0.17 ±  7%      +0.1        0.30 ±  7%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
      0.17 ± 10%      +0.2        0.32 ±  6%  perf-profile.self.cycles-pp.tcp_init_sock
      0.18 ±  8%      +0.2        0.35 ±  6%  perf-profile.self.cycles-pp.tcp_schedule_loss_probe
      0.42 ±  8%      +0.2        0.62 ±  7%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.25 ±  8%      +0.2        0.48 ±  7%  perf-profile.self.cycles-pp.__tcp_select_window
      0.28 ±  8%      +0.3        0.56 ±  5%  perf-profile.self.cycles-pp.tcp_ack_update_rtt
      0.71 ±  5%      +0.4        1.09 ±  6%  perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
      1.17 ±  7%      +0.7        1.84 ±  7%  perf-profile.self.cycles-pp.ipv4_dst_check


                                                                                
                               netperf.Throughput_tps                           
                                                                                
  22500 +-------------------------------------------------------------------+   
        |        ...+......                           ...+......+.....+.....|   
  22000 |.....+..          +.....+.....+.....+.....+..                      |   
        |                                                                   |   
        |                                                                   |   
  21500 |-+                                                                 |   
        |                                                                   |   
  21000 |-+                                                                 |   
        |                                                                   |   
  20500 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  20000 |-+                                                                 |   
        |     O     O            O     O                 O                  |   
  19500 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            netperf.Throughput_total_tps                        
                                                                                
  360000 +------------------------------------------------------------------+   
  355000 |-+      ...+.....                ...+.....   ...+..         +.....|   
         |.....+..         +.....+.....+...         +..                     |   
  350000 |-+                                                                |   
  345000 |-+                                                                |   
         |                                                                  |   
  340000 |-+                                                                |   
  335000 |-+                                                                |   
  330000 |-+                                                                |   
         |                                                                  |   
  325000 |-+                                                                |   
  320000 |-+                                                                |   
         |                                                                  |   
  315000 |-+   O     O     O     O     O      O     O     O     O     O     |   
  310000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                   netperf.workload                             
                                                                                
  1.08e+08 +----------------------------------------------------------------+   
           |        ...+.....+.....         ..+.....   ...+..         +.....|   
  1.06e+08 |.....+..               +.....+..        +..                     |   
  1.04e+08 |-+                                                              |   
           |                                                                |   
  1.02e+08 |-+                                                              |   
           |                                                                |   
     1e+08 |-+                                                              |   
           |                                                                |   
   9.8e+07 |-+                                                              |   
   9.6e+07 |-+                                                              |   
           |                                                                |   
   9.4e+07 |-+   O     O     O     O     O    O     O     O     O     O     |   
           |                                                                |   
   9.2e+07 +----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        netperf.time.voluntary_context_switches                 
                                                                                
   2.1e+08 +----------------------------------------------------------------+   
           |.....+.....+.....+.....+.....+....+.....   ...+..         +.....|   
  2.05e+08 |-+                                      +..                     |   
           |                                                                |   
           |                                                                |   
     2e+08 |-+                                                              |   
           |                                                                |   
  1.95e+08 |-+                                                              |   
           |                                                                |   
   1.9e+08 |-+                                                              |   
           |                                                                |   
           |                                                                |   
  1.85e+08 |-+   O     O     O     O     O          O     O                 |   
           |                                  O                 O     O     |   
   1.8e+08 +----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                                                                
                                                                                
   0.006 +------------------------------------------------------------------+   
         |                                                                  |   
         |                                                                  |   
  0.0058 |-+                                                                |   
         |                                                                  |   
         |                                                                  |   
  0.0056 |-+                                                                |   
         |                                                                  |   
  0.0054 |-+                                                                |   
         |                                                                  |   
         |                                                                  |   
  0.0052 |-+                                                                |   
         |                                                                  |   
         |                                                                  |   
   0.005 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                                                                
                                                                                
  3.25e+06 +----------------------------------------------------------------+   
           |.....   ...+....          ...+....+.....+.....+.....   ...+.....|   
   3.2e+06 |-+   +..        .   ...+..                          +..         |   
           |                 +..                                            |   
  3.15e+06 |-+                                                              |   
   3.1e+06 |-+                                                              |   
           |                                                                |   
  3.05e+06 |-+                                                              |   
           |                                                                |   
     3e+06 |-+                                                              |   
  2.95e+06 |-+                                                              |   
           |                                                                |   
   2.9e+06 |-+   O     O     O           O    O     O     O     O     O     |   
           |                       O                                        |   
  2.85e+06 +----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-icl-2sp1: 96 threads 2 sockets Ice Lake with 256G memory





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


[-- Attachment #2: config-5.15.0-rc4-00001-ga0918006f928.old --]
[-- Type: application/x-trash, Size: 183719 bytes --]

[-- Attachment #3: job-script --]
[-- Type: text/plain, Size: 8126 bytes --]

#!/bin/sh

export_top_env()
{
	export suite='netperf'
	export testcase='netperf'
	export category='benchmark'
	export disable_latency_stats=1
	export set_nic_irq_affinity=1
	export ip='ipv4'
	export runtime=300
	export nr_threads=16
	export cluster='cs-localhost'
	export job_origin='netperf-small-threads.yaml'
	export queue_cmdline_keys=
	export queue='vip'
	export testbox='lkp-csl-2ap3'
	export tbox_group='lkp-csl-2ap3'
	export kconfig='x86_64-rhel-8.3'
	export submit_id='617960e80b9a930a5af4f104'
	export job_file='/lkp/jobs/scheduled/lkp-csl-2ap3/netperf-cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006-debian-10.4-x86_64-20200603.cgz-a0918006f9284b77397ae4f163-20211027-68186-ja0nr3-6.yaml'
	export id='fbecc857f957790eb9cfac7363705ffadfda23f9'
	export queuer_version='/lkp/xsang/.src-20211027-151141'
	export model='Cascade Lake'
	export nr_node=4
	export nr_cpu=192
	export memory='192G'
	export ssd_partitions=
	export rootfs_partition='LABEL=LKP-ROOTFS'
	export kernel_cmdline_hw='acpi_rsdp=0x67f44014'
	export brand='Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz'
	export commit='a0918006f9284b77397ae4f163f055c3e0f987b2'
	export need_kconfig_hw='{"IGB"=>"y"}
BLK_DEV_NVME'
	export ucode='0x5003006'
	export enqueue_time='2021-10-27 22:23:36 +0800'
	export _id='617960f00b9a930a5af4f10a'
	export _rt='/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2'
	export user='lkp'
	export compiler='gcc-9'
	export LKP_SERVER='internal-lkp-server'
	export head_commit='955f175760f41ad2a80b07a390bac9a0444a47a6'
	export base_commit='519d81956ee277b4419c723adfb154603c2565ba'
	export branch='linux-devel/devel-hourly-20211025-030231'
	export rootfs='debian-10.4-x86_64-20200603.cgz'
	export result_root='/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/8'
	export scheduler_version='/lkp/lkp/.src-20211027-140142'
	export arch='x86_64'
	export max_uptime=2100
	export initrd='/osimage/debian/debian-10.4-x86_64-20200603.cgz'
	export bootloader_append='root=/dev/ram0
user=lkp
job=/lkp/jobs/scheduled/lkp-csl-2ap3/netperf-cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006-debian-10.4-x86_64-20200603.cgz-a0918006f9284b77397ae4f163-20211027-68186-ja0nr3-6.yaml
ARCH=x86_64
kconfig=x86_64-rhel-8.3
branch=linux-devel/devel-hourly-20211025-030231
commit=a0918006f9284b77397ae4f163f055c3e0f987b2
BOOT_IMAGE=/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/vmlinuz-5.15.0-rc4-00001-ga0918006f928
acpi_rsdp=0x67f44014
max_uptime=2100
RESULT_ROOT=/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/8
LKP_SERVER=internal-lkp-server
nokaslr
selinux=0
debug
apic=debug
sysrq_always_enabled
rcupdate.rcu_cpu_stall_timeout=100
net.ifnames=0
printk.devkmsg=on
panic=-1
softlockup_panic=1
nmi_watchdog=panic
oops=panic
load_ramdisk=2
prompt_ramdisk=0
drbd.minor_count=8
systemd.log_level=err
ignore_loglevel
console=tty0
earlyprintk=ttyS0,115200
console=ttyS0,115200
vga=normal
rw'
	export modules_initrd='/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/modules.cgz'
	export bm_initrd='/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/netperf_20210930.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/netperf-x86_64-2.7-0_20211027.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/mpstat_20200714.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/turbostat_20200721.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/turbostat-x86_64-3.7-4_20200721.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/perf_20211027.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/perf-x86_64-d25f27432f80-1_20211027.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/sar-x86_64-34c92ae-1_20200702.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/hw_20200715.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/cluster_20211026.cgz'
	export ucode_initrd='/osimage/ucode/intel-ucode-20210222.cgz'
	export lkp_initrd='/osimage/user/lkp/lkp-x86_64.cgz'
	export site='inn'
	export LKP_CGI_PORT=80
	export LKP_CIFS_PORT=139
	export last_kernel='5.15.0-rc6-wt-12022-g955f175760f4'
	export queue_at_least_once=0
	export schedule_notify_address=
	export kernel='/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/vmlinuz-5.15.0-rc4-00001-ga0918006f928'
	export dequeue_time='2021-10-28 03:03:20 +0800'
	export node_roles='server client'
	export job_initrd='/lkp/jobs/scheduled/lkp-csl-2ap3/netperf-cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006-debian-10.4-x86_64-20200603.cgz-a0918006f9284b77397ae4f163-20211027-68186-ja0nr3-6.cgz'

	[ -n "$LKP_SRC" ] ||
	export LKP_SRC=/lkp/${user:-lkp}/src
}

run_job()
{
	echo $$ > $TMP/run-job.pid

	. $LKP_SRC/lib/http.sh
	. $LKP_SRC/lib/job.sh
	. $LKP_SRC/lib/env.sh

	export_top_env

	run_setup $LKP_SRC/setup/cpufreq_governor 'performance'

	run_monitor $LKP_SRC/monitors/wrapper kmsg
	run_monitor $LKP_SRC/monitors/no-stdout/wrapper boot-time
	run_monitor $LKP_SRC/monitors/wrapper uptime
	run_monitor $LKP_SRC/monitors/wrapper iostat
	run_monitor $LKP_SRC/monitors/wrapper heartbeat
	run_monitor $LKP_SRC/monitors/wrapper vmstat
	run_monitor $LKP_SRC/monitors/wrapper numa-numastat
	run_monitor $LKP_SRC/monitors/wrapper numa-vmstat
	run_monitor $LKP_SRC/monitors/wrapper numa-meminfo
	run_monitor $LKP_SRC/monitors/wrapper proc-vmstat
	run_monitor $LKP_SRC/monitors/wrapper proc-stat
	run_monitor $LKP_SRC/monitors/wrapper meminfo
	run_monitor $LKP_SRC/monitors/wrapper slabinfo
	run_monitor $LKP_SRC/monitors/wrapper interrupts
	run_monitor $LKP_SRC/monitors/wrapper lock_stat
	run_monitor lite_mode=1 $LKP_SRC/monitors/wrapper perf-sched
	run_monitor $LKP_SRC/monitors/wrapper softirqs
	run_monitor $LKP_SRC/monitors/one-shot/wrapper bdi_dev_mapping
	run_monitor $LKP_SRC/monitors/wrapper diskstats
	run_monitor $LKP_SRC/monitors/wrapper nfsstat
	run_monitor $LKP_SRC/monitors/wrapper cpuidle
	run_monitor $LKP_SRC/monitors/wrapper cpufreq-stats
	run_monitor $LKP_SRC/monitors/wrapper turbostat
	run_monitor $LKP_SRC/monitors/wrapper sched_debug
	run_monitor $LKP_SRC/monitors/wrapper perf-stat
	run_monitor $LKP_SRC/monitors/wrapper mpstat
	run_monitor lite_mode=1 $LKP_SRC/monitors/no-stdout/wrapper perf-profile
	run_monitor $LKP_SRC/monitors/wrapper oom-killer
	run_monitor $LKP_SRC/monitors/plain/watchdog

	if role server
	then
		start_daemon $LKP_SRC/daemon/netserver
	fi

	if role client
	then
		run_test test='TCP_CRR' $LKP_SRC/tests/wrapper netperf
	fi
}

extract_stats()
{
	export stats_part_begin=
	export stats_part_end=

	env test='TCP_CRR' $LKP_SRC/stats/wrapper netperf
	$LKP_SRC/stats/wrapper kmsg
	$LKP_SRC/stats/wrapper boot-time
	$LKP_SRC/stats/wrapper uptime
	$LKP_SRC/stats/wrapper iostat
	$LKP_SRC/stats/wrapper vmstat
	$LKP_SRC/stats/wrapper numa-numastat
	$LKP_SRC/stats/wrapper numa-vmstat
	$LKP_SRC/stats/wrapper numa-meminfo
	$LKP_SRC/stats/wrapper proc-vmstat
	$LKP_SRC/stats/wrapper meminfo
	$LKP_SRC/stats/wrapper slabinfo
	$LKP_SRC/stats/wrapper interrupts
	$LKP_SRC/stats/wrapper lock_stat
	env lite_mode=1 $LKP_SRC/stats/wrapper perf-sched
	$LKP_SRC/stats/wrapper softirqs
	$LKP_SRC/stats/wrapper diskstats
	$LKP_SRC/stats/wrapper nfsstat
	$LKP_SRC/stats/wrapper cpuidle
	$LKP_SRC/stats/wrapper turbostat
	$LKP_SRC/stats/wrapper sched_debug
	$LKP_SRC/stats/wrapper perf-stat
	$LKP_SRC/stats/wrapper mpstat
	env lite_mode=1 $LKP_SRC/stats/wrapper perf-profile

	$LKP_SRC/stats/wrapper time netperf.time
	$LKP_SRC/stats/wrapper dmesg
	$LKP_SRC/stats/wrapper kmsg
	$LKP_SRC/stats/wrapper last_state
	$LKP_SRC/stats/wrapper stderr
	$LKP_SRC/stats/wrapper time
}

"$@"

[-- Attachment #4: job.yaml --]
[-- Type: text/plain, Size: 5549 bytes --]

---
:#! jobs/netperf-small-threads.yaml:
suite: netperf
testcase: netperf
category: benchmark
:# upto 90% CPU cycles may be used by latency stats:
disable_latency_stats: 1
set_nic_irq_affinity: 1
ip: ipv4
runtime: 300s
nr_threads: 16
cluster: cs-localhost
if role server:
  netserver:
if role client:
  netperf:
    test: TCP_CRR
job_origin: netperf-small-threads.yaml
:#! queue options:
queue_cmdline_keys: []
queue: vip
testbox: lkp-csl-2ap3
tbox_group: lkp-csl-2ap3
kconfig: x86_64-rhel-8.3
submit_id: 617960e80b9a930a5af4f104
job_file: "/lkp/jobs/scheduled/lkp-csl-2ap3/netperf-cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006-debian-10.4-x86_64-20200603.cgz-a0918006f9284b77397ae4f163-20211027-68186-ja0nr3-4.yaml"
id: d804900eed74d058b23143a825d247e0f8d03392
queuer_version: "/lkp/xsang/.src-20211027-151141"
:#! hosts/lkp-csl-2ap3:
model: Cascade Lake
nr_node: 4
nr_cpu: 192
memory: 192G
ssd_partitions:
rootfs_partition: LABEL=LKP-ROOTFS
kernel_cmdline_hw: acpi_rsdp=0x67f44014
brand: Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz
:#! include/category/benchmark:
kmsg:
boot-time:
uptime:
iostat:
heartbeat:
vmstat:
numa-numastat:
numa-vmstat:
numa-meminfo:
proc-vmstat:
proc-stat:
meminfo:
slabinfo:
interrupts:
lock_stat:
perf-sched:
  lite_mode: 1
softirqs:
bdi_dev_mapping:
diskstats:
nfsstat:
cpuidle:
cpufreq-stats:
turbostat:
sched_debug:
perf-stat:
mpstat:
perf-profile:
  lite_mode: 1
:#! include/category/ALL:
cpufreq_governor: performance
:#! include/queue/cyclic:
commit: a0918006f9284b77397ae4f163f055c3e0f987b2
:#! include/testbox/lkp-csl-2ap3:
need_kconfig_hw:
- IGB: y
- BLK_DEV_NVME
ucode: '0x5003006'
enqueue_time: 2021-10-27 22:23:36.647383432 +08:00
_id: 617960f00b9a930a5af4f108
_rt: "/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2"
:#! schedule options:
user: lkp
compiler: gcc-9
LKP_SERVER: internal-lkp-server
head_commit: 955f175760f41ad2a80b07a390bac9a0444a47a6
base_commit: 519d81956ee277b4419c723adfb154603c2565ba
branch: linux-devel/devel-hourly-20211025-030231
rootfs: debian-10.4-x86_64-20200603.cgz
result_root: "/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/0"
scheduler_version: "/lkp/lkp/.src-20211027-140142"
arch: x86_64
max_uptime: 2100
initrd: "/osimage/debian/debian-10.4-x86_64-20200603.cgz"
bootloader_append:
- root=/dev/ram0
- user=lkp
- job=/lkp/jobs/scheduled/lkp-csl-2ap3/netperf-cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006-debian-10.4-x86_64-20200603.cgz-a0918006f9284b77397ae4f163-20211027-68186-ja0nr3-4.yaml
- ARCH=x86_64
- kconfig=x86_64-rhel-8.3
- branch=linux-devel/devel-hourly-20211025-030231
- commit=a0918006f9284b77397ae4f163f055c3e0f987b2
- BOOT_IMAGE=/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/vmlinuz-5.15.0-rc4-00001-ga0918006f928
- acpi_rsdp=0x67f44014
- max_uptime=2100
- RESULT_ROOT=/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/0
- LKP_SERVER=internal-lkp-server
- nokaslr
- selinux=0
- debug
- apic=debug
- sysrq_always_enabled
- rcupdate.rcu_cpu_stall_timeout=100
- net.ifnames=0
- printk.devkmsg=on
- panic=-1
- softlockup_panic=1
- nmi_watchdog=panic
- oops=panic
- load_ramdisk=2
- prompt_ramdisk=0
- drbd.minor_count=8
- systemd.log_level=err
- ignore_loglevel
- console=tty0
- earlyprintk=ttyS0,115200
- console=ttyS0,115200
- vga=normal
- rw
modules_initrd: "/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/modules.cgz"
bm_initrd: "/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/netperf_20210930.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/netperf-x86_64-2.7-0_20211025.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/mpstat_20200714.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/turbostat_20200721.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/turbostat-x86_64-3.7-4_20200721.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/perf_20211027.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/perf-x86_64-d25f27432f80-1_20211027.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/sar-x86_64-34c92ae-1_20200702.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/hw_20200715.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/cluster_20211026.cgz"
ucode_initrd: "/osimage/ucode/intel-ucode-20210222.cgz"
lkp_initrd: "/osimage/user/lkp/lkp-x86_64.cgz"
site: inn
:#! /lkp/lkp/.src-20211026-143536/include/site/inn:
LKP_CGI_PORT: 80
LKP_CIFS_PORT: 139
oom-killer:
watchdog:
:#! runtime status:
last_kernel: 5.15.0-rc6-wt-12022-g955f175760f4
queue_at_least_once: 0
:#! /lkp/lkp/.src-20211026-205023/include/site/inn:
:#! user overrides:
schedule_notify_address:
kernel: "/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/vmlinuz-5.15.0-rc4-00001-ga0918006f928"
dequeue_time: 2021-10-28 02:02:15.174367353 +08:00
:#! /lkp/lkp/.src-20211027-140142/include/site/inn:
job_state: finished
loadavg: 6.09 10.58 5.53 1/1355 19660
start_time: '1635357811'
end_time: '1635358116'
version: "/lkp/lkp/.src-20211027-140222:5f87ddf4:8610dc698"

[-- Attachment #5: reproduce --]
[-- Type: text/plain, Size: 1090 bytes --]


for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
do
	online_file="$cpu_dir"/online
	[ -f "$online_file" ] && [ "$(cat "$online_file")" -eq 0 ] && continue

	file="$cpu_dir"/cpufreq/scaling_governor
	[ -f "$file" ] && echo "performance" > "$file"
done

netserver -4 -D
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
wait

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [fs]  a0918006f9:  netperf.Throughput_tps -11.6% regression
  2021-11-05  6:41   ` [fs] a0918006f9: netperf.Throughput_tps -11.6% regression kernel test robot
@ 2021-11-09 17:21     ` Kees Cook
  2021-11-10  1:54       ` Yin Fengwei
                         ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: Kees Cook @ 2021-11-09 17:21 UTC (permalink / raw)
  To: kernel test robot
  Cc: Mickaël Salaün, lkp, lkp, ying.huang, feng.tang,
	zhengjun.xing, fengwei.yin, Al Viro, Andrew Morton, Aleksa Sarai,
	Andy Lutomirski, Arnd Bergmann, Casey Schaufler,
	Christian Brauner, Christian Heimes, Deven Bowers, Dmitry Vyukov,
	Eric Biggers, Eric Chiang, Florian Weimer, Geert Uytterhoeven,
	James Morris, Jan Kara, Jann Horn, Jonathan Corbet,
	Lakshmi Ramasubramanian, Madhavan T . Venkataraman,
	Matthew Garrett, Matthew Wilcox, Miklos Szeredi, Mimi Zohar,
	Paul Moore, Philippe Trébuchet, Scott Shell, Shuah Khan,
	Steve Dower, Steve Grubb, Thibaut Sautereau, Vincent Strubel,
	kernel-hardening, linux-api, linux-fsdevel, linux-integrity,
	linux-kernel, linux-security-module, Mickaël Salaün

On Fri, Nov 05, 2021 at 02:41:59PM +0800, kernel test robot wrote:
> 
> 
> Greeting,
> 
> FYI, we noticed a -11.6% regression of netperf.Throughput_tps due to commit:
> 
> 
> commit: a0918006f9284b77397ae4f163f055c3e0f987b2 ("[PATCH v15 1/3] fs: Add trusted_for(2) syscall implementation and related sysctl")
> url: https://github.com/0day-ci/linux/commits/Micka-l-Sala-n/Add-trusted_for-2-was-O_MAYEXEC/20211013-032533
> patch link: https://lore.kernel.org/kernel-hardening/20211012192410.2356090-2-mic@digikod.net
> 
> in testcase: netperf
> on test machine: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
> with following parameters:
> 
> 	ip: ipv4
> 	runtime: 300s
> 	nr_threads: 16
> 	cluster: cs-localhost
> 	test: TCP_CRR
> 	cpufreq_governor: performance
> 	ucode: 0x5003006
> 
> test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
> test-url: http://www.netperf.org/netperf/
> 
> 
> please be noted we made out some further analysis/tests, as Fengwei mentioned:
> ==============================================================================
> Here is my investigation result of this regression:
> 
> If I add patch to make sure the kernel function address and data address is
> almost same even with this patch, there is almost no performance delta(0.1%)
> w/o the patch.
> 
> And if I only make sure function address same w/o the patch, the performance
> delta is about 5.1%.
> 
> So suppose this regression is triggered by different function and data address.
> We don't know why the different address could bring such kind of regression yet
> ===============================================================================
> 
> 
> we also tested on other platforms.
> on a Cooper Lake (Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz with 128G memory),
> we also observed regression but the gap is smaller:
> =========================================================================================
> cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
>   cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-cpl-4sp1/TCP_CRR/netperf/0x700001e
> 
> commit:
>   v5.15-rc4
>   a0918006f9284b77397ae4f163f055c3e0f987b2
> 
>        v5.15-rc4 a0918006f9284b77397ae4f163f
> ---------------- ---------------------------
>          %stddev     %change         %stddev
>              \          |                \
>     333492            -5.7%     314346 ±  2%  netperf.Throughput_total_tps
>      20843            -4.5%      19896        netperf.Throughput_tps
> 
> 
> but no regression on a 96 threads 2 sockets Ice Lake with 256G memory:
> =========================================================================================
> cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
>   cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-icl-2sp1/TCP_CRR/netperf/0xb000280
> 
> commit:
>   v5.15-rc4
>   a0918006f9284b77397ae4f163f055c3e0f987b2
> 
>        v5.15-rc4 a0918006f9284b77397ae4f163f
> ---------------- ---------------------------
>          %stddev     %change         %stddev
>              \          |                \
>     555600            -0.1%     555305        netperf.Throughput_total_tps
>      34725            -0.1%      34706        netperf.Throughput_tps
> 
> 
> Fengwei also helped review these results and commented:
> I suppose these three CPUs have different cache policy. It also could be
> related with netperf throughput testing.

Does moving the syscall implementation somewhere else change things?
That's a _huge_ performance change for something that isn't even called.
What's going on here?

-Kees

> 
> 
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <oliver.sang@intel.com>
> 
> 
> Details are as below:
> -------------------------------------------------------------------------------------------------->
> 
> 
> To reproduce:
> 
>         git clone https://github.com/intel/lkp-tests.git
>         cd lkp-tests
>         sudo bin/lkp install job.yaml           # job file is attached in this email
>         bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
>         sudo bin/lkp run generated-yaml-file
> 
>         # if come across any failure that blocks the test,
>         # please remove ~/.lkp and /lkp dir to run from a clean state.
> 
> =========================================================================================
> cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
>   cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2ap3/TCP_CRR/netperf/0x5003006
> 
> commit: 
>   v5.15-rc4
>   a0918006f9 ("fs: Add trusted_for(2) syscall implementation and related sysctl")
> 
>        v5.15-rc4 a0918006f9284b77397ae4f163f 
> ---------------- --------------------------- 
>          %stddev     %change         %stddev
>              \          |                \  
>     354692           -11.6%     313620        netperf.Throughput_total_tps
>      22168           -11.6%      19601        netperf.Throughput_tps
>  2.075e+08           -11.6%  1.834e+08        netperf.time.voluntary_context_switches
>  1.064e+08           -11.6%   94086163        netperf.workload
>       0.27 ± 35%      -0.1        0.22 ±  2%  mpstat.cpu.all.usr%
>    2207583            -6.3%    2068413        vmstat.system.cs
>    3029480 ±  6%     -23.3%    2324079 ±  7%  interrupts.CAL:Function_call_interrupts
>      13768 ± 25%     -35.6%       8872 ± 23%  interrupts.CPU30.CAL:Function_call_interrupts
>    2014617 ± 16%     -26.3%    1485200 ± 24%  softirqs.CPU180.NET_RX
>  3.268e+08           -12.1%  2.874e+08        softirqs.NET_RX
>     287881 ±  2%     +24.6%     358692        softirqs.TIMER
>    3207001            -9.6%    2899010        perf-sched.wait_and_delay.count.schedule_timeout.inet_csk_accept.inet_accept.do_accept
>       0.01 ± 15%     +67.1%       0.01 ±  9%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__release_sock.release_sock.sk_wait_data
>       0.02 ±  2%     +23.3%       0.03 ± 21%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_accept.do_accept
>       0.01           +20.0%       0.01        perf-sched.wait_time.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
>      63320 ±  2%     -10.6%      56615 ±  2%  slabinfo.sock_inode_cache.active_objs
>       1626 ±  2%     -10.6%       1454 ±  2%  slabinfo.sock_inode_cache.active_slabs
>      63445 ±  2%     -10.6%      56722 ±  2%  slabinfo.sock_inode_cache.num_objs
>       1626 ±  2%     -10.6%       1454 ±  2%  slabinfo.sock_inode_cache.num_slabs
>      49195            -3.2%      47624        proc-vmstat.nr_slab_reclaimable
>    4278441            -6.6%    3996109        proc-vmstat.numa_hit
>    4052317 ±  2%      -7.4%    3751341        proc-vmstat.numa_local
>    4285136            -6.5%    4006356        proc-vmstat.pgalloc_normal
>    1704913           -11.4%    1511123        proc-vmstat.pgfree
>  9.382e+09           -10.1%  8.438e+09        perf-stat.i.branch-instructions
>  1.391e+08           -10.0%  1.252e+08        perf-stat.i.branch-misses
>      13.98            +2.2       16.20        perf-stat.i.cache-miss-rate%
>   87082775           +14.0%   99273064        perf-stat.i.cache-misses
>    2231661            -6.4%    2088571        perf-stat.i.context-switches
>       1.65            +8.6%       1.79        perf-stat.i.cpi
>  7.603e+10            -2.1%  7.441e+10        perf-stat.i.cpu-cycles
>     907.53 ±  2%     -13.0%     789.92 ±  2%  perf-stat.i.cycles-between-cache-misses
>     920324 ± 19%     -20.3%     733572 ±  5%  perf-stat.i.dTLB-load-misses
>  1.417e+10           -10.3%  1.271e+10        perf-stat.i.dTLB-loads
>     182445 ± 16%     -57.6%      77419 ±  9%  perf-stat.i.dTLB-store-misses
>  8.254e+09           -10.3%  7.403e+09        perf-stat.i.dTLB-stores
>      88.23            -1.7       86.52        perf-stat.i.iTLB-load-miss-rate%
>   96633753           -11.0%   85983323        perf-stat.i.iTLB-load-misses
>   12277057            +4.0%   12766535        perf-stat.i.iTLB-loads
>  4.741e+10           -10.2%  4.259e+10        perf-stat.i.instructions
>       0.62            -8.2%       0.57        perf-stat.i.ipc
>       0.40            -2.1%       0.39        perf-stat.i.metric.GHz
>     168.88           -10.1%     151.87        perf-stat.i.metric.M/sec
>   16134360 ±  2%     +15.0%   18550862        perf-stat.i.node-load-misses
>    1576525 ±  2%     +10.0%    1734370 ±  2%  perf-stat.i.node-loads
>   10027868           -11.5%    8871598        perf-stat.i.node-store-misses
>     386034 ±  3%     -16.0%     324290 ±  7%  perf-stat.i.node-stores
>      13.15            +9.2%      14.36        perf-stat.overall.MPKI
>      13.97            +2.3       16.23        perf-stat.overall.cache-miss-rate%
>       1.60            +8.9%       1.75        perf-stat.overall.cpi
>     873.29           -14.2%     749.60        perf-stat.overall.cycles-between-cache-misses
>       0.00 ± 15%      -0.0        0.00 ±  9%  perf-stat.overall.dTLB-store-miss-rate%
>      88.73            -1.7       87.07        perf-stat.overall.iTLB-load-miss-rate%
>       0.62            -8.2%       0.57        perf-stat.overall.ipc
>     135778            +1.7%     138069        perf-stat.overall.path-length
>  9.351e+09           -10.1%   8.41e+09        perf-stat.ps.branch-instructions
>  1.387e+08           -10.0%  1.248e+08        perf-stat.ps.branch-misses
>   86797490           +14.0%   98949207        perf-stat.ps.cache-misses
>    2224197            -6.4%    2081616        perf-stat.ps.context-switches
>  7.578e+10            -2.1%  7.416e+10        perf-stat.ps.cpu-cycles
>     917495 ± 19%     -20.3%     731365 ±  5%  perf-stat.ps.dTLB-load-misses
>  1.412e+10           -10.3%  1.267e+10        perf-stat.ps.dTLB-loads
>     181859 ± 16%     -57.6%      77179 ±  9%  perf-stat.ps.dTLB-store-misses
>  8.227e+09           -10.3%  7.379e+09        perf-stat.ps.dTLB-stores
>   96313891           -11.0%   85700283        perf-stat.ps.iTLB-load-misses
>   12236194            +4.0%   12724086        perf-stat.ps.iTLB-loads
>  4.726e+10           -10.2%  4.245e+10        perf-stat.ps.instructions
>   16081690 ±  2%     +15.0%   18490522        perf-stat.ps.node-load-misses
>    1571411 ±  2%     +10.0%    1728755 ±  2%  perf-stat.ps.node-loads
>    9995103           -11.5%    8842824        perf-stat.ps.node-store-misses
>     385193 ±  3%     -16.0%     323588 ±  7%  perf-stat.ps.node-stores
>  1.445e+13           -10.1%  1.299e+13        perf-stat.total.instructions
>       1.51 ±  7%      -0.2        1.29 ±  7%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
>       1.53 ±  7%      -0.2        1.31 ±  7%  perf-profile.calltrace.cycles-pp.ret_from_fork
>       1.53 ±  7%      -0.2        1.31 ±  7%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
>       1.48 ±  7%      -0.2        1.26 ±  7%  perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread
>       1.49 ±  7%      -0.2        1.27 ±  7%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
>       1.50 ±  7%      -0.2        1.27 ±  7%  perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
>       1.47 ±  7%      -0.2        1.25 ±  7%  perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn
>       1.41 ±  7%      -0.2        1.19 ±  7%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd
>       1.25 ±  7%      -0.2        1.06 ±  7%  perf-profile.calltrace.cycles-pp.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start
>       1.21 ±  7%      -0.2        1.03 ±  7%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core
>       0.94 ±  7%      -0.1        0.80 ±  7%  perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch
>       0.62 ±  7%      +0.2        0.80 ±  9%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
>       1.51 ±  7%      -0.2        1.29 ±  7%  perf-profile.children.cycles-pp.smpboot_thread_fn
>       1.53 ±  7%      -0.2        1.31 ±  7%  perf-profile.children.cycles-pp.ret_from_fork
>       1.53 ±  7%      -0.2        1.31 ±  7%  perf-profile.children.cycles-pp.kthread
>       1.50 ±  7%      -0.2        1.27 ±  7%  perf-profile.children.cycles-pp.run_ksoftirqd
>       1.73 ±  6%      -0.2        1.51 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock_bh
>       1.25 ±  5%      -0.2        1.07 ±  6%  perf-profile.children.cycles-pp.lock_sock_nested
>       1.03 ±  7%      -0.1        0.88 ±  6%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
>       0.83 ±  6%      -0.1        0.72 ±  6%  perf-profile.children.cycles-pp.sk_clone_lock
>       0.84 ±  6%      -0.1        0.73 ±  6%  perf-profile.children.cycles-pp.inet_csk_clone_lock
>       0.45 ±  8%      -0.1        0.34 ±  6%  perf-profile.children.cycles-pp.__tcp_get_metrics
>       0.70 ±  6%      -0.1        0.60 ±  6%  perf-profile.children.cycles-pp.percpu_counter_add_batch
>       0.52 ±  8%      -0.1        0.42 ±  6%  perf-profile.children.cycles-pp.tcp_get_metrics
>       0.72 ±  5%      -0.1        0.62 ±  6%  perf-profile.children.cycles-pp.sk_forced_mem_schedule
>       0.32 ±  7%      -0.1        0.24 ±  7%  perf-profile.children.cycles-pp.sk_filter_trim_cap
>       0.49 ±  7%      -0.1        0.41 ±  8%  perf-profile.children.cycles-pp.tcp_v4_destroy_sock
>       0.26 ±  7%      -0.0        0.22 ±  8%  perf-profile.children.cycles-pp.ip_finish_output
>       0.29 ±  6%      -0.0        0.25 ±  9%  perf-profile.children.cycles-pp.tcp_write_queue_purge
>       0.16 ± 10%      -0.0        0.12 ±  8%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
>       0.10 ±  8%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__destroy_inode
>       0.10 ±  8%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.destroy_inode
>       0.10 ±  9%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.sock_put
>       0.10 ± 10%      -0.0        0.07 ±  8%  perf-profile.children.cycles-pp.d_instantiate
>       0.08 ± 11%      -0.0        0.06 ±  9%  perf-profile.children.cycles-pp.kmem_cache_alloc_trace
>       0.11 ±  8%      +0.0        0.15 ±  6%  perf-profile.children.cycles-pp.__inet_lookup_listener
>       0.08 ±  9%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.inet_lhash2_lookup
>       0.10 ±  7%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.tcp_ca_openreq_child
>       0.08 ±  9%      +0.0        0.13 ±  9%  perf-profile.children.cycles-pp.tcp_newly_delivered
>       0.08 ±  6%      +0.0        0.12 ±  9%  perf-profile.children.cycles-pp.tcp_mtup_init
>       0.09 ±  8%      +0.1        0.15 ±  6%  perf-profile.children.cycles-pp.tcp_stream_memory_free
>       0.24 ±  6%      +0.1        0.30 ±  8%  perf-profile.children.cycles-pp.ip_rcv_core
>       0.06 ±  9%      +0.1        0.12 ±  7%  perf-profile.children.cycles-pp.tcp_push
>       0.11 ±  9%      +0.1        0.17 ±  7%  perf-profile.children.cycles-pp.tcp_synack_rtt_meas
>       0.00 ±412%      +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.tcp_rack_update_reo_wnd
>       0.20 ±  8%      +0.1        0.28 ±  6%  perf-profile.children.cycles-pp.tcp_assign_congestion_control
>       0.34 ±  8%      +0.1        0.42 ±  6%  perf-profile.children.cycles-pp.tcp_init_metrics
>       0.14 ±  6%      +0.1        0.22 ±  8%  perf-profile.children.cycles-pp.tcp_sync_mss
>       0.33 ±  5%      +0.1        0.41 ±  8%  perf-profile.children.cycles-pp.inet_csk_route_req
>       0.31 ±  6%      +0.1        0.40 ±  6%  perf-profile.children.cycles-pp.inet_csk_route_child_sock
>       0.13 ±  8%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.skb_entail
>       0.21 ±  6%      +0.1        0.32 ±  7%  perf-profile.children.cycles-pp.ip_rcv_finish_core
>       0.24 ±  5%      +0.1        0.35 ±  7%  perf-profile.children.cycles-pp.ip_rcv_finish
>       0.20 ±  7%      +0.1        0.32 ±  5%  perf-profile.children.cycles-pp.tcp_select_initial_window
>       0.14 ±  5%      +0.1        0.26 ±  8%  perf-profile.children.cycles-pp.secure_tcp_ts_off
>       0.45 ±  6%      +0.1        0.58 ±  6%  perf-profile.children.cycles-pp.tcp_finish_connect
>       0.23 ±  5%      +0.1        0.35 ±  5%  perf-profile.children.cycles-pp.tcp_parse_options
>       0.17 ±  7%      +0.1        0.31 ±  6%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
>       0.20 ±  7%      +0.1        0.35 ±  6%  perf-profile.children.cycles-pp.tcp_openreq_init_rwin
>       0.27 ±  9%      +0.1        0.42 ±  7%  perf-profile.children.cycles-pp.tcp_connect_init
>       0.45 ±  7%      +0.2        0.60 ±  5%  perf-profile.children.cycles-pp.tcp_v4_init_sock
>       0.44 ±  7%      +0.2        0.60 ±  6%  perf-profile.children.cycles-pp.tcp_init_sock
>       0.23 ±  7%      +0.2        0.39 ±  6%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
>       0.35 ±  6%      +0.2        0.57 ±  7%  perf-profile.children.cycles-pp.inet_sk_rebuild_header
>       0.25 ±  9%      +0.2        0.49 ±  7%  perf-profile.children.cycles-pp.__tcp_select_window
>       0.35 ±  6%      +0.3        0.61 ±  6%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
>       0.76 ±  5%      +0.3        1.04 ±  6%  perf-profile.children.cycles-pp.ip_route_output_flow
>       0.78 ±  6%      +0.3        1.08 ±  6%  perf-profile.children.cycles-pp.tcp_init_transfer
>       1.78 ±  6%      +0.3        2.11 ±  6%  perf-profile.children.cycles-pp.tcp_conn_request
>       1.07 ±  4%      +0.4        1.44 ±  5%  perf-profile.children.cycles-pp.ip_route_output_key_hash
>       1.02 ±  5%      +0.4        1.40 ±  5%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
>       2.02 ±  5%      +0.5        2.50 ±  6%  perf-profile.children.cycles-pp.tcp_ack
>       1.04 ±  7%      +0.6        1.63 ±  7%  perf-profile.children.cycles-pp.__sk_dst_check
>       1.18 ±  7%      +0.7        1.86 ±  7%  perf-profile.children.cycles-pp.ipv4_dst_check
>       5.95 ±  5%      +0.9        6.87 ±  6%  perf-profile.children.cycles-pp.tcp_v4_connect
>       1.02 ±  7%      -0.2        0.87 ±  5%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
>       0.44 ±  8%      -0.1        0.34 ±  6%  perf-profile.self.cycles-pp.__tcp_get_metrics
>       0.69 ±  6%      -0.1        0.59 ±  6%  perf-profile.self.cycles-pp.percpu_counter_add_batch
>       0.71 ±  5%      -0.1        0.61 ±  6%  perf-profile.self.cycles-pp.sk_forced_mem_schedule
>       0.32 ±  6%      -0.1        0.26 ±  8%  perf-profile.self.cycles-pp.ip_finish_output2
>       0.35 ±  7%      -0.1        0.29 ±  5%  perf-profile.self.cycles-pp.tcp_recvmsg_locked
>       0.15 ±  7%      -0.0        0.12 ±  8%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
>       0.17 ±  6%      -0.0        0.14 ± 10%  perf-profile.self.cycles-pp.__skb_clone
>       0.07 ±  5%      -0.0        0.04 ± 43%  perf-profile.self.cycles-pp.sk_filter_trim_cap
>       0.09 ±  9%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.dequeue_task_fair
>       0.08 ±  7%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.release_sock
>       0.07 ± 10%      +0.0        0.09 ±  9%  perf-profile.self.cycles-pp.tcp_create_openreq_child
>       0.11 ±  7%      +0.0        0.15 ±  5%  perf-profile.self.cycles-pp.tcp_connect
>       0.08 ±  9%      +0.0        0.12 ±  8%  perf-profile.self.cycles-pp.inet_lhash2_lookup
>       0.09 ±  9%      +0.0        0.13 ±  6%  perf-profile.self.cycles-pp.inet_csk_get_port
>       0.08 ± 10%      +0.0        0.12 ±  8%  perf-profile.self.cycles-pp.tcp_init_transfer
>       0.08 ±  9%      +0.0        0.13 ±  8%  perf-profile.self.cycles-pp.tcp_newly_delivered
>       0.07 ±  7%      +0.0        0.12 ±  9%  perf-profile.self.cycles-pp.tcp_mtup_init
>       0.35 ±  5%      +0.1        0.40 ±  5%  perf-profile.self.cycles-pp.__ip_queue_xmit
>       0.16 ±  7%      +0.1        0.22 ±  6%  perf-profile.self.cycles-pp.__inet_bind
>       0.09 ±  8%      +0.1        0.15 ±  6%  perf-profile.self.cycles-pp.tcp_stream_memory_free
>       0.24 ±  6%      +0.1        0.30 ±  8%  perf-profile.self.cycles-pp.ip_rcv_core
>       0.06 ±  9%      +0.1        0.12 ±  6%  perf-profile.self.cycles-pp.tcp_push
>       0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.tcp_rack_update_reo_wnd
>       0.23 ±  8%      +0.1        0.30 ±  6%  perf-profile.self.cycles-pp.ip_output
>       0.20 ±  8%      +0.1        0.28 ±  5%  perf-profile.self.cycles-pp.tcp_assign_congestion_control
>       0.10 ±  8%      +0.1        0.18 ±  7%  perf-profile.self.cycles-pp.tcp_v4_syn_recv_sock
>       0.09 ±  7%      +0.1        0.17 ±  7%  perf-profile.self.cycles-pp.tcp_openreq_init_rwin
>       0.07 ± 10%      +0.1        0.16 ±  6%  perf-profile.self.cycles-pp.tcp_v4_send_synack
>       0.13 ±  7%      +0.1        0.22 ±  7%  perf-profile.self.cycles-pp.tcp_sync_mss
>       0.12 ±  8%      +0.1        0.20 ±  7%  perf-profile.self.cycles-pp.skb_entail
>       0.18 ±  8%      +0.1        0.27 ±  6%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
>       0.21 ±  5%      +0.1        0.31 ±  6%  perf-profile.self.cycles-pp.ip_rcv_finish_core
>       0.15 ±  9%      +0.1        0.26 ±  6%  perf-profile.self.cycles-pp.tcp_update_metrics
>       0.20 ±  8%      +0.1        0.31 ±  5%  perf-profile.self.cycles-pp.tcp_select_initial_window
>       0.12 ±  9%      +0.1        0.25 ±  8%  perf-profile.self.cycles-pp.tcp_connect_init
>       0.11 ±  8%      +0.1        0.24 ±  8%  perf-profile.self.cycles-pp.secure_tcp_ts_off
>       0.22 ±  5%      +0.1        0.35 ±  5%  perf-profile.self.cycles-pp.tcp_parse_options
>       0.13 ± 12%      +0.1        0.27 ±  7%  perf-profile.self.cycles-pp.tcp_init_metrics
>       0.17 ±  7%      +0.1        0.30 ±  7%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
>       0.17 ± 10%      +0.2        0.32 ±  6%  perf-profile.self.cycles-pp.tcp_init_sock
>       0.18 ±  8%      +0.2        0.35 ±  6%  perf-profile.self.cycles-pp.tcp_schedule_loss_probe
>       0.42 ±  8%      +0.2        0.62 ±  7%  perf-profile.self.cycles-pp.tcp_write_xmit
>       0.25 ±  8%      +0.2        0.48 ±  7%  perf-profile.self.cycles-pp.__tcp_select_window
>       0.28 ±  8%      +0.3        0.56 ±  5%  perf-profile.self.cycles-pp.tcp_ack_update_rtt
>       0.71 ±  5%      +0.4        1.09 ±  6%  perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
>       1.17 ±  7%      +0.7        1.84 ±  7%  perf-profile.self.cycles-pp.ipv4_dst_check
> 
> 
>                                                                                 
>                                netperf.Throughput_tps                           
>                                                                                 
>   22500 +-------------------------------------------------------------------+   
>         |        ...+......                           ...+......+.....+.....|   
>   22000 |.....+..          +.....+.....+.....+.....+..                      |   
>         |                                                                   |   
>         |                                                                   |   
>   21500 |-+                                                                 |   
>         |                                                                   |   
>   21000 |-+                                                                 |   
>         |                                                                   |   
>   20500 |-+                                                                 |   
>         |                                                                   |   
>         |                                                                   |   
>   20000 |-+                                                                 |   
>         |     O     O            O     O                 O                  |   
>   19500 +-------------------------------------------------------------------+   
>                                                                                 
>                                                                                                                                                                 
>                             netperf.Throughput_total_tps                        
>                                                                                 
>   360000 +------------------------------------------------------------------+   
>   355000 |-+      ...+.....                ...+.....   ...+..         +.....|   
>          |.....+..         +.....+.....+...         +..                     |   
>   350000 |-+                                                                |   
>   345000 |-+                                                                |   
>          |                                                                  |   
>   340000 |-+                                                                |   
>   335000 |-+                                                                |   
>   330000 |-+                                                                |   
>          |                                                                  |   
>   325000 |-+                                                                |   
>   320000 |-+                                                                |   
>          |                                                                  |   
>   315000 |-+   O     O     O     O     O      O     O     O     O     O     |   
>   310000 +------------------------------------------------------------------+   
>                                                                                 
>                                                                                                                                                                 
>                                    netperf.workload                             
>                                                                                 
>   1.08e+08 +----------------------------------------------------------------+   
>            |        ...+.....+.....         ..+.....   ...+..         +.....|   
>   1.06e+08 |.....+..               +.....+..        +..                     |   
>   1.04e+08 |-+                                                              |   
>            |                                                                |   
>   1.02e+08 |-+                                                              |   
>            |                                                                |   
>      1e+08 |-+                                                              |   
>            |                                                                |   
>    9.8e+07 |-+                                                              |   
>    9.6e+07 |-+                                                              |   
>            |                                                                |   
>    9.4e+07 |-+   O     O     O     O     O    O     O     O     O     O     |   
>            |                                                                |   
>    9.2e+07 +----------------------------------------------------------------+   
>                                                                                 
>                                                                                                                                                                 
>                         netperf.time.voluntary_context_switches                 
>                                                                                 
>    2.1e+08 +----------------------------------------------------------------+   
>            |.....+.....+.....+.....+.....+....+.....   ...+..         +.....|   
>   2.05e+08 |-+                                      +..                     |   
>            |                                                                |   
>            |                                                                |   
>      2e+08 |-+                                                              |   
>            |                                                                |   
>   1.95e+08 |-+                                                              |   
>            |                                                                |   
>    1.9e+08 |-+                                                              |   
>            |                                                                |   
>            |                                                                |   
>   1.85e+08 |-+   O     O     O     O     O          O     O                 |   
>            |                                  O                 O     O     |   
>    1.8e+08 +----------------------------------------------------------------+   
>                                                                                 
>                                                                                                                                                                 
>                                                                                 
>                                                                                 
>    0.006 +------------------------------------------------------------------+   
>          |                                                                  |   
>          |                                                                  |   
>   0.0058 |-+                                                                |   
>          |                                                                  |   
>          |                                                                  |   
>   0.0056 |-+                                                                |   
>          |                                                                  |   
>   0.0054 |-+                                                                |   
>          |                                                                  |   
>          |                                                                  |   
>   0.0052 |-+                                                                |   
>          |                                                                  |   
>          |                                                                  |   
>    0.005 +------------------------------------------------------------------+   
>                                                                                 
>                                                                                                                                                                 
>                                                                                 
>                                                                                 
>   3.25e+06 +----------------------------------------------------------------+   
>            |.....   ...+....          ...+....+.....+.....+.....   ...+.....|   
>    3.2e+06 |-+   +..        .   ...+..                          +..         |   
>            |                 +..                                            |   
>   3.15e+06 |-+                                                              |   
>    3.1e+06 |-+                                                              |   
>            |                                                                |   
>   3.05e+06 |-+                                                              |   
>            |                                                                |   
>      3e+06 |-+                                                              |   
>   2.95e+06 |-+                                                              |   
>            |                                                                |   
>    2.9e+06 |-+   O     O     O           O    O     O     O     O     O     |   
>            |                       O                                        |   
>   2.85e+06 +----------------------------------------------------------------+   
>                                                                                 
>                                                                                 
> [*] bisect-good sample
> [O] bisect-bad  sample
> 
> ***************************************************************************************************
> lkp-icl-2sp1: 96 threads 2 sockets Ice Lake with 256G memory
> 
> 
> 
> 
> 
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
> 
> 
> ---
> 0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
> https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation
> 
> Thanks,
> Oliver Sang
> 


> #!/bin/sh
> 
> export_top_env()
> {
> 	export suite='netperf'
> 	export testcase='netperf'
> 	export category='benchmark'
> 	export disable_latency_stats=1
> 	export set_nic_irq_affinity=1
> 	export ip='ipv4'
> 	export runtime=300
> 	export nr_threads=16
> 	export cluster='cs-localhost'
> 	export job_origin='netperf-small-threads.yaml'
> 	export queue_cmdline_keys=
> 	export queue='vip'
> 	export testbox='lkp-csl-2ap3'
> 	export tbox_group='lkp-csl-2ap3'
> 	export kconfig='x86_64-rhel-8.3'
> 	export submit_id='617960e80b9a930a5af4f104'
> 	export job_file='/lkp/jobs/scheduled/lkp-csl-2ap3/netperf-cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006-debian-10.4-x86_64-20200603.cgz-a0918006f9284b77397ae4f163-20211027-68186-ja0nr3-6.yaml'
> 	export id='fbecc857f957790eb9cfac7363705ffadfda23f9'
> 	export queuer_version='/lkp/xsang/.src-20211027-151141'
> 	export model='Cascade Lake'
> 	export nr_node=4
> 	export nr_cpu=192
> 	export memory='192G'
> 	export ssd_partitions=
> 	export rootfs_partition='LABEL=LKP-ROOTFS'
> 	export kernel_cmdline_hw='acpi_rsdp=0x67f44014'
> 	export brand='Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz'
> 	export commit='a0918006f9284b77397ae4f163f055c3e0f987b2'
> 	export need_kconfig_hw='{"IGB"=>"y"}
> BLK_DEV_NVME'
> 	export ucode='0x5003006'
> 	export enqueue_time='2021-10-27 22:23:36 +0800'
> 	export _id='617960f00b9a930a5af4f10a'
> 	export _rt='/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2'
> 	export user='lkp'
> 	export compiler='gcc-9'
> 	export LKP_SERVER='internal-lkp-server'
> 	export head_commit='955f175760f41ad2a80b07a390bac9a0444a47a6'
> 	export base_commit='519d81956ee277b4419c723adfb154603c2565ba'
> 	export branch='linux-devel/devel-hourly-20211025-030231'
> 	export rootfs='debian-10.4-x86_64-20200603.cgz'
> 	export result_root='/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/8'
> 	export scheduler_version='/lkp/lkp/.src-20211027-140142'
> 	export arch='x86_64'
> 	export max_uptime=2100
> 	export initrd='/osimage/debian/debian-10.4-x86_64-20200603.cgz'
> 	export bootloader_append='root=/dev/ram0
> user=lkp
> job=/lkp/jobs/scheduled/lkp-csl-2ap3/netperf-cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006-debian-10.4-x86_64-20200603.cgz-a0918006f9284b77397ae4f163-20211027-68186-ja0nr3-6.yaml
> ARCH=x86_64
> kconfig=x86_64-rhel-8.3
> branch=linux-devel/devel-hourly-20211025-030231
> commit=a0918006f9284b77397ae4f163f055c3e0f987b2
> BOOT_IMAGE=/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/vmlinuz-5.15.0-rc4-00001-ga0918006f928
> acpi_rsdp=0x67f44014
> max_uptime=2100
> RESULT_ROOT=/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/8
> LKP_SERVER=internal-lkp-server
> nokaslr
> selinux=0
> debug
> apic=debug
> sysrq_always_enabled
> rcupdate.rcu_cpu_stall_timeout=100
> net.ifnames=0
> printk.devkmsg=on
> panic=-1
> softlockup_panic=1
> nmi_watchdog=panic
> oops=panic
> load_ramdisk=2
> prompt_ramdisk=0
> drbd.minor_count=8
> systemd.log_level=err
> ignore_loglevel
> console=tty0
> earlyprintk=ttyS0,115200
> console=ttyS0,115200
> vga=normal
> rw'
> 	export modules_initrd='/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/modules.cgz'
> 	export bm_initrd='/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/netperf_20210930.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/netperf-x86_64-2.7-0_20211027.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/mpstat_20200714.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/turbostat_20200721.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/turbostat-x86_64-3.7-4_20200721.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/perf_20211027.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/perf-x86_64-d25f27432f80-1_20211027.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/sar-x86_64-34c92ae-1_20200702.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/hw_20200715.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/cluster_20211026.cgz'
> 	export ucode_initrd='/osimage/ucode/intel-ucode-20210222.cgz'
> 	export lkp_initrd='/osimage/user/lkp/lkp-x86_64.cgz'
> 	export site='inn'
> 	export LKP_CGI_PORT=80
> 	export LKP_CIFS_PORT=139
> 	export last_kernel='5.15.0-rc6-wt-12022-g955f175760f4'
> 	export queue_at_least_once=0
> 	export schedule_notify_address=
> 	export kernel='/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/vmlinuz-5.15.0-rc4-00001-ga0918006f928'
> 	export dequeue_time='2021-10-28 03:03:20 +0800'
> 	export node_roles='server client'
> 	export job_initrd='/lkp/jobs/scheduled/lkp-csl-2ap3/netperf-cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006-debian-10.4-x86_64-20200603.cgz-a0918006f9284b77397ae4f163-20211027-68186-ja0nr3-6.cgz'
> 
> 	[ -n "$LKP_SRC" ] ||
> 	export LKP_SRC=/lkp/${user:-lkp}/src
> }
> 
> run_job()
> {
> 	echo $$ > $TMP/run-job.pid
> 
> 	. $LKP_SRC/lib/http.sh
> 	. $LKP_SRC/lib/job.sh
> 	. $LKP_SRC/lib/env.sh
> 
> 	export_top_env
> 
> 	run_setup $LKP_SRC/setup/cpufreq_governor 'performance'
> 
> 	run_monitor $LKP_SRC/monitors/wrapper kmsg
> 	run_monitor $LKP_SRC/monitors/no-stdout/wrapper boot-time
> 	run_monitor $LKP_SRC/monitors/wrapper uptime
> 	run_monitor $LKP_SRC/monitors/wrapper iostat
> 	run_monitor $LKP_SRC/monitors/wrapper heartbeat
> 	run_monitor $LKP_SRC/monitors/wrapper vmstat
> 	run_monitor $LKP_SRC/monitors/wrapper numa-numastat
> 	run_monitor $LKP_SRC/monitors/wrapper numa-vmstat
> 	run_monitor $LKP_SRC/monitors/wrapper numa-meminfo
> 	run_monitor $LKP_SRC/monitors/wrapper proc-vmstat
> 	run_monitor $LKP_SRC/monitors/wrapper proc-stat
> 	run_monitor $LKP_SRC/monitors/wrapper meminfo
> 	run_monitor $LKP_SRC/monitors/wrapper slabinfo
> 	run_monitor $LKP_SRC/monitors/wrapper interrupts
> 	run_monitor $LKP_SRC/monitors/wrapper lock_stat
> 	run_monitor lite_mode=1 $LKP_SRC/monitors/wrapper perf-sched
> 	run_monitor $LKP_SRC/monitors/wrapper softirqs
> 	run_monitor $LKP_SRC/monitors/one-shot/wrapper bdi_dev_mapping
> 	run_monitor $LKP_SRC/monitors/wrapper diskstats
> 	run_monitor $LKP_SRC/monitors/wrapper nfsstat
> 	run_monitor $LKP_SRC/monitors/wrapper cpuidle
> 	run_monitor $LKP_SRC/monitors/wrapper cpufreq-stats
> 	run_monitor $LKP_SRC/monitors/wrapper turbostat
> 	run_monitor $LKP_SRC/monitors/wrapper sched_debug
> 	run_monitor $LKP_SRC/monitors/wrapper perf-stat
> 	run_monitor $LKP_SRC/monitors/wrapper mpstat
> 	run_monitor lite_mode=1 $LKP_SRC/monitors/no-stdout/wrapper perf-profile
> 	run_monitor $LKP_SRC/monitors/wrapper oom-killer
> 	run_monitor $LKP_SRC/monitors/plain/watchdog
> 
> 	if role server
> 	then
> 		start_daemon $LKP_SRC/daemon/netserver
> 	fi
> 
> 	if role client
> 	then
> 		run_test test='TCP_CRR' $LKP_SRC/tests/wrapper netperf
> 	fi
> }
> 
> extract_stats()
> {
> 	export stats_part_begin=
> 	export stats_part_end=
> 
> 	env test='TCP_CRR' $LKP_SRC/stats/wrapper netperf
> 	$LKP_SRC/stats/wrapper kmsg
> 	$LKP_SRC/stats/wrapper boot-time
> 	$LKP_SRC/stats/wrapper uptime
> 	$LKP_SRC/stats/wrapper iostat
> 	$LKP_SRC/stats/wrapper vmstat
> 	$LKP_SRC/stats/wrapper numa-numastat
> 	$LKP_SRC/stats/wrapper numa-vmstat
> 	$LKP_SRC/stats/wrapper numa-meminfo
> 	$LKP_SRC/stats/wrapper proc-vmstat
> 	$LKP_SRC/stats/wrapper meminfo
> 	$LKP_SRC/stats/wrapper slabinfo
> 	$LKP_SRC/stats/wrapper interrupts
> 	$LKP_SRC/stats/wrapper lock_stat
> 	env lite_mode=1 $LKP_SRC/stats/wrapper perf-sched
> 	$LKP_SRC/stats/wrapper softirqs
> 	$LKP_SRC/stats/wrapper diskstats
> 	$LKP_SRC/stats/wrapper nfsstat
> 	$LKP_SRC/stats/wrapper cpuidle
> 	$LKP_SRC/stats/wrapper turbostat
> 	$LKP_SRC/stats/wrapper sched_debug
> 	$LKP_SRC/stats/wrapper perf-stat
> 	$LKP_SRC/stats/wrapper mpstat
> 	env lite_mode=1 $LKP_SRC/stats/wrapper perf-profile
> 
> 	$LKP_SRC/stats/wrapper time netperf.time
> 	$LKP_SRC/stats/wrapper dmesg
> 	$LKP_SRC/stats/wrapper kmsg
> 	$LKP_SRC/stats/wrapper last_state
> 	$LKP_SRC/stats/wrapper stderr
> 	$LKP_SRC/stats/wrapper time
> }
> 
> "$@"

> ---
> :#! jobs/netperf-small-threads.yaml:
> suite: netperf
> testcase: netperf
> category: benchmark
> :# upto 90% CPU cycles may be used by latency stats:
> disable_latency_stats: 1
> set_nic_irq_affinity: 1
> ip: ipv4
> runtime: 300s
> nr_threads: 16
> cluster: cs-localhost
> if role server:
>   netserver:
> if role client:
>   netperf:
>     test: TCP_CRR
> job_origin: netperf-small-threads.yaml
> :#! queue options:
> queue_cmdline_keys: []
> queue: vip
> testbox: lkp-csl-2ap3
> tbox_group: lkp-csl-2ap3
> kconfig: x86_64-rhel-8.3
> submit_id: 617960e80b9a930a5af4f104
> job_file: "/lkp/jobs/scheduled/lkp-csl-2ap3/netperf-cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006-debian-10.4-x86_64-20200603.cgz-a0918006f9284b77397ae4f163-20211027-68186-ja0nr3-4.yaml"
> id: d804900eed74d058b23143a825d247e0f8d03392
> queuer_version: "/lkp/xsang/.src-20211027-151141"
> :#! hosts/lkp-csl-2ap3:
> model: Cascade Lake
> nr_node: 4
> nr_cpu: 192
> memory: 192G
> ssd_partitions:
> rootfs_partition: LABEL=LKP-ROOTFS
> kernel_cmdline_hw: acpi_rsdp=0x67f44014
> brand: Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz
> :#! include/category/benchmark:
> kmsg:
> boot-time:
> uptime:
> iostat:
> heartbeat:
> vmstat:
> numa-numastat:
> numa-vmstat:
> numa-meminfo:
> proc-vmstat:
> proc-stat:
> meminfo:
> slabinfo:
> interrupts:
> lock_stat:
> perf-sched:
>   lite_mode: 1
> softirqs:
> bdi_dev_mapping:
> diskstats:
> nfsstat:
> cpuidle:
> cpufreq-stats:
> turbostat:
> sched_debug:
> perf-stat:
> mpstat:
> perf-profile:
>   lite_mode: 1
> :#! include/category/ALL:
> cpufreq_governor: performance
> :#! include/queue/cyclic:
> commit: a0918006f9284b77397ae4f163f055c3e0f987b2
> :#! include/testbox/lkp-csl-2ap3:
> need_kconfig_hw:
> - IGB: y
> - BLK_DEV_NVME
> ucode: '0x5003006'
> enqueue_time: 2021-10-27 22:23:36.647383432 +08:00
> _id: 617960f00b9a930a5af4f108
> _rt: "/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2"
> :#! schedule options:
> user: lkp
> compiler: gcc-9
> LKP_SERVER: internal-lkp-server
> head_commit: 955f175760f41ad2a80b07a390bac9a0444a47a6
> base_commit: 519d81956ee277b4419c723adfb154603c2565ba
> branch: linux-devel/devel-hourly-20211025-030231
> rootfs: debian-10.4-x86_64-20200603.cgz
> result_root: "/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/0"
> scheduler_version: "/lkp/lkp/.src-20211027-140142"
> arch: x86_64
> max_uptime: 2100
> initrd: "/osimage/debian/debian-10.4-x86_64-20200603.cgz"
> bootloader_append:
> - root=/dev/ram0
> - user=lkp
> - job=/lkp/jobs/scheduled/lkp-csl-2ap3/netperf-cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006-debian-10.4-x86_64-20200603.cgz-a0918006f9284b77397ae4f163-20211027-68186-ja0nr3-4.yaml
> - ARCH=x86_64
> - kconfig=x86_64-rhel-8.3
> - branch=linux-devel/devel-hourly-20211025-030231
> - commit=a0918006f9284b77397ae4f163f055c3e0f987b2
> - BOOT_IMAGE=/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/vmlinuz-5.15.0-rc4-00001-ga0918006f928
> - acpi_rsdp=0x67f44014
> - max_uptime=2100
> - RESULT_ROOT=/result/netperf/cs-localhost-performance-ipv4-16-300s-TCP_CRR-ucode=0x5003006/lkp-csl-2ap3/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/0
> - LKP_SERVER=internal-lkp-server
> - nokaslr
> - selinux=0
> - debug
> - apic=debug
> - sysrq_always_enabled
> - rcupdate.rcu_cpu_stall_timeout=100
> - net.ifnames=0
> - printk.devkmsg=on
> - panic=-1
> - softlockup_panic=1
> - nmi_watchdog=panic
> - oops=panic
> - load_ramdisk=2
> - prompt_ramdisk=0
> - drbd.minor_count=8
> - systemd.log_level=err
> - ignore_loglevel
> - console=tty0
> - earlyprintk=ttyS0,115200
> - console=ttyS0,115200
> - vga=normal
> - rw
> modules_initrd: "/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/modules.cgz"
> bm_initrd: "/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/netperf_20210930.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/netperf-x86_64-2.7-0_20211025.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/mpstat_20200714.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/turbostat_20200721.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/turbostat-x86_64-3.7-4_20200721.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/perf_20211027.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/perf-x86_64-d25f27432f80-1_20211027.cgz,/osimage/pkg/debian-10.4-x86_64-20200603.cgz/sar-x86_64-34c92ae-1_20200702.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/hw_20200715.cgz,/osimage/deps/debian-10.4-x86_64-20200603.cgz/cluster_20211026.cgz"
> ucode_initrd: "/osimage/ucode/intel-ucode-20210222.cgz"
> lkp_initrd: "/osimage/user/lkp/lkp-x86_64.cgz"
> site: inn
> :#! /lkp/lkp/.src-20211026-143536/include/site/inn:
> LKP_CGI_PORT: 80
> LKP_CIFS_PORT: 139
> oom-killer:
> watchdog:
> :#! runtime status:
> last_kernel: 5.15.0-rc6-wt-12022-g955f175760f4
> queue_at_least_once: 0
> :#! /lkp/lkp/.src-20211026-205023/include/site/inn:
> :#! user overrides:
> schedule_notify_address:
> kernel: "/pkg/linux/x86_64-rhel-8.3/gcc-9/a0918006f9284b77397ae4f163f055c3e0f987b2/vmlinuz-5.15.0-rc4-00001-ga0918006f928"
> dequeue_time: 2021-10-28 02:02:15.174367353 +08:00
> :#! /lkp/lkp/.src-20211027-140142/include/site/inn:
> job_state: finished
> loadavg: 6.09 10.58 5.53 1/1355 19660
> start_time: '1635357811'
> end_time: '1635358116'
> version: "/lkp/lkp/.src-20211027-140222:5f87ddf4:8610dc698"

> 
> for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
> do
> 	online_file="$cpu_dir"/online
> 	[ -f "$online_file" ] && [ "$(cat "$online_file")" -eq 0 ] && continue
> 
> 	file="$cpu_dir"/cpufreq/scaling_governor
> 	[ -f "$file" ] && echo "performance" > "$file"
> done
> 
> netserver -4 -D
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> netperf -4 -H 127.0.0.1 -t TCP_CRR -c -C -l 300  &
> wait


-- 
Kees Cook

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [fs] a0918006f9: netperf.Throughput_tps -11.6% regression
  2021-11-09 17:21     ` Kees Cook
@ 2021-11-10  1:54       ` Yin Fengwei
  2021-11-10  8:52       ` Mickaël Salaün
  2021-11-12 12:25       ` Yin Fengwei
  2 siblings, 0 replies; 10+ messages in thread
From: Yin Fengwei @ 2021-11-10  1:54 UTC (permalink / raw)
  To: Kees Cook, kernel test robot
  Cc: Mickaël Salaün, lkp, lkp, ying.huang, feng.tang,
	zhengjun.xing, Al Viro, Andrew Morton, Aleksa Sarai,
	Andy Lutomirski, Arnd Bergmann, Casey Schaufler,
	Christian Brauner, Christian Heimes, Deven Bowers, Dmitry Vyukov,
	Eric Biggers, Eric Chiang, Florian Weimer, Geert Uytterhoeven,
	James Morris, Jan Kara, Jann Horn, Jonathan Corbet,
	Lakshmi Ramasubramanian, Madhavan T . Venkataraman,
	Matthew Garrett, Matthew Wilcox, Miklos Szeredi, Mimi Zohar,
	Paul Moore, Philippe Trébuchet, Scott Shell, Shuah Khan,
	Steve Dower, Steve Grubb, Thibaut Sautereau, Vincent Strubel,
	kernel-hardening, linux-api, linux-fsdevel, linux-integrity,
	linux-kernel, linux-security-module, Mickaël Salaün

Hi Kees,

On 11/10/21 1:21 AM, Kees Cook wrote:
>>     555600            -0.1%     555305        netperf.Throughput_total_tps
>>      34725            -0.1%      34706        netperf.Throughput_tps
>>
>>
>> Fengwei also helped review these results and commented:
>> I suppose these three CPUs have different cache policy. It also could be
>> related with netperf throughput testing.
> Does moving the syscall implementation somewhere else change things?
> That's a _huge_ performance change for something that isn't even called.
> What's going on here?

We just tried to do trick change to make sure the new function doesn't
make other kernel function address changed. But didn't try to move around
the new function even it's not called. We could try to move it around to
see the impact to netperf throughput.


We tried the original patch (without change to make sure no kernel function
address patch) on other box. As report, the regression are different on
different CPUs like:
       -11.6% vs -5.7% vs 0.1%

So my guess is that the different CPUs on these test box have different
cache policies which cause different performance impact when kernel
functions/data address are different. 

Yes. This is strange. We don't know exact reason. Need deeper investigation
on this.


Regards
Yin, Fengwei

> 
> -Kees
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [fs] a0918006f9: netperf.Throughput_tps -11.6% regression
  2021-11-09 17:21     ` Kees Cook
  2021-11-10  1:54       ` Yin Fengwei
@ 2021-11-10  8:52       ` Mickaël Salaün
  2021-11-11  3:29         ` Yin Fengwei
  2021-11-12 12:25       ` Yin Fengwei
  2 siblings, 1 reply; 10+ messages in thread
From: Mickaël Salaün @ 2021-11-10  8:52 UTC (permalink / raw)
  To: Kees Cook, kernel test robot
  Cc: lkp, lkp, ying.huang, feng.tang, zhengjun.xing, fengwei.yin,
	Al Viro, Andrew Morton, Aleksa Sarai, Andy Lutomirski,
	Arnd Bergmann, Casey Schaufler, Christian Brauner,
	Christian Heimes, Deven Bowers, Dmitry Vyukov, Eric Biggers,
	Eric Chiang, Florian Weimer, Geert Uytterhoeven, James Morris,
	Jan Kara, Jann Horn, Jonathan Corbet, Lakshmi Ramasubramanian,
	Madhavan T . Venkataraman, Matthew Garrett, Matthew Wilcox,
	Miklos Szeredi, Mimi Zohar, Paul Moore, Philippe Trébuchet,
	Scott Shell, Shuah Khan, Steve Dower, Steve Grubb,
	Thibaut Sautereau, Vincent Strubel, kernel-hardening, linux-api,
	linux-fsdevel, linux-integrity, linux-kernel,
	linux-security-module, Mickaël Salaün


On 09/11/2021 18:21, Kees Cook wrote:
> On Fri, Nov 05, 2021 at 02:41:59PM +0800, kernel test robot wrote:
>>
>>
>> Greeting,
>>
>> FYI, we noticed a -11.6% regression of netperf.Throughput_tps due to commit:
>>
>>
>> commit: a0918006f9284b77397ae4f163f055c3e0f987b2 ("[PATCH v15 1/3] fs: Add trusted_for(2) syscall implementation and related sysctl")
>> url: https://github.com/0day-ci/linux/commits/Micka-l-Sala-n/Add-trusted_for-2-was-O_MAYEXEC/20211013-032533
>> patch link: https://lore.kernel.org/kernel-hardening/20211012192410.2356090-2-mic@digikod.net
>>
>> in testcase: netperf
>> on test machine: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
>> with following parameters:
>>
>> 	ip: ipv4
>> 	runtime: 300s
>> 	nr_threads: 16
>> 	cluster: cs-localhost
>> 	test: TCP_CRR
>> 	cpufreq_governor: performance
>> 	ucode: 0x5003006
>>
>> test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
>> test-url: http://www.netperf.org/netperf/
>>
>>
>> please be noted we made out some further analysis/tests, as Fengwei mentioned:
>> ==============================================================================
>> Here is my investigation result of this regression:
>>
>> If I add patch to make sure the kernel function address and data address is
>> almost same even with this patch, there is almost no performance delta(0.1%)
>> w/o the patch.
>>
>> And if I only make sure function address same w/o the patch, the performance
>> delta is about 5.1%.
>>
>> So suppose this regression is triggered by different function and data address.
>> We don't know why the different address could bring such kind of regression yet
>> ===============================================================================
>>
>>
>> we also tested on other platforms.
>> on a Cooper Lake (Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz with 128G memory),
>> we also observed regression but the gap is smaller:
>> =========================================================================================
>> cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
>>   cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-cpl-4sp1/TCP_CRR/netperf/0x700001e
>>
>> commit:
>>   v5.15-rc4
>>   a0918006f9284b77397ae4f163f055c3e0f987b2
>>
>>        v5.15-rc4 a0918006f9284b77397ae4f163f
>> ---------------- ---------------------------
>>          %stddev     %change         %stddev
>>              \          |                \
>>     333492            -5.7%     314346 ±  2%  netperf.Throughput_total_tps
>>      20843            -4.5%      19896        netperf.Throughput_tps
>>
>>
>> but no regression on a 96 threads 2 sockets Ice Lake with 256G memory:
>> =========================================================================================
>> cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
>>   cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-icl-2sp1/TCP_CRR/netperf/0xb000280
>>
>> commit:
>>   v5.15-rc4
>>   a0918006f9284b77397ae4f163f055c3e0f987b2
>>
>>        v5.15-rc4 a0918006f9284b77397ae4f163f
>> ---------------- ---------------------------
>>          %stddev     %change         %stddev
>>              \          |                \
>>     555600            -0.1%     555305        netperf.Throughput_total_tps
>>      34725            -0.1%      34706        netperf.Throughput_tps
>>
>>
>> Fengwei also helped review these results and commented:
>> I suppose these three CPUs have different cache policy. It also could be
>> related with netperf throughput testing.
> 
> Does moving the syscall implementation somewhere else change things?
> That's a _huge_ performance change for something that isn't even called.
> What's going on here?

This regression doesn't make sense. I guess this is the result of a
flaky netperf test, maybe because the test machine was overloaded at
that time.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [fs] a0918006f9: netperf.Throughput_tps -11.6% regression
  2021-11-10  8:52       ` Mickaël Salaün
@ 2021-11-11  3:29         ` Yin Fengwei
  0 siblings, 0 replies; 10+ messages in thread
From: Yin Fengwei @ 2021-11-11  3:29 UTC (permalink / raw)
  To: Mickaël Salaün, Kees Cook, kernel test robot
  Cc: lkp, lkp, ying.huang, feng.tang, zhengjun.xing, Al Viro,
	Andrew Morton, Aleksa Sarai, Andy Lutomirski, Arnd Bergmann,
	Casey Schaufler, Christian Brauner, Christian Heimes,
	Deven Bowers, Dmitry Vyukov, Eric Biggers, Eric Chiang,
	Florian Weimer, Geert Uytterhoeven, James Morris, Jan Kara,
	Jann Horn, Jonathan Corbet, Lakshmi Ramasubramanian,
	Madhavan T . Venkataraman, Matthew Garrett, Matthew Wilcox,
	Miklos Szeredi, Mimi Zohar, Paul Moore, Philippe Trébuchet,
	Scott Shell, Shuah Khan, Steve Dower, Steve Grubb,
	Thibaut Sautereau, Vincent Strubel, kernel-hardening, linux-api,
	linux-fsdevel, linux-integrity, linux-kernel,
	linux-security-module, Mickaël Salaün


On 11/10/2021 4:52 PM, Mickaël Salaün wrote:
>>> ---------------- ---------------------------
>>>          %stddev     %change         %stddev
>>>              \          |                \
>>>     555600            -0.1%     555305        netperf.Throughput_total_tps
>>>      34725            -0.1%      34706        netperf.Throughput_tps
>>>
>>>
>>> Fengwei also helped review these results and commented:
>>> I suppose these three CPUs have different cache policy. It also could be
>>> related with netperf throughput testing.
>> Does moving the syscall implementation somewhere else change things?
>> That's a _huge_ performance change for something that isn't even called.
>> What's going on here?
> This regression doesn't make sense. I guess this is the result of a
> flaky netperf test, maybe because the test machine was overloaded at
> that time.

I agree the test result looks strange. But I don't think the test machine
or test methodology has issue. It's not possible the test box is overloaded
when test case is running. We did test several times (> 12 times) on different
days. Thanks.


Regards
Yin, Fengwei

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [fs] a0918006f9: netperf.Throughput_tps -11.6% regression
  2021-11-09 17:21     ` Kees Cook
  2021-11-10  1:54       ` Yin Fengwei
  2021-11-10  8:52       ` Mickaël Salaün
@ 2021-11-12 12:25       ` Yin Fengwei
  2 siblings, 0 replies; 10+ messages in thread
From: Yin Fengwei @ 2021-11-12 12:25 UTC (permalink / raw)
  To: Kees Cook, kernel test robot
  Cc: Mickaël Salaün, lkp, lkp, ying.huang, feng.tang,
	zhengjun.xing, Al Viro, Andrew Morton, Aleksa Sarai,
	Andy Lutomirski, Arnd Bergmann, Casey Schaufler,
	Christian Brauner, Christian Heimes, Deven Bowers, Dmitry Vyukov,
	Eric Biggers, Eric Chiang, Florian Weimer, Geert Uytterhoeven,
	James Morris, Jan Kara, Jann Horn, Jonathan Corbet,
	Lakshmi Ramasubramanian, Madhavan T . Venkataraman,
	Matthew Garrett, Matthew Wilcox, Miklos Szeredi, Mimi Zohar,
	Paul Moore, Philippe Trébuchet, Scott Shell, Shuah Khan,
	Steve Dower, Steve Grubb, Thibaut Sautereau, Vincent Strubel,
	kernel-hardening, linux-api, linux-fsdevel, linux-integrity,
	linux-kernel, linux-security-module, Mickaël Salaün

Hi Kees,

On 11/10/2021 1:21 AM, Kees Cook wrote:
>> I suppose these three CPUs have different cache policy. It also could be
>> related with netperf throughput testing.
> Does moving the syscall implementation somewhere else change things?
I moved the syscall implementation to a stand alone file and put the file
to net directory for testing. The new patch is like:
https://zerobin.net/?a2b782afadf3c428#Me8l4AJuhiSCfaLVWVzydAVIK6ves0EVIVD76wLnVQo=


The test result is like following:
     - on Cascade Lake: -10.4%
    356365           -10.4%     319180        netperf.Throughput_total_tps
     22272           -10.4%      19948        netperf.Throughput_tps


     - on Cooper Lake: -4.0%
    345772 ±  4%      -4.0%     331814        netperf.Throughput_total_tps
     21610 ±  4%      -4.0%      20738        netperf.Throughput_tps


     - on Ice Lake: -1.1%
    509824            -1.1%     504434        netperf.Throughput_total_tps
     31864            -1.1%      31527        netperf.Throughput_tps


Regards
Yin, Fengwei

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2021-11-12 12:34 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-12 19:24 [PATCH v15 0/3] Add trusted_for(2) (was O_MAYEXEC) Mickaël Salaün
2021-10-12 19:24 ` [PATCH v15 1/3] fs: Add trusted_for(2) syscall implementation and related sysctl Mickaël Salaün
2021-11-05  6:41   ` [fs] a0918006f9: netperf.Throughput_tps -11.6% regression kernel test robot
2021-11-09 17:21     ` Kees Cook
2021-11-10  1:54       ` Yin Fengwei
2021-11-10  8:52       ` Mickaël Salaün
2021-11-11  3:29         ` Yin Fengwei
2021-11-12 12:25       ` Yin Fengwei
2021-10-12 19:24 ` [PATCH v15 2/3] arch: Wire up trusted_for(2) Mickaël Salaün
2021-10-12 19:24 ` [PATCH v15 3/3] selftest/interpreter: Add tests for trusted_for(2) policies Mickaël Salaün

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).