linux-cxl.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Ben Widawsky <ben.widawsky@intel.com>
To: linux-cxl@vger.kernel.org, Chet Douglas <chet.r.douglas@intel.com>
Cc: Ben Widawsky <ben.widawsky@intel.com>,
	Alison Schofield <alison.schofield@intel.com>,
	Dan Williams <dan.j.williams@intel.com>,
	Ira Weiny <ira.weiny@intel.com>,
	Jonathan Cameron <Jonathan.Cameron@Huawei.com>,
	Vishal Verma <vishal.l.verma@intel.com>
Subject: [RFC PATCH v2 00/28] CXL Region Creation / HDM decoder programming
Date: Fri, 22 Oct 2021 11:36:41 -0700	[thread overview]
Message-ID: <20211022183709.1199701-1-ben.widawsky@intel.com> (raw)

Because v1 wasn't reviewed, and I quickly asked people to ignore it in favor of
this version, I'm not going to bother listing out the changes. I've only issued minor
edits to the original cover letter. If you already read that, feel free to move
along.

I don't have plans to send further RFCs until I receive feedback, good or bad.

CXL region creation
-------------------
An interleaved set of devices is known in the Compute Express Link [1]
specification as a region. In addition to a region being comprised of devices
that can be configured in a variety of orders there are other properties that
define a region. It is expected that a region may be created as part of
provisioning by the hardware vendor, or interactively via operating system
controls. This patch series implements both the interfaces to create and
configure a region, as well as the algorithm to program the HDM decoders to
enable CXL.mem traffic while obeying those configuration parameters. The series
stops short of storing the new regions in the Label Storage Area of the CXL
devices.

Some version of this functionality has all been posted previously, with the
exception of the actual HDM decoder programming. It's probably wise to forget
those exist, and take my apology in advance for not addressing feedback you may
have already given.

There are two branches I am using as development branches. The branch for
port/mem driver [2] is fairly solid. The branch for region creation [3] is less
baked.

cxl_port
========

The cxl_port driver is implemented within the cxl_port module. While loading of
this module is optional, the other new drivers depend on it. The port driver is
responsible for all activities around HDM decoder enumeration and programming.
Introduced earlier, the concept of a port is an abstraction over CXL components
with an upstream port, every host bridge, switch, and endpoint.

cxl_mem
=======

The cxl_mem driver's main job is to walk up the hierarchy to make the
determination if it is CXL.mem routed, meaning, all components above it in the
hierarchy are participating in the CXL.mem protocol. It is implemented within
the cxl_mem module. As the host bridge ports are added by a platform specific
driver, such as cxl_acpi, the scope of the mem driver can be reduced to scan for
switches and ask cxl_core to work on enumerating them. With this done, the
determination as to whether a device is CXL.mem routed can be done simply by
checking if the struct device has a driver bound to it.

This driver is also a logical place to migrate certain functionality from
cxl_pci. That is saved for later.

cxl_region
==========

Region verification and programming state are owned by the cxl_region driver
(implemented in the cxl_region module). It relies on cxl_mem to determine if
devices are CXL routed, and cxl_port to actually handle the programming of the
HDM decoders. Much of the region driver is an implementation of algorithms
described in the CXL Type 3 Memory Device Software Guide [4].

The patches for the region driver could be squashed. They're broken out to aid
review and because that's the order they were implemented in. My preference is
to keep those as they are.

Why RFC?
--------
While I think most of the architecture is sound, I don't believe anyone but
other developers should use this branch. Where I'd really like most eyes:
- Locking and device lifetimes. I suspect I have many bugs in this area.
- Region configuration. Should values have a default, should they all be
  explicit?
- What should/shouldn't be in core. I like how this ended up, but at this point
  I'm fairly biased (CXL.cache pun).
- What to extend to cxl_test.

What's missing
---------------
- CXL 2.0 switch support
- A full topology lock for programming HDM decoders. I'm looking for feedback on
  the best way to do this.
- Check that HDM decoder programming addresses are correct (must program higher
  addresses only)
- Volatile regions (or BIOS configured persistent regions)
- Connection to libnvdimm/labels. This includes many aspects, not the least of
  which is saving the region into the Label Storage Area so that it can be
  reestablished on reboot.

Here is an example of output when programming a x1 interleave region:
[   23.959814][  T645] cxl_core:cxl_add_region:406: cxl region0.0:0: Added region0.0:0 to decoder0.0
[   23.962972][  T645] cxl_port:cxl_commit_decoder:248: cxl_port port1: decoder1.0
[   23.962972][  T645] 	Base 0x0000004c00000000
[   23.962972][  T645] 	Size 268435456
[   23.962972][  T645] 	IG 256
[   23.962972][  T645] 	IW 1
[   23.962972][  T645] 	TargetList: 0 -1 -1 -1 -1 -1 -1 -1
[   23.965529][  T645] cxl_port:cxl_commit_decoder:248: cxl_port port3: decoder3.0
[   23.965529][  T645] 	Base 0x0000004c00000000
[   23.965529][  T645] 	Size 268435456
[   23.965529][  T645] 	IG 256
[   23.965529][  T645] 	IW 1
[   23.965529][  T645] 	TargetList: -1 -1 -1 -1 -1 -1 -1 -1

If you're wondering how I tested this, I've baked it into my cxlctl app [5] and
lib [6]. Eventually this will get absorbed by ndctl/cxl-cli/libcxl [7]. Region
deletion isn't implemented yet.

To get the detailed errors, trace-cmd can be utilized. Until a region device
exists, the region module will not be loaded, which means the region tracepoints
will not exist. To get around this, modprobe cxl_region before anything.

trace-cmd record -e cxl ./cxlctl create-region -n -a -s $((256<<20)) /sys/bus/cxl/devices/decoder0.0

Note: A minor bugfix is needed in QEMU if testing interleave configs. I've
pushed that to my v4 branch.
---

[1]: https://www.computeexpresslink.org/download-the-specification
[2]: https://gitlab.com/bwidawsk/linux/-/tree/cxl_port-v3
[3]: https://gitlab.com/bwidawsk/linux/-/tree/cxl_regions-v4
[4]: https://cdrdv2.intel.com/v1/dl/getContent/643805?wapkw=CXL%20memory%20device%20sw%20guide
[5]: https://gitlab.com/bwidawsk-cxl/cxlctl
[6]: https://gitlab.com/bwidawsk-cxl/cxl_rs
[7]: https://lore.kernel.org/linux-cxl/CAPcyv4joKOhTdaRBJVeoOtqhRjBvdtt9902TS=c39=zWTZXvuw@mail.gmail.com/
---

Ben Widawsky (28):
  cxl: Rename CXL_MEM to CXL_PCI
  cxl: Move register block enumeration to core
  cxl/acpi: Map component registers for Root Ports
  cxl: Add helper for new drivers
  cxl/core: Convert decoder range to resource
  cxl: Introduce endpoint decoders
  cxl/core: Move target population locking to caller
  cxl/port: Introduce a port driver
  cxl/acpi: Map single port host bridge component registers
  cxl/core: Store global list of root ports
  cxl/acpi: Rescan bus at probe completion
  cxl/core: Store component register base for memdevs
  cxl: Flesh out register names
  cxl: Hide devm host for ports
  cxl/core: Introduce API to scan switch ports
  cxl: Introduce cxl_mem driver
  cxl: Disable switch hierarchies for now
  cxl/region: Add region creation ABI
  cxl/region: Introduce concept of region configuration
  cxl/region: Introduce a cxl_region driver
  cxl/acpi: Handle address space allocation
  cxl/region: Address space allocation
  cxl/region: Implement XHB verification
  cxl/region: HB port config verification
  cxl/region: Record host bridge target list
  cxl/mem: Store the endpoint's uport
  cxl/region: Gather HDM decoder resources
  cxl: Program decoders for regions

 .clang-format                                 |   3 +
 Documentation/ABI/testing/sysfs-bus-cxl       |  63 ++
 .../driver-api/cxl/memory-devices.rst         |  28 +
 drivers/cxl/Kconfig                           |  28 +-
 drivers/cxl/Makefile                          |   8 +-
 drivers/cxl/acpi.c                            | 132 +++-
 drivers/cxl/core/Makefile                     |   2 +
 drivers/cxl/core/bus.c                        | 437 +++++++++++-
 drivers/cxl/core/core.h                       |   2 +
 drivers/cxl/core/memdev.c                     |   7 +-
 drivers/cxl/core/pci.c                        |  99 +++
 drivers/cxl/core/region.c                     | 453 +++++++++++++
 drivers/cxl/core/regs.c                       |  62 +-
 drivers/cxl/cxl.h                             |  85 ++-
 drivers/cxl/cxlmem.h                          |   8 +-
 drivers/cxl/mem.c                             | 162 +++++
 drivers/cxl/pci.c                             |  69 +-
 drivers/cxl/pci.h                             |  48 +-
 drivers/cxl/port.c                            | 491 ++++++++++++++
 drivers/cxl/region.c                          | 629 ++++++++++++++++++
 drivers/cxl/region.h                          |  57 ++
 drivers/cxl/trace.h                           |  75 +++
 tools/testing/cxl/Kbuild                      |   2 +
 tools/testing/cxl/mock_acpi.c                 |   4 +-
 tools/testing/cxl/test/mem.c                  |   3 +-
 25 files changed, 2806 insertions(+), 151 deletions(-)
 create mode 100644 drivers/cxl/core/pci.c
 create mode 100644 drivers/cxl/core/region.c
 create mode 100644 drivers/cxl/mem.c
 create mode 100644 drivers/cxl/port.c
 create mode 100644 drivers/cxl/region.c
 create mode 100644 drivers/cxl/region.h
 create mode 100644 drivers/cxl/trace.h

             reply	other threads:[~2021-10-22 18:37 UTC|newest]

Thread overview: 112+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-10-22 18:36 Ben Widawsky [this message]
2021-10-22 18:36 ` [RFC PATCH v2 01/28] cxl: Rename CXL_MEM to CXL_PCI Ben Widawsky
2021-10-29 20:15   ` Dan Williams
2021-10-29 21:20     ` Ben Widawsky
2021-10-29 21:39       ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 02/28] cxl: Move register block enumeration to core Ben Widawsky
2021-10-29 20:23   ` Dan Williams
2021-10-29 21:23     ` Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 03/28] cxl/acpi: Map component registers for Root Ports Ben Widawsky
2021-10-29 20:28   ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 04/28] cxl: Add helper for new drivers Ben Widawsky
2021-10-29 20:30   ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 05/28] cxl/core: Convert decoder range to resource Ben Widawsky
2021-10-29 20:50   ` Dan Williams
2021-10-29 21:26     ` Ben Widawsky
2021-10-29 22:22       ` Dan Williams
2021-10-29 22:37         ` Ben Widawsky
2021-11-01 14:33           ` Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 06/28] cxl: Introduce endpoint decoders Ben Widawsky
2021-10-29 21:00   ` Dan Williams
2021-10-29 22:02     ` Ben Widawsky
2021-10-29 22:25       ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 07/28] cxl/core: Move target population locking to caller Ben Widawsky
2021-10-29 23:03   ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 08/28] cxl/port: Introduce a port driver Ben Widawsky
2021-10-30  1:37   ` Dan Williams
2021-10-31 17:53     ` Dan Williams
2021-10-31 18:10       ` Dan Williams
2021-11-01 17:36         ` Ben Widawsky
2021-11-01 17:53     ` Ben Widawsky
2021-11-01 17:54       ` Ben Widawsky
2021-11-02  3:31       ` Dan Williams
2021-11-02 16:27         ` Ben Widawsky
2021-11-02 17:21           ` Dan Williams
2021-11-02 16:58         ` Ben Widawsky
2021-11-04 19:10           ` Dan Williams
2021-11-04 19:49             ` Ben Widawsky
2021-11-04 20:04               ` Dan Williams
2021-11-04 21:25                 ` Ben Widawsky
2021-11-04 16:37     ` Ben Widawsky
2021-11-04 19:17       ` Dan Williams
2021-11-04 19:46         ` Ben Widawsky
2021-11-04 20:00           ` Dan Williams
2021-11-04 21:26             ` Ben Widawsky
2021-11-03 15:18   ` Jonathan Cameron
2021-10-22 18:36 ` [RFC PATCH v2 09/28] cxl/acpi: Map single port host bridge component registers Ben Widawsky
2021-10-31 18:03   ` Dan Williams
2021-11-01 17:07     ` Ben Widawsky
2021-11-02  2:15       ` Dan Williams
2021-11-02 16:31         ` Ben Widawsky
2021-11-02 17:46           ` Dan Williams
2021-11-02 17:57             ` Ben Widawsky
2021-11-02 18:10               ` Dan Williams
2021-11-02 18:27                 ` Ben Widawsky
2021-11-02 18:49                   ` Dan Williams
2021-11-02 21:15                     ` Ben Widawsky
2021-11-02 21:34                       ` Dan Williams
2021-11-02 21:47                         ` Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 10/28] cxl/core: Store global list of root ports Ben Widawsky
2021-10-31 18:32   ` Dan Williams
2021-11-01 18:43     ` Ben Widawsky
2021-11-02  2:04       ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 11/28] cxl/acpi: Rescan bus at probe completion Ben Widawsky
2021-10-31 19:25   ` Dan Williams
2021-11-01 18:56     ` Ben Widawsky
2021-11-01 21:45       ` Ben Widawsky
2021-11-02  1:56         ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 12/28] cxl/core: Store component register base for memdevs Ben Widawsky
2021-10-31 20:13   ` Dan Williams
2021-11-01 21:50     ` Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 13/28] cxl: Flesh out register names Ben Widawsky
2021-10-31 20:18   ` Dan Williams
2021-11-01 22:00     ` Ben Widawsky
2021-11-02  1:53       ` Dan Williams
2021-11-03 15:53   ` Jonathan Cameron
2021-11-03 16:03     ` Ben Widawsky
2021-11-03 16:42       ` Jonathan Cameron
2021-11-03 17:05         ` Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 14/28] cxl: Hide devm host for ports Ben Widawsky
2021-10-31 21:14   ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 15/28] cxl/core: Introduce API to scan switch ports Ben Widawsky
2021-11-01  5:39   ` Dan Williams
2021-11-01 22:56     ` Ben Widawsky
2021-11-02  1:45       ` Dan Williams
2021-11-02 16:39         ` Ben Widawsky
2021-11-02 20:00           ` Dan Williams
2021-11-16 16:50         ` Ben Widawsky
2021-11-16 17:51           ` Dan Williams
2021-11-16 18:02             ` Ben Widawsky
2021-11-03 16:08   ` Jonathan Cameron
2021-11-10 17:49     ` Ben Widawsky
2021-11-10 18:10       ` Jonathan Cameron
2021-11-10 21:03         ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 16/28] cxl: Introduce cxl_mem driver Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 17/28] cxl: Disable switch hierarchies for now Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 18/28] cxl/region: Add region creation ABI Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 19/28] cxl/region: Introduce concept of region configuration Ben Widawsky
2021-12-15 17:47   ` Jonathan Cameron
2021-10-22 18:37 ` [RFC PATCH v2 20/28] cxl/region: Introduce a cxl_region driver Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 21/28] cxl/acpi: Handle address space allocation Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 22/28] cxl/region: Address " Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 23/28] cxl/region: Implement XHB verification Ben Widawsky
2022-01-06 16:55   ` Jonathan Cameron
2022-01-06 16:58     ` Ben Widawsky
2022-01-06 17:33       ` Jonathan Cameron
2022-01-06 18:10         ` Jonathan Cameron
2022-01-06 18:34           ` Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 24/28] cxl/region: HB port config verification Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 25/28] cxl/region: Record host bridge target list Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 26/28] cxl/mem: Store the endpoint's uport Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 27/28] cxl/region: Gather HDM decoder resources Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 28/28] cxl: Program decoders for regions Ben Widawsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20211022183709.1199701-1-ben.widawsky@intel.com \
    --to=ben.widawsky@intel.com \
    --cc=Jonathan.Cameron@Huawei.com \
    --cc=alison.schofield@intel.com \
    --cc=chet.r.douglas@intel.com \
    --cc=dan.j.williams@intel.com \
    --cc=ira.weiny@intel.com \
    --cc=linux-cxl@vger.kernel.org \
    --cc=vishal.l.verma@intel.com \
    --subject='Re: [RFC PATCH v2 00/28] CXL Region Creation / HDM decoder programming' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).