nvme/cdq: decouple the set-feature head update from the CDQ read

Mostly AI-Generated. Needs further review !!!!

Previously nvme_cdq_traverse() sent the set-feature that advances the
controller-side head synchronously, on every read, via
nvme_submit_sync_cmd(). This tied the read latency to the admin
round-trip: a slow set-feature stalled the data path even though the
host had already consumed the entries.

Split the head into curr_host_entry (advanced by the read path) and
curr_cntl_entry (the head the controller has acknowledged), and send the
set-feature asynchronously with blk_execute_rq_nowait(). The read path
now only advances curr_host_entry and kicks a send; the admin command
completes in nvme_cdq_feat_end_io() without blocking the reader.

At most one set-feature is in flight per CDQ. While one is outstanding,
further reads only bump curr_host_entry; on completion the callback
re-arms a single send if the host moved further, so sends coalesce to
the newest head instead of queueing one per read. A small feat_lock
serializes the in-flight/re-arm/teardown decision while the head values
stay lockless (single writer each). nvme_cdq_delete() marks the CDQ
dying and waits out any in-flight command so its completion cannot
dereference a freed cdq.

The set-feature send (including the tail-pointer-trigger arm in
nvme_cdq_set_tpt()) no longer returns the controller status to the
caller, which is an intended consequence of making the send asynchronous.

Signed-off-by: Joel Granados <joel.granados@kernel.org>

nvme: Add CDQ sysfs attribute

Add a sysfs attribute to expose the number of active CDQs for an
NVMe controller.

The num_cdqs attribute is read-only and displays the count of
currently allocated Controller Data Queues.

Signed-off-by: Joel Granados <joel.granados@kernel.org>

nvme: Add CDQ sysfs attribute

Add a sysfs attribute to expose the number of active CDQs for an
NVMe controller.

The num_cdqs attribute is read-only and displays the count of
currently allocated Controller Data Queues.

Signed-off-by: Joel Granados <joel.granados@kernel.org>

nvme: hold the mm read lock when pinning.

Signed-off-by: Joel Granados <joel.granados@kernel.org>

nvme/cdq: implement PRP list handling

Replace kernel memory allocation with user space memory mapping for CDQ
entries. Add comprehensive PRP (Physical Region Page) list setup to
handle scatter-gather memory layouts with proper DMA mapping. Remove
file descriptor based approach in favor of direct user memory access.

Key changes:
- Add nvme_cdq_alloc_from_usr() to pin and map user pages
- Implement nvme_cdq_setup_prps() for multi-page PRP list creation
- Remove anon_inode file descriptor mechanism
- Update UAPI to accept user virtual address instead of returning fd
- Add proper cleanup for PRP lists after command submission

This enables direct user space access to CDQ memory while maintaining
proper DMA coherency and memory management."

Signed-off-by: Joel Granados <joel.granados@kernel.org>

nvme/cdq: Remove all code to traverse CDQ

With the CDQ memory being mmaped into user space. The responsibility of
making sense of that memory is now on user space. The kernel remains
responsible of signaling an eventfd when it detects that a ONE_SHOT CDQ
AEN.

Interface is simplified as there is no need to define the number of
entries; we can get away with just the total CDQ size in number of
bytes. There is only need for the MOS and the CDS for CDQ creation, the
rest is removed. We are left with six values at CDQ creation: size of
the CDQ in bytes, eventfd for the TPT, the returned cdq identifier and
the internal kernel file descriptor, CQS and MOS.

Signed-off-by: Joel Granados <joel.granados@kernel.org>

nvme: Add AEN handler for CDQ tail pointer events

Add a new "one shot" ASYNC type event that will be called in
nvme_complete_async_event. Use the new "one shot" path to handle
Controller Data Queue Tail Pointer events that arrive through AENs.

For now we just print to the kernel log that an event has been received,
The logic of what to do with the CDQ AEN should be added to
nvme_cdq_aen_tpevent.

Signed-off-by: Joel Granados <joel.granados@kernel.org>

nvme: use cdqp_mask instead of hardcoded 0x1 in phase bit toggle

Replace hardcoded 0x1 with cdqp_mask for phase bit toggling in
nvme_cdq_next() to support variable phase bit widths.

Signed-off-by: Joel Granados <joel.granados@kernel.org>

nvme/cdq: Add a free_cdqs to controller init

Signed-off-by: Joel Granados <joel.granados@kernel.org>

Linux 6.15