NVMe-oF Architecture: RDMA (RoCEv2) vs NVMe/TCP for Enterprise Storage Fabrics

Written by: TrendSpike Engineering Desk  |  Published: October 10, 2026  |  Topic: High-Performance Storage & Enterprise Networking

Disaggregated storage separates compute resources from shared storage systems, allowing multiple servers to access centralized flash storage over a network. Traditional network storage commonly uses protocols such as iSCSI and Fibre Channel. NVM Express over Fabrics (NVMe-oF) extends the NVMe command interface beyond a locally attached PCIe device, allowing hosts to access remote NVMe storage through supported network transports.

Two important Ethernet transport options are NVMe over RDMA, including RDMA over Converged Ethernet version 2 (RoCEv2), and NVMe over TCP (NVMe/TCP). Both transport NVMe commands, but they differ in protocol processing, network requirements, hardware support, operational complexity, and potential performance. The better choice depends on the workload and the complete host, network, and storage implementation.

1. The Architectural Shift to NVMe-oF

A locally attached NVMe SSD communicates with its host through the PCIe transport. With NVMe-oF, the host communicates with a remote NVMe subsystem through a supported fabric transport. This makes it possible to share or disaggregate storage while retaining the NVMe command model.

Unlike iSCSI, which carries SCSI commands over TCP/IP, NVMe/TCP carries NVMe commands over TCP/IP. NVMe/RDMA maps NVMe-oF communication onto an RDMA transport. These approaches avoid translating NVMe commands into SCSI commands, but they still incur transport, network, host-processing, and storage-controller overhead.

NVMe-oF supports queue-based command processing designed for parallel I/O. The actual queue counts, queue depths, throughput, and latency available to a host depend on the applicable specifications, controller capabilities, driver implementation, and system configuration. Theoretical protocol limits should not be confused with the performance available in a production deployment.

2. NVMe-oF over RDMA (RoCEv2)

Remote Direct Memory Access (RDMA) allows supported network adapters to transfer data between registered memory regions with reduced host CPU involvement compared with conventional networking paths. NVMe/RDMA can therefore reduce transport-processing overhead and achieve low latency when the host adapters, storage targets, drivers, and network are configured appropriately.

• Reduced data-path overhead: RDMA-capable adapters can move data between registered memory regions without requiring the CPU to copy every payload through conventional networking buffers. The exact data path depends on the implementation, memory registration, and hardware capabilities.
• Lower CPU overhead: NVMe/RDMA can reduce host CPU consumption because supported network adapters offload parts of the transport processing. The benefit varies with hardware, workload, driver, and target implementation.
• RoCEv2 networking: RoCEv2 carries RDMA traffic over UDP/IP, enabling RDMA communication across routed IP networks where the network and implementation support it.
• Congestion management: Ethernet fabric design is important for reliable RoCEv2 performance. Priority-based Flow Control (PFC) and Explicit Congestion Notification (ECN) are techniques that may be used to manage congestion. Whether PFC is required, recommended, or avoided depends on the adapter vendor's guidance, the network design, and the supported congestion-control strategy.
• Operational planning: Teams should validate congestion behavior, switch configuration, routing, adapter compatibility, monitoring, and failure handling before deploying RoCEv2 at scale. Poorly designed flow-control configurations can create operational problems.

RoCEv2 is often attractive when predictable low latency and reduced CPU overhead are important and the organization can support the associated networking requirements. RDMA does not eliminate every software operation, and its performance advantage should be measured on the intended hardware and workload.

3. NVMe-oF over TCP (NVMe/TCP)

NVMe/TCP transports NVMe commands and data over standard TCP/IP networking. It is designed to make NVMe-oF available without requiring an RDMA-capable adapter or a network specifically configured for RDMA traffic.

• Standard Ethernet compatibility: NVMe/TCP can operate over supported conventional Ethernet and IP networks. It generally does not require the specialized RDMA configuration associated with RoCE deployments.
• TCP processing: TCP handles reliable byte-stream delivery, retransmission, and congestion control. This introduces processing work that differs from an RDMA data path, but the actual latency and CPU cost depend on the system and workload.
• Hardware acceleration: Some network adapters, SmartNICs, and storage platforms support TCP or storage-related offloads. These capabilities can reduce host processing in particular implementations; they do not guarantee that NVMe/TCP will match RDMA performance.
• Deployment flexibility: Because it uses standard TCP/IP networking, NVMe/TCP can simplify integration with existing data-center infrastructure and operational practices.
• Performance validation: NVMe/TCP performance depends on CPU availability, network congestion, packet processing, storage-controller behavior, queue configuration, and the I/O workload. It should not be assumed to be faster than iSCSI in every scenario.

NVMe/TCP is often a practical option for organizations that prioritize infrastructure compatibility, easier deployment, and broad hardware support. Its performance should be evaluated against the requirements of the specific application rather than against a universal latency estimate.

4. Operational Trade-Offs for Production Workloads

Choosing between NVMe/RDMA and NVMe/TCP involves more than comparing nominal network latency. Storage performance depends on the complete I/O path, including the application, host CPU, driver, network adapter, switches, storage target, and SSDs.

• Latency-sensitive applications: Applications with strict I/O latency requirements may benefit from NVMe/RDMA when a correctly configured RDMA environment delivers measurable improvements. Validate tail latency as well as average latency under realistic load.
• High-throughput AI and analytics: Large data pipelines and GPU clusters may benefit from high-throughput storage fabrics and reduced host processing. The best transport depends on aggregate bandwidth, concurrent I/O, CPU availability, and the storage system's ability to feed the workload.
• Cloud and virtualized infrastructure: NVMe/TCP can be attractive when teams want to use existing TCP/IP networks and standard operational tooling. Its suitability still depends on performance targets, network topology, isolation, and failure-recovery requirements.
• Operationally constrained environments: A team without experience managing RDMA networking may find NVMe/TCP simpler to deploy and troubleshoot. Teams already operating a validated RoCE or InfiniBand fabric may prefer RDMA for workloads that benefit from it.
• Cost and resource utilization: Compare adapter and switch costs, CPU utilization, licensing, support, monitoring, redundancy, and engineering effort. A lower-latency transport is not automatically the most cost-effective choice for every application.

5. Benchmarking and Validation Checklist

Before selecting a transport for production, test the complete configuration with a workload representative of the intended application. Results from a different controller, adapter, switch, firmware version, or queue configuration may not transfer to your environment.

• Record the host CPU, operating system, kernel and driver versions, network adapter, firmware, switch configuration, and storage target.
• Measure average and tail latency under both light load and sustained production-like load.
• Measure throughput, IOPS, CPU utilization, and performance as concurrent clients increase.
• Test realistic block sizes, read/write ratios, queue depths, and application access patterns.
• Evaluate congestion, packet loss, path failures, reconnect behavior, and recovery under load.
• For RoCEv2, validate the supported congestion-control and flow-control configuration with the adapter and switch vendors.
• For NVMe/TCP, check CPU headroom, TCP behavior, network utilization, and any supported adapter offloads.
• Repeat tests and document the exact configuration so results can be reproduced.

6. Practical Decision Framework

NVMe/RDMA and NVMe/TCP are different ways to transport NVMe commands and data across a network. RDMA may offer lower latency and lower CPU overhead, while TCP can provide broad compatibility with conventional IP networking. Neither option guarantees a particular performance level independent of implementation.

Choose NVMe/RDMA when measured workload benefits justify its hardware and network requirements. Choose NVMe/TCP when its operational simplicity and compatibility meet the application's performance objectives. For either option, production decisions should be based on repeatable benchmarks, failure testing, and the operational capabilities of the team responsible for the system.

Sources & Official References

• NVM Express Organization — NVMe Specifications, including NVMe over RDMA and NVMe over TCP
• NVM Express — NVMe/TCP: Specification Questions and Technical Guidance
• Linux Kernel Documentation — NVMe Subsystem
• NVIDIA Networking Documentation — NVMe over Fabrics and Target Offload
• IETF RFC 3168 — The Addition of Explicit Congestion Notification (ECN) to IP

Disclaimer: This article is for educational and architectural reference only. It is not a substitute for vendor-specific deployment guidance or testing. Storage and network performance vary with hardware, software, configuration, and workload. Validate the intended design in a representative environment before deploying it in production.