Home/Guides/NVMe vs. NVMe-oF: Understanding the Differences in Storage Protocols

NVMe vs. NVMe-oF: Understanding the Differences in Storage Protocols

By |5.9 min read|
crashed hard drive

Legacy storage systems used magnetic tapes and HDDs, with serial protocols controlling data access and interchange. SSDs revolutionized storage with fast, non-volatile memory (NVM) for enterprise-grade I/O operations. Serial transfer techniques were once considered inappropriate for large bandwidths and sub-second latency. Modern storage protocols allow parallel IOPS, lower latency, and faster command queueing.

This article compares NVMe and NVMe-oF (NVMe over Fabric), two current storage access protocols enabling distributed, cloud-native storage via dynamic disaggregation and abstraction.

What Exactly Is NVMe?

NVMe is a storage standard using multicore CPUs to execute concurrent I/O operations, improving data access. It removes CPU bottlenecks and increases throughput. The protocol employs a host controller interface to connect flash-based devices to the CPU via PCIe. NVMe applies to all Non-Volatile Memory (NVM) devices communicating with the CPU over PCIe.

The NVMe host interface has a submission-and-completion command queue with 13 commands, including ten system administration and three NVM input-output commands. It also splits up single jobs and conducts smaller I/O activities in parallel.

Features of NVMe Storage

NVMe is designed to provide efficient data transfer between servers and storage systems. This is accomplished using a variety of characteristics that the NVMe architecture inherently provides. Some of these features are:

Multi-stream Write Support

SSDs have a limited lifespan due to the finite amount of writes during program/erase cycles. NVMe, which supports multi-stream write, decreases write amplification by storing comparable data in neighboring spots, reducing system writes while increasing I/O writes, and extending SSD lifetime.

Zoned Namespaces for Aligned Data Placement

NVMe separates storage into zoned namespaces, logically separating data and applications. Each zone writes sequentially but can be read randomly, reducing write amplification and storage overprovisioning, improving speed, protecting sensitive data, and reducing costs.

IO Multipath Support

The NVMe standard permits the same NVM I/O request or answer to be transmitted across several pathways. When one path is congested or inaccessible, data can be retrieved through a duplicated path. This is useful in modern networks because it supports high availability, load balancing, and data redundancy.

Supports I/O Virtualization Techniques

NVMe separates physical storage from upper-layer protocols, allowing hyper-converged, hyper-scale server-network access for enterprise-grade SANs. It also supports SR-IOV, which allows numerous server VMs to share a single PCIe interface.

Supports Multiple Queues

While earlier standards (such as SATA) had just one command queue with up to 32 commands, NVMe is intended to accommodate up to 65,535 queues with a queue depth of 65,536 instructions. This enables the host controller to run a greater number of commands concurrently.

Benefits of NVMe

Though lower latency and increased IOPS are two of the most common benefits, the NVMe storage model also has the following advantages:

Faster Throughput

NVMe allows multiple queues with huge queue depths, reducing I/O latencies while increasing the number of requests for concurrent processing.

High I/O Consistency

NVMe provides optimized performance across workloads without partitioning them, leveraging the host software’s development of a single virtual environment for I/O operations.

Better Compatibility

The NVMe host interface interfaces directly with the CPU and is compatible with all available SSD form factors, allowing easy connectivity to storage controllers.

Offers Higher Bandwidth

Fourth-generation NVMs allow for faster transfer rates, with bandwidth and throughput of up to 7.9 GB/s and 3.9 GB/s, respectively.

Supports Tunneling for Data Privacy

Organizations can extend NVMe on a TCP over a Time Sensitive Network (TSN) to create a PCIe tunnel that secures communication between NVMe SSDs and hosts. Trusted security groups often offer tunneling protocols to help administer robust security controls such as purge-level erasure, easy access limits, and data-at-rest protection.

Compatible with All Major OSs Regardless of Form Factor

The NVMe protocol enables SSDs to handle multi-vendor form factors with a single device driver and standardized host interface. This eliminates the requirement for custom driver support, allowing SSDs to function with various hypervisors and operating systems.

What Is NVMe-oF (Non-Volatile Memory Express-over Fabrics)?

NVMe-oF is a storage protocol that improves NVMe performance on Fibernet and Ethernet channels. It exchanges data across media channels using NVMe commands, allowing faster communication between servers and storage while decreasing CPU utilization on host servers and improving overall performance. There are several types of nvme-of transport channels:

NVMe over TCP/IP

NVMe is a storage technology that enhances data access by executing concurrent I/O operations with multicore CPUs, eliminating CPU bottlenecks and increasing throughput. It connects flash-based devices via PCIe and applies to all Non-Volatile Memory (NVM) devices using a submission-and-completion command queue system.

NVMe over Fiber Channel (FC-NVMe)

FC-NVMe is a robust protocol that wraps I/O commands in Fiber Channel (FC) frames to facilitate data transport between servers and storage arrays. Although efficient and similar to the conventional FC protocol, its performance when parsing FC frames into NVMe commands could be better.

NVMe-oF over RDMA

NVMe-oF over RDMA facilitates message transport between servers and storage arrays via Remote Direct Memory Access (RDMA). RDMA allows direct data interchange between two hosts’ main memory without utilizing each machine’s operating system, processor, or cache resources.

Features of NVMe-oF-Based Storage

NVMe-oF is transport agnostic, leveraging NVMe characteristics across various network protocols to improve network storage performance. Some of these include:

Credit-based Flow Control

Before transferring the message, the sender waits for credits from the recipient, signaling that buffers are available. This offers a self-throttling connection that ensures delivery at the physical layer without lowering frames due to congestion.

Supports Multiple Transport Protocols

NVMe-oF enables organizations to create Storage Area Networks using various transport protocols, including RDMA over convergent Ethernet (RoCE), Fiber Channel, and TCP.

Low Network Latency

NVMe-oF makes use of low-latency fabrics to improve performance. The optimal maximum latency for fabrics is 10 microseconds from end to end.

Scales up to Thousands of Devices

The fabric can scale to tens of thousands of devices, if not more, depending on available resources and workload requirements.

Benefits of NVMe-oF

As an extension to the NVMe storage protocol, NVME-oF offers several benefits over and above those of NVMe, such as:

Reduces the Size of Server-Side OS Storage Stacks

The increased I/O capabilities of NVMe-oF enable several machines to share a storage array without each machine needing its own storage interface.

Enables Different Implementation Types for Different Use Cases

Storage administrators can design customized NVMe-oF networks to address individual resource requirements for quicker throughputs or greater workloads.

Supports Additional Parallel Requests

Larger network fabrics use the inherent parallelism of NVMe I/O queues and can be leveraged further to split down requests for faster data access and exchange.

Increased Overall Storage Performance

The fabric allows for concurrent data interchange between several hosts and storage arrays, accelerating I/O request processing for peak performance.

Conclusion

Flash-based memory devices enable high-speed data exchange between the CPU and storage. The widespread deployment of SSDs encouraged the development of storage protocols that matched their low latency and high throughput capabilities. Both protocols achieve this when comparing NVMe and NVMe-oF by directly linking the SSD storage to the CPU via the PCIe interface. These protocols provide various benefits over legacy protocols, including greater compatibility, bandwidth, and improved security management.