Last month, the SNIA Networking Storage Forum (NSF) hosted a live webcast “NVMe-oF: Looking Beyond Performance Hero Numbers.”It was extremely popular, in fact it has been viewed almost 800 times in just two weeks! If you missed it, it’s available on-demand, along with the presentation slides at the SNIA Educational Library. Our audience asked several great questions during the live event and our expert presenters, Erik Smith, Rob Davis and Nishant Lodha have kindly answered them all here.
Q. There are initiators for Linux but not for Windows? What are my options to connect NVMe-oF to Windows Server?
A. Correct. For many of the benchmarks, a standard Linux based initiator was used as it provided a consistent platform on which several of the compared NVMe Fabrics are supported/available.
Regarding what Fabrics are available on Microsoft Windows, it is best to check with your Microsoft representative for the most current information. As far as we are aware, Microsoft does not natively support NVMe-oF, but there are 3rd party Windows drivers available from other vendors, just search the web for “Windows NVMe-oF Initiator”.
In addition, some SmartNICs/DPU/Programmable NICs have the ability to terminate NVMe-oF and present remote NVMe devices as local, such options can be considered for your Windows deployment.
Q. Where do storage services like RAID, deduplication, and such come from in the Compute Storage Disaggregation model?
A. Storage services can either on the compute node in software (leveraging server CPU cycles), offloaded to a DPU (Data Processing Unit or SmartNIC) that is plugged into the compute node, or on the storage array node, or a combination depending on the implementation and use case.
Q. The “emerging heterogenous landscape” slide omits the fact that industry momentum, in both hyperscale and hyperconverged storage variants, has been and continues to be towards proprietary networked block storage protocols. Such providers have shunned both legacy and emerging (e.g. NVMe-oF) network block storage industry standards. Simple block storage does not suffice for them. Isn’t the shown “emerging landscape” a declining niche?
A. Definitely this was the direction of the “hyperscale and hyperconverged storage variants,” but with the maturity and industry wide support of NVMe, and NVMe-oF this is changing. The “hyperscale and hyperconverged storage variants” providers went the “proprietary networked block storage protocols” direction initially for their deployments because no standardized secure SAN protocols that met their cost, performance, and converged network requirements existed. With NVMe-oF on Ethernet at 100GbE or higher speeds these requirements are now met. It is also important to note that many of the Hyperscale datacenter companies were instrumental in driving the development of NVMe-oF standards, including NVMe/TCP and RDMA.
Q. At hardware components does the TCP-offloading take place? What TCP offload device was used to generate these numbers?
A. TCP Offload for NVMe/TCP can occur either in a standard NIC (if it has TCP offload capabilities) or a SmartNIC/DPU/Programmable NIC (via a dedicated hardware block or compute engines on the SmartNIC). The performance charts depicted in the slides used a variety of standard NICs and InfiniBand devices, including those from NVIDIA and Marvell. To the question about TCP offload, those tests were conducted with a Marvell 25GbE NIC. We recommend checking with your NIC, SmartNIC, or DPU vendor about the specific NVMe-oF offloads they can provide.
Q. Which Web and File applications use 4k random writes? Even for DBs 4k that is very uncommon.
A. We use a common block size (i.e., 4k) across all fabric types for the sake of creating a fair and realistic comparison. Small block I/Os (e.g., 512B) are great to use when you are trying to create hero numbers highlighting the maximum number of IOPS supported by an adapter or a storage system. Large block I/Os (e.g., 128k and above) are great to use when you are trying to create hero numbers that highlight the maximum throughput supported by an adapter or a storage system. We used 4k random writes for this testing because we’ve found (in general) the results obtained represent what end users can reasonably expect to observe at the application. We recommend looking at All Flash Array vendor published IO histograms to see the typical block size and IO operation seen most commonly on fabrics (See Figure 4 here.)
Q. Where could I get an overview around shipped ports FC vs. FCoE over time? Curious how impactful FCoE is after all.
A. For specific metrics, any of the market research firms like Crehan, IDC or Dell’Oro are a good place to start looking. In general, considering the significantly longer time Fibre Channel technologies have been in the market, one would expect the Fibre Channel ports to far outnumber the FCoE ports in datacenters.
Q. You said there would be a follow-up webcast on this topic. When is that happening?
A. We’re glad you asked! Our next webcast in this series is “Security of Data on NVMe over Fabrics, The Armored Truck Way” on May 12, 2021. You can register for it here. If you’re reading this blog after May 12th, you can view it on-demand.