Q&A on All Things iSCSI

In the recent SNIA Ethernet Storage Forum iSCSI pod webcast, from our “Everything You Wanted To Know About Storage Part Were Too Proud to Ask” series, we discussed all things iSCSI. If you missed the live event, it’s now available on-demand. As promised, we’ve compiled all the webcast questions with answers from our panel of experts. If you have additional questions, please feel free to ask them in the comment field of this blog. I also encourage you to check out the other on-demand webcasts in this “Too Proud To Ask” series here and stay informed on upcoming events in this series by following us on Twitter @SNIAESF.

Q. What does SPDK stand for?

A. SPDK stands for Storage Performance Development Kit. It is comprised of tools and libraries for developers to write high performance and scalable storage applications in user-mode. For details, see www.spdk.io.

Q. Can you elaborate on SPDK use? A quick search seems to indicate it is a “half-baked” solution, and available only on Linux systems.

A. SPDK isn’t a solution, per se – it’s a development kit, intended to provide common building blocks (NVMe drivers, NVMe over Fabrics targets & host/initiator, etc.) for solutions developers who care about latency, license (BSD) and efficiency.

Q. Is iSCSI ever going to be able to work with object storage?

A. iSCSI is a block storage protocol while object storage is normally accessed using a RESTful API such as Amazon’s S3 API or the Swift API. For this reason, iSCSI is unlikely to be used for direct access to object storage. However, an object storage system controller could use iSCSI—or other block protocols–to access remote storage enclosures or for data replication. There also could be storage systems that support both iSCSI/block and object storage access simultaneously.

Q. Does a high-density virtualized workload represent something better served with a full or partial offload solution?

A. The type of workload that is better served with full or partial offload will really depend more on what that workload is doing. If you are processing a lot of very large data segments, LSO or LRO might be very helpful. If you have a lot of smaller data sets, you might be able to benefit from checksum or chimney offload. Unfortunately, the best way to see is to test things out (but not on production, obviously).

Q. How does one determine if TOE NIC cards are worth the cost?

A. This is a really tough question to answer without context. The best way to look at it is do some digging into what your CPU and memory utilization and IO patters look like on your servers and try to map that to TCP connections. If you have a lot of iSCSI IO and a large amount of TCP connections on a server, that might be a candidate for TOE. That’s just a technical response, but then comes the really tricky part – the QUANTITY measurement of how many dollars it is worth… that’s way more challenging. For example, if I have a regular 10G NIC that costs $200 and a TOE card that costs 3x that and only saves 5% CPU, then it may not have enough value. On the other hand, if that 5% CPU can be used by your application to transact enough business to pay for the extra $400, then it’s worth it. Sorry to say that I have seen no scientific way to enumerate that value outside of specific hands-on testing of the solution with and without TOE NICs.

Q. What is the difference between a stateless and stateful TCP offload? Are RSS and TSS (receive-side and transmission-side scaling) offloads a type of TCP offload or are they operating at a lower level like Layer 2?

A. Stateless offloading is basically any offload function that can be done without the NIC needing to maintain a connection state table. Checksum offloads are an example. Stateful offloading is any offloading that requires the NIC to maintain a full state connection table. Receive Side Scaling has to do with distributing inbound connections in order to alternate connections coming into the server to different CPUs on a multi-CPU server. There are also some other performance-enhancements that can be done such as RPS, RFS, XPS and some others. These are more about how to get data from the network to the CPU, but are not really specifically TCP functions, as they have to do with uniform processing, not necessarily to do with the TCP stack.

Q. Is using the host CPU to run iSCSI really a downside?

A. There may be applications where this is a problem, but you’re generally right; it’s not too much of an issue today. But there are iSCSI-based storage solutions coming up where a consistent 100s of nanoseconds to low microseconds of latency from the device is possible – and that’s very fast indeed. So an iSCSI stack in these circumstances needs to ensure that its consumption of CPU doesn’t increase the latency (even very efficient stacks can add 100s of micro- to milliseconds of latency), or cause contention for the CPU (busy CPUs mean you may queue for compute resources).

Q. Is the term “onload” for iSCSI new – never heard this before?

A. It was intended as a quick shorthand word to stand in contrast to iSCSI offload. It will probably not catch on!

Update: If you missed the live event, it’s now available  on-demand. You can also  download the webcast slides.

 

 

 

NFS FAQ – Test Your Knowledge

How would you rate your NFS knowledge? That’s the question Alex McDonald and I asked our audience at our recent live Webcast, “What is NFS?.” From those who considered themselves to be an NFS expert to those who thought NFS was a bit of a mystery, we got some great questions. As promised, here are answers to all of them. If you think of additional questions, please comment in this blog and we’ll get back to you as soon as we can.

Q. I hope you touch on dNFS in your presentation

A. Oracle Direct NFS (dNFS) is a client built into Oracle’s database system that Oracle claims provides faster and more scalable access to NFS servers. As it’s proprietary, SNIA doesn’t really have much to say about it; we’re vendor neutral, and it’s not the only proprietary NFS client out there. But you can read more here if you wish at the Oracle site.

Q. Will you be talking about pNFS?

A. We did a series of NFS presentations that covered pNFS a little while ago. You can find them here.

Q. What is the difference between SMB vs. CIFS? And what is SAMBA? Is it a type of SMB protocol?

A. It’s best explained in this tutorial that covers SMB. Samba is the open source implementation of SMB for Linux. Information on Samba can be found here.

Q. Will you touch upon how file permissions are maintained when users come from an SMB or a non-SMB connection? What are best practices?

A. Although NFS and SMB share some common terminology for security (ACLs or Access Control Lists) the implementations are different. The ACL security model in SMB is richer than the NFS file security model. I touched on some of those differences during the Webcast, but my advice is; don’t expect the two security domains of SMB (or Samba, the open source equivalent) and NFS to perfectly overlap. Where possible, try to avoid the requirement, but if you do need the ability to file share, talk to your NFS server supplier. A Google search on “nfs smb mixed mode” will also bring up tips and best practices.

Q. How do you tune and benchmark NFSv4?

A. That’s a topic in its own right! This paper gives an overview and how-to of benchmarking NFS; but it doesn’t explain what you might do to tune the system. It’s too difficult to give generic advice here, except to say that vendors should be relied on to provide their experience. If it’s a commercial solution, they will have lots of experience based on a wide variety of use cases and workloads.

Q. Is using NFS to provide block storage a common use case?

A. No, it’s still fairly unusual. The most common use case is for files in directories. Object and block support are relatively new, and there are more NFS “personalities” being developed, see our ESF Webcast on NFSv4.2 for more information.  

Q. Can you comment about file locking issues over NFS?

A. Locking is needed by NFS to maintain file consistency in the face of multiple readers and writers. Locking in NVSv3 was difficult to manage; if a server failed or clients went AWOL, then the lock manager would be left with potentially thousands of stale locks. They often required manual purging. NFSv4 simplifies that by being a stateful protocol, and by integrating the lock management functions and employing timeouts and state, it can manage client and server recovery much more gracefully. Locks are, in the main, automatically released or refreshed after a failure.

 Q.  Where do things like AFS come into play? Above NFS? Below NFS? Something completely different?

A. AFS is another distributed file system, but it is not POSIX compliant. It influenced but is not directly related to NFS. Its use is relatively small; SMB and NFS dominate. Wikipedia has a good overview.

Q. As you said NFSv4 can hide some of the directories when exporting to clients. Can this operation hide different folders for different clients?

A. Yes. It’s possible to maintain completely different exports to expose or hide whatever directories on the server you wish. The pseudo file system is built separately for each server export. So you can have export X with subdirectories A B and C; or export Y with subdirectories B and C only.

Q. Similar to DFS-N and DFS-R in combination, if a user moves to a different location, does NFS have a similar methodology?

A. I’m not sure what DFS-N and DFS-R do in terms of location transparency. NFS can be set up such that if you can contact a particular server, and if you have the correct permissions, you should be able to see the same exports regardless of where the client is running.

Q. Which daemons should be running on server side and client side for accessing filesystem over NFS?

A. This is NFS server and client specific. You need to look at the documentation that comes with each.

Q. Regarding VMware 6.0. Why use NFS over FC?

A. Good question but you’ll need to speak to VMware to get that question answered. It depends on the application, your infrastructure, your costs, and the workload.

Update: If you missed the live event, it’s now available  on-demand. You can also  download the webcast slides.

Next Live Webcast: NFS 101

Need a primer on NFS? On March 23, 2016, The Ethernet Storage Forum (ESF) will present a live Webcast “What is NFS? An NFS Primer.” The popular and ubiquitous Network File System (NFS) is a standard protocol that allows applications to store and manage data on a remote computer or server. NFS provides two services; a network part that connects users or clients to a remote system or server; and a file-based view of the data. Together these provide a seamless environment that masks the differences between local files and remote files.

At this Webcast, Alex McDonald, SNIA ESF Vice Chair, will provide an introduction and overview presentation to NFS. Geared for technologists and tech managers interested in understanding:

  • NFS history and development
  • The facilities and services NFS provides
  • Why NFS rose in popularity to dominate file based services
  • Why NFS continues to be important in the cloud

As always, the Webcast will be live and Alex and I will be on hand to answer your questions. Register today. Alex and I look forward to hearing from you on March 23rd.

Update: If you missed the live event, it’s now available  on-demand. You can also  download the webcast slides.

How Ethernet RDMA Protocols iWARP and RoCE Support NVMe over Fabrics

NVMe (Non-Volatile Memory Express) over Fabrics is of tremendous interest among storage vendors, flash manufacturers, and cloud and Web 2.0 customers. Because it offers efficient remote and shared access to a new generation of flash and other non-volatile memory storage, it requires fast, low latency networks, and the first version of the specification is expected to take advantage of RDMA (Remote Direct Memory Access) support in the transport protocol.

Many customers and vendors are now familiar with the advantages and concepts of NVMe over Fabrics but are not familiar with the specific protocols that support it. Join us on January 26th for this live Webcast that will explore and compare the Ethernet RDMA protocols and transports that support NVMe over Fabrics and the infrastructure needed to use them. You’ll hear:

  • Why NVMe Over Fabrics requires a low-latency network
  • How the NVMe protocol is mapped to the network transport
  • How RDMA-capable protocols work
  • Comparing available Ethernet RDMA transports: iWARP and RoCE
  • Infrastructure required to support RDMA over Ethernet
  • Congestion management methods

The event is live, so please bring your questions. We look forward to answering them.

Update: If you missed the live event, it’s now available  on-demand. You can also  download the webcast slides.

New White Paper: An Updated Overview of NFSv4

Maybe you’ve asked yourself recently; “Hmm, I wonder what’s new in NFSv4?” Maybe (and more likely) you haven’t; but you should.

During the last few years, NFSv4 has become the version of choice for many users, and there are lots of great reasons for making the transition from NFSv3 to NFSv4. Not the least of which is that it’s a relatively straightforward transition.

But there’s more; NFSv4 offers features unavailable in NFSv3. Parallelization, better security, WAN awareness and many other features make it suitable as a file protocol for the next generation of applications. As a proof point, lately we’ve seen new clients of NFSv4 servers beyond the standard Linux client, including support in VMware’s vSphere for virtual machine datastores accessible via NFSv4.

In this updated white paper, An Updated Overview of NFSv4, we explain how NFSv4 is better suited to a wide range of datacenter and high performance compute (HPC) uses than its predecessor NFSv3, as well as providing resources for migrating from v3 to v4.

You’ll learn:

  • How NFSv4 overcomes statelessness issues associated with NFSv3
  • Advantages and features of NFSv4.1 & NFSv4.2
  • What parallel NFS (pNFS) and layouts do
  • How NFSv4 supports performant WAN access

We believe this document makes the argument that users should, at the very least, be evaluating and deploying NFSv4 for use in new projects; and ideally, should be using it wholesale in their existing environments. The information in this white paper  is meant to be comprehensive and educational and we hope you find it helpful.

If you have questions or comments after reading this white paper, please comment on this blog and we’ll get back to you as soon as possible.

NFS 4.2 Q&A

We received several great questions at our What’s New in NFS 4.2 Webcast. We did not have time to answer them all, so here is a complete Q&A from the live event. If you missed it, it’s now available on demand.

Q. Are there commercial Linux or windows distributions available which have adopted pNFS?

A. Yes. RedHat RHEL6.2, Suse SLES 11.3 and Ubuntu 14.10 all support the pNFS capable client. There aren’t any pNFS servers on Linux so far; but commercial systems such as NetApp (file pNFS), EMC (block pNFS), Panasas (object pNFS) and maybe others support pNFS servers. Microsoft Windows has no client or server support for pNFS.

Q. Are we able to prevent it from going back to NFS v3 if we want to ensure file lock management?

A. An NFSv4 mount (mount -t nfs4) won’t fall back to an nfs3 mount. See man mount for details.

Q. Can pNFS metadata servers forward clients to other metadata servers?

A. No, not currently.

Q. Can pNfs provide a way similar to synchronous writes? So data’s instantly safe in at least 2 locations?

A. No; that kind of replication is a feature of the data servers. It’s not covered by the NFSv4.1 or pNFS specification.

Q. Does hole punching depend on underlying file system in server?

A. If the underlying server supports it, then hole punching will be supported. The client & server do this silently; a user of the mount isn’t aware that it’s happening.

Q. How are Ethernet Trunks formed? By the OS or by the NFS client or NFS Server or other?

A. Currently, they’re not! Although trunking is specified and is optional, there are no servers that support it.

Q. How do you think vVols could impact NFS and VMware’s use of NFS?

A. VMware has committed to supporting NFSv4.1 and there is currently support in vSphere 6. vVols adds another opportunity for clients to inform the server with IO hints; it is an area of active development.

Q. In pNFS, does the callback call to the client must come from the original-called-to metadata server?

A. Yes, the callback originates from the MDS.

Q. Is hole punched in block units?

A. That depends on the server.

Q. Is there any functionality like SMB continuous availability?

A. Since it’s a function of the server, and much of the server’s capabilities are unspecified in NFSv4, the answer is – it depends. It’s a question for the vendor of your server.

Q. NFS has historically not been used in large HPC cluster environments for cluster-wide storage, for performance reasons. Do you see these changes as potentially improving this situation?

A. Yes. There’s much work being done on the performance side, and the cluster parallelism that pNFS brings will have it outperform NFSv3 once clients employ more of its capabilities.

Q. Speaking of the Amazon adoption for NFSv4.0. Do you have insight / guess on why Amazon did not select NFSv4.1, which has a lot more performance / scalability advantages over NFSv4.0?

A. No, none at all.

Next Webcast: What’s New in NFS 4.2

We’re excited to announce our next ESF Webcast on NFSv4.2. With NFSv4.1 implemented on several commercial NFS systems, an established Linux client and a new pNFS Linux server, there is a continued growth of NFS usage in the IT industry. NFSv4.1, first introduced in 2010, meets many needs in the modern datacenter, but there are still technologies and advanced techniques that NFS developers want to deliver.

Join me and J Metz on April 28th at 10:00 a.m. PT as we’ll cover a brief update of where we are with NFSv4.1 and more detail on the proposed features for NFSv4.2 that are currently being ratified at the IETF.  This will be a live, interactive session. Register now and please bring your questions.

If you need a primer on NFS before this event, I encourage you to check our 4-part Webcast mini-series available on demand:

Register today. I hope to see you on April 28th.

 

 

 

 

Cloud File Services: SMB/CIFS and NFS…in the Cloud – Q&A

Cloud File Services: SMB/CIFS and NFS…in the Cloud – Q&A

At our recent live ESF Webcast, “Cloud File Services: SMB/CIFS and NFS…in the Cloud” we talked about evaporating your existing file server into the cloud. Over 300 people have viewed the Webcast. If you missed it, it’s now available on-demand. It was an interactive session with a lot of great questions from attendees. We did not have time to address them all – so here is a complete Q&A from the Webcast. If you think of additional questions, please feel free to comment on this blog.

Q. Can your Storage OS take advantage of born-in-the-cloud File Storage like Zadara Storage at AWS and Azure?

A. The concept presented is generic in nature.   Whichever storage OS the customer chooses to use in the cloud will have its own requirements on the underlying storage beneath it.   Most Storage OS’s used for Cloud File Services will likely use block or object backends rather than a file backend.

Q. Regarding Cloud File Services for “Client file services,” since the traditional file services require the client and server to be in a connected mode, and in the same network. And, they are tied to identities available in the network. How can the SMB/NFS protocols be used to serve data from the cloud to the clients that could be coming from different networks (4G/Corporate)? Isn’t REST the appropriate interface for that model?

A. The answer depends on the use case.   There are numerous examples of SMB over the WAN, for example, so it’s not far fetched to imagine someone using Cloud File Services as an alternative to a “Sync & Share” solution for client file services.   REST (or similar) may be appropriate for some, while file-based protocols will work better for others.   Cloud File Services provides the capability to extend into this environment where it couldn’t before.

Q. Is Manila like VMware VSAN or VASA?

A. Please take a look at the Manila project on OpenStack’s website https://wiki.openstack.org/wiki/Manila

Q. How do you take care of data security while moving data from on-premises to cloud (Storage OS)?

A. The answer depends on the Storage OS you are using for your Cloud File Services platform.   If your Storage OS supports encryption, for example, in its storage-to-storage in-flight data transport, then data security in-flight would be taken care of.   There are many facets to security that need to be thought through, including security at rest, some of which may depend on the environment (private/on-premises, service provider, hyperscalar) the Storage OS is sitting in.

Q. How do you get the data out of the cloud?   I think that’s been a traditional concern with cloud storage.

A. That’s the beauty of Cloud File Services!   With data movement and migration provided at the storage-level by the same Storage OS across all locations, you can simply move the data between on-premises and off-premises and expect similar behavior on both ends.   If you choose to put data into a native environment specific to a hyperscalar or service provider, you run the risk of lock-in.

Q. 1. How does one address issue of “chatty” applications over the cloud?   2. File services have “poor” performance for small files. How does one address that issue? Block & Objects do address that issue 3. Why not expose SMB, NFS, Object Interface on the Compute note?

A. 1. We should take this opportunity to make the applications less chatty! 🙂   One possible solution here is to operate the application and Storage OS in the same environment, in much the same way you would have on-premises.   If you choose a hyperscalar or service provider, for chatty use cases, it may be best to keep the application and storage pieces “closer” together.

2. Newer file protocols are getting much better at this.   SMB 3.02 for instance, was optimized for 8K transactions.   With a modern Storage OS, you will be able to take advantage of new developments.

3. That is precisely the idea: the Storage OS operating in the “compute nodes,” serving out their interfaces, while taking advantage of different backend offerings for cost and scalability.

Q. Most storage arrays NetApp, EMC etc., can provide 5 9s of resilience, Cloud VMs typically offer 3 9’s.   How do you get to 5 9’s with CFS?

A. Cloud File Services (CFS) as a platform can span across all of your environments, and as such, the availability guarantees will depend upon each environment in which CFS is operating.

Q. Why are we “adding” another layer? Why can we just use powerful “NAS” devices that can have different media like NVMe, Flash SSD or HDDs?

A. Traditional applications may not want to change, but this architecture should suit those well.   It’s worth examining that “cloud-ready” model.   Is the goal to be “cloud-ready,” or is the goal to support the scaling, failover, and on-demand-ness that the cloud has the ability to provide?   Shared nothing is a popular way of accomplishing some of this, but it may not be the only way.

The existing interfaces provided by hyperscalars do provide abstraction, but if you are building an application, you run a strong risk of lock-in on any particular abstraction.   What is your exit strategy then?   How do you move your data (and applications) out?

By leveraging a common Storage OS across your entire infrastructure (on-premises, service providers, and hyperscalars), you have a very simple exit strategy, and your exit and mobility strategy become very similar, if not the same, with the ability to scale or move across any environment you choose.

Q. How do you virtualize storage OS? What happens to native storage OS hardware/storage?

A. A Storage OS can be virtualized similar to a PC or traditional server OS.   Some pieces may have to be switched or removed, but it is still an operating system.

Q. Why is your Storage displayed as part of your Compute layer?

A. In the hyperscalar model, the Storage OS is sitting in the compute layer because it is, in effect, running as a virtual machine the same as any other.   It can then take advantage of different tiers of storage offered to it.

Q. My concern is that it would be slower as a VM than a storage controller.   There’s really no guarantee of storage performance in the cloud in fact most hyperscalers won’t give me a good SLA without boatloads of money.   How might you respond to this?

A. Of course with on-premises infrastructure, a company or service provider will have more of a guarantee in the sense that they control the hardware behind it.   However, as we’ve seen, SLA’s continue to improve over time, and costs continue to come down for the Public Cloud.

Q. Does FreeNAS qualify as a Storage OS?

A. I recommend checking with their team.

Q. Isn’t this similar to Hybrid cloud?

A. Cloud File Services (CFS) is one way of looking at Hybrid Cloud.   Savvy readers and listeners will pick up that having the same Storage OS everywhere doesn’t necessarily limit you to only File Services. iSCSI or RESTful interfaces could work exactly the same.

Q. What do you mean by Storage OS? Can you give some examples?

A. As I work for NetApp, one example is Data ONTAP.   EMC has several as well, such as one for the VNX platform.   Most major storage vendors will have their own OS.

Q. I think one of the key questions is the data access latency over WAN, how I can move my data to the cloud, how I can move back when needed – for example, when the service is terminated?

A. Latency is a common concern, and connectivity is always important.   Moving your data into and out of the cloud is the beauty of the Cloud File Services platform, as I mentioned in other answers.   If one of your environments goes down (for example, your on-premises datacenter) then you would feasibly be able to shift your workloads over to one of your other environments, similar to a DR situation.   That is one example of where storage replication and application awareness across sites is important.

Q. Running applications like Oracle, Exchange through SMB/NFS (NAS), don’t you think it will be slow compared to FC (block storage)?

A. Oracle has had great success running over NFS, and it is extremely popular.   While Exchange doesn’t currently support running directly over SMB at this time, it’s not ludicrous to think that it may happen at some point in the future, in the same way that SQL has.

Q. What about REST and S3 API or are they just for object storage?   What about CINDER?

A. The focus of this presentation was only File Services, but as I mentioned in another answer, if your Storage OS supports these services (like REST or S3), it’s feasible to imagine that you could span them in the same way that we discussed CFS.

Q. Why SAN based application moving to NAS?

A. This was discussed in one of the early slides in the presentation (slide 10, I believe).   Data mobility and granular management were discussed, as it’s easier to move, delete, and otherwise manage files than LUNs, an admin can operate at a more granular level, and it’s easier to operate and maintain.   No HBA’s, etc.   File protocols are generally considered “easier” to use.

 

 

New Webcast: Cloud File Services: SMB/CIFS and NFS…in the Cloud

Imagine evaporating your existing file server into the cloud with the same experience and level of control that you currently have on-premises. On October 1st, ESF will host a live Webcast that introduces the concept of Cloud File Services and discusses the pros and cons you need to consider.

There are numerous companies and startups offering “sync & share” solutions across multiple devices and locations, but for the enterprise, caveats are everywhere. Register now for this Webcast to learn:

  • Key drivers for file storage
  • Split administration with sync & share solutions and on-premises file services
  • Applications over File Services on-premises (SMB 3, NFS 4.1)
  • Moving to the cloud: your storage OS in a hyperscalar or service provider
  • Accommodating existing File Services workloads with Cloud File Services
  • Accommodating  cloud-hosted applications over Cloud File Services

This Webcast will be a vendor-neutral and informative discussion on this hot topic. And since it’s live, we encourage your to bring your questions for our experts. I hope you’ll register today and we’ll look forward to having you attend on October 1st.    

 

Upcoming Plugfests at SDC

This year’s SNIA Storage Developer Conference (SDC) will take place in Santa Clara, CA Sept. 15-18.   In addition to an exciting agenda with great speakers, there is an opportunity for vendors to participate in SNIA Plugfests. Two Plugfests that I think are worth noting are: SMB2/SMB3 and iSCSI.

These Plugfests enable a vendor to bring their implementations of SMB2/SMB3 and/or iSCSI to test, identify, and fix bugs in a collaborative setting with the goal of providing a forum in which companies can develop interoperable products. SNIA provides and supports networks and infrastructure for the Plugfest, creating a collaborative framework for testing. Plugfest participants work together to define the testing process, assuring that objectives are accomplished.

Still Time to Register

Great news! There is still time to register. Setup for the Plugfest begins on September 13, 2014 and testing begins on the September 14th.

Register here for the SMB2/SMB3 Plugfest

Register here for the iSCSI Plugfest

What to Expect at a Plugfest

Learn more about what takes place at the Plugfests by watching the  video interview of Jeremy Allison, Co-Creator of Samba, as he candidly talks about what to expect at an SDC Plugfest.

Learn more about the Plugfest registration process. If you have additional questions, please contact Arnold Jones (arnold@snia.org).