In Part Vermillion of our SNIA Ethernet Storage Forum (ESF) “Everything You Wanted To Know About Storage But Were Too Proud To Ask” webcast series – we examined the terms and concepts are at the heart of where compute, networking and storage intersect. That’s why we called it “What if Programming and Networking Had a Storage Baby” If you missed the live webcast, you can watch it on-demand.
The discussion from our panel of experts generated a lot of good questions. As promised, here are answers to them all.
Q. With regard to persistent memory, how does one decide if it’s better to use load/store or access via I/O?
A. Legacy applications will not change and hence will access the persistent memory the way they were written. If your legacy application needs a lot of memory and you want to use the new persistent memory as just a big and cheap (volatile) memory, then the access will be byte addressable (load/store). If your legacy application uses block storage then it will use the persistent memory using block addressing. New applications can take advantage of using byte addressing and persistency. They can keep all their data structures in memory, and these data structures will also be persistent! This saves applications the need to serialize before storing and to de-serialize when retrieving from storage and enables many other ways to optimize the software.
Q. Can you go over again a bit more slowly how byte accessible and LBA change with persistent memory?
A. Persistent memory can be accessed in three different ways.
- Using byte addressing, in which case it behaves like a big (volatile) memory
- Using logical block addressing, in which case it behaves like a block storage device
- Using SNIA NVM Programming Model that enable byte addressing along with persistency. In this case byte being written into the device can be made persistent with special APIs
You can configure and decide what model is better use for your application.
Q. Is that like flash?
A. Persistent memory is a technology that is persistent like flash, but has byte addressing. It can be implemented using underlying flash, battery backed DRAM, Phase Change Memory and more.
Q. You were going to parse out flash vs. NVMe, I think. Also, how will the elements discussed during the session impact these evolving technologies?
A. Flash is a non-volatile memory technology that supports block addressing.
PCM is another non-volatile technology which is newer that supports byte addressing (which implies that it can also do block addressing by emulation). NVMe describes an interface to access non-volatile memory technology, by placing the non-volatile memory over the PCI bus. Storage Class Memory is yet another interface to access non-volatile memory, by placing the non-volatile memory over the memory bus.
With this in mind:
1) It is common to see NVMe devices with backing flash devices. They will support block addressable. They have the option to expose a small byte addressable memory buffer as well on the PCI (typically a DRAM), which may or may not be volatile.
2) It is common to see Storage Class Memory with backing PCM device, or with DRAM (that can backup itself to flash on power failure). They will support byte addressable.
Q. Regarding SMB & CIFS protocols, is SMB or CIFS the deprecated one?
A. The name CIFS hasn’t been used in a while; it’s now all SMB. SMB version1 is deprecated; see this Microsoft article. Don’t use CIFS!
Q. Are there any rules of thumb in regards to the efficiency of block vs. file vs. object stores from the storage capacity overhead and network “busyness”?
A. Effectively, as you get closer to the lower-level block storage devices, your storage networking architecture needs to become more deterministic. That is, you begin to start caring more about the number of hosts connecting to a particular storage target (fan-in ratio) and the ratio of bandwidth the target has compared to the bandwidth that the hosts connecting to it have (oversubscription).
Highly-transactional block storage protocols, such as Fibre Channel, FCoE and lossless iSCSI, for example, will need to have very low oversubscription ratios (sometimes as low as 4:1, depending on the type of application/workload). Most are somewhat more forgiving, and 16:1 and 20:1 are not uncommon. When you move into file-based systems, that oversubscription can be a lot higher (there is no general rule of thumb here, but the oversubscription can be in the low hundreds:1). Object-based systems are so scaled and distributed, that there really are no oversubscription limits at all, because those systems are not highly transactional.
Q. Does an object always have to be replaced in entirety? How do systems handle updates to large objects?
A. The rule is that you shouldn’t take a lock on an object. Classically, the whole object should be replaced. Updating is not straightforward.
Traditional “get/release” locking is too expensive in terms of latency over geographic distances, too hard to manage in a distributed environment, is hard to scale, needs recovery in the case of failure, and introduces state to what is basically storage built for stateless operations. Plus, the object may be sharded across multiple physical systems.
Some object systems do allow what they call “pessimistic locking” (take a lock for a fixed period of time, say 10 seconds) but it’s not a true lock that you obtain then release. It’s more like a window of opportunity and is often called, and acts like, a lease. There are also other techniques, like “optimistic concurrency” (using a unique identifier, try and then check if your identifier was successful) and “last writer wins” (as it says, the last write is the one that the storage system remembers). Many systems do this by snapshotting the object, allowing updates on the copy, and then atomically swapping them.
Object systems differ in what they permit. In general, applications need to be aware that they may, very occasionally, not be successful when modifying objects, and to have strategies to deal with it, like retrying or even simply giving up.
Again, you can check out the recorded version of the webcast at your convenience and you can download the webcasts slides as well if you’d like to follow along. Remember, this webcast was part of series. I encourage you to register today for our next one, which will be on August 1st at 10:00 am PT – Part Turquoise “Where Does My Data Go?” And please visit the SNIA ESF website for our full library of ESF webcasts.