Before we expanded the VAST Data Platform to understand the structure of database tables and process data through the VAST Data Engine, we called it Universal Storage. While support for all the common file and object protocols covered a wide swath of use cases, until now the universe of storage applications VAST could fill was big, maybe even galactic, but not quite Universal. With our latest software release we’re adding support for two more classes of data persistence: block storage and event streams with support for NVMe over TCP (NVMe/TCP) and the Apache Kafka APIs.
Each of these new protocols is a peer with all the existing file, object and database access protocols. The VAST Data Platform supports all of these data abstractions natively with our own more general abstraction of an element, the basic data unit any protocol accesses. For file and object protocols an element is a file/object; for database access an element is a table; and with NVMe/TCP each shared virtual SSD is just another type of element. All elements use the same database structures to map logical locations to the actual data on SSDs in a VAST cluster.
NVMe over TCP – Block Access
(Or Block Access over NVMe)
Block storage is the oldest of all shared storage metaphors. Even before the term SAN (Storage Area Network) was used to describe Fibre Channel in the 1990s, Ethernet LAN systems like 3+Share and Corvus Omninet allowed multiple PCs to share an expensive hard drive with each user seeing a partition of the shared disk as a local hard disk. Block storage’s advantage is its simplicity, back in the day that meant that block storage was always faster (had lower latency) than file storage. Several things have changed since back in the day, most significantly:
We went from single core array controllers to CNodes and EBoxes with 10s of cores to process metadata faster
Block storage added metadata-driven features including snapshots, thin-provisioning and data reduction, eliminating the “Block is faster because there’s less metadata” argument
File protocols have become less chatty and added performance enhancements like RDMA and multichannel
Even though we’ve clearly shown in the AI world that file access protocols can provide TB/s of performance and that many, if not most, workloads run in virtualized or containerized environments that store virtual disks as files (.VMDKs, .VHDs and the like), block storage remains the standard for many. Block storage also simplifies maintaining the boot images across large server farms. Once a master image is developed it’s easy to clone it and configure the servers to boot from SAN.
How We Did It
NVMe over Fabrics (NVMe-oF) is the modern block protocol and we at VAST have been using NVMe over Fabrics to connect CNodes their cluster’s SSDs since DASE was just a great whiteboard diagram. So it should come as no surprise that when we decided to offer block access that we chose NVMe-oF rather than iSCSI, Fibre Channel or other legacy SCSI based block protocols. NVMe over TCP uses TCP/IP as the transport protocol so it can run over standard data center networks with almost as low latency as the NVMe over RDMA VAST clusters run internally without the need for any specific network configuration. Our own VP of Architecture Sagi Grimberg was the primary author of the NVMe over TCP specification, and the engineers working on VAST’s implementation have sent him at least a dozen punctuation corrections.
As you should expect from VAST, our block protocols are designed to support vast implementations a single VAST cluster supports:
Up to 32,000 NVMe subsystems (virtual arrays)
Up to a million volume elements
Up to a million hosts
Providing block storage is easy. All an administrator has to do is, set up the VIP (Virtual IP address) pool that hosts will use for block access; create a view with block protocols enabled, which creates an NVMe subsystem; and then create the logical volumes (AKA LUNs for historical reasons) within that subsystem. Define which hosts should access the volumes and Robert is your father’s brother as they say.
See how it works here, including what it looks like to access a cluster from Linux over NVMe/TCP:
Each volume is an element contained in a folder in the cluster’s namespace. Placing each volume uniquely in a folder means that all the folder level data services of the VAST Data Platform, including snapshots, clones, replication, encryption key management and QoS, can also be applied to block volumes. Speaking of snapshots, remember that all snapshots on a VAST cluster taken at the same time - and therefore by the same protection policy - are inherently consistent, eliminating the need for explicit consistency groups.
Kafka – The Event Streaming Protocol
Apache Kafka has emerged as a de facto standard broker of the streaming events that transfer data between microservices (functions) in cloud-native computing architectures and more general compute stages in traditional data workflows. A Kafka system manages entries into message queues called Topics.
Clients and services that want to write entries into a Topic use the producer API to write to the topic on one of the Broker servers running Kafka, and services that want to read entries from the topic call the consumer API. Topic data is stored locally on the Brokers protected by replication. The whole thing is managed by a Zookeeper instance that also tracks the offsets (pointers) to the current entry in each topic.
As we thought about the VAST Data Engine it became clear that we would need an event broker and that the Kafka producer and consumer APIs looked a lot like a storage protocol. It turns out in this case the difference between a protocol and an API is a matter of perspective. To a programmer it’s an API, but just like a storage protocol that API defines how the messages on the wire will be formatted to perform each operation in the API.
So when we built the VAST event broker - part of the VAST Data Engine if you’re keeping score -, we implemented the Kafka consumer and producer APIs as a protocol module just like NFS and S3. This means VAST customers can manage event broker instances through the same virtual IP address pools and view policies they use to control access to their folders and object buckets. And like all VAST protocols, we wrote all the code in-house.
The VAST event broker stores each Topic as a table in the VAST DataBase, allowing not just streaming events from producers to consumers but also real-time analysis of the data streams in SQL. With Apache Kafka this requires an external database server and a Kafka connector. The VAST event broker is self-tuning with no need to manage replication factors, local disk storage and Kafka cluster management.
By treating Kafka as a protocol we’re providing the high performance, almost- infinitely scalable event broker infrastructure that will be at the core of the VAST Data Engine’s pipeline automation and making that same infrastructure available through standard APIs.
And of course, continuous enhancements
While block storage support and Kafka are the headliners of the new release, the engineers in their back rooms have also enhanced the VAST Data Platform’s security, including support for Fortinix and Hashicorp enterprise key managers, extended multi-tenancy to allow tenant managers access to the VAST GUI and dashboards, and the addition of DataBase Views.
For more details on this latest release, join me on a webinar on February 27th.
