Earlier this year, IDC stated “Not only is [VAST] a vendor to watch in enterprise storage, it is THE vendor to watch.” Why? Because, at the core, VAST is different from other storage technologies in the market, and forwards a new architectural paradigm for the modern all-flash data platform. This architectural paradigm is grounded in the idea of democratizing flash and democratizing data access for all customers at the absolute lowest financial and operational costs.
One of the areas this architecture shines in is VAST’s unique ability to make all data access protocols first class citizens within our implementation. With our 4.2 and 4.3 releases, VAST has rapidly leapfrogged a host of competitors in offering the most scalable high performance S3 Object Storage solution available today, featuring extensive API support along with true support for running multi-protocol (file/object) workflows on the same data - permitting customers to realize one of the core values of the VAST Data Platform, the ability to run both existing applications and emerging workflows within the same unified storage environment and the capacity to leverage the best access methods to meet their technical and business needs.
How VAST Is Inherently Different
Unlike legacy “scale-out” products, VAST stores your data in a strictly consistent, highly reducible, radically scalable and elegant structure we call the “Element Store” rather than a unix-like filesystem which is limited by multi-decade old architectural principles that cannot support high-performance access to folders/buckets with many millions of objects within them. Nor is VAST limited to constrained volumes or “tacked on” protocol implementations layered on top of each other (or even worse, completely disjointed and non-interoperable protocol heads). VAST is the only solution that can support high-performance access to 100PB+ buckets whilst also servicing all your file-based data storage needs concurrently.
Ramping UP S3 Support
So, back to S3. We’ve been a bit coy about it until now, but VAST has supported S3 object storage in our Data Platform product since day one.
S3 Object Storage provides a number of advantages over file access in a host of workflows - from AI and ML on one end, to archive and backup on the other, object storage provides the data immutability, rich metadata and APIs to drive programmatic workflows.
Over the past few years, we quietly added HTTPS support for S3, Object Versioning, Multi-Part Uploads, and IAM integrations which unify S3 user identities with the identities used for file access over SMB and NFS. In doing so, VAST has dramatically increased its API completeness to support its rapidly growing S3 user base.
A Look At Our Latest Enhancements
In the latest versions of the VAST VAST Data Platform (4.2 and 4.3), we have added a number of features focused on managing and securing your Object data at scale.
Bucket Management
For customers who require the ability to create and manage a large number of S3 Buckets and Endpoints, VAST has expanded its capabilities within the VMS UI and API to enable creation and control of both, integrating completely into the same NFS and SMB view creation flow. This means that not only is it easier to create and manage your buckets, but it is now also possible to present file-based data directories as buckets (or vice versa) and choose how that data’s access is controlled from the same interface.
With S3 Endpoints, administrators can create an endpoint from the UI and assign those to users for self-service bucket creation and management. An endpoint is a location in your directory structure to which those delegated buckets are assigned.
Multi-Protocol ACLs
For environments which leverage Access Control Lists to control access to their file data and wish to extend similar functionality to their object data, VAST has supported S3 ACLs from our first release. For mixed file/object workflows, additional mapping and access checks have been implemented in this version to enable true ACL interoperability and the ability to determine which flavor of Access Control checks to use in a given view.
Object Locks
For customers running critical backup applications and secure data vaults, we added support for the Object Lock API in version 4.3. With Object Lock, customers can set policies on buckets and objects which prevent users and applications from deleting or modifying an object before its expiry date.
User Policies (with Multi-Protocol Support)
Similar to the work done on ACLs in version 4.2, in our most recent version we add support for IAM User Policies to granularize access control by using native S3 style policies which can factor in additional attributes of a file or user to control data access in a given bucket. This greatly reduces the overhead of access control management, and allows customers to abstract access control above the individual element - preventing security and data leakage which can be caused by unintentional ACL modifications at the pre-object level.
Even better, you can apply these same policy controls to buckets which are presented both as buckets and directories. By using User Policies in mixed file/object workflows, you can ensure that your users can only access the data they truly have access to without having to manage permissions on billions of individual objects.
What Are Our Customers Doing With S3?
Fast Backup + Recovery
The exabyte enterprise can no longer rely on slow tape and hybrid solutions to service their most critical backup data. VAST Data’s revolutionary Data Platform makes it possible for organizations to finally adopt an all-flash system for their data backup and recovery requirements. (Enterprise Backup & Recovery Revolutionary Data Storage Technology)
In the modern secure computing environment, it is not just good enough to backup your data, you also need to ensure that your fast backup tier is protected against accidental (or worse, intentional) modification and deletion.
With the expanded VAST S3 Object Lock API, you can apply additional immutability and protection for your backup assets - ensuring that your objects cannot be changed or deleted within the retention period set by your policy.
Big Data Pipelines
One of the big use cases we have seen rapid momentum in this past year is in customers moving multi-petabyte data pipelines and warehouses from HDFS-based stacks to S3. Large commercial environments are tired of paying the redundancy tax on HDFS’ storage architecture while suffering under the performance constraints and spiraling costs of legacy scale-out vendors, are moving quickly to adopt S3 for big data applications. You can learn more about how VAST Data’s high-performance, low-cost all flash S3 solution allows Agoda to build a cost-effective, private cloud computing environment for Apache Spark and Apache Impala processing.
Multiprotocol Creation + Distribution
VAST’s multi-protocol story has seen adoption in a number of verticals, from finance to media and everything in between, who need a solution which can service both their creation and distribution requirements using both file and object semantics. Be that writing time-series data as objects and presenting them as R/O files, to editing media content as files and presenting them as R/O buckets, our customers are leveraging the power of VAST’s multi-protocol file/object storage to help their workflows take advantage of their full application ecosystem in one data platform.
By leveraging our multi-protocol ACLs and policies, customers are able to create strategies that can granularize the type of access given to an editor accessing the storage via Linux or Mac over NFS for a read-write workflow, and permit consumers consistent and read-only access to the same content presented as object data (or vice versa!).
Conclusion
VAST Data’s approach to S3 provides a much more comprehensive object storage solution, especially at a multi-petabyte and beyond scale. Unlike other vendors, we’ve integrated S3 in a way that provides a unified, multi-protocol experience that allows customers to reduce the storage footprint and workflow latencies, whilst bringing down the overall cost of storage and complexity of managing multiple versions of the same data asset.
Ready to see some of these features in action, then check out our latest demo.
