Ever since VAST's inception, we’ve aimed to deliver all-flash performance at a cost organizations could afford for their archival data. We knew that would allow organizations to turn their latest analysis tools, from Spark and Trino to AI inference engines, on those archives, extracting untold value from what was previously data in cold storage with negligible performance.
That archival data can now include archives from organizations in highly regulated industries that are required to store their archives on immutable storage. Cohasset Associates has validated that the VAST DataStore provides the immutable storage required by many financial regulations.
What’s the Requirement?
Many regulations, from HIPAA to Sarbanes-Oxley, include requirements that organizations retain the records covered by those regulations for some time, leaving the methods up to each organization. Some industries, particularly in finance, are subject to a more exacting regulatory regime that requires, in the words of The Securities Exchange Act of 1934 Rule 17a–4(f), The electronic storage media must preserve the records exclusively in a non-rewritable, non-erasable format.
The original interpretation of non-rewritable, non-erasable was that digital data had to be written to optical disks like DVD-R that could only be written to once, known as WORM media for Write Once Read Many. Over the years, the SEC has released interpretations of rule 17a-4(f) to allow compliance data to be stored on systems like VAST that prevent data from being rewritten or erased through software as long as the vendor, or a reputable third party, validates that data on the system is truly immutable until the end of its retention period.
With the possibility of multimillion-dollar fines and immeasurable reputational damage for non-compliance, the compliance officers at major banks wouldn’t rely on storage vendors’ self-certifications, so records management consultancy Cohasset Associates emerged as the gold standard of third-party validation.
Cohasset’s VAST DataStore Compliance Assessment validates that when configured properly, The VAST DataStore meets the requirements for immutability and audibility in SEC rules 17a-4(f), 18a-6(e), FINRA (Financial Industry Regulatory Authority) Rule 4511(c) and CFTC (Commodity Futures Trading Commission) rule 1.31(c)-(d).
Most organizations today meet their compliance requirements with dedicated compliance archives. Since they view those archives as strictly a compliance solution with no real read performance requirements, these archives rest on object stores or NAS systems using large hard drives and, therefore, extremely limited performance.
That limited performance just isn’t enough for those organizations to let their generative AI models feed on those archives and learn that certain background sounds were more common on customer phone calls that turned out to be fraudulent or that the model identified a shockingly high percentage of customers who complained about their brokers sounded drunk on one or more calls, even if those weren’t the transactions they complained about.
With VAST your compliance archive is unlocked for you to glean knowledge from on an all-flash system with support for GPU direct storage and NVIDIA SuperPods, but still locked against deletion and modification as required by SEC Rule 17a-4 and similar regulations.
Data Retention the VAST Way
VAST systems give administrators several ways to protect their data from accidental or intentional deletion or modification:
Indestructible Snapshots – Indestructible snapshots protect snapshots from deletion or modification until the end of their retention period. Snapshot policies are also protected to prevent deleting a schedule. Indestructible Snapshots are useful for RansomWare protection and recovery but snapshots are best for protecting active datasets. Archivists shouldn’t have to page through 100s of snapshots to find the data they’re looking for. In addition, Indestructible Snapshots can be deleted under certain conditions which isn’t allowed for compliance.
S3 Object Lock – S3 Object Lock prevents objects from being deleted (S3 objects are always non-modifiable) from VAST Views (Buckets) with Object Lock enabled until the end of their retention period. AWS defines the Object Lock API, and VAST systems are compatible with the AWS version.
File Lock – File Lock makes files read-only and prevents that read-only status from being changed until the file expires. When an administrator configures a VAST View for File Lock, they can set a default retention period and a grace period for the folder. Files in a folder with a seven-year retention period and a one-hour grace period will be locked for seven years one hour after they’re last written to. Users and applications can set individual file retention directly by setting the last accessed date (atime) metadata attribute of any file in a File Lock View to a future date and setting the file Read-Only. This is a common method for enterprise NAS systems supporting compliance locking so archive applications can use File Lock without additional coding.
Note that once a file or object is locked, retain-until-date can be extended further into the future but cannot be changed to shorten the retention period.
Compliance and Governance Mode
When you set a View for either Object Lock or File Lock, you can choose between compliance mode or the slightly less restrictive governance mode. What’s the difference? Whether the retention lock can be bypassed.
In compliance mode, once a file/object has had a Retain-Until-Date set, it can not be modified or erased until the Retain-Until-Date, and there is no way to override or change the retention date to shorten the retention period.
While compliance mode is great for compliance, it does have one significant drawback: since there’s no way to delete data from a compliance archive, customers have to be careful not to overfill them.
In governance mode, a sufficiently privileged user can override the retention lock and delete or edit files or objects. VAST’s Indestructible Snapshots provide the equivalent of governance mode because VAST support can provide a VAST customer organization with the time-limited token that unlocks the Indestructible Snapshot’s immutability.
Governance mode allows customers to perform triage and decide that temporarily reducing their retention from 90 to 80 days is better than having their applications crash because they’re out of storage space.
Put another way, Governance mode protects data against rogue users or processes running as rogue users like Ransomware. Compliance mode is intended to protect data against corrupt organizations that would change their records to cheat a customer or on their taxes.
Since every user at a corrupt organization is suspect, Cohasset, and therefore Wall Street, has interpreted non-modifiable, non-erasable to mean a system with no back doors.
Legal Hold
Legal Hold is a general do-not-delete flag on a file/object in a folder with Object Lock or File Lock Retention enabled. This flag is generally applied as part of processing an e-discovery request indicating it’s a file that can’t be deleted until the case is settled or the file is determined not to be germane.
For S3 Object lock, VAST implements the AWS API; for File Retention, a Legal Hold can be applied to any element via a REST call to VMS. Legal hold appears as a column in the VAST Catalog, making it easy to manage all the Elements on hold at once.
Clock Protection
Compliance storage is supposed to keep data immutable for a specified period. File and Object Lock protect the data by preventing changes and deletions, but keeping data safe also means the clock has to be protected.
VAST systems holding compliance mode Views must use an external NTP server. If the VAST Cluster’s internal clock ever gets more than 5 minutes out of sync with the NTP server, the cluster will block deletion for any file or object with enforced retention, even if the retention period for that file has expired.
VAST support can reset a system to allow the deletion of files after their expiration date if it has gotten out of sync with its NTP server, but support will only reset a system with a valid system time. That is, support will not re-enable deletion on a system with a clock set in the future.
Taking Archives from WORN to AI
All too many organizations today maintain their compliance archives for compliance alone. They store a copy of every customer communication in an immutable archive for seven years as the law requires and then hope they never have to retrieve data from that deep archive. They keep additional copies of the transaction records in a data lake and short-term archives to recover the phone call recording where Uncle Scrooge told his broker to sell all his Nvidia stock when he challenged the trade in arbitration.
Now that firms can store the compliance archive on VAST instead of a disk-based compliance silo, there’s no need for separate active and archive copies of the data. They can analyze and infer from the compliance archive directly. Even better, they can extend their analysis over the longer time frame of data in the compliance archive vs the typically smaller active second copy.
Read the Cohasset Associates Compliance Assessment here.
