Introduction
As data sets scale to exabytes it becomes increasingly difficult to manage as we have in the past. To date we’ve relied on humans to understand and keep track of workloads, growth, and utilization, it’s becoming increasingly practical to leverage artificial intelligence to keep pace with these challenges. With the release of VAST 4.6 we introduce Uplink Prediction, our first foray into applying machine learning to exascale data management.
Uplink is our cloud-based service for telemetry and centralized fleet management. Uplink collects telemetry from participating VAST clusters for proactive monitoring, allowing us to leverage machine learning in various innovative ways to benefit operations. Our focus in the 4.6 release is to give customers a look into the future of your VAST cluster and how capacity is consumed with the granularity to drill down for greater insights. Enter Uplink Prediction.
How does Uplink Prediction work?
Time series analysis of data sets is common across nearly every aspect of life: daily temperature graphs, heart rate monitoring (EKG), stock prices, etc. Automatic analysis of these data sets provides us with insights and helps us understand complex issues. We apply this same type of analysis to the temporal dynamics of file system units as well as the entire storage capacity of VAST clusters.
VAST time series data has its own unique properties: long-lasting trends, spikes of periodical activity (writes and deletes), weak seasonality, and prominent behavioral changes that are associated with new projects or major events.
We applied techniques such as Seasonal-Trend decomposition, Fast Fourier transform, and generalized linear regression as well as our own solutions for smoothing, trend anomalies detection, and periodical cleanups (delete) tracking. A key consideration is that data accumulation and data deletion are two distinct processes that are very different in nature. Here we combine human intelligence for greater accuracy. Customers use a simple slider to select the time period that they believe best represents “normal” and Uplink creates a prediction on the fly. For example, if a customer has recently gone through an acquisition or divestiture, the data patterns will be impacted, and the ability to select the right time window for analysis enables a more accurate forecast.
How do you use it?
Uplink Prediction graphically presents temporal capacity patterns gleaned from your VAST clusters in a simple web UI. Now you can see which folders are growing or shrinking at any point in the history of the cluster.
Drill down into individual folders to understand how workloads are growing
Here we display the predicted capacity for /scratch.
Looking into the future is how you can plan for growth – using the time slider you can now see a prediction with path confidence out to a year.
One year of growth analysis based on a customized time window
Keep in mind that while we are using the term “folder or directory,” VAST supports file and object equally in the same namespace. The data may be presented over NFS, SMB, S3, or some combination via VAST Views.
New Possibilities, New Potential
With this deep visibility customers can now answer questions that were difficult to answer at scale with traditional tools like du. Now it’s simple to answer questions like:
Which projects are growing the fastest?
When will I need to add more capacity?
How would removing a specific data set impact capacity?
Uplink Prediction is just the first application of AI to help customers optimize their VAST ownership experience. The possibilities to better understand infrastructure, performance, and even advanced capabilities like anomaly detection are on the horizon. Stay tuned.
