By storing triggers and functions as state in the VAST Data Platform, your code becomes dynamically managed by a global data store that supports global code versioning, global code distribution and global code security policies.

Deep learning data engineering is tough. Data engineers write large dataset files down to archive storage for training… creating a number of problems associated with rigid data management:

• If model training requires data variation, new datasets are written down to storage, often creating redundant data because datasets use overlapping training example data

• Because conventional datasets are not embedded with training code, it can often be difficult to reproduce training models as data and code continue to evolve independently

With the DataEngine, VAST is introducing a new concept called the VAST DataSet. This new approach to data management leverages the VAST Database to create materialized views of example data without copying and re-copying data into blunt data containers. DataSets can scale to exabytes. Each DataSet includes an indexed set of examples and the code used for training so that it’s easy to reproduce models on the fly.