Microsoft has announced the availability of Azure Managed Lustre for high-performance computing and AI workloads. Lustre is an alternative filesystem (like NTFS or FAT) that’s popular on supercomputers, including Frontier.
According to Microsoft, Azure Managed Lustre brings the Lustre filesystem to customers as a first-party managed service on Azure. It said:
“Long-time users of Lustre on-premises can now leverage the benefits of a complete HPC solution, including compute and high-performance storage, delivered on Azure. Users benefit from the high throughput and performance capabilities of Lustre as a seamless managed experience and can now focus on their business imperatives.”
The Lustre filesystem is already very popular on the world’s top supercomputers but thanks to the arrival of generative AI there’s renewed interest in Lustre. Outlining the benefits of Azure Managed Lustre, Microsoft lists:
- A managed PaaS experience for worry-free deployment and consumption of Lustre file system.
- Reduced TCO with pay-as-you-go pricing.
- Throughput options at different pricing levels for different workloads.
- Easy deployment of independent clusters with predictable performance, eliminating the noisy neighbor problem commonly experienced in on-premises Lustre deployments.
- Quick deployment in a few minutes, alleviating the need to plan months or years in advance and meeting compliance and data residency requirements.
- Region of choice for deployments across many Azure regions, supporting compliance and data residency requirements.
- Scalability to hundreds of petabytes (PB), with the associated durability, availability and cost benefits of tiering to Azure Blob storage.
- Seamless integration with existing Azure compute services including Azure HPC services, Azure Machine Learning and Azure Kubernetes service.
- Investment protection for customers’ existing automation and platform development when migrating from on-premises Lustre deployments.
Starting today, Microsoft has two persistent durable instances for customers to use. They are AMLFS Standard (125 MB/s) and AMLFS Premium (250 MB/s).
Lustre client packages can be downloaded from packages.microsoft.com for various Linux distributions and kernel versions. Microsoft says it’ll have support for HPC images pre-built with Lustre client packages for Ubuntu 18.04, 20.04, and 22.04 as well as Alma 8.7.
You can learn more on the Azure Managed Lustre product page.