The integration of the Lustre filesystem with OpenStack is a significant improvement in the world of large-scale computing. Lustre, an open-source, high-performance, scalable storage solution, has long been a popular choice for use in large-scale computing environments. By integrating Lustre with OpenStack, users can now benefit from the performance and large scale of Lustre storage systems deployed on HPC sites for the implementation of services provided by OpenStack, such as virtual machine services and archival data repositories.
One of the key advantages of this integration is the ability to use Lustre as a block storage backend for OpenStack virtual machines. This enables users to leverage the high performance and scalability of Lustre to support their virtualization needs, while also benefiting from the other features and services offered by OpenStack. From a technical standpoint, this integration is achieved by developing a Lustre connector for OpenStack Cinder.
Another important aspect of the integration is the optimization of Lustre as a file storage backend for OpenStack Swift object storage. This allows users to take advantage of the high-performance file storage capabilities of Lustre to support their object storage needs, while also benefiting from the integration with OpenStack and its other services.
In addition to these technical benefits, the integration of Lustre with OpenStack also provides a consistent view of data accessed directly in the Lustre filesystem and data accessed via the Swift object interface. This allows users to seamlessly access and manage their data across the two interfaces, without the need to copy or move data between two different storage systems.
This unified view is the trickiest point of this integration. This feature requires synchronization of the namespaces of the two systems. The conversion of Swift objects into Lustre files was accomplished by rehabilitating a tool called swift-on-file, which was ported to the latest version of OpenStack. The code was also modernized through its porting to Python 3. Finally, the tool was made more robust by ensuring nearly 100% test coverage.
The inverse operation of making Lustre files available through Swift is done through the processing of events generated by Lustre: changelogs. This mechanism allows for the tracking of changes made in the Lustre filesystem to make them visible through the Swift interface.
Overall, the integration of Lustre with OpenStack brings new possibilities to the architecture of computing centers and datacenters, offering users a powerful and scalable storage solution that is well-suited for use in a variety of different environments from HPC compute nodes to cloud usages.
With its open-source nature, the integration is also readily available to the broader community, allowing others to benefit from its capabilities.
