The “Swift Enhancements” R&D project is a collaboration between ICEI partners and IBM to extend an existing open project called SwiftHLM, developed by IBM Labs. The goal of this enhancement is to create a high-latency storage backend (like a tape archive system) that makes it easier for users to perform bulk operations of data tiering within a Swift data ring.
With the addition of SwiftHLM to OpenStack Swift, we allow users to control and query the state (on disk or HLM) of Swift object data, including the efficient prefetch of bulk objects from HLM to disk.
SwiftHLM has many backend connectors, and it can work with several high-latency backends that are POSIX/DMAPI compatible.
Tiered data flow that uses SwiftHLM in an OpenStack Swift environment.
SwiftHLM consists of:
-
SwiftHLM Middleware (Proxy nodes)
-
Proxy middleware exposing the enhanced API
-
-
SwiftHLM Dispatcher (Swift node)
-
Background daemon creates a list of objects, identifies the Storage Node for each object, and dispatches asynchronously to the appropriate Swift Storage Node
-
-
SwiftHLM Handler (Storage Nodes)
-
Provides/invokes generic interface towards SwiftHLM backend storage
-
Maps objects to files and submits the mapped list to the backend (via Connector)
-
SwiftHLM requires a backend-specific Connector module (the Connector is not part of the SwiftHLM packaging) that is supplied by the vendor of the backend software/hardware (e.g. IBM Spectrum Archive EE, IBM Spectrum Protect).
High-level architecture for high-latency, low-cost media storage solution
Overall SwiftHLM will enable user driven storage tiering that will augment cloud object storage with a low-cost, cold storage tier (Tape, optical), helping users and Data Centers to reduce the cost of long-term object storage (tape up to 6x cheaper than disk).
