HSM-RL

Efficient Hierarchical Storage Management Empowered by Reinforcement Learning.

Workflow of the RL-based HSM-RL migration policy
Workflow of HSM-RL on a three-tier hierarchical storage system. The controller decides upgrades from access patterns (step 1), while per-tier RL agents update cost functions (steps 2–3) to place files more efficiently.

Overview

Hierarchical storage systems (HSS) pool heterogeneous devices—fast but small SSDs, slower HDDs, and large object stores— into one storage hierarchy. The challenge is where to place each file as access patterns change: hot data should live in fast tiers, cold data in cheaper capacity, but naive or static policies waste bandwidth on unnecessary migrations and fail to adapt when workloads shift.

HSM-RL treats file placement as an online control problem. Each tier is associated with an RL agent whose state captures average file temperature, size-weighted temperature, and queueing delay. A fuzzy rule-based function approximates the tier cost, and agents update their parameters with TD(λ) learning. A file is upgraded only when the predicted system response after the move is lower than leaving it in place.

In our TKDE paper we evaluate the policy in both a simulator and a live cloud HSS, showing about 5–6× fewer transfers than rule-based baselines with comparable memory cost. In the later ESWA paper we extend the framework to scientific workloads including microscopy, CFD meshes, 1000 Genomes sequences, and phenotypic screening. By encapsulating dataset-specific interestingness features into the state variables of RL agents, we achieve lower response time than LRU, LFU, k-means, and even prefixed min/max policies.

Highlights

  • Problem: Adaptive file placement across storage tiers under changing access patterns.
  • Approach: Per-tier RL agents with fuzzy value approximation and TD(λ) updates.
  • Results: Far fewer migrations than rule-based policies (TKDE); faster scientific analysis pipelines vs. LRU/LFU/k-means (ESWA).

Key figures

Three-tier HSS with per-tier RL agents
Three-tier HSS with one RL agent per tier: incoming requests trigger migrations, then agents update from observed response.
Number of transfers and estimated system response time under each policy
Average number of transfers and estimated system response time under each policy (initial temperature 0.4–0.6, uniformly random requests). RL policies reach a similar response with far fewer migrations (TKDE).
System response time on phenotypic screening under HSM-RL and baselines
System response time on the first-phase phenotypic screening under RL vs. random, LRU, LFU, k-means, and min-intensity baselines (ESWA). The random-placement panel uses a different y-axis scale.

Links