VCF 9.1 Upgrade Benefits: Improve Efficiency with Memory Tiering

Memory remains one of the most expensive and capacity-constrained resources in modern virtualisation environments. As AI, analytics, VDI, and memory-intensive enterprise workloads continue to grow, scaling DRAM alone becomes increasingly costly and operationally challenging.

VMware Cloud Foundation (VCF) 9.1 introduces enhanced memory tiering capabilities designed to extend memory capacity using high-speed NVMe devices, enabling organisations to increase host memory density while optimising infrastructure cost efficiency.

As we continue to unpack the benefits and features of why you should upgrade to VCF 9.1, this product update will explore:

• What is memory tiering 
• How memory tiering works 
• NVMe device requirements 
• Capacity limits 
• Mirrored vs non-mirrored configurations 
• Cluster and host-level enablement 
• Networking considerations 
• Recommended and non-recommended workloads 
• Operational best practices 

What Is Memory Tiering?

Memory tiering in VMware Cloud Foundation 9.1 allows ESXi hosts to use locally attached NVMe storage devices as an extension of system memory. Instead of relying exclusively on DRAM, frequently accessed memory pages remain in DRAM, while less active “cold” pages are dynamically moved to a lower-cost NVMe memory tier. 

This creates a two-tier hierarchical memory architecture:

The objective is to:

• increase effective memory capacity
• reduce DRAM expansion costs
• improve workload consolidation ratios 
• support larger memory footprints per cluster

VCF 9.1 significantly enhances the operational experience through:

• cluster-wide configuration
• improved monitoring
• host profile integration 
• simplified lifecycle management

How Does Memory Tiering Work?

Memory tiering dynamically identifies inactive or less frequently accessed memory pages and migrates them from DRAM to NVMe-based memory tiers.

The ESXi NUMA-aware scheduler and memory management stack coordinate:

• memory placement 
• movement
• optimisation 

The feature operates entirely at the hypervisor level, and applications remain unaware of the tiering process, meaning that no guest OS modifications are required and no application redesign is necessary.

NVMe Device Requirements

To enable memory tiering, a minimum of one NVMe device per ESXi host is required while two NVMe devices per ESXi host are required for mirrored, highly available deployments.

Mirrored vs Non-Mirrored Memory Tiering

VCF 9.1 supports both mirrored and non-mirrored memory tier configurations.

Non-Mirrored Configuration

A non-mirrored deployment uses a single NVMe device, which maximises usable capacity and reduces hardware costs, but it also introduces a single point of failure for the memory tier.

Mirrored Configuration

A mirrored deployment requires two NVMe devices and synchronises tiered memory data to provide resiliency against device failure.

Minimum and Maximum Capacity Requirements

VCF 9.1 has the following per host limits for memory tier capacity.

Where Memory Tiering Can Be Enabled

VCF 9.1 introduces multiple deployment models for Memory Tiering.

Single Host Enablement

Administrators can configure Memory Tiering:

• directly through CLI
• on individual ESXi hosts. 

This is useful for testing, lab environments, validation scenarios, or staged rollouts. 

VCF includes:

• CLI-based configuration commands
• single-host operational workflows

Cluster-Level Enablement

This is one of the largest operational improvements in VCF 9.1.

Capabilities include:

• centralised policy management
• consistent configuration across hosts 
• host profile integration
• simplified lifecycle management 
• easier operational scaling

The platform includes:

• cluster level memory tiering configuration 
• configuration experience improvements 
• host profile overrides 
• monitoring enhancements 

Networking Requirements

A key architectural consideration for any VCF 9.1 environment is the networking layer and while Memory Tiering itself does NOT require dedicated networking the implication of having more memory in a host is more workloads are likely running on the host thus creating a higher demand on:

• Distributed Resource Scheduler (DRS) / vMotion
• vSAN
• External IP Based Storage

Although NVMe devices are locally attached to each ESXi host, memory migration between the DRAM and NVMe tiers occurs internally within the host, meaning there is no network dependency for accessing these tiers.

For VCF 9.1 deployments with vSAN, we recommend 4 x 25GB connections per host to ensure consistent high performance and redundancy, considering operational and failure scenarios.

NVMe Device Choice

Not all NVMe devices are suitable for memory tiering and only enterprise-grade NVMe devices should be used.

Enterprise grade devices ensure:

• high endurance ratings, 
• predictable latency, 
• power-loss protection, 
• consistent write performance, 
• and PCIe Gen4 or newer interfaces. 

Recommended Workloads

Memory Tiering works best with workloads that:

• have large memory footprints
• contain cold/inactive memory pages 
• benefit from increased consolidation 

Virtual Desktop Infrastructure (VDI)

VDI environments often contain:

• large memory pools
• repetitive memory patterns 
• moderate latency sensitivity

AI/ML Inference

Inference workloads frequently benefit from:

• expanded memory capacity, 
• large datasets, 
• moderate memory locality tolerance

Large Database Read Workloads

Read-heavy databases with:

• large caches
• moderate active memory sets 
• infrequently accessed data pages

General Virtualisation

Mixed enterprise workloads with:

• steady-state memory usage
• moderate overcommit ratios 
• predictable resource consumption

Kubernetes Worker Clusters

Containerised environments with:

• high density
• variable utilisation 
• elastic scaling patterns

Non-Recommended Workloads

Some workloads are highly latency-sensitive and may experience performance degradation if tiered pages are accessed frequently and form part of our suggested non-recommended workloads.

High-Frequency Trading (HFT)

Ultra-low latency workloads should remain DRAM-only.

Real-Time Analytics

Applications requiring deterministic latency may suffer from NVMe access penalties.

Ultra-Low-Latency Databases

Databases with:

• highly active memory pages 
• intensive write activity, 
• microsecond-sensitive response times

Real-Time Control Systems

Industrial or manufacturing control systems often demand deterministic memory response times, with zero tolerance for any latency caused by tier transitions.

Extremely Write-Intensive Memory Workloads

Heavy page churn can:

• increase NVMe wear
• reduce efficiency
• introduce latency variability

Operational Best Practices

Mirrored memory tiering is strongly recommended for:

• mission-critical workloads
• production clusters 
• high-availability environments

Given the minimal cost of NVMe compared to DRAM and the overall value of Memory Tiering, the investment in a second NVMe device per host is highly recommended.

Monitor Tier Utilisation

VCF 9.1 introduces:

• enhanced Memory Tiering dashboards 
• performance visibility
• health monitoring capabilities

Administrators should monitor:

• tier hit rates
• latency
• endurance 
• memory migration behaviour

Validate NUMA Alignment

VCF 9.1 includes NUMA scheduler improvements that complement Memory Tiering capabilities. Proper NUMA alignment remains critical for:

• predictable performance 
• reduced cross-node latency 
• efficient memory placement

Test Workloads Before Broad Deployment

Not every workload benefits equally from Memory Tiering. It is therefore recommended to conduct the below before enabling cluster-wide deployment.

• pilot testing 
• benchmark validation 
• performance characterisation

Key Takeaways on Memory Tiering

Memory Tiering in VMware Cloud Foundation 9.1 represents a significant advancement in infrastructure scalability and memory optimisation.

By leveraging high-speed NVMe devices as an extension of DRAM, organisations can:

• increase workload density
• reduce DRAM memory costs 
• extend hardware lifecycle value 
• support larger modern workloads 

VCF 9.1 further improves the operational maturity of memory tiering through:

• cluster-wide management 
• enhanced monitoring
• host profile integration 
• simplified deployment workflows 

When combined with enterprise-grade NVMe devices, thoughtful workload selection, and strong operational maturity, Memory Tiering becomes a powerful tool for building scalable, cost-efficient modern private cloud infrastructure. Architects should consider the hardware form factor carefully to maximise the value of VCF 9.1 deployments and features including memory tiering.

To explore how you can unlock the full potential of memory tiering and optimise your VCF 9.1 deployment, get in touch with the team at Zaleo Consulting. Our experts can help you design the right architecture, align workloads effectively, and maximise performance while controlling costs so you can build a future-ready private cloud with confidence.