CHAPTER 22

Configuring Host Clusters for Live VM Migration and High Availability

14 sections · 21 min read · SimpleVM 10.1

A SimpleVM host cluster is two or more SimpleVM hosts prepared to work together as one virtualization environment. Clustering lets you move a running workload to another host for maintenance or balancing, and recover a workload on another host after a host fails. In SimpleVM 10.1, the confirmed cluster use cases are live VM migration and high availability through manual failover when the VM disks are on shared storage that every participating host can reach. This chapter prepares the hosts to trust one another, communicate over the correct network path, and allow the services that migration and failover depend on.

Where Chapter 21, VM Migration covers moving a single VM between hosts, this chapter builds the host-to-host foundation underneath it: consistent hosts, a predictable migration network, the cluster firewall zone, passwordless administrative access, and shared storage for high availability. Administrators arriving from VMware can treat a SimpleVM cluster as the equivalent of an HA cluster and live migration as the equivalent of vMotion; that concept map is in Chapter 26, Advanced Administration and Command-Line Reference (see Section 26.8, VMware-to-SimpleVM Concepts).

Understand clustering — what a cluster provides and how live migration differs from high availability (Sections 22.1 and 22.2).
Choose a configuration — two-node, three-or-more node, and shared- versus local-storage trade-offs (Section 22.3).
Confirm readiness — before-you-begin, system, networking, and shared-storage requirements (Sections 22.4 through 22.7).
Build the cluster — prepare a two-node cluster, then extend the same pattern to three or more nodes (Sections 22.8 and 22.9).
Operate the cluster — live migration within the cluster and high availability through manual failover (Sections 22.10 and 22.11).
Maintain and recover — service a host safely, and diagnose cluster problems without making them worse (Sections 22.12 and 22.13).

22.1 Overview of Host Clustering

Overview

A SimpleVM cluster is a group of two or more hosts prepared to support coordinated virtualization operations. Instead of treating each host as an isolated system, you prepare a group of hosts so that virtual machines can move between them or be restarted on another host when needed. For SimpleVM 10.1, the confirmed cluster use cases are live VM migration and high availability through manual failover when shared storage is available.

Capability	What it means	Typical use
Live migration	Moves a running VM from one host to another with little or no interruption.	Planned host maintenance or resource balancing.
High availability	Recovers a protected workload on another host after a host becomes unavailable.	Unexpected host failure (manual failover in 10.1).
Shared storage	Lets multiple hosts access the same VM disk images.	Required for high-availability workloads.
Cluster networking	Lets hosts communicate for administration and migration.	Required before operating cluster hosts.

22.2 Purpose of Host Clustering

Overview

Host clustering gives administrators a safer way to operate multiple SimpleVM hosts as one virtualization environment. Instead of managing each host in isolation, you prepare a group of hosts so virtual machines can move or be restarted on another host when needed.

Live VM migration — Move a running VM from one host to another with little or no interruption when the VM, storage, network, and destination host meet migration requirements.
Maintenance flexibility — Move workloads away from a host before applying updates, replacing hardware, or troubleshooting host-level problems.
High availability through manual failover — Restart a VM on another cluster host after a host failure when the VM disks are on shared storage the remaining hosts can reach.
Resource balancing — Place workloads on hosts that have enough CPU, memory, storage, and network capacity for the expected load.

22.3 Common Cluster Configurations

Overview

Plan a cluster around the number of hosts available, the criticality of the workloads, and whether shared storage is available. The best configuration depends on the failure tolerance and maintenance expectations of the environment.

Configuration	Typical use	Strengths	Limits to understand
Two-node cluster	Small environments, branch offices, or simple redundancy.	Lower hardware cost; supports moving VMs between two hosts.	Less resilient than three or more hosts; one host may have to carry the remaining workloads, so capacity planning matters.
Three-or-more-node cluster	Production environments that need maintenance flexibility and fault tolerance.	Better spare capacity and improved operational resilience.	Requires more planning for networking, shared storage, and consistent host configuration.
Cluster with shared storage	Environments that need manual failover or smoother migration.	VM disk images can be reached by multiple hosts, which makes recovery and migration cleaner.	Shared storage becomes a critical dependency that must be designed, monitored, and backed up.
Cluster with local storage	Basic migration testing or lower-cost environments.	Can support some migration workflows when properly configured.	High availability is limited because a failed host's VM disks may not be reachable from another host.

For most production deployments, use at least three hosts and shared storage when the goal is high availability. A two-host cluster can be useful, but it leaves less spare capacity during maintenance or failure.

22.4 Before You Begin

Overview

Cluster preparation touches firewall behavior, remote access, migration paths, and VM placement, so confirm the host, storage, network, and administrative requirements are already in place before production workloads depend on the cluster.

Before you begin

SimpleVM is installed and verified on each host. See Chapter 4, Installing SimpleVM.
Each host is reachable by a stable hostname or IP address. See Chapter 15, Managing Host Networking.
The SimpleVM management interface is available on each host. See Chapter 6, Accessing the SimpleVM Management Interface.
You understand where VM disk images live and which storage pools are in use. See Chapter 12, Managing Storage Pools, and Chapter 13, Managing Host Storage.
The current firewall configuration is documented before you change it. See Chapter 16, Firewall and Security Settings.
Current backups exist for important VMs and for host configuration. See Chapter 20, Backup and Recovery.
You have administrative access to each host. The accounts and privilege model are covered in Chapter 17, User Access and Administrative Tasks (see Section 17.3, Privileged Tasks).

22.5 System Requirements

Overview

All cluster hosts should be as consistent as possible. Live migration is most reliable when hosts run the same SimpleVM version, use compatible CPUs, share similar virtualization settings, and keep enough spare capacity for the VMs they may receive.

All hosts run the same SimpleVM release and update level.
Hardware virtualization is enabled in BIOS or UEFI on every host.
CPU families are compatible across hosts, especially for live migration.
Each destination host has enough unused CPU and memory to receive the VMs it may host.
Time synchronization is working on every host.
Administrative credentials and SSH access are available for cluster setup and troubleshooting.
VMs that may be moved or recovered are documented, including their CPU, memory, network, and storage requirements.

22.6 Networking Requirements

Overview

Cluster networking must be predictable and low-latency. During live migration the running VM stays available while its memory state is copied from the source host to the destination host. If the migration network is slow, congested, blocked by a firewall, or routed through an unreliable path, migration can fail or take longer than expected.

Each host can resolve and reach every other cluster host by the name or IP address used in migration commands.
Hosts have reliable, low-latency connectivity between one another.
The host firewall allows ssh and the migration port range between cluster hosts. The cluster firewall zone is configured in Section 22.8, Configuring a Two-Node Cluster.
The VM networks used by the workload already exist on the destination host. See Chapter 14, Managing Virtual Networks.
Production environments separate management, storage, and VM traffic where the network design allows it.
Time synchronization (NTP) is configured consistently across hosts.

Figure 22-1. The host Networking page, used to confirm interfaces and the host-to-host path each peer uses for migration. Host networking is covered in Chapter 15, Managing Host Networking. — **Figure 22-1.** The host **Networking** **page, used to confirm interfaces and the host-to-host path each peer uses for migration. Host networking is covered in** **Chapter 15, Managing Host Networking**.

Figure 22-2. The Firewall zones. The cluster zone in the reference build allows ssh (TCP 22) for administration and qemu-nbd (TCP 49152–49215) for migration data between peer hosts — confirmed against the live firewall screenshot. — **Figure 22-2.** The **Firewall** **zones. The cluster zone in the reference build allows** **ssh** **(TCP 22) for administration and** **qemu-nbd** *(TCP 49152–49215) for migration data between peer hosts — confirmed against the live firewall screenshot.*

22.7 Shared Storage Requirements for High Availability

Overview

Shared storage is the key difference between simple migration planning and high-availability planning. If a VM's disk images exist only on the failed host, another host cannot safely restart that VM unless the disks are recovered or restored. When VM disks are on shared storage that every cluster host can reach, another host can start the VM after the original host fails.

Use shared storage for VMs that require high availability or manual failover.
Confirm that each cluster host can access the same shared storage path or storage pool.
Use storage designed for virtualization workloads, not casual file sharing.
Monitor shared-storage capacity, latency, and health as production dependencies.
Test failover with a non-critical VM before relying on the workflow for production.

Figure 22-3. The Storage pools page, used to confirm where VM disks are stored and whether a pool is backed by shared storage reachable from every cluster host. See Chapter 12, Managing Storage Pools. — **Figure 22-3.** The **Storage pools** **page, used to confirm where VM disks are stored and whether a pool is backed by shared storage reachable from every cluster host. See** **Chapter 12, Managing Storage Pools**.

Storage scenario	Live migration	High availability
Local storage only	May be possible depending on configuration and method, but usually offline because the disk must move with the VM.	Not recommended — surviving hosts may not be able to reach the VM disks.
Shared storage	Preferred; all hosts reach the same disk images, so only the running state has to transfer.	Required so a surviving host can access and start the VM.
Mixed storage	Validate carefully before production use.	Only VMs on shared storage should be treated as HA-ready.

22.8 Configuring a Two-Node Cluster

Overview

Use this procedure to prepare two SimpleVM hosts to support live migration between each other. It creates the host-to-host access that migration depends on: SSH is enabled, a cluster firewall zone allows the peer host, the migration port range is permitted, and key-based SSH authentication is configured for the administrative account used during migration. Repeat the configuration on both hosts so each host trusts and allows the other.

Before you begin

Both hosts are installed, updated, and reachable over the management network.
You know the hostname or IP address of each host. The examples use <ip-of-host-a> and <ip-of-host-b>.
The administrative account used for migration exists on both hosts.
A current backup exists for any VM used in testing.
For HA testing, shared storage is already configured and visible from both hosts (see Section 22.7, Shared Storage Requirements for High Availability).

Procedure: Prepare Host A

Sign in to Host A using the web terminal or SSH. Run command-line tasks under your named account with sudo rather than in a root shell; see Chapter 17, User Access and Administrative Tasks (and Section 17.5, Command-Line Access).
Enable SSH so Host B can authenticate to Host A during migration operations.

Create the cluster firewall zone, allow the peer host as a source, allow ssh, define the qemu-nbd migration service, and allow it in the cluster zone.

Create an SSH key for the administrative account if one does not already exist.

Copy the key to Host B so the administrative account can connect without a password.

Repeat the same steps on Host B, using <ip-of-host-a> as the cluster-zone source and copying the SSH key back to Host A. When both hosts are done, each host trusts and allows the other.

Verification

From Host A, connect to Host B over SSH without a password; from Host B, connect to Host A. Each should return the peer's hostname without prompting.

Confirm the cluster zone lists the peer host as an allowed source and allows ssh and qemu-nbd.

Test migration with a non-critical VM before moving production workloads (see Section 22.10, Live Migration in a Cluster).

Revert this change

To remove the two-node cluster preparation, remove the peer host's source from the cluster zone and reload the firewall.

If the SSH trust is no longer needed, remove the corresponding public key from the peer account's authorized_keys file, taking care not to remove keys used for other administrative access.

22.9 Configuring a Three-or-More Node Cluster

Overview

Use this procedure to prepare three or more SimpleVM hosts to support migration and manual failover across a larger environment. The core steps match the two-node procedure, but every host must trust and allow every other host that may participate in migration or failover. A larger cluster gives more flexibility because workloads can move to more than one destination, but it also requires more discipline: host naming, firewall sources, shared-storage access, and capacity planning must stay consistent across every node.

Before you begin

All hosts meet the requirements in Section 22.5, System Requirements through Section 22.7, Shared Storage Requirements for High Availability.
Each host has a stable hostname or IP address.
A list of every cluster host IP address is available before firewall rules are changed.
Shared storage is configured and visible from every host if high availability is required.
A destination-capacity plan identifies which hosts can absorb workloads during maintenance or host failure.

Procedure: Prepare each host

On each host, enable SSH.

On each host, create the cluster zone and add every other cluster host as an allowed source.

On each host, allow ssh and the migration port range in the cluster zone.

On each host, create or reuse an SSH key for the administrative account.

Copy the key from each host to every other host that may be used as a migration target.

Verification

From each host, SSH to every other host by the hostname or IP address used for migration.
Confirm each host's cluster zone lists all peer host source addresses.
Confirm the same VM networks are available on migration source and destination hosts.
Confirm shared-storage paths or storage pools are visible from every host where high availability is required.
Run a test migration with a non-critical VM between at least two hosts before expanding the test to all hosts.

Revert this change

To remove a host from cluster participation, remove its source address from the cluster zone on the remaining hosts and remove its trusted SSH key where appropriate.

22.10 Live Migration in a Cluster

Overview

Once the hosts are prepared as cluster peers, live migration moves a running VM from one to another while it keeps operating. The migration transfers the VM's running state to the destination, then performs a final switchover; storage and network placement must already be compatible with the destination before the migration begins. This section covers performing the move within a prepared cluster. The full single-VM migration mechanics — readiness checklist, dialog, command line, and troubleshooting — are detailed in Chapter 21, VM Migration.

Figure 22-4. The running-VM row menu in the Virtual machines list, showing the Migrate action alongside the power actions. The Migrate action is documented in Chapter 8, Creating and Managing Virtual Machines (see Section 8.7, Starting, Stopping, Pausing, and Rebooting VMs). — **Figure 22-4.** The running-VM row menu in the **Virtual machines** **list, showing the** **Migrate** **action alongside the power actions. The** **Migrate** **action is documented in** **Chapter 8, Creating and Managing Virtual Machines** **(see** **Section 8.7, Starting, Stopping, Pausing, and Rebooting VMs**).

Before you begin

The source and destination hosts are prepared as cluster peers (Sections 22.8 or 22.9).
The destination host has enough available CPU and memory for the VM.
The VM network used on the source host also exists on the destination host.
The destination can reach the VM disk images — shared storage is preferred — or the move is planned as offline. See Section 21.3, Local Storage Migration Considerations, and Section 21.4, Shared Storage Migration Considerations.
The VM is not performing a sensitive operation that should not be interrupted, such as a major guest OS upgrade or database maintenance task.
A recent backup exists for important workloads.

Procedure: Migrate a VM from the management interface

Sign in to the SimpleVM management interface on the source host.
Open Virtual machines and locate the VM you want to migrate.
Open the VM row action menu and select Migrate.
Select the destination SimpleVM host.
Review the migration summary, including destination host and storage behavior.
Select Migrate to begin, and watch the VM state until the migration completes. Do not restart either host during the migration.

Procedure: Migrate a VM from the command line

Use the command line when the management-interface workflow is unavailable or when support asks you to validate migration directly. The example uses an SSH-based libvirt connection to the destination host.

Use --undefinesource only when the move is permanent and the source host should remove its VM definition after the migration succeeds. Omit it for an intentional temporary migration, or when support directs you to preserve the source definition.

Verification

The VM appears on the destination host and remains running after the switchover.
The VM console opens from the destination host and the guest operating system is responsive.
Guest network connectivity and application services continue to work.
For a permanent move, the VM no longer appears as running on the source host.

Revert this change

A completed live migration is normally reverted by migrating the VM back to the original host. First confirm the original host still has access to the same storage and network resources and that the reason for the first migration has been resolved. A migration changes which host runs the VM; it does not undo guest-level changes.

22.11 High Availability and Manual Failover

Overview

High-availability planning focuses on what happens when a host is no longer available. In SimpleVM 10.1, operate high availability as manual failover unless the product confirms an automated failover workflow. Manual failover means an administrator starts or recovers the affected VM on another cluster host after confirming that the VM storage is available and that the failed host is not still running the same VM.

Manual failover requirements

The VM disk images are on shared storage reachable from the recovery host.
The recovery host has enough CPU and memory for the VM.
The required VM network exists on the recovery host.
The failed host is confirmed offline or fenced so it cannot continue running the VM.
The VM configuration is available on the recovery host or can be imported or registered.

Manual failover workflow

Confirm the original host is unavailable and will not resume running the affected VM unexpectedly.
Identify the most appropriate recovery host based on available CPU, memory, storage, and network access.
Confirm the shared storage containing the VM disk images is mounted or available on the recovery host.
Confirm the VM network is available on the recovery host.
Register, import, or start the VM on the recovery host according to the SimpleVM recovery workflow.
Open the VM console and verify the guest operating system starts.
Verify application and network access from a client system.
Document the failover event and the host where the VM is now running.

Revert this change

After the failed host is repaired, do not simply start the VM on it. First confirm where the VM is currently running, confirm storage consistency, then use a planned migration or a shutdown-and-start workflow to move it back. If there is any uncertainty about storage consistency, treat the situation as a recovery event and involve support before starting the VM on another host. See Chapter 20, Backup and Recovery.

22.12 Cluster Maintenance

Overview

Plan cluster maintenance so workloads stay protected while hosts are updated, rebooted, serviced, or removed. The safe pattern is to move or shut down workloads first, confirm that the remaining hosts have enough capacity, then perform maintenance on the empty or non-critical host.

Procedure: Prepare a host for maintenance

Review the VMs currently running on the host.
Confirm destination hosts have enough available CPU and memory.
Confirm shared storage and VM networks are healthy.
Migrate running VMs to other hosts (see Section 22.10, Live Migration in a Cluster), or shut down non-critical VMs that do not need to remain online.
Verify that no production VM is still running on the host being serviced.
Apply updates, reboot, replace hardware, or complete the required maintenance. Software updates are covered in Chapter 18, Software Updates.
After maintenance, verify host health from the SimpleVM management interface.
Move workloads back only if the host is stable and capacity planning supports it.

Verification

The serviced host is reachable through the SimpleVM management interface.
The host reports expected CPU, memory, storage, and network status.
The cluster firewall zone and its allowed services are still present.
SSH trust still works between the serviced host and its peers.
A non-critical test migration succeeds before production workloads are moved back.

22.13 Cluster Troubleshooting

Overview

Cluster problems usually come from one of four areas: host reachability, firewall rules, storage access, or VM compatibility. Start with the simplest dependency and move upward to migration-specific details.

Symptom	Likely cause	What to check first
Hosts cannot reach each other	DNS issue, wrong IP address, firewall block, or sshd not running.	Ping or SSH by the same name/IP used in migration; verify sshd and the cluster zone sources.
SSH asks for a password during migration	Key not copied, wrong user or host key, or home-directory permissions.	Re-run ssh-copy-id for the intended admin user and test SSH manually.
The Migrate action is missing or fails immediately	Graphical workflow unavailable, VM state not eligible, or host not configured as a peer.	Verify cluster preparation, VM state, and final UI behavior.
Migration starts but fails before completing	Insufficient destination memory, network interruption, incompatible CPU feature, or storage access issue.	Check destination capacity, migration network, CPU compatibility, and storage path access.
VM starts on destination but has no network	Destination host missing the VM network, bridge, VLAN, or firewall configuration.	Compare VM network configuration on source and destination hosts.
Manual failover cannot start the VM	Shared storage unavailable, VM definition missing, or original host may still own the VM.	Confirm shared storage, VM-definition availability, and failed-host state before starting the VM.
Cluster behavior is unstable	Time-synchronization mismatch or quorum instability.	Confirm time synchronization, node count, and peer communication health.

Useful command-line checks

Collecting information for support

When opening a support request, collect the information that shows host state, VM state, and the exact failure point.

SimpleVM version and update level on every host.
Source and destination hostnames or IP addresses.
VM name, guest OS, CPU count, memory size, disk location, and network attachment.
The time the migration or failover attempt started.
Screenshots of the VM action menu, the firewall zone, and any error message.
Output from the firewall, SSH, and virsh checks above.
Whether the VM uses local or shared storage.

Chapter review

After completing this chapter, an administrator should be able to explain what a SimpleVM cluster provides and how live migration differs from high availability; choose a cluster configuration based on host count and storage model; confirm system, networking, and shared-storage readiness; prepare a two-node cluster and extend the same pattern to three or more nodes; perform live migration within a prepared cluster from the management interface or the command line; recover a workload through manual failover without ever running it on two hosts at once; and maintain and troubleshoot the cluster.1 provides automated failover beyond manual failover; whether a graphical cluster-create or host-join workflow exists; the officially supported shared-storage types; whether two-node high availability is supported for production; the supported administrative-user model and SimpleVM version policy for SSH-based migration; storage path and pool consistency requirements across hosts; how a failed host is fenced before failover; and the final qemu-nbd service-definition name.