NAME

pvecm - Proxmox VE Cluster Manager

SYNOPSYS

pvecm <COMMAND> [ARGS] [OPTIONS]

pvecm add <hostname> [OPTIONS]

Adds the current node to an existing cluster.

<hostname> string

Hostname (or IP) of an existing cluster member.

-force boolean

Do not throw error if node already exists.

-nodeid integer (1 - N)

Node id for this node.

-ring0_addr string

Hostname (or IP) of the corosync ring0 address of this node. Defaults to nodes hostname.

-ring1_addr string

Hostname (or IP) of the corosync ring1 address, this needs an valid configured ring 1 interface in the cluster.

-votes integer (0 - N)

Number of votes for this node

pvecm addnode <node> [OPTIONS]

Adds a node to the cluster configuration.

<node> string

The cluster node name.

-force boolean

Do not throw error if node already exists.

-nodeid integer (1 - N)

Node id for this node.

-ring0_addr string

Hostname (or IP) of the corosync ring0 address of this node. Defaults to nodes hostname.

-ring1_addr string

Hostname (or IP) of the corosync ring1 address, this needs an valid bindnet1_addr.

-votes integer (0 - N)

Number of votes for this node

pvecm create <clustername> [OPTIONS]

Generate new cluster configuration.

<clustername> string

The name of the cluster.

-bindnet0_addr string

This specifies the network address the corosync ring 0 executive should bind to and defaults to the local IP address of the node.

-bindnet1_addr string

This specifies the network address the corosync ring 1 executive should bind to and is optional.

-nodeid integer (1 - N)

Node id for this node.

-ring0_addr string

Hostname (or IP) of the corosync ring0 address of this node. Defaults to the hostname of the node.

-ring1_addr string

Hostname (or IP) of the corosync ring1 address, this needs an valid bindnet1_addr.

-rrp_mode (active | none | passive) (default=none)

This specifies the mode of redundant ring, which may be none, active or passive. Using multiple interfaces only allows active or passive.

-votes integer (1 - N)

Number of votes for this node.

pvecm delnode <node>

Removes a node to the cluster configuration.

<node> string

The cluster node name.

pvecm expected <expected>

Tells corosync a new value of expected votes.

<expected> integer (1 - N)

Expected votes

pvecm help [<cmd>] [OPTIONS]

Get help about specified command.

<cmd> string

Command name

-verbose boolean

Verbose output format.

pvecm keygen <filename>

Generate new cryptographic key for corosync.

<filename> string

Output file name

pvecm nodes

Displays the local view of the cluster nodes.

pvecm status

Displays the local view of the cluster status.

pvecm updatecerts [OPTIONS]

Update node certificates (and generate all needed files/directories).

-force boolean

Force generation of new SSL certifate.

-silent boolean

Ignore errors (i.e. when cluster has no quorum).

DESCRIPTION

The Proxmox VE cluster manager pvecm is a tool to create a group of physical servers. Such group is called a cluster. We use the Corosync Cluster Engine for reliable group communication, and such cluster can consists of up to 32 physical nodes (probably more, dependent on network latency).

pvecm can be used to create a new cluster, join nodes to a cluster, leave the cluster, get status information and do various other cluster related tasks. The Proxmox Cluster file system (pmxcfs) is used to transparently distribute the cluster configuration to all cluster nodes.

Grouping nodes into a cluster has the following advantages:

  • Centralized, web based management

  • Multi-master clusters: Each node can do all management task

  • Proxmox Cluster file system (pmxcfs): Database-driven file system for storing configuration files, replicated in real-time on all nodes using corosync.

  • Easy migration of Virtual Machines and Containers between physical hosts

  • Fast deployment

  • Cluster-wide services like firewall and HA

Requirements

  • All nodes must be in the same network as corosync uses IP Multicast to communicate between nodes (also see Corosync Cluster Engine). Corosync uses UDP ports 5404 and 5405 for cluster communication.

    Note Some switches do not support IP multicast by default and must be manually enabled first.
  • Date and time have to be synchronized.

  • SSH tunnel on TCP port 22 between nodes is used.

  • If you are interested in High Availability, you need to have at least three nodes for reliable quorum. All nodes should have the same version.

  • We recommend a dedicated NIC for the cluster traffic, especially if you use shared storage.

Note It is not possible to mix Proxmox VE 3.x and earlier with Proxmox VE 4.0 cluster nodes.

Preparing Nodes

First, install Proxmox VE on all nodes. Make sure that each node is installed with the final hostname and IP configuration. Changing the hostname and IP is not possible after cluster creation.

Currently the cluster creation has to be done on the console, so you need to login via ssh.

Create the Cluster

Login via ssh to the first Proxmox VE node. Use a unique name for your cluster. This name cannot be changed later.

hp1# pvecm create YOUR-CLUSTER-NAME
Caution The cluster name is used to compute the default multicast address. Please use unique cluster names if you run more than one cluster inside your network.

To check the state of your cluster use:

hp1# pvecm status

Adding Nodes to the Cluster

Login via ssh to the node you want to add.

hp2# pvecm add IP-ADDRESS-CLUSTER

For IP-ADDRESS-CLUSTER use the IP from an existing cluster node.

Caution A new node cannot hold any VM´s, because you would get conflicts about identical VM IDs. Also, all existing configuration in /etc/pve is overwritten when you join a new node to the cluster. To workaround, use vzdump to backup and restore to a different VMID after adding the node to the cluster.

To check the state of cluster:

# pvecm status
Cluster status after adding 4 nodes
hp2# pvecm status
Quorum information
~~~~~~~~~~~~~~~~~~
Date:             Mon Apr 20 12:30:13 2015
Quorum provider:  corosync_votequorum
Nodes:            4
Node ID:          0x00000001
Ring ID:          1928
Quorate:          Yes

Votequorum information
~~~~~~~~~~~~~~~~~~~~~~
Expected votes:   4
Highest expected: 4
Total votes:      4
Quorum:           2
Flags:            Quorate

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes Name
0x00000001          1 192.168.15.91
0x00000002          1 192.168.15.92 (local)
0x00000003          1 192.168.15.93
0x00000004          1 192.168.15.94

If you only want the list of all nodes use:

# pvecm nodes
List Nodes in a Cluster
hp2# pvecm nodes

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes Name
         1          1 hp1
         2          1 hp2 (local)
         3          1 hp3
         4          1 hp4

Remove a Cluster Node

Caution Read carefully the procedure before proceeding, as it could not be what you want or need.

Move all virtual machines from the node. Make sure you have no local data or backups you want to keep, or save them accordingly.

Log in to one remaining node via ssh. Issue a pvecm nodes command to identify the node ID:

hp1# pvecm status

Quorum information
~~~~~~~~~~~~~~~~~~
Date:             Mon Apr 20 12:30:13 2015
Quorum provider:  corosync_votequorum
Nodes:            4
Node ID:          0x00000001
Ring ID:          1928
Quorate:          Yes

Votequorum information
~~~~~~~~~~~~~~~~~~~~~~
Expected votes:   4
Highest expected: 4
Total votes:      4
Quorum:           2
Flags:            Quorate

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes Name
0x00000001          1 192.168.15.91 (local)
0x00000002          1 192.168.15.92
0x00000003          1 192.168.15.93
0x00000004          1 192.168.15.94
Important at this point you must power off the node to be removed and make sure that it will not power on again (in the network) as it is.
hp1# pvecm nodes

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes Name
         1          1 hp1 (local)
         2          1 hp2
         3          1 hp3
         4          1 hp4

Log in to one remaining node via ssh. Issue the delete command (here deleting node hp4):

hp1# pvecm delnode hp4

If the operation succeeds no output is returned, just check the node list again with pvecm nodes or pvecm status. You should see something like:

hp1# pvecm status

Quorum information
~~~~~~~~~~~~~~~~~~
Date:             Mon Apr 20 12:44:28 2015
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000001
Ring ID:          1992
Quorate:          Yes

Votequorum information
~~~~~~~~~~~~~~~~~~~~~~
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           3
Flags:            Quorate

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes Name
0x00000001          1 192.168.15.90 (local)
0x00000002          1 192.168.15.91
0x00000003          1 192.168.15.92
Important as said above, it is very important to power off the node before removal, and make sure that it will never power on again (in the existing cluster network) as it is.

If you power on the node as it is, your cluster will be screwed up and it could be difficult to restore a clean cluster state.

If, for whatever reason, you want that this server joins the same cluster again, you have to

  • reinstall pve on it from scratch

  • then join it, as explained in the previous section.

Quorum

Proxmox VE use a quorum-based technique to provide a consistent state among all cluster nodes.

A quorum is the minimum number of votes that a distributed transaction has to obtain in order to be allowed to perform an operation in a distributed system.

Quorum (distributed computing)
— from Wikipedia

In case of network partitioning, state changes requires that a majority of nodes are online. The cluster switches to read-only mode if it loose quorum.

Note Proxmox VE assigns a single vote to each node by default.

Cluster Cold Start

It is obvious that a cluster is not quorate when all nodes are offline. This is a common case after a power failure.

Note It is always a good idea to use an uninterruptible power supply (UPS, also called battery backup) to avoid this state. Especially if you want HA.

On node startup, service pve-manager is started and waits for quorum. Once quorate, it starts all guests which have the onboot flag set.

When you turn on nodes, or when power comes back after power failure, it is likely that some nodes boots faster than others. Please keep in mind that guest startup is delayed until you reach quorum.

Copyright © 2007-2016 Proxmox Server Solutions GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see http://www.gnu.org/licenses/