mirror of
git://git.proxmox.com/git/pve-docs.git
synced 2025-02-01 05:47:08 +03:00
187 lines
5.1 KiB
Plaintext
187 lines
5.1 KiB
Plaintext
[[chapter_pveceph]]
|
|
ifdef::manvolnum[]
|
|
pveceph(1)
|
|
==========
|
|
:pve-toplevel:
|
|
|
|
NAME
|
|
----
|
|
|
|
pveceph - Manage Ceph Services on Proxmox VE Nodes
|
|
|
|
SYNOPSIS
|
|
--------
|
|
|
|
include::pveceph.1-synopsis.adoc[]
|
|
|
|
DESCRIPTION
|
|
-----------
|
|
endif::manvolnum[]
|
|
ifndef::manvolnum[]
|
|
pveceph - Manage Ceph Services on Proxmox VE Nodes
|
|
==================================================
|
|
endif::manvolnum[]
|
|
|
|
{pve} unifies your compute and storage systems, i.e. you can use the
|
|
same physical nodes within a cluster for both computing (processing
|
|
VMs and containers) and replicated storage. The traditional silos of
|
|
compute and storage resources can be wrapped up into a single
|
|
hyper-converged appliance. Separate storage networks (SANs) and
|
|
connections via network (NAS) disappear. With the integration of Ceph,
|
|
an open source software-defined storage platform, {pve} has the
|
|
ability to run and manage Ceph storage directly on the hypervisor
|
|
nodes.
|
|
|
|
Ceph is a distributed object store and file system designed to provide
|
|
excellent performance, reliability and scalability. For smaller
|
|
deployments, it is possible to install a Ceph server for RADOS Block
|
|
Devices (RBD) directly on your {pve} cluster nodes, see
|
|
xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]. Recent
|
|
hardware has plenty of CPU power and RAM, so running storage services
|
|
and VMs on the same node is possible.
|
|
|
|
To simplify management, we provide 'pveceph' - a tool to install and
|
|
manage {ceph} services on {pve} nodes.
|
|
|
|
|
|
Precondition
|
|
------------
|
|
|
|
To build a Proxmox Ceph Cluster there should be at least three (preferably)
|
|
identical servers for the setup.
|
|
|
|
A 10Gb network, exclusively used for Ceph, is recommmended. A meshed
|
|
network setup is also an option if there are no 10Gb switches
|
|
available, see {webwiki-url}Full_Mesh_Network_for_Ceph_Server[wiki] .
|
|
|
|
Check also the recommendations from
|
|
http://docs.ceph.com/docs/master/start/hardware-recommendations/[Ceph's website].
|
|
|
|
|
|
Installation of Ceph Packages
|
|
-----------------------------
|
|
|
|
On each node run the installation script as follows:
|
|
|
|
[source,bash]
|
|
----
|
|
pveceph install
|
|
----
|
|
|
|
This sets up an `apt` package repository in
|
|
`/etc/apt/sources.list.d/ceph.list` and installs the required software.
|
|
|
|
|
|
Creating initial Ceph configuration
|
|
-----------------------------------
|
|
|
|
After installation of packages, you need to create an initial Ceph
|
|
configuration on just one node, based on your network (`10.10.10.0/24`
|
|
in the following example) dedicated for Ceph:
|
|
|
|
[source,bash]
|
|
----
|
|
pveceph init --network 10.10.10.0/24
|
|
----
|
|
|
|
This creates an initial config at `/etc/pve/ceph.conf`. That file is
|
|
automatically distributed to all {pve} nodes by using
|
|
xref:chapter_pmxcfs[pmxcfs]. The command also creates a symbolic link
|
|
from `/etc/ceph/ceph.conf` pointing to that file. So you can simply run
|
|
Ceph commands without the need to specify a configuration file.
|
|
|
|
|
|
Creating Ceph Monitors
|
|
----------------------
|
|
|
|
On each node where a monitor is requested (three monitors are recommended)
|
|
create it by using the "Ceph" item in the GUI or run.
|
|
|
|
|
|
[source,bash]
|
|
----
|
|
pveceph createmon
|
|
----
|
|
|
|
|
|
Creating Ceph OSDs
|
|
------------------
|
|
|
|
via GUI or via CLI as follows:
|
|
|
|
[source,bash]
|
|
----
|
|
pveceph createosd /dev/sd[X]
|
|
----
|
|
|
|
If you want to use a dedicated SSD journal disk:
|
|
|
|
NOTE: In order to use a dedicated journal disk (SSD), the disk needs
|
|
to have a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT]
|
|
partition table. You can create this with `gdisk /dev/sd(x)`. If there
|
|
is no GPT, you cannot select the disk as journal. Currently the
|
|
journal size is fixed to 5 GB.
|
|
|
|
[source,bash]
|
|
----
|
|
pveceph createosd /dev/sd[X] -journal_dev /dev/sd[X]
|
|
----
|
|
|
|
Example: Use /dev/sdf as data disk (4TB) and /dev/sdb is the dedicated SSD
|
|
journal disk.
|
|
|
|
[source,bash]
|
|
----
|
|
pveceph createosd /dev/sdf -journal_dev /dev/sdb
|
|
----
|
|
|
|
This partitions the disk (data and journal partition), creates
|
|
filesystems and starts the OSD, afterwards it is running and fully
|
|
functional. Please create at least 12 OSDs, distributed among your
|
|
nodes (4 OSDs on each node).
|
|
|
|
It should be noted that this command refuses to initialize disk when
|
|
it detects existing data. So if you want to overwrite a disk you
|
|
should remove existing data first. You can do that using:
|
|
|
|
[source,bash]
|
|
----
|
|
ceph-disk zap /dev/sd[X]
|
|
----
|
|
|
|
You can create OSDs containing both journal and data partitions or you
|
|
can place the journal on a dedicated SSD. Using a SSD journal disk is
|
|
highly recommended if you expect good performance.
|
|
|
|
|
|
Ceph Pools
|
|
----------
|
|
|
|
The standard installation creates per default the pool 'rbd',
|
|
additional pools can be created via GUI.
|
|
|
|
|
|
Ceph Client
|
|
-----------
|
|
|
|
You can then configure {pve} to use such pools to store VM or
|
|
Container images. Simply use the GUI too add a new `RBD` storage (see
|
|
section xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
|
|
|
|
You also need to copy the keyring to a predefined location.
|
|
|
|
NOTE: The file name needs to be `<storage_id> + `.keyring` - `<storage_id>` is
|
|
the expression after 'rbd:' in `/etc/pve/storage.cfg` which is
|
|
`my-ceph-storage` in the following example:
|
|
|
|
[source,bash]
|
|
----
|
|
mkdir /etc/pve/priv/ceph
|
|
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring
|
|
----
|
|
|
|
|
|
ifdef::manvolnum[]
|
|
include::pve-copyright.adoc[]
|
|
endif::manvolnum[]
|