The Crush Map

The Crush map, which is the second part of this system, contains information about where each placement group is found in the cluster – that is, on which OSD (Figure 2). Replication is always executed at the PG level: All objects of a placement group are replicated between different OSDs in the RADOS cluster.

Figure 2: The elements of the RADOS universe, whose interaction is controlled by the Crush map.

The Crush map got its name from the algorithm it uses: Controlled Replication under Scalable Hashing [3]. The algorithm was developed by Weil specifically for such tasks in RADOS. Weil highlights one feature of Crush in particular: In contrast to hash algorithms, Crush remains stable when many storage devices leave or join the Cluster simultaneously. The rebalancing that other storage solutions require creates a lot of traffic with correspondingly long waiting periods. Crush-based clusters, on the other hand, transfer just enough data between storage nodes to achieve a balance.

Manipulating Data Placement

Of course, the admin also has a word to say about what data lands where. Practically all parameters that pertain to replication in RADOS can be configured by the admin, including, for example, how often an object should exist within an RADOS cluster (i.e., how many replicas of it should be made). Of course, the admin is also free to manipulate the allocation of the replicas to the OSDs. Thus, RADOS can take into account where specific racks are. Rules for replication specified by the admin control distribution of the replicas from placement groups to different OSD groups. A group can, for example, include all servers in the same data center room, another group in another room, and so on.

Administrators can define how the replication of data in RADOS is done by manipulating the corresponding Crush rules. The basic principle is the neither the allocation of the placement groups nor the results of the Crush calculations can be influenced directly. Instead, replication rules are set for each pool; subsequently, distribution of the placement groups and their positioning by means of the Crush map are done by RADOS.

By default, RADOS includes a rule that two replicas of each object must exist per pool. The number of replicas is always set for each pool; a typical example would be to raise this number to three, as is done here:

ceph osd pool set data size 3

To make the same change for the test pool, data would be replaced by test. Whether the cluster subsequently actually does what the admin expects can be investigated with ceph -v.

The use ceph in this kind of operation is certainly comfortable, but it does not provide the full range of functions. In the example, the pool continues to use the default Crush map, which does not consider properties such as the rack where the server is. If you want to distribute replicas according to such properties, you have to create your own Crush map.

A Crush Map of Your Own

With your own Crush map, you as administrator can have the objects in your pools distributed any way you want. crushtool is a valuable utility for this purpose because it creates corresponding templates. The following example creates a Crush map for a setup consisting of six OSDs (i.e., individual storage devices in servers) distributed over three racks:

crushtool --num_osds 6 -o crush.example.map --build host straw 1 rack straw 2 root straw 0

The --num_osds 6 parameter specifies that the cluster has six individual storage devices at its disposal. The --build option introduces a statement with three three-part parameters that follow the <Name> <Internal Crush Algorithm> <Number> syntax.

Name can be chosen freely; however, it is wise to choose something meaningful. host straw 1 specifies that one replica is allowed per host, and rack straw 2 tells RADOS that two servers exist per rack. root straw 0 refers to the number of racks and determines that the replicas should be distributed equally on all available racks.

Subsequently, if you want to see the resulting Crush map in plain text, you can enter

crushtool -d crush.example.map -o crush.example

The file with the map in plain text will then be called crush.example (Listing 1).

001 # begin crush map
002
003 # devices
004 device 0 device0
005 device 1 device1
006 device 2 device2
007 device 3 device3
008 device 4 device4
009 device 5 device5
010
011 # types
012 type 0 device
013 type 1 host
014 type 2 rack
015 type 3 root
016
017 # buckets
018 host host0 {
019         id -1            # do not change unnecessarily
020         # weight 1.000
021         alg straw
022         hash 0  # rjenkins1
023         item device0 weight 1.000
024 }
025 host host1 {
026         id -2            # do not change unnecessarily
027         # weight 1.000
028         alg straw
029         hash 0  # rjenkins1
030         item device1 weight 1.000
031 }
032 host host2 {
033         id -3            # do not change unnecessarily
034         # weight 1.000
035         alg straw
036         hash 0  # rjenkins1
037         item device2 weight 1.000
038 }
039 host host3 {
040         id -4            # do not change unnecessarily
041         # weight 1.000
042         alg straw
043         hash 0  # rjenkins1
044         item device3 weight 1.000
045 }
046 host host4 {
047         id -5            # do not change unnecessarily
048         # weight 1.000
049         alg straw
050         hash 0  # rjenkins1
051         item device4 weight 1.000
052 }
053 host host5 {
054         id -6            # do not change unnecessarily
055         # weight 1.000
056         alg straw
057         hash 0  # rjenkins1
058         item device5 weight 1.000
059 }
060 rack rack0 {
061         id -7            # do not change unnecessarily
062         # weight 2.000
063         alg straw
064         hash 0  # rjenkins1
065         item host0 weight 1.000
066         item host1 weight 1.000
067 }
068 rack rack1 {
069         id -8            # do not change unnecessarily
070         # weight 2.000
071         alg straw
072         hash 0  # rjenkins1
073         item host2 weight 1.000
074         item host3 weight 1.000
075 }
076 rack rack2 {
077         id -9            # do not change unnecessarily
078         # weight 2.000
079         alg straw
080         hash 0  # rjenkins1
081         item host4 weight 1.000
082         item host5 weight 1.000
083 }
084 root root {
085         id -10           # do not change unnecessarily
086         # weight 6.000
087         alg straw
088         hash 0  # rjenkins1
089         item rack0 weight 2.000
090         item rack1 weight 2.000
091         item rack2 weight 2.000
092 }
093
094 # rules
095 rule data {
096         ruleset 1
097         type replicated
098         min_size 2
099         max_size 2
100         step take root
101         step chooseleaf firstn 0 type rack
102         step emit
103 }
104
105 # end crush map

Of course, the names of the devices and hosts (device0, device 2, … and host1, host 2, …) must be adapted to the local conditions. Thus, the name of the device should correspond to the device name in ceph.conf (in this example: osd.0), and the hostname should agree with the hostname of the server.