HoguInside & ㅁ ㅇ ㄷ nail

[ ERROR 장애발생 ]

[root@loccalhost ~ ]# ceph-deploy osd activate osd-2:sda1
.
.
[osd-0][WARNIN] ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', u'1', '--monmap', '/var/lib/ceph/tmp/mnt.l7bamu/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.l7bamu', '--osd-journal', '/var/lib/ceph/tmp/mnt.l7bamu/journal', '--osd-uuid', u'a53f54e6-6dcb-4ed1-bfcf-3500e5ac5968', '--keyring', '/var/lib/ceph/tmp/mnt.l7bamu/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : 2018-04-18 16:13:17.313958 7f65fcbe1800 -1 filestore(/var/lib/ceph/tmp/mnt.l7bamu) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.l7bamu/journal: (13) Permission denied
[osd-0][WARNIN] 2018-04-18 16:13:17.313972 7f65fcbe1800 -1 OSD::mkfs: ObjectStore::mkfs failed with error -13
[osd-0][WARNIN] 2018-04-18 16:13:17.314012 7f65fcbe1800 -1  ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.l7bamu: (13) Permission denied
[osd-0][WARNIN] 
[osd-0][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/sda1

[ device 장치 확인 및 조치 ]


[ 정상 ]
[root@osd-1 ~]# ll /dev/sda*
brw-rw---- 1 root disk 8, 0  4월 18 16:41 /dev/sda
brw-rw---- 1 ceph ceph 8, 1  4월 18 16:41 /dev/sda1
brw-rw---- 1 ceph ceph 8, 2  4월 18 16:41 /dev/sda2

[ 장애 ]
[root@osd-2 ~]# ll /dev/sda*
brw-rw---- 1 root disk 8, 0  4월 18 16:06 /dev/sda
brw-rw---- 1 root disk 8, 1  4월 18 16:06 /dev/sda1
brw-rw---- 1 root disk 8, 2  4월 18 16:06 /dev/sda2


[ 해결 ]
[root@osd-2 ~]# chown ceph:ceph /dev/sda1
[root@osd-2 ~]# chown ceph:ceph /dev/sda2

또는

[root@osd-2 ~]# reboot
* 리부팅 후 해당 디바이스가 ceph:ceph 로 변경 됩니다.

[ ERROR. 장애발생 ]

[root@localhost ~]# ceph-deploy disk zap osd-0:sda 
.
.
[osd-0][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk', '--zap-all', '--', '/dev/sda']' returned non-zero exit status 2
[osd-0][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk zap /dev/sda

[ DISK 초기화 ]

[root@localhost ~ ]# dd if=/dev/zero of=/dev/sda bs=512 count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00852427 s, 60.1 kB/s

[root@localhost ~ ]# reboot

* disk 초기화 후 리부팅 하는 이유는 초기화 후 바로 "ceph-deploy disk zap" 를 진행하게 되면 정상적으로 작동은 하겠지만 종료가 되지 않는 버그(?)로 인해 리부팅 후 작업 진행을 추천합니다.

[ 정상 확인 ]

[root@localhost ~ ]# ceph-deploy disk zap osd-0:sda
.
.
.
[osd-0][DEBUG ] Creating new GPT entries.
[osd-0][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or
[osd-0][DEBUG ] other utilities.
[osd-0][DEBUG ] Creating new GPT entries.
[osd-0][DEBUG ] The operation has completed successfully.

[ CEPH 설치 - MON 서버 구성 ]

* 모든 작업은 mgmt 서버에서 진행 됩니다.
* ceph-deploy 는 ceph설치 및 운영시 사용되는 관리 패키지 입니다. ceph-deploy를 통해 설정 변경 시 일괄적으로 배포하고, osd추가 등을 편리하게 작업하게 해줍니다.
* yum update 해주세요. ceph-deploy 설치시 낮은 버젼으로 설치 되면 ceph 사이트 못찾아서  install 부터 장애가 발생됩니다.

[root@mgmt ~]# yum install ceph-deploy -y

* mon-0,mon-1 host 가 메인 모니터이고 mgmt는 개인적으로 서브로 넣어 두었습니다. mgmt를 포함하지 않으셔도 됩니다.

[root@mgmt ~]# ceph-deploy new mon-0 mon-1 mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.25): /usr/bin/ceph-deploy new mon-0 mon-1 mgmt
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[mon-0][DEBUG ] connected to host: mgmt 
[mon-0][INFO  ] Running command: ssh -CT -o BatchMode=yes mon-0
[mon-0][DEBUG ] connected to host: mon-0 
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
..
[mon-1][DEBUG ] connected to host: mgmt 
[mon-1][INFO  ] Running command: ssh -CT -o BatchMode=yes mon-1
[mon-1][DEBUG ] connected to host: mon-1 
...
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[mgmt][DEBUG ] connected to host: mgmt 
...
[ceph_deploy.new][DEBUG ] Monitor initial members are ['mon-0', 'mon-1', 'mgmt']
...
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...


* 위 명령어가 진행 되면 아래와 같이 ceph 관련 파일들이 생기게 됩니다.

[root@mgmt ~]# ll
-rw-r--r--   1 root root  1033  4월 18 11:38 .cephdeploy.conf
-rw-r--r--   1 root root   263  4월 18 11:38 ceph.conf
-rw-r--r--   1 root root  2395  4월 18 11:38 ceph.log
-rw-------   1 root root    73  4월 18 11:38 ceph.mon.keyring

* ceph.conf 설정

vi ceph.conf 

[global]
fsid = 0651cfa0-8ce4-4fbd-b35b-966e3d262851
mon_initial_members = mon-0, mon-1, mgmt
mon_host = 192.168.1.12,192.168.1.13,192.168.1.11
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true

# 여기서 부터 추가 설정 부분입니다. 
# osd_pool_default_size,rbd_default_features 기본으로 두 옵션만 넣고, 나머지 옵션들은 상황에 맞게 추가 합니다.
# ceph.conf 내용은 http://docs.ceph.com/docs/kraken/rados/configuration/ceph-conf/  참조하세요.

osd_pool_default_size = 2
rbd_default_features = 1
osd_journal_size = 10240
osd_pool_default_pg_num = 128
osd_pool_default_pgp_num = 128


[mon]
mon_host = mon-0, mon-1, mgmt
mon_addr = 192.168.1.12,192.168.1.13,192.168.1.11
mon_clock_drift_allowed = .3
mon_clock_drift_warn_backoff = 30    # Tell the monitor to backoff from this warning for 30 seconds
mon_osd_full_ratio = .90
mon_osd_nearfull_ratio = .85
mon_osd_report_timeout = 300
debug_ms = 1
debug_mon = 20
debug_paxos = 20
debug_auth = 20


[mon.0]
host = mon-0
mon_addr = 192.168.1.12:6789

[mon.1]
host = mon-1
mon_addr = 192.168.1.13:6789

[mon.2]
host = mgmt
mon_addr = 192.168.1.11:6789

[mds]
mds_standby_replay = true
mds_cache_size = 250000
debug_ms = 1
debug_mds = 20
debug_journaler = 20

[mds.0]
host = mgmt

[osd.0]
host = osd-0

[osd.1]
host = osd-1

[osd.2]
host = osd-2

[osd.3]
host = osd-3



* 각서버에 ceph관련 패키지를 설치 합니다. 아래와 같이 정상적으로 설치가 되며 그렇지 않을경우에는 방화벽, 네트워크 통신, resolv.conf, yum update(ceph-deploy 버전) 등을 확인해봐야 합니다.
* 설치는 host명으로 일괄 설치 합니다.
** 추가 사항 **
ubuntu ceph ver.10.x 에서는 아래와 같이 install 진행시 10.x 버전으로 설치가 되지만 centos는 가끔 상위 버젼으로 올라감..;; mgmt에서 ceph yumrepo를 jewel로 설정하고도 버전이 올라가면 다운 경로를 고정으로 잡고 진행.
ex)
ceph-deploy install --repo-url 'https://download.ceph.com/rpm-jewel/el7' --gpg-url 'https://download.ceph.com/keys/release.asc' mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3

[root@mgmt ~]# ceph-deploy install mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.25): /usr/bin/ceph-deploy install mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.install][DEBUG ] Installing stable version hammer on cluster ceph hosts mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.install][DEBUG ] Detecting platform for host mgmt ...
.
.
.

[osd-3][DEBUG ] 
[osd-3][DEBUG ] Complete!
[osd-3][INFO  ] Running command: ceph --version
[osd-3][DEBUG ] ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)


* ceph-mon 데몬 초기화 및 구동을 진행 합니다.

[root@mgmt ~]# ceph-deploy mon create-initial
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create-initial
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : 
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : 
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  keyrings                      : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts mon-0 mon-1 mgmt
[ceph_deploy.mon][DEBUG ] detecting platform for host mon-0 ...
.
.

* ceph-mon 데몬을 systemd에 등록 후 자동 실행으로 변경합니다.

[root@mgmt ~]# ceph-deploy mon create mon-0 mon-1 mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mon create mon-0 mon-1 mgmt
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : 
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  mon                           : ['mon-0', 'mon-1', 'mgmt']
[ceph_deploy.cli][INFO  ]  func                          : 
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  keyrings                      : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts mon-0 mon-1 mgmt
[ceph_deploy.mon][DEBUG ] detecting platform for host mon-0 ...

.
.
.

* ceph.conf 생성시 등록된 mon 서버들의 데몬을  초기화하고 구동 시킵니다. 각서버에서 " netstat -nlpt " 확인시 데몬 구동을 확인 합니다.

[root@mgmt ~]# netstat -nlpt | grep ceph
Active Internet connections (only servers)
Proto  Recv-Q Send-Q Local Address     Foreign Address    State       PID/Program name    
tcp         0      0 192.168.1.11:6789  0.0.0.0:*          LISTEN      14772/ceph-mon      

* mon-initial 후 키파일들이 생성 됩니다. mon 서버들이 ceph 서버에 정상적 접근하도록 합니다.
[root@mgmt ~]# ll
-rw-------   1 root root     71  4월 18 14:24 ceph.bootstrap-mds.keyring
-rw-------   1 root root     71  4월 18 14:24 ceph.bootstrap-mgr.keyring
-rw-------   1 root root    113  4월 18 14:24 ceph.bootstrap-osd.keyring
-rw-------   1 root root    113  4월 18 14:24 ceph.bootstrap-rgw.keyring
-rw-------   1 root root    129  4월 18 14:24 ceph.client.admin.keyring

* 생성된 키파일을 참조하게 합니다.
[root@mgmt ~]# ceph-deploy gatherkeys mon-0 mon-1 mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy gatherkeys mon-0 mon-1 mgmt
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x24e8cb0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  mon                           : ['mon-0', 'mon-1', 'mgmt']
[ceph_deploy.cli][INFO  ]  func                          : <function gatherkeys at 0x24cd500>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.gatherkeys][INFO  ] Storing keys in temp directory /tmp/tmpS0dvzG
[mon-0][DEBUG ] connected to host: mon-0 
[mon-0][DEBUG ] detect platform information from remote host
[mon-0][DEBUG ] detect machine type
[mon-0][DEBUG ] get remote short hostname
[mon-0][DEBUG ] fetch remote file
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.mon-0.asok mon_status
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.admin
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-mds
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-mgr
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-osd
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.client.admin.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-mds.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-mgr.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-osd.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-rgw.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] Destroy temp directory /tmp/tmpS0dvzG


* mon 서버 구성은 끝났습니다. 이제 osd 구성에 들어 가기 전에 상태값을 먼저 확인해 봅니다.
[root@mgmt ~]# ceph -s
    cluster 0651cfa0-8ce4-4fbd-b35b-966e3d262851
     health HEALTH_ERR
            clock skew detected on mon.mon-0, mon.mon-1
            64 pgs are stuck inactive for more than 300 seconds
            64 pgs stuck inactive
            64 pgs stuck unclean
            no osds
            Monitor clock skew detected 
     monmap e1: 3 mons at {mgmt=192.168.1.11:6789/0,mon-0=192.168.1.12:6789/0,mon-1=192.168.1.13:6789/0}
            election epoch 4, quorum 0,1,2 mgmt,mon-0,mon-1
     osdmap e1: 0 osds: 0 up, 0 in
            flags sortbitwise,require_jewel_osds
      pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
            0 kB used, 0 kB / 0 kB avail
                  64 creating

* 현재 health 상태는 ERR 입니다. 이제 OSD를 구성해서 ok 로  변경 합시다.

[ CEPH 설치 - OSD 서버 구성 ]

* osd 서버의 디스크를 확인 하면 아래와 같습니다. 
  ceph를 사용할 디스크는 sda 16TB RAID 디스크이고, osd 서버 4대를 ceph로 구성합니다.

[root@osd-0 ~]# parted -l
Error: /dev/sda: unrecognised disk label
Model: LSI MR9261-8i (scsi)                                               
Disk /dev/sda: 16.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: unknown
Disk Flags: 

Model: ATA P3-128 (scsi)
Disk /dev/sdb: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  128GB  128GB  primary  xfs          boot

* ceph-deploy disk zap 명령어를 통해 해당 osd서버들의 디스크의 Partition Table 을 GPT로 생성합니다.
* 주의할 점은 LSI MR9261-8i 해당 레이드 카드 사용시, 고질적인 버그로 인해 pci 우선순위가 마음대로 바뀌며, 리부팅시 해당 디바이스 장치가 안보일 때가 있습니다.
  아래 명령어를 하기전에 각 서버의 디스크 장치를 확인해 주세요. 그렇지 않으면 OS디스크를 초기화 시켜 재설치 해야 합니다..... ㅠㅠ

[root@mgmt ~]# ceph-deploy disk zap osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy disk zap osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : zap
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1649368>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function disk at 0x163e398>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : [('osd-0', '/dev/sda', None), ('osd-1', '/dev/sda', None), ('osd-2', '/dev/sda', None), ('osd-3', '/dev/sda', None)]
[ceph_deploy.osd][DEBUG ] zapping /dev/sda on osd-0
.
.
.
[osd-3][DEBUG ] Creating new GPT entries.
[osd-3][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or
[osd-3][DEBUG ] other utilities.
[osd-3][DEBUG ] Creating new GPT entries.
[osd-3][DEBUG ] The operation has completed successfully.

* osd-0에서 확인해 보니 gpt로 잘 초기화 되어 있습니다.

[root@osd-0 ~]# parted -l
Model: LSI MR9261-8i (scsi)
Disk /dev/sda: 16.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start  End  Size  File system  Name  Flags


* 다음 명령어들로 디스크 포맷 및 ceph-osd 데몬을 구동 진행 합니다.

[root@mgmt ~]# ceph-deploy osd prepare osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy osd prepare osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  block_db                      : None
[ceph_deploy.cli][INFO  ]  disk                          : [('osd-0', '/dev/sda', None), ('osd-1', '/dev/sda', None), ('osd-2', '/dev/sda', None), ('osd-3', '/dev/sda', None)]
[ceph_deploy.cli][INFO  ]  dmcrypt                       : False
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  bluestore                     : None
[ceph_deploy.cli][INFO  ]  block_wal                     : None
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : prepare
[ceph_deploy.cli][INFO  ]  dmcrypt_key_dir               : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x24805a8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  fs_type                       : xfs
[ceph_deploy.cli][INFO  ]  filestore                     : None
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x2474320>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  zap_disk                      : False
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks osd-0:/dev/sda: osd-1:/dev/sda: osd-2:/dev/sda: osd-3:/dev/sda:
.
.
.
[osd-3][INFO  ] checking OSD status...
[osd-3][DEBUG ] find the location of an executable
[osd-3][INFO  ] Running command: /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host osd-3 is now ready for osd use.

* osd에서 포맷 상태를 확인 합니다. 저널 파티션과, 데이터 파티션이 생성되고, 정상적으로 포맷이 되었습니다.

[root@osd-0 ~]# parted -l
Model: LSI MR9261-8i (scsi)
Disk /dev/sda: 16.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name          Flags
 2      1049kB  10.7GB  10.7GB               ceph journal
 1      10.7GB  16.0TB  16.0TB  xfs          ceph data


* ceph-osd 데몬을 구동 시키고 systemd에 등록 합니다.

[root@mgmt ~]# ceph-deploy osd activate osd-0:sda1 osd-1:sda1 osd-2:sda1 osd-3:sda1

* ceph activate 과정에서 장애-링크 가 발생 할 수 있습니다.
* osd 서버에서 데몬이 정상 적으로 올라왔는지 확인 합니다.

[root@osd-0 ~]# netstat -nlpt | grep ceph
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address    Foreign Address     State       PID/Program name    
tcp        0      0 0.0.0.0:6800     0.0.0.0:*           LISTEN      1382/ceph-osd       
tcp        0      0 0.0.0.0:6801     0.0.0.0:*           LISTEN      1382/ceph-osd       
tcp        0      0 0.0.0.0:6802     0.0.0.0:*           LISTEN      1382/ceph-osd       
tcp        0      0 0.0.0.0:6803     0.0.0.0:*           LISTEN      1382/ceph-osd       


* ceph-mds 데몬을 생성 합니다. 

[root@mgmt ~]# ceph-deploy mds create mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mds create mgmt
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x15de7e8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x15c6a28>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('mgmt', 'mgmt')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts mgmt:mgmt
[mgmt][DEBUG ] connected to host: mgmt 
[mgmt][DEBUG ] detect platform information from remote host
[mgmt][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to mgmt
[mgmt][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[mgmt][DEBUG ] create path if it doesn't exist
[mgmt][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.mgmt osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-mgmt/keyring
[mgmt][INFO  ] Running command: systemctl enable ceph-mds@mgmt
[mgmt][INFO  ] Running command: systemctl start ceph-mds@mgmt
[mgmt][INFO  ] Running command: systemctl enable ceph.target

[root@mgmt ~]# netstat -nlpt | grep ceph
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address     Foreign Address     State       PID/Program name    
tcp        0      0 0.0.0.0:6800      0.0.0.0:*           LISTEN      16122/ceph-mds 

tcp        0      0 192.168.1.11:6789  0.0.0.0:*           LISTEN      14772/ceph-mon 


* 설정된 ceph.conf,ceph.client.admin.keyring 파일을 각서버에 배포 합니다.

[root@mgmt ~]# ceph-deploy admin mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy admin mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x119dfc8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['mgmt', 'mon-0', 'mon-1', 'osd-0', 'osd-1', 'osd-2', 'osd-3']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x10f7cf8>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to mgmt
[mgmt][DEBUG ] connected to host: mgmt 
[mgmt][DEBUG ] detect platform information from remote host
[mgmt][DEBUG ] detect machine type
[mgmt][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
.
.
.
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to osd-3
[osd-3][DEBUG ] connected to host: osd-3 
[osd-3][DEBUG ] detect platform information from remote host
[osd-3][DEBUG ] detect machine type
[osd-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf


* 각 서버에 ceph.client.admin.keyrin 파일에 대한 권한을 수정 합니다.
* 600 => 644 로 변경.

[root@mgmt ~]# chmod +r /etc/ceph/ceph.client.admin.keyring


* ceph 구성이 완료 되었고 현재 상태 값을 확인 합니다.

[root@mgmt ~]# ceph -s
    cluster 0651cfa0-8ce4-4fbd-b35b-966e3d262851
     health HEALTH_OK
     monmap e1: 3 mons at {mgmt=192.168.1.11:6789/0,mon-0=192.168.1.12:6789/0,mon-1=192.168.1.13:6789/0}
            election epoch 12, quorum 0,1,2 mgmt,mon-0,mon-1
     osdmap e21: 4 osds: 4 up, 4 in
            flags sortbitwise,require_jewel_osds
      pgmap v49: 64 pgs, 1 pools, 0 bytes data, 0 objects
            141 MB used, 59556 GB / 59556 GB avail
                  64 active+clean

* ceph 의 모든 기본설치 및 구성이 완료 되었습니다.