HoguInside & ㅁ ㅇ ㄷ nail

[ ERROR. 장애발생 ]

[root@localhost ~]# ceph-deploy disk zap osd-0:sda 
.
.
[osd-0][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk', '--zap-all', '--', '/dev/sda']' returned non-zero exit status 2
[osd-0][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk zap /dev/sda

[ DISK 초기화 ]

[root@localhost ~ ]# dd if=/dev/zero of=/dev/sda bs=512 count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00852427 s, 60.1 kB/s

[root@localhost ~ ]# reboot

* disk 초기화 후 리부팅 하는 이유는 초기화 후 바로 "ceph-deploy disk zap" 를 진행하게 되면 정상적으로 작동은 하겠지만 종료가 되지 않는 버그(?)로 인해 리부팅 후 작업 진행을 추천합니다.

[ 정상 확인 ]

[root@localhost ~ ]# ceph-deploy disk zap osd-0:sda
.
.
.
[osd-0][DEBUG ] Creating new GPT entries.
[osd-0][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or
[osd-0][DEBUG ] other utilities.
[osd-0][DEBUG ] Creating new GPT entries.
[osd-0][DEBUG ] The operation has completed successfully.

[ CEPH 설치 - MON 서버 구성 ]

* 모든 작업은 mgmt 서버에서 진행 됩니다.
* ceph-deploy 는 ceph설치 및 운영시 사용되는 관리 패키지 입니다. ceph-deploy를 통해 설정 변경 시 일괄적으로 배포하고, osd추가 등을 편리하게 작업하게 해줍니다.
* yum update 해주세요. ceph-deploy 설치시 낮은 버젼으로 설치 되면 ceph 사이트 못찾아서  install 부터 장애가 발생됩니다.

[root@mgmt ~]# yum install ceph-deploy -y

* mon-0,mon-1 host 가 메인 모니터이고 mgmt는 개인적으로 서브로 넣어 두었습니다. mgmt를 포함하지 않으셔도 됩니다.

[root@mgmt ~]# ceph-deploy new mon-0 mon-1 mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.25): /usr/bin/ceph-deploy new mon-0 mon-1 mgmt
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[mon-0][DEBUG ] connected to host: mgmt 
[mon-0][INFO  ] Running command: ssh -CT -o BatchMode=yes mon-0
[mon-0][DEBUG ] connected to host: mon-0 
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
..
[mon-1][DEBUG ] connected to host: mgmt 
[mon-1][INFO  ] Running command: ssh -CT -o BatchMode=yes mon-1
[mon-1][DEBUG ] connected to host: mon-1 
...
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[mgmt][DEBUG ] connected to host: mgmt 
...
[ceph_deploy.new][DEBUG ] Monitor initial members are ['mon-0', 'mon-1', 'mgmt']
...
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...


* 위 명령어가 진행 되면 아래와 같이 ceph 관련 파일들이 생기게 됩니다.

[root@mgmt ~]# ll
-rw-r--r--   1 root root  1033  4월 18 11:38 .cephdeploy.conf
-rw-r--r--   1 root root   263  4월 18 11:38 ceph.conf
-rw-r--r--   1 root root  2395  4월 18 11:38 ceph.log
-rw-------   1 root root    73  4월 18 11:38 ceph.mon.keyring

* ceph.conf 설정

vi ceph.conf 

[global]
fsid = 0651cfa0-8ce4-4fbd-b35b-966e3d262851
mon_initial_members = mon-0, mon-1, mgmt
mon_host = 192.168.1.12,192.168.1.13,192.168.1.11
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true

# 여기서 부터 추가 설정 부분입니다. 
# osd_pool_default_size,rbd_default_features 기본으로 두 옵션만 넣고, 나머지 옵션들은 상황에 맞게 추가 합니다.
# ceph.conf 내용은 http://docs.ceph.com/docs/kraken/rados/configuration/ceph-conf/  참조하세요.

osd_pool_default_size = 2
rbd_default_features = 1
osd_journal_size = 10240
osd_pool_default_pg_num = 128
osd_pool_default_pgp_num = 128


[mon]
mon_host = mon-0, mon-1, mgmt
mon_addr = 192.168.1.12,192.168.1.13,192.168.1.11
mon_clock_drift_allowed = .3
mon_clock_drift_warn_backoff = 30    # Tell the monitor to backoff from this warning for 30 seconds
mon_osd_full_ratio = .90
mon_osd_nearfull_ratio = .85
mon_osd_report_timeout = 300
debug_ms = 1
debug_mon = 20
debug_paxos = 20
debug_auth = 20


[mon.0]
host = mon-0
mon_addr = 192.168.1.12:6789

[mon.1]
host = mon-1
mon_addr = 192.168.1.13:6789

[mon.2]
host = mgmt
mon_addr = 192.168.1.11:6789

[mds]
mds_standby_replay = true
mds_cache_size = 250000
debug_ms = 1
debug_mds = 20
debug_journaler = 20

[mds.0]
host = mgmt

[osd.0]
host = osd-0

[osd.1]
host = osd-1

[osd.2]
host = osd-2

[osd.3]
host = osd-3



* 각서버에 ceph관련 패키지를 설치 합니다. 아래와 같이 정상적으로 설치가 되며 그렇지 않을경우에는 방화벽, 네트워크 통신, resolv.conf, yum update(ceph-deploy 버전) 등을 확인해봐야 합니다.
* 설치는 host명으로 일괄 설치 합니다.
** 추가 사항 **
ubuntu ceph ver.10.x 에서는 아래와 같이 install 진행시 10.x 버전으로 설치가 되지만 centos는 가끔 상위 버젼으로 올라감..;; mgmt에서 ceph yumrepo를 jewel로 설정하고도 버전이 올라가면 다운 경로를 고정으로 잡고 진행.
ex)
ceph-deploy install --repo-url 'https://download.ceph.com/rpm-jewel/el7' --gpg-url 'https://download.ceph.com/keys/release.asc' mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3

[root@mgmt ~]# ceph-deploy install mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.25): /usr/bin/ceph-deploy install mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.install][DEBUG ] Installing stable version hammer on cluster ceph hosts mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.install][DEBUG ] Detecting platform for host mgmt ...
.
.
.

[osd-3][DEBUG ] 
[osd-3][DEBUG ] Complete!
[osd-3][INFO  ] Running command: ceph --version
[osd-3][DEBUG ] ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)


* ceph-mon 데몬 초기화 및 구동을 진행 합니다.

[root@mgmt ~]# ceph-deploy mon create-initial
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create-initial
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : 
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : 
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  keyrings                      : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts mon-0 mon-1 mgmt
[ceph_deploy.mon][DEBUG ] detecting platform for host mon-0 ...
.
.

* ceph-mon 데몬을 systemd에 등록 후 자동 실행으로 변경합니다.

[root@mgmt ~]# ceph-deploy mon create mon-0 mon-1 mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mon create mon-0 mon-1 mgmt
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : 
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  mon                           : ['mon-0', 'mon-1', 'mgmt']
[ceph_deploy.cli][INFO  ]  func                          : 
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  keyrings                      : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts mon-0 mon-1 mgmt
[ceph_deploy.mon][DEBUG ] detecting platform for host mon-0 ...

.
.
.

* ceph.conf 생성시 등록된 mon 서버들의 데몬을  초기화하고 구동 시킵니다. 각서버에서 " netstat -nlpt " 확인시 데몬 구동을 확인 합니다.

[root@mgmt ~]# netstat -nlpt | grep ceph
Active Internet connections (only servers)
Proto  Recv-Q Send-Q Local Address     Foreign Address    State       PID/Program name    
tcp         0      0 192.168.1.11:6789  0.0.0.0:*          LISTEN      14772/ceph-mon      

* mon-initial 후 키파일들이 생성 됩니다. mon 서버들이 ceph 서버에 정상적 접근하도록 합니다.
[root@mgmt ~]# ll
-rw-------   1 root root     71  4월 18 14:24 ceph.bootstrap-mds.keyring
-rw-------   1 root root     71  4월 18 14:24 ceph.bootstrap-mgr.keyring
-rw-------   1 root root    113  4월 18 14:24 ceph.bootstrap-osd.keyring
-rw-------   1 root root    113  4월 18 14:24 ceph.bootstrap-rgw.keyring
-rw-------   1 root root    129  4월 18 14:24 ceph.client.admin.keyring

* 생성된 키파일을 참조하게 합니다.
[root@mgmt ~]# ceph-deploy gatherkeys mon-0 mon-1 mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy gatherkeys mon-0 mon-1 mgmt
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x24e8cb0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  mon                           : ['mon-0', 'mon-1', 'mgmt']
[ceph_deploy.cli][INFO  ]  func                          : <function gatherkeys at 0x24cd500>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.gatherkeys][INFO  ] Storing keys in temp directory /tmp/tmpS0dvzG
[mon-0][DEBUG ] connected to host: mon-0 
[mon-0][DEBUG ] detect platform information from remote host
[mon-0][DEBUG ] detect machine type
[mon-0][DEBUG ] get remote short hostname
[mon-0][DEBUG ] fetch remote file
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.mon-0.asok mon_status
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.admin
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-mds
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-mgr
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-osd
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.client.admin.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-mds.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-mgr.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-osd.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-rgw.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] Destroy temp directory /tmp/tmpS0dvzG


* mon 서버 구성은 끝났습니다. 이제 osd 구성에 들어 가기 전에 상태값을 먼저 확인해 봅니다.
[root@mgmt ~]# ceph -s
    cluster 0651cfa0-8ce4-4fbd-b35b-966e3d262851
     health HEALTH_ERR
            clock skew detected on mon.mon-0, mon.mon-1
            64 pgs are stuck inactive for more than 300 seconds
            64 pgs stuck inactive
            64 pgs stuck unclean
            no osds
            Monitor clock skew detected 
     monmap e1: 3 mons at {mgmt=192.168.1.11:6789/0,mon-0=192.168.1.12:6789/0,mon-1=192.168.1.13:6789/0}
            election epoch 4, quorum 0,1,2 mgmt,mon-0,mon-1
     osdmap e1: 0 osds: 0 up, 0 in
            flags sortbitwise,require_jewel_osds
      pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
            0 kB used, 0 kB / 0 kB avail
                  64 creating

* 현재 health 상태는 ERR 입니다. 이제 OSD를 구성해서 ok 로  변경 합시다.

[ CEPH 설치 - OSD 서버 구성 ]

* osd 서버의 디스크를 확인 하면 아래와 같습니다. 
  ceph를 사용할 디스크는 sda 16TB RAID 디스크이고, osd 서버 4대를 ceph로 구성합니다.

[root@osd-0 ~]# parted -l
Error: /dev/sda: unrecognised disk label
Model: LSI MR9261-8i (scsi)                                               
Disk /dev/sda: 16.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: unknown
Disk Flags: 

Model: ATA P3-128 (scsi)
Disk /dev/sdb: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  128GB  128GB  primary  xfs          boot

* ceph-deploy disk zap 명령어를 통해 해당 osd서버들의 디스크의 Partition Table 을 GPT로 생성합니다.
* 주의할 점은 LSI MR9261-8i 해당 레이드 카드 사용시, 고질적인 버그로 인해 pci 우선순위가 마음대로 바뀌며, 리부팅시 해당 디바이스 장치가 안보일 때가 있습니다.
  아래 명령어를 하기전에 각 서버의 디스크 장치를 확인해 주세요. 그렇지 않으면 OS디스크를 초기화 시켜 재설치 해야 합니다..... ㅠㅠ

[root@mgmt ~]# ceph-deploy disk zap osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy disk zap osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : zap
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1649368>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function disk at 0x163e398>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : [('osd-0', '/dev/sda', None), ('osd-1', '/dev/sda', None), ('osd-2', '/dev/sda', None), ('osd-3', '/dev/sda', None)]
[ceph_deploy.osd][DEBUG ] zapping /dev/sda on osd-0
.
.
.
[osd-3][DEBUG ] Creating new GPT entries.
[osd-3][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or
[osd-3][DEBUG ] other utilities.
[osd-3][DEBUG ] Creating new GPT entries.
[osd-3][DEBUG ] The operation has completed successfully.

* osd-0에서 확인해 보니 gpt로 잘 초기화 되어 있습니다.

[root@osd-0 ~]# parted -l
Model: LSI MR9261-8i (scsi)
Disk /dev/sda: 16.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start  End  Size  File system  Name  Flags


* 다음 명령어들로 디스크 포맷 및 ceph-osd 데몬을 구동 진행 합니다.

[root@mgmt ~]# ceph-deploy osd prepare osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy osd prepare osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  block_db                      : None
[ceph_deploy.cli][INFO  ]  disk                          : [('osd-0', '/dev/sda', None), ('osd-1', '/dev/sda', None), ('osd-2', '/dev/sda', None), ('osd-3', '/dev/sda', None)]
[ceph_deploy.cli][INFO  ]  dmcrypt                       : False
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  bluestore                     : None
[ceph_deploy.cli][INFO  ]  block_wal                     : None
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : prepare
[ceph_deploy.cli][INFO  ]  dmcrypt_key_dir               : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x24805a8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  fs_type                       : xfs
[ceph_deploy.cli][INFO  ]  filestore                     : None
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x2474320>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  zap_disk                      : False
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks osd-0:/dev/sda: osd-1:/dev/sda: osd-2:/dev/sda: osd-3:/dev/sda:
.
.
.
[osd-3][INFO  ] checking OSD status...
[osd-3][DEBUG ] find the location of an executable
[osd-3][INFO  ] Running command: /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host osd-3 is now ready for osd use.

* osd에서 포맷 상태를 확인 합니다. 저널 파티션과, 데이터 파티션이 생성되고, 정상적으로 포맷이 되었습니다.

[root@osd-0 ~]# parted -l
Model: LSI MR9261-8i (scsi)
Disk /dev/sda: 16.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name          Flags
 2      1049kB  10.7GB  10.7GB               ceph journal
 1      10.7GB  16.0TB  16.0TB  xfs          ceph data


* ceph-osd 데몬을 구동 시키고 systemd에 등록 합니다.

[root@mgmt ~]# ceph-deploy osd activate osd-0:sda1 osd-1:sda1 osd-2:sda1 osd-3:sda1

* ceph activate 과정에서 장애-링크 가 발생 할 수 있습니다.
* osd 서버에서 데몬이 정상 적으로 올라왔는지 확인 합니다.

[root@osd-0 ~]# netstat -nlpt | grep ceph
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address    Foreign Address     State       PID/Program name    
tcp        0      0 0.0.0.0:6800     0.0.0.0:*           LISTEN      1382/ceph-osd       
tcp        0      0 0.0.0.0:6801     0.0.0.0:*           LISTEN      1382/ceph-osd       
tcp        0      0 0.0.0.0:6802     0.0.0.0:*           LISTEN      1382/ceph-osd       
tcp        0      0 0.0.0.0:6803     0.0.0.0:*           LISTEN      1382/ceph-osd       


* ceph-mds 데몬을 생성 합니다. 

[root@mgmt ~]# ceph-deploy mds create mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mds create mgmt
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x15de7e8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x15c6a28>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('mgmt', 'mgmt')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts mgmt:mgmt
[mgmt][DEBUG ] connected to host: mgmt 
[mgmt][DEBUG ] detect platform information from remote host
[mgmt][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to mgmt
[mgmt][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[mgmt][DEBUG ] create path if it doesn't exist
[mgmt][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.mgmt osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-mgmt/keyring
[mgmt][INFO  ] Running command: systemctl enable ceph-mds@mgmt
[mgmt][INFO  ] Running command: systemctl start ceph-mds@mgmt
[mgmt][INFO  ] Running command: systemctl enable ceph.target

[root@mgmt ~]# netstat -nlpt | grep ceph
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address     Foreign Address     State       PID/Program name    
tcp        0      0 0.0.0.0:6800      0.0.0.0:*           LISTEN      16122/ceph-mds 

tcp        0      0 192.168.1.11:6789  0.0.0.0:*           LISTEN      14772/ceph-mon 


* 설정된 ceph.conf,ceph.client.admin.keyring 파일을 각서버에 배포 합니다.

[root@mgmt ~]# ceph-deploy admin mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy admin mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x119dfc8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['mgmt', 'mon-0', 'mon-1', 'osd-0', 'osd-1', 'osd-2', 'osd-3']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x10f7cf8>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to mgmt
[mgmt][DEBUG ] connected to host: mgmt 
[mgmt][DEBUG ] detect platform information from remote host
[mgmt][DEBUG ] detect machine type
[mgmt][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
.
.
.
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to osd-3
[osd-3][DEBUG ] connected to host: osd-3 
[osd-3][DEBUG ] detect platform information from remote host
[osd-3][DEBUG ] detect machine type
[osd-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf


* 각 서버에 ceph.client.admin.keyrin 파일에 대한 권한을 수정 합니다.
* 600 => 644 로 변경.

[root@mgmt ~]# chmod +r /etc/ceph/ceph.client.admin.keyring


* ceph 구성이 완료 되었고 현재 상태 값을 확인 합니다.

[root@mgmt ~]# ceph -s
    cluster 0651cfa0-8ce4-4fbd-b35b-966e3d262851
     health HEALTH_OK
     monmap e1: 3 mons at {mgmt=192.168.1.11:6789/0,mon-0=192.168.1.12:6789/0,mon-1=192.168.1.13:6789/0}
            election epoch 12, quorum 0,1,2 mgmt,mon-0,mon-1
     osdmap e21: 4 osds: 4 up, 4 in
            flags sortbitwise,require_jewel_osds
      pgmap v49: 64 pgs, 1 pools, 0 bytes data, 0 objects
            141 MB used, 59556 GB / 59556 GB avail
                  64 active+clean

* ceph 의 모든 기본설치 및 구성이 완료 되었습니다.

[ yum repolist ]

* ceph 관련 저장소 및  epel-release 저장소를 설치 합니다.
* 다른 저장소 지정은  https://hoguinside.blogspot.kr/2017/11/yum.html  에서 등록하세요.

[root@localhost ~]# echo '[ceph]
name=Ceph packages for $basearch
baseurl=http://download.ceph.com/rpm-jewel/el7/$basearch
enabled=1
priority=2
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=http://download.ceph.com/rpm-jewel/el7/noarch
enabled=1
priority=2
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=http://download.ceph.com/rpm-jewel/el7/SRPMS
enabled=0
priority=2
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc ' > /etc/yum.repos.d/ceph.repo

[root@localhost ~]# yum install epel-release -y && yum repolist

[ 시간 동기화  ntp 설치 ]

* ceph의 장애 포인트중 하나는 시간 동기화 입니다. 서버 시간이 동일하지 않으면 데이터 동기화시 경고 메시지가 발생합니다.
  기본 설정은 0.3초 인데 동기화 단위 시간은 설정에서 변경 가능하며, 0.몇초 단위가 일치해야 경고 메시지가발생하기 않습니다.
* 해당 작업은 7대 서버에 각각 진행합니다.

[root@localhost ~]# yum install ntp -y
[root@localhost ~]# vi /etc/ntp.conf
.
.
.
 18 e public servers from the pool.ntp.org project.
 19 # Use public servers from the pool.ntp.org project..org/join.html).
 20 # Please consider joining the pool (http://www.pool.ntp.org/join.html).
 21 #server 0.centos.pool.ntp.org iburst    => 21 ~ 24 주석
 22 #server 1.centos.pool.ntp.org iburst
 23 #server 2.centos.pool.ntp.org iburst
 24 #server 3.centos.pool.ntp.org iburst

    server kr.pool.ntp.org    # => 해당 3줄 추가
    server time.bora.net
    server time.kornet.net

 25 #adcast 192.168.1.255 autokey        # broadcast server
 26 #broadcast 192.168.1.255 autokey        # broadcast server
 27 #broadcastclient autokey                # broadcast client
.
.
.

[root@localhost ~]# service ntpd restart
Redirecting to /bin/systemctl restart ntpd.service

[root@localhost ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.

[root@localhost ~]# netstat -nlp | grep ntp
udp        0      0 192.168.1.17:123         0.0.0.0:*                     22546/ntpd          
udp        0      0 127.0.0.1:123           0.0.0.0:*                     22546/ntpd          
udp        0      0 0.0.0.0:123             0.0.0.0:*                     22546/ntpd          
udp6       0      0 fe80::6eb3:11ff:fe3:123 :::*                          22546/ntpd          
udp6       0      0 ::1:123                 :::*                          22546/ntpd          
udp6       0      0 :::123                  :::*                          22546/ntpd   

[ HOST 설정 및 selinux off ]

[root@localhost ~]# vi /etc/hosts
192.168.1.11 MGTM
192.168.1.12 mon-0
192.168.1.13 mon-1
192.168.1.17 osd-0
192.168.1.18 osd-1
192.168.1.19 osd-2
192.168.1.20 osd-3


* 각 서버에서 HOSTNAME 변경
[root@localhost ~]# echo mgmt > /etc/hostname
[root@localhost ~]# echo mon-0 > /etc/hostname
[root@localhost ~]# echo mon-1 > /etc/hostname
[root@localhost ~]# echo osd-0 > /etc/hostname
[root@localhost ~]# echo osd-1 > /etc/hostname
[root@localhost ~]# echo osd-2 > /etc/hostname
[root@localhost ~]# echo osd-3 > /etc/hostname

[root@localhost ~]# setenforce 0  (해당 작업은 리부팅 하지 않았을 경우에 입력하고, 리부팅 예정이면 하지 않아도 무방합니다.)

[root@localhost ~]# vi /etc/selinux/config 
SELINUX=disabled  # => disabled 로 변경

[root@localhost ~]# reboot
.
.
[root@mgmt ~]# sestatus -v
SELinux status:                 disabled

[ SSH KEYGEN 생성 및 복사 ]

* 우분투나 타 메뉴얼등을 보면 ceph계정을 생성해서 진행하지만 본 포스팅은 root 로 진행합니다.
* 패스워드는 공란으로 엔터로 넘어갑니다.

[root@mgmt ~]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:26/AzqPSPY6Wdp48sn410UDwVOGD7pjl8JEM9iTeWXA root@mgmt
The key's randomart image is:
+---[RSA 2048]----+
|        .o+.E.   |
|         o.=     |
|        + +o+    |
|       o O.+..   |
|        S X.     |
|       . @o.     |
|     . o*.=.     |
|    . *=Bo .     |
|     ==OB+...    |
+----[SHA256]-----+

* 아래와 같이 7대 모두 공개키를 복사해 줍니다. 주의 할점은 mgmt 자기 자신에게도 복사를 해야 합니다. 
  ceph 설치시 7대 전부 진행해야 하기 때문에 해당 작업을 하지 않으면 일일이  패스워드를 입력해야 합니다.

[root@mgmt ~]# ssh-copy-id mgmt
.
.
[root@mgmt ~]# ssh-copy-id mon-0
.
.
[root@mgmt ~]# ssh-copy-id mon-1
.
.
.

[root@mgmt ~]# ssh osd-0
Last failed login: Wed Apr 18 11:20:35 KST 2018 from xxx.xxx.xx.xxx.ptr.cy4n.net on ssh:notty
There were 25 failed login attempts since the last successful login.
Last login: Wed Apr 18 11:10:12 2018 from xxx.xx.xx.x
[root@osd-0 ~]# 


* 테스트 후 정상적이면 CEPH 설치작업으로 넘어 갑니다.