HoguInside & ㅁ ㅇ ㄷ nail

[ ERROR 장애발생 ]

[root@loccalhost ~ ]# ceph-deploy osd activate osd-2:sda1
.
.
[osd-0][WARNIN] ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', u'1', '--monmap', '/var/lib/ceph/tmp/mnt.l7bamu/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.l7bamu', '--osd-journal', '/var/lib/ceph/tmp/mnt.l7bamu/journal', '--osd-uuid', u'a53f54e6-6dcb-4ed1-bfcf-3500e5ac5968', '--keyring', '/var/lib/ceph/tmp/mnt.l7bamu/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : 2018-04-18 16:13:17.313958 7f65fcbe1800 -1 filestore(/var/lib/ceph/tmp/mnt.l7bamu) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.l7bamu/journal: (13) Permission denied
[osd-0][WARNIN] 2018-04-18 16:13:17.313972 7f65fcbe1800 -1 OSD::mkfs: ObjectStore::mkfs failed with error -13
[osd-0][WARNIN] 2018-04-18 16:13:17.314012 7f65fcbe1800 -1  ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.l7bamu: (13) Permission denied
[osd-0][WARNIN] 
[osd-0][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/sda1

[ device 장치 확인 및 조치 ]


[ 정상 ]
[root@osd-1 ~]# ll /dev/sda*
brw-rw---- 1 root disk 8, 0  4월 18 16:41 /dev/sda
brw-rw---- 1 ceph ceph 8, 1  4월 18 16:41 /dev/sda1
brw-rw---- 1 ceph ceph 8, 2  4월 18 16:41 /dev/sda2

[ 장애 ]
[root@osd-2 ~]# ll /dev/sda*
brw-rw---- 1 root disk 8, 0  4월 18 16:06 /dev/sda
brw-rw---- 1 root disk 8, 1  4월 18 16:06 /dev/sda1
brw-rw---- 1 root disk 8, 2  4월 18 16:06 /dev/sda2


[ 해결 ]
[root@osd-2 ~]# chown ceph:ceph /dev/sda1
[root@osd-2 ~]# chown ceph:ceph /dev/sda2

또는

[root@osd-2 ~]# reboot
* 리부팅 후 해당 디바이스가 ceph:ceph 로 변경 됩니다.

[ ERROR. 장애발생 ]

[root@localhost ~]# ceph-deploy disk zap osd-0:sda 
.
.
[osd-0][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk', '--zap-all', '--', '/dev/sda']' returned non-zero exit status 2
[osd-0][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk zap /dev/sda

[ DISK 초기화 ]

[root@localhost ~ ]# dd if=/dev/zero of=/dev/sda bs=512 count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00852427 s, 60.1 kB/s

[root@localhost ~ ]# reboot

* disk 초기화 후 리부팅 하는 이유는 초기화 후 바로 "ceph-deploy disk zap" 를 진행하게 되면 정상적으로 작동은 하겠지만 종료가 되지 않는 버그(?)로 인해 리부팅 후 작업 진행을 추천합니다.

[ 정상 확인 ]

[root@localhost ~ ]# ceph-deploy disk zap osd-0:sda
.
.
.
[osd-0][DEBUG ] Creating new GPT entries.
[osd-0][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or
[osd-0][DEBUG ] other utilities.
[osd-0][DEBUG ] Creating new GPT entries.
[osd-0][DEBUG ] The operation has completed successfully.

[ CEPH 설치 - MON 서버 구성 ]

* 모든 작업은 mgmt 서버에서 진행 됩니다.
* ceph-deploy 는 ceph설치 및 운영시 사용되는 관리 패키지 입니다. ceph-deploy를 통해 설정 변경 시 일괄적으로 배포하고, osd추가 등을 편리하게 작업하게 해줍니다.
* yum update 해주세요. ceph-deploy 설치시 낮은 버젼으로 설치 되면 ceph 사이트 못찾아서  install 부터 장애가 발생됩니다.

[root@mgmt ~]# yum install ceph-deploy -y

* mon-0,mon-1 host 가 메인 모니터이고 mgmt는 개인적으로 서브로 넣어 두었습니다. mgmt를 포함하지 않으셔도 됩니다.

[root@mgmt ~]# ceph-deploy new mon-0 mon-1 mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.25): /usr/bin/ceph-deploy new mon-0 mon-1 mgmt
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[mon-0][DEBUG ] connected to host: mgmt 
[mon-0][INFO  ] Running command: ssh -CT -o BatchMode=yes mon-0
[mon-0][DEBUG ] connected to host: mon-0 
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
..
[mon-1][DEBUG ] connected to host: mgmt 
[mon-1][INFO  ] Running command: ssh -CT -o BatchMode=yes mon-1
[mon-1][DEBUG ] connected to host: mon-1 
...
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[mgmt][DEBUG ] connected to host: mgmt 
...
[ceph_deploy.new][DEBUG ] Monitor initial members are ['mon-0', 'mon-1', 'mgmt']
...
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...


* 위 명령어가 진행 되면 아래와 같이 ceph 관련 파일들이 생기게 됩니다.

[root@mgmt ~]# ll
-rw-r--r--   1 root root  1033  4월 18 11:38 .cephdeploy.conf
-rw-r--r--   1 root root   263  4월 18 11:38 ceph.conf
-rw-r--r--   1 root root  2395  4월 18 11:38 ceph.log
-rw-------   1 root root    73  4월 18 11:38 ceph.mon.keyring

* ceph.conf 설정

vi ceph.conf 

[global]
fsid = 0651cfa0-8ce4-4fbd-b35b-966e3d262851
mon_initial_members = mon-0, mon-1, mgmt
mon_host = 192.168.1.12,192.168.1.13,192.168.1.11
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true

# 여기서 부터 추가 설정 부분입니다. 
# osd_pool_default_size,rbd_default_features 기본으로 두 옵션만 넣고, 나머지 옵션들은 상황에 맞게 추가 합니다.
# ceph.conf 내용은 http://docs.ceph.com/docs/kraken/rados/configuration/ceph-conf/  참조하세요.

osd_pool_default_size = 2
rbd_default_features = 1
osd_journal_size = 10240
osd_pool_default_pg_num = 128
osd_pool_default_pgp_num = 128


[mon]
mon_host = mon-0, mon-1, mgmt
mon_addr = 192.168.1.12,192.168.1.13,192.168.1.11
mon_clock_drift_allowed = .3
mon_clock_drift_warn_backoff = 30    # Tell the monitor to backoff from this warning for 30 seconds
mon_osd_full_ratio = .90
mon_osd_nearfull_ratio = .85
mon_osd_report_timeout = 300
debug_ms = 1
debug_mon = 20
debug_paxos = 20
debug_auth = 20


[mon.0]
host = mon-0
mon_addr = 192.168.1.12:6789

[mon.1]
host = mon-1
mon_addr = 192.168.1.13:6789

[mon.2]
host = mgmt
mon_addr = 192.168.1.11:6789

[mds]
mds_standby_replay = true
mds_cache_size = 250000
debug_ms = 1
debug_mds = 20
debug_journaler = 20

[mds.0]
host = mgmt

[osd.0]
host = osd-0

[osd.1]
host = osd-1

[osd.2]
host = osd-2

[osd.3]
host = osd-3



* 각서버에 ceph관련 패키지를 설치 합니다. 아래와 같이 정상적으로 설치가 되며 그렇지 않을경우에는 방화벽, 네트워크 통신, resolv.conf, yum update(ceph-deploy 버전) 등을 확인해봐야 합니다.
* 설치는 host명으로 일괄 설치 합니다.
** 추가 사항 **
ubuntu ceph ver.10.x 에서는 아래와 같이 install 진행시 10.x 버전으로 설치가 되지만 centos는 가끔 상위 버젼으로 올라감..;; mgmt에서 ceph yumrepo를 jewel로 설정하고도 버전이 올라가면 다운 경로를 고정으로 잡고 진행.
ex)
ceph-deploy install --repo-url 'https://download.ceph.com/rpm-jewel/el7' --gpg-url 'https://download.ceph.com/keys/release.asc' mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3

[root@mgmt ~]# ceph-deploy install mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.25): /usr/bin/ceph-deploy install mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.install][DEBUG ] Installing stable version hammer on cluster ceph hosts mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.install][DEBUG ] Detecting platform for host mgmt ...
.
.
.

[osd-3][DEBUG ] 
[osd-3][DEBUG ] Complete!
[osd-3][INFO  ] Running command: ceph --version
[osd-3][DEBUG ] ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)


* ceph-mon 데몬 초기화 및 구동을 진행 합니다.

[root@mgmt ~]# ceph-deploy mon create-initial
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create-initial
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : 
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : 
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  keyrings                      : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts mon-0 mon-1 mgmt
[ceph_deploy.mon][DEBUG ] detecting platform for host mon-0 ...
.
.

* ceph-mon 데몬을 systemd에 등록 후 자동 실행으로 변경합니다.

[root@mgmt ~]# ceph-deploy mon create mon-0 mon-1 mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mon create mon-0 mon-1 mgmt
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : 
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  mon                           : ['mon-0', 'mon-1', 'mgmt']
[ceph_deploy.cli][INFO  ]  func                          : 
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  keyrings                      : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts mon-0 mon-1 mgmt
[ceph_deploy.mon][DEBUG ] detecting platform for host mon-0 ...

.
.
.

* ceph.conf 생성시 등록된 mon 서버들의 데몬을  초기화하고 구동 시킵니다. 각서버에서 " netstat -nlpt " 확인시 데몬 구동을 확인 합니다.

[root@mgmt ~]# netstat -nlpt | grep ceph
Active Internet connections (only servers)
Proto  Recv-Q Send-Q Local Address     Foreign Address    State       PID/Program name    
tcp         0      0 192.168.1.11:6789  0.0.0.0:*          LISTEN      14772/ceph-mon      

* mon-initial 후 키파일들이 생성 됩니다. mon 서버들이 ceph 서버에 정상적 접근하도록 합니다.
[root@mgmt ~]# ll
-rw-------   1 root root     71  4월 18 14:24 ceph.bootstrap-mds.keyring
-rw-------   1 root root     71  4월 18 14:24 ceph.bootstrap-mgr.keyring
-rw-------   1 root root    113  4월 18 14:24 ceph.bootstrap-osd.keyring
-rw-------   1 root root    113  4월 18 14:24 ceph.bootstrap-rgw.keyring
-rw-------   1 root root    129  4월 18 14:24 ceph.client.admin.keyring

* 생성된 키파일을 참조하게 합니다.
[root@mgmt ~]# ceph-deploy gatherkeys mon-0 mon-1 mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy gatherkeys mon-0 mon-1 mgmt
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x24e8cb0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  mon                           : ['mon-0', 'mon-1', 'mgmt']
[ceph_deploy.cli][INFO  ]  func                          : <function gatherkeys at 0x24cd500>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.gatherkeys][INFO  ] Storing keys in temp directory /tmp/tmpS0dvzG
[mon-0][DEBUG ] connected to host: mon-0 
[mon-0][DEBUG ] detect platform information from remote host
[mon-0][DEBUG ] detect machine type
[mon-0][DEBUG ] get remote short hostname
[mon-0][DEBUG ] fetch remote file
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.mon-0.asok mon_status
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.admin
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-mds
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-mgr
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-osd
[mon-0][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon-0/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.client.admin.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-mds.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-mgr.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-osd.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-rgw.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] Destroy temp directory /tmp/tmpS0dvzG


* mon 서버 구성은 끝났습니다. 이제 osd 구성에 들어 가기 전에 상태값을 먼저 확인해 봅니다.
[root@mgmt ~]# ceph -s
    cluster 0651cfa0-8ce4-4fbd-b35b-966e3d262851
     health HEALTH_ERR
            clock skew detected on mon.mon-0, mon.mon-1
            64 pgs are stuck inactive for more than 300 seconds
            64 pgs stuck inactive
            64 pgs stuck unclean
            no osds
            Monitor clock skew detected 
     monmap e1: 3 mons at {mgmt=192.168.1.11:6789/0,mon-0=192.168.1.12:6789/0,mon-1=192.168.1.13:6789/0}
            election epoch 4, quorum 0,1,2 mgmt,mon-0,mon-1
     osdmap e1: 0 osds: 0 up, 0 in
            flags sortbitwise,require_jewel_osds
      pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
            0 kB used, 0 kB / 0 kB avail
                  64 creating

* 현재 health 상태는 ERR 입니다. 이제 OSD를 구성해서 ok 로  변경 합시다.

[ CEPH 설치 - OSD 서버 구성 ]

* osd 서버의 디스크를 확인 하면 아래와 같습니다. 
  ceph를 사용할 디스크는 sda 16TB RAID 디스크이고, osd 서버 4대를 ceph로 구성합니다.

[root@osd-0 ~]# parted -l
Error: /dev/sda: unrecognised disk label
Model: LSI MR9261-8i (scsi)                                               
Disk /dev/sda: 16.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: unknown
Disk Flags: 

Model: ATA P3-128 (scsi)
Disk /dev/sdb: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  128GB  128GB  primary  xfs          boot

* ceph-deploy disk zap 명령어를 통해 해당 osd서버들의 디스크의 Partition Table 을 GPT로 생성합니다.
* 주의할 점은 LSI MR9261-8i 해당 레이드 카드 사용시, 고질적인 버그로 인해 pci 우선순위가 마음대로 바뀌며, 리부팅시 해당 디바이스 장치가 안보일 때가 있습니다.
  아래 명령어를 하기전에 각 서버의 디스크 장치를 확인해 주세요. 그렇지 않으면 OS디스크를 초기화 시켜 재설치 해야 합니다..... ㅠㅠ

[root@mgmt ~]# ceph-deploy disk zap osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy disk zap osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : zap
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1649368>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function disk at 0x163e398>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : [('osd-0', '/dev/sda', None), ('osd-1', '/dev/sda', None), ('osd-2', '/dev/sda', None), ('osd-3', '/dev/sda', None)]
[ceph_deploy.osd][DEBUG ] zapping /dev/sda on osd-0
.
.
.
[osd-3][DEBUG ] Creating new GPT entries.
[osd-3][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or
[osd-3][DEBUG ] other utilities.
[osd-3][DEBUG ] Creating new GPT entries.
[osd-3][DEBUG ] The operation has completed successfully.

* osd-0에서 확인해 보니 gpt로 잘 초기화 되어 있습니다.

[root@osd-0 ~]# parted -l
Model: LSI MR9261-8i (scsi)
Disk /dev/sda: 16.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start  End  Size  File system  Name  Flags


* 다음 명령어들로 디스크 포맷 및 ceph-osd 데몬을 구동 진행 합니다.

[root@mgmt ~]# ceph-deploy osd prepare osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy osd prepare osd-0:sda osd-1:sda osd-2:sda osd-3:sda
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  block_db                      : None
[ceph_deploy.cli][INFO  ]  disk                          : [('osd-0', '/dev/sda', None), ('osd-1', '/dev/sda', None), ('osd-2', '/dev/sda', None), ('osd-3', '/dev/sda', None)]
[ceph_deploy.cli][INFO  ]  dmcrypt                       : False
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  bluestore                     : None
[ceph_deploy.cli][INFO  ]  block_wal                     : None
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : prepare
[ceph_deploy.cli][INFO  ]  dmcrypt_key_dir               : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x24805a8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  fs_type                       : xfs
[ceph_deploy.cli][INFO  ]  filestore                     : None
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x2474320>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  zap_disk                      : False
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks osd-0:/dev/sda: osd-1:/dev/sda: osd-2:/dev/sda: osd-3:/dev/sda:
.
.
.
[osd-3][INFO  ] checking OSD status...
[osd-3][DEBUG ] find the location of an executable
[osd-3][INFO  ] Running command: /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host osd-3 is now ready for osd use.

* osd에서 포맷 상태를 확인 합니다. 저널 파티션과, 데이터 파티션이 생성되고, 정상적으로 포맷이 되었습니다.

[root@osd-0 ~]# parted -l
Model: LSI MR9261-8i (scsi)
Disk /dev/sda: 16.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name          Flags
 2      1049kB  10.7GB  10.7GB               ceph journal
 1      10.7GB  16.0TB  16.0TB  xfs          ceph data


* ceph-osd 데몬을 구동 시키고 systemd에 등록 합니다.

[root@mgmt ~]# ceph-deploy osd activate osd-0:sda1 osd-1:sda1 osd-2:sda1 osd-3:sda1

* ceph activate 과정에서 장애-링크 가 발생 할 수 있습니다.
* osd 서버에서 데몬이 정상 적으로 올라왔는지 확인 합니다.

[root@osd-0 ~]# netstat -nlpt | grep ceph
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address    Foreign Address     State       PID/Program name    
tcp        0      0 0.0.0.0:6800     0.0.0.0:*           LISTEN      1382/ceph-osd       
tcp        0      0 0.0.0.0:6801     0.0.0.0:*           LISTEN      1382/ceph-osd       
tcp        0      0 0.0.0.0:6802     0.0.0.0:*           LISTEN      1382/ceph-osd       
tcp        0      0 0.0.0.0:6803     0.0.0.0:*           LISTEN      1382/ceph-osd       


* ceph-mds 데몬을 생성 합니다. 

[root@mgmt ~]# ceph-deploy mds create mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mds create mgmt
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x15de7e8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x15c6a28>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('mgmt', 'mgmt')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts mgmt:mgmt
[mgmt][DEBUG ] connected to host: mgmt 
[mgmt][DEBUG ] detect platform information from remote host
[mgmt][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to mgmt
[mgmt][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[mgmt][DEBUG ] create path if it doesn't exist
[mgmt][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.mgmt osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-mgmt/keyring
[mgmt][INFO  ] Running command: systemctl enable ceph-mds@mgmt
[mgmt][INFO  ] Running command: systemctl start ceph-mds@mgmt
[mgmt][INFO  ] Running command: systemctl enable ceph.target

[root@mgmt ~]# netstat -nlpt | grep ceph
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address     Foreign Address     State       PID/Program name    
tcp        0      0 0.0.0.0:6800      0.0.0.0:*           LISTEN      16122/ceph-mds 

tcp        0      0 192.168.1.11:6789  0.0.0.0:*           LISTEN      14772/ceph-mon 


* 설정된 ceph.conf,ceph.client.admin.keyring 파일을 각서버에 배포 합니다.

[root@mgmt ~]# ceph-deploy admin mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy admin mgmt mon-0 mon-1 osd-0 osd-1 osd-2 osd-3
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x119dfc8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['mgmt', 'mon-0', 'mon-1', 'osd-0', 'osd-1', 'osd-2', 'osd-3']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x10f7cf8>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to mgmt
[mgmt][DEBUG ] connected to host: mgmt 
[mgmt][DEBUG ] detect platform information from remote host
[mgmt][DEBUG ] detect machine type
[mgmt][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
.
.
.
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to osd-3
[osd-3][DEBUG ] connected to host: osd-3 
[osd-3][DEBUG ] detect platform information from remote host
[osd-3][DEBUG ] detect machine type
[osd-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf


* 각 서버에 ceph.client.admin.keyrin 파일에 대한 권한을 수정 합니다.
* 600 => 644 로 변경.

[root@mgmt ~]# chmod +r /etc/ceph/ceph.client.admin.keyring


* ceph 구성이 완료 되었고 현재 상태 값을 확인 합니다.

[root@mgmt ~]# ceph -s
    cluster 0651cfa0-8ce4-4fbd-b35b-966e3d262851
     health HEALTH_OK
     monmap e1: 3 mons at {mgmt=192.168.1.11:6789/0,mon-0=192.168.1.12:6789/0,mon-1=192.168.1.13:6789/0}
            election epoch 12, quorum 0,1,2 mgmt,mon-0,mon-1
     osdmap e21: 4 osds: 4 up, 4 in
            flags sortbitwise,require_jewel_osds
      pgmap v49: 64 pgs, 1 pools, 0 bytes data, 0 objects
            141 MB used, 59556 GB / 59556 GB avail
                  64 active+clean

* ceph 의 모든 기본설치 및 구성이 완료 되었습니다.

[ yum repolist ]

* ceph 관련 저장소 및  epel-release 저장소를 설치 합니다.
* 다른 저장소 지정은  https://hoguinside.blogspot.kr/2017/11/yum.html  에서 등록하세요.

[root@localhost ~]# echo '[ceph]
name=Ceph packages for $basearch
baseurl=http://download.ceph.com/rpm-jewel/el7/$basearch
enabled=1
priority=2
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=http://download.ceph.com/rpm-jewel/el7/noarch
enabled=1
priority=2
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=http://download.ceph.com/rpm-jewel/el7/SRPMS
enabled=0
priority=2
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc ' > /etc/yum.repos.d/ceph.repo

[root@localhost ~]# yum install epel-release -y && yum repolist

[ 시간 동기화  ntp 설치 ]

* ceph의 장애 포인트중 하나는 시간 동기화 입니다. 서버 시간이 동일하지 않으면 데이터 동기화시 경고 메시지가 발생합니다.
  기본 설정은 0.3초 인데 동기화 단위 시간은 설정에서 변경 가능하며, 0.몇초 단위가 일치해야 경고 메시지가발생하기 않습니다.
* 해당 작업은 7대 서버에 각각 진행합니다.

[root@localhost ~]# yum install ntp -y
[root@localhost ~]# vi /etc/ntp.conf
.
.
.
 18 e public servers from the pool.ntp.org project.
 19 # Use public servers from the pool.ntp.org project..org/join.html).
 20 # Please consider joining the pool (http://www.pool.ntp.org/join.html).
 21 #server 0.centos.pool.ntp.org iburst    => 21 ~ 24 주석
 22 #server 1.centos.pool.ntp.org iburst
 23 #server 2.centos.pool.ntp.org iburst
 24 #server 3.centos.pool.ntp.org iburst

    server kr.pool.ntp.org    # => 해당 3줄 추가
    server time.bora.net
    server time.kornet.net

 25 #adcast 192.168.1.255 autokey        # broadcast server
 26 #broadcast 192.168.1.255 autokey        # broadcast server
 27 #broadcastclient autokey                # broadcast client
.
.
.

[root@localhost ~]# service ntpd restart
Redirecting to /bin/systemctl restart ntpd.service

[root@localhost ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.

[root@localhost ~]# netstat -nlp | grep ntp
udp        0      0 192.168.1.17:123         0.0.0.0:*                     22546/ntpd          
udp        0      0 127.0.0.1:123           0.0.0.0:*                     22546/ntpd          
udp        0      0 0.0.0.0:123             0.0.0.0:*                     22546/ntpd          
udp6       0      0 fe80::6eb3:11ff:fe3:123 :::*                          22546/ntpd          
udp6       0      0 ::1:123                 :::*                          22546/ntpd          
udp6       0      0 :::123                  :::*                          22546/ntpd   

[ HOST 설정 및 selinux off ]

[root@localhost ~]# vi /etc/hosts
192.168.1.11 MGTM
192.168.1.12 mon-0
192.168.1.13 mon-1
192.168.1.17 osd-0
192.168.1.18 osd-1
192.168.1.19 osd-2
192.168.1.20 osd-3


* 각 서버에서 HOSTNAME 변경
[root@localhost ~]# echo mgmt > /etc/hostname
[root@localhost ~]# echo mon-0 > /etc/hostname
[root@localhost ~]# echo mon-1 > /etc/hostname
[root@localhost ~]# echo osd-0 > /etc/hostname
[root@localhost ~]# echo osd-1 > /etc/hostname
[root@localhost ~]# echo osd-2 > /etc/hostname
[root@localhost ~]# echo osd-3 > /etc/hostname

[root@localhost ~]# setenforce 0  (해당 작업은 리부팅 하지 않았을 경우에 입력하고, 리부팅 예정이면 하지 않아도 무방합니다.)

[root@localhost ~]# vi /etc/selinux/config 
SELINUX=disabled  # => disabled 로 변경

[root@localhost ~]# reboot
.
.
[root@mgmt ~]# sestatus -v
SELinux status:                 disabled

[ SSH KEYGEN 생성 및 복사 ]

* 우분투나 타 메뉴얼등을 보면 ceph계정을 생성해서 진행하지만 본 포스팅은 root 로 진행합니다.
* 패스워드는 공란으로 엔터로 넘어갑니다.

[root@mgmt ~]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:26/AzqPSPY6Wdp48sn410UDwVOGD7pjl8JEM9iTeWXA root@mgmt
The key's randomart image is:
+---[RSA 2048]----+
|        .o+.E.   |
|         o.=     |
|        + +o+    |
|       o O.+..   |
|        S X.     |
|       . @o.     |
|     . o*.=.     |
|    . *=Bo .     |
|     ==OB+...    |
+----[SHA256]-----+

* 아래와 같이 7대 모두 공개키를 복사해 줍니다. 주의 할점은 mgmt 자기 자신에게도 복사를 해야 합니다. 
  ceph 설치시 7대 전부 진행해야 하기 때문에 해당 작업을 하지 않으면 일일이  패스워드를 입력해야 합니다.

[root@mgmt ~]# ssh-copy-id mgmt
.
.
[root@mgmt ~]# ssh-copy-id mon-0
.
.
[root@mgmt ~]# ssh-copy-id mon-1
.
.
.

[root@mgmt ~]# ssh osd-0
Last failed login: Wed Apr 18 11:20:35 KST 2018 from xxx.xxx.xx.xxx.ptr.cy4n.net on ssh:notty
There were 25 failed login attempts since the last successful login.
Last login: Wed Apr 18 11:10:12 2018 from xxx.xx.xx.x
[root@osd-0 ~]# 


* 테스트 후 정상적이면 CEPH 설치작업으로 넘어 갑니다.

[ febootstrap 설치 -  centos 6 ]

[root@centos6 ]# yum install febootstrap fake* -y
.
.
.
[root@centos6 ]# rpm -qa | grep -P "febootstrap|fake"
fakeroot-1.12.2-22.2.el6.x86_64
fakechroot-libs-2.9-24.5.el6_1.1.x86_64
febootstrap-supermin-helper-3.21-4.el6.x86_64
fakechroot-2.9-24.5.el6_1.1.x86_64
febootstrap-3.21-4.el6.x86_64
fakeroot-libs-1.12.2-22.2.el6.x86_64

[ febootstrap 설치 -  centos 7 ]

[root@centos7 ]# yum install febootstrap fake* -y
.
.
.
[root@centos7 ]# rpm -qa | grep -P "febootstrap|fake"
fakeroot-1.18.4-2.el7.x86_64
fakeroot-libs-1.18.4-2.el7.x86_64

[root@centos7 ]# cd /usr/local/src
[root@centos7 ]# wget https://rpmfind.net/linux/centos/6.9/os/x86_64/Packages/febootstrap-3.21-4.el6.x86_64.rpm
[root@centos7 ]# wget https://rpmfind.net/linux/centos/6.9/os/x86_64/Packages/febootstrap-supermin-helper-3.21-4.el6.x86_64.rpm
[root@centos7 ]# wget https://rpmfind.net/linux/centos/6.9/os/x86_64/Packages/fakechroot-2.9-24.5.el6_1.1.x86_64.rpm
[root@centos7 ]# wget https://rpmfind.net/linux/centos/6.9/os/x86_64/Packages/fakechroot-libs-2.9-24.5.el6_1.1.x86_64.rpm
[root@centos7 ]# ll
합계 604
drwxr-xr-x.  2 root root   4096  4월 10 14:08 .
drwxr-xr-x. 13 root root   4096  3월 27 11:15 ..
-rw-r--r--   1 root root  23576  9월 26  2011 fakechroot-2.9-24.5.el6_1.1.x86_64.rpm
-rw-r--r--   1 root root  23240  9월 26  2011 fakechroot-libs-2.9-24.5.el6_1.1.x86_64.rpm
-rw-r--r--   1 root root 209020 11월 25  2013 febootstrap-3.21-4.el6.x86_64.rpm
-rw-r--r--   1 root root 345436 11월 25  2013 febootstrap-supermin-helper-3.21-4.el6.x86_64.rpm

[root@centos7 ]# rpm -Uvh ./*
경고: ./fakechroot-2.9-24.5.el6_1.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
준비 중...                         ################################# [100%]
Updating / installing...
   1:febootstrap-supermin-helper-3.21-################################# [ 25%]
   2:fakechroot-libs-2.9-24.5.el6_1.1 ################################# [ 50%]
   3:fakechroot-2.9-24.5.el6_1.1      ################################# [ 75%]
   4:febootstrap-3.21-4.el6           ################################# [100%]

[root@centos7 ]# rpm -qa | grep -P "febootstrap|fake"
fakeroot-1.18.4-2.el7.x86_64
febootstrap-3.21-4.el6.x86_64
fakeroot-libs-1.18.4-2.el7.x86_64
febootstrap-supermin-helper-3.21-4.el6.x86_64
fakechroot-2.9-24.5.el6_1.1.x86_64
fakechroot-libs-2.9-24.5.el6_1.1.x86_64

*centos7 저장소에는 해당 패키지가 없기 때문에 centos6 패키지를 가져가 사용했습니다. 호환성에 문제가 없으니 잘되는군요.

[ febootstrap 이미지 생성 및 도킹 ]

febootstrap [--options] REPO TARGET [MIRROR]

[root@centos7]# febootstrap -u http://mirror.kakao.com/centos/7.4.1708/updates/x86_64/ centos7.4 centos7 http://mirror.kakao.com/centos/7.4.1708/os/x86_64/

또는

[root@centos7]# febootstrap -i yum -i bash -i wget -i net-tools -i iputils -i iproute -i openssh-server -i openssh-clients centos7.4 centos7 http://mirror.kakao.com/centos/7.4.1708/os/x86_64/

* 위 의 명령어로 진행하게 되면 updates 패키지들을 모두 다운 및 설치 진행 하고, 아래 명령어로 진행하게 되면 최소한의 패키지 설치로 이미지가 생성이됩니다. 해당 방식을 통해 원하는 패키지를 미리 설치 해서 이미지를 생성 준비를 합니다. 두 명령어들의 용량은 약 2배 정도 차이가 나게 됩니다.

[root@centos7]# ll
합계 0
drwxr-xr-x  7 root root  83  4월 10 15:08 .
drwxr-xr-x  4 root root  78  4월 10 10:38 ..
drwxr-xr-x  2 root root  30  4월 10 11:00 centos6
dr-xr-xr-x 17 root root 224  4월 10 15:08 centos7

[root@centos7]# tar -C centos7 -c . | docker import  - centos7.4

또는 
[root@centos7]# cd centos7
[root@centos7]# tar -c . | docker import  - centos7.4

[root@centos7]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos7.4           latest              79b78179cd69        53 seconds ago      490 MB
centos6-base        latest              4257dd712401        4 hours ago         2.3 GB

[root@centos7]# docker run -it --name centos7 centos7.4 /bin/bash
bash-4.2# cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 

* 정상적으로 도킹이 완료 되었습니다.

[ debootstrap 이미지 생성 및 도킹 ]

root@ubuntu:/home# apt-get install debootstrap -y
root@ubuntu:/home# debootstrap
I: usage: [OPTION]... <suite> <target> [<mirror> [<script>]]
debootstrap {code} {directory}

root@ubuntu:/home# debootstrap xenial xenial
I: Retrieving InRelease 
I: Checking Release signature
I: Valid Release signature (key id 790BC7277767219C42C86F933B4FE6ACC0B21F32)
I: Retrieving Packages 
I: Validating Packages 
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://archive.ubuntu.com/ubuntu...
I: Retrieving adduser 3.113+nmu3ubuntu4
.
.
.
I: Base system installed successfully.

root@ubuntu:/home# ll
total 233484
drwxr-xr-x  3 root root      4096 Apr 17 11:52 ./
drwxr-xr-x 23 root root      4096 Apr 13 15:21 ../
-rw-r--r--  1 root root 239073280 Apr 11 17:23 ubuntu16.04.tar
drwxr-xr-x 21 root root      4096 Apr 17 11:58 xenial/

root@ubuntu:/home# cd xenial/
root@ubuntu:/home/xenial# ll
total 84
drwxr-xr-x 21 root root 4096 Apr 17 11:58 ./
drwxr-xr-x  3 root root 4096 Apr 17 11:52 ../
drwxr-xr-x  2 root root 4096 Apr 17 11:57 bin/
drwxr-xr-x  2 root root 4096 Apr 13  2016 boot/
drwxr-xr-x  4 root root 4096 Apr 17 11:57 dev/
drwxr-xr-x 63 root root 4096 Apr 17 11:58 etc/
drwxr-xr-x  2 root root 4096 Apr 13  2016 home/
drwxr-xr-x 11 root root 4096 Apr 17 11:57 lib/
drwxr-xr-x  2 root root 4096 Apr 17 11:57 lib64/
drwxr-xr-x  2 root root 4096 Apr 17 11:57 media/
drwxr-xr-x  2 root root 4096 Apr 17 11:57 mnt/
drwxr-xr-x  2 root root 4096 Apr 17 11:57 opt/
drwxr-xr-x  2 root root 4096 Apr 13  2016 proc/
drwx------  2 root root 4096 Apr 17 11:57 root/
drwxr-xr-x  6 root root 4096 Apr 17 11:58 run/
drwxr-xr-x  2 root root 4096 Apr 17 11:58 sbin/
drwxr-xr-x  2 root root 4096 Apr 17 11:57 srv/
drwxr-xr-x  2 root root 4096 Feb  5  2016 sys/
drwxrwxrwt  2 root root 4096 Apr 17 11:58 tmp/
drwxr-xr-x 10 root root 4096 Apr 17 11:57 usr/
drwxr-xr-x 11 root root 4096 Apr 17 11:57 var/

* os관련 파일들이 정상적으로 다운 받아졌습니다. 도킹은 압축을 통해 사용하거나 rsync 등을 이용하여 docker 서비스로 넘겨 사용합니다.

[ docker 베이스 이미지 제작 - 현재 os 이미지화 ]

* 테스트 서버는 docker가 없는 centos6 os 서버 입니다. 

tar --numeric-owner --exclude=/proc --exclude=/sys --exclude=/boot --exclude=/usr/lib/firmware --exclude=/usr/lib/modules --exclude=/usr/lib/udev -cvf {os}.tar /

cat {os}.tar | docker import - {os명}


[root@centos6 ]# docker
-bash: docker: command not found
[root@centos6 ]# mkdir /home/docker-base-image
[root@centos6 ]# cd /home/docker-base-image

[root@centos6 ]# tar --numeric-owner --exclude=/proc --exclude=/sys --exclude=/boot --exclude=/usr/lib/firmware --exclude=/usr/lib/modules --exclude=/usr/lib/udev -cvf centos6-base.tar /
tar: Removing leading `/' from member names
/
/dev/
/dev/vcsa6
/dev/vcs6
/dev/vcsa5
/dev/vcs5
.
.
.
.
[root@centos6 ]# ll
합계 2295524
drwxr-xr-x  2 root root       4096 2018-04-10 10:47 .
drwxr-xr-x. 4 root root       4096 2018-04-10 10:46 ..
-rw-r--r--  1 root root 2350602240 2018-04-10 10:49 centos6-base.tar

* hub.docker 를 통해 이미지 업로드후 pull,run 명령어로 다운 및 실행 할 수 있습니다.
  지금은 그냥 scp를 이용해 테스트 합니다.

[root@centos6 ]# scp centos6-base.tar 192.168.229.224:/home/docker-imgaes/centos6
root@192.168.229.224's password: 
centos6-base.tar        100% 2242MB 124.5MB/s   00:18

* docker 서버 os는  centos7 입니다.

[root@docker]# cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core)

[root@docker]# pwd
/home/docker-imgaes/centos6

[root@docker]# ll
합계 2295512
drwxr-xr-x 2 root root         30  4월 10 11:00 .
drwxr-xr-x 6 root root         68  4월 10 11:00 ..
-rw-r--r-- 1 root root 2350602240  4월 10 11:00 centos6-base.tar

[root@docker]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

[root@docker]# time cat centos6-base.tar | docker import - centos6-base
sha256:4257dd712401282041cbc3501e1eb42f66359fc43aa985cdb56534aad3e21fe6

real 0m36.616s
user 0m1.088s
sys 0m7.615s

[root@docker]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE
centos6-base        latest              4257dd712401        About a minute ago   2.3 GB

[root@docker]# docker run -it --name cent6 centos6-base /bin/bash
[root@183dd65101ce /]# cat /etc/redhat-release 
CentOS release 6.9 (Final)

* 정상적으로 이미지 도킹 및 컨테이너 실행이 되었습니다. 해당 방식은 필요 한 패키지등을 모두 설치 후 제작하면 좋을 것 같고, 필요,불필요 디렉토리등은 제작 명령어의 --exclude 옵션을 이용하시면 되겠습니다.

[ Quick Start ]
[root@localhost ]# docker run -d -p 6080:80 --name ubuntu_dt dorowu/ubuntu-desktop-lxde-vnc
Unable to find image 'dorowu/ubuntu-desktop-lxde-vnc:latest' locally
Trying to pull repository docker.io/dorowu/ubuntu-desktop-lxde-vnc ... 
latest: Pulling from docker.io/dorowu/ubuntu-desktop-lxde-vnc
5ed55f09d535: Extracting [=====================>                             ] 184.9 MB/424.1 MB
e9ee3a2f4ce1: Download complete 
f228aa1b5acd: Download complete 

[root@localhost ]# docker images
REPOSITORY                  TAG                 IMAGE ID            CREATED             SIZE
docker.io/dorowu/ubuntu-…   latest              5077c595d94a        5 days ago          1.1 GB

[root@localhost ]# docker ps -a
CONTAINER ID    MAGE           COMMAND       CREATED     STATUS    PORTS                NAMES
bd95bd7d0599    dorowu/ubuntu… "/startup.sh" 56 seconds ago…       0.0.0.0:6080->80/tcp ubuntu_dt

[접속 확인]
Web Browse chrome

http://xxx.xxx.xxx.xxx:6080/



* start menu -> Internet -> 브라우저 실행
  브라우저에서 크롬은 실행이 안되네요.. 원인을 찾아봐야 할듯 합니다.
  아래 있는 크롬 사운드는 모듈이 없어서 실행이 안됩니다.
  파이어 폭스만 정상적으로 됩니다.

[ NOVnc 사운드 모드 ]

* centos7에는 기본적으로 모듈이 설치되어 있습니다. modprobe 로 모듈만 적재 시켜 주시면 될것 같습니다.
* 아래는 ubuntu 16.04 서버에서 진행 하였습니다.

[root@localhost ]# apt-get install linux-generic build-essential alsa-utils

[root@localhost ]# modprobe snd-aloop index=2
modprobe: FATAL: Module snd-aloop not found in directory /lib/modules/4.4.0-62-generic

* 모듈을 찾을 수 없다면 리부팅 진행 

docker run -it --rm -p 6080:80 --device /dev/snd -e ALSADEV=hw:2,0 dorowu/ubuntu-desktop-lxde-vnc

* 해당 명령어로 진행시 백그라운드 실행이 아닌 포그라운드 실행으로 로그 확인이 가능 합니다.
* ctrl+d 입력시 컨테이너 삭제및 종료 됩니다. 운영시에는 -d 옵션 추가와 --rm 옵션을 제거 하고 진행합니다.

[root@localhost ]# docker run -itd -p 6080:80 --device /dev/snd -e ALSADEV=hw:2,0 --name ubuntu_dt dorowu/ubuntu-desktop-lxde-vnc
Unable to find image 'dorowu/ubuntu-desktop-lxde-vnc:latest' locally
latest: Pulling from dorowu/ubuntu-desktop-lxde-vnc
5ed55f09d535: Downloading [==========================>                        ] 223.5 MB/424.1 MB
e9ee3a2f4ce1: Verifying Checksum 
f228aa1b5acd: Download complete 

* 기본 모드로 실행을 한번 이상했다면 다운로드 없이 진행됩니다.

[접속 확인]
* 일반 접속과는 달리  /?video   붙여 줍니다. 해당 주소 부분을 제거 하면 일반 모드로 접속 됩니다.

http://xxx.xxx.xxx.xxx:6080/#/?video



* 접속시 접근 속도가 느립니다. 반응속도 역시 느립니다...
  start menu -> Internet -> Chromium Web Browser Sound 실행합니다.
  youtube 접속을 통해 사운드 확인이 되었습니다. 하지만, 반응 속도 및 서버가 부하가 올라가네요.. 
  사용하기에는 효율적이지 못할 것 같습니다.

[VNC viewer]

* VNC 서비스 포트 5900를 호스트로 연결 됩니다.

[root@localhost ]# docker run -p 6080:80 -p 5900:5900 dorowu/ubuntu-desktop-lxde-vnc

* vnc 뷰어 기본포트 5900으로 연결 하고, 서비스 보호를 하려면 패스워드 옵션을 넣습니다.

[root@localhost ]# docker run -p 6080:80 -p 5900:5900 -e VNC_PASSWORD={"패스워드"} dorowu/ubuntu-desktop-lxde-vnc
* {"패스워드"} 에 패스워드를 입력 후 vnc 뷰어 접속시 패스워드 입력후 접근 가능합니다.

[ HTTP Base Authentication ]

* 해당 이미지는 브라우저 접근시 패스워드를 넣을 수 있습니다.
root@localhost ]#  docker run -p 6080:80 -e HTTP_PASSWORD=mypassword dorowu/ubuntu-desktop-lxde-vnc


* ssl, 해상도 설정, 사용자 설정 역시 옵션을 통해 지정 할 수 있습니다.  자세한 내용은 상단 참조 사이트에서 확인 해주세요.

[ 문제 발생 ]

* eth0 과 docker0 의 mtu가 다른 상황입니다. MTU가 다를때는 옵션을 줘서 동일 값으로 셋팅합니다.

[root@localhost]# ip link
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0:  mtu 1450 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether fa:16:3e:0a:b2:dc brd ff:ff:ff:ff:ff:ff
3: docker0:  mtu 1500 qdisc noqueue state DOWN mode DEFAULT 
    link/ether 02:42:86:b8:b7:01 brd ff:ff:ff:ff:ff:ff

* docker MTU 기본값을 변경 합니다.

[ centos7 ]
[root@localhost]# vi /lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target rhel-push-plugin.socket registries.service
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer

[Service]
Type=notify
NotifyAccess=all
EnvironmentFile=-/run/containers/registries.conf
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \
          --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
          --default-runtime=docker-runc \
          --exec-opt native.cgroupdriver=systemd \
          --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
          --seccomp-profile=/etc/docker/seccomp.json \
          --mtu 1450 \
          $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $ADD_REGISTRY \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY \
          $REGISTRIES
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
MountFlags=slave
KillMode=process

[Install]
WantedBy=multi-user.target


[ ubuntu 16.04]
[root@localhost]# vi /lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --mtu 1450
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target
~                              


[root@skhkil 04:41:22:/lib/systemd/system]# systemctl daemon-reload 
[root@skhkil 04:41:30:/lib/systemd/system]# service docker restart
Redirecting to /bin/systemctl restart docker.service

* centos7 ,ubuntu16.04 에서 MTU 값을 서버와 동일하게 셋팅하고, 재접속 또는 docker os 마운트해서 확인하면 mtu 변경 확인이 가능 합니다.

[root@localhost]# ip link
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0:  mtu 1450 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether fa:16:3e:0a:b2:dc brd ff:ff:ff:ff:ff:ff
3: docker0:  mtu 1450 qdisc noqueue state UP mode DEFAULT 
    link/ether 02:42:86:b8:b7:01 brd ff:ff:ff:ff:ff:ff
107: vethf134985@if106:  mtu 1450 qdisc noqueue master docker0 state UP mode DEFAULT 
    link/ether 42:82:96:46:44:b4 brd ff:ff:ff:ff:ff:ff link-netnsid 0