HoguInside & ㅁ ㅇ ㄷ nail

DATABASES

mysql InnoDB: Error: Table "mysql"."innodb_table_stats" not found.

By 때찌때찌맴매 - 10월 26, 2018

mysql - InnoDB: Error: Table "mysql"."innodb_table_stats" not found.

* 고객 db가 죽는 상황이 발생. mysql 재구동을 하자 pid 생성 오류로 데몬이 안올라옴.

* 일단 살려야 되니 innodb_force_recovery = 1 부터 올려서 구동

참조 : https://hoguinside.blogspot.com/2014/09/innodb-innodbforcerecovery.html

전에 사용중이던 mysql db를 복원하고 로그 확인 결과 해당 에러들 발견됨.

* 원인은 기존에 사용하던 mysql버전이 5.5 였는데 지금 서버는 5.6.

MySQL 5.6 부터 mysql database 에 통계정보나 slave 관련 정보를 저장하기 위해 새로운 InnoDB table 5개가 추가 (innodb_index_stats, innodb_table_stats, slave_master_info, slave_relay_log_info, slave_worker_info)되었는데 버전이 5.5 -> 5.6 변경시 mysql_upgrade 를 진행하지 않을 경우 해당 에러 발생. (물론 쿼리들을 해당 버전에 맞게 튜닝도 필요. 이 내용은 고객에게 안내)

내용 : https://dev.mysql.com/doc/refman/5.6/en/innodb-persistent-stats.html


[ database 에러 ]

[root@localhost01:33:26:~]# tail -f /usr/local/mysql/data/localhost.localdomain.err
2018-10-26 13:24:11 7f92494d6700 InnoDB: Error: Fetch of persistent statistics requested for table "SY_NOTI"."DMND_ORGNZ2" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.

2018-10-26 13:24:11 7f92494d6700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2018-10-26 13:24:11 7f92494d6700 InnoDB: Error: Fetch of persistent statistics requested for table "SY_NOTI"."PROD_CLAS_CODE" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.

2018-10-26 13:24:14 7f924af31700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2018-10-26 13:24:14 7f924af31700 InnoDB: Recalculation of persistent statistics requested for table "SY_SESSION"."SPRING_SESSION" but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.


[ 해결 ]

i. 위 언급한 mysql database에서 5개 테이블 제거

[root@localhost01:25:56:/usr/local/mysql/data]# mysql -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 52789
Server version: 5.6.10-log Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mysql;
mysql> show tables;
+---------------------------+
| Tables_in_mysql           |
+---------------------------+
| columns_priv              |
| db                        |
| event                     |
| func                      |
| general_log               |
| help_category             |
| help_keyword              |
| help_relation             |
| help_topic                |
| innodb_index_stats        |
| innodb_table_stats        |
| ndb_binlog_index          |
| plugin                    |
| proc                      |
| procs_priv                |
| proxies_priv              |
| servers                   |
| slave_master_info         |
| slave_relay_log_info      |
| slave_worker_info         |
| slow_log                  |
| tables_priv               |
| time_zone                 |
| time_zone_leap_second     |
| time_zone_name            |
| time_zone_transition      |
| time_zone_transition_type |
| user                      |
+---------------------------+
28 rows in set (0.00 sec)

mysql> drop table innodb_index_stats ; drop table innodb_table_stats; drop table slave_master_info; drop table slave_relay_log_info; drop table slave_worker_info;
ERROR 1051 (42S02): Unknown table 'mysql.innodb_index_stats'
ERROR 1051 (42S02): Unknown table 'mysql.innodb_table_stats'
ERROR 1051 (42S02): Unknown table 'mysql.slave_master_info'
ERROR 1051 (42S02): Unknown table 'mysql.slave_relay_log_info'
ERROR 1051 (42S02): Unknown table 'mysql.slave_worker_info'
mysql> show tables;
+---------------------------+
| Tables_in_mysql           |
+---------------------------+
| columns_priv              |
| db                        |
| event                     |
| func                      |
| general_log               |
| help_category             |
| help_keyword              |
| help_relation             |
| help_topic                |
| ndb_binlog_index          |
| plugin                    |
| proc                      |
| procs_priv                |
| proxies_priv              |
| servers                   |
| slow_log                  |
| tables_priv               |
| time_zone                 |
| time_zone_leap_second     |
| time_zone_name            |
| time_zone_transition      |
| time_zone_transition_type |
| user                      |
+---------------------------+
23 rows in set (0.00 sec)

mysql> quit

* table drop 시 에러메시지는  위에서 force recorvery 로 강제 복구시 정도가 제거 된듯;;
  확인 결과 정상적으로 테이블 제거

ii. ****.rbd 파일 제거 및 mysql 재구동

[root@localhost01:27:21:/usr/local/mysql/data]# /etc/init.d/mysqld stop
Shutting down MySQL....                                    [  OK  ]


[root@localhost01:27:21:/usr/local/mysql/data]#cd mysql
[root@localhost01:27:36:/usr/local/mysql/data/mysql]# rm -rf innodb_index_stats.ibd innodb_table_stats.ibd slave_master_info.ibd slave_relay_log_info.ibd  slave_worker_info.ibd

[root@localhost01:28:20:/usr/local/mysql/data/mysql]# /etc/init.d/mysqld start
Starting MySQL...                                          [  OK  ]

[root@localhost01:28:29:/usr/local/mysql/data/mysql]# /usr/local/mysql/bin/mysql_upgrade --force -uroot -p
Enter password: 
Looking for 'mysql' as: /usr/local/mysql/bin/mysql
Looking for 'mysqlcheck' as: /usr/local/mysql/bin/mysqlcheck
Running 'mysqlcheck' with connection arguments: '--port=3306' '--socket=/tmp/mysql.sock' 
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck' with connection arguments: '--port=3306' '--socket=/tmp/mysql.sock' 
Warning: Using a password on the command line interface can be insecure.
SY_ANLS.AUTO_ANLS                                  OK
SY_ANLS.AUTO_ANLS_GUGAN                            OK
SY_ANLS.AUTO_ANLS_LOG                              OK
SY_ANLS.AUTO_ANLS_PRF                              OK
SY_ANLS.AUTO_ANLS_RESULT                           OK
SY_ANLS.AUTO_ANLS_STAT_CD                          OK
SY_ANLS.QUARTILE                                   OK
SY_APP.CUST_APP                                    OK
SY_APP.CUST_APP_BIZ                                OK
SY_APP.SY_APP_MASTER                               OK
SY_BILCLT.BILCLT                                   OK
SY_BILCLT.BILCLT_AGR                               OK
SY_BILCLT.BILCLT_EXCCLC_RATE                       OK
SY_BILCLT.BILCLT_HIST                              OK
SY_BILCLT.BILCLT_HIST_BACK                         OK
SY_BILCLT.RCPTMN                                   OK
SY_BILCLT.RCPTMN_HIST                              OK
SY_BILCLT.TAXBL                                    OK
SY_CUST.ANLS_CHRGR_CHNG_HIST                       OK
SY_CUST.CUST                                       OK
SY_CUST.CUST_ACRSLT_AP_ST                          OK
SY_CUST.CUST_APP_CFG                               OK
SY_CUST.CUST_BIZ                                   OK
SY_CUST.CUST_BIZ_HIST                              OK
SY_CUST.CUST_CFG                                   OK
SY_CUST.CUST_CNCN_NOTI                             OK
SY_CUST.CUST_CNTR_MNGT                             OK
SY_CUST.CUST_COMSN_CNDTN                           OK
SY_CUST.CUST_CONSL                                 OK
SY_CUST.CUST_CONSL_KIND                            OK
SY_CUST.CUST_CRGRS                                 OK
SY_CUST.CUST_HIST                                  OK
SY_CUST.CUST_LICNS                                 OK
SY_CUST.CUST_LICNS_DEL_HIST                        OK
SY_CUST.CUST_LICNS_HIST                            OK
SY_CUST.CUST_LOGIN_HIST                            OK
SY_CUST.CUST_MEMO_HIST                             OK
SY_CUST.CUST_MXD_LICNS                             OK
SY_CUST.CUST_MXD_PRVNCL                            OK
SY_CUST.CUST_NOTI                                  OK
SY_CUST.CUST_NOTI_HIST                             OK
SY_CUST.CUST_OMTP_ECPT_NOTI                        OK
SY_CUST.CUST_PRFSH                                 OK
SY_CUST.CUST_PRTC_AMT                              OK
SY_CUST.CUST_PRTC_AMT_HIST                         OK
SY_CUST.CUST_SBID                                  OK
SY_CUST.CUST_SVC                                   OK
SY_CUST.CUST_TELNO                                 OK
SY_CUST.CUST_TKN                                   OK
SY_CUST.CUST_UTLZ_CLAUSE                           OK
SY_CUST.CUST_UTLZ_CLAUSE_HIST                      OK
SY_CUST.KAKAOTALK_USER                             OK
SY_CUST.NCST                                       OK
SY_CUST.NCST_BSN                                   OK
SY_CUST.NCST_CONSL                                 OK
SY_CUST.NCST_LICNS                                 OK
SY_CUST.NCST_MNGT_FLDR                             OK
SY_CUST.NCST_MNGT_FLDR_JNOW                        OK
SY_CUST.NCST_MNGT_FLDR_TARGT                       OK
SY_CUST.NCST_RECPTN_REJECT                         OK
SY_CUST.NCST_TELNO                                 OK
SY_CUST.NCST_TELNO_HIST                            OK
SY_CUST.TRANS                                      OK
SY_CUST.TRANS_TMPL                                 OK
SY_EMP.ACRSLT_NOTI                                 OK
SY_EMP.ACRSLT_ST_MNGT                              OK
SY_EMP.ATTEND_LVFFC                                OK
SY_EMP.AUTH                                        OK
SY_EMP.BDDPR_ASIST_MNGT                            OK
SY_EMP.EMP                                         OK
SY_EMP.EMP_ALAM                                    OK
SY_EMP.EMP_AUTHZ_REQ                               OK
SY_EMP.EMP_CNCN_NOTI                               OK
SY_EMP.EMP_GBS_LICNS                               OK
SY_EMP.EMP_HIST                                    OK
SY_EMP.EMP_LOGIN_HIST                              OK
SY_EMP.EMP_LONG_OFFDAY                             OK
SY_EMP.EMP_MEMO                                    OK
SY_EMP.EMP_MENU                                    OK
SY_EMP.EMP_MENU_AUTH                               OK
SY_EMP.EMP_MENU_SKIL                               OK
SY_EMP.EMP_NTBD                                    OK
SY_EMP.EMP_NTBD_JNOW                               OK
SY_EMP.EMP_PC_REGST                                OK
SY_EMP.EMP_REC                                     OK
SY_EMP.POSIT_TEAM                                  OK
SY_EMP.POSIT_TEAM_AUTH                             OK
SY_ETC.ETCLINK                                     OK
SY_ETC.SCHEDULED_RUN                               OK
SY_ETC.migrationCheck107                           OK
SY_NOTI.AREA_CODE                                  OK
SY_NOTI.BATCH_JOB_EXECUTION                        OK
SY_NOTI.BATCH_JOB_EXECUTION_CONTEXT                OK
SY_NOTI.BATCH_JOB_EXECUTION_PARAMS                 OK
SY_NOTI.BATCH_JOB_EXECUTION_SEQ                    OK
SY_NOTI.BATCH_JOB_HIST                             OK
SY_NOTI.BATCH_JOB_INSTANCE                         OK
SY_NOTI.BATCH_JOB_SEQ                              OK
SY_NOTI.BATCH_MASTER                               OK
SY_NOTI.BATCH_STEP_EXECUTION                       OK
SY_NOTI.BATCH_STEP_EXECUTION_CONTEXT               OK
SY_NOTI.BATCH_STEP_EXECUTION_SEQ                   OK
SY_NOTI.CFMT_UNDWRT_ACRSLT_MLTP                    OK
SY_NOTI.CFMT_UNDWRT_AVG_RATE                       OK
SY_NOTI.CFMT_UNDWRT_CDLT_DISTMK                    OK
SY_NOTI.CFMT_UNDWRT_CDLT_GROUP                     OK
SY_NOTI.CFMT_UNDWRT_CDLT_ST                        OK
SY_NOTI.CFMT_UNDWRT_ST                             OK
SY_NOTI.CFMT_UNDWRT_ST_DISTMK                      OK
SY_NOTI.CNTR_MTHD                                  OK
SY_NOTI.COMN_CODE                                  OK
SY_NOTI.COMN_CODE_GROUP                            OK
SY_NOTI.DMND_ORGNZ                                 OK
SY_NOTI.DMND_ORGNZ2                                OK
SY_NOTI.KEP_REQ_HDR                                OK
SY_NOTI.LICNS_CODE                                 OK
SY_NOTI.LICNS_NEED_TCNXPRT                         OK
SY_NOTI.NOTI                                       OK
SY_NOTI.NOTI_AREA                                  OK
SY_NOTI.NOTI_AREA_HIST                             OK
SY_NOTI.NOTI_ATCHMNFL                              OK
SY_NOTI.NOTI_BDDPR_PRTC_CMP                        OK
SY_NOTI.NOTI_COMPNO_PRDPRC                         OK
SY_NOTI.NOTI_HIST                                  OK
SY_NOTI.NOTI_LICNS                                 OK
SY_NOTI.NOTI_LICNS_HIST                            OK
SY_NOTI.NOTI_ORIGN                                 OK
SY_NOTI.NOTI_SBID_PRARG_CMP                        OK
SY_NOTI.PROD_CLAS_CODE                             OK
SY_NOTI.RPSNT_ORGNZ                                OK
SY_NOTI_TEST.AREA_CODE                             OK
SY_NOTI_TEST.BATCH_JOB_EXECUTION                   OK
SY_NOTI_TEST.BATCH_JOB_EXECUTION_CONTEXT           OK
SY_NOTI_TEST.BATCH_JOB_EXECUTION_PARAMS            OK
SY_NOTI_TEST.BATCH_JOB_EXECUTION_SEQ               OK
SY_NOTI_TEST.BATCH_JOB_HIST                        OK
SY_NOTI_TEST.BATCH_JOB_INSTANCE                    OK
SY_NOTI_TEST.BATCH_JOB_SEQ                         OK
SY_NOTI_TEST.BATCH_MASTER                          OK
SY_NOTI_TEST.BATCH_STEP_EXECUTION                  OK
SY_NOTI_TEST.BATCH_STEP_EXECUTION_CONTEXT          OK
SY_NOTI_TEST.BATCH_STEP_EXECUTION_SEQ              OK
SY_NOTI_TEST.CFMT_UNDWRT_ACRSLT_MLTP               OK
SY_NOTI_TEST.CFMT_UNDWRT_AVG_RATE                  OK
SY_NOTI_TEST.CFMT_UNDWRT_CDLT_DISTMK               OK
SY_NOTI_TEST.CFMT_UNDWRT_CDLT_GROUP                OK
SY_NOTI_TEST.CFMT_UNDWRT_CDLT_ST                   OK
SY_NOTI_TEST.CFMT_UNDWRT_ST                        OK
SY_NOTI_TEST.CFMT_UNDWRT_ST_DISTMK                 OK
SY_NOTI_TEST.CNTR_MTHD                             OK
SY_NOTI_TEST.COMN_CODE                             OK
SY_NOTI_TEST.COMN_CODE_GROUP                       OK
SY_NOTI_TEST.DMND_ORGNZ                            OK
SY_NOTI_TEST.DMND_ORGNZ2                           OK
SY_NOTI_TEST.LICNS_CODE                            OK
SY_NOTI_TEST.NOTI                                  OK
SY_NOTI_TEST.NOTI_AREA                             OK
SY_NOTI_TEST.NOTI_AREA_HIST                        OK
SY_NOTI_TEST.NOTI_ATCHMNFL                         OK
SY_NOTI_TEST.NOTI_BDDPR_PRTC_CMP                   OK
SY_NOTI_TEST.NOTI_COMPNO_PRDPRC                    OK
SY_NOTI_TEST.NOTI_HIST                             OK
SY_NOTI_TEST.NOTI_LICNS                            OK
SY_NOTI_TEST.NOTI_LICNS_HIST                       OK
SY_NOTI_TEST.NOTI_ORIGN                            OK
SY_NOTI_TEST.NOTI_SBID_PRARG_CMP                   OK
SY_NOTI_TEST.RPSNT_ORGNZ                           OK
SY_SESSION.SPRING_SESSION                          OK
SY_SESSION.SPRING_SESSION_ATTRIBUTES               OK
SY_TRSMIS.TRSMIS                                   OK
SY_TRSMIS.TRSMIS_ALAM                              OK
SY_TRSMIS.TRSMIS_ALAM_OBJPS                        OK
SY_TRSMIS.TRSMIS_ATCHMNFL                          OK
SY_TRSMIS.TRSMIS_SCLAS                             OK
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.general_log                                  OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.slow_log                                     OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
mysql_20181026.columns_priv                        OK
mysql_20181026.db                                  OK
mysql_20181026.event                               OK
mysql_20181026.func                                OK
mysql_20181026.general_log                         OK
mysql_20181026.help_category                       OK
mysql_20181026.help_keyword                        OK
mysql_20181026.help_relation                       OK
mysql_20181026.help_topic                          OK
mysql_20181026.innodb_index_stats
Error    : Table 'mysql_20181026.innodb_index_stats' doesn't exist
status   : Operation failed
mysql_20181026.innodb_table_stats
Error    : Table 'mysql_20181026.innodb_table_stats' doesn't exist
status   : Operation failed
mysql_20181026.ndb_binlog_index                    OK
mysql_20181026.plugin                              OK
mysql_20181026.proc                                OK
mysql_20181026.procs_priv                          OK
mysql_20181026.proxies_priv                        OK
mysql_20181026.servers                             OK
mysql_20181026.slave_master_info
Error    : Table 'mysql_20181026.slave_master_info' doesn't exist
status   : Operation failed
mysql_20181026.slave_relay_log_info
Error    : Table 'mysql_20181026.slave_relay_log_info' doesn't exist
status   : Operation failed
mysql_20181026.slave_worker_info
Error    : Table 'mysql_20181026.slave_worker_info' doesn't exist
status   : Operation failed
mysql_20181026.slow_log                            OK
mysql_20181026.tables_priv                         OK
mysql_20181026.time_zone                           OK
mysql_20181026.time_zone_leap_second               OK
mysql_20181026.time_zone_name                      OK
mysql_20181026.time_zone_transition                OK
mysql_20181026.time_zone_transition_type           OK
mysql_20181026.user                                OK
mysql_back.columns_priv                            OK
mysql_back.db                                      OK
mysql_back.event                                   OK
mysql_back.func                                    OK
mysql_back.general_log                             OK
mysql_back.help_category                           OK
mysql_back.help_keyword                            OK
mysql_back.help_relation                           OK
mysql_back.help_topic                              OK
mysql_back.innodb_index_stats
Error    : Table 'mysql_back.innodb_index_stats' doesn't exist
status   : Operation failed
mysql_back.innodb_table_stats
Error    : Table 'mysql_back.innodb_table_stats' doesn't exist
status   : Operation failed
mysql_back.ndb_binlog_index                        OK
mysql_back.plugin                                  OK
mysql_back.proc                                    OK
mysql_back.procs_priv                              OK
mysql_back.proxies_priv                            OK
mysql_back.servers                                 OK
mysql_back.slave_master_info
Error    : Table 'mysql_back.slave_master_info' doesn't exist
status   : Operation failed
mysql_back.slave_relay_log_info
Error    : Table 'mysql_back.slave_relay_log_info' doesn't exist
status   : Operation failed
mysql_back.slave_worker_info
Error    : Table 'mysql_back.slave_worker_info' doesn't exist
status   : Operation failed
mysql_back.slow_log                                OK
mysql_back.tables_priv                             OK
mysql_back.time_zone                               OK
mysql_back.time_zone_leap_second                   OK
mysql_back.time_zone_name                          OK
mysql_back.time_zone_transition                    OK
mysql_back.time_zone_transition_type               OK
mysql_back.user                                    OK
test.board                                         OK
test.board_reply                                   OK
test.test                                          OK
test.users                                         OK

Repairing tables
mysql_20181026.innodb_index_stats
Error    : Table 'mysql_20181026.innodb_index_stats' doesn't exist
status   : Operation failed
mysql_20181026.innodb_table_stats
Error    : Table 'mysql_20181026.innodb_table_stats' doesn't exist
status   : Operation failed
mysql_20181026.slave_master_info
Error    : Table 'mysql_20181026.slave_master_info' doesn't exist
status   : Operation failed
mysql_20181026.slave_relay_log_info
Error    : Table 'mysql_20181026.slave_relay_log_info' doesn't exist
status   : Operation failed
mysql_20181026.slave_worker_info
Error    : Table 'mysql_20181026.slave_worker_info' doesn't exist
status   : Operation failed
mysql_back.innodb_index_stats
Error    : Table 'mysql_back.innodb_index_stats' doesn't exist
status   : Operation failed
mysql_back.innodb_table_stats
Error    : Table 'mysql_back.innodb_table_stats' doesn't exist
status   : Operation failed
mysql_back.slave_master_info
Error    : Table 'mysql_back.slave_master_info' doesn't exist
status   : Operation failed
mysql_back.slave_relay_log_info
Error    : Table 'mysql_back.slave_relay_log_info' doesn't exist
status   : Operation failed
mysql_back.slave_worker_info
Error    : Table 'mysql_back.slave_worker_info' doesn't exist
status   : Operation failed
Running 'mysql_fix_privilege_tables'...
Warning: Using a password on the command line interface can be insecure.
OK

[root@localhost01:28:35:/usr/local/mysql/data/mysql]# mysql -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 333
Server version: 5.6.10-log Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mysql
Database changed
mysql> show tables;
+---------------------------+
| Tables_in_mysql           |
+---------------------------+
| columns_priv              |
| db                        |
| event                     |
| func                      |
| general_log               |
| help_category             |
| help_keyword              |
| help_relation             |
| help_topic                |
| innodb_index_stats        |
| innodb_table_stats        |
| ndb_binlog_index          |
| plugin                    |
| proc                      |
| procs_priv                |
| proxies_priv              |
| servers                   |
| slave_master_info         |
| slave_relay_log_info      |
| slave_worker_info         |
| slow_log                  |
| tables_priv               |
| time_zone                 |
| time_zone_leap_second     |
| time_zone_name            |
| time_zone_transition      |
| time_zone_transition_type |
| user                      |
+---------------------------+
28 rows in set (0.00 sec)

mysql> quit

* 해당 작업 후 에러 제거되고 정상 메시지만 출력. 근데 이 서버는 리플리케이션 제거 되고 단독으로 도는 서버.. mysql_upgrade 명령어를 하고 아래 에러 들이 출력 된것 보니 리플리케이션 관련 내용들이 없어서 뜬것...

0 Comments

CEPH

CEPH MGMT 재설치 및 MON 추가 feat. librados: client.bootstrap-mds authentication error (1) Operation not permitted

By 때찌때찌맴매 - 8월 02, 2018

CEPH MGMT 재설치 및 MON 추가 feat. librados: client.bootstrap-mds authentication error (1) Operation not permitted

* ceph 장애 메시지를 확인 해보니 ceph컨트롤 서버인 mgmt 서버가 다운됨.

확인 결과 mgmt 서버 디스크 초기화;;; ???? 어떻게 하면 초기화가..

os 자체가 날라갔기 때문에 데이터는 복구 할 수 없고 새로운 ssd os설치 및 mgmt 서버 복구 작업 진행.

* ceph 구성은 centos7 에서 진행하고, 새로운 키값들을 이용하여 복원을 진행해도 되지만, 현재 상품 운영중인 서버들이기 때문에 함부로 변경하기 보다는 기존에 있던 셋팅 값을 유지 하기로합니다.

* 참고 사이트 : http://docs.ceph.com/docs/mimic/rados/operations/add-or-rm-mons/?highlight=removing

* 주의 : 필자는 MGMT 서버에 MGMT,MDS,MON-2 세가지 용도로 사용합니다. 서버 비용 절감으로 인함이니 서버를 각각 사용하신 분들은 잘 필터해서 작업 진행 하시길 바랍니다.

[ 장애발생 ]

i.ceph 관련 7대의 서버중 mgmt 서버 다운.

[root@mon-0 ceph]# ceph -s
    cluster 0651cfa0-8ce4-4fbd-b35b-966e3d262851
     health HEALTH_WARN
            1 mons down, quorum 1,2 mon-0,mon-1
     monmap e5: 3 mons at {mgmt=192.168.1.11:6789/0,mon-0=192.168.1.12:6789/0,mon-1=192.168.1.13:6789/0}
            election epoch 58, quorum 1,2 mon-0,mon-1
     osdmap e68: 4 osds: 4 up, 4 in
            flags sortbitwise,require_jewel_osds
      pgmap v124967: 128 pgs, 1 pools, 64 bytes data, 9 objects
            149 MB used, 59556 GB / 59556 GB avail
                 128 active+clean

[ mgmt 서버 제거 ]

i. REMOVING MONITORS
============================================================================================
# ceph mon remove {mon-id}
============================================================================================

* mon 서버에 추가된 mgmt 서버 제거
* 해당 작업은 mgmt 서버가 아닌 mon 서버에서 진행합니다.

[root@mon-0 ceph]# ceph mon remove mgmt
removing mon.mgmt at 192.168.1.11:6789/0, there will be 2 monitors


[root@mon-0 ceph]# ceph -s
    cluster 0651cfa0-8ce4-4fbd-b35b-966e3d262851
     health HEALTH_OK
     monmap e6: 2 mons at {mon-0=192.168.1.12:6789/0,mon-1=192.168.1.13:6789/0}
            election epoch 60, quorum 0,1 mon-0,mon-1
     osdmap e68: 4 osds: 4 up, 4 in
            flags sortbitwise,require_jewel_osds
      pgmap v124974: 128 pgs, 1 pools, 64 bytes data, 9 objects
            149 MB used, 59556 GB / 59556 GB avail
                 128 active+clean


[root@mon-0 ceph]# ceph mon dump

dumped monmap epoch 6
epoch 6
fsid 0651cfa0-8ce4-4fbd-b35b-966e3d262851
last_changed 2018-08-01 13:37:51.939485
created 2018-04-18 14:24:34.356583
0: 192.168.1.12:6789/0 mon.mon-0
1: 192.168.1.13:6789/0 mon.mon-1


* ceph mon dump 로 확인 하면 현재 사용중인  mon 서버들이 나오는데, 제거 명령어 후 mgmt 서버 쪽은 빠짐.

[ os 기본구성 ]

i. os 설치, yum.repo 추가

국내 yum repo
https://hoguinside.blogspot.com/2017/11/yum.html


ii. ceph 기본구성

https://hoguinside.blogspot.com/2018/04/ceph-1.html

* yum update 해주세요. ceph-deploy 설치시 낮은 버젼으로 설치 되면 ceph 사이트 못찾아서  install 부터 장애가 발생됩니다.

[ MGMT 서버 복구 ]

i. MGMT 서버 복구 진행

[root@mgmt01:05:03:~]# yum install ceph-deploy
[root@mgmt01:05:37:~]# rpm -qa | grep ceph-deploy
ceph-deploy-1.5.39-0.noarch


[root@mgmt01:06:08:~]# ceph-deploy new mgmt
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy new mgmt
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  func                          : 
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : 
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['mgmt']
[ceph_deploy.cli][INFO  ]  public_network                : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds


[root@mgmt01:11:19:~]# ceph-deploy install mgmt
.
.
[mgmt][DEBUG ] Complete!
[mgmt][INFO  ] Running command: ceph --version
[mgmt][DEBUG ] ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)


[root@mgmt01:20:00:~]# ceph-deploy mon create-initial

* ceph instsall 및 mon 초기화를 진행 하면 bootstrap,admin.keyring 들이 생성이됨

[root@mgmt01:19:53:~]# ll | grep ceph
합계 128

-rw-r--r--   1 root root 32367  8월  1 13:19 ceph-deploy-ceph.log
-rw-------   1 root root   113  8월  1 13:19 ceph.bootstrap-mds.keyring
-rw-------   1 root root    71  8월  1 13:19 ceph.bootstrap-mgr.keyring
-rw-------   1 root root   113  8월  1 13:19 ceph.bootstrap-osd.keyring
-rw-------   1 root root   113  8월  1 13:19 ceph.bootstrap-rgw.keyring
-rw-------   1 root root   129  8월  1 13:19 ceph.client.admin.keyring
-rw-r--r--   1 root root   193  8월  1 13:19 ceph.conf
-rw-------   1 root root    73  8월  1 13:19 ceph.mon.keyring


* 해당 위치에 있는 파일들은 배포용이고, 실직적으로 import되어 있는 파일들은  /var/lib/ceph/ 위치에 있음.

[root@mgmt01:23:02:~]# ll /var/lib/ceph/
합계 4
drwxr-x---  10 ceph ceph  120  8월  1 13:15 .
drwxr-xr-x. 34 root root 4096  8월  1 13:15 ..
drwxr-x---   2 ceph ceph   25  8월  1 13:19 bootstrap-mds
drwxr-x---   2 ceph ceph   25  8월  1 13:19 bootstrap-osd
drwxr-x---   2 ceph ceph   25  8월  1 13:19 bootstrap-rgw
drwxr-x---   2 ceph ceph    6  7월 10 02:03 mds
drwxr-x---   3 ceph ceph   22  8월  1 13:19 mon
drwxr-x---   2 ceph ceph    6  7월 10 02:03 osd
drwxr-xr-x   2 root root    6  7월 10 02:03 radosgw


* conf 파일과 admin.keyring 들을 생성 했으면 기존의 conf 파일 내용으로 수정함
  ( 기존 conf 파일들은 mon-0, mon-1 등에도 있고, 필자의 구성대로 ceph를 설치 했었다면 다른 서버에서도 "ceph auth list" 명령어를 이용하여 keyring 값 확인 가능 )
  mgmt 를 mon 서버로 등록한 이유는 ceph는 안전빵으로 3대를 기본 구성으로 진행 했기 때문에 mgmt 도 mon 3번째 서버로 등록 했고, 또한 mds 도 같이 사용했음. ( 단독으로 mon, mds 를 쓰기에는 비용이...)

* ceph.conf 와 ceph.client.admin.keyring 의 key 값을 변경 했으면 복사.

[root@mgmt02:58:03:~]# ceph-deploy --overwrite-conf admin mgmt

또는 

[root@mgmt03:18:11:~]# \cp -arp ceph.conf ceph.client.admin.keyring /etc/ceph/

[root@mgmt03:39:25:~]# ceph-deploy mon create mgmt


[root@mgmt03:39:59:~]# ceph -s
2018-08-01 15:40:00.522889 7f983d142700  0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
[root@mgmt03:40:00:~]# ceph -s
    cluster 0651cfa0-8ce4-4fbd-b35b-966e3d262851
     health HEALTH_OK
     monmap e16: 2 mons at {mon-0=192.168.1.12:6789/0,mon-1=192.168.1.13:6789/0}
            election epoch 80, quorum 0,1 mon-0,mon-1
     osdmap e68: 4 osds: 4 up, 4 in
            flags sortbitwise,require_jewel_osds
      pgmap v125010: 128 pgs, 1 pools, 64 bytes data, 9 objects
            149 MB used, 59556 GB / 59556 GB avail
                 128 active+clean

* 확인 시 첫 ceph -s 했을때 ERROR 가 나온 이유는 conf 파일에 mgmt 서버의 ip가 입력 되어 있어거 최초에 mgmt  mon 쪽으로 접근했기 때문.  설정 값은 넣었지만 상태 값에는 해당 정보가 없고, /var/lib/ceph 디렉토리에 있는 bootstrap등 keyring 파일의 key값이 일치 하지 않아 생기는 현상. 실제로 ceph auth list 로 확인 되는 값들과  /var/lib/ceph 밑에 있는 파일들의 값이 다르기 때문임.

* 아직 정상은 아니지만 기존 정보들 확인되기 때문에 일단 넘어감.

[ mgmt mon 서버 추가. ]

i. ADDING MONITORS
============================================================================================
# mv /var/lib/ceph/mon/ceph-{mon-id} /var/lib/ceph/mon/ceph-{mon-id}_back

# mkdir /var/lib/ceph/mon/ceph-{mon-id}_back/tmp

# monmaptool --create --add {mon-id} {ip:port} --clobber tmp/monmap

# ceph auth get mon. -o /var/lib/ceph/mon/ceph-{mon-id}_back/tmp/{key-filename}

# ceph mon getmap -o /var/lib/ceph/mon/ceph-{mon-id}_back/tmp/{map-filename}

# ceph-mon -i {mon-id} --mkfs --monmap {tmp}/{map-filename} --keyring {tmp}/{key-filename}

# ceph-mon -i {mon-id} --public-addr {ip:port}
============================================================================================


[root@mgmt01:23:02:~]# mv /var/lib/ceph/mon/ceph-mgmt /var/lib/ceph/mon/ceph-mgmt_back

[root@mgmt01:59:31:~]# cd /var/lib/ceph/mon/ceph-mgmt_back

[root@mgmt01:59:31:/var/lib/ceph/mon/ceph-mgmt_back]# mkdir tmp

[root@mgmt02:31:43:/var/lib/ceph/mon/ceph-mgmt_back]# monmaptool --create --add mgmt 192.168.1.11:6789 --clobber tmp/monmap
monmaptool: monmap file tmp/monmap
monmaptool: generated fsid d8f0c51c-7314-4900-8bf6-41bc750152f1
monmaptool: writing epoch 0 to tmp/monmap (1 monitors)

[root@mgmt02:09:55:/var/lib/ceph/mon/ceph-mgmt_back]# ceph auth get mon. -o tmp/keyring
2018-08-01 14:10:12.096006 7f3414367700  0 -- :/363631262 >> 192.168.1.11:6789/0 pipe(0x7f34100663c0 sd=6 :0 s=1 pgs=0 cs=0 l=1 c=0x7f341005ea30).fault
exported keyring for mon.

[root@mgmt02:10:15:/var/lib/ceph/mon/ceph-mgmt_back]# ceph mon getmap -o tmp/monmap
got monmap epoch 6

[root@mgmt02:10:46:/var/lib/ceph/mon/ceph-mgmt_back]# ceph-mon -i mgmt --mkfs --monmap tmp/monmap --keyring tmp/keyring 
ceph-mon: set fsid to 0651cfa0-8ce4-4fbd-b35b-966e3d262851
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-mgmt for mon.mgmt

[root@mgmt02:11:17:/var/lib/ceph/mon/ceph-mgmt_back]# chown -R ceph.ceph ../ceph-mgmt

[root@mgmt05:00:24:/var/lib/ceph/mon/ceph-mgmt_back]# ceph-mon -i mgmt --public-addr 192.168.1.11:6789


[root@mgmt02:11:17:/var/lib/ceph/mon/ceph-mgmt_back]# ll /var/lib/ceph/mon/
합계 0
drwxr-x---  4 ceph ceph  43  8월  1 14:11 .
drwxr-x--- 10 ceph ceph 120  8월  1 13:15 ..
drwxr-xr-x  3 ceph ceph  35  8월  1 14:11 ceph-mgmt
drwxr-xr-x  4 ceph ceph  70  8월  1 13:49 ceph-mgmt_back



[root@mgmt03:00:47:/var/lib/ceph/mon/ceph-mgmt_back]# ceph mon add mgmt 192.168.1.11:6789
adding mon.mgmt at 192.168.1.11:3789/0

[root@mgmt12:55:45:~]# ceph mon dump
dumped monmap epoch 25
epoch 25
fsid 0651cfa0-8ce4-4fbd-b35b-966e3d262851
last_changed 2018-08-01 17:26:48.158918
created 2018-04-18 14:24:34.356583
0: 192.168.1.11:6789/0 mon.mgmt
1: 192.168.1.12:6789/0 mon.mon-0
2: 192.168.1.13:6789/0 mon.mon-1

[root@mgmt05:01:07:/var/lib/ceph/mon/ceph-mgmt_back]# ceph -s
    cluster 0651cfa0-8ce4-4fbd-b35b-966e3d262851
     health HEALTH_OK
     monmap e23: 3 mons at {mgmt=192.168.1.11:6789/0,mon-0=192.168.1.12:6789/0,mon-1=192.168.1.13:6789/0}
            election epoch 100, quorum 0,1,2 mgmt,mon-0,mon-1
     osdmap e68: 4 osds: 4 up, 4 in
            flags sortbitwise,require_jewel_osds
      pgmap v125025: 128 pgs, 1 pools, 64 bytes data, 9 objects
            149 MB used, 59556 GB / 59556 GB avail
                 128 active+clean

* ceph health 와 mon dump 로  확인 하면 mgmt mon을 포함한 3개의 mon 서버가 구동중인걸 확인할 수 있음.

[ MDS 구동 ]

[root@mgmt03:00:47:/var/lib/ceph/mon/ceph-mgmt_back]# cd /root
[root@mgmt03:23:07:~]# ceph-deploy gatherkeys mgmt
.
.
.
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.client.admin.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] Replacing 'ceph.bootstrap-mds.keyring' and backing up old key as 'ceph.bootstrap-mds.keyring-20180801170437'
[ceph_deploy.gatherkeys][INFO  ] Replacing 'ceph.bootstrap-mgr.keyring' and backing up old key as 'ceph.bootstrap-mgr.keyring-20180801170437'
[ceph_deploy.gatherkeys][INFO  ] Replacing 'ceph.mon.keyring' and backing up old key as 'ceph.mon.keyring-20180801170437'
[ceph_deploy.gatherkeys][INFO  ] Replacing 'ceph.bootstrap-osd.keyring' and backing up old key as 'ceph.bootstrap-osd.keyring-20180801170437'
[ceph_deploy.gatherkeys][INFO  ] Replacing 'ceph.bootstrap-rgw.keyring' and backing up old key as 'ceph.bootstrap-rgw.keyring-20180801170437'
[ceph_deploy.gatherkeys][INFO  ] Destroy temp directory /tmp/tmp2dpdwU

[root@mgmt05:15:26:~]# cat ceph.bootstrap-mds.keyring > /var/lib/ceph/bootstrap-mds/ceph.keyring 
[root@mgmt05:16:19:~]# cat ceph.bootstrap-osd.keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring 
[root@mgmt05:16:32:~]# cat ceph.bootstrap-rgw.keyring > /var/lib/ceph/bootstrap-rgw/ceph.keyring 

* ceph.bootstrap-mgr  키값은 /var/lib/ceph/  위치에 사용중이면 넣고 아니면 무시.

* gatherkeys 명령어는 ceph auth list 명령어로 확인 하면 나오는 키값을 bootstrap들의 keyring 파일로 배포를 합니다. mgmt 에서는 파일이 재생성 되고, mon 서버들은 /var/lib/ceph 밑에는 있는 keyring 에 입력이 됩니다. 고로 mgmt 에서는 위 작업을 진행해야 됩니다.

[root@mgmt05:29:39:~]# ceph-deploy mds create mgmt
.
.
[mgmt][INFO  ] Running command: systemctl enable ceph-mds@mgmt
[mgmt][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@mgmt.service to /usr/lib/systemd/system/ceph-mds@.service.
[mgmt][INFO  ] Running command: systemctl start ceph-mds@mgmt
[mgmt][INFO  ] Running command: systemctl enable ceph.target

[root@mgmt05:17:31:~]# netstat -nlpt | grep ceph
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address    Foreign Address    State       PID/Program name 
tcp        0      0 0.0.0.0:6800     0.0.0.0:*          LISTEN      10388/ceph-mds
tcp        0      0 192.168.1.11:6789 0.0.0.0:*          LISTEN      9619/ceph-mon

[ MDS 구동시 에러 ]

[mgmt][DEBUG ] create path if it doesn't exist
[mgmt][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.mgmt osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-mgmt/keyring
[mgmt][ERROR ] 2018-08-01 11:13:16.378595 7fe472685700  0 librados: client.bootstrap-mds authentication error (1) Operation not permitted
[mgmt][ERROR ] Error connecting to cluster: PermissionError
[mgmt][ERROR ] exit code from command was: 1
[ceph_deploy.mds][ERROR ] could not create mds

* 위와 같은 권한 에러는 대부분 keyring 파일의 key값이 일치 하지 않기 때문에 발생함. ceph auth list 의 키값과, /var/lib/ceph/ 아래 디렉토리들의 키값이 일치 해야됨

* 발생 이유는 전 명령어인 ceph-deploy gatherkeys mgmt 를 하지 않았기 때문.

* 모든 작업이 끝나고 결국 서버를 전부 분리 할까... 하다가 비용땜에.. 그만 둡니다..

0 Comments

LINUX

shell script 암호화시키기

By 때찌때찌맴매 - 7월 11, 2018

shell script 암호화시키기

* 리눅스에서 스크립트 작성 후 내용을 암호화 시켜야 하는 상황이 발생하였습니다.

암호화가 필요한 시점에 사용하면 되겠습니다.

[ shc 다운및 설치]

ex) ./shc -r -v -T -f {"스크립트파일"}

[root@localhost]# wget http://www.datsi.fi.upm.es/~frosal/sources/shc-3.8.7.tgz
[root@localhost]# tar zxvf shc-3.8.7.tgz 
shc-3.8.7/CHANGES
shc-3.8.7/Copying
shc-3.8.7/Makefile
shc-3.8.7/match
shc-3.8.7/pru.sh
shc-3.8.7/shc-3.8.7.c
shc-3.8.7/shc.1
shc-3.8.7/shc.README
shc-3.8.7/shc.c
shc-3.8.7/shc.html
shc-3.8.7/test.bash
shc-3.8.7/test.csh
shc-3.8.7/test.ksh
[root@localhost]# cd shc-3.8.7
[root@localhost]# make
cc -Wall -O6  shc.c -o shc
*** �Do you want to probe shc with a test script?
*** Please try... make test

[ 스크립트 암호화 ]

[root@localhost]# ./shc -r -v -T -f ../dceph 
shc shll=bash
shc [-i]=-c
shc [-x]=exec '%s' "$@"
shc [-l]=
shc opts=
shc: cc  ../dceph.x.c -o ../dceph.x
shc: strip ../dceph.x
shc: chmod go-r ../dceph.x

[root@localhost]# ll ../
합계 92
drwxr-xr-x   4 root root  4096 2018-07-11 12:07 .
drwxr-xr-x. 11 root root  4096 2018-07-11 12:00 ..
drwxr-xr-x   2 root root  4096 2018-07-11 12:05 shc-3.8.7
-rw-r--r--   1 root root 20498 2018-07-11 12:05 shc-3.8.7.tgz
-rwxr-xr-x   1 root root   948 2018-07-11 12:03 uceph
-rwx--x--x   1 root root 12688 2018-07-11 12:07 uceph.x
-rw-r--r--   1 root root 14413 2018-07-11 12:07 uceph.x.c

* 암호화 시킨 uceph.x 파일을 확작장 제거후 하용하시면 됩니다. vi 를 열게 되면 암호화된 파일을 볼 수 있습니다.

[ 참고사항 ]
i. {"스크립트파일"}.x 라고 생성된 파일이 바이너리 파일입니다.
ii. 스크립트 상단에 반드시 #!/bin/bash 가 있어야 변환됩니다.
iii. 스크립트 구동 가능 날짜 제한
# shc -e 11/07/2018 -m "The end" -f {"스크립트파일"}
// 2018년 11월 07일 이후에는 "The end" 라는 메세지가 출력되며 구동이 되지 않습니다.

0 Comments

HoguInside & ㅁ ㅇ ㄷ nail

IT. POST LIST

mysql InnoDB: Error: Table "mysql"."innodb_table_stats" not found.

mysql - InnoDB: Error: Table "mysql"."innodb_table_stats" not found.

CEPH MGMT 재설치 및 MON 추가 feat. librados: client.bootstrap-mds authentication error (1) Operation not permitted

CEPH MGMT 재설치 및 MON 추가 feat. librados: client.bootstrap-mds authentication error (1) Operation not permitted

shell script 암호화시키기

shell script 암호화시키기

About me

VIEW

cloud

Interested user

Blog Archive

Follow Us

LABEL

#ㅁㅇㄷ Nail by nailist Young