ceph-objectstore-tool工具使用详解

简介

ceph-objectstore-tool工具，能够操作到ceph最底层的数据，包括pg,对象层级。它能够对底层pg以及对象相关数据进行获取、修改。并能够对一些问题pg和对象进行简单修复。所以使用该工具进行操作的时候需要谨慎（涉及到修改的操作最好备份一份数据），防止误操作造成数据丢失。
该工具的使用主要针对对象和PG,所以这里我们的使用主要为以下两种

• PG的相关操作

  info #查看pg的信息
  log  #查看pg的元数据信息
  remove #从当前osd移除pg
  mkfs #重新初始化osd
  fsck #检查bluefs数据正确性
  export #导出pg元数据
  import #导入pg元数据
  list #列出pg内部对象或者当前osd所有对象
  fix-lost #修复pg丢失的对象
  list-pgs #列出当前osd所有pg
  rm-past-intervals 
  dump-journal #针对filestore，打印日志
  dump-super #打印osd的超级块信息
  meta-list #打印元数据信息列表
  get-osdmap #获取osdmap
  set-osdmap #设置osd map
  get-inc-osdmap #从当前osd 获取inc 信息
  set-inc-osdmap #将inc设置进入当前osd信息
  mark-complete #标记为complete，让bluestore认为可以pg可以选举出权威日志
  

• 对象的相关操作,这里主要列出常用的对象操作

   list-attrs  #列出对象的一些属性list-omap #列出对象的omp信息remove|removeall #移除对象或者移除所有对象dump #打印对象元数据信息
  

使用

基本命令使用如下：
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore xxx一般前半部分如左，指定osd路径，指定存储引擎（bluestore或者filestore）

PS：使用之前需要停止当前操作的osd，否则会报错

OSD相关操作

• 检查bluestore的bluefs是否文件系统被损坏，并尝试修复
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op fsck

• 列出当前osd所有对象
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op list

• 获取当前osd的osdmap信息，并指定对应的输出文件
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op get-osdmap --file 1.txt

  [root@node1 ceph]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op get-osdmap --file 1.txt
  osdmap#1457 exported.
  

• 导入一个osdmap信息到当前osd
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op set-osdmap --file 1.txt

  [root@node1 ceph]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op set-osdmap --file 1.txt
  Wrote osdmap.145
  

• 获取当前osd的超级块信息
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op dump-super 输出如下：

  {
  "cluster_fsid": "fa27f041-0ce9-4df1-a4bd-5e37678834bd",
  "osd_fsid": "c03c2fdb-cfd4-42bd-8e32-61e359281078",
  "whoami": 8,
  "current_epoch": 1457,
  "oldest_map": 784,
  "newest_map": 1457,
  "weight": 0.000000,
  "compat": {"compat": {},"ro_compat": {},"incompat": {"feature_1": "initial feature set(~v.18)","feature_2": "pginfo object","feature_3": "object locator","feature_4": "last_epoch_clean","feature_5": "categories","feature_6": "hobjectpool","feature_7": "biginfo","feature_8": "leveldbinfo","feature_9": "leveldblog","feature_10": "snapmapper","feature_11": "sharded objects","feature_12": "transaction hints","feature_13": "pg meta object","feature_14": "explicit missing set","feature_15": "fastinfo pg attr","feature_16": "deletes in missing set"}
  },
  "clean_thru": 1457,
  "last_epoch_mounted": 1456
  }
  

• 获取当前osd的所有pg
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op list-pgs 其中--op表示对当前osd或者当前pg的操作

  17.es1
  17.as0
  17.8s1
  17.4s2
  17.3s1
  17.3fs0
  17.3ds0
  17.3cs2
  17.23s2
  17.2es0
  

PG相关操作

• 针对单个pg，列出当前pg的基本信息
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op info输出如下

  "pgid": "17.es1",
  "last_update": "1380'5116",
  "last_complete": "1380'5116",
  "log_tail": "1376'4854",
  "last_user_version": 5118,
  "last_backfill": "MAX",
  "last_backfill_bitwise": 0,
  "purged_snaps": [],
  "history": {"epoch_created": 1290,"epoch_pool_created": 1290,"last_epoch_started": 1457,"last_interval_started": 1456,"last_epoch_clean": 1350,"last_interval_clean": 1349,"last_epoch_split": 0,"last_epoch_marked_full": 0,"same_up_since": 1456,"same_interval_since": 1456,"same_primary_since": 1290,"last_scrub": "0'0","last_scrub_stamp": "2019-08-12 11:14:04.515869","last_deep_scrub": "0'0","last_deep_scrub_stamp": "2019-08-12 11:14:04.515869","last_clean_scrub_stamp": "2019-08-12 11:14:04.515869"...
  

• 针对单个pg，列出当前pg元数据信息
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op log
  该命令的输出主要包括pg的历史版本信息，记录着每个对象的相关操作，以及当时pg epoch和对应的pg小版本

  "pg_log_t": {"head": "1380'5116","tail": "1376'4854","log": [{"op": "modify","object": "17:713bbf48:::rbd_data.18.cf83c74b0dc51.0000000000023ba0:head","version": "1376'4855","prior_version": "0'0","reqid": "client.852676.0:164001","extra_reqids": [],"mtime": "2019-08-12 14:40:08.571459","return_code": 0,"mod_desc": {"object_mod_desc": {"can_local_rollback": true,"rollback_info_completed": true,"ops": [{"code": "CREATE"}]}}},...
  

• 打印pg中对象的元数据
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op meta-list

  ["meta",{"oid":"osdmap.1372","key":"","snapid":0,"hash":168878088,"max":0,"pool":-1,"namespace":"","max":0}]
  ["meta",{"oid":"osdmap.1053","key":"","snapid":0,"hash":168945672,"max":0,"pool":-1,"namespace":"","max":0}]
  ["meta",{"oid":"osdmap.1266","key":"","snapid":0,"hash":168892424,"max":0,"pool":-1,"namespace":"","max":0}]
  ["meta",{"oid":"osdmap.1101","key":"","snapid":0,"hash":168920072,"max":0,"pool":-1,"namespace":"","max":0}]
  ...
  

• 将pg的元数据导出到指定文件，一般用作 备份
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op export --file /mnt/test.obj
• 将指定的pg元数据导入到当前pg
  这里导入前需要先移除当前pg（remove之前请先export备份一下pg数据）
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op remove
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op import --file /mnt/test.obj
• 将集群中incomplete的pg标记为complete，这里需要根据集群的副本数将相关的所有osd上的pg进行标记
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op mark-complete

对象相关操作

• 列出当前pg的所有对象
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op list
  输出如下：

  ["17.es1",{"oid":"rbd_data.18.cf83c74b0dc51.000000000001bda5","key":"","snapid":-2,"hash":1725693966,"max":0,"pool":17,"namespace":"","shard_id":1,"max":0}]
  ["17.es1",{"oid":"rbd_data.18.cf83c74b0dc51.0000000000033137","key":"","snapid":-2,"hash":1782448142,"max":0,"pool":17,"namespace":"","shard_id":1,"max":0}]
  ["17.es1",{"oid":"rbd_data.18.cf83c74b0dc51.000000000003f128","key":"","snapid":-2,"hash":1246789646,"max":0,"pool":17,"namespace":"","shard_id":1,"max":0
  ...
  ...
  

• 从以上获取的对象中取一个，打印对象的元数据信息
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a dump

  {
  "id": {"oid": "rbd_data.18.cf83c74b0dc51.000000000003925a","key": "","snapid": -2,"hash": 1962672078,"max": 0,"pool": 17,"namespace": "","shard_id": 1,"max": 0
  },
  "info": {"oid": {"oid": "rbd_data.18.cf83c74b0dc51.000000000003925a","key": "","snapid": -2,"hash": 1962672078,"max": 0,"pool": 17,"namespace": ""},...
  

  • 获取对象的属性信息
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore

     [root@node1 ceph]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a list-attrs
    _
    hinfo_key
    snapset
    

• 删除对象
  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a remove请慎重测试，如有pg备份，则可以进行测试

  [root@node1 ceph]# ceph-objectstore-tool-bak --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a remove
  remove 1#17:73ffdf2e:::rbd_data.18.cf83c74b0dc51.000000000003925a:head#
  

  查看pg中是否还有该对象
  ceph-objectstore-tool-bak --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op list|grep rbd_data.18.cf83c74b0dc51.000000000003925a并无输出

总结

由于该工具是对Ceph底层数据以及元数据进行操作，所以使用需谨慎，处处要备份。如有兴趣，可以对该工具的具体操作源码进行阅读
src/tools/ceph_objectstore_tool.cc路径，修改对应代码之后需要同时将libceph-common.so.0以及libceph-common.so.0拷贝到测试设备才能够正常使用