我是小葵
Toggle navigation
Home
Archives
Tags
ceph-objectstore-tool工具使用详解
2022-11-28 09:05:59
84
0
0
admin
简介 ceph-objectstore-tool工具，能够操作到ceph最底层的数据，包括pg,对象层级。它能够对底层pg以及对象相关数据进行获取、修改。并能够对一些问题pg和对象进行简单修复。所以使用该工具进行操作的时候需要谨慎（涉及到修改的操作最好备份一份数据），防止误操作造成数据丢失。 该工具的使用主要针对对象和PG,所以这里我们的使用主要为以下两种 PG的相关操作 info #查看pg的信息 log #查看pg的元数据信息 remove #从当前osd移除pg mkfs #重新初始化osd fsck #检查bluefs数据正确性 export #导出pg元数据 import #导入pg元数据 list #列出pg内部对象或者当前osd所有对象 fix-lost #修复pg丢失的对象 list-pgs #列出当前osd所有pg rm-past-intervals dump-journal #针对filestore，打印日志 dump-super #打印osd的超级块信息 meta-list #打印元数据信息列表 get-osdmap #获取osdmap set-osdmap #设置osd map get-inc-osdmap #从当前osd 获取inc 信息 set-inc-osdmap #将inc设置进入当前osd信息 mark-complete #标记为complete，让bluestore认为可以pg可以选举出权威日志 对象的相关操作,这里主要列出常用的对象操作 list-attrs #列出对象的一些属性 list-omap #列出对象的omp信息 remove|removeall #移除对象或者移除所有对象 dump #打印对象元数据信息 使用 基本命令使用如下： ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore xxx一般前半部分如左，指定osd路径，指定存储引擎（bluestore或者filestore） PS：使用之前需要停止当前操作的osd，否则会报错 OSD相关操作 检查bluestore的bluefs是否文件系统被损坏，并尝试修复 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –op fsck 列出当前osd所有对象 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –op list 获取当前osd的osdmap信息，并指定对应的输出文件 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –op get-osdmap –file 1.txt [root@node1 ceph]# ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –op get-osdmap –file 1.txt osdmap#1457 exported. 导入一个osdmap信息到当前osd ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –op set-osdmap –file 1.txt [root@node1 ceph]# ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –op set-osdmap –file 1.txt Wrote osdmap.145 获取当前osd的超级块信息 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –op dump-super 输出如下： { “cluster_fsid”: “fa27f041-0ce9-4df1-a4bd-5e37678834bd”, “osd_fsid”: “c03c2fdb-cfd4-42bd-8e32-61e359281078”, “whoami”: 8, “current_epoch”: 1457, “oldest_map”: 784, “newest_map”: 1457, “weight”: 0.000000, “compat”: { “compat”: {}, “ro_compat”: {}, “incompat”: { “feature_1”: “initial feature set(~v.18)”, “feature_2”: “pginfo object”, “feature_3”: “object locator”, “feature_4”: “last_epoch_clean”, “feature_5”: “categories”, “feature_6”: “hobjectpool”, “feature_7”: “biginfo”, “feature_8”: “leveldbinfo”, “feature_9”: “leveldblog”, “feature_10”: “snapmapper”, “feature_11”: “sharded objects”, “feature_12”: “transaction hints”, “feature_13”: “pg meta object”, “feature_14”: “explicit missing set”, “feature_15”: “fastinfo pg attr”, “feature_16”: “deletes in missing set” } }, “clean_thru”: 1457, “last_epoch_mounted”: 1456 } 获取当前osd的所有pg ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –op list-pgs 其中–op表示对当前osd或者当前pg的操作 17.es1 17.as0 17.8s1 17.4s2 17.3s1 17.3fs0 17.3ds0 17.3cs2 17.23s2 17.2es0 PG相关操作 针对单个pg，列出当前pg的基本信息 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –pgid 17.es1 –op info输出如下 “pgid”: “17.es1”, “last_update”: “1380’5116”, “last_complete”: “1380’5116”, “log_tail”: “1376’4854”, “last_user_version”: 5118, “last_backfill”: “MAX”, “last_backfill_bitwise”: 0, “purged_snaps”: [], “history”: { “epoch_created”: 1290, “epoch_pool_created”: 1290, “last_epoch_started”: 1457, “last_interval_started”: 1456, “last_epoch_clean”: 1350, “last_interval_clean”: 1349, “last_epoch_split”: 0, “last_epoch_marked_full”: 0, “same_up_since”: 1456, “same_interval_since”: 1456, “same_primary_since”: 1290, “last_scrub”: “0’0”, “last_scrub_stamp”: “2019-08-12 11:14:04.515869”, “last_deep_scrub”: “0’0”, “last_deep_scrub_stamp”: “2019-08-12 11:14:04.515869”, “last_clean_scrub_stamp”: “2019-08-12 11:14:04.515869” … 针对单个pg，列出当前pg元数据信息 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –pgid 17.es1 –op log 该命令的输出主要包括pg的历史版本信息，记录着每个对象的相关操作，以及当时pg epoch和对应的pg小版本 “pg_log_t”: { “head”: “1380’5116”, “tail”: “1376’4854”, “log”: [ { “op”: “modify”, “object”: “17:713bbf48:::rbd_data.18.cf83c74b0dc51.0000000000023ba0:head”, “version”: “1376’4855”, “prior_version”: “0’0”, “reqid”: “client.852676.0:164001”, “extra_reqids”: [], “mtime”: “2019-08-12 14:40:08.571459”, “return_code”: 0, “mod_desc”: { “object_mod_desc”: { “can_local_rollback”: true, “rollback_info_completed”: true, “ops”: [ { “code”: “CREATE” } ] } } }, … 打印pg中对象的元数据 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –pgid 17.es1 –op meta-list [“meta”,{“oid”:”osdmap.1372″,”key”:””,”snapid”:0,”hash”:168878088,”max”:0,”pool”:-1,”namespace”:””,”max”:0}] [“meta”,{“oid”:”osdmap.1053″,”key”:””,”snapid”:0,”hash”:168945672,”max”:0,”pool”:-1,”namespace”:””,”max”:0}] [“meta”,{“oid”:”osdmap.1266″,”key”:””,”snapid”:0,”hash”:168892424,”max”:0,”pool”:-1,”namespace”:””,”max”:0}] [“meta”,{“oid”:”osdmap.1101″,”key”:””,”snapid”:0,”hash”:168920072,”max”:0,”pool”:-1,”namespace”:””,”max”:0}] … 将pg的元数据导出到指定文件，一般用作 备份 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –pgid 17.es1 –op export –file /mnt/test.obj 将指定的pg元数据导入到当前pg 这里导入前需要先移除当前pg（remove之前请先export备份一下pg数据） ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –pgid 17.es1 –op remove ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –pgid 17.es1 –op import –file /mnt/test.obj 将集群中incomplete的pg标记为complete，这里需要根据集群的副本数将相关的所有osd上的pg进行标记 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –pgid 17.es1 –op mark-complete 对象相关操作 列出当前pg的所有对象 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –pgid 17.es1 –op list 输出如下： [“17.es1”,{“oid”:”rbd_data.18.cf83c74b0dc51.000000000001bda5″,”key”:””,”snapid”:-2,”hash”:1725693966,”max”:0,”pool”:17,”namespace”:””,”shard_id”:1,”max”:0}] [“17.es1”,{“oid”:”rbd_data.18.cf83c74b0dc51.0000000000033137″,”key”:””,”snapid”:-2,”hash”:1782448142,”max”:0,”pool”:17,”namespace”:””,”shard_id”:1,”max”:0}] [“17.es1”,{“oid”:”rbd_data.18.cf83c74b0dc51.000000000003f128″,”key”:””,”snapid”:-2,”hash”:1246789646,”max”:0,”pool”:17,”namespace”:””,”shard_id”:1,”max”:0 … … 从以上获取的对象中取一个，打印对象的元数据信息 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a dump { “id”: { “oid”: “rbd_data.18.cf83c74b0dc51.000000000003925a”, “key”: “”, “snapid”: -2, “hash”: 1962672078, “max”: 0, “pool”: 17, “namespace”: “”, “shard_id”: 1, “max”: 0 }, “info”: { “oid”: { “oid”: “rbd_data.18.cf83c74b0dc51.000000000003925a”, “key”: “”, “snapid”: -2, “hash”: 1962672078, “max”: 0, “pool”: 17, “namespace”: “” }, … 获取对象的属性信息 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore [root@node1 ceph]# ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a list-attrs _ hinfo_key snapset 删除对象 ceph-objectstore-tool –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a remove请慎重测试，如有pg备份，则可以进行测试 [root@node1 ceph]# ceph-objectstore-tool-bak –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a remove remove 1#17:73ffdf2e:::rbd_data.18.cf83c74b0dc51.000000000003925a:head# 查看pg中是否还有该对象 ceph-objectstore-tool-bak –data-path /var/lib/ceph/osd/ceph-8/ –type bluestore –pgid 17.es1 –op list|grep rbd_data.18.cf83c74b0dc51.000000000003925a并无输出 总结 由于该工具是对Ceph底层数据以及元数据进行操作，所以使用需谨慎，处处要备份。如有兴趣，可以对该工具的具体操作源码进行阅读 src/tools/ceph_objectstore_tool.cc路径，修改对应代码之后需要同时将libceph-common.so.0以及libceph-common.so.0拷贝到测试设备才能够正常使用
Pre:
Black Hat Go 之 端口扫描
Next:
时间戳引起的网站访问不了的问题(net.ipv4.tcp_timestamps)
0
likes
84
Weibo
Wechat
Tencent Weibo
QQ Zone
RenRen
Table of content
admin