Ceph/cephadm/memo をテンプレートにして作成
[
トップ
] [
新規
|
一覧
|
検索
|
最終更新
|
ヘルプ
|
ログイン
]
開始行:
***/var/log/secureに「pam_systemd(sudo:session): Failed t...
こんな出力
#code(nonumber){{
Aug 31 05:03:23 ceph01 sudo[4739]: ceph : PWD=/ ; USER...
Aug 31 05:03:23 ceph01 sudo[4739]: pam_systemd(sudo:sessi...
Aug 31 05:03:23 ceph01 sudo[4739]: pam_unix(sudo:session)...
Aug 31 05:03:23 ceph01 sudo[4739]: pam_unix(sudo:session)...
}}
これはユーザ「ceph」が cron, systemd, あるいは 非対話シェ...
PAMが反応して記録を残している模様. 無視しても構わない. ど...
#code(nonumber){{
[root@ceph-osd1 ~]# authselect current ...
[root@ceph-osd1 ~]# authselect create-profile custom-pam ...
[root@ceph-osd1 ~]# authselect select custom/custom-pam -...
[root@ceph-osd1 ~]# ls -l /etc/authselect/custom/custom-p...
:
-rw-r--r--. 1 root root 2355 Aug 31 05:42 smartcard-auth
-rw-r--r--. 1 root root 4778 Aug 31 05:42 system-auth
:
[root@ceph-osd1 ~]# vi /etc/authselect/custom/custom-pam/...
:
-session optional ...
↓
#-session optional ...
:
[root@ceph-osd1 ~]#
[root@ceph-osd1 ~]# authselect apply-changes <-- 変更を...
}}
これでログに記載されなくなります
***「ceph -s」 で「xx pgs not dee-scrubbed in time」「xx ...
PG（Placement Group）に対して定期的に行われる scrub / dee...
そもそも scrub とは複数ノードに分散したメタデータの整合性...
#code(nonumber){{
ceph health detail
}}
は残っている scrub 対象のリストがでます。遅いなら早くすれ...
#code(nonumber){{
ceph config set osd osd_max_scrubs 5 <-- 「ceph config...
}}
あるPGのscrubが終わって、osd_scrub_max_interval の時間が...
当然 rebalancing / recovery / backfill が起こっていれば s...
***ceph-19.2.3から libcrypto.so.3(OPENSSL_3.4.0)(64bit) ...
「libcrypto.so.3(OPENSSL_3.5.0)(64bit)」の提供は rockylin...
作るしかない
#code(nonumber){{
dnf install perl-Digest-SHA perl-FindBin perl-IPC-Cmd per...
wget https://mirror.stream.centos.org/9-stream/BaseOS/sou...
rpmbuild --rebuild ./openssl-3.5.1-2.el9.src.rpm
}}
できた
openssl-libs-3.5.1-2.el9.x86_64.rpm
openssl-fips-provider-3.5.1-2.el9.x86_64.rpm
openssl-3.5.1-2.el9.x86_64.rpm
を一斉にインストールします
***古いcephを入れるには [#x4d733a9]
「dnf --showduplicates search」で古いバージョンを見つけて...
#code(nonumber){{
[root@rockylinux9 ~]# dnf install centos-release-ceph-squ...
[root@rockylinux9 ~]# dnf --showduplicates search ceph-co...
Last metadata expiration check: 0:00:30 ago on Fri 12 Sep...
============================== Name Exactly Matched: ceph...
ceph-common-2:19.2.0-1.el9s.x86_64 : Ceph Common
ceph-common-2:19.2.0-2.el9s.x86_64 : Ceph Common
ceph-common-2:19.2.1-1.el9s.x86_64 : Ceph Common
ceph-common-2:19.2.2-1.el9s.x86_64 : Ceph Common
ceph-common-2:19.2.3-1.el9s.x86_64 : Ceph Common
================================== Name Matched: ceph-com...
python3-ceph-common-2:19.2.0-1.el9s.x86_64 : Python 3 uti...
python3-ceph-common-2:19.2.0-2.el9s.x86_64 : Python 3 uti...
python3-ceph-common-2:19.2.1-1.el9s.x86_64 : Python 3 uti...
python3-ceph-common-2:19.2.2-1.el9s.x86_64 : Python 3 uti...
python3-ceph-common-2:19.2.3-1.el9s.x86_64 : Python 3 uti...
[root@rockylinux9 ~]#
}}
と表示される。ここで 19.2.1 の「ceph-common」を入れるなら
#code(nonumber){{
[root@rockylinux9 ~]# dnf install ceph-common-2:19.2.1-1....
}}
とする。
***journal で流れるログを抑制したい [#e13dffb0]
各mgrデーモンのログ制御は「debug_mgr」、monデーモンは「de...
フォーマットは「ceph config set mon debug_mon <stderr>/<f...
#code(nonumber){{
[root@ceph01 ~]# ceph config set global debug_mgr 0/0
[root@ceph01 ~]# ceph config set global debug_mon 0/0
[root@ceph01 ~]# ceph config set global debug_osd 0/0
[root@ceph01 ~]# ceph config set global debug_bluestore 0/0
[root@ceph01 ~]# ceph config set global debug_rocksdb 0/0
[root@ceph01 ~]# ceph config set global debug_client 0/0
[root@ceph01 ~]# ceph config set global debug_rados 0/0
}}
確認は「ceph config get mon debug_mon」「ceph config get ...
「global」を「osd.2」とか特定でも構わない.
出力先は３つ用意されて ceph のファイル、journal、messages...
#code(nonumber){{
[root@ceph01 ~]# ceph config set global log_to_file false...
[root@ceph01 ~]# ceph config set global log_to_journald f...
[root@ceph01 ~]# ceph config set global log_to_stderr fal...
}}
上記は各ノードで働く各デーモンについて。
ceph全体のclusterログは「mon_cluster_log_to_level 」で定...
#code(nonumber){{
[root@ceph01 ~]# ceph config set global mon_cluster_log_l...
[root@ceph01 ~]# ceph config set global mon_cluster_log_t...
[root@ceph01 ~]# ceph config set global mon_cluster_log_t...
[root@ceph01 ~]# ceph config set global mon_cluster_log_t...
}}
***ワーニングメッセージをクリアする [#i0df438b]
問題が既に対処済みであるならクリアする
#code(nonumber){{
[root@ceph01 ~]# ceph -s
:
health: HEALTH_WARN
1 mgr modules have recently crashed
:
[root@ceph01 ~]# ceph health detail
HEALTH_WARN 1 mgr modules have recently crashed
[WRN] RECENT_MGR_MODULE_CRASH: 1 mgr modules have recentl...
mgr module cephadm crashed in daemon mgr.ceph01.cammq...
[root@ceph01 ~]# ceph crash ls
ID ...
2025-05-06T08:48:40.736559Z_4ad1da23-f2ea-4a4d-9b11-f68bd...
[root@ceph01 ~]#
[root@ceph01 ~]# ceph crash rm 2025-05-06T08:48:40.736559...
[root@ceph01 ~]# ceph -s
:
health: HEALTH_OK
:
[root@ceph01 ~]#
}}
***指定したcpu数でコンテナが動くのか？ [#mf8c84c0]
monitorを「"--cpus=4"」としたが本当に機能しているか確かめ...
ここでは「"--cpus=2"」としてmonitorを作って、そのなかに入...
#code(nonumber){{
[root@ceph01 ~]# vi mon.yaml
:
extra_container_args:
- "--cpus=2"
[root@ceph01 ~]# ceph orch apply -i mon.yaml
[root@ceph01 ~]# podman container inspect <CONTAINER ID> ...
:
"CpuPeriod": 100000, -> 「--cpus」の...
"CpuQuota": 200000,
:
[root@ceph01 ~]# podman ps
CONTAINER ID IMAGE ...
:
a47e3a21247c quay.io/ceph/ceph@sha256:8214ebff6133ac27d2...
[root@ceph01 ~]# podman exec -u root -it a47e3a21247c /...
[root@ceph01 /]#
[root@ceph01 /]# (timeout 15s yes > /dev/null & ) ; (time...
}}
大本のceph01のtopからは 各yesコマンドのプロセスが50%で動...
&ref(2025y05m06d_145347457.png,nolink);
***cephの運用形態 [#o3149398]
#code(nonumber){{
[root@ceph01 ~]# ceph orch status
Backend: cephadm
Available: Yes
Paused: No
[root@ceph01 ~]#
}}
***BlueStore [#a58718b8]
[[https://docs.ceph.com/en/reef/rados/configuration/blues...
cephのストレージデバイスの定義方法.
メインのHDD/SSDストーレジ(プライマリ デバイス)に１つまた...
-DBデバイス(block.db)
メタデータを保持します. メタデータが溢れたらプライマリ デ...
-WALデバイス(block.wal)
先行書き込みログ. ジャーナルまたは先行書き込みログを別の...
DBデバイスのサイズはプライマリ デバイスの1%-4%ほどで. cep...
DBデバイスとWALデバイスを１つのデバイスに割り当てても構わ...
&ref(2025y03m02d_120944083.png,nolink,noborder);
***ディスククォータ [#sa422c23]
いわゆるストレージの利用制限.
[[https://docs.ceph.com/en/latest/cephfs/quota/>+https://...
xfs/ext4なら uid/gid/project とかでそのファイルし済む全体...
「/emfs/user/ABC」に10TiBの制限を掛けたければ「setfattr -...
解除はサイズを０にする。「setfattr -n ceph.quota.max_byte...
ファイル数はceph.quota.max_bytes の代わりに ceph.quota.ma...
***cephを停止して、再開したらワーニングが出た [#j7d2c0d8]
様々なワーニングが発生するかと思いますが、基本dashboardに...
デーモンが起動してなくて起こるワーニング
#code(nonumber){{
[root@ceph01 ~]# ceph health detail
HEALTH_WARN 3 failed cephadm daemon(s); insufficient stan...
[WRN] CEPHADM_FAILED_DAEMON: 3 failed cephadm daemon(s)
daemon mgr.ceph01.ocyoth on ceph01 is in error state
daemon ceph-exporter.ceph01 on ceph01 is in error state
daemon mds.emfs.ceph01.etimbe on ceph01 is in error s...
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS ...
have 0; want 1 more
[root@ceph01 ~]# ceph -s
:
health: HEALTH_WARN
3 failed cephadm daemon(s)
insufficient standby MDS daemons available
:
[root@ceph01 ~]#
}}
この場合はceph health detailのデーモンを起動せよとdashboa...
#code(nonumber){{
[root@ceph01 ~]# ceph orch daemon start mgr.ceph01.ocyoth
[root@ceph01 ~]# ceph orch daemon start ceph-exporter.cep...
[root@ceph01 ~]# ceph orch daemon start mds.emfs.ceph01.e...
}}
で対処
***定義されているパラメータ [#ce291982]
#code(nonumber){{
[root@ceph01 ~]# ceph config dump
WHO MASK LEVEL OPTION ...
global advanced cluster_network ...
global basic container_image ...
mon advanced auth_allow_insecure_g...
mon advanced mon_allow_pool_delete...
mon advanced public_network ...
mgr advanced mgr/cephadm/container...
mgr advanced mgr/cephadm/migration...
mgr advanced mgr/dashboard/ALERTMA...
mgr advanced mgr/dashboard/GRAFANA...
mgr advanced mgr/dashboard/GRAFANA...
mgr advanced mgr/dashboard/PROMETH...
mgr advanced mgr/dashboard/ssl_ser...
mgr advanced mgr/orchestrator/orch...
osd host:ceph-osd1 basic osd_memory_target ...
osd host:ceph-osd2 basic osd_memory_target ...
osd host:ceph-osd3 basic osd_memory_target ...
osd advanced osd_memory_target_aut...
osd.0 basic osd_mclock_max_capaci...
osd.1 basic osd_mclock_max_capaci...
osd.6 basic osd_mclock_max_capaci...
mds.emfs basic mds_join_fs ...
[root@ceph01 ~]#
[root@ceph01 ~]# ceph config get osd osd_op_num_shards_hdd
1
[root@ceph01 ~]# ceph config get osd osd_op_num_threads_p...
5
[root@ceph01 ~]#
}}
***osdのHHDのキャッシュを停止する [#mba10185]
[[https://docs.ceph.com/en/reef/start/hardware-recommenda...
#code(nonumber){{
dnf install smartmontools hdparm
hdparm -W0 /dev/sda ...
echo "write through" > /sys/class/scsi_disk/0\:0\:0\:0/ca...
}}
「hdparm -W0 /dev/sda」で書き込みキャッシュは無効になるが...
***osdのDBサイズを確認する [#xceaa92d]
ノードに入ってcephadmを入れる
#code(nonumber){{
[root@ceph-osd1 ~]# dnf install centos-release-ceph-squid...
[root@ceph-osd1 ~]# dnf install cephadm python3-jinja2 ...
[root@ceph-osd1 ~]# cephadm shell
[ceph: root@ceph-osd1 /]# ceph daemon osd.0 perf dump | ...
{
"db_total_bytes": 53682888704,
"db_used_bytes": 27656192,
"wal_total_bytes": 0,
"wal_used_bytes": 0,
"slow_total_bytes": 0,
"slow_used_bytes": 0,
:
[ceph: root@ceph-osd1 /]#
}}
***windowsと接続 [#hb324811]
基本[[Ceph/windows]]でいいのかなと思いますが、sambaで提供...
っでここでは[[Ceph/cephadm]]で mgr を担っている ceph01 に...
クライアント接続に既に用意した「client.emfs」を外だしします
#code(nonumber){{
[root@ceph01 ~]# ceph auth get client.emfs | tee /etc/cep...
[client.emfs]
key = AQDTBbJnouWUBxAAhy6T8A6w8nTxcqDZrAaUhA==
caps mds = "allow rwp fsname=emfs"
caps mon = "allow r fsname=emfs"
caps osd = "allow rw tag cephfs data=emfs"
[root@ceph01 ~]# ls -l /etc/ceph/
total 20
-rw-------. 1 root root 151 Feb 17 01:50 ceph.client.admi...
-rw-r--r--. 1 root root 176 Feb 19 00:56 ceph.client.emfs...
-rw-r--r--. 1 root root 271 Feb 17 01:50 ceph.conf
-rw-r--r--. 1 root root 595 Feb 15 04:54 ceph.pub
-rw-r--r--. 1 root root 92 Feb 7 23:15 rbdmap
[root@ceph01 ~]#
}}
その上で
#code(nonumber){{
[root@ceph01 ~]# mount -t ceph emfs@.emfs=/ /emfs
[root@ceph01 ~]# df -Tht ceph
Filesystem Type Si...
emfs@4ce725c6-eb0d-11ef-980a-bc24112ffd94.emfs=/ ceph 25...
[root@ceph01 ~]#
}}
あとはsmbを入れて
#code(nonumber){{
[root@ceph01 ~]# dnf install samba
[root@ceph01 ~]# vi /etc/samba/smb.conf
[global]
workgroup = SAMBA
security = user
passdb backend = tdbsam
[emfs]
path = /emfs
browseable = yes
read only = No
inherit acls = Yes
[root@ceph01 ~]# systemctl enable smb --now
[root@ceph01 ~]# firewall-cmd --add-service=samba --zone=...
[root@ceph01 ~]# firewall-cmd --reload
(selinuxが有効なら)
[root@ceph01 ~]# dnf install policycoreutils-python-utils
[root@ceph01 ~]# semanage fcontext -a -t samba_share_t "/...
[root@ceph01 ~]# restorecon -R -v /emfs
(aclを付与して使えるようにします)
[root@ceph01 ~]# setfacl -m user:saber:rwx /emfs
[root@ceph01 ~]# setfacl -dm user:saber:rwx /emfs
}}
***OSDのHDDが怪しい..事前交換する(cephadmの場合) [#la51bb...
smartの値とかで微妙なHDDが存在したとする. これを抜き取っ...
まず微妙はHDDをoutにします
#code(nonumber){{
[root@ceph01 ~]# ceph osd out osd.3
marked out osd.3.
[root@ceph01 ~]#
(暫くして)
[root@ceph01 ~]# ceph osd safe-to-destroy osd.3
OSD(s) 3 are safe to destroy without reducing data durabi...
[root@ceph01 ~]# ceph osd tree
:
2 hdd 0.04880 osd.2 up 1.00000 ...
3 hdd 0.04880 osd.3 up 0 ...
:
[root@ceph01 ~]#
}}
とosd.3を外しても問題ないと確認してから、osd.3を外してdae...
#code(nonumber){{
[root@ceph01 ~]# ceph osd out osd.3
osd.3 is already out.
[root@ceph01 ~]# ceph orch daemon stop osd.3
Scheduled to stop osd.3 on host 'ceph-osd2'
(osd.3を持つceph-osd2では)
[root@ceph-osd2 ~]# podman ps
CONTAINER ID IMAGE ...
:
8021d6fd6825 quay.io/ceph/ceph@sha256:41d3f5e46ff7de2854...
20fd50080cdc quay.io/ceph/ceph@sha256:41d3f5e46ff7de2854...
28137df48c7e quay.io/ceph/ceph@sha256:41d3f5e46ff7de2854...
↓
8021d6fd6825 quay.io/ceph/ceph@sha256:41d3f5e46ff7de2854...
28137df48c7e quay.io/ceph/ceph@sha256:41d3f5e46ff7de2854...
[root@ceph-osd2 ~]#
}}
そうしてceph-osd2で抜き取ったHDDと同じ場所に新しいHDDを植...
#code(nonumber){{
[root@ceph01 ~]# ceph orch osd rm osd.3 --replace
[root@ceph01 ~]# ceph orch osd rm status
No OSD remove/replace operations reported
[root@ceph01 ~]# ceph device ls-lights <-- HDDスロ...
[root@ceph01 ~]# ceph device ls <-- HDDのデ...
}}
物理的にHDDを交換して
#code(nonumber){{
[root@ceph01 ~]# ceph orch daemon add osd ceph-osd2:/dev/...
}}
***OSDのHDDが飛んだ..(cephadmの場合) [#q063336c]
検証環境は Proxmox です. OSDのホスト ceph-osd3 のHDD(data...
当然ながら
#code(nonumber){{
[root@ceph01 ~]# ceph -s
cluster:
id: 4ce725c6-eb0d-11ef-980a-bc24112ffd94
health: HEALTH_WARN
1 failed cephadm daemon(s)
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph-osd1 (age 2h)
mgr: ceph01.ocyoth(active, since 2h), standbys: ceph0...
mds: 1/1 daemons up, 1 standby
osd: 8 osds: 7 up (since 35m), 7 in (since 26m)
data:
volumes: 1/1 healthy
pools: 3 pools, 66 pgs
objects: 24 objects, 587 KiB
usage: 604 MiB used, 349 GiB / 350 GiB avail
pgs: 66 active+clean
[root@ceph01 ~]#
}}
と警告になる.
どこか問題なのかは
#code(nonumber){{
[root@ceph01 ~]# ceph osd tree down
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT ...
-1 0.39038 root default
-7 0.14639 host ceph-osd3
1 hdd 0.04880 osd.1 down 0 ...
[root@ceph01 ~]# ceph orch ps --service_name osd --sort_b...
NAME HOST PORTS STATUS REFRESHED AGE M...
osd.1 ceph-osd3 error 5m ago 41h ...
osd.4 ceph-osd1 running (2h) 5m ago 43h ...
osd.6 ceph-osd1 running (2h) 5m ago 40h ...
:
[root@ceph01 ~]#
}}
で見れて、ceph-osd3の osd.1 が問題と.
このHDDを交換してみる.
まずは交換対象の osd.1 のデータが他に回るように手配します
#code(nonumber){{
[root@ceph01 ~]# ceph osd out 1
osd.1 is already out.
[root@ceph01 ~]#
}}
と既に他に回っているようなので停止させ、HDD交換手続きを行...
#code(nonumber){{
[root@ceph01 ~]# ceph orch daemon stop osd.1
Scheduled to stop osd.1 on host 'ceph-osd3'
[root@ceph01 ~]# ceph orch osd rm 1 --replace
Scheduled OSD(s) for removal.
VG/LV for the OSDs won't be zapped (--zap wasn't passed).
Run the `ceph-volume lvm zap` command with `--destroy` ag...
[root@ceph01 ~]#
}}
&color(red){*}; 注意： block.dbやwalを使ってosdを構成して...
なので交換したHDDを組み込むには block.db 、wal 領域を再作...
実体としてHDDを交換して
#code(nonumber){{
[root@ceph-osd3 ~]# lsblk
NAME ...
(略
mqceph--465e1cab--73d2--4083--95ff--b33bed556c53-osd--blo...
sdc ...
mqceph--19239b39--3cd8--4024--91f5--f42239c0e257-osd--blo...
sdd ...
[root@ceph-osd3 ~]#
(block.dbとwalを作り直す)
lvcreate -n sdc-db -l 100%FREE ceph-66205364-530c-4132-a1...
lvcreate -n sdc-wal -l 100%FREE ceph
その上で
ceph orch daemon add osd ceph-osd3:/dev/sdc,db_devices=/d...
}}
ここでは追加した /dev/sdd をcephに加える
#code(nonumber){{
[root@ceph01 ~]# ceph orch daemon add osd ceph-osd3:/dev/...
[root@ceph01 ~]# ceph osd tree
:
-7 0.14639 host ceph-osd3
0 hdd 0.04880 osd.0 up 1.000...
1 hdd 0.04880 osd.1 destroyed ...
5 ssd 0.04880 osd.5 up 1.000...
[root@ceph01 ~]#
(暫くすると)
[root@ceph01 ~]# ceph osd tree
:
-7 0.14639 host ceph-osd3
0 hdd 0.04880 osd.0 up 1.00000 ...
1 hdd 0.04880 osd.1 up 1.00000 ...
5 ssd 0.04880 osd.5 up 1.00000 ...
[root@ceph01 ~]#
}}
と回復する.
***ceph接続 clientパッケージを作る [#z560055c]
Squid(19.x)のrockylinux8向けパッケージが用意されていない....
#code(nonumber){{
[root@rockylinux8 ~]# cat /etc/redhat-release
Rocky Linux release 8.10 (Green Obsidian)
[root@rockylinux8 ~]# dnf install epel-release -y
[root@rockylinux8 ~]# dnf --enablerepo=devel install CUni...
libaio-devel libatomic libbabeltrace-devel libblkid-devel...
libicu-devel libnl3-devel liboath-devel librabbitmq-devel...
lua-devel lz4-devel nasm ncurses-devel ninja-build nss-de...
python3-sphinx re2-devel selinux-policy-devel snappy-deve...
gcc-toolset-11 gcc-toolset-11-build gcc-toolset-11-gcc-c+...
[root@rockylinux8 ~]# dnf install -y python3-jenkins
[root@rockylinux8 ~]# wget https://download.ceph.com/rpm-...
[root@rockylinux8 ~]# rpmbuild --rebuild ceph-19.2.1-0.el...
:
(紅茶とケーキの時間)
:
[root@rockylinux8 ~]#
}}
構築が終わったらceph-commonを入れます. 関連パッケージが沢...
&size(10){下記の実行で「/usr/bin/ceph」「/usr/sbin/mount....
#code(nonumber){{
[root@rockylinux8 ~]# cd rpmbuild/RPMS/x86_64
[root@rockylinux8 x86_64]# dnf localinstall \
ceph-common-19.2.1-0.el8.x86_64.rpm \
python3-ceph-argparse-19.2.1-0.el8.x86_64.rpm \
python3-ceph-common-19.2.1-0.el8.x86_64.rpm \
python3-cephfs-19.2.1-0.el8.x86_64.rpm \
python3-rados-19.2.1-0.el8.x86_64.rpm \
python3-rbd-19.2.1-0.el8.x86_64.rpm \
python3-rgw-19.2.1-0.el8.x86_64.rpm \
libcephfs2-19.2.1-0.el8.x86_64.rpm \
librados2-19.2.1-0.el8.x86_64.rpm \
libradosstriper1-19.2.1-0.el8.x86_64.rpm \
librbd1-19.2.1-0.el8.x86_64.rpm \
librgw2-19.2.1-0.el8.x86_64.rpm
[root@rockylinux8 x86_64]#
}}
***ファイルシステムの削除 [#q8fb9245]
現状確認、「ceph fs ls」でファイルシステム名称と使用して...
#code(nonumber){{
[root@ceph01 ~]# ceph fs ls
name: emfs, metadata pool: emfs-meta, data pools: [emfs-d...
[root@ceph01 ~]# ceph fs status
emfs - 0 clients
====
RANK STATE MDS ACTIVITY DN...
0 active cephfs.ceph-osd2.tjxksw Reqs: 0 /s 1...
POOL TYPE USED AVAIL
emfs-meta metadata 96.0k 47.4G
emfs-data data 0 252G
STANDBY MDS
cephfs.ceph01.ejtdsq
MDS version: ceph version 19.2.1 (58a7fab8be0a062d730ad7d...
[root@ceph01 ~]#
}}
STATEが「active」となっている。停止にはまず
#code(nonumber){{
[root@ceph01 ~]# ceph fs set emfs down true
emfs marked down.
[root@ceph01 ~]# ceph fs status
emfs - 0 clients
====
POOL TYPE USED AVAIL
emfs-meta metadata 96.0k 47.4G
emfs-data data 0 252G
STANDBY MDS
cephfs.ceph01.ejtdsq
cephfs.ceph-osd2.tjxksw
MDS version: ceph version 19.2.1 (58a7fab8be0a062d730ad7d...
[root@ceph01 ~]# ceph fs rm emfs --yes-i-really-mean-it
}}
と停止させる. っがまだmdsが残っているのでこれも削除/停止...
#code(nonumber){{
[root@ceph01 ~]# ceph orch ps --daemon_type=mds
NAME HOST PORTS STATUS REFR...
mds.emfs.ceph01.magcmf ceph01 running (9m) 4...
mds.emfs.ceph02.nuleix ceph02 running (9m) 9...
[root@ceph01 ~]# ceph orch rm mds.emfs
[root@ceph01 ~]# ceph orch ps --daemon_type=mds
No daemons reported
[root@ceph01 ~]#
}}
これで削除は完了です.
***cephadmで既に割り当てている サービスLAN 側のipアドレス...
&color(red){未完成};
[[https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm...
具体的には
|BGCOLOR(YELLOW):ホスト名|BGCOLOR(YELLOW):役目|BGCOLOR(YE...
|ceph01 |&color(crimson){Ceph Manager}; &br;&c...
|ceph02 |&color(crimson){Ceph Manager}; &br;&c...
|ceph-osd1 |&color(orangered){Ceph OSDs}; &br;...
|ceph-osd2 |&color(orangered){Ceph OSDs}; ...
|ceph-osd3 |&color(orangered){Ceph OSDs}; ...
ってな感じに.
#code(nonumber){{
[root@ceph01 ~]# ceph osd set noout
[root@ceph01 ~]# ceph osd set norecover
[root@ceph01 ~]# ceph osd set norebalance
[root@ceph01 ~]# ceph osd set nobackfill
[root@ceph01 ~]# ceph osd set nodown
[root@ceph01 ~]# ceph osd set pause
}}
#code(nonumber){{
[root@ceph-osd3 ~]# systemctl stop ceph.target
[root@ceph-osd3 ~]# systemctl disable ceph.target
[root@ceph-osd2 ~]# systemctl stop ceph.target
[root@ceph-osd2 ~]# systemctl disable ceph.target
[root@ceph-osd1 ~]# systemctl stop ceph.target
[root@ceph-osd1 ~]# systemctl disable ceph.target
[root@ceph02 ~]# systemctl stop ceph.target
[root@ceph02 ~]# systemctl disable ceph.target
[root@ceph01 ~]# systemctl stop ceph.target
[root@ceph01 ~]# systemctl disable ceph.target
}}
この後、各ノードをshutdownして移設して新規ネットワーク(17...
っでup.
***コンテナ毎にcpu数を定義するには [#xbe5f00e]
[[https://docs.ceph.com/en/reef/cephadm/services/#extra-c...
キーワード：cephadm osd "extra_container_args" cpus
***メモ [#a1496024]
#code(nonumber){{
[root@ceph01 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 192.168.0.47 _admin
ceph02 192.168.0.48
ceph-osd1 192.168.0.49
ceph-osd2 192.168.0.50
ceph-osd3 192.168.0.51
5 hosts in cluster
[root@ceph01 ~]# ceph osd lspools
1 .mgr
[root@ceph01 ~]# ceph config set mon mon_allow_pool_dele...
[root@ceph01 ~]# ceph osd pool delete .mgr .mgr --yes-i-r...
[root@ceph01 ~]# ceph osd tree
[root@ceph01 ~]# ceph orch osd rm 4 --zap --force
[root@ceph01 ~]# ceph orch osd rm status (OSD ...
[root@ceph01 ~]# ceph orch ps (ceph...
[root@ceph01 ~]# ceph orch host drain ceph-osd3 (ceph...
[root@ceph01 ~]# ceph orch ps ceph-osd3 (conta...
[root@ceph01 ~]# ceph orch host rm ceph-osd3 (ceph-...
[root@ceph01 ~]# ceph orch host drain ceph-osd2
[root@ceph01 ~]# ceph orch ps ceph-osd2
[root@ceph01 ~]# ceph orch host rm ceph-osd2
[root@ceph01 ~]# ceph orch apply mon --placement="ceph01"...
[root@ceph01 ~]# ceph orch apply mon --placement="ceph01"...
[root@ceph01 ~]# ceph orch host drain ceph-osd1
[root@ceph01 ~]# ceph orch ps ceph-osd1
[root@ceph01 ~]# ceph orch host rm ceph-osd1
[root@ceph01 ~]# ceph orch apply mgr --placement="ceph01"
[root@ceph01 ~]# ceph orch host drain ceph02
[root@ceph01 ~]# ceph orch daemon rm mon.ceph02 --force
[root@ceph01 ~]# ceph orch host rm ceph02
}}
「cephadm bootstrap」の後に取り消して無効にするには
#code(nonumber){{
[root@ceph01 ~]# systemctl list-unit-files |grep ceph
var-lib-ceph.mount ...
ceph-9ab38ad2-e6a9-11ef-9ba5-bc24112ffd94@.service ...
ceph-9ab38ad2-e6a9-11ef-9ba5-bc24112ffd94.target ...
ceph.target ...
[root@ceph01 ~]# systemctl disable ceph-9ab38ad2-e6a9-11e...
[root@ceph01 ~]# reboot
[root@ceph01 ~]# ls -l /etc/systemd/system/ceph*
-rw-r--r--. 1 root root 1181 Feb 9 14:51 /etc/systemd/sy...
-rw-r--r--. 1 root root 157 Feb 9 14:51 /etc/systemd/sy...
-rw-r--r--. 1 root root 88 Feb 9 15:18 /etc/systemd/sy...
[root@ceph01 ~]# rm -rf /etc/systemd/system/ceph*
[root@ceph01 ~]# rm -rf /var/lib/ceph/* /etc/ceph/*
}}
これで「cephadm bootstrap」がチャラになります.
***コンテナ毎にcpu数を定義するには [#xbe5f00e]
[[https://docs.ceph.com/en/reef/cephadm/services/#extra-c...
キーワード：cephadm osd "extra_container_args" cpus
***メモ [#a1496024]
#code(nonumber){{
[root@ceph01 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 192.168.0.47 _admin
ceph02 192.168.0.48
ceph-osd1 192.168.0.49
ceph-osd2 192.168.0.50
ceph-osd3 192.168.0.51
5 hosts in cluster
[root@ceph01 ~]# ceph osd lspools
1 .mgr
[root@ceph01 ~]# ceph config set mon mon_allow_pool_dele...
[root@ceph01 ~]# ceph osd pool delete .mgr .mgr --yes-i-r...
[root@ceph01 ~]# ceph osd tree
[root@ceph01 ~]# ceph orch osd rm 4 --zap --force
[root@ceph01 ~]# ceph orch osd rm status (OSD ...
[root@ceph01 ~]# ceph orch ps (ceph...
[root@ceph01 ~]# ceph orch host drain ceph-osd3 (ceph...
[root@ceph01 ~]# ceph orch ps ceph-osd3 (conta...
[root@ceph01 ~]# ceph orch host rm ceph-osd3 (ceph-...
[root@ceph01 ~]# ceph orch host drain ceph-osd2
[root@ceph01 ~]# ceph orch ps ceph-osd2
[root@ceph01 ~]# ceph orch host rm ceph-osd2
[root@ceph01 ~]# ceph orch apply mon --placement="ceph01"...
[root@ceph01 ~]# ceph orch apply mon --placement="ceph01"...
[root@ceph01 ~]# ceph orch host drain ceph-osd1
[root@ceph01 ~]# ceph orch ps ceph-osd1
[root@ceph01 ~]# ceph orch host rm ceph-osd1
[root@ceph01 ~]# ceph orch apply mgr --placement="ceph01"
[root@ceph01 ~]# ceph orch host drain ceph02
[root@ceph01 ~]# ceph orch daemon rm mon.ceph02 --force
[root@ceph01 ~]# ceph orch host rm ceph02
}}
「cephadm bootstrap」の後に取り消して無効にするには
#code(nonumber){{
[root@ceph01 ~]# systemctl list-unit-files |grep ceph
var-lib-ceph.mount ...
ceph-9ab38ad2-e6a9-11ef-9ba5-bc24112ffd94@.service ...
ceph-9ab38ad2-e6a9-11ef-9ba5-bc24112ffd94.target ...
ceph.target ...
[root@ceph01 ~]# systemctl disable ceph-9ab38ad2-e6a9-11e...
[root@ceph01 ~]# reboot
[root@ceph01 ~]# ls -l /etc/systemd/system/ceph*
-rw-r--r--. 1 root root 1181 Feb 9 14:51 /etc/systemd/sy...
-rw-r--r--. 1 root root 157 Feb 9 14:51 /etc/systemd/sy...
-rw-r--r--. 1 root root 88 Feb 9 15:18 /etc/systemd/sy...
[root@ceph01 ~]# rm -rf /etc/systemd/system/ceph*
[root@ceph01 ~]# rm -rf /var/lib/ceph/* /etc/ceph/*
}}
これで「cephadm bootstrap」がチャラになります.
***参照先 [#eb80e745]
[[https://yourcmc.ru/wiki/Ceph_performance>+https://yourc...
SSDなフラッシュストレージを作るなら、コンシューマー向けの...
&size(10){HDDがdataで、blockdb/walがコンシューマーなSSDで...
[[https://qiita.com/rev4t/items/0f5af561d8621c957bc3>+htt...
終了行:
***/var/log/secureに「pam_systemd(sudo:session): Failed t...
こんな出力
#code(nonumber){{
Aug 31 05:03:23 ceph01 sudo[4739]: ceph : PWD=/ ; USER...
Aug 31 05:03:23 ceph01 sudo[4739]: pam_systemd(sudo:sessi...
Aug 31 05:03:23 ceph01 sudo[4739]: pam_unix(sudo:session)...
Aug 31 05:03:23 ceph01 sudo[4739]: pam_unix(sudo:session)...
}}
これはユーザ「ceph」が cron, systemd, あるいは 非対話シェ...
PAMが反応して記録を残している模様. 無視しても構わない. ど...
#code(nonumber){{
[root@ceph-osd1 ~]# authselect current ...
[root@ceph-osd1 ~]# authselect create-profile custom-pam ...
[root@ceph-osd1 ~]# authselect select custom/custom-pam -...
[root@ceph-osd1 ~]# ls -l /etc/authselect/custom/custom-p...
:
-rw-r--r--. 1 root root 2355 Aug 31 05:42 smartcard-auth
-rw-r--r--. 1 root root 4778 Aug 31 05:42 system-auth
:
[root@ceph-osd1 ~]# vi /etc/authselect/custom/custom-pam/...
:
-session optional ...
↓
#-session optional ...
:
[root@ceph-osd1 ~]#
[root@ceph-osd1 ~]# authselect apply-changes <-- 変更を...
}}
これでログに記載されなくなります
***「ceph -s」 で「xx pgs not dee-scrubbed in time」「xx ...
PG（Placement Group）に対して定期的に行われる scrub / dee...
そもそも scrub とは複数ノードに分散したメタデータの整合性...
#code(nonumber){{
ceph health detail
}}
は残っている scrub 対象のリストがでます。遅いなら早くすれ...
#code(nonumber){{
ceph config set osd osd_max_scrubs 5 <-- 「ceph config...
}}
あるPGのscrubが終わって、osd_scrub_max_interval の時間が...
当然 rebalancing / recovery / backfill が起こっていれば s...
***ceph-19.2.3から libcrypto.so.3(OPENSSL_3.4.0)(64bit) ...
「libcrypto.so.3(OPENSSL_3.5.0)(64bit)」の提供は rockylin...
作るしかない
#code(nonumber){{
dnf install perl-Digest-SHA perl-FindBin perl-IPC-Cmd per...
wget https://mirror.stream.centos.org/9-stream/BaseOS/sou...
rpmbuild --rebuild ./openssl-3.5.1-2.el9.src.rpm
}}
できた
openssl-libs-3.5.1-2.el9.x86_64.rpm
openssl-fips-provider-3.5.1-2.el9.x86_64.rpm
openssl-3.5.1-2.el9.x86_64.rpm
を一斉にインストールします
***古いcephを入れるには [#x4d733a9]
「dnf --showduplicates search」で古いバージョンを見つけて...
#code(nonumber){{
[root@rockylinux9 ~]# dnf install centos-release-ceph-squ...
[root@rockylinux9 ~]# dnf --showduplicates search ceph-co...
Last metadata expiration check: 0:00:30 ago on Fri 12 Sep...
============================== Name Exactly Matched: ceph...
ceph-common-2:19.2.0-1.el9s.x86_64 : Ceph Common
ceph-common-2:19.2.0-2.el9s.x86_64 : Ceph Common
ceph-common-2:19.2.1-1.el9s.x86_64 : Ceph Common
ceph-common-2:19.2.2-1.el9s.x86_64 : Ceph Common
ceph-common-2:19.2.3-1.el9s.x86_64 : Ceph Common
================================== Name Matched: ceph-com...
python3-ceph-common-2:19.2.0-1.el9s.x86_64 : Python 3 uti...
python3-ceph-common-2:19.2.0-2.el9s.x86_64 : Python 3 uti...
python3-ceph-common-2:19.2.1-1.el9s.x86_64 : Python 3 uti...
python3-ceph-common-2:19.2.2-1.el9s.x86_64 : Python 3 uti...
python3-ceph-common-2:19.2.3-1.el9s.x86_64 : Python 3 uti...
[root@rockylinux9 ~]#
}}
と表示される。ここで 19.2.1 の「ceph-common」を入れるなら
#code(nonumber){{
[root@rockylinux9 ~]# dnf install ceph-common-2:19.2.1-1....
}}
とする。
***journal で流れるログを抑制したい [#e13dffb0]
各mgrデーモンのログ制御は「debug_mgr」、monデーモンは「de...
フォーマットは「ceph config set mon debug_mon <stderr>/<f...
#code(nonumber){{
[root@ceph01 ~]# ceph config set global debug_mgr 0/0
[root@ceph01 ~]# ceph config set global debug_mon 0/0
[root@ceph01 ~]# ceph config set global debug_osd 0/0
[root@ceph01 ~]# ceph config set global debug_bluestore 0/0
[root@ceph01 ~]# ceph config set global debug_rocksdb 0/0
[root@ceph01 ~]# ceph config set global debug_client 0/0
[root@ceph01 ~]# ceph config set global debug_rados 0/0
}}
確認は「ceph config get mon debug_mon」「ceph config get ...
「global」を「osd.2」とか特定でも構わない.
出力先は３つ用意されて ceph のファイル、journal、messages...
#code(nonumber){{
[root@ceph01 ~]# ceph config set global log_to_file false...
[root@ceph01 ~]# ceph config set global log_to_journald f...
[root@ceph01 ~]# ceph config set global log_to_stderr fal...
}}
上記は各ノードで働く各デーモンについて。
ceph全体のclusterログは「mon_cluster_log_to_level 」で定...
#code(nonumber){{
[root@ceph01 ~]# ceph config set global mon_cluster_log_l...
[root@ceph01 ~]# ceph config set global mon_cluster_log_t...
[root@ceph01 ~]# ceph config set global mon_cluster_log_t...
[root@ceph01 ~]# ceph config set global mon_cluster_log_t...
}}
***ワーニングメッセージをクリアする [#i0df438b]
問題が既に対処済みであるならクリアする
#code(nonumber){{
[root@ceph01 ~]# ceph -s
:
health: HEALTH_WARN
1 mgr modules have recently crashed
:
[root@ceph01 ~]# ceph health detail
HEALTH_WARN 1 mgr modules have recently crashed
[WRN] RECENT_MGR_MODULE_CRASH: 1 mgr modules have recentl...
mgr module cephadm crashed in daemon mgr.ceph01.cammq...
[root@ceph01 ~]# ceph crash ls
ID ...
2025-05-06T08:48:40.736559Z_4ad1da23-f2ea-4a4d-9b11-f68bd...
[root@ceph01 ~]#
[root@ceph01 ~]# ceph crash rm 2025-05-06T08:48:40.736559...
[root@ceph01 ~]# ceph -s
:
health: HEALTH_OK
:
[root@ceph01 ~]#
}}
***指定したcpu数でコンテナが動くのか？ [#mf8c84c0]
monitorを「"--cpus=4"」としたが本当に機能しているか確かめ...
ここでは「"--cpus=2"」としてmonitorを作って、そのなかに入...
#code(nonumber){{
[root@ceph01 ~]# vi mon.yaml
:
extra_container_args:
- "--cpus=2"
[root@ceph01 ~]# ceph orch apply -i mon.yaml
[root@ceph01 ~]# podman container inspect <CONTAINER ID> ...
:
"CpuPeriod": 100000, -> 「--cpus」の...
"CpuQuota": 200000,
:
[root@ceph01 ~]# podman ps
CONTAINER ID IMAGE ...
:
a47e3a21247c quay.io/ceph/ceph@sha256:8214ebff6133ac27d2...
[root@ceph01 ~]# podman exec -u root -it a47e3a21247c /...
[root@ceph01 /]#
[root@ceph01 /]# (timeout 15s yes > /dev/null & ) ; (time...
}}
大本のceph01のtopからは 各yesコマンドのプロセスが50%で動...
&ref(2025y05m06d_145347457.png,nolink);
***cephの運用形態 [#o3149398]
#code(nonumber){{
[root@ceph01 ~]# ceph orch status
Backend: cephadm
Available: Yes
Paused: No
[root@ceph01 ~]#
}}
***BlueStore [#a58718b8]
[[https://docs.ceph.com/en/reef/rados/configuration/blues...
cephのストレージデバイスの定義方法.
メインのHDD/SSDストーレジ(プライマリ デバイス)に１つまた...
-DBデバイス(block.db)
メタデータを保持します. メタデータが溢れたらプライマリ デ...
-WALデバイス(block.wal)
先行書き込みログ. ジャーナルまたは先行書き込みログを別の...
DBデバイスのサイズはプライマリ デバイスの1%-4%ほどで. cep...
DBデバイスとWALデバイスを１つのデバイスに割り当てても構わ...
&ref(2025y03m02d_120944083.png,nolink,noborder);
***ディスククォータ [#sa422c23]
いわゆるストレージの利用制限.
[[https://docs.ceph.com/en/latest/cephfs/quota/>+https://...
xfs/ext4なら uid/gid/project とかでそのファイルし済む全体...
「/emfs/user/ABC」に10TiBの制限を掛けたければ「setfattr -...
解除はサイズを０にする。「setfattr -n ceph.quota.max_byte...
ファイル数はceph.quota.max_bytes の代わりに ceph.quota.ma...
***cephを停止して、再開したらワーニングが出た [#j7d2c0d8]
様々なワーニングが発生するかと思いますが、基本dashboardに...
デーモンが起動してなくて起こるワーニング
#code(nonumber){{
[root@ceph01 ~]# ceph health detail
HEALTH_WARN 3 failed cephadm daemon(s); insufficient stan...
[WRN] CEPHADM_FAILED_DAEMON: 3 failed cephadm daemon(s)
daemon mgr.ceph01.ocyoth on ceph01 is in error state
daemon ceph-exporter.ceph01 on ceph01 is in error state
daemon mds.emfs.ceph01.etimbe on ceph01 is in error s...
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS ...
have 0; want 1 more
[root@ceph01 ~]# ceph -s
:
health: HEALTH_WARN
3 failed cephadm daemon(s)
insufficient standby MDS daemons available
:
[root@ceph01 ~]#
}}
この場合はceph health detailのデーモンを起動せよとdashboa...
#code(nonumber){{
[root@ceph01 ~]# ceph orch daemon start mgr.ceph01.ocyoth
[root@ceph01 ~]# ceph orch daemon start ceph-exporter.cep...
[root@ceph01 ~]# ceph orch daemon start mds.emfs.ceph01.e...
}}
で対処
***定義されているパラメータ [#ce291982]
#code(nonumber){{
[root@ceph01 ~]# ceph config dump
WHO MASK LEVEL OPTION ...
global advanced cluster_network ...
global basic container_image ...
mon advanced auth_allow_insecure_g...
mon advanced mon_allow_pool_delete...
mon advanced public_network ...
mgr advanced mgr/cephadm/container...
mgr advanced mgr/cephadm/migration...
mgr advanced mgr/dashboard/ALERTMA...
mgr advanced mgr/dashboard/GRAFANA...
mgr advanced mgr/dashboard/GRAFANA...
mgr advanced mgr/dashboard/PROMETH...
mgr advanced mgr/dashboard/ssl_ser...
mgr advanced mgr/orchestrator/orch...
osd host:ceph-osd1 basic osd_memory_target ...
osd host:ceph-osd2 basic osd_memory_target ...
osd host:ceph-osd3 basic osd_memory_target ...
osd advanced osd_memory_target_aut...
osd.0 basic osd_mclock_max_capaci...
osd.1 basic osd_mclock_max_capaci...
osd.6 basic osd_mclock_max_capaci...
mds.emfs basic mds_join_fs ...
[root@ceph01 ~]#
[root@ceph01 ~]# ceph config get osd osd_op_num_shards_hdd
1
[root@ceph01 ~]# ceph config get osd osd_op_num_threads_p...
5
[root@ceph01 ~]#
}}
***osdのHHDのキャッシュを停止する [#mba10185]
[[https://docs.ceph.com/en/reef/start/hardware-recommenda...
#code(nonumber){{
dnf install smartmontools hdparm
hdparm -W0 /dev/sda ...
echo "write through" > /sys/class/scsi_disk/0\:0\:0\:0/ca...
}}
「hdparm -W0 /dev/sda」で書き込みキャッシュは無効になるが...
***osdのDBサイズを確認する [#xceaa92d]
ノードに入ってcephadmを入れる
#code(nonumber){{
[root@ceph-osd1 ~]# dnf install centos-release-ceph-squid...
[root@ceph-osd1 ~]# dnf install cephadm python3-jinja2 ...
[root@ceph-osd1 ~]# cephadm shell
[ceph: root@ceph-osd1 /]# ceph daemon osd.0 perf dump | ...
{
"db_total_bytes": 53682888704,
"db_used_bytes": 27656192,
"wal_total_bytes": 0,
"wal_used_bytes": 0,
"slow_total_bytes": 0,
"slow_used_bytes": 0,
:
[ceph: root@ceph-osd1 /]#
}}
***windowsと接続 [#hb324811]
基本[[Ceph/windows]]でいいのかなと思いますが、sambaで提供...
っでここでは[[Ceph/cephadm]]で mgr を担っている ceph01 に...
クライアント接続に既に用意した「client.emfs」を外だしします
#code(nonumber){{
[root@ceph01 ~]# ceph auth get client.emfs | tee /etc/cep...
[client.emfs]
key = AQDTBbJnouWUBxAAhy6T8A6w8nTxcqDZrAaUhA==
caps mds = "allow rwp fsname=emfs"
caps mon = "allow r fsname=emfs"
caps osd = "allow rw tag cephfs data=emfs"
[root@ceph01 ~]# ls -l /etc/ceph/
total 20
-rw-------. 1 root root 151 Feb 17 01:50 ceph.client.admi...
-rw-r--r--. 1 root root 176 Feb 19 00:56 ceph.client.emfs...
-rw-r--r--. 1 root root 271 Feb 17 01:50 ceph.conf
-rw-r--r--. 1 root root 595 Feb 15 04:54 ceph.pub
-rw-r--r--. 1 root root 92 Feb 7 23:15 rbdmap
[root@ceph01 ~]#
}}
その上で
#code(nonumber){{
[root@ceph01 ~]# mount -t ceph emfs@.emfs=/ /emfs
[root@ceph01 ~]# df -Tht ceph
Filesystem Type Si...
emfs@4ce725c6-eb0d-11ef-980a-bc24112ffd94.emfs=/ ceph 25...
[root@ceph01 ~]#
}}
あとはsmbを入れて
#code(nonumber){{
[root@ceph01 ~]# dnf install samba
[root@ceph01 ~]# vi /etc/samba/smb.conf
[global]
workgroup = SAMBA
security = user
passdb backend = tdbsam
[emfs]
path = /emfs
browseable = yes
read only = No
inherit acls = Yes
[root@ceph01 ~]# systemctl enable smb --now
[root@ceph01 ~]# firewall-cmd --add-service=samba --zone=...
[root@ceph01 ~]# firewall-cmd --reload
(selinuxが有効なら)
[root@ceph01 ~]# dnf install policycoreutils-python-utils
[root@ceph01 ~]# semanage fcontext -a -t samba_share_t "/...
[root@ceph01 ~]# restorecon -R -v /emfs
(aclを付与して使えるようにします)
[root@ceph01 ~]# setfacl -m user:saber:rwx /emfs
[root@ceph01 ~]# setfacl -dm user:saber:rwx /emfs
}}
***OSDのHDDが怪しい..事前交換する(cephadmの場合) [#la51bb...
smartの値とかで微妙なHDDが存在したとする. これを抜き取っ...
まず微妙はHDDをoutにします
#code(nonumber){{
[root@ceph01 ~]# ceph osd out osd.3
marked out osd.3.
[root@ceph01 ~]#
(暫くして)
[root@ceph01 ~]# ceph osd safe-to-destroy osd.3
OSD(s) 3 are safe to destroy without reducing data durabi...
[root@ceph01 ~]# ceph osd tree
:
2 hdd 0.04880 osd.2 up 1.00000 ...
3 hdd 0.04880 osd.3 up 0 ...
:
[root@ceph01 ~]#
}}
とosd.3を外しても問題ないと確認してから、osd.3を外してdae...
#code(nonumber){{
[root@ceph01 ~]# ceph osd out osd.3
osd.3 is already out.
[root@ceph01 ~]# ceph orch daemon stop osd.3
Scheduled to stop osd.3 on host 'ceph-osd2'
(osd.3を持つceph-osd2では)
[root@ceph-osd2 ~]# podman ps
CONTAINER ID IMAGE ...
:
8021d6fd6825 quay.io/ceph/ceph@sha256:41d3f5e46ff7de2854...
20fd50080cdc quay.io/ceph/ceph@sha256:41d3f5e46ff7de2854...
28137df48c7e quay.io/ceph/ceph@sha256:41d3f5e46ff7de2854...
↓
8021d6fd6825 quay.io/ceph/ceph@sha256:41d3f5e46ff7de2854...
28137df48c7e quay.io/ceph/ceph@sha256:41d3f5e46ff7de2854...
[root@ceph-osd2 ~]#
}}
そうしてceph-osd2で抜き取ったHDDと同じ場所に新しいHDDを植...
#code(nonumber){{
[root@ceph01 ~]# ceph orch osd rm osd.3 --replace
[root@ceph01 ~]# ceph orch osd rm status
No OSD remove/replace operations reported
[root@ceph01 ~]# ceph device ls-lights <-- HDDスロ...
[root@ceph01 ~]# ceph device ls <-- HDDのデ...
}}
物理的にHDDを交換して
#code(nonumber){{
[root@ceph01 ~]# ceph orch daemon add osd ceph-osd2:/dev/...
}}
***OSDのHDDが飛んだ..(cephadmの場合) [#q063336c]
検証環境は Proxmox です. OSDのホスト ceph-osd3 のHDD(data...
当然ながら
#code(nonumber){{
[root@ceph01 ~]# ceph -s
cluster:
id: 4ce725c6-eb0d-11ef-980a-bc24112ffd94
health: HEALTH_WARN
1 failed cephadm daemon(s)
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph-osd1 (age 2h)
mgr: ceph01.ocyoth(active, since 2h), standbys: ceph0...
mds: 1/1 daemons up, 1 standby
osd: 8 osds: 7 up (since 35m), 7 in (since 26m)
data:
volumes: 1/1 healthy
pools: 3 pools, 66 pgs
objects: 24 objects, 587 KiB
usage: 604 MiB used, 349 GiB / 350 GiB avail
pgs: 66 active+clean
[root@ceph01 ~]#
}}
と警告になる.
どこか問題なのかは
#code(nonumber){{
[root@ceph01 ~]# ceph osd tree down
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT ...
-1 0.39038 root default
-7 0.14639 host ceph-osd3
1 hdd 0.04880 osd.1 down 0 ...
[root@ceph01 ~]# ceph orch ps --service_name osd --sort_b...
NAME HOST PORTS STATUS REFRESHED AGE M...
osd.1 ceph-osd3 error 5m ago 41h ...
osd.4 ceph-osd1 running (2h) 5m ago 43h ...
osd.6 ceph-osd1 running (2h) 5m ago 40h ...
:
[root@ceph01 ~]#
}}
で見れて、ceph-osd3の osd.1 が問題と.
このHDDを交換してみる.
まずは交換対象の osd.1 のデータが他に回るように手配します
#code(nonumber){{
[root@ceph01 ~]# ceph osd out 1
osd.1 is already out.
[root@ceph01 ~]#
}}
と既に他に回っているようなので停止させ、HDD交換手続きを行...
#code(nonumber){{
[root@ceph01 ~]# ceph orch daemon stop osd.1
Scheduled to stop osd.1 on host 'ceph-osd3'
[root@ceph01 ~]# ceph orch osd rm 1 --replace
Scheduled OSD(s) for removal.
VG/LV for the OSDs won't be zapped (--zap wasn't passed).
Run the `ceph-volume lvm zap` command with `--destroy` ag...
[root@ceph01 ~]#
}}
&color(red){*}; 注意： block.dbやwalを使ってosdを構成して...
なので交換したHDDを組み込むには block.db 、wal 領域を再作...
実体としてHDDを交換して
#code(nonumber){{
[root@ceph-osd3 ~]# lsblk
NAME ...
(略
mqceph--465e1cab--73d2--4083--95ff--b33bed556c53-osd--blo...
sdc ...
mqceph--19239b39--3cd8--4024--91f5--f42239c0e257-osd--blo...
sdd ...
[root@ceph-osd3 ~]#
(block.dbとwalを作り直す)
lvcreate -n sdc-db -l 100%FREE ceph-66205364-530c-4132-a1...
lvcreate -n sdc-wal -l 100%FREE ceph
その上で
ceph orch daemon add osd ceph-osd3:/dev/sdc,db_devices=/d...
}}
ここでは追加した /dev/sdd をcephに加える
#code(nonumber){{
[root@ceph01 ~]# ceph orch daemon add osd ceph-osd3:/dev/...
[root@ceph01 ~]# ceph osd tree
:
-7 0.14639 host ceph-osd3
0 hdd 0.04880 osd.0 up 1.000...
1 hdd 0.04880 osd.1 destroyed ...
5 ssd 0.04880 osd.5 up 1.000...
[root@ceph01 ~]#
(暫くすると)
[root@ceph01 ~]# ceph osd tree
:
-7 0.14639 host ceph-osd3
0 hdd 0.04880 osd.0 up 1.00000 ...
1 hdd 0.04880 osd.1 up 1.00000 ...
5 ssd 0.04880 osd.5 up 1.00000 ...
[root@ceph01 ~]#
}}
と回復する.
***ceph接続 clientパッケージを作る [#z560055c]
Squid(19.x)のrockylinux8向けパッケージが用意されていない....
#code(nonumber){{
[root@rockylinux8 ~]# cat /etc/redhat-release
Rocky Linux release 8.10 (Green Obsidian)
[root@rockylinux8 ~]# dnf install epel-release -y
[root@rockylinux8 ~]# dnf --enablerepo=devel install CUni...
libaio-devel libatomic libbabeltrace-devel libblkid-devel...
libicu-devel libnl3-devel liboath-devel librabbitmq-devel...
lua-devel lz4-devel nasm ncurses-devel ninja-build nss-de...
python3-sphinx re2-devel selinux-policy-devel snappy-deve...
gcc-toolset-11 gcc-toolset-11-build gcc-toolset-11-gcc-c+...
[root@rockylinux8 ~]# dnf install -y python3-jenkins
[root@rockylinux8 ~]# wget https://download.ceph.com/rpm-...
[root@rockylinux8 ~]# rpmbuild --rebuild ceph-19.2.1-0.el...
:
(紅茶とケーキの時間)
:
[root@rockylinux8 ~]#
}}
構築が終わったらceph-commonを入れます. 関連パッケージが沢...
&size(10){下記の実行で「/usr/bin/ceph」「/usr/sbin/mount....
#code(nonumber){{
[root@rockylinux8 ~]# cd rpmbuild/RPMS/x86_64
[root@rockylinux8 x86_64]# dnf localinstall \
ceph-common-19.2.1-0.el8.x86_64.rpm \
python3-ceph-argparse-19.2.1-0.el8.x86_64.rpm \
python3-ceph-common-19.2.1-0.el8.x86_64.rpm \
python3-cephfs-19.2.1-0.el8.x86_64.rpm \
python3-rados-19.2.1-0.el8.x86_64.rpm \
python3-rbd-19.2.1-0.el8.x86_64.rpm \
python3-rgw-19.2.1-0.el8.x86_64.rpm \
libcephfs2-19.2.1-0.el8.x86_64.rpm \
librados2-19.2.1-0.el8.x86_64.rpm \
libradosstriper1-19.2.1-0.el8.x86_64.rpm \
librbd1-19.2.1-0.el8.x86_64.rpm \
librgw2-19.2.1-0.el8.x86_64.rpm
[root@rockylinux8 x86_64]#
}}
***ファイルシステムの削除 [#q8fb9245]
現状確認、「ceph fs ls」でファイルシステム名称と使用して...
#code(nonumber){{
[root@ceph01 ~]# ceph fs ls
name: emfs, metadata pool: emfs-meta, data pools: [emfs-d...
[root@ceph01 ~]# ceph fs status
emfs - 0 clients
====
RANK STATE MDS ACTIVITY DN...
0 active cephfs.ceph-osd2.tjxksw Reqs: 0 /s 1...
POOL TYPE USED AVAIL
emfs-meta metadata 96.0k 47.4G
emfs-data data 0 252G
STANDBY MDS
cephfs.ceph01.ejtdsq
MDS version: ceph version 19.2.1 (58a7fab8be0a062d730ad7d...
[root@ceph01 ~]#
}}
STATEが「active」となっている。停止にはまず
#code(nonumber){{
[root@ceph01 ~]# ceph fs set emfs down true
emfs marked down.
[root@ceph01 ~]# ceph fs status
emfs - 0 clients
====
POOL TYPE USED AVAIL
emfs-meta metadata 96.0k 47.4G
emfs-data data 0 252G
STANDBY MDS
cephfs.ceph01.ejtdsq
cephfs.ceph-osd2.tjxksw
MDS version: ceph version 19.2.1 (58a7fab8be0a062d730ad7d...
[root@ceph01 ~]# ceph fs rm emfs --yes-i-really-mean-it
}}
と停止させる. っがまだmdsが残っているのでこれも削除/停止...
#code(nonumber){{
[root@ceph01 ~]# ceph orch ps --daemon_type=mds
NAME HOST PORTS STATUS REFR...
mds.emfs.ceph01.magcmf ceph01 running (9m) 4...
mds.emfs.ceph02.nuleix ceph02 running (9m) 9...
[root@ceph01 ~]# ceph orch rm mds.emfs
[root@ceph01 ~]# ceph orch ps --daemon_type=mds
No daemons reported
[root@ceph01 ~]#
}}
これで削除は完了です.
***cephadmで既に割り当てている サービスLAN 側のipアドレス...
&color(red){未完成};
[[https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm...
具体的には
|BGCOLOR(YELLOW):ホスト名|BGCOLOR(YELLOW):役目|BGCOLOR(YE...
|ceph01 |&color(crimson){Ceph Manager}; &br;&c...
|ceph02 |&color(crimson){Ceph Manager}; &br;&c...
|ceph-osd1 |&color(orangered){Ceph OSDs}; &br;...
|ceph-osd2 |&color(orangered){Ceph OSDs}; ...
|ceph-osd3 |&color(orangered){Ceph OSDs}; ...
ってな感じに.
#code(nonumber){{
[root@ceph01 ~]# ceph osd set noout
[root@ceph01 ~]# ceph osd set norecover
[root@ceph01 ~]# ceph osd set norebalance
[root@ceph01 ~]# ceph osd set nobackfill
[root@ceph01 ~]# ceph osd set nodown
[root@ceph01 ~]# ceph osd set pause
}}
#code(nonumber){{
[root@ceph-osd3 ~]# systemctl stop ceph.target
[root@ceph-osd3 ~]# systemctl disable ceph.target
[root@ceph-osd2 ~]# systemctl stop ceph.target
[root@ceph-osd2 ~]# systemctl disable ceph.target
[root@ceph-osd1 ~]# systemctl stop ceph.target
[root@ceph-osd1 ~]# systemctl disable ceph.target
[root@ceph02 ~]# systemctl stop ceph.target
[root@ceph02 ~]# systemctl disable ceph.target
[root@ceph01 ~]# systemctl stop ceph.target
[root@ceph01 ~]# systemctl disable ceph.target
}}
この後、各ノードをshutdownして移設して新規ネットワーク(17...
っでup.
***コンテナ毎にcpu数を定義するには [#xbe5f00e]
[[https://docs.ceph.com/en/reef/cephadm/services/#extra-c...
キーワード：cephadm osd "extra_container_args" cpus
***メモ [#a1496024]
#code(nonumber){{
[root@ceph01 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 192.168.0.47 _admin
ceph02 192.168.0.48
ceph-osd1 192.168.0.49
ceph-osd2 192.168.0.50
ceph-osd3 192.168.0.51
5 hosts in cluster
[root@ceph01 ~]# ceph osd lspools
1 .mgr
[root@ceph01 ~]# ceph config set mon mon_allow_pool_dele...
[root@ceph01 ~]# ceph osd pool delete .mgr .mgr --yes-i-r...
[root@ceph01 ~]# ceph osd tree
[root@ceph01 ~]# ceph orch osd rm 4 --zap --force
[root@ceph01 ~]# ceph orch osd rm status (OSD ...
[root@ceph01 ~]# ceph orch ps (ceph...
[root@ceph01 ~]# ceph orch host drain ceph-osd3 (ceph...
[root@ceph01 ~]# ceph orch ps ceph-osd3 (conta...
[root@ceph01 ~]# ceph orch host rm ceph-osd3 (ceph-...
[root@ceph01 ~]# ceph orch host drain ceph-osd2
[root@ceph01 ~]# ceph orch ps ceph-osd2
[root@ceph01 ~]# ceph orch host rm ceph-osd2
[root@ceph01 ~]# ceph orch apply mon --placement="ceph01"...
[root@ceph01 ~]# ceph orch apply mon --placement="ceph01"...
[root@ceph01 ~]# ceph orch host drain ceph-osd1
[root@ceph01 ~]# ceph orch ps ceph-osd1
[root@ceph01 ~]# ceph orch host rm ceph-osd1
[root@ceph01 ~]# ceph orch apply mgr --placement="ceph01"
[root@ceph01 ~]# ceph orch host drain ceph02
[root@ceph01 ~]# ceph orch daemon rm mon.ceph02 --force
[root@ceph01 ~]# ceph orch host rm ceph02
}}
「cephadm bootstrap」の後に取り消して無効にするには
#code(nonumber){{
[root@ceph01 ~]# systemctl list-unit-files |grep ceph
var-lib-ceph.mount ...
ceph-9ab38ad2-e6a9-11ef-9ba5-bc24112ffd94@.service ...
ceph-9ab38ad2-e6a9-11ef-9ba5-bc24112ffd94.target ...
ceph.target ...
[root@ceph01 ~]# systemctl disable ceph-9ab38ad2-e6a9-11e...
[root@ceph01 ~]# reboot
[root@ceph01 ~]# ls -l /etc/systemd/system/ceph*
-rw-r--r--. 1 root root 1181 Feb 9 14:51 /etc/systemd/sy...
-rw-r--r--. 1 root root 157 Feb 9 14:51 /etc/systemd/sy...
-rw-r--r--. 1 root root 88 Feb 9 15:18 /etc/systemd/sy...
[root@ceph01 ~]# rm -rf /etc/systemd/system/ceph*
[root@ceph01 ~]# rm -rf /var/lib/ceph/* /etc/ceph/*
}}
これで「cephadm bootstrap」がチャラになります.
***コンテナ毎にcpu数を定義するには [#xbe5f00e]
[[https://docs.ceph.com/en/reef/cephadm/services/#extra-c...
キーワード：cephadm osd "extra_container_args" cpus
***メモ [#a1496024]
#code(nonumber){{
[root@ceph01 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 192.168.0.47 _admin
ceph02 192.168.0.48
ceph-osd1 192.168.0.49
ceph-osd2 192.168.0.50
ceph-osd3 192.168.0.51
5 hosts in cluster
[root@ceph01 ~]# ceph osd lspools
1 .mgr
[root@ceph01 ~]# ceph config set mon mon_allow_pool_dele...
[root@ceph01 ~]# ceph osd pool delete .mgr .mgr --yes-i-r...
[root@ceph01 ~]# ceph osd tree
[root@ceph01 ~]# ceph orch osd rm 4 --zap --force
[root@ceph01 ~]# ceph orch osd rm status (OSD ...
[root@ceph01 ~]# ceph orch ps (ceph...
[root@ceph01 ~]# ceph orch host drain ceph-osd3 (ceph...
[root@ceph01 ~]# ceph orch ps ceph-osd3 (conta...
[root@ceph01 ~]# ceph orch host rm ceph-osd3 (ceph-...
[root@ceph01 ~]# ceph orch host drain ceph-osd2
[root@ceph01 ~]# ceph orch ps ceph-osd2
[root@ceph01 ~]# ceph orch host rm ceph-osd2
[root@ceph01 ~]# ceph orch apply mon --placement="ceph01"...
[root@ceph01 ~]# ceph orch apply mon --placement="ceph01"...
[root@ceph01 ~]# ceph orch host drain ceph-osd1
[root@ceph01 ~]# ceph orch ps ceph-osd1
[root@ceph01 ~]# ceph orch host rm ceph-osd1
[root@ceph01 ~]# ceph orch apply mgr --placement="ceph01"
[root@ceph01 ~]# ceph orch host drain ceph02
[root@ceph01 ~]# ceph orch daemon rm mon.ceph02 --force
[root@ceph01 ~]# ceph orch host rm ceph02
}}
「cephadm bootstrap」の後に取り消して無効にするには
#code(nonumber){{
[root@ceph01 ~]# systemctl list-unit-files |grep ceph
var-lib-ceph.mount ...
ceph-9ab38ad2-e6a9-11ef-9ba5-bc24112ffd94@.service ...
ceph-9ab38ad2-e6a9-11ef-9ba5-bc24112ffd94.target ...
ceph.target ...
[root@ceph01 ~]# systemctl disable ceph-9ab38ad2-e6a9-11e...
[root@ceph01 ~]# reboot
[root@ceph01 ~]# ls -l /etc/systemd/system/ceph*
-rw-r--r--. 1 root root 1181 Feb 9 14:51 /etc/systemd/sy...
-rw-r--r--. 1 root root 157 Feb 9 14:51 /etc/systemd/sy...
-rw-r--r--. 1 root root 88 Feb 9 15:18 /etc/systemd/sy...
[root@ceph01 ~]# rm -rf /etc/systemd/system/ceph*
[root@ceph01 ~]# rm -rf /var/lib/ceph/* /etc/ceph/*
}}
これで「cephadm bootstrap」がチャラになります.
***参照先 [#eb80e745]
[[https://yourcmc.ru/wiki/Ceph_performance>+https://yourc...
SSDなフラッシュストレージを作るなら、コンシューマー向けの...
&size(10){HDDがdataで、blockdb/walがコンシューマーなSSDで...
[[https://qiita.com/rev4t/items/0f5af561d8621c957bc3>+htt...
ページ名:
1
![[PukiWiki]](image/picc.png "[PukiWiki]")
