「古いファイルサーバが複数ある. これを一つにまとめてNFSサーバにしたい」
と要求が来た.
各ファイルサーバの容量はまちまちなので beeGFS やら ceph、GlusterFSらは厳しいかな.
Gfarmがあるけどあれはファイルシステムが別世界なので面倒.
っで基本に立ち直ってそのファイルサーバをiscsi-targetにして、iscsi-initiatorなるクライアントがLVMなファイルサーバにしてみた.
使うOSはCentOS7
iscsi-target †
「yum grouplist」を見ると "Network File System Client" に「targetcli」がある. まずはこの「tagetcli」でiscsi-targetを作る
[root@iscsi-target ~]# yum install targetcli
[root@iscsi-target ~]# yum info targetcli
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: ftp.nara.wide.ad.jp
* extras: ftp.nara.wide.ad.jp
* updates: ftp.nara.wide.ad.jp
Installed Packages
Name : targetcli
Arch : noarch
Version : 2.1.53
Release : 1.el7_9
Size : 254 k
Repo : installed
From repo : updates
Summary : An administration shell for storage targets
URL : https://github.com/open-iscsi/targetcli-fb
License : ASL 2.0
Description : An administration shell for configuring iSCSI, FCoE, and other
: SCSI targets, using the TCM/LIO kernel target subsystem. FCoE
: users will also need to install and use fcoe-utils.
[root@iscsi-target ~]#
ここではiscsiとして供出するデバイスは /dev/sdb まるごととします。パーティションは切ってないです.
ただ、/dev/sdbと指定するとデバイスが増えた際にIDが変更されてしまう.
なら可変しにくいデバイス名を付けたいもの. ファイルシステムにしてしまえば uuid やら label が使えるが
そのままなんので無理. さりとてLVMにして供出るのもいいのだが、供出した先でLVMでさいど纏うのは微妙...
なので /dev/disk/by-path で指定してみる
[root@iscsi-target ~]# cat /proc/partitions
major minor #blocks name
8 0 16777216 sda
8 1 2048 sda1
8 2 524288 sda2
8 3 1048576 sda3
8 4 1 sda4
8 5 15200256 sda5
8 16 33554432 sdb
11 0 4601856 sr0
[root@iscsi-target ~]#
[root@iscsi-target ~]# ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root 9 Feb 14 21:36 pci-0000:02:03.0-ata-1.0 -> ../../sr0
lrwxrwxrwx. 1 root root 9 Feb 14 21:36 pci-0000:03:00.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx. 1 root root 10 Feb 14 21:36 pci-0000:03:00.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Feb 14 21:36 pci-0000:03:00.0-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx. 1 root root 10 Feb 14 21:36 pci-0000:03:00.0-scsi-0:0:0:0-part3 -> ../../sda3
lrwxrwxrwx. 1 root root 10 Feb 14 21:36 pci-0000:03:00.0-scsi-0:0:0:0-part4 -> ../../sda4
lrwxrwxrwx. 1 root root 10 Feb 14 21:36 pci-0000:03:00.0-scsi-0:0:0:0-part5 -> ../../sda5
lrwxrwxrwx. 1 root root 9 Feb 14 23:03 pci-0000:03:00.0-scsi-0:0:1:0 -> ../../sdb
[root@iscsi-target ~]#
[root@iscsi-target ~]# targetcli
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> ls
o- / ............................................................................. [...]
o- backstores .................................................................. [...]
| o- block ...................................................... [Storage Objects: 0]
| o- fileio ..................................................... [Storage Objects: 0]
| o- pscsi ...................................................... [Storage Objects: 0]
| o- ramdisk .................................................... [Storage Objects: 0]
o- iscsi ................................................................ [Targets: 0]
o- loopback ............................................................. [Targets: 0]
/>
/> cd /backstores/block
/backstores/block> ls
o- block .......................................................... [Storage Objects: 0]
/backstores/block> create name=disk1 dev=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0
Created block storage object disk1 using /dev/sdb.
/backstores/block> ls
o- block .......................................................... [Storage Objects: 1]
o- disk1 [/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0 (32.0GiB) write-thru deactivated]
o- alua ........................................................... [ALUA Groups: 1]
o- default_tg_pt_gp ............................... [ALUA state: Active/optimized]
/backstores/block> cd /
/> ls
o- / ............................................................................. [...]
o- backstores .................................................................. [...]
| o- block ...................................................... [Storage Objects: 1]
| | o- disk1 [/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0 (32.0GiB) write-thru deactivated]
| | o- alua ....................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................... [ALUA state: Active/optimized]
| o- fileio ..................................................... [Storage Objects: 0]
| o- pscsi ...................................................... [Storage Objects: 0]
| o- ramdisk .................................................... [Storage Objects: 0]
o- iscsi ................................................................ [Targets: 0]
o- loopback ............................................................. [Targets: 0]
/>
次に IQN の設定
名前は自動で作らせてもいいけど「iqn.2022-02.local.iscsi-target:disk」とします
/> cd /iscsi
/iscsi> create iqn.2022-02.local.iscsi-target:disk
Created target iqn.2022-02.local.iscsi-target:disk.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/iscsi>
ただこれですと
/iscsi> ls
o- iscsi .................................................................. [Targets: 1]
o- iqn.2022-02.local.iscsi-target:disk ..................................... [TPGs: 1]
o- tpg1 ..................................................... [no-gen-acls, no-auth]
o- acls ................................................................ [ACLs: 0]
o- luns ................................................................ [LUNs: 0]
o- portals .......................................................... [Portals: 1]
o- 0.0.0.0:3260 ........................................................... [OK]
/iscsi>
次に /dev/sdb と繋げる
/iscsi> cd iqn.2022-02.local.iscsi-target:disk/tpg1/luns
/iscsi/iqn.20...isk/tpg1/luns>
/iscsi/iqn.20...isk/tpg1/luns> create /backstores/block/disk1
/iscsi/iqn.20...isk/tpg1/luns> cd /iscsi
/iscsi> ls
o- iscsi .................................................................. [Targets: 1]
o- iqn.2022-02.local.iscsi-target:disk ..................................... [TPGs: 1]
o- tpg1 ..................................................... [no-gen-acls, no-auth]
o- acls ................................................................ [ACLs: 0]
o- luns ................................................................ [LUNs: 1]
| o- lun0 [block/disk1 (/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0) (default_tg_pt_gp)]
o- portals .......................................................... [Portals: 1]
o- 0.0.0.0:3260 ........................................................... [OK]
/iscsi>
[root@iscsi-target ~]# targetcli saveconfig
次は接続しようとする iscsi-initiator の「iqn」をACLに登録する. これは後ほど.
FWの設定
[root@iscsi-target ~]# firewall-cmd --add-service=iscsi-target --permanent
[root@iscsi-target ~]# firewall-cmd --reload
[root@iscsi-target ~]# systemctl enable target.service
[root@iscsi-target ~]# systemctl start target.service
iscsi-initiator †
iscsi-initiator側の設定
[root@iscsi-initiator ~]# yum install iscsi-initiator-utils
その後に自分が名乗るiqnを付ける. 既定の値が一応あるけど、変更します
[root@iscsi-initiator ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:c13c411491f
↓
InitiatorName=iqn.2022-02.local.sybyl:iscsi-initiator
[root@iscsi-initiator ~]#
っでここでtarget側にこの「iqn.2022-02.local.sybyl:iscsi-initiator」を伝える
/iscsi> ls
o- iscsi .................................................................. [Targets: 1]
o- iqn.2022-02.local.iscsi-target:disk ..................................... [TPGs: 1]
o- tpg1 ..................................................... [no-gen-acls, no-auth]
o- acls ................................................................ [ACLs: 0]
o- luns ................................................................ [LUNs: 1]
| o- lun0 [block/disk1 (/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0) (default_tg_pt_gp)]
o- portals .......................................................... [Portals: 1]
o- 0.0.0.0:3260 ........................................................... [OK]
/iscsi> cd iqn.2022-02.local.iscsi-target:disk/tpg1/acls
/iscsi/iqn.20...isk/tpg1/acls> create iqn.2022-02.local.sybyl:iscsi-initiator
そのうえでinitiator側の操作に移る
まず現状のデバイスは
[root@iscsi-initiator ~]# lsblk -S
NAME HCTL TYPE VENDOR MODEL REV TRAN
sda 0:0:0:0 disk VMware Virtual disk 2.0
sdb 0:0:1:0 disk VMware Virtual disk 2.0
sr0 3:0:0:0 rom NECVMWar VMware SATA CD00 1.00 sata
[root@iscsi-initiator ~]#
である。
次にtargetを探して
[root@iscsi-initiator ~]# iscsiadm -m discovery -t st -p 192.168.0.142
192.168.0.142:3260,1 iqn.2022-02.local.iscsi-target:disk
[root@iscsi-initiator ~]# iscsiadm -m node --login
[root@iscsi-initiator ~]# lsblk -S
NAME HCTL TYPE VENDOR MODEL REV TRAN
sda 0:0:0:0 disk VMware Virtual disk 2.0
sdb 0:0:1:0 disk VMware Virtual disk 2.0
sdc 34:0:0:0 disk LIO-ORG disk1 4.0 iscsi
sr0 3:0:0:0 rom NECVMWar VMware SATA CD00 1.00 sata
[root@iscsi-initiator ~]#
(削除は)
iscsiadm -m node -T iqn.2022-02.local.iscsi-target:disk --portal 192.168.0.142:3260,1 --logout
っで一応見えている
めも †
(parted) mklabel gpt
(parted) mkpart primary 0% 100%
(parted) set 1 lvm on
(parted) p
[root@iscsi-initiator ~]# pvcreate /dev/sdb1
[root@iscsi-initiator ~]# pvcreate /dev/sdc1
[root@iscsi-initiator ~]# pvscan
PV /dev/sdb1 lvm2 [<32.00 GiB]
PV /dev/sdc1 lvm2 [31.96 GiB]
Total: 2 [63.96 GiB] / in use: 0 [0 ] / in no VG: 2 [63.96 GiB]
[root@iscsi-initiator ~]# vgcreate VG00 /dev/sdb1 /dev/sdc1
Volume group "VG00" successfully created
[root@iscsi-initiator ~]#
[root@iscsi-initiator ~]# lvcreate -n L0 -l 100%FREE VG00
[root@iscsi-initiator ~]# pvscan
PV /dev/sdb1 VG VG00 lvm2 [<32.00 GiB / 0 free]
PV /dev/sdc1 VG VG00 lvm2 [31.93 GiB / 0 free]
Total: 2 [<63.93 GiB] / in use: 2 [<63.93 GiB] / in no VG: 0 [0 ]
[root@iscsi-initiator ~]#
[root@iscsi-initiator ~]# lvscan
ACTIVE '/dev/VG00/L0' [<63.93 GiB] inherit
[root@iscsi-initiator ~]#
[root@iscsi-initiator ~]#
[root@iscsi-initiator ~]# mkfs.xfs -L data /dev/VG00/L0
[root@iscsi-initiator ~]# mkdir /data
[root@iscsi-initiator ~]# mount LABEL=data /data
[root@iscsi-initiator ~]# df -lTh /data
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/VG00-L0 xfs 64G 33M 64G 1% /data
[root@iscsi-initiator ~]#
fstabオプション
LABEL=data /data xfs defaults,_netdev 0 0
(あるいは)
LABEL=data /data xfs defaults,_netdev,nofail 0 0
「_netdev」これがないとinitiatorが有効(ネットワークが有効)になる前に処理してしまいエラーになる.なので必須かな. nfs mountにもつけたほうがいいのかも
「nofail」デバイスが存在しないなら無視する. initiatorでの運用の場合は initiatorが正しく働かない場合でも無理やりboot-upしますので、いいかは微妙. boot-up後にinitiator不調を感知、通知機能が必要かな.
initiatorでの運用なら「nofail」はやめたほうが良いみたい. 「nofail」を入れないとそのデバイスで行うサービス(nfsd)とかも止めてくれるので.
![[PukiWiki]](image/picc.png "[PukiWiki]")
