CentOS에 SAN 스토리지 연결하기.

작업환경

OS: CentOS7
HBA: Qlogic HBA 2 port 2/4g
Storage: EMC CX300
SAN 스위치: CISCO MDS 9148(기존 zoneset에 linux zone 추가)

1. HBA 장착후 인식 확인

# lspci | grep -i qlogic
07:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
07:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)

커널에 드라이버가 로드되어 있는지 확인

# lsmod | grep qla
qla2xxx               593236  0
scsi_transport_fc      59960  1 qla2xxx

2. HBA WWN 확인

# cat /sys/class/scsi_host/host*/device/fc_host/host*/port_name
0x500143800299f9be
0x500143800299f9bc

systool로 hba관련 사항 확인이 가능하다. systool 은 sysfsutils 에 포함되어 있으므로 systool이 없으면, 아래와 같이 설치하면 된다.

# yum install sysfsutils
# systool -vc fc_host | grep -i port_name
    port_name           = "0x500143800299f9be"
    port_name           = "0x500143800299f9bc"

3. SAN zoning(wwn zoning)
SAN 스위치에서 아래와 같은 방법으로 hba wwn과 스토리지의 wwn을 등록해준다.(첫 두 줄 hba, 나머지 네 줄 Storage)

switch(config)# zone name linux1 vsan 1
switch(config-zone)# member pwwn 50:01:43:80:02:99:f9:bc
switch(config-zone)# member pwwn 50:01:43:80:02:99:f9:be
switch(config-zone)# member pwwn 50:06:01:60:30:22:d9:87
switch(config-zone)# member pwwn 50:06:01:61:30:22:d9:87
switch(config-zone)# member  pwwn 50:06:01:68:30:22:d9:87
switch(config-zone)# member pwwn 50:06:01:69:30:22:d9:87
switch(config-zone)# member pwwn 50:01:43:80:02:99:f9:bf
switch(config-zone)# exit

기존에 존재하는 ‘zoneset1’ 에 새로만든 linux1 zone을 참여시킨다.

switch(config)# zoneset name zoneset1 vsan 1
switch(config-zoneset)# member linux1
switch(config-zoneset)# exit
switch(config)# exit

맨 아래쪽에 새로만든 ‘linux’ zone 이 보인다.

switch# sh zoneset
zoneset name zoneset1 vsan 1
  zone name zone1 vsan 1
    pwwn 10:00:00:00:c9:4f:25:00
...
  zone name linux1 vsan 1
    pwwn 50:01:43:80:02:99:f9:bc
    pwwn 50:01:43:80:02:99:f9:be
    pwwn 50:06:01:60:30:22:d9:87
    pwwn 50:06:01:61:30:22:d9:87
    pwwn 50:06:01:68:30:22:d9:87
    pwwn 50:06:01:69:30:22:d9:87

추가한 zone이 동작하도록 active 시켜주고 확인한다.

switch(config)# zoneset activate name zoneset1 vsan 1

switch# sh zoneset active
zoneset name zoneset1 vsan 1
  zone name zone1 vsan 1
    pwwn 10:00:00:00:c9:4f:25:00
...
  zone name linux vsan 1
  * fcid 0xce16ef [pwwn 50:01:43:80:02:99:f9:bc]
    pwwn 50:01:43:80:02:99:f9:be
    pwwn 50:06:01:60:30:22:d9:87
  * fcid 0xce1000 [pwwn 50:06:01:61:30:22:d9:87]
    pwwn 50:06:01:68:30:22:d9:87
  * fcid 0xce1100 [pwwn 50:06:01:69:30:22:d9:87]

설정한 구성을 저장한다.

switch# copy running-config startup-config
[########################################] 100%

4. 스토리지에서 HBA 등록, LUN 만들어 할당한다.(EMC navisphere)

SAN 스위치가 정상적으로 세팅되면, 아래처럼 서버가 보일 것이다.

storage 탭을 선택 -> 스토리지 선택 -> 마우스오른쪽 버튼을 눌러 Connectivity Status 를 선택한다.

새로 연결한 서버가 보이면, 선택한후, initiatior를 등록한다.

그 후, 스토리지 그룹을 만들고, connect hosts를 선택하여 스토리지그룹에 호스트를 등록한다.

LUN을 만들어, 스토리지 그룹에서 사용할 수 있도록 할당해 주면, 아래와 같이 사용할 준비가 끝난다.

5. Linux 에서 스토리지의  LUN을 인식하는지 확인하고, 파티셔닝 후, 화일시스템을 만들어 사용한다.

# yum install lsscsi
# lsscsi
[0:0:0:0]    disk    HP       LOGICAL VOLUME   3.52  /dev/sda
[0:3:0:0]    storage HP       P410i            3.52  -
[9:0:0:0]    disk    DGC      RAID 5           0219  /dev/sdb
[9:0:1:0]    disk    DGC      RAID 5           0219  /dev/sdc

아래 커맨드로도 확인 가능하다.(어떤 이유에서인지, fdisk에서는 lun이 보이지 않았다)

# parted -l | grep -i sd
Error: /dev/sdb: unrecognised disk label
Disk /dev/sda: 73.4GB
Partition Table: msdos
Disk /dev/sdb: 107GB
Disk /dev/sdc: 107GB
Partition Table: msdos

6. 기타

사용중에 스토리지에서 LUN 만들고, linux에서 인식시키려면, 아래와 같은 커맨드를 사용하고 기다리면 된다. host 뒤의 숫자는 자신의 시스템에 맞는 것을 사용하면 된다. log를 보면 대략 4분 정도 걸리는 것을 볼 수 있다.

# echo "1" > /sys/class/fc_host/host23/issue_lip

# tail -f /var/log/message

Nov 6 10:29:29 localhost kernel: rport-3:0-0: blocked FC remote port time out: removing target and saving binding
Nov 6 10:29:29 localhost kernel: rport-3:0-1: blocked FC remote port time out: removing target and saving binding
Nov 6 10:29:29 localhost kernel: sd 3:0:0:0: emc: Detached
Nov 6 10:29:29 localhost kernel: sd 3:0:0:2: emc: Detached
Nov 6 10:33:28 localhost kernel: qla2xxx [0000:07:00.0]-6009:3: Loop down - aborting ISP.
Nov 6 10:33:28 localhost kernel: qla2xxx [0000:07:00.0]-00af:3: Performing ISP error recovery - ha=ffff880192d78000.
Nov 6 10:33:32 localhost kernel: qla2xxx [0000:07:00.0]-505f:3: Link is operational (4 Gbps).
Nov 6 10:33:32 localhost kernel: scsi 3:0:0:0: Direct-Access DGC RAID 5 0219 PQ: 0 ANSI: 4
Nov 6 10:33:32 localhost kernel: scsi 3:0:0:0: emc: detected Clariion CX300, flags 0
Nov 6 10:33:32 localhost kernel: scsi 3:0:0:0: emc: connected to SP B Port 1 (owned, default SP A)
Nov 6 10:33:32 localhost kernel: sd 3:0:0:0: Attached scsi generic sg2 type 0
Nov 6 10:33:32 localhost kernel: sd 3:0:0:0: [sdb] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Nov 6 10:33:32 localhost kernel: sd 3:0:0:0: [sdb] Write Protect is off
Nov 6 10:33:32 localhost kernel: scsi 3:0:0:1: Direct-Access DGC RAID 5 0219 PQ: 0 ANSI: 4
Nov 6 10:33:32 localhost kernel: sd 3:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Nov 6 10:33:32 localhost kernel: scsi 3:0:0:1: emc: detected Clariion CX300, flags 0
Nov 6 10:33:32 localhost kernel: scsi 3:0:0:1: emc: connected to SP B Port 1 (owned, default SP B)
Nov 6 10:33:32 localhost kernel: sd 3:0:0:1: Attached scsi generic sg3 type 0
Nov 6 10:33:32 localhost kernel: sd 3:0:0:1: [sdc] 104857600 512-byte logical blocks: (53.6 GB/50.0 GiB)
Nov 6 10:33:32 localhost kernel: sd 3:0:0:1: [sdc] Write Protect is off
Nov 6 10:33:32 localhost kernel: sd 3:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Nov 6 10:33:32 localhost kernel: scsi 3:0:0:2: Direct-Access DGC RAID 5 0219 PQ: 0 ANSI: 4
Nov 6 10:33:32 localhost kernel: scsi 3:0:0:2: emc: detected Clariion CX300, flags 0
Nov 6 10:33:32 localhost kernel: scsi 3:0:0:2: emc: connected to SP B Port 1 (owned, default SP B)
Nov 6 10:33:32 localhost kernel: sd 3:0:0:2: Attached scsi generic sg4 type 0
Nov 6 10:33:32 localhost kernel: sd 3:0:0:2: [sdd] 104857600 512-byte logical blocks: (53.6 GB/50.0 GiB)
Nov 6 10:33:32 localhost kernel: sd 3:0:0:2: [sdd] Write Protect is off
Nov 6 10:33:32 localhost kernel: sd 3:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't su port DPO or FUA
Nov 6 10:33:32 localhost kernel: scsi 3:0:1:0: Direct-Access DGC LUNZ 0219 PQ: 0 ANS : 4
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: Attached scsi generic sg5 type 0
Nov 6 10:33:32 localhost kernel: sdd: unknown partition table
Nov 6 10:33:32 localhost kernel: sdb: unknown partition table
Nov 6 10:33:32 localhost kernel: sd 3:0:0:2: [sdd] Attached SCSI disk
Nov 6 10:33:32 localhost kernel: sd 3:0:0:0: [sdb] Attached SCSI disk
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde] READ CAPACITY failed
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde]
Nov 6 10:33:32 localhost kernel: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde]
Nov 6 10:33:32 localhost kernel: Sense Key : Illegal Request [current]
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde]
Nov 6 10:33:32 localhost kernel: Add. Sense: Logical unit not supported
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde] Test WP failed, assume Write Enabled
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde] Asking for cache data failed
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde] Assuming drive cache: write through
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde] READ CAPACITY failed
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde]
Nov 6 10:33:32 localhost kernel: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde]
Nov 6 10:33:32 localhost kernel: Sense Key : Illegal Request [current]
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde]
Nov 6 10:33:32 localhost kernel: Add. Sense: Logical unit not supported
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde] Test WP failed, assume Write Enabled
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde] Asking for cache data failed
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde] Assuming drive cache: write through
Nov 6 10:33:32 localhost kernel: sd 3:0:1:0: [sde] Attached SCSI disk
Nov 6 10:33:32 localhost kernel: sdc: unknown partition table
Nov 6 10:33:32 localhost kernel: sd 3:0:0:1: [sdc] Attached SCSI disk