Knowledge Layer :: Can I connect multiple servers to a single iSCSI LUN?

Knowledgebase Home | Favorites

Search the Knowledgebase	Browse by Category
[Advanced Search]

Can I connect multiple servers to a single iSCSI LUN?

Would you like to...

Print this page

Email this page

Add to favorites

Yes, this is possible, but with special requirements.
* The filesystem you use on the iSCSI mount must be cluster aware.
* Your applications must also be cluster aware which use this filesystem.
** The Linux ext3 filesystem is NOT cluster aware. You will need to use OCFS2, or another clustering filesystem for it to be clusterable on linux.

In this article, we will walk you through setting up the OCFS2 Clustering File System with RedHat 5 and our StorageLayer iSCSI system.

Requirements:
Redhat 5 servers updated (yum update)
ISCSI Target (only one needed)

The iSCSI target should already be setup on each server.You can verify this by running:

iscsiadm -m session

This should show you information like:

tcp: [1] 192.168.250.101:3260,1 iqn.snap01.iscsi0

Installation of Required Software (Done on every machine):

•Retrieve and install rpms: Module must match the kernel version running on the server.

We will need the OCFS2 module rpm for our kernel:
http://oss.oracle.com/projects/ocfs2/files/RedHat/

We also need the OCFS2-tools rpm
http://oss.oracle.com/projects/ocfs2-tools/files/RedHat/

Configuration (Done on every machine):

O2CB is disabled by default.We will run a simple perl script to enable it.This needs to be run on every server:

perl -pi.bak -e 's/O2CB_ENABLED=false/O2CB_ENABLED=true/' /etc/sysconfig/o2cb

You can verify this enabled O2CB by running:

grep O2CB_ENABLED /etc/sysconfig/o2cb

OCFS2 will then need to be configured.The file /etc/ocfs2/cluster.conf is this file.

node:
ip_port = 7777
ip_address = 192.168.250.8
number = 0
name = cluster01
cluster = ocfs2

node:
ip_port = 7777
ip_address = 192.168.250.9
number = 1
name = cluster02
cluster = ocfs2

node:
ip_port = 7777
ip_address = 192.168.250.10
number = 2
name = cluster03
cluster = ocfs2

cluster:
node_count = 3
name = ocfs2

The things to note in here are:

1)The node names should match the hostnames of each machine.
2)The ip addresses should match the private ips of each machine.
3)This file MUST be the same across all nodes in this cluster.
4)The cluster name needs to match the oc2b configuration file (ocfs2 is the default cluster name).

You can check the cluster name by looking at the O2CB_BOOTCLUSTER value in /etc/sysconfig/o2cb

grep O2CB_BOOTCLUSTER /etc/sysconfig/o2cb

# O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start.
O2CB_BOOTCLUSTER=ocfs2

We will need to create the /etc/ocfs2 directory and move our cluster.conf into it.This file move can be done via scp, sftp or any other method you choose.

mkdir -p /etc/ocfs2

Starting O2BC (On All servers)

You will now need to start the O2BC programs on each server to start the cluster.This is done by running:

/etc/init.d/o2cb start

Loading module "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Creating directory '/dlm': OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK

Partitioning and Formatting the disk (iSCSI Target)

This should only be done on ONE machine.Verify which disk the iSCSI target is located as (/dev/sdb in this case).If you select the wrong disk here, you will delete the data from another disk in your system.

First we will use parted to partition the disk:

parted -s /dev/sdb mklabel msdos
parted -s -- /dev/sdb mkpart primary 0 -1

These will partition the disk.To view the new partition table, run:

fdisk -l /dev/sdb

Disk /dev/sdb: 8453 MB, 8453619712 bytes
255 heads, 63 sectors/track, 1027 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 1028 8255487+ 83 Linux

Now, we are going to format the disk with some sane default settings and give it a label of cluster-storage:

mkfs.ocfs2 -b 4k -C 32k -N4 -L cluster-storage /dev/sdb1

mkfs.ocfs2 1.2.7
Filesystem label=cluster-storage
Block size=4096 (bits=12)
Cluster size=32768 (bits=15)
Volume size=8453586944 (257983 clusters) (2063864 blocks)
8 cluster groups (tail covers 32191 clusters, rest cover 32256 clusters)
Journal size=67108864
Initial number of node slots: 4

Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 2 block(s)
Formatting Journals: done
Writing lost+found: done
mkfs.ocfs2 successful

If you require more than 4 nodes, change -N<number> to a higher number than 4.This creates a file system with 4096 block size and 32768 (32k) cluster size.

Mounting the new partition (All servers)

We now need to have the partition table updated on all the servers in the cluster.In this case all the servers have /dev/sdb as the iSCSI target.

We will run the following to re-read the partition:

blockdev --rereadpt /dev/sdb

Next, we will want to create a mount point on the servers for this cluster.

mkdir /cluster-storage

Once the mount point is created, we will mount the partition. We are going to do this via its label we used in the format command. This will make it portable across all servers as the label will be the same, where the drive might be different (/dev/sdb1, or /dev/sdc1, etc).

mount -L cluster-storage /cluster-storage

Next, we will want to make sure it mounted.

mount | grep ocfs

This should display:

ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/sdb1 on /cluster-storage type ocfs2 (rw,_netdev,heartbeat=local)

This means the filesystem mounted on that server.

Now, we will want to check the o2cb status on each server:

/etc/init.d/o2cb status

Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold: 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active

This shows that the clustering is on and active.You will need to verify this on every machine in the cluster.

Does it work?

The servers now have the filesystem mounted and o2cb says they are working correctly, so we need to test it by writing files.
On each server, we will want to echo to a file to see if it works correctly.With the first server in the cluster, do the following:

echo testing1 >> /cluster-storage/test.txt

On the next machine, lets see if we can read that file:

cat /cluster-storage/test.txt

This should give you:

testing1

Now, let us echo to it again from the second machine:

echo testing2 >> /cluster-storage/test.txt

Then, lets echo from the third machine:

echo testing3 >> /cluster-storage/test.txt

On the first machine, lets read the file again to see if all the updates were done correctly:

cat /cluster-storage/test.txt

This should show:

testing1

testing2

testing3

Your filesystem is now clustering correctly.

Automatic Mounting (on bootup, on each machine).

To have the cluster mount at startup, we only need to make a few additions to your system.
The first is to make sure the netfs service is set to start at bootup:

chkconfig -list netfs

This should show:

netfs0:off1:off2:off3:on4:on5:on6:off

If your current runlevel (more than likely runlevel 3) is listed as off, run the following:

chkconfig --level 3 on

This will enable the netfs service to start at bootup on the server.
Next, we will want to modify the /etc/fstab file to add the mount point.
Here is an example for our configuration:

LABEL=cluster-storage/cluster-storageauto_netdev,defaults0 0

This tells us to use the label on the partition to mount to /cluster-storage.We are telling it to figure out the file system type automatically (via the auto entry)._netdev is used to tell the system not to try to mount it until the network is brought online.By then, iscsid should also be online, and it will mount successfully.The last two fields are for the mount order and fsck orders.0 for both these fields is a good default in this case.

For a list of other Cluster File systems, you can view the informative Wikipedia article: http://en.wikipedia.org/wiki/List_of_file_systems#Shared_disk_file_systems

Attachments

No attachments were found.