Month: April 2011

How to Setup GFS2 or GFS in Linux Centos

It has been a nightmare for me setting up GFS2 with my 3 shared hosting servers and 1 SAN Storage. I have been reading all over the internet and the solutions to this is either outdated or contains bug that cannot make my SAN storage SAN to work. Finally, i managed to setup my GFS2 on my Dell MD3200i with 10TB of disk space.

GFS2/GFS Test Environment

Here is the test environment equipment that i utilized for this setup.

  1. 3 Centos Web Server
  2. 1 MD3200i Dell SAN Storage
  3. 1 Switch to connect all these equipment together

Assumption

I will assume you would have setup all your 3 Centos servers to communicate with your SAN ISCSI storage. This means that all your 3 Centos servers will be able to view your newly created LUN using iscsiadmn. And you have switch off your iptabls and selinux. If your iscsi storage hasn't configure, you can do so at cyberciti.

Setup GFS2/GFS packages

On all of your 3 Centos servers, you must install the following packages:

  1. cman
  2. gfs-utils
  3. kmod-dlm
  4. modcluster
  5. ricci
  6. luci
  7. cluster-snmp
  8. iscsi-initiator-utils
  9. openais
  10. oddjobs
  11. rgmanager

Or you can simple type the following yum on all 3 Centos machine

yum install -y cman gfs-utils kmod-gfs kmod-dlm modcluster ricci luci cluster-snmp iscsi-initiator-utils openais oddjob rgmanager

Or even simplier, you can just add the cluster group via the following line

yum groupinstall -y Clustering
yum groupinstall -y "Storage Cluster"

Oh, remember to update your Centos before proceeding to do all of the above.

yum -y check-update
yum -y update

After you have done all of the above, you should have all the packages available to setup GFS2/GFS on all your 3 Centos machine.

Configuring GFS2/GFS Cluster on Centos

Once you have your required centos packages installed, you would need to setup your Centos machine. Firstly, you would need to setup all your hosts file with all 3 servers machine name. Hence, i appended all my 3 servers machine name across and in each machine i would have the following additional line in my /etc/hosts file.

111.111.111.1 gfs1.hungred.com
111.111.111.2 gfs2.hungred.com
111.111.111.3 gfs3.hungred.com

where *.hungred.com is each machine name and the ip beside it are the machine ip addresses which allows each of them to communicate with each other by using the ip stated there.

Next, we will need to setup the cluster configuration of the server. On each machine, you will need to execute the following instruction to create a proper cluster configuration on each Centos machine.

ccs_tool create HungredCluster
ccs_tool addfence -C node1_ipmi fence_ipmilan ipaddr=111.111.111.1 login=root passwd=machine_1_password
ccs_tool addfence -C node2_ipmi fence_ipmilan ipaddr=111.111.111.2 login=root passwd=machine_2_password
ccs_tool addfence -C node3_ipmi fence_ipmilan ipaddr=111.111.111.3 login=root passwd=machine_3_password

ccs_tool addnode -C gfs1.hungred.com -n 1 -v 1 -f node1_ipmi
ccs_tool addnode -C gfs2.hungred.com -n 2 -v 1 -f node2_ipmi
ccs_tool addnode -C gfs3.hungred.com -n 3 -v 1 -f node3_ipmi

Next, you will need to start cman.

service cman start
service rgmanager start

cman should starts without any error. If you have any error while starting cman, your GFS2/GFS will not work. If everything works fine, you should see the following when you type the command as shown below,

[root@localhost ]# cman_tool nodes
10.0.0.1
Node  Sts   Inc   Joined               Name
   1   M     16   2011-1-06 02:30:27  gfs1.hungred.com
   2   M     20   2011-1-06 02:30:02  gfs2.hungred.com
   3   M     24   2011-1-06 02:36:01  gfs3.hungred.com

If the above shows, this means that you have properly setup your GFS2 cluster. Next we will need to setup GFS2!

Setting up GFS2/GFS on Centos

You will need to start the following services.

  • service gfs start
  • service gfs2 start

Once, this two has been started. All you need to do is to partition your SAN storage LUN. If you want to use GFS2, partition it with gfs2

/sbin/mkfs.gfs2 -j 10 -p lock_dlm -t HungredCluster:GFS /dev/sdb

Likewise, if you like to use gfs, just change it to gfs instead of gfs2

/sbin/mkfs.gfs -j 10 -p lock_dlm -t HungredCluster:GFS /dev/sdb

A little explanation here. HungredCluster is the one we created while we were setup out GFS2 Cluster. /dev/sdb is the SAN storage lun space which was discovered using iscsiadm. -j 10 is the number of journals. each machine within the cluster will require 1 cluster. Therefore, it is good to determine the number of machine you will place into this cluster. -p lock_dlm is the lock type we will be using. There are other 2 more types beside lock_dlm which you can search online.

P.S: All of the servers that will belong to the GFS cluster will need to be located in the same VLAN. Contact support if you need assistance regarding this.
If you are only configuring two servers in the cluster, you will need to manually edit the file /etc/cluster/cluster.conf file on each server. After the tag, add the following text:

If you do not make this change, the servers will not be able to establish a quorum and will refuse to cluster by design.

Setup GFS2/GFS run on startup

Key the following to ensure that GFS2/GFS starts everytime the system reboot.

chkconfig gfs on
chkconfig gfs2 on
chkconfig clvmd on //if you are using lvm
chkconfig cman on
chkconfig iscsi on
chkconfig acpid off
chkconfig rgmanager on
echo "/dev/sdb /home gfs2 defaults,noatime,nodiratime 0 0" >>/etc/fstab
mount /dev/sdb

Once this is done, your GFS2/GFS will have mount on your system to /home. You can check whether it works using the following command.

[root@localhost ~]# df -h

You should now be able to create files on one of the nodes in the cluster, and have the files appear right away on all the other nodes in the cluster.

Optimize clvmd

We can try to optimize lvmd to control the type of locking lvmd is using.

vi /etc/clvmd/clvmd.conf
find the below variables and change it to the variable as shown below
locking_type = 3
fallback_to_local_locking = 0
service clvmd restart

credit goes to http://pbraun.nethence.com/doc/filesystems/gfs2.html

Optimize GFS2/GFS

There are a few ways to optimize your gfs file system. Here are some of them.
Set your plock rate to unlimited and ownership to 1 in /etc/cluster/cluster.conf

Set noatime and nodiratime in your fstab.

echo "/dev/sdb /home gfs2 defaults,noatime,nodiratime 0 0" >>/etc/fstab

lastly, we can tune gfs directy by decreasing how often GFS2 demotes its locks via this method.

echo "
gfs2_tool settune /GFS glock_purge 50
gfs2_tool settune /GFS scand_secs 5
gfs2_tool settune /GFS demote_secs 20
gfs2_tool settune /GFS quota_account 0
gfs2_tool settune /GFS statfs_fast 1
gfs2_tool settune /GFS statfs_slots 128
" >> /etc/rc.local

credit goes to linuxdynasty.

iptables and gfs2/gfs port

If you wish to have iptables remain active, you will need to open up the following ports.

-A INPUT -i 10.10.10.200 -m state --state NEW -p udp -s 10.10.10.0/24 -d 10.10.10.0/24 --dport 5404, 5405 -j ACCEPT
-A INPUT -i 10.10.10.200 -m state --state NEW -m multiport -p tcp -s 10.10.10.0/24 -d 10.10.10.0/24 --dports 8084 -j ACCEPT
-A INPUT -i 10.10.10.200 -m state --state NEW -m multiport -p tcp -s 10.10.10.0/24 -d 10.10.10.0/24 --dports 11111 -j ACCEPT
-A INPUT -i 10.10.10.200 -m state --state NEW -m multiport -p tcp -s 10.10.10.0/24 -d 10.10.10.0/24 --dports 14567 -j ACCEPT
-A INPUT -i 10.10.10.200 -m state --state NEW -m multiport -p tcp -s 10.10.10.0/24 -d 10.10.10.0/24 --dports 16851 -j ACCEPT
-A INPUT -i 10.10.10.200 -m state --state NEW -m multiport -p tcp -s 10.10.10.0/24 -d 10.10.10.0/24 --dports 21064 -j ACCEPT
-A INPUT -i 10.10.10.200 -m state --state NEW -m multiport -p tcp -s 10.10.10.0/24 -d 10.10.10.0/24 --dports 41966,41967,41968,41969 -j ACCEPT
-A INPUT -i 10.10.10.200 -m state --state NEW -m multiport -p tcp -s 10.10.10.0/24 -d 10.10.10.0/24 --dports 50006,50008,50009 -j ACCEPT
-A INPUT -i 10.10.10.200 -m state --state NEW -m multiport -p udp -s 10.10.10.0/24 -d 10.10.10.0/24 --dports 50007 -j ACCEPT

Once these ports are open on your iptables, your cman should be able to restart properly without getting start either on fencing or cman starting point. Good Luck!

Troubleshooting

You might face some problem setting up GFS2 or GFS. Here are some of them which might be of some help

CMAN fencing failed

You get something like the following when you start your cman

Starting cluster:
Loading modules... done
Mounting configfs... done
Starting ccsd... done
Starting cman... done
Starting daemons... done
Starting fencing... failed

One of the possibility that is causing this is that your gfs2 has already been mounted to a drive. Hence, fencing failed. Try to unmount it and start it again.

mount.gfs2 error

if you are getting the following error

mount.gfs2: can't connect to gfs_controld: Connection refused

you need to try to start the cman service

Reference

How to Integrate Extjs Tree Panel Drag And Drop Into Extjs HtmlEditor

This is a problem that i face during an integration with Extjs Tree Panel drag and drop functionality into extjs htmleditor. The research process doesn't take very long because the only clue online was extjs forum with the title "Drag and drop into child iframe document". Many said it was impossible. But there were actually people who had done this previously and successfully drag and drop tree panel items into extjs htmleditor. However, many will be clueless without a proper demo or actual code to see the effect of such implementation.  The forum page do exist code that could be of help in your implementation but you would really need to guess and try to see whether it works. In my case, i manage to get it work.

Htmleditor Problem

The problem was fairly straight forward. The drag and drop functionality in extjs cannot drag into iframe. Our Extjs htmleditor is using iframe for their editor implementation. Therefore, if you try to drag your items into the iframe, it will basically stuck somewhere outside of the htmleditor/iframe. Drop is also an issue since the drop behavior is being stopped by the implementation done by the htmleditor. Now, our objective here in this article is to make it so that the drag and drop will work within the iframe.

Drag and drop into htmleditor Solution

The solutions here is to overwrite the behavior of the htmleditor so that it accept the drag items into the iframe. This can be done using Extjs Event manager as shown below,

pushValue: function(){
	var ddm = Ext.dd.DragDropMgr;
	Ext.EventManager.on(this.iframe.contentDocument, "mousemove", ddm.handleMouseMove, ddm, true);
	Ext.EventManager.on(this.iframe.contentDocument, "mouseup", ddm.handleMouseUp, ddm, true);
},

The above code utilized htmleditor pushValue functionality to overwrite two behavior on extjs htmleditor. mousemove and mouseup is overwritten with the default behavior of Ext.dd.DragDropMgr. This will allow the behavior to properly mouseover the htmleditor content and also drop it accordingly. Once you have done that, your treepanel items should have no problem dragging around the htmleditor. However, drop will still be an issue since we haven't attached the appropriate drop zone into the htmleditor.

DropZone on Htmleditor

In order to drag the items on the treepanel into the extjs htmleditor, we will definitely needs to define a dragzone which is similiar to the dragzone on the treepanel. However to create a dragzone, the component must be initialized before any attachment can be done. Hence, we will create a listeners on htmleditor to see when it is render so that we can attach a dragzone into it.

listeners: {
	'render': function(){
		var DCZone = new Ext.dd.DropZone(Ext.get('ext-gen10'), {
			ddGroup: 'DCZone',
			/* If the mouse is over a target node, return that node. This is
			provided as the "target" parameter in all "onNodeXXXX" node event handling functions */
			getTargetFromEvent: function(e) {
				//console.log('getTargetFromEvent'+new Date().getTime());
				return e.getTarget();
			},

			/* On node drop, we can interrogate the target node to find the underlying
			application object that is the real target of the dragged data.
			We can use the data set up by the DragZone's getDragData method to read
			any data we decided to attach. */
			onNodeDrop : function(target, dd, e, data){
				console.log('onNodeDrop');
				// do your work here

				Ext.fly(this.getEl()).frame("00AE00");
				return true;
			}
		});

	}
}

Now, the important part here for our htmleditor iframe to work so that the drop items will land on it is the overwrite function of getTargetFromEvent. If this function doesn't exist, the whole thing must not work. Therefore, the method is important to have. Ext.get('ext-gen10') is the parent element of the iframe. Once this are all done properly, you should be able to get an iframe that is draggable and droppable into it.

 

Solution Demo

Like i have mention, a demo will be much more appropriate. Hence, here's an attachment of what i have been saying.

Making Mount DVD/CDROM Executable in Linux

Interestingly, if you try to mount your dvd or cdrom and try to run the files in your dvd/cdrom in linux, chances are you will most likely get an error stating that the file doesn't have the permission to perform the task. If you mount your media into linux and receives an error saying that your cd/dvd is write-protected and your mounted drive is only good for read-only, you will definitely get permission denial error when you try to execute any .sh files in your mounted drive.

The solutions for this is pretty simple. All you need to do is to fire up your fstab file at

vi /etc/fstab

and add/edit the following line so that you can mount and execute the files on your mounted drive.

/dev/dvd        /mnt/dvd        auto        ro,user,noauto,exec      0 0

The trick to make your dvd/cd executable is to set "exec" after "user" because by default once "user" is seen, it will automatically change your media to "noexec" and overwrite the "exec" you have defined either before or after you mount the dvd.

mount -o exec /dev/dvd

and you will see that it stills fail if on your fstab user is placed at the end or after exec statement. Cheers!

Changing SSH Port Fail To Login In Centos – No route to host

Recently i have been setting up with my own server in Centos playing around with Centos and understanding more about Linux. It has been a challenging and interesting process for me. From a beginner point of view, there is really a lot to learn and explore with hardware. One of this problem i faced was ssh giving me a headache when i change the ssh port to something different rather than port 22.

The whole process of changing SSH port 22 to something else was really to harden the security side for SSH. However, who would have though problem will come for something so simple such as changing SSH port to something else rather than 22?

If you are getting the following message

connect to host xxx.xxx.xxx.xxx port 2222: No route to host

and you are sure that you did the correct thing and started staring at your hardware switch. Don't. This should have nothing to do with your layer 3 switch if you hasn't touch it yet.

The reason why only port 22 is accessible via SSH and not other port was because Centos has its own Firewall called Iptables. If you are like me who suspect it might be Centos firewall who is causing the problem, you have found the right answer.

In order to determine whether is it the rule of Centos Iptables who is causing this problem, all you have to do is to initialize the following command,

iptables -F

this will flush the iptables rules and make it clean from centos default rules. Now, try to ssh to your machine and see whether it works?

If it does, you just found the culprit for your headache. Next, we will need to change the iptables rule so that it stays permanent on the iptables. Navigation down to 

/etc/sysconfig/iptables

look for the last 3rd line where you will see --dport 22, change it to your new ssh port and restart your iptables and sshd

service sshd restart
service iptables restart

and you should be able to ssh properly from another machine to your server. Cheers!