Difference between revisions of "High Availability on MySQL service faillure"

Latest revision as of 12:52, 11 May 2017

Overview

This document outlines the steps required to configure Swivel clusters to trigger fail-over on MySQL failure.

Originally any Swivel HA cluster is configured to fail-over on Tomcat service failure; this will add MySQL service motorization to trigger fail over.

Configuration Considerations

MySQL is extremly robust. If MySQL fails, the probability of having an underlying bigger problem (that could lead to database corruption) is high.

For this reason, the configuration doesn't perfectly mimmic the Tomcat Failure setting. In truth, the Tomcat setting is set not only to fail-over to a different node when one fails, but also to look for the primary node to come up, and then shift load to it.

Due to the causes of a MySQL service faillure, this "shift back to primary node" behaviour is not set.

Swivel configuration for both nodes

1- Enter the Swivel console, and then follow to command line, using Advanced menu (option 8), and then Command line (option 6).

2- To install the correct software packages that will support the monitoring functions, run command:

rpm -ivh http://vault.centos.org/4.9/centosplus/i386/RPMS/mysqlclient14-4.1.22-1.el4s1.1.i386.rpm

3- We need to update the mysql.monitor file by running the command:

wget –qN http://yum.swivelsecure.net/upgrades/mysql.monitor -O /usr/lib/mon/mon.d/mysql.monitor

4- Add executable permitions to the newlly updated file, by running command:

chmod +x /usr/lib/mon/mon.d/mysql.monitor

This next step can be performed directly by editing the files, or using webmin to do that form you.

Option A - Command line

1- Enter the Swivel console, and then follow to command line, using Advanced menu (option 8), and then Command line (option 6).

2- Run command:

nano /etc/mon.cf

3- Add the LocalHost hostgroup, and then add the LocalHost_IP watch section exemplified bellow.

4- Use ctrl+o to write the changes to the file, and ctrl-x to quit nano.

5- Use the 'exit' command to quit the command line and return to the appliance menu.

6- use 0 to return to the main menu and then use command 3 to enter Monitor service control

7- use 1 to stop monitor and then 1 to start it back on.

Option B - Webmin

1- Login to Webmin on https://serverIP:10000

2- Goto SYSTEM menu, then MON menu

3- Select Host groups icon, and then add the LocalHost_IP group to the last line, just as shown in the picture

4- On the watchlists menu, add a new watch list for Localhost, then add a service watch and configure it to:

- Name of service: musql

- Check every: 10 seconds

- Standard Monitor: mysql.monitor

- Monitor parameters: --database=pinsafe_rep

- Alerts for period: ha.alert on Service goes down

...according to the following image.

5- return to MON services and choose MON service restart

6- The final result on MON Status, should be something like this:

Swivel configuration to support the new HA settings

So far, we managed to setup the HA routines on Swivel secure appliance cluster to faill-over when the MySQL service stops responding.

By default it is setup to do so when the Tomcat service fails, bringing the RADIUS server down with it. However, in the event of a MySQL service failure without Tomcat failure, Radius will still be functional and responding to requests. This will generate an error, because most swivel integrations support radius servers as primary and secondary RADIUS. If the RADIUS service on the down machine accepts the RADIUS request, the integrated client hardware or software will not try the secondary RADIUS. The primary RADIUS however will fire an database connectivity error due to fact that the local host mysql service is down.

To avoid that, we need to assure that both appliance nodes connect to running MySQL service, and that is the on indicated by the appliance Virtual IP.

On both nodes:

1- Enter the Swivel web console, and then navigate to Database.

2- Choose General from the menu, and then edt the MySQL5 connection, replacing Localhost by the VIP assigned to the cluster, on the MYSQL5 connector URL setting