Appliance Disk full

From Swivel Knowledgebase
Revision as of 09:29, 17 May 2016 by Mtura (talk) (Reducing the MySQL transaction logs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Logo.gif

Symptoms

Log files fill the disk space as they are not correctly being purged.


Checking Disk Space

The following command from the command line may show the disk space usage as 100%

df –k

 /dev/sda3 	Use 100% Mounted on /backups


The disk space on a healthy system will look similar to the below sample output

 [admin@standby ~]# df -k
 Filesystem           1K-blocks      Used Available Use% Mounted on
 /dev/sda2             20641788   2196276  17396872  12% /
 /dev/sda3             10317860   1940240   7853500  20% /backups
 /dev/sda1               124427     13761    104242  12% /boot
 none                    517268         0    517268   0% /dev/shm
 /dev/sda7              7091968     48636   6683076   1% /support
 /dev/sda5              2063504     35952   1922732   2% /tmp


or df -m to see usuage in Mb


The following command may be of use in viewing where the disk usage is:

du -h --max-depth=1

Where depth is the number of folders to look within


Common file locations to check

As well as disk space, it is also important to check the number of files, as a large number of small or even 0 byte files can cause issues.


Disk space in /backups

View the backups using ls -la

Each backup is stored in the format as ddmmyy.number.tar.gz

The file size can be seen, large backups (200 Mb+) can cause the disk space to fill up, the backups contain some logging data as described below.


Disk space in /var/log/messages

Log files may build up in /var/log/messages


Disk space in /var/log/swivel

This folder contains Swivel log messages for differing error levels such as : Warning, Error, Fatal


Disk space in /var/lib/mysql

Transaction logs may build up in /var/lib/mysql, this is caused by A/A appliances being out of synchronisation. The transaction logs consist of binaries that contain data edits and/or relays that is information sent to a MySQL slave.


Disk space in /var/spool/clientmqueue

Mail logs for events are stored here


Disk space in /var/spool/mail

Mail logs for users are stored here

Swivel 3.7.3727

Swivel 3.7.3727 contains debugging information for Repository syncs that may produce a large number of files and the files may need to be purged. To prevent the files being generated upgrade to a more recent version of Swivel. As a temporary measure they can be deleted and a low frequency of repository sync can be set. The files are named:

profile.<date>.data

and reside in:

<path to Tomcat>/webapps/pinsafe/WEB-INF/logs

Prerequisites

The error has been seen on the following systems:

Appliance build 2.0.10, to 2.0.14


Error Messages

ERROR - Saving the XML config file "/usr/local/tomcat/webapps/pinsafe/WEB-INF/conf/config.xml" failed, error: java.io.IOException: No space left on device.

java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method)

The above message can be seen when there are Tomcat errors related to no disk space.


cp: cannot create regular file ‘/backups/.default/tomcat/logs………. No space left on device

The above error message can be seen when performing a backup.


Solutions

Purge script fix

Edit the following script /etc/cron.daily/PINsafe_backup_purge.sh

There are two lines in that script that should read

 p_arch=`grep "archives" /etc/pinsafe.conf|cut -d\= -f2`
 p_logs=`grep "logs" /etc/pinsafe.conf|cut -d\= -f2`


Logrotate

In the file /etc/logrotate.d/tomcat add notifempty as below

 /var/log/tomcat/*.log {
 	daily
 	missingok
 	copytruncate
 	rotate 30
 	missingok
 	compress
 }

To

 /var/log/tomcat/*.log {
 	daily
 	missingok
 	copytruncate
 	rotate 30
 	missingok
 	compress
 	notifempty
 }


Reducing the MySQL transaction logs

Removing Transaction logs Using MySQL Commands

From the MySQL command line run

STOP SLAVE;

RESET SLAVE;

RESET MASTER;


Setting Transaction Log Size

When the appliance is back in synchronisatuon, then the transaction logs should sync, by default these are stored for 7 days. It is possible to change these in /etc/my.cnf, look fior the following line:

  1. Delete BIN LOG Files After 7 Days.

expire_logs_days=7 max_binlog_size=256000000

Change expire_logs_days=7 to the required value

Manually deleting files

Note: Take care when using the rm command so that only the correct files are deleted. Deleted files can only be recovered from valid backups.

The following commands are run from the command line, see Command Line Access How to guide

  • Ensure that you are in the correct directory
  • Using ./filename or ./folder ensures that only the file or folder with the directory is removed.

If there are too many files to list then the following command can be used to remove the files. This example is for files within the Tomcat logs folder which will delete all instances of localhost.

find . -iname 'localhost.*' | xargs rm


Deleting log files

To delete log files older than 5 days the following commands can be used.


cd /var/log/tomcat


find /var/log/tomcat/ -iname 'localhost.*' -mtime +5 | xargs rm


find /var/log/tomcat/ -iname 'manager.*' -mtime +5 | xargs rm


find /var/log/tomcat/ -iname 'catalina.*' -mtime +5 | xargs rm


find /var/log/tomcat/ -iname 'admin.*' -mtime +5 | xargs rm


find /var/log/tomcat/ -iname 'host-manager.*' -mtime +5 | xargs rm


If the following error message is shown, and providing the commands have been correctly typed, then there are no files older than 5 days.

"rm: too few arguments
Try `rm --help' for more information."


Deleting Mail queue files

The following command will remove all files in the clientmqueue folder

ls /var/spool/clientmqueue/ xargs rm


and mail for the root user can be removed with the following command

echo > /var/spool/mail/root


Known Issues

Please be aware: As a side effect of a Full Disk, is that the config.xml file can become corrupt. After the disk space has been freed, please ensure that the config.xml is not showing as 0 bytes. This is located under:

v3.9.1 or newer - /home/swivel/.swivel/conf

v3.9 or older - /usr/local/tomcat/webapps/pinsafe/WEB-INF/conf

You must restore a config.xml from a valid and most recent backup.

Troubleshooting

/bin/rm: Argument list too long

This can occur where there are too many files to delete using the rm command. Either specify a specific file e.g. catalina.* or use the commands given above.