Appliance Disk full
Contents
Symptoms
Log files fill the disk space as they are not correctly being purged.
Checking Disk Space
The following command from the command line may show the disk space usage as 100%
df –k
/dev/sda3 Use 100% Mounted on /backups
The disk space on a healthy system will look similar to the below sample output
[admin@standby ~]# df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda2 20641788 2196276 17396872 12% / /dev/sda3 10317860 1940240 7853500 20% /backups /dev/sda1 124427 13761 104242 12% /boot none 517268 0 517268 0% /dev/shm /dev/sda7 7091968 48636 6683076 1% /support /dev/sda5 2063504 35952 1922732 2% /tmp
or df -m to see usuage in Mb
The following command may be of use in viewing where the disk usage is:
du -h --max-depth=1
Where depth is the number of folders to look within
Common file locations to check
As well as disk space, it is also important to check the number of files, as a large number of small or even 0 byte files can cause issues.
Disk space in /backups
View the backups using ls -la
Each backup is stored in the format as ddmmyy.number.tar.gz
The file size can be seen, large backups (200 Mb+) can cause the disk space to fill up, the backups contain some logging data as described below.
Disk space in /var/log/messages
Log files may build up in /var/log/messages
Disk space in /var/log/swivel
This folder contains Swivel log messages for differing error levels such as : Warning, Error, Fatal
Disk space in /var/lib/mysql
Transaction logs may build up in /var/lib/mysql, this is caused by A/A appliances being out of synchronisation. The transaction logs consist of binaries that contain data edits and/or relays that is information sent to a MySQL slave.
Disk space in /var/spool/clientmqueue
Mail logs for events are stored here
Disk space in /var/spool/mail
Mail logs for users are stored here
Swivel 3.7.3727
Swivel 3.7.3727 contains debugging information for Repository syncs that may produce a large number of files and the files may need to be purged. To prevent the files being generated upgrade to a more recent version of Swivel. As a temporary measure they can be deleted and a low frequency of repository sync can be set. The files are named:
profile.<date>.data
and reside in:
<path to Tomcat>/webapps/pinsafe/WEB-INF/logs
Prerequisites
The error has been seen on the following systems:
Appliance build 2.0.10, to 2.0.14
Error Messages
ERROR - Saving the XML config file "/usr/local/tomcat/webapps/pinsafe/WEB-INF/conf/config.xml" failed, error: java.io.IOException: No space left on device.
java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method)
The above message can be seen when there are Tomcat errors related to no disk space.
cp: cannot create regular file ‘/backups/.default/tomcat/logs………. No space left on device
The above error message can be seen when performing a backup.
Solutions
Purge script fix
Edit the following script /etc/cron.daily/PINsafe_backup_purge.sh
There are two lines in that script that should read
p_arch=`grep "archives" /etc/pinsafe.conf|cut -d\= -f2` p_logs=`grep "logs" /etc/pinsafe.conf|cut -d\= -f2`
Logrotate
In the file /etc/logrotate.d/tomcat add notifempty as below
/var/log/tomcat/*.log {
daily
missingok
copytruncate
rotate 30
missingok
compress
}
To
/var/log/tomcat/*.log {
daily
missingok
copytruncate
rotate 30
missingok
compress
notifempty
}
Reducing the MySQL transaction logs
Removing Transaction logs Using MySQL Commands
From the MySQL command line run
STOP SLAVE;
RESET SLAVE;
RESET MASTER;
Setting Transaction Log Size
When the appliance is back in synchronisatuon, then the transaction logs should sync, by default these are stored for 7 days. It is possible to change these in /etc/my.cnf, look fior the following line:
- Delete BIN LOG Files After 7 Days.
expire_logs_days=7 max_binlog_size=256000000
Change expire_logs_days=7 to the required value
Manually deleting files
Note: Take care when using the rm command so that only the correct files are deleted. Deleted files can only be recovered from valid backups.
The following commands are run from the command line, see Command Line Access How to guide
- Ensure that you are in the correct directory
- Using ./filename or ./folder ensures that only the file or folder with the directory is removed.
If there are too many files to list then the following command can be used to remove the files. This example is for files within the Tomcat logs folder which will delete all instances of localhost.
find . -iname 'localhost.*' | xargs rm
Deleting log files
To delete log files older than 5 days the following commands can be used.
cd /var/log/tomcat
find /var/log/tomcat/ -iname 'localhost.*' -mtime +5 | xargs rm
find /var/log/tomcat/ -iname 'manager.*' -mtime +5 | xargs rm
find /var/log/tomcat/ -iname 'catalina.*' -mtime +5 | xargs rm
find /var/log/tomcat/ -iname 'admin.*' -mtime +5 | xargs rm
find /var/log/tomcat/ -iname 'host-manager.*' -mtime +5 | xargs rm
If the following error message is shown, and providing the commands have been correctly typed, then there are no files older than 5 days.
"rm: too few arguments Try `rm --help' for more information."
Deleting Mail queue files
The following command will remove all files in the clientmqueue folder
ls /var/spool/clientmqueue/ xargs rm
and mail for the root user can be removed with the following command
echo > /var/spool/mail/root
Known Issues
Please be aware: As a side effect of a Full Disk, is that the config.xml file can become corrupt. After the disk space has been freed, please ensure that the config.xml is not showing as 0 bytes. This is located under:
v3.9.1 or newer - /home/swivel/.swivel/conf
v3.9 or older - /usr/local/tomcat/webapps/pinsafe/WEB-INF/conf
You must restore a config.xml from a valid and most recent backup.
Troubleshooting
/bin/rm: Argument list too long
This can occur where there are too many files to delete using the rm command. Either specify a specific file e.g. catalina.* or use the commands given above.
