Troubleshooting CentOS Web Server Logfile Diskspace Issue Tutorial
When it comes to categorizing the nature of major issues maintaining our RJM Programming domain’s CentOS Linux web server it’s easy to tell you what is top of the list …
diskspace
… both concerning …
- inode limits (a Linux/Unix/macOS problem only regarding the number of files on a web server) … and …
- diskspace (itself regarding the disk’s byte capacity, as for today’s scenario)
Constructing today’s tutorial picture we realized we could break the diskspace problem resolving issue into four parts …
- Identify diskspace problem … (yes, 94% is a problem on many Linux web servers, as you should allow at least 5% for the happiness of the Linux system’s running) …
df -k /
… resulting in Use% being (for us) …
94%
… in our case for a CentOS Linux web server at the RJM Programming domain - Hone in … and this is where we are constantly on the lookout for techniques that depend a bit on the problem, in that you may have more specific leads than we had for this occasion of “no leads” … so … looked it up on “the net” and arrived at the excellent disk space – Linux utility for finding the largest files/directories – Super User‘s excellent suggestion to try (having already “cd”ed to good “.” place on the command line ahead of time) …
find . -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
… the big test of which, for us, over tries of many different approaches we’ve tried being how long it takes … and this approach was reasonable (within half an hour) … and the resultant finding of immediate interest (and joy) for us was …
6.4G [theBiggestLogFileThatWeIdentifiedAsUnnecessaryToKeep]
- Fix for now …
rm -f [theBiggestLogFileThatWeIdentifiedAsUnnecessaryToKeep]
… resulting in …
df -k /
… resulting in Use% being (for us) …
88%
… aaaaahhhh! - Fix for from now on …
crontab -e # and create a record like (the 03:10 once a day timed task) below ...
10 3 * * * ksh -c "rm -f [theBiggestLogFileThatWeIdentifiedAsUnnecessaryToKeep]"
Compare this troubleshooting above to Troubleshooting CentOS Web Server Disk Zencart Issue Tutorial‘s more evidential web server diskspace issue.
Previous relevant Troubleshooting CentOS Web Server Disk Zencart Issue Tutorial is shown below.
Troubleshooting CentOS Web Server Disk Zencart Issue Tutorial
Out of the previous CentOS Exim Advanced Configuration Primer Tutorial‘s analysis phases …
- hard disk space
- hard disk inode count (for Linux and unix web servers)
… that latter one raised its-not-so-good-looking head for two days last Sunday (in the middle of the Sydney, Australia day) and Monday (up until resolving around 11am Sydney time). Not that I immediately knew that. Our symptoms of “something wrong” consisted of …
- all parts of Apache web server (for RJM Programming) websites working, including MySql ones
- cPanel hung
- FileZilla sftp got connection resets
… which is tempting to ignore, and hide the old noggin in the stuff that falls through the hourglass.
Luckily, though, my torpor was disturbed by needing to sftp an image over to the web server to complete yesterday’s tutorial.
Now, it’s inconvenient that the title gives “the general game away” about the source of the issue, but it would be an imaginative operator indeed who could jump to this “even general conclusion” from these symptoms, without getting access to this web server. And on reflection, had we used Power Management to see whether that would have had a chance of fixing the issue, it would have been a waste of time.
So what did we do first on Sunday? Well, we could still log in via ssh and on the RJM Programming web server, went …
# service sshd restart
Stopping sshd: [ OK ]
touch: cannot touch `/var/lock/subsys/sshd': No space left on device
- df -k /
- df -i /
… as per …
# df-k /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 133526580 112886392 13860736 90% /
root@vs-rmetcalfe [~]# df -k /tmp
Filesystem 1K-blocks Used Available Use% Mounted on
/usr/tmpDSK 544256 328968 187640 64% /tmp
root@vs-rmetcalfe [~]# df -k /var
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 133526580 112886456 13860672 90% /
root@vs-rmetcalfe [~]# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 8478720 8478720 0 100% /
… to arrive at “there is an inodes quota running out issue with the RJM Programming web server hard disk”.
So what did we do later on Sunday? We still could, believe it or not, from an already open FileZilla session, delete a goodly number of web server files, which we (recursively) did to get some thousands of inodes to become available again. That left the RJM Programming web server fully functional as we went to “shut eye places”. We gave the web server one task (via our Linux Disk Usage cPanel Heads Up Tutorial) to do for these sleeping hours …
# du -a / 2> /dev/null | sort -n -r | head -15
… but it petered out, alas. It is a very intensive “ask”, after all. So this takes us to Monday, where we woke up to …
- hard disk space
- hard disk inode count (for Linux and unix web servers)
… back to the original issue, again.
So what did we do on Monday? Well, amongst the research we lobbed onto this useful link, thanks, that got us to go …
find / -ctime -1 -print
… to get a list of files created over the last day. And that was the breakthrough, really, for us, as it lead us to seeing in that list thousands of entries for the filespec …
/home/virtfs/rjmprogr/home/rjmprogr/public_html/zencart/cache/myDEBUG-*.log
… and we noticed, after another round of file deletions to be okay, temporarily, again, when going through a few cycles of …
# ls -clt /home/virtfs/rjmprogr/home/rjmprogr/public_html/zencart/cache/myDEBUG-*.log
# ls -clt /home/virtfs/rjmprogr/home/rjmprogr/public_html/zencart/cache/myDEBUG-*.log
# ls -clt /home/virtfs/rjmprogr/home/rjmprogr/public_html/zencart/cache/myDEBUG-*.log
… each list would get considerably longer over the length of time it took to list the previous! Runaway train had us gobsmacked! Eventually, though, we tweaked to “the short term solution” …
# ksh -c 'for i in `find /home/virtfs/rjmprogr/home/rjmprogr/public_html/zencart/cache/ -name "myDEBUG-*.log"`; do rm -f $i; done'
# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 8478720 8428558 50162 100% /
# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 8478720 8429300 49420 100% /
… yayyyyy!!! (with a caveat of “just relief for now but must find the ultimate reason”) … but there were some back again on the next … a growing list again … plus the RJM Programming ZenCart eCommerce website came up with error message “WARNING: An Error occurred, please refresh the page and try again.” … mesmerized, we repeated the listings, watching the list grow, until it occurred to us, encouraged by that previous link talking about the error could be related to the MySql ZenCart database, to …
# cat /home/virtfs/rjmprogr/home/rjmprogr/public_html/zencart/cache/myDEBUG-1575848271-674789.log
[09-Dec-2019 07:37:51 Australia/Perth] PHP Fatal error: 145:Table './rjmprogr_zencart/sessions' is marked as crashed and should be repaired :: select value
from sessions
where sesskey = '42f9199e81e8c1eeb411921161c67ab9'
and expiry > '1575848271' in /home/rjmprogr/public_html/zencart/includes/classes/db/mysql/query_factory.php on line 101
… giving us the long term remedy …
- get into cPanel
- search for phpMyAdmin option, and click it
- access the rjmprogr_zencart database (and see that sessions table is indeed marked as crashed)
- within SQL tab textarea type …
REPAIR TABLE sessions
… click the GO button … successful
… sanity checked in that “the list” no longer grows!!! And as for the inodes quota issue …
# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 8478720 8425232 53488 100% /
… a lot better, especially as this time it only appears to be getting better (ie. IFree stays roughly the same or gets bigger (with our other crontab remedies we have implemented)). And ZenCart works again. What a relief!
Previous relevant CentOS Exim Advanced Configuration Primer Tutorial is shown below.
CentOS Exim Advanced Configuration Primer Tutorial
There are two file resource aspects to watch out for regarding a web server hard disk storage …
- hard disk space
- hard disk inode count (for Linux and unix web servers)
… which can, respectively, be monitored by Linux commands …
- df -k /
- df -i /
On our CentOS rjmprogramming.com.au web server recently we had occasion to do a check of this, and wanted to improve the web server hard disk situation for both measures, picking the /var folder of our web server. So we executed, respectively …
- find /var -xdev -type f -size +100M
- find /var -mtime -1 -ls
… the latter one being a list of files on /var modified in the last day being our idea to try to see what daily log filing might be contributing big time to hard disk inode usage. And that’s where we found out this way that our Exim Mail Server logs extensively, and that we could do without those in folders off …
/var/spool/exim/msglog/
We tried the latter of two deletion ideas we show below …
- all at once via …
cd /var/spool/exim
find msglog -type f -exec rm -rf {} \;
- broken up … via a series of deletions of the ilk …
cd /var/spool/exim/msglog
rm -f A/*
rm -f a/*
Then the next job was to stop Exim remaking these logs and wish to thank this useful link for great advice here, that led to this advice we give …
- log in to WHM cPanel
- in search bar type “Exim”
- click the “Exim Configuration Manager”
- click “Advanced Editor” tab (up the top)
- click the blue “Add additional configuration setting” button well down the webpage
- click “message_logs” in left hand drop down
- set value of this setting to “false”
- click blue “Save” button down the bottom of webpage to complete the steps here
You may find this is a web server configuration of interest to you too.
If this was interesting you may be interested in this too.
If this was interesting you may be interested in this too.
If this was interesting you may be interested in this too.
