echo "

Automatically clean up logs with a cron job

Since a lot of our applications generate enormous log files while running on Tomcat, Play, ... We have to intervene and clean them up and since zipping those logs makes a huge improvement on disk space consumption this is how I do it. Of course you can use ZFS with native disk compression too :)

I install this cron (as the user that runs the application). It will clean up all the logs that aren't in use and will clean up the archived logs after 30 days. And the neat thing is that recent versions of Vim kan still read the log while zipped, deflating it on the fly.

find /opt/ -type f \( -mtime +30 -iname "*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].log.gz" -exec rm -f {} \; -o -iname "*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].log" -exec gzip {} \; \)

 

Shushing a syslog spammer

Today I had a machine of which the /var/log/messages was drowning in DEBUG and TRACE messages from the spring framerwork used by Tomcat. To prevent the partition from running full I made a temporary workaround until our programmer disabled the extensive debugging. I found this sort of lines in /var/log/messages:

May 21 03:38:54 tomcatserver current: 03:38:54.431 [XNIO-3 I/O-3] TRACE org.xnio.nio.selector - Selected on sun.nio.ch.EPollSelectorImpl@15e3ed9
May 21 03:38:54 tomcatserver current: 03:38:54.431 [XNIO-3 task-15] TRACE org.xnio.safe-close - Closing resource io.undertow.servlet.core.ServletBlockingHttpExchange@3168bcd7
May 21 03:38:54 tomcatserver current: 03:38:54.431 [XNIO-3 I/O-3] TRACE org.xnio.nio.selector - Beginning select on sun.nio.ch.EPollSelectorImpl@15e3ed9 (with timeout)
May 21 03:38:54 tomcatserver current: 03:38:54.433 [XNIO-3 I/O-3] TRACE org.xnio.nio.selector - Selected on sun.nio.ch.EPollSelectorImpl@15e3ed9
May 21 03:38:54 tomcatserver current: 03:38:54.434 [XNIO-3 I/O-3] TRACE org.xnio.nio.selector - Selected key sun.nio.ch.SelectionKeyImpl@e68ade3 for java.nio.channels.SocketChannel[connected oshut local=/164.35.83.148:9000 remote=/10.68.64.38:45856]
May 21 03:38:54 tomcatserver current: 03:38:54.434 [XNIO-3 I/O-3] TRACE org.xnio.listener - Invoking listener io.undertow.util.ConnectionUtils$4@1d0ccab7 on channel org.xnio.conduits.ConduitStreamSourceChannel@56407229
May 21 03:38:54 tomcatserver current: 03:38:54.434 [XNIO-3 I/O-3] TRACE org.xnio.nio - Cancelling key sun.nio.ch.SelectionKeyImpl@e68ade3 of java.nio.channels.SocketChannel[connected oshut local=/164.35.83.148:9000 remote=/10.68.64.38:45856] (same thread)
May 21 03:38:54 tomcatserver current: 03:38:54.434 [XNIO-3 I/O-3] TRACE org.xnio.listener - Invoking listener io.undertow.server.AbstractServerConnection$CloseSetter@788b3120 on channel org.xnio.nio.NioSocketStreamConnection@5ab7faac

/etc/rsyslog.d/tomcat_silencer.conf

if $programname == 'current' then ~

Where current is the name of the application that is spamming. You could also use a filter for DEBUG and TRACE messages if you like.

:msg, contains, "DEBUG" ~
:msg, contains, "TRACE" ~

Either way after you adapt the config of syslog restart and optionally run a logrotate

service rsyslog restart
logrotate --force /etc/logrotate.d/syslog
Home