echo "

Automatically clean up logs with a cron job

Since a lot of our applications generate enormous log files while running on Tomcat, Play, ... We have to intervene and clean them up and since zipping those logs makes a huge improvement on disk space consumption this is how I do it. Of course you can use ZFS with native disk compression too :)

I install this cron (as the user that runs the application). It will clean up all the logs that aren't in use and will clean up the archived logs after 30 days. And the neat thing is that recent versions of Vim kan still read the log while zipped, deflating it on the fly.

find /opt/ -type f \( -mtime +30 -iname "*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].log.gz" -exec rm -f {} \; -o -iname "*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].log" -exec gzip {} \; \)

 

Make a CSV file of all the cronjobs on all the systems managed by ansible

To get a monthly overview of all the cronjobs on all the systems I wrote a wrapper (in bash) to create the CSV lines which with the help of an ansible playbook to generate a csv file.

This is the wrapper (collect_cron.sh):

#!/bin/bash
#this script will return a CSV file containing the server,user,cronjob
##
#this is set to be able to use filters on wildcards
shopt -s extglob
#here we store the hostname since we only need to declare this once
HOST=$(hostname|cut -d"." -f1)
#here we start looping through all the cron files exept the ones filtered by the pipe seperated list
for f in $(ls /var/spool/cron/*;ls /etc/cron.d/!(*@(sysstat|0hourly)) 2>>/dev/null )
do
        #here we store the content of the current cron file
        COMMAND=$(cat $f)
        #here we loop over the individual jobs in the file while filtering out comments and empty lines
        echo "$COMMAND" | sed /^#/d | sed /^\s*$/d | while read line;
        do
                #here we start printing a line for our CSV file
                #starting with the host
                printf $HOST","
                #here we check if it is a user or a system cron and we print accordingly
                if [[ $f == /var/spool/cron/* ]];
                then
                        USER=${f##*/}
                        printf $USER","
                else
                        printf "system,"
                fi
                #and finally here we print the actual command and since we desire a new line echo is used here instead of printf
                echo "$line"
        done
done

 And the matching playbook (made by a colleague):

- hosts: all
  #gather_facts: no
  tasks:
#  - name: create folder
#    local_action: file dest=/tmp/cron_collect state=directory owner=root group=root mode=0700
  - block:
    - name: "collect crons on system"
      script: "{{ playbook_dir}}/../../scripts/collect_cron.sh"
      register: crons
      ignore_errors: yes
    - name: move to csv file
      local_action: copy content={{ crons.stdout }} dest=/opt/systems/cron_collect/{{ansible_fqdn}}.csv
- hosts: localhost:ansibleserver01
  gather_facts: no
  tasks:
  - name: combine into one file
    assemble:
      src: /opt/systems/cron_collect/
      dest: /tmp/croncollection.csv
      owner: bescorli
      group: sysauto
      mode: 0640
  - name: remove blanks
    lineinfile:
      dest: /tmp/croncollection.csv
      regexp: '^\s$'
      state: absent

 

Home