Tomcat Log Rotation, Compression, Encryption (GDPR Compliance) and Copy to AWS S3

26/09/2018

In production environment the logs of applications are important, we need to store them for compliance reasons and make sure that the log files dont fill up the disk space. Since the EU GDPR Privacy law has been passed, the log files need to be encrypted.

In Linux we have a beautiful log management utility logrotate, which is run regularly to rotate the logs files of apt, dpkg, cups-daemon, rsyslog, ufw etc. configuration of these logs are found in /etc/logrotate.d

I am running Tomcat 9 and CATALINA_HOME=/opt/tomcat. You can install Tomcat 9 from here. Tomcat rotates the logs by itself in a default setup. which is configured in /opt/tomcat/conf/logging.properties file. we need explicitly configure tomcat to not to rotate the log files like catalina.out itself, since we will be rotating and compressing them using system logrotate utility.

Add this lines to /opt/tomcat/conf/logging.properties

I have added one line to each log entry org.apache.juli.AsyncFileHandler.rotatable = false to stop rotation of logs by tomcat, Once this is done tomcat will not rotate the logs by itself.

Logrotate and Compression

create a new file at /etc/logrotate.d/tomcat

compress – compress the log files compresscmd – command used for Compression, gzip compressoptions – options for compression command, Gzip Compression level 1-9 compressext – extention of compressed file

its just going to do gzip -9 filename , which compresses the file.

dateext dateformat -%Y-%m-%d-%s these lines add date extention to the rotated log file in the format catalina.out-2018-09-21-1537501441

nomail tells not mail the log files

copytruncate – copies the log file to new file and truncates the log file to zero bytes.

daily – rotates the log file daily

rotate 10 – keep the 10 past rotated logs

notifempty – do not rotate the log if it is empty

missingok – if the file is missing its ok

create 0644 tomcat tomcat – create the log file with these permissions, ignored if copy truncate is used.

postrotate – command to run after the log rotation.

Here we are running the a custom Bash command to encrypt and move the log files to AWS S3

Encryption and Copying to AWS S3

Create a Bash Shell Script vim /usr/share/encrypt

Give it execute permissions

chmod +x /usr/share/encrypt

This script will encrypt the file with .gz which are compressed files and accept the two arguments location of gz files and Encryption key

encrypt /opt/tomcat/log /data/key

Key is needed to encrypt and decrypt the file

cat /data/key
Big3secret

Encrypt command will get list of files in given location and if the file has .gz extention the it will encrypt it with gpg (GNU Privacy Guard), move all the encrypted files which has .gpg extention to aws s3 bucket and once uploaded it will delete the encrypted file from that location. For copying the encrypted logs to S3 make sure that the ec2 instance has write to S3 IAM Access Role assigned to it.

For Decryption of files

gpg --batch --passphrase-file /data/mykey -d catalina.out-2018-09-21-1537501441.gz.gpg > catalina.out-2018-09-21-1537501441.gz