Getting GCE instance to transfer log files to gcloud storage

Standard

We developers have a love / hate relationship with logs. Recently I struggled a lot to get them to transfer files automatically via cronjob to the bucket of what I want, but I finally managed to do it thanks to Google Support.

The following are the steps:

1) Prepare the bash script

#!/bin/bash
set -x
/usr/local/bin/gsutil -m mv <location of log files> gs://<Your bucket name>/<folder>

2) Prepare your crontab

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/home/<your username>/google-cloud-sdk/bin
HOME=/home/<your username>
BOTO_CONFIG="/home/<your username>/.config/gcloud/legacy_credentials/<folder/.boto"

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed

* * * * *  <your username

Do note that you need to set the PATH, HOME and BOTO_CONFIG on your crontab. If not it will not work. Make sure the paths are correct

3) Make sure that your instances have the access to google storage

Screen Shot 2015-02-27 at 7.32.22 PM

And you should be able to see your files saving to your desired google bucket

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s