Python 3 Script to Download Files From Google Drive & Upload to Amazon S3 Bucket Cloud Storage on Command Line

 

 

Download files from google drive and upload to S3

This repo contains a python tool to download files from a google drive folder, and to then upload the files to S3.

Features

  • Downloads the file to an in memory file handle and uploads from there without using precious disk space.
  • Operates on one file at a time.
  • Only speed limitation is network bandwith
  • Downloads can be picked up from where you left off using the paging feature.
  • Can take a file of known filenames and only upload files from google drive that match those.

First time authentication

  1. You will need to create a google drive app client for use with this script. You do this in your google API console.
  2. You will need to download the client secret file and call it client_secret.json. place it in the same folder as the script.
  3. On first run you’ll be asked to authenticate the app and allow it full access to your drive (needs this in order to access files shared with you)

Usage

This tool has built in help.

A typical command to download all files and upload to S3 would be:

A typical command to download only files which match a supplied checklist of files and upload to S3:

The checklist file contains a filename one per line.

You may need to process thousands of files but only want to work on them in distinct batches so that you can pick up where you left off.

The script defaults to 100 files per page, but this can be adjusted. This example processes a list of files in a google drive folder paging 10 at a time, starting at page 20 and ending at page 30.

 

 

 

Leave a Reply