Node.js Project to Stream Live Data & Files to Amazon S3 Cloud Storage Using multipart Upload API in Browser Using Javascript

 

 

s3-upload-stream Build Status Changelog Status

A pipeable write stream which uploads to Amazon S3 using the multipart file upload API.

Changelog

1.0.6 (2014-10-20)

Removing global state, and adding pause and resume functionality.

Historical Changelogs

Why use this stream?

  • This upload stream does not require you to know the length of your content prior to beginning uploading. Many other popular S3 wrappers such as Knox also allow you to upload streams to S3, but they require you to specify the content length. This is not always feasible.
  • By piping content to S3 via the multipart file upload API you can keep memory usage low even when operating on a stream that is GB in size. Many other libraries actually store the entire stream in memory and then upload it in one piece. This stream avoids high memory usage by flushing the stream to S3 in 5 MB parts such that it should only ever store 5 MB of the stream data at a time.
  • This package is designed to use the official Amazon SDK for Node.js, helping keep it small and efficient. For maximum flexibility you pass in the aws-sdk client yourself, allowing you to use a uniform version of AWS SDK throughout your code base.
  • You can provide options for the upload call directly to do things like set server side encryption, reduced redundancy storage, or access level on the object, which some other similar streams are lacking.
  • Emits “part” events which expose the amount of incoming data received by the writable stream versus the amount of data that has been uploaded via the multipart API so far, allowing you to create a progress bar if that is a requirement.
  • Support for pausing and later resuming in progress multipart uploads.

Limits

  • The multipart upload API does not accept parts less than 5 MB in size. So although this stream emits “part” events which can be used to show progress, the progress is not very granular, as the events are only per part. By default this means that you will receive an event each 5 MB.
  • The Amazon SDK has a limit of 10,000 parts when doing a mulitpart upload. Since the part size is currently set to 5 MB this means that your stream will fail to upload if it contains more than 50 GB of data. This can be solved by using the ‘stream.maxPartSize()’ method of the writable stream to set the max size of an upload part, as documented below. By increasing this value you should be able to save streams that are many TB in size.

Example

Usage

Before uploading you must configure the S3 client for s3-upload-stream to use. Please note that this module has only been tested with AWS SDK 2.0 and greater.

This module does not include the AWS SDK itself. Rather you must require the AWS SDK in your own application code, instantiate an S3 client and then supply it to s3-upload-stream.

The main advantage of this is that rather than being stuck with a set version of the AWS SDK that ships with s3-upload-stream you can ensure that s3-upload-stream is using whichever verison of the SDK you want.

When setting up the S3 client the recommended approach for credential management is to set your AWS API keys using environment variables or IAM roles.

If you are following this approach then you can configure the S3 client very simply:

However, some environments may require you to keep your credentials in a file, or hardcoded. In that case you can use the following form:

client.upload(destination)

Create an upload stream that will upload to the specified destination. The upload stream is returned immeadiately.

The destination details is an object in which you can specify many different destination properties enumerated in the AWS S3 documentation.

Example:

client.upload(destination, [session])

Resume an incomplete multipart upload from a previous session by providing a session object with an upload ID, and ETag and numbers for each part. destination details is as above.

Example:

Stream Methods

The following methods can be called on the stream returned by from client.upload()

stream.pause()

Pause an active multipart upload stream.

Calling pause() will immediately:

  • stop accepting data from an input stream,
  • stop submitting new parts for upload, and
  • emit a pausing event with the number of parts that are still mid-upload.

When mid-upload parts are finished, a paused event will fire, including an object with UploadId and Parts data that can be used to resume an upload in a later session.

stream.resume()

Resume a paused multipart upload stream.

Calling resume() will immediately:

  • resume accepting data from an input stream,
  • resume submitting new parts for upload, and
  • echo a resume event back to any listeners.

It is safe to call resume() at any time after pause(). If the stream is between pausing and paused, then resume() will resume data flow and the paused event will not be fired.

stream.maxPartSize(sizeInBytes)

Used to adjust the maximum amount of stream data that will be buffered in memory prior to flushing. The lowest possible value, and default value, is 5 MB. It is not possible to set this value any lower than 5 MB due to Amazon S3 restrictions, but there is no hard upper limit. The higher the value you choose the more stream data will be buffered in memory before flushing to S3.

The main reason for setting this to a higher value instead of using the default is if you have a stream with more than 50 GB of data, and therefore need larger part sizes in order to flush the entire stream while also staying within Amazon’s upper limit of 10,000 parts for the multipart upload API.

stream.concurrentParts(numberOfParts)

Used to adjust the number of parts that are concurrently uploaded to S3. By default this is just one at a time, to keep memory usage low and allow the upstream to deal with backpressure. However, in some cases you may wish to drain the stream that you are piping in quickly, and then issue concurrent upload requests to upload multiple parts.

Keep in mind that total memory usage will be at least maxPartSize * concurrentParts as each concurrent part will be maxPartSize large, so it is not recommended that you set both maxPartSize and concurrentParts to high values, or your process will be buffering large amounts of data in its memory.

Tuning configuration of the AWS SDK

The following configuration tuning can help prevent errors when using less reliable internet connections (such as 3G data if you are using Node.js on the Tessel) by causing the AWS SDK to detect upload timeouts and retry.

Leave a Reply