Documentation

Table of Contents API documentation

S3-compatible

Make sure to include druid-s3-extensions as an extension.

Deep Storage

S3-compatible deep storage is basically either S3 or something like Google Storage which exposes the same API as S3.

Configuration

Property Possible Values Description Default
druid.s3.accessKey S3 access key. Must be set.
druid.s3.secretKey S3 secret key. Must be set.
druid.storage.bucket Bucket to store in. Must be set.
druid.storage.baseKey Base key prefix to use, i.e. what directory. Must be set.

StaticS3Firehose

This firehose ingests events from a predefined list of S3 objects.

Sample spec:

"firehose" : {
    "type" : "static-s3",
    "uris": ["s3://foo/bar/file.gz", "s3://bar/foo/file2.gz"]
}
property description default required?
type This should be "static-s3" N/A yes
uris JSON array of URIs where s3 files to be ingested are located. N/A yes