March 3, 2017

Amazon S3 as a File System in Amazon EC2

As a cloud application developer, shared storage on EC2 is particularly challenging when GBs of data has to be accessed by 100s of EC2 instances. One particular option we explored is leveraging our Amazon S3 storage to mount on EC2 instances and use as a shared drive. Most popular software solutions for this approach are:

  • FusionOverAmazon (S3FS)
  • SubCloud NOTE: Subcloud offering was discontinued after this blog has been published.

Both solutions can mount Amazon S3 buckets as file system in Amazon EC2 instance and also on multiple instances at a same time. "use cache" option (provided by both) is particularly efficient when accessing same data multiple times on a same instance. When using this option, a local copy is maintained and operated on it to minimise the downloads from S3.

FusionOverAmazon is freely available where as SubCloud is commercial version. For an enterprise level application development, we are looking for a support and it is provided by SubCloud. Hence our choice is SubCloud.

Please feel free to comment on your approach/choice.

Topics: AWS, Big data technology, EC2, S3