At Amazon Web Services (AWS), the hybrid idea is by no means limited to allowing companies to extend the scope of their corporate network securely to the AWS Virtual Private Cloud with the help of various managed virtual private network (VPN) services or by way of AWS Direct Connect. The hybrid approach is hiding around almost every corner. The goal of AWS Storage Gateway is to provide local applications transparent access to the most important AWS storage service by far: Simple Storage Service (S3). AWS Storage Gateway extends existing enterprise environments with native integration into AWS services.

S3 Role

S3 is the oldest and most important cloud service. It existed long before the official launch of AWS in 2006. S3 is important to AWS as an object store, because the entire AWS service portfolio is multitiered, with higher level managed platform services built on the base of AWS infrastructure and foundation services, each of which needs to store data and states in some way.

Despite third-party solutions such as the S3 browser, S3 is primarily designed for direct queries, which means that AWS users can use S3 to perform advanced queries on stored data without extracting, transforming, or loading the data onto a separate analysis platform. Direct querying of data in S3 increases performance and keeps costs low for analysis applications that use S3 as a data pool. S3 has several direct query options, including the new S3 Select, Amazon Athena, or Amazon Redshift Spectrum.

With S3 Select and AWS Lambda, you can even create serverless apps that use direct processing of S3 Select. Amazon Athena, on the other hand, is an interactive query service designed to simplify analysis of the data in S3 with the use of standard SQL queries. Athena is also serverless, so customers do not need to set up and manage an infrastructure. Nevertheless, it is often inconvenient to allow on-premises applications to use AWS storage resources, which is why AWS introduced the Storage Gateway service in 2012.

Basics and Interfaces

AWS Storage Gateway was originally designed as a seamless cloud backup and disaster recovery solution for local data. Locally stored information is automatically saved in S3. The service simply allows hybrid storage between local environments and the AWS cloud. Organizations can seamlessly integrate local applications and workflows with Amazon block and object cloud storage services. The offer primarily provides for use scenarios such as backup, archiving, disaster recovery, cloud bursting, storage tiering, or the migration of data to AWS.

Technically, AWS provides the Storage Gateway in the form of a virtual machine (or EC2 instance) that the user starts on a local server or in their own data center. With the AWS Management Console, for example, gateway storage volumes with a capacity of up to 32TB can be created and integrated into existing systems as iSCSI devices. The user connects their local applications to Amazon S3 with the gateway appliance according to standard storage protocols like iSCSI or NFS.

The gateway not only provides space for volumes in AWS, but also for files and virtual tapes. For high-performance integration, you can find technologies such as an optimized data transfer mechanism, bandwidth management, automated network stability, and support for local cache storage, which enables even faster local access to the most frequently used data, with the data being permanently stored in the Amazon cloud in the background.

Each Storage Gateway supports three storage interfaces – file, volume, and tape – but can only serve one interface type at any given time. The Volume Gateway provides applications with block storage via the iSCSI protocol. These volumes are backed up in Amazon S3. The File Gateway lets users store and retrieve objects in Amazon S3 with file protocols such as NFS. In contrast to the Volume Gateway, the objects written by the File Gateway are directly accessible in S3. Finally, the Tape Gateway acts as an S3 entry point for classic backup applications by providing an interface for the iSCSI Virtual Tape Library (VTL), which comprises a virtual media changer, virtual tape drives, and virtual tapes. Virtual tape data can be either stored in Amazon S3 or archived in AWS Glacier.

Understanding the Volume Gateway

The Volume Gateway is basically an iSCSI target that creates volumes and assigns them to local servers (or EC2 instances) as iSCSI LUNs. The Volume Gateway can run in either cached or stored mode. Cached mode writes the user's primary data to S3 but keeps frequently accessed data in a local cache, allowing low-latency access.

In stored mode, the user's primary data is initially stored locally, so that all data remains available for quick access at all times. Backups in S3 only occur asynchronously in the background. In both modes, users can also create time-based and space-efficient snapshots of their volumes and store them in S3 for reuse at any time. Direct access to the volumes is not possible in this way. However, users can create new Elastic Block Store (EBS) volumes from the snapshots at any time and use them in AWS.