Cloud storage gateway

A cloud storage gateway is a hybrid cloud storage device, implemented in hardware or software, which resides at the customer premises and translates cloud storage APIs such as SOAP or REST to block-based storage protocols such as iSCSI or Fibre Channel or file-based interfaces such as NFS or SMB.^[1]^[2]

According to a 2011 report by Gartner Group, cloud gateways were expected to increase the use of cloud storage by lowering monthly charges and eliminating the concern of data security.^[3]

Technology

Features

Modern applications (aka "cloud native applications") use network attached storage by means of REST and SOAP with hypertext transfer protocol on the protocol layer. The related storage is provided from arrays that offer these as object storage. Classic applications use network attached storage by means of Network File System NFS, iSCSI or Server Message Block SMB. To make use of all the advantages of object storage, existing applications need to be rewritten, and new applications must be object storage aware, which is not the case by default. This problem is addressed by cloud storage gateways. They offer object storage via classic native storage protocols like Network File System NFS or Server Message Block SMB (and a very few offer iSCSI as well). As a rule of thumb, classic applications with cloud native object storage can now be used with cloud storage gateways.

Functionality

In enterprise infrastructures, NFS is mainly used by Linux systems whereas Windows systems are using SMB. Object storage needs data in the form of objects rather than files.^[4] For all cloud storage gateways, it is mandatory to cache the incoming files and destage them to object storage on a later step. The time of destaging is subject to the gateway and a policy engine allows functions like

pinning = bind specific files to the cache and destage them only for mirroring purpose
content based destaging = move only files with specific characteristics to object storage e.g. all MP3 files
multi-cloud mirroring = mirror all files to two different object stores
Least Recently use = fill the local cache to maximum, move all files to object storage and delete files in cache on a LRU algorithm
encrypt prior of destage = files are encrypted on the cloud storage gateway and destaged to object storage in an encrypted form
compress and/or deduplication prior of destage = files are deduplicated and/or compressed prior of destaging
backup data in a native backup format

Combinations of these functions are usual.^[5] Default sorting schematics spanning the retrieval interface generally rely on zero-fault content processing, which carries the obvious requirement that two or more of the above functions are synchronized.^[6]

Extensions

Nearly all object storage gateways support Amazon S3 protocol as a quasi-standard. Some offer as well Microsoft Azure Blob, Google Storage, or Openstack SWIFT. Most gateways support public cloud storage e.g. from Amazon or Microsoft as an object store and Dropbox as a file drive store, there are as well a lot of vendors that support private cloud storage as well – including off and on prem storage.^[7]

Deployment methods

There are multiple variants to deploy such gateways – and some vendors support as well different variants as of their product line:

bare metal hardware appliance
software appliance supporting different hypervisors
software on top of an operating system – aka FUSE based

Software appliances as well as FUSE-based gateways can be installed on public cloud infrastructures.

Advantages

Cloud storage gateways avoid the need to change existing applications by providing a standard interface.

Additionally, IT users are used to existing protocols – like SMB or NFS. They can make use of cloud storage with the advantage of still using their existing infrastructures (including e.g. Active Directory, LDAP integration, file share functions etc.).

While cloud storage gateways initially covered a niche only, they got more attraction as of multi-cloud technologies. As an example: It is possible to run a cloud storage gateway in the form of a software appliance on top of a public or private cloud infrastructure by offering docker volume drivers that enable containers for automatic provisioning of storage used by these containers in a consistent form. They are using the hypervisors disks as a cache only, but destage data on least recently used algorithm to the underlying cloud storage.

The de facto standard for object storage is Amazon S3 – it had the most popularity and capacity installed on object storage.^[8] But every object storage vendor can (and most of them do) offer Amazon S3 storage – even there is no real "standard" S3 API: Every vendor is a little bit different in implementing S3 API (as seen from the different cloud storage gateway vendors supporting the "specific" APIs of the different object storage vendors). Since 2018, an increasing number of cloud storage gateways hide this complexity by offering S3 on northbound (as of networking technologies, southbound relates to the storage used by a gateway, whereas northbound is the storage provided by the gateway). As such, one may utilize a richer S3 implementation on northbound than the southbound supports.

Disadvantages

By using cloud storage gateways the complexity to use object storage is hidden, but that also hides some of the advantages of object storage:

the ability of horizontal scaling
ability to add high efficient metadata to the data content
to use extended WORM and archiving capabilities of object storage

As applications change to cloud-aware applications (aka called cloud native applications), cloud storage gateways will change from multiprotocol gateways to multi-cloud gateways, providing access to multiple cloud providers as well as multiple southbound protocols and act as relays between different clouds.

Market

As of 2020^[update] the cloud storage gateway market was valued at over USD 2 billion and was predicted to reach USD 11 billion by 2026, based on a report by the market research firm Mordor intelligence.^[9]