Patent application number | Description | Published |
20100174731 | Distributed Storage System With Web Services Client Interface - A distributed, web-services based storage system. A system may include a web services interface configured to receive, according to a web services protocol, a given client request for access to a given data object, the request including a key value corresponding to the object. The system may also include storage nodes configured to store replicas of the objects, where each replica is accessible via a respective unique locator value, and a keymap instance configured to store a respective keymap entry for each object. For the given object, the respective keymap entry includes the key value and each locator value corresponding to replicas of the object. A coordinator may receive the given client request from the web services interface, responsively access the keymap instance to identify locator values corresponding to the key value and, for a particular locator value, retrieve a corresponding replica from a corresponding storage node. | 07-08-2010 |
20120173558 | Receiver-Side Data Deduplication In Data Systems - Methods and apparatus for receiving uploaded data from a sender at a receiver. A data deduplication technique is described that may reduce the bandwidth used in uploading data from the sender to the receiver. In the technique, the receiver, rather than the sender, maintains a fingerprint dictionary for previously uploaded data. When a sender has additional data to be uploaded, the sender extracts fingerprints for units of the data and sends the fingerprints to the receiver. The receiver checks its fingerprint dictionary to determine the data units to be uploaded and notifies the sender of the identified units, which then sends the identified units of data to the receiver. The technique may, for example, be applied in virtualized data store systems to reduce bandwidth usage in uploading data. | 07-05-2012 |
20120173656 | Reduced Bandwidth Data Uploading in Data Systems - Methods and apparatus for uploading data from a sender to a receiver. A data deduplication technique is described that may reduce the bandwidth used in uploading data from the sender to the receiver. In the technique, the receiver, rather than the sender, maintains a fingerprint dictionary for previously uploaded data. When a sender has additional data to be uploaded, the sender extracts fingerprints for units of the data and sends the fingerprints to the receiver. The receiver checks its fingerprint dictionary to determine the data units to be uploaded and notifies the sender of the identified units, which then sends the identified units of data to the receiver. The technique may, for example, be applied in virtualized data store systems to reduce bandwidth usage in uploading data. | 07-05-2012 |
20120226712 | Distributed Storage System With Web Services Client Interface - A distributed, web-services based storage system. A system may include a web services interface configured to receive, according to a web services protocol, a given client request for access to a given data object, the request including a key value corresponding to the object. The system may also include storage nodes configured to store replicas of the objects, where each replica is accessible via a respective unique locator value, and a keymap instance configured to store a respective keymap entry for each object. For the given object, the respective keymap entry includes the key value and each locator value corresponding to replicas of the object. A coordinator may receive the given client request from the web services interface, responsively access the keymap instance to identify locator values corresponding to the key value and, for a particular locator value, retrieve a corresponding replica from a corresponding storage node. | 09-06-2012 |
20130007183 | Methods And Apparatus For Remotely Updating Executing Processes - Methods, apparatus, and computer-accessible storage media for remotely updating an executing process that receives I/O requests on I/O port(s), stores write data to a write log on local storage, and uploads data from the write log to remote storage. An update for the process is detected and downloaded, and an updated process is instantiated from the update. The current process is directed to perform a shutdown for update during an update window. In response, the current process saves its current configuration, flushes an in-memory portion of the write log to local storage, and releases its I/O port(s). The updated process loads the saved configuration, detects that the port(s) have been released, and starts accepting I/O requests on the ports. During flushing, the current process flushes current data in memory while continuing to append new write data, stops accepting new write requests, and then flushes the new write data. | 01-03-2013 |
20130007219 | Shadowing Storage Gateway - Methods, apparatus, and computer-accessible storage media for shadowing data stored on a local store to a remote store provided by a service provider. A gateway may be configured as a shadowing gateway on a customer network in response to receiving configuration information. The shadowing gateway may receive reads and writes to the local store. The gateway passes the requests to the local store, and also uploads write data indicated by the writes to the service provider to update a snapshot of the local store maintained by the service provider on the remote store. The write data may be buffered to a write log for uploading, and may be uploaded as blocks according to a block storage format used by the service provider. The shadowing process may be transparent to processes on the customer network. The shadowed data may be used to recover data on the local store. | 01-03-2013 |
20130007854 | Storage Gateway Activation Process - Methods, apparatus, and computer-accessible storage media for activating a gateway to a remote service provider. The gateway serves as an interface between processes on a customer network and the provider, for example to store customer data to a remote data store. A gateway sends a public key and metadata describing the gateway to the provider. The gateway receives an activation key from the provider and exposes the activation key on the customer network. The customer obtains the key and communicates to the provider using the key to provide customer information including a name for the gateway and to authorize registration of the gateway. The provider provides the customer information to the gateway. The gateway requests security credentials from the provider using the customer information and the key. The provider sends a security credential to the gateway. The gateway may then obtain configuration information from the customer via the provider. | 01-03-2013 |
20130013571 | MANAGEMENT OF OBJECT MAPPING INFORMATION CORRESPONDING TO A DISTRIBUTED STORAGE SYSTEM - Systems and methods for managing mapping information for objects maintained in a distributed storage system are provided. The distributed storage system can include a keymap subsystem that manages the mapping information according to object keys. Requests for specific object mapping information are directed to specific keymap coordinators within the keymap subsystem. Each keymap coordinator can maintain a cache for caching mapping information maintained at various information sources. To manage the cache, the keymap system can utilize information placeholders that replace previously cached keymap information while a request to modify keymap information is being processed by the information sources. Each keymap coordinator can process subsequently received keymap information read requests in the event an information placeholder is cached as the current cached keymap information. | 01-10-2013 |
20130212165 | DISTRIBUTED STORAGE SYSTEM WITH WEB SERVICES CLIENT INTERFACE - A distributed, web-services based storage system. A system may include a web services interface configured to receive, according to a web services protocol, a given client request for access to a given data object, the request including a key value corresponding to the object. The system may also include storage nodes configured to store replicas of the objects, where each replica is accessible via a respective unique locator value, and a keymap instance configured to store a respective keymap entry for each object. For the given object, the respective keymap entry includes the key value and each locator value corresponding to replicas of the object. A coordinator may receive the given client request from the web services interface, responsively access the keymap instance to identify locator values corresponding to the key value and, for a particular locator value, retrieve a corresponding replica from a corresponding storage node. | 08-15-2013 |
20140046906 | ARCHIVAL DATA IDENTIFICATION - Methods and systems are provided herein that facilitate cost-effective and reliable data identification in an archival data storage system. In an embodiment, a data object stored in an archival data storage system is identified by a data object identifier which encodes storage location information that may be used to locate a data object stored in an archival data storage system that reduces the cost to store a global index. The data object identifier may also encode policy information such as access control information usable for validating subsequent access to the data object, payload validation information such as size and digest usable for verifying the integrity of the payload data, metadata validation information such as error-detection codes usable for validating integrity of the data object identifier and other information. | 02-13-2014 |
20140310390 | ASYMMETRIC PACKET FLOW IN A DISTRIBUTED LOAD BALANCER - A distributed load balancer in which a router receives packets from at least one client and routes packet flows to multiple ingress servers. For unknown packet flows, an ingress server cooperates with primary and secondary flow trackers to establish connections to server nodes. For known packet flows, the ingress server sends the packets to target server nodes. The server nodes randomly select egress servers for outgoing packets of the packet flows. The ingress servers, flow trackers, and egress servers are implemented by multiple load balancer nodes in a load balancer node layer. The ingress and egress servers for a given packet flow may be on different load balancer nodes. The load balancer nodes may use a consistent hash function to compute a consistent hash ring for the nodes according to packet flow client/public endpoint pairs so that nodes associated with given packet flows can be located. | 10-16-2014 |
20140310391 | MULTIPATH ROUTING IN A DISTRIBUTED LOAD BALANCER - A distributed load balancer in which a router receives packets from at least one client and routes packet flows to multiple load balancer nodes. The router exposes a public IP address and the load balancer nodes all advertise the same public IP address to the router. The router may implement a per-flow hashed multipath routing technique, for example an equal-cost multipath (ECMP) routing technique, to distribute the flows across the load balancer nodes. Thus, the multiple load balancer nodes may service a single public endpoint. The load balancer nodes may advertise to the router according to the Border Gateway Protocol (BGP). Rather than advertising itself, however, a load balancer node may be advertised to the router by one or more neighbor load balancer nodes; the neighbor nodes may terminate the BGP sessions with the router in response to determining that the load balancer node has failed. | 10-16-2014 |
20140310417 | CONNECTION PUBLISHING IN A DISTRIBUTED LOAD BALANCER - A connection publishing method for a distributed load balancer in which a router receives packets from at least one client and routes packet flows to multiple load balancer (LB) nodes, which in turn distribute the packet flows among multiple server nodes. A load balancer (LB) module on each server node determines and publishes a list of active connections between client(s) and the server (a connection publishing packet) back to the LB nodes. The LB module may randomly select a LB node to receive the packet. Upon receiving the packet, the LB node processes the connection information in the packet, distributing the connection information among appropriate LB nodes. Receiving the connection publishing information from the server nodes refreshes leases for the connections on the LB nodes. | 10-16-2014 |
20140310418 | DISTRIBUTED LOAD BALANCER - A distributed load balancer in which a router receives packets from at least one client and routes packet flows to multiple load balancer (LB) nodes according to a per-flow hashed multipath routing technique. For a given packet flow, the LB nodes randomly select a server node as a target for the packet flow from among multiple server nodes and send a connection request to the server node. A load balancer module on the server node makes the decision on whether to accept or reject the connection based on one or more metrics indicating a respective server's current load. If the module accepts the connection request, a connection is established between the server and the client. Otherwise, the load balancer nodes may select another server node and try again. The connections established between clients and servers pass through the load balancer nodes, but are not terminated at the load balancer nodes. | 10-16-2014 |
20140337483 | REDUNDANT STORAGE GATEWAYS - Methods, apparatus, and computer-accessible storage media for providing redundant storage gateways. A client may create a storage gateway group and add storage gateways to the group. The client may assign one or more volumes on a remote data store to each the storage gateways in the group. Volume data for each storage gateway in the group may be replicated to at least one other storage gateway in the group. If one of the gateways in the group becomes unavailable, one or more other gateways in the group may take over volumes previously assigned to the unavailable gateway, using the replicated data in the group to seamlessly resume gateway operations for the respective volumes. Client processes that previously communicated with the unavailable gateway may be manually or automatically directed to the gateway(s) that are taking over the unavailable gateway's volumes. | 11-13-2014 |
20140351906 | STORAGE GATEWAY ACTIVATION PROCESS - Methods, apparatus, and computer-accessible storage media for activating a gateway to a remote service provider. The gateway serves as an interface between processes on a customer network and the provider, for example to store customer data to a remote data store. A gateway sends a public key and metadata describing the gateway to the provider. The gateway receives an activation key from the provider and exposes the activation key on the customer network. The customer obtains the key and communicates to the provider using the key to provide customer information including a name for the gateway and to authorize registration of the gateway. The provider provides the customer information to the gateway. The gateway requests security credentials from the provider using the customer information and the key. The provider sends a security credential to the gateway. The gateway may then obtain configuration information from the customer via the provider. | 11-27-2014 |
20140359698 | SYSTEM AND METHOD FOR DISTRIBUTED LOAD BALANCING WITH DISTRIBUTED DIRECT SERVER RETURN - Embodiments may include a load balancer that receives a request packet sent by a remote client to an original destination address of multiple network addresses serviced by the load balancer, and selects according to a load balancing protocol, a host computer of a plurality of host computers to process the request. The load balancer may, from among a plurality of ports on the selected host computer, select a particular port having a one-to-one association with the original destination address, the association specified by mapping information accessible to the load balancer, and send the request packet to the selected port on the selected host computer. The mapping information accessible to the selected host computer specifies a one-to-one association between the selected port and the original destination address. Sending the request packet to the selected port conveys that address to the selected server without that address being included in that packet. | 12-04-2014 |
20140372381 | METHODS AND APPARATUS FOR DATA RESTORE AND RECOVERY FROM A REMOTE DATA STORE - Methods, apparatus, and computer-accessible storage media for restoring data from a snapshot to a data volume. The blocks in the volume may be treated as an implicit tree structure, for example a binary tree; each local block corresponds to a block on the snapshot. A local block on the volume may be marked, for example fingerprinted with metadata, to indicate that the local block has not been restored. Initially, the local block at the root node is marked. To restore a local block, the restore process may generate a list indicating all local blocks on a path from the root node of the tree to the target node that have not been restored. The marks in the local blocks are used in generating the list. For each block indicated in the list, children of the block are fingerprinted, and the block is restored from the snapshot. | 12-18-2014 |
20150116861 | HARD DISK DRIVE ASSEMBLY WITH FIELD-SEPARABLE MECHANICAL MODULE AND DRIVE CONTROL - A system for storing data includes a rack, one or more data storage drive assemblies coupled to the rack, and a data control module coupled to the rack. The data storage drive assemblies include one or more drive mechanical modules configured to store data and one or more drive control modules coupled to the drive mechanical modules. The drive control modules control mechanical operations in the drive mechanical modules. The drive mechanical modules and the associated drive control modules are separable from one another without removing the other module from the at least one data storage drive assembly. | 04-30-2015 |
20150242150 | METHODS AND APPARATUS FOR OPTIMIZING RESOURCE UTILIZATION IN DISTRIBUTED STORAGE SYSTEMS - Methods and apparatus for optimizing resource utilization in distributed storage systems. A data migration technique is described that may operate in the background in a distributed storage data center to migrate data among a fleet of storage units to achieve a substantially even and randomized data storage distribution among all storage units in the fleet. When new storage units are added to the fleet and coupled to the data center network, the new storage units are detected. Instead of processing and storing new data to the newly added storage units, as in conventional distributed storage systems, the new units are blocked from general client I/O to allow the data migration technique to migrate data from other, previously installed storage hardware in the data center onto the new storage hardware. Once the storage load on the new storage units is balanced with the rest of the fleet, the new storage units are released for general client I/O. | 08-27-2015 |
20150278243 | SCALABLE FILE STORAGE SERVICE - A client request, formatted in accordance with a file system interface, is received at an access subsystem of a distributed multi-tenant storage service. After the request is authenticated at the access subsystem, an atomic metadata operation comprising a group of file system metadata modifications is initiated, including a first metadata modification at a first node of a metadata subsystem of the storage service and a second metadata modification at a second node of the metadata subsystem. A plurality of replicas of at least one data modification corresponding to the request are saved at respective storage nodes of the service. | 10-01-2015 |
20150304407 | SYSTEM AND METHOD FOR DISTRIBUTED LOAD BALANCING WITH LOAD BALANCER CLIENTS FOR HOSTS - Embodiments may include a load balancer configured to, for a given packet received from a remote client, select a host computer of a plurality of host computers according to a load balancing protocol, the selection based on load information updates received from the plurality of host computers. Each load information update may indicate a measure of the respective host computer's capacity to service additional connections with remote clients. The load balancer may provide the given packet to the selected host computer for processing by an application component on the selected host computer, and subsequent to determining that a specific application component on one of the plurality of host computers does not have capacity for additional connections based on the load information updates, continue to send packets for existing connections serviced by the specific application component to the respective host computer without sending packets for new connections to that host computer. | 10-22-2015 |
20150365345 | REDUCED BANDWIDTH DATA UPLOADING IN DATA SYSTEMS - Methods and apparatus for uploading data from a sender to a receiver. A data deduplication technique is described that may reduce the bandwidth used in uploading data from the sender to the receiver. In the technique, the receiver, rather than the sender, maintains a fingerprint dictionary for previously uploaded data. When a sender has additional data to be uploaded, the sender extracts fingerprints for units of the data and sends the fingerprints to the receiver. The receiver checks its fingerprint dictionary to determine the data units to be uploaded and notifies the sender of the identified units, which then sends the identified units of data to the receiver. The technique may, for example, be applied in virtualized data store systems to reduce bandwidth usage in uploading data. | 12-17-2015 |
20160044108 | DISTRIBUTED STORAGE SYSTEM WITH WEB SERVICES CLIENT INTERFACE - A distributed, web-services based storage system. A system may include a web services interface configured to receive, according to a web services protocol, a given client request for access to a given data object, the request including a key value corresponding to the object. The system may also include storage nodes configured to store replicas of the objects, where each replica is accessible via a respective unique locator value, and a keymap instance configured to store a respective keymap entry for each object. For the given object, the respective keymap entry includes the key value and each locator value corresponding to replicas of the object. A coordinator may receive the given client request from the web services interface, responsively access the keymap instance to identify locator values corresponding to the key value and, for a particular locator value, retrieve a corresponding replica from a corresponding storage node. | 02-11-2016 |
20160085797 | ARCHIVAL DATA IDENTIFICATION - A request to retrieve a persistently stored data object is received, the request including a data object identifier that encodes at least storage location information and validation information related to the data object. The data object is retrieved using at least the storage location information to form a retrieved data object, and validation is performed using at least the validation information. | 03-24-2016 |