Basic information about Amazon SimpleDB Service:
AWS Free Tier availability:
- 25 SimpleDB Machine Hours
- 1GB of Storage
- data-sets organized into domains (vs. tables in relational DB’s)
- Domains are collections of items that are described by attribute-value pairs
- automatically creates an index for every field in a domain
- no need to pre-define a schema
- scale-out by creating new domains on different instances
- stores multiple geographically distributed copies of each domain to enable high availability and data durability.
- a successful write (using PutAttributes, BatchPutAttributes, DeleteAttributes, BatchDeleteAttributes, CreateDomain or DeleteDomain) means that all copies of the domain will durably persist
- by default, GetAttributes and Select perform an eventually consistent read (details below).
- a consistent read can potentially incur higher latency and lower read throughput therefore it is best to use it only when an application scenario mandates that a read operation absolutely needs to read all writes that received a successful response prior to that read. For all other scenarios the default eventually consistent read will yield the best performance.
- allows specifying consistency settings for each individual read request, so the same application could have disparate parts following different consistency settings.
- currently enables domains to grow up to 10 GB each
- initial allocation of domains is limited to 250
- CreateDomain — Create a domain that contains your dataset.
- DeleteDomain — Delete a domain.
- ListDomains — List all domains.
- DomainMetadata — Retrieve information about creation time for the domain, storage information both as counts of item names and attributes, as well as total size in bytes.
- PutAttributes — Add or update an item and its attributes, or add attribute-value pairs to items that exist already. Items are automatically indexed as they are received.
- BatchPutAttributes — For greater overall throughput of bulk writes, perform up to 25 PutAttribute operations in a single call.
- DeleteAttributes — Delete an item, an attribute, or an attribute value.
- BatchDeleteAttributes — For greater overall throughput of bulk deletes, perform up to 25 DeleteAttributes operations in a single call.
- GetAttributes — Retrieve an item and all or a subset of its attributes and values.
- Select — Query the data set in the familiar, “select target from domain_name where query_expression” syntax. Supported value tests are: =, !=, <, > <=, >=, like, not like, between, is null, is not null, and every (). Example: select * from mydomain where every(keyword) = ‘Book’. Order results using the SORT operator, and count items that meet the condition(s) specified by the predicate(s) in a query using the Count operator.
- Eventually Consistent Reads (Default) — the eventual consistency option maximizes read performance (in terms of low latency and high throughput). However, an eventually consistent read (using Select or GetAttributes) might not reflect the results of a recently completed write (using PutAttributes, BatchPutAttributes, DeleteAttributes, BatchDeleteAttributes). Consistency across all copies of data is usually reached within a second; repeating a read after a short time should return the updated data.
- Consistent Reads — in addition to eventual consistency, SimpleDB also gives flexibility to request a consistent read if your application, or an element of your application, requires it. A consistent read (using Select or GetAttributes with ConsistentRead=true) returns a result that reflects all writes that received a successful response prior to the read.
- Conditional Puts/Deletes — enable to insert, replace, or delete values for one or more attributes of an item if the existing value of an attribute matches the value specified. If the value does not match or is not present, the update is rejected. Conditional Puts/Deletes are useful for preventing lost updates when different sources write concurrently to the same item.
- Conditional puts and deletes are exposed via the PutAttributes and DeleteAttributes APIs by specifying an optional condition with an expected value.
- For example, if the application is reserving seats or selling tickets to an event, you might allow a purchase (i.e., write update) only if the specified seat was still available (the optional condition). These semantics can also be used to implement functionality such as counters, inserting an item only if it does not already exist, and optimistic concurrency control (OCC). An application can implement OCC by maintaining a version number (or a timestamp) attribute as part of an item and by performing a conditional put/delete based on the value of this version number
- Domain size: 10 GB per domain, 1 billion attributes per domain
- Domain name: 3-255 characters (a-z, A-Z, 0-9, ‘_’, ‘-‘, and ‘.’)
- Domains per account: 250
- Attribute name-value pairs per item: 256
- Attribute name length: 1024 bytes
- Attribute value length: 1024 bytes
- Item name length: 1024 bytes
- Attribute name, attribute value, and item name allowed characters: All UTF-8 characters that are valid in XML documents. Control characters and any sequences that are not valid in XML are returned Base64-encoded. For more information, seeWorking with XML-Restricted Characters.
- Attributes per
PutAttributes operation: 256
- Attributes requested per
Select operation: 256
- Items per
BatchDeleteAttributes operation: 25
- Items per
BatchPutAttributes operation: 25
- Maximum items in
Select response: 2500
- Maximum query execution time: 5 seconds
- Maximum number of unique attributes per
Select expression: 20
- Maximum number of comparisons per
Select expression: 20
- Maximum response size for
Basic information about Amazon DynamoDB Service:
AWS Free Tier availability:
- 100MB of Storage,
- 5 Units of Write Capacity,
- 10 Units of Read Capacity
- up to 40 million free operations each month with eventually consistent reads, or 25 million operations each month with strictly consistent reads
- Local secondary indexes
- automatic 3-way replication
- you pay a flat, hourly rate based on the capacity you reserve
- when creating or updating tables, you specify how much capacity you wish to reserve for reads and writes
- Table name: allowed characters are a-z, A-Z, 0-9, ‘_’ (underscore), ‘-‘ (dash), and ‘.’ (dot). Names can be between 3 and 255 characters long.
- Local secondary index name: allowed characters are a-z, A-Z, 0-9, ‘_’ (underscore), ‘-‘ (dash), and ‘.’ (dot). Names can be between 3 and 255 characters long.
- Table size: No practical limit in number of bytes or items
- Tables per account: 256 per region (default). Increase can be requested.
- Hash or hash-and-range primary key: No practical limit.
- Number of hash key values: No practical limit.
- Hash-and-range primary key: No practical limit for non-indexed tables (otherwise total sizes of all table and index items cannot exceed 10 GB)
- Number of range keys per hash value: No practical limit for non-indexed tables (otherwise total sizes of all table and index items cannot exceed 10 GB)
- Provisioned throughput capacity unit sizes: One read capacity unit = one strongly consistent read per second, or two eventually consistent reads per second, for items up 4 KB in size. One write capacity unit = one write per second, for items up to 1 KB in size.
- Provisioned throughput minimum per table: 1 read capacity unit and 1 write capacity unit
- Provisioned throughput limits: can request an increase but default values are:
- US East (Northern Virginia) Region:
- Per table – 40,000 read capacity units or 40,000 write capacity units
- Per account – 80,000 read capacity units or 80,000 write capacity units
- All Other Regions:
- Per table – 10,000 read capacity units or 10,000 write capacity units
- Per account – 20,000 read capacity units or 20,000 write capacity units
- UpdateTable: Limits when increasing provisioned throughput: You can call UpdateTable as often as necessary to increase provisioned throughput. You can increase ReadCapacityUnits or WriteCapacityUnits for a table, subject to these conditions:
- You can call the UpdateTable API to increase ReadCapacityUnits or WriteCapacityUnits (or both), up to twice their current values.
- The new provisioned throughput settings do not take effect until the UpdateTable operation is complete.
- You can call UpdateTable multiple times, until you reach the desired throughput capacity for your table.
- UpdateTable: Limits when decreasing provisioned throughput: You can reduce the provisioned throughput on a table no more than four times in a single UTC calendar day. These reductions can be any of the following operations:
- Decrease ReadCapacityUnits.
- Decrease WriteCapacityUnits.
- Decrease both ReadCapacityUnits and WriteCapacityUnits in a single request. This counts as one of your allowed reductions for the day.
- Maximum concurrent Control Plane API requests (includes cumulative number of tables in the
DELETING state): In general, you can have up to 10 of these requests running concurrently. The only exception is when you are
CREATING a table and you have defined a local secondary index on that table. You can only have one such request running at a time.
- Maximum number of local secondary indexes per table: You can define up to 5 local secondary indexes per table.
- Maximum number of projected attributes per table (local secondary indexes only): You can project a total of up to 20 attributes into all of a table’s local secondary indexes. This only applies to user-specified projected attributes.
CreateTable operation, if you specify a
INCLUDE, the total count of attributes specified in
NonKeyAttributes, summed across all of the local secondary indexes, must not exceed 20. If you project the same attribute name into two different indexes, this counts as two distinct attributes when determining the total.
This limit does not apply for indexes with a
- Attribute name lengths: The following attribute names are length-restricted:
- Primary key attribute names.
- The names of any user-specified projected attributes (applicable only to local secondary indexes). In a
CreateTableoperation, if you specify a
INCLUDE, then the names of the attributes in the
NonKeyAttributes parameter are length-restricted. The
ALL projection types are not affected.
- For any of the attribute names listed above, the name must be between 1 and 255 characters long, inclusive. The name can be any UTF-8 encodable character, but the total size of the UTF-8 string after encoding cannot exceed 255 bytes.
- Item size: Cannot exceed 64 KB which includes both attribute name binary length (UTF-8 length) and attribute value lengths (again binary length). The attribute name counts towards the size limit. For example, consider an item with two attributes: one attribute named “shirt-color” with value “R” and another attribute named “shirt-size” with value “M”. The total size of that item is 23 bytes. These limits apply to items stored in tables, and also to items in local secondary indexes.
- Attribute values: cannot be null or empty
- Attribute name-value pairs per item: The cumulative size of attributes per item must be under 64 KB
- Hash primary key attribute value: 2048 bytes
- Range primary key attribute value: 1024 bytes
- String: All strings must conform to the UTF-8 encoding. Since UTF-8 is a variable width encoding, string sizes are determined using the UTF-8 bytes.
- Number: A number can have up to 38 digits precision and can be between 10^-128 to 10^+126.
- Maximum number of values in an attribute set: No practical limit on the quantity of values, as long as the item containing the values fits within the 64 KB item limit.
- BatchGetItem item maximum per operation: Up to 100 items retrieved, with the request size not exceeding 1 MB.
- BatchWriteItem item maximum per operation: Up to 25 items put or delete operations, with the request size not exceeding 1 MB.
- Query: Result set limited to 1 MB per API call. You can use the
LastEvaluatedKey from the query response to retrieve more results.
- Scan: Scanned data set size maximum is 1 MB per API call. You can use the
LastEvaluatedKey from the scan response to retrieve more results
Continuing on my series of introductory posts on Redis DB, today i’ll address the subject of replication.
- Replication is a method by which other servers receive a continuously updated copy of the data as it’s being written, so that the replicas can service read queries.
Basic info (redis.io):
- Redis uses asynchronous replication. Starting with Redis 2.8 there is however a periodic (one time every second) acknowledge of the replication stream processed by slaves.
- A master can have multiple slaves.
- Slaves are able to accept other slaves connections. Aside from connecting a number of slaves to the same master, slaves can also be connected to other slaves in a graph-like structure.
- Redis replication is non-blocking on the master side, this means that the master will continue to serve queries when one or more slaves perform the first synchronization.
- Replication is non blocking on the slave side: while the slave is performing the first synchronization it can reply to queries using the old version of the data set, assuming you configured Redis to do so in redis.conf. Otherwise you can configure Redis slaves to send clients an error if the link with the master is down. However there is a moment where the old dataset must be deleted and the new one must be loaded by the slave where it will block incoming connections.
- Replications can be used both for scalability, in order to have multiple slaves for read-only queries (for example, heavy SORT operations can be offloaded to slaves), or simply for data redundancy.
- It is possible to use replication to avoid the saving process on the master side: just configure your master redis.conf to avoid saving (just comment all the “save” directives), then connect a slave configured to save from time to time.
How Redis replication works (redis.io):
- If you set up a slave, upon connection it sends a SYNC command. And it doesn’t matter if it’s the first time it has connected or if it’s a re-connection.
- The master then starts background saving, and collects all new commands received that will modify the dataset. When the background saving is complete, the master transfers the database file to the slave, which saves it on disk, and then loads it into memory. The master will then send to the slave all accumulated commands, and all new commands received from clients that will modify the dataset. This is done as a stream of commands and is in the same format of the Redis protocol itself.
- You can try it yourself via telnet. Connect to the Redis port while the server is doing some work and issue the SYNC command. You’ll see a bulk transfer and then every command received by the master will be re-issued in the telnet session.
- Slaves are able to automatically reconnect when the master <-> slave link goes down for some reason. If the master receives multiple concurrent slave synchronization requests, it performs a single background save in order to serve all of them.
- When a master and a slave reconnects after the link went down, a full re-sync is always performed. However starting with Redis 2.8, a partial re-synchronization is also possible.
In order to configure the replication, all you have to do is to add the line below (or issue the same as a CLI command from slave) to the redis.conf file of the slave.
- SLAVEOF <master_ip> <master_port> (ex. SLAVEOF 127.0.0.1 6379)
to tune the replication process you can play with following options in the redis.conf file:
- requirepass <password> – Require clients to issue AUTH <PASSWORD> before processing any other commands. This might be useful in environments in which you do not trust (eg. don’t run your own servers) others with access to the host running redis-server
- masterauth <master-password> – If the master is password protected (using the “requirepass” configuration directive above) it is possible to tell the slave to authenticate before starting the replication synchronization process, otherwise the master will refuse the slave request
- slave-serve-stale-data <yes|no> – When a slave loses its connection with the master, or when the replication is still in progress, the slave can act in two different ways:
- still reply to client requests, possibly with out-of-date data (the default behavior if the switch is set to “yes”)
- or reply with an error “SYNC with master in progress” to all the kind of commands, except for to INFO and SLAVEOF (otherwise)
- slave-read-only <yes|no> – You can configure a slave instance to accept writes or not. Writing against a slave instance may be useful to store some ephemeral data (because data written on a slave will be easily deleted after re-sync with the master anyway), but may also cause problems if clients are writing to it because of a misconfiguration
- repl-ping-slave-period <seconds> – Slaves send PINGs to server in a predefined interval. It’s possible to change this interval with the repl_ping_slave_period option from CLI. The default value is 10 seconds.
- repl-timeout <seconds> – This option sets a timeout for both Bulk transfer I/O timeout and master data or ping response timeout. The default value is 60 seconds. It is important to make sure that this value is greater than the value specified for repl-ping-slave-period otherwise a timeout will be detected every time there is low traffic between the master and the slave.
- repl-disable-tcp-nodelay <yes|no> – Controls whether to disable TCP_NODELAY on the slave socket after SYNC. If you select “yes” Redis will use a smaller number of TCP packets and less bandwidth to send data to slaves. But this can add a delay for the data to appear on the slave side, up to 40 milliseconds with Linux kernels using a default configuration. If you select “no” the delay for data to appear on the slave side will be reduced but more bandwidth will be used for replication. Default value of “no” is an optimization for low latency, but in very high traffic conditions or when the master and slaves are many hops away, turning this to “yes” may be a good idea.
- slave-priority <integer> – The slave priority is an integer number published by Redis in the INFO output. It is used by Redis Sentinel in order to select a slave to promote into a master if the master is no longer working correctly. A slave with a low priority number is considered better for promotion, so for instance if there are three slaves with priority 10, 100, 25 Sentinel will pick the one with priority 10, that is the lowest. However a special priority of 0 marks the slave as not able to perform the role of master, so a slave with priority of 0 will never be selected by Redis Sentinel for promotion.
Allowing writes only with N attached replicas (redis.io):
- Starting with Redis 2.8 it is possible to configure a Redis master in order to accept write queries only if at least N slaves are currently connected to the master, in order to improve data safety.
- However because Redis uses asynchronous replication it is not possible to ensure the write actually received a given write, so there is always a window for data loss.
- This is how the feature works:
- Redis slaves ping the master every second, acknowledging the amount of replication stream processed.
- Redis masters will remember the last time it received a ping from every slave.
- The user can configure a minimum number of slaves that have a lag not greater than a maximum number of seconds.
- If there are at least N slaves, with a lag less than M seconds, then the write will be accepted.
- There are two configuration parameters for this feature:
<number of slaves>
<number of seconds>
Have a nice weekend!