Amazon AWS – DynamoDB – advanced NoSQL

Basic information about Amazon DynamoDB Service:

 

AWS Free Tier availability:

  • 100MB of Storage,
  • 5 Units of Write Capacity,
  • 10 Units of Read Capacity
  • up to 40 million free operations each month with eventually consistent reads, or 25 million operations each month with strictly consistent reads

 

Developer Resources:

 

Features:

  • Local secondary indexes
  • SSD-storage
  • automatic 3-way replication
  • you pay a flat, hourly rate based on the capacity you reserve
  • when creating or updating tables, you specify how much capacity you wish to reserve for reads and writes

 

Limits:

  • Table name: allowed characters are a-z, A-Z, 0-9, ‘_’ (underscore), ‘-‘ (dash), and ‘.’ (dot). Names can be between 3 and 255 characters long.
  • Local secondary index name: allowed characters are a-z, A-Z, 0-9, ‘_’ (underscore), ‘-‘ (dash), and ‘.’ (dot). Names can be between 3 and 255 characters long.
  • Table size: No practical limit in number of bytes or items
  • Tables per account: 256 per region (default). Increase can be requested.
  • Hash or hash-and-range primary key: No practical limit.
  • Number of hash key values: No practical limit.
  • Hash-and-range primary key: No practical limit for non-indexed tables (otherwise total sizes of all table and index items cannot exceed 10 GB)
  • Number of range keys per hash value: No practical limit for non-indexed tables (otherwise total sizes of all table and index items cannot exceed 10 GB)
  • Provisioned throughput capacity unit sizes: One read capacity unit = one strongly consistent read per second, or two eventually consistent reads per second, for items up 4 KB in size. One write capacity unit = one write per second, for items up to 1 KB in size.
  • Provisioned throughput minimum per table: 1 read capacity unit and 1 write capacity unit
  • Provisioned throughput limits: can request an increase but default values are:
    • US East (Northern Virginia) Region:
      • Per table – 40,000 read capacity units or 40,000 write capacity units
      • Per account – 80,000 read capacity units or 80,000 write capacity units
    • All Other Regions:
      • Per table – 10,000 read capacity units or 10,000 write capacity units
      • Per account – 20,000 read capacity units or 20,000 write capacity units
  • UpdateTable: Limits when increasing provisioned throughput: You can call UpdateTable as often as necessary to increase provisioned throughput. You can increase ReadCapacityUnits or WriteCapacityUnits for a table, subject to these conditions:
    • You can call the UpdateTable API to increase ReadCapacityUnits or WriteCapacityUnits (or both), up to twice their current values.
    • The new provisioned throughput settings do not take effect until the UpdateTable operation is complete.
    • You can call UpdateTable multiple times, until you reach the desired throughput capacity for your table.
  • UpdateTable: Limits when decreasing provisioned throughput: You can reduce the provisioned throughput on a table no more than four times in a single UTC calendar day. These reductions can be any of the following operations:
    • Decrease ReadCapacityUnits.
    • Decrease WriteCapacityUnits.
    • Decrease both ReadCapacityUnits and WriteCapacityUnits in a single request. This counts as one of your allowed reductions for the day.
  • Maximum concurrent Control Plane API requests (includes cumulative number of tables in the CREATINGUPDATING or DELETING state): In general, you can have up to 10 of these requests running concurrently. The only exception is when you are CREATING a table and you have defined a local secondary index on that table. You can only have one such request running at a time.
  • Maximum number of local secondary indexes per table: You can define up to 5 local secondary indexes per table.
  • Maximum number of projected attributes per table (local secondary indexes only): You can project a total of up to 20 attributes into all of a table’s local secondary indexes. This only applies to user-specified projected attributes.

    In a CreateTable operation, if you specify a ProjectionType of INCLUDE, the total count of attributes specified in NonKeyAttributes, summed across all of the local secondary indexes, must not exceed 20. If you project the same attribute name into two different indexes, this counts as two distinct attributes when determining the total.

    This limit does not apply for indexes with a ProjectionType of KEYS_ONLY or ALL.

  • Attribute name lengths: The following attribute names are length-restricted:
    • Primary key attribute names.
    • The names of any user-specified projected attributes (applicable only to local secondary indexes). In a CreateTableoperation, if you specify a ProjectionType of INCLUDE, then the names of the attributes in the NonKeyAttributes parameter are length-restricted. The KEYS_ONLY and ALL projection types are not affected.
    • For any of the attribute names listed above, the name must be between 1 and 255 characters long, inclusive. The name can be any UTF-8 encodable character, but the total size of the UTF-8 string after encoding cannot exceed 255 bytes.
  • Item size: Cannot exceed 64 KB which includes both attribute name binary length (UTF-8 length) and attribute value lengths (again binary length). The attribute name counts towards the size limit. For example, consider an item with two attributes: one attribute named “shirt-color” with value “R” and another attribute named “shirt-size” with value “M”. The total size of that item is 23 bytes. These limits apply to items stored in tables, and also to items in local secondary indexes.
  • Attribute values: cannot be null or empty
  • Attribute name-value pairs per item: The cumulative size of attributes per item must be under 64 KB
  • Hash primary key attribute value: 2048 bytes
  • Range primary key attribute value: 1024 bytes
  • String: All strings must conform to the UTF-8 encoding. Since UTF-8 is a variable width encoding, string sizes are determined using the UTF-8 bytes.
  • Number: A number can have up to 38 digits precision and can be between 10^-128 to 10^+126.
  • Maximum number of values in an attribute set: No practical limit on the quantity of values, as long as the item containing the values fits within the 64 KB item limit.
  • BatchGetItem item maximum per operation: Up to 100 items retrieved, with the request size not exceeding 1 MB.
  • BatchWriteItem item maximum per operation: Up to 25 items put or delete operations, with the request size not exceeding 1 MB.
  • Query: Result set limited to 1 MB per API call. You can use the LastEvaluatedKey from the query response to retrieve more results.
  • Scan: Scanned data set size maximum is 1 MB per API call. You can use the LastEvaluatedKey from the scan response to retrieve more results

 

 

 

Resources:

Amazon AWS – Overview

Here’s some basic info about AWS (Amazon Web Services) in case you’re considering deploying your product in the Amazon cloud:

 

Core AWS components:

 

 

Resources:

Amazon AWS – EC2 – Elastic Compute Cloud

Basic information about Amazon EC2 Service:

 

AWS Free Tier availability:

  • 750 hours of Linux and Windows Micro Instances each month for one year.

 

Developer Resources:

 

Instance categories:

  • On-Demand Instances – pay for compute capacity by the hour with no long-term commitments.
    • Removes the need to buy “safety net” capacity to handle periodic traffic spikes.
  • Reserved Instances – make a low, one-time payment for each instance you want to reserve and in turn receive a significant discount on the hourly charge for that instance.
    • There are three Reserved Instance types (Light, Medium, and Heavy Utilization Reserved Instances) that enable you to balance the amount you pay upfront with your effective hourly price.
    • If your needs change, you can request to move your Reserved Instance to another Availability Zone in the same region, and Amazon will process your request based on available capacity.
    • You can also sell your Reserved Instances to other customers for cash via the Reserved Instance Marketplace (currently, only customers with US bank accounts may sell Reserved Instances).
  • Spot Instances –  bid on unused Amazon EC2 capacity and run those instances for as long as their bid exceeds the current Spot Price.
    • The Spot Price changes periodically based on supply and demand, and customers whose bids meet or exceed it gain access to the available Spot Instances.
    • If you have flexibility in when your applications can run, Spot Instances can significantly lower your Amazon EC2 costs.

 

Features:

  • Elastic Block Store (EBS) – persistent storage for EC2 instances.
    • EBS volumes are network-attached, and persist independently from the life of an instance.
    • they can be leveraged as an Amazon EC2 instance’s boot partition or attached to a running Amazon EC2 instance as a standard block device.
    • when used as a boot partition, EC2 instances can be stopped and subsequently restarted, enabling you to only pay for the storage resources used while maintaining your instance’s state.
    • EBS volumes are automatically replicated on the backend (in a single Availability Zone).
    • they can be used to create point-in-time consistent snapshots of your volumes that are then stored in Amazon S3, and automatically replicated across multiple Availability Zones.
    • such snapshots can be used as the starting point for new Amazon EBS volumes
    • EBS provides two volume types: Standard volumes and Provisioned IOPS volumes.
      • standard volumes offer cost effective storage that is ideal for applications with moderate or bursty I/O requirements.
      • provisioned IOPS volumes are designed to deliver predictable, high performance for I/O intensive applications such as databases.
  • EBS-Optimized Instances – enable EC2 instances to fully utilize the IOPS provisioned on an EBS volume.
    • For additional hourly fee you can launch selected EC2 instances types as “EBS-Optimized” instances.
    • EBS-Optimized instances deliver dedicated throughput between EC2 and EBS, with options between 500 Mbps and 1000 Mbps depending on the instance type used.
    • When attached to EBS-Optimized instances, Provisioned IOPS volumes are designed to deliver within 10% of their provisioned performance 99.9% of the time.
  • Multiple Locations – place instances in multiple locations.
    • Amazon EC2 locations are composed of Regions and Availability Zones.
    • Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region. By launching instances in separate Availability Zones, you can protect your applications from failure of a single location.
    • Regions consist of one or more Availability Zones, are geographically dispersed, and will be in separate geographic areas or countries. The Amazon EC2 Service Level Agreement commitment is 99.95% availability for each Amazon EC2 Region. EC2 is currently available in nine regions:
      • US East (Northern Virginia),
      • US West (Oregon),
      • US West (Northern California),
      • EU (Ireland),
      • Asia Pacific (Singapore),
      • Asia Pacific (Tokyo),
      • Asia Pacific (Sydney),
      • South America (Sao Paulo),
      • AWS GovCloud
  • Elastic IP Addresses – static IP addresses designed for dynamic cloud computing.
    • An Elastic IP address is associated with your account not a particular instance, and you control that address until you choose to explicitly release it.
    • Unlike traditional static IP addresses, however, Elastic IP addresses allow you to mask instance or Availability Zone failures by programmatically remapping your public IP addresses to any instance in your account.
    • Rather than waiting on a data technician to reconfigure or replace your host, or waiting for DNS to propagate to all of your customers, EC2 enables you to engineer around problems with your instance or software by quickly remapping your Elastic IP address to a replacement instance.
    • In addition, you can optionally configure the reverse DNS record of any of your Elastic IP addresses
  • Virtual Private Cloud (VPC) – lets you provision a logically isolated section of AWS Cloud where you can launch AWS resources in a virtual network that you define.
    • You have complete control over your virtual networking environment, including selection of your own IP address range, creation of subnets, and configuration of route tables and network gateways.
    • You can also create a Hardware Virtual Private Network (VPN) connection between your corporate datacenter and your VPC and leverage the AWS cloud as an extension of your corporate datacenter.
  • CloudWatch – a web service that provides monitoring for AWS cloud resources and applications.
    • It provides you with visibility into resource utilization, operational performance, and overall demand patterns—including metrics such as CPU utilization, disk reads and writes, and network traffic.
    • You can get statistics, view graphs, and set alarms for your metric data.
  • Auto Scaling – allows you to automatically scale your EC2 capacity up or down according to conditions you define.
    • With Auto Scaling, you can ensure that the number of Amazon EC2 instances you’re using scales up seamlessly during demand spikes to maintain performance, and scales down automatically during demand lulls to minimize costs.
    • Auto Scaling is particularly well suited for applications that experience hourly, daily, or weekly variability in usage.
    • Auto Scaling is enabled by Amazon CloudWatch and available at no additional charge beyond Amazon CloudWatch fees.
  • Elastic Load Balancing – automatically distributes incoming application traffic across multiple EC2 instances.
    • It enables you to achieve greater fault tolerance in your applications
    • detects unhealthy instances within a pool and automatically reroutes traffic to healthy instances until the unhealthy instances have been restored.
    • You can enable Elastic Load Balancing within a single Availability Zone or across multiple zones
    • CloudWatch can be used to capture a specific Elastic Load Balancer’s operational metrics, such as request count and request latency, at no additional cost beyond Elastic Load Balancing fees.
  • High Performance Computing (HPC) Clusters – complex computational workloads such as tightly coupled parallel processes, or applications sensitive to network performance, can achieve the same high compute and network performance provided by custom-built infrastructure while benefiting from the elasticity, flexibility and cost advantages of Amazon EC2.
    • Cluster Compute, Cluster GPU, and High Memory Cluster instances have been specifically engineered to provide high-performance network capability and can be programmatically launched into clusters – allowing applications to get the low-latency network performance required for tightly coupled, node-to-node communication.
    • Cluster instances also provide significantly increased throughput making them well suited for customer applications that need to perform network-intensive operations.
  • High I/O Instances – for customers requiring very high, low latency, random I/O access to their data:
    • High I/O instances are an Amazon EC2 instance type that can provide customers with random I/O rates over 100,000 IOPS.
    • High I/O instances are backed by Solid State Disk (SSD) technology and are ideally suited for customers running very high performance NoSQL and relational databases.
  • High Storage Instances – for customers requiring very high storage density per instance, and high sequential I/O for data-intensive applications like data warehousing and Hadoop:
    • High Storage instances are an Amazon EC2 instance type that can provide customers with sequential I/O throughout of 2.4 GB/s and provide customers with 48 TB of instance storage across 24 hard disk drives.
  • VM Import/Export – enables you to easily import virtual machine images from your existing environment to EC2 instances and export them back at any time.
    • By importing virtual machines as ready to use EC2 instances, you can leverage your existing investments in virtual machines that meet your IT security, configuration management, and compliance requirements.
    • You can export your previously imported EC2 instances back to your on-premise environment at any time.
    • available at no additional charge beyond standard usage charges for Amazon EC2 and Amazon S3.
  • AWS Marketplace – an online store that helps you find, buy and quickly deploy software that runs on AWS.
    • You can use AWS Marketplace’s 1-Click deployment to quickly launch pre-configured software and be charged for what you use, by the hour or month.
    • AWS handles billing and payments, and software charges appear on your AWS bill.

 

Operating Systems:

 

Databases:

 

Application Stacks:

 

 

 

Resources:

Amazon AWS – EC2 – Instance Types

Basic information about Amazon EC2 instances types:

 

  • Standard
    • First generation (M1) – balanced set of resources at a low cost
      • M1 Small Instance (Default):
        • 1.7 GiB of memory,
        • 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit),
        • 160 GB of local instance storage,
        • 32-bit or 64-bit platform
      • M1 Medium Instance:
        • 3.75 GiB of memory,
        • 2 EC2 Compute Units (1 virtual core with 2 EC2 Compute Units each),
        • 410 GB of local instance storage,
        • 32-bit or 64-bit platform
      • M1 Large Instance:
        • 7.5 GiB of memory,
        • 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each),
        • 850 GB of local instance storage,
        • 64-bit platform
      • M1 Extra Large Instance:
        • 15 GiB of memory,
        • 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each),
        • 1690 GB of local instance storage,
        • 64-bit platform
    • Second generation (M3) – balanced set of resources and a higher level of processing performance compared to M1.
      • M3 Extra Large Instance:
        • 15 GiB of memory,
        • 13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each),
        • EBS storage only,
        • 64-bit platform
      • M3 Double Extra Large Instance:
        • 30 GiB of memory,
        • 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each),
        • EBS storage only,
        • 64-bit platform
  • Micro
    • provide a small amount of consistent CPU resources and allow you to increase CPU capacity in short bursts when additional cycles are available.
      • t1.micro
        • 613 MiB of memory,
        • up to 2 ECUs (for short periodic bursts),
        • EBS storage only,
        • 32-bit or 64-bit platform
  • High-Memory
    • offer large memory sizes for high throughput applications, including database and memory caching applications.
      • High-Memory Extra Large Instance:
        • 17.1 GiB memory,
        • 6.5 ECU (2 virtual cores with 3.25 EC2 Compute Units each),
        • 420 GB of local instance storage,
        • 64-bit platform
      • High-Memory Double Extra Large Instance:
        • 34.2 GiB of memory,
        • 13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each),
        • 850 GB of local instance storage,
        • 64-bit platform
      • High-Memory Quadruple Extra Large Instance:
        • 68.4 GiB of memory,
        • 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each),
        • 1690 GB of local instance storage,
        • 64-bit platform
  • High-CPU
    • have proportionally more CPU resources than memory (RAM) and are well suited for compute-intensive applications
      • High-CPU Medium Instance:
        • 1.7 GiB of memory,
        • 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each),
        • 350 GB of local instance storage,
        • 32-bit or 64-bit platform
      • High-CPU Extra Large Instance:
        • 7 GiB of memory,
        • 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each),
        • 1690 GB of local instance storage,
        • 64-bit platform
  • Cluster Compute
    • provide proportionally high CPU resources with increased network performance and are well suited for High Performance Compute (HPC) applications and other demanding network-bound applications
      • Cluster Compute Eight Extra Large:
        • 60.5 GiB memory,
        • 88 EC2 Compute Units,
        • 3370 GB of local instance storage,
        • 64-bit platform,
        • 10 Gigabit Ethernet
  • High Memory Cluster
    • provide proportionally high CPU and memory resources with increased network performance, and are well suited for memory-intensive applications including in-memory analytics, graph analysis, and scientific computing
      • High Memory Cluster Eight Extra Large
        • 244 GiB memory,
        • 88 EC2 Compute Units,
        • 240 GB of local instance storage,
        • 64-bit platform,
        • 10 Gigabit Ethernet
  • Cluster GPU
    • provide general-purpose graphics processing units (GPUs) with proportionally high CPU and increased network performance for applications benefitting from highly parallelized processing, including HPC, rendering and media processing applications.
    • While Cluster Compute Instances provide the ability to create clusters of instances connected by a low latency, high throughput network, Cluster GPU Instances provide an additional option for applications that can benefit from the efficiency gains of the parallel computing power of GPUs over what can be achieved with traditional processors
      • Cluster GPU Quadruple Extra Large
        • 22 GiB memory,
        • 33.5 EC2 Compute Units,
        • 2 x NVIDIA Tesla “Fermi” M2050 GPUs,
        • 1690 GB of local instance storage,
        • 64-bit platform,
        • 10 Gigabit Ethernet
  • High I/O
    • provide very high disk I/O performance and are ideally suited for many high performance database workloads.
    • provide SSD-based local instance storage, and also provide high levels of CPU, memory and network performance
      • High I/O Quadruple Extra Large
        • 60.5 GiB memory,
        • 35 EC2 Compute Units,
        • 2 * 1024 GB of SSD-based local instance storage,
        • 64-bit platform,
        • 10 Gigabit Ethernet
  • High Storage
    • provide proportionally higher storage density per instance, and are ideally suited for applications that benefit from high sequential I/O performance across very large data sets
      • High Storage Eight Extra Large
        • 117 GiB memory,
        • 35 EC2 Compute Units,
        • 24 * 2 TB of hard disk drive local instance storage,
        • 64-bit platform,
        • 10 Gigabit Ethernet

 

EC2 Compute Unit (ECU) – One EC2 Compute Unit (ECU) provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

Amazon AWS – S3 – Simple Storage Service

Basic information about Amazon S3 Service:

 

AWS Free Tier availability:

  • 5GB storage,
  • 20,000 Get Requests,
  • 2,000 Put Requests

 

Developer Resources:

 

Functionality:

  • write, read, and delete objects containing from 1 byte to 5 terabytes of data each. The number of objects you can store is unlimited.
  • each object is stored in a bucket and retrieved via a unique, developer-assigned key.
  • a bucket can be stored in one of several Regions. You can choose a Region to optimize for latency, minimize costs, or address regulatory requirements. Amazon S3 is currently available in following regions:
    • US Standard (automatically routes requests to facilities in Northern Virginia or the Pacific Northwest using network maps)
    • US West (Oregon),
    • US West (Northern California),
    • EU (Ireland),
    • Asia Pacific (Singapore),
    • Asia Pacific (Tokyo),
    • Asia Pacific (Sydney),
    • South America (Sao Paulo),
    • GovCloud (US)
  • objects stored in a Region never leave the Region unless you transfer them out. For example, objects stored in the EU (Ireland) Region never leave the EU.
  • authentication mechanisms are provided to ensure that data is kept secure from unauthorized access. Objects can be made private or public, and rights can be granted to specific users.
  • options for secure data upload/download and encryption of data at rest are provided for additional data protection.
  • uses standards-based REST and SOAP interfaces designed to work with any Internet-development toolkit.
  • built to be flexible so that protocol or functional layers can easily be added. The default download protocol is HTTP.
  • a BitTorrent™ protocol interface is provided to lower costs for high-scale distribution.
  • provides functionality to simplify manageability of data through its lifetime. Includes options for segregating data by buckets, monitoring and controlling spend, and automatically archiving data to even lower cost storage options.
  • reliability backed with the Amazon S3 Service Level Agreement.

 

3 different S3 storage options are available:

  • standard storage
  • RRS
  • Glacier

 

Standard S3 storage:

  • designed for mission-critical and primary data storage
  • redundant storage of data in multiple facilities and on multiple devices within each facility.
  • synchronously stores data across multiple facilities before returning SUCCESS
  • calculates checksums on all network traffic to detect corruption of data packets when storing or retrieving data
  • performs regular, systematic data integrity checks and is built to be automatically self-healing.
  • further protection via Versioning to preserve, retrieve, and restore every version of every object stored in Amazon S3 bucket
  • by default, requests will retrieve the most recently written version. Older versions of an object can be retrieved by specifying a version in the request. Storage rates apply for every version stored.
  • backed with the SLA
  • designed for 99.999999999% durability and 99.99% availability of objects over a given year.
  • designed to sustain the concurrent loss of data in two facilities.

 

Reduced Redundancy Storage (RRS):

  • a storage option within S3 to reduce costs by storing non-critical, reproducible data at lower levels of redundancy than Amazon S3’s standard storage.
  • cost-effective, highly available solution for distributing or sharing content that is durably stored elsewhere, or for storing thumbnails, transcoded media, or other processed data that can be easily reproduced
  • stores objects on multiple devices across multiple facilities, providing 400 times the durability of a typical disk drive
  • does not replicate objects as many times as standard Amazon S3 storage
  • backed with the SLA
  • designed to provide 99.99% durability and 99.99% availability of objects over a given year (durability level corresponds to an average annual expected loss of 0.01% of objects).
  • designed to sustain the loss of data in a single facility.

 

Amazon Glacier:

  • extremely low-cost storage service as a storage option for data archival.
  • stores data for as little as $0.01 per gigabyte per month, and is optimized for data that is infrequently accessed and for which retrieval times of several hours are suitable. Examples include digital media archives, financial and healthcare records, raw genomic sequence data, long-term database backups, and data that must be retained for regulatory compliance.
  • like other S3 storage options (Standard or RRS), objects stored in Amazon Glacier using Amazon S3’s APIs or Management Console have an associated user-defined name.
  • you can get a real-time list of all of your Amazon S3 object names, including those stored using the Amazon Glacier option, only when using the Amazon S3 LIST API.
  • objects stored directly in Amazon Glacier using Amazon Glacier’s APIs cannot be listed in real-time, and have a system-generated identifier rather than a user-defined name. S3 maintains the mapping between your user-defined object name and the Amazon Glacier system-defined identifier
  • to restore Amazon S3 data that was stored in Amazon Glacier via the Amazon S3 APIs or Management Console, you first have to initiate a restore job.
  • Restore jobs typically complete in 3 to 5 hours. Once the job is complete, you can access your data through an Amazon S3 GET request.
  • backed with the SLA
  • designed for 99.999999999% durability and 99.99% availability of objects over a given year.
  • designed to sustain the concurrent loss of data in two facilities.

 

Common Use Cases:

  • Content Storage and Distribution
  • Storage for Data Analysis
  • Backup, Archiving and Disaster Recovery
  • Static Website Hosting

 

 

 

Resources:

Amazon AWS – SQS – Simple Queue Service

Basic information about Amazon SQS Service:

 

AWS Free Tier availability:

  • 1 million requests

 

Developer Resources:

 

Functionality:

  • no limits on number of queues and number of messages.
  • queue can be created in any region.
  • message payload can contain up to 256KB of text in any format. Each 64KB ‘chunk’ of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests.
  • messages can be sent, received or deleted in batches of up to 10 messages or 256KB. Batches cost the same amount as single messages, meaning SQS can be even more cost effective for customers that use batching.
  • long polling reduces extraneous polling to help you minimize cost while receiving new messages as quickly as possible. When your queue is empty, long-poll requests wait up to 20 seconds for the next message to arrive. Long poll requests cost the same amount as regular requests.
  • messages can be retained in queues for up to 14 days.
  • messages can be sent and read simultaneously.
  • when a message is received, it becomes “locked” while being processed. This keeps other computers from processing the message simultaneously. If the message processing fails, the lock will expire and the message will be available again. In the case where the application needs more time for processing, the “lock” timeout can be changed dynamically via the ChangeMessageVisibility operation.
  • developers can securely share Amazon SQS queues with others. Queues can be shared with other AWS accounts and Anonymously. Queue sharing can also be restricted by IP address and time-of-day.
  • when combined with Amazon SNS, developers can ‘fanout’ identical messages to multiple SQS queues in parallel. When developers want to process the messages in multiple passes, fanout helps complete this more quickly, and with fewer delays due to bottlenecks at any one stage. Fanout also makes it easier to record duplicate copies of your messages, for example in different databases.

 

Common design patterns with SQS and other AWS components:

  • Work Queues: Decoupling components of a distributed application that may not all process the same amount of work simultaneously.
  • Buffer and Batch Operations: Adding scalability and reliability to the architecture, and smooth out temporary volume spikes without losing messages or increasing latency.
  • Request Offloading: Moving slow operations off of interactive request paths by enqueing the request.
  • Fanout: Combined with SNS to send identical copies of a message to multiple queues in parallel for simultaneous processing.

 

Service interface:

  • CreateQueue: Create queues for use with your AWS account.
  • ListQueues: List your existing queues.
  • DeleteQueue: Delete one of your queues.
  • SendMessage: Add messages to a specified queue.
  • SendMessageBatch: Add multiple messages to a specified queue.
  • ReceiveMessage: Return one or more messages from a specified queue.
  • ChangeMessageVisibility: Change the visibility timeout of previously received message.
  • ChangeMessageVisibilityBatch: Change the visibility timeout of multiple previously received messages.
  • DeleteMessage: Remove a previously received message from a specified queue.
  • DeleteMessageBatch: Remove multiple previously received messages from a specified queue.
  • SetQueueAttributes: Control queue settings like the amount of time that messages are locked after being read so they cannot be read again.
  • GetQueueAttributes: Get information about a queue like the number of messages in it.
  • GetQueueUrl: Get the queue URL.
  • AddPermission: Add queue sharing for another AWS account for a specified queue.
  • RemovePermission: Remove an AWS account from queue sharing for a specified queue.

 

Message Lifecycle:

  • A system that needs to send a message will find an Amazon SQS queue, and use SendMessage to add a new message to it.
  • A different system that processes messages needs more messages to process, so it calls ReceiveMessage, and this message is returned.
  • Once a message has been returned by ReceiveMessage, it will not be returned by any other ReceiveMessage until the visibility timeout has passed. This keeps multiple computers from processing the same message at once.
  • If the system that processes messages successfully finishes working with this message, it calls DeleteMessage, which removes the message from the queue so no one else will ever process it. If this system fails to process the message, then it will be read by another ReceiveMessage call as soon as the visibility timeout passes.

 

 

 

Resources:

Displaying GIT build number (hash) in your REST API

The product i’m working on currently (a PaaS cloud offering) had a requirement to provide an API resource (GET call) throughout which a user could obtain basic details about the actual version of the API exposed (the api version, build time, corresponding git repo build number (hash id) and the jvm version used to compile the API). Except for the git repo hash part, everything else seemed to be quite easy to obtain. Below you’ll find the solution (step-by-step guide) i came up with.

 

End result (goal):

> curl http://api.my-system.company.com/1.0/
{
  "Implementation-Build" : "2951e7e",
  "Implementation-Build-Time" : "2013/09/17 12:40:02 AM,UTC",
  "Implementation-Jdk" : "1.7.0_15",
  "Implementation-Version" : "1.0-SNAPSHOT",
  "Implementation-Vendor" : "My Company, Inc.",
  "Implementation-Title" : "My System"
}

 

Technologies used:

 

Steps required: 

1. First let’s add the <scm> configuration tag to your master pom.xml file. The connection string represents the repository for which the buildnumber-maven-plugin will obtain the git hash id.

<scm>
    <!-- Replace the connection below with your project connection -->
    <connection>scm:git:git://github.com/mariuszprzydatek/hyde-park.git</connection>
</scm>

 

2. configure the maven-war-plugin to generate project’s MANIFEST.MF file, where the git hash id will be stored. Also, the Spring MVC controller will read this file in order to return its content as a result of GET call.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-war-plugin</artifactId>
    <version>2.3</version>
    <configuration>
        <archive>
            <addMavenDescriptor>false</addMavenDescriptor>
            <manifest>
                <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
            </manifest>
        </archive>
        <warName>1.0</warName>
    </configuration>
</plugin>

 

3. In the <properties> section of the pom we can define the format for the date timestamp that will be returned as the value of “Implementation-Build-Time” attribute.

<properties>
    <maven.build.timestamp.format>yyyy/MM/dd hh:mm:ss a,z</maven.build.timestamp.format>
</properties>

 

4. Next, let’s add the remaining pom sections that we’ll be storing in the MANIFEST.MF file for further read:

    <version>1.0-SNAPSHOT</version>
    <organization>
        <name>My Company, Inc.</name>
    </organization>
    <name>My System</name>

 

5. within the <archive> key of the maven-war-plugin <configuration> section, we need to add additional manifest entries including the one (<Implementation-Build>) that will be generated by the buildnumber-maven-plugin:

<archive>
    ...
    <manifestEntries>
        <Implementation-Build>${buildNumber}</Implementation-Build>
        <Implementation-Build-Time>${maven.build.timestamp}</Implementation-Build-Time>
    </manifestEntries>
</archive>

 

6. Add the buildnumber-maven-plugin itself which will do the hard work:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>buildnumber-maven-plugin</artifactId>
    <version>1.1</version>
    <executions>
        <execution>
            <phase>validate</phase>
            <goals>
                <goal>create</goal>
            </goals>
        </execution>
    </executions>
</plugin>

 

7. Finally, add the <configuration> section to the buildnumber-maven-plugin together with the <shortRevisionLength> key that is responsible for the length of git hash id we want to export:

<configuration>
    <shortRevisionLength>7</shortRevisionLength>
</configuration>

 

 

Now, let’s create the Spring MVC controller that will be handling the MANIFEST.FM file read and returning its content to the presentation layer.

@Controller
@RequestMapping
public class ApiController {

    /**
     * Handling GET request to retrieve details from MANIFEST.MF file
     * @return implementation details
     */
    @RequestMapping(method = RequestMethod.GET)
    public @ResponseBody Map<String, String> getBuildNumber(HttpServletRequest request) throws IOException {

        ServletContext context = request.getSession().getServletContext();
        InputStream manifestStream = context.getResourceAsStream("/META-INF/MANIFEST.MF");
        Manifest manifest = new Manifest(manifestStream);

        Map<String, String> response = new HashMap<>();
        response.put("Implementation-Vendor", manifest.getMainAttributes().getValue("Implementation-Vendor"));
        response.put("Implementation-Title", manifest.getMainAttributes().getValue("Implementation-Title"));
        response.put("Implementation-Version", manifest.getMainAttributes().getValue("Implementation-Version"));
        response.put("Implementation-Jdk", manifest.getMainAttributes().getValue("Build-Jdk"));
        response.put("Implementation-Build", manifest.getMainAttributes().getValue("Implementation-Build"));
        response.put("Implementation-Build-Time", manifest.getMainAttributes().getValue("Implementation-Build-Time"));

        return response;

    }

}

 

 

Hope you enjoyed this post.

Take care!