Inter AZ traffic costs
We have seen in many customers one of their main cloud bill (AWS) contributors is cross-AZ data transfer. Many companies suffer from this hidden cost, both in the aspect of not estimating it correctly or struggling with its analysis and optimizations.
This is so common due to:
- Highly Available Distributed Systems: to increase service availability we are all encouraged to use multiple AZ.
- Hard debugging: even using the correct tooling it’s pretty complicated to consider where the traffic is coming from and where it’s going. More challenging when running k8s across multiple availability zones with all its internal components, including control and data planes.
- Bi-directional pricing: Despite the quote “cross AZ data transfer within the region costs $0.01/GB” –> it’s 0.02$ per GB, once for sending and second for receiving. [InterZone-in and InterZone-out].
Depending upon the type of application and application architecture, data transfer usage differs. Though, most common data transfer usage line items are:
Data transfer out from Amazon S3 to either same AWS region or another AWS region
Data transfer out from Amazon Elastic Compute Cloud (Amazon EC2)
AWS NAT gateway charges
Inter-AZ data transfer
Inter-region data transfer
Setup purpose-built data transfer dashboard
The first step towards cost optimization is cost visibility with actionable findings. You need to analyze data transfer usage such as outbound internet, regional, and inter region data transfers from different AWS services.
We @2cloud.ai deploy a dashboard containing a number of views such as:
Usage and cost by AWS service and AWS Region
Data Transfer between AWS Regions
Data Transfer Out to Internet
Data Transfer within AWS Region
Data Transfer cost and usage of specific AWS services (For example, Elastic Load Balancing (ELB), AWS NAT Gateway, Amazon CloudFront, AWS Global Accelerator, and more)
From our dashboard you can also run custom queries like top NAT Gateway users, top resources with Inter-AZ traffic, Inter region usage and cost by resource, total data transfer out to internet usage and cost by resource or top S3 buckets with data transfer.
To deep dive in the cost contributors we also use Amazon VPC custom flow logs. Either:
- Publishing the flow log data to Amazon CloudWatch Logs, and then query the logs.
- Publishing the flow logs to an Amazon Simple Storage Service (Amazon S3) bucket, and then use Amazon Athena to query the logs.
BEST PRACTICES:
We have compiled a list of top best practices to follow to cost optimize the data transfer for your workload in AWS.
General:
- Enable CUR and use a Cost Intelligence dashboard to analyze data transfer components—cost and usage.
- Enable Amazon VPC flow logs to receive metadata about the actual traffic flow. This will provide an extra level of details compared to just the CUR.
- For high amounts of data transfer consider private pricing agreements.
- Use a Network Access Analyzer to detect unintended paths of traffic going through an AWS NAT Gateway.
Cost optimize Inter-AZ/regional data transfer:
- Design applications to be AZ aware. Use AZ affinity type of functionality available in the various infrastructure services you are using. As an example, consider an Aerospike NoSQL database in which the rack awareness feature can be used to reduce cross AZ traffic when possible.
- Use Application Load balancer and Classic Load balancer where possible as they have no cross AZ charges. This applies to traffic going to targets as well as traffic sourced from the same Amazon VPC going to the listener.
- Note that NLB charges for cross-AZ traffic to targets. Disable NLB cross-zone load balancing where not needed.
- Use an S3 AWS PrivateLink gateway endpoints (removing AWS NAT Gateway) for private subnet in your Amazon VPC to S3 bucket traffic within the same AWS Region.
- Amazon Route 53 Resolver Domain Name System resolution for Amazon Elastic File System endpoint returns the endpoint IP based on the AZ the query came from (AZ affinity), avoiding cross-AZ costs for the customer.
- Use Amazon VPC sharing, if using multiple account IDs in the same AZ ID, to reduce data transfer costs.
Cost optimize Inter region data transfer:
- Use private link network to connect in the same region.
- Use private link interface endpoint to reduce complexity of networking between regions.
Cost optimize data transfer out:
- Research if using AWS Direct Connect instead of the internet to talk between your AWS and on-premises environments is right for your use case. It’s usually multiple times less expensive.
- Note that adding AWS Transit Gateway for on-premises connectivity adds a data processing fee. Avoid AWS Transit Gateway for hybrid connectivity where expected data transfer volumes are high.
- Use edge service like Amazon CloudFront for static content delivery. Data transfer out to CloudFront is at no charge. CloudFront DTO is slightly cheaper than Amazon EC2 DTO.
- Use Amazon VPC flow logs (Amazon S3 + Amazon Athena) to identify traffic sent to Amazon Public IP ranges. This would indicate your Amazon VPC might not be using endpoints and is sending data out through the AWS NAT Gateway/internet gateway instead.
- Create an AWS NAT Gateway per AZ. This could help save on cross-AZ costs (note: there is a per hour charge for each AWS NAT Gateway).
Conclusion
- Tracking cross AZs data transfer cost may be pretty challenging but if you go with a data-driven approach based on the “correct data“, you’ll probably find a lot of space for optimizations.
- Cost issue analysis is not only for cost reduction. It’s also a great way to find bugs.
- Digging into the cross-AZ data transfer will help you knowing where it comes from, whether it makes sense, and whether you are willing to pay for it…
Reach out for help reducing your cloud costs!
