(Notes based mainly on the content from Ben Piper’s excellent Pluralsight course.)

Usually goes without saying, but especially when configuring VPC’s and hosts inside them on AWS - default is usually not the way to go. The course sort of reiterated the importance of designing the components (ENI, subnet etc) then adding them together based on your design, rather than clicking through the interface like you might in a lab.

The screenshots and NAT-instance walkthrough are from 2020. The managed alternatives are called out below where they change how we’d build this now.

What to design before clicking

For a respectable VPC design, decide these first:

  • CIDR ranges and how they avoid overlap with every other VPC, data center, partner, and VPN range.
  • Which subnets are public, private with egress, private with no egress, and inspection/edge subnets.
  • Whether outbound Internet is via NAT Gateway, egress-only Internet gateway for IPv6, firewall appliance, or no Internet route at all.
  • Whether networks need full routing through peering/Transit Gateway, or access to a particular service through PrivateLink. PrivateLink doesn’t provide general routing between VPCs.
  • How security groups, network ACLs, VPC Flow Logs, and route-table changes will be reviewed.

Quick notes:

  • A VPC can actually have more than one CIDR assigned to it.
  • The VPC’s implicit router uses the route table associated with the subnet. The router and its route table aren’t the same object.
  • An auto-assigned public IPv4 address normally survives a reboot, but is released when the instance is stopped. An Elastic IP stays allocated to the account until released. EC2 address lifecycle.
  • AWS reserves the subnet’s base address plus one for the VPC router. ip route shows the guest’s default gateway; don’t use ping to that reserved address as your test of Internet access. Reserved addresses.
  • For a public IPv4 instance, the Internet Gateway maps the public address to the instance’s private address. That is separate from a NAT Gateway providing outbound access for private instances.

VPC Peering

Adding peering is actually really simple, but we also need to update the route tables to point instances at the VPC peering link for the remote services. Also consider the security group configurations on both sides. One note is that the “target” field doesn’t always seem to do a great job of enumerating the potential targets, it took a few seconds after I type “pcx” for it to find the peering connection id created.

slow-route-modify

You need to go in and “accept” the VPC peering connection. It can feel unusual in the lab, but it makes sense when you consider that there’s a use case to peer to other VPC’s as well.

peering-to-customer

Peering limitations:

  • You can do same-region and cross-region peering. IPv6 over inter-region VPC peering is supported now, but both VPCs and the relevant instances/subnets need IPv6 configured and routed correctly.
  • You can’t peer overlapping IP CIDR blocks.
  • An instance in one VPC can’t use the peering connection and then the Internet Gateway of the remote VPC.
  • VPC peering is not transitive. If VPC A peers with VPC B, and VPC B peers with VPC C, A does not automatically reach C through B.
  • Full list: Unsupported VPC Peering Configurations

NAT

NAT instance can be used to allow outbound Internet access for otherwise non-Internet-connected subnets.

Historically, you could use “community AMIs” and search for amzn-ami-vpc-nat- for a pre-configured AMI that had the NAT configuration baked in. For current production work, prefer NAT Gateway unless you have a very specific reason to own the instance lifecycle, patching, scaling, and failover yourself.

When the machine comes up, don’t forget to disable the source/destination check that would otherwise prohibit the machine forwarding on things that don’t belong to it. (You do this from actions > source/destination ip check) in the EC2 console for the machine.

source-destination-check

Hang on, how come we aren’t just using a NAT Gateway?… Cost was the reason I was thinking about NAT instances in this lab. Depending on the needs, a NAT instance might be cheaper. The tradeoff is that it doesn’t scale as well, isn’t managed, and isn’t highly available unless you build that yourself. Avoid mixing the NAT and bastion roles in production unless you really know why you’re doing it; the convenience is not usually worth the blast-radius increase.

Outside of all that, the process for configuring it with subnets etc is similar.

Transit VPC

Lets you connect multiple VPC’s and sites together. We can minimize the number of VPN connections. We can also allow transitive routing (unlike peering).

You use some sort of virtual router instance. Cisco is the one in the example (CSR 1000V), but others are available.

As you’re setting up the VPC connection side in AWS the console provides you with a config file for the router (neat!)

csr-config

It is not done though, don’t forget to update the file with your IP addresses.

! You may need to populate these values throughout the config based on your setup:
! <interface_name/private_IP_on_outside_interface> - External interface of the CSR

Transit VPC should be part of your design plan from the start if you are maintaining older environments.

Transit VPC was a 2016 technology; Transit Gateway is the managed version that has been offered since 2018 and is where I would start for new multi-VPC designs. Amazon offers a tool to assist with migration, tgw-migrator.py, available here.

Monitoring

VPC flow logs record traffic metadata by network interface: source/destination IP, ports, protocol and other fields. They can be sent to CloudWatch Logs, S3 or Data Firehose. A VPC, subnet or individual ENI can be the collection scope. This is metadata, so it won’t tell us which HTTP URL was requested.

There are gaps: Amazon-provided DNS, DHCP, instance metadata (169.254.169.254) and Time Sync (169.254.169.123) traffic aren’t captured. The NLB exclusion is specifically traffic between an endpoint ENI and an NLB ENI, not all NLB traffic. Full limitations.

It isn’t real time. The maximum aggregation window is 10 minutes by default, with a 1-minute option; Nitro ENIs use 1 minute or less regardless. Delivery adds another delay. Keep that in mind before deciding a missing record proves the packet never arrived. Record format and timing.

CloudWatch gives us a convenient place to search; S3 with Athena is another option. Compare ingestion, retention and query costs for the amount of data you actually generate.

IPv6

This lab used an Amazon-provided /56 on the VPC and /64s on the subnets. AWS now also supports IPv6-only subnets for supported workloads, even though the VPC itself still needs an IPv4 CIDR. Don’t carry the old “every subnet needs IPv4” assumption into a new design. EC2 addressing options.

IPv6 is a place that administrators could easily accidentally open an instance to the Internet. Be careful. (::/0). Don’t forget, you test IPv6 addresses in a browser by surrounding them with [], like [2600:1f18:6135:4000::10].

Limitations are captured here

References