Private Networking: DNS, Routing, and TLS Failures

Moving workloads to private networks makes sense from a security perspective — no public IPs, no internet exposure, traffic stays within your cloud provider’s boundaries. But private networking introduces failure modes that don’t exist with public connectivity. DNS resolution that worked fine over the internet fails with private endpoints. Routing that seemed automatic now requires explicit configuration. TLS certificates that validated publicly get rejected privately.

Private networking isn’t “the same thing, but internal.” It’s a fundamentally different debugging domain where familiar tools give unfamiliar results and “connection refused” could mean ten different things.

I learned this the hard way during a database migration. We moved from a publicly-accessible RDS instance to a private endpoint. The application immediately failed with “connection refused.” We verified the endpoint URL was correct. DNS resolved… but to a different IP than expected. The IP was in a private subnet range, but our VPC couldn’t route to it. We added a route. Now we could reach the IP but got “connection reset.” The TLS handshake was failing because the certificate’s SAN didn’t include the private DNS name. We added the DNS name to the certificate. Handshake succeeded, but authentication failed — the database saw connections from an unexpected IP because we’d forgotten about NAT.

Each layer had its own failure. Each failure was masked by generic error messages. What should have been a 30-minute cutover turned into a 3-day debugging marathon. After that incident, we built a debugging playbook: DNS → routing → connectivity → TLS → application. Every subsequent migration followed that checklist.

Private DNS

DNS in private networks behaves differently than you’d expect. The same hostname can resolve to different IPs depending on where you query from, which resolver you use, and whether private hosted zones are properly configured. Understanding the resolution chain is the first step to debugging.

DNS Resolution Architecture

On a standard Linux system, resolution follows a predictable path: check /etc/hosts, then the local cache (if enabled), then query the nameserver from /etc/resolv.conf. Cloud VMs complicate this. AWS VPCs get a resolver at the VPC CIDR base address plus two (so 10.0.0.2 for a 10.0.0.0/16 VPC). This resolver handles Route 53 private hosted zones and falls back to public DNS for external names. GCP uses the metadata server at 169.254.169.254, Azure uses 168.63.129.16.

Kubernetes adds another layer. CoreDNS intercepts queries and applies search domains — a request for database becomes database.default.svc.cluster.local, then database.svc.cluster.local, then database.cluster.local before falling through to the node’s resolver. The ndots setting controls this behavior, and misconfiguring it causes subtle resolution failures.

The most common private DNS failures:

The diagram shows the critical branch point: if the private zone exists and is associated with your VPC, you get the private IP. If not, the query falls through to public DNS — which might return a public IP or NXDOMAIN, depending on whether the name exists publicly.

Private DNS Configuration

AWS private DNS centers on Route 53 private hosted zones — DNS zones that only resolve within associated VPCs.¹ You create a zone for internal.example.com, associate it with your VPCs, and add records for your private services. Simple in theory, but the gotchas accumulate.

VPC endpoints add another wrinkle. When you create an interface endpoint for an AWS service (like S3 or Secrets Manager) with private DNS enabled, Route 53 automatically creates a private hosted zone that overrides the public DNS. Requests to s3.us-east-1.amazonaws.com resolve to the endpoint’s private IPs instead of public AWS IPs. This requires enableDnsHostnames and enableDnsSupport on the VPC — settings that are easy to miss when troubleshooting.

For hybrid environments connecting AWS to on-premises networks, Route 53 Resolver endpoints bridge the gap. Inbound endpoints let on-premises DNS servers forward queries to AWS. Outbound endpoints let AWS workloads resolve on-premises DNS names. The configuration involves security groups, subnet placement, and forwarding rules — each a potential failure point.

The most common configuration mistake: creating a private hosted zone but forgetting to associate it with all the VPCs that need it. The zone exists, the records are correct, but queries from unassociated VPCs return NXDOMAIN. I’ve seen this take hours to diagnose because “the DNS is definitely configured.”

Routing and Connectivity

Once DNS resolves correctly, the next failure point is routing. Private networks require explicit route configuration — traffic doesn’t magically find its way to destinations outside your immediate subnet.

VPC Routing Fundamentals

Every VPC route table starts with an implicit local route that can’t be removed. For a VPC with CIDR 10.0.0.0/16, traffic to any 10.0.x.x address routes automatically within the VPC. Everything else needs explicit routes.

The common route targets:

Route selection follows the “most specific route wins” principle. If you have routes for 0.0.0.0/0 (internet), 10.0.0.0/16 (local), 10.1.0.0/16 (peered VPC), and 10.1.5.0/24 (specific subnet via transit gateway), traffic to 10.1.5.100 takes the /24 route — it’s the most specific match.

Routing failures in private networks present differently than on traditional networks. You won’t see clear error messages — timeouts look identical whether the cause is a missing route, a security group block, or a service that’s down. This ambiguity makes systematic debugging essential. These are the patterns you’ll encounter:

Connectivity Debugging

When you suspect routing works but connectivity still fails, you need to systematically test each layer. Start with ip route get <destination> to verify the kernel’s routing decision. Then test TCP connectivity with nc -zv -w 5 <host> <port>—this bypasses higher-level issues and tells you whether the port is reachable at all.

One gotcha that catches people: MTU issues. VPNs and tunnels often have lower MTU than the standard 1500 bytes. Large packets get silently dropped while small packets (like TCP handshakes) succeed. If connections establish but then hang when transferring data, test MTU with ping -M do -s 1472 <host> (1472 + 28 bytes header = 1500). Reduce the size until it works to find the actual path MTU.

I work through connectivity issues in a fixed order — each layer must pass before moving to the next:

Security Groups and NACLs

Security groups and NACLs both filter traffic, but they work differently and trip people up in different ways.

Security groups are stateful and operate at the instance (ENI⁵) level. You only define allow rules — there’s no explicit deny.

NACLs are stateless and operate at the subnet level. You can define both allow and deny rules, evaluated in order (lowest rule number first). Because they’re stateless, you must explicitly allow return traffic on ephemeral ports (1024-65535). The default NACL allows everything; custom NACLs start with deny-all. The classic NACL mistake: allowing outbound traffic but forgetting to allow inbound on ephemeral ports for the response. Connection times out, and you spend an hour checking security groups.

One more gotcha: security group rules can reference other security groups, but only within the same VPC (or across peered VPCs in the same region). Cross-VPC references don’t work with transit gateway — you must use CIDR blocks instead. And if your traffic goes through NAT, the destination sees the NAT IP, not the original source IP. Your security group rule allowing the source instance’s IP won’t match.

TLS Debugging

Once you’ve verified DNS resolution, routing, and basic connectivity, TLS becomes the next potential failure point. Private networking introduces TLS issues that don’t exist with public endpoints — hostname mismatches, untrusted internal CAs, and SNI problems are the usual suspects.

TLS Handshake Failures

The TLS handshake follows a specific sequence, and failures at each stage produce different error messages. The client sends a ClientHello with supported TLS versions, cipher suites, and the SNI (Server Name Indication). The server responds with its certificate. The client then verifies the certificate: Is the chain valid? Is it expired? Does the hostname match the certificate’s SAN (Subject Alternative Name)? Is the CA trusted?

Private networking introduces specific failure patterns:

The essential debugging command is openssl s_client:

# Connect and inspect certificate
echo | openssl s_client -connect api.internal:443 -servername api.internal 2>/dev/null | \
    openssl x509 -noout -text | grep -E "Subject:|Issuer:|Not After:|DNS:"

Always include -servername to send SNI. Without it, you might get a different certificate than your application receives, and your debugging results won’t match production behavior.

Certificate Management

Private services need certificates, and you have three options: public CAs, private CAs, or self-signed certificates.

Public CAs (Let’s Encrypt, DigiCert, AWS ACM) are trusted by all clients automatically. The limitation: the domain must be publicly verifiable via DNS or HTTP challenge. You can use public CA certificates for private endpoints if you own the domain and can complete the challenge — the certificate doesn’t care whether the service is publicly accessible.

Private CAs (AWS Private CA, HashiCorp Vault, step-ca, CFSSL⁶) issue certificates for internal services without public domain verification.⁷ The tradeoff: you must distribute the CA certificate to every client that needs to trust these certificates. Every language and runtime has its own trust store configuration:

Self-signed certificates require each certificate to be explicitly trusted — not just the CA. They work for development but don’t scale. Avoid them in production.

For certificate lifecycle, there are two common strategies: short-lived with automatic rotation (24 hours to 7 days, using tools like cert-manager or Vault), or long-lived with manual rotation (1-2 years). Short-lived certificates limit the damage window if compromised but require robust automation. Long-lived certificates are simpler to manage until they expire unexpectedly at 3am.

Whatever strategy you choose, monitor certificate expiry. Alert at 30 days (warning), 7 days (critical), and page immediately on expiry. Internal certificates don’t get the same attention as public ones, and they’ll bite you when you least expect it.

Debugging Playbook

The previous sections covered individual failure modes. This section puts them together into a systematic approach you can follow when something breaks.

Systematic Approach

Every private networking issue follows the same debugging sequence. Each layer depends on the previous one working correctly — there’s no point checking TLS if packets aren’t reaching the server.

At each step, the failure mode tells you where to look next. “Connection refused” at the port check means skip security groups and check whether the service is running. “Timeout” means go back to routing and security rules.

Common Scenarios

Two scenarios come up repeatedly: migrating existing services to private endpoints, and connecting services across VPCs. Both have predictable failure points.

Private Endpoint Migration

Moving from a public endpoint to a private one requires coordination across DNS, routing, security, and TLS. The migration itself is usually quick — the preparation takes longer.

Pre-migration checklist:

Migration patterns:

Post-migration verification:

Cross-VPC Connectivity

When services span multiple VPCs, you have three main options: peering, transit gateway, or Private Link. The choice depends on your topology and constraints.

Common debugging issues by pattern:

Conclusion

Private networking provides real security benefits — no public IPs to attack, traffic contained within provider boundaries, reduced attack surface. But those benefits come with operational complexity that catches teams off guard.

The core insight: private networks fail differently than public ones. DNS resolution depends on private hosted zones and VPC associations. Routing requires explicit configuration for every destination outside your subnet. TLS certificates need private hostnames that didn’t exist when you were public-only.

Build the debugging reflex before you need it: DNS → routing → connectivity → TLS → application. Each layer must work before the next can succeed. When something breaks at 3am, you don’t want to be guessing.

Three investments pay off:

The teams that handle private networking well aren’t the ones with the fanciest tools. They’re the ones who’ve internalized the debugging playbook before the first production incident.