Skip to content
labnotes.sh
ES

> note

VM-Series behind a Gateway Load Balancer on AWS

·18 min read# networking# cloud# security
In this note · 13 sections
  1. Architecture — What gets built
  2. Addressing — Five subnets per zone
  3. Scope — What gets inspected and what doesn't
  4. Costs — What it costs to keep this running
  5. Phase 1 — Infrastructure on AWS
  6. Terraform — The resources that matter
  7. Code — The complete configuration
  8. Phase 2 — Configuring the VM-Series
  9. Phase 3 — Enabling GENEVE parsing
  10. BGP — The control plane on the hybrid side
  11. Verification — What it looks like when it is right
  12. Mistakes — Ten that give you no message
  13. Limitations — What this design does not solve

Architecture — What gets built#

A Gateway Load Balancer distributes traffic to two VM-Series, one per zone. The firewall neither routes nor rewrites addresses: it receives the packet encapsulated in GENEVE, inspects it, and returns it down the same tunnel.

The per-zone symmetry is deliberate. With cross-zone load balancing disabled, traffic entering through one zone's endpoint is inspected on the firewall in that same zone and returns the way it came. That keeps the session on a single firewall, which is what a stateful device needs, and avoids cross-zone traffic charges.

Two-zone reference architecture: the virtual private gateway distributes to each zone's GWLB endpoint, which sends over GENEVE to the VM-Series in that same zone

Addressing from a real implementation. The GWLB node and the VM-Series share the fwdata subnet: that is why the health check arrives from an address on that same network and counts as intra-zone traffic.

Addressing — Five subnets per zone#

Each role needs its own subnet because each one carries a different route table. Mixing them is the quickest way to create a loop between the endpoint and the workload.

Role What lives there AZ a AZ b
public NAT gateway 10.20.0.0/24 10.20.10.0/24
mgmt VM-Series eth0 10.20.1.0/24 10.20.11.0/24
fwdata VM-Series eth1 and the GWLB nodes 10.20.2.0/24 10.20.12.0/24
gwlbe GWLB endpoint 10.20.3.0/24 10.20.13.0/24
app workloads 10.20.4.0/24 10.20.14.0/24

Which route each table carries#

Scope — What gets inspected and what doesn't#

Worth deciding before you write the policy, because not all three paths come for free.

Path How it reaches inspection Status
Hybrid on-prem ↔ workload Virtual private gateway edge route table works
Workload egress → internet 0.0.0.0/0 in the app table towards the endpoint works
East-west app-a ↔ app-b Needs routes more specific than the VPC local see limitations

Costs — What it costs to keep this running#

Almost 3 dollars an hour, and four out of every five go to the firewall licence, not to the AWS infrastructure.

us-east-1 rates: infrastructure queried against the AWS pricing API, software against the PAYG listing's dimension table. Data transfer not included.

Item USD/hour × USD/month
VM-Series licence — PAYG, m5.xlarge 1.1700 2 1,708.20
VM-Series — m5.xlarge compute 0.1920 2 280.32
NAT gateway 0.0450 2 65.70
Site-to-site VPN connection 0.0500 1 36.50
t3.micro workloads 0.0104 2 15.18
GWLB endpoint 0.0100 2 14.60
Public IPv4 in use 0.0050 4 14.60
EBS gp3 — 2×60 GB + 2×8 GB 0.0149 — 10.88
Gateway Load Balancer 0.0125 1 9.12
GWLB — capacity (minimum LCU) 0.0040 1 2.92
Total 2.9562 2,158.03
If it stays on for Cost Breakdown
one hour USD 2.96 licence 79 % · compute 13 % · network 8 %
one day USD 70.95
one month USD 2,158

The licence rules

The PAYG listing's software charge is six times the compute cost of the same instance: 1.17 against 0.192 an hour. It is billed through AWS Marketplace, which is why it never shows up in the pricing API or in estimates that only look at AWS resources.

The price goes by instance size, not family: every xlarge pays 1.17, large pays 0.99, 2xlarge 1.80 and 4xlarge 3.69. Dropping from m5.xlarge to m5.large would save USD 263 a month in licensing — but the GWLB needs at least 10.0.2 and an instance with enough NICs, so check support before you shrink it.

What to switch off#

Deployment reference: AMI PA-VM-AWS-11.1.15, product code e9yfvyj3uag5uo5j2hjikv74n, listing "VM-Series Next-Gen Virtual Firewall w/Advanced Threat Prevention (PAYG)".

Phase 1 — Infrastructure on AWS#

terraform init
terraform plan     # leerlo entero, incluidas las lineas con "-"
terraform apply

# comprobar que el direccionamiento interno del tunel quedo bien
terraform output pa440_tunnel1
terraform output pa440_tunnel2

# antes de dar por buena cualquier regla de security group
terraform plan -detailed-exitcode    # 0 = sin deriva

Read the whole plan, not just the summary. Plan: N to add, M to change does not distinguish between "adds a CIDR" and "revokes the ones that were there": on an aws_security_group with inline ingress blocks, an in-place change shows up as the old set with - and the new one with +.

The parameters that admit no variation:

Resource Value Why
Target group GENEVE port 6081 It is the only protocol a GWLB accepts
Target type ip With instance it registers the primary interface, which is the management one
Health check TCP port 80 The GWLB cannot query an arbitrary PAN-OS path
Stickiness source_ip_dest_ip_proto The firewall does not rewrite the 5-tuple; the flow must always return to the same device
Data ENI source_dest_check = false Without this, AWS drops traffic not addressed to the interface
Security group UDP 6081 and TCP 80 from the VPC CIDR GENEVE and health check

Careful with the AMI

Use the public SSM parameter of the listing you are subscribed to. An AMI from another listing boots just the same but bills against a subscription you do not hold. And do not solve it with most_recent: in Palo Alto's catalogue there are newer releases whose AMI carries an earlier date, so sorting by creation date picks the wrong version.

Verification

The endpoints in Available, and the edge route table associated with the virtual private gateway, with the app subnet routes pointing at their endpoints.

Terraform — The resources that matter#

The full deployment is 85 resources. Most of it is plumbing repeated per zone; these are the ones carrying the design's logic.

Group Resources Detail
Base network 1 + 10 VPC and five subnets per zone
Route tables 8 + 11 routes public, mgmt, fwdata, gwlbe, app and the VGW edge
Internet egress 1 + 2 + 2 IGW, NAT gateways and their EIPs
GWLB 5 balancer, listener, target group, endpoint service and 2 endpoints
Firewalls 2 + 4 + 2 instances, two ENIs each and the management EIPs
VPN 4 customer gateway, VGW, connection and route propagation
Security 3 one security group per role

The GWLB and its target group#

The GWLB lives in the fwdata subnets, alongside the firewalls. The target is registered by IP and not by instance: with instance, AWS would use the primary interface, which here is the management one.

hclThe GWLB and its target group30 lines download .txt
resource "aws_lb" "gwlb" {
  load_balancer_type               = "gateway"
  subnets                          = [for k, v in aws_subnet.fwdata : v.id]
  enable_cross_zone_load_balancing = false
}

resource "aws_lb_target_group" "fw" {
  target_type = "ip"
  protocol    = "GENEVE"
  port        = 6081

  health_check {
    protocol = "TCP"
    port     = 80
  }

  # El firewall no reescribe la 5-tupla: el flujo debe volver
  # siempre a la misma instancia.
  stickiness {
    type    = "source_ip_dest_ip_proto"
    enabled = true
  }
}

resource "aws_lb_target_group_attachment" "fw" {
  for_each          = local.subnets
  target_id         = aws_network_interface.fw_data[each.key].private_ip
  availability_zone = local.az[each.key]
  port              = 6081
}

The edge association#

This is the piece that makes inbound traffic get inspected. A route table associated with the virtual private gateway through gateway_id, not with a subnet, and carrying routes more specific than the VPC local.

resource "aws_route" "vgw_edge_to_app" {
  for_each               = local.subnets
  route_table_id         = aws_route_table.vgw_edge.id
  destination_cidr_block = each.value.app          # 10.20.4.0/24 y 10.20.14.0/24
  vpc_endpoint_id        = aws_vpc_endpoint.gwlbe[each.key].id
}

resource "aws_route_table_association" "vgw_edge" {
  gateway_id     = aws_vpn_gateway.vgw.id          # asociacion de BORDE
  route_table_id = aws_route_table.vgw_edge.id
}

The firewall interfaces#

Two NICs: management on device_index 0 and data on 1, which is the one the VM-Series sees as ethernet1/1. source_dest_check goes to false only on the data one.

hclThe data interface and the instance21 lines download .txt
resource "aws_network_interface" "fw_data" {
  for_each          = local.subnets
  subnet_id         = aws_subnet.fwdata[each.key].id
  security_groups   = [aws_security_group.fwdata.id]
  source_dest_check = false
}

resource "aws_instance" "fw" {
  for_each      = local.subnets
  instance_type = "m5.xlarge"

  network_interface { network_interface_id = ...fw_mgmt[each.key].id, device_index = 0 }
  network_interface { network_interface_id = ...fw_data[each.key].id, device_index = 1 }

  user_data = <<-EOT
    type=dhcp-client
    hostname=fw-gwlb-${each.key}
    dhcp-accept-server-hostname=no
    dhcp-accept-server-domain=no
  EOT
}

Selecting the AMI#

Through the public SSM parameter, never with most_recent: in Palo Alto's catalogue there are newer releases whose AMI carries an earlier date, so sorting by creation date picks the wrong version.

data "aws_ssm_parameter" "panos" {
  name = "/aws/service/marketplace/prod-<listing>/pan-os-<version>"
}

The three security groups#

Group Ingress From
fwdata UDP 6081, TCP 80, ICMP VPC CIDR
mgmt TCP 22, TCP 443, ICMP management list + on-prem prefixes
app everything VPC CIDR + on-prem prefixes

The management one takes a list of CIDRs, not a string. With a single value the only way out is adding the real IP by hand in the console, and since the ingress blocks are inline, the next apply silently revokes it.

Code — The complete configuration#

The seven files exactly as they are applied, with public addresses and account identifiers replaced by placeholders. This is the whole deployment: no external modules, no shared state.

File Contents
00-variables.tf Provider, variables and the per-zone subnet map.
01-vpc.tf VPC, the ten subnets, IGW, NAT gateways and the three security groups.
02-gwlb.tf Balancer, target group, listener, endpoint service and endpoints.
03-firewalls.tf AMI via SSM, the ENIs, the VM-Series and the workloads.
04-vpn.tf Customer gateway, VGW and the connection with its two tunnels.
05-routing.tf The six route tables, including the edge one associated with the VGW.
06-outputs.tf Outputs, among them the internal addressing of the tunnels.
hclProvider, variables and locals144 lines download .txt
terraform {
  required_version = ">= 1.5"
  required_providers {
    aws    = { source = "hashicorp/aws", version = "~> 5.0" }
    random = { source = "hashicorp/random", version = "~> 3.6" }
  }
}

provider "aws" {
  region = var.region
  default_tags {
    tags = {
      Project   = "kf-gwlb-lab"
      ManagedBy = "terraform"
      Owner     = "csegovia"
    }
  }
}

variable "region" {
  type    = string
  default = "us-east-1"
}

variable "vpc_cidr" {
  description = "Verificado libre contra la tabla de rutas del PA-410."
  type        = string
  default     = "10.20.0.0/16"
}

variable "azs" {
  type    = list(string)
  default = ["us-east-1a", "us-east-1b"]
}

# Cinco subnets por AZ. Cada una tiene un rol distinto en el path de GENEVE.
locals {
  az = { a = var.azs[0], b = var.azs[1] }

  subnets = {
    a = {
      public = "10.20.0.0/24" # NAT GW
      mgmt   = "10.20.1.0/24" # eth0 del VM-Series
      fwdata = "10.20.2.0/24" # eth1, GENEVE, target del GWLB
      gwlbe  = "10.20.3.0/24" # endpoints del GWLB
      app    = "10.20.4.0/24" # workloads
    }
    b = {
      public = "10.20.10.0/24"
      mgmt   = "10.20.11.0/24"
      fwdata = "10.20.12.0/24"
      gwlbe  = "10.20.13.0/24"
      app    = "10.20.14.0/24"
    }
  }

  # Rutas app -> GWLBE para cada prefijo del lab, por AZ.
  app_lab_routes = {
    for p in setproduct(keys(local.subnets), var.lab_cidrs) :
    "${p[0]}|${p[1]}" => { az = p[0], cidr = p[1] }
  }
}

# --- VPN / BGP ---------------------------------------------------------------

variable "cgw_public_ip" {
  description = "IP publica del PA-410 (ethernet1/1)."
  type        = string
  default     = "<IP-PUBLICA-CGW>"
}

variable "cgw_asn" {
  description = "AS local del PA-410, ya existente en el VR default."
  type        = number
  default     = 65000
}

variable "vgw_asn" {
  type    = number
  default = 64512
}

variable "tunnel1_inside_cidr" {
  description = "169.254.100.0/30 NO se puede: la usa el peer HEX-CASA."
  type        = string
  default     = "169.254.200.0/30"
}

variable "tunnel2_inside_cidr" {
  type    = string
  default = "169.254.201.0/30"
}

variable "lab_cidrs" {
  description = "Prefijos del home lab que se alcanzan por el tunel y se inspeccionan."
  type        = list(string)
  default = [
    "192.168.100.0/24",
    "192.168.106.0/24",
    "10.1.0.0/16",
    "10.142.0.0/16",
  ]
}

# --- Firewalls ---------------------------------------------------------------

# Parametro publico de SSM que publica AWS Marketplace. Resuelve al AMI
# correcto por region sin hardcodear IDs.
#
#   prod-<ID-DEL-LISTING> = listing "VM-Series Next-Gen Virtual Firewall
#                        w/Advanced Threat Prevention (PAYG)"
#
# OJO: usar el AMI de OTRO listing hace que la instancia facture contra una
# suscripcion que no tiene. El product code va atado al listing.
variable "panos_ssm_parameter" {
  type    = string
  default = "/aws/service/marketplace/prod-<ID-DEL-LISTING>/pan-os-11.1.15"
}

# Override manual. Si se setea, gana sobre el parametro SSM.
variable "panos_ami_id" {
  type    = string
  default = ""
}

variable "panos_instance_type" {
  description = "GWLB necesita minimo 10.0.2. m5.xlarge soporta las NICs necesarias."
  type        = string
  default     = "m5.xlarge"
}

variable "key_pair_name" {
  type = string
}

# Lista, no string: la IP publica de salida del equipo desde el que se
# administra NO siempre es la de ethernet1/1 del PA-410. Con un solo valor la
# unica salida era agregar la IP real a mano en la consola, y como los bloques
# ingress son inline, el siguiente terraform apply la revocaba en silencio.
variable "admin_cidrs" {
  description = "CIDRs con acceso de gestion a los VM-Series (SSH, HTTPS, ICMP)."
  type        = list(string)
  default     = ["<IP-PUBLICA-CGW>/32"]
}
hclThe network: VPC, subnets, NAT and security groups166 lines download .txt
resource "aws_vpc" "lab" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true
  tags                 = { Name = "kf-gwlb-vpc" }
}

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.lab.id
  tags   = { Name = "kf-gwlb-igw" }
}

# --- Subnets -----------------------------------------------------------------

resource "aws_subnet" "public" {
  for_each          = local.subnets
  vpc_id            = aws_vpc.lab.id
  cidr_block        = each.value.public
  availability_zone = local.az[each.key]
  tags              = { Name = "kf-public-${each.key}" }
}

resource "aws_subnet" "mgmt" {
  for_each          = local.subnets
  vpc_id            = aws_vpc.lab.id
  cidr_block        = each.value.mgmt
  availability_zone = local.az[each.key]
  tags              = { Name = "kf-mgmt-${each.key}" }
}

resource "aws_subnet" "fwdata" {
  for_each          = local.subnets
  vpc_id            = aws_vpc.lab.id
  cidr_block        = each.value.fwdata
  availability_zone = local.az[each.key]
  tags              = { Name = "kf-fwdata-${each.key}" }
}

resource "aws_subnet" "gwlbe" {
  for_each          = local.subnets
  vpc_id            = aws_vpc.lab.id
  cidr_block        = each.value.gwlbe
  availability_zone = local.az[each.key]
  tags              = { Name = "kf-gwlbe-${each.key}" }
}

resource "aws_subnet" "app" {
  for_each          = local.subnets
  vpc_id            = aws_vpc.lab.id
  cidr_block        = each.value.app
  availability_zone = local.az[each.key]
  tags              = { Name = "kf-app-${each.key}" }
}

# --- NAT Gateway por AZ ------------------------------------------------------
# Uno por AZ. Con uno solo compartido el retorno cruza AZ y el path se vuelve
# dificil de explicar. Son US$0.045/hr cada uno: el segundo item mas caro
# despues de los firewalls.

resource "aws_eip" "nat" {
  for_each   = local.subnets
  domain     = "vpc"
  tags       = { Name = "kf-eip-nat-${each.key}" }
  depends_on = [aws_internet_gateway.igw]
}

resource "aws_nat_gateway" "nat" {
  for_each      = local.subnets
  allocation_id = aws_eip.nat[each.key].id
  subnet_id     = aws_subnet.public[each.key].id
  tags          = { Name = "kf-nat-${each.key}" }
  depends_on    = [aws_internet_gateway.igw]
}

# --- Security groups ---------------------------------------------------------

resource "aws_security_group" "mgmt" {
  name        = "kf-sg-mgmt"
  description = "Gestion de los VM-Series"
  vpc_id      = aws_vpc.lab.id

  ingress {
    description = "HTTPS GUI"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = concat(var.admin_cidrs, var.lab_cidrs)
  }
  ingress {
    description = "SSH"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = concat(var.admin_cidrs, var.lab_cidrs)
  }
  ingress {
    description = "ICMP"
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = concat(var.admin_cidrs, var.lab_cidrs)
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
  tags = { Name = "kf-sg-mgmt" }
}

# GOTCHA GWLB #1: si falta el 6081 el target queda unhealthy y AWS no dice
# por que. Es el error mas comun de toda la integracion.
resource "aws_security_group" "fwdata" {
  name        = "kf-sg-fwdata"
  description = "Interfaz de datos: GENEVE + health check del GWLB"
  vpc_id      = aws_vpc.lab.id

  ingress {
    description = "GENEVE desde el GWLB"
    from_port   = 6081
    to_port     = 6081
    protocol    = "udp"
    cidr_blocks = [var.vpc_cidr]
  }
  ingress {
    description = "Health check del target group"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = [var.vpc_cidr]
  }
  ingress {
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = [var.vpc_cidr]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
  tags = { Name = "kf-sg-fwdata" }
}

resource "aws_security_group" "app" {
  name        = "kf-sg-app"
  description = "Workloads"
  vpc_id      = aws_vpc.lab.id

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = concat([var.vpc_cidr], var.lab_cidrs)
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
  tags = { Name = "kf-sg-app" }
}
hclThe Gateway Load Balancer and its endpoints81 lines download .txt
# =============================================================================
# Gateway Load Balancer
# El GWLB encapsula en GENEVE (UDP 6081) y reparte hacia los VM-Series.
# Los firewalls trabajan como bump-in-the-wire: no rutean, no hacen NAT.
# =============================================================================

resource "aws_lb" "gwlb" {
  name                             = "kf-gwlb"
  load_balancer_type               = "gateway"
  subnets                          = [for k, v in aws_subnet.fwdata : v.id]
  # Probado en true (2026-08-17) para descartar que fuera la causa de que el
  # GWLB no entregue al target: NO cambio nada, el trafico sigue muriendo entre
  # el endpoint y el target. Se vuelve a false, que es el diseno buscado
  # (inspeccion local por AZ) y evita cargos de trafico cross-AZ.
  enable_cross_zone_load_balancing = false

  tags = { Name = "kf-gwlb" }
}

# GOTCHA GWLB #2: el health check no puede pegarle a un path cualquiera del
# PAN-OS. Se usa TCP:80 y hay que habilitar HTTP en el interface management
# profile de la interfaz de datos, o el target nunca pasa a healthy.
resource "aws_lb_target_group" "fw" {
  name        = "kf-gwlb-tg"
  target_type = "ip"
  protocol    = "GENEVE"
  port        = 6081
  vpc_id      = aws_vpc.lab.id

  health_check {
    protocol            = "TCP"
    port                = 80
    interval            = 10
    healthy_threshold   = 3
    unhealthy_threshold = 3
  }

  # El firewall no reescribe la 5-tupla, asi que el flujo debe volver siempre
  # a la misma instancia.
  stickiness {
    type    = "source_ip_dest_ip_proto"
    enabled = true
  }

  tags = { Name = "kf-gwlb-tg" }
}

# Se registra la IP de la ENI de datos, no el instance-id: con target_type
# "instance" el GWLB usaria la interfaz primaria, que aca es la de gestion.
resource "aws_lb_target_group_attachment" "fw" {
  for_each          = local.subnets
  target_group_arn  = aws_lb_target_group.fw.arn
  target_id         = aws_network_interface.fw_data[each.key].private_ip
  availability_zone = local.az[each.key]
  port              = 6081
}

resource "aws_lb_listener" "gwlb" {
  load_balancer_arn = aws_lb.gwlb.arn
  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.fw.arn
  }
}

# --- Endpoint service + endpoints por AZ -------------------------------------

resource "aws_vpc_endpoint_service" "gwlb" {
  acceptance_required        = false
  gateway_load_balancer_arns = [aws_lb.gwlb.arn]
  tags                       = { Name = "kf-gwlb-endpoint-service" }
}

resource "aws_vpc_endpoint" "gwlbe" {
  for_each          = local.subnets
  vpc_id            = aws_vpc.lab.id
  service_name      = aws_vpc_endpoint_service.gwlb.service_name
  vpc_endpoint_type = "GatewayLoadBalancer"
  subnet_ids        = [aws_subnet.gwlbe[each.key].id]
  tags              = { Name = "kf-gwlbe-${each.key}" }
}
hclThe AMI and the two VM-Series102 lines download .txt
# No se usa data "aws_ami" con most_recent: en la lista de AMIs de Palo Alto
# la 11.1.15 es una release MAS nueva que la 11.1.6-h35, pero su AMI tiene
# fecha ANTERIOR. Ordenar por CreationDate elige la version equivocada.
data "aws_ssm_parameter" "panos" {
  count = var.panos_ami_id == "" ? 1 : 0
  name  = var.panos_ssm_parameter
}

locals {
  panos_ami = var.panos_ami_id != "" ? var.panos_ami_id : data.aws_ssm_parameter.panos[0].value
}

# Dos NICs: gestion y datos. Sin overlay routing, sin NAT, sin interfaz
# untrust. Todo el trafico entra y sale encapsulado en GENEVE por eth1.

resource "aws_network_interface" "fw_mgmt" {
  for_each        = local.subnets
  subnet_id       = aws_subnet.mgmt[each.key].id
  security_groups = [aws_security_group.mgmt.id]
  tags            = { Name = "kf-fw-${each.key}-mgmt" }
}

resource "aws_network_interface" "fw_data" {
  for_each          = local.subnets
  subnet_id         = aws_subnet.fwdata[each.key].id
  security_groups   = [aws_security_group.fwdata.id]
  source_dest_check = false
  tags              = { Name = "kf-fw-${each.key}-data" }
}

resource "aws_eip" "fw_mgmt" {
  for_each          = local.subnets
  domain            = "vpc"
  network_interface = aws_network_interface.fw_mgmt[each.key].id
  tags              = { Name = "kf-eip-fw-${each.key}-mgmt" }
  depends_on        = [aws_internet_gateway.igw]
}

resource "aws_instance" "fw" {
  for_each      = local.subnets
  ami           = local.panos_ami
  instance_type = var.panos_instance_type
  key_name      = var.key_pair_name

  network_interface {
    network_interface_id = aws_network_interface.fw_mgmt[each.key].id
    device_index         = 0
  }
  network_interface {
    network_interface_id = aws_network_interface.fw_data[each.key].id
    device_index         = 1
  }

  user_data = <<-EOT
    type=dhcp-client
    hostname=kf-fw-gwlb-${each.key}
    dns-primary=169.254.169.253
    dns-secondary=8.8.8.8
    dhcp-accept-server-hostname=no
    dhcp-accept-server-domain=no
  EOT

  root_block_device {
    volume_size = 60
    volume_type = "gp3"
  }

  tags = { Name = "kf-fw-gwlb-${each.key}" }
}

# --- Workloads ---------------------------------------------------------------

data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]
  filter {
    name   = "name"
    values = ["al2023-ami-2023.*-x86_64"]
  }
}

# Sin IP publica. Se prueban desde el home lab por el tunel, que es
# justamente el escenario que se quiere demostrar.
resource "aws_instance" "app" {
  for_each = local.subnets

  ami                    = data.aws_ami.al2023.id
  instance_type          = "t3.micro"
  subnet_id              = aws_subnet.app[each.key].id
  private_ip             = cidrhost(local.subnets[each.key].app, 10)
  vpc_security_group_ids = [aws_security_group.app.id]
  key_name               = var.key_pair_name

  user_data = <<-EOT
    #!/bin/bash
    dnf install -y nginx
    echo "kf-gwlb-lab · stack ${each.key} · $(hostname)" > /usr/share/nginx/html/index.html
    systemctl enable --now nginx
  EOT

  tags = { Name = "kf-app-${each.key}" }
}
hclThe site-to-site VPN and its PSKs70 lines download .txt
# PSK: 8-64 chars, alfanumerico mas . y _ , no puede empezar con 0.
resource "random_password" "psk" {
  count            = 2
  length           = 32
  special          = true
  override_special = "._"
  numeric          = true
  upper            = true
  lower            = true
}

resource "aws_customer_gateway" "cgw" {
  bgp_asn    = var.cgw_asn
  ip_address = var.cgw_public_ip
  type       = "ipsec.1"
  tags       = { Name = "kf-cgw-pa410" }
}

resource "aws_vpn_gateway" "vgw" {
  vpc_id          = aws_vpc.lab.id
  amazon_side_asn = var.vgw_asn
  tags            = { Name = "kf-vgw" }
}

resource "aws_vpn_connection" "s2s" {
  customer_gateway_id = aws_customer_gateway.cgw.id
  vpn_gateway_id      = aws_vpn_gateway.vgw.id
  type                = "ipsec.1"
  static_routes_only  = false # BGP

  # --- Tunel 1 ---------------------------------------------------------------
  tunnel1_inside_cidr   = var.tunnel1_inside_cidr
  tunnel1_preshared_key = random_password.psk[0].result
  tunnel1_ike_versions  = ["ikev2"]

  tunnel1_phase1_encryption_algorithms = ["AES256"]
  tunnel1_phase1_integrity_algorithms  = ["SHA2-256"]
  tunnel1_phase1_dh_group_numbers      = [14]
  tunnel1_phase1_lifetime_seconds      = 28800

  tunnel1_phase2_encryption_algorithms = ["AES256"]
  tunnel1_phase2_integrity_algorithms  = ["SHA2-256"]
  tunnel1_phase2_dh_group_numbers      = [14]
  tunnel1_phase2_lifetime_seconds      = 3600

  tunnel1_startup_action      = "start" # AWS inicia; tenemos IP fija
  tunnel1_dpd_timeout_action  = "restart"
  tunnel1_dpd_timeout_seconds = 30

  # --- Tunel 2 ---------------------------------------------------------------
  tunnel2_inside_cidr   = var.tunnel2_inside_cidr
  tunnel2_preshared_key = random_password.psk[1].result
  tunnel2_ike_versions  = ["ikev2"]

  tunnel2_phase1_encryption_algorithms = ["AES256"]
  tunnel2_phase1_integrity_algorithms  = ["SHA2-256"]
  tunnel2_phase1_dh_group_numbers      = [14]
  tunnel2_phase1_lifetime_seconds      = 28800

  tunnel2_phase2_encryption_algorithms = ["AES256"]
  tunnel2_phase2_integrity_algorithms  = ["SHA2-256"]
  tunnel2_phase2_dh_group_numbers      = [14]
  tunnel2_phase2_lifetime_seconds      = 3600

  tunnel2_startup_action      = "start"
  tunnel2_dpd_timeout_action  = "restart"
  tunnel2_dpd_timeout_seconds = 30

  tags = { Name = "kf-vpn-pa410" }
}
hclThe route tables and propagation155 lines download .txt
# =============================================================================
# ROUTING
#
# El path completo de un flujo de egress:
#   app -> GWLBE (misma AZ) -> GWLB -> firewall -> GWLB -> GWLBE -> NAT GW -> IGW
#
# El path de un flujo hibrido entrante:
#   home lab -> VGW -> [edge route table] -> GWLBE -> firewall -> GWLBE -> app
# =============================================================================

# --- Public: NAT GW + IGW ----------------------------------------------------
# El retorno del NAT GW tiene que volver por el firewall, si no el flujo queda
# asimetrico y solo se inspecciona la ida.

resource "aws_route_table" "public" {
  for_each = local.subnets
  vpc_id   = aws_vpc.lab.id
  tags     = { Name = "kf-rt-public-${each.key}" }
}

resource "aws_route" "public_default" {
  for_each               = aws_route_table.public
  route_table_id         = each.value.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.igw.id
}

resource "aws_route" "public_return_to_fw" {
  for_each               = local.subnets
  route_table_id         = aws_route_table.public[each.key].id
  destination_cidr_block = each.value.app
  vpc_endpoint_id        = aws_vpc_endpoint.gwlbe[each.key].id
}

resource "aws_route_table_association" "public" {
  for_each       = aws_subnet.public
  subnet_id      = each.value.id
  route_table_id = aws_route_table.public[each.key].id
}

# --- Mgmt --------------------------------------------------------------------
# Trafico de gestion, no se inspecciona a proposito: si el firewall se cae,
# igual se puede llegar a administrarlo.

resource "aws_route_table" "mgmt" {
  vpc_id = aws_vpc.lab.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
  tags = { Name = "kf-rt-mgmt" }
}

resource "aws_route_table_association" "mgmt" {
  for_each       = aws_subnet.mgmt
  subnet_id      = each.value.id
  route_table_id = aws_route_table.mgmt.id
}

# --- Fwdata ------------------------------------------------------------------
# Solo la ruta local. El GENEVE nace y muere dentro de la VPC.

resource "aws_route_table" "fwdata" {
  vpc_id = aws_vpc.lab.id
  tags   = { Name = "kf-rt-fwdata" }
}

resource "aws_route_table_association" "fwdata" {
  for_each       = aws_subnet.fwdata
  subnet_id      = each.value.id
  route_table_id = aws_route_table.fwdata.id
}

# --- GWLBE: salida despues de inspeccion -------------------------------------
# Aca cae el trafico ya inspeccionado que devuelve el firewall.

resource "aws_route_table" "gwlbe" {
  for_each = local.subnets
  vpc_id   = aws_vpc.lab.id
  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat[each.key].id
  }
  tags = { Name = "kf-rt-gwlbe-${each.key}" }
}

resource "aws_route_table_association" "gwlbe" {
  for_each       = aws_subnet.gwlbe
  subnet_id      = each.value.id
  route_table_id = aws_route_table.gwlbe[each.key].id
}

# Los prefijos del lab llegan por BGP. GOTCHA #1 del post anterior sigue
# vigente: la propagacion viene apagada por defecto.
resource "aws_vpn_gateway_route_propagation" "gwlbe" {
  for_each       = aws_route_table.gwlbe
  vpn_gateway_id = aws_vpn_gateway.vgw.id
  route_table_id = each.value.id
}

resource "aws_vpn_gateway_route_propagation" "mgmt" {
  vpn_gateway_id = aws_vpn_gateway.vgw.id
  route_table_id = aws_route_table.mgmt.id
}

# --- App: todo sale por el GWLBE ---------------------------------------------
# Sin propagacion del VGW aca: si se propagaran los prefijos del lab, el
# trafico iria directo al VGW sin pasar por el firewall.

resource "aws_route_table" "app" {
  for_each = local.subnets
  vpc_id   = aws_vpc.lab.id
  tags     = { Name = "kf-rt-app-${each.key}" }
}

resource "aws_route" "app_default" {
  for_each               = local.subnets
  route_table_id         = aws_route_table.app[each.key].id
  destination_cidr_block = "0.0.0.0/0"
  vpc_endpoint_id        = aws_vpc_endpoint.gwlbe[each.key].id
}

resource "aws_route" "app_to_lab" {
  for_each               = local.app_lab_routes
  route_table_id         = aws_route_table.app[each.value.az].id
  destination_cidr_block = each.value.cidr
  vpc_endpoint_id        = aws_vpc_endpoint.gwlbe[each.value.az].id
}

resource "aws_route_table_association" "app" {
  for_each       = aws_subnet.app
  subnet_id      = each.value.id
  route_table_id = aws_route_table.app[each.key].id
}

# --- Edge route table del VGW ------------------------------------------------
# Esto es lo que hace que el trafico hibrido se inspeccione. Sin esta
# asociacion el trafico del tunel entra directo a las subnets app.

resource "aws_route_table" "vgw_edge" {
  vpc_id = aws_vpc.lab.id
  tags   = { Name = "kf-rt-vgw-edge" }
}

resource "aws_route" "vgw_edge_to_app" {
  for_each               = local.subnets
  route_table_id         = aws_route_table.vgw_edge.id
  destination_cidr_block = each.value.app
  vpc_endpoint_id        = aws_vpc_endpoint.gwlbe[each.key].id
}

resource "aws_route_table_association" "vgw_edge" {
  gateway_id     = aws_vpn_gateway.vgw.id
  route_table_id = aws_route_table.vgw_edge.id
}
hclThe deployment outputs58 lines download .txt
output "firewalls" {
  description = "EIP de gestion de cada VM-Series y la IP de datos registrada en el target group."
  value = {
    for k in keys(local.subnets) : k => {
      mgmt_eip = aws_eip.fw_mgmt[k].public_ip
      data_ip  = aws_network_interface.fw_data[k].private_ip
    }
  }
}

output "app_servers" {
  description = "Workloads. Sin IP publica: se prueban desde el home lab por el tunel."
  value       = { for k, v in aws_instance.app : k => v.private_ip }
}

output "gwlb_endpoints" {
  value = { for k, v in aws_vpc_endpoint.gwlbe : k => v.id }
}

output "target_group_arn" {
  description = "Para vigilar el estado con: aws elbv2 describe-target-health --target-group-arn ..."
  value       = aws_lb_target_group.fw.arn
}

# --- Datos para armar el tunel en el PA-410 ---------------------------------

# inside_local / inside_remote salen de los atributos que devuelve AWS, no de
# cidrhost(). AWS asigna el PRIMER host del /30 al VGW y el segundo al customer
# gateway; calcularlo a mano invita a dejarlo al reves, que es justo lo que
# paso antes: el PA quedaba con la .1 (que es del VGW) y peereaba contra la .2.
output "pa440_tunnel1" {
  value = {
    aws_peer_ip   = aws_vpn_connection.s2s.tunnel1_address
    inside_local  = aws_vpn_connection.s2s.tunnel1_cgw_inside_address
    inside_remote = aws_vpn_connection.s2s.tunnel1_vgw_inside_address
    bgp_peer_asn  = var.vgw_asn
  }
}

output "pa440_tunnel2" {
  value = {
    aws_peer_ip   = aws_vpn_connection.s2s.tunnel2_address
    inside_local  = aws_vpn_connection.s2s.tunnel2_cgw_inside_address
    inside_remote = aws_vpn_connection.s2s.tunnel2_vgw_inside_address
    bgp_peer_asn  = var.vgw_asn
  }
}

# CREDENCIAL VIVA. Nunca en el blog ni en screenshots.
output "psk_tunnel1" {
  value     = aws_vpn_connection.s2s.tunnel1_preshared_key
  sensitive = true
}

output "psk_tunnel2" {
  value     = aws_vpn_connection.s2s.tunnel2_preshared_key
  sensitive = true
}

Phase 2 — Configuring the VM-Series#

Identical on both devices: the data interface takes its IP by DHCP, so there is nothing to customise per zone.

configure

set network profiles interface-management-profile GWLB-HC http yes
set network profiles interface-management-profile GWLB-HC ping yes

set network interface ethernet ethernet1/1 layer3 dhcp-client enable yes
set network interface ethernet ethernet1/1 layer3 dhcp-client create-default-route no
set network interface ethernet ethernet1/1 layer3 interface-management-profile GWLB-HC

set network virtual-router default interface ethernet1/1
set zone gwlb network layer3 ethernet1/1

The http yes in the management profile is what makes the firewall answer the health check. The zone command depends on multi-vsys: with off it is set zone; with on it is set vsys vsys1 zone. Check it with show system info | match multi-vsys before writing the line: if the zone is not created, every rule that references it fails afterwards.

Policy: a single zone, everything intra-zone#

Traffic comes in and goes out through the same interface, so source and destination land in the same zone. The policy has to allow that explicitly.

The health check needs a rule of its own

Enabling HTTP in the management profile is necessary but not sufficient. The health check leaves a GWLB node on the same subnet towards the firewall's data IP: it is intra-zone traffic and it goes through the policy. Any any/any cleanup rule kills it first, and the target stays unhealthy forever without AWS explaining a thing.

set service TCP-80 protocol tcp port 80
set address-group GWLB-NODES static [ FWDATA-A FWDATA-B ]

set rulebase security rules GWLB-HEALTHCHECK from gwlb
set rulebase security rules GWLB-HEALTHCHECK to gwlb
set rulebase security rules GWLB-HEALTHCHECK source GWLB-NODES
set rulebase security rules GWLB-HEALTHCHECK destination any
set rulebase security rules GWLB-HEALTHCHECK application any
set rulebase security rules GWLB-HEALTHCHECK service TCP-80
set rulebase security rules GWLB-HEALTHCHECK action allow

move rulebase security rules GWLB-HEALTHCHECK top

The move is not cosmetic: new rules are appended at the end, which is below the cleanup, where they never match.

For the inspection rules use service any. The combination of application any with service application-default matches nothing: with no concrete applications in the rule there is no default port to derive the service from.

Verification

Both targets healthy. It takes about 30 seconds with the default interval.

aws elbv2 describe-target-health --region <region> \
  --target-group-arn <arn>

Phase 3 — Enabling GENEVE parsing#

This is the step that makes the design work, and the only one that produces no error at all if you skip it.

PAN-OS does not decapsulate GENEVE until you ask it to. Without that it receives the UDP 6081 packets, never sees the inner addresses, inspects nothing, and never returns them to the tunnel.

On both devices

request plugins vm_series aws gwlb inspect enable yes
request plugins vm_series aws gwlb overlay-routing enable no
show plugins vm_series aws gwlb

GWLB enabled has to say True. The message vpc-endpoint association not found is normal in bump-in-the-wire: that association belongs to overlay routing mode, which is deliberately not used here.

INSPECT ENABLE NO GWLB GENEVE 6081 VM-Series does not decapsulate nothing returns to the tunnel rules at 0 hits · no deny · no log INSPECT ENABLE YES GWLB GENEVE 6081 decap → policy → re-encap App-ID sees the inner packet GENEVE GWLB endpoint workload The TCP 80 health check crosses both paths unchanged: that is why the target looks healthy in either case.

The health check does not depend on parsing, which is why the AWS dashboard looks healthy while not a single packet is being inspected. That is what makes this mistake so hard to find.

Final verification

The workload answering proves traffic is passing, not that it is being inspected. The real proof is in the log: the identified application has to be the real one and not not-applicable.

show rule-hit-count vsys vsys-name vsys1 rule-base security rules all
show log traffic rule equal <REGLA>

web-browsing   gwlb   49880   <origen>
HIBRIDO-IN     allow  gwlb    80    <workload>
                              tcp-fin

Do not use direction equal backward: it returns the oldest entries in the log, and with health checks running every few seconds you will never find your test there.

BGP — The control plane on the hybrid side#

Only applies if you also terminate a VPN against the same architecture. Real output from the edge device, with the router identifier and the pre-existing peer replaced.

One screen with everything#

summary is the command that pays off most: in one view it gives you the state of each session and how many prefixes go in and out of each.

consolaBGP summary on the edge device18 lines download .txt
admin@fw-borde> show routing protocol bgp summary

  ==========
  router id:                     <router-id>
  virtual router:                default
  Local AS:                      65000
  Install BGP routes:            yes
  Graceful Restart:              supported
  Default local preference:      100
  mp-bgp-enable:                 yes
  afi-safi-ipv4-unicast:         yes
  rib-out entries:               current 140, peak 142
    peer PEER-PREEXISTENTE:      AS 65010, Established, IP <inside-peer>
      bgpAfiIpv4/unicast pfx:    Accepted pfx: 1, Advertised pfx: 54
    peer AWS-VGW-1:              AS 64512, Established, IP 169.254.200.1
      bgpAfiIpv4/unicast pfx:    Accepted pfx: 3, Advertised pfx: 43
    peer AWS-VGW-2:              AS 64512, Established, IP 169.254.201.1
      bgpAfiIpv4/unicast pfx:    Accepted pfx: 3, Advertised pfx: 43

What to read here: both AWS peers Established, 3 prefixes accepted from each — the VPC and its two app subnets — and 43 advertised towards each. Both tunnels showing identical numbers is the sign that redundancy is active and not merely configured.

The peer that already existed keeps its own counter: Advertised pfx: 54. If that number changes when you add the new group, you touched a policy that belonged to another peer.

The detail of one session#

consolaThe detail of one BGP session32 lines download .txt
admin@fw-borde> show routing protocol bgp peer

  ==========
  Peer:                          AWS-VGW-2 (id 5)
  virtual router:                default
  Peer router id:                169.254.201.1
  Remote AS:                     64512
  Peer group:                    AWS (id 1)
  Peer status:                   Established, for 20118 seconds
  Passive:                       no
  Multi-hop TTL:                 1
  Remote Address:                169.254.201.1:179
  Local Address:                 169.254.201.2:40411
  Prefix limit:                  100
  Holdtime:                      30 (config 30)
  Keep-Alive interval:           10 (config 10)
  Update messages:               in        4, out        5
  Total messages:                in     2018, out     2318
  Last error:
  Flap counts:                   1, established 1 times
  Nexthop set to self:           no
  ----------
  remove private AS number:      no
  ----------
  Capability:                    Multiprotocol Extensions(1)  value: IPv4 Unicast
  Capability:                    Route Refresh(yes)
  Capability:                    4-Byte AS Number(65)  value: 64512
  ----------
  Prefix counter for:            bgpAfiIpv4 / unicast
  Incoming Prefix:               Accepted 3, Rejected 0, Policy Rej 0, Total 3
  Outgoing Prefix:               43
  Advertised Prefix:             43

Four lines worth looking at that are usually skipped:

What ends up in the forwarding table#

admin@fw-borde> show routing route destination 10.20.0.0/16

flags: A:active, ?:loose, C:connect, H:host, S:static, ~:internal, R:rip, O:ospf, B:bgp,
       Oi:ospf intra-area, Oo:ospf inter-area, O1:ospf ext-type-1, O2:ospf ext-type-2, E:ecmp, M:multicast

VIRTUAL ROUTER: default (id 1)
  ==========
destination            nexthop              metric flags   age    next-AS
10.20.0.0/16           169.254.201.1        100    A?B     20117  64512
10.20.4.0/24           169.254.201.1        100    A?B     20117  64512
10.20.14.0/24          169.254.201.1        100    A?B     20117  64512
total routes shown: 3

All three prefixes come in over both tunnels, but only one stays in the FIB: the best path, with flags A?B — active, incomplete origin, learned by BGP. If that tunnel drops, the other takes over with no intervention. To see all six entries before selection you have to look at the BGP table with show routing protocol bgp loc-rib.

If the firewall advertises and AWS does not accept

When summary shows advertised prefixes but the AWS console reports zero accepted routes, the suspect is export-nexthop: with resolve, routes relearned from a third peer can leave with a next-hop AWS discards. use-self fixes it.

Verification — What it looks like when it is right#

Screenshots from a working implementation. The two that matter most are the edge association and the log with the identified application: the first explains why traffic enters inspection, the second proves it is actually inspected.

In the AWS console#

What the control plane has to report.

Target group · targets tab#

Target group · targets tab

Both targets Healthy, one per zone, both on port 6081. Careful: this screen looks exactly the same with inspection working and without it, because the health check is TCP 80 and does not depend on GENEVE parsing. Green here proves nothing about inspection.

Edge route table · routes#

Edge route table · routes

The two routes that divert inbound traffic to inspection: each app subnet pointing at its zone's endpoint. They are more specific than the VPC local, which is what makes them win.

Edge route table · edge association#

Edge route table · edge association

The piece that is usually missing: the table associated with the virtual private gateway. Without this association the routes in the previous screenshot are never evaluated, and tunnel traffic goes straight into the workload without passing through the firewall.

Gateway Load Balancer endpoints#

Gateway Load Balancer endpoints

One per zone, both Available. Each lives in its zone's gwlbe subnet.

VPN connection · tunnel details#

VPN connection · tunnel details

If you are also connecting on-prem: both tunnels up and with accepted BGP routes. This is the authoritative source for BGP state, above whatever the firewall reports.

Deployment instances#

Deployment instances

Four instances: two VM-Series as firewalls and two workloads, split across the two zones.

On the VM-Series#

What confirms there is inspection and not just transit.

Policies · Security#

Policies · Security

The correct rulebase order. The health check rule goes first; if it ends up below the final deny, the target never reaches healthy. The counters confirm what is being used: hybrid traffic and egress accumulate hits, east-west stays at zero because in this design it does not go through the firewall.

Monitor · Traffic filtered by rule#

Monitor · Traffic filtered by rule

The verification that matters. The Application column says web-browsing, not not-applicable: App-ID is seeing the inner packet, which means the GENEVE decap works. If not-applicable shows up there, traffic is passing but not being inspected.

Network · Interfaces#

Network · Interfaces

The data interface in Layer3, with its virtual router, the inspection zone and the management profile applied. The IP is assigned by DHCP and has to match the target registered in the group.

On the edge device#

Only if you also terminate a VPN against the same architecture.

IPSec Tunnels on the edge device#

IPSec Tunnels on the edge device

Both tunnels towards AWS green, each on its own tunnel interface. The blank rows are production tunnels on the same device, hidden on purpose.

BGP · peers#

BGP · peers

Both AWS peers Established, with the local on .2 and the peer on .1 of each /30. Above, a pre-existing peer with its own uptime: when adding the new peer group, it is worth confirming that the ones already there did not flap.

Mistakes — Ten that give you no message#

All of them turned up while building this architecture. None produces an error that points at the cause.

E1 · GWLB

GENEVE parsing is not enabled by default#

Symptom — Everything looks healthy and not a single packet is inspected. No errors.

Cause — PAN-OS does not decapsulate UDP 6081 until you ask it to. overlay-routing is a different setting.

Fix — request plugins vm_series aws gwlb inspect enable yes

E2 · policy

The final deny stops the target from reaching healthy#

Symptom — Permanent Target.Timeout, even with HTTP enabled in the management profile.

Cause — The health check is intra-zone and goes through the policy, where the final deny matches first.

Fix — An allow rule for TCP 80 from the data subnets, in first position.

E3 · policy

application any with application-default matches nothing#

Symptom — Inspection rules at 0 hits from day one, with no explanation.

Cause — With no concrete applications there is no default port to derive the service from.

Fix — service any, or list explicit applications.

E4 · policy

New rules are born below the final deny#

Symptom — You add a correct rule and it never matches.

Cause — set rulebase security rules appends to the end of the rulebase.

Fix — move ... top or before <cleanup>, and check the order before committing.

E5 · CLI

The zone command path changes with multi-vsys#

Symptom — Invalid syntax when creating the zone, and then cascading not a valid reference failures.

Cause — With multi-vsys off the vsys node does not exist at the root.

Fix — set zone with multi-vsys off; set vsys vsys1 zone with multi-vsys on.

E6 · hybrid

AWS gives the first host of the /30 to the gateway, not the client#

Symptom — IPsec comes up and BGP never leaves Connect.

Cause — In the tunnel's inner /30, .1 belongs to the virtual private gateway and .2 to the customer gateway.

Fix — Read the attributes AWS returns instead of working the addresses out by hand.

E7 · hybrid

A final any/any rule also kills intra-zone traffic#

Symptom — BGP does not establish over the tunnel /30s.

Cause — A from any to any rule is of type universal: it matches intra-zone as well as inter-zone.

Fix — An explicit intra-zone rule for the tunnel's inner IPs, above the deny.

E8 · Terraform

Inline ingress blocks revoke whatever you add by hand#

Symptom — You lose management access after an apply that did not touch that resource.

Cause — With inline blocks, Terraform is authoritative over the entire security group.

Fix — Make the access variable a list and keep the IPs in the code. Check with terraform plan -detailed-exitcode.

E9 · CLI

An 80-column PTY corrupts long commands#

Symptom — Intermittent Invalid syntax on commands that are correct.

Cause — The CLI wraps by inserting a space in the middle: ethernet1/1 arrives as ethern et1/1.

Fix — Enlarge the PTY before the spawn, or descend with edit <node> to shorten each line.

E10 · logs

direction equal backward shows the oldest entries#

Symptom — Your test traffic never appears in the log even though the rule is accumulating hits.

Cause — It returns the oldest entries, and the log is dominated by health checks.

Fix — show log traffic rule equal <NAME>

Limitations — What this design does not solve#

East-west between app subnets is not inspected#

In each app subnet's route table, the VPC local route beats the default towards the endpoint on longest-prefix match. Traffic between workloads goes direct, and the rule that covers it stays at zero hits.

Adding routes more specific than local is not enough: each direction would enter through its own zone's endpoint and be inspected by a different firewall. With a stateful device that does not work, and stickiness does not save it because when source and destination swap, the hash changes. The real options are sending east-west through a single endpoint — losing independence between zones — or accepting that this path is not inspected and removing the rule so as not to promise something that does not happen.

No MSS clamp on the tunnel interfaces#

If you also terminate a VPN against the same device, bear in mind that adjust-tcp-mss does not exist under tunnel units in PAN-OS 11.1: the node only appears on ethernet subinterfaces. Without a clamp, ping goes through and large TCP transfers hang. The alternative is clamping on the ingress LAN interfaces, at the cost of affecting all their traffic.

The workloads depend on inspection in order to provision themselves#

Their internet egress goes through the endpoint, the GWLB and the firewall. While inspection is not working they have no internet: cloud-init cannot reach the metadata service and user_data never runs, with no visible error in the console. That is why the end-to-end test comes at the end of the procedure and not in the middle.


← all notes