> note
VM-Series behind a Gateway Load Balancer on AWS
In this note · 13 sections
- Architecture — What gets built
- Addressing — Five subnets per zone
- Scope — What gets inspected and what doesn't
- Costs — What it costs to keep this running
- Phase 1 — Infrastructure on AWS
- Terraform — The resources that matter
- Code — The complete configuration
- Phase 2 — Configuring the VM-Series
- Phase 3 — Enabling GENEVE parsing
- BGP — The control plane on the hybrid side
- Verification — What it looks like when it is right
- Mistakes — Ten that give you no message
- Limitations — What this design does not solve
Architecture — What gets built#
A Gateway Load Balancer distributes traffic to two VM-Series, one per zone. The firewall neither routes nor rewrites addresses: it receives the packet encapsulated in GENEVE, inspects it, and returns it down the same tunnel.
The per-zone symmetry is deliberate. With cross-zone load balancing disabled, traffic entering through one zone's endpoint is inspected on the firewall in that same zone and returns the way it came. That keeps the session on a single firewall, which is what a stateful device needs, and avoids cross-zone traffic charges.

Addressing from a real implementation. The GWLB node and the VM-Series share the fwdata subnet: that is why the health check arrives from an address on that same network and counts as intra-zone traffic.
Addressing — Five subnets per zone#
Each role needs its own subnet because each one carries a different route table. Mixing them is the quickest way to create a loop between the endpoint and the workload.
| Role | What lives there | AZ a | AZ b |
|---|---|---|---|
| public | NAT gateway | 10.20.0.0/24 | 10.20.10.0/24 |
| mgmt | VM-Series eth0 | 10.20.1.0/24 | 10.20.11.0/24 |
| fwdata | VM-Series eth1 and the GWLB nodes | 10.20.2.0/24 | 10.20.12.0/24 |
| gwlbe | GWLB endpoint | 10.20.3.0/24 | 10.20.13.0/24 |
| app | workloads | 10.20.4.0/24 | 10.20.14.0/24 |
Which route each table carries#
-
app —
0.0.0.0/0to that zone's endpoint. Everything leaving the workload goes to inspection. -
gwlbe —
0.0.0.0/0to the NAT gateway, plus propagation from the virtual private gateway. This is the table that decides where already-inspected traffic goes. -
fwdata — the local route only. GENEVE is born and dies inside the VPC.
-
VGW edge — the app subnets pointing at their endpoint. Without this table, traffic arriving from the tunnel goes straight into the workload without passing through the firewall.
Scope — What gets inspected and what doesn't#
Worth deciding before you write the policy, because not all three paths come for free.
| Path | How it reaches inspection | Status |
|---|---|---|
| Hybrid on-prem ↔ workload | Virtual private gateway edge route table | works |
| Workload egress → internet | 0.0.0.0/0 in the app table towards the endpoint |
works |
| East-west app-a ↔ app-b | Needs routes more specific than the VPC local |
see limitations |
Costs — What it costs to keep this running#
Almost 3 dollars an hour, and four out of every five go to the firewall licence, not to the AWS infrastructure.
us-east-1 rates: infrastructure queried against the AWS pricing API, software against the PAYG listing's dimension table. Data transfer not included.
| Item | USD/hour | × | USD/month |
|---|---|---|---|
VM-Series licence — PAYG, m5.xlarge |
1.1700 | 2 | 1,708.20 |
VM-Series — m5.xlarge compute |
0.1920 | 2 | 280.32 |
| NAT gateway | 0.0450 | 2 | 65.70 |
| Site-to-site VPN connection | 0.0500 | 1 | 36.50 |
t3.micro workloads |
0.0104 | 2 | 15.18 |
| GWLB endpoint | 0.0100 | 2 | 14.60 |
| Public IPv4 in use | 0.0050 | 4 | 14.60 |
| EBS gp3 — 2×60 GB + 2×8 GB | 0.0149 | — | 10.88 |
| Gateway Load Balancer | 0.0125 | 1 | 9.12 |
| GWLB — capacity (minimum LCU) | 0.0040 | 1 | 2.92 |
| Total | 2.9562 | 2,158.03 |
| If it stays on for | Cost | Breakdown |
|---|---|---|
| one hour | USD 2.96 | licence 79 % · compute 13 % · network 8 % |
| one day | USD 70.95 | |
| one month | USD 2,158 |
The licence rules
The PAYG listing's software charge is six times the compute cost of the same instance: 1.17 against 0.192 an hour. It is billed through AWS Marketplace, which is why it never shows up in the pricing API or in estimates that only look at AWS resources.
The price goes by instance size, not family: every xlarge pays 1.17, large pays 0.99, 2xlarge 1.80 and 4xlarge 3.69. Dropping from m5.xlarge to m5.large would save USD 263 a month in licensing — but the GWLB needs at least 10.0.2 and an instance with enough NICs, so check support before you shrink it.
What to switch off#
-
Stopping both firewalls cuts 92 % of the hourly spend — licence plus compute. It is the only lever that moves the needle.
-
With everything off you still run at USD 0.20/hour — around 143 a month: the NAT gateways, the GWLB, the endpoints, the VPN and the public IPv4 addresses all bill by the hour even if not a single packet goes through.
-
For an on-and-off lab,
terraform destroyand rebuild beats stopping: it is 85 resources and the full cycle takes minutes.
Deployment reference: AMI PA-VM-AWS-11.1.15, product code e9yfvyj3uag5uo5j2hjikv74n, listing "VM-Series Next-Gen Virtual Firewall w/Advanced Threat Prevention (PAYG)".
Phase 1 — Infrastructure on AWS#
terraform init
terraform plan # leerlo entero, incluidas las lineas con "-"
terraform apply
# comprobar que el direccionamiento interno del tunel quedo bien
terraform output pa440_tunnel1
terraform output pa440_tunnel2
# antes de dar por buena cualquier regla de security group
terraform plan -detailed-exitcode # 0 = sin deriva
Read the whole plan, not just the summary. Plan: N to add, M to change does not distinguish between "adds a CIDR" and "revokes the ones that were there": on an aws_security_group with inline ingress blocks, an in-place change shows up as the old set with - and the new one with +.
The parameters that admit no variation:
| Resource | Value | Why |
|---|---|---|
| Target group | GENEVE port 6081 |
It is the only protocol a GWLB accepts |
| Target type | ip |
With instance it registers the primary interface, which is the management one |
| Health check | TCP port 80 |
The GWLB cannot query an arbitrary PAN-OS path |
| Stickiness | source_ip_dest_ip_proto |
The firewall does not rewrite the 5-tuple; the flow must always return to the same device |
| Data ENI | source_dest_check = false |
Without this, AWS drops traffic not addressed to the interface |
| Security group | UDP 6081 and TCP 80 from the VPC CIDR |
GENEVE and health check |
Careful with the AMI
Use the public SSM parameter of the listing you are subscribed to. An AMI from another listing boots just the same but bills against a subscription you do not hold. And do not solve it with most_recent: in Palo Alto's catalogue there are newer releases whose AMI carries an earlier date, so sorting by creation date picks the wrong version.
Verification
The endpoints in Available, and the edge route table associated with the virtual private gateway, with the app subnet routes pointing at their endpoints.
Terraform — The resources that matter#
The full deployment is 85 resources. Most of it is plumbing repeated per zone; these are the ones carrying the design's logic.
| Group | Resources | Detail |
|---|---|---|
| Base network | 1 + 10 | VPC and five subnets per zone |
| Route tables | 8 + 11 routes | public, mgmt, fwdata, gwlbe, app and the VGW edge |
| Internet egress | 1 + 2 + 2 | IGW, NAT gateways and their EIPs |
| GWLB | 5 | balancer, listener, target group, endpoint service and 2 endpoints |
| Firewalls | 2 + 4 + 2 | instances, two ENIs each and the management EIPs |
| VPN | 4 | customer gateway, VGW, connection and route propagation |
| Security | 3 | one security group per role |
The GWLB and its target group#
The GWLB lives in the fwdata subnets, alongside the firewalls. The target is registered by IP and not by instance: with instance, AWS would use the primary interface, which here is the management one.
resource "aws_lb" "gwlb" {
load_balancer_type = "gateway"
subnets = [for k, v in aws_subnet.fwdata : v.id]
enable_cross_zone_load_balancing = false
}
resource "aws_lb_target_group" "fw" {
target_type = "ip"
protocol = "GENEVE"
port = 6081
health_check {
protocol = "TCP"
port = 80
}
# El firewall no reescribe la 5-tupla: el flujo debe volver
# siempre a la misma instancia.
stickiness {
type = "source_ip_dest_ip_proto"
enabled = true
}
}
resource "aws_lb_target_group_attachment" "fw" {
for_each = local.subnets
target_id = aws_network_interface.fw_data[each.key].private_ip
availability_zone = local.az[each.key]
port = 6081
}
The edge association#
This is the piece that makes inbound traffic get inspected. A route table associated with the virtual private gateway through gateway_id, not with a subnet, and carrying routes more specific than the VPC local.
resource "aws_route" "vgw_edge_to_app" {
for_each = local.subnets
route_table_id = aws_route_table.vgw_edge.id
destination_cidr_block = each.value.app # 10.20.4.0/24 y 10.20.14.0/24
vpc_endpoint_id = aws_vpc_endpoint.gwlbe[each.key].id
}
resource "aws_route_table_association" "vgw_edge" {
gateway_id = aws_vpn_gateway.vgw.id # asociacion de BORDE
route_table_id = aws_route_table.vgw_edge.id
}
The firewall interfaces#
Two NICs: management on device_index 0 and data on 1, which is the one the VM-Series sees as ethernet1/1. source_dest_check goes to false only on the data one.
resource "aws_network_interface" "fw_data" {
for_each = local.subnets
subnet_id = aws_subnet.fwdata[each.key].id
security_groups = [aws_security_group.fwdata.id]
source_dest_check = false
}
resource "aws_instance" "fw" {
for_each = local.subnets
instance_type = "m5.xlarge"
network_interface { network_interface_id = ...fw_mgmt[each.key].id, device_index = 0 }
network_interface { network_interface_id = ...fw_data[each.key].id, device_index = 1 }
user_data = <<-EOT
type=dhcp-client
hostname=fw-gwlb-${each.key}
dhcp-accept-server-hostname=no
dhcp-accept-server-domain=no
EOT
}
Selecting the AMI#
Through the public SSM parameter, never with most_recent: in Palo Alto's catalogue there are newer releases whose AMI carries an earlier date, so sorting by creation date picks the wrong version.
data "aws_ssm_parameter" "panos" {
name = "/aws/service/marketplace/prod-<listing>/pan-os-<version>"
}
The three security groups#
| Group | Ingress | From |
|---|---|---|
fwdata |
UDP 6081, TCP 80, ICMP | VPC CIDR |
mgmt |
TCP 22, TCP 443, ICMP | management list + on-prem prefixes |
app |
everything | VPC CIDR + on-prem prefixes |
The management one takes a list of CIDRs, not a string. With a single value the only way out is adding the real IP by hand in the console, and since the ingress blocks are inline, the next apply silently revokes it.
Code — The complete configuration#
The seven files exactly as they are applied, with public addresses and account identifiers replaced by placeholders. This is the whole deployment: no external modules, no shared state.
| File | Contents |
|---|---|
00-variables.tf |
Provider, variables and the per-zone subnet map. |
01-vpc.tf |
VPC, the ten subnets, IGW, NAT gateways and the three security groups. |
02-gwlb.tf |
Balancer, target group, listener, endpoint service and endpoints. |
03-firewalls.tf |
AMI via SSM, the ENIs, the VM-Series and the workloads. |
04-vpn.tf |
Customer gateway, VGW and the connection with its two tunnels. |
05-routing.tf |
The six route tables, including the edge one associated with the VGW. |
06-outputs.tf |
Outputs, among them the internal addressing of the tunnels. |
terraform {
required_version = ">= 1.5"
required_providers {
aws = { source = "hashicorp/aws", version = "~> 5.0" }
random = { source = "hashicorp/random", version = "~> 3.6" }
}
}
provider "aws" {
region = var.region
default_tags {
tags = {
Project = "kf-gwlb-lab"
ManagedBy = "terraform"
Owner = "csegovia"
}
}
}
variable "region" {
type = string
default = "us-east-1"
}
variable "vpc_cidr" {
description = "Verificado libre contra la tabla de rutas del PA-410."
type = string
default = "10.20.0.0/16"
}
variable "azs" {
type = list(string)
default = ["us-east-1a", "us-east-1b"]
}
# Cinco subnets por AZ. Cada una tiene un rol distinto en el path de GENEVE.
locals {
az = { a = var.azs[0], b = var.azs[1] }
subnets = {
a = {
public = "10.20.0.0/24" # NAT GW
mgmt = "10.20.1.0/24" # eth0 del VM-Series
fwdata = "10.20.2.0/24" # eth1, GENEVE, target del GWLB
gwlbe = "10.20.3.0/24" # endpoints del GWLB
app = "10.20.4.0/24" # workloads
}
b = {
public = "10.20.10.0/24"
mgmt = "10.20.11.0/24"
fwdata = "10.20.12.0/24"
gwlbe = "10.20.13.0/24"
app = "10.20.14.0/24"
}
}
# Rutas app -> GWLBE para cada prefijo del lab, por AZ.
app_lab_routes = {
for p in setproduct(keys(local.subnets), var.lab_cidrs) :
"${p[0]}|${p[1]}" => { az = p[0], cidr = p[1] }
}
}
# --- VPN / BGP ---------------------------------------------------------------
variable "cgw_public_ip" {
description = "IP publica del PA-410 (ethernet1/1)."
type = string
default = "<IP-PUBLICA-CGW>"
}
variable "cgw_asn" {
description = "AS local del PA-410, ya existente en el VR default."
type = number
default = 65000
}
variable "vgw_asn" {
type = number
default = 64512
}
variable "tunnel1_inside_cidr" {
description = "169.254.100.0/30 NO se puede: la usa el peer HEX-CASA."
type = string
default = "169.254.200.0/30"
}
variable "tunnel2_inside_cidr" {
type = string
default = "169.254.201.0/30"
}
variable "lab_cidrs" {
description = "Prefijos del home lab que se alcanzan por el tunel y se inspeccionan."
type = list(string)
default = [
"192.168.100.0/24",
"192.168.106.0/24",
"10.1.0.0/16",
"10.142.0.0/16",
]
}
# --- Firewalls ---------------------------------------------------------------
# Parametro publico de SSM que publica AWS Marketplace. Resuelve al AMI
# correcto por region sin hardcodear IDs.
#
# prod-<ID-DEL-LISTING> = listing "VM-Series Next-Gen Virtual Firewall
# w/Advanced Threat Prevention (PAYG)"
#
# OJO: usar el AMI de OTRO listing hace que la instancia facture contra una
# suscripcion que no tiene. El product code va atado al listing.
variable "panos_ssm_parameter" {
type = string
default = "/aws/service/marketplace/prod-<ID-DEL-LISTING>/pan-os-11.1.15"
}
# Override manual. Si se setea, gana sobre el parametro SSM.
variable "panos_ami_id" {
type = string
default = ""
}
variable "panos_instance_type" {
description = "GWLB necesita minimo 10.0.2. m5.xlarge soporta las NICs necesarias."
type = string
default = "m5.xlarge"
}
variable "key_pair_name" {
type = string
}
# Lista, no string: la IP publica de salida del equipo desde el que se
# administra NO siempre es la de ethernet1/1 del PA-410. Con un solo valor la
# unica salida era agregar la IP real a mano en la consola, y como los bloques
# ingress son inline, el siguiente terraform apply la revocaba en silencio.
variable "admin_cidrs" {
description = "CIDRs con acceso de gestion a los VM-Series (SSH, HTTPS, ICMP)."
type = list(string)
default = ["<IP-PUBLICA-CGW>/32"]
}
resource "aws_vpc" "lab" {
cidr_block = var.vpc_cidr
enable_dns_support = true
enable_dns_hostnames = true
tags = { Name = "kf-gwlb-vpc" }
}
resource "aws_internet_gateway" "igw" {
vpc_id = aws_vpc.lab.id
tags = { Name = "kf-gwlb-igw" }
}
# --- Subnets -----------------------------------------------------------------
resource "aws_subnet" "public" {
for_each = local.subnets
vpc_id = aws_vpc.lab.id
cidr_block = each.value.public
availability_zone = local.az[each.key]
tags = { Name = "kf-public-${each.key}" }
}
resource "aws_subnet" "mgmt" {
for_each = local.subnets
vpc_id = aws_vpc.lab.id
cidr_block = each.value.mgmt
availability_zone = local.az[each.key]
tags = { Name = "kf-mgmt-${each.key}" }
}
resource "aws_subnet" "fwdata" {
for_each = local.subnets
vpc_id = aws_vpc.lab.id
cidr_block = each.value.fwdata
availability_zone = local.az[each.key]
tags = { Name = "kf-fwdata-${each.key}" }
}
resource "aws_subnet" "gwlbe" {
for_each = local.subnets
vpc_id = aws_vpc.lab.id
cidr_block = each.value.gwlbe
availability_zone = local.az[each.key]
tags = { Name = "kf-gwlbe-${each.key}" }
}
resource "aws_subnet" "app" {
for_each = local.subnets
vpc_id = aws_vpc.lab.id
cidr_block = each.value.app
availability_zone = local.az[each.key]
tags = { Name = "kf-app-${each.key}" }
}
# --- NAT Gateway por AZ ------------------------------------------------------
# Uno por AZ. Con uno solo compartido el retorno cruza AZ y el path se vuelve
# dificil de explicar. Son US$0.045/hr cada uno: el segundo item mas caro
# despues de los firewalls.
resource "aws_eip" "nat" {
for_each = local.subnets
domain = "vpc"
tags = { Name = "kf-eip-nat-${each.key}" }
depends_on = [aws_internet_gateway.igw]
}
resource "aws_nat_gateway" "nat" {
for_each = local.subnets
allocation_id = aws_eip.nat[each.key].id
subnet_id = aws_subnet.public[each.key].id
tags = { Name = "kf-nat-${each.key}" }
depends_on = [aws_internet_gateway.igw]
}
# --- Security groups ---------------------------------------------------------
resource "aws_security_group" "mgmt" {
name = "kf-sg-mgmt"
description = "Gestion de los VM-Series"
vpc_id = aws_vpc.lab.id
ingress {
description = "HTTPS GUI"
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = concat(var.admin_cidrs, var.lab_cidrs)
}
ingress {
description = "SSH"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = concat(var.admin_cidrs, var.lab_cidrs)
}
ingress {
description = "ICMP"
from_port = -1
to_port = -1
protocol = "icmp"
cidr_blocks = concat(var.admin_cidrs, var.lab_cidrs)
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = { Name = "kf-sg-mgmt" }
}
# GOTCHA GWLB #1: si falta el 6081 el target queda unhealthy y AWS no dice
# por que. Es el error mas comun de toda la integracion.
resource "aws_security_group" "fwdata" {
name = "kf-sg-fwdata"
description = "Interfaz de datos: GENEVE + health check del GWLB"
vpc_id = aws_vpc.lab.id
ingress {
description = "GENEVE desde el GWLB"
from_port = 6081
to_port = 6081
protocol = "udp"
cidr_blocks = [var.vpc_cidr]
}
ingress {
description = "Health check del target group"
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = [var.vpc_cidr]
}
ingress {
from_port = -1
to_port = -1
protocol = "icmp"
cidr_blocks = [var.vpc_cidr]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = { Name = "kf-sg-fwdata" }
}
resource "aws_security_group" "app" {
name = "kf-sg-app"
description = "Workloads"
vpc_id = aws_vpc.lab.id
ingress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = concat([var.vpc_cidr], var.lab_cidrs)
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = { Name = "kf-sg-app" }
}
# =============================================================================
# Gateway Load Balancer
# El GWLB encapsula en GENEVE (UDP 6081) y reparte hacia los VM-Series.
# Los firewalls trabajan como bump-in-the-wire: no rutean, no hacen NAT.
# =============================================================================
resource "aws_lb" "gwlb" {
name = "kf-gwlb"
load_balancer_type = "gateway"
subnets = [for k, v in aws_subnet.fwdata : v.id]
# Probado en true (2026-08-17) para descartar que fuera la causa de que el
# GWLB no entregue al target: NO cambio nada, el trafico sigue muriendo entre
# el endpoint y el target. Se vuelve a false, que es el diseno buscado
# (inspeccion local por AZ) y evita cargos de trafico cross-AZ.
enable_cross_zone_load_balancing = false
tags = { Name = "kf-gwlb" }
}
# GOTCHA GWLB #2: el health check no puede pegarle a un path cualquiera del
# PAN-OS. Se usa TCP:80 y hay que habilitar HTTP en el interface management
# profile de la interfaz de datos, o el target nunca pasa a healthy.
resource "aws_lb_target_group" "fw" {
name = "kf-gwlb-tg"
target_type = "ip"
protocol = "GENEVE"
port = 6081
vpc_id = aws_vpc.lab.id
health_check {
protocol = "TCP"
port = 80
interval = 10
healthy_threshold = 3
unhealthy_threshold = 3
}
# El firewall no reescribe la 5-tupla, asi que el flujo debe volver siempre
# a la misma instancia.
stickiness {
type = "source_ip_dest_ip_proto"
enabled = true
}
tags = { Name = "kf-gwlb-tg" }
}
# Se registra la IP de la ENI de datos, no el instance-id: con target_type
# "instance" el GWLB usaria la interfaz primaria, que aca es la de gestion.
resource "aws_lb_target_group_attachment" "fw" {
for_each = local.subnets
target_group_arn = aws_lb_target_group.fw.arn
target_id = aws_network_interface.fw_data[each.key].private_ip
availability_zone = local.az[each.key]
port = 6081
}
resource "aws_lb_listener" "gwlb" {
load_balancer_arn = aws_lb.gwlb.arn
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.fw.arn
}
}
# --- Endpoint service + endpoints por AZ -------------------------------------
resource "aws_vpc_endpoint_service" "gwlb" {
acceptance_required = false
gateway_load_balancer_arns = [aws_lb.gwlb.arn]
tags = { Name = "kf-gwlb-endpoint-service" }
}
resource "aws_vpc_endpoint" "gwlbe" {
for_each = local.subnets
vpc_id = aws_vpc.lab.id
service_name = aws_vpc_endpoint_service.gwlb.service_name
vpc_endpoint_type = "GatewayLoadBalancer"
subnet_ids = [aws_subnet.gwlbe[each.key].id]
tags = { Name = "kf-gwlbe-${each.key}" }
}
# No se usa data "aws_ami" con most_recent: en la lista de AMIs de Palo Alto
# la 11.1.15 es una release MAS nueva que la 11.1.6-h35, pero su AMI tiene
# fecha ANTERIOR. Ordenar por CreationDate elige la version equivocada.
data "aws_ssm_parameter" "panos" {
count = var.panos_ami_id == "" ? 1 : 0
name = var.panos_ssm_parameter
}
locals {
panos_ami = var.panos_ami_id != "" ? var.panos_ami_id : data.aws_ssm_parameter.panos[0].value
}
# Dos NICs: gestion y datos. Sin overlay routing, sin NAT, sin interfaz
# untrust. Todo el trafico entra y sale encapsulado en GENEVE por eth1.
resource "aws_network_interface" "fw_mgmt" {
for_each = local.subnets
subnet_id = aws_subnet.mgmt[each.key].id
security_groups = [aws_security_group.mgmt.id]
tags = { Name = "kf-fw-${each.key}-mgmt" }
}
resource "aws_network_interface" "fw_data" {
for_each = local.subnets
subnet_id = aws_subnet.fwdata[each.key].id
security_groups = [aws_security_group.fwdata.id]
source_dest_check = false
tags = { Name = "kf-fw-${each.key}-data" }
}
resource "aws_eip" "fw_mgmt" {
for_each = local.subnets
domain = "vpc"
network_interface = aws_network_interface.fw_mgmt[each.key].id
tags = { Name = "kf-eip-fw-${each.key}-mgmt" }
depends_on = [aws_internet_gateway.igw]
}
resource "aws_instance" "fw" {
for_each = local.subnets
ami = local.panos_ami
instance_type = var.panos_instance_type
key_name = var.key_pair_name
network_interface {
network_interface_id = aws_network_interface.fw_mgmt[each.key].id
device_index = 0
}
network_interface {
network_interface_id = aws_network_interface.fw_data[each.key].id
device_index = 1
}
user_data = <<-EOT
type=dhcp-client
hostname=kf-fw-gwlb-${each.key}
dns-primary=169.254.169.253
dns-secondary=8.8.8.8
dhcp-accept-server-hostname=no
dhcp-accept-server-domain=no
EOT
root_block_device {
volume_size = 60
volume_type = "gp3"
}
tags = { Name = "kf-fw-gwlb-${each.key}" }
}
# --- Workloads ---------------------------------------------------------------
data "aws_ami" "al2023" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = ["al2023-ami-2023.*-x86_64"]
}
}
# Sin IP publica. Se prueban desde el home lab por el tunel, que es
# justamente el escenario que se quiere demostrar.
resource "aws_instance" "app" {
for_each = local.subnets
ami = data.aws_ami.al2023.id
instance_type = "t3.micro"
subnet_id = aws_subnet.app[each.key].id
private_ip = cidrhost(local.subnets[each.key].app, 10)
vpc_security_group_ids = [aws_security_group.app.id]
key_name = var.key_pair_name
user_data = <<-EOT
#!/bin/bash
dnf install -y nginx
echo "kf-gwlb-lab · stack ${each.key} · $(hostname)" > /usr/share/nginx/html/index.html
systemctl enable --now nginx
EOT
tags = { Name = "kf-app-${each.key}" }
}
# PSK: 8-64 chars, alfanumerico mas . y _ , no puede empezar con 0.
resource "random_password" "psk" {
count = 2
length = 32
special = true
override_special = "._"
numeric = true
upper = true
lower = true
}
resource "aws_customer_gateway" "cgw" {
bgp_asn = var.cgw_asn
ip_address = var.cgw_public_ip
type = "ipsec.1"
tags = { Name = "kf-cgw-pa410" }
}
resource "aws_vpn_gateway" "vgw" {
vpc_id = aws_vpc.lab.id
amazon_side_asn = var.vgw_asn
tags = { Name = "kf-vgw" }
}
resource "aws_vpn_connection" "s2s" {
customer_gateway_id = aws_customer_gateway.cgw.id
vpn_gateway_id = aws_vpn_gateway.vgw.id
type = "ipsec.1"
static_routes_only = false # BGP
# --- Tunel 1 ---------------------------------------------------------------
tunnel1_inside_cidr = var.tunnel1_inside_cidr
tunnel1_preshared_key = random_password.psk[0].result
tunnel1_ike_versions = ["ikev2"]
tunnel1_phase1_encryption_algorithms = ["AES256"]
tunnel1_phase1_integrity_algorithms = ["SHA2-256"]
tunnel1_phase1_dh_group_numbers = [14]
tunnel1_phase1_lifetime_seconds = 28800
tunnel1_phase2_encryption_algorithms = ["AES256"]
tunnel1_phase2_integrity_algorithms = ["SHA2-256"]
tunnel1_phase2_dh_group_numbers = [14]
tunnel1_phase2_lifetime_seconds = 3600
tunnel1_startup_action = "start" # AWS inicia; tenemos IP fija
tunnel1_dpd_timeout_action = "restart"
tunnel1_dpd_timeout_seconds = 30
# --- Tunel 2 ---------------------------------------------------------------
tunnel2_inside_cidr = var.tunnel2_inside_cidr
tunnel2_preshared_key = random_password.psk[1].result
tunnel2_ike_versions = ["ikev2"]
tunnel2_phase1_encryption_algorithms = ["AES256"]
tunnel2_phase1_integrity_algorithms = ["SHA2-256"]
tunnel2_phase1_dh_group_numbers = [14]
tunnel2_phase1_lifetime_seconds = 28800
tunnel2_phase2_encryption_algorithms = ["AES256"]
tunnel2_phase2_integrity_algorithms = ["SHA2-256"]
tunnel2_phase2_dh_group_numbers = [14]
tunnel2_phase2_lifetime_seconds = 3600
tunnel2_startup_action = "start"
tunnel2_dpd_timeout_action = "restart"
tunnel2_dpd_timeout_seconds = 30
tags = { Name = "kf-vpn-pa410" }
}
# =============================================================================
# ROUTING
#
# El path completo de un flujo de egress:
# app -> GWLBE (misma AZ) -> GWLB -> firewall -> GWLB -> GWLBE -> NAT GW -> IGW
#
# El path de un flujo hibrido entrante:
# home lab -> VGW -> [edge route table] -> GWLBE -> firewall -> GWLBE -> app
# =============================================================================
# --- Public: NAT GW + IGW ----------------------------------------------------
# El retorno del NAT GW tiene que volver por el firewall, si no el flujo queda
# asimetrico y solo se inspecciona la ida.
resource "aws_route_table" "public" {
for_each = local.subnets
vpc_id = aws_vpc.lab.id
tags = { Name = "kf-rt-public-${each.key}" }
}
resource "aws_route" "public_default" {
for_each = aws_route_table.public
route_table_id = each.value.id
destination_cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.igw.id
}
resource "aws_route" "public_return_to_fw" {
for_each = local.subnets
route_table_id = aws_route_table.public[each.key].id
destination_cidr_block = each.value.app
vpc_endpoint_id = aws_vpc_endpoint.gwlbe[each.key].id
}
resource "aws_route_table_association" "public" {
for_each = aws_subnet.public
subnet_id = each.value.id
route_table_id = aws_route_table.public[each.key].id
}
# --- Mgmt --------------------------------------------------------------------
# Trafico de gestion, no se inspecciona a proposito: si el firewall se cae,
# igual se puede llegar a administrarlo.
resource "aws_route_table" "mgmt" {
vpc_id = aws_vpc.lab.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.igw.id
}
tags = { Name = "kf-rt-mgmt" }
}
resource "aws_route_table_association" "mgmt" {
for_each = aws_subnet.mgmt
subnet_id = each.value.id
route_table_id = aws_route_table.mgmt.id
}
# --- Fwdata ------------------------------------------------------------------
# Solo la ruta local. El GENEVE nace y muere dentro de la VPC.
resource "aws_route_table" "fwdata" {
vpc_id = aws_vpc.lab.id
tags = { Name = "kf-rt-fwdata" }
}
resource "aws_route_table_association" "fwdata" {
for_each = aws_subnet.fwdata
subnet_id = each.value.id
route_table_id = aws_route_table.fwdata.id
}
# --- GWLBE: salida despues de inspeccion -------------------------------------
# Aca cae el trafico ya inspeccionado que devuelve el firewall.
resource "aws_route_table" "gwlbe" {
for_each = local.subnets
vpc_id = aws_vpc.lab.id
route {
cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.nat[each.key].id
}
tags = { Name = "kf-rt-gwlbe-${each.key}" }
}
resource "aws_route_table_association" "gwlbe" {
for_each = aws_subnet.gwlbe
subnet_id = each.value.id
route_table_id = aws_route_table.gwlbe[each.key].id
}
# Los prefijos del lab llegan por BGP. GOTCHA #1 del post anterior sigue
# vigente: la propagacion viene apagada por defecto.
resource "aws_vpn_gateway_route_propagation" "gwlbe" {
for_each = aws_route_table.gwlbe
vpn_gateway_id = aws_vpn_gateway.vgw.id
route_table_id = each.value.id
}
resource "aws_vpn_gateway_route_propagation" "mgmt" {
vpn_gateway_id = aws_vpn_gateway.vgw.id
route_table_id = aws_route_table.mgmt.id
}
# --- App: todo sale por el GWLBE ---------------------------------------------
# Sin propagacion del VGW aca: si se propagaran los prefijos del lab, el
# trafico iria directo al VGW sin pasar por el firewall.
resource "aws_route_table" "app" {
for_each = local.subnets
vpc_id = aws_vpc.lab.id
tags = { Name = "kf-rt-app-${each.key}" }
}
resource "aws_route" "app_default" {
for_each = local.subnets
route_table_id = aws_route_table.app[each.key].id
destination_cidr_block = "0.0.0.0/0"
vpc_endpoint_id = aws_vpc_endpoint.gwlbe[each.key].id
}
resource "aws_route" "app_to_lab" {
for_each = local.app_lab_routes
route_table_id = aws_route_table.app[each.value.az].id
destination_cidr_block = each.value.cidr
vpc_endpoint_id = aws_vpc_endpoint.gwlbe[each.value.az].id
}
resource "aws_route_table_association" "app" {
for_each = aws_subnet.app
subnet_id = each.value.id
route_table_id = aws_route_table.app[each.key].id
}
# --- Edge route table del VGW ------------------------------------------------
# Esto es lo que hace que el trafico hibrido se inspeccione. Sin esta
# asociacion el trafico del tunel entra directo a las subnets app.
resource "aws_route_table" "vgw_edge" {
vpc_id = aws_vpc.lab.id
tags = { Name = "kf-rt-vgw-edge" }
}
resource "aws_route" "vgw_edge_to_app" {
for_each = local.subnets
route_table_id = aws_route_table.vgw_edge.id
destination_cidr_block = each.value.app
vpc_endpoint_id = aws_vpc_endpoint.gwlbe[each.key].id
}
resource "aws_route_table_association" "vgw_edge" {
gateway_id = aws_vpn_gateway.vgw.id
route_table_id = aws_route_table.vgw_edge.id
}
output "firewalls" {
description = "EIP de gestion de cada VM-Series y la IP de datos registrada en el target group."
value = {
for k in keys(local.subnets) : k => {
mgmt_eip = aws_eip.fw_mgmt[k].public_ip
data_ip = aws_network_interface.fw_data[k].private_ip
}
}
}
output "app_servers" {
description = "Workloads. Sin IP publica: se prueban desde el home lab por el tunel."
value = { for k, v in aws_instance.app : k => v.private_ip }
}
output "gwlb_endpoints" {
value = { for k, v in aws_vpc_endpoint.gwlbe : k => v.id }
}
output "target_group_arn" {
description = "Para vigilar el estado con: aws elbv2 describe-target-health --target-group-arn ..."
value = aws_lb_target_group.fw.arn
}
# --- Datos para armar el tunel en el PA-410 ---------------------------------
# inside_local / inside_remote salen de los atributos que devuelve AWS, no de
# cidrhost(). AWS asigna el PRIMER host del /30 al VGW y el segundo al customer
# gateway; calcularlo a mano invita a dejarlo al reves, que es justo lo que
# paso antes: el PA quedaba con la .1 (que es del VGW) y peereaba contra la .2.
output "pa440_tunnel1" {
value = {
aws_peer_ip = aws_vpn_connection.s2s.tunnel1_address
inside_local = aws_vpn_connection.s2s.tunnel1_cgw_inside_address
inside_remote = aws_vpn_connection.s2s.tunnel1_vgw_inside_address
bgp_peer_asn = var.vgw_asn
}
}
output "pa440_tunnel2" {
value = {
aws_peer_ip = aws_vpn_connection.s2s.tunnel2_address
inside_local = aws_vpn_connection.s2s.tunnel2_cgw_inside_address
inside_remote = aws_vpn_connection.s2s.tunnel2_vgw_inside_address
bgp_peer_asn = var.vgw_asn
}
}
# CREDENCIAL VIVA. Nunca en el blog ni en screenshots.
output "psk_tunnel1" {
value = aws_vpn_connection.s2s.tunnel1_preshared_key
sensitive = true
}
output "psk_tunnel2" {
value = aws_vpn_connection.s2s.tunnel2_preshared_key
sensitive = true
}
Phase 2 — Configuring the VM-Series#
Identical on both devices: the data interface takes its IP by DHCP, so there is nothing to customise per zone.
configure
set network profiles interface-management-profile GWLB-HC http yes
set network profiles interface-management-profile GWLB-HC ping yes
set network interface ethernet ethernet1/1 layer3 dhcp-client enable yes
set network interface ethernet ethernet1/1 layer3 dhcp-client create-default-route no
set network interface ethernet ethernet1/1 layer3 interface-management-profile GWLB-HC
set network virtual-router default interface ethernet1/1
set zone gwlb network layer3 ethernet1/1
The http yes in the management profile is what makes the firewall answer the health check. The zone command depends on multi-vsys: with off it is set zone; with on it is set vsys vsys1 zone. Check it with show system info | match multi-vsys before writing the line: if the zone is not created, every rule that references it fails afterwards.
Policy: a single zone, everything intra-zone#
Traffic comes in and goes out through the same interface, so source and destination land in the same zone. The policy has to allow that explicitly.
The health check needs a rule of its own
Enabling HTTP in the management profile is necessary but not sufficient. The health check leaves a GWLB node on the same subnet towards the firewall's data IP: it is intra-zone traffic and it goes through the policy. Any any/any cleanup rule kills it first, and the target stays unhealthy forever without AWS explaining a thing.
set service TCP-80 protocol tcp port 80
set address-group GWLB-NODES static [ FWDATA-A FWDATA-B ]
set rulebase security rules GWLB-HEALTHCHECK from gwlb
set rulebase security rules GWLB-HEALTHCHECK to gwlb
set rulebase security rules GWLB-HEALTHCHECK source GWLB-NODES
set rulebase security rules GWLB-HEALTHCHECK destination any
set rulebase security rules GWLB-HEALTHCHECK application any
set rulebase security rules GWLB-HEALTHCHECK service TCP-80
set rulebase security rules GWLB-HEALTHCHECK action allow
move rulebase security rules GWLB-HEALTHCHECK top
The move is not cosmetic: new rules are appended at the end, which is below the cleanup, where they never match.
For the inspection rules use service any. The combination of application any with service application-default matches nothing: with no concrete applications in the rule there is no default port to derive the service from.
Verification
Both targets healthy. It takes about 30 seconds with the default interval.
aws elbv2 describe-target-health --region <region> \
--target-group-arn <arn>
Phase 3 — Enabling GENEVE parsing#
This is the step that makes the design work, and the only one that produces no error at all if you skip it.
PAN-OS does not decapsulate GENEVE until you ask it to. Without that it receives the UDP 6081 packets, never sees the inner addresses, inspects nothing, and never returns them to the tunnel.
On both devices
request plugins vm_series aws gwlb inspect enable yes
request plugins vm_series aws gwlb overlay-routing enable no
show plugins vm_series aws gwlb
GWLB enabled has to say True. The message vpc-endpoint association not found is normal in bump-in-the-wire: that association belongs to overlay routing mode, which is deliberately not used here.
The health check does not depend on parsing, which is why the AWS dashboard looks healthy while not a single packet is being inspected. That is what makes this mistake so hard to find.
Final verification
The workload answering proves traffic is passing, not that it is being inspected. The real proof is in the log: the identified application has to be the real one and not not-applicable.
show rule-hit-count vsys vsys-name vsys1 rule-base security rules all
show log traffic rule equal <REGLA>
web-browsing gwlb 49880 <origen>
HIBRIDO-IN allow gwlb 80 <workload>
tcp-fin
Do not use direction equal backward: it returns the oldest entries in the log, and with health checks running every few seconds you will never find your test there.
BGP — The control plane on the hybrid side#
Only applies if you also terminate a VPN against the same architecture. Real output from the edge device, with the router identifier and the pre-existing peer replaced.
One screen with everything#
summary is the command that pays off most: in one view it gives you the state of each session and how many prefixes go in and out of each.
admin@fw-borde> show routing protocol bgp summary
==========
router id: <router-id>
virtual router: default
Local AS: 65000
Install BGP routes: yes
Graceful Restart: supported
Default local preference: 100
mp-bgp-enable: yes
afi-safi-ipv4-unicast: yes
rib-out entries: current 140, peak 142
peer PEER-PREEXISTENTE: AS 65010, Established, IP <inside-peer>
bgpAfiIpv4/unicast pfx: Accepted pfx: 1, Advertised pfx: 54
peer AWS-VGW-1: AS 64512, Established, IP 169.254.200.1
bgpAfiIpv4/unicast pfx: Accepted pfx: 3, Advertised pfx: 43
peer AWS-VGW-2: AS 64512, Established, IP 169.254.201.1
bgpAfiIpv4/unicast pfx: Accepted pfx: 3, Advertised pfx: 43
What to read here: both AWS peers Established, 3 prefixes accepted from each — the VPC and its two app subnets — and 43 advertised towards each. Both tunnels showing identical numbers is the sign that redundancy is active and not merely configured.
The peer that already existed keeps its own counter: Advertised pfx: 54. If that number changes when you add the new group, you touched a policy that belonged to another peer.
The detail of one session#
admin@fw-borde> show routing protocol bgp peer
==========
Peer: AWS-VGW-2 (id 5)
virtual router: default
Peer router id: 169.254.201.1
Remote AS: 64512
Peer group: AWS (id 1)
Peer status: Established, for 20118 seconds
Passive: no
Multi-hop TTL: 1
Remote Address: 169.254.201.1:179
Local Address: 169.254.201.2:40411
Prefix limit: 100
Holdtime: 30 (config 30)
Keep-Alive interval: 10 (config 10)
Update messages: in 4, out 5
Total messages: in 2018, out 2318
Last error:
Flap counts: 1, established 1 times
Nexthop set to self: no
----------
remove private AS number: no
----------
Capability: Multiprotocol Extensions(1) value: IPv4 Unicast
Capability: Route Refresh(yes)
Capability: 4-Byte AS Number(65) value: 64512
----------
Prefix counter for: bgpAfiIpv4 / unicast
Incoming Prefix: Accepted 3, Rejected 0, Policy Rej 0, Total 3
Outgoing Prefix: 43
Advertised Prefix: 43
Four lines worth looking at that are usually skipped:
-
Local Address: 169.254.201.2againstRemote Address: 169.254.201.1— the local one is the.2. Inverted, the tunnel comes up just the same and the session never establishes. -
remove private AS number: no— with every ASN private, stripping them from the AS_PATH removes loop detection. -
Flap counts: 1, established 1 times— it established once and never dropped again. A rising counter points to MTU or tunnel instability. -
Accepted 3, Rejected 0, Policy Rej 0— separates "no routes arrive" from "they arrive and an import policy discards them".
What ends up in the forwarding table#
admin@fw-borde> show routing route destination 10.20.0.0/16
flags: A:active, ?:loose, C:connect, H:host, S:static, ~:internal, R:rip, O:ospf, B:bgp,
Oi:ospf intra-area, Oo:ospf inter-area, O1:ospf ext-type-1, O2:ospf ext-type-2, E:ecmp, M:multicast
VIRTUAL ROUTER: default (id 1)
==========
destination nexthop metric flags age next-AS
10.20.0.0/16 169.254.201.1 100 A?B 20117 64512
10.20.4.0/24 169.254.201.1 100 A?B 20117 64512
10.20.14.0/24 169.254.201.1 100 A?B 20117 64512
total routes shown: 3
All three prefixes come in over both tunnels, but only one stays in the FIB: the best path, with flags A?B — active, incomplete origin, learned by BGP. If that tunnel drops, the other takes over with no intervention. To see all six entries before selection you have to look at the BGP table with show routing protocol bgp loc-rib.
If the firewall advertises and AWS does not accept
When summary shows advertised prefixes but the AWS console reports zero accepted routes, the suspect is export-nexthop: with resolve, routes relearned from a third peer can leave with a next-hop AWS discards. use-self fixes it.
Verification — What it looks like when it is right#
Screenshots from a working implementation. The two that matter most are the edge association and the log with the identified application: the first explains why traffic enters inspection, the second proves it is actually inspected.
In the AWS console#
What the control plane has to report.
Target group · targets tab#

Both targets Healthy, one per zone, both on port 6081. Careful: this screen looks exactly the same with inspection working and without it, because the health check is TCP 80 and does not depend on GENEVE parsing. Green here proves nothing about inspection.
Edge route table · routes#

The two routes that divert inbound traffic to inspection: each app subnet pointing at its zone's endpoint. They are more specific than the VPC local, which is what makes them win.
Edge route table · edge association#

The piece that is usually missing: the table associated with the virtual private gateway. Without this association the routes in the previous screenshot are never evaluated, and tunnel traffic goes straight into the workload without passing through the firewall.
Gateway Load Balancer endpoints#

One per zone, both Available. Each lives in its zone's gwlbe subnet.
VPN connection · tunnel details#

If you are also connecting on-prem: both tunnels up and with accepted BGP routes. This is the authoritative source for BGP state, above whatever the firewall reports.
Deployment instances#

Four instances: two VM-Series as firewalls and two workloads, split across the two zones.
On the VM-Series#
What confirms there is inspection and not just transit.
Policies · Security#

The correct rulebase order. The health check rule goes first; if it ends up below the final deny, the target never reaches healthy. The counters confirm what is being used: hybrid traffic and egress accumulate hits, east-west stays at zero because in this design it does not go through the firewall.
Monitor · Traffic filtered by rule#

The verification that matters. The Application column says web-browsing, not not-applicable: App-ID is seeing the inner packet, which means the GENEVE decap works. If not-applicable shows up there, traffic is passing but not being inspected.
Network · Interfaces#

The data interface in Layer3, with its virtual router, the inspection zone and the management profile applied. The IP is assigned by DHCP and has to match the target registered in the group.
On the edge device#
Only if you also terminate a VPN against the same architecture.
IPSec Tunnels on the edge device#

Both tunnels towards AWS green, each on its own tunnel interface. The blank rows are production tunnels on the same device, hidden on purpose.
BGP · peers#

Both AWS peers Established, with the local on .2 and the peer on .1 of each /30. Above, a pre-existing peer with its own uptime: when adding the new peer group, it is worth confirming that the ones already there did not flap.
Mistakes — Ten that give you no message#
All of them turned up while building this architecture. None produces an error that points at the cause.
E1 · GWLB
GENEVE parsing is not enabled by default#
Symptom — Everything looks healthy and not a single packet is inspected. No errors.
Cause — PAN-OS does not decapsulate UDP 6081 until you ask it to. overlay-routing is a different setting.
Fix — request plugins vm_series aws gwlb inspect enable yes
E2 · policy
The final deny stops the target from reaching healthy#
Symptom — Permanent Target.Timeout, even with HTTP enabled in the management profile.
Cause — The health check is intra-zone and goes through the policy, where the final deny matches first.
Fix — An allow rule for TCP 80 from the data subnets, in first position.
E3 · policy
application any with application-default matches nothing#
Symptom — Inspection rules at 0 hits from day one, with no explanation.
Cause — With no concrete applications there is no default port to derive the service from.
Fix — service any, or list explicit applications.
E4 · policy
New rules are born below the final deny#
Symptom — You add a correct rule and it never matches.
Cause — set rulebase security rules appends to the end of the rulebase.
Fix — move ... top or before <cleanup>, and check the order before committing.
E5 · CLI
The zone command path changes with multi-vsys#
Symptom — Invalid syntax when creating the zone, and then cascading not a valid reference failures.
Cause — With multi-vsys off the vsys node does not exist at the root.
Fix — set zone with multi-vsys off; set vsys vsys1 zone with multi-vsys on.
E6 · hybrid
AWS gives the first host of the /30 to the gateway, not the client#
Symptom — IPsec comes up and BGP never leaves Connect.
Cause — In the tunnel's inner /30, .1 belongs to the virtual private gateway and .2 to the customer gateway.
Fix — Read the attributes AWS returns instead of working the addresses out by hand.
E7 · hybrid
A final any/any rule also kills intra-zone traffic#
Symptom — BGP does not establish over the tunnel /30s.
Cause — A from any to any rule is of type universal: it matches intra-zone as well as inter-zone.
Fix — An explicit intra-zone rule for the tunnel's inner IPs, above the deny.
E8 · Terraform
Inline ingress blocks revoke whatever you add by hand#
Symptom — You lose management access after an apply that did not touch that resource.
Cause — With inline blocks, Terraform is authoritative over the entire security group.
Fix — Make the access variable a list and keep the IPs in the code. Check with terraform plan -detailed-exitcode.
E9 · CLI
An 80-column PTY corrupts long commands#
Symptom — Intermittent Invalid syntax on commands that are correct.
Cause — The CLI wraps by inserting a space in the middle: ethernet1/1 arrives as ethern et1/1.
Fix — Enlarge the PTY before the spawn, or descend with edit <node> to shorten each line.
E10 · logs
direction equal backward shows the oldest entries#
Symptom — Your test traffic never appears in the log even though the rule is accumulating hits.
Cause — It returns the oldest entries, and the log is dominated by health checks.
Fix — show log traffic rule equal <NAME>
Limitations — What this design does not solve#
East-west between app subnets is not inspected#
In each app subnet's route table, the VPC local route beats the default towards the endpoint on longest-prefix match. Traffic between workloads goes direct, and the rule that covers it stays at zero hits.
Adding routes more specific than local is not enough: each direction would enter through its own zone's endpoint and be inspected by a different firewall. With a stateful device that does not work, and stickiness does not save it because when source and destination swap, the hash changes. The real options are sending east-west through a single endpoint — losing independence between zones — or accepting that this path is not inspected and removing the rule so as not to promise something that does not happen.
No MSS clamp on the tunnel interfaces#
If you also terminate a VPN against the same device, bear in mind that adjust-tcp-mss does not exist under tunnel units in PAN-OS 11.1: the node only appears on ethernet subinterfaces. Without a clamp, ping goes through and large TCP transfers hang. The alternative is clamping on the ingress LAN interfaces, at the cost of affecting all their traffic.
The workloads depend on inspection in order to provision themselves#
Their internet egress goes through the endpoint, the GWLB and the firewall. While inspection is not working they have no internet: cloud-init cannot reach the metadata service and user_data never runs, with no visible error in the console. That is why the end-to-end test comes at the end of the procedure and not in the middle.