AWS Streaming Monitor

Overview

The AWS streaming monitor sends CloudTrail events, VPC Flow Logs, Application Load Balancer (ALB) logs, or Network Load Balancer (NLB) logs from your AWS account to Fencer’s SIEM in near-real time. It replaces the batch setup, which copied log files out of an S3 bucket on a schedule. Events travel from your trail, VPC, or load balancer through Amazon Data Firehose to a Fencer endpoint, and appear in Hunts a few minutes later.

Each source streams one data source type: AWS CloudTrail, AWS VPC Flow Logs, AWS ALB Access Logs, AWS ALB Connection Logs, or AWS NLB Access Logs. To stream more than one, create one source for each type.

Note

Fencer turns on streaming for each organization on request. Contact us to have it enabled.

Prerequisites

• Streaming enabled for your organization.
• Admin role in Fencer.
• An active subscription with the SIEM feature enabled.
• For CloudTrail: a trail in your AWS account, and permission in that account to create IAM roles, an S3 bucket, a Firehose stream, and CloudWatch Logs subscription filters.
• For VPC Flow Logs: one or more VPCs in your AWS account, and permission in that account to create IAM roles, an S3 bucket, a Firehose stream, and flow logs. The console steps list the flow log permissions.
• For ALB logs: one or more Application Load Balancers in your AWS account, and permission in that account to create an S3 bucket, a Firehose stream, and log deliveries. The permissions section lists the actions.
• For NLB logs: one or more Network Load Balancers with a TLS listener in your AWS account, and permission in that account to create an S3 bucket, a Firehose stream, and log deliveries. The NLB permissions section lists the actions.

How it works

For CloudTrail, your trail delivers events to a CloudWatch Logs log group. A subscription filter on that log group forwards every event to an Amazon Data Firehose stream, and the stream posts them over HTTPS to an endpoint that belongs to your Fencer data source. An access token authenticates each delivery. Firehose writes any record it cannot deliver to an S3 bucket in your account.

For VPC Flow Logs, each flow log publishes straight to the Firehose stream. There is no log group and no subscription filter. The rest of the path is the same.

For ALB logs, a log delivery on each load balancer publishes straight to the Firehose stream, in JSON format. There is no log group and no subscription filter here either.

NLB logs follow the same path as ALB logs.

The setup has two halves. In Fencer you create the data source and generate a token. In AWS you build the delivery path, either by hand in the console or with the Terraform module that Fencer publishes.

Usage / Configuration

Create the source in Fencer

1. Navigate to Settings > Integrations and find the AWS Monitor (Streaming) integration.
2. Click Connect.
3. Choose the Data source type: AWS CloudTrail, AWS VPC Flow Logs, AWS ALB Access Logs, AWS ALB Connection Logs, or AWS NLB Access Logs. The default is AWS CloudTrail.
4. Enter a Display name, such as AWS CloudTrail, and click Create source.
5. Wait for the status badge to change from Provisioning to Ready.
6. Under Access tokens, click Generate token. Give the token a label that says where you will use it, such as production-cloudtrail.
7. Copy the token and store it in your secrets manager. Then check I’ve stored this token in a safe place and click Done.
8. Copy the URL under Firehose endpoint.

Caution

Fencer shows the full token once. If you lose it, revoke it and generate a new one.

Set up AWS in the console

These steps are for CloudTrail. For VPC Flow Logs, see Stream VPC Flow Logs. For ALB logs, see Stream ALB access logs and connection logs. For NLB logs, see Stream NLB access logs.

Create every resource in the AWS account and region that hold your trail’s log group.

1. Make sure your trail delivers to a CloudWatch Logs log group. To turn this on, go to CloudTrail > Trails, open your trail, choose Edit, and enable CloudWatch Logs. AWS creates the delivery role for you. Write down the log group name.

2. Create an S3 bucket for failed deliveries.

3. Create an Amazon Data Firehose stream with source Direct PUT and destination HTTP Endpoint, using these settings:

   Setting	Value
   HTTP endpoint URL	The Firehose endpoint URL from Fencer
   Access key	Your Fencer access token
   Content encoding	GZIP
   Buffer size	1 MiB
   Buffer interval	60 seconds
   Backup settings	Failed data only, to the bucket from step 2
   Service access	Create or update IAM role (the default)

   The role that Firehose creates lets it write failed records to the bucket and log delivery errors to CloudWatch.

4. Create an IAM role that CloudWatch Logs can assume to write to the stream. In IAM > Roles > Create role, choose Custom trust policy and paste the policy below. Replace <YOUR_ACCOUNT_ID> with your AWS account ID.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": { "Service": "logs.amazonaws.com" },
               "Action": "sts:AssumeRole",
               "Condition": {
                   "StringEquals": { "aws:SourceAccount": "<YOUR_ACCOUNT_ID>" }
               }
           }
       ]
   }

   On the Add permissions step, choose Create inline policy and paste the policy below into the JSON tab. Replace <YOUR_FIREHOSE_STREAM_ARN> with the ARN of the stream from step 3.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": ["firehose:PutRecord", "firehose:PutRecordBatch"],
               "Resource": "<YOUR_FIREHOSE_STREAM_ARN>"
           }
       ]
   }

5. Subscribe the log group to the stream. Do this in the CloudWatch console, not in CloudTrail.

   1. Open Logs > Log Management > Log groups and select the log group from step 1.
   2. Choose Actions > Subscription filters > Create Amazon Data Firehose subscription filter.
   3. Set the destination account to Current account and pick the stream from step 3.
   4. Under Grant permission, select the role from step 4.
   5. Set Log format to Other, leave the filter pattern empty to send all events, and name the filter fencer-siem-cloudtrail.
   6. Choose Start streaming.

Set up AWS with Terraform

Fencer publishes a Terraform module, Fencer-Security/fencer-monitor/aws, that creates the Firehose stream, its IAM roles, the error log group, the failed-delivery bucket, and the subscription filter.

variable "fencer_access_key" {
  description = "Fencer access token (generate it in the Fencer AWS Monitor page)"
  type        = string
  sensitive   = true
}

variable "cloudtrail_log_group_name" {
  description = "CloudWatch Logs group that receives your CloudTrail events"
  type        = string
}

module "fencer_monitor" {
  source  = "Fencer-Security/fencer-monitor/aws"
  version = "~> 1.3"

  fencer_endpoint_url = "<your Firehose endpoint URL>"
  fencer_access_key   = var.fencer_access_key

  # A key of your choice to the log group name. The key is the Terraform instance key.
  cloudwatch_log_groups = { cloudtrail = var.cloudtrail_log_group_name }

  # Each module instance in one AWS account must have a different name_prefix.
  name_prefix = "fencer-cloudtrail"
}

Apply the module in the same account and region as your trail’s log group. Pass fencer_access_key from a secrets manager or through TF_VAR_fencer_access_key, and do not commit it. Terraform stores the key in plaintext in its state file, so keep the state in an encrypted backend with restricted access.

If your trail does not deliver to CloudWatch Logs yet, the module README covers the trail setup that matches yours.

Set up AWS with Pulumi

Pulumi uses the Terraform module directly. Add it once as a local package:

pulumi package add terraform-module Fencer-Security/fencer-monitor/aws 1.3.0 fencermonitor

Then declare the module in your program:

import * as pulumi from "@pulumi/pulumi";
import * as fencermonitor from "@pulumi/fencermonitor";

const config = new pulumi.Config();

const monitor = new fencermonitor.Module("fencer-monitor", {
    fencer_endpoint_url: "<your Firehose endpoint URL>",
    fencer_access_key: config.requireSecret("fencerAccessKey"),
    // A key of your choice to the log group name.
    cloudwatch_log_groups: { cloudtrail: config.require("cloudtrailLogGroupName") },
    // Each module instance in one AWS account must have a different name_prefix.
    name_prefix: "fencer-cloudtrail",
});

export const firehoseDeliveryStreamName = monitor.firehose_delivery_stream_name;
export const backupBucketName = monitor.backup_bucket_name;

Set both config values before you deploy:

pulumi config set cloudtrailLogGroupName <log-group>
pulumi config set --secret fencerAccessKey <token>

Tip

The source page in Fencer shows these snippets with your endpoint URL filled in. Its Agent prompt tab holds a prompt that you can paste into a coding agent, which then adapts the Terraform module to your trail, account layout, and infrastructure-as-code tool.

Verify the stream

Within about five minutes, the monitoring tab of the Firehose stream shows successful deliveries to the HTTP endpoint (2xx responses), and the failed-delivery bucket stays empty. Open Hunts in Fencer to see the events. VPC Flow Logs, ALB connection logs, and NLB access logs arrive as OCSF Network Activity records. ALB access logs arrive as OCSF HTTP Activity records.

Rotate or revoke a token

A source can hold more than one active token. To rotate a token, generate a new one, set it as the access key on the Firehose stream, and then click Revoke next to the old token.

Caution

Revoking takes effect at once and cannot be undone. Fencer denies any stream that still uses the token.

Remove the source

Remove the connection with the delete action on its card in your integration settings. Fencer revokes every active token for the source and stops accepting its events. Delete the subscription filter and the Firehose stream in AWS as well, or Firehose keeps writing rejected records to the failed-delivery bucket.

Change the data source type

You can change the type of a source until you generate its first access token. Open the source page and click Change type next to the source name. In the Change data source type dialog, choose the new Data source type and click Change type. The source must be Ready.

After the first token the type is fixed, even if you revoke the token, because the source can already hold events of that type. To use another type, remove the source and create a new one. The new source has a new endpoint URL, so give the Firehose stream the new URL and a new token.

Stream VPC Flow Logs

VPC Flow Logs record the IP traffic that enters and leaves the network interfaces in your VPCs. Choose AWS VPC Flow Logs as the Data source type when you create the source. Fencer maps each flow log record to OCSF Network Activity (class 4001) and enriches the source and destination IP addresses with GeoIP location and autonomous system (ASN) data. It also adds cloud metadata, looked up by instance ID or network interface ID.

Fencer accepts the AWS default format (version 2) and the full field set of versions 3, 4, 5, 7, 8, 9, 10, and 11. Accepted flow log formats lists the fields of each. The Terraform module and the console steps below use the version 10 field set, which has 42 fields. It leaves out the tag fields of version 11, because AWS needs a TagFieldSpecifications setting on the flow log before it records them.

After you set up AWS with one of the methods below, follow Verify the stream.

Set up AWS in the console

Create every resource in the AWS account and region that hold your VPCs.

1. Create an S3 bucket for failed deliveries, then create the Firehose stream with the same settings as steps 2 and 3 of the CloudTrail console setup. Use the Firehose endpoint URL and access token from Fencer.

2. Open the VPC console. Go to Your VPCs, select a VPC, and choose Actions > Create flow log. Repeat this for each VPC that you want to monitor.

3. Set the flow log options:

   • Filter: All
   • Maximum aggregation interval: 1 minute
   • Destination: Send to Amazon Data Firehose in the same account
   • Amazon Data Firehose stream: the stream from step 1

4. Set Log record format to Custom format, and select every field of versions 2 to 10 in the order below.

   ${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${vpc-id} ${subnet-id} ${instance-id} ${tcp-flags} ${type} ${pkt-srcaddr} ${pkt-dstaddr} ${region} ${az-id} ${sublocation-type} ${sublocation-id} ${pkt-src-aws-service} ${pkt-dst-aws-service} ${flow-direction} ${traffic-path} ${ecs-cluster-arn} ${ecs-cluster-name} ${ecs-container-instance-arn} ${ecs-container-instance-id} ${ecs-container-id} ${ecs-second-container-id} ${ecs-service-name} ${ecs-task-definition-arn} ${ecs-task-arn} ${ecs-task-id} ${reject-reason} ${resource-id} ${encryption-status}

   Fencer also accepts the AWS default format (version 2) and the full field set of any other version. See Accepted flow log formats.

5. Check that the AWS user or role that creates the flow log has these permissions: logs:CreateLogDelivery, logs:DeleteLogDelivery, iam:CreateServiceLinkedRole, firehose:TagDeliveryStream, ecs:ListClusters, ecs:ListContainerInstances, ecs:ListServices, ecs:ListTaskDefinitions, and ecs:ListTasks. The ECS permissions are needed because the version 10 field set includes ECS fields.

Set up AWS with Terraform

The Terraform module Fencer-Security/fencer-monitor/aws creates the Firehose stream, its IAM role, the error log group, the failed-delivery bucket, and one flow log for each entry of vpc_flow_logs, a map of a key of your choice to the VPC ID. The key is the flow log’s Terraform instance key, so a VPC that the same apply creates works.

variable "fencer_access_key" {
  description = "Fencer access token (generate it in the Fencer AWS Monitor page)"
  type        = string
  sensitive   = true
}

variable "vpcs" {
  description = "VPCs to collect flow logs from: a key of your choice to the VPC ID"
  type        = map(string)
}

module "fencer_vpc_flow_logs" {
  source  = "Fencer-Security/fencer-monitor/aws"
  version = "~> 1.3"

  # vpc_flow_log_format defaults to the version 10 field set (42 fields). For another version,
  # see "Accepted flow log formats" on this page.
  fencer_endpoint_url = "<your Firehose endpoint URL>"
  fencer_access_key   = var.fencer_access_key
  vpc_flow_logs       = var.vpcs

  # Each module instance in one AWS account must have a different name_prefix.
  name_prefix = "fencer-flowlogs"
}

Apply the module in the same account and region as the VPCs. The default value of vpc_flow_log_format is the version 10 field set. To publish another version, set it to the full field set of that version as $${field} tokens. The module’s examples/vpc-flow-logs/formats.tf holds the string for each accepted version, and Accepted flow log formats lists the fields. Use one module instance for each Fencer data source: do not send CloudTrail and flow logs through the same instance, and give each instance in an AWS account a different name_prefix. The module requires the input: the IAM role, the Firehose stream and the log group take their names from it, so two instances with one prefix collide. The principal that applies the module needs the flow log permissions listed in the console steps, plus permission to create the module’s resources: IAM roles, the Firehose stream, the S3 bucket, and the log group. The module repository has a complete example in examples/vpc-flow-logs, and the module README lists every permission.

Pass fencer_access_key from a secrets manager or through TF_VAR_fencer_access_key, and do not commit it. Terraform stores the key in plaintext in its state file, so keep the state in an encrypted backend with restricted access.

Set up AWS with Pulumi

Pulumi uses the Terraform module directly. Add it once as a local package:

pulumi package add terraform-module Fencer-Security/fencer-monitor/aws 1.3.0 fencermonitor

Then declare the module in your program:

import * as pulumi from "@pulumi/pulumi";
import * as fencermonitor from "@pulumi/fencermonitor";

const config = new pulumi.Config();

// The module default for vpc_flow_log_format is the version 10 field set. For another version,
// pass the full field set of that version (see "Accepted flow log formats" on this page).
const monitor = new fencermonitor.Module("fencer-vpc-flow-logs", {
    fencer_endpoint_url: "<your Firehose endpoint URL>",
    fencer_access_key: config.requireSecret("fencerAccessKey"),
    // A key of your choice to the VPC ID, for example { main: "vpc-0123456789abcdef0" }.
    vpc_flow_logs: config.requireObject<Record<string, string>>("vpcs"),
    // Each module instance in one AWS account must have a different name_prefix.
    name_prefix: "fencer-flowlogs",
});

export const firehoseDeliveryStreamName = monitor.firehose_delivery_stream_name;
export const backupBucketName = monitor.backup_bucket_name;
export const flowLogIds = monitor.flow_log_ids;

Set both config values before you deploy:

pulumi config set --path 'vpcIds[0]' <vpc-id>
pulumi config set --secret fencerAccessKey <token>

Accepted flow log formats

Fencer reads the fields of a flow log record by position. A flow log must therefore use one of the nine field sets below, with the fields in this order. Each version has every field of the rows above it, followed by the fields in its own row. There is no version 6.

Version	Fields	Fields the version adds, in order
2 (AWS default)	14	version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
3	21	vpc-id subnet-id instance-id tcp-flags type pkt-srcaddr pkt-dstaddr
4	25	region az-id sublocation-type sublocation-id
5	29	pkt-src-aws-service pkt-dst-aws-service flow-direction traffic-path
7	39	ecs-cluster-arn ecs-cluster-name ecs-container-instance-arn ecs-container-instance-id ecs-container-id ecs-second-container-id ecs-service-name ecs-task-definition-arn ecs-task-arn ecs-task-id
8	40	reject-reason
9	41	resource-id
10	42	encryption-status
11	54	instance-tag instance-tag-2 interface-tag interface-tag-2 asg-tag asg-tag-2 interface-type next-hop-interface-id next-hop-subnet-id next-hop-az-id next-hop-vpc-id next-hop-interface-type

Fencer picks the row from the number of fields in a record and checks it against the record’s version field. It also checks that the account ID, interface ID, addresses, action, log status, and start and end times hold values those fields can have. A record that fails a check lands as an error row with no OCSF fields. These checks catch most flow logs that have the fields in a different order. A swap of two fields of the same kind, such as source and destination, passes them, and the values land in the wrong fields. Keep the fields in this order and leave none out, unless the source has a custom transformation.

To publish a version other than 10: in the console, select every field of that version in this order. With Terraform or Pulumi, set vpc_flow_log_format to those fields as ${field} tokens ($${field} in Terraform). The module’s examples/vpc-flow-logs/formats.tf holds a ready string for each version. In that example, the flow_log_version variable selects one; it is a variable of the example, not of the module.

Version 11 adds tag fields. AWS fills them only from tag field specifications set when the flow log is created, so set the module input vpc_flow_log_tag_field_specifications with the resource type (instance, network-interface or auto-scaling-group) and the tag keys to publish. The module refuses a version 11 format without it. The principal also needs ec2:DescribeTags for instance and network interface tags and autoscaling:DescribeTags for Auto Scaling group tags.

Stream ALB access logs and connection logs

An Application Load Balancer (ALB) keeps two kinds of log. Access logs hold one entry for each request. Connection logs hold one entry for each TLS connection that a client opens. Choose AWS ALB Access Logs or AWS ALB Connection Logs as the Data source type when you create the source. Fencer maps access logs to OCSF HTTP Activity (class 4002) and connection logs to OCSF Network Activity (class 4001). It enriches the client address with GeoIP location and autonomous system (ASN) data. For access logs it also adds the cloud metadata of the load balancer.

AWS sends these logs through a log delivery on each load balancer. The delivery writes to your Firehose stream, and the output format of the delivery must be JSON. The Terraform module creates the delivery for you. In the console, you create it yourself.

Caution

One source receives one log type. To stream both access logs and connection logs, create two sources, generate a token for each, and give each source its own Firehose stream. Fencer rejects a record of the other type and records an ingestion error that names the source type to use. A stream that carries both log types loses the records of the other type.

One source can receive its log type from many load balancers. Each load balancer gets one log delivery, all pointing at the same stream. AWS allows one vended log delivery source per load balancer and log type, so a load balancer that already sends the same log type through a log delivery cannot send it to Fencer as well.

Create the Firehose stream in the console

Create every resource in the AWS account and region that hold your load balancers. Create an S3 bucket for failed deliveries, then create the Firehose stream with the same settings as steps 2 and 3 of the CloudTrail console setup. Use the Firehose endpoint URL and access token from Fencer.

The log delivery writes only to a stream that carries the tag LogDeliveryEnabled with the value true. AWS adds the tag when you create the delivery in the console. If you create the delivery with the CLI, check that the stream has the tag. The Terraform module sets it.

Create the Firehose stream with Terraform

The Terraform module Fencer-Security/fencer-monitor/aws creates the Firehose stream, its IAM role, the error log group, the failed-delivery bucket, and one log delivery for each entry of alb_load_balancers, a map of a key of your choice to the load balancer ARN. The key names the delivery source (<name_prefix>-<key>) and is its Terraform instance key, so a load balancer that the same apply creates works. The plan refuses a load balancer from another AWS account or region than the module. The module repository has a complete example for each log type in examples/alb-access-logs and examples/alb-connection-logs, and examples/alb-logs for both log types from one configuration.

variable "fencer_alb_access_logs_access_key" {
  description = "Fencer access token (generate it in the Fencer AWS Monitor page)"
  type        = string
  sensitive   = true
}

variable "fencer_alb_access_logs_load_balancers" {
  description = "Load balancers to collect logs from: a key of your choice to the load balancer ARN"
  type        = map(string)
}

module "fencer_alb_access_logs" {
  source  = "Fencer-Security/fencer-monitor/aws"
  version = "~> 1.3"

  fencer_endpoint_url = "<your Firehose endpoint URL>"
  fencer_access_key   = var.fencer_alb_access_logs_access_key
  name_prefix         = "fencer-alb-access-logs"

  # One module instance delivers one log type. The module creates the log delivery
  # for each load balancer; the key names it (<name_prefix>-<key>).
  alb_log_type       = "ALB_ACCESS_LOGS"
  alb_load_balancers = var.fencer_alb_access_logs_load_balancers
}

output "fencer_alb_access_logs_stream_arn" {
  value = module.fencer_alb_access_logs.firehose_delivery_stream_arn
}

The module sets the LogDeliveryEnabled tag on the stream and keeps it across applies, because Terraform removes a tag that it does not manage and the delivery stops without it.

Each log type has its own names, so that you can run both in one account and one Terraform configuration. The name_prefix also keeps the two streams from sharing the module default name.

Name	Access logs	Connection logs
AWS log type	ALB_ACCESS_LOGS	ALB_CONNECTION_LOGS
name_prefix and destination	fencer-alb-access-logs	fencer-alb-connection-logs
Terraform module	fencer_alb_access_logs	fencer_alb_connection_logs
Terraform variable	fencer_alb_access_logs_access_key	fencer_alb_connection_logs_access_key
Terraform load balancers	fencer_alb_access_logs_load_balancers	fencer_alb_connection_logs_load_balancers
Terraform output	fencer_alb_access_logs_stream_arn	fencer_alb_connection_logs_stream_arn
Pulumi secret	fencerAlbAccessLogsAccessKey	fencerAlbConnectionLogsAccessKey
Pulumi load balancers config	fencerAlbAccessLogsLoadBalancers	fencerAlbConnectionLogsLoadBalancers
Pulumi resource name	fencer-alb-access-logs	fencer-alb-connection-logs
Pulumi module constant	fencerAlbAccessLogsMonitor	fencerAlbConnectionLogsMonitor
Pulumi stream ARN export	fencerAlbAccessLogsStreamArn	fencerAlbConnectionLogsStreamArn
Pulumi stream name export	fencerAlbAccessLogsStreamName	fencerAlbConnectionLogsStreamName
Pulumi backup bucket export	fencerAlbAccessLogsBackupBucketName	fencerAlbConnectionLogsBackupBucketName

Apply the module in the same account and region as the load balancers. Use one module instance for each Fencer data source. Pass the access key from a secrets manager or through the matching TF_VAR_ environment variable, and do not commit it. Terraform stores the key in plaintext in its state file, so keep the state in an encrypted backend with restricted access. The module README lists the permissions that the principal needs.

Create the Firehose stream with Pulumi

Pulumi uses the Terraform module directly. Add it once as a local package:

pulumi package add terraform-module Fencer-Security/fencer-monitor/aws 1.3.0 fencermonitor

Then declare the module in your program. The snippet is for access logs. For connection logs, use the names in the table above.

import * as pulumi from "@pulumi/pulumi";
import * as fencermonitor from "@pulumi/fencermonitor";

const config = new pulumi.Config();

// One module instance delivers one log type. The module creates the log delivery for each
// load balancer; the key names it (<name_prefix>-<key>).
const fencerAlbAccessLogsMonitor = new fencermonitor.Module("fencer-alb-access-logs", {
    fencer_endpoint_url: "<your Firehose endpoint URL>",
    fencer_access_key: config.requireSecret("fencerAlbAccessLogsAccessKey"),
    name_prefix: "fencer-alb-access-logs",
    alb_log_type: "ALB_ACCESS_LOGS",
    alb_load_balancers: config.requireObject<Record<string, string>>("fencerAlbAccessLogsLoadBalancers"),
});

export const fencerAlbAccessLogsStreamArn = fencerAlbAccessLogsMonitor.firehose_delivery_stream_arn;
export const fencerAlbAccessLogsStreamName =
    fencerAlbAccessLogsMonitor.firehose_delivery_stream_name;
export const fencerAlbAccessLogsBackupBucketName = fencerAlbAccessLogsMonitor.backup_bucket_name;

Set the secret and, once per load balancer, the load balancer before you deploy:

pulumi config set --secret fencerAlbAccessLogsAccessKey <token>
pulumi config set --path fencerAlbAccessLogsLoadBalancers.my-lb <load balancer ARN>

Create the log delivery without the module

Skip this section if you use the Terraform module: it creates the log delivery. If you created a delivery with these commands before you switched to the module, delete it first with aws logs delete-delivery --id <id>; the module adopts a delivery source and destination of the same name, but a second delivery for the pair fails.

A log delivery has three parts. The delivery destination points at your Firehose stream and fixes the output format. The delivery source stands for one load balancer and one log type. The delivery joins the two.

In the console, open the load balancer in the EC2 console and add a log delivery for its access logs or connection logs. Send it to Amazon Data Firehose, choose your stream, and set the output format to JSON. Repeat this for each load balancer that you want to monitor.

With the AWS CLI, create the destination once. Then create a source and a delivery for each load balancer. A delivery source belongs to one load balancer, and its name must be unique in the account, so the commands add the load balancer name to the source name.

For access logs:

aws logs put-delivery-destination --name fencer-alb-access-logs --output-format json \
  --delivery-destination-configuration destinationResourceArn=<Firehose stream ARN>
aws logs put-delivery-source --name fencer-alb-access-logs-<load balancer name> \
  --resource-arn <load balancer ARN> --log-type ALB_ACCESS_LOGS
aws logs create-delivery --delivery-source-name fencer-alb-access-logs-<load balancer name> \
  --delivery-destination-arn <delivery destination ARN>

For connection logs:

aws logs put-delivery-destination --name fencer-alb-connection-logs --output-format json \
  --delivery-destination-configuration destinationResourceArn=<Firehose stream ARN>
aws logs put-delivery-source --name fencer-alb-connection-logs-<load balancer name> \
  --resource-arn <load balancer ARN> --log-type ALB_CONNECTION_LOGS
aws logs create-delivery --delivery-source-name fencer-alb-connection-logs-<load balancer name> \
  --delivery-destination-arn <delivery destination ARN>

Replace the placeholders as follows:

• <Firehose stream ARN> is the ARN of the stream. With Terraform, it is the fencer_alb_access_logs_stream_arn output (or the connection logs output).
• <delivery destination ARN> is the ARN that put-delivery-destination returns.
• <load balancer name> is the name of one load balancer. It is the segment between loadbalancer/app/ and the next / in its ARN. For ...:loadbalancer/app/my-lb/50dc6c495c0c9188, the name is my-lb. The name must not contain a slash.
• <load balancer ARN> is the full ARN of that load balancer. aws elbv2 describe-load-balancers prints it.

The output format must be JSON. Fencer does not accept the plain format: the stream delivers successfully, but no events appear in Hunts.

After the delivery exists, follow Verify the stream.

Permissions for the log delivery

The AWS user or role that creates the delivery needs these actions: logs:PutDeliverySource, logs:PutDeliveryDestination, logs:CreateDelivery, logs:GetDelivery, firehose:TagDeliveryStream, iam:CreateServiceLinkedRole, and elasticloadbalancing:AllowVendedLogDeliveryForResource on the load balancers. AWS uses the service-linked role to write to your stream. The first five are the actions in the AWS documentation for log delivery to Firehose; the last one is checked when the delivery source is created, on the load balancer ARN. Terraform also needs the Get, Describe and Delete counterparts of the logs actions. Check the AWS documentation if your account policy blocks the delivery.

What Fencer maps from ALB logs

ALB writes - for a value it does not have. Fencer drops those values, so a missing field is absent in Hunts and never shows a placeholder.

Access logs become HTTP Activity events:

ALB fields	OCSF fields
request_line	http_request.http_method, http_request.url (host, port, path, query), version
user_agent, trace_id	http_request.user_agent, http_request.uid
elb_status_code	http_response.code, and status: below 400 is Success, 400 and above is Failure
client_port (client address and port)	src_endpoint.ip, src_endpoint.port
elb, ip_address, domain_name	dst_endpoint.name, dst_endpoint.ip, dst_endpoint.hostname, load_balancer
target_port, target_status_code	load_balancer.dst_endpoint, load_balancer.code
request_processing_time, target_processing_time, response_processing_time	load_balancer.metrics
received_bytes, sent_bytes	traffic.bytes_out (request size), traffic.bytes_in (response size)
ssl_protocol, ssl_cipher	tls.version, tls.cipher
time, request_creation_time	end_time, start_time, and duration in milliseconds
actions_executed	action: Allowed when the last action is forward, redirect or fixed-response, Denied when it is waf, waf-failed or authenticate, otherwise Unknown
error_reason, classification	status_detail, load_balancer.classification
conn_trace_id	metadata.correlation_uid

The activity comes from the HTTP method, such as Get or Post. Fencer keeps the method exactly as the client sent it, and picks the activity from its upper-case form. An unusual method becomes the activity Other. A request that the load balancer could not parse (the request line - - -) has no method and no URL, and its activity is Unknown. The target group ARN, the certificate ARN, the rule priority, and the other fields without an OCSF match go into unmapped.

Connection logs become Network Activity events:

ALB fields	OCSF fields
client_ip, client_port	src_endpoint.ip, src_endpoint.port
elb, ip_address, listener_port	dst_endpoint.name, dst_endpoint.ip, dst_endpoint.port
tls_protocol, tls_cipher	tls.version, tls.cipher
tls_handshake_latency	tls.handshake_dur, converted from seconds to milliseconds
leaf_client_cert_subject, leaf_client_cert_serial_number	tls.certificate.subject, tls.certificate.serial_number
leaf_client_cert_validity	tls.certificate.created_time, tls.certificate.expiration_time
tls_verify_status	status, and the activity: Open, or Fail when the status starts with Failed
conn_trace_id	metadata.uid, metadata.correlation_uid

For a failed connection, status_detail holds the reason that follows the colon in the verify status. tls_keyexchange goes into unmapped.

Fencer adds GeoIP location and ASN data for the client address on both log types. For access logs it also looks up the load balancer by its full ARN and adds its tags and a link to its asset page, when Fencer already knows the load balancer from your AWS integration. The lookup runs only when Fencer can build the ARN. It builds the ARN from the target group ARN of the record, or else from the ACM certificate ARN. A request with neither, such as one that a WAF rule blocked, gets no lookup. Connection logs carry no target group or certificate, so Fencer cannot build the load balancer ARN, and it adds no cloud metadata.

Join access logs and connection logs

To see the connection behind a request, join the two sources on metadata.correlation_uid. It holds the ALB conn_trace_id on both log types. Do not join on load_balancer.uid: on access logs it holds the full ARN, or the load balancer ID when Fencer cannot build the ARN, and on connection logs it holds the load balancer ID. The load balancer ID has the form app/<name>/<id>, as in the elb field. dst_endpoint.name holds this ID on both.

Stream NLB access logs

A Network Load Balancer (NLB) writes one access log entry for each TLS connection that a client opens on a TLS listener. A TCP or UDP listener produces no entries, and a load balancer without a TLS listener produces none at all. AWS delivers the entries on a best-effort basis, so the log is not a complete count of connections. Choose AWS NLB Access Logs as the Data source type when you create the source. Fencer maps each entry to OCSF Network Activity (class 4001). It enriches the client address with GeoIP location and autonomous system (ASN) data, and it adds the cloud metadata of the load balancer.

AWS sends these logs through a log delivery on each load balancer, with the log type NLB_ACCESS_LOGS. The delivery writes to your Firehose stream, and the output format of the delivery must be JSON. The Terraform module creates the delivery for you. In the console, you create it yourself.

Note

The nlb_load_balancers module input needs module version 1.4 or later. Earlier versions do not accept it.

Caution

One source receives one log type. Do not send ALB logs, CloudTrail events, or flow logs to an NLB source, and give each source its own Firehose stream. Fencer rejects a record of another type and records an ingestion error that names the fix.

One source can receive NLB access logs from many load balancers. Each load balancer gets one log delivery, all pointing at the same stream. AWS allows one vended log delivery source per load balancer and log type, so a load balancer that already sends NLB access logs through a log delivery cannot send them to Fencer as well.

Create the NLB Firehose stream in the console

Create every resource in the AWS account and region that hold your load balancers. Create the S3 bucket and the Firehose stream as in Create the Firehose stream in the console, with the endpoint URL and access token from your NLB source.

The log delivery writes only to a stream that carries the tag LogDeliveryEnabled with the value true. AWS adds the tag when you create the delivery in the console. If you create the delivery with the CLI, check that the stream has the tag. The Terraform module sets it.

Create the NLB Firehose stream with Terraform

The Terraform module Fencer-Security/fencer-monitor/aws creates the Firehose stream, its IAM role, the error log group, the failed-delivery bucket, and one log delivery for each entry of nlb_load_balancers, a map of a key of your choice to the load balancer ARN. The key names the delivery source (<name_prefix>-<key>) and is its Terraform instance key, so a load balancer that the same apply creates works. The module sets the log type NLB_ACCESS_LOGS itself.

variable "fencer_nlb_access_logs_access_key" {
  description = "Fencer access token (generate it in the Fencer AWS Monitor page)"
  type        = string
  sensitive   = true
}

variable "fencer_nlb_access_logs_load_balancers" {
  description = "Load balancers to collect logs from: a key of your choice to the load balancer ARN"
  type        = map(string)
}

module "fencer_nlb_access_logs" {
  source  = "Fencer-Security/fencer-monitor/aws"
  version = "~> 1.4"

  fencer_endpoint_url = "<your Firehose endpoint URL>"
  fencer_access_key   = var.fencer_nlb_access_logs_access_key
  name_prefix         = "fencer-nlb-access-logs"

  # One module instance delivers one log type. The module creates the log delivery
  # for each load balancer; the key names it (<name_prefix>-<key>).
  nlb_load_balancers = var.fencer_nlb_access_logs_load_balancers
}

output "fencer_nlb_access_logs_stream_arn" {
  value = module.fencer_nlb_access_logs.firehose_delivery_stream_arn
}

The module sets the LogDeliveryEnabled tag on the stream and keeps it across applies, because Terraform removes a tag that it does not manage and the delivery stops without it.

Give the NLB module instance its own name_prefix. Two instances with the same prefix create resources with the same names. Apply the module in the same account and region as the load balancers. Use one module instance for each Fencer data source. Pass the access key from a secrets manager or through the matching TF_VAR_ environment variable, and do not commit it. Terraform stores the key in plaintext in its state file, so keep the state in an encrypted backend with restricted access. The module README lists the permissions that the principal needs.

Create the NLB Firehose stream with Pulumi

Pulumi uses the Terraform module directly. Add it once as a local package:

pulumi package add terraform-module Fencer-Security/fencer-monitor/aws 1.4.0 fencermonitor

Then declare the module in your program:

import * as pulumi from "@pulumi/pulumi";
import * as fencermonitor from "@pulumi/fencermonitor";

const config = new pulumi.Config();

// One module instance delivers one log type. The module creates the log delivery for each
// load balancer; the key names it (<name_prefix>-<key>).
const fencerNlbAccessLogsMonitor = new fencermonitor.Module("fencer-nlb-access-logs", {
    fencer_endpoint_url: "<your Firehose endpoint URL>",
    fencer_access_key: config.requireSecret("fencerNlbAccessLogsAccessKey"),
    name_prefix: "fencer-nlb-access-logs",
    nlb_load_balancers: config.requireObject<Record<string, string>>("fencerNlbAccessLogsLoadBalancers"),
});

export const fencerNlbAccessLogsStreamArn = fencerNlbAccessLogsMonitor.firehose_delivery_stream_arn;
export const fencerNlbAccessLogsStreamName =
    fencerNlbAccessLogsMonitor.firehose_delivery_stream_name;
export const fencerNlbAccessLogsBackupBucketName = fencerNlbAccessLogsMonitor.backup_bucket_name;

Set the secret and, once per load balancer, the load balancer before you deploy:

pulumi config set --secret fencerNlbAccessLogsAccessKey <token>
pulumi config set --path fencerNlbAccessLogsLoadBalancers.my-lb <load balancer ARN>

Create the NLB log delivery without the module

Skip this section if you use the Terraform module: it creates the log delivery. If you created a delivery with these commands before you switched to the module, delete it with aws logs delete-delivery --id <id> first, because a second delivery for the same source and destination fails.

A log delivery has three parts. The delivery destination points at your Firehose stream and fixes the output format. The delivery source stands for one load balancer and one log type. The delivery joins the two.

In the console, open the load balancer in the EC2 console and go to its Integrations tab. Add a CloudWatch log delivery for access logs, send it to Amazon Data Firehose, choose your stream, and set the output format to JSON. Repeat this for each load balancer that you want to monitor.

With the AWS CLI, create the destination once. Then create a source and a delivery for each load balancer. A delivery source belongs to one load balancer, and its name must be unique in the account, so the commands add the load balancer name to the source name.

aws logs put-delivery-destination --name fencer-nlb-access-logs --output-format json \
  --delivery-destination-configuration destinationResourceArn=<Firehose stream ARN>
aws logs put-delivery-source --name fencer-nlb-access-logs-<load balancer name> \
  --resource-arn <load balancer ARN> --log-type NLB_ACCESS_LOGS
aws logs create-delivery --delivery-source-name fencer-nlb-access-logs-<load balancer name> \
  --delivery-destination-arn <delivery destination ARN>

Replace the placeholders as follows:

• <Firehose stream ARN> is the ARN of the stream. With Terraform, it is the fencer_nlb_access_logs_stream_arn output.
• <delivery destination ARN> is the ARN that put-delivery-destination returns.
• <load balancer name> is the name of one load balancer. It is the segment between loadbalancer/net/ and the next / in its ARN. For ...:loadbalancer/net/my-lb/50dc6c495c0c9188, the name is my-lb. The name must not contain a slash.
• <load balancer ARN> is the full ARN of that load balancer. aws elbv2 describe-load-balancers prints it.

The output format must be JSON. Fencer does not accept the plain format. The stream delivers successfully, but Fencer records an ingestion error for each entry and no events appear in Hunts.

After the delivery exists, follow Verify the stream.

Permissions for the NLB log delivery

The AWS user or role that creates the delivery needs the same actions as for ALB logs: logs:PutDeliverySource, logs:PutDeliveryDestination, logs:CreateDelivery, logs:GetDelivery, firehose:TagDeliveryStream, iam:CreateServiceLinkedRole, and elasticloadbalancing:AllowVendedLogDeliveryForResource on the network load balancers. AWS checks the last action when the delivery source is created, on the load balancer ARN. AWS uses the service-linked role to write to your stream. Terraform also needs the Get, Describe and Delete counterparts of the logs actions. Check the AWS documentation if your account policy blocks the delivery.

What Fencer maps from NLB logs

NLB writes - for a value it does not have. Fencer drops those values, so a missing field is absent in Hunts and never shows a placeholder. A duration, byte count, TLS alert or port that is not a number leaves its field empty, and its raw text appears in unmapped under the field name.

Each entry becomes a Network Activity event:

NLB fields	OCSF fields
client_port (client address and port)	src_endpoint.ip, src_endpoint.port
destination_port (load balancer address and port), elb, domain_name	dst_endpoint.ip, dst_endpoint.port, dst_endpoint.name, dst_endpoint.hostname, load_balancer
tls_protocol_version, tls_cipher	tls.version (for example TLSv1.2), tls.cipher
domain_name (server name indication, SNI)	tls.sni
tls_handshake_time	tls.handshake_dur in milliseconds, and the activity and status
incoming_tls_alert	tls.alert, status_code, and status_detail (TLS alert <n>)
alpn_fe_protocol, alpn_be_protocol, alpn_client_preference_list	tls.extension_list: one ALPN extension that holds the three values
received_bytes, sent_bytes	traffic.bytes_out (from the client), traffic.bytes_in (to the client), traffic.bytes (the sum)
time, tls_connection_creation_time	end_time, start_time
connection_time	duration in milliseconds; Fencer uses the time between start and end when this field is missing

The activity comes from the TLS handshake. A connection with a handshake time is activity Traffic and status Success. A connection without one is activity Fail and status Failure. When the client sent a TLS alert, status_code holds the alert number. The listener, the entry type and version, the key exchange, the raw TLS version, and the certificate ARN and serial go into unmapped. OCSF requires an issuer and a serial number on a certificate, and an NLB entry has neither.

Fencer adds GeoIP location and ASN data for the client address. It also looks up the load balancer by its full ARN and adds its tags and a link to its asset page, when Fencer already knows the load balancer from your AWS integration. An NLB entry holds no account ID and no region, so Fencer builds the ARN from the ACM certificate ARN in chosen_cert_arn. An entry without one, such as a connection with no valid client hello, gets no lookup and no cloud region or account.

Records that Fencer rejects

Fencer records an ingestion error, in place of an event, for these records:

• A plain-format record. The error says that NLB access logs must use output format JSON. Set the output format of the delivery destination to JSON.
• A record from an Application Load Balancer, which has an elb value that starts with app/. The error says to send ALB logs to a source of an AWS ALB type.
• A record whose time is missing or is not a valid timestamp.

Examples

Streaming an organization trail with Terraform:

1. In Fencer, create a source named AWS CloudTrail and generate a token labeled organization-trail. Store the token in your secrets manager.

2. Find the log group that the trail delivers to:

   aws cloudtrail get-trail --name my-trail \
     --query 'Trail.CloudWatchLogsLogGroupArn'

3. Add the module to the Terraform configuration of your management account, in the region of that log group, and apply it.

4. Open Hunts in Fencer. One module instance covers every member account, because an organization trail collects all their events into one log group.

Troubleshooting

• AWS Monitor (Streaming) is missing from the integrations list — Streaming is not enabled for your organization. Contact us to turn it on.
• Generate token is disabled — The source is still provisioning. Wait for the Ready badge.
• Provisioning fails or does not complete — Click Retry provisioning. Contact support if it keeps failing.
• No events in Hunts — Check the monitoring tab and the error logs of the Firehose stream. Confirm that the endpoint URL matches the one in Fencer, that the access key is an active token, and that the subscription filter sits on the log group your trail delivers to.
• Records in the failed-delivery bucket — Firehose could not deliver them to Fencer. Check whether the token on the stream was revoked.
• Rows in Hunts show an error and no OCSF fields — For VPC Flow Logs, the flow log fields are not the full field set of one version, or they are in a different order. AWS does not let you change a flow log after you create it, so create a new flow log with the full field set of one version, in the order of Accepted flow log formats, and delete the old one. With Terraform, set vpc_flow_log_format to the default or to a string from the module’s examples/vpc-flow-logs/formats.tf, and apply again.
• VPC Flow Logs rows show values in the wrong fields — The flow log has the fields of one version in a different order. Fencer reads the fields by position and cannot detect a swap of two fields of the same kind, such as source and destination. Create a new flow log with the fields in the order of Accepted flow log formats.
• ALB logs: the stream delivers, but Hunts shows no events — The log delivery probably uses the plain output format. Fencer accepts only JSON for ALB logs. Set the output format of the delivery destination to JSON.
• ALB ingestion errors say request_line or listener_port is missing — The source receives the other log type. A source takes one log type. Create a second source for the other type, with its own token and its own Firehose stream, and point the delivery of that log type at the new stream.
• ALB logs stopped after a Terraform apply — The apply removed the LogDeliveryEnabled tag from the stream. Module 1.3 sets the tag when alb_load_balancers is set. With an older module or a hand-written stream, add LogDeliveryEnabled = "true" to the stream tags and apply again.
• NLB logs: no entries arrive — A Network Load Balancer writes entries only for connections on a TLS listener. A load balancer with only TCP or UDP listeners writes none. Check the listeners of the load balancer.
• NLB logs: the stream delivers, but Hunts shows no events — The log delivery probably uses the plain output format. Fencer accepts only JSON for NLB logs. Set the output format of the delivery destination to JSON.
• NLB ingestion errors say not an NLB access log record — The source receives ALB logs. A source takes one log type. Create a source of an ALB type for them, with its own token and its own Firehose stream, and point the ALB log delivery at the new stream.
• NLB logs stopped after a Terraform apply — The apply removed the LogDeliveryEnabled tag from the stream. Module 1.4 sets the tag when nlb_load_balancers is set. With an older module or a hand-written stream, add LogDeliveryEnabled = "true" to the stream tags and apply again.
• Cannot create the subscription filter — A log group allows five subscription filters, and AWS does not raise this quota. To see what already uses them, run aws logs describe-subscription-filters --log-group-name <log-group>.

Related pages

• Security Monitoring (SIEM)
• AWS Log Sources
• AWS Integration
• Threat Hunting