Skip to content

Custom Source

A custom source takes JSON events from any system you run and lands them in Fencer’s SIEM in near-real time. Your application, a log shipper, or an Amazon Data Firehose stream posts the events to an HTTPS endpoint that belongs to your Fencer organization. A transformation that you write in Vector Remap Language (VRL) shapes each event before Fencer stores it, and the events appear in Threat Hunting a few seconds later.

Use a custom source for logs that no built-in integration covers: your own services, an internal tool, a vendor that can post webhooks, or a log pipeline such as Vector or Fluent Bit. For AWS service logs, use the AWS Streaming Monitor instead, which comes with a transformation for each log type.

  • Custom sources enabled for your organization.
  • Admin role in Fencer.
  • An active subscription with the SIEM feature enabled.
  • A system that can send HTTPS POST requests with a bearer token, or an AWS account in which you can create a Firehose stream.
  • A few sample events from that system, as JSON, to test the transformation against.

Your system posts JSON events to the Fencer ingestion endpoint. Every request carries an access token that identifies your organization and the source, so the endpoint URL is the same for every custom source. Fencer checks the token, writes the events to its ingestion stream, runs your transformation on each event, and stores the result in a table that belongs to this source.

The transformation is the part that makes the source yours. Fencer has no knowledge of your event format, so the program you save decides which fields to keep, rename, or compute. It must set two fields on every event: the event id and the event timestamp. Everything else it emits is available in Threat Hunting and to detection rules.

The setup has three parts, all in Fencer: create the source, save a transformation, and generate a token. Then you point your system at the endpoint.

  1. Go to Admin > Integrations and find the Custom Source integration.
  2. Click Connect.
  3. Enter a Display name that says where the events come from, such as auth-api production logs, and click Create source.
  4. Wait for the status badge to change from Provisioning to Ready.

The source page shows two endpoints under Endpoints: HTTP push for clients that post directly, and Firehose for an Amazon Data Firehose stream. Both feed the same source and accept the same tokens.

A new custom source has no transformation, and it cannot generate a token until you save one.

  1. On the source page, click Edit transformation under Transformation.
  2. Under Sample events, paste one of your events into the empty event_1 sample, as one JSON object. Click add for each further sample. Use real events from the system you plan to connect.
  3. Write the program in the VRL transformation editor. It must set metadata.uid to the event id and time_dt to the event timestamp, as an ISO 8601 string, on every event. A timestamp with an offset is converted to UTC. A time_dt that does not parse is replaced with the ingestion time.
  4. Click Test on samples. The Output card shows each sample after the transformation, or the error the program raised on it.
  5. Click Save.

A field that is missing from an input event reads as null in VRL, so set the two required fields with a fallback. This program takes an event with event_id and timestamp fields and keeps the rest of the event as it is:

.metadata.uid = string(.event_id) ?? uuid_v4()
.time_dt = string(.timestamp) ?? to_string(now())

A program that parses a timestamp in another format converts it first:

.metadata.uid = string(.id) ?? uuid_v4()
ts, err = parse_timestamp(.ts, "%Y-%m-%d %H:%M:%S")
if err == null {
.time_dt = format_timestamp!(ts, "%+")
} else {
.time_dt = to_string(now())
}

The editor refuses to save a program that can leave either field unset or null on some event. If the program rebuilds the whole event, for example with . = compact(.), assign the two fields after that step.

Fencer stores the transformed event, so the field names in the output are the names you search for in Threat Hunting. Name them after the OCSF attributes where one fits, such as actor.user.name or src_endpoint.ip, so that Threat Hunting shows them in the same place as events from other sources.

Fencer checks the type of a few top-level OCSF fields on the transformed event: time must be epoch milliseconds, and class_uid, category_uid, type_uid, activity_id and severity_id must be integers. A source field with one of these names and another type, such as a time string in an application log, records an error on every row. Rename or drop such a field, for example with del(.time), or convert it to the OCSF type.

  1. On the source page, under Access tokens, click Generate token. In the Generate access token dialog, enter a Token label that says where you will use the token, such as production-ingest or edge-collector-eu, and click Generate token.
  2. Copy the token and store it in your secrets manager. Then check I’ve stored this token in a safe place and click Done.
  3. Copy the URL under HTTP push, or under Firehose if you send events from a Firehose stream.

Post the events to the HTTP push URL with the token in the Authorization header:

Terminal window
curl -X POST <your HTTP push URL> \
-H "Authorization: Bearer $FENCER_INGEST_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"event_id": "evt_2026-05-20T10:30:00Z_8f3c",
"timestamp": "2026-05-20T10:30:00Z",
"message": "user alice signed in",
"severity": "INFO",
"service": "auth-api"
}'

The endpoint accepts three body formats:

Content type Body
application/json One JSON object, or a JSON array of objects, one event per element
application/x-ndjson (also application/jsonl) One JSON object per line
application/json with a Firehose access-key header The Firehose HTTP endpoint envelope (see below)

A body may be gzip-compressed, with Content-Encoding: gzip. Send events in batches where you can: one array or one NDJSON body of up to a few hundred events costs one request.

The response to a successful request is 200 with a summary:

{
"status": "ingested",
"ingested": 2,
"skipped": 1,
"failures": [{ "index": 2, "error": "element is not a JSON object" }]
}

ingested counts the events Fencer accepted. skipped counts array elements or lines it could not read as a JSON object, and failures says which ones and why. Fencer accepts the rest of the batch, so a client does not need to retry a request that returns 200. A request with a missing, revoked, or unknown token returns 401. A body that is not valid JSON at all returns 500 on every attempt, so fix the body instead of retrying. A transient failure in the pipeline also returns 500; retry those with a delay.

An Amazon Data Firehose stream with an HTTP Endpoint destination can deliver to the Firehose URL. Create the stream in the AWS console with the same settings as for the AWS Streaming Monitor, with the Firehose URL from the custom source page as the HTTP endpoint URL and the token as the access key. Firehose sends the access key in its own header, so there is no Authorization header to set.

Each record in the stream is one event: a JSON object, or one JSON object per line when a record holds several. Records that come from a CloudWatch Logs subscription filter arrive in the CloudWatch envelope, and Fencer unwraps them: a message that is a JSON object becomes the event, and any other message becomes an event with the text under message. Your transformation then parses that text.

Firehose retries a delivery that Fencer answers with 401 or 500, and writes a delivery that keeps failing to the backup bucket of the stream. Fencer acknowledges a delivery with 200 even when some of its records cannot be read as JSON objects. Those records are dropped: they do not reach the backup bucket, and the acknowledgement carries no failure list. Test the record format with the HTTP push endpoint first, which reports every record it could not read.

Post a few events and open Operations > Threat Hunting. The events appear within about a minute, with the Data Source Type Application Logs and the Data Source set to the display name of your source. Open an event to see the fields your transformation emitted.

An event that the transformation could not process is stored too, without the fields your program sets. Threat Hunting does not show the error. To find the cause, paste that event as a sample in the editor, click Test on samples, fix the program, and save it again.

A source can hold more than one active token. To rotate a token, generate a new one, deploy it to the system that sends the events, and then click Revoke next to the old token.

Remove the connection with the delete action on its card in your integration settings. Fencer revokes every active token for the source and stops accepting its events. Stop the system that sends them as well, or it keeps receiving 401 responses, and a Firehose stream keeps writing rejected records to its backup bucket.

A custom detection rule reaches a custom source through the Sigma logsource product: application and service: logs. The field names in the rule’s detection are the paths your transformation emits, such as severity or actor.user.name. Two names are reserved: timestamp is the row’s timestamp, taken from time_dt, and event_id is the row’s id, taken from metadata.uid. A payload field with one of those names is not reachable under it. See Detection Rules for how to add and manage custom rules.

Shipping application logs with Vector:

  1. In Fencer, create a source named auth-api production logs, save a transformation that sets metadata.uid and time_dt from your log fields, and generate a token labeled vector-prod.

  2. Add an http sink to your Vector configuration that posts newline-delimited JSON with the token:

    [sinks.fencer]
    type = "http"
    inputs = ["auth_api_logs"]
    uri = "<your HTTP push URL>"
    method = "post"
    encoding.codec = "json"
    framing.method = "newline_delimited"
    auth.strategy = "bearer"
    auth.token = "${FENCER_INGEST_TOKEN}"
  3. Open Operations > Threat Hunting and filter on Data Source is auth-api production logs.

Posting a batch from a script:

Terminal window
curl -X POST <your HTTP push URL> \
-H "Authorization: Bearer $FENCER_INGEST_TOKEN" \
-H "Content-Type: application/json" \
-d '[
{"event_id": "a1", "timestamp": "2026-05-20T10:30:00Z", "message": "user alice signed in"},
{"event_id": "a2", "timestamp": "2026-05-20T10:30:04Z", "message": "user alice changed password"}
]'
  • Custom Source is missing from the integrations list — Custom sources are not enabled for your organization. Contact us to turn them on.
  • Generate token is disabled — The source is still provisioning, or it has no saved transformation yet. Wait for the Ready badge, then write the transformation.
  • Provisioning fails or does not complete — Click Retry provisioning. Contact support if it keeps failing.
  • The editor will not save the transformation — The program can leave metadata.uid or time_dt unset or null on some event. Set both with a fallback, such as string(.id) ?? uuid_v4() and string(.ts) ?? to_string(now()), after any step that rebuilds the whole event.
  • Requests return 401 — The token is missing or revoked, or the header is malformed. It must be exactly Authorization: Bearer <token> with the full token. A token from another custom source does not fail: every source shares the endpoint, and the token decides where the events land. If events are missing, check that the token is listed under Access tokens on this source’s page.
  • Requests return 200 but skipped is not 0 — Some array elements or lines are not JSON objects. The failures list names them by index or line number. Send one object per element or per line.
  • Requests return 500 — The body is not valid JSON, or the ingestion pipeline had a transient failure. Check the body with a JSON validator first: an invalid body fails on every attempt. Retry with a delay only when the body is valid.
  • No events in Threat Hunting — Confirm that the request returned 200 with ingested greater than 0, that the URL is the one from the source page, and that the time range in Threat Hunting covers the time_dt your transformation set. A timestamp in the past or the future moves the event out of the default range.
  • Rows show the original event instead of your fields — The transformation raised an error on those events. Threat Hunting does not show the error text. Paste one of them as a sample in the editor, click Test on samples to see the error, fix the program, and save. Events that arrive after the save use the new program.
  • Events have a random id or the ingestion time — The transformation left metadata.uid null, or left time_dt null or set it to a value that does not parse as a timestamp. The editor stops the null case for new saves. Review a program saved before that check and set both fields with a fallback, and check the timestamp format with Test on samples.
  • Firehose delivers, but some events are missing — Records that are not JSON objects are dropped without a failure signal. Send the same records to the HTTP push endpoint, which lists every record it could not read, and fix the format.
  • Firehose keeps retrying — Fencer answered with 401 or 500. Check that the access key on the stream is an active token and that the HTTP endpoint URL is the Firehose URL from the source page.