Custom Source
Overview
Section titled “Overview”A custom source takes JSON events from any system you run and lands them in Fencer’s SIEM in near-real time. Your application, a log shipper, or an Amazon Data Firehose stream posts the events to an HTTPS endpoint that belongs to your Fencer organization. A transformation that you write in Vector Remap Language (VRL) shapes each event before Fencer stores it, and the events appear in Threat Hunting a few seconds later.
Use a custom source for logs that no built-in integration covers: your own services, an internal tool, a vendor that can post webhooks, or a log pipeline such as Vector or Fluent Bit. For AWS service logs, use the AWS Streaming Monitor instead, which comes with a transformation for each log type.
Prerequisites
Section titled “Prerequisites”- Custom sources enabled for your organization.
- Admin role in Fencer.
- An active subscription with the SIEM feature enabled.
- A system that can send HTTPS POST requests with a bearer token, or an AWS account in which you can create a Firehose stream.
- A few sample events from that system, as JSON, to test the transformation against.
How it works
Section titled “How it works”Your system posts JSON events to the Fencer ingestion endpoint. Every request carries an access token that identifies your organization and the source, so the endpoint URL is the same for every custom source. Fencer checks the token, writes the events to its ingestion stream, runs your transformation on each event, and stores the result in a table that belongs to this source.
The transformation is the part that makes the source yours. Fencer has no knowledge of your event format, so the program you save decides which fields to keep, rename, or compute. It must set two fields on every event: the event id and the event timestamp. Everything else it emits is available in Threat Hunting and to detection rules.
The setup has three parts, all in Fencer: create the source, save a transformation, and generate a token. Then you point your system at the endpoint.
Usage / Configuration
Section titled “Usage / Configuration”Create the source in Fencer
Section titled “Create the source in Fencer”- Go to Admin > Integrations and find the Custom Source integration.
- Click Connect.
- Enter a Display name that says where the events come from, such as
auth-api production logs, and click Create source. - Wait for the status badge to change from Provisioning to Ready.
The source page shows two endpoints under Endpoints: HTTP push for clients that post directly, and Firehose for an Amazon Data Firehose stream. Both feed the same source and accept the same tokens.
Write the transformation
Section titled “Write the transformation”A new custom source has no transformation, and it cannot generate a token until you save one.
- On the source page, click Edit transformation under Transformation.
- Under Sample events, paste one of your events into the empty
event_1sample, as one JSON object. Click add for each further sample. Use real events from the system you plan to connect. - Write the program in the VRL transformation editor. It must set
metadata.uidto the event id andtime_dtto the event timestamp, as an ISO 8601 string, on every event. A timestamp with an offset is converted to UTC. Atime_dtthat does not parse is replaced with the ingestion time. - Click Test on samples. The Output card shows each sample after the transformation, or the error the program raised on it.
- Click Save.
A field that is missing from an input event reads as null in VRL, so set the two required fields
with a fallback. This program takes an event with event_id and timestamp fields and keeps the
rest of the event as it is:
.metadata.uid = string(.event_id) ?? uuid_v4().time_dt = string(.timestamp) ?? to_string(now())A program that parses a timestamp in another format converts it first:
.metadata.uid = string(.id) ?? uuid_v4()ts, err = parse_timestamp(.ts, "%Y-%m-%d %H:%M:%S")if err == null { .time_dt = format_timestamp!(ts, "%+")} else { .time_dt = to_string(now())}The editor refuses to save a program that can leave either field unset or null on some event. If
the program rebuilds the whole event, for example with . = compact(.), assign the two fields after
that step.
Fencer stores the transformed event, so the field names in the output are the names you search for
in Threat Hunting. Name them after the OCSF attributes where one fits,
such as actor.user.name or src_endpoint.ip, so that Threat Hunting shows them in the same place
as events from other sources.
Fencer checks the type of a few top-level OCSF fields on the transformed event: time must be
epoch milliseconds, and class_uid, category_uid, type_uid, activity_id and severity_id
must be integers. A source field with one of these names and another type, such as a time string
in an application log, records an error on every row. Rename or drop such a field, for example with
del(.time), or convert it to the OCSF type.
Generate a token
Section titled “Generate a token”- On the source page, under Access tokens, click Generate token. In the Generate access
token dialog, enter a Token label that says where you will use the token, such as
production-ingestoredge-collector-eu, and click Generate token. - Copy the token and store it in your secrets manager. Then check I’ve stored this token in a safe place and click Done.
- Copy the URL under HTTP push, or under Firehose if you send events from a Firehose stream.
Send events over HTTPS
Section titled “Send events over HTTPS”Post the events to the HTTP push URL with the token in the Authorization header:
curl -X POST <your HTTP push URL> \ -H "Authorization: Bearer $FENCER_INGEST_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "event_id": "evt_2026-05-20T10:30:00Z_8f3c", "timestamp": "2026-05-20T10:30:00Z", "message": "user alice signed in", "severity": "INFO", "service": "auth-api" }'The endpoint accepts three body formats:
| Content type | Body |
|---|---|
application/json |
One JSON object, or a JSON array of objects, one event per element |
application/x-ndjson (also application/jsonl) |
One JSON object per line |
application/json with a Firehose access-key header |
The Firehose HTTP endpoint envelope (see below) |
A body may be gzip-compressed, with Content-Encoding: gzip. Send events in batches where you can:
one array or one NDJSON body of up to a few hundred events costs one request.
The response to a successful request is 200 with a summary:
{ "status": "ingested", "ingested": 2, "skipped": 1, "failures": [{ "index": 2, "error": "element is not a JSON object" }]}ingested counts the events Fencer accepted. skipped counts array elements or lines it could not
read as a JSON object, and failures says which ones and why. Fencer accepts the rest of the batch,
so a client does not need to retry a request that returns 200. A request with a missing, revoked,
or unknown token returns 401. A body that is not valid JSON at all returns 500 on every attempt,
so fix the body instead of retrying. A transient failure in the pipeline also returns 500; retry
those with a delay.
Send events from Amazon Data Firehose
Section titled “Send events from Amazon Data Firehose”An Amazon Data Firehose stream with an HTTP Endpoint destination can deliver to the Firehose
URL. Create the stream in the AWS console with the same settings as for the
AWS Streaming Monitor, with the Firehose
URL from the custom source page as the HTTP endpoint URL and the token as the access key. Firehose
sends the access key in its own header, so there is no Authorization header to set.
Each record in the stream is one event: a JSON object, or one JSON object per line when a record
holds several. Records that come from a CloudWatch Logs subscription filter arrive in the CloudWatch
envelope, and Fencer unwraps them: a message that is a JSON object becomes the event, and any other
message becomes an event with the text under message. Your transformation then parses that text.
Firehose retries a delivery that Fencer answers with 401 or 500, and writes a delivery that
keeps failing to the backup bucket of the stream. Fencer acknowledges a delivery with 200 even
when some of its records cannot be read as JSON objects. Those records are dropped: they do not
reach the backup bucket, and the acknowledgement carries no failure list. Test the record format
with the HTTP push endpoint first, which reports every record it could not read.
Verify the source
Section titled “Verify the source”Post a few events and open Operations > Threat Hunting. The events appear within about a minute, with the Data Source Type Application Logs and the Data Source set to the display name of your source. Open an event to see the fields your transformation emitted.
An event that the transformation could not process is stored too, without the fields your program sets. Threat Hunting does not show the error. To find the cause, paste that event as a sample in the editor, click Test on samples, fix the program, and save it again.
Rotate or revoke a token
Section titled “Rotate or revoke a token”A source can hold more than one active token. To rotate a token, generate a new one, deploy it to the system that sends the events, and then click Revoke next to the old token.
Remove the source
Section titled “Remove the source”Remove the connection with the delete action on its card in your integration settings. Fencer
revokes every active token for the source and stops accepting its events. Stop the system that sends
them as well, or it keeps receiving 401 responses, and a Firehose stream keeps writing rejected
records to its backup bucket.
Detection rules
Section titled “Detection rules”A custom detection rule reaches a custom source through the Sigma logsource product: application
and service: logs. The field names in the rule’s detection are the paths your transformation
emits, such as severity or actor.user.name. Two names are reserved: timestamp is the row’s
timestamp, taken from time_dt, and event_id is the row’s id, taken from metadata.uid. A
payload field with one of those names is not reachable under it. See
Detection Rules for how to add and manage custom rules.
Examples
Section titled “Examples”Shipping application logs with Vector:
-
In Fencer, create a source named
auth-api production logs, save a transformation that setsmetadata.uidandtime_dtfrom your log fields, and generate a token labeledvector-prod. -
Add an
httpsink to your Vector configuration that posts newline-delimited JSON with the token:[sinks.fencer]type = "http"inputs = ["auth_api_logs"]uri = "<your HTTP push URL>"method = "post"encoding.codec = "json"framing.method = "newline_delimited"auth.strategy = "bearer"auth.token = "${FENCER_INGEST_TOKEN}" -
Open Operations > Threat Hunting and filter on Data Source is
auth-api production logs.
Posting a batch from a script:
curl -X POST <your HTTP push URL> \ -H "Authorization: Bearer $FENCER_INGEST_TOKEN" \ -H "Content-Type: application/json" \ -d '[ {"event_id": "a1", "timestamp": "2026-05-20T10:30:00Z", "message": "user alice signed in"}, {"event_id": "a2", "timestamp": "2026-05-20T10:30:04Z", "message": "user alice changed password"} ]'Troubleshooting
Section titled “Troubleshooting”- Custom Source is missing from the integrations list — Custom sources are not enabled for your organization. Contact us to turn them on.
- Generate token is disabled — The source is still provisioning, or it has no saved transformation yet. Wait for the Ready badge, then write the transformation.
- Provisioning fails or does not complete — Click Retry provisioning. Contact support if it keeps failing.
- The editor will not save the transformation — The program can leave
metadata.uidortime_dtunset or null on some event. Set both with a fallback, such asstring(.id) ?? uuid_v4()andstring(.ts) ?? to_string(now()), after any step that rebuilds the whole event. - Requests return 401 — The token is missing or revoked, or the header is malformed. It must be
exactly
Authorization: Bearer <token>with the full token. A token from another custom source does not fail: every source shares the endpoint, and the token decides where the events land. If events are missing, check that the token is listed under Access tokens on this source’s page. - Requests return 200 but
skippedis not 0 — Some array elements or lines are not JSON objects. Thefailureslist names them by index or line number. Send one object per element or per line. - Requests return 500 — The body is not valid JSON, or the ingestion pipeline had a transient failure. Check the body with a JSON validator first: an invalid body fails on every attempt. Retry with a delay only when the body is valid.
- No events in Threat Hunting — Confirm that the request returned
200withingestedgreater than 0, that the URL is the one from the source page, and that the time range in Threat Hunting covers thetime_dtyour transformation set. A timestamp in the past or the future moves the event out of the default range. - Rows show the original event instead of your fields — The transformation raised an error on those events. Threat Hunting does not show the error text. Paste one of them as a sample in the editor, click Test on samples to see the error, fix the program, and save. Events that arrive after the save use the new program.
- Events have a random id or the ingestion time — The transformation left
metadata.uidnull, or lefttime_dtnull or set it to a value that does not parse as a timestamp. The editor stops the null case for new saves. Review a program saved before that check and set both fields with a fallback, and check the timestamp format with Test on samples. - Firehose delivers, but some events are missing — Records that are not JSON objects are dropped without a failure signal. Send the same records to the HTTP push endpoint, which lists every record it could not read, and fix the format.
- Firehose keeps retrying — Fencer answered with
401or500. Check that the access key on the stream is an active token and that the HTTP endpoint URL is the Firehose URL from the source page.