Kafka Consumer Trigger¶
Kafka Consumer Trigger starts a workflow when a message arrives on a configured Kafka topic. Use it to process events from other systems without someone manually launching the flow.
- Category: Triggers
- Purpose: Starts the workflow when a message arrives on a Kafka topic
- Inputs: None
- Outputs:
kafka_data- The received message with its key, topic, and metadata - Configuration — Required:
- Credential: Kafka brokers and client configuration
- Topic: Topic to listen on. Must match the producer's topic exactly
- Group ID: Consumer group. Consumers sharing a group id divide the messages between them; consumers in different groups each receive a copy
- Configuration — Behaviour:
- Allow Topic Creation: Default on. Creates the topic if it does not exist. Usually disabled in production so a typo cannot create a stray topic
- Read Messages From Beginning: Default off. When enabled, the entire message history is replayed on every start
- Only Message: Default off. Returns just the message value, without topic and metadata
- Return Headers: Default on
- JSON Parse Message: Default on. Parses the message body as JSON so its fields become addressable
- Keep Message as Binary Data: For non-text payloads
- Configuration — Performance:
- Auto Commit Threshold (1), Auto Commit Interval (5000 ms), Batch Size (1048576), Fetch Max Bytes (52428800), Fetch Min Bytes (1), Max Number of Requests (500), Heartbeat Interval (3000 ms), Session Timeout (45000 ms), Rebalance Timeout (60000 ms), Retry Delay on Error (1000 ms). These are standard Kafka settings with sensible defaults — leave them alone until you have a measured reason to change them
- Usage: The listener attaches shortly after the workflow is activated, not instantly. Messages published before it connects are missed unless Read Messages From Beginning is enabled.