Messaging Queue Triggers

Messaging queue triggers send your scraped data to message queue systems. Message queues are used in modern software architectures to pass data between different services, applications, and systems. They act as a “buffer” that receives messages and delivers them to consumers when they’re ready.

RTILA X supports three messaging queue systems:

Trigger	System	Best For
RabbitMQ	RabbitMQ	General-purpose message queuing, microservices
Kafka	Apache Kafka	High-throughput data streaming, log processing
AWS SQS	Amazon Simple Queue Service	AWS-native applications, serverless architectures

What Are Message Queues?

If you’re new to message queues, here’s a simple explanation:

A message queue is like a mailbox. Your automation (the sender) drops a message into the queue. Another application (the receiver) picks up the message when it’s ready. The queue sits in between, ensuring messages are delivered reliably even if the receiver is temporarily busy.

Why use message queues instead of direct API calls?

Feature	Direct API Call	Message Queue
Receiver must be online	✅ Yes	❌ No — messages wait in the queue
Handles traffic spikes	❌ Can overload the receiver	✅ Queue absorbs the spike
Retry on failure	❌ Must implement manually	✅ Built-in retry mechanisms
Multiple consumers	❌ One-to-one only	✅ Many consumers can read from one queue
Guaranteed delivery	❌ Depends on the API	✅ Built-in delivery guarantees

RabbitMQ

RabbitMQ is the most popular open-source message broker. It is used by companies of all sizes to pass messages between applications.

Description

The RabbitMQ trigger publishes a message to a RabbitMQ queue. The message contains your scraped data (or a summary of it) and is picked up by whatever application is listening on that queue.

Parameters

Parameter	Type	Required	Default	Description
amqp_url	string	✅ Yes	—	The AMQP connection URL for your RabbitMQ server. Format: amqp://username:password@hostname:port/vhost
queue_name	string	✅ Yes	—	The name of the queue to publish to. Example: scraping_results
message_body	string	✅ Yes	—	The message content to publish. Supports ${variable} substitution.

AMQP URL Format

The AMQP URL follows this format:

amqp://username:password@hostname:port/vhost

Part	Description	Example
amqp://	Protocol (use amqps:// for TLS)	amqp://
username	RabbitMQ username	admin
password	RabbitMQ password	secret123
hostname	RabbitMQ server address	rabbit.example.com
port	RabbitMQ port (default: 5672)	5672
vhost	Virtual host (default: /)	/

Examples

Publish scraped data to a queue:

{
  "amqp_url": "amqp://admin:secret123@rabbit.example.com:5672/",
  "queue_name": "scraping_results",
  "message_body": "${dataset}"
}

Publish a summary message:

{
  "amqp_url": "amqp://admin:secret123@rabbit.example.com:5672/",
  "queue_name": "scraping_alerts",
  "message_body": "Scrape completed: ${record_count} records from ${target_url}"
}

Publish with TLS encryption:

{
  "amqp_url": "amqps://admin:secret123@rabbit.example.com:5671/",
  "queue_name": "scraping_results",
  "message_body": "${dataset}"
}

Behavior Notes

• The message is published to the specified queue. If the queue doesn’t exist, RabbitMQ will create it automatically (if configured to do so).
• The message body is sent as a string. If you’re sending JSON data, make sure it’s properly formatted.
• Use amqps:// instead of amqp:// for TLS-encrypted connections.
• The default RabbitMQ port is 5672 (or 5671 for TLS).
• If the RabbitMQ server is unreachable, the trigger will fail and the error will be logged.

---

Kafka

Apache Kafka is a distributed event streaming platform used for high-throughput data pipelines. It is used by companies like LinkedIn, Netflix, and Uber for real-time data processing.

Description

The Kafka trigger publishes a message to a Kafka topic. Kafka is designed for very high throughput — it can handle millions of messages per second. This makes it ideal for large-scale scraping operations where you need to stream data to downstream systems in real time.

Parameters

Parameter	Type	Required	Default	Description
bootstrap_servers	string	✅ Yes	—	Comma-separated list of Kafka broker addresses. Format: hostname:port. Example: kafka1:9092,kafka2:9092
topic	string	✅ Yes	—	The Kafka topic to publish to. Example: scraping-events
message	string	✅ Yes	—	The message content to publish. Supports ${variable} substitution.

Kafka Concepts

Concept	Description
Broker	A Kafka server. A Kafka cluster has one or more brokers.
Topic	A named stream of messages. Think of it like a channel or category.
Producer	The application that sends messages (your RTILA automation).
Consumer	The application that reads messages from the topic.
Bootstrap Servers	The list of broker addresses your client connects to.

Examples

Publish to a single-broker Kafka cluster:

{
  "bootstrap_servers": "kafka.example.com:9092",
  "topic": "scraping-events",
  "message": "${dataset}"
}

Publish to a multi-broker Kafka cluster:

{
  "bootstrap_servers": "kafka1:9092,kafka2:9092,kafka3:9092",
  "topic": "product-data",
  "message": "${dataset}"
}

Publish a summary message:

{
  "bootstrap_servers": "kafka.example.com:9092",
  "topic": "scraping-alerts",
  "message": "Scrape completed: ${record_count} records extracted from ${target_url}"
}

Behavior Notes

• Kafka topics are created automatically if they don’t exist (depending on your Kafka configuration).
• The message is published to the specified topic. Consumers listening on that topic will receive it.
• For high-availability setups, provide multiple bootstrap servers separated by commas.
• Kafka is designed for high throughput. It can handle very large volumes of data.
• The default Kafka port is 9092.
• If all bootstrap servers are unreachable, the trigger will fail.

---

AWS SQS

Amazon Simple Queue Service (SQS) is AWS’s managed message queue service. It requires no server management — AWS handles everything for you.

Description

The AWS SQS trigger sends a message to an Amazon SQS queue. SQS is a fully managed service, meaning you don’t need to install or maintain any software. You just create a queue in the AWS console and start sending messages.

Parameters

Parameter	Type	Required	Default	Description
aws_access_key_id	string	✅ Yes	—	Your AWS access key ID. Found in the AWS IAM console.
aws_secret_access_key	string	✅ Yes	—	Your AWS secret access key. Found in the AWS IAM console.
region_name	string	✅ Yes	—	The AWS region where your SQS queue is located. Example: us-east-1, eu-west-1
queue_url	string	✅ Yes	—	The full URL of your SQS queue. Found in the AWS SQS console.
message_body	string	✅ Yes	—	The message content to send. Supports ${variable} substitution.

AWS SQS Queue URL Format

The SQS queue URL looks like this:

https://sqs.us-east-1.amazonaws.com/123456789012/my-queue

You can find this URL in the AWS SQS console by selecting your queue.

AWS Regions

Region Code	Location
us-east-1	US East (N. Virginia)
us-west-2	US West (Oregon)
eu-west-1	EU (Ireland)
eu-central-1	EU (Frankfurt)
ap-southeast-1	Asia Pacific (Singapore)
ap-northeast-1	Asia Pacific (Tokyo)

Examples

Send scraped data to SQS:

{
  "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
  "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "region_name": "us-east-1",
  "queue_url": "https://sqs.us-east-1.amazonaws.com/123456789012/scraping-results",
  "message_body": "${dataset}"
}

Send a notification message:

{
  "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
  "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "region_name": "eu-west-1",
  "queue_url": "https://sqs.eu-west-1.amazonaws.com/123456789012/scraping-alerts",
  "message_body": "Scrape completed: ${record_count} records from ${target_url}"
}

Behavior Notes

• The SQS queue must already exist in your AWS account. Create it in the AWS SQS console before using this trigger.
• The message is sent to the specified queue. Whatever application is polling that queue will receive it.
• The region_name must match the region where your queue was created.
• AWS credentials must have sqs:SendMessage permission on the target queue.
• The default AWS region is us-east-1. If your queue is in a different region, specify it.

AWS Security Best Practices

Practice	Why It Matters
Use IAM roles instead of access keys	More secure; no keys to manage or leak
Use the principle of least privilege	Give only sqs:SendMessage permission, not full SQS access
Rotate access keys regularly	Reduces the risk of compromised keys
Use AWS Secrets Manager	Store credentials securely instead of hardcoding them

---

Choosing the Right Message Queue

Scenario	Recommended System	Why
General-purpose messaging	RabbitMQ	Mature, feature-rich, easy to set up
High-throughput data streaming	Kafka	Designed for millions of messages per second
AWS-native applications	AWS SQS	No server management, integrates with AWS services
Serverless architectures	AWS SQS	Works with AWS Lambda, Step Functions
Microservices communication	RabbitMQ	Flexible routing, multiple consumers
Log processing / event sourcing	Kafka	Built for high-volume event streams
Simple, low-maintenance setup	AWS SQS	Fully managed, no infrastructure to manage
On-premises deployment	RabbitMQ or Kafka	Both can be self-hosted

Message Body Best Practices

Practice	Why It Matters
Send structured data (JSON)	Makes it easy for consumers to parse the message
Include metadata	Add timestamps, source URLs, and record counts for context
Keep messages reasonably sized	Very large messages can cause performance issues
Use variables for dynamic content	${dataset}, ${record_count}, ${target_url}

| Handle errors gracefully | If the queue is unreachable, the trigger will fail — check logs |

##