Data Stores & Search Triggers

Data store triggers let you read from and write to specialized data storage systems — caches, search engines, and enterprise databases. These systems serve different purposes than traditional relational databases: Redis is for fast caching and real-time data, Elasticsearch is for full-text search and analytics, Teradata and Sybase are enterprise-grade data warehouses.

RTILA X provides four data store triggers, each targeting a different type of storage system:

Trigger	System Type	Best For
Redis	In-memory cache / data store	Caching scraped data, real-time feeds, pub/sub messaging
Elasticsearch	Search engine / analytics	Full-text search, log analysis, data indexing
Teradata	Enterprise data warehouse	Large-scale enterprise analytics
Sybase ASE	Enterprise database	Legacy enterprise systems

Redis

Redis is an in-memory data store that is incredibly fast because it keeps data in RAM rather than on disk. It’s used for caching, session management, real-time analytics, message queues, and more. Think of Redis as a super-fast sticky note that applications can read from and write to instantly.

Description

The Redis trigger connects to a Redis server and performs one of several operations: storing a value, retrieving a value, adding to a list, reading from a list, publishing a message, or deleting a key. This makes it versatile for many use cases — from caching scraped data to feeding real-time dashboards.

Parameters

Parameter	Type	Required	Default	Description
host	string	✅ Yes	—	The hostname or IP address of your Redis server. Example: localhost, redis.company.com, or 192.168.1.100.
port	integer	❌ No	6379	The port your Redis server listens on. The default Redis port is 6379.
operation	string	✅ Yes	—	The operation to perform. Options: set, get, lpush, rpush, lrange, publish, delete.
password	string	❌ No	—	Your Redis password. Leave empty if your Redis server doesn’t require authentication.
db	integer	❌ No	0	The Redis database number. Redis supports 16 databases numbered 0–15. Default is 0.
key	string	❌ No	—	The Redis key for set, get, or delete operations. Example: scraped:product:123. Supports ${variable} substitution.
value	string	❌ No	—	The value to store (for set) or the item to push to a list (for lpush/rpush). Supports ${variable} substitution.
list_name	string	❌ No	—	The name of the Redis list for lpush, rpush, and lrange operations.
channel	string	❌ No	—	The pub/sub channel name for the publish operation.
message	string	❌ No	—	The message to publish to the channel (for publish). Supports ${variable} substitution.
start	integer	❌ No	0	The start index for lrange (read list) operation. 0 means start from the beginning.
end	integer	❌ No	-1	The end index for lrange. -1 means read to the end of the list.
ttl	integer	❌ No	—	Time-to-live in seconds. When set, the key automatically expires after this many seconds. Useful for caching.

Operations Reference

Operation	What It Does	Required Parameters	Example Use Case
set	Stores a value under a key	key, value	Cache a product price
get	Retrieves the value of a key	key	Read a cached value
lpush	Adds an item to the beginning of a list	list_name, value	Add a new item to a feed
rpush	Adds an item to the end of a list	list_name, value	Append to a queue
lrange	Reads items from a list	list_name	Read all items from a queue
publish	Publishes a message to a channel	channel, message	Notify subscribers of new data
delete	Removes a key and its value	key	Clear a cache entry

Example: Cache Scraped Data

{
  "host": "redis.company.com",
  "port": 6379,
  "operation": "set",
  "password": "redis-password",
  "key": "scraped:product:${product_id}",
  "value": "${product_data}",
  "ttl": 86400
}

This caches the scraped product data for 24 hours (86400 seconds).

Example: Add Items to a Redis List

{
  "host": "localhost",
  "port": 6379,
  "operation": "rpush",
  "list_name": "scraped_products_queue",
  "value": "${product_name}|${product_price}|${product_url}"
}

Example: Publish a Notification

{
  "host": "redis.company.com",
  "port": 6379,
  "operation": "publish",
  "channel": "scraping_alerts",
  "message": "Scrape completed: ${record_count} records from ${target_url}"
}

Example: Read All Items from a List

{
  "host": "localhost",
  "port": 6379,
  "operation": "lrange",
  "list_name": "scraped_products_queue",
  "start": 0,
  "end": -1
}

TTL (Time-to-Live) Explained

TTL tells Redis how long to keep a key before automatically deleting it. This is perfect for caching:

TTL Value	Duration	Use Case
60	1 minute	Very short-lived cache
3600	1 hour	Hourly cache
86400	24 hours	Daily cache
604800	7 days	Weekly cache
(empty)	No expiry	Data stays until manually deleted

Output

Operation	Output
set	OK if successful
get	The stored value, or null if the key doesn’t exist
lpush / rpush	The new length of the list
lrange	An array of list items
publish	The number of subscribers that received the message
delete	1 if the key was deleted, 0 if it didn’t exist

---

Elasticsearch

Elasticsearch is a powerful search and analytics engine. It’s used for full-text search, log analysis, monitoring, and data analytics. If you need to make your scraped data searchable or feed it into an analytics pipeline, Elasticsearch is the tool.

Description

The Elasticsearch trigger connects to an Elasticsearch cluster and can index documents (store data), search for documents, retrieve a specific document, delete a document, or bulk-index multiple documents at once.

Parameters

Parameter	Type	Required	Default	Description
host	string	✅ Yes	—	The hostname or IP address of your Elasticsearch server. Example: localhost, es.company.com.
port	integer	❌ No	9200	The HTTP port. Default Elasticsearch HTTP port is 9200.
operation	string	✅ Yes	—	The operation to perform. Options: index_doc, search, get_doc, delete_doc, bulk_index.
index_name	string	❌ No	—	The name of the Elasticsearch index (like a database table). Example: products, scraping_logs.
doc_id	string	❌ No	—	The unique ID of a document. Used with get_doc and delete_doc.
document	object	❌ No	—	The document body to index. Used with index_doc. Supports ${variable} substitution.
query	object	❌ No	—	The search query in Elasticsearch Query DSL. Used with search.
documents	list	❌ No	—	A list of documents for bulk_index. Use ${dataset} to index all scraped data.
use_ssl	boolean	❌ No	false	If true, connects using HTTPS instead of HTTP. Enable this for production clusters.
username	string	❌ No	—	Username for authentication. Required if your cluster has security enabled.
password	string	❌ No	—	Password for authentication.

Operations Reference

Operation	What It Does	Required Parameters	Example Use Case
index_doc	Stores a single document in an index	index_name, document	Index a scraped product
search	Searches for documents matching a query	index_name, query	Find products matching criteria
get_doc	Retrieves a specific document by ID	index_name, doc_id	Get a specific record
delete_doc	Deletes a specific document by ID	index_name, doc_id	Remove outdated data
bulk_index	Indexes multiple documents at once	index_name, documents	Bulk-load all scraped data

Example: Index Scraped Data

{
  "host": "es.company.com",
  "port": 9200,
  "operation": "index_doc",
  "index_name": "scraped_products",
  "document": {
    "name": "${product_name}",
    "price": "${product_price}",
    "url": "${product_url}",
    "scraped_at": "${date}"
  },
  "use_ssl": true,
  "username": "elastic",
  "password": "es-password"
}

Example: Bulk Index All Scraped Data

{
  "host": "es.company.com",
  "port": 9200,
  "operation": "bulk_index",
  "index_name": "scraped_products",
  "documents": "${dataset}",
  "use_ssl": true,
  "username": "elastic",
  "password": "es-password"
}

Example: Search for Documents

{
  "host": "es.company.com",
  "port": 9200,
  "operation": "search",
  "index_name": "scraped_products",
  "query": {
    "match": {
      "name": "wireless headphones"
    }
  },
  "use_ssl": true,
  "username": "elastic",
  "password": "es-password"
}

Output

Operation	Output
index_doc	The index result with document ID and status
search	An array of matching documents with scores
get_doc	The requested document
delete_doc	Confirmation of deletion
bulk_index	Summary of indexed documents and any errors

---

Teradata

Teradata is an enterprise-grade relational database management system designed for large-scale data warehousing and analytics. It’s used by large enterprises for business intelligence and data analytics. If your organization uses Teradata, this trigger lets you write scraped data directly to it.

Description

The Teradata trigger connects to a Teradata database and executes a SQL query. You can use INSERT, UPDATE, SELECT, or any other SQL statement supported by Teradata.

Parameters

Parameter	Type	Required	Default	Description
host	string	✅ Yes	—	The Teradata server hostname or IP address. Example: teradata.company.com.
user	string	✅ Yes	—	Your Teradata username.
password	string	✅ Yes	—	Your Teradata password.
database	string	✅ Yes	—	The Teradata database name.
query	string	✅ Yes	—	The SQL query to execute. Supports ${variable} substitution.
timeout	integer	❌ No	60	Connection timeout in seconds. Increase this for large queries or slow networks.

Example: Insert Scraped Data into Teradata

{
  "host": "teradata.company.com",
  "user": "etl_user",
  "password": "teradata-password",
  "database": "SCRAPING_DB",
  "query": "INSERT INTO products (name, price, url, scraped_at) VALUES ('${product_name}', ${product_price}, '${product_url}', CURRENT_TIMESTAMP)",
  "timeout": 120
}

Example: Query Teradata for Verification

{
  "host": "teradata.company.com",
  "user": "etl_user",
  "password": "teradata-password",
  "database": "SCRAPING_DB",
  "query": "SELECT COUNT(*) AS total FROM products WHERE scraped_at >= CURRENT_DATE"
}

Output

Query Type	Output
SELECT	An array of row objects
INSERT / UPDATE / DELETE	Number of affected rows

---

Sybase ASE

Sybase ASE (Adaptive Server Enterprise) is a relational database management system commonly found in enterprise environments, particularly in banking, finance, and telecommunications. If your organization still runs Sybase, this trigger lets you write scraped data to it.

Description

The Sybase ASE trigger connects to a Sybase database and executes a SQL query. You can use INSERT, UPDATE, SELECT, or any other SQL statement supported by Sybase ASE.

Parameters

Parameter	Type	Required	Default	Description
host	string	✅ Yes	—	The Sybase server hostname or IP address. Example: sybase.company.com.
port	integer	❌ No	5000	The Sybase server port. Default is 5000.
user	string	✅ Yes	—	Your Sybase username.
password	string	✅ Yes	—	Your Sybase password.
database	string	✅ Yes	—	The Sybase database name.
query	string	✅ Yes	—	The SQL query to execute. Supports ${variable} substitution.
timeout	integer	❌ No	30	Connection timeout in seconds.

Example: Insert Scraped Data into Sybase

{
  "host": "sybase.company.com",
  "port": 5000,
  "user": "etl_user",
  "password": "sybase-password",
  "database": "scraping_db",
  "query": "INSERT INTO products (name, price, url) VALUES ('${product_name}', ${product_price}, '${product_url}')",
  "timeout": 60
}

Example: Query Sybase

{
  "host": "sybase.company.com",
  "port": 5000,
  "user": "etl_user",
  "password": "sybase-password",
  "database": "scraping_db",
  "query": "SELECT name, price FROM products WHERE price > 50"
}

Output

Query Type	Output
SELECT	An array of row objects
INSERT / UPDATE / DELETE	Number of affected rows

---

Choosing the Right Data Store Trigger

Scenario	Recommended Trigger
Caching scraped data for fast access	Redis
Feeding a real-time dashboard or feed	Redis (pub/sub)
Making scraped data searchable	Elasticsearch
Feeding log analysis or monitoring	Elasticsearch
Writing to an enterprise data warehouse	Teradata
Writing to a legacy enterprise database	Sybase ASE
Bulk-loading large datasets	Elasticsearch (bulk_index)
Temporary data storage with auto-expiry	Redis (with TTL)

Tips and Best Practices

Tip	Why It Matters
Use TTL for Redis caching	Prevents stale data from accumulating forever
Use bulk_index for Elasticsearch	Much faster than indexing documents one at a time
Test credentials before running	Verify your connection works before adding to production
Use ${variable} substitution	Makes your queries dynamic and reusable
Increase timeout for large queries	Default timeouts may be too short for complex operations
Use SSL/TLS for production	Enable use_ssl for Elasticsearch in production environments
Store credentials as project variables	Avoids hardcoding passwords in your trigger configuration