Cloud Warehouses & Databases

Cloud warehouse triggers let you write scraped data directly to cloud data platforms and enterprise databases. These are the heavy-duty data storage systems used by organizations for analytics, reporting, and business intelligence. When you scrape data and need it in a data warehouse for analysis, these triggers handle the delivery.

RTILA X supports five cloud warehouse and enterprise database triggers:

Trigger	Platform	Best For
Snowflake	Cloud data warehouse	Modern analytics, data sharing, multi-cloud
Google BigQuery	Google Cloud data warehouse	Google Cloud users, serverless analytics
Databricks	Data lakehouse	Machine learning, data engineering
Microsoft SQL Server	Enterprise database	Microsoft shops, Windows environments
Oracle Database	Enterprise database	Oracle shops, enterprise applications

Snowflake

Snowflake is a modern cloud data warehouse that runs on AWS, Azure, or Google Cloud. It’s known for its speed, ease of use, and ability to share data between organizations. If your company uses Snowflake for analytics, this trigger lets you write scraped data directly to it.

Description

The Snowflake trigger connects to your Snowflake account and executes a SQL query. You can use INSERT, UPDATE, SELECT, or any other SQL statement. Snowflake uses a unique architecture with separate compute (warehouses) and storage, so you specify which warehouse to use for your query.

Parameters

Parameter	Type	Required	Default	Description
account	string	✅ Yes	—	Your Snowflake account identifier. This is the first part of your Snowflake URL. Example: mycompany.us-east-1 (from https://mycompany.us-east-1.snowflakecomputing.com).
user	string	✅ Yes	—	Your Snowflake username.
password	string	✅ Yes	—	Your Snowflake password.
warehouse	string	✅ Yes	—	The Snowflake warehouse to use for the query. This is the compute resource that processes your query. Example: COMPUTE_WH.
database	string	✅ Yes	—	The Snowflake database name. Example: SCRAPING_DB.
schema	string	✅ Yes	—	The schema within the database. Example: PUBLIC or RAW_DATA.
query	string	✅ Yes	—	The SQL query to execute. Supports ${variable} substitution.
role	string	❌ No	—	The Snowflake role to assume. If not specified, your default role is used.
timeout	integer	❌ No	30	Login timeout in seconds. Increase for slow connections.

Example: Insert Scraped Data into Snowflake

{
  "account": "mycompany.us-east-1",
  "user": "ETL_USER",
  "password": "snowflake-password",
  "warehouse": "COMPUTE_WH",
  "database": "SCRAPING_DB",
  "schema": "RAW_DATA",
  "query": "INSERT INTO products (name, price, url, scraped_at) SELECT '${product_name}', ${product_price}, '${product_url}', CURRENT_TIMESTAMP()",
  "role": "ETL_ROLE",
  "timeout": 60
}

Example: Query Snowflake for Verification

{
  "account": "mycompany.us-east-1",
  "user": "ETL_USER",
  "password": "snowflake-password",
  "warehouse": "COMPUTE_WH",
  "database": "SCRAPING_DB",
  "schema": "RAW_DATA",
  "query": "SELECT COUNT(*) AS total FROM products WHERE scraped_at >= CURRENT_DATE()"
}

Finding Your Snowflake Settings

Setting	Where to Find It
Account Identifier	Your Snowflake URL: https://mycompany.us-east-1.snowflakecomputing.com → account is mycompany.us-east-1
Warehouse	Snowflake → Warehouses tab
Database	Snowflake → Databases tab
Schema	Snowflake → Select a database → Schemas tab
Role	Snowflake → Account → Roles

Output

Query Type	Output
SELECT	An array of row objects
INSERT / UPDATE / DELETE	Number of affected rows

---

Google BigQuery

Google BigQuery is Google’s serverless, highly scalable data warehouse. It can analyze billions of rows in seconds using SQL-like queries. If your company uses Google Cloud, BigQuery is the natural choice for storing and analyzing scraped data.

Description

The Google BigQuery trigger connects to your Google Cloud project and executes a SQL query against BigQuery. It uses a service account for authentication, which you provide as a JSON credentials string.

Parameters

Parameter	Type	Required	Default	Description
gcp_credentials_json	string	✅ Yes	—	Your Google Cloud service account JSON credentials. This is the full JSON string from your service account key file. Paste the entire contents of the JSON file.
project_id	string	✅ Yes	—	Your Google Cloud project ID. Example: my-analytics-project. Found in the Google Cloud Console dashboard.
query	string	✅ Yes	—	The SQL query to execute. Supports ${variable} substitution. Uses BigQuery SQL syntax.
dataset_id	string	❌ No	—	The default dataset to use. If specified, you can use short table names in your query. Example: scraping_data.
timeout	integer	❌ No	60	Query timeout in seconds. Increase for complex queries on large datasets.

Example: Insert Scraped Data into BigQuery

{
  "gcp_credentials_json": "{\"type\": \"service_account\", \"project_id\": \"my-analytics-project\", \"private_key\": \"-----BEGIN PRIVATE KEY-----\\n...\", \"client_email\": \"etl@my-analytics-project.iam.gserviceaccount.com\"}",
  "project_id": "my-analytics-project",
  "dataset_id": "scraping_data",
  "query": "INSERT INTO products (name, price, url, scraped_at) VALUES ('${product_name}', ${product_price}, '${product_url}', CURRENT_TIMESTAMP())",
  "timeout": 120
}

Example: Query BigQuery

{
  "gcp_credentials_json": "{\"type\": \"service_account\", ...}",
  "project_id": "my-analytics-project",
  "dataset_id": "scraping_data",
  "query": "SELECT COUNT(*) AS total FROM products WHERE scraped_at >= CURRENT_DATE()"
}

Setting Up BigQuery Authentication

1. Go to the Google Cloud Console → IAM & Admin → Service Accounts.
2. Click Create Service Account and give it a name (e.g., “RTILA ETL”).
3. Grant it the BigQuery Data Editor role.
4. Click the service account → Keys → Add Key → JSON.
5. Download the JSON file and copy its entire contents.
6. Paste the JSON into the gcp_credentials_json field.

Output

Query Type	Output
SELECT	An array of row objects
INSERT / UPDATE / DELETE	Number of affected rows

---

Databricks

Databricks is a unified data analytics platform built on Apache Spark. It combines data warehousing, machine learning, and data engineering in one platform. If your company uses Databricks, this trigger lets you execute SQL queries against Databricks SQL Warehouses.

Description

The Databricks trigger connects to a Databricks SQL Warehouse and executes a SQL query. It uses a personal access token for authentication and connects via the warehouse’s HTTP endpoint.

Parameters

Parameter	Type	Required	Default	Description
host	string	✅ Yes	—	Your Databricks workspace URL. Example: https://mycompany.cloud.databricks.com. Found in your browser’s address bar when logged into Databricks.
http_path	string	✅ Yes	—	The HTTP path of your SQL Warehouse. Example: /sql/1.0/warehouses/a1b2c3d4e5f6. Found in Databricks → SQL Warehouses → Select a warehouse → Connection details.
token	string	✅ Yes	—	Your Databricks personal access token. Generated in Databricks → User Settings → Access Tokens.
query	string	✅ Yes	—	The SQL query to execute. Supports ${variable} substitution.
catalog	string	❌ No	"hive_metastore"	The Databricks catalog to use. Default is hive_metastore. If using Unity Catalog, specify your catalog name.
schema	string	❌ No	"default"	The schema (database) within the catalog. Default is default.

Example: Insert Scraped Data into Databricks

{
  "host": "https://mycompany.cloud.databricks.com",
  "http_path": "/sql/1.0/warehouses/a1b2c3d4e5f6",
  "token": "dapi1234567890abcdef1234567890abcdef",
  "query": "INSERT INTO products (name, price, url) VALUES ('${product_name}', ${product_price}, '${product_url}')",
  "catalog": "hive_metastore",
  "schema": "scraping_data"
}

Example: Query Databricks

{
  "host": "https://mycompany.cloud.databricks.com",
  "http_path": "/sql/1.0/warehouses/a1b2c3d4e5f6",
  "token": "dapi1234567890abcdef1234567890abcdef",
  "query": "SELECT COUNT(*) AS total FROM products WHERE price > 50",
  "catalog": "hive_metastore",
  "schema": "scraping_data"
}

Finding Your Databricks Settings

Setting	Where to Find It
Workspace URL	Your browser’s address bar when logged into Databricks
HTTP Path	Databricks → SQL Warehouses → Select a warehouse → Connection details → HTTP Path
Access Token	Databricks → Click your username → Settings → Access Tokens → Generate New Token
Catalog	Databricks → Data → Catalogs
Schema	Databricks → Data → Select a catalog → Schemas

Output

Query Type	Output
SELECT	An array of row objects
INSERT / UPDATE / DELETE	Number of affected rows

---

Microsoft SQL Server

Microsoft SQL Server is one of the most widely used enterprise relational databases. If your company runs on Microsoft technology, SQL Server is likely your primary database. This trigger lets you write scraped data directly to it.

Description

The Microsoft SQL Server trigger connects to a SQL Server instance and executes a SQL query. You can use INSERT, UPDATE, SELECT, or any other T-SQL statement.

Parameters

Parameter	Type	Required	Default	Description
host	string	✅ Yes	—	The SQL Server hostname or IP address. Example: sqlserver.company.com, localhost, or 192.168.1.100.
port	integer	✅ Yes	1433	The SQL Server port. Default is 1433. Named instances may use different ports.
user	string	✅ Yes	—	Your SQL Server username. Use sa for the system administrator, or a specific user account.
password	string	✅ Yes	—	Your SQL Server password.
database	string	✅ Yes	—	The database name. Example: ScrapingDB.
query	string	✅ Yes	—	The T-SQL query to execute. Supports ${variable} substitution.
driver	string	❌ No	—	The ODBC driver name. This is informational only and doesn’t affect the connection. Example: ODBC Driver 17 for SQL Server.
timeout	integer	❌ No	30	Connection timeout in seconds.

Example: Insert Scraped Data into SQL Server

{
  "host": "sqlserver.company.com",
  "port": 1433,
  "user": "etl_user",
  "password": "sqlserver-password",
  "database": "ScrapingDB",
  "query": "INSERT INTO products (name, price, url, scraped_at) VALUES ('${product_name}', ${product_price}, '${product_url}', GETDATE())",
  "timeout": 60
}

Example: Query SQL Server

{
  "host": "sqlserver.company.com",
  "port": 1433,
  "user": "etl_user",
  "password": "sqlserver-password",
  "database": "ScrapingDB",
  "query": "SELECT COUNT(*) AS total FROM products WHERE scraped_at >= CAST(GETDATE() AS DATE)"
}

SQL Server Authentication Notes

Authentication Type	How to Use
SQL Server Authentication	Provide user and password in the trigger parameters
Windows Authentication	Not directly supported. Use SQL Server Authentication instead.
Named Instances	Use the format hostname\instancename for the host parameter, or specify the port directly

Output

Query Type	Output
SELECT	An array of row objects
INSERT / UPDATE / DELETE	Number of affected rows

---

Oracle Database

Oracle Database is one of the oldest and most widely used enterprise database systems. It’s common in banking, telecommunications, government, and large enterprises. If your organization runs Oracle, this trigger lets you write scraped data to it.

Description

The Oracle Database trigger connects to an Oracle database instance and executes a SQL query. Oracle uses a slightly different connection model than other databases — instead of a database name, you specify a service name or SID.

Parameters

Parameter	Type	Required	Default	Description
host	string	✅ Yes	—	The Oracle database server hostname or IP address. Example: oracle.company.com.
port	integer	✅ Yes	1521	The Oracle listener port. Default is 1521.
user	string	✅ Yes	—	Your Oracle database username.
password	string	✅ Yes	—	Your Oracle database password.
service_name	string	✅ Yes	—	The Oracle service name or SID. Example: ORCL, ORCLPDB1, or PROD. Found in your Oracle tnsnames.ora file or from your DBA.
query	string	✅ Yes	—	The SQL query to execute. Supports ${variable} substitution. Uses Oracle SQL syntax.
timeout	integer	❌ No	30	Connection timeout in seconds.

Example: Insert Scraped Data into Oracle

{
  "host": "oracle.company.com",
  "port": 1521,
  "user": "ETL_USER",
  "password": "oracle-password",
  "service_name": "ORCLPDB1",
  "query": "INSERT INTO products (name, price, url, scraped_at) VALUES ('${product_name}', ${product_price}, '${product_url}', SYSDATE)",
  "timeout": 60
}

Example: Query Oracle

{
  "host": "oracle.company.com",
  "port": 1521,
  "user": "ETL_USER",
  "password": "oracle-password",
  "service_name": "ORCLPDB1",
  "query": "SELECT COUNT(*) AS total FROM products WHERE scraped_at >= TRUNC(SYSDATE)"
}

Oracle SQL Syntax Notes

Oracle SQL is slightly different from MySQL or SQL Server:

Feature	Oracle Syntax	MySQL Equivalent
Current date/time	SYSDATE	NOW()
Today’s date	TRUNC(SYSDATE)	CURDATE()
Auto-increment	Use SEQUENCE.NEXTVAL	AUTO_INCREMENT
String concatenation	'Hello' || ' World'	CONCAT('Hello', ' World')
Row limit	FETCH FIRST 10 ROWS ONLY	LIMIT 10

Finding Your Oracle Settings

Setting	Where to Find It
Host	Ask your DBA or check tnsnames.ora
Port	Default is 1521. Check listener.ora for custom ports.
Service Name	Check tnsnames.ora or ask your DBA. Common names: ORCL, ORCLPDB1, PROD
Username / Password	Your Oracle database credentials

Output

Query Type	Output
SELECT	An array of row objects
INSERT / UPDATE / DELETE	Number of affected rows

---

Choosing the Right Cloud Warehouse Trigger

Scenario	Recommended Trigger
Your company uses Snowflake	Snowflake
Your company uses Google Cloud	Google BigQuery
Your company uses Databricks	Databricks
Your company uses Microsoft SQL Server	Microsoft SQL Server
Your company uses Oracle	Oracle Database
You need serverless analytics	Google BigQuery
You need data sharing between organizations	Snowflake
You need ML + analytics in one platform	Databricks
You’re in a Microsoft-centric environment	Microsoft SQL Server
You’re in a traditional enterprise	Oracle Database

Common Patterns

Pattern 1: Scrape → Insert → Verify

Project Run → Extract Data → Snowflake (INSERT) → Snowflake (SELECT COUNT)

Insert scraped data, then verify the row count matches expectations.

Pattern 2: Scrape → BigQuery → Notify

Project Run → Extract Data → BigQuery (INSERT) → Slack Webhook (notify)

Insert data into BigQuery and send a Slack notification to your team.

Pattern 3: Scrape → Export → SQL Server

Project Run → Extract Data → Export to CSV → SQL Server (BULK INSERT)

Export to CSV first, then bulk-insert into SQL Server for large datasets.

Pattern 4: Scrape → Oracle → Email Report

Project Run → Extract Data → Oracle (INSERT) → Email (send summary)

Insert data into Oracle and email a summary report.

Tips and Best Practices

Tip	Why It Matters
Test credentials before running	Verify your database credentials work before adding to production
Use ${variable} substitution in queries	Makes your inserts dynamic
Use batch inserts for large datasets	Insert multiple rows in one query instead of one at a time
Store credentials as project variables	Avoids hardcoding passwords in your trigger configuration
Use the correct SQL dialect	Each database has slightly different SQL syntax (see Oracle notes above)
Increase timeout for large operations	Default timeouts may be too short for bulk inserts
Use database-specific date functions	SYSDATE for Oracle, GETDATE() for SQL Server, CURRENT_TIMESTAMP() for Snowflake
Create tables before inserting	Ensure the target table exists before running INSERT statements
Use transactions for multi-statement operations	Ensures data integrity if one statement fails
Monitor database performance	Large inserts can slow down production databases

Error Handling

Error	Cause	Solution
Connection refused	Database server is unreachable	Check host, port, and firewall settings
Authentication failed	Wrong username or password	Verify your database credentials
Database/schema not found	Wrong database or schema name	Verify the database/schema name
Table not found	Table doesn’t exist	Create the table before inserting
Syntax error	Wrong SQL dialect	Check the SQL syntax for your specific database
Timeout	Query takes too long	Increase the timeout parameter
Permission denied	Insufficient database permissions	Grant INSERT/SELECT permissions to your user