Smart Network API Interception

Smart Network API Interception is a unique RTILA X feature that lets you extract data from a website’s background API calls instead of parsing the visible HTML. Many modern websites load their data through invisible API requests behind the scenes. Instead of scraping the rendered HTML (which can be messy and fragile), you can intercept these API responses directly and get clean, structured JSON data.

Think of it like this: instead of reading a restaurant menu posted on the wall (HTML scraping), you’re intercepting the order slip going to the kitchen (API interception). The data is cleaner, more complete, and easier to work with.

---

Why Use Network Interception?

Benefit	Description
Cleaner data	API responses return structured JSON, not messy HTML. No need to parse through divs and spans.
More reliable	API endpoints have consistent response formats. HTML structure changes frequently, but API schemas are more stable.
Faster execution	No need to wait for full page rendering. API responses arrive quickly and contain all the data at once.
Bypasses DOM changes	If the website redesigns its HTML but keeps the same API, your automation keeps working.
Access to hidden data	Some data is only available via API and never appears in the visible HTML.
Pagination built-in	API responses often include pagination metadata (total count, next cursor) for automatic multi-page extraction.

---

How It Works

Network Interception follows a simple three-phase process:

Phase 1: Enable and Run

1. Open your project settings.
2. Go to the Advanced tab.
3. Enable Smart Network API Interception.
4. Make sure your project has 0 datasets and 0 commands (leave them empty).
5. Run the project.

Phase 2: Intercept and Analyze

When the project runs:

1. RTILA navigates to your target URL.
2. A status overlay appears: “🔍 RTILA is scanning live API traffic…”
3. RTILA listens for all network requests (XHR and Fetch).
4. When JSON API responses arrive, they are captured.
5. The captured data is analyzed to find arrays of objects (the actual data).

Phase 3: Auto-Generate Configuration

Once data is captured:

1. RTILA shows a configuration dialog asking you to confirm settings.
2. You can adjust the JSON path to the data array.
3. You can configure pagination parameters.
4. RTILA auto-generates:
   • A dataset configuration with properties matching the JSON fields
   • Extraction commands to retrieve the data
   • Pagination logic (if detected)

---

Example Workflow

Let’s say you want to scrape products from a modern e-commerce site that loads products via API.

Step 1: Create a New Project

1. Go to Automation Projects → New.
2. Set the starting URL: https://example-store.com/products
3. Do NOT add any datasets or commands. Leave them empty.

Step 2: Enable Interception

1. Click Settings → Advanced tab.
2. Check Enable Smart Network API Interception.
3. Click Save.

Step 3: Run the Project

1. Click Run.
2. RTILA navigates to the page and displays the scanning overlay.
3. The page loads and makes API calls in the background.
4. RTILA captures the API response: GET /api/v1/products?page=1&limit=20

Step 4: Configure the Interception

A dialog appears showing the detected data:

📡 Intercepted API Data Detected!

Request: GET /api/v1/products?page=1&limit=20
Data Path: data.products (20 items found)

Pagination detected: Yes
Parameter: page (type: page, step: 20)
Total count field: data.total_count

[Start Extraction] [Continue (Single Page)] [Cancel]

Step 5: Review Generated Configuration

After confirmation, RTILA generates:

• A dataset named Auto_Intercepted_Data with properties matching the API fields
• An http_request command to fetch the API data
• An extract_data command to process the results
• Pagination logic using a while loop

Step 6: Run Again

Run the project again with the generated configuration. This time it will:

1. Call the API directly
2. Extract all products
3. Paginate through all pages automatically
4. Save everything to your scraped data

---

Supported Data Types

Network Interception captures these types of network requests:

Request Type	Description	Example
XHR (XMLHttpRequest)	Traditional AJAX requests	XMLHttpRequest calls
Fetch API	Modern JavaScript fetch calls	fetch('/api/data')

Response Format Requirements

Requirement	Details
Content-Type	Must contain application/json or +json
HTTP Method	GET, POST, PUT, DELETE, PATCH (OPTIONS is excluded)
Response Body	Must be valid JSON
Data Structure	Must contain at least one array of objects

---

Pagination Detection

One of the most powerful features of Network Interception is automatic pagination detection. RTILA analyzes the API request and response to determine how to paginate.

Detected Pagination Types

Type	How It Works	Example
Offset-based	Increments an offset parameter	?offset=0, ?offset=20, ?offset=40
Page-based	Increments a page number	?page=1, ?page=2, ?page=3
Cursor-based	Uses a cursor/token from the response	?cursor=abc123 → next: ?cursor=def456

Pagination Parameters Detected

Parameter Name	Type	Example
page, p, pageIndex, pageNum	Page-based	?page=2
offset, start	Offset-based	?offset=20
cursor, after, token, page_token	Cursor-based	?cursor=abc123
limit, rows, size, per_page	Page size	?limit=20

Total Count Fields Detected

RTILA looks for these fields in the API response to know when to stop:

Field Name	Example
total, totalcount, total_count	"total": 150
count, numfound, num_found	"count": 150
total_results, total_records	"total_results": 150
nbpages, nb_pages, totalpages	"total_pages": 8
totalitems, total_items	"total_items": 150

Has-More Flags Detected

True Flags	False Flags
has_more, hasmore	is_last, islast
hasnext, has_next	is_final, is_end
more, more_results
hasnextpage, has_next_page

---

The Interception Configuration Dialog

When data is detected, RTILA shows a configuration dialog where you can fine-tune the interception:

Setting	Description	Default
API Request	Which intercepted request to use	The one with the most items
JSON Path to Data Array	The path to the array of objects in the JSON	Auto-detected
Parameter Name	The pagination parameter name	Auto-detected
Parameter Type	offset, page, or cursor	Auto-detected
Step Size	How much to increment per page	Auto-detected from limit param
Total Count Field	JSON path to the total count field	Auto-detected
Cursor Field	JSON path to the next cursor field	Auto-detected

Actions in the Dialog

Button	Action
Start Extraction	Generate full pagination configuration and extract all pages
Continue (Single Page)	Extract only the current page (no pagination)
Cancel	Abort the interception

---

Limitations

Limitation	Details
JSON only	Only JSON API responses are supported. XML, HTML, or binary responses are ignored.
Must be called during page load	The API must be called when the page loads. APIs triggered only by user interaction won’t be captured.
Authentication may be required	Some APIs require auth tokens or cookies. The interception uses the browser’s session context.
Rate limiting	Aggressive pagination may trigger rate limits. RTILA adds delays between requests.
CORS restrictions	Some APIs may block cross-origin requests. These are captured from the browser’s own requests.

---

Combining with DOM Scraping

Network Interception doesn’t replace DOM scraping — it complements it. You can use both approaches in the same project:

When to Use Network Interception

• The page loads data via API calls
• You need structured JSON data
• The HTML structure is complex or changes frequently
• You need pagination metadata (total count, cursors)

When to Use DOM Scraping

• The data is only in the HTML (no API calls)
• You need to interact with the page (click, scroll, fill forms)
• The data is embedded in the HTML and not loaded via API
• You need to handle CAPTCHAs or login flows

Combining Both

You can start with Network Interception to discover the API structure, then add DOM scraping commands for additional data:

1. Run the project with interception enabled (empty datasets/commands).
2. Let RTILA generate the API-based extraction.
3. Add DOM scraping commands for any data not available via API.
4. Both approaches will run in the same project execution.

Tip: If interception finds the data you need, prefer it over DOM scraping. API data is more reliable and less likely to break when the website redesigns.