Network Commands

Network commands let your automation interact with web APIs, crawl websites by following links, and handle infinite scrolling. These commands are essential for building complex scraping workflows that go beyond simple page navigation.

http_request

Makes an HTTP request from the browser context. This command is useful for calling APIs, submitting form data, or fetching data without navigating to a new page.

Description

The http_request command sends an HTTP request (GET, POST, PUT, DELETE, or PATCH) to a specified URL. It can include custom headers, query parameters, and a request body. The response is returned and can be stored in a variable for use in later commands.

This command runs on the server side (in the Deno engine), not in the browser. This means it has full access to the network stack and can handle complex HTTP operations.

Parameters

Parameter	Type	Required	Default	Description
`url`	string	✅ Yes	`""`	The URL to send the request to. Supports `${variable}` substitution.
`method`	string	No	`"GET"`	The HTTP method to use. Options: `GET`, `POST`, `PUT`, `DELETE`, `PATCH`.
`headers`	object	No	`{}`	Custom HTTP headers as key-value pairs. Example: `{"Authorization": "Bearer ${api_key}"}`.
`params`	object	No	`{}`	URL query parameters as key-value pairs. Example: `{"page": "1", "limit": "20"}`.
`body`	string	No	`""`	The request body for POST, PUT, or PATCH requests. Can be a JSON string. Supports `${variable}` substitution.
`output_variable`	string	No	`""`	The name of the variable to store the response in.
`timeout`	number	No	`30000`	Maximum time to wait for the response, in milliseconds.
`retries`	number	No	`0`	Number of times to retry the request if it fails.
`retry_delay`	number	No	`1000`	Delay between retries, in milliseconds.
`retry_on_status`	array	No	`[429, 500, 502, 503, 504]`	HTTP status codes that trigger a retry.
`use_browser_context`	boolean	No	`false`	If `true`, uses the browser’s network context (cookies, session) for the request.

Examples

Simple GET request:

{
  "command": "http_request",
  "params": {
    "url": "https://api.example.com/products",
    "method": "GET",
    "output_variable": "api_response"
  }
}

POST request with JSON body:

{
  "command": "http_request",
  "params": {
    "url": "https://api.example.com/login",
    "method": "POST",
    "headers": {
      "Content-Type": "application/json"
    },
    "body": "{\"username\": \"${username}\", \"password\": \"${password}\"}",
    "output_variable": "login_response"
  }
}

Request with query parameters:

{
  "command": "http_request",
  "params": {
    "url": "https://api.example.com/search",
    "method": "GET",
    "params": {
      "q": "${search_term}",
      "page": "1",
      "limit": "50"
    },
    "output_variable": "search_results"
  }
}

Request with retries:

{
  "command": "http_request",
  "params": {
    "url": "https://api.example.com/data",
    "method": "GET",
    "retries": 3,
    "retry_delay": 2000,
    "retry_on_status": [429, 500, 503],
    "output_variable": "data"
  }
}

Response Format

The response is stored in the output_variable as an object with these fields:

Field	Type	Description
`status`	number	The HTTP status code (e.g., 200, 404).
`statusText`	string	The HTTP status text (e.g., “OK”, “Not Found”).
`headers`	object	Response headers as key-value pairs.
`body`	string	The raw response body as a string.
`json`	object or null	The parsed JSON response, or `null` if the response is not valid JSON.

Behavior Notes

If use_browser_context is true, the request uses the browser’s cookies and session. This is useful for APIs that require authentication via browser cookies.
If use_browser_context is false (default), the request is made directly from the Deno engine without browser context.
The retry_on_status parameter lets you specify which HTTP status codes should trigger a retry. Common codes include 429 (Too Many Requests), 500 (Internal Server Error), and 503 (Service Unavailable).
The body parameter should be a string. If you’re sending JSON, convert it to a JSON string first.

crawl_links

Finds and follows links on a page to crawl a website, running actions on each new page. This command is essential for building web crawlers that visit multiple pages.

Description

The crawl_links command starts from the current page, finds all links matching your criteria, and visits each one. For each page visited, it runs the commands in the do block. This lets you build complex crawlers that extract data from many pages.

The crawler respects the max_depth and max_urls limits to prevent runaway crawls. It also supports concurrency for faster crawling.

Parameters

Parameter	Type	Required	Default	Description
`do`	array	✅ Yes	`[]`	Commands to run on each crawled page.
`link_type`	string	No	`"internal"`	Which links to follow. Options: `internal` (same domain), `external` (different domains), `both`.
`max_depth`	number	No	`2`	Maximum depth to crawl. Depth 0 means only the starting page. Depth 1 means the starting page and pages linked from it.
`max_urls`	number	No	`100`	Maximum number of URLs to visit. Set to 0 for unlimited.
`concurrency`	number	No	`1`	Number of pages to crawl simultaneously. Higher values are faster but use more resources.
`allowed_extensions`	array	No	`["html", "htm", "php", "aspx", "asp", "jsp", ""]`	File extensions to follow. Empty string means URLs without extensions.
`randomize`	boolean	No	`false`	If `true`, visits URLs in random order instead of the order they appear on the page.
`delay`	number	No	`1000`	Delay between visiting pages, in milliseconds. Helps avoid rate limiting.
`wait_until`	string	No	`"load"`	When to consider a page loaded. Options: `load`, `domcontentloaded`, `networkidle`, `commit`.

Examples

Crawl all internal links up to depth 2:

{
  "command": "crawl_links",
  "params": {
    "link_type": "internal",
    "max_depth": 2,
    "max_urls": 50,
    "do": [
      {
        "command": "extract_data",
        "params": {
          "dataset": "products"
        }
      }
    ]
  }
}

Crawl with concurrency and delay:

{
  "command": "crawl_links",
  "params": {
    "link_type": "internal",
    "max_depth": 3,
    "max_urls": 200,
    "concurrency": 3,
    "delay": 2000,
    "do": [
      {
        "command": "extract_data",
        "params": {
          "dataset": "articles"
        }
      }
    ]
  }
}

Crawl external links only:

{
  "command": "crawl_links",
  "params": {
    "link_type": "external",
    "max_depth": 1,
    "max_urls": 20,
    "allowed_extensions": ["html", "htm"],
    "do": [
      {
        "command": "screenshot",
        "params": {
          "path": "external_page.png"
        }
      }
    ]
  }
}

Behavior Notes

The crawler starts from the current page and follows links found on that page.
max_depth controls how many “hops” the crawler makes. Depth 0 means only the starting page. Depth 1 means the starting page and pages linked from it. Depth 2 means the starting page, pages linked from it, and pages linked from those pages.
max_urls limits the total number of URLs visited across all depths. This prevents runaway crawls.
concurrency controls how many pages are crawled simultaneously. Higher values are faster but use more memory and may trigger rate limiting on the target site.
randomize is useful for avoiding detection by anti-bot systems that look for sequential crawling patterns.
The allowed_extensions parameter filters which links to follow. For example, setting it to ["html", "htm"] will only follow links to HTML pages, ignoring links to images, PDFs, or other file types.
The crawler respects robots.txt if respect_robots_txt is enabled in your project settings.

infinite_scroll

Scrolls until the bottom is reached or a max count is hit. Executes nested blocks first (e.g., extracting data step-by-step).

Description

The infinite_scroll command handles websites that load more content as you scroll down (infinite scroll). It scrolls the page repeatedly, running the commands in the do block after each scroll. This lets you extract data from pages that load content dynamically.

The command stops when:

The bottom of the page is reached (no new content loads).
The max_scrolls limit is reached.
The page height stops changing after multiple scroll attempts.

Parameters

Parameter	Type	Required	Default	Description
`selector`	string	No	`""`	The CSS selector for the scrollable container. If empty, scrolls the entire page.
`max_scrolls`	number	No	`0`	Maximum number of scroll actions. Set to 0 for unlimited (stops when no new content loads).
`scroll_delay`	number	No	`1500`	Delay between scroll actions, in milliseconds. Gives the page time to load new content.
`do`	array	No	`[]`	Commands to run after each scroll. Typically used for data extraction.

Examples

Scroll the entire page:

{
  "command": "infinite_scroll",
  "params": {
    "max_scrolls": 10,
    "scroll_delay": 2000,
    "do": [
      {
        "command": "extract_data",
        "params": {
          "dataset": "products"
        }
      }
    ]
  }
}

Scroll a specific container:

{
  "command": "infinite_scroll",
  "params": {
    "selector": "css=.product-list",
    "max_scrolls": 5,
    "scroll_delay": 1500,
    "do": [
      {
        "command": "extract_data",
        "params": {
          "dataset": "products"
        }
      }
    ]
  }
}

Unlimited scrolling with data extraction:

{
  "command": "infinite_scroll",
  "params": {
    "max_scrolls": 0,
    "scroll_delay": 2000,
    "do": [
      {
        "command": "extract_data",
        "params": {
          "dataset": "articles"
        }
      }
    ]
  }
}

Behavior Notes

If selector is empty, the command scrolls the entire page (the window object).
If selector is specified, the command scrolls only that specific element. This is useful for pages with multiple scrollable areas.
The command uses a “jiggle” technique to trigger lazy-loaded content: it scrolls up slightly, waits briefly, then scrolls down. This tricks many lazy-loading implementations into loading more content.
The command stops when the page height stops changing after multiple scroll attempts. This indicates that no more content is available.
The scroll_delay parameter gives the page time to load new content after each scroll. If content loads slowly, increase this value.
The do block is executed after each scroll. This is typically where you put your extract_data command to capture data as it loads.
If max_scrolls is 0, the command scrolls until no new content loads. Be careful with this setting on very long pages.

Tips

Use max_scrolls to limit the number of scroll actions and prevent runaway scrolling.
Increase scroll_delay if the page loads content slowly.
Use the selector parameter to scroll only the relevant container, not the entire page.
Combine infinite_scroll with extract_data to capture data as it loads.
If the page uses a “Load More” button instead of infinite scroll, use a while loop with a click command instead.