Recipe · Web scraping

List harvest

Opens each of this profile's rows in Scrape targets, reads the list page with the row's own selector, and files the items into Collected items — updating rows rather than adding them. The selectors are illustrative: every site draws a list differently, so expect to adjust the row selector once per source.

On this page

What it does

Walks this profile's sources, reads each list page with the row's own selector, and files the items.

It runs against one profile, through that profile's own proxy and with its own cookies — so the pages load the way that account's owner would see them. It navigates, so give it a profile whose session you are happy to have in use.

One profileIts own proxy and cookies2 tables

The 12 steps

The real tree, read out of the pack the launcher ships. Indentation is nesting: the steps under a loop run once per row, and the ones under a branch run only when the condition above them holds. Every {{…}} is a value filled in at run time — from the row being walked, from a setting on the run, or from the profile itself.

loadRowsLoad this profile's targetsLoad Scrape targets into rows

evaluateStamp the passRun script

loopEach sourceLoop forEach

Each time

setVarStart this source as unfinishedSet verdict

gotoOpen the list pageGo to {{loop.item.url}}on failure: continue

waitWait

evaluateRead the listRun script

evaluateWhat happened here?Run scripton failure: continue

saveRowsFile the itemsSave to Collected itemson failure: continue

saveRowsWrite the verdict backSave to Scrape targetson failure: continue

screenshotScreenshoton failure: continue

waitWait

Each of those is one of the step types on the automation reference, with the same fields the editor shows and the same fields an agent is handed over the local API.

The tables it uses

These load with the recipe. A dataset is a typed table your workspace owns — the columns are named and typed up front so the steps can address them, and every one of them is yours to rename, extend or fill from a file afterwards.

Scrape targets

readswrites

One row per source: its list page, the selectors that read it, the profile that reads it, and what the last pass found.

label: Label · text
url: URL · url
row_selector: Row selector · text
link_selector: Link selector · text
profile: Profile · profile
status: Status · select · Unchecked | Collected | Nothing matched | Blocked | Needs attention
found: Found · number
checked_at: Checked at · datetime
notes: Notes · longText

Collected items

writes

What the passes write: one row per item, keyed so a re-run updates it instead of adding a second copy.

row_key: Key · text
source: Source · text
title: Title · text
url: URL · url
detail: Detail · longText
collected_at: Collected at · datetime
profile: Profile · profile

Re-running updates the row it already wrote rather than adding a second one. That is what the match column in the save step is for, and it is the difference between a status check and a table that doubles in size every pass.

Loading it

In the launcher, open Automations, choose Load a starter pack, and pick Web scraping. Untick anything you do not want. The tables land first, then the workflows, then a project called Collection that links them together.

What arrives is an ordinary automation row. Open it in the editor, change a step, rename it, delete it — nothing in the app treats it as special afterwards, and loading the pack a second time makes a fresh copy rather than overwriting the one you edited. Tables are the opposite: an existing table of the same name is reused and keeps its rows.

Download Argus The whole library

The rest of the web scraping pack

They load together, share the tables above, and land in one Collection project.

Target audit

Flags target rows that are missing a label, a URL, a row selector or a profile of their own. Opens no page.

5 stepsopens no page

One profile · Opens no page · The Scrape targets table

Warm up a session

Opens this profile's sources and reads them like a person — scroll, pause, scroll back — without collecting anything.

9 steps

One profile · Its own proxy and cookies · The Scrape targets table

Paged harvest

Walks one list through its "next page" control, filing every page into the same keyed table.

9 steps

One profile · Its own proxy and cookies · The Collected items table · 4 settings on the run