Recipe · Web scraping
Warm up a session
Opens each of this profile's target pages, scrolls it the way a reader would, and screenshots — without reading a single selector or writing a single row. For a profile that is about to start collecting, or one that has been idle.
On this page
What it does
Opens this profile's sources and reads them like a person — scroll, pause, scroll back — without collecting anything.
It runs against one profile, through that profile's own proxy and with its own cookies — so the pages load the way that account's owner would see them. It navigates, so give it a profile whose session you are happy to have in use.
The 9 steps
The real tree, read out of the pack the launcher ships. Indentation is nesting: the steps under a loop run once per row, and the ones under a branch run only when the condition above them holds. Every {{…}} is a value filled in at run time — from the row being walked, from a setting on the run, or from the profile itself.
loadRowsLoad this profile's targetsLoad Scrape targets into rowsloopEach sourceLoop forEachEach time
gotoOpen the sourceGo to {{loop.item.url}}on failure: continuewaitWaitscrollDown the pageScroll to bottomon failure: continuewaitWaitscrollBack to the topScroll to topon failure: continuescreenshotScreenshoton failure: continuewaitWaitEach of those is one of the step types on the automation reference, with the same fields the editor shows and the same fields an agent is handed over the local API.
The tables it uses
These load with the recipe. A dataset is a typed table your workspace owns — the columns are named and typed up front so the steps can address them, and every one of them is yours to rename, extend or fill from a file afterwards.
Scrape targets
readsOne row per source: its list page, the selectors that read it, the profile that reads it, and what the last pass found.
- label
- Label · text
- url
- URL · url
- row_selector
- Row selector · text
- link_selector
- Link selector · text
- profile
- Profile · profile
- status
- Status · select · Unchecked | Collected | Nothing matched | Blocked | Needs attention
- found
- Found · number
- checked_at
- Checked at · datetime
- notes
- Notes · longText
Re-running updates the row it already wrote rather than adding a second one. That is what the match column in the save step is for, and it is the difference between a status check and a table that doubles in size every pass.
Loading it
In the launcher, open Automations, choose Load a starter pack, and pick Web scraping. Untick anything you do not want. The tables land first, then the workflows, then a project called Collection that links them together.
What arrives is an ordinary automation row. Open it in the editor, change a step, rename it, delete it — nothing in the app treats it as special afterwards, and loading the pack a second time makes a fresh copy rather than overwriting the one you edited. Tables are the opposite: an existing table of the same name is reused and keeps its rows.
The rest of the web scraping pack
They load together, share the tables above, and land in one Collection project.
Target audit
Flags target rows that are missing a label, a URL, a row selector or a profile of their own. Opens no page.
One profile · Opens no page · The Scrape targets table
List harvest
Walks this profile's sources, reads each list page with the row's own selector, and files the items.
One profile · Its own proxy and cookies · 2 tables
Paged harvest
Walks one list through its "next page" control, filing every page into the same keyed table.
One profile · Its own proxy and cookies · The Collected items table · 4 settings on the run