Turn any web page into clean Markdown for your LLM

Why Markdown, not HTML.

The HTML of a web page is mostly not the article. It is navigation, scripts, class names and wrappers around a few paragraphs. Hand it to a model and you spend context window, and money, on markup.

Markdown keeps what a model can use: headings, lists, code blocks, tables and links, as plain text. It also splits cleanly on headings when you build a retrieval index. Vera's scrape endpoint returns it for any public page you could open without signing in.

One request.

Create an API key in your workspace and keep it on your server. Send the page's URL with markdown set to true, and the main content of the page comes back as Markdown.

curl

curl --fail-with-body --max-time 45 \
  -X POST "https://vera.waggle.ae/api/vera/v1/scrape" \
  -H "Authorization: Bearer $VERA_API_KEY" \
  -H 'Content-Type: application/json' \
  --data '{"url":"https://example.com","markdown":true}'

Replace https://example.com with the page you want. It is a plain POST with a JSON body, so any HTTP client does the same job. Vera has no SDK to install.

What comes back.

The response is JSON. The Markdown is in data.markdown. Beside it are the page's title, its final URL, its links, its metadata and a quality score, and credits_used says what the call cost. This is a trimmed, illustrative response, not the output of a live page.

Response, trimmed

{
  "request_id": "...",
  "data": {
    "url": "https://example.com/",
    "title": "Example Domain",
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents.",
    "status_code": 200
  },
  "credits_used": 1
}

Without markdown set to true, data.markdown holds the whole page, navigation and footer included. With it, you get the main content only.

Put it in a prompt or an index.

Three ways to use data.markdown, and one way to read many pages.

To read one page, pass data.markdown to your model as it is, with the source URL beside it so you can cite where it came from.
To build a retrieval index, split the Markdown on its headings. Each section then keeps its own heading.
To keep thin results out, set min_quality_score. If a page comes back below your floor, the call returns an error and costs nothing.
To read many pages, create a collection: up to 1,000 URLs, or a same-site crawl, at 1 credit for each page that completes. Each page's JSON result carries its Markdown, with the main-content version in main_content.markdown. The CSV export has the URL, title, text and status only.

The honest limits.

Markdown from a page is only as good as the page, and Vera does not try to be more than a careful reader.

A normal request does not run JavaScript. A page that builds its content in the browser can come back almost empty. For those pages set render to true: a rendered request costs 5 credits, runs in an isolated browser, and is only offered where that browser is turned on.
Vera reads public pages only. A login, a paywall or a page behind a form is out of scope, and Vera does not solve CAPTCHAs or bypass bot protection.
It checks robots.txt and each source's policy before it fetches, so some pages are refused rather than forced.
Vera has no residential or rotating proxies and cannot fetch a page through a country you choose. If a site blocks ordinary requests, Vera will not get through.
The conversion is mechanical: images are left out, and a table comes out as pipe-separated rows with blank lines between the rows, which you may want to tidy before a renderer draws it as a table.
The main content is taken from the page's main or article element. A page with neither falls back to the body without its navigation, sidebars, headers and footers, so it can carry more page furniture. The Markdown is capped at 50,000 characters.

What it costs.

A scrape costs 1 credit for a static page and 5 credits for a rendered page. Failed or blocked requests cost nothing, and so does a response below a quality floor you set.

A credit costs between $0.00029 and $0.00054 depending on your plan, so a thousand static pages come to between $0.285 and $0.543.

New accounts get free credits once for each confirmed email address, with no card, so you can run this on your own pages before you pay anything. The amount is shown below.