Convert a Web Page to Markdown for an LLM in Python

A tested Python pipeline: fetch a page, extract the main content, convert it to Markdown, check it and chunk it for RAG without splitting code blocks.

The short version is four steps: fetch the HTML, cut it down to the main content, convert that to Markdown, then split the Markdown into chunks a retriever can use. Python has a mature library for each step, and a composed version of the whole pipeline, with the library versions it was tested against, is near the end of this guide. The rest is the parts that decide whether the output is any good: pages that render with JavaScript, tables and code that converters mangle, chunk boundaries, and the responses that look like success and contain nothing.

To look at the conversion step on one page before writing any code, paste its HTML into the HTML to Markdown converter. It runs in your browser with its own main-content heuristic (the <main> element, else the largest <article>, minus navigation, headers, footers and banner-like blocks). That is simpler than the readability scoring the Python code below uses, so the two can keep different parts of the same page.

The minimal script

Install the packages:

pip install httpx readability-lxml markdownify "lxml[html_clean]"

The html_clean extra matters: recent lxml releases moved the HTML cleaner into a separate lxml_html_clean package, and readability-lxml 0.9 fails on import without it.

Then fetch, extract and convert:

import httpx
from readability import Document
from markdownify import markdownify

HEADERS = {
    "User-Agent": "my-research-bot/1.0 (+https://example.com/bot)",
    "Accept": "text/html,application/xhtml+xml",
    "Accept-Language": "en",
}

def page_to_markdown(url: str) -> dict:
    with httpx.Client(headers=HEADERS, timeout=20, follow_redirects=True) as client:
        r = client.get(url)
    r.raise_for_status()

    doc = Document(r.text)
    main_html = doc.summary(html_partial=True)   # readability's main-content pick
    markdown = markdownify(
        main_html,
        heading_style="ATX",                      # "## Heading", not underlines
        bullets="-",
        strip=["script", "style"],
    )
    return {"url": str(r.url), "title": doc.short_title(), "markdown": markdown.strip()}

if __name__ == "__main__":
    out = page_to_markdown("https://en.wikipedia.org/wiki/Markdown")
    print(out["title"])
    print(out["markdown"][:1500])

readability-lxml is a Python port of the Arc90 Readability algorithm, the ancestor of Firefox's Reader View. It scores blocks by text density and link density and returns the one that looks like an article, which removes navigation, footers and sidebars without a selector per site. It also removes small blocks it scores as clutter, which on some pages includes a short table, so check a few converted pages per site by eye. markdownify then walks the cleaned HTML and writes Markdown. heading_style="ATX" matters more than it looks: underlined Setext headings only go two levels deep, and the chunker below only recognizes # headings.

Send a User-Agent that names your client and gives a contact URL. A browser string bolted onto a script is easy to spot, and the request fingerprint tool shows what yours actually sends.

Pages that render with JavaScript

Some sites send an HTML shell and build the content in the browser. httpx gets the shell, readability finds nothing worth keeping, and the Markdown comes back as a cookie notice and a footer. The fix is to let a real browser render the page and hand you the finished DOM:

pip install playwright
playwright install chromium

The part that needs care is knowing when the page has finished. Waiting for a main, article or h1 element is not enough: many applications put those on the page first, holding a "Loading..." placeholder, and fill them in later. Wait for the content itself instead. Save this as browser.py:

from playwright.sync_api import sync_playwright

# True once any element matching the selector holds at least minChars of visible text.
READY = """([selector, minChars]) => [...document.querySelectorAll(selector)]
    .some(el => el.innerText.trim().length >= minChars)"""

def rendered_html(url: str, selector: str = "main, article", min_chars: int = 500,
                  user_agent: str = "my-research-bot/1.0 (+https://example.com/bot)",
                  proxy: dict | None = None, timeout_ms: int = 15_000) -> str:
    """Load url in Chromium and return the DOM once `selector` holds real text.

    Raises playwright's TimeoutError when nothing matching `selector` reaches
    `min_chars` of visible text in time: the page is still a shell, or the
    selector does not fit this site.
    """
    with sync_playwright() as p:
        browser = p.chromium.launch(proxy=proxy)
        try:
            page = browser.new_page(user_agent=user_agent)
            page.goto(url, wait_until="domcontentloaded", timeout=30_000)
            page.wait_for_function(READY, arg=[selector, min_chars], timeout=timeout_ms)
            return page.content()
        finally:
            browser.close()

The defaults are a generic starting point. For a site you crawl repeatedly, pass the selector of the record you actually want, such as selector=".product-price" with min_chars=1, or selector="article" with a floor below that site's shortest real page. innerText skips elements hidden with CSS, so a hidden template does not count as content. A consent or login page can still pass this wait, because it is real visible text; the checks further down catch those. Prefer this kind of wait over wait_until="networkidle": pages that poll an analytics endpoint never go idle, and the timeout fires on a page that rendered long ago. See Playwright's wait_for_function for polling options.

A browser costs far more per page than an HTTP request, so decide per site rather than per page. Fetch with httpx first, and fall back to Playwright only when the plain fetch produces too little text (the validation step below gives you that number). Many sites that look dynamic ship their content in the initial HTML or in a JSON blob in a <script> tag, and reading that blob is cheaper still.

Default converter settings are tuned for prose. Three things need attention for LLM input.

Tables. markdownify writes GitHub-style pipe tables, which models read well. A pipe table needs a header row, and when the HTML table has no <th> cells, markdownify 1.2 writes an empty header row by default and keeps the first row as data. Passing table_infer_header=True promotes the first row to the header instead. That is right when the first row holds column labels in <td> cells and wrong when it holds data, so set it per site. (The browser converter above always promotes the first row; markdownify does not unless you ask.) Layout tables, one column holding the whole page and still common on older sites, come out as a one-column table; if that happens on a site you care about, unwrap them before converting:

from lxml import html as lxml_html

def unwrap_layout_tables(fragment: str) -> str:
    tree = lxml_html.fragment_fromstring(fragment, create_parent="div")
    for table in tree.xpath("//table"):
        rows = table.xpath("./tr|./tbody/tr")
        if rows and all(len(r.xpath("./td|./th")) <= 1 for r in rows):
            table.tag = "div"
            for el in table.xpath(".//tbody|.//tr|.//td|.//th"):
                el.drop_tag()
    return lxml_html.tostring(tree, encoding="unicode")

Code. Documentation pages put the language in a class such as language-python. markdownify writes a bare fence by default; pass a callback so the fenced block keeps its tag, which helps both the model and any code-aware splitter:

import re

def code_language(el):
    for node in (el, el.find("code")):
        if node is not None:
            m = re.search(r"(?:language|lang)-([\w+#.-]+)", " ".join(node.get("class", [])))
            if m:
                return m.group(1)
    return ""

def convert(main_html: str) -> str:
    return markdownify(main_html, heading_style="ATX", code_language_callback=code_language)

Links. Relative links are useless once the text leaves the page. Resolve them against the final URL before converting, so a citation the model produces points somewhere real:

def absolutize(fragment: str, base: str) -> str:
    tree = lxml_html.fragment_fromstring(fragment, create_parent="div")
    tree.make_links_absolute(base)
    return lxml_html.tostring(tree, encoding="unicode")

If you do not need the URLs at all, converting links to plain text (add "a" to the strip list) shortens every chunk without losing the words.

Chunking for RAG, with metadata

Splitting on a fixed character count cuts tables in half, breaks code fences and separates a heading from the paragraph it introduces. The chunker below reads the Markdown as blocks first, then packs whole blocks into chunks section by section, carrying the heading path into each chunk so it still makes sense when retrieved alone. Save it as chunking.py:

import hashlib
import re

HEADING = re.compile(r"^ {0,3}(#{1,6})[ \t]+(.+?)[ \t]*$")
FENCE = re.compile(r"^ {0,3}(`{3,}|~{3,})(.*)$")

def _closes(line: str, opener: str) -> bool:
    """A fence closes on the same character, at least as long, and nothing else."""
    s = line.strip()
    return (len(line) - len(line.lstrip(" ")) <= 3 and len(s) >= len(opener)
            and set(s) == {opener[0]})

def markdown_blocks(md: str) -> list[tuple[str, str]]:
    """Split Markdown into ("heading" | "fence" | "text", source) blocks.

    Supported subset: ATX headings, fenced code opened with ``` or ~~~, and
    text blocks separated by blank lines (paragraphs, lists, pipe tables).
    Everything inside a fence is code, including lines that start with "#".
    An unclosed fence runs to the end of the document. Setext headings and
    indented code blocks are treated as ordinary text.
    """
    blocks, lines, opener = [], [], None
    for line in md.splitlines():
        if opener:
            lines.append(line)
            if _closes(line, opener):
                blocks.append(("fence", "\n".join(lines)))
                lines, opener = [], None
            continue
        fence, heading = FENCE.match(line), HEADING.match(line)
        if fence and not (fence.group(1)[0] == "`" and "`" in fence.group(2)):
            if lines:
                blocks.append(("text", "\n".join(lines)))
            lines, opener = [line], fence.group(1)
        elif heading or not line.strip():
            if lines:
                blocks.append(("text", "\n".join(lines)))
            lines = []
            if heading:
                blocks.append(("heading", line.strip()))
        else:
            lines.append(line)
    if lines:
        blocks.append(("fence" if opener else "text", "\n".join(lines)))
    return blocks

def _join(blocks):
    return "\n\n".join(text for _, text in blocks)

def _overlap(blocks, limit):
    """Trailing whole blocks, at most `limit` characters, never a heading."""
    tail = []
    for block in reversed(blocks):
        if block[0] == "heading" or len(_join([block] + tail)) > limit:
            break
        tail.insert(0, block)
    return tail

def chunk_markdown(md: str, url: str, title: str,
                   max_chars: int = 3000, overlap: int = 300) -> list[dict]:
    # 1. Group blocks into sections, each with its heading path.
    sections, path, current = [], [], []
    for kind, text in markdown_blocks(md):
        if kind == "heading":
            sections.append((" > ".join(path), current))
            hashes, name = HEADING.match(text).groups()
            path = path[: len(hashes) - 1] + [name]
            current = [(kind, text)]
        else:
            current.append((kind, text))
    sections.append((" > ".join(path), current))

    # 2. Pack whole blocks into chunks. A block is never cut.
    chunks, seen = [], {}
    for section, blocks in sections:
        if all(kind == "heading" for kind, _ in blocks):
            continue                       # a bare heading lives on in its children's path
        occurrence = seen.get(section, 0)  # the same path twice, e.g. two "## Notes"
        seen[section] = occurrence + 1
        parts, cur = [], []
        for block in blocks:
            if cur and len(_join(cur + [block])) > max_chars:
                parts.append(cur)
                cur = _overlap(cur, overlap)
                if cur and len(_join(cur + [block])) > max_chars:
                    cur = []
            cur.append(block)
        parts.append(cur)
        for part, part_blocks in enumerate(parts):
            text = _join(part_blocks)
            key = "\n".join([url, section, str(occurrence), str(part), text])
            chunks.append({
                "url": url,
                "title": title,
                "section": section,
                "part": part,
                "text": text,
                "id": hashlib.sha256(key.encode()).hexdigest()[:32],
                "content_hash": hashlib.sha256(text.encode()).hexdigest()[:32],
                "oversized": len(text) > max_chars,
            })
    return chunks

The rules it follows, so you can change them on purpose:

The id identifies a place in a page: it hashes the URL, the section path, which occurrence of that path it is, the chunk's position in the section, and its text. Re-crawling an unchanged page produces the same ids, so your vector store can skip them. Editing a section changes ids in that section only, and renaming a heading changes the ids under it. Two identical "Note" paragraphs under different parent headings get different ids, because they are different places a citation can point to. If you would rather store repeated text once, key on content_hash instead and keep a list of locations for each hash; that is a deliberate deduplication choice, not this chunker's default. Store url and section with every embedding either way: they are what lets an answer cite its source, and what lets you delete a page's chunks when the page goes away.

Check every page before it reaches the index

A 200 response is not proof of content. The common failures are a JavaScript shell, a consent wall, a login redirect, and a block page served with a success status. Web scraping best practices covers the general version of this problem; for a Markdown pipeline the checks are short:

import re
from urllib.parse import urlsplit

REFUSED = {401, 403, 407, 429, 451}
BLOCK_HINTS = ("access denied", "verify you are human", "unusual traffic",
               "are you a robot", "request blocked")
GATE_HINTS = ("accept all cookies", "manage cookie preferences", "sign in to continue",
              "log in to continue", "enable javascript")

def check_page(result: dict, status: int, min_words: int = 80) -> tuple[str, str]:
    """Return (verdict, reason): ok, refused, http_error, challenge, gated or thin."""
    if status in REFUSED:
        return "refused", "status %d" % status
    if status >= 400:
        return "http_error", "status %d" % status
    head = (result["title"] + " " + result["markdown"][:600]).lower()
    if any(h in head for h in BLOCK_HINTS):
        return "challenge", "block or challenge page"
    if any(h in head for h in GATE_HINTS) or re.search(
            r"/(login|signin|sign-in|auth)\b", urlsplit(result["url"]).path):
        return "gated", "consent or login page"
    words = len(result["markdown"].split())
    if words < min_words:
        return "thin", "only %d words: a JavaScript shell, or a page shorter than min_words" % words
    return "ok", "%d words" % words

The hint lists are examples, not a complete detector; add the phrases your targets actually use. Pick the word floor per site, because a product page can be legitimately short. The verdicts lead to different actions. A thin page is the one case where the Playwright path can help. A refused or challenge page should be retried later at a lower rate, or dropped; a browser does not change the answer. Do not try to defeat a challenge: the site is telling you it does not want automated traffic on that path, and the robots.txt and terms for that site are where that decision belongs. A gated page needs a decision about whether that content is meant for you at all. The robots.txt tester checks a path against a pasted file.

The whole pipeline in one file

The stages above, composed. Save browser.py and chunking.py from the sections above in the same folder; pipeline.py repeats the conversion helpers and check_page so it runs on its own. It was tested on Python 3.14 with these versions:

pip install httpx==0.28.1 readability-lxml==0.9 lxml==6.1.3 lxml_html_clean==0.4.5 markdownify==1.2.3 playwright==1.64.0

Playwright is only needed for the render fallback, which the command-line part at the bottom uses. The type hints need Python 3.10 or later.

import re
from urllib.parse import urlsplit

import httpx
from lxml import html as lxml_html
from markdownify import markdownify
from readability import Document

from chunking import chunk_markdown

HEADERS = {
    "User-Agent": "my-research-bot/1.0 (+https://example.com/bot)",
    "Accept": "text/html,application/xhtml+xml",
    "Accept-Language": "en",
}
REFUSED = {401, 403, 407, 429, 451}
BLOCK_HINTS = ("access denied", "verify you are human", "unusual traffic",
               "are you a robot", "request blocked")
GATE_HINTS = ("accept all cookies", "manage cookie preferences", "sign in to continue",
              "log in to continue", "enable javascript")

def unwrap_layout_tables(fragment: str) -> str:
    tree = lxml_html.fragment_fromstring(fragment, create_parent="div")
    for table in tree.xpath("//table"):
        rows = table.xpath("./tr|./tbody/tr")
        if rows and all(len(r.xpath("./td|./th")) <= 1 for r in rows):
            table.tag = "div"
            for el in table.xpath(".//tbody|.//tr|.//td|.//th"):
                el.drop_tag()
    return lxml_html.tostring(tree, encoding="unicode")

def absolutize(fragment: str, base: str) -> str:
    tree = lxml_html.fragment_fromstring(fragment, create_parent="div")
    tree.make_links_absolute(base)
    return lxml_html.tostring(tree, encoding="unicode")

def code_language(el):
    for node in (el, el.find("code")):
        if node is not None:
            m = re.search(r"(?:language|lang)-([\w+#.-]+)", " ".join(node.get("class", [])))
            if m:
                return m.group(1)
    return ""

def html_to_markdown(html: str, url: str, infer_table_header: bool = False) -> dict:
    if not html.strip():
        return {"url": url, "title": "", "markdown": ""}
    doc = Document(html)
    main_html = absolutize(unwrap_layout_tables(doc.summary(html_partial=True)), url)
    markdown = markdownify(
        main_html,
        heading_style="ATX",
        bullets="-",
        strip=["script", "style"],
        code_language_callback=code_language,
        table_infer_header=infer_table_header,
    )
    return {"url": url, "title": doc.short_title() or "", "markdown": markdown.strip()}

def check_page(result: dict, status: int, min_words: int = 80) -> tuple[str, str]:
    """Return (verdict, reason): ok, refused, http_error, challenge, gated or thin."""
    if status in REFUSED:
        return "refused", "status %d" % status
    if status >= 400:
        return "http_error", "status %d" % status
    head = (result["title"] + " " + result["markdown"][:600]).lower()
    if any(h in head for h in BLOCK_HINTS):
        return "challenge", "block or challenge page"
    if any(h in head for h in GATE_HINTS) or re.search(
            r"/(login|signin|sign-in|auth)\b", urlsplit(result["url"]).path):
        return "gated", "consent or login page"
    words = len(result["markdown"].split())
    if words < min_words:
        return "thin", "only %d words: a JavaScript shell, or a page shorter than min_words" % words
    return "ok", "%d words" % words

def process(url: str, render=None, min_words: int = 80, proxy: str | None = None,
            infer_table_header: bool = False) -> tuple[str, str, list[dict]]:
    """Fetch, convert, check and chunk one URL. Returns (verdict, reason, chunks).

    render: optional callable(url) -> html, such as browser.rendered_html,
    tried only when the plain fetch comes back thin.
    """
    try:
        with httpx.Client(headers=HEADERS, timeout=20, follow_redirects=True,
                          proxy=proxy) as client:
            r = client.get(url)
    except httpx.HTTPError as e:
        return "network_error", type(e).__name__, []
    final_url = str(r.url)
    if r.status_code >= 400:
        verdict, reason = check_page({"url": final_url, "title": "", "markdown": ""},
                                     r.status_code, min_words)
        return verdict, reason, []
    result = html_to_markdown(r.text, final_url, infer_table_header)
    verdict, reason = check_page(result, r.status_code, min_words)
    if verdict == "thin" and render is not None:
        try:
            result = html_to_markdown(render(final_url), final_url, infer_table_header)
        except Exception as e:  # playwright's TimeoutError, a crashed browser, ...
            return "render_failed", str(e).splitlines()[0], []
        verdict, reason = check_page(result, r.status_code, min_words)
    if verdict != "ok":
        return verdict, reason, []
    return "ok", reason, chunk_markdown(result["markdown"], final_url, result["title"])

if __name__ == "__main__":
    import sys
    from functools import partial
    from browser import rendered_html

    render = partial(rendered_html, selector="main, article", min_chars=300)
    for line in open(sys.argv[1]):
        if line.strip():
            verdict, reason, chunks = process(line.strip(), render=render)
            print(verdict, len(chunks), reason, line.strip(), sep="\t")

What was tested: every function in this guide ran against local fixture pages, with no network access, embeddings or live sites. The fixtures covered a documentation page with a labeled table, relative links and a language-python block; a JavaScript shell; a page that shows "Loading..." in its main element before the article arrives; a consent page; a login redirect; 403 and 429 answers; a connection error; and Markdown with tilde fences, a code block longer than max_chars, and repeated headings under different parents. One run on fixtures does not prove extraction quality on your sites. Read a sample of each site's output before trusting a crawl.

Running it at volume through proxies

One script on one address works for a few hundred pages. At a few thousand, many sites start answering a single address with 429s or the block pages the check above catches. Spreading requests across exit addresses, at a rate each site tolerates, is the usual next step. With httpx that is one argument:

PROXY = "http://USERNAME:PASSWORD@gw-rotating_datacenter.node4.io:8083"

with httpx.Client(proxy=PROXY, headers=HEADERS, timeout=30, follow_redirects=True) as client:
    r = client.get(url)

In pipeline.py, pass the same string as process(url, proxy=PROXY).

That is Node4's Rotating Datacenter gateway. It picks an exit from the pool at random, and when it picks depends on the URL scheme. For an https:// URL the client opens a tunnel through the gateway, the exit is picked once for that tunnel, and every request a client sends over that kept-alive connection uses the same address. For a plain http:// URL the gateway picks an exit for each request, even when the client reuses its connection to the gateway. Either way a later pick can land on an address you had before, so do not count on a different address each time. To spread an HTTPS crawl, open a new client per site or per batch. To pin one exit across several requests (pagination that the site ties to an address, for example), add a session id to the username, USERNAME-session-docs42, which holds the exit for up to 30 minutes. For a full walkthrough of proxy settings in Python clients, see the Python requests proxy guide.

Playwright takes the same gateway at launch, through the proxy argument of rendered_html:

html = rendered_html(url, proxy={
    "server": "http://gw-rotating_datacenter.node4.io:8083",
    "username": "USERNAME",
    "password": "PASSWORD",
})

The Puppeteer proxy setup guide and the Puppeteer and Playwright help page cover the browser side in more depth, and proxies for browser automation compares the options for headless work.

Choosing a plan comes down to volume. Rotating Datacenter is billed per gigabyte, starting at $5.90/month for 10 GB ($0.59/GB), which suits a crawl of text-heavy pages where Markdown extraction discards most of what was downloaded. Rotating Unmetered draws from the same exits but is priced by concurrent connections with no byte meter, from $29.00/month for 250 connections ($0.116 per connection), which suits a crawler that runs all day. Both rotate across Node4's own datacenter addresses in the US, Italy and Spain; the rotating pricing page lists every size. Datacenter addresses are classified as such by sites that check, and some refuse them. For those, the residential gateway exits through household connections and costs more per gigabyte.

Or wait for one API call

Everything above is code you run and maintain: the browser fleet, the fallbacks, the block checks. Web scraping API vs proxies sets out when that trade is worth making. Node4 is building a Scrape API that will take a URL and return the rendered page as Markdown, HTML or a screenshot. It is in development and not available yet, with no launch date set. The waitlist is open, and until it launches the proxies above work with the pipeline in this guide today.