Shadow DOM web scraping: why your selectors keep silently failing

If you've scraped or automated a page built with web components and gotten back an empty result — no error, no stack trace, just nothing where a button should be — you've hit the shadow DOM problem. It's one of the more annoying failure modes in browser automation because it doesn't look like a failure. Your script runs, your selector matches zero elements, and you're left guessing whether the page changed, the timing was off, or something more fundamental broke.

Usually it's the third one.

What shadow DOM actually breaks

Shadow DOM is a browser feature that lets a component encapsulate its own internal markup, styling, and structure, hiding it from the rest of the page. It's how design systems like Shoelace, Material Web, and a growing number of component libraries ship self-contained, style-isolated widgets. From a frontend engineering standpoint, it's a genuine improvement — no more CSS leaking across component boundaries, no more naming collisions.

From a scraping standpoint, it's a wall.

document.querySelectorAll() does not cross shadow boundaries by default. If a button lives inside a component's shadow root, a standard DOM query run against the main document simply doesn't see it. It's not hidden, not disabled, not lazy-loaded — it's structurally invisible to anything that isn't shadow-DOM-aware. Most scraping and browser-automation tooling was written assuming a single flat DOM tree, so this is where a lot of "why is my selector returning nothing" debugging sessions end up.

There's a second, quieter version of the same problem: slotted content. Web components use <slot> elements to let you pass content from the light DOM into specific positions inside the shadow tree. A component like Shoelace's <sl-button> might render as:

<sl-button>
  <span>Add to cart</span>
</sl-button>

The visible text lives in the light DOM, but the actual interactive element — the shadow-rendered <button> inside <sl-button>'s shadow root — has no text content of its own. If your accessible-name resolution logic only looks at textContent or aria-label on the element itself, it comes back empty, even though a sighted user (or a screen reader, which handles slot assignment correctly) sees "Add to cart" plainly.

This is the bug that's easy to miss in testing and painful to find in production: everything looks fine when you eyeball the rendered page, and everything is fine when you inspect the light DOM markup. The failure only shows up when you try to programmatically resolve what that button is actually called.

Traversal: getting past the boundary

The fix for the first problem is mechanical but has to be applied consistently. Any DOM-walking code — query selectors, element enumeration, accessibility tree construction — needs to recursively descend into shadowRoot wherever one exists, not just walk children or childNodes. In practice that means a helper that checks element.shadowRoot at every node and, if present, continues the traversal from there instead of stopping.

The complication is that shadow roots nest. A design system component might itself be built from other design system components, each with its own shadow root. A traversal function that only goes one level deep will work on toy examples and then quietly stop resolving elements the moment someone nests a <sl-dropdown> inside a <sl-card>. The traversal has to be genuinely recursive, and it needs to handle open shadow roots (the common case, accessible via element.shadowRoot) while gracefully skipping closed ones, which are — by design — not inspectable from outside the component at all.

Slotted content: resolving the real accessible name

Getting past the shadow boundary only solves half the problem. Once you can see the shadow-rendered <button> inside <sl-button>, you still need to figure out what it's called — and the answer isn't inside the shadow root, it's in the light DOM content that got projected into the component's <slot>.

The correct resolution order, roughly, mirrors how the browser's accessibility tree actually computes accessible names:

1. Check aria-label on the host element first — an explicit label always wins.

2. If there's a <slot> inside the shadow root, resolve slot.assignedNodes() (or assignedElements() for element nodes) to find what content the light DOM actually assigned to it.

3. Fall back to the shadow-rendered element's own text content only if neither of the above produces anything.

That middle step is the one that's easy to skip and the one that actually matters. assignedNodes() gives you the real projected content — text nodes, nested elements, whatever the page author put between the component's opening and closing tags — regardless of how deep the shadow tree that eventually renders it goes. Once you resolve names this way, <sl-button>, <sl-dropdown>, <sl-menu-item>, and similar slotted components stop being blank spots and start resolving to their actual, human-meaningful labels.

Why this stays broken silently

The reason shadow DOM bugs are worse than most scraping bugs is that they fail without an error. A missing element because of a race condition throws a timeout. A missing element because of a shadow boundary just isn't there — your code has no signal that anything went wrong. You ship it, it works on every page you tested against, and then it silently drops actions on the first site that adopts a shadow-DOM-based component library.

The practical mitigation, beyond fixing the traversal itself, is instrumentation: log when a query returns zero results against a page that clearly has interactive content, and treat suspiciously empty extractions as a signal worth investigating rather than an edge case to ignore. Golden regression tests — a captured, hand-verified snapshot of what a page's interactive elements should resolve to — are worth the setup cost specifically because this class of bug doesn't announce itself any other way. We added one against a real Shoelace-based product page after finding this exact bug, and it's caught regressions since that a manual smoke test never would have.

The takeaway

Shadow DOM and slotted content aren't edge cases anymore — they're the default rendering strategy for a growing share of modern component libraries. Any scraping, testing, or agent-facing tooling that assumes a flat DOM tree will eventually hit a page where it returns confidently empty results for elements that are sitting right there, one shadow boundary away. The fix isn't exotic: recursive shadow-root traversal, plus accessible-name resolution that actually checks slot assignment before giving up. It's a small amount of code. It's just code most tooling doesn't have, because the failure mode doesn't show up until someone's production page uses it.