Skip to content

Guard scraper against private network URLs#5137

Open
pragnyanramtha wants to merge 1 commit into
Aider-AI:mainfrom
pragnyanramtha:fix-5075-scraper-ssrf-guard
Open

Guard scraper against private network URLs#5137
pragnyanramtha wants to merge 1 commit into
Aider-AI:mainfrom
pragnyanramtha:fix-5075-scraper-ssrf-guard

Conversation

@pragnyanramtha
Copy link
Copy Markdown

@pragnyanramtha pragnyanramtha commented May 16, 2026

Summary

  • block scraper requests to private, loopback, link-local, metadata, multicast, and otherwise non-public network targets before fetching
  • resolve and validate hostname targets, revalidate redirects, disable proxy/env routing for HTTPX, and bind HTTPX connections to the validated resolved address
  • avoid browser DNS rebinding by only using Playwright for literal IP URLs, with route-level blocking for unsafe browser requests

Testing

  • .venv/bin/python -m pytest tests/scrape/test_scrape.py -k "private or multicast or rebinding or network_backend or playwright_route"
  • .venv/bin/python -m pytest tests/scrape/test_scrape.py tests/scrape/test_playwright_disable.py
  • .venv/bin/python -m flake8 aider/scrape.py tests/scrape/test_scrape.py tests/scrape/test_playwright_disable.py
  • .venv/bin/python -m compileall aider/scrape.py tests/scrape/test_scrape.py tests/scrape/test_playwright_disable.py
  • git diff --check

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 16, 2026

CLA assistant check
All committers have signed the CLA.

@pragnyanramtha pragnyanramtha force-pushed the fix-5075-scraper-ssrf-guard branch from e301c89 to 63db679 Compare May 16, 2026 01:34
@pragnyanramtha pragnyanramtha force-pushed the fix-5075-scraper-ssrf-guard branch from 63db679 to a9a1db6 Compare May 16, 2026 01:43
@pragnyanramtha pragnyanramtha marked this pull request as ready for review May 16, 2026 21:48
Copilot AI review requested due to automatic review settings May 16, 2026 21:48
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants