Unverified60% confidenceFactExact time
What a crawler retrieves is only unrendered HTML source code, because crawlers lack the rendering capabilities of a browser.
2
Sources
60%
Confidence
Long-term
Relevance
7/8/2026
First Seen
Sources
Learning Python Web Scraping from Scratch: How It Works, the Full Workflow, and Legal Boundaries
bilibiliPython全套学习7/6/2026
Related Claims
UnverifiedA crawler program sends HTTP requests to a server via code, and the server returns raw content; however, since the crawler lacks a browser's rendering capability, what it receives is the raw HTML source code of the webpage77% similarUnverified如果页面内容需要JavaScript才能显示，在使用非JS爬虫抓取时无法获取该内容，这本身就是一个发现68% similarUnverifiedAI search crawlers generally do not execute JavaScript or execute only very limited scripts, so what they see is the initial HTML returned by the server; the body text of client-side rendered (CSR) pages may be empty in the crawler's eyes.67% similarUnverifiedAI crawlers only download HTML and do not execute JavaScript, so website analytics tools that rely on browser-side JavaScript cannot count AI crawler visits.66% similarUnverifiedMainstream AI crawlers such as GPTBot, ClaudeBot and PerplexityBot currently largely do not execute JavaScript, so for sites that depend on JavaScript rendering, even if both robots.txt and the WAF let them through, the content they get is still empty.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/281187API
curl https://kongchang.com/api/v1/knowledge/claims/281187MCP
get_claim(id=281187)