Most website problems that reach an admin’s desk are not mysterious: a page answers 404, a redirect loops, a deploy dropped the sitemap, a preview card shows the wrong image, or a robots.txt change quietly blocked a crawler. Each is quick to confirm with the right check — and easy to misdiagnose from a browser that has cached everything and runs every script.
The Web & SEO Utilities on NetTools.tech look at websites the way servers, crawlers and link-preview bots see them: status codes and headers, redirect chains, the HTML as delivered, well-known site files, links and resource sizes. This guide explains what the group covers, which tool fits which problem, and how to combine them.
What these tools cover
The category is organized in four groups:
- Website status & HTTP — HTTP Status (status code, headers, redirects), Is It Down (down for everyone or just you), Redirect Checker (every hop with status, Location and timing) and Page Size Checker (transfer size, compression and resource breakdown).
- SEO & metadata — Meta Tag Analyzer (title, description, canonical, robots, hreflang), Open Graph Checker (share previews from
og:*andtwitter:*tags), Sitemap Checker (XML sitemap validation), Broken Link Checker (404s and errors on a page or small crawl) and Website Authority Checker (Open PageRank score, rank and referring domains). - Crawlers & site policies — Robots.txt Checker (rules, syntax and a path tester), Robots.txt LLM Policy Checker (rules per AI and search crawler), llms.txt Checker (the proposed llms.txt convention) and security.txt Checker (RFC 9116 security contacts).
- URL & content utilities — URL Encoder / Decoder (percent-encoding in your browser) and URL Shortener (short links with expiry and no click tracking).
Server-side checks share one SSRF-protected fetcher: public addresses only, a limited set of ports, every redirect re-checked, size and time limits, per-client rate limits and a per-target politeness budget. Pages are parsed, never executed.
Why they matter
Search engines, social apps, AI crawlers and monitoring systems all consume your site automatically, and they react to details people never see: a 302 where a 301 was meant, a canonical pointing at the wrong host, an og:image with a relative path, a User-agent line with a typo, an expired security contact. These details decide whether pages are crawled, indexed, previewed correctly and reported to you when something is wrong.
The tools also keep you honest about what is not a fault. A link answering 403 to bots may work fine for people; a domain that is “not ranked” in a link dataset is not penalized; a missing llms.txt is normal. Each tool’s guide spells out these limits.
The tools at a glance
| Tool | What it does | Best for | Guide |
|---|---|---|---|
| HTTP Status | Check HTTP response codes and headers. | Fetch a URL and see its HTTP status code, response headers and redirect chain. | Guide |
| Is It Down | Check if a website is down for everyone or just you. | Find out whether a website is down for everyone or only for you by checking it from our server. | Guide |
| URL Encoder / Decoder | Percent-encode or decode URLs and query strings. | Encode or decode URLs, query parameters and form data (percent-encoding) instantly in your browser, with a breakdown of every URL part. | Guide |
| URL Shortener | Turn a long URL into a short link. | Create a short link for a long URL, with an optional custom alias and expiry. | Guide |
| Redirect Checker | Follow a URL’s redirect chain hop by hop. | Follow every redirect of a URL and see each hop with its HTTP status code, Location header and timing, up to the final destination. | Guide |
| Robots.txt Checker | Fetch, validate and test a site’s robots.txt. | Check a website’s robots.txt: formatted rules per user agent, syntax validation, listed sitemaps, and a tester showing whether a URL path is allowed. | Guide |
| Sitemap Checker | Validate an XML sitemap and preview its URLs. | Validate an XML sitemap or sitemap index: structure, URL count, lastmod dates and common errors, with a formatted XML preview and a table of entries. | Guide |
| Meta Tag Analyzer | Check a page’s title, description, canonical and robots tags. | Analyze a web page’s SEO metadata: title, meta description, canonical URL, robots directives, hreflang, viewport and every other meta tag, with length checks and a search-result preview. | Guide |
| Website Authority Checker | Check a domain’s Open PageRank authority score and rank. | Check the link-based authority of one or more websites: the Open PageRank score (0–10), global rank and referring domains reported by the provider, with source attribution and the data release date. | Guide |
| Open Graph Checker | Preview how a link looks when shared. | Check a page’s Open Graph and Twitter Card tags and preview the title, description and image that appear when the link is shared in social networks and chat apps. | Guide |
| Broken Link Checker | Find broken links on a web page or small site. | Find broken links: crawl a page or a small part of a site and check every link and resource for 404s, server errors, timeouts and redirects. | Guide |
| Page Size Checker | Measure a page’s download size and resources. | Check how heavy a web page is: total transfer size, compression, response headers and a breakdown of HTML, CSS, JavaScript, images and fonts. | Guide |
| Robots.txt LLM Policy Checker | See how robots.txt treats AI and search crawlers. | Check which robots.txt rules apply to AI and search crawlers such as GPTBot, ClaudeBot, Google-Extended and PerplexityBot, and whether each is allowed, restricted or partly restricted. | Guide |
| llms.txt Checker | Check a site’s llms.txt file and its structure. | Check whether a website publishes an llms.txt file and validate its Markdown structure: title, summary, sections and links, with clear recommendations. | Guide |
| security.txt Checker | Validate a site’s security.txt contact file (RFC 9116). | Check whether a website publishes security.txt for vulnerability disclosure and validate it against RFC 9116: Contact and Expires, date formats, Canonical, location and signature. | Guide |
Which tool should I use?
| Problem | Tool | Guide |
|---|---|---|
| “Is the site down or is it just me?” | Is It Down | Is It Down guide |
| What status code and headers does this URL return? | HTTP Status | HTTP Status Checker guide |
| Where does this URL redirect, and is it 301 or 302? Is there a loop? | Redirect Checker | Redirect Checker guide |
| Which links or images on my site are broken? | Broken Link Checker | Broken Link Checker guide |
| Why is this page so heavy? Is compression on? | Page Size Checker | Page Size Checker guide |
| Are title, description, canonical and robots tags correct? | Meta Tag Analyzer | Meta Tag Analyzer guide |
| Why does my link preview show the wrong title or no image? | Open Graph Checker | Open Graph Checker guide |
| Is my XML sitemap valid, and what does it list? | Sitemap Checker | Sitemap Checker guide |
| May a crawler fetch this path? Is robots.txt valid? | Robots.txt Checker | Robots.txt Checker guide |
| Which AI crawlers are allowed or blocked by robots.txt? | Robots.txt LLM Policy Checker | AI crawler policy guide |
| Is our llms.txt structured correctly? | llms.txt Checker | llms.txt Checker guide |
| Is our vulnerability reporting contact published and current? | security.txt Checker | security.txt Checker guide |
| How well-linked is this domain compared with others? | Website Authority Checker | Website Authority Checker guide |
| How do I encode a query parameter or decode a mangled URL? | URL Encoder / Decoder | URL Encoder / Decoder guide |
| I need a short link with an expiry and no tracking. | URL Shortener | URL Shortener guide |
If a site does not resolve at all, start one layer lower with the DNS tools; if it resolves but does not connect, use the network diagnostics (see the related guides below).
Typical workflows
Launch checklist for a new site
- HTTP Status on the home page and a few key URLs: 200, sensible headers.
- Redirect Checker on
http://and the non-preferred host (wwwor not): one permanent hop to the canonical HTTPS URL. - Robots.txt Checker to make sure the staging-era
Disallow: /is gone, then the Sitemap Checker on the sitemap it lists. - Meta Tag Analyzer and Open Graph Checker on the main page templates.
- Broken Link Checker with Crawl the site, then Page Size Checker on the heaviest template.
Domain or URL-structure migration
- List old URLs and run them through the Redirect Checker: each should reach its new address in one 301 or 308 hop.
- Crawl the new site with the Broken Link Checker and fix internal links that still redirect.
- Check canonical tags on the new pages with the Meta Tag Analyzer, and that the new sitemap lists only new URLs.
- Compare old and new domains in the Website Authority Checker over the following provider data releases.
Deciding and verifying an AI crawler policy
- Review the current file with the Robots.txt LLM Policy Checker, selecting training, search and user-fetch crawlers.
- Edit robots.txt, paste the draft into the checker and test important paths per crawler — watch for named groups that drop your
*rules. - Confirm with the Robots.txt Checker that search crawlers such as Googlebot are unaffected.
- Optionally publish an llms.txt and validate it with the llms.txt Checker, keeping in mind it is a proposed convention, not a standard.
Quarterly site-policy hygiene
- security.txt Checker: is Expires still in the future and the contact current?
- Robots.txt Checker and Sitemap Checker: did a CMS update change either file?
- Broken Link Checker on the most-linked pages for outbound link rot.
Related guides
Tool guides in this category:
- HTTP Status Checker, Is It Down, Redirect Checker, Page Size Checker
- Meta Tag Analyzer, Open Graph Checker, Sitemap Checker, Broken Link Checker, Website Authority Checker
- Robots.txt Checker, Robots.txt AI crawler policy, llms.txt Checker, security.txt Checker
- URL Encoder / Decoder, URL Shortener
Neighboring categories:
- DNS & domain tools — when a site does not resolve or points to the wrong server.
- Network diagnostics — when a host resolves but does not respond.
- Security testing tools — TLS certificates and a non-intrusive website security assessment.
All tools are listed on the tools overview.
Frequently asked questions
Do these tools run in my browser or on a server?
It depends on the tool. The URL Encoder runs entirely in your browser, and the robots.txt, llms.txt, security.txt, meta tag, Open Graph and sitemap checkers can analyze pasted content locally. Fetching live pages, following redirects, crawling links and measuring page size are done by the NetTools.tech server through an SSRF-protected fetcher.
Why does a tool show different results than my browser?
The server requests pages from its own network, with its own user agent, no cookies and no JavaScript execution. Sites that vary content by location, consent, login or user agent, or that block bots, can answer differently than they do for you.
Can these tools check internal or staging sites?
Not by fetching. Server-side requests only go to public internet addresses on ports 80, 443, 8080 and 8443. For staging pages, use paste mode where a tool offers it.
Are checked URLs stored?
No. The web tools return results to your browser and do not store the URLs or fetched content; the URL Shortener stores only the code, destination, creation time and expiry for each link it creates. The Website Authority Checker sends domains to its data provider, Open PageRank, and keeps results briefly in a cache.
Does a good result in these tools guarantee better search rankings?
No. They check technical prerequisites — reachable pages, correct redirects, readable metadata, crawl rules — and report link-based authority from one dataset. Rankings depend on many factors none of these tools can measure.
Which tool do I use for AI crawlers?
The Robots.txt LLM Policy Checker shows which robots.txt rules apply to AI crawlers such as GPTBot, ClaudeBot or PerplexityBot. The llms.txt Checker validates the proposed llms.txt file, which points AI tools to content but grants or blocks nothing.