VigilVigil

Sole engineer: the verification engine, the run manager, the reports and the bilingual interface.

A status code is not an answer. Plenty of "dead" links open fine in a browser, and plenty of 200 OK pages are parked domains, empty pages, server welcome pages or "not found" pages in disguise. Simple link checkers either cry wolf or miss real problems, and when your own connection drops they mark a thousand healthy sites as dead.

Vigil collects independent evidence and only calls a link dead when the layers agree. Each address is normalised (missing schemes, www, Arabic and other international domains, stray punctuation, duplicates) and checked as given first. DNS failures get a second opinion from independent resolvers, HTTP follows redirects and fingerprints the content, TCP and TLS diagnostics explain connection and certificate problems, and anything still unclear, such as a bot challenge or a JavaScript-only page, goes to a real browser.

When the original address fails, Vigil tries the other scheme, the www variant and the site root and reports which one works. Failures are re-checked at low pressure before they are trusted, and canary checks pause the run if the machine itself goes offline. A site that only opens with certificate checks turned off (expired, self-signed or issued for another name) gets its own "certificate problem" verdict, and the reason names the problem. Every link ends with one of nine verdicts plus issue flags, a confidence score, a plain-language reason in English or Arabic and the full probe timeline.

A Python FastAPI engine with httpx (HTTP/2), dnspython, cryptography for certificate details, selectolax for content analysis and Playwright Chromium for the browser layer. Runs live in SQLite, survive restarts and resume where they stopped; an adaptive governor keeps CPU and memory under their ceilings and a per-host limit keeps runs polite. A Next.js 16 static interface with React 19, Tailwind CSS 4 and Motion is served by the API and streams progress over server-sent events. Links come in from pasted text, spreadsheets, a web page, a sitemap, a REST API or a CLI.

Telling apart the many ways a site can look broken or healthy without being so: bot challenges, rate limits, soft 404s, login walls, maintenance pages, parked domains, expired or self-signed certificates and JavaScript-only apps. Keeping very large runs fast, with a server-side filtered results table and reports that open safely in Excel.

Use the cheapest layer that gives a sure answer and escalate to a real browser only when needed. Never hide a failure behind a working variant: report both. Neutralise spreadsheet formula injection in exports and protect the API against DNS rebinding.

On a CRM-style spreadsheet of 85 rows written the way people write addresses (with and without https, bare domains, www names, an Arabic domain, duplicates and a typo) against 30 test sites that reproduce real-world cases, Vigil found the address column on its own and finished in 52 s with an average confidence of 95.9%. It caught a payment page behind a bot challenge (decided by the real browser), a rate-limited API, a soft 404, expired, self-signed and soon-expiring certificates, a redirect loop, a login wall and a domain that only works with www. The same file through the CLI gave identical counts.