EvidaLux Tools Validation Report
Every run of our 64 audit tools — behaviour lock, agreement with external references, and real-site oracles. Methodology over marketing — bad numbers publish too.
Generated:
2026-05-21T20:51:25.847276+00:00This is an archived snapshot. For the latest, see here.
Section 1 — Cross-Tool Agreement
Diff against industry-reference tools on the same 10-site cohort.
cohort: 2026-05-cohort117 of 18 plugins measured · median agreement 1.00
Cross-tool snapshot from 2026-05-16 — 5 days stale, awaiting next cron run.
PEND rows mean the reference-tool integration is queued. Each integration may use a different per-site match metric (see plugin.metric). Median agreement is taken across implemented rows only.
| Plugin | Reference tool | Band | Agreement | Agreed / Compared | Comparison metric |
|---|---|---|---|---|---|
accessibility.axe | Pa11y (htmlcs runner) | GREEN | 1.00 | 10 / 10 | hierarchical WCAG agreement (Pa11y htmlcs vs plugin axe-core): T1 filtered-SC Jaccard ≥0.50 · T2 violation-presence boolean · T3 noise tolerance (one side empty ≤ engine noise floor 2). WCAG 4.1.1 (deprecated in 2.2) excluded from both sides. |
compliance.iab_tcf | httpx + lxml + Set-Cookie scan | GREEN | 1.00 | 10 / 10 | boolean TCF-surface agreement (independent vantage: Set-Cookie inspection for euconsent-v2 + raw-text JS-marker scan + lxml DOM parse — plugin uses BeautifulSoup + script-body scan only; empty/empty = both agree page has no IAB CMP) |
compliance.iab_tcf_verified | Playwright + iab-tcf (TC-string decode) | GREEN | 0.90 | 9 / 10 | cmp_id agreement (independent Playwright probe + iab-tcf library decode of tcString vs plugin's JS-callback cmpId; divergence = CMP misconfiguration. Empty/empty = both unable to probe — common when playwright is missing from env) |
quality.lighthouse_perf | Google PageSpeed Insights API | GREEN | 0.90 | 9 / 10 | accessibility delta ≤10 vs PSI v5 (strict, DOM-static — the environment-stable axis); performance + best-practices deltas ≤45 each (sanity cap — Lighthouse FAQ documents these categories as throttle/network-divergent between PSI's GCE upstream + production Chromium and our local subprocess; agreement metric reports per-category deltas in detail for transparency rather than averaging the strict a11y delta into the noisy perf/bp deltas) |
quality.owasp_zap_scan | nuclei cve+exposure+misconfig templates | GREEN | 1.00 | 10 / 10 | HIGH/CRITICAL presence boolean (nuclei cve+exposure+misconfig templates vs plugin's ZAP spider+active-scan alerts; mirror of vulnerability_nuclei↔ZAP — set-Jaccard on rule IDs is undefined across two engines with disjoint namespaces; agreement = both surface at least one HIGH/CRITICAL OR both are clean of HIGH/CRITICAL) |
quality.vulnerability_nuclei | OWASP ZAP REST API (baseline + passive) | PEND | — | — | Integration queued |
search.lighthouse_seo | Google PageSpeed Insights API | GREEN | 1.00 | 9 / 9 1 of 10 sites unmeasured | seo score within ±10 of PSI v5 (mobile+desktop avg) |
security.exposed_files | nuclei http/exposures/{files,configs} templates | GREEN | 1.00 | 10 / 10 | leaked-path set-Jaccard ≥0.50 (nuclei http/exposures/{files,configs} templates vs plugin's custom path-prober + content-sniffer; both emit a set of leaked URL paths per origin; empty/empty = both agree origin has no exposures) |
security.headers | Mozilla Observatory v2 (MDN) | YELLOW | 0.80 | 8 / 10 | overall score within ±15 of Mozilla Observatory v2 score |
security.tls_deep | SSL Labs API | GREEN | 1.00 | 9 / 9 1 of 10 sites unmeasured | A-F grade-band agreement within ±1 letter (Qualys SSL Labs API v3 vs plugin's score-deduction grader; engines disagree on HSTS-preload + OCSP-stapling weightings the plugin's v1.0 doesn't probe, so exact-grade equality would over-penalize — the broad health bucket is the defensible signal) |
seo.broken_links | linkchecker (W3C) | YELLOW | 0.78 | 7 / 9 1 of 10 sites unmeasured | in-domain set-Jaccard ≥0.50 OR union ≤2 small-set noise floor (linkchecker --recursion-level=1 --check-extern; external broken findings counted in transparency but excluded from agreement; small-set floor recognises that heterogeneous crawlers — linkchecker BFS vs our SharedFetcher — legitimately diverge on the single-broken-URL case from JS-injection, sitemap-only references, or anti-bot 403/200 flips) |
seo.canonical_audit | lxml + httpx canonical-chain follower | GREEN | 0.90 | 9 / 10 | 4-axis canonical classification agreement ≥3/4 (canonical_present / self_canonical / cross_host / chain_two_hop). Reference = vanilla httpx + lxml clean-room reimplementation of the plugin's chain-follow logic. Independent fetcher (no SharedFetcher etag/cache layer) and parser (lxml vs BeautifulSoup4). Plan §3.2 nominally maps this to Lighthouse SEO subset, but PSI's canonical audit doesn't follow chains — would silently drop the plugin's distinguishing checks; this row's purpose is to cross-tool exactly those. |
seo.hreflang_validator | langcodes (BCP 47) + lxml | GREEN | 1.00 | 10 / 10 | set-equality on invalid hreflang codes (langcodes/BCP 47 vs plugin regex; same page parsed independently via lxml vs BeautifulSoup; empty/empty = no hreflang or all codes pass) |
seo.meta_tags | Google PageSpeed Insights API (Lighthouse SEO audits subset) | GREEN | 0.90 | 9 / 10 | 6-axis meta-tag feature-vector agreement ≥5/6 (title / description / viewport / canonical / is-crawlable / html-has-lang). Reference = PSI Lighthouse SEO audits (charitable OR across mobile+desktop strategies); plugin = BeautifulSoup raw-HTML parse. H1-count + Open Graph completeness excluded — no Lighthouse equivalent (kept in plugin output for customer reports). Lighthouse renders real Chromium with JS, so SPA-injected meta tags surface as divergence — the cross-tool's most valuable signal in this row. |
seo.robots_txt_audit | Google robotstxt parser | GREEN | 1.00 | 10 / 10 | boolean homepage indexability agreement (Protego vs plugin verdict on User-agent: *; empty/empty = both report no readable robots.txt) |
seo.structured_data | validator.schema.org | GREEN | 0.90 | 9 / 10 | schema.org type-set Jaccard ≥0.50 (validator.schema.org vs plugin JSON-LD/Microdata/RDFa; Google Rich Results API retired 2024) |
tech.dns_health | dig +dnssec + Google DNS-over-HTTPS | GREEN | 1.00 | 10 / 10 | 7-axis DNS posture feature-vector agreement ≥6/7 (AAAA / NS≥2 / SPF / DMARC-present / DMARC-strong / CAA / DNSSEC). Reference = OR of dig (BIND binary, system resolver) and Google DoH (independent resolver via HTTPS); plugin = dnspython. DKIM excluded — selector probing is informational only (selectors are private, miss ≠ absence). |
tech.stack_detection | python-Wappalyzer (open-source fingerprint engine) | YELLOW | 0.78 | 7 / 9 1 of 10 sites unmeasured | set-Jaccard ≥0.30 within the plugin-detectable tech universe (python-Wappalyzer 2,000+ fingerprints filtered to the ~40 techs our PATTERNS table claims to fingerprint, so detector-coverage gaps — databases, OSes, build tools — don't count as plugin failures; name aliases normalised; small-set floor when filtered union ≤2; full Wappalyzer detections surfaced in ours_only_outside_universe / ref_only_outside_universe for transparency) |
Section 2 — Real-World Correctness
Plugin findings on random real sites (US/Europe/Turkey/Arab/South America) scored against a real-browser + HTTP ground truth. Unreachable sites are INCONCLUSIVE (never gate); a gate failure is a false positive/negative on a reachable site.
| Region | When | Verdict | Reachable | Inconclusive | FP | FN |
|---|---|---|---|---|---|---|
| arab, europe, south_america, turkey, us | 2026-07-20 02:41 UTC | FAIL | 86 | 14 | 1 | 0 |
| arab, europe, south_america, turkey, us | 2026-07-19 02:37 UTC | FAIL | 88 | 12 | 1 | 0 |
| arab, europe, south_america, turkey, us | 2026-07-18 20:31 UTC | PASS | 85 | 15 | 0 | 0 |
| arab, europe, south_america, turkey, us | 2026-07-18 19:22 UTC | FAIL | 84 | 16 | 5 | 0 |
| arab, europe, south_america, turkey, us | 2026-07-18 18:18 UTC | PASS | 18 | 7 | 0 | 0 |
| arab, europe, south_america, turkey, us | 2026-07-18 18:00 UTC | FAIL | 20 | 5 | 3 | 0 |
| arab, europe, south_america, turkey, us | 2026-07-18 16:09 UTC | FAIL | 20 | 5 | 44 | 1 |
| arab, europe, south_america, turkey, us | 2026-07-04 06:08 UTC | FAIL | 20 | 5 | 1 | 1 |
| arab, europe, south_america, turkey, us | 2026-07-03 06:20 UTC | FAIL | 19 | 6 | 1 | 1 |
Latest run — 2026-07-20 02:41 UTC
| Plugin | Confirmed | FP | FN | Inconclusive |
|---|---|---|---|---|
accessibility.eaa_mapping | 0 | 0 | 0 | 155 |
compliance.accessibility_statement | 74 | 0 | 0 | 98 |
compliance.age_verification | 0 | 0 | 0 | 86 |
compliance.ai_disclosure | 0 | 0 | 0 | 86 |
compliance.child_consent | 0 | 0 | 0 | 86 |
compliance.cookie_consent | 0 | 0 | 0 | 210 |
compliance.cross_border_transfer | 0 | 0 | 0 | 172 |
compliance.dark_pattern | 86 | 0 | 0 | 172 |
compliance.data_subject_request | 8 | 0 | 0 | 78 |
compliance.dpo_contact | 0 | 0 | 0 | 86 |
compliance.eu_representative | 0 | 0 | 0 | 86 |
compliance.geo_consistency | 0 | 0 | 0 | 172 |
compliance.iab_tcf | 0 | 0 | 0 | 86 |
compliance.iab_tcf_verified | 6 | 0 | 0 | 80 |
compliance.legal_disclosure | 82 | 0 | 0 | 176 |
compliance.odr_link | 86 | 0 | 0 | 0 |
compliance.pay_or_consent_wall | 0 | 0 | 0 | 86 |
compliance.pricing_indication | 0 | 0 | 0 | 172 |
compliance.privacy_policy_content | 36 | 0 | 0 | 116 |
compliance.purchase_disclosure | 0 | 0 | 0 | 172 |
compliance.required_pages | 274 | 0 | 0 | 70 |
quality.api_test | 0 | 0 | 0 | 222 |
security.exposed_files | 157 | 0 | 0 | 13 |
security.headers | 608 | 0 | 0 | 303 |
security.tls_deep | 246 | 1 | 0 | 93 |
seo.broken_links | 38 | 0 | 0 | 28 |
seo.canonical_audit | 51 | 0 | 0 | 4 |
seo.duplicate_content | 0 | 0 | 0 | 85 |
seo.freshness | 34 | 0 | 0 | 28 |
seo.hreflang_validator | 74 | 0 | 0 | 5 |
seo.meta_tags | 158 | 0 | 0 | 0 |
seo.robots_txt_audit | 17 | 0 | 0 | 1 |
seo.sitemap | 84 | 0 | 0 | 3 |
seo.structured_data | 80 | 0 | 0 | 9 |
tech.dns_health | 344 | 0 | 0 | 172 |
tech.stack_detection | 0 | 0 | 0 | 86 |
Wrong findings — latest run (gate)
false_positivesecurity.tls_deep/tls.no_https — https://www.isbank.com.tr/ — TLS handshake succeeded independently