EvidaLux Tools Validation Report
Every run of our 64 audit tools — behaviour lock, agreement with external references, and real-site oracles. Methodology over marketing — bad numbers publish too.
Generated:
2026-07-17T14:01:18.481216+00:00This is an archived snapshot. For the latest, see here.
Section 1 — Cross-Tool Agreement
Diff against industry-reference tools on the same 10-site cohort.
cohort: pool-v111 of 22 plugins measured · cohort coverage 89% · median agreement 0.93
PEND rows mean the reference-tool integration is queued. Each integration may use a different per-site match metric (see plugin.metric). Median agreement is taken across implemented rows only, and each row's agreement is over the sites that could be measured — read it next to coverage, not alone. A row whose coverage is at or below 50% cannot be GREEN (band_capped_by_coverage).
| Plugin | Reference tool | Band | Agreement | Agreed / Compared | Comparison metric |
|---|---|---|---|---|---|
accessibility.axe | Pa11y (htmlcs runner) | ERR | — | — | https://www.hepsiburada.com/: FileNotFoundError: [Errno 2] No such file or directory: 'pa11y'; https://www.zalando.de/: |
compliance.iab_tcf | httpx + lxml + Set-Cookie scan | GREEN | 1.00 | 13 / 13 2 of 15 sites unmeasured | boolean TCF-surface agreement (independent vantage: Set-Cookie inspection for euconsent-v2 + raw-text JS-marker scan + lxml DOM parse — plugin uses BeautifulSoup + script-body scan only; empty/empty = both agree page has no IAB CMP) |
compliance.iab_tcf_verified | Playwright + iab-tcf (TC-string decode) | PEND | — | — | Integration queued |
quality.api_test | schemathesis (OpenAPI / Swagger spec loader) | GREEN | 1.00 | 10 / 10 5 of 15 sites unmeasured | discovered-endpoint set-Jaccard ≥0.40 with mutual-empty match + |union| ≤2 small-set noise floor. Reference = schemathesis OpenAPI 2.0 / 3.x loader, probing 13 common spec locations (/openapi.json, /swagger.json, /v3/api-docs, /api-docs, /api/openapi.json, ...); spec content-sniffed (must contain `openapi`/`swagger` token) so HTML catch-alls don't false-positive. Plugin = quality.api_test's `_discover_endpoints` discovery sub-component called DIRECTLY (bypassing the registry runner) so the 50-request rate-limit burst probe doesn't fire — the cross-tool is comparing the discovery surface, and firing the burst against 10 cohort sites daily would add ~500 net-new requests/day of noise without signal. Paths normalised both sides: {param} placeholders + concrete digit / UUID / long-hex segments → {}, lowercased, trailing slash stripped — so spec-driven enumeration and convention-probe heuristics meet at the static-prefix shape. 0.40 threshold (vs 0.50 elsewhere) reflects fundamental engine heterogeneity (spec enumeration vs convention probing); most cohort sites hit mutual-empty match (no public OpenAPI spec AND no responding conventional /api* path). |
quality.lighthouse_perf | Google PageSpeed Insights API | ERR | — | — | https://www.hepsiburada.com/: RuntimeError: lighthouse runner failed: lighthouse binary not on PATH; https://www.zalando |
quality.owasp_zap_scan | nuclei cve+exposure+misconfig templates | ERR | — | — | https://www.hepsiburada.com/: FileNotFoundError: [Errno 2] No such file or directory: 'nuclei'; https://www.zalando.de/: |
quality.responsive_test | Google PageSpeed Insights API (Lighthouse mobile-friendly audits subset) | ERR | — | — | https://www.hepsiburada.com/: RuntimeError: GOOGLE_PSI_API_KEY env var is required for the PSI cross-tool integration (s |
quality.vulnerability_nuclei | OWASP ZAP REST API (baseline + passive) | PEND | — | — | Integration queued |
search.lighthouse_seo | Google PageSpeed Insights API | ERR | — | — | https://www.hepsiburada.com/: RuntimeError: lighthouse runner failed: lighthouse binary not on PATH; https://www.zalando |
security.exposed_files | nuclei http/exposures/{files,configs} templates | ERR | — | — | https://www.hepsiburada.com/: FileNotFoundError: [Errno 2] No such file or directory: 'nuclei'; https://www.zalando.de/: |
security.headers | Mozilla Observatory v2 (MDN) | RED | 0.64 | 9 / 14 1 of 15 sites unmeasured | overall score within ±15 of Mozilla Observatory v2 score. CAVEAT (#661): the two scores are not on the same scale — Mozilla grades 50 as C and 30 as D, the same numbers are D and E here — and Observatory v2 no longer exposes the per-test breakdown that would allow a header-by-header comparison. So ±15 across two different rulers is a rough proxy, not a like-for-like measurement, and this row should be read as weaker evidence than the others. |
security.tls_deep | SSL Labs API | GREEN | 0.93 | 14 / 15 | A-F grade-band agreement within ±1 letter (Qualys SSL Labs API v3 vs plugin's score-deduction grader; engines disagree on HSTS-preload + OCSP-stapling weightings the plugin's v1.0 doesn't probe, so exact-grade equality would over-penalize — the broad health bucket is the defensible signal) |
seo.broken_links | linkchecker (W3C) | ERR | — | — | https://www.hepsiburada.com/: FileNotFoundError: [Errno 2] No such file or directory: 'linkchecker'; https://www.zalando |
seo.canonical_audit | lxml + httpx canonical-chain follower | GREEN | 1.00 | 13 / 13 2 of 15 sites unmeasured | 4-axis canonical classification agreement ≥3/4 (canonical_present / self_canonical / cross_host / chain_two_hop). Reference = vanilla httpx + lxml clean-room reimplementation of the plugin's chain-follow logic. Independent fetcher (no SharedFetcher etag/cache layer) and parser (lxml vs BeautifulSoup4). Plan §3.2 nominally maps this to Lighthouse SEO subset, but PSI's canonical audit doesn't follow chains — would silently drop the plugin's distinguishing checks; this row's purpose is to cross-tool exactly those. |
seo.freshness | httpx + lxml independent sitemap lastmod walker | GREEN | 0.91 | 10 / 11 4 of 15 sites unmeasured | stale_ratio agreement within ±0.20 (httpx + lxml independent sitemap.xml lastmod walker vs plugin's combined sitemap + JSON-LD + HTTP Last-Modified scan; same 540-day stale cut-off, 50-URL cap; clean-room ref intentionally skips secondary signals so any plugin regression in sitemap parsing surfaces in the agreement number) |
seo.hreflang_validator | langcodes (BCP 47) + lxml | GREEN | 1.00 | 13 / 13 2 of 15 sites unmeasured | set-equality on invalid hreflang codes (langcodes/BCP 47 vs plugin regex; same page parsed independently via lxml vs BeautifulSoup; empty/empty = no hreflang or all codes pass) |
seo.meta_tags | Google PageSpeed Insights API (Lighthouse SEO audits subset) | ERR | — | — | https://www.hepsiburada.com/: RuntimeError: GOOGLE_PSI_API_KEY env var is required for the PSI cross-tool integration (s |
seo.robots_txt_audit | Google robotstxt parser | YELLOW | 0.87 | 13 / 15 | boolean homepage indexability agreement (Protego vs plugin verdict on User-agent: *; empty/empty = both report no readable robots.txt) |
seo.sitemap | httpx + lxml inline-XSD sitemap.org 0.9 validator | YELLOW | 0.80 | 12 / 15 | sitemap presence boolean parity + URL count delta ≤ max(10% of larger, 2 small-set noise floor). Reference = vanilla httpx + lxml with inline sitemap.org 0.9 XSD (urlset + sitemapindex schemas embedded clean-room, not fetched). Independent fetcher (no SharedFetcher), independent XSD-validation pipeline (plugin only well-formedness-checks). Both sides mirror plugin discovery (robots.txt Sitemap: directives → /sitemap.xml fallback, cap 5 docs) and count top-level <loc> only — no recursion into sitemap-index children — so URL counts are apples-to-apples; ref XSD-valid count surfaced in per-site detail for schema-conformance transparency. |
seo.structured_data | validator.schema.org | PEND | — | — | Integration queued |
tech.dns_health | dig +dnssec + Google DNS-over-HTTPS | GREEN | 1.00 | 15 / 15 | 7-axis DNS posture feature-vector agreement ≥6/7 (AAAA / NS≥2 / SPF / DMARC-present / DMARC-strong / CAA / DNSSEC). Reference = OR of dig (BIND binary, system resolver) and Google DoH (independent resolver via HTTPS); plugin = dnspython. DKIM excluded — selector probing is informational only (selectors are private, miss ≠ absence). |
tech.stack_detection | python-Wappalyzer (open-source fingerprint engine) | GREEN | 0.92 | 12 / 13 2 of 15 sites unmeasured | set-Jaccard ≥0.30 within the plugin-detectable tech universe (python-Wappalyzer 2,000+ fingerprints filtered to the ~40 techs our PATTERNS table claims to fingerprint, so detector-coverage gaps — databases, OSes, build tools — don't count as plugin failures; name aliases normalised; small-set floor when filtered union ≤2; full Wappalyzer detections surfaced in ours_only_outside_universe / ref_only_outside_universe for transparency) |
Section 2 — Real-World Correctness
Plugin findings on random real sites (US/Europe/Turkey/Arab/South America) scored against a real-browser + HTTP ground truth. Unreachable sites are INCONCLUSIVE (never gate); a gate failure is a false positive/negative on a reachable site.
| Region | When | Verdict | Reachable | Inconclusive | FP | FN |
|---|---|---|---|---|---|---|
| arab, europe, south_america, turkey, us | 2026-10-21 02:59 UTC | FAIL | 85 | 15 | 0 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-20 03:03 UTC | FAIL | 87 | 13 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-19 02:57 UTC | FAIL | 87 | 13 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-18 03:36 UTC | FAIL | 86 | 14 | 2 | 4 |
| arab, europe, south_america, turkey, us | 2026-10-18 02:57 UTC | FAIL | 87 | 13 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-17 02:54 UTC | FAIL | 87 | 13 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-16 03:05 UTC | FAIL | 86 | 14 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-15 03:02 UTC | FAIL | 87 | 13 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-14 02:54 UTC | FAIL | 85 | 15 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-13 03:04 UTC | FAIL | 87 | 13 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-12 03:02 UTC | FAIL | 85 | 15 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-11 03:00 UTC | FAIL | 84 | 16 | 2 | 4 |
| arab, europe, south_america, turkey, us | 2026-10-10 03:02 UTC | FAIL | 85 | 15 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-09 03:01 UTC | FAIL | 84 | 16 | 2 | 4 |
| arab, europe, south_america, turkey, us | 2026-10-08 02:54 UTC | FAIL | 86 | 14 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-07 03:01 UTC | FAIL | 84 | 16 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-06 02:56 UTC | FAIL | 84 | 16 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-05 02:58 UTC | FAIL | 85 | 15 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-04 03:38 UTC | FAIL | 86 | 14 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-04 03:01 UTC | FAIL | 85 | 15 | 3 | 4 |
| arab, europe, south_america, turkey, us | 2026-10-03 02:58 UTC | FAIL | 85 | 15 | 2 | 4 |
| arab, europe, south_america, turkey, us | 2026-10-02 02:53 UTC | FAIL | 85 | 15 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-10-01 02:54 UTC | FAIL | 87 | 13 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-09-30 02:48 UTC | FAIL | 86 | 14 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-09-29 03:00 UTC | FAIL | 86 | 14 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-09-28 02:49 UTC | FAIL | 86 | 14 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-09-27 03:36 UTC | FAIL | 86 | 14 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-09-27 02:52 UTC | FAIL | 86 | 14 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-09-26 02:42 UTC | FAIL | 86 | 14 | 2 | 5 |
| arab, europe, south_america, turkey, us | 2026-09-25 03:00 UTC | FAIL | 86 | 14 | 2 | 5 |
Latest run — 2026-10-21 02:59 UTC
| Plugin | Confirmed | FP | FN | Inconclusive |
|---|---|---|---|---|
accessibility.eaa_mapping | 0 | 0 | 0 | 138 |
compliance.accessibility_statement | 58 | 0 | 0 | 94 |
compliance.age_verification | 0 | 0 | 0 | 85 |
compliance.ai_disclosure | 0 | 0 | 0 | 85 |
compliance.child_consent | 0 | 0 | 0 | 85 |
compliance.cookie_consent | 0 | 0 | 0 | 207 |
compliance.cross_border_transfer | 0 | 0 | 0 | 152 |
compliance.dark_pattern | 67 | 0 | 0 | 152 |
compliance.data_subject_request | 16 | 0 | 0 | 69 |
compliance.dpo_contact | 0 | 0 | 0 | 85 |
compliance.eu_representative | 0 | 0 | 0 | 85 |
compliance.geo_consistency | 0 | 0 | 0 | 170 |
compliance.iab_tcf | 0 | 0 | 0 | 85 |
compliance.iab_tcf_verified | 5 | 0 | 0 | 80 |
compliance.legal_disclosure | 72 | 0 | 0 | 147 |
compliance.odr_link | 67 | 0 | 0 | 18 |
compliance.pay_or_consent_wall | 0 | 0 | 0 | 85 |
compliance.pricing_indication | 0 | 0 | 0 | 170 |
compliance.privacy_policy_content | 14 | 0 | 0 | 151 |
compliance.purchase_disclosure | 0 | 0 | 0 | 152 |
compliance.required_pages | 158 | 0 | 0 | 128 |
quality.api_test | 0 | 0 | 0 | 216 |
security.exposed_files | 156 | 0 | 0 | 12 |
security.headers | 580 | 0 | 0 | 324 |
security.tls_deep | 239 | 0 | 0 | 93 |
seo.broken_links | 19 | 0 | 0 | 22 |
seo.canonical_audit | 46 | 0 | 0 | 3 |
seo.duplicate_content | 0 | 0 | 0 | 85 |
seo.freshness | 36 | 0 | 0 | 25 |
seo.hreflang_validator | 71 | 0 | 0 | 3 |
seo.meta_tags | 90 | 0 | 5 | 0 |
seo.robots_txt_audit | 16 | 0 | 0 | 2 |
seo.sitemap | 74 | 0 | 0 | 8 |
seo.structured_data | 83 | 0 | 0 | 9 |
tech.dns_health | 340 | 0 | 0 | 170 |
tech.stack_detection | 0 | 0 | 0 | 85 |
Wrong findings — latest run (gate)
false_negativeseo.meta_tags/meta.title.missing — https://www.imdb.com/ — oracle has no title; plugin didn't flagfalse_negativeseo.meta_tags/meta.title.missing — https://www.booking.com/ — oracle has no title; plugin didn't flagfalse_negativeseo.meta_tags/meta.title.missing — https://www.dw.com/ — oracle has no title; plugin didn't flagfalse_negativeseo.meta_tags/meta.title.missing — https://www.amazon.ae/ — oracle has no title; plugin didn't flagfalse_negativeseo.meta_tags/meta.title.missing — https://www.petra.gov.jo/ — oracle has no title; plugin didn't flag