Limitations and Manual Review

What A11YSmith Can Do

A11YSmith provides automated accessibility auditing across four platforms:

Web

• 26 Target-specific rules layered on top of axe-core
• Live browser automation via Playwright with axe-core integration
• Authenticated flow support via storage state, scripted login, and headers-based auth profiles
• Computed style analysis for color contrast and visual properties
• Structured reports in JSON, Markdown, and HTML formats
• Diff engine for comparing assessments and tracking finding changes over time

Android

• 15 rules for Jetpack Compose and XML accessibility patterns
• Tree-sitter Kotlin parsing with regex fallback
• Artifact analysis: view hierarchies, Espresso output, lint output
• Four audit modes: source only, artifact only, hybrid, manual assist

iOS

• 18 rules for SwiftUI and UIKit accessibility patterns
• Tree-sitter Swift parsing with regex fallback
• Artifact analysis: view hierarchies, XCTest output
• Same four audit modes as Android

Documents

• 28 rules for Office (.docx, .xlsx, .pptx) and PDF accessibility

Current Limitations

Web

• No JavaScript event detection: Only inline event handlers (onclick) are detected. JavaScript-added listeners via addEventListener or framework bindings require runtime inspection.
• No Playwright-based target size measurement: Target size is checked via axe-core results. Live bounding box measurement via Playwright is planned but not yet implemented.
• No screenshot analysis: Screenshots are captured as artifacts but are not analyzed programmatically. Visual checks require manual review.

Native (Android and iOS)

• No full AST parsing: Source analysis uses tree-sitter with regex fallback. Full AST parsing via SwiftSyntax (Swift) and kotlin-compiler-embeddable (Kotlin) remains a future enhancement.
• No runtime inspection: Native auditing analyzes source code and pre-collected artifacts (view hierarchies, test output). Live device interaction is not performed.
• Screenshots are metadata-only: Uploaded screenshots provide visual context but cannot be analyzed programmatically.

General

• No CSV report format: Reports are generated in JSON, Markdown, and HTML only.
• Automation does not replace manual review: Automated checks accelerate expert review but cannot verify all accessibility requirements. See the manual review section below.

Manual Review

Manual review is first-class in A11YSmith. Every assessment includes:

1. Unresolved manual checks: Structured check items that require human validation (e.g., "Verify modal focus trap with NVDA")
2. Confidence caveats: When a finding's confidence is medium or low, the caveats explain what manual confirmation is needed
3. Manual follow-up guidance: Each rule specifies what manual testing should follow automated detection

Recommended Manual Validation

For web findings, validate with:

1. Chrome on Windows with JAWS
2. Firefox on Windows with NVDA
3. Keyboard only
4. 200% browser zoom
5. Safari on iOS with VoiceOver
6. Chrome on Android with TalkBack

Native Validation

For Android findings, validate with:

• TalkBack on a physical device or emulator
• Switch Access
• Keyboard navigation (when available)

For iOS findings, validate with:

• VoiceOver on a physical device or simulator
• Switch Control
• Full Keyboard Access

Honesty Principles

1. A11YSmith is honest about what was inspected, what was inferred, and what still requires human validation.
2. Findings include confidence scores and caveats.
3. Limitations are declared in every assessment.
4. Automation accelerates expert review — it does not replace it.