Add Apple crash symbolication skill

[kotlarmilos]

Description

Adds automation, test coverage, and review fixes for the iOS crash symbolication skill.

Automation Script (Symbolicate-Crash.ps1)

New 664-line PowerShell script that automates the full .ips crash log symbolication workflow:

• Parses two-part .ips JSON format (iOS 15+)
• Identifies .NET runtime libraries (libcoreclr, libmonosgen-2.0, libSystem.*) in usedImages
• Searches for dSYM debug symbols: user-provided paths → SDK packs → NuGet cache
• Verifies UUID match via dwarfdump --uuid
• Batch-symbolicates with atos (groups addresses per library for efficiency)
• Identifies .NET runtime version by matching UUIDs against local packs
• Supports -ParseOnly, -CrashingThreadOnly, -SkipVersionLookup, -DsymSearchPaths

Mirrors the Android sibling's Symbolicate-Tombstone.ps1 for structural parity.

SKILL.md Updates

• Added Automation Script and Runtime Version Identification sections
• Fixed atos -o to point inside dSYM bundle (Contents/Resources/DWARF/) — not the bundle itself
• Added dwarfdump and Symbolicate-Crash.ps1 to INVOKES in frontmatter
• Added "MAUI" keyword for trigger matching (matches Android sibling's phrasing)
• Wrapped steps under ## Workflow heading for consistency with Android skill
• Fixed misleading "rebuild" guidance per @rolfbjarne's review — dSYM mismatch means locating the original build artifacts, not rebuilding

Test Suite (7 scenarios)

Scenario	Tests
Mono crash symbolication	UUID extraction, atos commands, ASI/NullRefException
No .NET frames (pure Swift)	Correctly stops, no false symbolication
CoreCLR crash	Identifies CoreCLR (not Mono), EXC_BAD_ACCESS
NativeAOT crash	Recognizes static linking, libSystem.* BCL libs
Multiple .NET libraries	Distinct UUIDs per library, separate atos calls
ASI field priority	Checks managed exception before native symbolication
Reject Android tombstone	Wrong format detection, suggests Android skill

Validation

• Multi-model review (Sonnet 4, GPT-5.1-Codex, Opus 4.5): 4/5 across all 3 models
• skill-validator A/B testing: 5/7 scenarios show improvement (NativeAOT +2.0, Android rejection +2.0, ASI priority +1.0, multi-lib +1.0), 2 ties on baseline-strong scenarios
• Overfitting score: ✅ 0.12 (low — eval tests outcomes, not skill vocabulary)
• All pre-submission checklist items pass: trigger coverage 8/8, stop signals explicit, domain examples present, token budget ~2K (under 4K limit)

Co-author: @steveisok

[Copilot]

Pull request overview

Adds a new ios-crash-symbolication skill under the .NET plugin to guide retrieval and symbolication of iOS .ips crash logs, focused on resolving .NET runtime native frames via dSYMs and atos.

Changes:

• Introduces a new skill markdown (SKILL.md) documenting an end-to-end workflow for .ips parsing, runtime image identification, dSYM discovery, and atos invocation.
• Adds validation criteria, stop signals, and common pitfalls specific to iOS crash logs and .NET runtime components.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[ViktorHofer]

@kotlarmilos looks like this needs a bit more work

[steveisok]

@kotlarmilos looks like this needs a bit more work

I think the scoring for this type of skill is tough.

For the MAUI scenario:

• Quality:
  0.0 — baseline 2.3/5, skilled 3/5... but the pairwise judge said tie (position-swap inconsistent → defaulted to 0). So the 0.7 quality improvement gets zeroed out.
• Pairwise:
  0.0 — same reason, inconsistent swap → tie
• The only non-zero terms are time (-0.54, penalizing the skill for 75s→116s) and tool calls (-0.29)

So the skill gets zero credit for quality improvement because the pairwise judge couldn't make up its mind when the response positions were swapped. Then the small overhead penalties push it negative.

This is an eval robustness issue — the pairwise judge is position-swap-sensitive, and when it defaults to "tie," the skill gets no quality credit despite the rubric judge scoring it 0.7 points higher. The outputs are genuinely different in quality (the skilled version actually symbolicates frames),
but the pairwise comparison is noisy enough that swapping A/B changes the winner.

Summary: The skill isn't performing poorly — the scoring is fragile for scenarios where the improvement is "more of the same kind of work but better." The skill's clear wins (symbol server, .NET library identification, actual symbolication) get discounted by a noisy pairwise judge, and then the
overhead from doing that valuable work pushes the score negative.

[ViktorHofer]

If you have suggestions how to improve the scoring, please let us know and submit a PR. We need to trust and rely on the scoring of skill-validator. We will eventually also make as positive required for merging.

What about the "not activated" scenario that also shows a negative verdict?

[steveisok]

What about the "not activated" scenario that also shows a negative verdict?

I fixed that to be really basic. The problem before was that it was "too good". The wording caused it to execute the actual android-symbolication skill instead of skip the apple skill outright lol.

[ViktorHofer]

/evaluate

[github-actions]

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
apple-crash-symbolication	Parse .NET frames and locate dSYMs from an iOS crash log	3.0/5 → 3.7/5 🟢	✅ apple-crash-symbolication; tools: skill	✅ 0.19	✅
apple-crash-symbolication	Investigate root cause of a .NET MAUI iOS crash	2.7/5 → 3.3/5 🟢	✅ apple-crash-symbolication; tools: skill	✅ 0.19	✅
apple-crash-symbolication	Reject Android tombstone passed as iOS crash log	5.0/5 → 4.7/5 🔴	ℹ️ not activated (expected)	✅ 0.19	❌

⏰ timeout — run(s) hit the (300s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

📖 See InvestigatingResults.md for how to diagnose failures. Additional debugging guidance may be provided by your workflow.

🔍 Full Results - additional metrics and failure investigation steps

[ViktorHofer]

@steveisok looks like in this run (this whole process is non-deterministic as we know) the scenario that we talked about now has a winning verdict but the non-activate scenario is still failing.

[steveisok]

@steveisok looks like in this run (this whole process is non-deterministic as we know) the scenario that we talked about now has a winning verdict but the non-activate scenario is still failing.

Interesting analysis:

Baseline (no skills, 2 tool calls): Gives a perfect textual analysis — "here's how to symbolicate" — instructions only, no actual work. Score: 5.0/5.

Isolated (apple skill loaded, 8 tool calls): Apple skill correctly doesn't activate. But the model goes further — it actually downloads symbols with dotnet-symbol, runs addr2line, verifies BuildIds with readelf. Real symbolication work! But the captured output is just "Cleaned up temporary symbol
files." — the model cleaned up after itself and the final message is useless. Judge gave 4.7/5 noting the analysis was excellent mid-conversation.

Plugin (all dotnet-diag skills, 10 tool calls): The android-tombstone-symbolication skill correctly activates (skillEventCount: 3). The model runs its PowerShell symbolication script, which times out at 300s. Final output: "Let me run the symbolication script" — incomplete.

The irony: The skilled runs are actually doing better work (real symbolication vs. just instructions), but:

1. The baseline gets 5/5 for telling you what to do
2. The skilled runs get penalized for actually trying to do it (more tokens/tools/time)
3. Isolated mode's final captured output is a cleanup message
4. Plugin mode's real work timed out

Root causes:

• Plugin mode: Cross-skill contamination — the android skill correctly activates for an Android tombstone, but its overhead is charged to the apple skill's eval. This is working as designed (user would have both skills), but the eval can't distinguish "good activation of sibling skill" from "bad
  overhead."
• Isolated mode: Even without activation, the model does 4x more work with the skill loaded. It's going from "give instructions" to "actually symbolicate" — arguably better behavior, but the eval penalizes the overhead.

What I'd recommend: The expectActivation: false scoring path needs special handling — when a skill correctly doesn't activate, efficiency penalties should be heavily dampened. The skill did its job (stayed out of the way).

[steveisok]

@ViktorHofer one additional point. The validator runs on linux, but this is by and large a mac-based skill. For example, atos does not exist on linux and (seemingly) no matter how much you try, the model will try to execute and fail. That is extra time and tokens spent on a dead-end. The relative scores are also higher on the mac as a result.

[ViktorHofer]

If the skill only works on Mac, should it require such an environment in the description or later in the skill content?

[steveisok]

If the skill only works on Mac, should it require such an environment in the description or later in the skill content?

You would think, but even the most aggressive stop signals have a tendency to be 'suggestions'.

[ViktorHofer]

Sorry for the naive question but did you try system prompt terms like CRITICAL: ..., etc in the description? We previously had that in our msbuild skills and that worked well.

[danmoseley]

You would think, but even the most aggressive stop signals have a tendency to be 'suggestions'.

That's fine it's just an optimization it's ok if it's not perfect

[steveisok]

Sorry for the naive question but did you try system prompt terms like CRITICAL: ..., etc in the description? We previously had that in our msbuild skills and that worked well.

Early commits in this PR showed it didn't really matter. Worth another try though. I'll push something up.

[steveisok]

Sorry for the naive question but did you try system prompt terms like CRITICAL: ..., etc in the description? We previously had that in our msbuild skills and that worked well.

Early commits in this PR showed it didn't really matter. Worth another try though. I'll push something up.

It doesn't address the actual failure. The Android rejection scenario fails because:

1. Isolated: The apple skill isn't activated — the model just does more work with bash (8 vs 2 tool calls). No amount of "CRITICAL" in the apple skill description helps when the skill isn't even invoked.
2. Plugin: The android skill activates and times out. Stronger language in the apple skill can't prevent a sibling skill from activating.

We already learned this anti-pattern. The original stop signal said "use ndk-stack/addr2line instead" — teaching the model about Android tools caused it to use them. Adding "CRITICAL: DO NOT execute atos on Linux" would similarly draw attention to atos on a platform where it doesn't exist. The model
discovers atos doesn't exist in ~1 tool call; the "CRITICAL" preamble wouldn't save much.

It pollutes the skill for real users. This skill targets macOS developers. Adding Linux-avoidance language to help CI pass is the tail wagging the dog.

Where "CRITICAL" does work (and why Viktor saw success with msbuild): when the problem is the model misapplying the skill — doing the wrong thing when the skill IS activated. That's a routing/behavior problem. Here, the skill correctly doesn't activate. The problem is environmental (Linux CI) and
scoring (expectActivation: false penalties).

[steveisok]

1. Isolated: The apple skill isn't activated — the model just does more work with bash (8 vs 2 tool calls). No amount of "CRITICAL" in the apple skill description helps when the skill isn't even invoked.
2. Plugin: The android skill activates and times out. Stronger language in the apple skill can't prevent a sibling skill from activating.

I'm translating this to mean that the skill is getting punished for correctly not activating the skill to be tested, but instead activating the android-symbolication skill and getting hung up there. Nothing we can do in the skill itself will prevent sibling skills from activating.

[JanKrivanek]

/evaluate

[github-actions]

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
apple-crash-symbolication	Parse .NET frames and locate dSYMs from an iOS crash log	3.0/5 → 3.0/5	✅ apple-crash-symbolication; tools: skill	✅ 0.19	❌ [1]
apple-crash-symbolication	Investigate root cause of a .NET MAUI iOS crash	3.0/5 → 4.0/5 🟢	✅ apple-crash-symbolication; tools: skill, bash	✅ 0.19	✅ [2]
apple-crash-symbolication	Reject Android tombstone passed as iOS crash log	3.7/5 → 4.3/5 🟢	ℹ️ not activated (expected)	✅ 0.19	❌ [3]

[1] ⚠️ High run-to-run variance (CV=22.67) — consider re-running with --runs 5. (Isolated) Quality unchanged but weighted score is -16.9% due to: judgment, quality
[2] ⚠️ High run-to-run variance (CV=0.64) — consider re-running with --runs 5
[3] ⚠️ High run-to-run variance (CV=2.67) — consider re-running with --runs 5. (Plugin) Quality improved but weighted score is -14.2% due to: completion (✓ → ✗), tokens (27515 → 70333), tool calls (2 → 7), time (33.0s → 44.7s)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions

[JanKrivanek]

The skils themselves seems solid.
The actual scenarios or rubrics might need some tunning - but that can be done as on optional followup (as of now - the eval scores are informational)