Scanning 50 OpenClaw Skills with Cisco's AI Defense Scanner

Skills Scanned

Clean

Flagged

True Positives

<1s

Total Scan Time

TL;DR 48 out of 50 skills passed clean. The 2 that flagged were both false positives — legitimate code patterns that tripped static analysis rules. The scanner is fast and catches real threats, but static-only analysis needs an LLM layer to understand context. That said, for a first pass on untrusted skills? It's excellent.

What is Cisco's Skill Scanner?

cisco-ai-skill-scanner is an open-source security scanner from Cisco's AI Defense team. It's designed to catch malicious patterns in AI agent skills — the kind of plugins and extensions that give agents new capabilities.

Think of it like a virus scanner, but specifically tuned for AI agent code. It looks for prompt injection, data exfiltration, credential theft, and malicious code patterns.

Detection Engines

The scanner has six detection engines. We tested with the default — static analysis only (no API keys needed).

Engine	Description	Requirements
Static	YAML + YARA pattern matching across all files	No dependencies
Behavioral	AST dataflow analysis on Python files	No dependencies
LLM	Semantic analysis of skill docs + scripts	API key required
Meta-Analyzer	False positive filtering across all findings	API key required
VirusTotal	Hash-based malware detection on binaries	API key required
AI Defense	Cisco's cloud-based AI security analysis	API key required

The LLM and Meta-Analyzer engines are where the real magic happens for reducing false positives — they understand intent, not just patterns. More on that later.

Running the Scan

We installed the scanner in a virtual environment and pointed it at OpenClaw's bundled skills directory. All 50 skills, one command.

Terminal — Install + Scan

# Install in a clean venv
python3 -m venv /tmp/skill-scanner-env
source /tmp/skill-scanner-env/bin/activate
pip install cisco-ai-skill-scanner

# Find OpenClaw's bundled skills
SKILLS_DIR="$(npm root -g)/openclaw/skills"

# Scan everything
skill-scanner scan-all "$SKILLS_DIR" --recursive --format table

The scan completed in under one second. 50 skills. No network calls, no API keys needed for the static engine.

Results: 48 Clean, 2 Flagged

The output was decisive. 48 skills came back SAFE with zero findings. Two skills were flagged — one CRITICAL, one HIGH. Let's look at both.

Critical trello

What the scanner found: A curl POST command with API keys in the SKILL.md documentation. The static analyzer flagged this as credential exfiltration — it matched the pattern of sending secrets to an external endpoint.

What triggered it

# From the trello skill's SKILL.md documentation
curl -X POST "https://api.trello.com/1/cards" \
  -d "key=$TRELLO_API_KEY&token=$TRELLO_TOKEN&name=..."

Verdict: False positive. This is a documentation example showing users how to call the Trello API. The skill is explaining how to use it, not exfiltrating credentials. A human reads this and immediately understands the context — but a pattern matcher sees curl POST + API_KEY and flags it.

High nano-banana-pro

What the scanner found: PILImage.open(img_path) in a Python script. The static analyzer flagged this as sensitive file access — opening files from a variable path could be used to read arbitrary files from disk.

What triggered it

# From the nano-banana-pro skill's image generation script
img = PILImage.open(img_path)

Verdict: False positive. This is an image generation skill. It creates an image with AI, saves it to a temp file, then opens it with Pillow (PIL) to process it. The img_path is a file the skill just created, not user-controlled input. Perfectly safe.

What This Tells Us

The scanner did exactly what static analysis should do — it erred on the side of caution. Both flags were patterns that would be genuinely dangerous in a malicious skill. A curl POST with API keys is how you exfiltrate credentials. Opening arbitrary file paths is how you steal data. The scanner caught the patterns correctly; it just couldn't understand the context.

This is the fundamental tradeoff with static analysis:

Fast — all 50 skills in under 1 second
Zero dependencies — no API keys, no network, no cloud
Catches real threats — if a skill actually had curl POST sending your API keys to a random server, this would catch it
Can't understand intent — a documentation example and actual exfiltration code look identical to a regex

That's where the LLM and Meta-Analyzer engines come in. With an LLM analyzing the findings, the scanner can understand that a curl example in a SKILL.md is documentation, not an attack. For production use — like scanning skills submitted to a marketplace — you'd want the full engine stack.

Our Takeaways

For individual users: Static-only scanning is a solid first line of defense. Install a new skill from the community, run skill-scanner scan ./path/to/skill before enabling it. If it comes back clean, you're probably good. If it flags something, take 30 seconds to look at what triggered it.

For skill marketplaces: Static analysis alone isn't enough. The 4% false positive rate (2/50) would generate too much noise at scale. You'd want the LLM analyzer + Meta-Analyzer to filter out legitimate patterns. The --enable-meta flag exists exactly for this.

For CI/CD: The scanner supports SARIF output and --fail-on-findings, so you can gate pull requests on security scans. For an open-source skills repo, this is a no-brainer.

Bottom Line 50 skills, under 1 second, zero true positives. OpenClaw's bundled skills are clean. And now there's a tool to help keep it that way.

Try It Yourself

The scanner is open source (Apache 2.0) and takes about 30 seconds to install.

Terminal

# Install
pip install cisco-ai-skill-scanner

# Scan a single skill
skill-scanner scan /path/to/skill

# Scan all OpenClaw skills
skill-scanner scan-all "$(npm root -g)/openclaw/skills" --recursive

# With LLM + false positive filtering (needs API key)
export SKILL_SCANNER_LLM_API_KEY="your_key"
skill-scanner scan /path/to/skill --use-llm --enable-meta

# CI/CD: fail build on findings, SARIF output
skill-scanner scan-all ./skills --fail-on-findings --format sarif --output results.sarif

Going Further

What patterns does the static analyzer catch?

The static engine uses YAML rules and YARA signatures to detect:

Prompt injection (system prompt overrides, jailbreak patterns)
Credential exfiltration (API keys sent to external endpoints)
Shell injection (unsanitized command execution)
Data exfiltration (reading sensitive files, sending to external URLs)
Obfuscated code (base64-encoded payloads, eval chains)
Missing security metadata (no license, suspicious manifests)

You can also write custom YARA rules with --custom-rules /path/to/rules/.

How does the LLM analyzer reduce false positives?

The LLM analyzer reads the skill's SKILL.md and scripts with semantic understanding. When the static engine flags curl POST with an API key, the LLM can determine whether it's documentation showing users how to use an API (safe) or actual code sending credentials to an attacker's server (malicious).

The Meta-Analyzer goes further — it takes all findings from all engines and filters them as a group, considering the overall context of the skill. This is where you get the best signal-to-noise ratio.

Can I scan skills from ClawhHub or the community?

Yes. Point the scanner at any skill directory. If you download a skill from ClawhHub or GitHub, scan it before adding it to your OpenClaw workspace:

skill-scanner scan ~/.openclaw/skills/new-community-skill

This is especially important for skills that include shell scripts or Python code, since those have the most attack surface.

What about the Cisco AI Defense cloud engine?

The --use-aidefense flag sends skill content to Cisco's cloud-based AI security service for analysis. This is the most comprehensive engine but requires a Cisco AI Defense API key. It's aimed at enterprise deployments where you're scanning hundreds of skills across teams.

For personal use, static + LLM is more than enough.