Every security vendor now claims "AI-powered" capabilities. The term has become so overused that it's nearly meaningless. Some wrap GPT in a nice UI and call it innovation. Others use basic machine learning for pattern matching and market it as artificial intelligence.
So what can AI actually do in cybersecurity? What are its real capabilities and limitations? And how do you evaluate AI claims when every vendor makes them?
The Current State of AI in Security
AI and machine learning have been used in cybersecurity for years—long before the current hype cycle. Understanding what's proven versus what's experimental helps set realistic expectations:
Anomaly Detection
ML models trained on normal behavior can identify deviations. Used in SIEM, UEBA, and network monitoring for years. Effective for detecting unknown threats.
Malware Classification
AI excels at classifying malware variants. Trained on millions of samples, ML models can identify malware families and predict behavior of new variants.
Phishing Detection
NLP models analyze email content, URLs, and sender patterns. Combined with visual analysis, AI catches phishing attempts that bypass rule-based filters.
Automated Pentesting
AI agents that can execute security testing methodologies. Requires sophisticated reasoning and tool use. ManticoreAI's ShieldProbe represents this frontier.
Security Copilots
LLMs that assist security analysts with investigation, code review, and threat intelligence. Useful but require human oversight for accuracy.
Autonomous Defense
AI that independently detects, responds to, and remediates threats without human intervention. Still largely aspirational due to reliability concerns.
What AI Can Actually Do in Pentesting
For penetration testing specifically, AI enables capabilities that weren't possible before:
Contextual Understanding
LLMs can understand application context—recognizing that an e-commerce site has checkout flows, or that a healthcare app handles PHI. This enables testing for
Multi-Step Reasoning
AI can chain together attack steps: "This IDOR might combine with this session handling issue to create account takeover." Traditional scanners test endpoints in isolation—AI reasons about relationships.
Tool Operation
Advanced AI agents can operate security tools like humans do—navigating interactive consoles, responding to prompts, handling errors. This enables using professional-grade tools at scale.
Visual Analysis
Multimodal AI can analyze screenshots and UI elements. Identify login forms, admin panels, and sensitive data displays. Extract information from images that might contain credentials.
Report Generation
AI can transform technical findings into coherent reports with executive summaries, remediation guidance, and risk prioritization. What took hours now takes minutes.
What AI Can't Do (Yet)
Honest assessment of AI limitations is essential for setting appropriate expectations:
Guarantee No Hallucinations
LLMs can generate plausible-sounding but incorrect information. In security, this means potential false positives or missed vulnerabilities. Human validation remains essential for critical findings.
Understand Novel Attack Techniques
AI is trained on existing knowledge. Truly novel attack techniques—ones not in training data—may be missed. Cutting-edge offensive security still requires human creativity.
Make Business Risk Decisions
AI can identify vulnerabilities but struggles with business context: "This vulnerability is low severity technically but critical for our specific regulatory requirements." Human judgment needed.
Replace Human Expertise Entirely
The best results come from human + AI collaboration. AI handles scale and coverage; humans provide judgment, creativity, and validation. Neither alone matches the combination.
The Human + AI Collaboration Model
The most effective security operations combine AI capabilities with human expertise. Here's how the division of labor works:
AI Handles
- Scale (thousands of test cases)
- Consistency (never forgets steps)
- Speed (24/7 operation)
- Pattern recognition
- Documentation and reporting
Humans Handle
- Business context understanding
- Novel attack ideation
- Finding validation
- Risk prioritization
- Complex logic exploitation
This model delivers better outcomes than either approach alone: AI coverage at human quality levels.
How to Evaluate AI Security Claims
When a vendor claims "AI-powered" security, ask these questions to separate substance from marketing:
What specifically does the AI do?
Ask for concrete examples. "AI-powered scanning" could mean sophisticated reasoning or basic pattern matching. Vague answers suggest marketing over substance.
How are findings validated?
AI can generate false positives. What's the validation process? Human review? Automated verification? Proof-of-concept exploitation? Unvalidated AI findings create noise, not security.
What are the limitations?
Vendors that acknowledge limitations are more trustworthy than those claiming AI solves everything. Every AI system has constraints—honest vendors will explain them.
How does it handle business logic?
Business logic testing is the hardest challenge for AI. Ask for examples. If the vendor only demonstrates technical vulnerability detection, their "AI" may be limited.
What's the human involvement?
Pure automation sounds impressive but often means lower quality. The best AI security tools have clear human oversight, especially for validation and remediation guidance.
Where AI Security Is Heading
Based on current trajectories, here's what's likely in the near and medium term:
AI Copilots Become Standard
Security analysts will routinely use AI assistants for investigation, code review, and threat intelligence. Quality will improve as models get more security-specific training.
Autonomous Testing Matures
AI agents capable of comprehensive penetration testing with minimal human intervention. Human validation remains for critical findings, but coverage becomes fully automated.
Real-Time Attack Prevention
AI that can identify and block novel attacks in real-time by understanding attack intent, not just patterns. Virtual patching becomes intelligent and adaptive.
Continuous Security Validation
The line between "testing" and "monitoring" blurs. AI continuously validates security posture as code changes, providing always-current security assessments.
ManticoreAI's Approach to AI Security
ManticoreAI's ShieldProbe represents the current frontier of AI-powered penetration testing:
Autonomous Agent Architecture
ShieldProbe is an AI agent that operates security tools like a human expert—not a wrapper around ChatGPT. It navigates interactive consoles, handles errors, and maintains state across sessions.
Multi-Agent Validation
Findings pass through adversarial validation. A separate agent attempts to verify each vulnerability by reproducing the exploit. No theoretical findings—only confirmed exploitables.
Human Expert Oversight
CREST-certified consultants validate all findings, assess business impact, and ensure audit-grade quality. AI scales the testing; humans ensure the quality.
Business Logic Detection
Contextual understanding enables detection of business logic flaws—payment bypasses, privilege escalation, workflow abuse—that pattern-based tools miss.
The Reality of AI in Security
AI is transforming cybersecurity, but not in the way marketing materials suggest. The reality:
- AI augments humans—it doesn't replace them. The best results come from collaboration.
- Validation matters—AI can generate false positives. Human or automated verification is essential.
- Specificity reveals substance—vendors that explain exactly what their AI does are more credible.
- Business logic is hard—this is where AI pentesting is differentiated. Ask about it.
- The technology is real—despite the hype, AI genuinely enables new security capabilities.
Cut through the noise by asking specific questions and demanding demonstrations. The vendors with real AI capabilities will be happy to show you. The ones relying on marketing will deflect.
See AI-Powered Pentesting in Action
ManticoreAI's ShieldProbe demonstrates what AI can actually do in penetration testing. Not marketing claims—real capabilities you can see.