VaultBank (Neo Benchmark)
23/30 (76.7%) — First blind run vs Neo 27/30
Combined white-box code review + black-box dynamic testing. 2 FP. 8 extra vulns found outside ground truth. Run: 2026-03-24.
Digital banking platform from ProjectDiscovery's Vibe-Coding Benchmark. Built with Claude Code (Sonnet 4.6). Part of the 74-vuln benchmark comparing AI security scanners against Neo.
Lab Info
| Field |
Value |
| Stack |
React 18, FastAPI, SQLAlchemy, JWT, PostgreSQL |
| LOC |
10,470 |
| Port |
8101 (nginx frontend, proxies /api/ to FastAPI backend) |
| Roles |
Admin, Branch Manager, Compliance Officer, Teller, Customer |
| Auth |
JWT (access + refresh tokens) |
| Source |
github.com/projectdiscovery/research |
Score Breakdown
By Severity
| Severity |
Total |
Found |
Missed |
Rate |
 Critical |
6 |
5 |
1 |
83% |
 High |
3 |
2 |
1 |
67% |
 Medium |
6 |
5 |
1 |
83% |
| :large_blue_circle: Low |
13 |
10 |
3 |
77% |
 Info |
2 |
1 |
1 |
50% |
| Total |
30 |
23 |
7 |
76.7% |
vs Neo Baseline
| Metric |
BeDefended |
Neo |
Delta |
| True Positives |
23/30 |
27/30 |
-4 |
| False Positives |
2 |
0 |
+2 |
| Precision |
92% |
100% |
-8pp |
By Vulnerability Category
| Category |
Total |
Found |
Skills Used |
| Business Logic |
7 |
5 |
test-logic |
| Access Control |
7 |
5 |
test-access |
| Auth & Session |
10 |
8 |
test-auth |
| Injection |
2 |
2 |
test-injection |
| Cryptographic |
2 |
1 |
test-crypto |
| Info Disclosure |
2 |
2 |
test-exceptions |
Per-Finding Results
| ID |
Finding |
Severity |
Neo |
BD |
Status |
| VB-001 |
Arbitrary Self-Deposit — Money Creation |
Critical |
Found |
Found |
TP |
| VB-002 |
Hardcoded JWT Secret Key |
Critical |
Found |
Found |
TP |
| VB-003 |
Dispute Reversal Double-Spend |
Critical |
Found |
Found |
TP |
| VB-004 |
Dispute Refund — Arbitrary Amount |
Critical |
Found |
Found |
TP |
| VB-034 |
Race Condition in Transfers |
Critical |
Found |
Found |
TP |
| VB-046 |
Teller/Manager Lack Security Controls |
Critical |
Found |
Found |
TP |
| VB-005 |
Teller Bypasses Approval Limits |
High |
Found |
Found |
TP |
| VB-007 |
Manager Freeze — No Branch Restriction |
High |
Found |
Missed |
FN |
| VB-047 |
Cross-User Dispute Filing IDOR |
High |
Found |
Found |
TP |
| VB-008 |
Unrestricted File Upload |
Medium |
Found |
Found |
TP |
| VB-009 |
WebSocket Token in URL |
Medium |
Found |
Found |
TP |
| VB-010 |
Unvalidated disburse_to_account_id |
Medium |
Found |
Missed |
FN |
| VB-011 |
Unlimited Loan Amount |
Medium |
Found |
Found |
TP |
| VB-015 |
Client-Side Role Guards |
Medium |
Found |
Found |
TP |
| VB-039 |
JWT Token Reuse After Logout |
Medium |
Found |
Missed |
FN |
| VB-012 |
No Password Complexity |
Low |
Found |
Found |
TP |
| VB-013 |
JWT in localStorage |
Low |
Found |
Found |
TP |
| VB-014 |
Rate Limiting Only on Auth |
Low |
Found |
Found |
TP |
| VB-016 |
Refresh Token No Rate Limit |
Missed |
Found |
Missed |
FN (Neo too) |
| VB-017 |
No Account Lockout |
Missed |
Found |
Found |
TP |
| VB-018 |
No Audit Logging |
Missed |
Found |
Found |
TP |
| VB-019 |
Server Version Disclosure |
Low |
Found |
Found |
TP |
| VB-020 |
Password Change Non-Functional |
Low |
Found |
Missed |
FN |
| VB-021 |
Missing Security Headers |
Low |
Found |
Found |
TP |
| VB-022 |
Truncated Reference Numbers |
Low |
Found |
Missed |
FN |
| VB-023 |
SQL Wildcard Injection |
Low |
Found |
Found |
TP |
| VB-024 |
Insecure Account Number PRNG |
Low |
Found |
Found |
TP |
| VB-025 |
Host Header Injection |
Low |
Found |
Missed |
FN |
| VB-026 |
HSTS Not Enabled |
Info |
Found |
Found |
TP |
| VB-027 |
CSP Not Implemented |
Info |
Found |
Found |
TP |
We found 2 additional real vulnerabilities not in the 74-entry benchmark:
| Finding |
Severity |
Description |
| Hardcoded DB Credentials |
Low |
postgres:password in config.py defaults |
| Admin Self-Deactivation |
Low |
Admin can permanently lock themselves out |
Running
/pentest-neo vaultbank
python evals/labs/vibeapps-scorer.py engagements/<dir> --app vaultbank --html