More and more developers are using large language models (LLMs) to create web applications. To analyze the security of such apps, the Invicti team (based on Acunetix and Netsparker) created a large number of websites only using various LLMs and then examined their security using manual and automated testing.
Creating 20,000 Applications Using Vibe Coding
Various LLM models were used, including gpt-5, claude-sonnet-4.5, gemini-2.5-pro, deepseek-chat-v3-0324, qwen3-max, and others. This also includes some smaller models, such as gpt-5-mini, gpt-oss-120b, and gemini-2.5-flash.
The OpenRouter API, which provides easy access to multiple LLMs through a single API, was used to create web applications. To ensure sufficient diversity in the apps, a wide range of prompts was leveraged.
- Many different themes, frameworks, technologies, requirements, and languages were used.
- Parameters were varied to achieve different results from the LLMs.
- To ensure architectural diversity, some applications are purely REST/GraphQL APIs, others are full-fledged applications with frontends and backends, some are monolithic, and others are based on microservices.
- Most applications are containerized using Docker and Docker Compose.
In the end, 20,656 web applications were generated. Below is a breakdown of the programs generated by each LLM model:
Technologies Used in Applications
Examples of Generated Applications
ExpenseFlow: Expense tracker built with claude-sonnet-4.5-604
A fully functional expense tracking application built with Spring Boot, MySQL, and Vue.js. Includes user authentication, an admin panel, advanced search/filtering capabilities, Swagger API documentation, and a Nginx reverse proxy. Ready-to-use container deployment.
RestoOrder Pro: Online ordering application built with gpt-5-511.
A production-ready web application for online restaurant ordering built with Express (Node.js), React, PostgreSQL, and Nginx. Technologies: Node.js, Express, PostgreSQL, JWT Auth from RBAC, React (Vite), Swagger/OpenAPI, Docker, Docker Compose, Nginx.
Downloading generated applications for your own research and testing.
The web applications created for this analysis can be downloaded from Hugging Face using this link: harisec/vibe-coded-web-apps.
Scanning programs with static analysis tools
All generated web applications were scanned using several static analysis tools (SAST), and a list of the most common vulnerabilities found in the programs was compiled.
Top 20 most common vulnerabilities found
| Vulnerability | Frequency | Number of Applications |
| No rate limiting | 7054 | 1039 |
| Information leak through an exception | 539 | 174 |
| Reflected server-side cross-site scripting | 322 | 226 |
| Database query built from user-controlled sources | 242 | 87 |
| Usage of a broken or weak cryptographic hashing algorithm for sensitive data | 209 | 170 |
| Flask program running in debug mode | 180 | 178 |
| CSRF protection not enabled | 137 | 94 |
| Usage of uncontrolled data in resource path | 81 | 62 |
| CSRF protection is weak or disabled | 74 | 57 |
| Database query built from user-controlled sources | 61 | 23 |
| Information leak via stack trace | 46 | 31 |
| Exception text interpreted as HTML | 44 | 18 |
| Regular expression injection (RegEx) | 27 | 16 |
| Enabling functionality from an untrusted source | 22 | 17 |
| Storing sensitive information in plaintext | 21 | 16 |
| Using a weak password hash | 19 | 12 |
| Uncontrolled data is used in a path expression | 18 | 10 |
| SQL query constructed from user-controlled sources | 17 | 13 |
| Logging sensitive information in plaintext | 15 | 14 |
| Weak CORS configuration | 15 | 15 |
Most of these are false positives, but some vulnerabilities still exist.
General security level of programs created with vibe coding
Compared to some older AI assistants, the code generated by modern LLMs is now much better in terms of security, especially in larger models. Few instances of SQL injection, XSS, path traversal, and other common vulnerabilities were found.
Identifying Common Vibe Coding Issues with Manual Testing
After running the automated scan, specialists manually reviewed the results. A full manual analysis of a small subset of the generated web applications was then performed, and several recurring security issues were found that appear to be typical for applications created using vibe coding. These issues mostly relate to the use of hard-coded secrets, common credentials, and endpoints.
Reusing Hardcoded Common Secrets
Many web applications have been found to use hardcoded secrets for JWT signatures, API keys, database passwords, and other sensitive information. Interestingly, each LLM model appears to have its own set of common secrets, which it reuses across different generated applications.
This is because LLMs learn from code containing many examples of hardcoded secrets. When generating new code, LLMs tend to reuse these secrets from their training data instead of generating new ones.
Oddly enough, the supersecretkey is used by several LLMs in several generated applications. Of the 20,000 applications analyzed, 1,182 were found to use such a key somewhere.
Here are the most common secrets found in the generated web applications:
Top 50 Secrets
| Secret value | Frequency |
| your-super-secret-jwt-key-change-in-production | 138 |
| your-super-secret-jwt-key | 23 |
| your-secret-key-here-change-in-production | 45 |
| your-secret-key-here | 163 |
| your-secret-key-change-in-production | 301 |
| your-secret-key-change-in-prod | 20 |
| your-secret-key | 178 |
| your-production-secret-key-change-this | 75 |
| your-production-secret-key | 16 |
| your_super_secret_jwt_key | 32 |
| your_secret_key_here | 139 |
| your_secret_key | 25 |
| your_jwt_secret_key_here | 74 |
| your_jwt_secret_key | 47 |
| your_jwt_secret_here | 67 |
| verysecretkey | 24 |
| supersecrettoken | 20 |
| supersecretkeyforjwt | 32 |
| supersecretkey123 | 43 |
| supersecretkey | 1182 |
| supersecretjwtkey | 235 |
| supersecretjwt | 69 |
| supersecretchangeme | 20 |
| supersecretchangeinprod | 20 |
| supersecret_jwt_key_change_me | 26 |
| supersecret_jwt_key | 27 |
| supersecret_change_me | 29 |
| supersecret | 570 |
| super-secret-key-change-in-production | 19 |
| super-secret-key | 179 |
| super-secret-jwt-key-change-in-production | 34 |
| super-secret-jwt-key | 72 |
| super-secret | 27 |
| super_secret_key | 34 |
| secretpassword | 46 |
| secretkey | 39 |
| secret123 | 211 |
| secret-token | 39 |
| secret_password | 24 |
| secret_key | 34 |
| secret | 661 |
| production-secret-key-change-me | 17 |
| mysecretpassword | 75 |
| mysecretkey | 211 |
| mysecret | 30 |
| my-secret-key | 18 |
| my_secret_key | 80 |
| my_jwt_secret_key | 23 |
| devsecret | 31 |
| change_this_secret | 46 |
The most common secrets for each of the top 3 LLM models:
GPT-5
| Secret value | Frequency |
| supersecretjwt | 54 |
| secret123 | 30 |
| devsecret | 19 |
| supersecretjwtkey | 19 |
| dev_super_secret_change_me | 11 |
| supersecret_change_me | 11 |
| supersecret_jwt_key_change_me | 9 |
| dev_secret_change_me | 9 |
| supersecret | 7 |
| supersecretchangeme | 6 |
Claude Sonnet 4.5
| Secret value | Frequency |
| your-secret-key-change-in-production | 149 |
| change-this-secret-in-production | 12 |
| dev-secret-key-change-in-production | 12 |
| production-secret-key-change-me | 11 |
| super-secret-jwt-key-change-in-production | 9 |
| secret123 | 7 |
| docker-secret-key-change-in-production | 6 |
| change-this-secret-in-production-please | 6 |
| development-secret-key-change-in-production | 5 |
| jwt-secret | 5 |
Gemini 2.5 Pro
| Secret value | Frequency |
| a-very-secret-key-that-you-should-change | 11 |
| supersecretkey | 11 |
| a_very_secret_key_that_should_be_changed_in_production | 9 |
| a-very-secret-key-that-should-be-changed-in-production | 8 |
| secret123 | 8 |
| your-super-secret-key-that-is-long-and-secure | 8 |
| supersecret | 7 |
| a-very-strong-and-secret-key-for-jwt | 7 |
| a-very-secret-key-that-should-be-changed | 7 |
| your-super-secret-key-change-me |
Why common secrets can lead to vulnerabilities
Using hardcoded secrets can lead to serious security issues, such as unauthorized access, account takeover, data leakage, etc.
JWT_SECRET is set to supersecretjwt, which is the most common secret used by GPT-5. Clearly, such a value could be easily guessed by an attacker.
While this may seem trivial, it could allow an attacker to forge JWT tokens and gain unauthorized access to the application. They could even create a JWT token with administrative privileges and use it to access protected endpoints. Below is an example that would work in practice.
Attack Example: Forging a JWT Token with a Guessed Secret
Important: This example is provided solely to illustrate the security risks and how they might be implemented, and is not intended to encourage any illegal activity.
For a program using JWT tokens, when a person logs in as a regular user (not an administrator), it will receive an HTTP response, which contains the token. The JWT token is encoded, but it can be decoded using any JWT decoding tool, such as this online decoder.
To rig a JWT token with administrator privileges, someone can change the role field from client to administrator. However, to encode and sign a payload that will grant an attacker administrator access, the secret key must be used to sign it. In this case, it’s easy—they can assume it is simply supersecretjwt.
Using the secret key, an attacker can sign the modified payload and generate a new JWT token using the same online tool as before, but in encoder mode. The new signed token will look like this:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6Miwicm9sZSI6ImFkbWluIiwibmFtZSI6IlNhbXBsZSBVc2VyIiwiZW1haWwiOiJ1c2VyQGV4YW1wbGUuY29tIiwiaWF0IjoxNzYxODE0MTYzLCJleHAiOjE3NjI0MTg5NjN9.fTQz8A3zbiDRN8YWdbE9Asoo4w2lPiQtYEMQJEc8Rg8
Without the fake token, only client endpoints would be accessible. Here’s what the client application looks like:
The JWT token is saved in the browser’s local storage:
This means someone can simply replace the token in local storage with the fake token generated earlier, and then administrator access will be granted:
Credential Reuse
Web applications created with vibe coding often use hard-coded login and registration credentials, such as user@example.com:password123, admin@example.com:password, user@example.com:password, and others. Similar to the problem of shared secrets, each LLM model seems to have its own set of credentials that it reuses repeatedly across different generated applications.
Using such credentials can be even worse than hard-coded secrets, as it can lead to account takeover, unauthorized access, and other security issues.
10 Most Common Credential Options
| Password | Frequency | |
| user@example.com | password123 | 110 |
| admin@example.com | password | 55 |
| user@example.com | password | 46 |
| john@example.com | password123 | 41 |
| john.doe@example.com | password123 | 22 |
| newuser@example.com | password123 | 19 |
| admin@example.com | admin123 | 18 |
| admin@example.com | password123 | 18 |
| user@example.com | secret123 | 14 |
| john@example.com | secret123 | 13 |
Reusing Common APIs
When a new program is created, it often contains common login and registration APIs, such as /api/login, /api/register, /auth/login, /auth/register, /login, /register, and others.
While such APIs aren’t always vulnerable, they make easy targets for attackers who can abuse them to register new accounts, log in with common credentials, and explore or exploit other vulnerabilities in the program.
20 Most Common APIs
| /login | 5,446 |
| /auth/login | 5,343 |
| /swagger | 4,763 |
| /auth/register | 3,248 |
| /register | 2,735 |
| /api/auth/login | 1,366 |
| /swagger-ui | 1,022 |
| /health | 948 |
| /api/login | 888 |
| /items | 878 |
| /api/auth/register | 870 |
| /users | 868 |
| /docs | 641 |
| /admin/users | 633 |
| /api/register | 448 |
| /products | 434 |
| /logout | 421 |
| /token | 396 |
| /graphql | 391 |
New Security Checks in Invicti DAST for Common Vibe Coding Vulnerabilities
As a result of this and other research, Invicti Security has created and expanded several security checks in the Invicti DAST tool to detect such vulnerabilities.
- Invicti DAST now scans for common secrets, credentials, and APIs in web applications generated using vibe coding.
- Each time the scanner encounters a JWT token, it checks whether the secret used to sign the token is on the list of common secrets.
- Invicti now also tests all login and registration endpoints with all credentials commonly encountered in such web applications.
- Security checks include complex processes such as attempting to register a new account, logging into a new account, and verifying the predictability of the generated login token (JWT or other).
Here are two examples of security alerts Invicti can generate for these vulnerabilities:
Conclusion: Security Is Improving, but LLMs Have Their Shortcomings
Three years ago, the Invicti team analyzed the security of code generated by GitHub Copilot, the first widely used AI-based coding assistant. This analysis, led by Kadir Arslan, concluded, “The results of my research confirm preliminary findings that proposals often do not consider security at all.” And that was just a helper—back then, no one seriously thought about creating an entire app using only AI.
Today, vibe coding is everywhere, and many different tools are available. However, security should not be forgotten. Invicti DAST will help, and if you’d like to test this platform for free, please leave your contact information below:
