AI-Generated Code Mistakes You Should Review (and How to Spot Them)

AI-generated code often looks finished but hides problems: wrong business logic, missing security checks, weak error handling, unreliable tests and made-up dependencies. Catching them takes more than checking the code runs. You need to test it against your business rules, permissions and edge cases before it reaches production. 

At Velocity AI, we offer AI-driven development and seasoned engineering management, human code review and testing before anything goes live.

Why Does AI-Generated Code Need Review? 

AI can cut software development time, but it can also produce code that looks complete while missing key requirements. 

An AI can help you create a sign up page for a customer. But it may not ask you if you want to ensure that only authorized users can view customer information. 

It will seem as if the software is meeting its requirements when it is not. 

This is why we need to use AI when reviewing code. Not only to see if the application will run but also to see how it interacts with other aspects of your program. If it meets business requirements and how it will behave when given certain scenarios. 

The 6 Mistakes We Catch Most in AI Code Review 

1. Code That Passes Tests but Breaks Business Rules 

Probably going to be that your code works but your business logic is wrong. 

For example you could have an ecommerce website that uses an AI assistant to figure out refund amounts. You would write a function to help with this. The problem being that it only helps calculate how much money is being refunded. It does not check if the item can be refunded. 

Look at what the functions are doing not just how its coded. 

2. Anyone Can See Everything: Missing Access Controls 

For instance, an administrative API could be able to get the customer data but shouldn't check if the user making the request is permitted to access the data. 

How to spot it: Check who can access each function and what data they can see. 

Pay extra attention to functions that handle: 

• Data about customers and monetary information about transactions. 
• Admin functions and actions 
• External API calls and file uploads 

Enforce security on the server, not just in the user interface. 

3. Packages and APIs That Don’t Actually Exist 

AI coding assistants might offer libraries, functions, or API methods not included or supported in current versions of the libraries you are using in your project. 

How to spot it: check if suggested packages are available in official package registries, and make sure that API methods are in line with the latest documentation from vendors. 

Check each new dependency for compatibility, maintenance activity, licensing and known vulnerabilities. 

4. Fine on a Good Day, Broken on a Bad One: Missing Error Handling 

Take an ecommerce checkout as an example. If the payment provider times out, the app shouldn't make an incorrect payment confirmation or double order. 

How to spot it: Application performance testing if an application denies a request, if the information is not found, or if a process is interrupted. 

Check whether the code can handle failed APIs, timeouts, invalid inputs, repeated requests, and so on. 

5. Green Tests That Prove Nothing 

For instance, a test could be created by an AI helper to determine the correct application of the discount but not to determine whether the discount should be applied to that specific customer. 

How to spot it: See the definitions of each of the tests. 

Make sure to include business rules, failure cases, permission limitations, and unexpected inputs in your test. 

Of particular concern is that if any test has been removed, weakened or modified in any way to make the test suite pass the AI change, one should be very careful. 

6. Code Nobody Can Explain (or Maintain) 

It can create redundant functions and additional logic/dependencies that you don't need for your feature. This leads to code that will be difficult to maintain. Which is why AI Maintenance and Support is important. 

How to spot it: If there is code that is duplicated in it, there are unrelated changes in the code, or there are additional dependencies. 

Ask whether another developer could explain the code without relying on the AI’s explanation. 

Which AI Code Review Tools Help Catch These Errors? 

AI code review tools can be used to review pull requests, detect suspicious patterns, and make recommendations prior to code being merged. 

GitHub Copilot Code Review 

With the code review feature, GitHub Copilot will read the proposed changes and provide feedback in various GitHub workflows. 

However, it should support, not replace, the developer who approves the change. 

CodeRabbit AI Code Review 

Another tool that you can use is CodeRabbit. It provides line by line comments and alerts to potential problems in your PR. 

The best option will depend on your review process. 

There are many aspects to keep in mind. Being able to review your entire repository, understand code outside of your diff, security features, testing capabilities and developer usage when it returns false positives. 

There are free versions and free AI code review tools available to help review your code. These will work for trying out but most likely wont suffice for your needs if your team or company is dealing with secure information. 

Use a tool with some pull requests that have known bugs to test the tool prior to selection. Record which known bugs each tool catches and which it misses. 

AI Code Review Checklist Before Deployment 

• Screen Requirements: Does the code meet objectives and rules of business? 
• Go Over the Changes: Do they appear to be unrelated editing, unnecessary dependencies, or not clear changes? 
• Security: Are user access and permissions correct, and is sensitive data handled safely? 
• Perform the appropriate tests: Do the tests include successful operation, failure, and unusual situations? 
• Check validity of dependencies: Do they exist and are they valid, supported and compatible? 
• Human sign-off: Is there a human who can sign off on this change. 

A more advanced level of security review/testing may be needed for larger changes. 

Why Human Review Still Matters 

At Velocity, AI speeds up coding, and senior engineers review and test every change before it reaches production. That keeps quality, security and maintainability under control. 

In building an application (or analysing a codebase) you want to discover more problems. It's to find out the problems that can occur with the real customers and rectify them prior to deployment. 

AI is able to write code for you however what comes after that is what will make your software better.  

Need someone to look over code that was made with AI? Reach out to Velocity about code review done by humans.