AI (LLM) code reviews for Thrive have been tested a bit lately with a few PRs receiving a set of AI review comments. These have been quite good with almost all of the detected problems being real, and this type of review catching problems that human reviewers have not (a big problem with Thrive is the lack of volunteer code reviewers as most PRs only get a quick review from me and if they are lucky then maybe one other team member). So it does seem like it would be very helpful for Thrive development if we got automated LLM reviews done on PRs before a human looked at them as this could catch a lot of simple logic errors and typos.
With this human review effort could be focused on more high level things and the game’s code quality would improve as more PRs would get a higher quality overall review than me just mostly skimming the code (if it is a big PR my brain will melt from trying to review such a thing so I just start skimming the code at some point).
I’ve just tried to enable Github Copilot as a low-effort solution to try out, but it seems like it cannot be enabled repository-wide for everyone. It seems the signup for Copilot licenses for organizations is currently closed to non-enterprise customers. So basically right now we cannot start using Copilot for automated reviews. Or at least that’s how I understand it.
I did just try to look for alternatives but everything seems to have really bad pricing as being per user per month (which is a non-starter for us). It does seem like there’s at least SonarQube which is priced per lines of code in the repo, but that also seems a bit expensive. Luckily it seems that CodeAnt AI and SonarQube both have free offerings for open source projects. I did sent an unquiry for CodeAnt to see what they say. So if we do qualify for open source free usage from either of those companies it seems like a viable alternative to Copilot and who knows they might even do better reviews.
So yeah my hope is to at least experiment with AI code reviews as they seem like they could catch real issues and style guide violations at a good rate without producing many (if at all) false positives. If we do get a lot of noise from false positives or other problems then I am willing to reconsider but for now I think it would help a lot for Thrive to get some better PR reviews as it is basically an eternal issue that I’m the only regular code reviewer so I don’t often have enough energy to really review PRs as well as I wish I could.
I did also look into doing something like coding a custom code review integration, which could be quite cost-effective, however the amount of code needed to do all relevant diff extraction and then validation of found issues, it would quickly turn into a 2-week project for me, which I’m not sure I want to undertake at the current moment. The big plus of this would be that using something like openrouter our code review implementation could just very quickly swap between any needed AI agents so we would be basically safe from any provider ending their service.
Edit: I just found this project which could be an alternative (somehow some lower quality open source solutions came up first in Google so I didn’t see this until now): GitHub - The-PR-Agent/pr-agent: 🚀 PR Agent: The Original Open-Source PR Reviewer. This project is not the Qodo free tier. · GitHub as an open source ready-made review tool, we’d just need to provide the AI credentials to run it.