AI (LLM) code reviews for Thrive have been tested a bit lately with a few PRs receiving a set of AI review comments. These have been quite good with almost all of the detected problems being real, and this type of review catching problems that human reviewers have not (a big problem with Thrive is the lack of volunteer code reviewers as most PRs only get a quick review from me and if they are lucky then maybe one other team member). So it does seem like it would be very helpful for Thrive development if we got automated LLM reviews done on PRs before a human looked at them as this could catch a lot of simple logic errors and typos.
With this human review effort could be focused on more high level things and the game’s code quality would improve as more PRs would get a higher quality overall review than me just mostly skimming the code (if it is a big PR my brain will melt from trying to review such a thing so I just start skimming the code at some point).
I’ve just tried to enable Github Copilot as a low-effort solution to try out, but it seems like it cannot be enabled repository-wide for everyone. It seems the signup for Copilot licenses for organizations is currently closed to non-enterprise customers. So basically right now we cannot start using Copilot for automated reviews. Or at least that’s how I understand it.
I did just try to look for alternatives but everything seems to have really bad pricing as being per user per month (which is a non-starter for us). It does seem like there’s at least SonarQube which is priced per lines of code in the repo, but that also seems a bit expensive. Luckily it seems that CodeAnt AI and SonarQube both have free offerings for open source projects. I did sent an unquiry for CodeAnt to see what they say. So if we do qualify for open source free usage from either of those companies it seems like a viable alternative to Copilot and who knows they might even do better reviews.
So yeah my hope is to at least experiment with AI code reviews as they seem like they could catch real issues and style guide violations at a good rate without producing many (if at all) false positives. If we do get a lot of noise from false positives or other problems then I am willing to reconsider but for now I think it would help a lot for Thrive to get some better PR reviews as it is basically an eternal issue that I’m the only regular code reviewer so I don’t often have enough energy to really review PRs as well as I wish I could.
I did also look into doing something like coding a custom code review integration, which could be quite cost-effective, however the amount of code needed to do all relevant diff extraction and then validation of found issues, it would quickly turn into a 2-week project for me, which I’m not sure I want to undertake at the current moment. The big plus of this would be that using something like openrouter our code review implementation could just very quickly swap between any needed AI agents so we would be basically safe from any provider ending their service.
Edit: I just found this project which could be an alternative (somehow some lower quality open source solutions came up first in Google so I didn’t see this until now): GitHub - The-PR-Agent/pr-agent: 🚀 PR Agent: The Original Open-Source PR Reviewer. This project is not the Qodo free tier. · GitHub as an open source ready-made review tool, we’d just need to provide the AI credentials to run it.
5 Likes
I’ve been using Sonarcube for IDE(formerly Sonarlint, a local IDE plugin that add extra rules for format checking) for years, and if I understand correctly, Sonarcube doesn’t use LLM to review codes; instead, it uses rules to scan possible problems just like the IDE plugin. Basically we just get the same result as we do in the IDE, execpt we can track them in the front end. So overall speaking, I don’t think it’s a candidate for AI code review.
However, if we can self-deploy it or subscribe it with a very low price, I think it would be nice to add an additional layer to enhance the code quality.
Edit: I’ve noticed that Gitar has been acquired by Sonarcube, it uses LLM to review code and have free plan for plublic repo on github, so probably that’s what you are referring?
I think it would be good to apply for the free plan, but my concern is that if they don’t disclose what the model they are using, they might use outdated model and provide worse results.
Yes, that’s what I was referring to, it talks about AI pull request reviews on their page.
After looking a bit more carefully, I did discover that Codacy, CodeRabbit, and Qodo have mentions on their site as being free for open source projects (or qualifying open source projects). So I think we actually do have quite many choices to try to use for free.
The only alternative to this kind of LLM code review I see is that I spend a week or two developing a new DevCenter feature that would do the following:
- Detect open PR and assign them WaitingCI tag
- Then once CI passes the PR would move to a InitialReview status and the system would automatically randomly pick 2 reviewers from a pool of reviewers (reviewers could sign up on the DevCenter)
- If those people don’t complete a review within 24 hours, then the system would automatically grab the next people from the pool (preferring people who didn’t recently review anything to spread out the effort)
- Once the initial review cycle is completed, then the PR moves to SecondLevelReview status and then I would receive a review request (for now, I think I would be the only second level reviewer)
- Once I approve the PR, then it moves to PlayTesting status and now a separate playtester pool would have it be sent to 1 person who again has 24 hours to make a play review before the pool tries the next person
- Now if the gameplay testing is not good the PR goes either back to InitialReview or SecondLevelReview status depending on how badly broken it is
- If the playtesting passes (after potentially multiple cycles through the review process) then the PR status is changed to ReadyToMerge and can be merged by a code merger (currently only me)
Until the pool is huge, I think I would make it so that my PRs that aren’t draft would go automatically to PlayTesting status to conserve other programmer’s review efforts on PRs that most need it.
I think I could build that kind of automation system in a week or two (including bug fixing when it goes wrong). And I could see how that could be very effective also for this problem but only if we get enough people into the review and playtesting pools.
So I’m not sure if I should put in the time to build such a system as it depends on how many people could be activated by such a system who right now don’t do PR reviews but could with personally picked out PRs every week or two they would receive.
So nobody was interested in that review pool concept that could be an alternative? If we got enough pool members everyone might need to do just 1 or 2 PR reviews per week, even if we get back to the usual PR numbers (this end of summer has been relatively more quiet in terms of new PRs coming in).
I think the current problem is that we lack PR reviewers… and a mechanism like reviewer pool cannot solve this problem..
So my personal opinion is still that AI review will be a better plan.
1 Like
I installed Codecacy on the Thrive repo. It says it should be a free tool for open source projects. So if it seems good we can keep using it. The other option was CodeRabbit which also should be free for us but based on some reviews it seems like it might be lower quality results from it (though it was just one review I read, but that’s why I tried Codecacy first). So it should now be up and running and we can see how it works.
So initial results from Codecacy code review is that it isn’t that good… So that is starting to look like a failed try.
1 Like
I’m going to disable the Codecacy AI review now as it was not good.
Next going to test CodeRabbit which seems to use a better AI model.
1 Like