While working on redoing the game logic using a custom physics engine and just delegating rendering to Godot, I’ve started to re-evaluate using a full Entity Component System (ECS) approach. I just spent a couple of days basically just commenting out code and putting an absolute ton of not implemented exceptions into the code. The most egregious place being the Microbe class, which even when split into multiple files is quite a behemoth. I had originally said that redoing the game with a full ECS approach would be a lot of extra work. Currently we kind of have code in that direction with various quite independent “systems” (the S in ECS). So we already have a bit of a hybrid design where fundamentally many designs are obviously modelled after the ECS model, but our entities aren’t really done like that. So my opinion has changed from what I last said on this.
So as a quick summary, I think it would be good to go full ECS for Thrive:
We already have a kind of a hybrid thing and with at least the Microbe class being a lot of dense code, I think it would actually clarify the game architecture
ECS approach is said to be a lot faster with many entities thanks to cache locality and vastly easier multithreading (I was thinking how I’d do that while refactoring the game logic, and I came to the conclusion that I wouldn’t be able to do this well without basically copying an ECS approach)
Turns out that the logic rework is already difficult, but also boring to me at the same time as this is like the 4th time I’m working on the base microbe logic (thanks to all the engine changes etc.)
So a solution to all of those would be to go all in on an ECS and rebuild the game entities from the ground up using the ECS paradigm and making all of the game systems with smaller parts. Especially making the Microbe class work again after the conversion to the custom physics, seems to be a pretty huge and complicated monolith of a task, which would be easier to do piecemeal with ECS components.
I picked those potential candidates based on a benchmark where those 3 were the fastest ones:
Those Arch and HypEcs are basically the fastest ones (their order depends on the test), with DefaultEcs still being pretty fast, but having an absolute ton more features. Arch is the most basic one but it allows basically direct access to the data for custom threading etc. to be done so it might be the fastest possible one (HypEcs has a threading implementation drawback when there aren’t a lot of distinct entity kinds).
Even though basic Arch might be the way to start as it has the bare minimum features, so converting from it to any other ECS framework should be pretty easy as it only really has the most basic of features. Also it is the fastest in many tests so it would serve well our goal of improving game performance.
If I go ahead with this, I’ll leave the prototypes as is, but I’ll be able to ditch quite a lot of duplicate code that I needed to already add to allow custom simulated and Godot entities. The ECS approach would thus allow me to ditch some unnecessary boilerplate and start reimplementing the microbe logic in a lot cleaner way. I already saw a bunch of bugs / workarounds in the Microbe class that needs to be removed even with a slight logic rewrite so I really think a full rework would be faster to do, partly because I have more motivation to work on that.
Thoughts on my plan on redoing the game as ECS properly?
Here’s some extra links on ECS performance and what’s the philosophy like:
I’m so far not the best-placed one to answer to this but I’ve tried to understand it overall.
ECS sounds like a double-edged solution where it seems to really work faster and easier for programmers, but in another way have quite a tiny amount of documentation.
So I might tell you to go for it only if you really think you can handle it without counting much on external help o.o’
Another “negative” point would be that it will (at first) slow down again the completion of the microbe stage that is being developed for ages.
Again, I’m probably not the one that is really concerned about that change. The only reference I have with programming is the visual blueprint system from Unreal engine x)
From the way you’re describing this, it sounds like this is a win-win situation where the refactor would go faster and have better results this way. Given that performance is such a big pain point to the players, I’d say it’s worth the investment.
As with the previous refactor, I’d want to ask what areas are NOT being impacted, so that other programmers would know what kinds of changes aren’t going to turn into a mess of conflicts.
GUI stuff is mostly safe, there’s only a few tiny changes needed to the C# side of the microbe HUD, but all other GUI stuff should be untouched. Also most graphics are untouched, there’s just a few entity related scenes I need to rework. A bunch of the general and species etc. code will be also untouched.
The most conflict prone stuff will be gameplay Node and microbe stage related code. Especially the Microbe class and to a lesser extend the other entities and the microbe stage gameplay systems will be pretty nightmarish to fix conflicts on. I don’t think that many conflicts will arise as it’s been pretty quiet development-wise the past month. So if that continues even after my vacation (I’m very unlikely to be able to finish this before that), there could be as low as 0 conflicts.
That’s just that one library. And that one library was so small that it only takes 15 minutes to read all of the code in there to figure out what it does. Anyway it seems that Arch has at least pretty good documentation for how to use it, so it should all be fine if I pick that.
It’s of course impossible to say and fully evaluate your mental energy, but I’d guestimate that doing this ECS conversion will give me enough of an increased enthusiasm towards this whole thing that overall the game logic rework will also finish faster. So I’d estimate this ECS conversion to actually have a negative time cost. Programming is funny that way that a single fully motivated workday with good flow can result in more finished work than a week of not very motivated pace.
Well judging by this you seems to love programming as much as I love making 3D x)
I happened to work around 75 hours in the same week on the same project. This rewarded me to be the most downloaded Space Engineers’s mod for 2 months !
So yeah, okay, even when we point negative side you’re like “I don’t care I wanna do it anyway”.
I fully agree with you that one motivated guy can work waaay faster than anyone (even multiple people) that is not!
I sat down and tried to understand what an ECS is and what it entails. Now fully understanding it, I agree that this could be a large advantage. It’s a decision that seems to make a lot of sense for a game like Thrive with tons of different entities.
Question: Do you plan to treat each species of microbe as an individual entity type with its own components, or do you plan to use it in a different way?
I think the usual model is that you can have only one instance of a component per entity (and that seems to be the case for all the libraries I looked at). So for example if I decide to model the organelles as entity components (which I’m not yet fully sure I should as they are quite internal to the microbe), there’d need to be some bundling like all flagella are bundled into one component. So the end result is that each species is not going to have that different entities. I think about the only difference between the microbe entities will be that the AI microbes will have an AI component to store the AI memory, and the player obviously doesn’t need that.
One design I’m not fully sure yet on how it would best make sense to create the visual nodes and physics, which are external to the ECS world. One potential way I’m thinking right now is that the components will have some ephemeral data that is generated by some systems when microbes are missing that stuff. For example the Godot nodes and then they will be attached by a different system that also takes care of QueueFree’ing unused things. This allows very easily saving the world state to JSON and re-loading the component instances afterwards. And the system then after load will setup all the external data automatically without any special save load handling. I don’t suppose anyone else knows a different approach for this?
Turns out that out of the 3 libraries I posted earlier, only one of them is compatible with Godot 3. The other 2 require much newer .NET runtime than mono can provide. I don’t think I’m in any shape or form ready to consider also converting things over to Godot 4 at the same time…
Exploring the work in progress overhaul, I realized that I needed a way for entity and its components to be easily reused, stored and spawned without much code involved, something akin to a prefab or more specifically a Godot scene now that we seem to move away from it. What would be the equivalent in this method?
That sounds a lot like what is needed for saving. For saving my idea right now is to use the access to all entities and their components as basically a list to create a JSON tree from that. Then loading is just the reverse process of recreating the entities and setting their component data back on them. Obviously network syncing will need a more granular approach to cutdown on bandwidth requirements to not send unchanged properties, but the basic approach to accessing all components on an entity and later re-applying the component values should be very similar in nature.
I asked this because my question relates to object replication. In Godot 4, scenes are needed for node replication, the game authority sends a scene id and game clients instantiates the scene, these work as a “skeleton” for which an initial data or spawn state from the authority is then applied but I imagine with ECS things will be slightly different.
I suppose this is an inherent feature in DefaultECS? If so is there some docs I can read? I haven’t delve too deep on understanding the framework yet.
But yeah, save system (including serialization & deserialization) and network syncing are quite similar process. I’d be happy to have the ongoing ECS overhaul largely compatible with multiplayer out of the box.
This is not going to work at all, or well at least that nicely. The source of truth about the gamestate will be the ECS components with Godot Nodes just playing the role of displaying the data to the user. I’ve designed things so that deleting all Godot nodes should allow the game to automatically just resume where it left off with all systems just recreating the needed Godot nodes from the ECS components which hold all the game state. So relying on Godot networking would only allow the visual and sound aspects of the game to work. No actual game logic could exist on the client. I think it is much much better to design the networking from the ground up to work in syncing the entity component values.
I actually did this kind of approach in Leviathan. Each component type there had a function for taking a snapshot of that component’s state and those snapshots had a method to delta compress them based on the latest acknowledged packets from the client. That worked by only sending the fields in the component that the client still had old values for. And I had an interpolation system for component values to smooth out those received game states on the client for display purposes to make objects move smoothly.
They have an inbuilt serialization system but I want to keep our JSON tree structure for saves so I need to do a something bit different.
That’s the documentation on the DefaultEcs serialization. Though, if once we move to Godot 4, we swap to a slightly fasater ECS library (Arch) then the serialization will need to be remade to support the different library.
Indeed, I figured Node-based replication won’t work here. Was trying to get a clearer picture of the ECS way and your description of component-first synchronization definitely clears it. Thanks for the Leviathan reference, I almost forgot that it also uses ECS.
Am I correct to say that the save system will be its own thing and is separate from any other serialization system?