Macroscopic Design Documents

[Kashnox]

As we approach the full start of the Macroscopic stage, I want to discuss what we can do to make sure it starts smoothly. The Macroscopic and Aware stages will likely be the most complicated stages in the entire game. Considering the amount of time spent on the Microbe and Multicellular stages, we might very well be working on the Macroscopic and Aware stages for the next decade, so it’s important we start them off strong with proper design.

At a minimum, each design document should be reviewed by the lead programmer (or another experienced programmer if the lead made it) and the lead game designer before any code is written, but more eyes on the document are always helpful.

Why?

I’ll discuss what I think we should include in design documents further down, but these are the big benefits to using them. I know writing documents isn’t fun, but I think they will lead to us having a more fun and productive time developing the Macroscopic stage once they’re made.

1. It forces programmers to think about how their code will interact with the big picture, leading to inherently more scalable and maintainable code.
   1. Right now, Hyyrylainen is often the only one who sees the big picture, which adds extra strain on him during code reviews
2. It reduces the risk of reworks and redesigns after features are implemented because all stakeholders have a chance to give input before the feature is developed
3. It makes it easier for new volunteers to jump in, either by implementing a system designed by someone else or by reading through a design document to understand the basics of an already implemented system.
   1. Currently, it’s very hard for new volunteers to figure out where they fit in, which I think is why we have so many people making up new features to implement or tweaking existing systems that don’t necessarily need tweaking, rather than helping move Thrive definitively towards a more complete game.
4. It helps reduce merge conflicts when two programmers are both working on major systems because it’s easier to see how those systems will interact.

Systems vs Features

When thinking about how to create designs for Thrive, I separated things into “systems” and “features”. Systems are the major foundations that power the stage while features are the specific things that use the systems. For example, Mouths would be a feature that would utilize these systems

• Metaball Manipulation to place the mouth
• Procedural Animation to animate biting and chewing
• Nutrient Storage and Consumption, because the mouth consumes nutrients
• AI, as the AI organisms need to know how to use their mouths
• Auto-Evo, so AI species can evolve mouths

My hope is that, if we can identify all (or at least most) of the systems needed before we dive into development, we can design and implement those first. Then, if we do it right, adding features will be as simple as plugging them into the interfaces and helper methods of the systems they utilize.

So, this would lead to a seemingly slow start to the Macroscopic stage as most of our effort would be on designs and implementing systems that the players can’t interact with much, but once the initial setup is finished, we should be able to start pumping features out quickly and finish the Macroscopic stage much faster than we otherwise would have.

Interface Design Document

To guide the individual design documents and the order we design and implement systems, I think a simple Interface Design Document (IDD) would be useful. The point of this document is to show a high-level overview of how each system interacts with each other system. We don’t need to get too detailed with it. I think a diagram showing all the interactions and a one sentence description of each interaction would be plenty.

I made a very rough draft of what that diagram could look like as a starting point. Auto-evo has no arrows because basically everything affects it and I didn’t want to clutter the diagram. Reproduction has no arrows because I don’t know what would affect that yet. Some systems have question marks because I haven’t seen them mentioned anywhere yet, but they seem pretty important to the stage, so I assume they’ll exist in some form.

We’ll need a lot more discussion about what systems should exist before anything can be finalized, and the final document would have descriptions of all those arrows too, but hopefully that gives an idea of what the IDD means to accomplish. It helps plan out what types of methods and properties need to be exposed when implementing a system to make implementing related systems cleaner.

Design Document Template

This is what I would like to see in each design document.

1. General summary of what this system does
2. What systems relate to this?
   1. What methods does this system need to expose for those systems?
   2. What methods do those systems need to expose for this system?
3. What performance concerns are there with this system? How can those be mitigated?
4. UI mockups
5. Limitations and assumptions

Example Design Document

This isn’t at all a fleshed-out design, so don’t focus on the content. It’s just something I threw together to show the idea of what these documents should contain.

Terrain Generation

Summary

The Terrain Generation system is responsible for creating the physical environment the game takes place in. It generates rocks, hills, caves, lakes, etc., and it decides how to place fully sessile organisms determined by the LOD system to be “terrain”, like grass and trees.

Related Systems

1. The LOD system relates to this system because it determines what organisms should be treated as terrain. The LOD system will also be utilized to improve performance by lowering the detail of terrain that’s far from the player.
   1. The LOD system needs a public method called “GetTerrainOrganisms” that accepts a patch as a parameter and returns all the organisms that should be handled by the Terrain Generation system
   2. The Terrain Generation system needs a public method called “GetDistantObjects” that returns a collection of terrain features far enough to be simplified by the LOD system
2. The Patch system relates to this system because it determines what environmental conditions exist, which will impact terrain generation. For example, a dry patch will have fewer bodies of water.
   1. The Patch system needs a public method called “GetEnvironmentalConditions” that returns an EnvironmentalConditions struct representing a specific patch. The struct will be created as part of the Terrain Generation system
3. The Nutrient Storage and Consumption system relates to this system because this system determines what rocks can be consumed.
   1. The Terrain Generation system needs a public method called “GetEdibleRocks” that returns a collection of all rocks that can be consumed

Performance Concerns

We want the terrain to feel rich and interactive, but the more interesting the terrain is, the worse it will be for performance. The main way to mitigate this is by utilizing the LOD system to make far away objects simpler.

UI Mockups

This should have no impact on UI, other than some new options for world generation when starting a new game.

Limitations and Assumptions

1. Each patch will only be 1 square kilometer, and players will be unable to move between patches without entering the editor, so we only need to generate 1 square kilometer at a time

[CheviLevi]

I think it would be a good idea to include more design documents, but I have some questions first:

1. what’s the difference and relationship between game design document and interface design document?
2. who are supposed to write the IDD? Programmers, or the designers?

[Kashnox]

@CheviLevi

1. what’s the difference and relationship between game design document and interface design document?

I’m from the simulation industry, not the game industry, so I don’t usually work with game design documents, but IDDs are meant to show how things relate in code, not necessarily in game. For example, whether an organism can smell its food might have a huge gameplay impact on how much nutrient storage it needs, but I didn’t draw an arrow between Senses and Nutrient Storage because there shouldn’t be any reason they need to reference each other in code.

So, I would say a GDD is meant to make sure all the mechanics make sense together for a player and the game is generally fun, while an IDD is meant to make sure the implementation of each mechanic works together.

The IDD also wouldn’t include any in-depth descriptions of the systems. Those will be saved for the individual design documents. The IDD is only there to show how things connect.

@CheviLevi

2. who are supposed to write the IDD? Programmers, or the designers?

It should be written by the programmers.

[hhyyrylainen]

@Kashnox

It makes it easier for new volunteers to jump in, either by implementing a system designed by someone else or by reading through a design document to understand the basics of an already implemented system.

3. Currently, it’s very hard for new volunteers to figure out where they fit in, which I think is why we have so many people making up new features to implement or tweaking existing systems that don’t necessarily need tweaking, rather than helping move Thrive definitively towards a more complete game.

I’d argue that extensive documentation makes it harder for new people to jump in. If you need to read like 5 long documents before being able to work on anything, that’s going to discourage a lot of people.

Even back when I joined the team, our “getting started” wiki page said to not to try to read everything.

@Kashnox

My hope is that, if we can identify all (or at least most) of the systems needed before we dive into development, we can design and implement those first. Then, if we do it right, adding features will be as simple as plugging them into the interfaces and helper methods of the systems they utilize.

I guess, whether I think it is a good idea to design these systems now, depends on the level of detail.

@Kashnox

We’ll need a lot more discussion about what systems should exist before anything can be finalized, and the final document would have descriptions of all those arrows too, but hopefully that gives an idea of what the IDD means to accomplish. It helps plan out what types of methods and properties need to be exposed when implementing a system to make implementing related systems cleaner.

@Kashnox

So, I would say a GDD is meant to make sure all the mechanics make sense together for a player and the game is generally fun, while an IDD is meant to make sure the implementation of each mechanic works together.

I think that we actually need a GDD first before being able to design the systems. Like what systems interact with what would quite heavily come down to the game design. And we have kind of overtaken the GDDs. There are some design threads on the macroscopic stage but not an overall GDD.

Also in a game almost nothing survives the initial contact with player feedback and designers. So any systems design document would basically get thrown out the second someone implements it and shows the implementation.

It might be useful to have some kind of GDD / supplement that would use technical terminology to explain the major parts.

Though, actually the start of macroscopic development would be converting over the prototypes to an ECS approach, and basically redoing / making compatible many microbe systems. So we already kind of do have a blueprint for what we need for macroscopic. These are all systems that exists in the microbe stage and need an equivalent in multicellular:

• Creature movement and commands system
• AI system that hooks into commands
• Player input that hooks into commands
• Spawn system for spawning and despawning creatures that are too far away
• Compound gaining system (game design is probably not done for this)
• Terrain system and inert terrain parts system
• Plant and other stationary creature spawning system
• Process system for creatures (I hope the microbe one is flexible enough to actually work as-is)
• Creature growth system
• Damage / dying system
• Healing system
• Day/night cycle feature
• Weather cycle feature (edit: if you count the glaciation, but otherwise this is basically a new system for macroscopic, but I listed it here as the ideas were flowing)

All of the systems should be designed as ECS systems that communicate basically entirely through components. The microbe stage is not as pure ECS as I would like due to the way the code was ported over, but I want to make macroscopic a cleaner ECS architecture.

---

When we get to macroscopic development, I expect that I would take a few months to set up all the ECS stuff and do kind of placeholder systems and components for most things. During this time I would greatly appreciate if nobody else tried to touch the things, or at least asks me beforehand as there would be major code (architecture) changes each week.

After that phase is done working on macroscopic would be a lot more relaxed with all the major systems and components defined.

@Kashnox

1. Each patch will only be 1 square kilometer, and players will be unable to move between patches without entering the editor, so we only need to generate 1 square kilometer at a time

I wish those were the real requirements for the system.

What is actually the requirement here is that a player can start off looking at the galaxy map and then just scroll in, in, in and in and end up looking at a some environment and a creature walking there. So there’s absolutely no loading screens which makes the terrain / LOD systems almost the hardest part of the game to implement.

So for that reason I kind of think that LOD / terrain needs to be almost the last system to implement for macroscopic.

[Kashnox]

@hhyyrylainen

I’d argue that extensive documentation makes it harder for new people to jump in. If you need to read like 5 long documents before being able to work on anything, that’s going to discourage a lot of people.

That can definitely be true, depending on what the volunteer is trying to do. If they’re just tweaking a system or fixing a bug, they shouldn’t feel like they need to read an exhaustive design document first. I think the main challenge with jumping in and working on Thrive (at least the challenge I’ve had) is knowing what’s ready to be worked on and is also useful to work on, and design documents don’t help much with that challenge.

@hhyyrylainen

I guess, whether I think it is a good idea to design these systems now, depends on the level of detail.

I think the level of detail can be pretty low, unless the programmer who’s making the document is really excited about writing their designs down. The main point is just making sure there’s a clear vision for how a system will be implemented and how it will connect with the rest of the game, and making sure any major challenges with the system have been thought about already, before development starts.

@hhyyrylainen

I think that we actually need a GDD first before being able to design the systems.

That would be ideal, yeah. I don’t think these system design documents need to be done immediately, I just think they should be done before implementing a feature, so they can wait until a GDD is made, if that’s something we plan to do.

I’m also not married to these exact documents I’ve described being the design documents we make. I think they’ll be really helpful, but if someone has better ideas for how to structure our design documents, I’d be wide open to hearing them. I really just care that we’re planning out the big picture in some way before we start implementing the stage.

@hhyyrylainen

Also in a game almost nothing survives the initial contact with player feedback and designers. So any systems design document would basically get thrown out the second someone implements it and shows the implementation.

This type of rework is one of the big things I think design documents can help avoid. With how small our team is, having to redo large parts of a system hurts a lot. The goal is that, if the game design team and the other programmers can see the implementation laid out before it’s worked on, everyone can make sure it meets their vision before any code is written.

Of course, some rework is inevitable if players don’t think a system is as fun as we expected, but we shouldn’t expect the initial implementation of most systems to be completely different from the final implementation.

@hhyyrylainen

Though, actually the start of macroscopic development would be converting over the prototypes to an ECS approach, and basically redoing / making compatible many microbe systems. So we already kind of do have a blueprint for what we need for macroscopic. These are all systems that exists in the microbe stage and need an equivalent in multicellular:

That’s definitely a big help, but I think most of those probably still deserve new design documents (which would be heavily based on the existing implementation) to make sure the prototype implementation still makes sense with the current vision for the stage. It would stink if we spent tons of time building features off those systems only to realize some of them are inherently flawed or were based on an outdated vision for the stage, so those systems and the features built off them have to be redone later on.

Basically, I think we should triple-check that the foundation is good before we start building anything on top of it.

@hhyyrylainen

The microbe stage is not as pure ECS as I would like due to the way the code was ported over, but I want to make macroscopic a cleaner ECS architecture.

This sounds great to me. This is as close as we’ll get to a fresh start until we eventually get to the Aware or Society stage, so we should take full advantage of it to do things right.

@hhyyrylainen

When we get to macroscopic development, I expect that I would take a few months to set up all the ECS stuff and do kind of placeholder systems and components for most things. During this time I would greatly appreciate if nobody else tried to touch the things, or at least asks me beforehand as there would be major code (architecture) changes each week.

This sounds like the perfect time for the rest of the team to start on design documents so we can hit the ground running once you finish converting everything to ECS. Or, if the gameplay team is still working on the core of the game design throughout that, we can at least start getting design documents together for the safer systems that are unlikely to change.

@hhyyrylainen

I wish those were the real requirements for the system.

What is actually the requirement here is that a player can start off looking at the galaxy map and then just scroll in, in, in and in and end up looking at a some environment and a creature walking there. So there’s absolutely no loading screens which makes the terrain / LOD systems almost the hardest part of the game to implement.

Yeesh, I knew the assumption I made would be completely wrong because I made it up on the spot just to write something in that section, but I didn’t realize how intense the real requirement was.

[hhyyrylainen]

@Kashnox

Of course, some rework is inevitable if players don’t think a system is as fun as we expected, but we shouldn’t expect the initial implementation of most systems to be completely different from the final implementation.

I don’t think even the world’s best game designers can (always) make a design right the first time around without first implementing it.

So even if we had world’s best designers we couldn’t avoid reworks. Though, yeah, most designs wouldn’t be totally different from the end result.

@Kashnox

That’s definitely a big help, but I think most of those probably still deserve new design documents (which would be heavily based on the existing implementation) to make sure the prototype implementation still makes sense with the current vision for the stage. It would stink if we spent tons of time building features off those systems only to realize some of them are inherently flawed or were based on an outdated vision for the stage, so those systems and the features built off them have to be redone later on.

I honestly think that it is just faster for me to program most of these systems myself rather than writing any design documents first, because I expect the skeleton (pun intended) of the stage to take only a few weeks from me to do.

@Kashnox

This sounds like the perfect time for the rest of the team to start on design documents so we can hit the ground running once you finish converting everything to ECS.

If someone else writes it at the same time, it would end up different than the actual implementation… Someone could document what code I end up writing, but doing the both at the same time seems a bit silly to me as this condition probably lasts less than a month and that’s basically nothing in the overall Thrive development timeline.

---

Overall after writing that list of systems, which I think probably counts for like 80% of stuff already needed for the stage, I’m not sure we need that much documentation? Though I did not write any of the graphics systems just the gameplay ones. But a lot of the graphics will just interact with other graphics systems and only need a little bit of customization from the gameplay side. So I still think that a proper GDD outlining the gameplay and visual effects would go a lot farther than technical design of systems that might end up unsuitable for implementation.

[CheviLevi]

@hhyyrylainen

I think that we actually need a GDD first before being able to design the systems. Like what systems interact with what would quite heavily come down to the game design. And we have kind of overtaken the GDDs. There are some design threads on the macroscopic stage but not an overall GDD.

I totally agree with that. When I worked in the game industry, GDD is a must when designing new features. When I joined the team, I actually had thought about suggesting writing GDD mandatorily, but I didn’t do it because I’m not sure if it suits our workflow and project management styles. But I think it would be very useful and to avoid situation that let programmers to decide feature details.

@Kashnox

IDDs are meant to show how things relate in code, not necessarily in game. For example, whether an organism can smell its food might have a huge gameplay impact on how much nutrient storage it needs, but I didn’t draw an arrow between Senses and Nutrient Storage because there shouldn’t be any reason they need to reference each other in code.

Well, IDD sounds like a detailed UML graph to me. In my previous company, I seldom see programmers write that (maybe because we are too “agile” and no one really had the time to).
IMHO, a good GDD would cover what systems should be write/changes, which is part of the meaning of IDD(the other part is how the systems are gonna interact with each other I guess). I would have a nutrual attitude for implementing IDD to our team.

@Kashnox

I think the main challenge with jumping in and working on Thrive (at least the challenge I’ve had) is knowing what’s ready to be worked on and is also useful to work on, and design documents don’t help much with that challenge.

This sounds more like a project management problem to me. Now github issues works as a project management tool(jira, linear, slack, e.g.) to us. But since we are trying to keep issues as less as possible, we are actually not trying to track every develop action. If we did, we might have a better who picture of the current status of Thrive development.

Also, when someone starts to work on thrive, I think it’s always better to dispatch small and deterministic task such as bug fix to them.

[Kashnox]

If the consensus is to create a good GDD and skip making designs documents for individual systems, then that’s fine with me, as long as we actually do make the GDD (and we probably shouldn’t drop that whole task on the game design team alone). I still think designing individual systems as well would be worth it to help avoid having to rework those systems later on, but maybe a GDD will be good enough for that.

@CheviLevi

This sounds more like a project management problem to me. Now github issues works as a project management tool(jira, linear, slack, e.g.) to us. But since we are trying to keep issues as less as possible, we are actually not trying to track every develop action. If we did, we might have a better who picture of the current status of Thrive development.

I agree with this and it’s something I’d very much like to help with eventually, since it’s a big part of what I do in my work, but I want to get a lot more experience on this team before I start asking people to let me help with project management.

@CheviLevi

Also, when someone starts to work on thrive, I think it’s always better to dispatch small and deterministic task such as bug fix to them.

Definitely. Sometimes at work, we purposefully keep very minor bugs around so new hires have good introductory tasks.

[hhyyrylainen]

Apparently we do have a page for the macroscopic GDD, but it is mostly just blank:

Macroscopic Stage GDD

@Kashnox

Definitely. Sometimes at work, we purposefully keep very minor bugs around so new hires have good introductory tasks.

I was about to post this link, but apparently we are down to just 6 good first issues: Issues · Revolutionary-Games/Thrive · GitHub

@Kashnox

I agree with this and it’s something I’d very much like to help with eventually, since it’s a big part of what I do in my work, but I want to get a lot more experience on this team before I start asking people to let me help with project management.

I do see the need of opening a massive wave of task issues once we get to macroscopic. I’ve wanted to avoid that before as dumping a massive wave of issues would bury all the issues for the stages we are actively working on, but that won’t be a problem once the multicellular stage is done.

Actually, I’ll open one right now: Add a keybinding for the macroscopic editor rotation mode (currently always middle mouse) · Issue #7419 · Revolutionary-Games/Thrive · GitHub

[CheviLevi]

@Kashnox

I agree with this and it’s something I’d very much like to help with eventually, since it’s a big part of what I do in my work, but I want to get a lot more experience on this team before I start asking people to let me help with project management.

I think on thing we could do right now is to discuss and explore the future project manage workflow for Thrive. For example, what is still lacking by using our current workflow and how can we do better, etc. We can also bring our previous experience into discussion and see what is suitable and what is not.

@hhyyrylainen

I do see the need of opening a massive wave of task issues once we get to macroscopic. I’ve wanted to avoid that before as dumping a massive wave of issues would bury all the issues for the stages we are actively working on, but that won’t be a problem once the multicellular stage is done.

Yeah I think we should look through and update the status of current issues, and I also think as a part of project manage improvement, we can see if we can tweak any of the issue tag types.

[hhyyrylainen]

@CheviLevi

Yeah I think we should look through and update the status of current issues, and I also think as a part of project manage improvement, we can see if we can tweak any of the issue tag types.

I try to do that occasionally but last time I only got through like the first 5 pages of open issues, and was only able to update / close a few.

So for anyone needing more stuff to work on there are about 500 things to pick from, most still valid.