If you ask someone in the industry what the most important skill they learned in college was, I think you’d get different answers from everyone. If you ask me for my particular role, it’s the ability to debug a program. This isn’t just when the program is completely yours either. This is when there’s:
- Legacy code
- Poorly written code
- A lack of tools
- Incorrect comments
- Topics you don’t know anything about
- Code you had no hand in creating
- Threads
- Distributed Systems
- Owners who have left the team
Clearly, there’s a lot that can make debugging a problem an absolute nightmare. Well, guess what, there might be times in your professional career where literally every single one of the above bullet points are true. In the industry, you will probably not be working on a project that just started. Thus, a large portion of the code, if not all of it, will not be yours. Does that mean you go to your team and say “I can’t because I didn’t make this”?
No.
I don’t know anything about physics. I got great grades in pretty much every class except for physics, in which I got a C+ (PHY 200) and a D (PHY 250). Thank God it’s not a core class. Thinking about physics just makes me cringe a little bit. Despite that, I’ve debugged physics code on multiple teams multiple times.
I’m not a graphics programmer. I only have a semi-decent guess as to what a vertex buffer is. My linear algebra skills are an absolute joke. When I found a solution to some linear algebra problem in our engine, one of my teammates would look at it and instantly know a single line solution or know why my solution was completely broken. Despite that, I, again, have debugged and optimized graphics multiple times.
Let’s talk about how.
Mental Approach
Never code tired. If you’re trying to trace code and you just feel this rage build up as you look at other people’s code, you’re not in the mental state to be debugging something. Debugging is a stressful, tiresome and relentless process of making educated guesses and having nearly all of them be wrong. If you’re tired, hungry and stressed out, you’ll never make any progress.
There’s been some bugs that I traced for literally days if not weeks on end. I’m not talking about it being one of many issues I’m tracking. I’m talking about a clear singular focus of killing one major bug. By day two, that can feel pretty devastating.
Get yourself a drink. Go for a walk outside. Put on some music and just completely forget about the code for a little bit.
I personally have found that giving myself a specific amount of time to break for has mixed results. What I tend to do now is simply say “Once I feel ready to attack it again, I’ll go back”. That might take five minutes. That might take three hours, Tarantino film and some ice cream. Do what you have to do to be ready. You’re prepping for a marathon. Act like it.
Reducing The Problem
It’s tempting to assume you know exactly what your code is doing and how it works. You might have tested it. It might have gone through code reviews. It might be a staple of your engine that has been around for months if not years.
Well, we were also pretty sure that speculative execution was working, too.
You need to prove you know exactly what is working and what is not. Often times, I’ll hear someone say something like “I know this works” or “There’s no way it could be returning that value”. This causes a lot of internal screaming in me. Always verify your assumptions.
The goal is to simplify the problem. If you can verify something is not the problem, you no longer need to factor it into your investigations or fix. I like to think of debugging as an equation with variables that we have mixed confidence in our knowledge of what their value is.
PlayerDoesWallClimb + AnotherPlayerShoots + LotsOfColliders + IsLevel3 = HARD_CRASH_IN_PHYSICS
You think, based on various testimony from your teammates, that this set of events and conditions causes a hard crash in physics. Well, that’s a lot of variables to have to track. Some of those are in physics code. Some in the movement controller. Another in the weapon controller. Another points to level specific code.
Rather than tracking every single one of these, reduce the problem until the bug goes away. Let’s say you run the game like this:
PlayerDoesWallClimb + AnotherPlayerShoots + LotsOfColliders + IsEmptyWorld = HARD_CRASH_IN_PHYSICS
Well, now we know it’s not specific to level 3 and we can completely ignore level 3 specific code. What if we remove the gameplay code?
LotsOfColliders + IsEmptyWorld = NO_CRASH
Okay, closer.
PlayerDoesWallClimb + LotsOfColliders + IsEmptyWorld = HARD_CRASH_IN_PHYSICS
Getting there…
PlayerDoesWallClimb + IsEmptyWorld = HARD_CRASH_IN_PHYSICS
Bingo. The problem is way simpler than we thought. Someone is doing something dumb in the movement controller. Possibly a raycast. Maybe some weird math that doesn’t work. Who knows? At the very least, we’ve prevented ourselves from going down rabbit holes with nothing in them.
We can reduce this even further. Try commenting out portions of the movement code. Which line do you have to remove to make the bug go away? You might see that the problem goes away when a raycast is removed. At this point, you’re close enough to the answer that you might find our that it’s operating on a nullptr world, or it’s being handed bad values, or something is fundamentally wrong with your raycasting.
At this point, notice that you haven’t touched anything with physics whatsoever. No gravity changes. No messing with colliders. You don’t need to understand what dynamical mean field theory is (I sure don’t). You’re just identifying the source of the problem. In general terms, most bugs can be reduced down to a few things:
- Something was suppose to be there, but wasn’t.
- Something wasn’t suppose to be there, but was.
- Something was an illegal value.
- Something did something when it wasn’t suppose to.
- Something did the wrong thing.
Giving up because it broke in someone else’s code is lazy, and it’s not going to fly in the professional world. Get as close as you can, and only pass it off when it would take you way too long to fix it compared to someone else.
Read the Error
I realize that a large portion of you can probably read, but there’s also a large portion of you who don’t read errors. They skim them. The red hurts their eyes so they put tape over it. They see a few words that look scary or unfamiliar and assume that they know where to look. They jump to the call stack instead of the error. They click out immediately and run it again. They see a system mentioned that they didn’t make and punt it to the teammate that made it.
Do not do this.
There’s nothing more humiliating than going to your manager or teammate, asking them about a bug, having them look at the error and it turning out to be something incredibly stupid.
Know what a more experienced programmer is going to do instead of immediately punting? They’re going to copy paste that error and throw it into Google. They’re going to Ctrl + F and see where in the code base the exception is thrown from. They’re going to read the documentation for the system that failed. They’re going to check the master branch to see if this problem exists there.
The error is the scene of the crime. Don’t start cleaning the apartment without looking into the clues right in front of your face!
Learn
Sometimes, a bug is just above your head. It involves concept you haven’t touched before. It touches systems you didn’t help create. It uses construct in the language you’ve never seen.
There might be no way around it. No one else on the team understands it either (physics), but someone has to fix it. This is where it’s completely acceptable and encouraged to take some time to learn about the code you’re looking at. Google the words you don’t know. Watch some YouTube videos about it. Study the documentation.
I’ve even taken multiple days to just write toy programs. One bug I had dealt with dependency injection, so I ended up writing a simple toy program to make sure I knew how to use it. There’s a big difference between reading some code that does it properly and entering in the code yourself! What was even better was that the knowledge I gained from that toy program assisted me in debugging many other problems as well. Those subsequent problems were much easier to understand and solve because I took that time to learn.
Rubber Ducking
In short, rather than poking the person next to you and solving the bug in front of them, just read your code out loud and line by line to ensure you know exactly what you’re doing.
Ever write an essay and someone reads it back and you wonder what terrible decisions led to the mess of a sentence you just heard? Obvious words are missing. Another word has no vowels in it. You even proofread it before you submitted it! There’s just no way this could have happened.
Well, you were skimming, and you were probably skimming in your head. When you speak out loud, your brain has to do a lot more work in processing what you wrote. I even tend to point to each token as I go along so I know I’m not skipping something. If I’m really unsure about a piece of code, I’ll even say where the variables/functions come from:
aFrog…which is an argument to this function and should be allocated because we check for it on the first line, has a function named Jump() which we hand the value five to.
You’ll notice the previous quote uses the word “should”. Remember to verify every assumption or “should” statement as you’re rubber ducking!
If you skim, you’re debugging different code than what is in front of you. So many bugs have come down to two variables that look the same but are from completely different sources, but I missed it because I skipped over a whole section. Take your time. Be thorough and you’ll find bugs much faster.
Asking for Help
If you get the chance to do an internship, one of the most common questions interns have is “how many questions do you ask”? I don’t think this really is the right question. The question should be “Under what circumstances should you ask a question?”. In general, you should try to follow the tips above first before asking for help. Remember, part of your job is solving problems, not getting other people to do it for you. It’ll make you a much stronger programmer in the long run, too. That said, there’s a few situations where asking a question is a great idea:
You Don’t Know What You Don’t Know
It’s a great idea to ask for references and possible places to look for answers. There’s little techniques that other people know that they use for debugging. It might be a web page, a piece of documentation they didn’t give you, or another person who has special knowledge of the topic. They might even have a hunch on what the main issue is. If you’re just starting out and feel light on tools, it might be good to ask what tools are available.
Bug Identified, Solution Unclear
This is when you’ve taken the time to perform the steps above and you know what’s wrong, but have no idea how to fix it. That person can then guide you to the right person to talk to or simply pop in the few lines of code that fixes because they know the answer.
Bug Exists on Master
If you are programming and hit an error, try running the code on master and see if the problem persists. If it does, this might be a known issue. There might be a common workaround for it. It might even just be a simple command line flag you didn’t know about.
If this happens, this is a great opportunity to submit an issue, fix up an error message or update some documentation so the next newbie doesn’t hit this same problem!
Lack of Tools To Debug
Let’s say you’re trying to debug something involving a database, but you have no idea how to get information out of it in your engine. It’s completely reasonable to ask somebody for a few simple commands that do that. A lot of my questions will come down to “Is there a way to get this information?” It might be a visualization of something, metrics or just logs. You’re not asking them to solve the problem, you’re asking them for tools to solve it. Don’t ask them to build the house, ask them for a hammer.
That said, you should check your normal places for tools/tricks as well! Google has the answers to most questions you have!
This last one gets into another debugging technique:
Find a Way, Or Make a Tool
As I’ve mentioned in earlier posts, sometimes a problem simply cannot be debugged because it’s too complicated in its current form. Making visualizations, creating tools to probe for information, automating something to write a test and many many other tools can help you when you’re really stuck.
We once had a problem with our particle system and couldn’t figure out why. So, I built a tool to help debug it:
When I ran this before, I could tell that we were been extremely inefficient with our particle slots, so I came up with a new way of managing them. I even found a way to share them better between multiple emitters to avoid crashing!
A tool doesn’t always need to be this grandiose enterprise masterpiece that could ship to a billion clients tomorrow. Sometimes you write a quick and dirty python script to get some information. That’s a tool. Write them.
Pull Master
Sometimes, you’ve been working on a bug for a while and in that time, someone else has already fixed the bug by accident. This is why it is important to pull from master every single day, if not multiple times per day. An argument against this is that you may also pull down bugs on master. There’s two big problems with not pulling from master constantly.
If you pull a bug, it will be fixed or reverted faster. You might be the only person affected by it. When there’s fewer pull requests that have gone up in the time since the bug was introduced, it’s much easier to identify and revert. If you go weeks without pulling master, there may be hundreds of pull requests that could have caused the bug. It might also be covered but a bunch of other pull requests that need to now be reverted as well!
If you pull a fix, you’ll know why something was a fix. The list of possible pull requests that could have fixed your bug will be smaller so you’ll both know why your bug is fixed and how to avoid it being a problem in the future. If you just get a random gift from pulling after a few weeks, there’s no telling when someone will accidentally reintroduce that bug!
Remember that by not pulling master, you’ll delaying your bugs and stacking them up. Pull early. Pull often!