Running Team Knowledge Surveys

(Originally drafted in October 2017 during my tenure at Xero. Republished here in September 2026 with updated context and removed proprietary references)

Some domains are just plain complicated. I think it’s safe to say that the most complicated domain I’ve personally worked in during my career so far fits that bill.

The combination of differing business rules, calculation methodologies, and the sheer number of edge cases baked into a mature system can cause your brain to fall out of your ears. This is without even starting to consider the various crazy-ass date maths that has to be performed in some parts of the system (and we all know how much developers love dates).

Starting as a newbie developer on a legacy codebase can be daunting at best. There’s a great deal of mental gymnastics you have to go through to even ensure that the code you’ve written works for all the weird and wonderful possible scenarios and edge cases that users can (and will) bend the system to do.

Peeps

From a people perspective, teams churn. People join, people move onto greener pastures. It’s these people that have gained knowledge and, as such, when they’ve left they’ve taken it with them. Those that remain either have areas of the system they completely understand, areas they partially understand, and areas they have absolutely no idea about.

This also puts incredible pressure on those that remain – any time there is a bug in a particular area, the person that has the most knowledge of that particular area is automatically the one that gets assigned to fix it. Conversely, those who are newer to the team probably feel useless because they have no idea what’s going on, they’ve got no idea where they need to make changes, and probably feel incredibly alienated.

Thus, actually spreading this knowledge amongst team members became paramount if we were to ensure the ongoing success of the product. Coupled with the idea of ‘cross-functional’ teams (where anyone can work on anything), it makes absolutely no sense if a Quality Engineer fully understands an area of a system but conversely a Developer has absolutely no idea.

The Knowledge

However, it’s not enough to just understand the business rules for a particular area - if someone hasn’t got a clue how that area works under the hood, then making changes to that area becomes even more perilous. What if I break something else without knowing? What if there’s a better way? What if I miss a business rule?

On the other hand, a developer can probably look at some code and think “OK, cool, I understand what this does…but why?!” This gives rise to code changes being made that introduce unwanted ‘features’ because the developer didn’t know that particular rule was there for a specific reason.

Therefore, the knowledge of the system falls into one of two camps: business knowledge and technical knowledge.

Topics

So, we’ve figured out the categories of knowledge that we need to capture. But what about the topics? What areas of the system should we have knowledge of?

Well, if your system is grouped reasonably into functional namespaces, you can actually derive the areas of the system that you should have knowledge of. In my case, the codebase was grouped by functional area.

Naturally, this doesn’t contain everything you’d need to know about – but it gives you a starting point.

The Lie of the Land

Next is to put these topics into a format that we can start asking the team about. Any decent survey tool will do the job – we used Google Forms, as it’s straightforward, easy to use, and everyone (you would hope) has a Google account.

Oh, and look – there’s a multiple choice grid 🙂

We set about creating our questionnaire with a ranking of 1 to 5, each of which has a description of what that rating means:

1. What?
2. Seen it
3. Little Idea
4. Understands
5. Fully Understands

On the left-hand side for each row we add a topic, and for each column we add a rating. Split this into two: one for business and one for technical – and we’ve got our survey.

What’s great about this approach is we can now take the survey data (provided people fill it in – don’t even get me started) and start to gather metrics from that data.

Peeps Again

But wait, who do we send this to?

Well, the answer is anybody and everybody who is or has been involved with the product in some way. This includes the Product Owner, Product Manager, Quality Engineers, Developers (front-end, back-end and anywhere in-between), and even DBAs. Essentially, everyone who has touched the product has some idea how one small part of that product works.

What’s also crucial about this is that it gets sent to every team member regardless of their tenure or experience. The idea here is not to single out people for what they don’t know but for what they do know.

OK, now we’ve got our data, what do we do with it? We just leave it to rot on a server somewhere, right?

Keep It Going…

Well, not exactly.

Every month we send out the survey to everyone involved with the product. Once we get the results, we then look at those areas of the product that have the lowest scores. In the majority of cases, this is pretty clear. But don’t just take the numbers at face value: there may be other reasons why the numbers for that area are so low. Could it be a piece of new functionality? Did the person who knows the most about that area move to another team?

Regardless, once we’ve identified those areas that we think we have the least knowledge of, you can probably guess what we do next?

Spreading the Knowledge

Yup, we look at those people who have scored the highest in that area!

Then we need to actually get that knowledge spread around to other members of the team. So, we take an hour once a week and that person talks about that area of the system – be it business, technical, or otherwise.

To further reinforce the learnings by the other people at that session, we then have people do some action off the back of that session. Some of these for us included:

• Taking a story or bug in that area of the system and fixing it, together, as a group (see: Mob Programming). What better way to understand a system than to debug it, let alone fix a bug at the same time!
• Writing documentation around that particular area of the system. Now, this doesn’t have to be Death by Documentation, as in most cases a picture is a thousand words – this could be a flowchart, diagram or just a wiki page of bullet points. This knowledge is then captured forever in time, for all to see.

Making History

The by-product of sending out these surveys is that we also have a historical record of how well our knowledge within the team has improved. One of the team created an awesome chart that shows just this – the mean scores across all members of the team across all areas of the system over the past 6 months.

Note that there are areas of the chart where it dips massively? This might be because someone joined the team and dragged the scores down (remember, they know nothing at this point) or someone moved on and that knowledge went with them.

Wrap-up

Hopefully this has given you some ideas for your own team going forward. It takes discipline as a group to ensure this happens – both through actively filling out the survey and taking the time to present to other members of the team.

But I can guarantee it is worth it – I felt incredibly more confident in my own work in the system, and I’m pretty sure others felt the same way. This put us on a much better path to becoming a truly ‘cross-functional’ team where everyone is responsible for everything and nobody is afraid of touching anything 🙂

When Teams Shrink

I originally wrote about this technique while working in a team that would later be restructured out of existence. Looking back, the irony isn’t lost on me: the very thing that motivated us to run these surveys - knowledge walking out the door - happened all at once when redundancies hit, and no amount of charts could stop it.

If you’re leading a team, run these surveys before the storm. Not to build a case against anyone, but because knowing where your knowledge lives is the difference between losing a person and losing a decade. And if you’re an engineer sitting in a team where everything lives in one person’s head - maybe yours – that fragility cuts both ways. Your expertise is your leverage right up until the day it becomes your replacement plan’s biggest obstacle.

Nobody is afraid of touching anything, the saying goes. Turnover or not, keeping it that way is everyone’s job.