Distributed Teams at DataStax Followup
Due to rapid growth in engineering here at DataStax, the OpsCenter and DSE teams needed their own engineering directors. Mike Bulman, who used to run both teams, now gets to concentrate on OpsCenter and I came on board to manage the highly distributed DSE team. Coming from a more traditional co-located setup where pretty much the entire team was in one office, I found there to be a number of notable differences (some positive, some negative, some just different). This is a little bit of a followup on Mike Bulman’s writeup.
We use HipChat for all intra team communication (well, I am old school so I sometimes fall back to old fashioned email). What makes HipChat such a good choice for us:
- Instant communication – Either directly to the person you need, or to a group of people that can help.
- Multitasking-friendly – It is possible to carry multiple conversations at the same time if needed, share code snippets, etc.
- Controllable Interruptions – People can choose to not participate/respond, or do it at a more convenient time. One person chatting does not affect neighboring people either (for those few that are actually sitting together).
- Constant Information Sharing – Engineers can see what others are discussing and doing, almost completely eliminating the need for status meetings (we still have one once a week, mostly for my own comfort, but that may go away).
- Traceability – All conversations are logged, so prior nuggets of wisdom are readily available. Well, search could probably be improved, so yes, that’s on my todo list for a DSE integration.
With Hipchat we are actually able to keep up with more conversations between more people than if we were in the same room!
Coming together as a team is important, so having a technology that facilitates regular meetings is critical. It is interesting how everyone has a strong preference, there is no perfect solution yet. Skype , Google Hangout , Join.me all have come up, and all have their pluses and minuses. Here are some of my wish list items:
- Facilitates our team size.
- Can record the meeting if needed.
- Works well even on low bandwidth connections.
- Runs on all our diverse OS platforms (yes, we have it all).
- Is affordable, yes I can be cheap.
We tend to use WebEx for meetings with more than 10 participants and also when we need to record the information (like a knowledge sharing talk, which we try to have at least once ever two weeks). For meetings that don’t have these requirements we use Google Hangouts, they are just easy to set up in Google Calendar and work very well. For personal one on one conversations I used to use Skype, but am now adopting Google Hangout more and more (one less application to run and update).
Most conversations for planning and design happen in HipChat or Skype. To share designs and other permanent information within the team, but also with other teams in the organization, we have found that a wiki (Atlassians Confluence ) combined with Google Docs is a great way to collaboratively develop and share content. I wish Confluence could index and search Google Docs properly, maybe someday that will work better.
Having a quiet remote office can help get into the flow. Pair programming can be a little challenging in a distributed environment (actually, it’s challenging in a co-located environment), but we found that proactive communication with the rest of the team and a strong review process work fine.
For our development repository, we use github , which enforces/facilitates distributed development very nicely. Not only through local repositories, but also through an easily enforceable code review process, which improves code quality and also the sharing of ideas, solutions and decisions made. For any commit that goes into a system, there is usually a traceable lively discussion between developers, so if there is ever any question on why something was done, it’s right there.
We use cloud infrastructure for running our continuous integration builds as well as for testing. It makes scaling as well as remote access very easy, while allowing sharing of resources as needed. One neat aspect of how our QA team operates is that bug reports document how to reproduce an issue with a few commands that deploy a cloud based set of machines, install the correct version of our setup, and reproduce the problem. That is of course not always possible, but even for the cases where it’s not, the cloud aspect makes it easy to just preserve the failing environment without blocking progress on other areas.
Like many other companies our size, Zendesk is our choice to manage support tickets. All development engineers have access to that system. It is tied into our Jira bug database, so the handoff/interaction between support and engineering is pretty clear. On top of that, our engineers monitor the support room in HipChat, which allows close cooperation between suppor and engineering.
One very nice advantage of having such a distributed team is that when things go hot for one reason or another, we can pretty easily hand off tasks between different time zones to keep things moving. When folks here in the US go to bed, the developers in Europe start to come online (yes, we sometimes have crazy work hours, like any startup does) and vice versa.
This is such a big part of working in a remote environment that I created a separate blog post.
This is a hard one in a distributed team. Not actually interacting on a personal level on a daily basis does change the relationship. So we try to make sure that people feel a part of the organization even when they are remote. For example, we bring the team together on a regular basis. We use that time to socialize, but also to discuss process, design and all other things around the team. The last one was a few month ago, and we are planning the next one now.
We strongly encourage participation of our engineers at events like conferences and meetups. They are encouraged to submit papers, participate in writing books, do whatever helps build their career while helping the Apache Cassandra™ community and DataStax. It is mutually beneficial, and has worked great so far.
There are a number of small things that I believe help with building strong team relations (and if others have more to share, please do). I find it important that my team knows that DataStax will take care of them. So if there is something they need, we make sure they get it. The occasional/when time allows/optional TeamFortress 2 match up certainly can be a lot of fun, not everyone is a hard core gamer, so no shame in being fragged. I like to find ways to remember particular milestone releases, so for the last release we had this comic book page made (nothing fancy, just something to have fun with).
While this is a new level of distributed team development for me, I have to say that it is working very well for us. There is always room for improvement, but having extremely talented engineers makes that a fun ride towards perfection. If you would like to be part of that journey, see if there is a position that matches your skill at DataStax Careers.