Back to top
Picture this: your data team is a well-oiled machine, tackling complex projects with finesse. But behind the scenes, chaos can ensue without proper organization and strategy.
The need for proper management and leadership for data teams cannot be underscored; reports say that nearly 80% of leaders recognize the importance of having the right people, processes, and culture to transform into a data-driven enterprise (Source: Exploding Topics). But what does this really look like in real life, and how can teams make sure they are set up for success?
Welcome to the world of data team management, where the right approach can make all the difference! Join us as we delve into five essential practices for building and leading successful data teams.
Let’s start with an underrated one – documentation! If you’ve ever led a data team, you know that there are multiple moving pieces at any given time. You have new features being shipped, pipelines breaking and being put back together, production rollouts, user interfaces being revamped and discovery sessions ongoing for yet another web app, analytics suite, or AI product – sometimes all for the same project or client.
Without standardized documentation, you won’t have any single source of truth to track and understand these processes or the different versions of the same data product. New team members being onboarded to a project will lose themselves in the clutter and nouce and have zero context about what’s happening or why, leading to frustration, missed deadlines and inefficient usage of time.
Executing projects successfully is just as important as tracking and documenting them, so make sure you have a framework setup for how and when this documentation will take place.
Sound familiar? That’s because it is increasingly becoming clear to data professionals that a project approach towards data-centric initiatives tends to be limiting and even unrealistic. Seckin Dinc, Head of Data at a startup in Berlin, writes about this concept, pointing out that while projects usually have a fixed scope and a defined beginning and end, the process of building data products is much more iterative without a clearly defined end (Source: Medium). In reality, your work continues until the business user is satisfied.
For example, you may have built a dashboard with some requested KPIs and marked off the “project” as complete. According to the initial scoping, it is complete, but then the business user you built for comes back with a request to add another filter or change the chart type. The success criteria of this data product is not measured according to the completion of project scope but the ability of the user to get real value out of the product. Having a product approach towards building the dashboard by understanding business needs and how to fulfil them will help you deliver successfully and with impact.
Data teams are often under intense pressure to deliver. How do you make sure you’re setting team members up for success?
One way to do this is to help direct their efforts in the right direction. You want teams to be fully focused on execution to bring data products to life. To that end, they should have very clear directions for how tasks will be managed and tracked, what the deliverables are and when to share updates. Standard Operating Procedures should be airtight, and the project manager or scrum master should ensure everyone is fully caught up on these.
Good leadership can make or break – leading a data team also means taking care of people who are responsible for execution. Making people feel valued is intrinsically important and helps optimize team performance.
Are you working in silos? Want to reach out for help but hesitate? These could be signs that the data culture at your organization needs improvement. With the data landscape changing every day, and teams solving more and more complex problems, it’s essential to foster an open data culture (Source: Atlan).
Although we all value collaboration, curiosity, and teamwork, we may miss out on emulating these values in our everyday work. It doesn’t necessarily require putting on a big production with all-hands sessions and fancy workshops. Small actions such as reaching out to a colleague to see if they need help, or letting people know it’s alright to admit if they don’t know something, can go a long way. No one can know everything, but it’s important to be open about what you know and don’t know so that everyone can learn together, and mistakes are minimized.
With new tools and technologies coming to the fore every day, it’s never been more important to arm data teams with the right training plans and resources. Continuous upskilling is especially vital if you wish to lead data science or AI teams, both of which are highly evolving and sophisticated fields.
Instead of reinventing the wheel, prioritize the most popular courses and trainings for your team. Data teams benefit from the versatility of both open and closed source software – offering the best of both worlds (Source: Secoda). A training plan should be engaging and targeted, with clearly defined learning goals. If your team works in sprints, it also helps to have a fixed number of hours allotted to training in each sprint to make sure progress is easily measurable.
Moreover, people often learn best when they have a good mix of tools or resources to work with. Instead of just enrolling in self-paced courses, you can add exciting guest sessions and webinars to ensure a balanced and effective approach – this can also offer the added benefit of networking!
A real-life example of a data organization structure that works well can be found at Shopify. Levi Bowles, Manager of Shopify’s Banking and Accounting data team, calls this the ‘Diamond Defense’ Framework. Levi explains that this comes from the defensive strategy followed in basketball that focuses on concentrating players in the toughest defensive zones and allocating fewer resources to areas that don’t require as much support (Source: Shopify). The point is to win the game, and that means prioritization above all else!
Within data teams, this can look like allocating team members to certain specialized tasks within a project (that also fit their skillset) and then readjusting this allocation as development progresses. Let’s consider the data engineering team structure. If a data engineer runs into a major blocker, for instance, other data engineers on the project can be pulled in to provide support. Levi also mentions the importance of having a ‘bullpen’ of additional team members available to fill any vacuum that may be created.
This approach can help prioritize the most important issues for the end user, and encourage collaboration and openness, while ensuring that team members are allowed to deeply specialize in their area of work. However, a major disadvantage can be the instability that comes with rotating assignments too much – for some people, this could even be a positive factor if they enjoy variety in their work. But it does pose the risk of distraction and must be carefully monitored to ensure that it is not doing more harm than good.
In the fast-paced world of data, success hinges on strategy, adaptability, and continuous learning. Data science best practices, generative AI development, data pipeline automation and prescriptive analytics are major trends, to name a few, that all data teams want to get caught up on. By embracing the practices outlined – from standardized documentation to fostering an open data culture – leaders can steer their teams towards excellence. As we've seen through the 'Diamond Defense' framework at Shopify, effective team structures prioritize collaboration, specialization, and flexibility (note that this may require some tailoring to fit your company’s needs).
Let's empower our data teams, equip them with the right tools and training, and watch as they conquer the challenges of tomorrow's data landscape.
.png)
.png)
.png)
.png)
.png)
