Drive business outcomes through effective collaboration
In this webinar, Eric talks with Max Werner, senior data engineer, and owner of Obsessive Analytics Consulting, about how data engineers drive business outcomes through effective collaboration with downstream teams. The session will give you role defining clarity to enhance your day-to-day workflow, and deliver insights into data team optimization.
Max will pull from his experience at companies like Proposify and Warner Media to give you a framework for successful cross-functional collaboration at every stage. He'll detail how the role of a data engineer morphs as a company scales - duties look different, the level of specialization required changes, and the number of teams the data engineer works with changes. He'll introduce helpful data engineer archetypes, and he'll cover the pros and cons of different data team structures.
Here's what Eric and Max cover:
The role of the data engineer at each stage: startup, SMB, & enterprise
Data engineer archetypes: the wrangler, the plumber, & the planner
Working with external partners
Pros and cons of common data team structures
Eric leads growth at RudderStack and has a long history of helping companies architect customer data stacks to use their data to grow.
Max lives and breathes data, from collection to transformation and analysis. He's the digital plumber that keeps data infrastructure running.
Eric Dodds (00:00)
Okay, let's get going. Welcome everyone. We've still got a couple of people trickling in, but we'll go ahead and get started since we're five minutes past the hour. I'm Eric Dodds. I lead customer success at RudderStack and we can tell you a little bit more about RudderStack later, but really what we want to do is talk to Max Werner who has been a friend of RudderStack for a long time and he has been a data engineer in multiple contexts. And the thing we're going to talk about today is how data engineers work with other teams and there are all sorts of different structures that we see as far as data engineers and the way they interact and even the term data engineer itself can mean different things depending the size of the company, depending on the stack and depending on sort of the data needs. So we're going to ask Max all about that and I think we have an intro slide, so I'm probably jumping ahead. Do we need to [crosstalk 00:00:58]-
Max Werner (00:59)
Oh, we do. We do. I mean, you covered it basically all there. Data engineering is like a catch all term for you do things with data, which is like if it's car mechanic that you do things with metal. Lots of things in there.
Eric Dodds (01:20)
Yes. Yes, indeed. Well, let's go ahead and dive in. You want to click to the next slide and we can do some intros since we've already covered that? I'll just kind of give you some accolades, but I'll just give a little bit of background around why we love talking with you and why I'm excited to chat with you about this subject. You've done data engineering in a bunch of different contexts. So for Proposify as a start-up, in the BDB space and you're currently doing some work with Warner Brothers, which is a massive company with massive, massive amounts of data in hundreds or thousands of data pipelines, and then you also do some consulting through Obsessive Analytics and you get to see all sorts of companies. So you've sort of lived and get to experience through consulting just a wide spectrum of different sort of ways that data engineering happens inside of companies.
Max Werner (02:15)
Yeah, absolutely. It gets crazy at times and we'll go into the difference on the start-ups versus the enterprise space, and there's also of course different maturity stages that companies have with data, and there's also of course a number of companies that I do consulting with that kind of they don't have a data stack, they're kind of just on the level of we're collecting some data with Google Analytics and that's kind of it and what do you mean you can do things with data that go beyond running a Google Analytics report? But there's a lot of variety like you said, data engineering is almost never the same from company to company or even within the same company. Oops. I double clicked there.
Right, so different roles for data engineering. I just put SMBs here, I think it's small, medium businesses but that covers start-ups as well. And to start off with a little thing that came across my meme list that kind of summarizes this and it says, dear recruiters, if you're looking for a Java, Python, PHP, React, Angular, PostgreS, Redis, Mongo, AWS and so on, and so on, that's not a full stack developer, that's an entire IT department. And if you look at job postings for data engineers or even data analysts, this is basically the same. You can replace a couple of things here, I mean, instead of Java and Python, it will probably say Python, but instead of Java, PHP it might say Salesforce and those kinds of things. But generally the breadth of the skillset that companies are generally looking for when they're hiring a data engineer is crazy.
Eric Dodds (04:06)
Right, I mean, it's kind of like, okay, you have expertise in SQL, and you can do the odd stuff to connect to downstream tools like Marketo and Salesforce, and you can deploy pipelines on Kubernetes clusters.
Max Werner (04:21)
Yeah, basically you can do it all from start to finish.
Eric Dodds (04:28)
Okay, we know that's not actually reality and sort of the keyword stuffing for engineering focused job descriptions is horrendous and I think that's a whole other webinar topic. And so let's talk through the differences depending on stage and what skillsets do you actually need at sort of the stage of the business.
Max Werner (04:51)
Absolutely. Right, so if we go kind of from left to right from basically small to ever large it comes kind of down to these three different categories, how specialized are you, what do your day to day duties include in the given job and what are the teams that you work with? And of course at a start-up space when there just isn't that many people you are generally not specializing. You actually do mirror that screenshot from before where you kind of have to know a little bit about everything because there is just not a dedicated data operations person that can help you with the Airflow DAGs or Kafka consumers. It's just you kind of have to figure it out as you go, which [crosstalk 00:05:39]-
Eric Dodds (05:39)
And I think in a lot of those situations at least what we've seen is someone who is a leader on the engineering team is sort of responsible for the data stack, whether that's just because they're there and they know it needs to be done or if that's intentional. It's just sort of shepherded by an early leader in the engineering department whether or not they're formally a data engineer those responsibilities roll off to them generally.
Max Werner (06:08)
Yeah, for sure. Of course, that's exactly it, it depends. In some start-ups you have that engineer that is slightly data leaning, or maybe it's even the CPO that kind of does it in his spare time, or you come in and you have a little bit of that server background and you can do it as the engineer, it varies widely. So your duties they are everything or some of those things depending on how small the start-up is. And naturally that means the teams that you work with is all of them, you've got to talk to the engineers to help you with the stack and more so the Devops side of things there you have to talk to the engineers [inaudible 00:06:51] analytics code implemented usually inside the application.
I've seen very few instances where data engineers actually touch the main product codes or be it a web app, a mobile app, a website, whatever. The data engineers often don't interact with that code base I think, usually because they're not trusted but it's a different kind of work, but it helps to have that skillset a little bit, kind of if you know enough about let's say you're start-up has a mobile app and you know enough about how mobile apps work, lifecycle stages for entering iOS apps and these kinds of things that you can communicate more directly with the engineers that actually built the app, that's great.
As we kind of mature a little bit and we get into the small, medium business stage you will naturally specialize more and also of course that means the job postings for data engineers will be more specialized. So if there actually starts to be a data team you're not just the Jack of all trades, you have someone who is more focused on the pipelines, someone maybe that's more focused on maybe even some analysis, or someone that's even directly just specialized on you make sure that Salesforce works from the data perspective, which means [crosstalk 00:08:19]-
Eric Dodds (08:19)
Here's a question for you on the SMBs. So I think the start-up is fairly straight forward in terms of you generally have someone from engineering who's kind of doing a little bit of everything. On the other end we'll talk about enterprise, that's a little bit more straight forward just because the roles become more defined. But in this SMB space there tends to be... And I'm interested on your perspective on this because I know you have done a lot of work sort of in the marketing ops type space as a data engineer. And one thing that we see that's kind of interesting, marketing tends to be the tip of the spear when it comes to data, which makes sense. I mean, they need a lot of data to optimize what they're doing, to understand what's going on, the top of the funnel tends to produce a lot of data, you're driving a ton of traffic and then downstream data sort of subsets of that by conversion rate.
And so in the SMB space, what do you see around sort of there's roles that can blend in different directions? So if someone from engineering has been leading the charge they generally sort of move into a more formal data engineering role that is working closely with marketing, but it's also the case when someone from marketing ops who doesn't really have an engineering skillset kind of glades into the data engineer role and becomes increasingly technical. Do you see any patterns around that? Is there sort of benefits or drawbacks to either approach? I mean, it's not something that you can control, but it's something that we see pretty often.
Max Werner (09:58)
Yeah, for sure. I mean, I've seen it more often on the marketer that becomes more technical side of things rather than the engineer that slides more into the data side of things, just because most start-ups have a hard enough time to get the qualified engineers already so they want to keep them focused on building the main product. And so it's more often in my experience case that the marketers just lots of times out of necessity become more technical.
Eric Dodds (10:30)
Max Werner (10:30)
And it almost never backfires because a marketer already has that goal oriented mindset and usually with these data types of things having a goal and working your way backwards of how you get to that goal works out a lot better than just throwing stuff at the wall and seeing what sticks.
Eric Dodds (10:55)
Sure. Super interesting and I think we'll just continue to see I think the blending of data engineering and sort of fill in the blank ops, whether its sales ops or marketing ops depending on the business because the ops roles are just having to become more technical, as apposed to being really good at configuring a single SaaS tool, if that makes sense.
Max Werner (11:20)
Yes, for sure. I mean, as we try to scale more and do more things with fewer people or be more efficient at some point you need to be able to talk to the marketer in terms of the Saleforce objects and fields, rather than saying this your workflow that you put together. So you sort of need to be able to talk with them at a more technical level to really understand what their requirements are.
Eric Dodds (11:50)
Sure. One other thing I would say in the SMB space and then I want to move to talk about enterprise and there's lots more stuff, but just I think a lot of companies find themselves in this space and this is where the lines are blurred the most is leveraging a data warehouse to do a lot of data engineering type work. So in the start-up world you may not be and really in many cases it doesn't necessarily make sense early on to be really warehouse heavy in terms of moving data around, A, because you don't have that much data and B, because a lot of times you'll maybe just replicate your production database from the app and that has basically all of your data points and you can sort of use that as a common thing we've seen with early stage companies.
You don't need to provision an entirely separate warehouse for analytics, you can sort of if you're running PostgreS you can replicate PostgreS and sort of get the data where you need to go and probably doing a lot of just initial manual querying to figure out what's going on. But as you get into SMB space you start to adopt the warehouse, analyti