Sam: Hi, this is Sam Ramji and you're listening to open source data today on open-source data, we have Melody Meckfessel who is the Co-founder and CEO of Observable the destination for data visualization that helps people explore, analyze, and visualize data on the web together. Observable helps everyone make sense of the world with data in a more approachable, accessible, and social and maybe even a fun way.

Previously, Melody was a Vice President of engineering at Google, where she led dev ops for Google, including Google cloud platforms, tools, and systems with a global team of engineers over a thousand people worldwide. Her team powered, the world's most advanced continuously delivered software, enabling development teams to turn ideas into reliable, scalable production systems.

Melody's belief is that great things happen when developers are allowed to use their creativity together, that software engineering is a team sport and she has spent her career working to empower developers to do their best work.

Melody. Welcome to the show. It's a pleasure to have you on.

Melody: Thanks so much, Sam. It's so great to be here.

Sam: It's almost like old times when I got to be your product manager, as you were leading those thousand engineers.

Melody: I know, I know I'm so grateful to be able to spend this time with you.

Sam: We like to start each conversation by asking what open source data means to our guests. So what does open source data mean to you?

Melody: So I have to ask a question. I'm going to put you on the spot, Sam.

Sam: Sure.

Melody: So are you familiar with improv?

Sam: Improv in terms of improvisational theater?

Melody: Yes

Sam: Absolutely.

Melody: So. This is my theory on open source. So improv has these rules and there's two rules. You have to agree and you have to add two. So have you heard this phrase, "yes, and"?

Sam: Yeah.

Melody: Yeah. Right. So, bear with me here. So open source to me is about people coming together to create something out of nothing. Which is exactly what improv is. And it immediately means, you know, people working together and collaborating. And thank you for that very kind introduction. I've seen thousands of developers working on problems from UI tweaks to managing, you know, complexity at Epic scale, billions of lines of code of Google and millions of builds and massive deployments to, you know, millions and millions of users, you know, from the little small tweaks to the big, hairy infrastructure challenges.

Now I've seen developers work together on open source projects all over the world, right? So whether they're gearing up for release or they're debugging issues together, and I think there's something that's really beautiful. About this principle of building something and putting it out there into the world, having someone, a human, a person, find it, share it and have more people build upon that.

And so I think we can all pretty much agree that open source is the dominant development model in the world. I mean, if you think about five to 10 years ago, the acceleration of enterprises using open source technology it's like table stakes. These days. We know that today's software is either open source itself or builds on extensively. And we, as people, as software developers are more dependent on code written by other people. And we're more likely to share code that we've written while we might work on a PR solo, it's our teammates and our collaborators that are helping with reviews and helping us push it out to our communities. And I think in the times that we're in now, it's even more important to collaborate more deeply on code and our understanding of data together.

So it's just this idea of yes. And this idea of people coming together to create something out of nothing is to me, very inspiring and beautiful about what open source is doing in the world.

Sam: I really appreciate that answer. There's so much depth behind it when I think about open source and yes, and I think about the get mechanism and get hub as a service for being able to look at different people's opinions, you know, fork them, contributing them back, pull requests.

And I look at what you and Mike have been building or with your team at Observable on top of and around D3, right. How could we start to fork data? How could we take the data and the visualization that makes a human being smart about the data? Cause we weren't really evolved to you do much more than run away from tigers and, you know, catch things and be nice to each other.

Right. But, but you know, a 32 dimensional set of data is not something that is comprehensible by people until you start to put into these amazing charts, but the charts themselves are computational. So how do you fork and share those? There's something happening I think at the core of what you're working on every day, that is a huge frame of open source data.

Melody: Yeah. And I think you talk about software complexity, right? What GitHub's latest Octoverse reports said 50 million plus developers expected to grow to a hundred million by 2025, which feels to me like it's right around the corner. When you look at the intersection of like software complexity, increasing, well, there's also data complexity that's increasing and how people are communities going to come together to navigate that?

Sam: Yeah, that's a really interesting question, right? We have generated most data in the world from apps. So we have app driven data, but there's this big shift towards data-driven apps. So that change is coming by 2025. I think data-driven apps will outpace app driven data. Add into this, a new community of 50 million developers who aren't yet developing and there's a big challenge. So this is something you look at constantly like looking at how communities work together to solve open source challenges and specifically in data. What are you seeing in your day to day? What are you seeing in people using the three and what are you seeing beyond? Because you're connected to a lot more than simply the company that you're operating.

Melody: We're definitely more powerful together, when we collaborate. But I just think if there are more people and there is more collaboration, complexity is to me, it's a natural outcome of that. So that growing complexity in the world, to your point in our software and our data, it is demanding new ways for us to understand.

And these questions that we have around how to make sense of data in a meaningful way. As that complexity increases, it's super easy for us as developers for the technical details to consume all of our attention. And to distract us from what we're trying to do, which is gain insight, and to think with the data. We have to come back to, are we asking the right questions? Is this the right data for us to even be looking at? And what are we trying to do to understand it? And I think this is where visualization really comes into play because visualization techniques help us. Makes sense out of the data.

So you brought up D3, right? Development of libraries, like D3, I think their power and their positive impact is because it brought together people into this community around these tools to build dynamic data visualizations. And it was the community and the sharing where people were riffing and iterating off each other's code and examples. That we have to do that same thing with data.

So when you talk about open source communities, it starts to be like, how do we look at the workflow to help data be more approachable and interactive and transparent and shareable? Right. I could keep doing the ANDs to add on to that, but I think the people in the community, learning from each other and building on each other's work, we need to do the same thing for data. And that means establishing new techniques and sharing them with the world. And. And not allowing ourselves to get all consumed by the technical matters, meaning like the development, tweaks, whatever the takes our attention away from is this the right data? What are we trying to understand? What are our insights?

Sam: Yeah, you bring up a lot of things there, you know, between your teamwork and workflow and numeracy.

One of the things that Jesse Anderson talked about on the open source data podcast recently is the change in data teams. And in the beginning, like what's happening with data engineering, data operations, data science.

What you've seen with D3 is pretty fascinating when I look at D3.js, which just turned 10 years old. Congratulations.

Melody: Yeah. Kudos to Mike Bostock and the entire D3 community. Yeah.

Sam: Yeah, it's huge. And I kind of look at D3, almost as a dark art personally, because, you know, I'm just a simple software engineer and I look at these incredible visualizations in the New York times or any of these other, digital media renderings and it's just mind boggling. Like first, how did they get the data? How did they make it sing and dance? How did you do this chart where you see the flow, of the change of the shape of the world across a graph or across a map? I'm just not able to think about how you do that. It's moving from being a dark art to something that's quite transparent, right? How do you change the shape of the data team so that people can actually easily interact and work on these kinds of things? So maybe you could share with our audience a bit about how is that developer workflow changing for data visualization as it becomes a bit more mainstream?

Melody: That's a really important point because when you think about the people and their lens. So whether they're a data scientist or a data analyst or data engineer right, we all have these different sorts of perspectives that we're bringing into it. And the workflow needs to continue to evolve. I mean, of course it has in software engineering, but specifically around data.

So I believe writing software as a team sport. And if you think about data visualization itself, it is a type of software development. So right. D3 is a library. What people do with it is the software development. You could go create a data viz, Sam and I could go create it. We could create it for ourselves.

It's often something that we're doing to share insights with other people. And as developers, we have strong opinions, right? We care about the outcomes. But the data helps us think, and we can use the data to ask questions and identify patterns, and hopefully make better decisions.

And when we think about the developer workflow, we can't just talk about software. We have to talk about data because data is the foundation of most things in our world. Oftentimes. Like we'll get stuck around the data. Well, how do you get unstuck? And I'm oversimplifying here, but how do we get unstuck around? Like, okay, what are we doing here? What are we trying to understand? You bring the visualization into it. You visualize the data to better explore. And when I think about developers and our workflow, I think about the fact that we just talked about it, complexity, software complexity, data complexity. I don't know about you, but like I have a finite amount of time and I want to best use my brain for what I'm trying to understand in my workflow. And so I think, when it comes to the developer's workflow, Database, right. It really can help us understand. And I'm going to just go through the gamut here. Our code structure flows invocations with the intent to make our code better. So to refactor it, to clean it, to improve it, we can model directory structures.

We can explore hierarchies and I'm not even getting into the production operation side of it, of which data visualizations are extremely helpful to understand the state of your system. And what's how data is flowing and what's happening. But you know, my big claim that I've been making around what I've seen in the market around developer workflow is that not just developers, but everyone who uses visualization, who has access to data can use visualization to be smarter, to be better.

Visualizing integration tasks that are flaky using flame graphs to profile your code and find hot sprouts, distributed tracing the more data, the more complexity, the more opportunity for us to use visualization to make sense out of it. And I think it's this really fundamental biological topic where if you think about visualization, it is.

Tapping into Sam and Melody's human visual system and our incredible ability to pattern, match, and recognize and understand. But that's all with the intention that for developers, we want to build better products. We want to ship faster and we hopefully want to have fun doing it. So if we see the information, like we can learn through interaction and I use this quote all the time, Sam, I feel like it's so profound. It's by Brett Victor, he said, "People understand what they can see. If a programmer cannot see what the program is doing, she can't understand it." And it's that belief really, that has motivated us. To try and improve access to data visualization in the world, and really to create Observable this intention of making data more accessible and approachable, and inclusive through visualization.

I'm an optimist, I think visualization through collaboration and our shared understanding. We can make the world better together.

Sam: It's fascinating. You evoked for me two totally different lines of developer data visualization. One is kind of developers helping themselves a little bit. Like what, you know, we were building a Google with stack driver and integration to Grafana, right?

That's visualizing the behavior of the complex systems, or you look at the ELK stack, right? elastic Logstash and Kibana. It's about developers visualizing, you know, their production environment. But going beyond that, to say, here's a tool that you can use to understand all sorts of elements of your work. Yeah, it's kind of the dataviz for developers helping themselves.

There's another entire movement around, how do you communicate complex quantitative information to broader teams that need to make big decisions? Like when I receive data visualizations, often they've already been flattened into a graphic and pasted into a presentation.

Melody: Yes!

Sam: Right. And if you could step back beyond that, you might have the next read only version, which would be an app that's maybe a little bit interactive, but I can't see much with, but there's a step before that, which is what if we could interact on the code that's generated on the graph, switch the data a little bit, make a few different assumptions in the code and we can all be literate or maybe the word that I've I've heard you and others use is instead of literacy, numeracy, we can start to be more intelligent about different.

Tools, right. What are a couple of different statistical methods? What are a couple of different standard visualizations or views, not milling, a graph that are appropriate for particular fields of data? How do I look at a cohort? Analysis of, should I look at cross tabs in a survey, right? There's this different field of work that is evolving by pushing back from flattened data.

That's been dumbed down into a ping file and go back to the origin, but make it simple enough that I can spend my limited brain power and heartbeats interacting with other people around the class, a decision we ought to make.

Melody: Yeah.

Sam: How do you think about numeracy? One of the things that you and Michael both talked about is that you're on a mission to increase numeracy in the world and help share tools and raise the basic level of education about how to think about numbers.

Melody: When I look at, how do you amplify understanding, how do you help people learn and question, and what are the challenges or obstacles that go along with that? How do you help people learn? I often talk about data literacy too, as another way of thinking about it. Like we think about our ability to learn, to read and write what is our ability to understand and explore data? And I think what's motivated us and Observable is, how do we enable that in service of the people that are trying to understand, not just the concepts around how you find your data, you transform it, you clean it, you get it in a form you can use?

So, so much work goes into that. Sam, when you bring up the topic of numeracy or we talk about data literacy or. It's the exploratory visualization and that process of tapping into our intuitive learning through that visualization where interesting things start to happen. So you have anyone out there has an idea in their head and they want to understand it, but it's not until it actually gets shared and other humans start to interact with it, that it becomes powerful and transformative. And so to me, when I think about what we're trying to do at Observable, it comes back to the community, the other people, the community of insight that we're trying to support. We want to have this seamless process where we can support people, not just wrangling their data, but getting to the point where they feel like they've maybe try it a couple of visualizations and they found the right one, and then they've brought interactivity to it. And it's the interactivity piece that's so fascinating, because then you don't feel the pressure to export to an image and then put it in a slide deck. You can have Chet from DataStax interacting with data around decisions that you're trying to make or any enterprise company.

And so I guess the one note that I really want to call out here is that I believe that thinking, thinking with data is an essential skill that we all need to have, and it should not be relegated to a specialized role or siloed or constrained. It should have expressivity. And when everyone is participating in the organization, right, from C-level to developers, to scientists, to a business analyst, everyone needs to participate for us to make better decisions.

And when we're all thinking better, if we're able to explore and visualize the data without the limits and it's, I don't know, it's this dream, right? That everyone is kind of, part Data Explorer, part Visualizer, part Data Communicator, because it's the communication that really catalyzes the change.

Sam: Yeah. It's about empowerment, right? Enriching people's ability to get things done that they care about in the world. And that kind of takes us pretty quickly to democratization, right? If you're starting to do empowerment, you want to empower as many people as possible.

So you've made a really interesting decision, right? To support JavaScript instead of R and Python wrapped around what you're doing, which I think at first it's like, what are all the data viz people use R and Python. And then you start with the numbers that you were talking about, you know, another 50 million developers by 2025. Do you know what language the next 50 million are gonna be using?

It's going to be JavaScript. Maybe there's maybe a million R and Python developers. If you kind of look at people who are using those skills in this field and 15 million at least already in. Self-described JavaScript centric, professional roles. So that's a big change, but you've had to think about this real differently.

So I'd love to hear you talk about the sort of the connection between empowerment and democratization and the very technical decision you've made to focus on JavaScript and not other languages for data viz.

Melody: Why don't we start with a technical bed and then come back to the, talk to the empowerment. So I have this conversation at least multiple times in a week when I'm talking with people that are looking at Observable or companies, people to partner with, and we support JavaScript.

And, I get asked, why don’t you support these other languages? And what are you doing? Like Melody? This is crazy. Yeah, it's bold. But follow me through here. Visualization is a means of communication, right? We keep coming back to that, like deepen your understanding and communicate. We want visualization to help enable and empower how data analysis has a positive impact within an organization.

So we want those exploratory visualizations. To be accessible by everyone in the organization. So now we need it to run in the browser. Right. And we need JavaScript. JavaScript is the language of the web for collaboration. This is a "yes, and" Sam, there can still be upstream. Offline analysis that happens in R and Python and other languages that gets to the sort of distillation of the data in a form that's suitable for the interactive exploration.

And I just want to share this person, Ben Smith. He wrote this great article recently it's called JavaScript and the next decade of data programming. And he has some observations, which I think are relevant. He shares that he thinks the languages for data analysis have failed to keep up. Now, new data formats are making the differences between Python and R and JavaScript less important. And that JavaScript, which has been viewed as a front end language can be doing backend data work. He does quote, Observable in his article. So I should mention that. But for visualization really JavaScript is becoming more and more prevalent. And so it's this idea of, how do you bring the entire data analysis and data viz processes into the browser. And if you look at the advancements that have been made in the browser, it's totally possible.

At Observable, technically people are writing code and interacting with data in a notebook. And in the notebook we use data flow. We're compiling the code. You don't have to rerun the cells like you have to remember to do in Jupyter. And because it's in the browser, you have this one click, you can share it, share the URL, but you can also use your browser tools to debug and you get this immediate feedback, which you and I know when developers have immediate feedback, it completely changes the game in terms of speed of delivery.

You can use libraries, like we talked about D3 or WebGL, import codes you can reuse. I mean, I can sort of go on and on and on about how the shift, I think setting people, humans up to be able to do this work faster and easier. And I want to say, I think it can be a compliment to other backend data processing and analysis that's happening in organizations.

Sam: That's really a story about ecosystems, right? Ecosystems of people, tools. Those are the ones that create the most power for change. And when you look at this JavaScript ecosystem it is big. It is diverse.

Melody: Yeah.

Sam: So it was from the front end of the backend. So if I had to make a business bet, I would bet on JavaScript because the, the proliferation of new approaches, new tools that are popping up every day.

And it's fundamental, integratedness into the web experience. So you can see it on a mobile device, or you can see it on anything else that you can imagine. It's pretty powerful.

Melody: Yeah.

Sam: You made a super interesting point that I want to connect to Kubernetes, actually, you and I got to do a lot of work on Kubernetes based tooling, right?

Google cloud dev ops, and the ability to make a lean flow from concept to production and iteration, right through an instrumented set of container-based. Tools all orchestrated by Kubernetes. When you say JavaScript on the server to do data of his work, I immediately go back to node and containers as kind of this magical combination of stateless processing, where all of a sudden, Hey, it didn't matter if node was scaling properly or not, or if the process locked, right, the system could just restart the container, and it was fine. So that was six, seven years ago that we started seeing that pattern. Now, I'm starting to see a big movement as people have laid in their Kubernetes estate to manage compute. They're starting to bring data to that game and bring in stateful workloads.

I'm really interested in your perspective on how data is moving to Kubernetes and how that relates to more work for JavaScript and developer access.

Melody: You know, I have to be honest, it is hard to keep up with that space. It's I think there's so much tooling innovation that's happening there and ease that the ecosystem is bringing to managing workloads, but also being able to navigate just the volume of data that folks have to interact with.

I will say I've had my lens, my focus on data viz, and I'm really excited to start doing some, some projects and some prototyping around how to bring data viz into that story of data and managing workloads coming together. Because I think there's a real opportunity there for making better decisions within the ecosystem.

And again, bringing open source into it, like how can people innovate faster on the tooling side, in that space? So it's kind of like, hang on, there's some interesting things that are going to be coming out, in the coming year.

Sam: Totally. Server-side devas with JavaScript. It's going to happen.

I'm curious about what guidance do you give to people who are interested in getting into this space? Right? Maybe somebody is a data engineer or data operator, or maybe someone's a new college grad and thinking about what they want to do with their next few years of their life, what guidance would you give them about how to jump deeper into this, or how to connect to people who care about it as much as you do?

Melody: Yeah, maybe I could share a little bit about Observable because I think when I look out in the space, there really isn't a place for people to come together. And so one of the things that we're doing at Observable is trying to create this place for people to come together, to learn and to be inspired, whether you're a teacher or a developer or data scientist, there's really no place to interact and kind of learn from each other.

So we're really inspired by the power of open source and we believe those same principles apply to data viz. We kind of think about Observable as this gigantic repo of functioning interactive data viz to get you started with this community of folks that are really passionate about it. That actually will share what they're doing.

And you, as a person who's new to the space can fork it and modify it and add your own data. And it's not just hobbyists, right? Like I think if you're a data engineer and analyst and you're at your company and you're trying to do interactive data viz, you can come together with other people to share the challenges that you're working through.

So my advice is really, it's pretty simple. I think that we're stronger together than we are alone. And I think collaboration is the way and it's fun to do it. We can do it around data. We can do it around data as we've proven. We're, we're good at doing around software development, but all that creative energy and innovation happens when we're working together.

And if we can make data more accessible and we can build on each other's work, share techniques, share tools, share insights. You just mentioned, Kubernetes in the ecosystem, look at what's been happening there. Then we have to enable everyone to collaborate, not just the experts. And I think the way that we do that is we take the open source model and we bring it to data. We bring it specifically to visualization, which is how we're going to. Share the insights that we're getting from the data itself. "Yes. And" - let's do it together, Sam.

Sam: That's awesome. You've carved out such a unique path through your career and through the industry. I know that you inspire me and you inspire many other people who are pursuing technical excellence and trying to create change in the world.

So I think folks will be equally inspired to see what you're doing at Observable as you're bringing the spirit of GitHub and open collaboration to data technology and data visualization.

Melody: Thank you, Sam.

I am grateful to continue to have this discussion with you and other folks that are working through these challenges as we try and make sense of the world. Make sense of our data.

Thanks so much for having me.

Open Source Data

A Podcast Series With Charna Parkey

Share

Season 1 · Episode 14

Data Visualization, Democratization, and Javascript with Melody Meckfessel

Episode Guest

Melody Meckfessel

Episode Transcript