Secret Management & Safeguarding Your Product Data with Brian Vallelunga, Founder and CEO of Doppler

[00:00:00] [SPEAKER_02]: Welcome to the Convergence Podcast. I'm your host, Ashok Sivanand.

[00:00:06] [SPEAKER_02]: Revenue alone does not replace strategy. Revenue is vanity, profitability is sanity, and then there's got to be something that rhymes that talks about strategy there.

[00:00:19] [SPEAKER_02]: On this show, we'll deconstruct the best practices, principles, and the underlying philosophies behind the most engaged product teams who ship the most successful products.

[00:00:36] [SPEAKER_02]: This is what teams are made of.

[00:00:38] [SPEAKER_02]: Welcome back, folks. On this episode, I am joined by Brian Vallelunga, the CEO of Doppler Technologies.

[00:00:47] [SPEAKER_02]: Before founding Doppler in 2018, Brian was a lead engineer at Uber, where he worked on special projects for the C-suite.

[00:00:55] [SPEAKER_02]: Doppler is a secret ops platform backed by industry heavy hitting venture capitalists like CRV, Google Ventures, Sequoia, Greylock, Kleiner Perkins, and their own companies.

[00:01:05] [SPEAKER_02]: We're also a Y Combinator company. Now, if you've worked closely with a product team, chances are that secret management has been a topic that you've had to wrangle with at some point.

[00:01:15] [SPEAKER_02]: This is either because your development or deployment teams don't have access to the right secrets, and that slows down how quickly you can get your code to production,

[00:01:24] [SPEAKER_02]: or worse, way worse, because your secrets were exposed to the public and put your data and your customers' data at risk through a data breach.

[00:01:34] [SPEAKER_02]: On the episode, we get to hear about Brian, about what secrets are, why your development team might be bad at managing secrets,

[00:01:43] [SPEAKER_02]: and four of his crucial questions that he believes all executives should ask about their teams in order to understand where your areas for improvement are,

[00:01:54] [SPEAKER_02]: to be highly secure and efficient at how you manage your secrets.

[00:01:59] [SPEAKER_02]: We also get to hear from Brian about ways to proactively train your team to avoid such issues,

[00:02:06] [SPEAKER_02]: as well as playbooks on what to do if your team accidentally exposes secrets in your code,

[00:02:12] [SPEAKER_02]: or worse, if you have a data breach.

[00:02:14] [SPEAKER_02]: As well as war stories about companies like Twitter and Toyota,

[00:02:19] [SPEAKER_02]: their data breaches, and their highly preventable reasons what led to their data breaches.

[00:02:25] [SPEAKER_02]: I also enjoyed zooming out a little bit with Brian and hearing about how he came about founding this business,

[00:02:30] [SPEAKER_02]: scaling it, and how customer research is an extremely critical part to how they go about building delightful developer tools,

[00:02:39] [SPEAKER_02]: prioritizing their product roadmaps, as well as setting the roadmap for their business.

[00:02:45] [SPEAKER_02]: Subscribe to the podcast to get future episodes as soon as they're published.

[00:02:49] [SPEAKER_02]: If you find this helpful, give the podcast a five-star rating on your podcast app,

[00:02:54] [SPEAKER_02]: or hit that like button on YouTube.

[00:02:57] [SPEAKER_02]: Here is Brian.

[00:03:04] [SPEAKER_02]: I appreciate you taking some time here, and this is definitely a pretty big problem area,

[00:03:10] [SPEAKER_02]: and I've been on teams that have kind of had a rough day as a result of some of the things we're about to talk about.

[00:03:17] [SPEAKER_02]: So, you know, for maybe the less technical folks that do listen to the show,

[00:03:22] [SPEAKER_02]: why don't we start with defining what secrets are?

[00:03:26] [SPEAKER_01]: Yes, super important.

[00:03:28] [SPEAKER_01]: Okay, so secrets are API keys, database URLs,

[00:03:33] [SPEAKER_01]: or anything that would be generally configured by the environment

[00:03:36] [SPEAKER_01]: that are either used to configure your application or communicate with another service.

[00:03:41] [SPEAKER_01]: So, like an API key could authenticate you to Stripe or Twilio for payment processing or messaging,

[00:03:48] [SPEAKER_01]: and a database URL would obviously connect you to a database,

[00:03:51] [SPEAKER_01]: and maybe you have some encryption keys.

[00:03:52] [SPEAKER_01]: They're usually stored in an ENV file or in like a cloud secrets manager,

[00:03:57] [SPEAKER_01]: like AWS secrets manager, or GCP secrets manager.

[00:04:02] [SPEAKER_01]: And I think the most important part about secrets that's often overlooked

[00:04:07] [SPEAKER_01]: is that you're actually using them almost every day.

[00:04:08] [SPEAKER_01]: I can rarely think of an occasion where, as a developer, I was not using secrets.

[00:04:14] [SPEAKER_01]: It's not like this new concept that's been around for a long time,

[00:04:17] [SPEAKER_01]: or that's brand new.

[00:04:18] [SPEAKER_01]: It's actually a concept that's been around for a very long time

[00:04:21] [SPEAKER_01]: because we generally write code that is meant to be work in development and production,

[00:04:27] [SPEAKER_01]: so it needs to be configurable.

[00:04:29] [SPEAKER_02]: Absolutely.

[00:04:30] [SPEAKER_02]: And keep me honest here, correct me if I'm wrong.

[00:04:33] [SPEAKER_02]: Anytime really two systems are hitting each other,

[00:04:37] [SPEAKER_02]: especially if they're housed on different servers or different environments,

[00:04:41] [SPEAKER_02]: both systems want to know that there's an approved, authorized, authenticated interaction

[00:04:48] [SPEAKER_02]: that's happening through this, whether it's an API call or you're hitting a database directly,

[00:04:52] [SPEAKER_02]: and that bad actors can't get in.

[00:04:55] [SPEAKER_02]: And so the secret is kind of like the key to get in

[00:04:57] [SPEAKER_02]: and validate who you are and that you do have access.

[00:05:02] [SPEAKER_02]: And then, of course, the challenge is that now our software engineers

[00:05:06] [SPEAKER_02]: are all kind of holding on to this ring of keys

[00:05:11] [SPEAKER_02]: that if we don't keep it in a safe place, it exposes us to bad actors, right?

[00:05:15] [SPEAKER_02]: So now that we've gotten to understand a little bit about what secrets are,

[00:05:19] [SPEAKER_02]: what do you see today in terms of how teams,

[00:05:23] [SPEAKER_02]: whether they're at big companies like Uber or, from your experience, other companies,

[00:05:26] [SPEAKER_02]: how do software teams tend to use secrets today?

[00:05:30] [SPEAKER_01]: Yeah, you're 100% spot on.

[00:05:32] [SPEAKER_01]: Secrets are the digital keys to your data kingdom is the way I like to look at it.

[00:05:36] [SPEAKER_01]: And there's a pretty wide spectrum of how people manage secrets today.

[00:05:40] [SPEAKER_01]: I'd say a lot of small startups are doing the most bare bones thing possible.

[00:05:45] [SPEAKER_01]: They have a literal file on every developer's computer called an env file per project

[00:05:50] [SPEAKER_01]: that holds these secrets in unencrypted text, which is bad for a lot of reasons.

[00:05:55] [SPEAKER_01]: It's bad, A, because you're not supposed to be having secrets on disk.

[00:06:00] [SPEAKER_01]: That's a really big security problem there.

[00:06:03] [SPEAKER_01]: And then, B, it's kind of like in the days before Notion or Google Docs

[00:06:07] [SPEAKER_01]: where you have like one doc that everyone's editing in real time.

[00:06:10] [SPEAKER_01]: It's more like I have a Google Doc and you have a Google Doc and everyone has a Google Doc.

[00:06:14] [SPEAKER_01]: And these docs are all changing all the time.

[00:06:16] [SPEAKER_01]: And I have to share it with each other in Slack or email,

[00:06:19] [SPEAKER_01]: these insecure places that you should never be sharing them over just to keep up with your team.

[00:06:23] [SPEAKER_01]: And then you have to figure out what are the secrets that should change,

[00:06:27] [SPEAKER_01]: i.e. my coworker just added some code that relies on the secret versus secrets that shouldn't change

[00:06:33] [SPEAKER_01]: because I've changed them to work for my local setup.

[00:06:36] [SPEAKER_01]: And so you have this like constant game of like conflict merge resolution at the local level.

[00:06:42] [SPEAKER_01]: And then at the production level, it's very much copy and paste it into AWS Secrets Manager,

[00:06:48] [SPEAKER_01]: GCP Secrets Manager, Kubernetes.

[00:06:50] [SPEAKER_01]: And the flow typically there is like let me open a ticket, put the secret in the ticket,

[00:06:55] [SPEAKER_01]: which is already really bad because those systems were not meant to hold highly sensitive data.

[00:06:59] [SPEAKER_01]: Again, this one piece of text is like the key to a door that unlocks all your data, right?

[00:07:05] [SPEAKER_01]: So you really don't want to put that key in a lot of places.

[00:07:07] [SPEAKER_01]: You want to put it in a bank vault essentially.

[00:07:10] [SPEAKER_01]: And so now you have this sitting ticket and eventually some DevOps engineer or security engineer

[00:07:16] [SPEAKER_01]: will copy the secret out of that ticket and paste it into AWS.

[00:07:20] [SPEAKER_01]: And the big problem with that outside of it being in an insecure place

[00:07:23] [SPEAKER_01]: is that your code may get to production faster than that ticket gets closed.

[00:07:27] [SPEAKER_01]: And now you have an outage.

[00:07:29] [SPEAKER_01]: You have a guaranteed outage on top of all the security problems we just chatted about.

[00:07:32] [SPEAKER_01]: So it's pretty bad how companies are doing it today at the small scale, the small end of the market.

[00:07:37] [SPEAKER_01]: And then for much larger companies, they typically have either built out their own solution,

[00:07:43] [SPEAKER_01]: which tends to be quite brittle because it was built for that moment in time.

[00:07:47] [SPEAKER_01]: It wasn't built for like all the needs of their scale, right?

[00:07:50] [SPEAKER_01]: A lot of startups by definition, like startup equals growth.

[00:07:53] [SPEAKER_01]: So they're constantly growing, but the engineering team will build it for that moment in time.

[00:07:57] [SPEAKER_01]: And so like six months, a year later, they're like, wow, the system's starting to fail us again.

[00:08:00] [SPEAKER_01]: So let's do another rebuild.

[00:08:02] [SPEAKER_01]: And every time you do that, you introduce a lot of risk.

[00:08:04] [SPEAKER_01]: You may be introducing a ton of vulnerabilities.

[00:08:06] [SPEAKER_01]: You may be introducing downtime.

[00:08:07] [SPEAKER_01]: So it's a very risky business to be changing a secrets manager.

[00:08:10] [SPEAKER_01]: And it's also a distraction because that's not the stuff that drives value for your customers.

[00:08:15] [SPEAKER_01]: So it's also a pretty big source of distraction on top of risk.

[00:08:19] [SPEAKER_01]: Or the last case is they're using something like HashiCorp Vault,

[00:08:23] [SPEAKER_01]: which really wasn't meant to solve this problem.

[00:08:25] [SPEAKER_01]: It was meant to more like store like personal identical information,

[00:08:30] [SPEAKER_01]: like a social security number or credit card number,

[00:08:32] [SPEAKER_01]: things like I would consider like personal secrets or data secrets, not application secrets.

[00:08:37] [SPEAKER_01]: And so what they'll typically do is they'll take this basically a very secure version of Amazon S3

[00:08:43] [SPEAKER_01]: is the way I like to think about it.

[00:08:44] [SPEAKER_01]: It's like file-based path storage.

[00:08:46] [SPEAKER_01]: That's very secure.

[00:08:47] [SPEAKER_01]: And they add a bunch of wrappers on top of it to build out this orchestration layer

[00:08:51] [SPEAKER_01]: so that the secrets automatically get into production or get into developers' hands.

[00:08:56] [SPEAKER_01]: But it's very messy and typically doesn't scale well.

[00:09:00] [SPEAKER_01]: Usually the common theme we hear is that HashiCorp customers are very unhappy

[00:09:03] [SPEAKER_01]: if they're trying to use it for application secrets.

[00:09:05] [SPEAKER_01]: And that's where they find us at some point.

[00:09:07] [SPEAKER_02]: This is a very understandable problem.

[00:09:09] [SPEAKER_02]: So teams are trying to go really fast.

[00:09:11] [SPEAKER_02]: And especially as your organization gets bigger and maybe the engineers have some level of platform team

[00:09:18] [SPEAKER_02]: or support team that's helping push stuff to production,

[00:09:21] [SPEAKER_02]: there is a human and process element that causes delays in that speed.

[00:09:26] [SPEAKER_02]: And on the flip side, teams looking to go fast sometimes may end up having that ENV file on a local disk

[00:09:34] [SPEAKER_02]: or put it in the ticket management system,

[00:09:37] [SPEAKER_02]: which is basically like having your passwords.text file on your computer

[00:09:42] [SPEAKER_02]: where you stare your passwords

[00:09:43] [SPEAKER_02]: or you have an email going back and forth and your family would like,

[00:09:46] [SPEAKER_02]: here are the passwords.

[00:09:47] [SPEAKER_02]: So if someone hacks into your email or your device,

[00:09:50] [SPEAKER_02]: they're just going to look up the word passwords

[00:09:51] [SPEAKER_02]: and looking up ENV is kind of a similar way here.

[00:09:54] [SPEAKER_02]: And so it's hard because we're trying to go fast.

[00:09:58] [SPEAKER_02]: And then it's important because security is sort of an underlying foundation

[00:10:03] [SPEAKER_02]: that protects the business and protects the customers and everyone around.

[00:10:07] [SPEAKER_02]: So in your experience before starting Doppler and at Doppler,

[00:10:13] [SPEAKER_02]: why are teams so bad at keeping secrets, if you will?

[00:10:18] [SPEAKER_01]: The core thing is twofold.

[00:10:20] [SPEAKER_01]: One is that, again, developers want to move fast.

[00:10:24] [SPEAKER_01]: So they're going to do the most frictionless thing possible.

[00:10:26] [SPEAKER_01]: And then it just so happens that the most frictionless thing possible,

[00:10:29] [SPEAKER_01]: if you don't have a tool dedicated to being productive for developers.

[00:10:34] [SPEAKER_01]: So if you have either no tooling at all, which would be like ENV files,

[00:10:37] [SPEAKER_01]: or you have the basic set of tooling that exists out there,

[00:10:39] [SPEAKER_01]: like an AWS Secrets Manager, is they're not designed to be productive.

[00:10:43] [SPEAKER_01]: And so what ends up happening is you kind of go to the least secure

[00:10:47] [SPEAKER_01]: or developers naturally gravitate towards the least secure,

[00:10:50] [SPEAKER_01]: fastest or most productive thing possible.

[00:10:53] [SPEAKER_01]: And another way of kind of like looking at this is like these sets of secrets

[00:10:57] [SPEAKER_01]: or this ENV file is going to be a hodgepodge of secrets and actually non-secrets.

[00:11:00] [SPEAKER_01]: Like you may have a port variable in there that's not sensitive at all, right?

[00:11:05] [SPEAKER_01]: And or maybe some feature flags that aren't sensitive.

[00:11:08] [SPEAKER_01]: And so when developers are looking at this,

[00:11:09] [SPEAKER_01]: half the time they're touching non-sensitive things,

[00:11:12] [SPEAKER_01]: half the times they're touching very sensitive things like API keys,

[00:11:15] [SPEAKER_01]: but it's all in one file.

[00:11:16] [SPEAKER_01]: And so they all treat it the same.

[00:11:17] [SPEAKER_01]: And because the tools are like really high friction out there,

[00:11:22] [SPEAKER_01]: they gravitate towards the low friction, high productivity things

[00:11:24] [SPEAKER_01]: because they're like, oh, I'm just changing port variables and feature flags.

[00:11:27] [SPEAKER_01]: It's totally fine for me to move this fast

[00:11:29] [SPEAKER_01]: until the time that they realize that there's 40 or 50 other secrets in there

[00:11:33] [SPEAKER_01]: and one of them can basically end the company.

[00:11:36] [SPEAKER_01]: And so that's why I think it's really, really hard to manage secrets

[00:11:40] [SPEAKER_01]: without a dedicated tool that's designed for developers.

[00:11:43] [SPEAKER_01]: Because if it's not designed for developers,

[00:11:45] [SPEAKER_01]: it will be a high friction tool leading to very low productivity

[00:11:50] [SPEAKER_01]: and developers will find workarounds around that

[00:11:53] [SPEAKER_01]: because their job is not to think about the security of the company.

[00:11:57] [SPEAKER_01]: Their job is to ship product.

[00:11:59] [SPEAKER_01]: And the best way for them to do that is to stay in flow states,

[00:12:02] [SPEAKER_01]: i.e. no friction or reducing friction.

[00:12:05] [SPEAKER_01]: And that's kind of like one of the things that led to Doppler's inspiration,

[00:12:08] [SPEAKER_01]: which we can talk about a little bit later,

[00:12:10] [SPEAKER_01]: is design security tools that make developers more productive.

[00:12:14] [SPEAKER_01]: And the kind of like general theme we have in the company is make vegetables taste like candy.

[00:12:19] [SPEAKER_01]: Vegetables being the security, candy being the developer productivity.

[00:12:22] [SPEAKER_01]: We want developers to naturally use this tool and see it as a productivity tool.

[00:12:27] [SPEAKER_01]: And our target goal is get two hours a week back per developer.

[00:12:31] [SPEAKER_01]: So they are naturally trying to go to their boss or their teammates and say,

[00:12:34] [SPEAKER_01]: hey, you should use this tool.

[00:12:34] [SPEAKER_01]: I got more productive by it.

[00:12:36] [SPEAKER_01]: And then by doing that, they actually got more secure in the process.

[00:12:38] [SPEAKER_02]: There's a lot of risks you just talked about that come to being that are vulnerabilities that are created in the systems.

[00:12:47] [SPEAKER_02]: And what are some of the bigger implications or stories in the industry that you've come across that feel most relevant here where secrets was the compromise?

[00:12:59] [SPEAKER_01]: Yeah, great.

[00:13:00] [SPEAKER_01]: So I have two stories that I can talk about from the lens of public companies that have been deeply impacted by this problem.

[00:13:10] [SPEAKER_01]: Both companies everyone's heard about.

[00:13:12] [SPEAKER_01]: And then I can actually share another story as well that's personal to me about what happens once that data gets out.

[00:13:18] [SPEAKER_01]: So we can talk a little bit about how it happened to companies and then the effects of that breach on people like me.

[00:13:27] [SPEAKER_01]: And so the first one that really comes to mind is Twitter.

[00:13:29] [SPEAKER_01]: Twitter had their secrets in source code.

[00:13:32] [SPEAKER_01]: Again, going back to what is the least friction thing possible.

[00:13:35] [SPEAKER_01]: The least friction thing is put in a file and put in source control because I know when I push it up, every developer gets it.

[00:13:40] [SPEAKER_01]: And when I push it, the production infrastructure gets it.

[00:13:43] [SPEAKER_01]: So super simple.

[00:13:44] [SPEAKER_01]: Well, they did that.

[00:13:46] [SPEAKER_01]: And that code got leaked.

[00:13:47] [SPEAKER_01]: And when that code got leaked, all the secrets were in that code.

[00:13:51] [SPEAKER_01]: And so now they have all the passwords or the keys to the data kingdom.

[00:13:56] [SPEAKER_02]: Let me clarify a few things before you get a little bit further in, right?

[00:13:59] [SPEAKER_02]: So source control is using things like GitHub and GitLab, which makes it really easy for multiple software engineers to be working on code at the same time.

[00:14:08] [SPEAKER_02]: And it really maintains that repo of your library as well and maintains who worked on what at what time.

[00:14:16] [SPEAKER_02]: And it really takes away the friction of folks working on stuff simultaneously or in parallel.

[00:14:22] [SPEAKER_02]: This is, for the less technical folks, kind of like working on a Google Doc that has really good logs and stuff in the past,

[00:14:30] [SPEAKER_02]: which also means if you put something in the code or in this Google Doc that is highly sensitive, like a key or password,

[00:14:38] [SPEAKER_02]: and even if you delete it, you're able to go back and look at revisions of that file.

[00:14:46] [SPEAKER_02]: And so the history needs to be cleansed and managed as well if this mistake comes to being.

[00:14:51] [SPEAKER_02]: And then the second thing I think you mentioned is when the code at Twitter, when the code was made public or when the code –

[00:14:59] [SPEAKER_02]: how did the code get out in that case?

[00:15:02] [SPEAKER_02]: Was that open source code?

[00:15:03] [SPEAKER_01]: Yeah, so it wasn't open source code.

[00:15:05] [SPEAKER_01]: Actually, it was a leak.

[00:15:06] [SPEAKER_01]: So they got hacked.

[00:15:08] [SPEAKER_01]: And that internal code was leaked publicly.

[00:15:10] [SPEAKER_01]: And that leak then led to a cascading attack where the secrets became available.

[00:15:14] [SPEAKER_01]: So now they can leverage those secrets for further attacks.

[00:15:17] [SPEAKER_02]: Yep.

[00:15:18] [SPEAKER_02]: That's massively, massively a challenge.

[00:15:20] [SPEAKER_02]: And what was the other one that you had in mind?

[00:15:23] [SPEAKER_01]: Yeah, Toyota.

[00:15:24] [SPEAKER_01]: So very similar story.

[00:15:26] [SPEAKER_01]: They also had their secrets in their code.

[00:15:30] [SPEAKER_01]: And funny enough, they had their secrets in their code from 2017 all the way to 2022.

[00:15:38] [SPEAKER_01]: And they didn't realize that those secrets had been exposed for that five-year period.

[00:15:43] [SPEAKER_01]: So it was in the code.

[00:15:45] [SPEAKER_01]: It was exposed.

[00:15:46] [SPEAKER_01]: So a breach technically did happen.

[00:15:49] [SPEAKER_01]: And they didn't even realize it for five years.

[00:15:51] [SPEAKER_01]: Now that's like something really scary to me.

[00:15:53] [SPEAKER_01]: Like if you're a security professional of a company that large and you don't even realize you have a breach for five years because the secrets in the code.

[00:16:01] [SPEAKER_01]: And you're not really scanning for it or looking for it.

[00:16:04] [SPEAKER_01]: And you don't have these policies set up and tools set up so that developers can do the right thing.

[00:16:08] [SPEAKER_01]: That was a lot of time.

[00:16:10] [SPEAKER_01]: A lot of time for hackers to really execute multiple stages of attacks.

[00:16:17] [SPEAKER_01]: And to kind of put in perspective, if you put a secret out there in code today, like the general thinking is that if it gets to GitHub and it's in a repo that anyone on the internet can look at,

[00:16:29] [SPEAKER_01]: a bot will pick up a bot that is deployed by a hacking group or a hacker will find that secret in less than a couple seconds.

[00:16:38] [SPEAKER_01]: So you basically have like one to three seconds before a whole suite of hackers have found that secret and then they get to decide what they want to do with it.

[00:16:49] [SPEAKER_01]: So five years is a lot of time.

[00:16:51] [SPEAKER_01]: Like that is an enormous amount of time to wait to fix a problem or even discover the other problem to then fix.

[00:16:58] [SPEAKER_02]: And then what's the one that impacted you personally?

[00:17:03] [SPEAKER_01]: Yeah.

[00:17:04] [SPEAKER_01]: So I think one thing that most people forget about when they hear data breaches is like the natural sentiment I hear all the time is,

[00:17:11] [SPEAKER_01]: oh, it's really bad that that company had a data breach and they're going to get fined for it or they're going to have lost a customer reputation and such.

[00:17:18] [SPEAKER_01]: Maybe even get sued.

[00:17:19] [SPEAKER_01]: And I would strongly encourage you to think about that from the legal side,

[00:17:23] [SPEAKER_01]: because if you're not managing your secrets, it's negligence.

[00:17:26] [SPEAKER_01]: But at the end day, like the most fundamental thing is that all these companies that are storing data are storing customers' data.

[00:17:33] [SPEAKER_01]: Customers are trusting these companies to store their data securely.

[00:17:36] [SPEAKER_01]: And when you have a breach, it's really, I think, the people like me and you and everyone else, you know, that pay the price for this.

[00:17:44] [SPEAKER_01]: And so a good example of this is I was in – I live in Austin, Texas, and I just moved and I brought my mom out from California to come visit

[00:17:54] [SPEAKER_01]: and convince her that this is a great city to live in.

[00:17:56] [SPEAKER_01]: Maybe she can move someday.

[00:17:57] [SPEAKER_01]: And we were at barbecue, and I get this call.

[00:18:00] [SPEAKER_01]: And it's from someone claiming to be at the Texas Borders and Customs.

[00:18:05] [SPEAKER_01]: And again, I had just moved.

[00:18:07] [SPEAKER_01]: And they're like, hey, we have found this illegal package with drugs and money in it, and we are now formally investigating you.

[00:18:15] [SPEAKER_01]: Do you have time for a conversation?

[00:18:17] [SPEAKER_01]: And I'm like, this is a phone call where my life ends.

[00:18:20] [SPEAKER_01]: Like, I'm like, I've gone from highs to lows really, really quickly in a day.

[00:18:25] [SPEAKER_01]: And I'm freaking out at this point.

[00:18:27] [SPEAKER_01]: And they started rattling off information about me, like where I've lived, sites I've been to, just like so much information that like really makes you think this is a government.

[00:18:38] [SPEAKER_01]: Like the government sees all and like who else would have this information but the government.

[00:18:42] [SPEAKER_01]: So naturally, we get lawyers on the phone to help make sure I'm not making any mistakes.

[00:18:47] [SPEAKER_01]: And we're about an hour into this process of them just asking questions, getting more and more information out of me.

[00:18:53] [SPEAKER_01]: Because I think that they're building a case to prosecute me, and I have to answer honestly.

[00:18:57] [SPEAKER_01]: And they said some very scary line, like if you hang up this call, it's like a federal crime.

[00:19:00] [SPEAKER_01]: So like, I was like, we were pretty freaking scared at this point.

[00:19:05] [SPEAKER_01]: And at one point, they slipped up.

[00:19:07] [SPEAKER_01]: And we realized it was a scammer.

[00:19:10] [SPEAKER_01]: And that scammer got me, a CEO of a security company.

[00:19:15] [SPEAKER_01]: And they only got me because of all this other information that they had on me from previous data breaches.

[00:19:22] [SPEAKER_01]: And now they got even more information about me from this one that they can use for future attacks to look even more credible.

[00:19:30] [SPEAKER_01]: And like, imagine if I didn't have lawyers on the phone.

[00:19:33] [SPEAKER_01]: Imagine if we weren't like, if I didn't have the training that I do.

[00:19:36] [SPEAKER_01]: Like I actually just did cybersecurity training.

[00:19:37] [SPEAKER_01]: And myself, I have to take it.

[00:19:39] [SPEAKER_01]: Every employee at the company has to take that.

[00:19:40] [SPEAKER_01]: I just did a couple of days ago.

[00:19:42] [SPEAKER_01]: If I didn't have that training, maybe I wouldn't have detected it.

[00:19:45] [SPEAKER_01]: I would have given away a lot more.

[00:19:46] [SPEAKER_01]: Maybe they would have hacked my bank account because of that and such.

[00:19:50] [SPEAKER_01]: Like I can totally imagine normal people who have not had this training really being deeply affected by this.

[00:19:56] [SPEAKER_01]: And like they absolutely would have gotten enough information over time with that conversation to do something incredibly meaningfully damaging that I probably would not have been able to recover from, from that one phone call because of all this other data.

[00:20:08] [SPEAKER_01]: And so I think the important thing for people to remember here, both the people that are listening because they're in charge of protecting this data, but also just normal people who are not.

[00:20:17] [SPEAKER_01]: And they're like, every time I put this data in the system, like I traveled here, like they referenced where I traveled because they had some data from past trips from like Uber's had a data breach at one point.

[00:20:27] [SPEAKER_01]: And so they had a lot of information that felt very credible.

[00:20:31] [SPEAKER_01]: And so the thing to remember here is that this data, when it gets out, can then be used against you to get more data.

[00:20:36] [SPEAKER_01]: And then eventually they get enough data where they can take serious action against you, like compromising your social security number or draining your bank accounts or whatever it may be.

[00:20:46] [SPEAKER_01]: Like they'll be able to answer the forgot password set of questions there pretty well, or they'll be able to compromise your email and such.

[00:20:55] [SPEAKER_01]: And so the real people pay the price for these data breaches and real people are most of the time not equipped to fight these scammers.

[00:21:02] [SPEAKER_01]: And so once that data gets out, it gets really, really bad.

[00:21:05] [SPEAKER_02]: And on the team side, typically there's three or four steps that we're always looking to do, right?

[00:21:11] [SPEAKER_02]: In terms of trying to number one, avoid this in the first place.

[00:21:16] [SPEAKER_02]: So what is the proactive training and other steps in your systems and processes that you can put in place?

[00:21:22] [SPEAKER_02]: You mentioned cybersecurity training.

[00:21:24] [SPEAKER_02]: Monitoring is the second one, like that in the case of Toyota, if something does happen, hopefully it doesn't take you five years to find out that you always want to be the first to know when you've got a, whether it's a downtime on the availability side or a breach on the security side.

[00:21:42] [SPEAKER_02]: And then there's a respond where once you know about it, has your team run fire drills of some kind where it isn't the first time they're thinking about how do we respond to such a problem?

[00:21:50] [SPEAKER_02]: They've got some level of thought that's gotten into it that was developed when everyone was in calm mind versus now when we're in a heightened state of stress.

[00:22:01] [SPEAKER_02]: Now we're, you know, we don't want to be doing too much decision making.

[00:22:04] [SPEAKER_02]: We want to be doing, doing right.

[00:22:06] [SPEAKER_02]: And then the last one, of course, I think is once you've cleared this up, you got to go back and update the training or whatever we're doing proactively.

[00:22:12] [SPEAKER_02]: So that, um, that the training in itself continues to be, um, as up to date as hopefully the bad actors out there.

[00:22:26] [SPEAKER_02]: Fostering an engaged product organization and aligning them with the principles around lean, human centered design and agile will more than likely lead to successful business outcomes for your organization.

[00:22:39] [SPEAKER_02]: But getting started or getting unblocked can be hard.

[00:22:42] [SPEAKER_02]: This podcast is brought to you by the player coaches over at integral.

[00:22:46] [SPEAKER_02]: They help ambitious companies like you build amazing product teams and ship products in artificial intelligence, cloud, web, and mobile.

[00:22:57] [SPEAKER_02]: Listeners to the podcast can head on over to integral.io slash convergence and get a free product success lab.

[00:23:06] [SPEAKER_02]: During this session, the integral team will facilitate a problem solving exercise that gives you clarity and confidence to solve a product design or engineering problem.

[00:23:17] [SPEAKER_02]: That's integral.io slash convergence.

[00:23:22] [SPEAKER_02]: Now back to the show.

[00:23:27] [SPEAKER_01]: My career kind of started around Uber.

[00:23:30] [SPEAKER_01]: I was in college student and, uh, while in college, I kind of applied to Uber.

[00:23:35] [SPEAKER_01]: And, um, after many submissions, finally got an internship there.

[00:23:39] [SPEAKER_01]: And while I was there, uh, I started working on, um, pretty crazy projects, uh, directly for the VPs and directors.

[00:23:46] [SPEAKER_01]: And through that, I got to see a lot of, a lot of projects all at once.

[00:23:51] [SPEAKER_01]: Um, and I noticed from time to time that, uh, secrets was an issue.

[00:23:57] [SPEAKER_01]: And I was also at the same time working on my own side project.

[00:24:01] [SPEAKER_01]: And my side project was kind of like, uh, pushing a boulder up a hill.

[00:24:05] [SPEAKER_01]: Um, for every one foot, uh, that I'd move forward, I'd slip five feet back from exhaustion.

[00:24:10] [SPEAKER_01]: And, um, at some point I was kind of at my breaking point and I was like, I need to really look long and hard and see if this is the right thing to work on.

[00:24:17] [SPEAKER_01]: And after a lot of looking, I realized it wasn't.

[00:24:20] [SPEAKER_01]: And so I kind of inspired by, uh, Stuart Butterfield.

[00:24:23] [SPEAKER_01]: I think he's like the best in the industry at failing upwards, right?

[00:24:26] [SPEAKER_01]: Creates a video game born out of that was a flicker goes and tries to create another video game born out of that again was slack.

[00:24:32] [SPEAKER_01]: Um, and so I was like, what can I learn from, uh, this project that's, that's miserably failing?

[00:24:37] [SPEAKER_01]: Um, and out of that was a real deep pain for managing secrets.

[00:24:42] [SPEAKER_01]: And I noticed that I saw it from place to place to place that I went and every founder that I chatted with, every developer I chatted with, I'd say about 50% of the time they were having real issues too.

[00:24:53] [SPEAKER_01]: It wasn't just me.

[00:24:54] [SPEAKER_01]: Um, and that, that's what kind of got me motivated to, to start this.

[00:24:58] [SPEAKER_01]: I was looking, I was like, okay, what are all the problems in the space?

[00:25:01] [SPEAKER_01]: And there were a couple of big themes that I saw.

[00:25:03] [SPEAKER_01]: And one of them was that the tools were not developer centric.

[00:25:06] [SPEAKER_01]: They were not made for developers.

[00:25:08] [SPEAKER_01]: And I was a developer at the time and I was like, well, I know exactly what I would want.

[00:25:12] [SPEAKER_01]: And so why don't I try to build a version of this for me and see and get and build something that I would truly love.

[00:25:17] [SPEAKER_01]: And from there I can, uh, get feedback and see if others would like it too.

[00:25:21] [SPEAKER_02]: Uh, I'd love to hear kind of more, you've spent way more time in this space obsessing over it.

[00:25:27] [SPEAKER_02]: And I'd love to hear from, from your perspective, you know, at a high level, uh, what is something that executives maybe could be thinking about more?

[00:25:37] [SPEAKER_02]: And then we'll, I know we want to talk about some very specific playbooks.

[00:25:39] [SPEAKER_02]: We'll get into that next.

[00:25:41] [SPEAKER_01]: Yeah, sure.

[00:25:41] [SPEAKER_01]: Sure.

[00:25:42] [SPEAKER_01]: So I think the first thing before we kind of jump into anything else is really, I think there's like four questions that every executive should be able to.

[00:25:49] [SPEAKER_01]: And honestly, it's like executive all the way down to senior engineer.

[00:25:53] [SPEAKER_01]: Like, I think that's the band that, that should be, uh, really asking these four questions and it's the, sorry, I'm going to interrupt you just because I think this is going to be a really good short.

[00:26:02] [SPEAKER_02]: Can you start that from the beginning and go like, um, I think every executive to senior engineer should be thinking about this and then go into it and we'll get one clean shot of that whole thing.

[00:26:11] [SPEAKER_02]: All right.

[00:26:12] [SPEAKER_01]: Sure.

[00:27:01] [SPEAKER_01]: What's the answer?

[00:27:05] [SPEAKER_01]: them. Then the next one is where are, sorry, who has access to all those secrets and when did they

[00:27:12] [SPEAKER_01]: access them? So again, it comes back to do I know where all my secrets are? Can I control who has

[00:27:19] [SPEAKER_01]: access them? Do I have a log of every time they were accessed and when? And then the last one

[00:27:24] [SPEAKER_01]: you've already mentioned, which is a data breach. Can I stop a data breach once it happens?

[00:27:30] [SPEAKER_01]: So if you are figuring out what to do when a data breach happens, it's not going to be good

[00:27:36] [SPEAKER_01]: because now you have to think and do at the same time. When you get into situations like a data

[00:27:42] [SPEAKER_01]: breach, literally every second counts. Because if you have a hundred megabit or gigabit line,

[00:27:48] [SPEAKER_01]: that's a lot of data per second that can be leaving your system, right? So literally every second

[00:27:52] [SPEAKER_01]: counts. And at that point, you don't want to be thinking, you want to be doing. You want to have

[00:27:56] [SPEAKER_01]: a playbook and up and execute that playbook very, very quickly.

[00:28:01] [SPEAKER_02]: Got it. So the questions you want to know, number one, where are the secrets being housed and are they

[00:28:08] [SPEAKER_02]: all the secrets or are they most of the secrets? Number two, who has access to these secrets?

[00:28:14] [SPEAKER_02]: Number three, what's the logging? When are people accessing these secrets? And number four,

[00:28:21] [SPEAKER_02]: when you have a data breach, can you jump on it, make every second count and minimize the implication

[00:28:28] [SPEAKER_02]: of that breach? I got that right. Yep. That's perfect.

[00:28:33] [SPEAKER_02]: Well, and then, so maybe let's start with that one, right? Like say I'm a CIO and I'm in charge of

[00:28:42] [SPEAKER_02]: including security and we get a tip that there's been a data breach somewhere.

[00:28:48] [SPEAKER_02]: Um, what is the playbook that you wish I had already? And my team was team was trained on to

[00:28:56] [SPEAKER_01]: jump on this. Yeah. Okay. So let's reverse engineer all the things that you need to have

[00:29:01] [SPEAKER_01]: before this point happens so that you could do it so that you can handle it very quickly. And well,

[00:29:05] [SPEAKER_01]: the first thing is that you're going to want all your secrets centralized in one place. You're going

[00:29:09] [SPEAKER_01]: to want to have a really complete story about where all those secrets are, who has access to them,

[00:29:14] [SPEAKER_01]: meaning that you a know who has access, but also be, you can control that access. So you have policy

[00:29:20] [SPEAKER_01]: set up like this group of engineers do have access to production. These do not, um, for a set of

[00:29:26] [SPEAKER_01]: projects. Um, you have that secrets manager that's storing all your secrets natively integrated with

[00:29:33] [SPEAKER_01]: your infrastructure. This is really, really important because it comes down to speed, um, and zero

[00:29:39] [SPEAKER_01]: downtime when you, when you do, uh, the rotation, which I'll talk about in a second. And so what does

[00:29:43] [SPEAKER_01]: that mean by natively integrated with the infrastructure that comes in two parts? The

[00:29:47] [SPEAKER_01]: first is that you can rotate secrets automatically. So rotating secrets is like the fancy way of saying,

[00:29:53] [SPEAKER_01]: we're going to swap out the lock on the door. Um, and so you want to be able to click a button and

[00:29:57] [SPEAKER_01]: say this API key that just got leaked is dead and gone, does not work anymore. And this new one is fully

[00:30:02] [SPEAKER_01]: functional. And you want to be able to do that very quickly. And so if you have to do that manually,

[00:30:06] [SPEAKER_01]: which I'll talk about in a second, we had a customer, uh, that actually had a malicious actor

[00:30:10] [SPEAKER_01]: in their company and, uh, it took, uh, three security engineers six months to rotate 2000 secrets.

[00:30:18] [SPEAKER_01]: So if you are doing this manually, it will take months. You need to do this automatically now with

[00:30:23] [SPEAKER_01]: Doppler and the setup that I'm recommending, they can literally do this in seconds. Um, so you want to

[00:30:27] [SPEAKER_01]: have the ability to rotate all your critical services with a click of a button. The next part of deeply

[00:30:33] [SPEAKER_01]: integrated with your infrastructure is you want to be able to then push those secrets directly into, uh,

[00:30:38] [SPEAKER_01]: your cloud infrastructure. So like AWS, GCP, Azure, Kubernetes, wherever may be, uh, Vercel, Heroku and

[00:30:46] [SPEAKER_01]: such. You want to be able to get that new secret into that infrastructure and then restart those

[00:30:50] [SPEAKER_01]: applications immediately, uh, with those new secrets without any downtime. And the last thing is you're

[00:30:58] [SPEAKER_01]: going to want to log, uh, of, of every time a secret is read, every time a secret was written.

[00:31:04] [SPEAKER_01]: And also the change request approval flow. So like when new secrets get added, updated or changed,

[00:31:09] [SPEAKER_01]: you want to have someone that's kind of like a pull request that says, I approve this change,

[00:31:14] [SPEAKER_01]: roll it out. Um, so just like how you have the same flow for, for code, you want the same thing

[00:31:19] [SPEAKER_01]: for secrets. Uh, we call them change requests, get a, uh, our code calls and pull requests.

[00:31:24] [SPEAKER_01]: So once you have all those things in place, you can basically say, okay, there's been a breach. I

[00:31:28] [SPEAKER_01]: have the logs that show that either this person did the breach and it's the set of secrets. I'm

[00:31:34] [SPEAKER_01]: going to click the rotate button on all of them. And 30 seconds later, the old keys are gone and the

[00:31:39] [SPEAKER_01]: new keys are in your infrastructure. And that's the perfect setup instead of spending three to six

[00:31:45] [SPEAKER_02]: months trying to solve this. And you mentioned rotating keys there, and you spoke of it in the lens of,

[00:31:53] [SPEAKER_02]: doing it reactively or responding to a breach. And I also know of teams that do it proactively

[00:32:00] [SPEAKER_02]: and do it as a best practice. And then there's a little bit debate about whether we do it on a

[00:32:05] [SPEAKER_02]: time bound basis or an event bound basis. And I'm curious if you've got any opinions for the team,

[00:32:11] [SPEAKER_02]: uh, around if you're going to proactively rotate secrets, what are the times, best times to do it,

[00:32:18] [SPEAKER_01]: or what's the best practice there? Yeah, great question. So I think it's highly contextual to

[00:32:22] [SPEAKER_01]: what you're trying to do. So I'll paint two use cases for you. The first one is you have a flow

[00:32:28] [SPEAKER_01]: where, uh, a CICD workflow is deploying code into production and you need an AWS credential to

[00:32:34] [SPEAKER_01]: basically get that code from your workflow into, into production. And so with that,

[00:32:40] [SPEAKER_01]: it's actually should be event driven. And those are typically called, uh, dynamic secrets. So

[00:32:44] [SPEAKER_01]: there's secrets that are generated on the fly. So like generating an AWS credential on the fly,

[00:32:48] [SPEAKER_01]: that expires after the job is done. So usually you'd give it like a life of like 20 minutes or so,

[00:32:54] [SPEAKER_01]: uh, just enough for the job to complete. And then that, that, that credential is completely expired.

[00:32:58] [SPEAKER_01]: So those are con, uh, just in time credentials that have an expiration attached to it.

[00:33:03] [SPEAKER_01]: Uh, that'd be the event driven one that you were talking about. And that's a really good model for

[00:33:07] [SPEAKER_01]: those kinds of workflows where they spin up and they spin down very quickly. The other is a database.

[00:33:12] [SPEAKER_01]: So if you're using a database, your database is never not going to be talked to, right? You're

[00:33:15] [SPEAKER_01]: never going to turn off your website and say, Nope, we're not doing traffic right now. Um, and so you

[00:33:19] [SPEAKER_01]: will always need a database, uh, up, uh, connection up and running. And so with that, you want to do it

[00:33:24] [SPEAKER_01]: on a time, uh, cadence. And usually what we recommend is a two key system. And so there are two keys that

[00:33:30] [SPEAKER_01]: can talk to the database at any given time and you rotate through them. So you have one primary that's

[00:33:35] [SPEAKER_01]: being used and then that primary will become the secondary, um, at the fifth, let's just say you're

[00:33:41] [SPEAKER_01]: rotating every 30 days at the 15 day mark. And then at another 15 day mark, that secondary will

[00:33:45] [SPEAKER_01]: get swapped to a new credential and become the new primary. And so in that model, you never have

[00:33:50] [SPEAKER_01]: any downtime and you are rotating credentials every 15 days. So you know that if there's ever a breach

[00:33:55] [SPEAKER_01]: more than a month ago, you're totally fine. There is no chance of it harming you at all.

[00:34:01] [SPEAKER_02]: And, um, while we're getting into this and this is sort of in the back office of decision-making,

[00:34:06] [SPEAKER_02]: the stuff that no one wants to do, but you're kind of thankful for the folks who did it once,

[00:34:11] [SPEAKER_02]: uh, something bad happens. Right. The other thing is what's the correct number of secrets

[00:34:16] [SPEAKER_02]: or how do you think about that? Right. Cause I've definitely been guilty of this. I've been in an

[00:34:20] [SPEAKER_02]: early stage business where there'd be three engineers that kind of, and there was a licensing

[00:34:26] [SPEAKER_02]: thing there where it was more feasible for us to have just one account at this place,

[00:34:31] [SPEAKER_02]: but we need three people hitting it. And we ended up just using the same secret and, um,

[00:34:36] [SPEAKER_02]: that I know was wrong. And, uh, we generally avoided, but I feel like you're going to have

[00:34:42] [SPEAKER_02]: like a little bit more thought through best practice around, like, how do we allocate for

[00:34:45] [SPEAKER_01]: how many roles and what's too much and too few maybe. I may have a hot take here. And the hot take

[00:34:51] [SPEAKER_01]: here is that if you're using a secrets manager that has all the tools, all the things I just

[00:34:56] [SPEAKER_01]: talked about, it's centralized. You have a full audit story. You have the easy ability to rotate

[00:35:00] [SPEAKER_01]: secrets. Um, you have the ability to generate secrets on the fly and such all that stuff.

[00:35:06] [SPEAKER_01]: Then it actually really doesn't matter for the most parts. Um, and, uh, so like generally what we

[00:35:11] [SPEAKER_01]: do, even at Doppler's we'll have like one credential for production, one for staging and one for

[00:35:16] [SPEAKER_01]: development. So it's really actually environment contingent instead of developer contingent.

[00:35:20] [SPEAKER_01]: And the reason why that's, that works for us is because you have a complete audit story of

[00:35:24] [SPEAKER_01]: every time those secrets are read. And if anything ever does happen, we have a one click button that

[00:35:27] [SPEAKER_01]: we can do to rotate that secret for everyone and push that into new infrastructure. And so it becomes

[00:35:32] [SPEAKER_01]: not a problem for us anymore. Like we've basically shifted the burden of like managing and auditing

[00:35:38] [SPEAKER_01]: from that service provider all the way to, to that secrets manager. And then you had that guarantee

[00:35:43] [SPEAKER_01]: across every secret, even secrets that are like ports or other things where you will never,

[00:35:48] [SPEAKER_01]: you'd never get those kinds of guarantees anyways. Uh, and if you push it to that service provider,

[00:35:53] [SPEAKER_02]: going back to what we talked about with the source control and, and pushing secrets up,

[00:35:59] [SPEAKER_02]: I I'll admit this is something I did myself when I kind of switched jobs one time. And

[00:36:04] [SPEAKER_02]: the company that I used before wasn't quite as rigorous at source control. And then the new

[00:36:09] [SPEAKER_02]: company was like CICD and just very, very modern in terms of their check-in process. And I was running

[00:36:17] [SPEAKER_02]: some kind of spike and, um, I got a key. I was like, you know, I'll make the ENV file and I'll do

[00:36:24] [SPEAKER_02]: things the right way, but I just want to check and see if it works or not right now first.

[00:36:28] [SPEAKER_02]: And I put this, put it in there. And then I ended up pushing this code up, right? Versus just even

[00:36:35] [SPEAKER_02]: running it on my local device. And now everyone has access to it and it's sits in the revision history

[00:36:42] [SPEAKER_02]: of, of this code and that anyone can go back and look to, even though I deleted it. And so it's not

[00:36:48] [SPEAKER_02]: just a matter of deleting, obviously there's a little bit more of an involved process. So

[00:36:53] [SPEAKER_02]: same creating process or one of the folks at my startup comes and says, Hey, whoops,

[00:36:58] [SPEAKER_02]: I kind of made this mistake. My bad. I need your help. What do we do? What's the process to kind

[00:37:02] [SPEAKER_01]: of clean up that mistake? It really depends on when you catch it. In my opinion. Um, if you caught

[00:37:08] [SPEAKER_01]: it pretty quickly, what you can do, like it's one or two commits in the stack, it's like, and it's an

[00:37:13] [SPEAKER_01]: isolated commits. So I think that the key thing here is if you have one isolated commits, uh, for anyone

[00:37:19] [SPEAKER_01]: who doesn't know a commit is kind of like a bundle of code that gets all, all submitted together as

[00:37:23] [SPEAKER_01]: one big change. Um, it can be across multiple files too. So if you have one commit that has those sets

[00:37:29] [SPEAKER_01]: of secrets in it and there's no other commits that are really involving those secrets, it's actually

[00:37:34] [SPEAKER_01]: pretty simple. You can just basically go into the get history. You cherry pick that one commits and you

[00:37:39] [SPEAKER_01]: say, we're going to delete that one. And then you basically do a force push that updates master

[00:37:43] [SPEAKER_01]: in all the branches. Uh, still can be quite tricky if you have a lot of branches off of

[00:37:48] [SPEAKER_01]: our master slash main. So even then it's still a problem. If you have it in multiple commits across

[00:37:53] [SPEAKER_01]: multiple files, then it gets far more challenging. Um, and then you'll probably need to actually bring

[00:37:58] [SPEAKER_01]: in like a security team that goes through, uh, each file one by one and goes through like a pruning

[00:38:04] [SPEAKER_01]: process. And it's, it's quite difficult. I think it'd be a little bit tricky to like share over the,

[00:38:08] [SPEAKER_01]: over a webinar, but basically you have to go file by file, line by line and, and, and prune out that

[00:38:14] [SPEAKER_01]: history. And it's a, a very time consuming, expensive process. And there's a very high

[00:38:19] [SPEAKER_01]: chance of error too, because you may, you may miss somewhere in the history or even worse,

[00:38:24] [SPEAKER_01]: you may actually corrupt the history, um, in a way where the code just doesn't make sense now,

[00:38:28] [SPEAKER_02]: uh, where it did before. That sounds like a nightmare, especially depending on how far these

[00:38:33] [SPEAKER_02]: tentacles go. Um, the cleanup crew gets just more and more expensive and there's more of their time

[00:38:38] [SPEAKER_02]: that you're doing all the while. This is just something that's going on that is not adding any

[00:38:43] [SPEAKER_02]: value to your business or your customer, if anything, compromising value from all of it. Right. So it's

[00:38:49] [SPEAKER_02]: the distraction and a risk, the whole, the product, the business and everything. And so inevitably,

[00:38:56] [SPEAKER_02]: I'm sure everyone's wondering like, okay, what do we do to avoid this? Uh, you know, I can send out an

[00:39:02] [SPEAKER_02]: email tomorrow to my team saying, Hey, remember, don't put secrets in the code, but there's probably

[00:39:08] [SPEAKER_02]: something better than that in terms of systematizing this, um, this proactively. So what do you think

[00:39:14] [SPEAKER_01]: there? Yeah. And before we, before I jumped in, I just want to call out that, like, I'm not perfect

[00:39:19] [SPEAKER_01]: either. Like it was a quick, funny story. Um, one, our head of engineering reaches out, I think it was

[00:39:26] [SPEAKER_01]: a couple of months ago and he says, Hey, we've had this bug bounty, um, ticket. And it basically is

[00:39:30] [SPEAKER_01]: the, uh, the CEO of Doppler, uh, has a, uh, a public repo of a past company that has some secrets in the

[00:39:37] [SPEAKER_01]: source code. Uh, you should probably clean that up because it looks bad. Um, and so like even past

[00:39:42] [SPEAKER_01]: Brian years ago has made this exact same mistake that we're talking about now. We're all human.

[00:39:47] [SPEAKER_01]: We also have, uh, we all start from one starting point. We don't start perfect. It's just how do we

[00:39:52] [SPEAKER_01]: choose to move forward? So I'll just call it out that even I've made this mistake plenty of times

[00:39:55] [SPEAKER_01]: and even into the company, I didn't retroactively go clean up my past mistakes. Uh, so it is quite a

[00:40:00] [SPEAKER_01]: tricky thing. Um, okay. So I think the question was like, uh, how do you actually operationalize

[00:40:06] [SPEAKER_01]: this? What, uh, what does it look like, uh, uh, uh, to, to use a secrets manager? Is that the right

[00:40:12] [SPEAKER_02]: question if I remember correctly? I, it could be a secrets manager, but it, it could be outside of,

[00:40:18] [SPEAKER_02]: it could be beyond just the tool. I think you've got, if you're running a dev team, I know you're

[00:40:23] [SPEAKER_02]: scaling your team at Doppler there's new engineers coming in and they're smart folks that have

[00:40:27] [SPEAKER_02]: worked in very modern environments like this. And there's also folks who've maybe, you know,

[00:40:31] [SPEAKER_02]: worked on a very specific problem that is critical to you, but have done it in a more traditional

[00:40:35] [SPEAKER_02]: environment where they weren't going as fast or they didn't have sort of CICD and checking in code as often.

[00:40:43] [SPEAKER_02]: And, um, I think there requires sort of a set of tools for sure, but also some level of

[00:40:51] [SPEAKER_02]: enablement and onboarding to make that system complete. So I'm curious, kind of, I want to know

[00:40:56] [SPEAKER_02]: the, the, the, the tools for sure, but I also want to know beyond that, how you lead a team that you

[00:41:02] [SPEAKER_02]: feel very confident is going to have the, you know, a minimal risk exposure.

[00:41:08] [SPEAKER_01]: Sure. So I think the first thing you need to do is you need to have a secrets manager that is

[00:41:11] [SPEAKER_01]: developer, developer friendly. Um, and it has a set of functionality that deeply integrates within

[00:41:17] [SPEAKER_01]: your company. And so what do I mean by that? That's we've talked about the bare minimums of

[00:41:23] [SPEAKER_01]: a secrets manager, like all your secrets centralizes integrated infrastructure. But outside of that,

[00:41:27] [SPEAKER_01]: what you really want is you want, um, something that's directly connected to your like payroll

[00:41:31] [SPEAKER_01]: provider. So that when an engineer joins a company, they automatically get an account on the secrets

[00:41:35] [SPEAKER_01]: manager. So they don't have to go hunt down those secrets and then share it in insecure ways.

[00:41:39] [SPEAKER_01]: Uh, you want that, uh, uh, payroll or identity provider, whichever route you choose,

[00:41:44] [SPEAKER_01]: if you choose like rippling or author one login, uh, that to be connected to the secrets manager

[00:41:49] [SPEAKER_01]: and then have group setups. You can say, well, if they're in this team, then automatically they

[00:41:54] [SPEAKER_01]: get access to these sets of projects and the development environment. So they don't have to go

[00:41:57] [SPEAKER_01]: hunt down their secrets anymore. They're immediately available to them. The second thing you're going

[00:42:01] [SPEAKER_01]: to want is you're going to want, uh, these tools directly integrated in the developer, uh, workflow.

[00:42:06] [SPEAKER_01]: So for us, that typically means like we have a script that we run on every developer's laptop when

[00:42:11] [SPEAKER_01]: they join the company and this kind of like sets up their laptop for development. And part of that

[00:42:16] [SPEAKER_01]: is setting that auto configures Doppler. So it will automatically authenticate them in on their machine.

[00:42:21] [SPEAKER_01]: It'll set it up so that VS code, the Doppler VS code extension is automatically installed, um,

[00:42:27] [SPEAKER_01]: in VS code for them. It'll make sure the CLI, the command line interface is automatically available

[00:42:31] [SPEAKER_01]: for them in the terminal. It authenticates both of those, uh, to their Doppler accounts. Um,

[00:42:36] [SPEAKER_01]: and it checks in their, um, or, or activates their personal branches that they can use for development.

[00:42:41] [SPEAKER_01]: And so you'll want that. You'll want to, you'll basically want to have it so that when a, when

[00:42:46] [SPEAKER_01]: a developer joins day one, they don't have to hunt down their secrets. They automatically have it

[00:42:50] [SPEAKER_01]: available to them and all their tooling is set up when you're setting up their laptop and all the

[00:42:53] [SPEAKER_01]: other tooling Doppler, Doppler or another secrets manager set up with it. That is, I think one of the

[00:42:59] [SPEAKER_01]: best ways you can, uh, you can get the developer on a good footing day one, because if he's not,

[00:43:04] [SPEAKER_01]: if they're not on a good footing day one, then they're going to go back to their old paradigm,

[00:43:09] [SPEAKER_01]: which may be deeply insecure and not part of the process you have today. The next part is you're

[00:43:13] [SPEAKER_01]: going to want to leverage, um, you're going to want to set up policies in your secrets manager

[00:43:17] [SPEAKER_01]: so that all changes have to go through an approval process. You have to have another set of eyes,

[00:43:21] [SPEAKER_01]: um, to basically get things into a higher orders environments like staging and production.

[00:43:27] [SPEAKER_01]: And so you can, um, enable that in Doppler for example, by having, uh, a policy around this,

[00:43:34] [SPEAKER_01]: like, uh, we must have reviewers on production staging and that will then, um, uh, require

[00:43:40] [SPEAKER_01]: developers when they're changing secrets to use a change request. Um, and that is kind of like the

[00:43:46] [SPEAKER_01]: nuts and bolts from a, from a developer side is make sure that when they're developing, uh, they're

[00:43:50] [SPEAKER_01]: using a secrets manager that will handle all the things like making sure secrets never touch disk,

[00:43:55] [SPEAKER_01]: handles all the synchronization securely, handles all the permissioning and auditing securely.

[00:43:59] [SPEAKER_01]: It's integrated into the developer workflows like VS code and the terminal.

[00:44:02] [SPEAKER_01]: Um, and then the second part is when they need to get secrets into higher order environments,

[00:44:06] [SPEAKER_01]: they have to go through a change request or pull request flow like they do for code.

[00:44:09] [SPEAKER_01]: If you do those two things, uh, 99% of developers will just do the right things by default

[00:44:14] [SPEAKER_01]: and they will be in great shape and you have very low chance of a breach.

[00:44:20] [SPEAKER_02]: And for, we talked about monitoring, right? And, um, this is whether it's on the product

[00:44:28] [SPEAKER_02]: development side and you're looking at analytics or on the security availability infrastructure

[00:44:33] [SPEAKER_02]: side, and you're looking for problems. Um, there is always this challenge of

[00:44:40] [SPEAKER_02]: there's access to so much information now and there's infinite things we can go out there and

[00:44:46] [SPEAKER_02]: and monitor and log. And, you know, in this case, I imagine there are code scanners

[00:44:52] [SPEAKER_02]: that the big repos like GitHub offer to look for if secrets are checked in. There's ways that you can

[00:44:58] [SPEAKER_02]: scan the log. I'm sure you can, there's a lot of AI tools now jumping up on top of it. And it can,

[00:45:04] [SPEAKER_02]: there's a lot of availability of new resources. And then there's also a risk whenever there's an

[00:45:11] [SPEAKER_02]: availability of all these things that you're kind of going beyond maybe the 80-20 rule. Maybe the

[00:45:16] [SPEAKER_02]: 20 rule doesn't apply here because it's security. Maybe it does. And so I'm curious to know in your

[00:45:22] [SPEAKER_02]: case, you know, what are the maybe different tiers of precision to look for when you're setting up your

[00:45:29] [SPEAKER_02]: monitoring hooks in terms of like, Hey, here's the bare minimum. And this is maybe overkill stuff

[00:45:34] [SPEAKER_01]: that you see as well. Yeah. So I'd say there's like two basic hooks that you should have. Uh, the first

[00:45:40] [SPEAKER_01]: one is that your secrets manager is integrated into a logging platform like Sumo Logic, Datadog,

[00:45:46] [SPEAKER_01]: wherever. And those logs are analyzed for anomalies. So you set, you see a set of IP

[00:45:52] [SPEAKER_01]: ranges that don't make sense to you, or you see a set of actions like, Oh, we're seeing a high

[00:45:56] [SPEAKER_01]: number of downloads of secrets from a token or user that, uh, has never done this before. Something

[00:46:04] [SPEAKER_01]: like that. Um, and for what it's worth, I think a lot of those like logging platforms kind of do that

[00:46:08] [SPEAKER_01]: out of the box. You just send logs over, they kind of get a picture of what looks normal and then they

[00:46:12] [SPEAKER_01]: can show you an anomaly or anomalies. Um, but just having that connection there and using the

[00:46:18] [SPEAKER_01]: bare bones, uh, anomaly detection will actually do you a lot of good. The second is having secret

[00:46:23] [SPEAKER_01]: scanners like Trufflehog, get guardian, uh, that are actively scanning your code, uh, your

[00:46:29] [SPEAKER_01]: communication platforms like Slack and email, um, and, uh, any other tooling that it connects to.

[00:46:35] [SPEAKER_01]: And then what you really want to do is you want to kind of have a pro a policy that says,

[00:46:39] [SPEAKER_01]: if secrets are found, uh, that, uh, now you have to take action and you have to get those secrets

[00:46:45] [SPEAKER_01]: out of, uh, out of code or out of those, uh, systems and put them in a secrets manager.

[00:46:49] [SPEAKER_01]: And so kind of like if you had two charts, what you really want to see is on one chart, you want to

[00:46:53] [SPEAKER_01]: see the line of secrets and, uh, code Slack, email, Microsoft teams going down over time.

[00:47:00] [SPEAKER_01]: And you want to see another graph that shows that the number of secrets entering a secrets manager

[00:47:05] [SPEAKER_01]: and staying in the secrets manager are going up over time. And so if you can kind of have those two sets

[00:47:09] [SPEAKER_01]: of, uh, like scanning and reporting, you'll be in really good shape, I think.

[00:47:15] [SPEAKER_02]: All right. So we're in this situation here where we're trying to move really fast and any friction

[00:47:23] [SPEAKER_02]: that's added to your developers lives comes with a cost. Sometimes that cost is a worthwhile

[00:47:29] [SPEAKER_02]: investment. Sometimes you're just throwing the money away and there's inefficiency that is

[00:47:34] [SPEAKER_02]: compounding because not only your developer is slower, but if you have an environment that is not

[00:47:39] [SPEAKER_02]: fun or highly productive for developers, your best folks tend to leave to where they can focus on what

[00:47:44] [SPEAKER_02]: they're good at versus the friction, then, uh, you've got executives that may or may not be able to

[00:47:50] [SPEAKER_02]: answer all four of the questions that we talked about earlier in the episode really confidently.

[00:47:56] [SPEAKER_02]: And, um, you're trying to really build a business, build really great products and take care of your

[00:48:02] [SPEAKER_02]: customers really well. I would imagine that Doppler solves this in some capacity. So tell us about what

[00:48:10] [SPEAKER_01]: Doppler does for, for the dev teams out here. Yeah. I think what makes developer, uh, Doppler really

[00:48:17] [SPEAKER_01]: special is that do you want to, if I read you that real quick, that, uh, what makes Doppler really

[00:48:23] [SPEAKER_01]: special is that we challenge that notion that we have to be introducing friction into the system.

[00:48:28] [SPEAKER_01]: Uh, if anything, we want to grease the system. So moves a bit faster. Um, again, going back to making

[00:48:33] [SPEAKER_01]: vegetables taste like candy. And so that thing that makes Doppler special is that it's a productivity

[00:48:36] [SPEAKER_01]: tool for developers. They get two hours a week back per developer. Um, and so they naturally

[00:48:42] [SPEAKER_01]: want to use it because they're moving faster. And then from doing that, they get all the benefits and

[00:48:46] [SPEAKER_01]: that kind of starts to bubble up right now. They they're individually more secure. Their team is

[00:48:51] [SPEAKER_01]: more secure and thus the company is more secure and we can provide analytics and data to those

[00:48:56] [SPEAKER_01]: executives to be able to answer those four questions very confidently and also be able to set up

[00:49:00] [SPEAKER_01]: workflows and policies, um, that keep securing the company. And so what I look at, what makes Doppler great

[00:49:06] [SPEAKER_01]: is that developers truly love using this tool. And because of that, it affords us all the other

[00:49:12] [SPEAKER_01]: stuff we want to do, like securing the company, having those policies and workflows and orchestration

[00:49:17] [SPEAKER_01]: layers that actually work when you change a secret, another, uh, usually like a good flow that you'll

[00:49:22] [SPEAKER_01]: see is like a developer adds a secret locally. He rolls it out to his team. Uh, he can do that all

[00:49:27] [SPEAKER_01]: in one click directly from VS code. Um, and then they create a change request to get this into

[00:49:32] [SPEAKER_01]: production. So on the dev ops team clicks the merge, uh, approve and merge button, and that naturally

[00:49:38] [SPEAKER_01]: gets rolled out directly into your infrastructure without any downtime. That entire thing worked

[00:49:43] [SPEAKER_01]: end to end because we put a rigorous focus on making it natively integrated into all the workflows

[00:49:48] [SPEAKER_01]: developers have in the ways that they want it to work instead of forcing them to work into a system

[00:49:53] [SPEAKER_01]: that doesn't feel natural to them. Um, that is the real key to Doppler success. And because we,

[00:49:59] [SPEAKER_01]: we do that very, very well developers by default are doing the most secure thing, um, leading companies

[00:50:05] [SPEAKER_01]: to be far more successful and have a, and a higher security posture. That's what makes Doppler really,

[00:50:10] [SPEAKER_02]: really special. And then in terms of, if I was the executive struggling with the four questions,

[00:50:16] [SPEAKER_02]: um, how do you, like, I think I'm remembering them. You want to know where your secrets are,

[00:50:20] [SPEAKER_02]: who's got access to them? When have they been accessing them? And then how do I, how do I minimize the

[00:50:26] [SPEAKER_02]: data breach once it happens? Right? So how does, how does Doppler help with those four?

[00:50:31] [SPEAKER_01]: Great question. So the first thing we'll do when you get on board with Doppler and you'll work

[00:50:35] [SPEAKER_01]: with one of our solutions engineers, if you're a larger stage company, um, or have a more complex

[00:50:40] [SPEAKER_01]: environment is we'll, uh, help you pull all your secrets from all the different places, code, Slack,

[00:50:45] [SPEAKER_01]: email, wherever it may be into one central place. And you'll organize them by projects and environments.

[00:50:51] [SPEAKER_01]: From there, we'll get all your teams on board and we'll set up policies that say that this team has

[00:50:57] [SPEAKER_01]: access to these projects in these environments. These, uh, these teams have access to these projects

[00:51:03] [SPEAKER_01]: in these environments, and it will all be automated and connect your identity provider and your payroll

[00:51:08] [SPEAKER_01]: systems. From there, we'll give you a complete audit story of every time every action has taken place

[00:51:15] [SPEAKER_01]: in that, uh, organization from setting changes to secret changes to every read, uh, of a secret on,

[00:51:22] [SPEAKER_01]: uh, with, uh, the IP address and the, uh, the device that was used and the time. So you're going to get

[00:51:26] [SPEAKER_01]: a very complete audit picture and you'll be able to send that to your logging providers as well.

[00:51:31] [SPEAKER_01]: Um, and you'll also get a complete version history. So just like how you have a version history of code,

[00:51:35] [SPEAKER_01]: you'll get those secrets. And then the last piece is we'll deeply integrate that with your

[00:51:39] [SPEAKER_01]: infrastructure so that we need to rotate secrets on the fly. You can click a button or you can, uh, set it up to say,

[00:51:45] [SPEAKER_01]: hey, every 10 days or 15 days, rotate my secrets automatically for me or dynamically generate

[00:51:50] [SPEAKER_01]: them in CICD workflows. And then, um, when those change, when those secrets do get generated and

[00:51:55] [SPEAKER_01]: or unswapped, push them directly into my infrastructure and we'll give, uh, your developers the workflows

[00:52:01] [SPEAKER_01]: to do this all naturally. And, uh, so that they actually want to use these tools instead of run away

[00:52:06] [SPEAKER_01]: from them. And that's going to help you answer all four of those questions very confidently because

[00:52:11] [SPEAKER_01]: you know that, uh, they're going to be wanting to use these tools and running towards them.

[00:52:14] [SPEAKER_01]: Um, and embracing these tools along the way.

[00:52:17] [SPEAKER_02]: Uh, we've definitely had a number of guests on speaking directly to developer tools and building

[00:52:24] [SPEAKER_02]: really delightful developer experiences. I think that's maybe a function of my bias toward it in

[00:52:31] [SPEAKER_02]: part because I think it's a overlap of human centered design and software engineering. And as a software

[00:52:37] [SPEAKER_02]: engineer that felt like I was always doing stuff on the behind the curtain side of human centered design,

[00:52:42] [SPEAKER_02]: uh, it really feels very special to me on the stage where, um, the developers are the ones that are

[00:52:49] [SPEAKER_02]: being kind of crafted for and, and, and the sort of, let me try that again. The developers are the

[00:52:57] [SPEAKER_02]: folks that are the ones that are crafted for and the end user in this case. And also I think, uh, to some

[00:53:04] [SPEAKER_02]: extent I'm excited about it because I feel like there's been a lot of sins committed by myself in the

[00:53:09] [SPEAKER_02]: past where it's like, Hey, you know what? I've consumed a lot of APIs. I know how to write an API

[00:53:14] [SPEAKER_02]: and let me just put this out there. And I realized, Oh, I'm making the same mistake that a lot of

[00:53:18] [SPEAKER_02]: executives done before that said, Hey, I use mobile apps all the time. I'm going to force my way into

[00:53:22] [SPEAKER_02]: designing this mobile app instead of taking a little bit more of a systematic customer oriented research

[00:53:29] [SPEAKER_02]: approach. So I might be leading the witness here, but tell us about how you gain confidence that your

[00:53:36] [SPEAKER_02]: developer tools are super delightful and are integrating well into the workflows of what

[00:53:42] [SPEAKER_02]: would delight your developer customers. Yeah. Happy to. And by the way, I really appreciate

[00:53:46] [SPEAKER_01]: what you said about, um, that I, I, I strongly agree. I do think software is eating up the world,

[00:53:51] [SPEAKER_01]: which means that, uh, uh, developers are building the future, right? And so if you build great tools

[00:53:56] [SPEAKER_01]: for developers, they can continue to build great tools for all of us. Um, so how do we know that

[00:54:01] [SPEAKER_01]: what we're doing is, is really loved by developers? Well, I think the first part is where we take a very

[00:54:06] [SPEAKER_01]: customer centric approach. Um, we are extremely detail oriented, like our, our fundamental process

[00:54:10] [SPEAKER_01]: in Doppler is not to ship that many things a year, but the things that we ship have extremely high

[00:54:14] [SPEAKER_01]: impact. And so we do an immense amount of customer research across the board from individual developers,

[00:54:19] [SPEAKER_01]: all the way up to the very large enterprises that we work with on a day to day basis.

[00:54:23] [SPEAKER_01]: And from that, you can just see it in the metrics. Um, we can see that there's high usage across

[00:54:28] [SPEAKER_01]: the board in a lot of different areas that we care about how many time secrets are fetched. Are they

[00:54:32] [SPEAKER_01]: using tools like change requests to have approval flows? Are they using the VS code extension to,

[00:54:37] [SPEAKER_01]: to manage their secrets natively where they, uh, they manage their code? Um, and so it's a combination

[00:54:42] [SPEAKER_01]: for us of really listening very intently to feedback, um, and acting on it paired with, uh, looking at that

[00:54:49] [SPEAKER_02]: data that we're seeing from our analytics tool. And this is, um, you mentioned that you do a lot of

[00:54:55] [SPEAKER_02]: research. So I want to double click over there a little bit, a question that our folks at integral

[00:55:00] [SPEAKER_02]: would get asked a lot or a friction point for executives that we had to convince,

[00:55:04] [SPEAKER_02]: especially at larger companies to do this research was while getting the folks to research is going

[00:55:10] [SPEAKER_02]: to be really hard and it's going to slow us down and I don't know how to do it. And we would oftentimes

[00:55:15] [SPEAKER_02]: go in there and help work through making sure that the right folks are the ones who are researching

[00:55:20] [SPEAKER_02]: and take a lot of the pain off their hands, uh, both in terms of getting the work done,

[00:55:24] [SPEAKER_02]: but also helping strategize and not be inefficient in terms of how that work is deployed.

[00:55:29] [SPEAKER_02]: How's it done at Doppler and how do you maintain a discipline around it? Cause I,

[00:55:34] [SPEAKER_02]: you know, it's still a minority of the industry that's putting in that level of customer research,

[00:55:38] [SPEAKER_01]: especially on the dev tool side. The Seltzer side, we've identified a set of customers that are

[00:55:43] [SPEAKER_01]: like really active, like Doppler advocates, and we just have a database of them and we reach out to

[00:55:49] [SPEAKER_01]: them and we're like, Hey, we're going to be releasing this thing in a couple months or we're thinking

[00:55:53] [SPEAKER_01]: about solving this problem. Uh, does this resonate as a problem to you? Does it feel deep in a priority

[00:55:58] [SPEAKER_01]: problem? Or is this more like not a problem that you don't care about? And if it got solved or it

[00:56:01] [SPEAKER_01]: didn't, it wouldn't move the deal for you. And we have those sets of conversations and the enterprise

[00:56:05] [SPEAKER_01]: side, we have the exact same thing. Uh, we have a database of like really strong champions that are

[00:56:10] [SPEAKER_01]: actively engaged and invested in, in their success through Doppler. And so they are very willing to shout out to us.

[00:56:16] [SPEAKER_01]: And so anytime that we're thinking about spinning up a new, uh, feature or, uh, part of the product,

[00:56:23] [SPEAKER_01]: we first ask ourselves, what is the deep problem that we're solving and how can we deliver outsized

[00:56:28] [SPEAKER_01]: value against it? Um, and so we will reach out to that strong advocacy base. The other big area that

[00:56:35] [SPEAKER_01]: we look at is we have a lot of customers because they're advocates. They, they reach out to support a

[00:56:39] [SPEAKER_01]: lot and they're like, I want this feature. And so it is very well known to us, the features that are,

[00:56:44] [SPEAKER_01]: that are heavily demanded. They, they very much, uh, uh, surface themselves through support, um,

[00:56:50] [SPEAKER_01]: and solutions engineering. And then the other part that we started to do a little bit more recently

[00:56:55] [SPEAKER_01]: is, um, I would say more like, like blind feedback in a way of like, there's like, uh, systems like

[00:57:01] [SPEAKER_01]: user testing that can, you can say, I want to talk to people that work at these types of companies with

[00:57:06] [SPEAKER_01]: these roles and, uh, you'll pay them like 50 bucks to attend. And you get a talk with like four or five

[00:57:12] [SPEAKER_01]: of these people that you've never met before that are not customers that are coming in completely fresh.

[00:57:16] [SPEAKER_01]: And you can ask them a set of questions and be like, okay, what are your most pressing problems?

[00:57:19] [SPEAKER_01]: And they have no incentive to lie to you because they're never going to see you again. They're not

[00:57:22] [SPEAKER_01]: a customer to your product. Um, and so you get really high quality data out of that in my opinion.

[00:57:26] [SPEAKER_01]: Um, and so we've done a fair amount of that. Um, and then the last thing we do is we'll pop up in landing

[00:57:33] [SPEAKER_01]: pages and we'll be like, okay, uh, let's, let's put these landing pages up that we think may solve,

[00:57:38] [SPEAKER_01]: uh, some sets of problems. Uh, we're going to make sure that it tells a story well enough that it

[00:57:42] [SPEAKER_01]: would convince someone to put in their email address to talk to us. And then we're going to

[00:57:45] [SPEAKER_01]: run some ads against it on keywords or put on our, uh, on our website and where it's easily discoverable

[00:57:51] [SPEAKER_01]: and we'll see how much traffic we get into people sign up. And if it does, that's also another big

[00:57:54] [SPEAKER_01]: signal for us that, uh, that we have something there. So that's, we do it in a number of different

[00:57:59] [SPEAKER_01]: ways and we're kind of like taking all that data from different places to build a

[00:58:03] [SPEAKER_01]: complete picture of like, what should we be building? What should we be building and why?

[00:58:06] [SPEAKER_01]: And what's our unique differentiator there? Like we always try to say, what's the Doppler way of doing

[00:58:11] [SPEAKER_02]: this, man. I I'm sure there's going to be some audience members that are a little irritated with

[00:58:16] [SPEAKER_02]: me that we didn't get to this part of the discussion around how you build really great developer tools

[00:58:22] [SPEAKER_02]: or how you do your customer research. I love the various examples in there and, um, we'll definitely

[00:58:28] [SPEAKER_02]: put some shown outs on different best practices for those different approaches that Brian just talked

[00:58:33] [SPEAKER_02]: about. And Brian, so you're, you're the CEO of the business. Um, you are, uh, head of strategy in many

[00:58:42] [SPEAKER_02]: ways. You're also looking to go make sure that there's enough revenue and fundraising and fuel that

[00:58:49] [SPEAKER_02]: continues to allow the engines to go. Right. And I know for a fact that this kind of customer research

[00:58:56] [SPEAKER_02]: is extremely helpful for resource allocation of your people, your time, your budget, and ultimately

[00:59:04] [SPEAKER_02]: gets into product roadmapping. And I'm curious to know for someone who has been, uh, this disciplined

[00:59:10] [SPEAKER_02]: around that, how much of that flows into the, the business strategy and prioritization and roadmapping?

[00:59:17] [SPEAKER_01]: A way to take a step back to just kind of like share a little bit how the process works. We set board goals,

[00:59:22] [SPEAKER_01]: uh, and CEO goals for revenue targets. So this, uh, the board goal will be a little bit lower than the

[00:59:27] [SPEAKER_01]: CEO goal. The CEO goes a little bit more ambitious. Um, we said that once a year that gets broken down

[00:59:31] [SPEAKER_01]: into quarter and then from quarter gets broken down into month. Um, so at a monthly level, we're tracking

[00:59:36] [SPEAKER_01]: our revenue progression. And so we kind of know exactly what we need to do to hit our goals. Um, and so one,

[00:59:42] [SPEAKER_01]: one new practice that we've started doing the last, I would say six months that's worked decently well,

[00:59:47] [SPEAKER_01]: um, is for enterprise features. We actually put a dollar value associated with the enterprise feature.

[00:59:53] [SPEAKER_01]: And so the kind of going idea is like, well, like the common thing we'll hear is, oh, we love Doppler,

[00:59:58] [SPEAKER_01]: but if we had this one feature, it would help us expand to two XR, uh, the number of users on the,

[01:00:04] [SPEAKER_01]: uh, uh, that were, that are on Doppler. Right. And so we can go, well, we know how much we're charging

[01:00:08] [SPEAKER_01]: you proceed. We know how much revenue that will generate. If we unlock this feature that, that basically

[01:00:12] [SPEAKER_01]: allows you to double your team. And then we sum that up by all the customers that are requesting that

[01:00:16] [SPEAKER_01]: and get, uh, and so we can very easily say, well, for this enterprise feature, we know that this one

[01:00:20] [SPEAKER_01]: will literally drive more revenue than this one will like to the number. We know we have a very

[01:00:25] [SPEAKER_01]: strong estimation of what will move the needle there. And you don't always want to be driven

[01:00:29] [SPEAKER_01]: by that. Like that alone is, is a recipe for disaster. You have to take that in with the rest

[01:00:33] [SPEAKER_01]: of the strat, the product strategy of like, what do you want to accomplish long-term? What future bets

[01:00:37] [SPEAKER_01]: are you willing to make that may not have a dollar association with it? What are things that are

[01:00:41] [SPEAKER_01]: like really critical table stakes for you that differentiate your product, all that. But like one,

[01:00:45] [SPEAKER_01]: one area we've looked at least from a revenue side that that's helped us is we can literally put

[01:00:49] [SPEAKER_01]: a dollar on a value on a lot of the features we build and we can say, well, we know if we ship

[01:00:53] [SPEAKER_01]: this, it'll drive this, this much in enterprise revenue. And we get a looser approximation on

[01:00:58] [SPEAKER_01]: self-serve as well, because, um, like self-serve is about 60% or sorry, 40% of our revenue.

[01:01:03] [SPEAKER_01]: So it's, it's, it's, uh, it's not insignificant. So, uh, we, we generally factor that into our estimations

[01:01:09] [SPEAKER_01]: too, of, uh, feature to revenue impact. Um, but that's been a really critical tool for us.

[01:01:15] [SPEAKER_01]: Uh, it's helped us dodge a lot of bad decisions where like there was some sexy thing that we

[01:01:20] [SPEAKER_01]: thought would be really cool to build, but turns out a enterprise customers didn't really want it

[01:01:25] [SPEAKER_01]: that much and B something far less sexy and simpler. They actually wanted really, really bad.

[01:01:31] [SPEAKER_01]: Um, and, and so much so that there was like hundreds of thousands of dollars that were on the line,

[01:01:36] [SPEAKER_01]: maybe per customer for that feature. Um, and so that, that's been a really good way of helping us

[01:01:41] [SPEAKER_01]: like guide, um, our decision-making and all props to our, our PM James at that, uh, at Doppler, uh,

[01:01:48] [SPEAKER_01]: who joined pretty recently. Uh, he helped institute this, uh, from GitLab and it's been very effective.

[01:01:53] [SPEAKER_01]: Um, so props to him, uh, for implementing it.

[01:01:56] [SPEAKER_02]: That's awesome. You've given me something to chase after around, uh, finding out more of the,

[01:02:01] [SPEAKER_02]: the genesis of this at GitLab because it's music to my ears, what you're hearing.

[01:02:04] [SPEAKER_02]: I happen to span a lot of roles between engineering and product and solutions,

[01:02:10] [SPEAKER_02]: engineering and sales even where I carried a bag and a quota myself. And so I'm not afraid

[01:02:14] [SPEAKER_02]: of the revenue part. And I find myself a minority in the product community where a lot of the folks

[01:02:20] [SPEAKER_02]: like myself have grown up in engineering, have a shyness away from a revenue commit. And,

[01:02:25] [SPEAKER_02]: and then sitting in the product hat too, there's nothing more validating than when

[01:02:29] [SPEAKER_02]: a credible salesperson puts a revenue number or a revenue commit, as we called it in some past

[01:02:34] [SPEAKER_02]: lives that I had associated with a feature request and whether it went into the roadmap or we did it

[01:02:39] [SPEAKER_02]: as non-recurring engineering. If they were able to get us some level, a letter of intent, or sometimes

[01:02:43] [SPEAKER_02]: I just do a ride along to a sales call with a C suite at a fortune 500 company. Those folks are not

[01:02:50] [SPEAKER_02]: going to BS you, um, at that level. They're looking to build long-term partnerships as well with

[01:02:54] [SPEAKER_02]: companies like you. And when they say that, Hey, yeah, we will sign this. It's pretty easy for them

[01:02:59] [SPEAKER_02]: to get that letter of intent done when they're at that high level, even if legal processes can be hard.

[01:03:04] [SPEAKER_02]: And from a product perspective, it makes a night and day difference of how you're deploying like a

[01:03:09] [SPEAKER_02]: six, seven figure investment in terms of one feature or another feature. And depending on the

[01:03:14] [SPEAKER_02]: stage of your company, it can make or break a lot of things. And so creating that fluid flow between what

[01:03:19] [SPEAKER_02]: customers want, understanding they need, quantifying that to some level of problem statement or dollar

[01:03:24] [SPEAKER_02]: value, and then incorporating that into the product roadmap is like, and it sounds so linear and easy

[01:03:31] [SPEAKER_02]: when I say it, but with discipline, it does over time start to feel linear, but there's a J curve

[01:03:37] [SPEAKER_02]: or valley of death where I think a lot of folks kind of lose faith. So it's awesome to see that, uh,

[01:03:41] [SPEAKER_02]: you've got a team and a PM and a great kind of customer advisory board that you can tap into for a lot

[01:03:48] [SPEAKER_01]: of this. I want to call out one thing that I think goes like, while it's very amazing,

[01:03:52] [SPEAKER_01]: there is one set of risks I'll call out for anyone who's listening. Um, which is, it's very easy to get

[01:03:57] [SPEAKER_01]: like narrowly focused on that number. Once you start looking at it, because then what you do is

[01:04:01] [SPEAKER_01]: you start sorting features by which drives the most revenue. And like, it's kind of like shifting

[01:04:07] [SPEAKER_01]: from like a startup, which is a long-term, uh, like you make long-term big bets as a startup.

[01:04:12] [SPEAKER_01]: Then that's how you fuel these next like big, uh, big like jumps of growth. And I feel like once you go

[01:04:17] [SPEAKER_01]: like public as a company, you're, uh, held to the shareholder value in that moment. So you start

[01:04:22] [SPEAKER_01]: making short-term bets that move the needle on, on, on the share price. And I feel like doing,

[01:04:27] [SPEAKER_01]: uh, this with your features can actually create that same psychological effect where you're like,

[01:04:32] [SPEAKER_01]: okay, I know that this can drive revenue immediately in the short term. So, and I only

[01:04:36] [SPEAKER_01]: have so much engineering resources, so why not go do that? That'll add a couple hundred thousand,

[01:04:40] [SPEAKER_01]: that'll add a million, that will add a million. Um, and you may do that for so long and concentrate that

[01:04:46] [SPEAKER_01]: you're, uh, your resources so aggressively against that strategy that you may actually miss this

[01:04:50] [SPEAKER_01]: bigger picture. That's a much riskier bet for the company, but may pay off big time and like

[01:04:56] [SPEAKER_01]: completely transform the success curve of the company. And so I think it's really important

[01:05:01] [SPEAKER_01]: to know that this is just one property of, or one, one tool in the toolkit for you to use. It is not

[01:05:08] [SPEAKER_01]: the tool to use. It's just one of many. And if you too narrowly focus, you'll, uh, I really think

[01:05:14] [SPEAKER_01]: you'll hurt yourself in the, in the longterm because you'll just iterate so aggressively in

[01:05:18] [SPEAKER_02]: the short term. I love that you mentioned that I've been, I wanted to double click on it when

[01:05:23] [SPEAKER_02]: you mentioned it earlier as well, where revenue alone does not replace strategy. And I think, uh,

[01:05:31] [SPEAKER_02]: the coaching I was given was something along the lines of revenue is vanity. Profitability is sanity.

[01:05:37] [SPEAKER_02]: And you, and then there's gotta be something that rhymes that talks about strategy there,

[01:05:41] [SPEAKER_02]: where it's a good clarification where the revenue goals for something that maybe has the same,

[01:05:47] [SPEAKER_02]: maybe a two by two, like you talked about before, where you prioritize based on the longterm for your

[01:05:51] [SPEAKER_02]: company, but also the short term revenue. And this revenue method really comes into play only when

[01:05:57] [SPEAKER_02]: you're comparing similar grade things on the strategy where they're equally, maybe similarly

[01:06:04] [SPEAKER_02]: important to your business in the longterm, but you have to decide which one you work on first.

[01:06:09] [SPEAKER_02]: And then that's where this revenue works in the short term. And in terms of deciding which,

[01:06:15] [SPEAKER_02]: whether it's a or B that you do first with that limited amount of budget. Right. And, um,

[01:06:20] [SPEAKER_02]: another way that I have really liked thinking about this is, is in the form of debt, right?

[01:06:25] [SPEAKER_02]: There's different forms of debt we can take on to get something done today that we maybe don't

[01:06:29] [SPEAKER_02]: have the cash for. We can put it on our credit cards. We can buy a house with a mortgage. We can, uh,

[01:06:34] [SPEAKER_02]: get investors involved to, to help and share some equity with them. And it's a similar kind of

[01:06:41] [SPEAKER_02]: approach that can be taken here where sometimes you just need to get a press release done or you

[01:06:46] [SPEAKER_02]: need to sign this big customer. And so you're going to go build this feature that strays from what your

[01:06:50] [SPEAKER_02]: longterm vision is. And if you don't have a strategy and longterm vision, then you're going to get dragged

[01:06:57] [SPEAKER_02]: in multiple directions by customers that don't necessarily have a strategy on their own that are

[01:07:02] [SPEAKER_02]: similar. But if you have your own, then you can think about it as, you know what, I just need to

[01:07:07] [SPEAKER_02]: pay the bills this month, but I know that next month this is going to get paid off. So I'm going to pay

[01:07:12] [SPEAKER_02]: the high interest on this credit card, but I know that it's for a short amount of time. And once we get

[01:07:17] [SPEAKER_02]: back on track, we are, we, we don't have credit card debt on our balance sheet and, um, it's hard to

[01:07:24] [SPEAKER_02]: quantify. And I think that's also a risk. Most of us have decent instincts around which ones the

[01:07:30] [SPEAKER_02]: credit card debt are and which ones feel more like mortgages or equity investments. And using

[01:07:37] [SPEAKER_02]: that gut in itself is a good way to sort of have a conversation with your team or gut check yourself

[01:07:42] [SPEAKER_02]: around, Hey, are we, are we just chasing after short-term revenue and won't end up with something

[01:07:48] [SPEAKER_01]: of value in the longterm? Yeah, I agree strongly. I I've always thought about it, like for anyone who's

[01:07:53] [SPEAKER_01]: a visual learner, for some people, this will make sense. Some of them won't, but if you know what a

[01:07:57] [SPEAKER_01]: step, uh, a step graph looks like, where you have like a bunch of boxes that kind of like go up in

[01:08:01] [SPEAKER_01]: steps, those steps are the strategy of long-term planning and long-term bets where they pay off,

[01:08:07] [SPEAKER_01]: they create a big jump for you and like opportunity. Um, and then all, and then what you can do is with

[01:08:13] [SPEAKER_01]: those iterations, you can kind of fill in the curve to the next step, right? Like you can, you can,

[01:08:17] [SPEAKER_01]: and that is like the, yeah. And like, that is that little step curve or, um, or filling in the step

[01:08:24] [SPEAKER_01]: curve that is, uh, looking at that revenue and going, well, this feature is more valuable than

[01:08:29] [SPEAKER_01]: that feature. And I can add a hundred K there. It's not going to get you to the next, it's not

[01:08:33] [SPEAKER_01]: going to take you from 1 million to 10 million or 10 million to a hundred million, but it will take you

[01:08:37] [SPEAKER_02]: from one to two or one to three. A hundred percent. Hey, um, one of the things that I really love

[01:08:44] [SPEAKER_02]: asking all my guests is, uh, what's a product or service that you have just really loved. And I'll

[01:08:52] [SPEAKER_02]: take an answer from work or at home. Yeah. Perplexity. Love it. It's like,

[01:09:00] [SPEAKER_01]: it's chat GPT plus Google search. And I find it pretty much any question I can, I can think of

[01:09:07] [SPEAKER_01]: asking. It has a very sound, like reasonable answer to, um, it gets better and better every day.

[01:09:13] [SPEAKER_01]: And like, like they just added the ability to like add checkout to it. So like, if you,

[01:09:17] [SPEAKER_01]: like you could search, like, um, I dunno, what is a great t-shirt brand for men of this size that

[01:09:25] [SPEAKER_01]: work out daily? And then I'll show you a bunch of options and they can literally just buy it right

[01:09:28] [SPEAKER_01]: on, right within the, uh, in the app. And so I, I've used it to almost entirely replace Google for me.

[01:09:35] [SPEAKER_01]: Um, and I think of it as like with Google, you ask a question and you have to, they can get a bunch of

[01:09:40] [SPEAKER_01]: possible answers from a bunch of different sites. You have to go find that answer versus perplexity

[01:09:45] [SPEAKER_01]: will kind of do that second order step. Well, they'll look at every website that comes up from

[01:09:49] [SPEAKER_01]: that search and then they will build knowledge based off that. So they'll extract the learnings

[01:09:54] [SPEAKER_01]: from all those different websites, cross-reference information and give you a complete picture answer

[01:09:59] [SPEAKER_01]: in like five seconds. Um, and so that's really powerful.

[01:10:03] [SPEAKER_02]: This is a little personal. So did you go from Google to chat GPT to perplexity or did you go

[01:10:08] [SPEAKER_01]: straight? I went straight. I was pretty disappointed with chat GPT for the most part,

[01:10:13] [SPEAKER_01]: to be honest. I know I'm probably one of the very few that is, but outside of some things like tell

[01:10:18] [SPEAKER_01]: me a story every once in a while or translate something for me, I haven't found a ton of like

[01:10:23] [SPEAKER_01]: really powerful use cases for chat GPT. Um, I always felt like it was a little bit limited because

[01:10:28] [SPEAKER_01]: it didn't have access to the internet and it couldn't do like that next order step for me of like

[01:10:31] [SPEAKER_01]: generating knowledge from my search. Um, and so I just, I tried chat GPT for a tiny bit and I was

[01:10:38] [SPEAKER_01]: like, you know, this is cool, but maybe not for me. Um, and then when I found perplexity from a friend,

[01:10:43] [SPEAKER_01]: I was like, Oh, I'm using this every day. And I use perplexity multiple times per day.

[01:10:48] [SPEAKER_02]: It's, uh, still something where when I do a talk about LLMs or how to incorporate it into systems

[01:10:54] [SPEAKER_02]: and processes is very typical of the moderator will do a raise of hands of how many people use it daily,

[01:11:00] [SPEAKER_02]: weekly, weekly, monthly, and it's still staggering how folks still tend to kind of go with the search

[01:11:06] [SPEAKER_02]: engine route when there are much more efficient tools out there. And the advice usually is just

[01:11:12] [SPEAKER_02]: changing your homepage to, or your native search from the address bar from one to the other.

[01:11:17] [SPEAKER_02]: That's what I did.

[01:11:18] [SPEAKER_02]: And then it just kind of, you create a friction to go to Google and, um, and then once you learn

[01:11:25] [SPEAKER_02]: the benefits and you overcome the personal learning adoption friction, then you're at a place where

[01:11:31] [SPEAKER_02]: you are, where you use it multiple times daily. Awesome. And, um, Brian, how do people get ahold of

[01:11:36] [SPEAKER_02]: you? And if they're interested in what they heard about Doppler, how do they reach out to your team?

[01:11:41] [SPEAKER_01]: Yeah. So you can reach out to me personally at Brian, B R I N at doppler.com, D O P P L E R.com.

[01:11:48] [SPEAKER_01]: Uh, or you go to doppler.com and we have a contact us page, uh, there we can reach out to our team.

[01:11:53] [SPEAKER_01]: Also, if you become a customer, then you can reach out to us on, um, our, through our support systems.

[01:11:58] [SPEAKER_01]: Um, so there's a lot of ways to get in contact with us, um, just for whatever your use case may be,

[01:12:04] [SPEAKER_01]: uh, but very happy to chat, uh, or give advice or whatever it may be.

[01:12:08] [SPEAKER_02]: Awesome, man. Well, Hey, thank you so much for all the education around secrets, secret

[01:12:13] [SPEAKER_02]: management, and the proactive steps and playbooks that we can put in play as well as a window into

[01:12:20] [SPEAKER_02]: your product design management and prioritization process. I think the audience is going to really

[01:12:25] [SPEAKER_02]: love it. I wish we had a lot more time and maybe we can find some more time when you're at your next

[01:12:31] [SPEAKER_02]: stage of growth and hear some more stories about how you're doing. Thanks a lot for making the time for

[01:12:35] [SPEAKER_01]: us. Would love to thank you so much for the time. This is a absolute pleasure.

[01:12:44] [SPEAKER_02]: I enjoyed that conversation with Brian a lot. It's something that I have had to manage on teams.

[01:12:51] [SPEAKER_02]: And I believe that secrets management can be a really powerful way to not just security team,

[01:12:58] [SPEAKER_02]: but also speed up and make their development process way more efficient so that you are spending

[01:13:03] [SPEAKER_02]: way less time with things like chasing after secrets and making sure they're managed well.

[01:13:09] [SPEAKER_02]: And more importantly, you can get value add features to your customers and for your business

[01:13:14] [SPEAKER_02]: from code to deployment as quickly as possible. One of the things that I really enjoyed and agree

[01:13:21] [SPEAKER_02]: with Brian on is about his approach to proactive training and enablement, as well as having playbooks

[01:13:29] [SPEAKER_02]: in place. And I think these four steps that I learned along the way at past lives has been extremely

[01:13:35] [SPEAKER_02]: helpful for me to think about infrastructure related resiliency and reliability, whether it's around

[01:13:42] [SPEAKER_02]: the availability of your infrastructure or securing the data that sits in it. And the four steps are

[01:13:48] [SPEAKER_02]: number one, you want to train folks and be proactive so that the risk is minimized. If the risk were to come

[01:13:57] [SPEAKER_02]: true, you want to be the first to find out. You certainly don't want any compliance teams externally

[01:14:03] [SPEAKER_02]: or let alone your customer to know before you do. When you find out, you want to make sure that you

[01:14:10] [SPEAKER_02]: have a playbook in place so that the right person has the right procedure in order to mitigate,

[01:14:15] [SPEAKER_02]: reduce the risk as well as rectify the issue. And lastly, you want to create a feedback loop wherein

[01:14:22] [SPEAKER_02]: in after the issue occurs, you do a retrospective or a postmortem and that that's going back into

[01:14:29] [SPEAKER_02]: the training as well as any other automation that you have in place. So when it comes to training,

[01:14:34] [SPEAKER_02]: I think Brian talked about really great ways in which you can help your team understand the trade-offs

[01:14:41] [SPEAKER_02]: between the risks introduced when you're under managing your secrets, as well as some of the

[01:14:48] [SPEAKER_02]: efficiency losses associated if there's over management or micromanagement of the secrets.

[01:14:54] [SPEAKER_02]: Secondly, around observability, monitoring and alerting, this is a place where you want to have

[01:15:01] [SPEAKER_02]: the 80-20 rule in place, I think. And when it comes to something as sensitive as security,

[01:15:05] [SPEAKER_02]: that's hard to think about. But the fact of the matter is if you have too many alerts

[01:15:10] [SPEAKER_02]: or your dashboards, your single plane of glass, if you will, have too many gauges,

[01:15:14] [SPEAKER_02]: it loses the focus on the really high risk, highly important things on that gauge or on the alerting.

[01:15:23] [SPEAKER_02]: So again, similar to your product roadmap, I highly encourage you to think about where you feel the

[01:15:28] [SPEAKER_02]: biggest risks, the biggest assumptions and prioritize your dashboards, your monitoring and your alerting

[01:15:34] [SPEAKER_02]: for the really highly critical ones. And take the leap of faith around being okay with maybe not having

[01:15:42] [SPEAKER_02]: an immediate alert and having maybe a digest type notification go out the next morning if it's

[01:15:48] [SPEAKER_02]: something that was more on the green to yellow or yellow severity and save that alerting for the

[01:15:55] [SPEAKER_02]: pagers and everyone being woken up for the orange and red alert things that need to be responded to right

[01:16:01] [SPEAKER_02]: away. Playbooks, I mean fire drills are run in different ways I think where teams get to exercise and feel

[01:16:08] [SPEAKER_02]: comfortable with their playbooks on what to do when things are not going so great. One of the things I

[01:16:15] [SPEAKER_02]: really like about it is that we get to think about what we might do when we are of sound mind versus

[01:16:21] [SPEAKER_02]: when the fire alarm is going off. And secondly, we get to do this drill so we get to familiarize ourselves

[01:16:26] [SPEAKER_02]: with the work that comes out of the playbooks and the good judgment we exercised so that our teams can

[01:16:32] [SPEAKER_02]: respond to the issues really fast. It also I think really helps if you're able to have a fire drill

[01:16:41] [SPEAKER_02]: around the alerting systems so that whether it's you're using pager duty or some other similar system

[01:16:47] [SPEAKER_02]: to alert the various folks that are on call, that that is also consistently being tested so that

[01:16:53] [SPEAKER_02]: a risk introduced in your alerting system isn't the reason that you have a delay in responding to a problem.

[01:17:00] [SPEAKER_02]: Lastly, of course, I always encourage doing whether it's a five wise root cause analysis,

[01:17:07] [SPEAKER_02]: a no blame postmortem or a retrospective that that information is circulated across the team

[01:17:14] [SPEAKER_02]: as well as incorporated into training and onboarding materials for new teams. And you look at any

[01:17:20] [SPEAKER_02]: architectural or system changes that come out from that that you need to put into play fairly soon so

[01:17:26] [SPEAKER_02]: that the chances of the implications of something like a data breach are minimized as you get better

[01:17:33] [SPEAKER_02]: at this. Thank you again for listening. I hope you enjoyed this episode with Brian. We will see you on

[01:17:40] [SPEAKER_02]: the next episode that comes out on Tuesday. Thank you for joining me on the Convergence podcast today.

[01:17:55] [SPEAKER_02]: Subscribe to the Convergence podcast on Apple podcast, Spotify, YouTube, or wherever you get your content.

[01:18:03] [SPEAKER_02]: If you're listening and found this helpful, please give us a five star review. And if you're watching

[01:18:08] [SPEAKER_02]: on YouTube, hit that like button and tell me what you think about what you heard today.