EPISODE 1595

[EPISODE]

[0:00:00] ANNOUNCER: Josh Prismon is a veteran software architect having worked at FICO for 17 years before shifting to Index Exchange in 2022.

In this episode, Josh joins the podcast to speak with host, Lee Atchison, who has deep experience in software architecture from his time at Amazon, New Relic and other companies. Josh and Lee discussed different paradigms for thinking about software architecture, including the importance of storytelling to build a shared vision for architecture across a team. They also talk about the tension between emergent and intentional design, how to allow architecture to evolve dynamically without sacrificing structure, and more.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His bestselling book, Architecting for Scale, is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business, produced for people looking to build and grow their digital business. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

[INTERVIEW]

[0:01:27] LA: Josh, thank you for joining me in Software Engineering Daily.

[0:01:30] JP: It's great to be back.

[0:01:31] LA: So, I thought it might be useful to talk about your approach to software architecture. So, why don't we start out with this. What's the first thing you think of when you think about a high-performant software architect?

[0:01:45] JP: I think the biggest thing in terms of high performance is the ability to kind of understand the core business, and then translate from the core business into systems that either already exist, and you're using them in new and creative ways, or systems that don't even exist at all that you have to imagine on paper. I think the key to that second part, though, isn't necessarily just the imagination. It's actually the ability to communicate. It's the ability to tell a story about the castles that you have in your head. You're describing something that maybe something was never designed for, or you were describing something or some new use of technology that people may not be familiar with. Or frankly, you might be telling someone, “Hey, I'm not sure that that business context is correct. But if we did this, it might give you a little bit more flexibility.” That's really a communication skill at the end of the day. 

So, when I think about what makes a really effective architect, it's actually all about that storytelling aspect. It's about coming up with a story, making the story interesting, and then telling other people about, and getting them excited about a shared vision to go match a problem.

[0:02:55] LA: I love the term storyteller. I think that describes it very effectively. What I often say is that a software architect has to be multilingual. They have to understand business, they have to understand developer, and they have to understand product management, and all of those the deep understanding of the language that’s important to those individuals, because they have to communicate the business need to people who don't know anything about business, necessarily, or connected to the business very closely, and need to understand in the language of a software developer, and understand how to talk to them in their own language and how to talk to a product management in their own language, et cetera. Your right. I think the term storyteller is a good model for that, is you have to understand the language to the point where you can explain what you're trying to do in a way that they all can understand, which is a form of storytelling.

[0:03:47] JP: Yes. In fact, one of the things I've really found is, people from non-traditional software backgrounds typically do really well, in a lot of cases, because they might be coming from a biochem background. They might be coming from, people have history degrees. You might have people who are coming from a musical background. You might have people who are coming from an operations or a QA background, and that ability to kind of step between two different worlds, I think is really effective. Not to say that there's not as many people who are successful coming at it, and the more traditional mathematician, computer science background. But it does give people a different frame of reference, which tends to make them able to engage in multiple points of view a little bit easier, sometimes. 

[0:04:30] LA: Absolutely. One of the things I always struggle with is when I see the very senior engineer who is clearly an engineer, clearly the engineering mindset that says they want to become an architect. I'd like to say, “You're much better a senior engineer. Maybe you want to stay on that track and go that direction. But software architecture takes a whole different set of skills that are different than what you're doing. Not saying good senior engineers can turn into good software architects. But I think, you know the person I'm talking about, the person who is very much a developer, very much talks well to developers and very much doesn't really communicate outside the developer community. That person is not going to make a good software architect.

[0:05:13] JP: Well, I think like all skills, it can be mastered, it can be taught, it can be a skill set that you pick up. And I think there's actually a lot more resources now, in terms of what it means to be a good software architect than there were certainly 10 or 20 years ago. So, for example, part of my background, and I think something that's helped me be successful, as I came up more on the IT side of the house. I came up with setting up routers and setting up computers and servers, and early days of the Internet, and all of that. What that really enabled was kind of thinking at this very holistic, large level, where you have to incorporate a lot of people into it.

So, I think it's just like anything else, you have to want to master the craft, if you're going to be really successful at it. There are some people, where that's not what they're interested in mastering, and more power to them. They're great at what they do, don't get in their way. But I will say, I do think that skill can be learned. I think it can be mastered, if you've got a heart to go and do it.

[0:06:08] [0:06:07] LA: That's fair enough. I agree with you. It is definitely a learnable skill, but you have to have the mindset to want to learn it. So, I completely agree with what you're saying there. Now, we're going to get into where I think people disagree, and so I'd love to get – I’m looking for your opinion on this.

[0:06:23] JP: Hot take time.

[0:06:24] LA: Is software architecture a formal process or an informal process?

[0:06:29] JP: See, you're jumping right into it. So, I think that there's this classic give and take, and I think you'll see it all the time. I think there's a whole bunch of, certainly 10, 20 years ago, people would say, “Oh, software development needs to be incredibly structured. We're going to follow a waterfall methodology. We're going to plan everything out beforehand. We're going to approach this, like we're building a space shuttle, with every single risk identified, every single possible contingency, and maybe even get to the point where you mathematically model the entire system.” So, that's one spectrum.

Then, you've got the other side of the spectrum. “Oh, you don't need all that. Early days of Agile, just do these basic simple things, and you just need to get people in a group, and you need to structure it this way. But you don't need to invest in all that upfront.” What ends up happening always is we get these wild pendulum swings, from one extreme to the other. I'm going to start out incredibly structured, and then I'm going to go incredibly unstructured, and then I'm going to swing back rapidly to the other side. I think the exact same thing happens in software architecture. And I don't like to think of it as structured versus unstructured. But what I like to think of it instead is emergent versus intentional.

What I mean by that is, when you think about emergent architecture, emergent architecture is when you're making architectural decisions, and no matter what, architectural decisions always exist. If you're connecting two systems, that's an architectural decision, right? But emergent is when you're making that architectural decision based off of what you know, in that exact moment, you're writing code, and the next moment, you give it a little bit of thought, you make the right decision, a local maximum decision for that exact moment, and you make that decision as close or in that team as humanly possible.

Then, there's the intentional school of architecture. We're going to sit down. We're going to think about every single possible use case. If you want to be derogatory about this, you can call ivory tower architecture at times, right? You might have a separate team that you take off your architects in your company, and you say, “Your job is to think heavy thoughts about architecture, and then tell us where we need to go.” But you're going to think about the next two years, you're going to have a plan for the next two years. Neither of those are sufficient in and of themselves.

So, I think modern architecture really is about finding the right combination of enabling the emergent architecture, and guiding it with some very specific, intentional architecture that's thinking beyond the next line of code, beyond the next iteration, beyond the next sprint, beyond the next quarter. By having a structure where what you really have is an ability to think about where a North Star is, and march to that North Star, and then combining it with making most of your architectural decisions inside of an emergent structure as you meet to make those decisions, you preserve both the agility and the velocity that you're trying to get with Agile or lightweight development methodologies. But at the same time, having some confidence that you're not going to throw away all the work that you're currently doing, that you're anticipating what your future needs are, that you have a general idea of where you're going. 

That's not to say that your North Sar doesn't wander. That's not to say that you can't change your mind. You absolutely can. You need to refresh that often. But I think modern architecture is about marrying those two concepts.

[0:09:40] LA: That makes a lot of sense. It's very similar to the Agile architecture approach that I like to talk about, which is, you need to have an idea in your head of what you're trying to accomplish, and that picture of the universe, the picture of the world needs to be constantly updated, changed, and modified as you go along. But your decisions are in the moment decisions have to be, like you said, locally maximized, and be relevant to what's currently going on, and be flexible enough to be able to be made on the spot with what's going on, with the knowledge you have available at that moment.

So, you're never going to have complete knowledge set, you're never going to have a perfect solution. But what you're going to have is a constant picture in your mind, that's evolving of where you want to be, and these spot decisions that occur all the time, should all drive eventually towards that. But individually, they may or may not, depending on the specific needs of the case.

[0:10:36] JP: Yes. The reality of it is, is that technology changes so quickly. You can't have a multi-year horizon anymore. You can have a general idea of where you eventually want to go towards. For example, I want to build a platform because I see these adjacent markets coming up, that's a business goal that you might want to target to, and you can certainly support that from an architectural point of view. But outside of like the really big trends, you really can't count on technology being stable enough so that an architecture decision that you make today is exactly the right architecture for you to be doing in a quarter, six months, a year. Chances are, it’s going to be.

[0:11:09] LA: Right. That makes perfect sense. So, how do you create a system with a solid architecture as opposed to a solid – I'm not asking how do you create a solid architecture. How do you create a system with a solid architecture? How do you create a system with a solid architecture in the highly agile environments that are popular today? How do you that?

[0:11:32] JP: Yes, so I tend to think of architecture as kind of a tripod. What I mean by that is, there's sort of three legs to this stool. The first one, let's think about what you need to do from a process point of view, right? What kind of things do you need to do to make sure that you're communicating effectively with the entire organization that you're listening to what's going on in the organization, that you understand what business needs, that you understand what your developers are trying to do. How do you do architectural governance in one sense?

In some cases, if you're coming from a more traditional design approach, you might be thinking in terms of large upfront UML specification, software architecture definitions. I think there's much lighter-weight ways of doing that now. So, there's things like architecture decision records, ADRs. I think there is lighter weight visual representation. So, things like C4 models for describing and helping you to communicate. So, I think that governance layer, how are we going to make architecture decisions? How do we make it so that architectural decisions are carefully scoped, that the right people are involved in making those decisions, that the business needs are being reflected in those requirements? That's the first leg.

The second leg is what I would say are your guardrails. What are the things that you need to tell people you can't do to protect the company? What is your open-source policy? Are you allowed to copy and paste lines of code from a proprietary code base or a GPLv3 codebase? What is your security policies look like? What is your tech stack look like? Do you have a tech stack? Do you have a list of technologies that the company has invested in, that you want to fit into? So, those are kind of your guardrails?

Then, I think the third piece of that is really community, right? Who are the champions inside of your organization who are going to be really, really disruptive? Who are the people who are going to go say, “Hey, here's something new we can do with technology.” Who are the crafts people inside of your organization, who really are going to spread this a sense of accountability and ownership in your organization? What you want to do is you want to take that community, you want to put them into a process, an architectural governance process that allows them to be creative, and you want to give them some guidelines in the form of the process.

So, when we think about effective architecture, I think effective architecture rests on that tripod of three things, and that I think, is what helps build a really positive, encouraging, inclusive architectural community at a company.

[0:13:55] LA: That sounds a little bit more on the formal side than the informal side. Or using your terminology more on the intentional than the emergent side. Is that the way you –

[0:14:05] JP: Yes, it's a ratio. So, what I would like to say is, this entire swing between formal and informal, is it news to anybody who's been around in this industry for a bit? They understand it. And they'll actually point blank, say, in some circumstances, “Yes, I do that because I don't have better ways to actually mitigate between those two extremes. And there are times that I need to be formal, and there are times that I need to be informal.”

What I would say is, in my experience, you want about 10% to 20% of your architectural decisions, closer to 10, 10% to 15%, let's say, to be very intentional, where are we going? How are we going to get there? What structures do we need to put into place? What are our tech stacks? What are the technologies we want to invest in? Are we all going to write and go? Or is somebody going to write and go? Is somebody going to write in Python? Is somebody going to write in Rust? Is somebody going to write in Java or Perl, for whatever you're dealing with, right?

That, I think, in a good company and my experience that tends to be like 10% to 15%. I think the other 85, 80% to 85% needs to be emergent. It needs to be making the right decisions at the right time. But in order to make the right decisions at the right time, you still need to make sure that you've got the right processes in place so that if somebody is making a really important decision, the right people have visibility into it. If somebody is saying, “Hey, I'm going to bring a new language into my tech stack, then I have no chance whatsoever in finding people who actually know that language.” That's a decision that you want to make sure that people have thought through the ramifications of before they do it.

So, it's a little bit more intentional than having nothing, but I don't think it's overly formal, at least in my experience.

[0:15:39] LA: Makes sense. One of the things I talk a lot about in my writings, and one of my books focuses on this specifically is complexity. IT complexity specifically, but complexity in general. One of the things I've been putting some attention on recently, is the balance between choice and complexity. Choice leads to complexity. And giving people choices, has advantages, but one of the disadvantages is increasing complexity. Just a simple example, pushing decisions downwards within an organization tends to mean you have more smaller decisions, and therefore more variety of outcome of those decisions, and that variety of outcomes leads to complexity.

I mean, we can get into sandboxing, and talking about all of that. But in general, how do you manage the balance between giving people choices and the complexity that ensues from those decisions?

[0:16:40] JP: So, we could spend hours on this topic. There is so much in there that I mean, we usually could spend the rest of time just picking apart that question, because there's so many different aspects to it. Remind me, because I'm going to do two separate things. But I could go so deep into either of these, I could lose my place on them. But let's talk about complexity in two forms. Intentional complexity versus emergent complexity. Then, let's talk specifically in terms of architecture, about the scaling and the rule of choice and complexity in scaling things.

So, when we're talking about complexity, let's talk about intentional complexity and emergent complexity. Intentional complexity, from my perspective, is I'm looking at a problem, it's a very complex problem, I need to solve it and my solution to it is going to be complex. That is complexity that you're signing up for. It's complexity that you are aware of, that you know, that you're going to end up taking, that is just the cost of doing business. It's table stakes for your problem.

A good example is we deal with incredibly high transactional systems, right? We handle hundreds of billions of transactions a day. There is a certain amount of complexity to that, because of the fact that you've got 400 billion transactions that you need to process in under 100 milliseconds for every single one of them, and you're signing up for that complexity. So, that is kind of your intentional complexity. But there is emergent complexity. And unlike the emergent behavior that we want to encourage in architecture, emergent complexity is really difficult. Emergent complexity is, “Oh, I have a new kernel version. That new kernel version changed something, and all of a sudden, I'm seeing an elevated level of timeouts and do I have all the structure in place to actually monitor that.” It's unintended consequences from your decisions.

So, part of architecture, and the part that I think everybody likes to focus on is the very intentional complexity. “Oh, if I use a microservices architecture, I get this, this and this benefit. I understand that there's this risk about it. But I'm going to make that decision, because it's going to enable me to do something I can't do otherwise.” We tend to think a lot like 90% of our time about that. But the thing that really tends to bite us is actually that second category. It's something new happened that I didn't see coming, and I have no idea how to manage, and I don't know if I've got the right systems in place. I don't know if I've got the right ility, scalability, security, et etcetera, in place to actually manage that complexity.

When we think about complexity, just making a choice in a complex matter, you'd have to bear in mind that no matter what, you're going to hit both that first category of problems and that second category of problems, and you probably haven't thought enough about that second category of problems.

[0:19:12] LA: Right. That makes sense. There is a different type of complexity too that is probably somewhere between those two. I'm not exactly sure where it sits on that continuum between those. But it's the decisions you make that enable increasing of complexity later. Let's look at a simple example. This example is what made me think of this question in the first place when you mentioned decisions, like what language we're going to use. We're going to use Go. We're going to do whatever. But there are a lot of companies, for instance, that say, “No, we built service teams, and service teams make their own decisions about things like what language they're going to use.”

So, they push the decision down of the language choice. And the side effect of that is there'll be more languages in use within the application which increases the complexity. The application has lots of other disadvantages. Has some advantages, but other disadvantages about being able to move people around and learning curves, all that stuff. But it's a form of complexity. It's a simple fact of enabling choices, increases complexity. I'm not sure if that's intentional, or not.

[0:20:22] JP: Actually, it goes into the second category I was talking about, which is thinking about scaling the organization. Or another way to put it is, architecting things beyond software, right? When you think about the role of an architect, it's not just architecting software, right? Long ago, it started including architecting your network connections as well. But I would actually argue it goes well beyond that as well. When you think about, for example, what we're talking about, let's think about language introductions. The primary impact on that isn't necessarily just in the software pieces. It's actually in your organization. Do you have the ability to go and hire top line Go programmers? Do you have the ability to hire people with operational experience with Rust? Do you have enough programmers so that if you've got a new business space, you are going to go after, you've got enough mass to go solve that problem, right? Do you have people who are skilled enough in quality? Do you have people who understand the operations with the technology that you're talking about? Do you have enough of a runway in the business to say we can take six months to re-architect something and put it out there, or I have to do a short term.

So, I tend to think of the problems that you're talking about is very much being architecture problems. But these are examples where you're not really just architecting for software, you're actually architecting for what do the processes I need to have in place look like? What do my teams need to look like? What skill sets do they need to have? What training do they need to have? How good is the language at being agile? How good is a language in terms of people being able to pick it up? If I'm doing a lot of ml, do I even have the right hardware in place. That complexity, that architecting complexity is all about architecting its role, not just with software, but in the entire business. If you do it successfully, it's not that you're more successful in building complex systems, it's that you're being successful in scaling the business, right? That tends to be the outcome. You're able to go after business outcomes. So, you're architecting, but you're architecting for the business outcomes that you're shooting for, not necessarily the particular line of code that you might be writing.

[0:22:28] LA: Which gets back to the mindset we talked about the very beginning, which is an architect isn't just someone that helps developers figure out what to do. It's something that works with the entire business, because it's the connection that the glue that connects the business owners, to the product managers, to the software engineers, to the testers, to the IT people. All of that is interconnected. Yes, there's connections elsewhere, but one of the main connections is through the software architecture team.

[0:22:55] JP: Absolutely. I would take it a little bit further than that. I mean, I think you're exactly right. I think architects need to understand law in a real way. If you're in financial services, if you're in ad tech, if you're in medical, any of those fields that have any form of regulation in them, there is a considerable amount of your time that you're going to be spending thinking, “Hey, Am I doing the right things to comply with this law? Am I doing the right things to protect my user’s privacy? Am I doing the right things to protect the business from an ethical point of view? All of those aspects also fit into it. So, I think a modern architect is really much more about a tech being a technology architect, than it is just about being a software architect.

[0:23:39] LA: Yes, I absolutely agree. I think even saying using the word technology is, in some respects, too limiting. It's the business architect is really what it is. Everything having to do with what it takes to deliver business value, which is legal, it’s security, it’s software, it’s infrastructure, it's everything. 

[0:23:59] JP: It's storytelling. Are you telling the right story to the right people? And are you factoring in enough stuff that it's a realistic story? We'd all love to write fantasy stories. We don't always have that ability.

[0:24:09] LA: Make sense. Some people would argue. I've heard this argument a lot. Let's put it that way, that architectural processes are less formal, and less developed than software developer processes. First of all, do you agree with that? Or is that too much of a generalization? And two, what’s your thoughts about that?

[0:24:32] JP: Have you ever been tormented by Rational Rose, would be my first question.

[0:24:37] LA: Boy, that brings back a long time ago, a long time ago. I've managed to forget, a lot of things between then and now.

[0:24:45] JP: Yes. So, for those who might be a little bit younger and have a little bit less gray in their hair than I do. Again, it kind of comes and goes in waves, right? I think that there was a very strong point of view in the industry that architecture needed to be extremely formal. So, you saw a lot of good things come out of this. UML, like it or hate it, is a form of a model definition. It's a formal model. There was Rational Rose or Rational Later, and then eventually Eclipse’s EMF, which were various approaches to take a more formal modeling-based approach on it.

Those systems can be incredibly powerful. A model driven architecture approach, if you get absolutely everything correct, is incredibly powerful, and it can be a massive force multiplier, not only for your architects, but for your entire development organization. It's a very big if. It's hard to execute, but I do believe that every architect should understand domain-driven design, for example. Should understand some of those very basic core concepts.

I think architecture is kind of swung widely to that. And again, I think it's kind of gone back to the, “Hey, just at one point”, I've done my own version of this swing. I tried to replace a very large complex review system with what I was calling whiteboard designs, which was basically throw a PowerPoint template up there, draw on a whiteboard, take a picture, and then present it to a group and get everybody to agree, right? Theoretically, that's a good form of storytelling. You can absolutely do architecture, specifically at places that are maybe smaller in scope, or that you've got great alignment, or really great technical architecture expertise in the teams, you can absolutely make something like that work.

In reality, I think you do need a little bit more formal approach. So, there are again, systems like ADRs, architecture decision records. There's a whole school of thought around those. There are modeling techniques like C4. And then there's actually just plain old good tooling, like Lucidchart, or Gliffy, or Visio. For those of you stuck in the Microsoft world, whatever, that are great for enabling communication. And at the end of the day, architecture is still about communication. Whatever you need to do to tell that story, you need to tell, and get the consensus and the buy-in that you need to be able to move forward in an organization that's really well. That’s really what you need to do.

Now, different organizations have different levels of going back to your choice versus complexity thing, different architecture practices in place. If you go to someplace that has to operate at an incredible scale, say like Google, your architecture decisions in a lot of cases are made for you. They have systems that they have already scaled out to an unbelievable level, and constraints that they're going to give you, that they've learned about that will make your life as a developer much, much, much, much easier, and it will reduce complexity. Those are great systems. They're absolutely great systems. But not everybody operates in that world. The more that you're approaching something that is a Greenfield, the more choices that you have, the more you need to think about, do I have the right structure in place? Do I have the right structured decision-making process in place to make sure that I'm not making a dumb decision that I've actually thought out what the actual implications of it are? Because if you make a mistake upfront in the architecture process, it's going to cost a lot to fix it on the back end of the outcome.

[0:27:56] LA: You've kind of alluded to something which is also on my list of things I wanted to talk about. And that is the role of company culture in architecture. We know that doing software architecture for Google, or software architecture for HP, or for Verizon is very different than doing software architecture for the technology startup, that’s the three-person startup company that you're the CTO for, but simply because you had the most senior tech person in the company. Talk about the difference there about, in the startup world, what should you do? And what should you do to prepare yourself for the growth of your company so that you don't get caught up and architecture doesn't become a problem?

[0:28:43] JP: So, it's a great set of questions. And for a lot of startup use cases, you have the ability to be agile. You have the ability to move at velocity. When you're trying to achieve –

[0:28:55] LA: You have the responsibility too –

[0:28:58] JP: You have the responsibility too. When you're trying to achieve product-market fit and discover your marketplace, you have to move as quickly as possible. That is your survival instinct. If you cannot do that, you will not be successful.

On the other hand, when you are a large organization, and let's say that you're working in the pharmaceutical space, or the medical space, or aviation, or one of these fields, where people's lives absolutely depend on your decisions, you have to have a certain maturity in how you're approaching things. What I would say is in the model that I've tried adopted is this three-phase model, where you have an approach where you start with a discovery or an experimentation phase. In that, what you're really trying to do is get to the point where you can rapidly do experiments, see if they work, move on to the next thing, if they don't. If they do work, start to invest in them as quickly as possible and have a structure in place, which is just as architectural as anything else. But have a structure in place that allows you to do that rapid experimentation.

Good feature flagging. Continuous integration, so that you can push out builds on a case by case basis. Serverless, so that you don't have a huge upfront bill and standing up servers, and you're not worrying about that, right? Those are still architectural. There's still things that you need to do as an architect, even though they're unstructured. I don't think they're really unstructured. It's just you're maximizing for that discovery phase. Then, you've got kind of a maturity or a scaling phase, if you will. The maturity or scaling phase is all about, okay, we've learned X, Y, and Z, we've got product market fit, we need to grow this, but we need to start investing in some serious things and make it so that I'm not up at 2am every single night, kicking the server to make sure that it's still running.

That's all about starting to layer in the ilities, if you will. How am I going to scale this? How am I going to make sure that it's secure across the entire board, beyond just the basic can anybody get in, right? Am I doing things like data provenance? Am I doing things like additional pieces that I might need to do to make it easier to do additional pieces of analysis? What are my non-core, non-functional pieces of things that I need to layer on top of it. These, again, are still architecture, but what you're doing is you're balancing what you learned from that experimentation phase, into that middle phase into that maturity phase and saying, “Okay, now we're going to integrate with an SSO provider because my enterprise customer needs them.”

Then finally, I think the last phase is kind of this mature system. I've got something out there, people depend on it, it cannot go down, because of X, Y, and Z. I need to make sure I've got a dedicated ops team to support this. But I also need to make sure that I can still do development, and I can still go in and I can still add features in a reasonable way. But maybe I'm going to give it a little bit more, “Hey, what are the unintended consequences of taking a look at this?” “Hey, I've got a customer who needs this incredible feature, it's coming six months down the road. It's going to take me a while. Let's actually break this apart into multiple phases. So, I know how that works going to do.”

That mature phase is really all about not necessarily making the big flashy decisions, but about making a lot of little decisions in an emergent way, in a way that allows you to sustain that over a period of time. There's actually kind of a fourth phase in all this, which is okay, what do I do now that this entire thing is defunct and I need to replace it with something else. You figure out a phase to want or migrate that system often or go back to the discovery phase, and learn what you need to do again. But that is really kind of the way I think of it. I would actually challenge a little bit of your assertion about one end being the startup and the other end being the corporate. I actually think that this process occurs inside of large corporations, and there's absolutely times where the startup needs to get into a mature phase.

[0:32:32] LA: Absolutely. But there is still a cultural aspect in the larger corporations that dictate how the startup process works.

[0:32:40] JP: Absolutely. But what I will say is, culture is another thing that you architect for. If you architect for building a solution that encourages daily pushes, then it's going to change the culture of the company. This, I think, is a really important thing. I've worked, where I spent the bulk of my career was in financial services. I started out as employee number 22, through mergers acquisitions, organic growth. It was 4,000 some odd people when I left.

When you're tracing that, what I have a deep respect for is the ability for a very small number of individuals to change the engineering culture of a company, just by having the right sets of goals and having the right attitude, and the right things that they're architecting. Going in and introducing, for example, a platform, and then re-platforming a company, even at a very large financial services company, is something that's very doable, and the company will get huge benefits for you, actually advocating, for doing the right thing architecturally overtime.

[0:33:32] LA: I absolutely agree with you. I would think that that would be a great place to end the interview. But I want to actually end with a softball question. I want to ask you another question that I get asked a lot when I go out and talk to people and the answer, I give, again, is it depends. It depends on the organization. But the question is this, are software architects the best part of the development team, or best part in centralized architectural role?

[0:34:00] JP: It depends a lot. What I will say, the way that I've structured things in my current gig is I have a very small team of incredibly well respected people who understand the space, architects who have been acting as architects for a long period of time, who know how to interface with the organization, who understand that they're not there to make every decision, but they are there to be force multipliers for the rest of the organization. Those individuals, these are people who understand this space, because they grew up in this space. They literally, they've internalized this space, they understand what's going on. Whatever background they come from, they have a mastery of it. They have a mastery of how to communicate about it, and they are dedicated to the idea of taking ownership of things and driving it forward.

But that's about 20%, 10% to 20% of the architects that we have, right? So, it goes back to that entire intentional versus emergent. I want that core team that's focused on the intentional, to really have that be their day job. We don't exactly follow that. We have other architects who are responsible for some of the core intentionality pieces that are not in that core architecture team, but they still have a clearly expressed ownership. We expect you to be the person who's coordinating the architecture on these things. But then the vast majority of architecture decisions, we want those to be the people in the trenches, we want those to be the people talking to the customers, we want those to be the people looking at the unit tests, figuring out if the unit tests are right. We want the owners, the people who are the accountable owners for something, to be making those architectural decisions with as much support and as much force multiplying as we can possibly give them.

[0:35:41] LA: Yes. The model I like to use, which I think is basically very similar to what you're describing is, the Center of Excellence model, right? You have the core group of architectures or whatever technology you're talking about, but let's just talk general architecture for a moment. And they're the ones responsible for the overall architecture of the system. But they can't work in an ivory tower. They have to work with fingers tangled, connected throughout the organization. So, I like to have the central group, but individual architects are geolocated within individual teams that are part of those teams, that are part of that day to day process of those teams, where they can make the emergent decisions that come up, yet they still have a responsibility to the central group.

[0:36:29] JP: Yes. Let me draw two very small differences. But I think that are kind of illuminating. When we think about responsible for making decisions. One of the things that I've done is I've introduced the concept of accountable ownership and accountable ownership, except for the artifacts that are explicitly intentional based. So, things like what our roadmaps are, and things like that, are not the architects. They're actually the teams that are responsible for building the developer or doing the development themselves. What we put into place as part of our governance model is, the VP of Engineering, at the end of the day, is the accountable owner for that. He's got the final say on that.

If I disagree with the VP of engineering, fine, we'll take it up to our mutual boss, who's the CTO of the company, and he'll work it out. That's really not the problem. The real goal here is, I want to empower that individual to do the right thing, to make architecture decisions without being blocked on an outside org. So, he, at the end of the day, still has responsibility for that decision. But we're going to give them a lot of input, and chances are, if we really disagree, then we're going to have it out at some point at that top level if it's a decision that's worth that. That's where I think I differ from a CoE model, which in a lot of cases has, what I like to call parachuting. You parachute in an architect, you co-locate them for a period of time, but their line of responsibility is up to the architecture organization, as opposed to through the other organization.

My one challenge with that, and again, all these models, you can make work depending on the culture of what you're trying to do and what your goals really are. If you're really in an organization, where you don't have kind of that organic architecture strengths, sometimes you need to do that. But I do think that what ends up happening a lot of times is you get the development manager saying, “Hey, I'm blocked on architecture. Architecture can't do this fast enough. I'm not going to meet my goals because of them.” And the architect is saying, “Well, I designed a perfect system.” And because I designed a perfect system, every flaw in it is because you didn't implement my vision correctly. I don't want that. I want a collaboration. I want people to very clearly state what they're doing, why they're doing it, and what the concerns are, and then find a way to negotiate within the context of what we're trying to deliver as an organization. That's maybe a little less structured than some of the CoE models. But I do think that that's a good way to break the tie, if you will.

[0:38:43] LA: Fair enough. Fair enough. I'm a firm believer in what the model that I call STOSA, but it's the single team ownership, the ownership at the team level, and I'm full, full, strong advocate of that model. When I think of the architecture CoE model, I think about architects as a resource to the teams and part of the decision-making process, not the ones that deliver decisions. So, I think we're talking the same thing, just maybe in slightly different ways.

[0:39:12] JP: Yes. I agree. And there's such as shade of grey in how people implement that. A lot of that times that can get into something that's really destructive, which is politics. Politics is really, in some cases can really cause problems, because as an architect, your job is to communicate. If people aren't buying into your architectural vision. That's actually a sign that you're not communicating well, or that you haven't figured out your entire problem. So, architecture, it's incredibly rare that you run into this situation where you basically say, “It's my way or the highway.” I've never seen that successfully work from an architecture point of view.

What I have seen work is, hey, we built this system, we're going to ask everybody to use it. This is why. And yes, in some circumstances, every now and then you're going to get to a situation where you're just going to have somebody who has a different technical vision and you're going to have to have a long and candid conversation with that individual about what they're doing. But at some point, you need the oars going in this same general direction. But that's the outcome of a collaborative process. That's not the input to a collaborative process.

[0:40:07] LA: That makes perfect sense. The other tool I like to use is sandboxing. The idea where part of what the architect does is they create the boundaries and say, any decision within these boundaries, is a good decision, right? And then let the teams make the decisions from there. So, the boundaries help guide the system into the direction you want the architecture to go. This works well for things like language choices. They work well for lots of different decisions and it helps reduce complexity and helps organize towards the right architecture, without removing choice and without taking the choice away. So, it helps get ownership because people making the choices feel ownership, yet, still controls and manages the direction you're moving.

[0:40:51] JP: Yes, absolutely. And that's where things like tech stack and your guidelines, your safety boundaries really come into play. Those are critical. Again, more as you're moving into that maturing phase, especially when you're in discovery phase, yes, every now and then the best solution to something is a Perl script to deal with it. But as you're trying to scale a business, as you're trying to make it sustainable, as you're trying to go beyond that initial product market fit, I really feel that that's the place in which you really have to be thinking about, how do I take the technology that I already have, have the right patterns on them so that we can scale them very quickly, to go into new problems, solve new problems, go into adjacent problem spaces, things like that.

[0:41:27] LA: Make sense. Well, my guest today has been Josh Prismon, the chief architect of Index Exchange. Josh, thank you for joining me on Software Engineering Daily.

[0:41:37] JP: No problem. Always a pleasure.

[END]