Jekyll2023-10-10T11:34:12+00:00http://petergillardmoss.github.io/feed.xmlPeter Gillard-MossPeter Gillard-MossOn Forests and Factories2023-10-10T06:00:00+00:002023-10-10T06:00:00+00:00http://petergillardmoss.github.io/blog/2023/10/10/on-forests-and-factories<h1 id="on-forests-and-factories">On Forests and Factories</h1>
<p>A few miles from where I live is the The Pilgrims’ Way, a prehistoric trackway which stretches from Stone Henge and Avebury to the English Channel. It earned its name in the middle ages as the route followed by pilgrims to the tomb of Thomas Becket. A journey immortalised by Chaucer’s The Canterbury Tales.</p>
<p>The stretch where I live is through an avenue of trees on the edge of forest as old as Chaucer’s tale. My son has found a rope swing hanging off the gnarled branch of a veteran Hornbeam. There is a sign stuck to its trunk urging people not to climb it. The tree is over 300 years old and has just been pruned to encourage new growth which needs to be protected and nurtured.</p>
<p>As I look around I spot a number of old Hornbeams forming a canopy with a variety of other species. Oak, beech, ash, chestnut. Some of which are thought to be up to 800 years old. I reach out, touch the trees and realise I am connecting with a living organism which has stood on this earth for ten or twenty times my own lifespan. Who else has passed below its branches? What other children, in another century climbed its trunk or swung from a rope it patiently held firm? </p>
<p>The sign in this story is more significant though. These trees have reached these old ages thanks to the people who tended them and managed this woodland. From generation to generation the responsibility has been handed over. I am appreciating the trees they tended at an age they would never witness.</p>
<p>Managing a forest requires a very different approach to the ones we are used to. Most of us have been trained in the ideas of optimising organisations to become a well oiled machine. To treat problems like a factory.</p>
<p><img src="/img/mondrian.jpg" alt="Mondrian style image" /></p>
<p>A factory is a stable system. It has a hard boundary with predictable inputs and predictable outputs. It’s made up of human engineered processes which aim to make production robust, repeatable and efficient. Sometimes these engineered processes become so sophisticated they may make the factory very complicated and hard to reason about. Even so the factory maintains the overall relationship between cause and effect.</p>
<p>Our engineering brains default to seeing things as factories. Our first move is to break everything down into its component parts and then optimise the individual elements.</p>
<p>An engineering approach works well for things that are a factory. The parts of the system where inputs and outputs are stable and need to be predictable and processes need to be repeatable and efficient. Like a build pipeline or a payroll system.</p>
<p>The risk is with what happens when you take this factory approach and apply it to a forest?</p>
<p><img src="/img/deforestation.jpg" alt="A forest destroyed by logging" /></p>
<p>Forests are a complex system formed by natural processes interacting with each other to produce a whole different to its parts. Their boundaries are fuzzy; where does the forest begin and end? They are in constant states of flux, continuously changing yet the forest as a whole is self regulating and resilient.</p>
<blockquote>
<p>“Many leaders are tempted to lead like a chess master, striving to control every move, when they should be leading like gardeners, creating and maintaining a viable ecosystem in which the organization operates.” - General Stanley A. McChrystal</p>
</blockquote>
<p>When thinking of an organisation as a forest we switch from an engineering approach to a cultivation approach. One which acknowledges the fact that you are working within the natural processes that already exist. A forester cannot actually “grow” trees, or ferns or fungi. They can’t give them targets or objectives to hit. They “can only foster an environment in which the plants do so.”.</p>
<p>The way we solve problems in the forest is different from the factory. Rather than assign experts to diagnose cause and effect and propose solutions, we become responsible for spotting patterns, both desirable and undesirable and discovering ways to amplify or dampen them.</p>
<p><img src="/img/Descanso_Gardens.jpg" alt="Forest light show" /></p>
<p>It would be a mistake to view Forests and Factories as good verses bad. Two adversarial approaches which demand a shift in mindset from the way I label as wrong (your way) to the way I label as right (my way).</p>
<p>There are parts of the forest that benefit from the engineers carefully designed solution. Technology which improves surveillance, firebreaks, pathways, improvements to soil, coppicing and production of sustainable wood products.</p>
<p>As long as we tread carefully and ensure that our engineered processes work in balance with the forest and, at best promote its health and at worst do no harm, then we should encourage the pursuit of efficiency in those areas which make sense.</p>
<p>One challenge I see for many of us is that the engineering approach is so deeply ingrained that we lack the exposure to the thinking, tools and techniques of the organisational forester. The difference between what is often condescendingly labelled as hard and soft skills.</p>
<p>Hopefully though, by simply acknowledging whether we are dealing with a Forest or a Factory we can shift our approach, define different outcomes and look for different signals before we jump straight to our engineering toolbox.</p>Peter Gillard-MossOn Forests and Factories A few miles from where I live is the The Pilgrims’ Way, a prehistoric trackway which stretches from Stone Henge and Avebury to the English Channel. It earned its name in the middle ages as the route followed by pilgrims to the tomb of Thomas Becket. A journey immortalised by Chaucer’s The Canterbury Tales. The stretch where I live is through an avenue of trees on the edge of forest as old as Chaucer’s tale. My son has found a rope swing hanging off the gnarled branch of a veteran Hornbeam. There is a sign stuck to its trunk urging people not to climb it. The tree is over 300 years old and has just been pruned to encourage new growth which needs to be protected and nurtured. As I look around I spot a number of old Hornbeams forming a canopy with a variety of other species. Oak, beech, ash, chestnut. Some of which are thought to be up to 800 years old. I reach out, touch the trees and realise I am connecting with a living organism which has stood on this earth for ten or twenty times my own lifespan. Who else has passed below its branches? What other children, in another century climbed its trunk or swung from a rope it patiently held firm? The sign in this story is more significant though. These trees have reached these old ages thanks to the people who tended them and managed this woodland. From generation to generation the responsibility has been handed over. I am appreciating the trees they tended at an age they would never witness. Managing a forest requires a very different approach to the ones we are used to. Most of us have been trained in the ideas of optimising organisations to become a well oiled machine. To treat problems like a factory. A factory is a stable system. It has a hard boundary with predictable inputs and predictable outputs. It’s made up of human engineered processes which aim to make production robust, repeatable and efficient. Sometimes these engineered processes become so sophisticated they may make the factory very complicated and hard to reason about. Even so the factory maintains the overall relationship between cause and effect. Our engineering brains default to seeing things as factories. Our first move is to break everything down into its component parts and then optimise the individual elements. An engineering approach works well for things that are a factory. The parts of the system where inputs and outputs are stable and need to be predictable and processes need to be repeatable and efficient. Like a build pipeline or a payroll system. The risk is with what happens when you take this factory approach and apply it to a forest? Forests are a complex system formed by natural processes interacting with each other to produce a whole different to its parts. Their boundaries are fuzzy; where does the forest begin and end? They are in constant states of flux, continuously changing yet the forest as a whole is self regulating and resilient. “Many leaders are tempted to lead like a chess master, striving to control every move, when they should be leading like gardeners, creating and maintaining a viable ecosystem in which the organization operates.” - General Stanley A. McChrystal When thinking of an organisation as a forest we switch from an engineering approach to a cultivation approach. One which acknowledges the fact that you are working within the natural processes that already exist. A forester cannot actually “grow” trees, or ferns or fungi. They can’t give them targets or objectives to hit. They “can only foster an environment in which the plants do so.”. The way we solve problems in the forest is different from the factory. Rather than assign experts to diagnose cause and effect and propose solutions, we become responsible for spotting patterns, both desirable and undesirable and discovering ways to amplify or dampen them. It would be a mistake to view Forests and Factories as good verses bad. Two adversarial approaches which demand a shift in mindset from the way I label as wrong (your way) to the way I label as right (my way). There are parts of the forest that benefit from the engineers carefully designed solution. Technology which improves surveillance, firebreaks, pathways, improvements to soil, coppicing and production of sustainable wood products. As long as we tread carefully and ensure that our engineered processes work in balance with the forest and, at best promote its health and at worst do no harm, then we should encourage the pursuit of efficiency in those areas which make sense. One challenge I see for many of us is that the engineering approach is so deeply ingrained that we lack the exposure to the thinking, tools and techniques of the organisational forester. The difference between what is often condescendingly labelled as hard and soft skills. Hopefully though, by simply acknowledging whether we are dealing with a Forest or a Factory we can shift our approach, define different outcomes and look for different signals before we jump straight to our engineering toolbox.Improving flow with Team Topologies and Business Capability Mapping2023-09-25T06:00:00+00:002023-09-25T06:00:00+00:00http://petergillardmoss.github.io/blog/2023/09/25/business-capability-mapping-and-team-topologies<p>You’ve read Team Topologies and everyone has come together to do some event storming and map out some domains. You’ve used this to produce a new technology strategy around microservices and an organisational structure which matches. A year later you’ve rolled this out but your problems didn’t go away. What went wrong?</p>
<p>In order to diagnose we have to understand the relationship between customers, business capabilities and teams.</p>
<p>Business capabilities are a concept used by organisations to enable them to architect their business and operating models. Capabilities describe <strong>what</strong> a business does not <strong>how</strong> it does it. They are very similar in many ways to bounded contexts and domains in Domain Driven Design. However, the definition of a capability is technology agnostic and determined by what the business needs to provide value to the customer and operate succesfully. For this reason I prefer talking about <em>Business Capabilities</em> over <em>domains</em> or <em>bounded contexts</em>.</p>
<p>A strong relationship between business capabilities and teams is something which should be encouraged and Team Topologies provides a pattern language to achieve this. But here is a trap as it can be tempting to assume that organisational design is simply a case of mapping out your capabilities and then structuring your teams around them. Unfortunately this is not the case.</p>
<p>The mapping between teams and capabilities is more complicated in practice and every organisation will be different. So copying a model from somewhere else is not recommended. But, like Team Topologies itself, there are patterns and principles which can help.</p>
<h1 id="how-to-relate-teams-and-capabilities">How to relate teams and capabilities</h1>
<p>Let’s start by keeping things as simple as possible. You are providing a really simple product, you have a single team who have built a monolith and they are responsible for everything. Take this blog site for example. In simplistic terms, the purpose of this site is to share my experiences with fellow technologists so they can learn from me. That’s the Job To Be Done. I’m achieving that by providing content. That’s the product. Powering that product are a bunch of capabilities:</p>
<ul>
<li>Content Experience</li>
<li>Content Management</li>
<li>User Analytics</li>
<li>Promotion (through LinkedIn)</li>
<li>Content Syndication</li>
<li>Etc.</li>
</ul>
<p>Some of these capabilities (content experience) are direct to the end user. In service design these are classed “frontstage”. Some of them (like Usage Analytics and Content Editing) enable internal operations (me!) to carry out tasks which are necessary but not visible to the user. In service design these are classed as “backstage” and “support”. In some organisations these may be referred to as “front office” and “back office”.</p>
<p>These are the product (or business capabilities). There are also a bunch of technology capabilities which provide infrastructure and allow me to manage changes to my product. These include:</p>
<ul>
<li>Web serving (GitHub Pages)</li>
<li>Change management (GitHub)</li>
<li>Deployment pipeline (GitHub Actions)</li>
</ul>
<p>As CTO, I have one team providing the entire product and all of its capabilities. Some are frontstage, some are backstage, some are support. My team is also responsible for a set of technology capabilities (operational and delivery infrastructure). All of these things work together nicely, enabling fast flow of change.</p>
<p>Now let’s go to the other extreme. My little website has evolved to become an international educational content publisher producing thousands of pieces of content for millions of global learners. We’ve gone from one team to a big engineering department and lots of operations people doing things like writing and editing content. I’ve had to introduce lots of new capabilities like product search, payments and invoicing. It all grew inside that old monolith and change is really hard. So I read Team Topologies and microservices and set a tech strategy built around MACH architecture.</p>
<p><img src="/img/org-design.png" alt="Visualisation of the org design described next" /></p>
<p>I show the exec team an organisational structure which has stream aligned teams responsible for providing content experiences over mobile and web (Technology, Health, Mathematics etc.) and stream aligned teams responsible for providing capabilities over APIs as microservices (content editing, payments, account management etc.), a Platform team to provide the operational and delivery infrastructure and an SRE and AppSec enabling teams to build those capabilities in the stream aligned teams. The cognitive load of each stream-aligned team is kept low as they are responsible for one capability each and the platform and enabling teams give them everything they need to build and operate software. Job done!</p>
<p>Not so fast. There’s a whole messy reality behind that clean, well presented abstract model which is going to ruthlessly get in your way.</p>
<p>Before I explain why though, let’s look at this idealised case. We have stream-aligned teams who provide a single capability either through a front-end user experience or via an API of some sort. This keeps their cognitive load down from a domain perspective (germane). Each team is a design-it-build-it-run-it team, so they are responsible for their operational and delivery infrastructure too. To minimise the teams cognitive load we create a set of Platform teams responsible for making those capabilities easy to consume and manage and enabling teams to help build skills within the stream aligned teams.</p>
<p>The result is a really strong one-to-one relationship between the team and the capability including interface (user or API), operational infrastructure (compute, serverless etc.) and delivery infrastructure (source control and CD pipelines).</p>
<p>The ideal assumes that we can simply map experiences and capabilities and the structure teams around them. Whilst this makes sense from a cognitive load perspective it overlooks many crucial factors.</p>
<h1 id="problem-1-one-to-one-mapping-may-harm-flow">Problem #1: One-to-one mapping may harm flow</h1>
<p>From a flow perspective, there are many reasons why you may not want a one-to-one mapping. These will be entirely contextual and based on the current environment. In our org chart we have split content experience and content editing into two different teams, and content editing provides an API. This might make sense if there were several content experience teams who all needed to respond fast to changes in user experience and content editing needed to respond at a different cadence. However, if there was only one content experience team and a lot of their features required changes from the content editing team, then flow is hampered by high levels of coordination. In this case, splitting into two teams may not make sense.</p>
<p>Avoid falling into the trap of using the same pattern for all teams. Using flow to guide you it may make sense for one team to own vertical capabilities (e.g. UI and API), and, in the same organisation, flow might guide you to having a single team to own two horizontal capabilities (e.g. APIs for both Payment and Invoicing). There isn’t a single right answer which all teams should follow.</p>
<h1 id="problem-2-people-and-capacity-limitations">Problem #2: People and capacity limitations</h1>
<p>The ideal assumes limitless people. And that’s not realistic. Whilst the org chart might say there is one team per capability for financial and market reasons you will often not have enough people.</p>
<p>This means you will have to compromise and have single teams responsible for two or more capabilities.</p>
<p>This will bring challenges around capacity as the team will not be able to (and probably shouldn’t) give equal attention to each capability. This requires the team to carefully organise their workflows to ensure each capability gets the attention it deserves. This is when stakeholder management comes in play.</p>
<p>When a single team has to provide multiple capabilities ensure that there is high coherence between the capabilities. This will keep cognitive load down and maximise flow of change. Be careful though, when one team owns multiple highly-coherent capabilities it can be very tempting to trade-off architecture concerns like modularisation and decoupling. E.g. If the same team owns both services they may skip providing an API.</p>
<p>Another anti-pattern is the team looking after multiple capabilities grows in size and gently tips into being too large. Team size is equally important factor to flow and cognitive load as large teams find it harder to build trust. </p>
<h1 id="problem-3-capabilities-are-not-homogenous">Problem #3: Capabilities are not homogenous</h1>
<p>Two teams are independently developing a similar capability, surely they should be brought together as one? The reasoning for this is similar to DRY principles but at an architecture/team level.</p>
<p>For example, our learning content product may have official provided content and a learning blogging site. Doesn’t it make sense to introduce a single Content Management team and provide a unified editing experience and an API for the two experience teams?</p>
<p>There are many reasons why this might be the wrong thing to do. The needs of someone creating a lightweight blog are very different from someone creating high quality, highly governed content. This means the feature set of the capability may be very different.</p>
<p>When two similar looking capabilities are brought together in one team the effect could be to slow things down as each consumer team waits for changes. It might also increase cognitive load as the Content Management team has to understand both different use cases and different users. And they will probably find themselves struggling to “manage competing stakeholders” and ever increasing costs.</p>
<h1 id="problem-4-the-platform-isnt-always-clear">Problem #4: The Platform isn’t always clear</h1>
<p>It is tempting to argue that all infrastructure capabilities should be provided via a Platform team. If one team needs feature toggles, then surely the platform should provide one way of providing feature toggles to every team?</p>
<p>Whilst it is likely this is true for many capabilities be warned of premature optimisation. If only one team is doing feature toggles, that doesn’t mean that the Platform team has to rush in and take on managing that capability.</p>
<p>Use the rule of three (only extract to a common service on the third use) along with the principles of cognitive load and flow to double check whether the Platform team should be adding the capability to their portfolio.</p>
<h1 id="problem-5-vendors-dont-provide-discrete-capabilities">Problem #5: Vendors don’t provide discrete capabilities</h1>
<p>Now these problems are all difficult enough when assuming that all our teams are providing their capabilities through bespoke software. In reality you will be working with a healthy mix of software you build and software you buy.</p>
<p>Unfortunately many vendor solutions aren’t designed to enable fast flow and low cognitive load in their customers’ engineering teams. Often it’s the exact opposite! They will often provide a wide (and ever growing) range of capabilities which are tightly coupled together.</p>
<p>Even if you are fortunate enough to find vendors who provide discrete capabilities which play nicely together (and they do exist), that doesn’t mean they will fit exactly with your business model and org structure.</p>
<p>In reality the vendor will end up influencing your team design. There are things you can do to dampen the vendor’s influence and minimize the blast radius so it doesn’t impact other teams. For example treat them as a complicated sub-system team or create an anti-corruption layer by providing integrations through your API platform.</p>
<h1 id="problem-6-capabilities-arent-as-stable-as-you-think">Problem #6: Capabilities aren’t as stable as you think</h1>
<p>They can change! They can divide, join back together. New ones can be born, old ones can retire. They have lifecycles during which they may have rapid change and flow is critical, and at other times they may become stable and change very gradually. They move through the Wardley Cycle of Genesis, Bespoke, Product, Commodity. Sometimes they may be strategically critical and during other times they may be in a sustain and maintain mode.</p>
<p>When these things happen your organisational structure will need to change and respond.</p>
<h1 id="make-the-relationships-first-class">Make the relationships first class</h1>
<p>The right answer to organisational structure is always going to be dependent on flow of change and business strategy and no two teams will be the same. It will always be shifting and changing as it has to respond to the business shifting and changing.</p>
<p>Customers/users, experiences, capabilities, infrastructure and teams all need to be first class concepts when thinking about organisational structure. The way these relate and interact will have a large effect on flow and cognitive load.</p>
<p>Rather than using Team Topologies and Capability Maps to draw new organisational structures, instead start with where you are today. Begin at the team level and establish the experiences and capabilities they provide, who they provide them to, how they interact and their underlying operational and delivery infrastructure. These will give you the clues you need to establish The Next Right step to improving flow.</p>Peter Gillard-MossYou’ve read Team Topologies and everyone has come together to do some event storming and map out some domains. You’ve used this to produce a new technology strategy around microservices and an organisational structure which matches. A year later you’ve rolled this out but your problems didn’t go away. What went wrong? In order to diagnose we have to understand the relationship between customers, business capabilities and teams. Business capabilities are a concept used by organisations to enable them to architect their business and operating models. Capabilities describe what a business does not how it does it. They are very similar in many ways to bounded contexts and domains in Domain Driven Design. However, the definition of a capability is technology agnostic and determined by what the business needs to provide value to the customer and operate succesfully. For this reason I prefer talking about Business Capabilities over domains or bounded contexts. A strong relationship between business capabilities and teams is something which should be encouraged and Team Topologies provides a pattern language to achieve this. But here is a trap as it can be tempting to assume that organisational design is simply a case of mapping out your capabilities and then structuring your teams around them. Unfortunately this is not the case. The mapping between teams and capabilities is more complicated in practice and every organisation will be different. So copying a model from somewhere else is not recommended. But, like Team Topologies itself, there are patterns and principles which can help. How to relate teams and capabilities Let’s start by keeping things as simple as possible. You are providing a really simple product, you have a single team who have built a monolith and they are responsible for everything. Take this blog site for example. In simplistic terms, the purpose of this site is to share my experiences with fellow technologists so they can learn from me. That’s the Job To Be Done. I’m achieving that by providing content. That’s the product. Powering that product are a bunch of capabilities: Content Experience Content Management User Analytics Promotion (through LinkedIn) Content Syndication Etc. Some of these capabilities (content experience) are direct to the end user. In service design these are classed “frontstage”. Some of them (like Usage Analytics and Content Editing) enable internal operations (me!) to carry out tasks which are necessary but not visible to the user. In service design these are classed as “backstage” and “support”. In some organisations these may be referred to as “front office” and “back office”. These are the product (or business capabilities). There are also a bunch of technology capabilities which provide infrastructure and allow me to manage changes to my product. These include: Web serving (GitHub Pages) Change management (GitHub) Deployment pipeline (GitHub Actions) As CTO, I have one team providing the entire product and all of its capabilities. Some are frontstage, some are backstage, some are support. My team is also responsible for a set of technology capabilities (operational and delivery infrastructure). All of these things work together nicely, enabling fast flow of change. Now let’s go to the other extreme. My little website has evolved to become an international educational content publisher producing thousands of pieces of content for millions of global learners. We’ve gone from one team to a big engineering department and lots of operations people doing things like writing and editing content. I’ve had to introduce lots of new capabilities like product search, payments and invoicing. It all grew inside that old monolith and change is really hard. So I read Team Topologies and microservices and set a tech strategy built around MACH architecture. I show the exec team an organisational structure which has stream aligned teams responsible for providing content experiences over mobile and web (Technology, Health, Mathematics etc.) and stream aligned teams responsible for providing capabilities over APIs as microservices (content editing, payments, account management etc.), a Platform team to provide the operational and delivery infrastructure and an SRE and AppSec enabling teams to build those capabilities in the stream aligned teams. The cognitive load of each stream-aligned team is kept low as they are responsible for one capability each and the platform and enabling teams give them everything they need to build and operate software. Job done! Not so fast. There’s a whole messy reality behind that clean, well presented abstract model which is going to ruthlessly get in your way. Before I explain why though, let’s look at this idealised case. We have stream-aligned teams who provide a single capability either through a front-end user experience or via an API of some sort. This keeps their cognitive load down from a domain perspective (germane). Each team is a design-it-build-it-run-it team, so they are responsible for their operational and delivery infrastructure too. To minimise the teams cognitive load we create a set of Platform teams responsible for making those capabilities easy to consume and manage and enabling teams to help build skills within the stream aligned teams. The result is a really strong one-to-one relationship between the team and the capability including interface (user or API), operational infrastructure (compute, serverless etc.) and delivery infrastructure (source control and CD pipelines). The ideal assumes that we can simply map experiences and capabilities and the structure teams around them. Whilst this makes sense from a cognitive load perspective it overlooks many crucial factors. Problem #1: One-to-one mapping may harm flow From a flow perspective, there are many reasons why you may not want a one-to-one mapping. These will be entirely contextual and based on the current environment. In our org chart we have split content experience and content editing into two different teams, and content editing provides an API. This might make sense if there were several content experience teams who all needed to respond fast to changes in user experience and content editing needed to respond at a different cadence. However, if there was only one content experience team and a lot of their features required changes from the content editing team, then flow is hampered by high levels of coordination. In this case, splitting into two teams may not make sense. Avoid falling into the trap of using the same pattern for all teams. Using flow to guide you it may make sense for one team to own vertical capabilities (e.g. UI and API), and, in the same organisation, flow might guide you to having a single team to own two horizontal capabilities (e.g. APIs for both Payment and Invoicing). There isn’t a single right answer which all teams should follow. Problem #2: People and capacity limitations The ideal assumes limitless people. And that’s not realistic. Whilst the org chart might say there is one team per capability for financial and market reasons you will often not have enough people. This means you will have to compromise and have single teams responsible for two or more capabilities. This will bring challenges around capacity as the team will not be able to (and probably shouldn’t) give equal attention to each capability. This requires the team to carefully organise their workflows to ensure each capability gets the attention it deserves. This is when stakeholder management comes in play. When a single team has to provide multiple capabilities ensure that there is high coherence between the capabilities. This will keep cognitive load down and maximise flow of change. Be careful though, when one team owns multiple highly-coherent capabilities it can be very tempting to trade-off architecture concerns like modularisation and decoupling. E.g. If the same team owns both services they may skip providing an API. Another anti-pattern is the team looking after multiple capabilities grows in size and gently tips into being too large. Team size is equally important factor to flow and cognitive load as large teams find it harder to build trust. Problem #3: Capabilities are not homogenous Two teams are independently developing a similar capability, surely they should be brought together as one? The reasoning for this is similar to DRY principles but at an architecture/team level. For example, our learning content product may have official provided content and a learning blogging site. Doesn’t it make sense to introduce a single Content Management team and provide a unified editing experience and an API for the two experience teams? There are many reasons why this might be the wrong thing to do. The needs of someone creating a lightweight blog are very different from someone creating high quality, highly governed content. This means the feature set of the capability may be very different. When two similar looking capabilities are brought together in one team the effect could be to slow things down as each consumer team waits for changes. It might also increase cognitive load as the Content Management team has to understand both different use cases and different users. And they will probably find themselves struggling to “manage competing stakeholders” and ever increasing costs. Problem #4: The Platform isn’t always clear It is tempting to argue that all infrastructure capabilities should be provided via a Platform team. If one team needs feature toggles, then surely the platform should provide one way of providing feature toggles to every team? Whilst it is likely this is true for many capabilities be warned of premature optimisation. If only one team is doing feature toggles, that doesn’t mean that the Platform team has to rush in and take on managing that capability. Use the rule of three (only extract to a common service on the third use) along with the principles of cognitive load and flow to double check whether the Platform team should be adding the capability to their portfolio. Problem #5: Vendors don’t provide discrete capabilities Now these problems are all difficult enough when assuming that all our teams are providing their capabilities through bespoke software. In reality you will be working with a healthy mix of software you build and software you buy. Unfortunately many vendor solutions aren’t designed to enable fast flow and low cognitive load in their customers’ engineering teams. Often it’s the exact opposite! They will often provide a wide (and ever growing) range of capabilities which are tightly coupled together. Even if you are fortunate enough to find vendors who provide discrete capabilities which play nicely together (and they do exist), that doesn’t mean they will fit exactly with your business model and org structure. In reality the vendor will end up influencing your team design. There are things you can do to dampen the vendor’s influence and minimize the blast radius so it doesn’t impact other teams. For example treat them as a complicated sub-system team or create an anti-corruption layer by providing integrations through your API platform. Problem #6: Capabilities aren’t as stable as you think They can change! They can divide, join back together. New ones can be born, old ones can retire. They have lifecycles during which they may have rapid change and flow is critical, and at other times they may become stable and change very gradually. They move through the Wardley Cycle of Genesis, Bespoke, Product, Commodity. Sometimes they may be strategically critical and during other times they may be in a sustain and maintain mode. When these things happen your organisational structure will need to change and respond. Make the relationships first class The right answer to organisational structure is always going to be dependent on flow of change and business strategy and no two teams will be the same. It will always be shifting and changing as it has to respond to the business shifting and changing. Customers/users, experiences, capabilities, infrastructure and teams all need to be first class concepts when thinking about organisational structure. The way these relate and interact will have a large effect on flow and cognitive load. Rather than using Team Topologies and Capability Maps to draw new organisational structures, instead start with where you are today. Begin at the team level and establish the experiences and capabilities they provide, who they provide them to, how they interact and their underlying operational and delivery infrastructure. These will give you the clues you need to establish The Next Right step to improving flow.Beyond Team Topologies with team portfolios2023-09-18T09:00:00+00:002023-09-18T09:00:00+00:00http://petergillardmoss.github.io/blog/2023/09/18/team-topologies-portfolio<p>You’ve been successfully using team topologies to optimize for flow and reduce cognitive load at the team level. You have broken down barriers and re-organised into a number of different teams, some stream aligned, some enabling, some platform, some complicated-subsystem. Now your next question is “how do I manage all these different teams and unlock flow at the organisational level?”</p>
<p>As a leader you will be responsible for multiple things beyond team level flow. You have to execute on strategy, allocate funding, manage capacity, reduce impediments to collaboration and improve capability across teams whether for security, compliance or other cross functionals. How do leaders build on Team Topologies to achieve these things?</p>
<p>In the next series of articles I am going to explain how Team Topologies combined with the principles and ideas from Value Driven Portfolio Management, such as EDGE, can enable leaders to improve flow at the next level. </p>
<h1 id="managing-teams-as-a-portfolio">Managing teams as a portfolio</h1>
<p>Team Topologies stresses that teams should be cross-functional with a strong purpose aligned to the flow of value. EDGE stresses that portfolios should be managed by value and delivered by self-sufficient cross functional teams, again aligned to value.</p>
<p>Where EDGE helps leaders manage and govern the portfolio of value, team topologies helps leaders design the organizational structure to deliver. The two approaches go hand in hand..</p>
<p>So what does a Team Topologies portfolio look like? Chapter 7 of EDGE talks about Capability-Based Portfolios. This gives a complete picture of all the business and technology capabilities the organization needs to deliver along with the set of key performance indicators (KPIs) and other measures used to monitor the health of those capabilities. By combining this with Team Topologies leaders have the sensing mechanisms and tools to refine team structures to improve alignment and flow of value. This creates a virtuous cycle.</p>
<h1 id="thinking-in-portfolios">Thinking in portfolios</h1>
<p>At this level, it is critical to be able to think of the portfolio in terms of the capabilities provided by the teams. The key responsibility of leaders is to effectively manage this portfolio in order to improve flow of value at the organizational level. This is achieved by creating greater and greater alignment between value and the team and greater and greater self-sufficiency of each team.</p>
<p>There are a few principles which are really critical to achieving this:</p>
<ul>
<li>Everything (costs, security, compliance, legal etc.) are linked to outcomes</li>
<li>Teams are first class and responsible for discrete outcomes</li>
<li>Flow is optimized at the org level, beyond team level flow</li>
<li>Portfolio is cross-functional and collectively owned by all leaders (not just one)</li>
</ul>
<p>This is very different from other management methods where functional leaders are accountable for managing projects based on their own functional deliverables. Instead capability portfolios should include everything needed to successfully deliver value to the customer and run the organization and each capability is provided with everything it needs to design, build and run the software.</p>
<p>Another thing is that the portfolio is not rigid like an architecture diagram of components or an organizational chart (it is neither of these things). It should change and adapt as the organization responds to changes in the environment and teams identify ways to re-organise in order to improve the flow of value. The role of leaders is to govern the portfolio as a whole and ensure that it is well balanced and is creating an impact greater than the sum of its parts.</p>
<h1 id="introducing-stream-aligned-teams-into-the-portfolio">Introducing Stream Aligned Teams into the Portfolio</h1>
<p>In order to govern the portfolio each team needs to be clear on the capabilities they provide. It’s easier to start by bringing your stream aligned teams as identifying their capabilities will be simpler.</p>
<p>For each capability the team provides they must expose the following information:</p>
<ul>
<li>Outcome the capability provides</li>
<li>Measures of Success linked to those outcomes</li>
<li>Key Performance Indicators which indicate operational health (inc business and engineering)</li>
<li>Risks</li>
<li>Costs</li>
</ul>
<p>For example, if the capability was Product Search then:</p>
<ul>
<li><strong>Outcome</strong>: Customers are finding the right products</li>
<li><strong>MoS</strong>: % of searches which convert to purchase (driving revenue)</li>
<li><strong>KPIs</strong>: response time, click-through rate, relevance, availability, 4KMs etc.</li>
<li><strong>Risks</strong>: security, scalability, team health, delivery etc.</li>
<li><strong>Costs</strong>: $60k per month inc 5 HC and infra costs</li>
</ul>
<p>This data then serves two purposes:</p>
<ol>
<li>The stream aligned teams use this to sense and respond within the team</li>
<li>Leaders re-use this same information to sense and respond across the organization</li>
</ol>
<p>Gathering this information is where the functional leaders come in. Because the portfolio is collectively owned, everyone works from the same model. Rather than each function (e.g. engineering, security, compliance, finance) doing governance independently and reporting back in silos, they are responsible for enabling the stream aligned teams to gather and interpret the information. They do this first for the benefit of the team, then expose this to the portfolio for the benefit of leaders. For example, finance are responsible for helping each team understand their costs and budgets and adding their insights to each capability in the portfolio.</p>
<p>This information can then be used, alongside a healthy dose of human judgment (from the team) to RAG status their capabilities. Is there a drop in KPIs? Or have costs gone up suddenly? Then mark the capability as Amber or Red.</p>
<p>This gives leaders the ability to look across the portfolio and get the big picture they need and make appropriate decisions. Because the teams are first class in the portfolio this acts as a forcing function on cross-functional collaboration as different leaders have to make decisions collectively based on the data. For example, if a number of teams are showing red on security, then there is an easy case to prioritize those concerns over delivery if that is mostly green. It also dramatically reduces the number of reports and meetings because everything comes together in one place through the lens of teams, not split by functional needs.</p>
<p>Another benefit to this view is that it removes a lot of the discussions around engineering productivity. This is because the relationship between inputs and outputs are organized around the capability and the value it produces. Costs are linked to outcomes. This makes it really easy for leaders to see the relationship between engineers and the value they are creating.</p>
<h1 id="introducing-platform-teams-into-the-portfolio">Introducing Platform Teams into the Portfolio</h1>
<p>Where stream aligned teams are responsible for business capabilities, Platform teams are responsible for technology capabilities. By technology capability I mean things from production infrastructure (compute, API gateways, message queues, databases etc.) to delivery infrastructure (source code, deployment pipelines etc.).</p>
<p>The collection of Platform teams can be captured in their own sub-portfolio. They should report their capabilities in a similar way to the stream-aligned teams. One key difference is that the platform team’s outcomes must be related to helping the stream aligned teams improve their technical KPIs. For example, stream aligned teams will have 4KMs as a capability KPI, so the outcome of the Deployment platform capability should be “making it easy for stream aligned teams to deploy to production” as its outcome with its MoS related to improvement in the 4KM.</p>
<p>Again, the portfolio makes this easier for the platform teams as the stream aligned teams are already making this data visible there. Rather than the platform teams having to do all their own measuring and data collection they are able to work from the exact same view that everyone uses. </p>
<h1 id="introducing-enabling-teams-into-the-portfolio">Introducing Enabling Teams into the Portfolio</h1>
<p>Remember earlier I mentioned that functional leaders are responsible for enabling the stream aligned teams to gather and interpret information and turn this into KPIs? This is essentially where the Enabling team type comes in.</p>
<p>If we look at our portfolio, we may see that 60% of our teams have KPIs we would class as high availability. And we notice the teams are struggling to reach those KPIs because of a lack of skills. This is where leaders would make the decision to add a Site Reliability Engineering enabling team to the portfolio in order to shift the needle on those KPIs by building those skills across the teams.</p>
<p>The success of the enabling team is based on that outcome: “identified teams have the skills to build high availability services”. Again, as with the platform team, the governance is all done through the same portfolio view.</p>
<p>Sometimes the relationship between enabling teams goes the other way. They may decide that they wish to introduce a new KPI to specific capabilities and then they become responsible for helping them meet that KPI. For example, when GDPR was introduced, specific capabilities which need to process personal data would require introducing KPIs around data handling. Again, by using the portfolio view, this information can be clearly seen as each team with GDPR responsibilities now has the information they need to measure their performance.</p>
<p>Another pattern which emerges is that the functional leads switch to being responsible for enabling teams and are able to use the portfolio view to show their impact through the results of the stream aligned teams. This creates a big change in how they interact within the organization as their success becomes the team’s success.</p>
<h1 id="introducing-complicated-subsystem-teams-into-the-portfolio">Introducing Complicated Subsystem Teams into the Portfolio</h1>
<p>Complicated subsystem teams are, well, complicated. Whilst it is best to try and express the value they provide in a similar way to the stream aligned teams, this isn’t always easy and is very dependant on what components they are developing and how they are used.</p>
<p>However, it is important to try and articulate the outcomes they are trying to provide to the stream aligned teams. This might naturally force them into operating more like an enabling team or a platform team. </p>
<h1 id="managing-the-complete-portfolio">Managing the complete portfolio</h1>
<p>Now your leaders and teams should have a shared view of the portfolio. But it doesn’t have to end there. By capturing the right information at the portfolio it should enable different teams and leaders to slice and dice based on their current needs. For example, if the business needs to IPO and compliance becomes a priority, the portfolio view can help you identify which teams will be impacted by this change. Not only does this give a sense of scope it helps formulate strategies and agree priorities for making effective change.</p>
<p>The portfolio also doesn’t have to be limited to the information I suggested. Depending on the organization other information can be brought in. The point is the unit of granularity is always the capabilities provided by the team. And the role of the portfolio is to bring visibility of all those capabilities in one place.</p>
<h1 id="portfolios-within-portfolios">Portfolios within portfolios</h1>
<p>Smaller organizations will be able to achieve this with a single portfolio. However, as organizations get bigger they may need to break their portfolios down. This should be done in a fractal manner where sub-portfolios operate at a different scale enabling leaders to zoom in and out. A common way of doing this is by creating a sub-portfolio per domain. This should also match the organisational structure by giving next level leaders responsiblity for their portfolio and the teams within them.</p>
<p>Making teams and the capabilities they provide first class and governing them as portfolios makes it much easier for teams to optimize at the organizational level and identify strategies for improving organizational performance. It also gives teams the much needed view of how they fit into the bigger picture.</p>Peter Gillard-MossYou’ve been successfully using team topologies to optimize for flow and reduce cognitive load at the team level. You have broken down barriers and re-organised into a number of different teams, some stream aligned, some enabling, some platform, some complicated-subsystem. Now your next question is “how do I manage all these different teams and unlock flow at the organisational level?” As a leader you will be responsible for multiple things beyond team level flow. You have to execute on strategy, allocate funding, manage capacity, reduce impediments to collaboration and improve capability across teams whether for security, compliance or other cross functionals. How do leaders build on Team Topologies to achieve these things? In the next series of articles I am going to explain how Team Topologies combined with the principles and ideas from Value Driven Portfolio Management, such as EDGE, can enable leaders to improve flow at the next level. Managing teams as a portfolio Team Topologies stresses that teams should be cross-functional with a strong purpose aligned to the flow of value. EDGE stresses that portfolios should be managed by value and delivered by self-sufficient cross functional teams, again aligned to value. Where EDGE helps leaders manage and govern the portfolio of value, team topologies helps leaders design the organizational structure to deliver. The two approaches go hand in hand.. So what does a Team Topologies portfolio look like? Chapter 7 of EDGE talks about Capability-Based Portfolios. This gives a complete picture of all the business and technology capabilities the organization needs to deliver along with the set of key performance indicators (KPIs) and other measures used to monitor the health of those capabilities. By combining this with Team Topologies leaders have the sensing mechanisms and tools to refine team structures to improve alignment and flow of value. This creates a virtuous cycle. Thinking in portfolios At this level, it is critical to be able to think of the portfolio in terms of the capabilities provided by the teams. The key responsibility of leaders is to effectively manage this portfolio in order to improve flow of value at the organizational level. This is achieved by creating greater and greater alignment between value and the team and greater and greater self-sufficiency of each team. There are a few principles which are really critical to achieving this: Everything (costs, security, compliance, legal etc.) are linked to outcomes Teams are first class and responsible for discrete outcomes Flow is optimized at the org level, beyond team level flow Portfolio is cross-functional and collectively owned by all leaders (not just one) This is very different from other management methods where functional leaders are accountable for managing projects based on their own functional deliverables. Instead capability portfolios should include everything needed to successfully deliver value to the customer and run the organization and each capability is provided with everything it needs to design, build and run the software. Another thing is that the portfolio is not rigid like an architecture diagram of components or an organizational chart (it is neither of these things). It should change and adapt as the organization responds to changes in the environment and teams identify ways to re-organise in order to improve the flow of value. The role of leaders is to govern the portfolio as a whole and ensure that it is well balanced and is creating an impact greater than the sum of its parts. Introducing Stream Aligned Teams into the Portfolio In order to govern the portfolio each team needs to be clear on the capabilities they provide. It’s easier to start by bringing your stream aligned teams as identifying their capabilities will be simpler. For each capability the team provides they must expose the following information: Outcome the capability provides Measures of Success linked to those outcomes Key Performance Indicators which indicate operational health (inc business and engineering) Risks Costs For example, if the capability was Product Search then: Outcome: Customers are finding the right products MoS: % of searches which convert to purchase (driving revenue) KPIs: response time, click-through rate, relevance, availability, 4KMs etc. Risks: security, scalability, team health, delivery etc. Costs: $60k per month inc 5 HC and infra costs This data then serves two purposes: The stream aligned teams use this to sense and respond within the team Leaders re-use this same information to sense and respond across the organization Gathering this information is where the functional leaders come in. Because the portfolio is collectively owned, everyone works from the same model. Rather than each function (e.g. engineering, security, compliance, finance) doing governance independently and reporting back in silos, they are responsible for enabling the stream aligned teams to gather and interpret the information. They do this first for the benefit of the team, then expose this to the portfolio for the benefit of leaders. For example, finance are responsible for helping each team understand their costs and budgets and adding their insights to each capability in the portfolio. This information can then be used, alongside a healthy dose of human judgment (from the team) to RAG status their capabilities. Is there a drop in KPIs? Or have costs gone up suddenly? Then mark the capability as Amber or Red. This gives leaders the ability to look across the portfolio and get the big picture they need and make appropriate decisions. Because the teams are first class in the portfolio this acts as a forcing function on cross-functional collaboration as different leaders have to make decisions collectively based on the data. For example, if a number of teams are showing red on security, then there is an easy case to prioritize those concerns over delivery if that is mostly green. It also dramatically reduces the number of reports and meetings because everything comes together in one place through the lens of teams, not split by functional needs. Another benefit to this view is that it removes a lot of the discussions around engineering productivity. This is because the relationship between inputs and outputs are organized around the capability and the value it produces. Costs are linked to outcomes. This makes it really easy for leaders to see the relationship between engineers and the value they are creating. Introducing Platform Teams into the Portfolio Where stream aligned teams are responsible for business capabilities, Platform teams are responsible for technology capabilities. By technology capability I mean things from production infrastructure (compute, API gateways, message queues, databases etc.) to delivery infrastructure (source code, deployment pipelines etc.). The collection of Platform teams can be captured in their own sub-portfolio. They should report their capabilities in a similar way to the stream-aligned teams. One key difference is that the platform team’s outcomes must be related to helping the stream aligned teams improve their technical KPIs. For example, stream aligned teams will have 4KMs as a capability KPI, so the outcome of the Deployment platform capability should be “making it easy for stream aligned teams to deploy to production” as its outcome with its MoS related to improvement in the 4KM. Again, the portfolio makes this easier for the platform teams as the stream aligned teams are already making this data visible there. Rather than the platform teams having to do all their own measuring and data collection they are able to work from the exact same view that everyone uses. Introducing Enabling Teams into the Portfolio Remember earlier I mentioned that functional leaders are responsible for enabling the stream aligned teams to gather and interpret information and turn this into KPIs? This is essentially where the Enabling team type comes in. If we look at our portfolio, we may see that 60% of our teams have KPIs we would class as high availability. And we notice the teams are struggling to reach those KPIs because of a lack of skills. This is where leaders would make the decision to add a Site Reliability Engineering enabling team to the portfolio in order to shift the needle on those KPIs by building those skills across the teams. The success of the enabling team is based on that outcome: “identified teams have the skills to build high availability services”. Again, as with the platform team, the governance is all done through the same portfolio view. Sometimes the relationship between enabling teams goes the other way. They may decide that they wish to introduce a new KPI to specific capabilities and then they become responsible for helping them meet that KPI. For example, when GDPR was introduced, specific capabilities which need to process personal data would require introducing KPIs around data handling. Again, by using the portfolio view, this information can be clearly seen as each team with GDPR responsibilities now has the information they need to measure their performance. Another pattern which emerges is that the functional leads switch to being responsible for enabling teams and are able to use the portfolio view to show their impact through the results of the stream aligned teams. This creates a big change in how they interact within the organization as their success becomes the team’s success. Introducing Complicated Subsystem Teams into the Portfolio Complicated subsystem teams are, well, complicated. Whilst it is best to try and express the value they provide in a similar way to the stream aligned teams, this isn’t always easy and is very dependant on what components they are developing and how they are used. However, it is important to try and articulate the outcomes they are trying to provide to the stream aligned teams. This might naturally force them into operating more like an enabling team or a platform team. Managing the complete portfolio Now your leaders and teams should have a shared view of the portfolio. But it doesn’t have to end there. By capturing the right information at the portfolio it should enable different teams and leaders to slice and dice based on their current needs. For example, if the business needs to IPO and compliance becomes a priority, the portfolio view can help you identify which teams will be impacted by this change. Not only does this give a sense of scope it helps formulate strategies and agree priorities for making effective change. The portfolio also doesn’t have to be limited to the information I suggested. Depending on the organization other information can be brought in. The point is the unit of granularity is always the capabilities provided by the team. And the role of the portfolio is to bring visibility of all those capabilities in one place. Portfolios within portfolios Smaller organizations will be able to achieve this with a single portfolio. However, as organizations get bigger they may need to break their portfolios down. This should be done in a fractal manner where sub-portfolios operate at a different scale enabling leaders to zoom in and out. A common way of doing this is by creating a sub-portfolio per domain. This should also match the organisational structure by giving next level leaders responsiblity for their portfolio and the teams within them. Making teams and the capabilities they provide first class and governing them as portfolios makes it much easier for teams to optimize at the organizational level and identify strategies for improving organizational performance. It also gives teams the much needed view of how they fit into the bigger picture.Seven Leadership Focus Areas to Cultivate Engineering Excellence2023-09-12T09:00:00+00:002023-09-12T09:00:00+00:00http://petergillardmoss.github.io/blog/2023/09/12/seven-leadership-focus-areas-to-cultivate-engineering-excellence<p>If you’ve been following my posts you’ll know I’m passionate about cultivating cultures of Engineering Excellence and why leaders need to go beyond numbers and data and focus on teamwork and what makes a team exceptional. But I haven’t explained what technology leaders can do to achieve this.</p>
<p>I’ve come up with seven areas technology leaders can focus on to create these cultures of engineering excellence by developing mechanisms which provide teams with the feedback loops they need so they can autonomously drive towards improving customer, business, team and individual outcomes.</p>
<p>Here’s the key: leaders <em>must</em> provide these tools <strong>in the teams first</strong> and bring value directly to them. Once in place leaders can reuse these same mechanisms to better connect what’s really happening on the ground with the broad organisational goals they are responsible for driving.</p>
<h3 id="1-align-everything-to-customer-and-business-impact"><strong>1. Align everything to Customer and Business Impact</strong></h3>
<p>To excel in engineering, teams must first understand the customer and user deeply and the relationship between them and business success. Clarify each team’s purpose in customer terms, demonstrating how it contributes to the business. This alignment fosters a sense of purpose and motivation among your engineers.</p>
<p>Develop feedback loops that facilitate a deeper understanding of customer needs and behaviours. For example by capturing customer activity and interactions. These signals provide invaluable insights into user needs and preferences.</p>
<p>Next, measure your business capability metrics to gauge operational effectiveness. How efficient are your business operational processes in delivering value? For example, are payments completed with accuracy and speed?</p>
<h3 id="2-monitor-delivery-logistics-and-quality-of-service"><strong>2. Monitor Delivery Logistics and Quality of Service</strong></h3>
<p>To continously improve on your customers satisfaction and business outcomes you need to be confident that you have the capability to be responsive to new ideas whilst providing reliable services.</p>
<p>Establish feedback loops that give teams insights into the health and agility of their Software Development Life Cycle (SDLC) and Operations (Ops) by using metrics like flow efficiency, DORA 4KM and Service Level Indicators (SLIs).</p>
<p>Ensure that you have a strong understanding of your cross-functional requirements (security, availability, accessibility, compliance etc.) and that you have visibility of how you are performing against them.</p>
<p>Focus on continous improvement and help teams identify waste and impediments and give them the tools and time so they can proactively elimate them independantly.</p>
<h3 id="3-build-pride-in-practice-and-craft"><strong>3. Build Pride in Practice and Craft</strong></h3>
<p>As humans we are intrinsically motivated to become good at what we do and develop pride in the quality of our work.</p>
<p>Supercharge this instinct by working with the community to co-create practices and standards. Empower your community to create and own frameworks that align with their context and discipline. Let them be the torchbearers of progress as they advance ways of working.</p>
<p>Help individuals assess their skill levels against these practices. For instance, if Test-Driven Development (TDD) is a practice, provide a clear path for developers to self-assess, gather feedback from others and grow their expertise.</p>
<h3 id="4-accelerate-high-quality-decision-making"><strong>4. Accelerate High Quality Decision-Making</strong></h3>
<p>Empower your teams by pushing decision-making closer to them. Embeding the above feedback loops at team, rather than management level, will provide them with the framing and data they need to make fast, high quality decisions and learn from them.</p>
<p>However, be mindful of consistency and alignment risks. Invest in decision support mechanisms such as joint solutioning forums, knowledge management, and lightweight, co-owned governance structures. These mechanisms enhance collaboration, speed up decision-making, and maintain alignment without becoming rigid.</p>
<h3 id="5-foster-individual-growth"><strong>5. Foster Individual Growth</strong></h3>
<p>People are your greatest asset, and the more you can help them grow the greater the impact they will have.</p>
<p>To nurture engineering excellence, provide individuals with feedback loops that enhance their intuition and accelerate learning. Imagine it like a musician who uses a metronome to improve their sense of rythm and timing.</p>
<p>Offer coaching and mentoring and use yours and others expertise and experience to provide an objective sounding board, increasing empowerment and growth.</p>
<p>Proactively look for and create opportunities for people to take on more responsibility and leverage your own leadership.</p>
<h3 id="6-develop-sense-making-and-ethnography"><strong>6. Develop Sense Making and Ethnography</strong></h3>
<p>Whilst leaders should invest in building these feedback loops to empower the teams, it’s important that they are not blind themselves.</p>
<p>Leaders should don the hat of researchers, striving to grasp both the big picture and the minutiae of obstacles. Leverage existing rituals like retrospectives, one-on-ones, and team catch-ups to paint a comprehensive picture of what’s happening on the ground.</p>
<p>Gemba sessions, when leaders go to where the work is and observe, are fantastic ways to build trust and relationships and give leaders the feedback loops to understand the real impact of their decisions and the challenges teams are facing.</p>
<h3 id="7-advocate-and-role-model"><strong>7. Advocate and Role Model</strong></h3>
<p>Leaders play a pivotal role in shaping engineering cultures. They need to be role models for their teams, embodying the highest standards of their craft. They must also represent their teams’ interests in decision-making forums, bridging the gap between the ground reality and strategic decisions. Ultimately, leaders are the human feedback loops that foster continuous improvement across organisational boundaries and help avoid sub-optimisation.</p>
<p>Nurturing a culture of engineering excellence is a holistic endeavor. It involves a team centric approach creating feedback loops, rather than reporting against targets. These feedback loops need to exist at multiple levels, connecting different aspects of your organization. By building these feedback loops at team level you empower them to improve and individuals to grow.</p>
<p>By embracing these principles, you can accelerate high-quality, data-driven decision-making and foster a culture of excellence that motivates your engineering teams to reach new heights.</p>Peter Gillard-MossIf you’ve been following my posts you’ll know I’m passionate about cultivating cultures of Engineering Excellence and why leaders need to go beyond numbers and data and focus on teamwork and what makes a team exceptional. But I haven’t explained what technology leaders can do to achieve this. I’ve come up with seven areas technology leaders can focus on to create these cultures of engineering excellence by developing mechanisms which provide teams with the feedback loops they need so they can autonomously drive towards improving customer, business, team and individual outcomes. Here’s the key: leaders must provide these tools in the teams first and bring value directly to them. Once in place leaders can reuse these same mechanisms to better connect what’s really happening on the ground with the broad organisational goals they are responsible for driving. 1. Align everything to Customer and Business Impact To excel in engineering, teams must first understand the customer and user deeply and the relationship between them and business success. Clarify each team’s purpose in customer terms, demonstrating how it contributes to the business. This alignment fosters a sense of purpose and motivation among your engineers. Develop feedback loops that facilitate a deeper understanding of customer needs and behaviours. For example by capturing customer activity and interactions. These signals provide invaluable insights into user needs and preferences. Next, measure your business capability metrics to gauge operational effectiveness. How efficient are your business operational processes in delivering value? For example, are payments completed with accuracy and speed? 2. Monitor Delivery Logistics and Quality of Service To continously improve on your customers satisfaction and business outcomes you need to be confident that you have the capability to be responsive to new ideas whilst providing reliable services. Establish feedback loops that give teams insights into the health and agility of their Software Development Life Cycle (SDLC) and Operations (Ops) by using metrics like flow efficiency, DORA 4KM and Service Level Indicators (SLIs). Ensure that you have a strong understanding of your cross-functional requirements (security, availability, accessibility, compliance etc.) and that you have visibility of how you are performing against them. Focus on continous improvement and help teams identify waste and impediments and give them the tools and time so they can proactively elimate them independantly. 3. Build Pride in Practice and Craft As humans we are intrinsically motivated to become good at what we do and develop pride in the quality of our work. Supercharge this instinct by working with the community to co-create practices and standards. Empower your community to create and own frameworks that align with their context and discipline. Let them be the torchbearers of progress as they advance ways of working. Help individuals assess their skill levels against these practices. For instance, if Test-Driven Development (TDD) is a practice, provide a clear path for developers to self-assess, gather feedback from others and grow their expertise. 4. Accelerate High Quality Decision-Making Empower your teams by pushing decision-making closer to them. Embeding the above feedback loops at team, rather than management level, will provide them with the framing and data they need to make fast, high quality decisions and learn from them. However, be mindful of consistency and alignment risks. Invest in decision support mechanisms such as joint solutioning forums, knowledge management, and lightweight, co-owned governance structures. These mechanisms enhance collaboration, speed up decision-making, and maintain alignment without becoming rigid. 5. Foster Individual Growth People are your greatest asset, and the more you can help them grow the greater the impact they will have. To nurture engineering excellence, provide individuals with feedback loops that enhance their intuition and accelerate learning. Imagine it like a musician who uses a metronome to improve their sense of rythm and timing. Offer coaching and mentoring and use yours and others expertise and experience to provide an objective sounding board, increasing empowerment and growth. Proactively look for and create opportunities for people to take on more responsibility and leverage your own leadership. 6. Develop Sense Making and Ethnography Whilst leaders should invest in building these feedback loops to empower the teams, it’s important that they are not blind themselves. Leaders should don the hat of researchers, striving to grasp both the big picture and the minutiae of obstacles. Leverage existing rituals like retrospectives, one-on-ones, and team catch-ups to paint a comprehensive picture of what’s happening on the ground. Gemba sessions, when leaders go to where the work is and observe, are fantastic ways to build trust and relationships and give leaders the feedback loops to understand the real impact of their decisions and the challenges teams are facing. 7. Advocate and Role Model Leaders play a pivotal role in shaping engineering cultures. They need to be role models for their teams, embodying the highest standards of their craft. They must also represent their teams’ interests in decision-making forums, bridging the gap between the ground reality and strategic decisions. Ultimately, leaders are the human feedback loops that foster continuous improvement across organisational boundaries and help avoid sub-optimisation. Nurturing a culture of engineering excellence is a holistic endeavor. It involves a team centric approach creating feedback loops, rather than reporting against targets. These feedback loops need to exist at multiple levels, connecting different aspects of your organization. By building these feedback loops at team level you empower them to improve and individuals to grow. By embracing these principles, you can accelerate high-quality, data-driven decision-making and foster a culture of excellence that motivates your engineering teams to reach new heights.Beyond metrics: creating cultures of engineering excellence2023-09-03T09:00:00+00:002023-09-03T09:00:00+00:00http://petergillardmoss.github.io/blog/2023/09/03/beyond-metrics-creating-cultures-of-engineering-excellence<p>McKinsey’s report “Yes, you can measure developer productivity” has caused a strong reaction in the engineering community. As an experienced engineering manager I have made it my job to read McKinsey’s report and the response to it.</p>
<p>The report’s conclusions are tempting and it comes across reasonable. It talks about the collaborative nature of software development, leverages industry recognised metrics such as DORA and SPACE and even warns about the perils of misuse. Yet there is something about its approach which feels regressive. So what is the problem?</p>
<blockquote>
<p>“Eliminate exhortations for the workforce asking for new levels of productivity” - Deming’s 10th principle of management.</p>
</blockquote>
<p>McKinsey’s report falls instantly into this trap. It misleads the C-Suite into believing that developer productivity can be controlled by prescribing of a set of specific metrics on the engineering function. And it completely misses the truth, that metrics are only a tool. And that true engineering excellence comes from having the right engineering leadership who are empowered to cultivate the right culture.</p>
<h1 id="why-is-developer-productivity-suddenly-so-important">Why is Developer Productivity suddenly so important?</h1>
<p>The last decade has witnessed a technology gold rush, driven by the proliferation of ‘digital native’ companies and accelerated by the challenges posed by the COVID-19 pandemic. As organizations emerge from this era of digital transformation, they find themselves in a new landscape where their IT functions, once relegated to backend support, have evolved into integral engineering departments embedded throughout their business models. What’s more, despite the recent tech layoffs, the size of these engineering departments is growing at an unprecedented pace.</p>
<p>It’s no surprise that McKinsey has responded by offering a set of tools to ‘manage’ these increasingly complex engineering departments through the lens of crude metrics. The C-Suite, now discovering they are blind now engineering has replaced business operations as their strategic execution engine, seeks to measure developer productivity, desperately searching for answers to an urgent problem—how to navigate this transformation and ensure business success.</p>
<h1 id="mckinseys-metrics-driven-approach">McKinsey’s Metrics-Driven Approach</h1>
<p>McKinsey’s report is great news for engineers as it is a testament to the growing recognition within the C-Suite of the business-critical nature of high-performing engineering departments.</p>
<p>However, the model proposed by McKinsey may inadvertently take organizations back to an era of poor management. McKinsey’s report fails to take heed on its own warnings. Engineers, drawing from their experience, argue that this approach is a step in the wrong direction.</p>
<h1 id="learning-from-past-mistakes">Learning from past mistakes</h1>
<p>To understand why McKinsey’s approach might falter, we need to look back to the 1980s when US and European businesses faced an existential crisis from Japanese manufacturers. Western companies, having deemed manufacturing a ‘solved problem,’ proliferated methods from the pre-war era pioneered by General Motors. In contrast, the Japanese, having rebuilt their country from scratch, rewrote the rule book on manufacturing and disrupted established brands by producing faster, cheaper, and higher-quality products.</p>
<p>A wave of ‘Lean Transformation’ swept through America, as companies implemented the Japanese methods now rebranded as Lean. However, the Japanese continued to crush the competition because too many companies missed the vital element—the culture. This is the very mistake McKinsey is repeating. Metrics and methods are essential, but as the saying goes, “It’s the culture, stupid.” Without cultivating the right culture, we cannot expect meaningful impact on the business.</p>
<h1 id="the-role-of-leadership">The role of leadership</h1>
<p>Deming’s wisdom echoes through time: “Quality is made in the boardroom.” While McKinsey provides a solution to the measurement problem, engineering excellence is not a problem of having the right metrics; it is a problem of leadership. We must avoid the pitfalls of the past and empower the boardroom and the C-Suite to foster exceptional engineering leadership, creating a competitive advantage.</p>
<p>Just as in manufacturing, the temptation to skip changing culture and leadership is because creating cultures of engineering excellence is hard. Really hard. It requires a combination of engineering instinct, organisational systems, people management and strong role models. It’s not just about improving the productivity of engineers who work on the coal face. It’s about hiring the best engineering leaders who can enable that.</p>
<p>To create cultures of excellence engineering leaders must know how to instill passion for their craft and know how to “remove barriers that rob people in management and in engineering of their right to pride of workmanship” (Deming’s 12th principle of management). The engineers they lead must have faith that their leaders are advocates for them, liberating them to deliver better, more impactful solutions.</p>
<h1 id="experts-leading-experts">Experts Leading Experts</h1>
<p>The <a href="https://www.forbes.com/sites/joshbersin/2012/03/09/why-leaders-must-be-experts-keys-to-success-from-ge/">highest performing companies in every industry</a>, <a href="https://hbr.org/2020/11/how-apple-is-organized-for-innovation">like Apple</a>, adhere to the principle of Experts leading Experts, recognizing that it’s easier to train an expert to manage effectively than to train a manager to be an expert. Research supports this approach across various industries, from business to healthcare and education. Deep expertise leads to better decision-making, effective communication, and career growth.</p>
<p><a href="https://hbr.org/podcast/2023/07/the-best-leaders-are-also-technical-experts">Amanda Goodall’s research</a> explains why “if your boss understands the nature of the work, then they can actually help you. They can assess you well, and they can encourage you in the right direction to advance in your career, and that is a very important element for job satisfaction.”.</p>
<h1 id="metrics-cant-replace-expertise">Metrics can’t replace expertise</h1>
<p>Metrics are important. They create feedback loops to help you set direction and identify progress, they provide diagnostic information to understand side effects, they provide signals to spot unintended consequences or exciting new opportunities.</p>
<p>Leaders and engineers alike need metrics. They create transparency and enable a shared understanding. Gergerly Orosz and Kent Beck’s response to McKinsey’s report has great examples of how engineers can use metrics effectively.</p>
<p>The problem with McKinsey’s report is not with the need for metrics. It’s not even whether they are using the right or wrong metrics. The problem is it misses the need for capable engineering leads and engineers who have developed the deep instinct to understand how to use a metric appropriately, when to take it seriously and when to ignore it. And most importantly, how to develop that intuition throughout the engineering organization. Data is fantastic but you need humans who can use data to make good quality decisions.</p>
<p>Think about all the things your car measures. Speed, fuel, revs, oil, temperature etc. These are all indicators that your car is working correctly. And you want to keep them within reasonable boundaries and act on them when they move outside.</p>
<p>The problem is when metrics are used as targets not indicators. Targets destroy culture. As a driver you need those indicators, but reporting them up to management so they can set targets on them removes autonomy and judgment. When you are trying to get somewhere, you don’t make having a fuel tank of petrol your target. It’s nonsensical, it would result in people doing stupid things. As the driver you have enough knowledge and experience to know when to fill up your car. And you don’t just use the fuel gauge to make that decision. You are using a lot of other information too (price of fuel, optimum point to refill, whether a detour is required).</p>
<p>So a big part of management, and measurement, is knowing what measures are indicators (or signals) and what they indicate. And trusting others to know how to act when they look wrong.</p>
<p>And that requires expertise, experience, intuition and instinct. And you cannot replace those things with measures.</p>
<h1 id="a-call-for-change">A call for change</h1>
<p>Engineering has caught the attention of the C-Suite and as the engineering community we must equip organisations with the tools and thought leadership to:</p>
<ul>
<li>
<p>Cultivate sustainable cultures of engineering excellence</p>
</li>
<li>
<p>Foster the growth of future engineering leaders</p>
</li>
</ul>
<p>To achieve this, we must elevate Engineering Leadership to a critical role within the C-Suite, empowered to drive business impact and serve as role models for aspiring engineers.</p>
<p>Breaking down barriers and dispelling stereotypes is crucial. Negative perceptions about engineers and a culture of contempt for management hinder potential engineering leaders. Having navigated this journey myself, I understand the fulfillment and impact of empowered engineering leadership roles. I’m here to encourage and support others on this journey.</p>
<h1 id="leadership-over-metrics">Leadership over metrics</h1>
<blockquote>
<p>While there is value in the items on the right, we value the items on the left more.</p>
</blockquote>
<p>Engineering excellence is deeply rooted in culture and leadership. McKinsey’s metrics-based approach, while well-intentioned, risks repeating historical mistakes. We must focus on nurturing cultures of excellence, where experts lead experts, and metrics complement expertise.</p>
<p>The engineering community has a unique opportunity to shape the narrative and empower organizations to thrive in this post-digital world. It starts with leadership, culture, and a shared commitment to investigate and understand what truly drives engineering excellence.</p>Peter Gillard-MossMcKinsey’s report “Yes, you can measure developer productivity” has caused a strong reaction in the engineering community. As an experienced engineering manager I have made it my job to read McKinsey’s report and the response to it.Performance Management the Trading Card Way2023-08-06T15:20:00+00:002023-08-06T15:20:00+00:00http://petergillardmoss.github.io/blog/2023/08/06/performance-management-the-trading-card-way<p>Many software engineers are put off of leadership. And performance management is one reason why. Yet growing other engineers can be one of the most rewarding aspects of leadership. It’s critical engineers take on management responsibilities because research shows that <a href="https://hbr.org/podcast/2023/07/the-best-leaders-are-also-technical-experts">the best leaders and managers are technical experts</a>. And their effectiveness in performance management is the main reason:</p>
<blockquote>
<p>If your boss understands the nature of the work, then they can actually help you. They can assess you well, and they can encourage you in the right direction to advance in your career, and that is a very important element for job satisfaction - <a href="https://amandagoodall.com/">Dr Amanda Goodall</a></p>
</blockquote>
<p>In my role as a technology leader, I avoided the concept of performance management for far too long. The very word <em>performance</em> put me off. “People aren’t engines,” I thought “you can’t simply tune them in order to get better outputs”.</p>
<p>Eventually it became a responsibility I couldn’t avoid. I couldn’t rely alone on motivating presentations, visions and sharing my past experience. I realised great engineering organisations had leaders who had the skills to guide and grow their engineers. And that was what people meant when they said “performance management”.</p>
<p>Over the years I found an approach to performance management built on a philosophy of cultivation and quality research and it has become one of the most satisying parts of my job!</p>
<h1 id="performance-management-is-coaching">Performance management is coaching</h1>
<p>People who make great performance managers are good coaches who can guide their team through their journey as they grow. Performance managers need to answer two key questions for their people <em>“What do I need to do to be more successful?”</em> and, <em>“What does my future look like?”</em>.</p>
<p>To help people answer these questions they need a performance manager that can act as a <em>credible</em> guide. Where guidance means providing direction, feedback and as objective assessment as possible. I’ve made the word credible bold for a reason. If the manager providing guidance isn’t seen as credible by the person they are managing then they will not be successful. This was something clearly demonstrated by Dr Goodall’s research.</p>
<p>For most people they need to trust their guide is credible in answering the two key questions for:</p>
<ul>
<li>Their role/craft and seniority/grade</li>
<li>Their contribution to the team and organisational mission</li>
</ul>
<p>Putting this in the context they must have someone who would give them credible guidance when asked:</p>
<ul>
<li>What do I need to do to be more successful as a senior (grade) software engineer (role)?</li>
<li>What does my future as a software engineer look like?</li>
<li>What do I need to do to help our team be successful for our customers?</li>
<li>What does my future in this team and this organisation look like?</li>
</ul>
<p>These things are all very closely related. Being more successful in your role needs to be aligned to being more successful for the team and organisational mission. And the success of the mission is dependent on growing people in their craft and seniority.</p>
<p>My first bit of advice, for any new engineering managers, is to focus on how you provide credible guidance to engineers in their role and grade.</p>
<h1 id="build-shared-awareness">Build Shared Awareness</h1>
<p>We have established you as a credible guide whom the software developer sees as a valuable mentor for their role and grade. The first step in your role as a coach is to foster a mutual understanding and shared awareness with the developer. This involves aligning your perspective (as the guide) with that of the developer, ensuring a shared view of the role and grade expectations and how the developer is doing against those expectations.</p>
<p>This shared view has to also be aligned with the organisation’s view too. The guide and developer can’t decide their own version of role and grade. If they are going to talk about what makes a good software engineer, then it needs to be based on your organisation’s collective understanding of a good software engineer.</p>
<p>This shared awareness, between the organisation, the guide and the developer, forms the foundation of the relationship. The role of the guide is essentially this! To guide the developer towards this shared understanding by providing direction, feedback and assessment.</p>
<p>So how do we achieve this?</p>
<h1 id="the-trading-card-game">The Trading Card Game</h1>
<p>When it comes to growth everyone must choose their own path. Whilst this is incredibly empowering, it can also be overwhelming if we don’t have the tools to be able to do that.</p>
<p>This is where the idea of the trading game comes into play. People who play trading games have large collections of cards. These cards provide a limitless number of possibilities and styles. However, the player can’t use all the cards. They select those which best suit their playing style to make their own customised decks. A collector has to select from the cards they have whilst they work hard to collect the cards they really want to become a stronger player.</p>
<p>This is the same with growth. The cards are skills and competencies. Just like trading cards, we have a very large selection of competencies and no one person can have them all! And just like trading card games, the developer has to collect new competencies. The developer collects the card by demonstrating that they have consistently applied that capability on their team.</p>
<p>In Trading Card games there are base sets which provide a ready mix of cards required to play. These base sets are like roles (data engineer, software engineer, back-end, UI etc.). You don’t need every single skill as long as you have the right mix to enable you to play the role.</p>
<p>Roles aren’t the only way of understanding collections of cards. As people take on more responsibilitiy there are other sets of general <em>leadership</em> skills which overlap across roles. For example Stakeholder Management is important for many different roles (tech leads, business analysts, designers). These overlaps can help us see which competencies are more powerful and worth investing in acquiring. Just like in a card game.</p>
<p>Skills, roles and leader expectations are what provide us with the collective understanding of what people need to be successful in their teams. Guides and Thoughtworkers can use these to begin their journey of shared awareness.</p>
<h1 id="dont-try-to-be-good-at-everything">Don’t try to be good at everything</h1>
<p>Many people look at all of these expectations and quickly become overwhelmed. If you’ve ever tried to complete the skills section of your LinkedIn profile you’ll know what I mean! The more you read the long list of skills and what each one requires it can quickly lead to a feeling of lacking.</p>
<p>It’s essential that these expectations are not seen as targets. People musn’t feel that the goal is to collect the complete set. Nobody (really, nobody) has the complete collection. Everybody has a group of competencies they are strong in and others they will never develop beyond the basic needs. This is why teams are special. If we had a base set of capabilities that everyone had to achieve, then we would become mediocre.</p>
<p>This where your role of the guide is very important. By building this shared awareness you learn which areas the developer is strong in and should accelerate to become more impactful and which ones are fine to leave. There is no hard rule that says you are “now playing the role” or that you “achieve this grade” when you reach the right level 60, 70 or 80% of the competencies, so we have to use our own judgement.</p>
<h1 id="understanding-the-level">Understanding the level</h1>
<p>In trading card games, two characters may have the same skills but not at the same level. Scaling skills is not easy, and a model like the Dreyfus model can help (novice, advanced beginner, competent, proficient, expert).</p>
<p>The easiest way to do this is through a self-assessement. For each competency start at the bottom. Ask “what would a novice doing this look like?” and then ask “do you at least three examples where you did this in the last 6 months?”. If you get three convincing examples then ask the question again, but for the next level up.</p>
<p>Once you’ve done this quick assessment you can decide together if the competency is a strength, if it needs to be a learning priority or perhaps they don’t need to focus on it at all.</p>
<h1 id="its-a-team-game">It’s a Team Game</h1>
<p>Not only is nobody good at everything, it would be quite wasteful if they were. This is a team game and the competencies in the team need to complement each other.</p>
<p>The next level of shared awareness is knowing the shape of the whole team and where the developer fits in. For example, Front End Web Development may be a competency for the Engineer role. Whilst it is reasonable to expect most developers in the team to do some front end development, there may be some team members who are particularly good at it and can lead. However, there may be less expertise in Pipeline Design and Automation. This would trigger a conversation about whether Pipeline Design should be a learning priority, given the gaps in the overall team’s competencies.</p>
<h1 id="context-is-key">Context is Key</h1>
<p>Players need to choose which deck they’ll play with based on the type of gameplay and the style of their opponents. This is true for us too.</p>
<p>Beyond knowing the team, we need to understand the context of the organisation. And we need to match the desired competencies to that context. If you are working for a trading company and building a high frequency trading platform, then you’d want to be focussing heavily on Performance and scalability engineering. Front End Web Development is probably irrelevant for this context.</p>
<p>Likewise, working in heavily regulated environments might mean everyone needs to focus on Change and release governance and compliance to be successful.</p>
<p>This builds an even broader shared awareness, helping the developer focus on the essential competencies they need to be more successful.</p>
<h1 id="a-more-accurate-view">A more accurate view</h1>
<p>Doing a self assessment, with the aid of a guide, is a really great starting point. But self-assessments suffer from bias. Dunning-Kruger plays a big role and we often over assess ourselves when we are starting to learn and under assess ourselves as we become more of an expert.</p>
<p>To help build a better shared awareness we need to find ways of including the perspectives of the people we work with day-to-day. This is why we collect feedback.</p>
<p>We can use feedback to provide evidence that we are demonstrating a competency at a specific level. The more feedback which correlates to the competency, the more confident we can be.</p>
<p>It is also important to get feedback from a range of roles. For example, whilst a Project Manager may not feel qualified to tell a developer if they are good at Test Driven Development, they will have a much more relevant opinion on how well they do Estimation. If we only hear from developers, then we may miss these insights.</p>
<h1 id="playing-the-game">Playing the game</h1>
<p>Remember the two key questions:</p>
<ul>
<li>What do I need to do to be more successful (in my role and grade)?</li>
<li>What does my future (in my role) look like?</li>
</ul>
<p>By thinking in capabilities, and approaching it like a trading card game, performance partners have the tools they need to guide their developers through these questions. By bringing in the wider context of the team and the organisation it is possible to narrow down the wide range of competencies to the ones which would be most impactful to develop. This process creates a shared awareness between guide and developer which forms the foundation of their relationship.</p>
<p>These expectations won’t tell you everything though. There are many things which aren’t captured in the capability model which will still need to be explored. Hopefully though it will provide 80% of the picture with only 20% of the effort.</p>Peter Gillard-MossMany software engineers are put off of leadership. And performance management is one reason why. Yet growing other engineers can be one of the most rewarding aspects of leadership. It’s critical engineers take on management responsibilities because research shows that the best leaders and managers are technical experts. And their effectiveness in performance management is the main reason:How Agile scaffolds enable emergence2021-10-15T08:20:00+00:002021-10-15T08:20:00+00:00http://petergillardmoss.github.io/blog/2021/10/15/How-Agile-scaffolds-enable-emergence<p>Ann Pendleton-Jullian coined the metaphor of scaffolding when she was working on a case study of General Stan McChrystal’s transformation of JSOC. She wanted to contrast frameworks with scaffolds.</p>
<p>Frameworks are <em>“a complete structure, usually permanent, and gives forms to that which it supports, or encloses, or solves”.</em> Where as a scaffold acts as a <em>“temporary structure for supporting something until that something can stand on its own”</em>.</p>
<p>I’ve found myself in that place where an idea resonates and pulls on some underlying truth but I struggle with its application (does that put me in aporetic domain?).</p>
<p>When I’m stuck here I need to have a play. Take the ideas and throw them at something, see what fits and what falls. Often with a big wet slap.</p>
<p>As enabling emergence is a key quality of agile practice then surely, if I root about in the agile practices box I might find some things to throw?</p>
<p>Basically, this is the result of what I think might have successfully stuck.</p>
<p>Ann, along with others, have proposed <a href="https://cynefin.io/wiki/Scaffolding">five basic types of scaffold</a>. I’ll attempt to see if I can find agile practices which might fit these metaphors.</p>
<p><strong>Building scaffolds</strong> <em>enable building crews to construct new buildings and are erected with the knowledge of the final shape. When work is done they are taken down.</em></p>
<p>A good example of this is probably the Story. The articulation of value (<em>As a, I want, So that</em>) along with acceptance criteria (<em>Given, When, Then</em>) allow the building crew (in this case the developers) to find the best solution to meet the value. When the story is done (delivered) the card has served its purpose and is archived away.</p>
<p>An important quality Stories share with real building scaffold, is neither contain any implementation detail. Anything could appear within in them.</p>
<p>This also highlights why poor practices around stories can cause issues. A common anti-pattern is to turn them into implementation specific tasks or requirements. Rather than acting as scaffolds which enable solutions to emerge they become blueprints.</p>
<p><strong>Neural lattices</strong> <em>stay part of the structure but are transformed in the process. The example used is a Bionic Cardiac Patch which both enables the regeneration of cardiac tissue and also provides pacemaker technology which detects and corrects cardiac arrhythmia.</em></p>
<p>I feel this metaphor perfectly explains Continuous Delivery. The delivery pipeline both acts as a seed for the production software and provides the pulse and rhythm for change.</p>
<p>Practices like Test Driven Design and Fitness Functions may fit into this category. The test provides the initial scaffold which seed an emergent design. Whilst the tests themselves don’t form part of the final structure they remain in place checking for anomalies, like the cardiac patch.</p>
<p><strong>Skin grafts or nutrient lattices</strong> <em>work by providing the nutrients which activate the generation of new skin. As the skin reforms the scaffolding dissolves away, leaving no trace.</em></p>
<p>I would put a collection of techniques under here from User Journeys to Personas to Story Mapping to Domain Driven Design to Event Storming.</p>
<p>Unlike a building scaffold these activities don’t provide knowledge of the final shape and act as a container. Instead, like the skin graft, they provide the scaffold from which we generate ideas from.</p>
<p>We can start off with a user journey and plot out the key stories we need to deliver to start, or form a domain model which identifies the key parts of our language and relationships between data.</p>
<p>Once the exercises are done, usually using low fidelity tooling like white boards, they dissolve away leaving the real user journeys or the real domain model in place.</p>
<p>A common trap is to over analyse these stages and make them specifications. And a common frustration is often that these things have dissolved away leaving no permanent artefacts.</p>
<p>A common explanation for the lack of need of permanent artefacts, for example around domain models, is often that the code is living documentation. By seeing these activities through the metaphor of a nutrient scaffold adds weight to this reasoning.</p>
<p>Practices like walking skeletons and tracer bullets also fit under these types of scaffold. Again they provide the initial scaffolding for a emergent solution to grow from and then dissolve away as the solution takes over rather than being removed like building scaffold.</p>
<p><strong>Keystone scaffolds</strong> <em>provide foundations for other structures to be built on. Arches are the example used where the archway is built around the keystone. Removing the keystone can cause the arch to collapse.</em></p>
<p>These scaffolds I have really struggled to understand. Everything I throw at them slips off too easily or fits better somewhere else. Maybe because they become invisible and forgotten?</p>
<p>A few practices which make sense are things like Feature Leads, or Story Leads. Here these people act as knowledge keystones. Team’s learn that lack of continuity on a story or across features can cause the solution to collapse.</p>
<p>I also wonder if practices like source control, especially Configuration as Code and Infrastructure as Code count here. These are often keystones to enable emergence in a safe way. When teams unwittingly remove these practices they begin to experience configuration drift which fits with the metaphor of removing a keystone and higher structures slipping unexpectedly and uncontrollably.</p>
<p><strong>Shadow scaffolding</strong> <em>enables emergence across multiple interactions over long periods of time. Examples used are extreme sports infrastructure like training, peer group interaction, technology developments and apprentice type practices.</em></p>
<p>Practices such as pair programming, rotations and retrospectives sit here. They enable emergence through improved team dynamics.</p>
<p>I feel retrospectives done well are a perfect example of a shadow scaffold which enables emergence through reflection and improvement of processes. And yet, perhaps because they aren’t explicitly part of finding a solution, they become tempting to drop.</p>
<p><strong>Agile Frameworks vs Agile Scaffolding</strong></p>
<p>At the end of this I hit a realisation. Scaled Agile Framework (SAFe). Framework! <strong><em>Framework!</em></strong></p>
<p>Perhaps this metaphor sums up some people’s aversion towards SAFe? Would they rather employ agile as a set of scaffolds to be used to enable emergence? Where as SAFe is a framework to enclose and force structure?</p>
<p><strong>Any more for any more?</strong></p>
<p>This feels a good start to thinking in this way. I wonder if I’ve got them right or are there any interesting ones you think would fit the metaphor?</p>Peter Gillard-MossAnn Pendleton-Jullian coined the metaphor of scaffolding when she was working on a case study of General Stan McChrystal’s transformation of JSOC. She wanted to contrast frameworks with scaffolds.Agile is a decision making method not a project management method2021-05-19T08:20:00+00:002021-05-19T08:20:00+00:00http://petergillardmoss.github.io/blog/2021/05/19/agile-is-a-decision-making-method<p>Agile is primarily seen as a project management methodology but its real power comes when used as a decision making method. When used this way Agile becomes about finding ways to break down big complex decisions with high risk and uncertainty into small decisions with empirical feedback loops which are safe to fail.</p>
<p>When only used as a project management tool it simply takes that same big hairy high risk decision and creates a linear schedule of tasks broken into iterations. None of the advantages are realised and switching to agile will probably bring little benefit.</p>
<p>To make this switch Agile techniques moves delivery from something that happens post “business decision making” to melding decision making and delivery into one process. How else do you interpret “Customer collaboration over contract negotiation” and “Business people and developers must work together daily throughout the project” from the Agile Manifesto? Agile is all about collaborative decision making!</p>
<p>Yet all decisions aren’t the same. And therefore they shouldn’t be delivered in the same way. The type of decision determines which development and delivery techniques are appropriate. Not the other way around! I don’t switch electricity provider by breaking it down into vertical stripes by the value it delivers each end user (decrease cost of son watching television by 5%, decrease spoilt food due to outages). That would be a nonsense. Likewise you don’t develop and roll-out a novel vaccine based solely based on stakeholder requirements and deploy it world wide in one big bang. That would be deadly.</p>
<p>In the mid-60s thinkers such as C. West Churchman and Horst Rittel began to talk about how different types of decision needed different approaches. They distinguished between simple problems, which are easily identified and a solution can be discovered, verses wicked problems which have no consensus on the problem or the solution. More recently, the Cynefin framework expanded on these ideas to introduce four different problem domains (Clear, Complicated, Complex and Chaotic).</p>
<p>The Cynefin framework allows decision makers to decide which agile techniques and delivery approach are best suited based on the problem domain they are operating in.</p>
<h1 id="clear">Clear</h1>
<p>The Clear domain is where cause and effect are very clear and so is the solution. This blog is a great example. I want to publish a blog. I write a post, it appears on the internet. It is a known problem with a known solution. If you’ve ever setup one blog you know you could setup a thousand. I may have some requirements but nothing special to me. I just need to find something which meets my requirements and use it.</p>
<p>The main decisions are around ensuring the selected solution is delivered to time, budget and with adequate scope. These constraints are reasonably fixed and so we follow the classic process of collect and analyse the requirements, find a matching solution that fits, implement it pattern. Essentially waterfall.</p>
<p>Some solutions may be clear but that doesn’t make them easy to implement. This is often due to the size of the solution. In these situations, breaking the solution down into smaller parts which can be delivered independently is a sensible approach.</p>
<p>How you break that solution down depends on whether you can only establish value when the full solution is delivered or whether you can realise value in smaller pieces.</p>
<p>Agile techniques can be applied in these domains based around creating the appropriate decision points in response to size of the implementation and when you want to get return on investment. Delivering in these ways won’t be doing “true agile” and may look something more similar to what people call <strong>Water-Scrum-Fall</strong>.</p>
<h2 id="iterative-delivery">Iterative delivery</h2>
<p>Iterative delivery works well when you only need to realise value when the full solution is delivered and there is no point in delivering something partially done. By breaking the solution down into smaller deliverables this enables smaller decisions which can be solved and delivered independently.</p>
<p>A good example of this is installing a new kitchen. You start by building all the cabinets first and then check they position correctly. Then you may start installing electrics and pipework and check those. Then doors and fixing and check those. Then surfaces. Again checking. Then install appliances. Again a final check before producing a “snagging” list.</p>
<p>At regular intervals (iterations) you introduce decision points enabling you to Plan Do Check Act. With each iteration you ensure that the solution is being delivered according to plan and project managing as necessary.</p>
<p>Regular integration enables clear and regular decisions around delivery. For example, aligning schedules or responding to problems as they arise. Working iteratively simply makes those decision points as regular and often as is reasonable.</p>
<p>What you are not doing is delivering a usable kitchen with each iteration. What you are not doing is gathering requirements as you go along. What you are not doing is discovering new ways to solve the problem. All of those things are fixed. This is really important to understand. Because if you <strong>are</strong> trying to introduce those things as decision points, then you need to introduce them appropriately.</p>
<p>In software the equivalent is iterative development with continuous integration. Development practices, such as automated checks (or tests), provide scaffolding throughout the process which speed up integration and validation. You may even deliver in horizontal layers to fixed contracts (UI first, then backend, then infrastructure) if the requirements of each layer are very clear.</p>
<h2 id="incremental-delivery">Incremental delivery</h2>
<p>The solution is clear and there is a way of breaking it down so that you can deliver parts independently and gain value from them. There are two reasons you may chose to do this. It could be because you want to realise your return on investment as early as possible or you want to minimise disruption.</p>
<p>A domestic example of this is refurbishing a property. Rather than gutting the entire property in one go, you decide to split it into smaller sequential projects which are fully delivered. Essentially refurbish one room at a time. This means you get to enjoy the newly renovated room as soon as possible (early return on investment) and you don’t have to relocate during the process (minimise disruption).</p>
<p>This approach has many advantages for decision making. Rather than making all decisions up-front in one go they can be made between each increment. You can decide the theme for the master bedroom, renovate, then decide the bathroom. You can also change priority (swap the bathroom for the guest suite).</p>
<p>An incremental approach means you deliver a useable room with each increment. Before each increment you gather the requirements for the next deliverable. Once an increment starts those things are fixed. What you are not doing is discovering new ways to solve the problem.</p>
<p>In software delivery this is when we typically slice solutions “vertically” delivering a slice of each layer as part of the overall solution. Other ways are slicing by segments such as a persona or market and delivering to them in series one at a time. Practices such as Continuous Delivery put the decision for “going live” into the hands of the business owners.</p>
<p>One challenge with vertical slicing is that it can easily slip into becoming complicated if the way to slice is non-obvious.</p>
<h1 id="complicated">Complicated</h1>
<p>These are situations when cause and effect exist but are not clear. Whilst the problem is well understood the requirements and the solution itself are only discoverable with some diagnosis and expertise. This introduces risk due to known unknowns.</p>
<p>Finding a good solution may take some effort but once you have enough information to land on one you can commit to it. In this domain decisions are much more around solution discovery and design through the creation of options and deciding on the trade-offs between them.</p>
<p>Using the domestic example again, this may be that you know you want more space for a growing family. With the help of experts, from financial advisors to property agents to architects you can gather the requirements and generate various options from a range of different homes to move into to different ways to extend your existing property.</p>
<p>Each solution will have different pros and cons. And each one will require completely different approaches. Trade-offs will need to be made, alternatives generated.</p>
<p>From a decision making perspective what you want is to maximise optionality and bring risks forward as much as possible. These risks are your known unknowns and are usually easily verifiable once detected.</p>
<h2 id="emergent-solutions">Emergent solutions</h2>
<p>This is the problem domain Agile was born in and showed its superiority over waterfall methods. Due to the highly malleable nature of software, requirements gathering, solution discovery and production delivery could be bundled together rather than as separate phases.</p>
<p>This is much harder with physical solutions because decisions aren’t easily reversible. If I sell my home I can’t move back if things didn’t work out.</p>
<p>In contrast, with software we can pull the risk forward by making decisions reversible. This enables the discovery of requirements and solutions to be generated through an empirical process: through small feedback loops of trial and error.</p>
<p>To achieve this, Agile methods break problems down into the smallest pieces of value, deliver just enough solution and verify it against real feedback. Based on the feedback, requirements are refined and the solution is adjusted. This is the purpose of a showcase.</p>
<p>This process reduces the burden on up-front decision making and heavy analysis in preference to enabling a solution to emerge based on proof (empirically). This enables you to push as many decisions to the last responsible moment because you can improve the solution as you go. This creates a very large scope for change.</p>
<h2 id="incremental-delivery-1">Incremental delivery</h2>
<p>These cycles of feedback are bundled into iterations and released to production in increments in a similar manner to the Clear domain. However, in the Clear domain the solution is fixed and we break the solution down into smaller increments. In the complicated domain the solution is emergent and we break the problem down into smaller problems to solve.</p>
<p>This means each iteration includes decisions around the problem. The first decision is whether enough of the problem has been solved to move onto the next problem. This is done based on the feedback from each release. If the problem is solved, the decision is which problem to tackle next.</p>
<p>As with the clear domain the decision of when to release value to the end users is held by the business stakeholders. One difference is that because the solution isn’t clear this decision is based on reaching an acceptable amount of certainty. Have we solved enough of the problem to bring some value?</p>
<p>The classic metaphor for this approach is the “skateboard, scooter, bicycle, motorbike, car” evolution.</p>
<p>Agile techniques optimise for establishing that certainty as early as possible. This is why “pushing to production” is an important tenant as the greatest certainty comes from real production users.</p>
<p>Fidelity is often used as a technique for deferring decisions. This means that more naive solutions are chosen early on and are replaced with something more feature rich latter as the solution becomes more sophisticated. This is referred to as Evolutionary Architecture. A common example is using a flat text file over a database. This defers the decision and effort about “what is the right database to use” until the real need arises. This doesn’t mean a sacrifice in quality however.</p>
<p>To achieve this the delivery process invests heavily in maximising malleability and feedback loops. Practices from Pair Programming to Test Driven Development to Continuous Delivery to Architectural Fitness Functions prioritise scaffolding for fast feedback and making change as cheap and easy as possible.</p>
<p>In highly advanced teams the concepts of iterations and increments can almost completely disappear to the point where new features are released to users multiple times a day. This creates an environment of very rapid decision making.</p>
<h1 id="complex">Complex</h1>
<p>Not all problems exist in the ordered world. In the unordered domain cause and effect are difficult to understand or are even in a state of flux.</p>
<p>Simply put these are not predictable. Whilst you do know the desired outcome you want (the effect) you neither know the cause of the problem nor the solution to it. No amount of analysis or debate will bring you to a solution. You won’t know what will or won’t bring about your desired effect until you try it for real. You need to probe first, then sense, then respond.</p>
<p>From a decision making perspective reversibility becomes crucial. Any change needs to be small and safe to fail. A common trap is trying to make decisions on Complex matters by applying the same techniques from the Clear or Complicated domain. Instead of achieving the desired reversibility you end up over committing either to implementing a fixed solution or to solving a fixed problem.</p>
<p>The first time many teams are impacted by the complex domain is when they experience continuous changes in priorities or requirements. This is also part of Agile’s original allure with the promise of <a href="[https://agilemanifesto.org/">“Responding to change over following a plan”</a>. By delivering in incremental slices, which realise the value as early as possible, when factors outside the system result in a need to pivot the team are able to respond.</p>
<p>In reality many teams struggle with priority changes and find them a source of frustration. This is because change in priority is the symptom not the cause. Priorities change due to complexity. People realise that the solution isn’t delivering the desired outcome because they are solving the wrong problem. So they switch to a different problem to solve. Or something else in the environment - loss of customers, competitor launch - forces them to react.</p>
<p>The pain teams feel from this reprioritisation are the result of trying to force the complex world into the ordered world of software delivery too early. In the complicated domain the solution is discovered. In the complex domain the problem is.</p>
<p>Some organisations have managed to successfully exploit Agile when dealing with the Complex domain. Many of the techniques around Agile, reduction in the cost of change, incremental delivery etc. provide a strong foundation for delivery process which enables emergence. To not only discover solutions but discover the problems too.</p>
<h2 id="shift-to-experimentation">Shift to experimentation</h2>
<p>Decision making has now shifted away from implementing solutions to focus on discovering how to achieve desired effects (outcomes).</p>
<p>This is typically done by identifying and designing experiments and observing the outcomes. The results of the experiment inform your next decision which is usually to commit to more investigation, switch to another hypothesis or abandon the area altogether. At some point enough certainty is established to decide to invest and deliver a solution (usually in the complicated domain).</p>
<p>Teams achieve this by adding the capability to experiment both outside their software through prototypes or proof of concepts and within production software. At an infrastructure level this might include things like A/B testing with sophisticated analytics and telemetry. At a code level it may be things like feature toggles or entire experimental microservices.</p>
<p>One challenge for teams is the switch between the complex domain and the complicated. In the complicated domain the solution evolves and becomes more sophisticated as new requirements are discovered. Because the problem is known and fixed there is a commitment to the solution as it evolves. Therefore a focus on cross functional criteria such as scalability or operability is critical.</p>
<p>In the complex domain the problem isn’t known. It doesn’t make sense to commit to any one solution as the goal is to discover the problem. The solution itself should be readily abandoned. This means framing decisions based on the desired outcome not the problem.</p>
<p>Teams can find this very hard. A lot of Agile practices focus on aspects of quality which may hinder their ability to “Probe-sense-respond”. Yet there is a point when the problem will switch into the complicated domain and these concerns will become important. Detecting that moment is not easy. This grey area between complex and complicated can create challenges around things like effective management of technical debt.</p>
<h1 id="crossing-the-domains">Crossing the domains</h1>
<p>Problems aren’t fixed in one domain. As we learn more the problem can shift. A great example is infection control. Go back a few centuries and the cause and effect relationships between infection and disease was unknown. Many guesses were made until germ theory finally discovered the relationship. With this understanding we were able to identify different diseases and their causes moving them one by one into the complicated domain. This gave scientists the opportunity to identify common causes of disease and find solutions to tackle them. When Joseph Lister developed sanitation in medical settings he was solving a complicated problem. Now sanitary practices (washing and cleaning wounds with antiseptic wipes) are clear to everyone.</p>
<p>The same is true for business challenges and software development. As we learn what works overtime the domains shift. A problem which was Complex a few years ago may now be Complicated. And several years later becomes Clear. There were times before version control, and for many years <a href="https://ask.slashdot.org/story/05/01/20/1657218/who-doesnt-use-source-control">its benefits were much debated</a>. Understanding this shift is one function of Wardley Mapping.</p>
<p>As a methodology Agile sits in the space (liminal) between complex and complicated. This means there isn’t a sudden line you cross and therefore teams need to be able to shift between the two.</p>
<p>Likewise, when you vertically slice from a desired outcome to a solution you’ll find that the whole problem doesn’t sit in one domain. Different parts of the problem will sit in different domains. For example, you may be working on a complicated problem but you will still need to chose a version control system or a database, decisions which sit in the Clear domain.</p>
<h1 id="helping-to-decide">Helping to decide</h1>
<p>Agile techniques can be used effectively in each domain if they are applied appropriately and in a way which supports decision making. To help here is a quick summary:</p>
<ul>
<li>
<p><em>Clear</em>: known problem known solution. Prefer fixed roadmaps which break <strong>solutions</strong> down into iterative and/or incremental release.</p>
</li>
<li>
<p><em>Complicated</em>: known problems, unknown solution. Prefer roadmaps of which break <strong>known problems</strong> down into incrementally releases. Focus on emergent solutions which are delivered iteratively.</p>
</li>
<li>
<p><em>Complex</em>: unknown problems, unknown solution. Focus on experiments which enable <strong>problems to emerge</strong>.</p>
</li>
</ul>
<p>Don’t jump straight to a delivery approach. Instead understand which domain best fits your decision. Then align your delivery method to fit. This will create the right opportunities which will improve decision making and overall delivery.</p>Peter Gillard-MossAgile is primarily seen as a project management methodology but its real power comes when used as a decision making method. When used this way Agile becomes about finding ways to break down big complex decisions with high risk and uncertainty into small decisions with empirical feedback loops which are safe to fail.Signals are just as important as indicators2021-05-04T08:20:00+00:002021-05-04T08:20:00+00:00http://petergillardmoss.github.io/blog/2021/05/04/signals-are-just-as-important-as-indicators<p>As a runner I am very used to setting goals and tracking my progress towards them. With the rise of fitness watches and apps like Strava I have more information available than ever. Some of this information provides key indicators which relate to my goals.</p>
<p>Yet not all information provides a measure of performance. Sometimes it is nothing more than a weak hint towards more complex things. If you become overly focused on your performance indicators then these signals are easily overlooked. Whether your goals are athletic or business the consequences can be disastrous.</p>
<p>This is what happened to me in 2015. All my performance indicators pointed to a personal best and a sub-three hour marathon. Then a couple of months before the event I had to bail out due to an injury. It took over two years to fully return to form. When I look back, all the signals were there. Why did I ignore them?</p>
<h1 id="setting-goals-and-targets">Setting goals and targets</h1>
<p>Goal setting for runners isn’t too different from organisational goals. I generally set my sights on a future point and give myself an ambitious performance target. This might be a sub-20 5k or a sub-3 hour marathon. These performance goals are my lagging indicators.</p>
<p>I then take my goals but rather than a strategic business plan, I design a training plan which track different key indicators with milestones which are more immediately in my control. These are my leading indicators. For example, when running a marathon I focus on improving my endurance. I can measure this through my overall weekly distance because and slowly increasing it until I hit an optimum amount (about 100km). If I am after a good 5k I will swap milage for speed and instead I will focus on intervals. This gives me short term aims such as “increase weekly distance from 40km to 50km in four weeks” or “improve average 800m interval time from 4:00 to 3:50”.</p>
<p>I also have a collection of other leading indicators I know affect performance such as my optimal weight.</p>
<p>By creating a training plan which focuses on these leading indicators I can feel confident that I am making progress towards my goals and if I’m not then I can make the necessary adjustments.</p>
<p>This isn’t too different from organisational metrics where a lagging indicator might be Revenue but you focus the business on something more near term like “number of customer referrals”.</p>
<h1 id="performance-indicators-alone-will-mislead">Performance Indicators alone will mislead</h1>
<p>This is exactly what I did in 2015. I was tracking both my distance and speed indicators and, if anything, was slightly ahead of plan. I was looking forward to the event with confidence. Then one day disaster struck when less than two kilometres into my run I experienced an agonising pain in my right knee.</p>
<p>I rested for a few days but when I tried to return to running the pain came back. So I rested longer and tried again. Same thing. Then the pain got worse, spreading to my lower back. Eventually I was diagnosed with a knee bursitis and sciatica. A deadly combination which took two years to fully eradicate.</p>
<h1 id="signals-can-be-warnings">Signals can be warnings</h1>
<p>The purpose of this post is not to share details of my injury or garner sympathy. The point is that athletes, like organisations, face threats and these threats aren’t always external, from your competition. The greatest threats are often internal and caused by your own decision making. For athletes this is injury. The same is true for organisations.</p>
<p>My poor decision making was obvious with hindsight. All the signs were there. I just ignored them because I was too focused on my indicators. For two years I kicked myself for my foolishness.</p>
<p>Here’s a simple example. The first thing my physio did was put me on a running machine and then made me balance on each leg. Straight away they said “when you run your right leg sweeps outwards. Your left leg is overdeveloped whilst key muscles in your right leg have atrophied”. My glutes and quads had lost strength, and other muscles were compensating. This was putting my biomechanics off, causing too much movement in my hip and knee.</p>
<p>Also, I had lost flexibility in my hips. This tightening was affecting my biomechanics. The two were in a vicious loop. Tight hips caused my body to compensate which lead to poor muscle development which lead to tighter hips.</p>
<p>To add insult to injury my physio told me that this was entirely predictable. Ninety percent of their patients are being treated for injuries from similar causes. I pointed out I had ten years of experience and hadn’t suffered any problems before. To which they simply replied “you got lucky”.</p>
<p>There was nothing in any of the indicators I use for my goals that told me this. Despite being one the single most significant factor preventing my success. Worse than that my indicators where giving me a false sense of security as I over reached them.</p>
<p>Yet, what the physio told me made perfect sense. The pain had actually started in my hip, several weeks before, not in my knee. And even before that I had lost a lot of flexibility in my legs. These were weaker signals both of which I ignored. I’m used to mild pain during training so I dismissed them with a simple heuristic mis-categorising them as usual training aches. I had also noticed the difference size of my quads, but again, had ignored that too. My wife had even committed on how I was walking strange.</p>
<p>Eventually these weak signals developed into strong ones: debilitating pain in my knee.</p>
<h1 id="act-on-weaker-signals">Act on weaker signals</h1>
<p>I realised that understanding threats and how to spot weaker signals and act on them before they become strong was critical to long term success. This is true in business and software development. With such a focus on short term results we need to keep a careful eye on signals which may pose a future threat.</p>
<p>Injury and over training are two key threats to an athlete’s performance. A strong signal of overtraining is fatigue. This is measurable through my heart rate and shows up when I struggle to push into higher zones. By this point I know I have overtrained and performance has already suffered.</p>
<p>To prevent reaching that point I look for a weaker signal which I can act on earlier. If I notice that too many sessions have been in high heart rate zones, and not enough in lower zones, then I can easily predict where I will end up. I can take immediate action.</p>
<p>Spotting the weaker signals prompts a decision. This is true in organisations too. We can see relationships between weak signals like customer queries for issues which, overtime might turn into stronger signals such as customer complaints. Weaker signals such as employee motivation or engagement might turn into attrition.</p>
<p>The same is true in software development teams where we may look at things like code complexity as a weak signal of defects. If we can spot the weaker signals, and act on them, we can prevent them becoming stronger.</p>
<h1 id="turn-signals-into-preventative-measures">Turn signals into preventative measures</h1>
<p>Keeping an eye on signals is one thing. But once cause and effect are established it is more useful to add preventative practices by “shifting left”. Rather than waiting for injury signals to appear and then take action I do targeted stretching and strength training and incorporate these into my training plan. To avoid overtraining I add in rest days and weeks. These are both measurable and form leading indicators against injury and overtraining.</p>
<p>To achieve this I’ve essentially added new objectives. “Stay injury free” and “Maintain good physical energy”. Again, these are measurable. And my practices contribute to achieving those outcomes.</p>
<p>As a side-effect, now I have those outcomes I am much more attuned to any weak signals which might hint that I am missing something.</p>
<p>The same is true in business. We can define objective which describe the business running in a healthy state and listen to the signals.</p>
<p>In software development we also know which signals indicate a healthy team. Whether that is ease of deployment, levels of technical debt, de-coupled architecture, time spent on manual tasks. All of these things are weak signals which, if left, turn into strong ones which threaten the teams ability to deliver.</p>
<p>Lastly, having an outcome doesn’t mean that I assume all signals are now captured as part of a new indicator. I still need to stay tuned to my body and how it is responding to the training schedule. This includes external factors which can affect my training (like weather or social events). Some can’t be measured by a device (like pain or feelings of positivity) but some can. For example, I look at trends in my heart rate. By correlating it with my pace I can quickly spot signs that I might be over training.</p>
<p>By using both indicators and signals I become a better runner. And a better leader too. Signals can come from many places and inform many different things. By helping teams become attuned to them and understand how to act on them we can all improve our performance.</p>Peter Gillard-MossAs a runner I am very used to setting goals and tracking my progress towards them. With the rise of fitness watches and apps like Strava I have more information available than ever. Some of this information provides key indicators which relate to my goals.A method which learns faster improves faster2021-03-26T08:20:00+00:002021-03-26T08:20:00+00:00http://petergillardmoss.github.io/blog/2021/03/26/a-method-which-learns-faster-improves-faster<p>When Satya Nadella took over as CEO of Microsoft in 2014 it was seen as a company heading towards irrelevance. Not only had it missed out on mobile and social but its core market was being attacked on multiple fronts by Apple and Google.</p>
<p>Many credit Nadella for pulling off a historical corporate turnaround. Nadella focused on cultural changes to Microsoft which, over five years rewarded the company with a $1 trillion valuation and placing it amongst the wealthiest publicly listed companies in the world.</p>
<p>In an <a href="https://www.bloomberg.com/features/2016-satya-nadella-interview-issue/">early interview</a> Nadella talked about changing Microsoft from a “know-it-all” to a “learn-it-all” culture. “Take two people, one of them is a learn-it-all and the other one is a know-it-all, the learn-it-all will always trump the know-it-all in the long run, even if they start with less innate capability” he said.</p>
<p>Not all companies pull out of the same nosedive Microsoft experienced. Many successful companies fail to respond to big changes in their environment and never recover.</p>
<p>We often assume that this is because the company ignores the problem or fails to respond. Donald Sull researched <a href="https://hbr.org/1999/07/why-good-companies-go-bad">why good companies go bad</a> and found that this wasn’t the case. Companies identified the trends and threat and moved rapidly by investing heavily in countering them. They were full of action. Unfortunately, their action was based around more of the same. He described this phenomenon as <em>Active Inertia</em>.</p>
<blockquote>
<p>Active inertia is an organization’s tendency to follow established patterns of behavior — even in response to dramatic environmental shifts. Stuck in the modes of thinking and working that brought success in the past, market leaders simply accelerate all their tried-and-true activities. In trying to dig themselves out of a hole, they just deepen it.</p>
</blockquote>
<p>In my previous post I talked about the relationship between Method, Product and Performance. I used the example of how Dick Fosbury transformed the High Jump sport with his new method called the <em>Fosbury Flop.</em></p>
<p>Deming said “<a href="http://www.curiouscat.com/management/deming/bestefforts">Best efforts are not enough, you have to know what to do.”</a>. Active inertia is the corporate equivalent of working harder on the scissor method when all your competitors are mastering the Flop. Essentially working harder at the same method.</p>
<p>This means there are two main concerns for any product or service to answer. The first is to improve performance by focusing on the method and ensuring that it is fit for the product we are trying to produce for our customers.</p>
<p>The second is to focus on knowing how to improve. Customers expect continuous improvement. They expect you to deliver solutions that they didn’t even know they wanted. If you don’t, then when your competitors do, they will switch. And if your competitors don’t but do just as well as you then your customers have little to lose so may still switch.</p>
<p>Essentially our method must be fit for two dimensions:</p>
<ol>
<li>Does it meet the customer’s needs today?</li>
<li>Can it adapt to meet the customer’s needs tomorrow?</li>
</ol>
<p>The first one assumes a fixed target that we must reach. We know the customer’s demands and we simply improve our method to improve quality, reduce defects, reduce costs and price. Essentially to become increasingly lean.</p>
<p>The second one assumes a moving target. One where our method must include sense checking and the ability to adjust and change. Essentially to become increasingly agile.</p>
<h1 id="the-same-method">The same method</h1>
<p>The difference between Nadella’s vision of Microsoft and companies which suffer from <em>Active Inertia</em> is the recognition that both questions must be answered by the same method.</p>
<p>Companies which suffer from Active Inertia believe that they can answer the second question through their current method of working. Donald Sull describes this like countries who fight new wars with the tactics from the last war. They believe their previous victories grant them great insight and advantage. Essentially they are “know best” companies.</p>
<p>Companies which focus on building a single method to answer both are “learn best” companies. They realise that the existing method can only be improved from new learning. Not from previous experience. And that must be built into the method itself.</p>
<p>To understand how well the method achieves this ask yourself: how fast can we go from having an idea to improve to successfully embedding that idea across the company?</p>
<h1 id="learning-at-speed">Learning at speed</h1>
<blockquote>
<p>The ability to learn faster than your competitors may be the only sustainable competitive advantage. - De Geus</p>
</blockquote>
<p>Having a method which can respond to ideas for improvement is not enough. It must actually learn. Companies which suffer from active inertia work under a fatal assumption. That their ideas will work. This causes them to focus on delivering ever bigger change faster.</p>
<p>A learning organisation doesn’t just focus on changing its method at speed. It focuses on rapidly eliminating the bad ideas from the good. They assume nothing about the validity of each individual idea. All of them are equally likely to be wrong.</p>
<p>Whilst they target ambitious change they do so by always keeping change small. This is because big ideas are full of small ones. And even the small ones need validating to avoid wasting time on committing to them.</p>
<p>Which means a key metric of the method is: how fast can we go from having an idea to validating whether it is worth committing to?</p>
<h1 id="learning-methods">Learning methods</h1>
<p>There are dozens of different methodologies which all achieve both of these things. The list includes Agile, Design Thinking, Systems Thinking, Lean Startup’s Build, Measure, Learn, Lean’s Plan Do Check Act, Improvement Kata’s or Observe, Orient, Decide, Act, Hypothesis Driven Development.</p>
<p>All of these methods have learning built in. All of these method rely the systematic observation, measurement, experimentation, formulation, testing, and modification of hypotheses. They are all essentially applications of the scientific method.</p>
<p>In all of these methods experiments are kept as small as possible. A succession of rapid loops gradually increasing in scale. They do not leap from one small experiment to one giant change.</p>
<p>A method which leaps from small to large is not learning. It is simply a “know all” method with the facade of experimentation. It isn’t experimenting to learn but experimenting to confirm existing biases. It learns nothing.</p>
<p>There are three things to conclude from this:</p>
<ol>
<li>Improving the method improves the product which improves performance.</li>
<li>A method which learns faster improves the product faster which improves performance faster.</li>
<li>They are the same method.</li>
</ol>Peter Gillard-MossWhen Satya Nadella took over as CEO of Microsoft in 2014 it was seen as a company heading towards irrelevance. Not only had it missed out on mobile and social but its core market was being attacked on multiple fronts by Apple and Google.