Agile Product Development

Team Leadership and Self Organization

In the early days of Scrum, we talked a lot about chaos theory and self-organization.  We wanted to believe that “management” was unnecessary and even counter productive to getting software built.  Well, after 15 years and many, many agile projects (some more successful than others), I’ve formed some pretty strong opinions on how self organization and team leadership impact a software project’s success. 

(1)    A team has to have a strong, management supported leader.  For purposes of this discussion, this is the team lead.  Its not important what title is on the team lead’s business card.   It may be Development Manager, Project Manager, ScrumMaster, Program Manager or whatever other label your organization is comfortable with.  What is important is the knowledge and experience of the team lead.

(2)    The team leader must have at least five years hands-on experience actually building software (I am assuming here that we’re talking about building software products).  By this, I mean that the lead has spent 5 years designing, coding, testing, delivering and supporting software products.  Ideally, the team lead has some experience with domain for which the product is being built, but this is not a requirement.

(3)    The team lead has equal and balanced knowledge, respect and appreciation for all disciplines required to get software built.  This is essential to successfully manage a cross-functional agile product development team.  As such, the team lead can dive into details with team members to help remove impediments regardless of whether it’s a development, test, scope, or documentation issue.  If one discipline is favored, the product will be out-of-balance.  The classic pitfall is for a team lead to be too engineering focused.  The end result is often that “technology” is produced rather than a product.  The features become internally driven.  Market focus and the customer connection is lost.

(4)    The team lead owns the resources and schedule.  The team lead has authority and control to direct and modify project resources.  Also, operating with-in bounds defined by the business, the team lead can alter the schedule.

(5)    With authority and control comes accountability.  The team lead is held accountable for resource and schedule issues.

(6)    The team lead does not control scope.  The primary owner of scope is the “customer proxy.”  The customer proxy may actually be the end user on consultative and internal projects, a business analyst or business unit representative, or a Product Manager for commercial software products.  The team lead can influence and negotiate scope changes with the customer proxy.  I’m going to avoid a deeper discussion on scope change control at this point.

So… self-organization…  As a leader in many past businesses, I am unwilling to put my business at risk.  I am a strong advocate of preemptive, proactive business and team leadership.  There is an important balance though.  Management and the team lead must not squelch innovation with heavy-handed oversight and control.  There does need to be self organization within the team.  During any project of significant scope and duration, priorities will shift.  Certain skills and attributes will become more important than others.  A well functioning agile team reflexively compensates and the team member with appropriate skills and knowledge steps up.  This is important.  If the team is functional, this just happens.  It’s not planned, managed, or dictated.  A good team lead recognizes when its happening, and when its not, the team lead takes action.

Who's driving?

Sometimes I feel like half the trash I read on the agile internet groups and blogs is written by counter-culture contrarian anarchists that smoke pot and sleep in yurts on the Oregon coast (I highly recommend at least the yurt on the Oregon coast).  These extremists abhor most formal process and gods forbid someone tried to profit from writing a piece of software – especially software that can help them get their job done.

Most of the rest of the crap on these same groups feels like it comes from neo-McCarthy-ist control freaks that seek to institute PMI-like controls and measures on agile development including mean time between potty breaks (which of course at which you’ll be randomly drug tested).  They storm the hallways wielding their ten square meter Gantt chart like a light saber unwilling to accept that any project of material scope or effort could be managed without it.

Why am I so agitated today?  I am very worried that something that we built that was a simple and efficient way of building software products is being co-opted and convoluted by the mindless hoard.  Can this web-based zeitgeist guide, evolve, and improve Scum and other agile methods?  My experience with design, whether it be software or process design, is that elegant designs are not created by committee, they are the result of creative genius that is usually encompassed in a single individual.

Yep.  I am a bit tired, grumpy, and burnt out today.  Transitioning from a long holiday weekend directly into a 6am flight to SFO will do it to you.  So for the 1% of you that I haven’t offended, please give me a smile, hug, or perhaps buy me a shot of Patron when you see me bleary eyed at the bar at SFO tonight at 8pm.

Developers build stuff, Testers break stuff

I've seen some recent posts in the user groups about eliminating the distinction between developers and testers on teams.  One word – Hogwash (that’s the polite version).

The best developers I've worked with create beautifully engineered and usually bug-resistant code.  They do this with passion and will commit every once of their soul to the purpose.  Now, if they're a bit more enlightened, they'll like the idea of unit testing and build out a [j|n]unit test suite for their component.  Often, here the passion dissipates and testing isn't quite as complete as you'd like it to be.  I've yet to truly experience test-first, but I am sure its out there somewhere.

Now testers are a different beast entirely.  They are devious.  They will take this piece of artwork, conspire with others, and meticulously commence punish it in every way conceivable.  They will do this over and over again with a fervor that it at times frightening.  They can't be stopped in their search and destroy quest.  And worse, yet, they will document and publish every last detail of their escapades.

These are two very different personality types that are very good at specific activities.  There is a difference between good testers and developers and good leaders will recognize and leverage this to build the best possible products.

Be agile about agile!

The democrats and republicans are doing a fine job of "fire bombing" the centrists in both their parties (I grabbed that quote from an article "It's not easy being George" in this months Vanity Fair).  And, as I scan the agile groups and blogosphere, it seems the methodologists are doing it as well.  Some of us take the "extreme" in extreme programming too literally, and others are holding on to some of agile developments long disproved stigma for far too long.

My approach always has been (and hopefully always will be, unless of course I turn out like my father, which all of us eventually do...) a pragmatic one.  There are many things I love about Scrum and agile - driving elaboration and development off the backlog, test-driven development, burndown charts, and much more.  And there are a some that I could do without (self-organizing teams, developers and testers are interchangeable,... I blog on these later).

An agile approach needs to be tailored to your people, process and technology.  You can start-off "by the book" but be sure to incrementally adjust your process in the same way you incrementally modify your products under development.  If you have a particularly gnarly architectural component to build feel free to build a UML-based design first.  It’s okay.  Really!  These are just items that are added to the sprint backlog and monitored along side all of the other demand that flows through the process.  And god forbid you build a Gantt chart to model out some complex project dependencies.  How will you be able to look yourself in the mirror?

Come on now; let’s be agile about agile and use the tool that works best for the job at hand.

Picking the right projects

One of the most important things I’ve learned from my recent stint in the project and portfolio management (PPM) space is that business success is not just about efficient and effective project execution (agile or traditional).  Success is more determined by making sure that your precious resources are working on projects that return the most value to the organization.  I can execute the hell out of a project and bring it in 25% ahead of schedule with a satisfied customer.  But if my competitors are delivering new higher value products that satisfy a broader customer base, then I will be left in the dust.  The irony is that they may have worst project execution and still beat me in the end if they’ve selected the right projects to invest in!

In the PPM space, determining what project investments to make is typically considered part of demand management.  Demand management considers inputs from a variety of internal and external sources including customers, analysts, partners, IT, business units, and more.  All of this demand needs to be reviewed and appropriate selections made.

If the incoming demand is not structured and managed, effective decision making will be impossible (garbage in, garbage out).  Each request should include a standardized business justification for the new “investment” that includes meaningful expected costs and benefits. The requests should also demonstrate alignment with corporate goals, compliance with technical architecture, and other measures important to the organization.  If a standardized structure to incoming demand is applied, a selection process can be realized that allows comparison of all incoming demand and with ongoing projects in enterprise portfolio.

There is a broader business context that agile projects need to exist in for an organization to succeed.  The larger and more complex the organization, the more standards and processes are required.  This may seem like blasphemy to us agile zealots, but its the cold hard reality of managing a successful business.  The key is to put in appropriate controls and at the same time leave enough degrees of freedom to foster an environment of creativity where agile teams can thrive.

"Demand Management" covers the front end of the funnel.  In a future post, I'll discuss the impact on the tail-end of the funnel and the impact of product release on the broader organization.  This has a whole other set of issues that impact agile teams.

Why be Agile?

I have been repeatedly asked why agile is better than traditional approaches.  The most succinct answer I have is that agile software enables you to build the right software at the right time.  Agile is not about in-depth analysis and design that leads to a detailed work breakdown structure that predicts on what day of the year eighteen months in the future a team will deliver a certain set of features.  Agile is about continuous prioritization and customer review to insure that meaningful features are delivered when they are ready and when they are needed.

The ability to make significant changes to design and behavior throughout the evolution of a system is perhaps one of the biggest differentiators of software development from other engineering disciplines.  And because we can, we should, and usually do.  This allows organization who embrace agile principles to have agility in how their business execute as well.  Being agile implies applying approaches and processes that maximize an organization's effectiveness in creating software.  Agility is not a chaotic undisciplined approach to hacking software.  If this is what you are doing, you are not being agile, you're being stupid!

Agile software development approaches promote high levels visibility, predictability, and quality.  Visibility is achieved through frequent, regular team status checks and product demos.  Predictability is achieved through continuous monitoring and updating of the project backlog and burn-down.  And, quality is maximized by embracing test-first approaches eliminating the "over-the-fence" paradigms of legacy software development methodologies.

An essential characteristic of agile software development is that barriers that separate people that work together are removed.  Legacy software processes and methodologies formalize these barriers as entry and exit criteria, documentation, and sign-offs.  Developers, testers and documenters are far removed from the actual users of the system and must often interpret documentation rather than directly ascertaining desirable system behavior.

Successful agile teams are collaborative and cross-functional.  All disciplines required to build the product are involved in all aspects of development.  The customer is pulled deeper into development and testers and writers are pulled earlier into design and elaboration.  The barriers to success are broken down.  Information is now instantaneously available to all who need it.  Errors resulting from interpretation can be resolved in real-time.

View this photo

Worlds Collide: Agile Development meets PPM

I just returned from the 2006 Project & Portfolio Management Summit.  Its an event that is put on by Gartner every year where they roll-out their annual analysis of the PPM market.  Since this is Pacific Edge Software's primary market, it was critical that we made a good showing.  And since were a small company in this market (CA, IBM, Microsoft, Compuware, Mercury and others have acquired their way in over the last couple years), we decided we wanted to look and feel different from the others.

So I presented "Worlds Collide: Agile Development meets PPM."  PPM advocates tend to have a traditional project management and methodology focus.  I was expecting my talk to be a bit on the controversial side as PPM-types tend to focus on change control, well defined business processes, approvals, and lots of other things that repulse most agile types.  I was pleasantly surprised.

I basically structured my presentation as a case study on Pacific Edge.  We are a hardcore agile shop with respect to product development, but apply many Project and Portfolio Management best practices as well.  One of the surprises was the amount of attendees that were familiar with Agile development.  When I queried the audience, it was well over half.  I had several approach me after the presentation to discuss how they could incorporate more agility into their PPM processes.  While the perceived informality of agile approaches is perceived as a problem with the tactically oriented, its value in making quick, good-enough decisions outweighed the risk with most.

For me, this is a clear sign that agile is well on its way into the mainstream.  When CIOs of large organizations are discussing the details of Scrum, we have truly arrived!

Rhythm and Pace

An essential element of agile development is rhythm.  As with musical composition where there are different levels of periodicity in the same passage of music, an agile development approach has nested levels of rhythm that serve different needs and interleave to  form an efficient, vibrant software development approach.

The build automation system is the metronome of the software engineers.  It sets the pace for code submissions, defect resolution, unit testing, peer code integration and more.

Weekly feature drops are the metronome for quality assurance.  Stabile delivery of weekly features drive the test organization.

Sprints are the metronome of the product development team.  They provide high-level integration points, usability analysis and feedback, and project visibility.

The highest level of periodicity is the release.  It is the synchronization point for the entire organization with the customer.

Which customer? I've got 100,000+

The success of agile approaches depends on how close you are to your customer.  This is an indisputable fact.  So if your an independent software vendor (ISV) that is producing commercial software products (like me at Pacific Edge Software), then how do I get my customer "on the team?"  Its likely not possible or practical and this is where effective Product Management kicks in.

The Product Manager (PM) is the proxy for the customer on an agile team in an ISV or for that matter any company building products for customers that they are at least once removed from.  Being on an agile team will change how a product manager elaborates and drives their product to market.  And I've had some interesting challenges on retooling classically schooled PMs (some of them didn't make required adjustments and didn't survive the organizational change).  While the roles and responsibilities are the same, the behavior patterns must change.

The PM is integrated into the development team.  He elaborates the feature and sees it through all stages of development.  He does not produce monolithic MRD and PRD documents and hand them over the fence to engineering.  He produces lightweight artifacts and insures they are implemented in a way that meets his customer's requirements.  He makes sure that the testers understand the features in detail.  And, he insures that the technical documentation is spot-on.  In the end, if the product and/or feature does not meet the customer's requirements, its the PMs failure.

Producing monolithic MRD, PRD and requirements documents is a failure mode.  The elaboration should be done as mini documents.  Preferably as discrete "user stories."  We dog-food our own PMM system (Mariner) and store them as individual records in its database.  That was each one can be individually prioritized and triaged into the appropriate "sprint" and/or product release.  Monolithic documents also reinforce "stove-piped" organizations where the PM can disengage from the product until test has verified and validated the final deliverable.  This is another failure mode.

Making these adjustments has greatly improved the product development process at Pacific Edge.  Our products are better aligned with customer needs and expectations, and our through-put of feature development is higher.

Creating Scrum

My first job was at Litton Itek Optical Systems in Lexington, Massachusetts.  Itek's business was mostly defense-oriented (This was true when I was there.  During the 1970s and 1980s Itek was a much larger more diversified company).  The most I can say is that Itek built land and space based reconnaissance systems.  This was during the Regan Star Wars craze.  There was lots of money being spent on lots of esoteric research programs.  There technology and tools were very cool, and I had the opportunity to work with some of the very brightest scientist in the world.  It was a very fun place for college grad to land.  This is where I cut my teeth on software technology and process.  I saw first hand how software could be harnessed to do amazing things and I learned how important quality and reliability is.  Bugs in my software could do millions of dollars in damage and in worst case scenarios injure or kill someone.  While we weren't practicing test-first development, the attention to process and quality was acute.  My accomplishments in this area were recognized by senior management and I was assigned to a team responsible for defining and deploying software methodology and standards.  As a result, I learned more about TQM, DOD-STD-2167A, and ISO-9000 than any 25 year old engineer should!  While the intense technical environment of defense-oriented engineering was very rewarding, I left Itek in 1991 for Easel Corporation to learn a bit about what this business application stuff was all about.

Easel Corporation was a small software tools company located along the 128 technology belt in Burlington, MA.  I started off in there consulting organization.  There, I was immersed into their Fortune 500 customer base and got exposure to an amazing variety of business systems and applications.  I worked in the field for three years and in 1993 I came off the road and joined engineering to lead a team of Smalltalkers that was blazing a trail on the bleeding edge of software technology and process.  Client-server development was the rage and several vendors were developing tools at a frantic pace.  We could see the PowerBuilder office across the street from us.  They we're bigger, better funded, and we could feel the heat.  PowerBuilder chose the 4GL approach.  Point, click and watch your application go.  We wanted to do it right.  We wanted to do it with objects.  We did it right, PowerBuilder ate us for lunch.

The ride was fun and we learned many lessons, some of them painful.  The highly productive Smalltalk development tools and environments we were using allowed the developers to churn out features at an incredible pace.  This allowed for rapid time to market and allowed the team to explore (prototype) many different alternative approaches.  Analysis and specification was just not practical when it was quicker to just code it.  The developers loved it.  The testers hated it.  And other downstream disciplines, like technical documentation, were just in a mad scramble trying to keep up.  In retrospect, it now easy to understand why it had to be this way.  The speed of development and innovation was so great that other non-code oriented aspects of product development were left in the dust.

I had the great fortune to work for a very supportive boss, Jeff Sutherland, who took a keen interest in not only moving our products and technology forward, but also the process we employed to construct the software.  Additionally, we employed Jeff McKenna as technology, process and business consultant.  Jeff McKenna's involvement with Smalltalk and object-oriented design since the early 1980's was invaluable in determining our technical and process approaches.  Together, we developed an agile software development approach we called Scrum (http://www.agilealliance.org/articles/reviews/Sutherland1/articles/InventingSCRUM.pdf).