Lean and Green

Lean and Scrum - Chicken and the Egg

For some reason I can't explain, metaphor's abound with respect to farm animals and agile (Pigs and Chickens for instance).  So I figure I'll pile on a bit here!

I've seen, read, and listened to several agile and lean experts discuss how lean and agile are related.  The discussion is a bit frivolous in my opinion, but after hearing it enough times, I wanted to set the record straight on Scrum.

We built Scrum for one reason.  We wanted a process that reflected how software actually got built rather than one that tried to control how people should think and structure code.  While at Easel Corporation, as the software development manager for the first ever Scrum (Ken Schwaeber hadn't yet started the CSM marketing machine, so there was no such thing as a "Scrum Master"), I had to figure out how to manage my team in an environment where most of the requirements weren't well understood, and those that we did understand seemed to change as quickly as they could be committed to paper(?).  And I had to figure out how to describe what we were doing to my management team (Jeff Sutherland, John Dove, and others).

I sat in my cube working tirelessly to maintain my Gantt charts.  I'd meet with my team members and review our requirements and designs.  I'd try and extrapolate tasks for the team.  I'd roll-up estimates and make predictions.  I'd then sit back and look at what I'd created and think it was total bullshit (another gratuitous farm reference).  There had to be a better way.  This is where Scrum came from.  We abandoned traditional planning and developed the notion of the "backlog".  We also re-used the notion of incremental development from Sprial and RAD/JAD and introduced Sprints.  We found the Backlog and Sprint to be very effective tools in planning and managing work i our volatile environment.

We felt like we were on to something here.

After some research, Jeff Sutherland found some similar models for how work was defined and managed that aligned with what we were doing.  This was documented in a 1996 Harvard Business Review paper by Takeuchi and Nonaka.

Meanwhile back at the ranch... (yep, back to the farm) Kent Beck et al were busy refining XP.  In the early days there was no communication between the Easel team and Kent's team.  So I can't personally speak to the degree to which XP was influenced by lean.  But I can inequivocally say that the primary innovations introduced by Scrum were done so independently of what we know of today as "Lean".

Are there common practices across Lean and Scrum.  Yes, of course there are.  Is this a coincidence?  Not really.  As with XP, some of the basic concepts that Scrum introduced are about how we can more efficiently deliver value on software projects to our customers.  So does it really matter which is the chicken and which is the egg?

It's Feature Time! Making Vision Actionable

User Stories and Epics are inadequate to capture and express critical elements of product design.  User Stories are fine grained and atomic – remember they are “implementable” within a single Sprint.  Epic Stories tend to be semantic free containers of Stories.  Creating sophisticated applications with rich application interfaces requires pre-production activities to validate Epic and User Stories with product design considerations such as usability and architecture. 

We experienced this on our latest project.  Ironically, this was while building Serena Agile On Demand, a work management tool for teams (and teams of teams) using agile methods to create software!  The team had been Sprinting together for quite some time.  Probably in the neighborhood of 15 or more 3 week Sprints that included 3 releases to production of a SaaS product.  As we approached the completion of the third release, we started getting pretty clear feedback from some team members that they didn’t really understand the product vision.

This perplexed some of us.  We always maintain a product vision.  We refresh it often, especially at the beginning of the release.  It’s not a heavy weight document.  It tends to be about 15 or so PowerPoint slides that capture market, business and product information.  The product stuff is usually high-level Use Cases, release themes, goals, and key high-level features.  We had created these artifacts and reviewed them with the teams before every release.  And in fact, after review, we agreed that the team actually executed against them effectively as well.  So we had a vision, we communicated the vision, our teams delivered on that vision, yet they said they didn’t understand it!

It took some discussion with the team, but we did get to what we believe the root causes were and took action to remedy them.  Firstly, we now communicate vision early and often.  Previously, we were doing the early part.  Communicating vision to teams at the beginning of the release, but we fell short on often.  Now we’ve formalized reminding the team about key elements of our vision at the end of each Sprint in the Sprint demo meeting.  We’ve also employed corporate marketing communication tactics by posting release goals and themes in many of the public areas.

The second remedy wasn’t quite so obvious.  While the team said they didn’t understand the vision, they also meant they didn’t understand why certain User Stories were so important or didn’t agree that others were the right User Stories.  Part of this was that they “couldn’t see the forest but for the trees”.

The teams do use “Story Time” for Sprint pre-planning.  These meetings typically occur towards the end of a Sprint in preparation for the next Sprint.  In Story Time meetings, the Product Owners bring candidate user stories and other items to the team for vetting and discussion.  This allows the Product Owner to have more effective (and shorter!) Sprint planning meetings once the next Sprint starts.  Story Time is a great tool for teams to use to improve User Story quality and make Sprint planning more effective and shorter but typically important planning discussion have already occurred where these Stories were decomposed from broader themes, goals, or Epics.  In our case important people and important considerations were missed during the pre- story time elaboration.

Our solution was to introduce “Feature Time”.  Peter Goubert, our Chief Product Owner coined the term.  Feature Time is a regular (at least weekly) meeting.  It will have one or more specific topics that will be discussed.  The Chief Product Owner coordinates the meeting.  Participants include Product Owners, software engineering leads (or architects), usability experts and others depending on specific topics.  We try and keep attendance as compact as possible.  The purpose of feature time is to look at course grained groups of features, Epics, and Stories and to understand and evolve the product design.  Outputs include preliminary and revised Story backlogs, user interface mock-ups, and high-level domain models.  These artifacts are used to translate the product vision into actionable work for the team.  We also make participants in Feature Time “holders of the vision” and responsible for evangelizing the vision to their teammates.

Keeping teams focused and motivated is both challenging and rewarding.  While agile and Scrum methods are great steps forward for building energy and momentum in application development organizations, practicing them at scale requires us to extend the agile toolkit a bit and we’ve found Feature Time to be an effective new tool!

Agile ROI Webinar on September 3, 2009

I am presenting my post on Agile ROI in a brief webinar today.  I will basically go over my previous Agile ROI post with the inclusion of some new materials.  You can find the spreadsheet of the ROI model here and the PDF of the presentation here.  Enjoy.

Why Scrum isn't enough for agile success - The Webinar!

So I blogged on this topic a few months back.  You can find that blog entry here.  It was a popular post.  So much so that we made a webinar out of it with the Agile Journal.  Jeff McKenna, Serena's Chief Agile Evangelist and the first Scrum Coach, and Paul Dupuy, an agile black belt and the Product Owner for Serena Agile On Demand joined me for the session.  We had a great time and hopefully the attendees found it valuable.  You can find the information on the webinar here and a recording of the webinar here.

We had far more Q&A than could be addressed in time allotted.  I guess the discussion going a little long didn't help much either!  So I decided to cross post the Q&A here in my blog.  Enjoy (or ignore)!

Question: What if the business process isn't defined how does Agile handle business reengineering?

Answer: When the business process is not defined, an agile approach provides an incremental model to explore and define it!  Using Scrum for instance, you would work with your Product Owner (customer) to elaborate user stories that define some aspect of the business process.  Don't boil the ocean.  Pick a small "chunk" that can be easily described and understood.  If the end goal is to have a system that is implemented using software, then you should strive to implement this process as well.  To the extent possible, this should be accomplished in the same sprint.  Avoid using the sprints to only document the process.  Your goal should be to produce working software as early as often as possible.


Q: Thanks for this presentation. One issue we struggle with is using scrum teams that do all of the things you are describing but we have trouble bidding competitively on small professional services projects. There is some overhead with scrum when applied to smallish projects (lasting 2 - 6 weeks) that we struggle with. If you have any suggestions or examples for how to insulate the customer from our internal costs of using scrum on small projects I would love to hear it. We've been using scrum for about 4 years and it works great for our product team. Any thoughts you have would be appreciated.    

A: When using Scrum on professional services engagements, we highly recommend giving your customers visibility (and getting their buy in!) to the process you use to deliver the requested features.  If they've agreed to the approach, then this should minimize the impedance mismatch and associated overhead costs for spinning up the project.  Also, try and keep your teams together.  Remember, with Scrum, we focus on building and optimizing teams.  If your building up and tearing down teams for each project, this will certainly introduce overhead and also decrease the predictability of teams velocity as it can take up to three sprints for a team's velocity to normalize.


Q: Which is more important between Individuals/Interaction and Processes/Tools? What I should do if my team members do not have cross-functional skills?

A: In agile we always favor direct communication.  Whenever and where ever possible get people in a room together hashing through the problems and issues.  The reality of "agile in the enterprise" is that teams are distributed by time and distance.  This is where tools are essential in enabling team collaboration.   When teams are first formed, its common that there is some specialization of skills.  This moderates over time.  Pairing, an extreme programming best practice, is an excellent tool to use to help cross train team members.


Q: How can you help a large, compliance-oriented, traditional development organization move to a more agile approach without jumping all the way into Scrum?  

A: If you don't have expert help on staff, we recommend getting help with the organizational change management issues that you will deal with in adopting agile.  That is why Serena partnered with Valtech, a leading provider of Agile transformational services, with our Serena On Demand product.
Also set it up for early success.  Identify a few of your best teams that are starting up new projects.  Plant an experienced agile coach on the team, get them norming and storming.  You'll be able to use the experience from this team to seed (and motivate) others.
A tool like Serena Agile On Demand helps as well.  It provides a prescriptive Scrum framework for the teams to follow.


Q: What tools to you offer to facilitate yearly budgeting and goal setting in a way that could feed into incremental deliveries while supporting financial accountability within key cost centers?   

A: The "team" is the key unit for agile projects.  We want to fund teams and keep them stable for as long as possible.  This allows us to optimize predictability.  So you want to fund as many teams as possible, keep them together for as long as possible, and bring the work (projects) to them.  At the portfolio level, its also helpful to consider relative investment levels for key initiative requested from the business units.  This will dictate the mix of projects that you direct to the teams.


Q: Can we implement these tools based on traditional methods and then seamlessly transition to agile?  

A: No.  Adopting agile has significant impacts on the people and processes within an organization.


Q: How does this approach handle complex systems integrations with third parties and systems involving end-to-end financial settlement processes such as GL posting.   

A: This is hard regardless of whether your following a traditional or agile approach!  When we scale agile, we typically have many teams collaborating on a single appliation/product/service.  The integration of the work developed by these teams of course needs to be closely managed.  From a program management perspective, we can use the "Scrum of Scrums" to manage the inter-dependencies between teams and other cross team issues.  We can also use the "Super Scrum" as longer range product planning tool to make sure that work is appropriately sequenced so that cross team dependencies will be satisfied.


Q: What if the business perception is that small, incremental releases do not generate adequate business value to warrant business change, training, and communication required? Can this process handle small internal increments that culminate into large releases?
   
A: Absolutely.  Agile (and Scrum) are based on incremental approaches that produce potentially shippable code.  Whether or not the increment is shipped is a business decision based on product, market, and customer readiness.  


Q: (Loaded question) What if the user stories to be delivered in a sprint get delayed, and the spring demo cannot be completed on time?   

A: We love loaded questions!  The duration of a sprint never changes.  If a story is not completed, its not accepted and not delivered as part of the sprint.  The story will either be returned to the product backlog or moved into the next sprint.


Q: How is Serena differentiated from competition (such as Rally)?   

A: Serena Agile On Demand was built from the ground up to deal with agile at the enterprise scale.  Multiple teams, multiple releases, multiple products.  We offer unique capabilities in our ability to deal with enterprise complexity.  Serena Agile On Demand comes fully loaded for prescriptive Scrum development.  While complete in its ability to deal with Scrum, its hyper-configurable to meet your specific needs as well.  Serena Agile On Demand not only delivers the product features as a service, but it also delivers on demand agile coaching through Coach Live in cooperation with Valtech. 


Q: How can the tool support allocating planned resource capacity to specific initiatives while allowing freedom from scrum masters to manage specific task assignments for team members. Need to measure that actual utilization is not exceeding targeted resource allocations.    

A: A major difference with agile approaches from traditional approaches with respect to resource management is that the team is the unit of planning, not an individual resource.  We try and put cross functional teams together and keep them together.  The team then establishes a velocity - basically the amount of work they can complete in a sprint.  Once the teams velocity is established, the team commits to deliver the amount of work that matches their velocity.  Agile teams are always looking for ways to optimize their throughput (increase their velocity).  This is contrasted with traditional approaches where management attempts to maximize utlization.


Q: Can the Agile On Demand task board be customized to reflect custom statuses ("swim lanes")?"   

A: Yes.  The swim lanes in the Card Wall are completely configurable.


Q: Who selects scrum team

A: When making a transition from traditional methodologies to SCRUM we think it is a best practice to "invite" members of the team to join the SCRUM. It's important in the early stages to get enthusiastic members on the team who will actively engage and support adoption. Hard to get people like that if they are 'forced' to be on the team.


Q: Since agile / scrum utilizes an Empirical model for managing costs, How can management make an assessment of the total cost of the program in order to fund development.    

A: Plan the work as far as the release horizon using coarse grain items. Have the team size the items using rough estimates. Use historical information from the team's activity or make an educated guess about how much work the team can do in a sprint. Determine the cost of a team for a sprint. Calculate how many sprints you need to finish the work in the release. This gives you the estimated cost to complete the work. You'll need to use scope negotiation skills to prevent "scope creep" and decrease the priority of low priority items. You'll also need to recognize that as the software is developed, changes will occur (which is a good thing -- you'll find that what you wanted at the beginning isn't what you need) and trade-offs will have to be made.


Q: I manage a small unit responsible for several projects. Some were started over a year ago and were designed to follow waterfall. Subsequent projects are now developed via Scrum Agile method. With at least a year left on the waterfal project, how or should we switch mid-stream to agile?   

A: It is fairly simple to switch to Scrum to manage your work. If your waterfall planning resulted in a pretty gantt chart with tasks, you can use them as the work items in your product backlog. Read up on Scrum and get a certified scrum master if you can.


Q: Have you ever heard of ACTUAL customers (in addition to product owners and PMs) participating in a sprint review? Any ideas how this could be implemented? Thanks!   

A: Absolutely.  This is done quite frequently in fact.  If possible invite the customer to attend in person.  Otherwise their are many virtualized meeting tools that can be used quite effectively.


Q: Could you expand some on non-product stories, how they get prioritized since the product owner may not care, how you track those vs. product user stories in metrics, etc?"   

A: Pay me now or pay me later.  Sometimes the team has to go slow for a few sprints to go faster in future sprints.  There are many examples of non-product stories where technical debt is addressed that may not result in direct value to the customer.  This is where the team and the Scrum Master must lobby and compel the product owner to include these stories into a sprint.  Without this balance, an insurmountable amount of technical debt will build up.


Q: Is Serena overkill for a team just looking for a requirements/project management?   

A: No, not at all. Serena's Agile On Demand has the ability to manage many types of backlogs and is ideal for requirements/project management. Internally here, the Marketing department has several teams actively sprinting and using Agile On Demand to manage projects and requirements. Multiple product backlogs, about nine is our case, contain stories for different projects the team is working on, each managed by a separate 'product owner' (usually a Product Marketing Manager). Then there are separate TEAM backlogs that contain the sprints and cross functional teams of marketing types who all contribute to the completion of stories with the project backlogs. The benefits of have a team manage requirements this way include time management, capacity planning for the sprint and minimizing the randomizing effects of fire drills.


Q: How did you convince your managment to use XP?   

A: XP is a collection of best practices that increase the likelihood a team will deliver running tested features that meet customer needs in a predictable and repeatable way. Surely management can see the value in that result? ;-) The book "Extreme Programming Applied" has a good discussion of this. Also see the discussion on the c2 wiki: http://c2.com/cgi/wiki?WhyItIsSoHardToSellExtremeProgramming


Q: So, Product Managers can equate to Business Analyst in a traditional organization and Scrum Master is played by Project Managers?   

A: The Product Manager's roles and responsibilities align quite well with the Product Owner's.  BUT, the approach they take to elaborating requirements and engaging with team varies significantly.  A Project Manager can be a Scrum Master, but be cautious.  Project Managers are typically in a top-down command and control roll where they plan and dictate to the team how to complete work.  A Scrum Master is a passive leader that facilitates the team.  The Scrum Master coordinates meetings, interfaces with external teams, and manages the impediments that the team can't resolve themselves.


Q: How do we handle Standup meetings for geographically distributed teams?"  

A: Tools are essential.  Use planning and collaborations tools like Serena Agile On Demand and Virtual Meeting tools.


Q: What estimation method do you use from sprint planning?    

A: We use story points with "unit-less" value. Our teams use planning poker to estimate sizes.  There is an interesting video of it's creator talking about it here: http://www.youtube.com/watch?v=1oIaz7aZsno&feature=channel_page and more info on this site: http://www.planningpoker.com/


Q: Would you recommend adding to the backlog a "regression testing" item?"   

A: Using stories to manage work that is difficult to complete or track can work well. If the team is having trouble doing regression testing in a predictable way during the course of a sprint, using a story is recommended. If the team does a consistent amount of regression testing every sprint, you could instead consider it "overhead" and choose not to track it.


Q: is serena agile license per user based?  

A: Yes. $35 per user per month.


Q: Where would a Business/System Analyst fit into the Scrum team/role   

A: Often times we see Business/System Analysts as part of the Scrum team in primarily a Product Owner role.


Q: IS a user story different than a story board. Or is the story board where you post all the user stories for selection?  

A: The card wall (story board) is a technique for showing stories and their current state in a single view. It is typically used to manage the state for stories in a sprint.


Q: Where is the current "high-bar" for portfolio management across an enterprise for full Scrum methodology used on across large teams of developers? Team size of 500 + developers? 800+ developers?   

A: 1000s of resources.

Get over it, scaling software development is hard!

<rant>

I am so sick of listening to people propagate the myth that agile methods don't scale.  Guess what everyone, scaling software development is hard!  Regardless of approach or methodology.  I am not going to waste your or my time quoting all the well know (but somewhat outdated studies), but suffice it to say that the success or failure of a software development effort correlates pretty damn well with the size of the effort.  So if its gotta be big, its gonna be hard. Count on it.

And that’s why one of the most important things we do with agile development efforts is to decompose them into smaller chunks.  Ideally, incrementally delivering to our customers production ready capabilities rather than embarking on the classic 24-month death march to preordained failure and/or obsolescence.

</rant>  

Ok.  I feel better now.

For those edge cases where the minimum marketable set of features translates to a large software development effort, lets be smart about it and make sure we’ve got the right people, processes, and tools in place so we have a better chance at success!  I'll have some follow-up posts to my blog under the "Scaling Agile" category with specific techniques agile teams can use to scale their agile development efforts.

Motherhood and Agile Pie

There is an idiom in American English that I use quite often.  I assumed that it was commonly used – at least here in the States.  The full idiom is “as American as motherhood and apple pie”.  It’s typically used when you are describing something that is quintessentially American and an idea that very few would disagree with.  Its often abbreviated to “motherhood and apple pie” and use when describing something obvious.  When I enumerate the benefits of agile methods, they are, well, mother hood and apple pie!  Here they are:

• Ensure the right software gets built through customer collaboration
• Deliver value early and often with incremental development
• Optimize production throughput and communication with collaborative, cross-functional teams
• Rapidly adapt to changing business, market and customer requirements
• Deliver higher-quality software through test-driven development
• Provide predictable execution and real-time visibility

Ensure the right software gets built through customer collaboration.  Agile methods are customer-driven.   Where possible, agile teams will embed the customer into the agile team where they create and prioritize requirements (in Scrum, this is the Product Owner role and typically requirements are captured as User Stories).  If its not possible to actually embed the customer on the team, a proxy for the customer is used.  The customer proxy may be a Business Analyst or Product Manager depending on your business and organizational structure.  With the customer closely aligned with team, the team will always be focused on delivering the most valuable features and will have a higher likelihood of meeting customer expectations.

Deliver value early and often with incremental development.  Almost all agile methods espouse some type of incremental development where potentially shippable features are developed and delivered incrementally.  Assuming the requirements are being effectively prioritized (see previous bullet), and that we are developing incrementally, we are then optimizing the frequency of value delivery to the customer.  Now, if we are incrementally delivering value rather than in the traditional 18+ month cycles, basic finance tells us that the time value of money will yield higher ROI with the incremental model (see my blog post on this here).

Optimize production throughput and communication with collaborative cross-functional teams.  From its inception, agile methods have been all about optimizing the throughput of agile development teams.  Depending on whom you talk to, this was either borrowed from or developed in parallel to the work that was done in Lean Manufacturing.  In agile, we optimize in a variety of ways.  Perhaps the most profound is the creation of cross-functional teams.  The cross-functional team moves us away from stove-piped organizations that use documents as a mechanism for communication.  Agile cross-functional teams produce just enough documentation and focus value delivery over formal process documentation.

Rapidly adapt to changing business, market and customer requirements.  This benefit is inherent with incremental development and effective prioritization of requirements.  If we are continuously prioritizing and re-ranking the team’s future work, we will optimize the value delivered with the market requirements.

Deliver higher-quality software through test-driven development.  Test-driven development is one of the essential practices that teams employ to ensure that not only do the features that they deliver work, but that they continue to work over lifecycle of the product, application, or service.  The combination of test-driven, development, automation, and continuous integration practices allow teams to improve the quality of their software deliverables.

Provide predictable execution and real-time visibility.  One of they myths about agile methods is that there are no processes to ensure predictable execution and that stakeholders get no visibility into status, progress, and risks.   Well, if this is the case, then you are not practicing agile, you’re just not managing your software development efforts.  Agile methods can provide high levels of CMMI process maturity (see Jeff Sutherland's post on this here).  Additionally, teams that are effectively planning and performing incremental development will deliver higher levels of visibility to their stakeholders.

To me, this all motherhood and apple (agile?) pie.  The benefits are obvious and it would be negligent or at least irresponsible not to pursue them.