Notes on Agile/Scrum

Russell Bateman
last update:

Please peruse my tutorial, Toward Scrum, and presentations linked at Tutorials and articles on Java and other topics.

(These are my notes though they're writting in an inhabitually pedantic style here. I'm writing as if to teach the reader what Agile/Scrum is. More simply "notish" stuff comes along the deeper you read into this page.)

The point of Agile is to release early and release often. It is the opposite of waterfall development which spends a great deal of time in design, proof, planning and execution only to end up costly, missing a target that sailed long before release and being a waste of everyone's effort.

The rationale for Agile is to release something, however small, immediately based on the customer's wishes, something that all parties can see as either beginning to meet the needs and requirements or not. Another way of saying this is: fail early, fail often, correct immediately.

In order to do Agile well, you have to have teams, you have to have backlog, and you have to have the ability to produce a working tested increment of product at the end of each iteration.

Scrum is the best of Agile.


Scrum and the pigs and chickens metaphor

This famous cartoon explains breakfast pigs and chickens in the Scrum process. It's a way to discern between roles in the Scrum/Agile world. Roles are key in Agile. They are constant and form the basis of every decision and define the very basic human relationships in Scrum.

For example, chickens have input on the product that will be developed, but pigs decide how it's going to be done and the rate at which it can be accomplished. Don't be put off by chicken being demeaning. The customer, vendor, executives and other important people with vision are chickens in Scrum.

Formally speaking, the pigs are developers including development managers, the produt owner/manager, who represents the chickens, and the Scrummaster, a sort of master of ceremonies especially during Scrum stand-ups.

There is a rule that during stand-ups, pigs can talk, but not chickens. If chickens wish to talk, they must wait for a "meeting after the meeting." There is no regularly scheduled meeting after the meeting and, if institutionalized, it destroys the effectiveness of the stand-up in not wasting time or taking the pigs away from what they do best.

Who are pigs? The developers, their manager and the product manager (when he shows up, but he's especially important during sprint planning. Chickens are other stakeholders like customers, vendors, dotted-line managers, maybe interested parties like executives. But, chickens are NOT allowed to talk!

Correspondingly, however, pigs are required to talk. They must attend every stand-up.


Scrum is so much more

The principles of Scrum/Agile cannot be studied casually (like reading the few notes on this page).


The Scrum stand-up

This is the most important meeting to pigs and to the essence of Scrum.

Pigs must attend stand-up. Chickens may attend, if they have enough discipline to keep their mouths shut. In some organizations who pray at the Scrum altar, there may be fines for pigs missing meetings or chickens opening their mouths.

What else?

The stand-up is all about accomplishing the meeting's objectives. These include sync'ing up, making sure there's no disaster in the making and noting obstacles.

The stand-up answers three questions. Each pig states...

  1. what he or she did since the last stand-up, ("Yesterday, I...")
  2. what he or she expects to work on before the next stand-up ("Today, I...")
  3. and whether or not there are any impediments. ("No blockers" or "I'm blocked on...")

These are absolutely, depending on how fastidious the organization is, the only sounds that go on in the stand-up proper. In strict Scrum, nothing else may be said during stand-up.

The stand-up happens in the same way, every day, in the same place, at the same time, on time. All other meetings are subjugated to the Scrum stand-up in importance.

The name implies that the meeting is so short and so succinct that a team need not even sit down. Some (those fastidious Scrummers) even impose fines for sitting down.


Other Scrum meetings and practices

In Scrum there are these meetings:

  1. Stand-up
  2. Release planning
  3. Sprint planning
  4. Sprint retrospective

(The stand-up is covered elsewhere.)

Release planning (not really Scrum)

Release planning is conducted by the product owner/manager. He maintains the back-log, a list of epics and stories that will accomplish what's needed to produce the product. Often, only team leads and development managers attend.

Release planning is aberrant to Scrum, but a crutch in some organizations for product planning. It has to be done sometime by someone whether or not it's institutionalized and given this name.

Typically, the output from release planning is a list of epics or stories for the product owner's back-log. Some degree of estimation, requiring cooperation from pigs, can be done, but there is never any break-down deeper than the epic, never deeper than the story level. Formal epics and stories need not come out of it, but these must obviously be in place for a back-log to exist in time for spring planning.

Release planning, because it's not Scrum, is a good opportunity for chickens to get their licks in. They insist that the product owner see product requirements their way and be able to plan accordingly.

Release planning happens ahead of beginning any sprints to work on the features planned. This would not be every iteration, but more on a quarterly or yearly basis depending on the size and scope of the product. (This is why it's not an Agile concept.)

The product owner

In all of this planning, the product owner is the key not only to planning, but to the ultimate success of the product. The product owner creates and maintains the back-log, prioritizes it based on customer and management needs, and actively promotes its accomplishment. His job isn't done once the back-log is full and the sprint planning underway. He gives vision and order to the team. The absence of a good product owner can literally destroy a products chances to see the light of day. In many cases, the product owner is really a very savvy developer with a strong streak of customer- and marketing sense, or an extremely talented customer and marketing person who totally groks the developers' side of the equation.

Sprint (iteration) planning

Sprint planning is conducted by the product owner. In some organizations, it is a two-step process spread across two meetings, sometimes two days. This doesn't mean it takes that long, just that there are two phases. Obviously, if the iteration is two weeks long, spring planning must not waste more than a few hours of it.

First, what's to be done in the sprint is estimated in terms of what can be done based on historical team velocity. (Velocity is the observed rate, in story points, at which a team typically works.) The availability of the team members is scheduled (who's on vacation or out of the office, when, etc.) and matched up against the story points to be accomplished. Stories that can't be done are returned to the back-log for a later sprint.

The dynamic is that the product owner promotes as many stories as he can reasonably get out of the team while the team pushes back to ensure they can successfully complete them.

Second, team members take time personally to break down the planned stories into tasks. A task is a subunit of work, never taking more than a few hours (some organizations place a strict maximum like 4, 6, 8 hours, etc.) that is required for most developers to complete. A task is not necessarily a programming assignment. It can include research, learning some required new technology, writing documentation, etc. Stories are broken down into 2, 4, 6 or even more tasks. Stories that cannot be decomposed into fewer than half a dozen tasks might be epics and not stories.

When the sprint planning resumes, the stories are all broken down, ready to do and (or) a sanity check is possible, to wit, the sprint as planned may have been proven during break-down to be too much.

The longer an iteration is, say 30 days instead of a fortnight, the more likely the two-part sprint planning will be needed.

Retrospective

The spring retrospective is a short meeting, usually at the end of one iteration and prior to sprint planning, attended mandatorily by all pigs during which the following questions are answered:

  1. what went well?
  2. what didn't go well?
  3. what can we do better next time?

Bookkeeping

The foregoing makes obvious the need to remember what goes on during planning, iterations, retrospectives, etc. Unless the number of stories successfully completed and their points tabulated, it's impossible to gain an idea of a team's velocity, something that takes several sprints (at least).

Obviously, knowing a team's velocity factors very usefully into estimation which is essential to budget, resources, etc.

The results of the sprint retrospective must be noted for follow-up, usually under the auspices of the Scrummaster.

Tools

This is where a good Agile tool comes in to play. This tool must be able to cope with

  • epics, stories and tasks along with points and hours spent,
  • team member's availability,
  • notes from the retrospective,
  • the state of stories and tasks (unbegun, in progress, finished),
  • the arithmetic (tabulation, matching availability to plans, etc.),
  • the burn-down chart (shows how much left to do),
  • all this in a fairly simple yet aesthetic representation.

Some good tools include:

(What Assembla does not do is most of what's above—barely tracking the state of things with no hierarchy in the relationship between epics, stories and tasks.)

Sprint cadence

What happens during a sprint is:

  1. planning and buy-off,
  2. testing underway (TDD: test first, code second!),
  3. coding underway,
  4. demonstration of finished tasks or stories,
  5. the retrospective.

The pigs buy off on the sprint planning symbolically by expressing their perception as to the likelihood of success, giving a fist of five for full confidence, or a single (or no) finger when most pessimistic.

Then, they get to work, hold stand-ups daily. An important aspect of that work includes test-driven development (TDD). This is an Agile concept that ensures:

Scrum preconizes the term finished by saying what it means. Finished is the final state of every task and story (it is to be hoped) by the end of the iteration. Finished is defined as

  1. test-complete
  2. code-complete
  3. documentation if needed
  4. a demonstration capability

Demonstration is most important to chickens, but it's useful to pigs as well.

To the greatest extent possible, the effect of the work must be shown in the light of how it contributes to the product. This proves to the developer, manager and product owner that the story is finished, it shows chickens like the customer, vendor, salesperson, marketer, etc. the finished product will do.

Back-end effects, like database or server work, are shown if possible by the side effects of product (UI) use. Often there is work whose demonstration is too arcane to be done for chickens.

Most frequently, the demo is accomplished in a great-room setting showing off the entire team's accomplishment for the iteration.

The epic...

...is simply a huge story that can be broken down into smaller stories that, unlike the original story, do not defy decomponsition into tasks. The epic is a relatively new concept to Scrum.

Writing stories

In Scrum, there's a peculiar way of expressing stories. These are couched in the following language:

  1. "As a _______________,
  2. I need ______________,
  3. so that _____________!"

For example, "As an iPad user, I need a button on this application, so that I can save my work."

It's that simple. In the context of any good tool for managing stories, one often elaborates by talking more about the stories, its requirements, expectations or any other important information.

Typically, this expression is not applied to tasks, which are more telegraphic, though it can be applied to epics.


Don't estimate stories during sprint planning

...because you know too much about them as compared to what you know about stories that aren't likely to be done immediately. Instead, estimate them at least a couple of months ahead of time so that the estimates have the same weight and don't falsify velocity.

"So, if I estimate or reestimate the story in sprint planning, I can't estimate it relatively to the rest of the backlog. I don't know that level of detail about the rest of the stories. I'm now thinking at a different level of detail, using a different kind of thinking than when we estimated the other stories.

"We know our estimates are wrong. We knew they were wrong when we estimated in release planning, and we know they are still wrong when we get to sprint planning. That's why we have velocity—an empirical measure that takes all that wrongness into account."


Agile and management

As we're trying to do a project in a Scrum way, and as Scrum/Agile isn't reassuring to management, I realize that I'm going to have to learn the part of the Agile story that fixes this reservation. Specifically, "Agile is fine, when do I get what?"

I looked around. I found some help. One article I especially liked was Agile Release Planning: Estimate, Predice, Commit. Here is the best of that article. The pyramid metaphor comes out of it.

In the dark ages before Agile, one made estimates and commitments. These were never exactly met, but no one really noticed. It was waterfall. That was how the game was played. A commitment was made, everyone knew it would be wrong, but it was expected anyway. Maybe management handicapped the commitment, removing scope, lowering expectations, padding the schedule. It's been the recipe for success since the pyramids were planned.

  1. Early estimates are wrong. Arithmetic shows they will be wrong. Program Evaluation Review Technique (PERT)'s law of large numbers helps aggregate (read: fudge), but it will still be wrong. (When, in my waterfall experience, were we ever right?)

  2. The outside demands, availability, etc. of the resources, especially people, will change including unplanned illness, attrition, personal level of commitment. It's people stuff. It changes.

  3. Customer needs and priorities will change. And they don't really even know what they need or want in the first place. That's one of the things Agile is all about: growing with the customer understanding, reflecting educated needs live rather than in a cycle of work, disappointment, renegotiation, rework, etc. Agile processes are designed to help deliver what the customer actually needs, not what he asks for. Waterfall processes encourage one to deliver the wrong thing, come hell or high water, as close to on-time as possible.

Under waterfall, one committed to delivering a couple of pyramids. After spending twice the time, double the budget, the first pyramid rolls off the line and, guess what? sphinxes are now all the rage. Oops.

Unfortunately, the Pharaoh still wants a commitment to deliver a couple of pyramids. How to remain true to Agile (not because it's a question of religion, but because of the outcome that makes it the True Way), but mollify management.

One way is just to give the estimate. Not, "This will take two months." Instead, "I expect, with 95% confidence, that this will take no more than two months."

There's an uncertainty cone that develops as a result of predicting the time it will take to produce the range of undefined tasks and even stories combined with not being able to predict illness, attrition, etc. or customer lunacy (because this is Agile and you're not leaving him out, therefore, he will change what he wants as he sees the product develop because he'll begin to realize what it is he really wants).

Because agile, one is willing to get smarter about the product over time. (And not just the developer either.) Therefore, the uncertainty cone should shrink over time.

But, a commitment is still required.

Backlog!

The backlog is an ordered, prioritized list of stories (and bugs). The backlog represents what and the order in which stories will be executed. The product manager will refine the backlog, changing the order and break-downs as the team (and he, and also the customer) get smarter.

So, given the backlog and given iterations, it's easier to predict how long because we commit to how much and how much is what's in the sprint this go-around. So, it becomes not how long, but how much, and how long is the iteration, so how much is (this much: what's on the backlog).

What will we finish?

The objective then is to move management from asking how long to asking how much or rather what. Commit to a subset of what you predict you can complete (in the iteration, or in the iterations planned).

Over time and with good care (i.e.: commitment to some rules of Scrum/Agile), a team's velocity is known. Starting out a release with a backlog of 500 points (projected), one can predict completing n points worth of stories, e.g.: a 5-member team, velocity average of 40 points per sprint, 8 sprints in the release timeframe equals 320 points completed. Beginning at the top of the backlog and working down, one draws a cut-line 320 points worth of stories down the backlog: that's how much will get done. That's a prediction.

For a commitment, however, it depends on comfort. Maybe one is comfortable committing to one-half (the 320 points), maybe more, maybe not even that much (especially if the early requirements—read: stories—themselves are not convincingly quantified or followed). Similarly, measure down the backlog to determine what can be committed to.

Over the ensuing months, however, maybe things will go better, maybe worse. Maybe the client will switch gears after seeing what's beginning to roll off the assembly line.* One increases or decreases his scope of commitment. You reach the point where you can say,

"We didn't know this at the start of the release, but X is really important to our customers. We will be able to deliver X in addition to what we have already committed. Without slipping the release date."

Making commitments in Agile isn't impossible. It just needs approaching differently than in waterfall. In waterfall, commitments aren't ever ultimately worth the medium in which they're made. With time, experience and a stable team, commitments in Agile become a great deal more believable because they're based on reality and shorter, more quantifiable periods of work.

* In particular, examine the boss' responses at the beginning and end of I want to run an Agile project, but client involvement, perhaps represented by the product manager, is crucial.

Similarly, I want to run an Agile project, part 2.


Continuous planning

There is a problem with commitments and the need for planning an entire quarter. This is waterfall thinking.

Commitment is the usual goal, but really, it's not what's needed. What's needed is delivery. The more often delivery happens, the better because the more that can be changed. Points:

  1. Even with small stories, estimates aren't perfect. Something might (hopefully) get finished sooner. Agile makes it so that a new story is pulled from the backlog and worked on. That's good.
  2. For too-large stories, under-estimation is common.
  3. If too much changes mid-quarter, it's "broken arrow." Do you want to be able to change and realign or just fail? Do you wait until quarter's end to realign?

Committing to a shorter cadence permits managing these problems. A novel idea is not to commit to anything. Instead, use continuous planning.

Do planning with small stories, short iterations. A longer roadmap is okay, but it's a wishlist and must not be committed to because it likely won't happen. Why commit to something that's impossible?

Commit to one iteration at a time. This encourages everyone to...

  1. Think small, short stories, managing work in progress and maintaining momentum.
  2. See interdependencies: the smaller the features, the clearer these will be.
  3. Plan smaller thing in less time reducing also the planning load for the iteration. A full quarter or release planning is a huge investment in time for something that, as noted, likely will not happen that way and gradual replannings will take place anyway. So just count on doing the "gradual" plannings.
  4. Use deliverable planning (a list of features) in a rolling wave, replanning as needed and as teams deliver.

Small planning means not gathering everyone together for long hours of pointless, time-wasting meetings. Small planning means involving those you must when you must, different ones at different stages.

Do a short kick-off with everyone, to be sure, so they all know each other and feel like a big team. This is, however, different from a big planning session where people are planning instead of delivering.


Scaled Agile Framework (SAFe)

I found this interesting in a heads-up sort of way. Most organizations suck at implementing Agile because of a) over-willingness to get religious and build empires and b) valuing tools and process over human intelligence and interaction because they think they can control the former while the latter is just herding cats. (Well, okay, it is herding cats, but tools and systems don't build software, people do.)

Fail SAFe

I think SAFe is onto something though I'm no kind of crusader, just an interested by-stander. This video introduction to SAFe is short and sweet. In the first couple of minutes, it does a remarkable job of illustrating the essence of Scrum, something people who think they know Scrum, but never did and bastardized it in order to build their own religious empire around it, should watch because they need to learn or be reminded of what Scrum really is.

SAFe in 7 minutes


Dumping Scrum for Kanban?

If you must react to changes and surprises in less time than an iteration, choose Kanban over Scrum:

When to dump Scrum for Kanban


Good links