“Zero to Metrics-Driven Hero”: Courtney Shar with Splunk (Video + Transcript)

In this ELEVATE session, Courtney Shar (Senior Program Manager at Splunk) discusses organizing the flow of work to sophisticated forecasting models. She shares how the team learned to bring metrics into each part of their own development lifecycle, and how the team was able to measure success with their consumers.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies!
• Watch more ELEVATE 2024 videos from the event, or just the “Best Of 2024” Videos!
• Does your company want to sponsor a Girl Geek Dinner or Virtual Conference? Talk to us!

Transcript of ELEVATE Session:

Courtney Shar:

Thank you. I am so happy to talk to all of you today like they introduced. I am Courtney Shar. Welcome to Zero to Metrics-Driven Hero. I am a senior program manager who works at Splunk

Today I’m going to take you on a journey from zero to hero. I hope that by sharing this journey of a new team, learning how to use metrics that you gain ideas and inspiration you can take with you and make your own heroic team.

Like any good story of a hero’s journey, we must start at the very beginning. The team lead, Sean joined the company in 2012 as a software engineer and ended up on a backend services team where he spent the first five years of his career. He enjoyed working on this team a lot and after five years you can imagine he was an expert in how the team operated, including how they practice agile methodologies. He was then offered to become a team lead for a brand new team and was excited to take the opportunity on day one. The so-called team was just him and his new manager and wasn’t long after, however, they were given two software engineers and a project manager, all three of whom were brand new.

Now, when they walked into the first planning meeting, they didn’t do it as a team. Everyone had different assumptions based on their previous experiences. As expected things started falling apart almost immediately there was confusion around how and what to story point and scope on the stories. They ended up abandoning, story pointing and any attempt to do iteration planning failed to.

There were a couple more issues in that meeting that I’d like to mention as well. The fact that the project manager was new and was on a new team resulted in a tuck of war between three different needs. The need for Sean and his manager to help coach him as he grows into his role. The need for the project manager to be recognized leader on the team by leading the agile ceremonies, including the planning meeting and the need for the team to sort out all of the issues they were running into.

The result was that they did a mixture of trying to satisfy all three, but in the end they didn’t get anywhere. The last thing I’ll mention about this meeting is the retrospective portion. Sean had questions that had worked on his previous team, assuming it would work here as well. As they began to work through the questions, they didn’t understand ’em all. Also, not all the questions even applied to the teams since they were operating somewhat differently than the team they had stolen the questions from.

Lastly, and I believe this is the most important part, since they had blindly copied the questions, they didn’t understand why they were answering them, so they didn’t get any value out of doing so. They were just going through the motions. After all that, they decided to end the meeting, think through things and regroup. At a later time in the aftermath of the first planning meeting, Sean began giving the processes the attention they deserved.

He was by no means an expert at agile development methodologies, but did recognize some things that needed changing, such as the story breakdowns. It was then that he took the team’s very first step towards greatness. He took the current projects and reorganized the task into much smaller bite-sized that could be completed in a single iteration and showed these smaller tasks to the team as an example of how to organize things moving forward.

It wasn’t a change he was able to make easily on a personal level because he spent years working with tasks that were vague and broad and had grown to accept that as an appropriate way to track projects. But he pushed himself and before long the positive results began to show. In the second planning meeting, they came in knowing they were going to point the stories and that they were now broken down into more manageable tasks.

They began the first story pointing session. It was rough, but this time around it was possible. They were able to understand each story scope and give something more than a wild yes as to its complexity. Now when I say it was rough this time I’m mostly referring to what many of you probably would recognize as typical difficulties for a new team when story pointing, coming to a common understanding of what a particular number of points means, as well as not being afraid to disagree with other team members and voice a unique opinion.

These difficulties would prove to naturally smooth out as they all continue to storypoint together. Another step the team took in the right direction was to take an agile bootcamp course as a team. This helped them level set on the agile processes, giving all the new members and idea of how things should ideally be done.

After they did this, the whole team was able to understand where they were in the process and where they wanted to be and could help contribute to minimizing that gap. At this point, the team had started making some progress towards functioning well and in an agile way. Things were still rough, but they were story pointing, breaking out tasks and retrospecting. They started to get into something resembling a successful rhythm. That is until the first bigger and planning session.

If you’re not familiar with the SAFE framework, bigger and planning also called program increment or PI planning, it’s where related teams work together to establish their mission, vision, and high level iteration planning over a certain set of time. This team did this quarterly. The first bigger planning session was stressful and confusing. Sean’s team was new to the organization and no one on the team had experienced it before.

As a new team joining a different organization than been doing it for a while. They were missing context that was key to getting value from it. Instead of understanding why they were going through this process, they suddenly found themselves scrambling just to check all the boxes that everyone wanted ’em to check with no time to understand why the result was two days hold up in a meeting room, struggling to pull together some sort of educated guess as to what they could accomplish in the next quarter.

Unfortunately, no one on the team had the knowledge and experience to really understand this yet or where they needed to go from here. They were stuck and needed help. This is where I come in. I was pulled aside by a manager and asked to come in and look into the team. We’d realized that a new project manager with a brand new team wasn’t the best idea at the time.

I’d already had five agile teams, so I thought I could go in quick fix and then I could work on maintaining team stability. I did a one week observation of the team and immediately had come up with kick fixes. However, I noticed the team constantly had a changing scope and we needed to start pulling numbers.

I looked into both actual Agile and Kanbans with the intention of getting the best of both. I’ve linked both of them here. Now the cool thing with actual Agile was that you could use it with any tracking tool that we used at the company at that time. In this team’s case, it was Jira and I could run it locally on my device with kanon sim, I listened to a presentation about how you could use it for PI planning. That allowed me to start including out of office time and my guesses of when we would finish.

Now I’ll go over the different phases that the team went through. For phase one, the team focused on establishing work in progress limits with color visuals when limits were exceeded based on one piece of work per engineer, we ensured the board visualized the workflow by removing the block status and ensured that each column had a clear definition of done before moving to the next status.

For this team, we removed the block status and instead added a flag to blocked items as they could become blocked anywhere in the workflow. We also made sure that we had clear done criteria such as you have to have testing in your code before you put it out for review. In phase two, we began measuring and using metrics in earnest.

Specifically, we needed to focus on measuring and understanding the team’s cycle time and throughput. Measuring the data wasn’t enough, however we needed to start discussing it in our retrospectives. The cycle time scatterplot is a graph representation of all our JIRAs and how long it took for them to close. For this team, we decided that any story taking longer than 10 business days needs to be discussed during our retrospective. So when preparing for that meeting, we check the data for stories that fall in this category. As you can see in the screenshot, there was an outlier task that took 25 days to close. So you can bet we discussed this in the retrospective meeting.

Cumulative flow diagrams are basically fancy ways to monitor the CU of the team. This is particularly helpful in practicing lane where there is a focus on keeping work in progress items and cycle time. At a minimum for this report, the team generally ignored the graph and instead focused on the table, which listed out the average amount of time all the stories during a specific time span spent in each state. When looking at the data, the team focused the most on our time spent in review as that was a status that we found we had a tendency to get stuck in for too long. Also, we had the most control over this as the tendencies were always within our team.

With this chart, we can see how the team is doing throughout the project and see if there’s any stories that go over the threshold goal. The team had a goal to finish all stories under 10 business days and with this chart we can review all the stories that were close to the 10 day threshold and be sure to discuss them before they become an issue. For phase three, the team focused on prediction models for forecasting and deriving estimates. Mike Carlo is a technique for forecasting the probability of finishing the deliverable. On time within this team, we decided we’ll have a goal with 70% chance that we’ll finish on time. For our executive, we communicate an 85% chat state. Then for our clients, we worked off a 95% chat state that we would finish on time. This is all due to how comfortable we would feel about communicating risk acceptance to the stakeholders, the team, the executive and the client.

On this slide you’ll see two different visuals for the Monte Carlo simulation. For the first one, how many, we used this to talk to the team about how many stories we’d be able to complete and tell the next retrospective to make sure everyone was on the same page. We also looked at this version of Monte Carlo win to see if we were still on track for our end of quarter or release commitments. This was pulled every retrospective so that the team needed to stay late. We discovered this before the week something was due. No actionable agile will only flex based off of previous cycle times,

This is where I’ve introduced a new tool with the team. When I’m at this phase, we started looking at what we can track and making my simulation more intelligent. As I learned more about the team, what the Monte Carlo tool before could not show me was if I should be worried. Earlier on. For example, during December there were always many engineers taking a vacation or traveling, and my previous simulations I showed before will only tell me how we’ve even doing so far and not that I need to count for being short one or more people for an entire month, which would put our commits at risk before even starting the project.

For this team in particular, I focused on scope creep. I then start to gather numbers around how much scope creep we take in on average and use it for running my simulation, which Al Agiles Monte Carlos simulation did not run through. I’m also able to calculate any holidays or vacations that come up and take those out of my simulation. I continue to use the cycle time generated earlier, which I can get auto generated from actual Agile or calculate from a CSV export and then I can use my calculations for the average amount of scope creep that team has. I also included how often development bugs were found and included that in my simulation.

I would first run the simulation with 25 Monte Carlo cycles and that showed a 96% chance of it finishing on the 15th. Then if for fun, I wanted to increase it to running 10,000 cycles. You can see it only changed the completion date by one day, which wasn’t a large enough difference for me to communicate anything different to the stakeholders.

By using the tools I demonstrated to you, I was able to find targeted areas for improvement. In addition to the normal biweekly retrospective, I worked with the team on two others. We first focused on a project retrospective where I’d ask the team questions, they would rate themselves on a scale of one to 10, which then ended with a grade. As the team was constantly working with outside organizations, we came up with polling the teams we worked with every quarter.

By doing this, we were able to recognize ways to improve things for future projects and fix any issues that we have for longstanding projects such as those that we knew we’d be working on for more than a quarter.

Here are the things we came up with for our projects. Did we update the project page in a timely fashion? Did we use real life numbers, metrics to drive project scope and feature decisions? Did we design, develop and test collaboratively? Did we deploy frequently and integrate early? Did we plan and execute the project well and without last minute, all hands on deck effort? Were agile ceremonies well run and effective?

Did we design for passivity? Did we also validate passivity before we deployed? Did we meet our release milestone? Did we resolve all identified functional issues prior to release? Did we tie up all loose ends after the project was finished and did we resolve any previously existing technical debt? Did we document project scope, leftovers requirements, concept pages, et cetera? Did we measure performance impact on the system and create no harm? Did we build a product that was good enough to make available to end users right away?

Did we use our time effectively? Do we invite all the appropriate stakeholders to the meetings? Did we avoid the need for a break fix cycle for issues uncovered by early adopters? And finally, did we identify and implement success metrics to track early adoption? Are they showing us the success?

Here were the items that we came up with for the curly survey that we sent out to teams that we worked with. We asked them, did we leave any functional issues unresolved prior release? Did we meet our release milestones? How well did our team respond to your request for changes or updates in a timely manner?

We asked several process questions. Well once again asking about documentation, how we documented scope requirements designed, were our scrum sync meetings held regularly and were they effective? Did we plan and execute the project well and without an all hands on deck effort? What other feedback did they have for us regarding the project’s process? And we also asked, how well did our team follow your documented development processes? In the kickoff, we wanted to know how well the project scope and project roles were communicated and understood.

We also wanted to know about specific collaboration that we use with them. Did we develop functionality that team was going to own longterm or did we both develop functionality that needed to integrate together implementation? How well did our team design for passivity? Did we leave any loose ends? Did they have adequate time to give attention to this project? And how well did we deploy frequently and integrate? And then for the feedback loop, we want to know about any other feedback they had about the collaboration effort and if we needed to have additional meetings to discuss any future efforts that we would be working out together.

By making the retrospectives metrics driven, we were able to start measuring and putting action plans based on feedback and collaboration. By doing this, we were able to move the team towards becoming lean by focusing on turnaround time and amount of work completed. Our standups turned into updates and daily planning. The team changed to story pointing everything at a three, as that was determined to be a smallest amount of work. We used that breakdown to continue to have the story pointing discussions that we had while using Scrum, but we turned the talking points and the focusing on how we can make our stories even smaller. Then we looked into how we were doing our estimates. Of course, there was a learning curve and even things that our team or stakeholders thought were implied in the scope. We started to measure how many new stories and tasks were logged outside of our initial estimate to come up with an average of scope pre log per quarter that we then used to make our overall project estimation even better.

Things that we learned while going through these changes such as the team is from scope creep, so we track their scope creep stories and by catering retrospectives to results, we included our numbers and discussed them with the team so that they could have full visibility. Tour went on and not be surprised by anything we were tracking on for releases or for bigger planning. And by including the engineers in the team and coming up with the retrospective format, we were able to get better feedback and cut down on the amount of stories that were greater than 10 business days. This helped to build a safe space for the team to talk about issues that were going on with them and allowed them to feel safe with constantly contributing feedback to make the team better. Overall, I think we did a pretty good job now to help your own team go down in history.

There’s a few things to keep in mind. First, use metrics to drive your decisions when planning to sprints and in bigger in planning. Next, find the metrics that your team needs to use to estimate. As I said before, this team is prone to scope creep, so they tracked the scope creep stories and added the percentage of scope creep stories from previous quarters to the estimates for future quarters. Finally, be persistent in your pursuit of becoming agile hero. Not everything that you try will work and not everything that works for all the teams around you will work for your team.

Last but not least, remember to ensure that you and your team understands why you are following the processes that you are or else you’ll receive. You’ll risk not receiving any value from them. I know we’re short on time, so if there were any questions that were asked and we don’t have time to answer. Once again, thank you for your presentation and reach out to me on LinkedIn.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies!
• Watch more ELEVATE 2024 videos from the event, or just the “Best Of 2024” Videos!
• Does your company want to sponsor a Girl Geek Dinner or Virtual Conference? Talk to us!