Product #33
OpenAI debacle, Discovery Sprints, LLMs and more.
Sprint (2016)
I was lucky enough to come across this book just in time for it to greatly contribute to a piece of discovery work I was working on back in 2016. I felt like I learnt a lifetime of facilitation tips from a few hours of reading. It's incredibly practical and even though the one week Sprint format may seem a bit rigid, every time I revisit this book, I find something new and exciting.
“Every interview draws you and your team closer to the people you’re trying to help with your product or service.”
Key Highlights
Building a minimal product to validate an idea is a huge investment. First focus on getting real data from a realistic prototype.
A Sprint helps you assess what customer reaction to your final product will be like. Even Sprints that expose critical flaws have a great return on investment (imagine if you built the thing!).
Focus on the biggest question. Go after your most important problem. The bigger the challenge the better the Sprint. Make it high stakes. No problem is too large for a Sprint, not having enough time fosters creativity and focus.
Solve the surface first. Focus on the customer interaction parts of the product. Get the human interaction right, and you can work out backstage systems and processes later.
Limit your team to 7 people or less. You need some builders and a blend of expertise. Bring the troublemaker. It’s better that they feel invested in the project.
Pick a facilitator to keep things to schedule and organise everyone.
Clear the entire week. Stay in the same place so you can decorate it with ideas. Don’t allow folks to check devices inside the room. Make sure you have lots of whiteboard space.
Monday
Set a long-term goal: Why are we doing this project? Where do we want to be in 6 months, 1 year or even 5 years from now?
Capture the difficult questions that must be answered. What assumptions need to be true for us to succeed? What are the reasons why our project might fail?
Create a map of the experience from beginning to end. List the actors on the left, write the ending on the right, add words and arrows in between, keep it simple and ask your experts for help.
Knowledge is distributed across your organisation. Nobody knows everything. So interview a bunch of stakeholders about your problem space.
Choose a target for your Sprint. Narrow things down to the most important customer and most critical moment. Clustering How Might We (HMW) notes should make it clear what to focus on. Add the best HMWs to your map.
Tuesday
Start with lightning demos for inspiration. Think inside and outside of your company and industry. Everything should contain something you can learn from. Get everyone to demo their favourite relevant product or service 3 minutes or less. Capture big ideas a you go → create a whiteboard of ideas. Combine ideas with HMWs or the map from yesterday.
Start coming up with and sketching solutions. Sketching helps you explain what you mean. It’s the fastest and easiest way to transform abstract ideas into concrete solutions and communicate them. Work alone together → you’ll generate more solutions, you can do more research, find inspiration and think about the problem more deeply this way.
Four-Step Sketching Technique:
Gather key info in note form {review your inspiration} (20 mins)
Doodle rough solutions (20 mins)
Try rapid variations {using crazy eights = folding paper into 8) (8 mins)
Figure out the details {sketch the solution} (30+ mins)
The best sketches are:
self explanatory
kept anonymous,
words are chosen carefully (copywriting)
have a catchy title
Wednesday
Decide what sketches to prototype. Put the sketches on the wall, look at solutions in silence, use dot stickers to mark interesting parts. Quickly discuss highlights of each solution, use sticky notes to capture big ideas. Get each person to choose one solution (by voting with a sticker). The decider(s) make the final decision (by voting with stickers).
More than one great idea? Either combine them or build them both and pit them against each other in user testing on Friday
Storyboard your winning ideas in more detail (15 frames). Choose an opening scene. Fill in all the detail. Make sure you can test it in 15 minutes.
Thursday
Build a goldilocks prototype: do just enough work to build something that gets a genuine reaction, without wasting too much time on polish.
Focus on the facade, the bit the customer interacts with.
You should have everything you need to move fast, Thursday is just about building. You should have basic components, copywriting and details already.
Make sure you use the right tools (keep it rough with Keynote)
Assigning different roles will help you move faster
Makers: create individual components.
Stitcher: collect components and combine them seamlessly
Writer: make all the text realistic
Asset collector: grabs images, icons and sample content
Interviewer: writes the interview script
Stitch everything together at the end and make sure the story is consistent (e.g. numbers, dates, prices, names, labels)
Friday
85% of problems will be exposed after just 5 interviews. The fastest way to find issues it to run 5 interviews, address problems and test again.
Interviews give you a signal as to wether the product is effective or not. They also tell you why. They’re easy to do well, just be friendly, curious and open to having your assumptions proved wrong.
The five act interview:
A friendly welcome
Some general open-ended context questions about the customer
Introduce the prototype (make it clear you didn’t design it and want frank feedback)
Tasks designed to get the customer to react to the prototype. Nudge: what’s this? what do you think that’s for? What do you expect that will do?
A quick debrief (to capture customer’s overarching thoughts, impressions). How does it compare to what you do now? How would you describe the product to a friend? If you had 3 wishes to improve the product, what would you do?
If comparing prototypes… Ask which is best, pros and cons and how would you combine them
Collect interview insights on a whiteboard grid. Columns are customers, rows are parts of your product or user experience. Get everyone to put up their sticky notes of quotes, observations or interpretations from the interviews . Colour code the notes (Green for positive, red for negative, black for neutral). Look for patterns across the whiteboard (the more customers react in the same way the stronger the signal). List them on another and label them as positive, negative or neutral.
Review your long-term goal and sprint questions. Can you answer them now?
Full Book Summary · Purchase on Amazon
In the News
The OpenAI saga grabbed the headlines this week. Rapid progress in the capabilities of LLMs have hinted at a possible text based path to AGI. It's fascinating to see major companies in the field recognising the transformative potential of AGI and even hiring science fiction writers to envision the consequences of its genesis. It’s not surprising there have been power struggles in the space, this isn’t even the first. Many think Microsoft played a great hand this week. The truth is they they got away with one. Investing $10B without appropriate oversight was a big risk.
Quick Links
Is Andrej Karpathy the Richard Feynman of AI? Watch his 101 on LLMs · Video
Avoid these 10 common Product Strategy Mistakes (Roman Pichler) · Video
106 Cognitive Biases & Principles That Affect Your UX · Website
Best practices for naming design system components · Website
Could this antifragile ABC goal system work in product? · Article
LLMs can strategically deceive their users when put under pressure · Paper
Sleep regularity is a stronger predictor of mortality risk than sleep duration · Paper
The Unreasonable Effectiveness of Data (2009)
Alon Halevy, Peter Norvig, and Fernando Pereira.
Sciences that involve human beings rather than elementary particles have proven more resistant to elegant mathematics….But if that’s so, we should stop acting as if our goal is to author extremely elegant theories, and instead embrace complexity and make use of the best ally we have: the unreasonable effectiveness of data.
Key Highlights
Even in 2009 it was clear. The authors argue for the importance of leveraging the vast amounts of available data in improving natural language processing and other machine learning applications, suggesting a shift from seeking elegance in models to embracing the complexity and richness of large-scale data.
View the Paper
"If you ever find yourself saying something like that’s not my job, that’s probably a thing you should do.
And you know what? It probably isn’t your job and is someone else’s job. And you can spend you life getting frustrated at that. Or you can just get over it and get the work done.
People who are willing to just get the work done will move faster.”
Maggie Crowley, Product at Toast
Book Highlights
When you’re deciding how to fix a usability problem, the question you should always be asking is “What’s the smallest, simplest change we can make that’s likely to keep people from having the problem we observed?”
Steve Krug · Rocket Surgery Made Easy
Every time we do this we find holes. We find things that we thought another team should be taking care of, but it didn’t know. We find the necessary stuff in between the big important features that we’d forgot to talk about.
Jeff Patton · User Story Mapping
Quotes and Tweets
It's been over a decade since I started working as a product manager. I've worked with, reported to, and mentored some of the most amazing PMs in the world. And you know what I've noticed? Most of them think of success metrics as a complicated rocket-sciency subject. When it is actually a simple thought exercise instead. So here is everything I know today.
Things to remember:
Limit total metrics to 5-8
Don't over complicate them
Metric is an answer to a question
The goal is to ask the right questions
Don't use metrics that can be cheated
There is never a "perfect" list of metrics
Think of ALL users when creating metrics
Keep your approach simple
1. Think in broad buckets
2. Determine primary aspect to measure
3. Think of questions you want to answer
4. Find a metric that answers the question
5. Avoid common mistakes
Just Another PM · X JustAnotherPM