Product #42
Build by Tony Fadell, How Complex Systems Fail, Large Context Windows and More
Build · Tony Fadell · 2022
Tony Fadell was the driving force behind the iPhone, iPod, and Nest Thermostat, some of the most iconic products of all time. His book, "Build," is a unique blend of autobiography and product development playbook. His insights into Google's acquisition of Nest resonated with my own experiences as a Googler. The world needed his chapter on ‘Assholes’ and how to deal with them. Fadell's framework for effective product messaging is invaluable. Anyone passionate about creating exceptional products should consider this a must-read.
Key Takeaways
There are too many mediocre companies and products out there. Strive for excellence - push those around you to do the same.
Dedicate yourself to a problem you're passionate about. Find your community. If nobody is working on it - you may be too early.
There are two types of decisions:
Data-Driven: Acquire, study, and debate facts and numbers that will allow you to be fairly confident in your choice. Easy to make, easy to defend, and easy to get agreement.
Opinion-driven: Follow your gut and your vision - without the benefit of sufficient data to guide you or back you up. Hard to make and always questioned.
You can't turn an opinion-driven decision into a data-driven one.
Know the different types of assholes (political, controlling, aggressive, passive-aggressive, mission-driven). Pushing for greatness, challenging assumptions, and not tolerating mediocrity doesn't automatically make you an asshole.
Quit if you're no longer passionate about the mission, or you've tried everything and spoken to everyone and it's still not working.
Design and prototype the entire user journey. Don't just make a thing - make a better user journey. Every customer touchpoint is an expression of your brand and product.
Test prototypes with real customers as soon as possible.
If hardware doesn't absolutely need to exist to enable the overall experience, then it should not exist.
Product storytelling is really important - for both the customers and your team. Once you understand why your product is needed - you can focus on how it works.
Version 1 of your product should be disruptive. Opinion-driven vision > Customer Insights > Data. Aiming for product-market fit, not profitability.
Version 2 is typically an evolution. Decisions can be based on data and customer insights once you are evolving your product - look for opportunities to disrupt yourself while aiming for profitability.
You make the product. You fix the product. You build the business. Every product. Every company. Every time.
Tony Fadell
The disruption tradeoff: Not so disruptive that you won't be able to execute - Not so easy to execute that nobody will care.
Handcuff yourself and your team to a deadline - constraints make you more creative. Then codify your delivery process into a heartbeat - that sets the pace for product development.
Good ideas solve a customer need that's important and frequent. Good ideas will follow you around - they persist in your mind.
On Org Design: Break your org into product-specific groups so each product gets the attention it deserves. New products need new teams, otherwise, they'll never get made. Decisions speed up and everyone has a shared goal rather than conflicting priorities.
Design thinking means thinking through a problem and finding an elegant solution. Don't outsource a problem unless you've tried to solve it yourself. Try to notice things and avoid habituation, getting used to the inconveniences.
Product messaging is key. What should you say? - Where should you say it? You can't say everything everywhere, so you need to get it right.
Understand your customers' pains → map them to a pain-killer in your product.
Test the messaging to check if it resonates.
Map your customer touchpoints - and work out where each piece of messaging should be displayed.
The product manager's responsibility is to build the right products for the right customers.
The things a CEO pays attention to become the priorities for the company.
Do something meaningful - make something worth making.
In the News
During PR training at Google, you are instructed to avoid discussing the competition and instead highlight the positive aspects of your product. Meanwhile, at Meta… Zuck Video
Google's new Gemini 1.5 Pro model has a context window 10 times the size of Chat GPT-4 (which could already swallow a book). This is super impressive, and recall doesn't seem to degrade when it's full. This made me wonder if we'll actually need Retrieval Augmented Generation in the future? Vivek has a great video on this, and it turns out we'll still be using RAG in the future because of latency, cost, and the injection of live data from APIs. Google Announcement · Vivek Video
Quick Links
Eminem's Boxes of Notes · Article
Product Discovery: Pitfalls and Anti-Patterns · Article
The Gartner Hype Cycle Methodology · Article
In praise of shadows · Article
Thoughts on a Global Design System · Article
How to build better AI products with user research · Article
The ultimate guide to willingness-to-pay · Article
How to Balance Keeping The Lights On With Innovation · Article
Measuring experiences, not product use · Article
The Life-Changing Power of Shutting Up · Article
The 9 Fundamental Ingredients to Thrive with Product Management · Article
How Complex Systems Fail · Richard Cook · 1998
This paper provides crucial insights into the failure mechanisms of complex systems. The work is foundational in safety critical fields like healthcare, aviation, and technology.
Book Highlights
A recommender system can be greatly optimised by adding the output of several algorithms. Hybrid recommenders enable you to combine the forces of different recommenders to get better results… The feature-weighted linear stacking (FWLS) algorithm enables the system to use feature recommenders in a function-weighted way, which makes it strong.
Kim Falk · Practical Recommender Systems
Very rarely does a product initially hit the market in its final, successful form. Instead, it evolves with users over time to eventually fit into their lives. By doing user research and understanding what the market needs, you can save yourself the disappointment (not to mention cost) of a failed product launch.
Amber Case · Calm Technology
Often, design feedback from stakeholders isn’t taking these goals into account at the outset. People look at something and react, often without considering the original intent of the project.
Tom Greever · Articulating Design Decisions
From the start, I wanted PayPal to be tightly knit instead of transactional. I thought stronger relationships would make us not just happier and better at work but also more successful in our careers even beyond PayPal. So we set out to hire people who would actually enjoy working together. They had to be talented, but even more than that they had to be excited about working specifically with us. That was the start of the PayPal Mafia.
Peter Thiel · Zero to One
Quotes and Tweets
Strategy is hard because you can't A/B test your business. There's only one business but infinite opportunities. Once you make a choice, you will never know if the other opportunity was better or not. This is why successful founders all had strong belief in what they were doing. Any Murphy
*RAG = Retrieval augmented generation
7 Key Metrics to Evaluate RAG Pipelines with LLMs
RAGs leverage external data to enrich the context of LLMs, thereby enhancing their ability to generate more accurate and relevant responses. As the adoption of RAGs grows, so does the complexity of evaluating their performance effectively.
1. Faithfulness:
This metric assesses the degree to which the generated text accurately reflects the information present in the source documents retrieved by the RAG system. Faithfulness is crucial for ensuring that the augmentation process does not introduce inaccuracies or distortions, maintaining the integrity of the generated content.
2. Answer Relevancy:
It measures how relevant the generated answers are to the queries posed. This metric is vital for determining the utility of the RAG pipeline in practical applications, where the goal is to provide users with information that is not just accurate but also directly applicable to their questions.
3. Context Recall:
This evaluates the RAG system's ability to retrieve all relevant information from the external data sources. High context recall is indicative of a system that can comprehensively utilise available data, a critical factor for generating well-informed and complete responses.
4. Context Precision:
In contrast to recall, context precision measures the proportion of retrieved information that is relevant to the task at hand. This metric ensures that the RAG system efficiently filters out extraneous data, focusing on quality rather than quantity in its augmentation process.
5. Context Relevancy:
It combines aspects of both recall and precision, evaluating the overall relevancy of the context utilised by the RAG system. This metric underscores the importance of a balanced approach to data retrieval, where both the breadth and specificity of information are optimised.
6. Answer Semantic Similarity:
This metric gauges the semantic alignment between the generated answers and the ground truth (or expected answers), taking into account the nuances of language. It is essential for verifying that the RAG system captures the underlying meanings and not just the superficial aspects of responses.
7. Answer Correctness:
Beyond relevance and semantic similarity, answer correctness directly assesses the accuracy of the information provided in the generated text. This metric is fundamental to ensuring that RAG-augmented LLMs act as reliable sources of information.