🧘‍♂️
Jin's Muse on Engineering
  • About Me
  • Leadership
    • About Work
      • Work Like Watching a Movie
      • The day you leave …
      • The inner game of work
      • Routinization - how great institutions become mediocre
      • Have You Lost Your Stories
      • I wish I knew when I was facing layoffs
      • How to get out of this boring job
      • Why Setbacks Can Be Good for Your Career
      • What do we get from returning to office?
      • What if this is the last year you work here
      • Learning, Balancing, Falling - Don’t Stop
      • Who is the boss?
      • Under the Emperor’s New Clothes
      • Are you getting the right opportunities?
      • The elements of a healthy organization
      • Do What You Love vs. Love What You Do
    • About Life
      • Accept the things we cannot change ...
      • Live backwards
      • Life, one breath away, one breath a time
      • Will to Power
      • Learn from stand-up comedian
      • Connect the dots - submit to life’s entropy
    • Leadership Pinciples
      • The thing about Amazon’s leadership principles
      • Vision, Mission, Roadmap - But First, Eat What You Cook
      • Calibration - how we compare
      • But We are on the same side!
      • Leaders need two plans
      • Whose results are you delivering?
      • Coaching through open ended questions
      • This is the new year, and who will you delight?
      • Anti-fragility and Over-thinking
      • So ... what’s your decision
      • No Super Hero in Team
      • Congrats, you are the owner!
    • Communication
      • What, So What, Now What
        • Just tell me what’s next!
      • The Power of Repetition
      • Passing the Story Along
      • Lost in Translation
      • Chopstick War
      • How you speak about the most impactful project
      • Are we really listening
    • Problem Solving
      • How we resolve conflicts
      • Problem Solving: Leader Define Process and Better Leader Define StandardPage
      • If Kant and Nietzsche read Annie Duke's "How to Decide"
  • Engineering
    • Best Practices
      • Accuracy Ain't Always the Truth
      • No excuse for bad engineering
      • Secret of good engineering: constraint, not more time, or resource
      • The Art of Picking the Right Problems
      • Choosing the Right Battle: The Path to Productivity and Satisfaction
      • The Power Balance in Software Development
      • Three Ways of Invent and Simplify
      • Work backwards from demo-able sprint goals
      • That last 1%: from good to great
      • When it is too hard ... don't do it!
      • What Engineering can learn from Sports: Time it!
      • Struggling with Sprint Goals? Try Working Backward
      • Why working on a “legacy service” is a good thing
    • Operational Excellence
      • On dashboard
        • Dashboard Quality Vs. Product Quality
      • How we turn operation problems into big data innovation
      • Don’t automate 100%
        • 99% Automated - deleted an on-call rotation!
      • How we got rid of log diving
      • No perfect runbook - map is not territory
      • Someone have to be oncall, why they can’t be you?
        • Why being oncall is a good thing
        • If Sherlock is On-Call ...
        • Inner Peace During Oncall
      • Monitoring Your Service Health Like Human
    • Software Architecture
      • The Curse of the "Next Generation Project"
      • Absurdity in Decoupling
        • Tension of Decoupling and Cohesion
      • The ideal software form
        • Composition Over Inheritance
      • Security and Safety
      • Architecture Properties for Security Services
      • Learn system design from Git - Multilayered Architecture
        • Learn system design from Git - Immutable content-addressable datastore
        • Learn system design from Git - Event Sourcing
        • Learn system design from Git - Evolutionary Architecture
      • Quality starts with local stack
      • Intention-Revealing Interfaces with Examples
      • Design Principles
    • Distributed Computing
      • The beauty of multi-process applications
      • Cook Concurrent, Parallel, Async, and Non-blocking Together (1)
      • Cook Concurrent, Parallel, Async and Non-blocking Together (2)
      • Cook Concurrent, Parallel, Async and Non-Blocking Together (3) - The Multitasking Myth
      • The Local and Global Optimization Trade-Off in System Design
      • Throttling, Traffic Shaping, Traffic Shedding and Circuit Breakers in System Design
      • Differences Between Service Discovery and Load Balancing
        • Load balancer
      • Power of Two Choices
      • Using High Percentile Latency to Detect Resource Constraints
      • Queuing's Impact on High Percentile Latency
    • Machine Learning
      • Are you underfitting or overfitting?
      • How to lead an underfitting team
      • How to lead an overfitting team
      • Project Management and Neural Networks
      • Cryptography and Machine Learning
    • Programming
      • Can you create a language interpreter?
    • Cryptography
      • Explore KMS with CodeWhisperer (and a Dash of Cryptography) - AES-GCM
      • Exploring KMS with a Dash of Philosophy - HASH
      • Explore KMS' RSA encryption: a tale of two keys
      • Explore KMS - the monkey business of Elliptic Curve Cryptography (ECC)Page
  • Project Management
    • Setting goals
      • The Words of Goals
      • Why deadline is a good thing
      • Can you ship it?
      • One Thing Worth Remembering Per Quarter
    • The art of project planning
      • Are you still playing poker at scrum planning
      • Plan and Planning - How we start a project
      • A project’s journey - from inception to reflection
      • Component Team, Feature Team and Tiger Team
      • Sprint demo is a stage
      • Milestone and date
      • Onboarding project
      • Have You Sent Your Status Reports
      • When do you know a project smells
      • Hackathon is coming
    • How to prioritize
      • How to prioritize your 700 ticket backlog
      • Kitchen Sink and Tech Debt
  • Product Management
    • Find Problem
      • Dare to be 10x better
      • It is better to do the right problem the wrong way …
      • The Art of Goal setting: reactive goals, proactive goals, and inspirational goals
      • The things we don’t do …
      • No, I don’t want a platform
  • SDE Career Development
    • Promotion
      • The thing about promotion
        • Write your own promo doc
          • How to write your promo doc
        • The Reason Not To Promote
        • The myth of promotion project
      • The thing about leveling: are you SDE I, II or III?
      • What a L5 SDE is expected in Amazon
      • Customer Obsession - Of Your Own Promotion
      • Still waiting for that perfect manager?
      • You and Your Manager
    • Being SDE
      • What does a KMS SDE do?
      • From Knowing to Mastery: SDE Levels and Guild Ranks
      • How do SDE paint a Tiger from a Cat
      • The need to code
        • So ... where is your code
        • Quantity and Quality - How many line of code did you write?
      • Learn the Machine First
      • Is college major important?
      • Become an industry expert
      • How, What and Why in Problem Solving
      • Integrate both Breath and Depth in Life Long Learning
      • Do one thing well
      • CS students should know computer science history
      • Why Micromanagement is good for you
      • The power of imprinting
    • The Different Levels of Diving Deep
    • Becoming a tech lead
      • The traits of top engineers
      • Road to L6 SDE
      • There is no such thing called L6 task
      • Become a SME
      • Owning a Project vs. Owning a Problem Space
      • Leading, coaching and instructing
  • SDM Career Development
    • What does a SDM do
      • The thing about SDM - organization leader
      • The thing about SDM - manager of managers
      • The thing about SDM - line manager
      • The satisfaction of being a manager
      • Coach first, manager second, learner always - how I learn to coach
      • Gardener vs. SDM
      • SDM wearing three hats
      • What is a SDM doing in a design review?
    • How to be a SDM
      • How not to Micro-Manage - Stopping Being A Task Master
      • How SDM Build Momentum And Connect the Dots
      • How a SDM shoots at a moving target
      • The Danger of Over-Abstraction for Managers
        • How AWS SDM Dive Deep (1) - Dashboard
        • How AWS SDM Dive Deep (2) - Datastore
        • How AWS SDM Dive Deep (3) - Infrastructure
        • How AWS SDM Dive Deep (4) - Deployment
        • How AWS SDM Dive Deep (5) - Correction of Error (COE)
        • How AWS SDM Dive Deep (6) - OPS Review
        • How AWS SDM Dive Deep (7) - Bar Raiser
      • You are a manager, but are you a coach?
      • What an SDM can learn from swimming coach
      • Building a Team like a Waterpolo Coach
      • What a SDM can learn from a new programming language?
      • Should a SDM be able to code like a SDE?
      • Manager, Leader and Ruler
      • Performance Evaluation - Scope, Complexity and Impact
      • A Manager’s Vision
      • A Manager's Judgment
      • Build team with a soul
      • Build a learning organization - tech talk
      • Should You Become a Manager?
      • Becoming a SDM
    • Lead Business Application Team
      • Platform Team and Business Application Team
      • Page
    • What should you expect from your manager?
    • Build a Team like a Human Learning System
    • A New Manager's Tribal Survival Guide
    • SDM should Advocate for Intention Revealing Interface
    • A 30 minutes daily standup? You are doing it wrong!
    • Project Status Meeting that Takes an Hour? You are doing it wrong!
    • How to Ask Questions as a New Manager
    • Constructive Feedback: A Managerial Dialogue
    • Distinguishing "Improvement" from "Development" Feedback
  • Interview Tips
    • Level of Your LinkedIn Profile?
    • Make it personal
    • Tell me about yourself in 3 minutes
  • Import
Powered by GitBook
On this page

Was this helpful?

  1. Engineering
  2. Operational Excellence
  3. Someone have to be oncall, why they can’t be you?

If Sherlock is On-Call ...

PreviousWhy being oncall is a good thingNextInner Peace During Oncall

Last updated 1 year ago

Was this helpful?

Imagine Sherlock Holmes pacing around 221B Baker Street, when suddenly his pager blares to life. "Ah, Watson! The database has collapsed, and all of London's e-commerce is in peril!" Before he dives into a forensic investigation, what's the first thing he'd do? Mitigate, of course. Because even a legendary detective knows you've got to stop the bleeding before you can find out who's holding the knife. So, here's how Sherlock might go about it, with a dash of wit and a whole lot of wisdom. "Stop the Presses, Watson!" * Mitigation: The Immediate Response Before Sherlock can solve the mystery, he has to make sure London can still buy their trinkets online. In tech-speak, this is all about isolating the issue and implementing a quick fix—like diverting traffic, rolling back a release, or launching backup servers. It's not about knowing who did it; it's about making sure no more damage is done. -- Why It's Priority #1: Stopping a crisis in its tracks ensures you'll still have a job tomorrow. It buys you the time you need to then step back and put on your detective hat. * Investigation 1. Data Enumeration: The Unbiased Witness "The Game is Afoot, But First, Data!" Once the immediate crisis is under control, Sherlock turns his attention to gathering evidence. Just the facts, please. He doesn't want opinions, assumptions, or early hypotheses to muddle the waters. -- Why This is Vital: If you don't have clean, unbiased data, you're starting your investigation on shaky ground. It's like accusing Mrs. Hudson of foul play just because she makes a mean cup of tea. 2. Hypothesis Formation: The Usual—and Unusual—Suspects "No Watson, We Can't Blame Moriarty Yet!" After collecting factual data, Sherlock formulates hypotheses. Did Moriarty tamper with the firewall? Did Watson forget to update the SSL certificate? Hypotheses abound, but they're kept separate from the data. -- Why This Matters: Letting hypotheses and data mix early on is like jumping to conclusions without a parachute—you're setting yourself up for a fall. 3. Action Plans & Iterative Analysis: The Controlled Experiment "The Science of Deduction Meets A/B Testing" With mitigations in place and facts separated from hypotheses, now comes the plan to test each theory. Only one variable is changed at a time, as controlled as Sherlock's use of a magnifying glass. It's essential to tackle one hypothesis at a time to know exactly what fixes or fails. Keep the data untainted by opinions to make your conclusions unassailable. So, when your pager buzzes with the next crisis, think What Would Sherlock Holmes Do? Remember, prioritize mitigation before diving into any form of problem-solving. Keep a analytical mindset: collect data - form hypothesis - run controlled experiment and iterate!Activate to view larger image,Activate to view

Image preview