Principal Software Engineer – Site Reliability

Job Description:

• Lead the definition, advocacy, and adoption of SRE principles across engineering teams
• Partner with leadership to shape long-term reliability, resiliency, and observability strategies
• Champion distributed tracing, real user monitoring (RUM), and key performance metrics such as Largest Contentful Paint (LCP) to improve system visibility and user experience
• Build and scale self-healing systems to minimize manual intervention and reduce downtime
• Drive enterprise-wide improvements to incident response processes, including those related to Machine Learning systems
• Collaborate closely with Development Productivity and Quality teams to improve engineering velocity without sacrificing reliability
• Influence technical and operational roadmaps through data-driven insights and hands-on technical contributions
• Own and deliver cross-functional initiatives from concept through execution, applying program management skills to align stakeholders and achieve results

Requirements:

• 10+ years combined experience across Software Engineering and Site Reliability Engineering, with a balanced background in both disciplines
• Proven track record as an SRE thought leader and evangelist, driving adoption of reliability best practices across organizations
• Strong communication and mentoring skills to influence engineers across disciplines
• Proficiency in Python, Go, and JavaScript/TypeScript
• Proficiency with Infrastructure as Code (Terraform, CDK, CloudFormation, etc.)
• Experience building internal tooling from scratch in agile development environments
• Expertise with observability, distributed tracing, RUM, LCP, and performance monitoring tools (e.g., Datadog, Prometheus)
• Experience with on-call and incident management, including large-scale or ML-related incidents
• Strong background in automation and building self-healing systems
• Hands-on experience with LLM/GenAI to improve SRE efficiency and processes
• Program management skills, including the ability to propose innovative solutions, influence leadership, improve processes, and drive cross-functional projects to completion

Benefits:

• Competitive compensation, including base pay, bonus opportunities, and annual equity grants that vest quarterly
• Generous 401(k) plan with Upstart matching $2 for every $1 contributed, up to $15,000 per year
• Employee Stock Purchase Plan (ESPP) with discounted stock purchase options for eligible employees
• Affordable medical, dental, and vision coverage, with multiple plan options - Upstart covers 90% to 100% of the cost depending on the plans you choose
• Health Savings Account contributions from Upstart for eligible plans
• Income protection benefits, including company-paid Basic Life, AD&D, and Short- and Long-Term Disability coverage, with options to purchase supplemental coverage
• Paid time off, sick and safe time, and company holidays
• Paid family and parental leave to support caregiving and major life moments
• Family-centered benefits through Carrot and Cleo, supporting fertility, parenthood, and caregiving
• Employee Assistance Program (EAP) offering mental health support and life-centered resources
• Financial wellness resources, including access to financial planning tools and a financial concierge service
• Annual wellness allowance to support your physical and emotional well-being and personal development, based on what matters most to you
• Annual productivity allowance to invest in relevant tools and resources you need to do your best work, no matter where you work from
• Connection and community through team events and onsites, all-company updates, and employee resource groups (ERGs)
• Onsite perks, including catered lunches and fully stocked micro-kitchens when working from one of our four offices, located in the Bay Area, Austin, Columbus, and New York City (opening Summer 2026!).

Apply tot his job Apply To this Job