San Mateo, California, United States

Backend Software Developer

Posted on January 9th

A bit about Scalyr

Scalyr’s mission is to provide a different approach to unified observability and log management that is built for modern application development and deployment practices. Founded by Steve Newman, who is also the Writely (aka Google Docs) founder and lead engineer, and led by tech industry veteran and CEO Christine Heckart, Scalyr offers an integrated and extensible suite of monitoring, management, visualization and analysis tools that aggregate and search all the signals needed for real-time observability, including logs, metrics and traces. We are the only observability and log management provider that does not index data and scales horizontally, is blazing fast and is ultra-affordable. The opportunity in front of us is huge and we are still in the very early days. This is going to be one of those companies where people will look back and say “I wish I’d been there when…” well, this is your chance to be part of “when”..

What the Distributed Systems team does

Scalyr’s business revolves around ingesting a firehose of log data from our customers and making that data available for real-time search.  The Distributed Systems team is responsible for the custom distributed system and storage system powering it all. Most companies use databases, we built ours to gain a competitive advantage.  We are proud that many customers choose Scalyr because our search is orders of magnitude faster than our competitors.

And we have only gotten started.  We’ve built a lot, and we need you to help us go farther and faster.  Here’s some of the projects the team has worked on in the last few months:

“I was tasked with adding encryption at rest for all customer data, and at first I knew almost nothing about the database that I was supposed to work on.  It's amazing how much I've learned during this project. One of the best parts was performance tweaking, I'm still happy for what I've came up with using minimal memory copies/allocation.”


“We rebuilt Scalyr’s log storage engine on S3 to improve the efficiency and operability of our platform.  We solved many interesting performance and scalability challenges to enable storing and querying gigabytes of data per node in seconds.  This new architecture also enables us to converge towards more stateless machines. The end result is a more cost-effective and scalable solution.”

-Jim & Oliver

“I worked with our founder, Steve Newman, to design a novel scatter-gather algorithm for query distribution that dynamically lets fast nodes compensate for slow nodes without any preconceived server weighting.  We call it QueryMaster, and internally the work assignment logic is known as 'boustrophedon races'. At some point there will be an awesome blog post explaining just what that is supposed to mean…”


Your skills

  • You build high-scale distributed systems.
  • You love to dogfood the product you work on.  We are heavy users of our own product; it's fun!
  • You want to work on high-volume analytics, but not with an end goal of serving ads.
  • Can thread the needle between “our business logic dictates X” and “external clients will want Y” and come up with a minimal API that supports both
  • You are looking for a challenge.

Bonus points

  • You have delivered elegant distributed solutions to complex problems, at scale
  • You can identify relevant algorithms in the literature, and bring them into production when they fit.
  • You write functional Java, and prefer composition to inheritance

Here are some additional blog posts with more detail about what we’ve been up to:

Built for Speed: Custom Parser for Regex at Scale

Searching 1TB/sec: Systems Engineering Before Algorithms

Impossible Engineering Problems Often Aren’t

Tonight We Monitor, For Tomorrow, We Test in Production!

We believe in showcasing transparency in our process. Check out the below blog posts that dive into the greater detail of your interview:

Scalyr Distributed Systems Engineer Hiring Overview

Being Transparent in Our Hiring Process

Your benefits:

  • Competitive salary & equity at a fast growing startup
  • Fully funded comprehensive medical, dental, and vision coverage
  • Short-term disability, long term disability, and life insurance
  • 401k
  • Free lunch
  • Flexible schedule
  • Unlimited vacations
  • Paid parental leave
  • Easygoing, low-stress, considerate and fun office environment
  • Whatever equipment you need to get the job done

Our commitment to diversity

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.