Skyline + Anomify

Skyline is a robust, real-time anomaly detection system with a rich history, now made accessible to a wider audience through Anomify, while proudly remaining open-source.

Our journey began in 2015 when we took the reins of the pioneering open-source Skyline project. Today, nearly a decade later we’re proud to introduce Anomify.ai, a culmination of years of dedicated development, refining and perfecting anomaly detection algorithms and data pipelines.

From Open Source Roots to Enterprise

Skyline’s journey began at Etsy. It was engineered to passively monitor hundreds of thousands of metrics without the demanding task of configuring individual models or thresholds for each one, a significant advancement over traditional monitoring systems like Nagios.

Its core design catered to environments with a vast number of high-resolution time series requiring constant attention. Once a metric stream (from sources like statsd or Graphite) was established, Skyline would automatically incorporate new metrics for analysis, anomaly detection, and alerting, with results briefly available in its web interface. However, active maintenance by its original creators ceased around 2014. This is where one of our team - Gary Wilson stepped in (aka Earthgecko).

The original approach highlighted a challenge: a “one-size-fits-all” approach to anomaly detection proved less universally useful than initially hoped. Tuning out the noise from such a vast number of metrics without individual configuration remained a hurdle.

Why Anomify Championed and Evolved Skyline

Despite these initial challenges, the fundamental architecture and processing pipeline of Skyline were recognized as exceptionally solid, fast, and battle-tested. Capable of handling tens of thousands of time series in near real-time, Skyline stood out for its speed, especially within the Python data analysis landscape. It can process and analyze a massive volume of constantly changing time series, every minute of every day, across multiple resolutions, all on relatively modest commodity hardware.

Recognizing this powerful foundation, development continued with the aim of overcoming its limitations and extending its capabilities. The core of the original Skyline with its simple and elegant methodology has been largely preserved, testament to its excellent conceptual design and implementation. Enhancements have focused on performance, logic optimizations, and a restructured package.

Today’s Skyline is still a near real-time anomaly detection system but has evolved significantly. It now features a modular design with various applications like Horizon, Analyzer, Mirage, Boundary, Crucible, Panorama, Webapp, and Ionosphere. This modularity means users only need to enable the specific apps they require. Crucially, Skyline can now be fed, query, and analyze time series on an ad-hoc basis, extending its utility beyond just machine/app metric fed systems to include the analysis of static data.

The system employs a multi-stage analysis pipeline, utilizing a range of algorithms from simple 3-sigma checks to more complex methods like matrix profiling, spectral residual analysis, and tsfresh-based time series similarity comparisons for machine learning. This staged approach ensures computationally expensive analyses are only performed when necessary, optimizing for speed.

Making Complex Anomaly Detection Accessible

Skyline, in its open-source form, is a powerful but complex system to set up, implement, and upgrade, with many different parts and dependencies. It requires a significant investment of time and expertise to get running and maintain.

This is where Anomify steps in.

Anomify provides Skyline as a managed service. We handle the complexities of installation, configuration, maintenance, and upgrades, allowing you to leverage the full power of Skyline without the operational overhead. With Anomify, you get:

• Simplified Access: Harness advanced anomaly detection without the steep learning curve or resource commitment of a self-managed deployment.
• Expert Support: Benefit from the knowledge and experience of developers who have extensive experience with numerous Skyline integrations.
• Focus on Insights, Not Infrastructure: Concentrate on deriving value and insights from your data, while we manage the underlying system.

Committed to Open Source

Even as Anomify offers a version of Skyline as a managed service, we remain deeply committed to its open-source roots. The core Skyline project continues to be open-source and available, allowing the community to benefit from its capabilities, contribute to its development, and ensure its continued evolution as a transparent and powerful tool.

The effectiveness of Skyline’s approach has seen industry giants like Splunk openly reference utilizing Earthgecko Skyline methods to enhance their own anomaly detection capabilities. This adoption by a leading enterprise platform serves as powerful validation of the foundational principles we’ve continued to build upon.

Find out more by visitng the Skyline docs or talk to us today.