Cribl Stream with Cloudian HyperStore S3 Data Lake

Data Observability to Ingest and Analyze Massive Amounts of Data at Cloud Economics

Challenges

In today’s IT world, monitoring only helps enterprises answer well-known questions about their infrastructure and environment. In order to get new signals or opportunities, in addition to a monitoring platform, enterprises also need an observability platform that can ingest data into multiple tools from multiple sources without adding new infrastructure and agents. With this, IT teams get full visibility and insight into their environment, from applications to infrastructure assets.

Solution

Cribl Stream is an observability pipeline that collects data from any source and can send and replay data to Cloudian HyperStore, a scale-out S3 data lake designed to manage massive amounts and varieties of data, forming a modern observability platform. HyperStore can scale up to thousands of nodes across multiple data centers, supporting millions of users and hundreds of petabytes of data. Together, the solution built on Cribl and Cloudian lets you parse, restructure, and enrich data in flight – ensuring that you get the right data, where you want, and in the formats you need. Customers can convert their logs into metrics, reduce cost, and increase search speed.

Scalable for Long-Term Compliance

Cribl Stream allows you to implement an observability pipeline that can collect data from any source and send it to Cloudian’s HyperStore data lake. The data is stored in HyperStore with full fidelity and is always available to search and analyze. It can be reformatted via a pre-parser, helping you mask, reduce, or restructure and route to multiple destinations, like low-cost storage for long-term retention. Stream helps you enrich data in flight – ensuring that you get the right data, where you want, and in the formats you need, behind the security of your firewall. Cloudian drives down the cost of on-prem, disk-based storage to ½¢/GB/month or less and makes it economical to retain data long-term with full fidelity for compliance.

Replay Data

Cribl’s observability solution allows customers to replay multiple data formats stored in a Cloudian HyperStore data lake to popular analytics and search platforms. Stream can ingest from any machine data source, store in Cloudian HyperStore with all fidelity and schedule batch collection from multiple APIs, or recall data from Cloudian data tiers and “replay” those logs to analytical tools for later investigations with ad hoc data collection.

Hybrid Cloud Ready

The observability platform built with Cribl and Cloudian is hybrid cloud ready. Customers can choose to start with an on-prem HyperStore deployment and employ policy-based tools to replicate or tier data to AWS, GCP, Azure, or another Cloudian HyperStore cluster for offsite DR, capacity expansion, or data analysis in the cloud. This allows organizations to costeffectively combine storage across environments into a single pool, while consolidating storage management enterprise-wide to a single screen. This flexibility is ideal for cloudbursting or disaster recovery (DR) in public cloud(s).

Secure

The solution provides government-certified, military-grade security capabilities that make it possible to deploy and operate cost-effective data observability and keep customers’ data extremely secure. Features include data encryption and transparent key management, AES-256 server-side encryption for data stored at rest, SSL for data in transit (HTTPS), rolebased access controls (RBAC) with specified levels of access, audit trail logging, and WORM (Write Once Read Multiple) for storage of immutable data.

Data Resiliency

An observability data platform built with Cribl and Hyperstore provides 14 nine’s of resiliency. HyperStore supports storage policies (administrator selectable) for implementing data resiliency based on Replication (RF) or Erasure Coding (EC). Administrators can configure the number of replicas or type of erasure code scheme required to meet SLA and cost objectives. Storage policies also provide fine grain control of data placement across data centers, taking into consideration factors such as cost efficiency, security levels, and proximity.

provider:    youtube
url:         https://youtu.be/v9lPB_9guWY

src:         https://www.youtube-nocookie.com/embed/v9lPB_9guWY?wmode=transparent&rel=0&feature=oembed
src mod:     https://www.youtube-nocookie.com/embed/v9lPB_9guWY?wmode=transparent&rel=0
src gen:     https://www.youtube-nocookie.com/embed/v9lPB_9guWY

Keep Watching

SOLUTION BENEFITS

• Modern Cloudian S3 data lake based observability platform to ingest and store data with full fidelity
• Replay right data through expansive analytics and search tools and resources
• Scale-out modular design with centralized data management at cloud economics
• Native S3-APIs with industry-leading compatibility
• Military-grade security and regulatory compliance certifications
• Hybrid and multi-cloud ready

Get Solution Brief