Log Analysis & Correlation

Use processing pipelines to parse and enrich raw log data

Datadog Log Management offers simple yet powerful tools for teams to transform disparate, unstructured streams of raw log data into centralized, structured datasets. For instance, Datadog will automatically parse logs sent in JSON format. Datadog can automatically parse logs in other formats as well. For example, logs coming from any of the integrations in the integrations pipeline library will be automatically parsed and enriched. Teams can also define custom pipelines using patterns-based processing recommendations to implement complex data transformation strategies.

In addition to structuring logs, Datadog standardizes key attributes like URL and IP address to unify data across log sources, and supports the enrichment of logs with custom reference data. And, because pipelines are run on Datadog’s SaaS platform, organizations never have to worry about manual scaling to support processing complexity or growing log volumes.

Search and analyze logs at any scale

With Datadog’s user-friendly UI, team members of any technical experience can immediately start slicing and dicing large volumes of log data and performing complex investigations without having to learn a complex query language. Once the relevant logs are found, Datadog makes it easy to combine and compare the results of multiple queries and uncover trends across high cardinality data. For example, teams can quickly calculate the proportion of service logs with errors or compute statistics like average cart size per user. The results of any query can be displayed in top lists, timeseries graphs, geomap graphs, and other rich visualizations.

Accelerate investigations by grouping logs into Patterns and Transactions

Datadog offers a variety of aggregated, intelligent entry points to guide investigations. For example, the Log Patterns view intelligently clusters logs based on shared format to cut through noisy patterns and uncover outliers quickly.

To view individual log events in the context of a larger user journey or business process, engineers can use Log Transaction Queries, which aggregate logs from multiple services based on a common attribute and order them in sequence. Datadog will automatically calculate critical transactional insights like duration and max severity level to help users isolate significant flows. For example, an e-commerce application can group all logs related to the cart checkout process, across services and systems, to quickly identify bottlenecks and reasons for cart abandonment. Once a Log Transaction Query view is created, it can be saved for quick future reference.

Augmented troubleshooting with Watchdog Insights

When investigating an incident, an individual user may lack the system or application context to know which parts of an organization's systems and applications are responsible. Datadog’s Watchdog Insights is a recommendation engine that automatically detects if a specific host, service, or other log attribute is showing a disproportionate number of errors. This guides on-call engineers and incident commanders even when an investigation involves systems they’re unfamiliar with, and helps them bring the right people into the conversation. Watchdog Insights fits seamlessly into existing Log Explorer workflows, directing engineers to unusual errors within the scope of their current query.