Application Performance Monitoring (APM): Achieving Full-Stack Visibility

In the contemporary landscape of software development, applications are no longer monolithic. They have evolved into complex distributed systems. Consider a simple user interaction like a button click. It may:

• Invoke an API hosted on a separate domain
• Query multiple databases to fetch data
• Pass through a myriad of CDNs, proxies, and caches

To gain a comprehensive understanding of performance across this intricate chain, we employ a concept known as Application Performance Monitoring (APM).

APM tools are powerful resources that assist developers and operations teams in the following ways:

• They enable tracing of user requests across multiple services
• They facilitate the measurement of backend latency and error rates
• They allow correlation of logs, metrics, and traces
• They aid in diagnosing system slowdowns and outages

In essence, APM brings backend observability to the frontend-aware engineer.

The Breadth of APM Monitoring

APM tools function by weaving together telemetry from various sources such as:

• HTTP request latency: Measures the time it takes for an HTTP request to be fulfilled
• Database queries and timing: Keeps track of the queries made and the time taken to execute each query
• Cache usage and misses: Monitors the efficiency of cache usage and logs instances where data is not found in the cache, resulting in a miss
• Error rates and stack traces: Records the frequency of errors and keeps track of the call stack at the point where an exception was thrown
• Resource usage: Monitors the consumption of resources like CPU and memory

Some sophisticated tools also incorporate:

• User session traces: Provides visibility into user behavior during a session
• Frontend to backend span tracking: Tracks requests from frontend to backend
• Distributed transaction mapping: Helps visualize and understand distributed transactions

Key Features of APM Tools

APM tools come packed with several essential features such as:

• Distributed Tracing: Provides visibility into requests as they traverse through various services
• Alerting and anomaly detection: Monitors metrics and sends alerts when anomalies are detected
• Performance dashboards: Presents a visual representation of application performance metrics
• Service maps and dependency graphs: Visualizes service dependencies to help understand system behavior
• Error tracking: Tracks and reports errors in the application
• Data retention and analytics: Stores and analyzes historical data to identify trends and patterns

Real-World APM Examples

To understand how these tools function in a real-world setting, let’s delve into a few examples.

Datadog APM

Datadog APM is a robust tool that offers:

• Distributed tracing with support for OpenTelemetry, a set of APIs, libraries, and instrumentation that standardizes the generation, collection, and description of telemetry data
• Flamegraphs that provide a visualization of service execution, helping identify bottlenecks
• Metrics and logs presented in the context of traces, providing comprehensive visibility

New Relic

New Relic offers:

• Full-stack tracing, spanning from the browser to the backend and the database
• Monitoring of transaction times and external service tracking
• Custom dashboards featuring Service Level Indicators (SLIs) and Service Level Objectives (SLOs)

Sentry Performance

Sentry Performance is a tool that:

• Focuses on frontend and API timing, providing insights into frontend performance and API response times
• Provides spans from user input to backend API response, offering end-to-end visibility
• Combines error tracking with performance monitoring to provide context

Tracing Across Frontend and Backend

With the help of OpenTelemetry, you can track a full user journey as follows:

1. A button click triggers an API request
2. A Trace ID is added to the request headers
3. The backend carries the trace through its services
4. The final response is correlated with the frontend span

Here's a practical example of how this works:

Frontend:

fetch("/api/data", {
  headers: {
    "traceparent": "00-abc...-1234...-01"
  }
});

The JavaScript fetch function makes a request to "/api/data". The headers include a "traceparent" header, which carries the Trace ID.

Backend:

func handler(w http.ResponseWriter, r *http.Request) {
  span := tracer.StartSpanFromRequest(r)
  defer span.End()
  ...
}

In this Go code snippet, a new span is started from the incoming HTTP request, and it's ensured that the span ends when the function returns, using the defer keyword. This span carries the Trace ID through the backend services.

Choosing the Right Tool

When it comes to selecting an APM tool, it's essential to consider the strengths of each:

| Tool | Strength | |-------------|----------------------------------| | Datadog | Enterprise scale, full tracing | | New Relic | Deep backend profiling | | Sentry | JavaScript and API tracing, with integrated error tracking | | Lightstep | Long-term trace retention | | Elastic APM | Integration with open-source stack|

APM Anti-Patterns

Avoiding the following anti-patterns can significantly improve your APM practices:

• Logging without correlation IDs: Without correlation IDs, it's difficult to trace logs back to their original requests
• Only tracing the frontend or backend, not both: This provides an incomplete picture of your application's behavior
• Ignoring service latency in dashboards: Service latency can have a significant impact on user experience
• Not setting up alerting on critical user flows: This can lead to slow responses to issues affecting important user activities

The Intersection of APM and Frontend Development

Frontend developers can greatly benefit from APM. It not only concerns servers but also helps answer critical questions such as:

• Why is the /api/products endpoint slow?
• Which database query is causing a bottleneck during checkout?
• Are failed logins related to authentication service latency?

APM tools help connect the dots between code and its consequence, providing a comprehensive view of the system.

Conclusion: Trace It or Chase It

You can either trace the user journey — gaining insights and understanding the behavior of your application — or chase bug reports around your communication channels.

APM tools provide you with:

• Context: Understand the circumstances surrounding system behavior
• Timelines: See when specific events occur
• Causality: Determine the cause-and-effect relationship between events

And ultimately, they provide a better understanding of the reality of your application.

Because excellent performance is not just about speed — it's about observability. The ability to observe the inner workings of your system is crucial to maintaining and improving performance.