dermetfan

Monitoring Go Apps with Distributed Tracing and OpenTelemetry

2022-04-28T00:00:00Z

The following is a guest article written by Vladimir Mihailenco from Uptrace.

Monitoring Go Apps with Distributed Tracing and OpenTelemetry

What is tracing?

Distributed tracing allows to precisely pinpoint the problem in complex systems, especially those built using a microservices architecture.

Tracing allow to follow requests as they travel through distributed systems. You get a full context of what is different, what is broken, and which logs & errors are relevant.

Distributed Tracing

Using tracing, you can break down requests into spans. A span is an operation (unit of work) your app performs handling a request, for example, a database query or a network call.

A trace is a tree of spans that shows the path that a request makes through an app. The root span is the first span in a trace.

Trace

To learn more about tracing, see Distributed tracing using OpenTelemetry.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework for distributed tracing (including logs and errors) and metrics.

Otel allows developers to collect and export telemetry data in a vendor agnostic way. With OpenTelemetry, you can instrument your application once and then add or change vendors without changing the instrumentation, for example, here is a list popular DataDog alternatives that support OpenTelemetry.

How to use OpenTelemetry?

OpenTelemetry is available for most programming languages and provides interoperability across different languages and environments.

You can get started with OpenTelemetry by following these 5 steps:

Step 1. Install a distributed tracing tool.
Step 2. Browse OpenTelemetry instrumentations to find examples for your framework and libraries.
Step 3. Configure the OpenTelemetry SDK to export telemetry data to the tracing tool.
Step 4. Learn about OpenTelemetry Tracing API for your programming language to create custom instrumentations.
Step 5. Install OpenTelemetry Collector to gather infrastructure metrics.

OpenTelemetry Go API

You can create spans using OpenTelemetry Go API like this:

import "go.opentelemetry.io/otel"

var tracer = otel.Tracer("app_or_package_name")

func someFunc(ctx context.Context) error {
	ctx, span := tracer.Start(ctx, "some-func")
	defer span.End()

	// the code you are measuring

	return nil
}

You can also record attributes and errors:

func someFunc(ctx context.Context) error {
	ctx, span := tracer.Start(ctx, "some-func")
	defer span.End()

	if span.IsRecording() {
		span.SetAttributes(
			attribute.Int64("enduser.id", userID),
			attribute.String("enduser.email", userEmail),
		)
	}

	if err := someOtherFunc(ctx); err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
	}

	return nil
}

What is Uptrace?

Uptrace is an open source DataDog alternative that helps developers pinpoint failures and find performance bottlenecks. Uptrace can process billions of spans on a single server and allows to monitor your software at 10x lower cost.

Uptrace accepts data from OpenTelemetry and stores it in a ClickHouse database. ClickHouse is a columnar database that is much more efficient for traces and logs than, for example, Elastic Search. On the same hardware, ClickHouse can store 10x more traces and, at the same time, provide much better performance.

You can install Uptrace by downloading a DEB/RPM package or a pre-compiled binary.

Uptrace tracing tool

What’s next?

Next, you can continue exploring OpenTelemetry or start instrumenting your app using popular instrumentations:

Trying Zig with C web servers

2020-03-26T00:00:00Z

Tags:
code
zig
nix
h2o
facil.io
lwan

I wanted to try using Zig to build an application server.

The C integration should, in theory, allow me to use an existing web server library. Here are my results.

I tried the following web servers:

H2O is usable with some workarounds.
facil.io works in principle but is unusable in practice.
LWAN was a complete failure.

TL;DR: C translation is obviously not production ready, but I was able to get it to work with H2O.

H2O

This is the only one I had success with.

I ported the simple.c example to Zig, following this intro to libh2o.

Here you can download the entire project.

A few problems came up which I want to briefly discuss below.

Zig cannot translate C bitfields

H2O makes use of C bitfields throughout its codebase. Unfortunately, Zig does not (yet) understand them. There was an effort to fix this but it seems to have stagnated:

issue #179 "translate-c: support C bitfields" (open)
PR #4165 "Add support for C bitfields" (closed)

Consequently we have to remove all bitfields from H2O. Fortunately using Nix it is trivial to apply a patch and use the recompiled H2O:

pkgs.h2o.overrideAttrs (oldAttrs: {
  patches = [ ./h2o-no-bitfields.patch ];
});

libuv

H2O has its own event loop implementation that is faster than libuv, so for a simple demo I did not want to link against libuv. So without libuv in the build environment obviously the header file cannot be found.

./src/c.zig:1:20: error: C import failed
pub usingnamespace @cImport({
                   ^
/nix/store/…-h2o-2.3.0-beta2-dev/include/h2o/socket/uv-binding.h:27:10: note: 'uv.h' file not found
#include

We can fix that be setting the H2O_USE_LIBUV preprocessor variable.

pub usingnamespace @cImport({
    @cDefine("H2O_USE_LIBUV", "0");
    @cInclude("h2o.h");
});

If you want to compile H2O without libuv support you need to pass that to H2O’s CMakeLists.txt. Using Nix that looks like this:

pkgs.h2o.overrideAttrs (oldAttrs: {
  cmakeFlags = [ "-DDISABLE_LIBUV=1" ];
});

`#define H2O_STRLIT`

H2O passes strings around with an additional length parameter using the preprocessor.

h2o_iovec_init(H2O_STRLIT("default");

That expands to:

h2o_iovec_init("default", 7)

That H2O_STRLIT macro cannot be used in Zig code, so we will have to type the length ourselves.

For common use cases you can write little wrapper functions to make it more comfortable:

pub fn iovec_init(data: var) h2o_iovec_t {
    return h2o_iovec_init(data, data.len);
}

Zig stdlib bug when linking libc

Setting up the listening socket is easier than in C. Zig’s standard library handles everything for us:

var server = std.net.StreamServer.init(.{});
defer server.deinit();
try server.listen(try std.net.Address.parseIp4("127.0.0.1", 12345));

Here I hit issue #4797 in the standard library, but that was quickly solved by a Zig contributor.

In case you are using a version before commit 778dbc17acce77588a46a27074ecb8b418786d94, there is a workaround.

Make a copy of the stdlib.

Replace the setsockopt() signature in std/os.zig:

- pub fn setsockopt(fd: fd_t, level: u32, optname: u32, opt: []const u8) SetSockOptError!void {
+ pub fn setsockopt(fd: fd_t, level: u32, optname: u32, opt: []u8) SetSockOptError!void {

Use a compiler binary that is at a path relative to your stdlib:

Zig locates lib files relative to executable path by searching up the filesystem tree for a sub-path of lib/zig/std/std.zig or lib/std/std.zig. Typically the former is an install and the latter a git working tree which contains the build directory.

— Contribution Guidelines

H2O_TOKEN_…

Similarly Zig can not parse the H2O_TOKEN_… macros. There is a h2o_lookup_token() function that we can use though.

To avoid looking up tokens all the time, we can assign all the tokens on startup of our program to our own constants.

// lookup_token() cannot run comptime so these have to be vars...
pub var TOKEN_CONTENT_LENGTH: *const h2o_token_t;
pub var TOKEN_CONTENT_LOCATION: *const h2o_token_t;
pub var TOKEN_CONTENT_RANGE: *const h2o_token_t;
pub var TOKEN_CONTENT_TYPE: *const h2o_token_t;
// …

pub fn _init() void {
    TOKEN_CONTENT_LENGTH = lookup_token("content-length");
    TOKEN_CONTENT_LOCATION = lookup_token("content-location");
    TOKEN_CONTENT_RANGE = lookup_token("content-range");
    TOKEN_CONTENT_TYPE = lookup_token("content-type");
    // …
}

pub fn lookup_token(name: []const u8) *const h2o_token_t {
    return h2o_lookup_token(&name[0], name.len);
}

Admittedly that is not very nice. Maybe we could build the H2O_TOKEN_… structs directly and assign them to actual consts.

Converting `h2o_iovec_t` to slices

We can simplify here with a little helper.

pub fn _from_iovec(data: h2o_iovec_t) []u8 {
    return data.base[0..data.len];
}

That allows us to use std.mem.eql() instead of h2o_ismem(), if desired.

if (!std.mem.eql(u8, c.h2o._from_iovec(req.*.method), c.h2o.METHOD_GET)) return -1;

Conclusion

Since this was the only web server I could get to work, there is still work to do in the C translation. But that is to be expected from such an early version of the language.

I find it remarkable that C translation works so well already. Very promising!

facil.io

I was able to start the server and answer a request.

Unfortunately Zig is unable to translate fiobj_free(). Therefore I did not think it was worth investigating further.

Download all files.

const std = @import("std");

const c = @cImport({
    @cInclude("fio.h");
    @cInclude("http.h");
});

pub fn main() !void {
    const HTTP_HEADER_X_DATA: c.FIOBJ = c.fiobj_str_new("X-Data", "X-Data".len);
    defer c.fiobj_free(HTTP_HEADER_X_DATA); // FIXME https://github.com/ziglang/zig/blob/8ea0a00f406bb04c08a8fa4471c3a3895f82b24a/src-self-hosted/translate_c.zig

    const socket_uuid = c.http_listen("3000", null, .{
        .on_request = on_request,
        .log = 1,
        .on_upgrade = null,
        .on_response = null,
        .on_finish = null,
        .udata = null,
        .public_folder = null,
        .public_folder_length = 0,
        .max_header_size = 0,
        .max_body_size = 0,
        .max_clients = 0,
        .tls = null,
        .reserved1 = 0,
        .reserved2 = 0,
        .reserved3 = 0,
        .ws_max_msg_size = 0,
        .timeout = 0,
        .ws_timeout = 0,
        .is_client = 0,
    });
    if (socket_uuid == -1) return error.Listen;

    c.fio_start(.{ .threads = 1, .workers = 1, });
}

fn on_request(request: [*c]c.http_s) callconv(.C) void {
    const body_ = "Hello World!\r\n";
    var body: [body_.len]u8 = undefined;
    std.mem.copy(u8, body[0..], body_[0..]);

    _ = c.http_send_body(request, body[0..], body.len);
}

./zig-cache/o/…/cimport.zig:2918:24: error: unable to translate function
pub const fiobj_free = @compileError("unable to translate function");
                       ^
./src/main.zig:8:12: note: referenced here
    defer c.fiobj_free(HTTP_HEADER_X_DATA);
           ^

LWAN

Those macros can clearly not easily be translated to Zig, so the journey ends abruptly.

#define LWAN_HANDLER_REF(name_) lwan_handler_##name_
#define LWAN_HANDLER(name_)                                                    \
    static enum lwan_http_status lwan_handler_##name_(                         \
        struct lwan_request *, struct lwan_response *, void *);                \
    static const struct lwan_handler_info                                      \
        __attribute__((used, section(LWAN_SECTION_NAME(lwan_handler))))        \
            lwan_handler_info_##name_ = {.name = #name_,                       \
                                         .handler = lwan_handler_##name_};     \
    static enum lwan_http_status lwan_handler_##name_(                         \
        struct lwan_request *request __attribute__((unused)),                  \
        struct lwan_response *response __attribute__((unused)),                \
        void *data __attribute__((unused)))

dermetfan

Monitoring Go Apps with Distributed Tracing and OpenTelemetry

Monitoring Go Apps with Distributed Tracing and OpenTelemetry

What is tracing?

What is OpenTelemetry?

How to use OpenTelemetry?

OpenTelemetry Go API

What is Uptrace?

What’s next?

Trying Zig with C web servers

H2O

Zig cannot translate C bitfields

libuv

#define H2O_STRLIT

Zig stdlib bug when linking libc

H2O_TOKEN_…

Converting h2o_iovec_t to slices

Conclusion

facil.io

LWAN

`#define H2O_STRLIT`

Converting `h2o_iovec_t` to slices