Kubernetes with Fluent Bit to send logs to Loggly

I’m going to show you how easy is to deploy Fluent Bit into your Kubernetes cluster. I’ll configure Fluent Bit to work together with Loggly, an external logging tool to manage all your cluster logs.

But first, some quick concepts about the tools we’re going to use.

Kubernetes, in short, is a tool that allows you to manage, in a better organized and simpler way, your containerized applications.

Fluent Bit, is a log processor and forwarder. It will allow you to collect logs from different sources, process them and send them to different locations.

Loggly, is a cloud based too, to unify all your logs in one place and analyse them. You could also use it as monitoring tool, sending you alerts when some events occur.

Now that you know briefly the tools we’re going to use, let’s start with the configuration.

 

Why Fluent Bit instead of Fluentd ?

In the past I experienced big performance problems when working with Fluentd, specially if you have a lot of traffic in your cluster generating a lot of logs.

After some research I found that the same company started a new project Fluent Bit, written in C, with a pluggable architecture, much more efficient, providing high performance. It’s good to mention that all I/O operations are made in asynchronous mode (event-driven).

Fluent Bit Service Account

First thing we’re going to do is create a Service Account in our Kubernetes cluster for our new Fluent Bit service.

A Service Account provides identity for the process running in the pods, which means that those pods will connect to and authenticate with the apiserver using the parameters defined in that Service Account instead of using the defaults (admin).

Create a new file called: fluent-bit-service-account.yaml, with this content.


apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: monitoring

As you see, we defined the name of this new Service Account and in which namespace is going to run. All the yaml files we’re going to create must use the same namespace as well.

You can run this command to create your new Service Account in your cluster.


$ kubectl create -f fluent-bit-service-account.yaml

To check that the new Service Account has been created you can use these commands.


$ kubectl --namespace=monitoring get sa
NAME         SECRETS   AGE
default      1         2h
fluent-bit   1         1h

$ kubectl --namespace=monitoring describe sa fluent-bit
Name:         fluent-bit
Namespace:    monitoring
Labels:
Annotations:  

Loggly account settings

Inside our Kubernetes cluster, Fluent Bit will collect all the logs from our containers, parse them and then send them to Loggly using the HTTP Output plugin.

In order to post your logs from Fluent Bit to Loggly we need to specify some Loggly account configuration values. You can find all the Loggly details about configuration here. Basically we’ll need a token and a tag.

We’ll create a Kubernetes Secret to hold those values and keep them safe without being hardcoded inside our yaml files.

A Secret is an object living in your cluster that contains small amount of sensitive data, such as passwords, tokens, etc.

Create a new file called: fluent-bit-secrets-loggly.yaml, with this content.

Pleas replace, the content of loggly_tag and loggly_token with your values.


apiVersion: v1
kind: Secret
metadata:
  name: fluent-bit-secrets
type: Opaque
data:
  loggly_tag: c29tZXRoaW5nIHJlYWxseSBjb29s
  loggly_token: YTU5ZmU0ZmMtMzNlYS00ODEwLTk5YTktNmU2NWE0OGEzOWRh

Create the new Secret in your Kubernetes cluster by running this command


$ kubectl create -f fluent-bit-secrets-loggly.yaml

To check that the new Secret has been created you can use these commands.


$ kubectl --namespace=monitoring get secrets
NAME                     TYPE                                  DATA      AGE
default-token-njcjx      kubernetes.io/service-account-token   3         2h
fluent-bit-secrets       Opaque                                2         1h
fluent-bit-token-cnwxj   kubernetes.io/service-account-token   3         1h

$ kubectl --namespace=monitoring describe secret fluent-bit-secrets
Name:         fluent-bit-secrets
Namespace:    monitoring
Labels:
Annotations:  

Type:  Opaque

Data
====
loggly_tag:    7 bytes
loggly_token:  36 bytes

Fluent Bit Configuration using Config Map

Now it’s time to configure our Fluent Bit. I strongly recommend you to read the Fluent Bit documentation it’s quite good and explain all the core concepts about Fluent Bit in a simple and clear way.

We’re going to create a Kubernetes Config Map that will hold all the needed configuration for Fluent Bit.

Please, create a file called: fluent-bit-configmap-loggly.yaml and add this content into it.


apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: monitoring
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   Off
    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-loggly.conf
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           On
        K8S-Logging.Parser  On
  output-loggly.conf: |
    [OUTPUT]
        Name             http
        Match            *
        Host             ${LOGGLY_HOSTNAME}
        Port             443
        Tls              On
        URI              /bulk/${LOGGLY_TOKEN}/tag/${LOGGLY_TAG}/
        Format           json_lines
        Json_Date_Key    timestamp
        Json_Date_Format iso8601
        Retry_Limit      False
  parsers.conf: |
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
        # Command      |  Decoder | Field | Optional Action
        # =============|==================|=================
        Decode_Field_As   escaped    log

As you can see we’re defining all the config files we’re going to use within Fluent Bit.

The file fluent-bit.conf will hold the basic configuration for the service and will refer to other files containing the input, parsers and outputs.

For our example, will get all the logs from Kubernetes containers using the tail input plugin. Then we’ll parse them using the docker parser. Finally we’ll send them to loggly using the HTTP output plugin. Just before parse them we’ll enrich our logs using the Kubernetes filter plugin. This plugin allows you to add more info into your logs like, container name, container ID, pod name, namespace, etc.

Create the new Config Map in your Kubernetes cluster by running this command


$ kubectl create -f fluent-bit-configmap-loggly.yaml

To check that the new Config Map has been created correctly you can use these commands.


$ kubectl --namespace=monitoring get cm
NAME                DATA      AGE
fluent-bit-config   5         1h

$ kubectl --namespace=monitoring describe cm fluent-bit-config
Name:         fluent-bit-config
Namespace:    monitoring
Labels:       k8s-app=fluent-bit
Annotations:  

Data
====
fluent-bit.conf:
----
[SERVICE]
    Flush         1
    Log_Level     info
    Daemon        off
    Parsers_File  parsers.conf
    HTTP_Server   Off
@INCLUDE input-kubernetes.conf
@INCLUDE filter-kubernetes.conf
@INCLUDE output-loggly.conf
...
... THE REST OF THE INFO IS OMITTED
...

Fluent Bit DaemontSet

Finally, we’re going to use a Kubernetes DaemonSet which will ensure that our new Fluent Bit pod is running in all the nodes. This is exactly what we want, all the nodes in our cluster will hold pods and we want to collect all the logs from all the nodes that have pods running in there.

Create a file called: fluent-bit-ds-loggly.yaml with this content.


apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: monitoring
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:0.13.7
        imagePullPolicy: IfNotPresent
        env:
        - name: LOGGLY_HOSTNAME
          value: "logs-01.loggly.com"
        - name: LOGGLY_TAG
          valueFrom:
            secretKeyRef:
              name: fluent-bit-secrets
              key: loggly_tag
        - name: LOGGLY_TOKEN
          valueFrom:
            secretKeyRef:
              name: fluent-bit-secrets
              key: loggly_token
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule

As you can see, in the container specification we’ve referenced the loggly_tag and the loggly_token from our Secret.

Create the new daemont set in your cluster by running this command:


$ kubectl create -f fluent-bit-ds-loggly.yaml

To check that the new DaemonSet has been created correctly you can use these commands.


$ kubectl --namespace=monitoring get ds
NAME         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluent-bit   1         1         1         1            1                     1h

Now you can go to your Loggly account and you’ll see your logs appearing there.

More info

You’ll find all the files for this example in this repository:

https://github.com/mbelchin/kubernetes-fluent-bit-loggly/

 

github

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: