I’m going to show you how easy is to deploy Fluent Bit into your Kubernetes cluster. I’ll configure Fluent Bit to work together with Loggly, an external logging tool to manage all your cluster logs.
But first, some quick concepts about the tools we’re going to use.
Kubernetes, in short, is a tool that allows you to manage, in a better organized and simpler way, your containerized applications.
Fluent Bit, is a log processor and forwarder. It will allow you to collect logs from different sources, process them and send them to different locations.
Loggly, is a cloud based too, to unify all your logs in one place and analyse them. You could also use it as monitoring tool, sending you alerts when some events occur.
Now that you know briefly the tools we’re going to use, let’s start with the configuration.
Why Fluent Bit instead of Fluentd ?
In the past I experienced big performance problems when working with Fluentd, specially if you have a lot of traffic in your cluster generating a lot of logs.
After some research I found that the same company started a new project Fluent Bit, written in C, with a pluggable architecture, much more efficient, providing high performance. It’s good to mention that all I/O operations are made in asynchronous mode (event-driven).
Fluent Bit Service Account
First thing we’re going to do is create a Service Account in our Kubernetes cluster for our new Fluent Bit service.
A Service Account provides identity for the process running in the pods, which means that those pods will connect to and authenticate with the apiserver using the parameters defined in that Service Account instead of using the defaults (admin).
Create a new file called: fluent-bit-service-account.yaml, with this content.
apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit namespace: monitoring
As you see, we defined the name of this new Service Account and in which namespace is going to run. All the yaml files we’re going to create must use the same namespace as well.
You can run this command to create your new Service Account in your cluster.
$ kubectl create -f fluent-bit-service-account.yaml
To check that the new Service Account has been created you can use these commands.
$ kubectl --namespace=monitoring get sa NAME SECRETS AGE default 1 2h fluent-bit 1 1h $ kubectl --namespace=monitoring describe sa fluent-bit Name: fluent-bit Namespace: monitoring Labels: Annotations:
Loggly account settings
Inside our Kubernetes cluster, Fluent Bit will collect all the logs from our containers, parse them and then send them to Loggly using the HTTP Output plugin.
In order to post your logs from Fluent Bit to Loggly we need to specify some Loggly account configuration values. You can find all the Loggly details about configuration here. Basically we’ll need a token and a tag.
We’ll create a Kubernetes Secret to hold those values and keep them safe without being hardcoded inside our yaml files.
A Secret is an object living in your cluster that contains small amount of sensitive data, such as passwords, tokens, etc.
Create a new file called: fluent-bit-secrets-loggly.yaml, with this content.
Pleas replace, the content of loggly_tag and loggly_token with your values.
apiVersion: v1 kind: Secret metadata: name: fluent-bit-secrets type: Opaque data: loggly_tag: c29tZXRoaW5nIHJlYWxseSBjb29s loggly_token: YTU5ZmU0ZmMtMzNlYS00ODEwLTk5YTktNmU2NWE0OGEzOWRh
Create the new Secret in your Kubernetes cluster by running this command
$ kubectl create -f fluent-bit-secrets-loggly.yaml
To check that the new Secret has been created you can use these commands.
$ kubectl --namespace=monitoring get secrets NAME TYPE DATA AGE default-token-njcjx kubernetes.io/service-account-token 3 2h fluent-bit-secrets Opaque 2 1h fluent-bit-token-cnwxj kubernetes.io/service-account-token 3 1h $ kubectl --namespace=monitoring describe secret fluent-bit-secrets Name: fluent-bit-secrets Namespace: monitoring Labels: Annotations: Type: Opaque Data ==== loggly_tag: 7 bytes loggly_token: 36 bytes
Fluent Bit Configuration using Config Map
Now it’s time to configure our Fluent Bit. I strongly recommend you to read the Fluent Bit documentation it’s quite good and explain all the core concepts about Fluent Bit in a simple and clear way.
We’re going to create a Kubernetes Config Map that will hold all the needed configuration for Fluent Bit.
Please, create a file called: fluent-bit-configmap-loggly.yaml and add this content into it.
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: monitoring
labels:
k8s-app: fluent-bit
data:
# Configuration files: server, input, filters and output
# ======================================================
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server Off
@INCLUDE input-kubernetes.conf
@INCLUDE filter-kubernetes.conf
@INCLUDE output-loggly.conf
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
filter-kubernetes.conf: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
Merge_Log On
K8S-Logging.Parser On
output-loggly.conf: |
[OUTPUT]
Name http
Match *
Host ${LOGGLY_HOSTNAME}
Port 443
Tls On
URI /bulk/${LOGGLY_TOKEN}/tag/${LOGGLY_TAG}/
Format json_lines
Json_Date_Key timestamp
Json_Date_Format iso8601
Retry_Limit False
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
# Command | Decoder | Field | Optional Action
# =============|==================|=================
Decode_Field_As escaped log
As you can see we’re defining all the config files we’re going to use within Fluent Bit.
The file fluent-bit.conf will hold the basic configuration for the service and will refer to other files containing the input, parsers and outputs.
For our example, will get all the logs from Kubernetes containers using the tail input plugin. Then we’ll parse them using the docker parser. Finally we’ll send them to loggly using the HTTP output plugin. Just before parse them we’ll enrich our logs using the Kubernetes filter plugin. This plugin allows you to add more info into your logs like, container name, container ID, pod name, namespace, etc.
Create the new Config Map in your Kubernetes cluster by running this command
$ kubectl create -f fluent-bit-configmap-loggly.yaml
To check that the new Config Map has been created correctly you can use these commands.
$ kubectl --namespace=monitoring get cm NAME DATA AGE fluent-bit-config 5 1h $ kubectl --namespace=monitoring describe cm fluent-bit-config Name: fluent-bit-config Namespace: monitoring Labels: k8s-app=fluent-bit Annotations: Data ==== fluent-bit.conf: ---- [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server Off @INCLUDE input-kubernetes.conf @INCLUDE filter-kubernetes.conf @INCLUDE output-loggly.conf ... ... THE REST OF THE INFO IS OMITTED ...
Fluent Bit DaemontSet
Finally, we’re going to use a Kubernetes DaemonSet which will ensure that our new Fluent Bit pod is running in all the nodes. This is exactly what we want, all the nodes in our cluster will hold pods and we want to collect all the logs from all the nodes that have pods running in there.
Create a file called: fluent-bit-ds-loggly.yaml with this content.
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: fluent-bit namespace: monitoring labels: k8s-app: fluent-bit-logging version: v1 kubernetes.io/cluster-service: "true" spec: template: metadata: labels: k8s-app: fluent-bit-logging version: v1 kubernetes.io/cluster-service: "true" spec: containers: - name: fluent-bit image: fluent/fluent-bit:0.13.7 imagePullPolicy: IfNotPresent env: - name: LOGGLY_HOSTNAME value: "logs-01.loggly.com" - name: LOGGLY_TAG valueFrom: secretKeyRef: name: fluent-bit-secrets key: loggly_tag - name: LOGGLY_TOKEN valueFrom: secretKeyRef: name: fluent-bit-secrets key: loggly_token volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: fluent-bit-config mountPath: /fluent-bit/etc/ terminationGracePeriodSeconds: 10 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: fluent-bit-config configMap: name: fluent-bit-config serviceAccountName: fluent-bit tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule
As you can see, in the container specification we’ve referenced the loggly_tag and the loggly_token from our Secret.
Create the new daemont set in your cluster by running this command:
$ kubectl create -f fluent-bit-ds-loggly.yaml
To check that the new DaemonSet has been created correctly you can use these commands.
$ kubectl --namespace=monitoring get ds NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE fluent-bit 1 1 1 1 1 1h
Now you can go to your Loggly account and you’ll see your logs appearing there.
More info
You’ll find all the files for this example in this repository:
https://github.com/mbelchin/kubernetes-fluent-bit-loggly/
MOISES BELCHIN 
Leave a comment