Implement Centralized Logging for Tanzu Kubernetes Grid With Fluent Bit
Monitoring is one of the most important aspects of a production deployment. Capturing event logs from the infrastructure pieces is very critical and logs are essential to determine the causes of any application or infrastructure failures . Day-2 operations become easy if you have comprehensive logging and alerting mechanism in place as it allows for a quick response to failures in infrastructure. With the increasing footprint of containerized workloads in data centers, application monitoring is equally important along with infrastructure monitoring.
Importance of Centralized Logging
The application developers are more focused on developing and deploying containerized applications and may not usually be well versed with infrastructure pieces. Applications logs are usually stored within the application. In the event of an application failure, if centralized logging is not in place and since infrastructure logs are stored in a different location, the developer may find errors in the application logs but is unable to determine the actual cause of failure.
This document explains the log forwarding procedure from TKG to vRealize Log Insight Cloud.
Log Forwarding for Tanzu Kubernetes Clusters
Log processing and forwarding in Tanzu Kubernetes Grid is provided via Fluent Bit which is available under TKG extensions and enables you to gather logs from TKG Management and Workload clusters and forward the logs to the supported destinations including the following:
- Elastic Search
- HTTP Endpoint (for e.g vRealize Log Insight)
Follow the instruction below to configure and forward the logs.
Before you can configure your TKG instance to send logs to vRealize Log Insight (vRLI) Cloud, you must create an API Key which will be used to authenticate against the vRLI Cloud instance.
To generate a new API key, log in to the vRLI Cloud instance through the VMware Cloud Console portal and navigate to Configuration > API Keys page, and click on the New API Key.
Provide a name for the API key and click Create.
Once the API key is generated, a URL and API key is displayed on the screen. Make a note of both the items as you will need them when configuring Fluent-Bit in later steps.
Before installing the Fluent Bit extensions, ensure that you have met the following prerequisites on the TKG bootstrapper machine
- TKG Workload cluster is deployed.
- cert manager is installed on workload cluster
Install the Carvel tools
- Download the latest TKG Extensions from .
- Upload the TKG Extension tar file on the machine from where you are managing your TKG clusters.
- Extract the extension file using tar or a similar extraction tool - # tar -xzf tkg-extensions-manifests-v1.3.1-vmware.1.tar
- Install Cert Manager on Workload cluster. Switch to the TKG workload cluster context and run the below commands to install and validate the cert manager extension.
# cd tkg-extensions-v1.3.1 # kubectl apply -f cert-manager/ # kubectl get pods -n cert-manager NAME READY STATUS RESTARTS AGE cert-manager-7c58cb795-jw7mk 1/1 Running 0 2m38s cert-manager-cainjector-765684c9d6-qgcw9 1/1 Running 0 2m38s cert-manager-webhook-ccc946479-gnbvh 1/1 Running 0 2m37s
3.1: Create fluent-bit namespace - This command will create a Fluent Bit namespace, a service account, and necessary role bindings.
# cd ~/tkg-extensions-v1.3.1/extensions/logging/fluent-bit/ # kubectl apply -f namespace-role.yaml namespace/tanzu-system-logging created serviceaccount/fluent-bit-extension-sa created role.rbac.authorization.k8s.io/fluent-bit-extension-role created rolebinding.rbac.authorization.k8s.io/fluent-bit-extension-rolebinding created clusterrole.rbac.authorization.k8s.io/fluent-bit-extension-cluster-role created clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-extension-cluster-rolebinding created
3.2: Prepare the yaml file for Fluent Bit deployment.
Copy ‘<LOG_BACKEND>/fluent-bit-data-values.yaml.example’ to ‘<LOG_BACKEND>/fluent-bit-data-values.yaml’
Note: vRLI is configured as HTTP endpoint in the Fluent Bit configuration, so the corresponding command is shown below.
# cp http/fluent-bit-data-values.yaml.example http/fluent-bit-data-values.yaml
3.3: Configure fluent-bit data values.
http/fluent-bit-data-values.yaml file as shown below:
Name of the management cluster
workload cluster where you are installing Fluent Bit extension.
logging: image: repository: projects.registry.vmware.com/tkg tkg: instance_name: "tkg13-mgmt" cluster_name: "tkg13-wld01" fluent_bit: output_plugin: "http" http: host: "data.mgmt.cloud.vmware.com" port: "443" uri: "/le-mans/v1/streams/ingestion-pipeline-stream" format: "json" header_key_value: "Authorization Bearer XXXXXXXXXX"
3.4: Create a secret for your log backend
# kubectl create secret generic fluent-bit-data-values --from-file=values.yaml=http/fluent-bit-data-values.yaml -n tanzu-system-logging
3.5: Deploy fluent-bit extension
# kubectl apply -f fluent-bit-extension.yaml
3.6: Retrieve the status of fluent-bit extension
# kubectl get app fluent-bit -n tanzu-system-logging
Fluent Bit app status should change to ‘Reconcile Succeeded’ once fluent-bit is deployed successfully.
# kubectl get app fluent-bit -n tanzu-system-logging NAME DESCRIPTION SINCE-DEPLOY AGE fluent-bit Reconcile succeeded 3m10s 4m8s
Log in to the vRLI Cloud instance and execute the queries to search for the Tanzu Kubernetes Cluster where you have installed the Fluent Bit extension.
You can run queries against the received logs and save the search queries and build intelligent dashboards on top of that.