Implement Centralized Logging for Tanzu Kubernetes Grid With Fluent Bit
Introduction
Monitoring is one of the most important aspects of a production deployment. Capturing event logs from the infrastructure pieces is very critical and logs are essential to determine the causes of any application or infrastructure failures . Day-2 operations become easy if you have comprehensive logging and alerting mechanism in place as it allows for a quick response to failures in infrastructure. With the increasing footprint of containerized workloads in data centers, application monitoring is equally important along with infrastructure monitoring.
Importance of Centralized Logging
The application developers are more focused on developing and deploying containerized applications and may not usually be well versed with infrastructure pieces. Applications logs are usually stored within the application. In the event of an application failure, if centralized logging is not in place and since infrastructure logs are stored in a different location, the developer may find errors in the application logs but is unable to determine the actual cause of failure.
Scope
This document explains the log forwarding procedure from TKG to vRealize Log Insight Cloud.
Log Forwarding for Tanzu Kubernetes Clusters
Log processing and forwarding in Tanzu Kubernetes Grid is provided via Fluent Bit which is available under TKG extensions and enables you to gather logs from TKG Management and Workload clusters and forward the logs to the supported destinations including the following:
- Elastic Search
- Kafka
- Splunk
- HTTP Endpoint (for e.g vRealize Log Insight)
- Syslog
Follow the instruction below to configure and forward the logs.
Step 1: Configure vRealize Log Insight Clouds
Before you can configure your TKG instance to send logs to vRealize Log Insight (vRLI) Cloud, you must create an API Key which will be used to authenticate against the vRLI Cloud instance.
To generate a new API key, log in to the vRLI Cloud instance through the VMware Cloud Console portal and navigate to Configuration > API Keys page, and click on the New API Key.
Provide a name for the API key and click Create.
Once the API key is generated, a URL and API key is displayed on the screen. Make a note of both the items as you will need them when configuring Fluent-Bit in later steps.
Step 2: Install Carvel Tools
Before installing the Fluent Bit extensions, ensure that you have met the following prerequisites on the TKG bootstrapper machine
- TKG Workload cluster is deployed.
yttis installedkappis installed- cert manager is installed on workload cluster
Install the Carvel tools
- Download the latest TKG Extensions from My VMware.
- Upload the TKG Extension tar file on the machine from where you are managing your TKG clusters.
- Extract the extension file using tar or a similar extraction tool - # tar -xzf tkg-extensions-manifests-v1.3.1-vmware.1.tar
- Install Cert Manager on Workload cluster. Switch to the TKG workload cluster context and run the below commands to install and validate the cert manager extension.
# cd tkg-extensions-v1.3.1
# kubectl apply -f cert-manager/
# kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-7c58cb795-jw7mk 1/1 Running 0 2m38s
cert-manager-cainjector-765684c9d6-qgcw9 1/1 Running 0 2m38s
cert-manager-webhook-ccc946479-gnbvh 1/1 Running 0 2m37s
Step 3: Deploy Fluent Bit Extension on Workload cluster
3.1: Create fluent-bit namespace - This command will create a Fluent Bit namespace, a service account, and necessary role bindings.
# cd ~/tkg-extensions-v1.3.1/extensions/logging/fluent-bit/
# kubectl apply -f namespace-role.yaml
namespace/tanzu-system-logging created
serviceaccount/fluent-bit-extension-sa created
role.rbac.authorization.k8s.io/fluent-bit-extension-role created
rolebinding.rbac.authorization.k8s.io/fluent-bit-extension-rolebinding created
clusterrole.rbac.authorization.k8s.io/fluent-bit-extension-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-extension-cluster-rolebinding created
3.2: Prepare the yaml file for Fluent Bit deployment.
Copy ‘<LOG_BACKEND>/fluent-bit-data-values.yaml.example’ to ‘<LOG_BACKEND>/fluent-bit-data-values.yaml’
Note: vRLI is configured as HTTP endpoint in the Fluent Bit configuration, so the corresponding command is shown below.
# cp http/fluent-bit-data-values.yaml.example http/fluent-bit-data-values.yaml
3.3: Configure fluent-bit data values.
Modify the http/fluent-bit-data-values.yaml file as shown below:
|
Instance Name |
Name of the management cluster |
|
Cluster Name |
workload cluster where you are installing Fluent Bit extension. |
|
Authorization Key |
API key that you have generated from the vRLI Cloud instance. Point the host entry to 'data.mgmt.cloud.vmware.com'. |
logging:
image:
repository: projects.registry.vmware.com/tkg
tkg:
instance_name: "tkg13-mgmt"
cluster_name: "tkg13-wld01"
fluent_bit:
output_plugin: "http"
http:
host: "data.mgmt.cloud.vmware.com"
port: "443"
uri: "/le-mans/v1/streams/ingestion-pipeline-stream"
format: "json"
header_key_value: "Authorization Bearer XXXXXXXXXX"
3.4: Create a secret for your log backend
# kubectl create secret generic fluent-bit-data-values --from-file=values.yaml=http/fluent-bit-data-values.yaml -n tanzu-system-logging
3.5: Deploy fluent-bit extension
# kubectl apply -f fluent-bit-extension.yaml
3.6: Retrieve the status of fluent-bit extension
# kubectl get app fluent-bit -n tanzu-system-logging
Fluent Bit app status should change to ‘Reconcile Succeeded’ once fluent-bit is deployed successfully.
# kubectl get app fluent-bit -n tanzu-system-logging
NAME DESCRIPTION SINCE-DEPLOY AGE
fluent-bit Reconcile succeeded 3m10s 4m8s
Step 4: Verify TKG log forwarding
Log in to the vRLI Cloud instance and execute the queries to search for the Tanzu Kubernetes Cluster where you have installed the Fluent Bit extension.
You can run queries against the received logs and save the search queries and build intelligent dashboards on top of that.
Associated Content
From the action bar MORE button.