Implement Centralized Logging for Tanzu Kubernetes Grid With Fluent Bit

Introduction

Monitoring is one of the most important aspects of a production deployment. Capturing event logs from the infrastructure pieces is very critical and logs are essential to determine the causes of any application or infrastructure failures . Day-2 operations become easy if you have comprehensive logging and alerting mechanism in place as it allows for a quick response to failures in infrastructure. With the increasing footprint of containerized workloads in data centers, application monitoring is equally important along with infrastructure monitoring.

Importance of Centralized Logging

The application developers are more focused on developing and deploying containerized applications and may not usually be well versed with infrastructure pieces. Applications logs are usually stored within the application. In the event of an application failure, if centralized logging is not in place and since infrastructure logs are stored in a different location, the developer may find errors in the application logs but is unable to determine the actual cause of failure.

Scope

This document explains the log forwarding procedure from TKG to vRealize Log Insight Cloud.

Log Forwarding for Tanzu Kubernetes Clusters

Log processing and forwarding in Tanzu Kubernetes Grid is provided via Fluent Bit which is available under TKG extensions and enables you to gather logs from TKG Management and Workload clusters and forward the logs to the supported destinations including the following:

  • Elastic Search
  • Kafka
  • Splunk
  • HTTP Endpoint (for e.g vRealize Log Insight)
  • Syslog

Follow the instruction below to configure and forward the logs.

Step 1: Configure vRealize Log Insight Clouds

Before you can configure your TKG instance to send logs to vRealize Log Insight (vRLI) Cloud, you must create an API Key which will be used to authenticate against the vRLI Cloud instance.

To generate a new API key, log in to the vRLI Cloud instance through the VMware Cloud Console portal and navigate to Configuration > API Keys page, and click on the New API Key.

Provide a name for the API key and click Create.

Once the API key is generated, a URL and API key is displayed on the screen. Make a note of both the items as you will need them when configuring Fluent-Bit in later steps.

 

Step 2: Install Carvel Tools

Before installing the Fluent Bit extensions, ensure that you have met the following prerequisites on the TKG bootstrapper machine

  • TKG Workload cluster is deployed.
  • ytt is installed
  • kapp is installed
  • cert manager is installed on workload cluster

Install the Carvel tools

  1. Download the latest TKG Extensions from My VMware.
  2. Upload the TKG Extension tar file on the machine from where you are managing your TKG clusters.
  3. Extract the extension file using tar or a similar extraction tool - # tar -xzf tkg-extensions-manifests-v1.3.1-vmware.1.tar
  4. Install Cert Manager on Workload cluster. Switch to the TKG workload cluster context and run the below commands to install and validate the cert manager extension.
# cd tkg-extensions-v1.3.1
 
# kubectl apply -f cert-manager/
 
# kubectl get pods -n cert-manager
 
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7c58cb795-jw7mk               1/1     Running   0          2m38s
cert-manager-cainjector-765684c9d6-qgcw9   1/1     Running   0          2m38s
cert-manager-webhook-ccc946479-gnbvh       1/1     Running   0          2m37s

Step 3: Deploy Fluent Bit Extension on Workload cluster

3.1: Create fluent-bit namespace - This command will create a Fluent Bit namespace, a service account, and necessary role bindings.

# cd ~/tkg-extensions-v1.3.1/extensions/logging/fluent-bit/
 
# kubectl apply -f namespace-role.yaml
namespace/tanzu-system-logging created
serviceaccount/fluent-bit-extension-sa created
role.rbac.authorization.k8s.io/fluent-bit-extension-role created
rolebinding.rbac.authorization.k8s.io/fluent-bit-extension-rolebinding created
clusterrole.rbac.authorization.k8s.io/fluent-bit-extension-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-extension-cluster-rolebinding created

3.2: Prepare the yaml file for Fluent Bit deployment.

Copy ‘<LOG_BACKEND>/fluent-bit-data-values.yaml.example’ to ‘<LOG_BACKEND>/fluent-bit-data-values.yaml’

Note: vRLI is configured as HTTP endpoint in the Fluent Bit configuration, so the corresponding command is shown below.

# cp http/fluent-bit-data-values.yaml.example http/fluent-bit-data-values.yaml

3.3: Configure fluent-bit data values.

Modify the http/fluent-bit-data-values.yaml file as shown below:

Instance Name

Name of the management cluster

Cluster Name

workload cluster where you are installing Fluent Bit extension. 

Authorization Key

API key that you have generated from the vRLI Cloud instance. Point the host entry to 'data.mgmt.cloud.vmware.com.

 
logging:
  image:
    repository: projects.registry.vmware.com/tkg
tkg:
  instance_name: "tkg13-mgmt"
  cluster_name: "tkg13-wld01"
fluent_bit:
  output_plugin: "http"
  http:
    host: "data.mgmt.cloud.vmware.com"
    port: "443"
    uri: "/le-mans/v1/streams/ingestion-pipeline-stream"
    format: "json"
    header_key_value: "Authorization Bearer XXXXXXXXXX"
 

3.4: Create a secret for your log backend

# kubectl create secret generic fluent-bit-data-values --from-file=values.yaml=http/fluent-bit-data-values.yaml -n tanzu-system-logging

3.5: Deploy fluent-bit extension

# kubectl apply -f fluent-bit-extension.yaml

3.6: Retrieve the status of fluent-bit extension

# kubectl get app fluent-bit -n tanzu-system-logging

Fluent Bit app status should change to ‘Reconcile Succeeded’ once fluent-bit is deployed successfully.

# kubectl get app fluent-bit -n tanzu-system-logging
 
NAME         DESCRIPTION           SINCE-DEPLOY   AGE
 
fluent-bit   Reconcile succeeded   3m10s          4m8s

Step 4: Verify TKG log forwarding

Log in to the vRLI Cloud instance and execute the queries to search for the Tanzu Kubernetes Cluster where you have installed the Fluent Bit extension.

You can run queries against the received logs and save the search queries and build intelligent dashboards on top of that.

Author and Contributors

Manish Jha has authored this article.

 

 

 

 

Filter Tags

App Modernization General Kubernetes Tanzu VMware Cloud on AWS Document Technical Guide Intermediate Deploy Manage