Configure Kubernetes Logging from the Tanzu Mission Control Catalog

Configure Log Forwarding through Tanzu Mission Control Catalog Packages

Deploying Kubernetes resources usually involves creating or editing YAML manifests in a text editor before finally applying a desired state to the cluster. A YAML manifest saved in version control is a great way to manage configuration options in your Kubernetes clusters, but this might not be the preferred option for some organizations.

Tanzu Mission Control announced a new "Catalog" capability where platform operators can select an application from the catalog and deploy it to one of their Kubernetes clusters managed by Tanzu Mission Control. VMware provides a list of packages included with your TMC version to help you get your clusters up and running with little effort. Things like ingress routing with Contour, cert-manager for applying TLS certificates to your apps, Fluent-bit for logging, and Prometheus for monitoring, to name a few.

This post focuses on one of these packages (fluent-bit) and how you can use the catalog to deploy fluent-bit to your Kubernetes clusters and push logs to vRealize Log Insight Cloud.

Deploy the Fluent-bit Package

Log in to the Tanzu Mission Control portal and select the "Catalog" menu item to deploy a package from the catalog. The catalog screen will open and prompt you to deploy the package on a cluster. Then select the package you wish to install. For this post, we'll use the fluent-bit package.

Once you've selected the package and the cluster, you will be presented with a new screen that shows details about the package, including version and release notes. The most important part of this screen is the configuration values. These configuration values show what variables can be changed, or are required, to deploy your packages. This screen is purely informational, but you'll need to configure these values in the next step.

Tanzu Mission Control will now prompt you for deployment information in a wizard-based menu. Give the package a deployment name and select a package version. Click "Next."

On the next screen, you'll be prompted with some radio buttons. You should be able to leave these settings the default if you're just getting started, but you can change the package's namespace to a custom namespace if you like.

NOTE: this namespace is for the package object and NOT the actual package contents. For example, we might deploy this package to a "Packages" namespace. However, Fluent-bit itself might still be deployed into a logging namespace or wherever you decide to set that variable (seen in the next phase).

Once you've decided about your package resources, you'll need to set the configuration values for the package. These values were shown during a previous step and are shown again here with an option to modify them. It should be noted that these configuration values will change depending on the type of package used. In our case, we need to make a few changes to the default fluent-bit configuration to push logs to vRealize Insight Cloud using our API token.

In my case, I needed to change the configurations for the "fluent_bit" object. My full configuration is shown below, which I've pasted into the "Configured values" section.

fluent_bit:
  config:
    service: |
      [Service]
        Flush         3
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
    outputs: |
      [OUTPUT]
        Name          http
        Match         *
        tls           On
        tls.verify    Off
        Host          data.mgmt.cloud.vmware.com
        Port          443
        URI           /le-mans/v1/streams/ingestion-pipeline-stream
        Header        Authorization Bearer “Your-API-Token-Goes-Here”
        Format        json
    inputs: |
      [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Parser docker
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
      [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On
    filters: |
      [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
      [FILTER]
        Name                modify
        Match               *
        Rename message text
    parsers: |
      [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    streams: ""
    plugins: ""
  #! optional configuration for the daemonset
  daemonset:
    resources:
    #! limits:
    #!   cpu: 100m
    #!   memory: 128Mi
    #! requests:
    #!   cpu: 100m
    #!   memory: 128Mi
    podAnnotations: {}
    podLabels: {}

Once your configuration is in place, click the "Install Package" button to begin the deployment.

Within the Tanzu Mission Control portal, you can look at the installed packages under the "Add-ons" tab of the cluster view. You can see from the screenshot below that two packages were already successfully deployed, and our fluent-bit package is in the process of being deployed.

Once the reconciliation loops are complete, you should see a "Succeeded" status message or an error with some information about what failed during the deployment.

If your fluent-bit configuration succeeded, you should start seeing logs in your vRealize Log Insight instance within a couple of minutes. Your next steps would be to build fancy dashboards or alerts out of those new logs that vRealize Log Insight is receiving.

Summary and Additional Resources

The Tanzu Mission Control catalog feature is a nice way to provide platform operators a GUI based method of deploying software to Kubernetes clusters. Platform operators can use these catalog items to configure any Kubernetes clusters registered with TMC so they can quickly update their fleet of clusters. VMware Cloud on AWS customers are entitled to this catalog feature for free with their Tanzu Mission Control Essentials licensing that comes with VMware Cloud on AWS.

Additional Resources

Tanzu Mission Control Documentation

Changelog

The following updates were made to this guide.

Date	Description of Changes
2022-01-03	Initial publication

About the Author and Contributors

Eric Shanks has spent two decades working with VMware and cloud technologies focusing on hybrid cloud and automation. Eric has obtained some of the industry's highest distinctions, including two VMware Certified Design Expert (VCDX #195) certifications and many others across various solutions, including Microsoft, Cisco, and Amazon Web Services.

Eric's acted as a community contributor as a Chicago VMUG Users Group leader, blogger at theITHollow.com, and Tech Field Day delegate.

• Eric Shanks, Sr. Technical Marketing Architect, Cloud Services Business Unit, VMware