New Relic on Kubernetes

New Relic is a SaaS (Software as a Service) observability tool that supports the performance monitoring of applications (application performance monitoring), mobile users (real-time user monitoring), databases, clouds and on-prem infrastructure. It supports all four pillars of observability which are usually referred to by the observability industry as M.E.L.T – metrics, events, logs and traces.  

New Relic fulfils all four pillars of observability 

Metrics – metrics depict the status of a system by giving different numeric measurements pertaining to a system or application – uptime, downtime, number of pods and nodes, etc. These metrics help you in assessing the performance of the system or application. To provide you with metric data, New Relic has integrations with third-party services – OpenTelemetry, Prometheus and DropWizard.  

Events – events are any user activity or change made to a system or application. In New Relic, event data is stored as key-values pairs, charts and tables, and can be queried by users using NRQL query language. In Kubernetes clusters, New Relic uses a deployment to collect cluster events which can then be viewed on the console.  

Logs – logs are a historical description of a past activity with the timestamp and are used for forensics and troubleshooting purposes.  New Relic provides a central platform in its UI to help you check logs along with the metric data. In Kubernetes, New Relic has its own plugin for Fluent Bit to forward the logs to itself.  

Traces – traces are the paths of a request from the end-user all the way to the backend of a system. This can help you in identifying bottlenecks in the path and other performance issues. New Relic provides insight on how requests travel from one service to another.  

Data security in New Relic: 

By default, data stored in New Relic is encrypted at rest and in-transit to help prevent unauthorised access, breaches or threats.  

1. At rest – data stored in New Relic is encrypted at rest with the strongest cipher suite available in the environment. It uses FIPS 140-2 compliant hardware modules for encryption of data at rest. This setting cannot be disabled.  
2. In transit – data pertaining to agents, APIs and data traversing to third-party integrations that use TLS is encrypted in transit. The preferred protocol is TLS 1.2. 

Installation on Kubernetes: 

New Relic can be installed onto Kubernetes with the help of helm. The official helm repository for New Relic is available on GitHub which can be cloned and then used for the installation of the New Relic Infrastructure Bundle or the nri-bundle.  

  helm repo add newrelic https://helm-charts.newrelic.com 

The nri-bundle consists of several different components, each enhancing a different feature. These different components work together in fulfilling the different pillars of observability.  

The nri-bundle installation will result in the deployment of the components onto the Kubernetes cluster in the form of DaemonSets and deployments. Best practise is to deploy all these components in a separate namespace to isolate them from the rest of the workloads.  

You can also choose to install only the specific components you would need as well. This can be done by setting the values.yaml file to enable only the required components before installation.  

  helm upgrade –install newrelic-bundle newrlic/nri-bundle -f your-custom-values.yaml 

Potential problems in regulated environments: 

1. Supply chain security – whilst installing the nri-bundle in highly regulated environments like banks, you may need to whitelist the New Relic registry to be able to pull images. Alternatively, you could upload the images onto your local repository and pull images from there. 
2. Helm installation – sometimes all components do not get installed. Ensure you have enabled all components in the values file. You can use the debug option with helm to view the manifests that are being implemented onto Kubernetes. This will let you know how many resources are being created and if there are any errors with them. 
3. Kubernetes resources – components of the nri-bundle might have errors if there is a lack of sufficient resources on the Kubernetes cluster. Ensure you have sufficient CPU and memory on GKE prior to installation.  

Conclusion 

In my working environment, I installed New Relic onto GKE by using GitHub Action workflows. For the GitHub runners, I used an asmcli image available in the GCP GitHub repository for ASM packages. The reason for this choice is because it has the packages for gcloud, helm and kubectl already pre-installed and removes the burden of including their installation in the workflows. I did face the issues mentioned above and was able to resolve them with some troubleshooting. Hence, I could successfully deploy New Relic onto GKE and provide best practices to avoid potential issues that can come up along the way.