GCP - ELK Stack on Ubuntu 21.04

GCP Marketplace Image User Guide

This AMI features a built-in ELK (Elasticsearch, Logstash, and Kibana) stack on Google Cloud Platform which can be customize to each users’ preference. This solution is ideal for user who wants the full functionality of an ELK stack and want to save some time on installing the software. We have installed all the ELK components on an Ubuntu environment, so that you can start your data analysis immediately. As easy as it sound, with just a few clicks, you can provision an ELK instance in less than a minute.

After you have provisioned the VM instance, go to the GCP Compute Engine console. Check that the status is running. Note the External IP address to access Kibana on your browser.

Wait a few minutes to provision Kibana. Then paste the IPv4 Public IP onto your web browser and it will bring you to the Kibana launch page:

{https://{External IP}:5601 

You can further customize the ELK stack by SSH into the instance using the following command:

gcloud beta compute ssh --zone "us-west1-a" "elk"  --project "classmethod-can-public"

GCP allow users to SSH instance directly on the console. Click on the arrow next to SSH of your instance to see more SSH options. If you wish to SSH using terminal, make sure you set up gcloud and OAuth on your local machine.

SSH documentation: https://cloud.google.com/sdk/gcloud/reference/compute/ssh

Elasticsearch

This documentation guide you through how to get started with Elasticsearch.

https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

Run this command to see if Elasticsearch is running.

curl -X GET "localhost:9200/" 

It should give the following if successful:

{
 “name” : “ip-172–31–28–88”,
 “cluster_name” : “my-application”,
 “cluster_uuid” : “dLfzIIfwSEWOnuu5i1I6OQ”,
 “version” : {
 “number” : “7.2.0”,
     “build_flavor” : “default”,
     “build_type” : “deb”,
     “build_hash” : “508c38a”,
     “build_date” : “2019–06–20T15:54:18.811730Z”,
     “build_snapshot” : false,
     “lucene_version” : “8.0.0”,
     “minimum_wire_compatibility_version” : “6.8.0”,
     “minimum_index_compatibility_version” : “6.0.0-beta1”
 },
 “tagline” : “You Know, for Search”
}

Alternatively, you can also check the status by typing

sudo service elasticsearch status

If the service is down, restart or stop elasticsearch service.

sudo service elasticsearch restart 

or

sudo service elasticsearch stop 

Wait at least 30s to check the status again.

By default, all three services will automatically start when you launch the instance.

Start using Elasticsearch with Introduction to Indexing.

Logstash

Check status using this command:

sudo service logstash status 

Follow this tutorial to get started on a Logstash pipeline.

Kibana

Check status using this command:

 sudo service kibana status

Refer to these official user guides to start adding sample data!

Configuring Elasticsearch

Kibana Guide - Get Started

Troubleshooting

Scenario 1.

If Kibana doesn’t show up on web browser, reboot the instance on AWS console.

Alternatively, you can also SSH into the instance and run:

sudo reboot 

Scenario 2.

If you encounter the following page indicating that the connection is not private, there’s a work around.

Select anywhere on the error page, and type the following code. This will bring you to the Kibana launch page. (Refer to reference number 4 & 5)

thisisunsafe

References

1. Install Elasticsearch with Debian Package

2. Installing Logstash

3. Install Kibana with Debian Package

4. No “Proceed Anyway” option on NET::ERR_CERT_INVALID in Chrome on MacOS

5. NET::ERR_CERT_REVOKED Chrome MacOS home server