실시간 데이터 수집-6 (Elasticsearch & kibana 설치)

filebeat로 수집한 데이터를
metric 수집 및 적재에 특화되어 있는 promethues에 적재하는 것은 적절하지 않다고 판단하여
적재 장소를 Elasticsearch로 변경하게 되었다. (Filebeat와 호환성 보장)
또한 아키텍쳐에 필수요소는 아니나 Elasticsearch 데이터조회의 편의성을 위해 Kibana도 함께 설치를 진행하였다.

Elasticsearch 설치 및 kafka를 통한 데이터 적재를 기록할 예정이다.

 

Elasticsearch 설치 

• Elasticsearch 설치

$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.12.2-amd64.deb
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.12.2-amd64.deb.sha512
$ shasum -a 512 -c elasticsearch-8.12.2-amd64.deb.sha512 
$ sudo dpkg -i elasticsearch-8.12.2-amd64.deb
Selecting previously unselected package elasticsearch.
(Reading database ... 131249 files and directories currently installed.)
Preparing to unpack elasticsearch-8.12.2-amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (8.12.2) ...
Setting up elasticsearch (8.12.2) ...
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : GmUQFtHAn3-yg8PHrc*N

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

• 외부에서 접근 가능하도록 설정

$ sudo vi /etc/elasticsearch/elasticsearch.yml
========================================================
...
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: 0.0.0.0
...

인증서 발급 및 적용

• instances.yml 작성

# vi /etc/elasticsearch/instances.yml
instances:
    - name: "ip-172-31-9-181"
      ip:
          - "172.31.9.181"
    - name: "kibana"
      ip:
          - "172.31.9.181"
wq!

• 인증서 발급

# /usr/share/elasticsearch/bin/elasticsearch-certutil ca --days 36500
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.p12]:
Enter password for elastic-stack-ca.p12 :elastic

# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --silent --in /etc/elasticsearch/instances.yml --out certs.zip --ca elastic-stack-ca.p12 --days 36500
Enter password for CA (elastic-stack-ca.p12) :elastic
Enter password for ip-172-31-9-181/ip-172-31-9-181.p12 :elastic
Enter password for kibana/kibana.p12 :elastic

# mv /usr/share/elasticsearch/certs.zip /usr/share/elasticsearch/elastic-stack-ca.p12 /etc/elasticsearch/certs/
# cd /etc/elasticsearch/certs
# unzip certs.zip
Archive:  certs.zip
   creating: ip-172-31-9-181/
  inflating: ip-172-31-9-181/ip-172-31-9-181.p12
   creating: kibana/
  inflating: kibana/kibana.p12
# cp ip-172-31-9-181/ip-172-31-9-181.p12 http.p12

• keystore / truststore 비밀번호 업데이트

# /usr/share/elasticsearch# bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
Setting xpack.security.transport.ssl.keystore.secure_password already exists. Overwrite? [y/N]y
Enter value for xpack.security.transport.ssl.keystore.secure_password:elastic
# /usr/share/elasticsearch# bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
Setting xpack.security.transport.ssl.truststore.secure_password already exists. Overwrite? [y/N]y
Enter value for xpack.security.transport.ssl.truststore.secure_password:elastic
# /usr/share/elasticsearch# bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
Setting xpack.security.http.ssl.keystore.secure_password already exists. Overwrite? [y/N]y
Enter value for xpack.security.http.ssl.keystore.secure_password:elastic
# /usr/share/elasticsearch# bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password
Setting xpack.security.http.ssl.truststore.secure_password already exists. Overwrite? [y/N]y
Enter value for xpack.security.http.ssl.truststore.secure_password:elastic

• 인증서 elasticsearch.yml에 적용
  
  • xpack.security.http.ssl.keystore.path: certs/http.p12
  • xpack.security.transport.ssl.keystore.path: certs/http.p12
  • xpack.security.transport.ssl.truststore.path: certs/http.p12

# vi /etc/elasticsearch/elasticsearch.yml
...
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 17-03-2024 12:17:14
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  #keystore.path: certs/transport.p12
  #truststore.path: certs/transport.p12
  keystore.path: certs/http.p12
  truststore.path: certs/http.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["ip-172-31-9-181"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

Elasticsearch 실행

• 서비스 실행 / 부팅 시 서비스 자동실행 설정

$ sudo systemctl daemon-reload
$ sudo systemctl enable elasticsearch.service
$ sudo systemctl start elasticsearch.service

• 비밀번호 변경

# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i
This tool will reset the password of the [elastic] user.
You will be prompted to enter the password.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]:elastic
Re-enter password for [elastic]:elastic
Password for the [elastic] user successfully reset.

• 서비스 실행 확인

$ curl -X GET -k -u elastic:$ELASTIC_PASSWORD https://localhost:9200
{
  "name" : "ip-172-31-9-181",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "Jrs5Z5JTR6KLiQZGajzPzA",
  "version" : {
    "number" : "8.12.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "48a287ab9497e852de30327444b0809e55d46466",
    "build_date" : "2024-02-19T10:04:32.774273190Z",
    "build_snapshot" : false,
    "lucene_version" : "9.9.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Elasticsearch 삭제

$ sudo apt-get --purge autoremove elasticsearch
$ sudo rm -rf /var/lib/elasticsearch
$ sudo rm -rf /etc/elasticsearch

 

Kibana 설치 

• kibana 설치

$ sudo apt-get install apt-transport-https
$ wget https://artifacts.elastic.co/downloads/kibana/kibana-8.12.2-amd64.deb
$ shasum -a 512 kibana-8.12.2-amd64.deb
$ sudo dpkg -i kibana-8.12.2-amd64.deb

• 외부 접근 허용하도록 설정

$ sudo vi /etc/kibana/kibana.yml
# For more configuration options see the configuration guide for Kibana in
# https://www.elastic.co/guide/index.html

# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"
server.host: "0.0.0.0"
wq!

Kibana 실행

• 서비스 실행 / 부팅 시 서비스 자동실행 설정

$ sudo /bin/systemctl daemon-reload
$ sudo /bin/systemctl enable kibana.service
$ sudo systemctl start kibana.service

• kibana 접속코드 확인

# systemctl status kibana
● kibana.service - Kibana
     Loaded: loaded (/lib/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2024-03-24 10:53:30 UTC; 13s ago
       Docs: https://www.elastic.co
   Main PID: 11276 (node)
      Tasks: 11 (limit: 19165)
     Memory: 307.9M
     CGroup: /system.slice/kibana.service
             └─11276 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist

Mar 24 10:53:42 ip-172-31-9-181 kibana[11276]: [2024-03-24T10:53:42.514+00:00][INFO ][plugins-service] Plugin "securitySolutionServerless" is disabled.
Mar 24 10:53:42 ip-172-31-9-181 kibana[11276]: [2024-03-24T10:53:42.515+00:00][INFO ][plugins-service] Plugin "serverless" is disabled.
Mar 24 10:53:42 ip-172-31-9-181 kibana[11276]: [2024-03-24T10:53:42.515+00:00][INFO ][plugins-service] Plugin "serverlessObservability" is disabled.
Mar 24 10:53:42 ip-172-31-9-181 kibana[11276]: [2024-03-24T10:53:42.515+00:00][INFO ][plugins-service] Plugin "serverlessSearch" is disabled.
Mar 24 10:53:43 ip-172-31-9-181 kibana[11276]: [2024-03-24T10:53:43.176+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
Mar 24 10:53:43 ip-172-31-9-181 kibana[11276]: [2024-03-24T10:53:43.345+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
Mar 24 10:53:43 ip-172-31-9-181 kibana[11276]: [2024-03-24T10:53:43.370+00:00][INFO ][preboot] "interactiveSetup" plugin is holding setup: Validating Elasticsearch connection configur>
Mar 24 10:53:43 ip-172-31-9-181 kibana[11276]: [2024-03-24T10:53:43.407+00:00][INFO ][root] Holding setup until preboot stage is completed.
Mar 24 10:53:43 ip-172-31-9-181 kibana[11276]: i Kibana has not been configured.
Mar 24 10:53:43 ip-172-31-9-181 kibana[11276]: Go to http://0.0.0.0:5601/?code=038469 to get started.

• kibana 접속 (http://<<ec2 주소>>:5601/?code=위에서 확인한 접속코드)
  • 하단의 Configure manually 클릭하여 Elasticsearch 연동

• kibana_system 유저 비밀번호 재설정 방법

#  /usr/share/elasticsearch/bin/elasticsearch-reset-password -u kibana_system -i
This tool will reset the password of the [kibana_system] user.
You will be prompted to enter the password.
Please confirm that you would like to continue [y/N]y


Enter password for [kibana_system]: elastic
Re-enter password for [kibana_system]: elastic
Password for the [kibana_system] user successfully reset.

Kibana 삭제 

$ sudo apt-get --purge autoremove kibana
$ sudo rm -rf /var/lib/kibana
$ sudo rm -rf /etc/kibana

Kafka Connector 설정

Confluent Platform용 Elasticsearch Sink 커넥터는 버전 8.x를 포함하여 Elasticsearch 버전 7.x 이상에 대한 지원을 제공한다.

• connector 설치

$ confluent-hub install confluentinc/kafka-connect-elasticsearch:14.0.12

• connector 설정

{
  "name": "ElasticsearchSinkConnectorConnector_0",
  "config": {
    "key.converter.schemas.enable": "false",
    "value.converter.schemas.enable": "false",
    "name": "ElasticsearchSinkConnectorConnector_0",
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "topics": "worldmoney.filebeat",
    "errors.deadletterqueue.context.headers.enable": "false",
    "connection.url": "https://172.31.9.181:9200",
    "connection.username": "elastic",
    "connection.password": "*******",
    "key.ignore": "true",
    "schema.ignore": "true",
    "behavior.on.null.values": "ignore",
    "behavior.on.malformed.documents": "fail",
    "write.method": "insert",
    "elastic.security.protocol": "ssl",
    "elastic.https.ssl.keystore.location": "/etc/elasticsearch/certs/http.p12",
    "elastic.https.ssl.keystore.password": "*******",
    "elastic.https.ssl.truststore.location": "/etc/elasticsearch/certs/http.p12",
    "elastic.https.ssl.truststore.password": "*******",
    "elastic.https.ssl.keystore.type": "PKCS12",
    "elastic.https.ssl.truststore.type": "PKCS12"
  }
}

적재된 데이터 확인 

• kibana의 devtools를 통해 쉽고 빠르게 확인이 가능.
• 인덱스 이름 = confluent topic 이름
• kibana 접속 : http://<퍼블릭 IP DNS>:5601

• devtools 클릭

• 인덱스 조회 및 document 조회

 

 

---

참고

Elasticsearch 설치 : https://www.elastic.co/guide/en/elasticsearch/reference/8.12/deb.html#deb-repo

kibana 설치 : https://www.elastic.co/guide/en/kibana/8.12/deb.html

elasticsearch 인증서 발급 및 적용 : https://realkoy.tistory.com/entry/elasticsearch-840-%EC%9D%B8%EC%A6%9D%EC%84%9C-%EC%83%9D%EC%84%B1%ED%95%98%EC%97%AC-%EC%84%A4%EC%B9%98