실시간 데이터 수집-5 (Filebeat 설치)

환율 데이터를 API로 받아보니 json array 형태로 되어있어서 kafka로 직접적으로 넣기가 힘들었다.
filestream connector를 사용하면 json array를 하나하나 풀어서 파일에 write 한 후 읽어갈 수 있지만
실제 운영환경에서는 사용을 권하지 않는다고 공식문서에 적혀있었다.

그렇다면 log 파일 형태로 관리했을때 어떻게 kafka로 수집할 수 있을지 찾아보다
ELK stack에서 자주 사용되는 filebeat를 사용하기로 결정했다.
connector의 역할을 충분히 대체할 수 있고 제한적인 환경에서 logstash보다 적은 리소스를 차지한다.

 

환율 log 파일 생성

현재 환율 api를 통해 가져오는 데이터가 json형태의 데이터가 여러개 들어있는 array 형태이기 때문에 
array안에 들어있는 json을 하나씩 가져와야한다.
현재 형태 : [json1, json2, json3, ...]
따라서, array를 풀어주는 가공 후 log 파일에 각 json을 한 줄 씩 append하도록 shell script를 작성한다.

• shell script

#!/bin/bash

GETDATE=$(date "+%Y-%m-%d")
money=$(curl -X GET 'https://www.koreaexim.go.kr/site/program/financial/exchangeJSON?authkey=trbi20dGwQLpPdFaGwOta4QA13A1NIDd&searchdate='$GETDATE'&data=AP01')

money_array=$(echo ${money:1:-1} | sed -e "s/},{/}?{/g")

(echo $money_array | tr '?' '\n') >>world_money.log

• world_money.log 

{"result":1,"cur_unit":"KRW","ttb":"0","tts":"0","deal_bas_r":"1","bkpr":"1","yy_efee_r":"0","ten_dd_efee_r":"0","kftc_bkpr":"1","kftc_deal_bas_r":"1","cur_nm":"한국 원"}
{"result":1,"cur_unit":"KWD","ttb":"4,330.84","tts":"4,418.33","deal_bas_r":"4,374.59","bkpr":"4,374","yy_efee_r":"0","ten_dd_efee_r":"0","kftc_bkpr":"4,374","kftc_deal_bas_r":"4,374.59","cur_nm":"쿠웨이트 디나르"}
{"result":1,"cur_unit":"MYR","ttb":"281.49","tts":"287.18","deal_bas_r":"284.34","bkpr":"284","yy_efee_r":"0","ten_dd_efee_r":"0","kftc_bkpr":"284","kftc_deal_bas_r":"284.34","cur_nm":"말레이지아 링기트"}
{"result":1,"cur_unit":"NOK","ttb":"123.26","tts":"125.75","deal_bas_r":"124.51","bkpr":"124","yy_efee_r":"0","ten_dd_efee_r":"0","kftc_bkpr":"124","kftc_deal_bas_r":"124.51","cur_nm":"노르웨이 크로네"}
{"result":1,"cur_unit":"NZD","ttb":"797.94","tts":"814.06","deal_bas_r":"806","bkpr":"806","yy_efee_r":"0","ten_dd_efee_r":"0","kftc_bkpr":"806","kftc_deal_bas_r":"806","cur_nm":"뉴질랜드 달러"}
{"result":1,"cur_unit":"SAR","ttb":"355.23","tts":"362.4","deal_bas_r":"358.82","bkpr":"358","yy_efee_r":"0","ten_dd_efee_r":"0","kftc_bkpr":"358","kftc_deal_bas_r":"358.82","cur_nm":"사우디 리얄"}
{"result":1,"cur_unit":"SEK","ttb":"125.18","tts":"127.71","deal_bas_r":"126.45","bkpr":"126","yy_efee_r":"0","ten_dd_efee_r":"0","kftc_bkpr":"126","kftc_deal_bas_r":"126.45","cur_nm":"스웨덴 크로나"}
{"result":1,"cur_unit":"SGD","ttb":"987.76","tts":"1,007.71","deal_bas_r":"997.74","bkpr":"997","yy_efee_r":"0","ten_dd_efee_r":"0","kftc_bkpr":"997","kftc_deal_bas_r":"997.74","cur_nm":"싱가포르 달러"}
{"result":1,"cur_unit":"THB","ttb":"36.61","tts":"37.34","deal_bas_r":"36.98","bkpr":"36","yy_efee_r":"0","ten_dd_efee_r":"0","kftc_bkpr":"36","kftc_deal_bas_r":"36.98","cur_nm":"태국 바트"}
{"result":1,"cur_unit":"USD","ttb":"1,332.34","tts":"1,359.25","deal_bas_r":"1,345.8","bkpr":"1,345","yy_efee_r":"0","ten_dd_efee_r":"0","kftc_bkpr":"1,345","kftc_deal_bas_r":"1,345.8","cur_nm":"미국 달러"}
...

Filebeat 설치 및 실행

• tar 파일 다운로드 (ubuntu 20.04 환경)

$ wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.12.2-linux-x86_64.tar.gz

• tar 파일 압축해제

$ tar -zxvf filebeat-8.12.2-linux-x86_64.tar.gz

• filebeat.yml 작성

$ vi kafka_filebeat.yaml
===============================================================
filebeat.inputs:
- type: log
  enabled: true
  #document_type: BROWSER
  paths:
    - /home/ubuntu/world_money/world_money.log
  json.keys_under_root: true
  fields:
          #index_name: "CHROME"
    topic_name: "worldmoney.filebeat"

  encoding: utf-8


output.kafka:
  enabled: true
  hosts: [ 'localhost:9092' ]
  topic: '%{[fields.topic_name]}'

  partition.round_robin:
    reachable_only: false

    #required_acks: 1

  #sasl.mechanism: PLAIN
  #username: producer
  #password: dochi

  codec.json:
    pretty: false


logging.level: debug
path.data: /home/ubuntu/filebeat/data
path.logs: /home/ubuntu/filebeat/logs

• filebeat.yml start

$ ./filebeat -e -c kafka_filebeat.yaml

 

실행 결과 

{"log.level":"info","@timestamp":"2024-03-10T10:53:55.010Z","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":45136109}},"memory":{"mem":{"usage":{"bytes":45268992}}}},"cpu":{"system":{"ticks":80},"total":{"ticks":220,"time":{"ms":10},"value":220},"user":{"ticks":140,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":13},"info":{"ephemeral_id":"b1246739-d647-42a7-8133-01b8ec7f93db","uptime":{"ms":330074},"version":"8.12.2"},"memstats":{"gc_next":38455632,"memory_alloc":19568488,"memory_total":66588864,"rss":100323328},"runtime":{"goroutines":43}},"filebeat":{"events":{"active":0},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.06,"15":0.09,"5":0.11,"norm":{"1":0.015,"15":0.0225,"5":0.0275}}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2024-03-10T10:53:55.039Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input.(*Runner).Run","file.name":"input/input.go","file.line":137},"message":"Run input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:53:55.040Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).Run","file.name":"log/input.go","file.line":228},"message":"Start next scan","service.name":"filebeat","input_id":"3ec9ee6e-855c-444a-85d3-eabcbe7d806d","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:53:55.040Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.getFileState","file.name":"log/input.go","file.line":476},"message":"Check file for harvesting: /home/ubuntu/world_money/world_money.log","service.name":"filebeat","input_id":"3ec9ee6e-855c-444a-85d3-eabcbe7d806d","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:53:55.040Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).harvestExistingFile","file.name":"log/input.go","file.line":574},"message":"Update existing file for harvesting: /home/ubuntu/world_money/world_money.log, offset: 9492","service.name":"filebeat","input_id":"3ec9ee6e-855c-444a-85d3-eabcbe7d806d","source_file":"/home/ubuntu/world_money/world_money.log","state_id":"native::774157-51713","finished":false,"os_id":"774157-51713","old_source":"/home/ubuntu/world_money/world_money.log","old_finished":false,"old_os_id":"774157-51713","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:53:55.040Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).harvestExistingFile","file.name":"log/input.go","file.line":627},"message":"Harvester for file is still running: /home/ubuntu/world_money/world_money.log","service.name":"filebeat","input_id":"3ec9ee6e-855c-444a-85d3-eabcbe7d806d","source_file":"/home/ubuntu/world_money/world_money.log","state_id":"native::774157-51713","finished":false,"os_id":"774157-51713","old_source":"/home/ubuntu/world_money/world_money.log","old_finished":false,"old_os_id":"774157-51713","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:53:55.040Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).cleanupStates","file.name":"log/input.go","file.line":292},"message":"input states cleaned up. Before: 1, After: 1, Pending: 0","service.name":"filebeat","input_id":"3ec9ee6e-855c-444a-85d3-eabcbe7d806d","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.044Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.044Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"bkpr\": \"362\",\n  \"yy_efee_r\": \"0\",\n  \"cur_nm\": \"아랍에미리트 디르함\",\n  \"result\": 1,\n  \"log\": {\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    },\n    \"offset\": 9492\n  },\n  \"cur_unit\": \"AED\",\n  \"kftc_deal_bas_r\": \"362.49\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"ten_dd_efee_r\": \"0\",\n  \"ttb\": \"358.86\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"deal_bas_r\": \"362.49\",\n  \"kftc_bkpr\": \"362\",\n  \"tts\": \"366.11\",\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.044Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.044Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"log\": {\n    \"offset\": 9708,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"yy_efee_r\": \"0\",\n  \"kftc_deal_bas_r\": \"870.47\",\n  \"result\": 1,\n  \"ten_dd_efee_r\": \"0\",\n  \"ttb\": \"861.76\",\n  \"tts\": \"879.17\",\n  \"cur_nm\": \"호주 달러\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"agent\": {\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\",\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\"\n  },\n  \"deal_bas_r\": \"870.47\",\n  \"bkpr\": \"870\",\n  \"kftc_bkpr\": \"870\",\n  \"cur_unit\": \"AUD\"\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.044Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.044Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"tts\": \"3,566.97\",\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"ten_dd_efee_r\": \"0\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"bkpr\": \"3,531\",\n  \"cur_unit\": \"BHD\",\n  \"log\": {\n    \"offset\": 9909,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"cur_nm\": \"바레인 디나르\",\n  \"result\": 1,\n  \"kftc_bkpr\": \"3,531\",\n  \"ttb\": \"3,496.34\",\n  \"deal_bas_r\": \"3,531.66\",\n  \"yy_efee_r\": \"0\",\n  \"kftc_deal_bas_r\": \"3,531.66\",\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.044Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.044Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"kftc_bkpr\": \"990\",\n  \"cur_nm\": \"브루나이 달러\",\n  \"bkpr\": \"990\",\n  \"yy_efee_r\": \"0\",\n  \"result\": 1,\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"tts\": \"1,000.12\",\n  \"deal_bas_r\": \"990.22\",\n  \"ten_dd_efee_r\": \"0\",\n  \"cur_unit\": \"BND\",\n  \"log\": {\n    \"offset\": 10128,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"kftc_deal_bas_r\": \"990.22\",\n  \"ttb\": \"980.31\",\n  \"agent\": {\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\",\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.045Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.045Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"kftc_deal_bas_r\": \"985.6\",\n  \"deal_bas_r\": \"985.6\",\n  \"ttb\": \"975.74\",\n  \"cur_nm\": \"캐나다 달러\",\n  \"kftc_bkpr\": \"985\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"log\": {\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    },\n    \"offset\": 10337\n  },\n  \"result\": 1,\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  },\n  \"tts\": \"995.45\",\n  \"bkpr\": \"985\",\n  \"cur_unit\": \"CAD\",\n  \"ten_dd_efee_r\": \"0\",\n  \"yy_efee_r\": \"0\",\n  \"input\": {\n    \"type\": \"log\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.045Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.045Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  },\n  \"deal_bas_r\": \"1,512.87\",\n  \"cur_unit\": \"CHF\",\n  \"ttb\": \"1,497.74\",\n  \"yy_efee_r\": \"0\",\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"bkpr\": \"1,512\",\n  \"kftc_bkpr\": \"1,512\",\n  \"ten_dd_efee_r\": \"0\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"log\": {\n    \"offset\": 10539,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"kftc_deal_bas_r\": \"1,512.87\",\n  \"cur_nm\": \"스위스 프랑\",\n  \"result\": 1,\n  \"tts\": \"1,527.99\"\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.045Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.045Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"bkpr\": \"184\",\n  \"kftc_deal_bas_r\": \"184.76\",\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"result\": 1,\n  \"deal_bas_r\": \"184.76\",\n  \"yy_efee_r\": \"0\",\n  \"log\": {\n    \"offset\": 10755,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  },\n  \"cur_nm\": \"위안화\",\n  \"tts\": \"186.6\",\n  \"cur_unit\": \"CNH\",\n  \"ttb\": \"182.91\",\n  \"ten_dd_efee_r\": \"0\",\n  \"kftc_bkpr\": \"184\"\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.045Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.045Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"ttb\": \"191.86\",\n  \"bkpr\": \"193\",\n  \"log\": {\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    },\n    \"offset\": 10951\n  },\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"agent\": {\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\",\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\"\n  },\n  \"kftc_deal_bas_r\": \"193.8\",\n  \"tts\": \"195.73\",\n  \"result\": 1,\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"ten_dd_efee_r\": \"0\",\n  \"cur_unit\": \"DKK\",\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"cur_nm\": \"덴마아크 크로네\",\n  \"yy_efee_r\": \"0\",\n  \"deal_bas_r\": \"193.8\",\n  \"kftc_bkpr\": \"193\",\n  \"input\": {\n    \"type\": \"log\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.045Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.045Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"kftc_deal_bas_r\": \"1,444.7\",\n  \"result\": 1,\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  },\n  \"log\": {\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    },\n    \"offset\": 11159\n  },\n  \"kftc_bkpr\": \"1,444\",\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"deal_bas_r\": \"1,444.7\",\n  \"yy_efee_r\": \"0\",\n  \"cur_unit\": \"EUR\",\n  \"cur_nm\": \"유로\",\n  \"ttb\": \"1,430.25\",\n  \"bkpr\": \"1,444\",\n  \"tts\": \"1,459.14\",\n  \"ten_dd_efee_r\": \"0\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.045Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.045Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"cur_unit\": \"GBP\",\n  \"result\": 1,\n  \"deal_bas_r\": \"1,688.75\",\n  \"bkpr\": \"1,688\",\n  \"tts\": \"1,705.63\",\n  \"cur_nm\": \"영국 파운드\",\n  \"log\": {\n    \"offset\": 11363,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"ttb\": \"1,671.86\",\n  \"ten_dd_efee_r\": \"0\",\n  \"kftc_deal_bas_r\": \"1,688.75\",\n  \"yy_efee_r\": \"0\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"kftc_bkpr\": \"1,688\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.045Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.045Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  },\n  \"ttb\": \"168.46\",\n  \"tts\": \"171.87\",\n  \"bkpr\": \"170\",\n  \"result\": 1,\n  \"deal_bas_r\": \"170.17\",\n  \"kftc_deal_bas_r\": \"170.17\",\n  \"kftc_bkpr\": \"170\",\n  \"ten_dd_efee_r\": \"0\",\n  \"yy_efee_r\": \"0\",\n  \"cur_nm\": \"홍콩 달러\",\n  \"cur_unit\": \"HKD\",\n  \"log\": {\n    \"offset\": 11579,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.045Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.045Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"kftc_deal_bas_r\": \"8.52\",\n  \"bkpr\": \"8\",\n  \"tts\": \"8.6\",\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"result\": 1,\n  \"cur_unit\": \"IDR(100)\",\n  \"log\": {\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    },\n    \"offset\": 11780\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  },\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"ten_dd_efee_r\": \"0\",\n  \"deal_bas_r\": \"8.52\",\n  \"yy_efee_r\": \"0\",\n  \"kftc_bkpr\": \"8\",\n  \"ttb\": \"8.43\",\n  \"cur_nm\": \"인도네시아 루피아\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.046Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.045Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"ttb\": \"874.73\",\n  \"bkpr\": \"883\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"agent\": {\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\",\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"result\": 1,\n  \"ten_dd_efee_r\": \"0\",\n  \"tts\": \"892.4\",\n  \"deal_bas_r\": \"883.57\",\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"yy_efee_r\": \"0\",\n  \"cur_nm\": \"일본 옌\",\n  \"cur_unit\": \"JPY(100)\",\n  \"kftc_bkpr\": \"883\",\n  \"log\": {\n    \"offset\": 11985,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"kftc_deal_bas_r\": \"883.57\",\n  \"input\": {\n    \"type\": \"log\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.046Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.046Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"ttb\": \"0\",\n  \"cur_unit\": \"KRW\",\n  \"cur_nm\": \"한국 원\",\n  \"kftc_bkpr\": \"1\",\n  \"deal_bas_r\": \"1\",\n  \"yy_efee_r\": \"0\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"tts\": \"0\",\n  \"ten_dd_efee_r\": \"0\",\n  \"log\": {\n    \"offset\": 12187,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"bkpr\": \"1\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"result\": 1,\n  \"kftc_deal_bas_r\": \"1\"\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.046Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.046Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"bkpr\": \"4,326\",\n  \"deal_bas_r\": \"4,326.38\",\n  \"log\": {\n    \"offset\": 12361,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"ttb\": \"4,283.11\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"agent\": {\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\",\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\"\n  },\n  \"cur_nm\": \"쿠웨이트 디나르\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"kftc_deal_bas_r\": \"4,326.38\",\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"tts\": \"4,369.64\",\n  \"result\": 1,\n  \"ten_dd_efee_r\": \"0\",\n  \"cur_unit\": \"KWD\",\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"kftc_bkpr\": \"4,326\",\n  \"yy_efee_r\": \"0\"\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.046Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.046Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"ten_dd_efee_r\": \"0\",\n  \"cur_nm\": \"말레이지아 링기트\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  },\n  \"tts\": \"281.49\",\n  \"ttb\": \"275.92\",\n  \"bkpr\": \"278\",\n  \"kftc_bkpr\": \"278\",\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"yy_efee_r\": \"0\",\n  \"cur_unit\": \"MYR\",\n  \"result\": 1,\n  \"deal_bas_r\": \"278.71\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"kftc_deal_bas_r\": \"278.71\",\n  \"log\": {\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    },\n    \"offset\": 12583\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.046Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.046Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"tts\": \"127.87\",\n  \"log\": {\n    \"offset\": 12796,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"agent\": {\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\",\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\"\n  },\n  \"kftc_bkpr\": \"126\",\n  \"ten_dd_efee_r\": \"0\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"result\": 1,\n  \"yy_efee_r\": \"0\",\n  \"ttb\": \"125.34\",\n  \"kftc_deal_bas_r\": \"126.61\",\n  \"cur_unit\": \"NOK\",\n  \"bkpr\": \"126\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"deal_bas_r\": \"126.61\",\n  \"cur_nm\": \"노르웨이 크로네\"\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.046Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.046Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"deal_bas_r\": \"821.34\",\n  \"yy_efee_r\": \"0\",\n  \"ttb\": \"813.12\",\n  \"tts\": \"829.55\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"log\": {\n    \"offset\": 13006,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"bkpr\": \"821\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"ten_dd_efee_r\": \"0\",\n  \"kftc_deal_bas_r\": \"821.34\",\n  \"kftc_bkpr\": \"821\",\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  },\n  \"cur_unit\": \"NZD\",\n  \"result\": 1,\n  \"cur_nm\": \"뉴질랜드 달러\"\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.046Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.046Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"cur_unit\": \"SAR\",\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"log\": {\n    \"offset\": 13213,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"ttb\": \"351.44\",\n  \"kftc_bkpr\": \"354\",\n  \"tts\": \"358.53\",\n  \"deal_bas_r\": \"354.99\",\n  \"kftc_deal_bas_r\": \"354.99\",\n  \"cur_nm\": \"사우디 리얄\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"bkpr\": \"354\",\n  \"ten_dd_efee_r\": \"0\",\n  \"yy_efee_r\": \"0\",\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"result\": 1,\n  \"input\": {\n    \"type\": \"log\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.046Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.046Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"cur_nm\": \"스웨덴 크로나\",\n  \"cur_unit\": \"SEK\",\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"kftc_deal_bas_r\": \"129.49\",\n  \"ttb\": \"128.19\",\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"result\": 1,\n  \"log\": {\n    \"offset\": 13417,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"deal_bas_r\": \"129.49\",\n  \"bkpr\": \"129\",\n  \"yy_efee_r\": \"0\",\n  \"ten_dd_efee_r\": \"0\",\n  \"kftc_bkpr\": \"129\",\n  \"tts\": \"130.78\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.046Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.046Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"agent\": {\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\",\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\"\n  },\n  \"yy_efee_r\": \"0\",\n  \"cur_unit\": \"SGD\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"bkpr\": \"990\",\n  \"ten_dd_efee_r\": \"0\",\n  \"kftc_bkpr\": \"990\",\n  \"cur_nm\": \"싱가포르 달러\",\n  \"result\": 1,\n  \"kftc_deal_bas_r\": \"990.22\",\n  \"tts\": \"1,000.12\",\n  \"ttb\": \"980.31\",\n  \"deal_bas_r\": \"990.22\",\n  \"log\": {\n    \"offset\": 13624,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.047Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.046Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"cur_unit\": \"THB\",\n  \"ttb\": \"36.7\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"agent\": {\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\",\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"yy_efee_r\": \"0\",\n  \"kftc_deal_bas_r\": \"37.08\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"result\": 1,\n  \"deal_bas_r\": \"37.08\",\n  \"log\": {\n    \"offset\": 13833,\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    }\n  },\n  \"bkpr\": \"37\",\n  \"kftc_bkpr\": \"37\",\n  \"cur_nm\": \"태국 바트\",\n  \"tts\": \"37.45\",\n  \"ten_dd_efee_r\": \"0\"\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.047Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-03-10T10:54:00.047Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\"\n  },\n  \"kftc_bkpr\": \"1,331\",\n  \"kftc_deal_bas_r\": \"1,331.4\",\n  \"yy_efee_r\": \"0\",\n  \"ten_dd_efee_r\": \"0\",\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"log\": {\n    \"file\": {\n      \"path\": \"/home/ubuntu/world_money/world_money.log\"\n    },\n    \"offset\": 14027\n  },\n  \"cur_nm\": \"미국 달러\",\n  \"ttb\": \"1,318.08\",\n  \"result\": 1,\n  \"bkpr\": \"1,331\",\n  \"tts\": \"1,344.71\",\n  \"fields\": {\n    \"topic_name\": \"worldmoney.filebeat\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"host\": {\n    \"name\": \"ip-172-31-9-181\"\n  },\n  \"agent\": {\n    \"id\": \"b37c9704-162c-461a-ba9b-6216185d5a2e\",\n    \"name\": \"ip-172-31-9-181\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"b1246739-d647-42a7-8133-01b8ec7f93db\"\n  },\n  \"cur_unit\": \"USD\",\n  \"deal_bas_r\": \"1,331.4\"\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-10T10:54:00.047Z","log.logger":"input.harvester","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Log).Read","file.name":"log/log.go","file.line":111},"message":"End of file reached: /home/ubuntu/world_money/world_money.log; Backoff now.","service.name":"filebeat","input_id":"3ec9ee6e-855c-444a-85d3-eabcbe7d806d","source_file":"/home/ubuntu/world_money/world_money.log","state_id":"native::774157-51713","finished":false,"os_id":"774157-51713","old_source":"/home/ubuntu/world_money/world_money.log","old_finished":true,"old_os_id":"774157-51713","harvester_id":"28daba71-9bd6-4bf6-958d-40af0fe74ac3","ecs.version":"1.6.0"}

 

 

서버 재기동 시 자동으로 실행되도록 설정

$ vi /etc/rc.local
=====================================================
#!/bin/bash

export CONFLUENT_HOME=/home/ubuntu/confluent/confluent-7.6.0

/home/ubuntu/confluent/confluent-7.6.0/bin/confluent local services start
nohup /home/ubuntu/filebeat/filebeat-8.12.2-linux-x86_64/filebeat -c /home/ubuntu/filebeat/filebeat-8.12.2-linux-x86_64/kafka_filebeat.yaml -e > /home/ubuntu/filebeat/filebeat-8.12.2-linux-x86_64/logs/kafka.log 2>&1&

exit 0