Study of the Big Data Collection Scheme Based Apache Flume for Log Collection

Abstract

With the advances in IT technology and the rapid adoption of smart devices, users can more easily produce, distribute and consume data through network access anytime, anywhere.The data generated by users in response to these changes has increased dramatically.This has required companies to collect large amounts of logs, and these companies are actively researching and developing big data collection technologies.In this paper, we have studied the big data collection technology based on Apache Flume for bulk log collection.The structure for bulk log processing is designed to be matched with one web server and one Flume agent, and the Flume agents connected to the web server are connected to the Flume agent that plays the role of storing in the Hadoop distributed file system.This makes the collection of big data logs more efficient.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 8 citations exist upstream.