ETL Big Data Pipeline

A real-time data streaming pipeline built around GitHub API data: Kafka handles asynchronous message brokering, Flink (via PyFlink) performs stream processing and transformations, Hadoop/HDFS with Hive and HBase provide distributed storage and querying, and Elasticsearch + Kibana handle indexing and visualization dashboards. Includes a word-count analysis example, Hive table loading from CSV, and HBase schema design with column families, with attention to cross-tool version compatibility.