Big Data Integration and Processing

This introductory module welcomes students to the third course in the Big Data Specialization, focusing on basic concepts in big data integration and processing. Students are guided through installing Docker, downloading the course datasets, and learning how to work with Jupyter notebooks. It builds on concepts from previous courses and introduces the unique challenges of big data processing.

This module covers various aspects of data retrieval and relational querying, focusing on the fundamentals of retrieving data from databases. Topics include basic data retrieval concepts, querying techniques for working with multiple relations, writing and using subqueries, and practical experience with the Postgres database system. Students learn essential SQL querying skills for working with relational data.

This module extends the data retrieval concepts to cover NoSQL data, focusing on data aggregation and working with data frames. Students are introduced to MongoDB and Aerospike databases, and learn how to use Pandas to retrieve and manipulate data from these sources. The module covers querying JSON data, using aggregation functions, and exploring Pandas DataFrames for data analysis.

This module introduces students to data integration tools and provides practical insight into information integration processes. Students learn about Splunk and Datameer for big data management and processing, with hands-on experience installing and using Splunk Enterprise. The module covers integration scenarios, multichannel customer analytics, and practical applications like connected cars with Ford's OpenXC and Splunk. Students gain practical skills in writing and exploring Splunk queries and creating pivot reports.

This module introduces big data pipelines and workflows along with processing and analysis techniques using Apache Spark. Students learn about high-level processing operations, aggregation operations, and typical analytical operations in big data pipelines. The module covers an overview of big data processing systems, the integration and processing layer, and provides hands-on experience with Apache Spark including the classic WordCount example. Students gain practical skills in designing and implementing big data workflows.

This module delves deeper into big data processing by exploring the inner workings of the Spark Core and introducing key tools in the Spark toolkit. Students learn about programming in Spark using Resilient Distributed Datasets (RDDs) in pipelines, Spark transformations and actions, and various Spark components including Spark SQL, Spark Streaming, Spark MLlib for machine learning, and GraphX for graph processing. The module provides hands-on experience with SparkSQL, Spark DataFrames, and analyzing sensor data with Spark Streaming.

This module provides practical hands-on experience applying the concepts learned about Spark and MongoDB to analyze Twitter data. Students work with soccer tweets, learning to express analytical questions as MongoDB queries, export data from MongoDB to CSV format, and analyze tweets about countries. The module consolidates knowledge from previous modules through practical application to real-world social media data analysis.