Big Data Modeling and Management Systems

This module introduces the concepts behind big data modeling and management, setting the stage for the remainder of the course. It covers essential topics including data ingestion, storage, quality, operations, scalability, and security. The module also presents real-world data management challenges through case studies from the energy sector (ConEd), gaming industry (Apmetrix), and flight data management (FlightStats).

This module explores the principles of big data modeling, focusing on various factors including data structure, operations that can be performed on the data, and constraints placed on the models. Students learn about different data model structures such as relational data models, semistructured data models, and array data models through practical examples using CSV files, JSON data, images, and sensor data. The module provides practical skills for modeling big data projects.

This module continues the exploration of big data modeling with specific approaches including vector space models, graph data models, and other specialized data models. Students gain practical experience with the Lucene search engine's vector data model and exploring graph data models using Gephi. The module builds on the previous modeling concepts to provide a comprehensive understanding of various modeling approaches for big data.

This module focuses on working with different types of data formats, with a special emphasis on streaming data and its unique characteristics compared to static data. Students learn about data models versus data formats, what data streams are, why streaming data requires different approaches, and the concept of data lakes. The module provides practical hands-on experience working with streaming data including weather data and Twitter feeds.

This module explores how managing big data requires different approaches to database management systems due to variations in data structure. It introduces the transition from traditional Database Management Systems (DBMS) to Big Data Management Systems (BDMS), and covers various applications suitable for big data management, including Redis (key-value store), Aerospike, AsterixDB (for semistructured data), Solr (for text management), and Vertica (for relational data).

This module provides a practical application of big data modeling and management concepts through a fictitious online game called "Catch the Pink Flamingo." Students learn to design a data model for the game, apply analytical tasks to improve the game, and utilize the data model effectively. This hands-on approach helps consolidate the knowledge gained throughout the course by applying it to a realistic scenario.