Big Data Mining and Applications, Fall 2023

Notes on Online Sessions of this Course

[NOTE] For those who really cannot attend the class in person due to some unforeseeable reasons, you could join the corresponding online sessions in Teams. During course sessions, remember to sign up your name and student ID in Teams, when the TA will remind you.

Please read the notes to the course very carefully about the requirements for the programming homework and quiz. [*** NOTE ***]: This course is NOT an introductory course for undergraduates. You are strongly suggested to take the first course of "Introduction to Big Data Analytcis" or "Data Mining" in the undergraduate-level as a prerequisite.
It contains very heavy loads of parallel programming, which is a totally different concept from sequential programming.
Please think twice before taking this course!

Course Information

• Instructor: Jenq-Haur Wang
• E-mail: mailbox at the domain of NTUT, and my username is jhwang
• Office: R1534, Technology Building (ext. 4238)
• Office Hours: Tue. & Fri. 9:10-12:00am
• Class Hours: Thu. 9:10am-12:00pm, R231, Technology Building
• TA: Lawrence (R1424, Technology Building)
  e-mail: (TBA)
• Prerequisite: data structures (and algorithms), database systems, discrete math (probability), linear algebra, basic data mining, and working experience of high-level programming languages
• Textbook: (Selected chapters)
  • Jure Leskovec, Anand Rajaraman, Jeffrey David Ullman, Mining of Massive Datasets, 3rd Edition, Cambridge University Press, Feb. 2020. ([MMDS3])
  • Jimmy Lin and Chris Dyer, Data-Intensive Text Processing with MapReduce, Morgan & Claypool Publishers, 2010. ([MR])
• References:
  • Jiawei Han, Jian Pei, and Hanghang Tong, Data Mining: Concepts and Techniques, 4th ed., Morgan Kaufmann Publishers, Oct. 2022. ([DM4])
  • Tom White, Hadoop: The Definitive Guide, 4th ed., O'Reilly Media, 2015.
  • Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia, Learning Spark: Lightning-Fast Big Data Analysis, O'Reilly Media, January 2015.
  • Others: Web documents, tutorials, and selected academic papers.

Latest News

• Sep. 1, 2023: The homepage for Big Data Mining and Applications (Fall 2023) has been setup.

(Tentative) Schedule

Programming Assignments and Projects

Program Submission Instructions

If you cannot successfully submit your work, please contact with the TA or the instructor.

Distributed Programming Environment

1. Computers:
   You can build your own distributed environment by commodity computers. Usually, Hadoop/Spark does not require very poweful capability of each machine.
2. VMs on the cloud:
   You can use existing cloud computing services from Google, Amazon, or Microsoft Azure to setup distributed environemnt. Most cloud services provide free quota for limited usage (CPU, memory, network bandwidth, storage).

Homework

In the case of very large datasets, you still need to analyze all data objects in the whole dataset. Please design your program accordingly to first partition the data into several batches, and then merge the final result.

1. HW#0 : Big data analysis platform setup & simple statistics in MapReduce (or Python in Jupyter Notebook)
   Due: Oct. 19, 2023
2. HW#1 : Analyzing documents in MapReduce (or Python in Jupyter Notebook)
   Due: Nov. 2, 2023
3. HW#2 : Statistics (counting, co-occurrence) of various data types in MapReduce (or Python in Jupyter Notebook)
   Due: Nov. 23, 2023
4. HW#3 : Finding similar documents (by calculating minhash signatures, LSH, and KNN search) in MapReduce (or Python in Jupyter Notebook)
   Due: Dec. 7, 2023
5. HW#4 : Recommender systems using collaborative filtering in MapReduce (or Python in Jupyter Notebook)
   Due: Jan. 4, 2024
6. HW#5 (Optional) : Analyzing web graphs in MapReduce (Connectivity, PageRank, ...)
   Due: Jan. 4, 2024

Projects

1. Term Project: paper presentation or system demonstration

   Item	Description	Time
   Proposal	You are required to submit a proposal for term project one week after midterm exam.	Nov. 23, 2023 (Tue.)
   Schedule	Due to our time limits, we might have to start the term project presentation as early as Dec. 28, 2023 (Thu.). * [NOTE] All presentations *must* be finished within the scheduled time slots, which will be the last *three* weeks in this semester. No other time slots will be avbailable.	Dec. 28, 2023, Jan. 4, 11, 2024
   Report	Each team is *required* to upload the final report after finishing your presentation. The final report should contain at least the following: presentation slides (for all teams), and source code, installation/execution instructions, team members and task responsibility (for system projects)	Jan. 12, 2024 (Fri.)

Exams

1. Midterm Exam: Nov. 6-10, 2023
   
   • Date: Nov. 16, 2023 (Thu.)
   • Time: 9:10am-12:00pm
   • Location: R231, Technology Building
   • Range: Ch. 1, 3, 7, 9, 11
2. Final Exam: Jan. 8-12, 2024
   
   • Note: There will be no final exam in this course. Instead, you are required to finish a term project for system development.

Scores