Big Data Mining and Applications, Spring 2018

This course discusses issues in scalable mining of big data in various applications, which is gaining more popularity in recent years.
The major focus in this course will be utilizing parallel progamming in distributed platforms for efficient mining of different kinds of big data.
The course is offered at graduate-level, and will be taught in English.

[*** NOTE ***]: This course is NOT an introductory course for undergraduates. You are required to take the first course of "Introduction to Big Data Analytcis" or "Data Mining" in the undergraduate-level as as prerequisite.
It contains very heavy loads of parallel programming, which is a totally different concept from sequential programming.
Please think twice before taking this course!

Please read the notes to the course very carefully about the requirements for the programming homework and quiz.

Course Information

Instructor: Jenq-Haur Wang
E-mail: mailbox at the domain of csie within NTUT, and my username is jhwang
Office: R1534, Technology Building (ext. 4238)
Office Hours: Mon. & Wed. 4:10-5:00pm
Class Hours: Fri. 9:10-12:00am, R327, 6th Teaching Building
TA: Jack Lin (R1424, Technology Building)
e-mail: jack888835 AT GMail DOT COM
Prerequisite: data structures (and algorithms), database systems, discrete math (probability), linear algebra, basic data mining, and working experience of high-level programming languages
Textbook: (Selected chapters)
- Jure Leskovec, Anand Rajaraman, Jeffrey David Ullman, Mining of Massive Datasets, 2nd Edition, Cambridge University Press, November 2014. ([MMDS2])
- Jimmy Lin and Chris Dyer, Data-Intensive Text Processing with MapReduce, Morgan & Claypool Publishers, 2010. ([MR])
References:
- Jiawei Han, Micheline Kamber and Jian Pei, Data Mining: Concepts and Techniques, 3rd ed., Morgan Kaufmann Publishers, July 2011. ([DM3])
- Tom White, Hadoop: The Definitive Guide, 4th ed., O'Reilly Media, 2015.
- Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia, Learning Spark: Lightning-Fast Big Data Analysis, O'Reilly Media, January 2015.
- Others: Web documents, tutorials, and selected academic papers.

Latest News

Jul. 2, 2018: [NOTE] The total score has been posted. Please check if there's any questions before 5:00pm, Jul. 4, 2018 (Wed.).
Jun. 1, 2018: The schedule of term project presentation has been posted. Also, the score of HW#4 has been announced. Please check if there's any questions.
May 18, 2018: HW#5 has been announced. Also, the score for HW#3 has been posted.
May 4, 2018: The scores for HW#1, HW#2, and Quiz#1 have been posted.
May 2, 2018: Bonus Question Set #1 has been announced.
Apr. 27, 2018: HW#4 has been announced.
Apr. 19, 2018: [NOTE] The midterm exam has been extended to May 11, 2018.
Apr. 16, 2018: HW#3 has been announced.
Mar. 23, 2018: HW#2 has been announced.
Mar. 20, 2018: [NOTE] Due to the delay in VM application, the deadline of HW#1 has been extended for one week, to Mar. 30, 2018.
Mar. 18, 2018: The rules of applying virtual machines on the server of CSIE have been posted. Please check it before applying.
Mar. 14, 2018: [NOTE] Due to the lack of computer room on Friday morning of the scheduled quiz (Apr. 20), we have to re-schedule our time for quiz to Apr. 21 (Sat.). Please check the details of the time and location.
Mar. 9, 2018: HW#1 has been announced. The details about the dataset will be given later.
Mar. 2, 2018: The homepage for Big Data Mining and Applications (Spring 2018) has been setup.

(Tentative) Schedule

Week	Date	Content	Reading	Note
1	Mar. 2, 2018	Course Overview
2	Mar. 9, 2018	Introduction to distributed platforms & MapReduce: Hadoop, Spark	[MMDS2] Ch.1 & 2 [MR] Ch.1 & 2	HW#1
3	Mar. 16, 2018	MapReduce programming: pairs & stripes, relational algebra	[MMDS2] Ch. 2 [MR] Ch. 3
4	Mar. 23, 2018	MapReduce programming: matrix multiplication, relational joins	[MMDS2] Ch. 2 [MR] Ch. 3	HW#2
5	Mar. 30, 2018	Finding similar items	[MMDS2] Ch.3	3/30 Due: HW#1
6	Apr. 6, 2018	(Compensation Leave for Spring Break)
7	Apr. 13, 2018	TA: Homework Q&A, term project proposal, and team member registration (Leave for ACM SAC 2018)		HW#3 Term Project Proposal
8	Apr. 20, 2018	(4/21 (Sat.), 9:00am-12:00: Quiz -- Hadoop and Spark)		4/20 (extended) Due: HW#2
9	Apr. 27, 2018	Clustering	[MMDS2] Ch.7	HW#4
10	May 4, 2018	Dimension reduction Recommender systems	[MMDS2] Ch.11 [MMDS2] Ch.9	Due: HW#3
11	May 11, 2018	(Midterm Exam)
12	May 18, 2018	Link analysis - PageRank & HITS More about PageRank	[MMDS2] Ch.5	Due: Proposal Due: HW#4 HW#5
13	May 25, 2018	Mining social network graphs Community Detection in Graphs	[MMDS2] Ch.10
14	Jun. 1, 2018	Large-scale machine learning: SVM	[MMDS2] Ch.12
15	Jun. 8, 2018	Term Project Presentation (Week 1): 6 teams completed.		Due: HW#5
16	Jun. 15, 2018	Term Project Presentation (Week 2)
17	Jun. 22, 2018	Term Project Presentation (Week 3)
18	Jun. 29, 2018	Term Project Presentation (Week 4)

Programming Assignments and Projects

Please hand in your assignment before deadline according to the following instructions.

Program Submission Instructions

NOTE: Programs or projects in electronic files must be submitted directly to the TA online at Open Cyber Classrooms.
Please follow the instructions before your first login for this course.

Account: Use your student ID as the account and password at your first login. Please change the password *as soon as possible* for better security.
- Note: Even if you already have accounts for other courses, you are still *required* to do it for this course.
Filename: Compress your source code, output, documents, and related files into one compressed file. Please name it according to your ID and each homework. For example, [id]_HW1.zip.

If you cannot successfully submit your work, please contact with the TA or the instructor.

Distributed Programming Environment

You can use your own computers, or virtual machines on any cloud computing services to setup your distributed programming environment for this course.

Computers:
You can build your own distributed environment by commodity computers. Usually, Hadoop/Spark does not require very poweful capability of each machine.
VMs on the cloud:
You can use existing cloud computing services from Google, Amazon, or Microsoft Azure to setup distributed environemnt. Most cloud services provide free quota for limited usage (CPU, memory, network bandwidth, storage).
VMs on CSIE server: See the following section for more details.

VM Application on CSIE server

To service the undergraduate students who might not have fixed computers to complete the homework, students might apply for the virtual machines on CSIE server.
Please read the Rules for VM Application before applying this service.

Quota: 2 VMs per student
VM Specification:
- Hardware: 4 CPU, 2GB RAM, 40GB storage
- OS: ubuntu 16.04 64-bit
Contact: jack888835 AT GMail DOT Com

Important Notes

By applying this service, you agree to follow the rules of Taipei Tech and academic networks.
Since a single IP address is shared among VMs on the server, an upper limit on the network traffic of 10GB/day will be enforced by the computer center.
Due to the limited physical memory size (64GB) on CSIE server, at most 15 applications will be granted in a first-come-first-serve manner except that: Requests from undergraduate students without any lab equipment are given higher priority.

Homework

There will be about five programming exercises (and some written exercises) that target at different data analysis tasks.
Given large dataset, all homeworkas require parallel programming using either Hadoop or Spark, in distributed mode.
Since it's totallly different from sequential programming, it takes much more efforts and time to complete. Please think twice before taking this course! r

HW#1: Hadoop/Spark distributed mode setup & simple calculation in MapReduce
Details about Task and Data
Due: extended to Mar. 30, 2018
HW#2: Statistics of various data types in MapReduce (co-occurrence)
Due: extended (again) to Apr. 20, 2018
HW#3: Similarity estimation using MapReduce (for computing minhash signatures, LSH, and KNN search)
Due: extended to May 4, 2018
HW#4: Matrix multiplication using MapReduce (for dimension reduction using SVD, or CUR)
Due: May 18, 2018
Bonus Question Set#1: Statistics of crime data using MapReduce
HW#5: Analyzing web graphs in MapReduce (Connectivity, PageRank, ...)
Due: Jun. 8, 2018

Projects

Term Project: paper presentation or system demonstration

Item	Description	Time
Proposal	You are required to submit a proposal for term project one week after midterm exam.	extended to May 18, 2018 (Fri.)
Topics	For paper presentations, the paper quality will greatly affect your score in term project. Please carefully select good papers to read.
Schedule	Due to our time limits, we might have to start the term project presentation as early as Jun. 8, 2018 (Fri.). You can check the current schedule of term porject presentations. (As of Jun. 15, 2018, 38 teams in total) * [NOTE] All presentations must be finished within the scheduled time slots, which will be the last four weeks in this semester. No other time slots will be avbailable.	Jun. 8, 15, 22, 29, 2018
Report	Each team is required to upload the final report after finishing your presentation. The final report should contain at least the following: presentation slides (for all teams), and source code, installation/execution instructions, team members and task responsibility (for system projects)	Jun. 29, 2018 (Fri.)

Exams

Quiz: Apr. 21, 2018 (Sat.)
- Range: Hadoop & Spark Implementation
- Location: R312 & R313 (3F), Computer Center
- Timer: 9:00am-12:00
- Task & Data: to be given onsite
- You have to:
  - Start a small cluster in Hadoop or Spark (in cluster mode)
  - Implement specified tasks using parallel programs in Hadoop or spark, with Java, Scala, Python, or R
  - Submit the source code, and generated output
Midterm Exam: Apr. 23-27, 2018
- Date: May 11, 2018 (Fri.)
- Time: 9:10-12:00am
- Location: R327, 6th Teaching Building
- Range:Ch. 1-3, 7, 9, 11
Final Exam: Jun. 25-29, 2018
- Note: There will be no final exam in this course. Instead, you are required to finish a term project for system development or paper presentation.

Scores

Please check the homework submission site for more details.

You can check the total score. (including HW#1-HW#5, Quiz#1, Midterm exam, Bonus#1, and Term Project, as of Jul. 3, 2018)
Please check if there's any questions about the scores with the TA as soon as possible, before 5:00pm, Jul. 4, 2018.

E-mail: jhwang AT csie . ntut . edu . tw
Created: Mar. 2, 2018.
Last Updated: Jul. 3, 2018.