WARNING: All lectures of the First Semester of the academic year 2020/21, until 31/12/2020, will be provided exclusively remotely, through the Teams team named “599AA 20/21 - BIG DATA ANALYTICS [WDS-LM]” (https://bit.ly/35yJ65c).

ATTENZIONE: Tutte le lezioni frontali del Primo Semestre dell’a.a. 2020/21, fino al 31/12/2020, verranno erogate esclusivamente in modalità a distanza, attraverso il canale Teams “599AA 20/21 - BIG DATA ANALYTICS [WDS-LM]” (https://bit.ly/35yJ65c).

Instructors - Docenti:

• Luca Pappalardo, Fosca Giannotti
  • KDD Laboratory, Università di Pisa and ISTI-CNR, Pisa
  • http://www-kdd.isti.cnr.it
  • luca [dot] pappalardo [at] isti [dot] cnr [dot] it
  • fosca [dot] giannotti [at] isti [dot] cnr [dot] it

Timetable (http://bit.ly/unipi_timetable_2020)

• Monday 16:15 - 18:00 Aula WDS/1
• Tuesday 16:15 - 18:00 Aula WDS/1

Team Registration: build up teams of 3 or 4 students and register your team here, by September 27th: https://forms.gle/rbsV4dF6RuAnCBWz9

For students without a team: send an email to Luca Pappalardo to notify that you are without a team by September 30th.

Only for the registered teams, express your preference for the datasets by September 30th https://forms.gle/HVheaScCgQJw4o616

Dataset assignment: at thie following link, each team can find the dataset assigned for the project –> https://bit.ly/33eTfC9

Instructions for mid term 1: The first mid term presentation (data understanding and project proposal) will be on October 19th (BigProblem, Global, MMG, I TeamIDI) and October 20th (Bei Dati Acrobatici, Malucs, AMS Group).

• presentation: prepare a presentation describing the data understanding and a proposal of the problem you want to solve. Motivate your decisions and choices (e.g., which variables you delete, how you deal with missing values and noise, the new variables you created, if you integrated your data with external datasets, etc.). The presentation should last 20 minutes (+ 10 minutes questions). The presentation must be sent through the google form (see below) in pdf format;
• report: the report must be done in latex, using this template: latex_template_bda.zip. It must be a maximum of 5 pages long. Summarize the data understanding and describe and motivate your project proposal. A zipped folder (.zip file) containing the .tex file, the .cls file, the .pdf file, and the files of all figures must be sent through the google form. In the report, put the name of the title of your project and the names of the members of your team.
• code: the python code in .ipynb format (Jupyter Notebook) or .py format used to generate the computations and the plots must be sent through the google form. Please document adequately your notebooks using the markdown language.
• Google form: upload the material by October 18th using this form: https://forms.gle/h2SAKFmkdXv4itiU6
• name the files using the format midterm1_teamname_type, where teamname is the name of the team (do not use spaces, use lowercase only), type is the type of the file (i.e., presentation, report, or code). Examples: midterm1_iteamidi_presentation.pdf, midterm1_beidatiacrobatici_report.zip, midterm1_amsgroup_code.ipynb

Instructions for mid term 2: The second mid term presentation (model(s) implementation and evaluation) will be on November 16th (BigProblem, Global, MMG, I TeamIDI) and November 17th (Bei Dati Acrobatici, Malucs, AMS Group).

• presentation: prepare a presentation describing the models you tried (e.g., Decision Trees, SVMs, etc.), the baselines used as a comparison (e.g., DummyClassifiers), how you perform the hyper-parameter tuning, the evaluation technique used (i.e., holdout, repeated holdout, cross validation), and the metrics chosen to evaluate the performance of the models. Motivate your decisions and choices (e.g., which evaluation metrics you chose, how you deal with the unbalancing of the dataset). The presentation should last 20 minutes (+ 10 minutes questions). The presentation must be sent through the google form (see below) in pdf format;
• report: the report must be done in latex, using the same template as the first mid term: latex_template_bda.zip. It must be a maximum of 10 pages long, including the part of the report regarding mid term 1. Summarize the model construction and evaluation and motivate your choices. A zipped folder (.zip file) containing the .tex file, the .cls file, the .pdf file, and the files of all figures must be sent through the google form. In the report, put the name of the title of your project and the names of the members of your team.
• code: the python code in .ipynb format (Jupyter Notebook) or .py format used to generate the computations and the plots must be sent through the google form. Please document adequately your notebooks using the markdown language.
• Google form: upload the material by November 15th using this form: https://forms.gle/kr3uq2PyyMqraRn78
• name the files using the format midterm2_teamname_type, where teamname is the name of the team (do not use spaces, use lowercase only), type is the type of the file (i.e., presentation, report, or code). Examples: midterm2_iteamidi_presentation.pdf, midterm2_beidatiacrobatici_report.zip, midterm2_amsgroup_code.ipynb

Paper presentation:

• each student will present, during a talk of most 7 minutes, a paper on Big Data Analytics. The presentations of the papers are scheduled on November 23rd and 24th. The presentation should last 7 minutes (+ 3 minutes questions).
• Express your preference for 5 papers here: https://forms.gle/B9rCmpJ8jnQ4vzWN8. We'll take into account your preference as much as possible. Fill the form by Nov 3rd. I'll assign you the paper within Nov 5th.
• During the presentation (with slides) you should highlight the following aspects: the data set used, the feature engineering and/or selection (if any), the problem addressed, the models/algorithms used to solve the problem, and finally the explanations of the model constructed (if any).
• The paper assigned to each student, and the date of presentation, are here: https://bit.ly/2I10Uw2

Instructions for mid term 3: The third mid term presentation (model(s) interpretation and explanation) will be on December 7th (BigProblem, Global, MMG, I TeamIDI) and December 8th (Bei Dati Acrobatici, Malucs, AMS Group).

• presentation: prepare a presentation in which you show how to interpret the model(s) you created and how to explain the reasoning the model.Example: if you use a decision tree (or similar) you can show the feature importance and show its structure to describe the rules it is composed of; if you use a (logistic or linear) regressor you can show the value of the computed weights; if you use a geometric model you can perform a dimensionality reduction and try to interpret it in two or three dimensions; if you use a non-interpretable model (e.g., a neural network) you can use explanability models (e.g., Shap, LIME). Provide also examples of how to interpret the model on specific records that are correctly classified and records that are incorrectly classified by your model. The presentation should last 20 minutes (+ 10 minutes questions). The presentation must be sent through the google form (see below) in pdf format;
• report: the report must be done in latex, using the template as the other mid terms. It must be a maximum of 15 pages long. Extend/modify the previous report. A zipped folder (.zip file) containing the .tex file, the .cls file and the files of all figures/plots must be sent through the google form. In the report, put the name of the title of your project and the names of the members of your team.
• code: the python code in .ipynb format (Jupyter Notebook) or .py format used to generate the computations and the plots must be sent through the google form. Please document adequately your notebooks using the markdown language.
• Google form: upload the material by December 6th using this form: https://forms.gle/65rRgYiNwz9ofdkn8
• name the files using the format midterm3_teamname_type, where teamname is the name of the team (do not use spaces, use lowercase only), type is the type of the file (i.e., presentation, report, or code). Examples: midterm3_iteamidi_presentation.pdf, midterm3_beidatiacrobatici_report.zip, midterm3_amsgroup_code.ipynb

Examples of projects from past years:

• Credit Risk Prediction, final report: credit_risk_prediction_heloc_case.pdf
• Ted Talks, final report: dataworms_project_ted_talks_report.pdf

Instructions for exam:

• remember to register in time to the first appello (Jan 14th) or the second appello (Feb 4th);
• by the day before your appello (i.e., Jan 13th or Feb 3rd), upload the python code that will be used during the exam (in .ipynb format or .py format). Name the file as exam_teamname_code, where teamname is the name of your team;
• during the exam, we'll provide you with a .txt file (using the same separator as the one in your dataset) with some records (with the corresponding labels) extracted from the dataset assigned to your team. You must run the code to generate predictions and explanations using your model (this means that the model should be already trained). Your code must also show which predictions are correct and which are not.
• If you made some improvements or changes on your project, describe them in the report and upload it. Name the file exam_teamname_report, where teamname is the name of your team.
• To upload the material you can use the following form: https://forms.gle/Ty1WfGQCN9c6esqJ7
• There will two exam sessions on Feb 14th: 10am - 12pm and 4pm - 6pm. Groups can connect to the session they prefer. We'll use the same Teams channel used for the lessons.

In our digital society, every human activity is mediated by information technologies, hence leaving digital traces behind. These massive traces are stored in some, public or private, repository: phone call records, movement trajectories, soccer-logs and social media records are all examples of “Big Data”, a novel and powerful “social microscope” to understand the complexity of our societies. The analysis of big data sources is a complex task, involving the knowledge of several technological and methodological tools. This course has three objectives:

• introducing to the emergent field of big data analytics and social mining;
• introducing to the technological scenario of big data, like programming tools to analyze big data, query NoSQL databases, and perform predictive modeling;
• guide students to the development of a open-source and reproducible big data analytics project, based on the analyis of real-world datasets.

In this module, analytical methods and processes are presented thought exemplary cases studies in challenging domains, organized according to the following topics:

• The Big Data Scenario and the new questions to be answered
• Sport Analytics:
  1. Soccer data landscape and injury prediction
  2. Analysis and evolution of sports performance
• Mobility Analytics
  1. Mobility data landscape and mobility data mining methods
  2. Understanding Human Mobility with vehicular sensors (GPS)
  3. Mobility Analytics: Novel Demography with mobile-phone data
• Social Media Mining
  1. The social media data landscape: Facebook, Linked-in, Twitter, Last_FM
  2. Sentiment analysis. example from human migration studies
  3. Discussion on ethical issues of Big Data Analytics
• Well-being&Now-casting
  1. Nowcasting influenza with retail market data
  2. Predicting well-being from human mobility patterns
• Paper presentations by students

This module will provide to the students the technologies to collect, manipulate and process big data. In particular the following tools will be presented:

• Python for Data Science
• The Jupyter Notebook: developing open-source and reproducible data science
• MongoDB: fast querying and aggregation in NoSQL databases
• GeoPandas: analyze geo-spatial data with Python
• Scikit-learn: machine learning in Python
• Keras: deep learning in Python

During the course, teams of students will be guided in the development of a big data analytics project. The projects will be based on real-world datasets covering several thematic areas. Discussions and presentation in class, at different stages of the project execution, will be performed.

• 1st Mid Term: Data Understanding and Project Formulation
• 2nd Mid Term: Model(s) construction and evaluation
• 3rd Mid Term: Model interpretation/explanation
• Exam: Final Project results

14/09 (Mod. 1) Introduction to the course, The Big Data scenario lesson1_introduction_to_the_course_bda2021.pdf

15/09 (Mod. 2) Python for Data Science and the Jupyter Notebook: developing open-source and reproducible data science

• How to install Jupyter notebook: https://jupyter.readthedocs.io/en/latest/install.html
• Python notebooks: http://bit.ly/bda2021_notebooks_1

21/09 No Lesson (Election Day in Italy)

22/09 (Mod. 3) Presentation of datasets for projects bda20_21_datasets_1_.pdf

28/09 (Mod. 2) Scikit-learn: programming tools for data mining (part 1): http://bit.ly/bda_notebooks_2

29/09

• (Mod. 2) Scikit-learn: programming tools for data mining (part 2): http://bit.ly/bda_notebooks_2
• (Mod. 1) Reproducing and Explaining Human Evaluations of Soccer Performance with Artificial Intelligence evaluting_soccer_performance_1_.pdf

05/10 No Lesson (SocInfo2020 conference)

06/10 No Lesson (SocInfo2020 conference)

12/10 (Mod. 2) Geopandas and scikit-mobility: managing geographic data in Python (part 1) bda2021_geopandas.zip

13/10 (Mod. 2) Geopandas and scikit-mobility: managing geographic data in Python (part 2) https://github.com/scikit-mobility/tutorials/tree/master/mda_masterbd2020

19/10 (Mod. 3) 1st Mid Term - first group of teams

20/10 (Mod. 3) 1st Mid Term - second group of teams

26/10 (Mod. 3) Discussion and group working on projects

27/10 (Mod. 3) Discussion and group working on projects

02/11 (Mod. 1) Nowcasting well-being with big data bda_wellbeing.pdf

03/11 (Mod. 1) Injury prediction in sports with AI bda_2020_injury_forecasting.pdf

09/11 (Mod. 3) Discussion and group working on projects

10/11 (Mod. 1) Trustworthy data mining and Explainable AI parti1.explainableai-10.11.2020.pdf

16/11 (Mod. 3) 2nd Mid Term - first group of teams

17/11 (Mod. 3) 2nd Mid Term - second group of teams

23/11 (Mod. 3) Discussion and group working on projects

24/11 - No Lesson

30/11 (Mod. 3) Paper presentation

01/12 (Mod. 3) Paper presentation

07/12 (Mod. 3) 3rd Mid Term - first group of teams

08/12 (Mod. 3) 3rd Mid Term - second group of teams

• January 14th, 2021
• February 4th, 2021

Big Data Analytics A.A. 2019/20

Big Data Analytics A.A. 2018/19

Big Data Analytics A.A. 2017/18

Big Data Analytics A.A. 2016/17

Big Data Analytics A.A. 2015/16