Instructors - Docenti:

• Dino Pedreschi
  • KDD Laboratory, Università di Pisa ed ISTI - CNR, Pisa
  • http://www-kdd.isti.cnr.it
  • dino [dot] pedreschi [at] unipi [dot] it

Teaching assistant - Assistente:

• Riccardo Guidotti
  • KDD Laboratory, Università di Pisa and ISTI - CNR, Pisa
  • guidotti [at] di [dot] unipi [dot] it

Instructors:

• Mirco Nanni, Dino Pedreschi
  • KDD Laboratory, Università di Pisa and ISTI - CNR, Pisa
  • http://www-kdd.isti.cnr.it
  • mirco [dot] nanni [at] isti [dot] cnr [dot] it
  • dino [dot] pedreschi [at] unipi [dot] it

Instructors:

• Dino Pedreschi, Anna Monreale
  • KDD Laboratory, Università di Pisa and ISTI - CNR, Pisa
  • http://www-kdd.isti.cnr.it
  • mirco [dot] nanni [at] isti [dot] cnr [dot] it
  • dino [dot] pedreschi [at] unipi [dot] it
  • anna [dot] monreale [at] unipi [dot] it

Teaching assistant - Assistente:

• Riccardo Guidotti
  • KDD Laboratory, Università di Pisa and ISTI - CNR, Pisa
  • guidotti [at] di [dot] unipi [dot] it

• Results of DM2 exam of 10.07.2019 are out: Results
• Results of DM2 exam of 19.06.2019 are out: Results
• Results of DM2 2nd mid-term exam are out: Results
• In preparation for the 2nd mid-term of DM2, you can find the exam of last year and its solutions: text and solutions.
• Due to a strike, the lesson of May 31th 2019 will not take place. In the course calendar you can find some additional material that might be useful to you, especially for the project.
  There are no other classes in program, therefore the course is now officially closed. Thanks to all my students for attending, and good luck for the exam.
• The project for DM2 is out!
• Results of DM2 mid-term exam are out: Results
• Last exam session on Feb, 14. Please register your name here: https://doodle.com/poll/6dgc5du4fgpnbyyx
• Results of the written exam of Feb dm_evaluation_1819_-_appello-feb.pdf
• Results of the written exam of January dm_evaluation_1819-jan-session.pdf
• Dates for exam registration: (a) Jan 21: slot 14 - 15, 16-17; (b) Jan 22: slot 10 - 11; ( c ) Jan 23: slot 09 - 10. Location: Monreale's office.
• I setup 3 days for the oral exam: 25, 28, 29 January. Other dates will we available after the written exam of Feb. For booking your oral exam please use the doodle indicating you Surname and Name: https://doodle.com/poll/3wunys9yd8s9q8ay
• Final results including project evaluation available here: dm_evaluation_1819.pdf. If you do not find your evaluation please write an email to Anna Monreale.
• New project is available!
• *Results of the Second mid-term test. After the evaluation of the project we will propose you the average grade considering: first and second midterm tests and project. *
• Get clusters from scipy dendogram: https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.fcluster.html#scipy.cluster.hierarchy.fcluster
• Help for installing Pyfim library https://anaconda.org/conda-forge/pyfim, https://pypi.org/project/fim/, http://www.borgelt.net/pyfim.html
• *Results of the first mid-term test.*
• Students need to decide the group composition for the project and fill this spreadsheet within October 1, 2018. It is strongly recommended an heterogenous composition with respect to the master degree. The number of members of each group can be 3 or 4.

… a new kind of professional has emerged, the data scientist, who combines the skills of software programmer, statistician and storyteller/artist to extract the nuggets of gold hidden under mountains of data. Hal Varian, Google’s chief economist, predicts that the job of statistician will become the “sexiest” around. Data, he explains, are widely available; what is scarce is the ability to extract wisdom from them.

Data, data everywhere. The Economist, Special Report on Big Data, Feb. 2010.

La grande disponibilità di dati provenienti da database relazionali, dal web o da altre sorgenti motiva lo studio di tecniche di analisi dei dati che permettano una migliore comprensione ed un più facile utilizzo dei risultati nei processi decisionali. L'obiettivo del corso è quello di fornire un'introduzione ai concetti di base del processo di estrazione di conoscenza, alle principali tecniche di data mining ed ai relativi algoritmi. Particolare enfasi è dedicata agli aspetti metodologici presentati mediante alcune classi di applicazioni paradigmatiche quali il Basket Market Analysis, la segmentazione di mercato, il rilevamento di frodi. Infine il corso introduce gli aspetti di privacy ed etici inerenti all’utilizzo di tecniche inferenza sui dati e dei quali l’analista deve essere a conoscenza. Il corso consiste delle seguenti parti:

1. i concetti di base del processo di estrazione della conoscenza: studio e preparazione dei dati, forme dei dati, misure e similarità dei dati;
2. le principali tecniche di datamining (regole associative, classificazione e clustering). Di queste tecniche si studieranno gli aspetti formali e implementativi;
3. alcuni casi di studio nell’ambito del marketing e del supporto alla gestione clienti, del rilevamento di frodi e di studi epidemiologici.
4. l’ultima parte del corso ha l’obiettivo di introdurre gli aspetti di privacy ed etici inerenti all’utilizzo di tecniche inferenza sui dati e dei quali l’analista deve essere a conoscenza

• Data, data everywhere. The Economist, Feb. 2010 download
• Data scientist: The hot new gig in tech, CNN & Fortune, Sept. 2011 link
• Welcome to the yotta world. The Economist, Sept. 2011 download
• Data Scientist: The Sexiest Job of the 21st Century. Harvard Business Review, Sept 2012 link
• Il futuro è già scritto in Big Data. Il SOle 24 Ore, Sept 2012 link
• Special issue of Crossroads - The ACM Magazine for Students - on Big Data Analytics download
• Peter Sondergaard, Gartner, Says Big Data Creates Big Jobs. Oct 22, 2012: YouTube video

• Towards Effective Decision-Making Through Data Visualization: Six World-Class Enterprises Show The Way. White paper at FusionCharts.com. download

Classes - Lezioni

Office hours - Ricevimento:

• Prof. Pedreschi: Lunedì/Monday h 14:00 - 16:00, Dipartimento di Informatica
• Prof. Monreale: by appointment, Room 374/DO, Dept. of Computer Science.
• Dr. Guidotti: class-appointment (see calendar)

Classes - Lezioni

Office hours - Ricevimento:

• Nanni : appointment by email, c/o ISTI-CNR

• Pang-Ning Tan, Michael Steinbach, Vipin Kumar. Introduction to Data Mining. Addison Wesley, ISBN 0-321-32136-7, 2006
  • http://www-users.cs.umn.edu/~kumar/dmbook/index.php
  • I capitoli 4, 6, 8 sono disponibili sul sito del publisher. – Chapters 4,6 and 8 are also available at the publisher's Web site.
• Berthold, M.R., Borgelt, C., Höppner, F., Klawonn, F. GUIDE TO INTELLIGENT DATA ANALYSIS. Springer Verlag, 1st Edition., 2010. ISBN 978-1-84882-259-7
• Laura Igual et al. Introduction to Data Science: A Python Approach to Concepts, Techniques and Applications. 1st ed. 2017 Edition.
• Jake VanderPlas. Python Data Science Handbook: Essential Tools for Working with Data. 1st Edition.

• The slides used in the course will be inserted in the calendar after each class. Most of them are part of the the slides provided by the textbook's authors Slides per "Introduction to Data Mining".

Le slide utilizzate durante il corso verranno inserite nel calendario al termine di ogni lezione. In buona parte esse sono tratte da quelle fornite dagli autori del libro di testo: Slides per "Introduction to Data Mining"

* Some text of past exams on DM1 (6CFU):

• 2017-1-19.pdf, 2017-9-6.pdf, 2016-05-30-dm1-seconda.pdf

* Some solutions of past exams containing exercises on KNN and Naive Bayes classifiers DM1 (9CFU):

• dm2_exam.2017.06.13_solutions.pdf, dm2_exam.2017.07.04_solutions.pdf, dm2_mid-term_exam.2017.06.06_solutions.pdf

* Some exercises (partially with solutions) on sequential patterns and time series can be found in the following texts of exams from the last years:

• dm2_exam.2015.04.13.results.pdf, dm2_exam.2016.04.4_sol.pdf, dm2_exam.2016.04.5_sol.pdf, dm2_exam.2016.06.20_sol.pdf, dm2_exam.2016.07.08_sol.pdf

• Some very old exercises (part of them with solutions) are available here, most of them in Italian, not all of them on topics covered in this year program:
  • Verifica 2006, Verifica 2005 (con soluzioni), Verifica 2004
  • Verifica 5 giugno 2007, Verifica 26 giugno 2007, Verifica 24 luglio 2007 (e Soluzioni)
  • Verifica 3 aprile 2008 (e Soluzioni), Verifica 18 luglio 2008 - parte 1, Verifica 18 luglio 2008 - parte 2
  • Exam with solution 2010-06-01 Exam with solution 2010-06-22 Exam with solution 2010-09-09 Exam with solution 2010-07-13

• KNIME The Konstanz Information Miner. Download page
• Python - Anaconda (2.7 version!!!): Anaconda is the leading open data science platform powered by Python. Download page (the following libraries are already included)
• Scikit-learn: python library with tools for data mining and data analysis Documentation page
• Pandas: pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. Documentation page
• WEKA Data Mining Software in JAVA. University of Waikato, New Zealand Download page

The exam is composed of three parts:

• A written exam, with exercises and questions about methods and algorithms presented during the classes. It can be substitute with the first and second mid-term tests of November and December.

• An oral exam (optional) , that includes: (1) discussing the project report with a group presentation; (2) discussing topics presented during the classes, including the theory of the parts already covered by the written exam. It is optional for students passing the written part by ONLY mid-term tests.

• A project consists in exercises that require the use of data mining tools for analysis of data. Exercises include: data understanding, clustering analysis, frequent pattern mining, and classification. The project has to be performed by min 3, max 4 people. It has to be performed by using Knime, Python or a combination of them. The results of the different tasks must reported in a unique paper. The total length of this paper must be max 20 pages of text including figures. The paper must emailed to datamining [dot] unipi [at] gmail [dot] com. Please, use “[DM 2018-2019] Project 2” in the subject. Students who will decide to perform the project during the summer exam sessions will find the dataset of the project online after 31/05/2019. In this case the project must be delivered at least 2 days before the oral exam.

Tasks of the project:

1. Data Understanding (Collective discussion on: 19/10/2018): Explore the dataset with the analytical tools studied and write a concise “data understanding” report describing data semantics, assessing data quality, the distribution of the variables and the pairwise correlations. (see Guidelines for details)
2. Clustering analysis (Collective discussion on: 21/11/2018): Explore the dataset using various clustering techniques. Carefully describe your's decisions for each algorithm and which are the advantages provided by the different approaches. (see Guidelines for details)
3. Classification (Collective discussion on: 12/12/2018): Explore the dataset using classification trees and random forest. Use them to predict the target variable. (see Guidelines for details)
4. Association Rules (Collective discussion on: 12/12/2018): Explore the dataset using frequent pattern mining and association rules extraction. Then use them to predict a variable either for replacing missing values or to predict target variable. (see Guidelines for details)

• Project 1
  1. Dataset: Credit Card Default
  2. Assigned: 01/10/2018
  3. Deadline: 05/01/2019, 09/01/2019
  4. Link: https://www.kaggle.com/t/5d7277746f8d45d6a10686506f602a9b

• Project 2
  1. Dataset: Telco Customer Churn
  2. Assigned for the summer session.
  3. Deadline: 2 days before the oral exam.
  4. Link: https://www.kaggle.com/blastchar/telco-customer-churn

Guidelines for the project are here.

The exam is composed of three parts:

• A written exam, with exercises and questions about methods and algorithms presented during the classes. It can be substitute with the first and second mid-term tests of April and June.

• A small online test for the data ethics part. The test can be taken at the following link: Link to "First Aid for Data Scientist" web site (pwd: datamining_2018). Register, and enroll to the “First Aid for Data Scientist” course. Take the quizzes of the 3 units. Then, download your certificate and send it to mirco [dot] nanni [at] isti [dot] cnr [dot] it before the oral exam.

• An oral exam, that includes: (1) discussing the project report with a group presentation; (2) discussing topics presented during the classes, including the theory of the parts already covered by the written exam.

• A project, that consists in exercises that require the use of data mining tools for analysis of data. Exercises include: sequential patterns, time series, classification (alternative methods and validation), outlier detection. The project has to be performed by max 3 people. It has to be performed by using Knime, Python, other software or a combination of them. The results of the different tasks must reported in a unique paper. The total length of this paper must be max 20 pages of text including figures. The project must be delivered at least 2 days before the oral exam.
  • Dataset: the data is a time series dataset on air quality, which can be downloaded here: Dataset.
  • Task 1: Time series: Consider only attribute “PT08.S1(CO)” and split the corresponding time series into daily series, deleting those with too many missing values (value = -200) and fixing the others in some way. Make also sure that all time series have 24 values. Compute clustering (with an algorithm of your choice) based on DTW and Euclidean distances and compare the results.
  • Task 2: Sequential patterns: discover contiguous sequential patterns of at least length 4. Before that, time series should be discretized in some way.
  • Task 3:Classification methods: define a target variable “WE” for the time series data set to “true” for weekend days, and “false” for the others. Test the K-NN classification method using DTW as distance measure, and at least another classification method using the 24 values as separate variables.
  • Task 4: Outlier detection: from the original dataset (i.e. the raw records with all attributes, not the time series built only on the “PT08.S1(CO)” attribute), identify the top 1% outliers. Adopt at least two different methods belonging to different families (i.e. model-based, distance-based, density-based, angle-based, …) to identify the 1% of input records with the highest likelihood of being outliers, and compare the results. Before doing the analysis, the records containing missing values should be deleted to avoid trivial results.

• Data Mining A.A. 2017/18
• Data Mining A.A. 2016/17
• Data Mining A.A. 2015/16
• Data Mining A.A. 2014/15
• Data Mining A.A. 2013/14
• Data Mining A.A. 2012/13
• Data Mining A.A. 2011/12
• Data Mining A.A. 2010/11
• Data Mining A.A. 2009/10
• Data Mining A.A. 2008/09
• Data Mining A.A. 2007/08
• Data Mining A.A. 2006/07
• PhD Workshop 2011
• Analisi delle Reti Sociali
• Social and Information Networks
• ICT for BI & CRM - Part III: Data Mining
• Data Mining for Customer Relationship Management 2015
• Data Mining for Customer Relationship Management 2016
• Data Mining for Customer Relationship Management 2017
• Data Mining for Customer Relationship Management 2018
• Data Mining and Machine Learning -- Master MAINS 2019
• Instructions for camera ready and copyright transfer
• Storie dell'Altro Mondo