CORSO: Big Data Architectures (B028463, B028462), Università degli studi di Firenze

Le  versioni aggiornate di questi contenuti sono accessibili su Moodle di UNIFI per gli anni 2021

B028462 - B028463 - BIG DATA ARCHITECTURES
Corso di Studio: B070 - INGEGNERIA INFORMATICA,
Università degli Studi di Firenze.

Contenuti:
--Overview del corso e del Lab--Big data architectures
--Cloud: virtualizzazione, HA, DRS, FT, architetture
--Container: Marathon, Mesos, Docker
--Big Data Analytics: prediction, early warning, anomaly detection
--IOT Architecture: AWS, Azure IOT, Google IOT, Snap4City IOT/IOE
--Tools: snap4city.org
--Big data computing, Performance analysis
--Big Data Analytics: prediction, early warning, anomaly detection
--Visual Analytics: Business intelligence, Dashboard
 

--Tools: Twitter vigilance (opzionale)
--Hadoop, MapReduce, Spark (opzionale)
--Batch and Data Stream processing: NIFI, Kafka, ETL, … (opzionale)
--Indexing and Search: SOLR, SOLR Sharded, Elastic Search (opzionale)
--Big Data Store: Hbase, Mongo (opzionale)

----- version 2023-2024 ---------

OVERVIEW

Cloud and container

Architecture

Data Analytics

TimeSeries and predictions, clustering, XAI

Mobility Data Analytics

Mobility Data Analytics Tensor Flow (Optional)

3D Digital Twin, Deck.GL

Object tracking Vision

NOSQL databases: Mongo, Hadoop, SOLR, ElasticSearch (Optional)

---------------------------------- old versions --------------------------------

CORSO 2020: Big Data Architectures, overview 

Snap4City: IOT/IOE smart city https://www.snap4city.org Trafair: CEF project con varie Citta  Mosaic: Mobility and transport model Smart City of Florence Metropolitan Area Km4City: http://www.km4city.org REPLICATE H2020, SCC1, EC flagship http://replicate-project.eu/ Sii-Mobility SCN MIUR: http://www.sii-mobility.org Feedback: reail and GDO Big Data analytics 5G with 3-Wind Open Fiber Estra Coll@bora Social Innovation, MIUR: http://www.disit.org/5479 RESOLUTE H2020, EC: http://www.resolute-eu.org TRACE-IT, RAISSS, TESYSRAIL, ETC. ETC.

Video 2020: Lex1 (all'inizio vi sono svariati minuti di prove da saltare),

 

 

Elements of cloud Computing Motivation and definitions Virtualization Cloud Computing High Availability vSphere Infrastructure Security on the Cloud Conversions among VM and physical machines vCenter, datacenters and cluster management Comparison among virtual computing solutions How to work with Virtual Machines IaaS solutions SaaS Solutions PaaS Solutions ICARO project
 
Video 2020: Lex5, Lex6, Lex7,  Lex8,  Lex9,  Lex10, Lex11, Lex12

 

 Corso BigDataArc: Hadoop, Hbase, MapReduce
BigDataArc: Hadoop, Hbase, MapReduce

Video 2020: Lex13, Lex14, Lex13-14,

 

 

 Corso BigDataArc: Hadoop vs performance, SOLR vs Elastic Search
BigDataArc: Hadoop, performance, SOLR vs Elastic Search


Video 2020:  Lex15a, Lex15b, Lex16, Lex 15b-16 

 

 

Corso BigDataArc: Big Data, NoSQL, MongoDB

Big Data, NoSQL, MongoDB


Video 2020  Lex17
Corso BigDataArc: DataLake vs Data Ware House
Corso BigDataArc: DataLake vs Data Ware House

Video 2020: Video1, Video2


Corso BigDataArc: IOT generico e Snap4City: IOT and IOE, Architetture, Dashboard, scaling, etc.

(Corso della Magistrale di Ingegneria dell Informazione) General architecture for Smart City KB vs Smart City API IOT Solutions IOT Architectures Comparison Snap4City IOT/IOE Solution and architecture IOT Device Registration IOT Devices vs Open Hardware / Open Software IOT APPLICATIONS, THE LOGIC AND THE SMARTNESS IOT APPLICATIONS VS IOT EDGE DEVICES End-to-end security on IOT infrastructures Managing Data Analytics tools: processes vs Containers Data Protection, Personal Data vs GDPR From Data to Applications and Dashboards Sentient City Control Room Dashboard Production Protocols vs Dashboards Dashboards vs mobile app User Interface Further Reading


Video 2020: video1, video2, video3, video4, video5, video6

 

Corso BigData Arc: data analytic

Data Analytics: Examples from Snap4City Smart parking: Predictions Smart Bike Sharing User Behavior Analysis, via Wi-Fi, OD, trajectories Recognition of Used Transportation means Traffic Flow Predictions, Traffic Flow Reconstruction, from Traffic Sensors Data Covid-19 vs other data: traffic and environmental Quality of Public Transport Service Origin Destination Matrices from: Wi-Fi, Mobile Apps, etc. Demand of Mobility vs Offer of Transportation Modal and Multimodal Routing for Navigation and Travel Planning Environmental Data Analysis and Predictions, early Warning Prediction of Air Quality Conditions Anomaly Detection What-IF Analysis Data Analytics: Enforcing and Exploiting Real Time Data Analytics: using R Studio Exploitation in IOT Applications Engaging City users Towards a Virtuous behavior Decision Support Systems, Smart DS and Resilience DS Twitter Vigilance: Social Media Analysis: Early Warning, Predictions

Video 2020: video1, video2, video3, video4

 

Tensor Flow for Traffic flow Reconstruction, partial differential equation

Video 2020: Video
 

Corso BigDataArc: Container Docker

Container Docker

Video 2020:  Video

 


  ----------------- versioni precedenti------------------

 

 

Corso BigDataArc: Visual Analytics for IOT Data Flow and traffic flow

Visual Analytics for IOT Data Flow and traffic flow. From NIFI, SOLR and Banana vs Elastic Search with Kibana.

 


PER IOT App esercitazioni vedi slide: https://www.snap4city.org/download/video/barc19/iot/
oppure per il PDF: https://www.snap4city.org/download/video/barc19/iot/iot.pdf

 

 

 

Corso BigDataArc: Big Data Analytics, examples

Big data analytic. Analisi Dati raccolti via Social Media Predicting presences Analisi Dati raccolti via Mobile App Tracce, matrici OD, heatmap Regency e frequency Impatto in uscita da Firenze Analisi Dati raccolti dai Flussi di traffico Ricostruzione del traffico in punti non misurati Analisi Dati raccolti dai Parcheggi Predizione dei posti liberi Analisi Dati raccolti via Firenze WiFi Tracce, matrici OD, heatmap Predicting presences Analisi Dati Dati raccolti via Cellular Valutazione comparativa TIM-VODA Valutazione comparativa FirenzeWiFi-Tim-Voda

 

 

Corso BigDataArc: 2018-19 elaborati e tesi

Corso BigDataArc: 2018-19 elaborati e tesi


si veda anche https://www.disit.org/5492

 

 


Durante il corso saranno effettuate esercitazioni su cluster hadoop, mapreduce, IOT application, containers, elastic search, twitter vigilance.
Questo stesso portale DISIT.ORG contiene molte slide e contenuti nel tema Big Data Architetture, Cloud, IOT, etc. fate delle ricerche. 

Modalità di verifica dell'apprendimento: un elaborato, small project
per elaborati, stage e Tesi si veda sopra le slide ma anche https://www.disit.org/5492

Si veda
https://www.disit.org
Https://www.km4city.org
https://www.disit.org/tv
https://www.snap4city.org

Docente: Paolo Nesi, mailto:paolo.nesi@unifi.it