Homepage

CORSO: Big Data Architectures (B028463, B028462), Università degli studi di Firenze

B028462 - B028463 - BIG DATA ARCHITECTURES
Corso di Studio: B070 - INGEGNERIA INFORMATICA,
Università degli Studi di Firenze.

Contenuti:
--Overview del corso e del Lab
--Big data architectures
--Cloud: virtualizzazione, HA, DRS, FT, architetture
--Container: Marathon, Mesos, Docker
--Hadoop, MapReduce, Spark
--Batch and Data Stream processing: NIFI, Kafka, ETL, …
--Indexing and Search: SOLR, SOLR Sharded, Elastic Search
--Big Data Store: Hbase, Mongo
--Big Data Analytics: prediction, early warning, anomaly detection
--Visual Analytics: Business intelligence, Dashboard
--IOT Architecture: AWS, Azure IOT, Google IOT, Snap4City IOT/IOE
--Tools: snap4city.org, Km4city.org, Twitter vigilance
--Big data computing, Performance analysis


newCORSO 2020: Big Data Architectures, overview 
Snap4City: IOT/IOE smart city https://www.snap4city.org Trafair: CEF project con varie Citta  Mosaic: Mobility and transport model Smart City of Florence Metropolitan Area Km4City: http://www.km4city.org REPLICATE H2020, SCC1, EC flagship http://replicate-project.eu/ Sii-Mobility SCN MIUR: http://www.sii-mobility.org Feedback: reail and GDO Big Data analytics 5G with 3-Wind Open Fiber Estra Coll@bora Social Innovation, MIUR: http://www.disit.org/5479 RESOLUTE H2020, EC: http://www.resolute-eu.org TRACE-IT, RAISSS, TESYSRAIL, ETC. ETC.

newVideo 2020: Lex1 (all'inizio vi sono svariati minuti di prove da saltare),
 

Corso BigDataArc: Virtualizzazione e cloud
Elements of cloud Computing Motivation and definitions Virtualization Cloud Computing High Availability vSphere Infrastructure Security on the Cloud Conversions among VM and physical machines vCenter, datacenters and cluster management Comparison among virtual computing solutions How to work with Virtual Machines IaaS solutions SaaS Solutions PaaS Solutions ICARO project

newVideo 2020: Lex5Lex6Lex7,  Lex8,  Lex9,  Lex10Lex11Lex12
new Corso BigDataArc: Hadoop, Hbase, MapReduce
BigDataArc: Hadoop, Hbase, MapReduce



newVideo 2020: Lex13Lex14Lex13-14,
  •  
new Corso BigDataArc: Hadoop vs performance, SOLR vs Elastic Search
BigDataArc: Hadoop, performance, SOLR vs Elastic Search



newVideo 2020:  Lex15aLex15bLex16Lex 15b-16 


newCorso BigDataArc: Big Data, NoSQL, MongoDB
Big Data, NoSQL, MongoDB



newVideo 2020  Lex17

 
Corso BigDataArc: DataLake vs Data Ware House
Corso BigDataArc: DataLake vs Data Ware House

newVideo 2020: Video1, Video2


newCorso BigDataArc: IOT generico e Snap4City: IOT and IOE, Architetture, Dashboard, scaling, etc.
(Corso della Magistrale di Ingegneria dell Informazione) General architecture for Smart City KB vs Smart City API IOT Solutions IOT Architectures Comparison Snap4City IOT/IOE Solution and architecture IOT Device Registration IOT Devices vs Open Hardware / Open Software IOT APPLICATIONS, THE LOGIC AND THE SMARTNESS IOT APPLICATIONS VS IOT EDGE DEVICES End-to-end security on IOT infrastructures Managing Data Analytics tools: processes vs Containers Data Protection, Personal Data vs GDPR From Data to Applications and Dashboards Sentient City Control Room Dashboard Production Protocols vs Dashboards Dashboards vs mobile app User Interface Further Reading


newVideo 2020: video1, video2, video3, video4, video5, video6
Corso BigData Arc: data analytic
Data Analytics: Examples from Snap4City Smart parking: Predictions Smart Bike Sharing User Behavior Analysis, via Wi-Fi, OD, trajectories Recognition of Used Transportation means Traffic Flow Predictions, Traffic Flow Reconstruction, from Traffic Sensors Data Covid-19 vs other data: traffic and environmental Quality of Public Transport Service Origin Destination Matrices from: Wi-Fi, Mobile Apps, etc. Demand of Mobility vs Offer of Transportation Modal and Multimodal Routing for Navigation and Travel Planning Environmental Data Analysis and Predictions, early Warning Prediction of Air Quality Conditions Anomaly Detection What-IF Analysis Data Analytics: Enforcing and Exploiting Real Time Data Analytics: using R Studio Exploitation in IOT Applications Engaging City users Towards a Virtuous behavior Decision Support Systems, Smart DS and Resilience DS Twitter Vigilance: Social Media Analysis: Early Warning, Predictions
newVideo 2020: video1, video2, video3, video4

 
Corso BigDataArc: Tensor Flow for Traffic flow Reconstruction
Tensor Flow for Traffic flow Reconstruction, partial differential equation

newVideo 2020: Video
 

Corso BigDataArc: Container Docker
Container Docker

newVideo 2020:  Video






  ----------------- versioni precedenti------------------

Corso BigDataArc: Visual Analytics for IOT Data Flow and traffic flow
Visual Analytics for IOT Data Flow and traffic flow. From NIFI, SOLR and Banana vs Elastic Search with Kibana.




PER IOT App esercitazioni vedi slide: https://www.snap4city.org/download/video/barc19/iot/
oppure per il PDF: https://www.snap4city.org/download/video/barc19/iot/iot.pdf

Corso BigDataArc: Big Data Analytics, examples
Big data analytic. Analisi Dati raccolti via Social Media Predicting presences Analisi Dati raccolti via Mobile App Tracce, matrici OD, heatmap Regency e frequency Impatto in uscita da Firenze Analisi Dati raccolti dai Flussi di traffico Ricostruzione del traffico in punti non misurati Analisi Dati raccolti dai Parcheggi Predizione dei posti liberi Analisi Dati raccolti via Firenze WiFi Tracce, matrici OD, heatmap Predicting presences Analisi Dati Dati raccolti via Cellular Valutazione comparativa TIM-VODA Valutazione comparativa FirenzeWiFi-Tim-Voda


Corso BigDataArc: 2018-19 elaborati e tesi
Corso BigDataArc: 2018-19 elaborati e tesi

si veda anche https://www.disit.org/5492




Durante il corso saranno effettuate esercitazioni su cluster hadoop, mapreduce, IOT application, containers, elastic search, twitter vigilance.
Questo stesso portale DISIT.ORG contiene molte slide e contenuti nel tema Big Data Architetture, Cloud, IOT, etc. fate delle ricerche. 


Modalità di verifica dell'apprendimento: un elaborato, small project
per elaborati, stage e Tesi si veda sopra le slide ma anche
https://www.disit.org/5492

Si veda
https://www.disit.org
Https://www.km4city.org
https://www.disit.org/tv
https://www.snap4city.org

Docente: Paolo Nesi, mailto:paolo.nesi@unifi.it
0
Il tuo voto: Nessuno


Warning: Table '.\drupal\sessions' is marked as crashed and should be repaired query: UPDATE sessions SET uid = 0, cache = 0, hostname = '192.168.0.44', session = 'hidePDA|s:1:\"0\";', timestamp = 1733888297 WHERE sid = 'a5b8bd763aa4841615852ff76b08a3f1' in C:\Programmi\Apache Software Foundation\Apache2.2\htdocs\drupal\includes\database.mysqli.inc on line 128