Data Engineering and Big Data: A Comprehensive Overview, Summaries of Computer science

Download Data Engineering and Big Data: A Comprehensive Overview and more Summaries Computer science in PDF only on Docsity!

Data engineering

and big data

U N D E R S TA N D I N G D ATA E N G I N E E R I N G Hadrien Lacroix Content Developer at DataCamp

About the course

Conceptual course No coding involved Objectives Being able to exchange with data engineers Provide a solid foundation to learn more

Chapter 2

How data storage works

1. Structured vs unstructured data
2. SQL
3. Data warehouse and data lakes

Chapter 3

How to move and process data

1. Processing data
2. Scheduling data
3. Parallel computing
4. Cloud computing

Data workflow

Data workflow

Data workflow

Data engineers

A data engineer's responsibilities

Ingest data from different sources Optimize databases for analysis Remove corrupted data Develop, construct, test and maintain data architectures

Data engineers and big data

Big data becomes the norm =>

Big data growth

Sensors and devices Social media Enterprise data VoIP (voice communication, multimedia sessions) Data Age 2025, Seagate, November 2018 1

The five Vs

Volume (how much?) Variety (what kind?) Velocity (how frequent?) Veracity (how accurate?) Value (how useful?)

Let's practice!

Data engineers vs.

data scientists

U N D E R S TA N D I N G D ATA E N G I N E E R I N G Hadrien Lacroix Content Developer at DataCamp