Data Engineering: Processing, Scheduling, Parallel Computing, and Cloud Computing, Summaries of Computer science

Download Data Engineering: Processing, Scheduling, Parallel Computing, and Cloud Computing and more Summaries Computer science in PDF only on Docsity!

Processing data

U N D E R S TA N D I N G D ATA E N G I N E E R I N G Hadrien Lacroix Content Developer at DataCamp

Data processing value

Conceptually Remove unwanted data Optimize memory, process and network costs Convert data from one type to another At Spotflix No long term need for testing feature data Can't afford to store and stream files this big

Data processing value

Conceptually Remove unwanted data To save memory Convert data from one type to another Organize data To fit into a schema/structure Increase productivity At Spotflix No need for lossless format Can't afford to store files this big Convert songs from .flac to .ogg Reorganize data from the data lake to data warehouses Employee table example Enable data scientists

The difference between batch and stream will be explained in the next lesson! 1

Let's practice!

Scheduling data

U N D E R S TA N D I N G D ATA E N G I N E E R I N G Hadrien Lacroix Content Developer at DataCamp

Manual, time and sensor scheduling

Manually Manually update the employee table