Course · Training · Workshop
Apache Spark
Two intensive days on Apache Spark with Scala and Python: distributed in-memory computing for data engineering, data science, machine learning and AI pipelines.
Introduction to Apache Spark, currently the most powerful open source platform to run data engineering, data science, machine learning and AI pipelines. Two days of intensive dive into Spark core concepts, features and best practices. Using Scala and Python, we will ensure you build up your skills to start your journey with Spark.
Trainers
What participants say
These customers booked courses in the same topic cluster.More customers →
Content
Apache Spark is a multi-language engine for executing data engineering, data science, and machine learning tasks. Thousands of companies, including 80% of the Fortune 500, use Apache Spark™. Spark can process data in batches and streams, it can run on a single-node or on clusters. The Spark open source project has over 2,000 contributors from industry and academia.
We will cover all the fundamentals so that you are ready to run your next big data processing pipelines.
Amongst many topics, we will cover Spark architecture, using Scala, Python, R and SQL, Data Sources, DataFrames, Datasets, Joins, Aggregations, Spark Types, SparkUI, MLlib, etc.
The actual course content may differ from the above depending on the trainer, delivery, duration and the composition of participants.
Request this course in-house
Request a public date
No suitable public date? Register without obligation — once there is enough interest we schedule a new public date and let you know first.
More about Apache Spark
Apache Spark is a distributed computing system for big data processing that combines in-memory computing and DAG-based execution. It provides high-level APIs in Scala, Java, Python, and R, along with an optimized engine that automatically handles distributed data processing and fault tolerance.Further resources:
History
Spark was developed in 2009 by Matei Zaharia at UC Berkeley's AMPLab. The project emerged from the realization that MapReduce was inefficient for iterative algorithms and interactive data analysis. The first version was released as an open-source project in 2010.
Development was significantly influenced by the founding of Databricks by Zaharia, Ali Ghodsi, Ion Stoica, and other AMPLab members. In 2013, Spark was donated to the Apache Software Foundation. The introduction of DataFrames in Spark 1.3 and the structured streaming API in Spark 2.0 were major milestones.
Today, Spark is the largest open-source project in big data processing and is used by companies like Netflix, Yahoo, and Alibaba. It has fundamentally changed how companies process large amounts of data and set new standards for distributed data processing. Integration with cloud platforms and support for deep learning have further strengthened Spark's position as a universal engine for big data analytics.