Apache Spark Getting Started Exam with
Complete Solutions
Which of the following statements is true? - The SparkSession encapsulates the
SparkContext, SQLContext, and HiveContext
Among the following types of RDD objects, which can be converted into a Spark
DataFrame? - 2-dimensional list
Row object
Which of these statements related to the transformation operations on RDDs are true? -
RDDs use metadata to track what series of transformations have been applied within it
Transformation operations are applied on RDDs only when you request for it
Let's say you have defined a transformation operation on a DataFrame called "data"
using the map function and you store this mapping on a variable called "tennis". When
is this mapping executed? - After calling the tennis.collect() command
Which of these functions is the Driver component in the Spark architecture responsible
for? - Sets up the SparkSession so that applications can interact with the environment
It is responsible for launching all the tasks that are executed by the worker nodes
How are Spark DataFrames and RDDs related? - DataFrames are higher level
abstractions built on top of RDDs
In the following piece of code, what command would you use to print all the elements of
the "simple_data" variable?
import pyspark from pyspark import SparkContext sc=SparkContext()
spark=SparkSession(sc) simple_data=sc.parallelize(1, "Djokovic", 31) -
simple_data.take(3)
simple_data.collect()
Consider the following lines of code. What command would you use to extract only the
"Rank" and "Player_name" column from the "tennis" DataFrame in the form of an RDD?
import pyspark from pyspark import SparkContext from pyspark import SQLContext
from pyspark import SparkSession sc=SparkContext() sqlContext=SQLContext(sc)
spark=SparkSession(sc) data= sc. parallelize ([ Row (1, "Djokovic", 31), Row (2,
"Nadal", 32), Row (3, "Federer", 37]) tennis=data.map(lambda r: Row('Rank',
'Player_name', 'age')(*r)) - tennis.rdd\.map(lambda x: (x.Rank,
x.Player_name))\.collect()
