Spark with Scala Course - #16 Writing a DataFrame test case

#apachespark #scala #course #tutorial #livecode #code
Apache Spark with Scala Course: From Beginner to Pro
Spark Rockstar Academy: https://academy.philipp-brunenberg.de
Spark Rockstar Newsletter: https://newsletter.philipp-brunenberg.de
Entire playlist: https://www.youtube.com/playlist?listPLeEh_6coH9EpIwsiNiDCW0V-UjNZElF0d
Next video: https://youtu.be/59-5MeGl_Q4
In this video, we implement a full test case to assert that our DataFrame transformations actually produce a correct result. We'll discover how to create a DataFrame containing test data, pass it into our unit and asserting that the result is correct.
00:00 Intro & Outline
00:58 Instantiating a SparkSession
01:44 Creating a test DataFrame with schema and encoder
03:56 Defining the schema
06:14 Specifying test data
08:21 Creating a test DataFrame using the schema and encoder
10:01 Calling the unit under test
11:27 Asserting result correctness
13:33 Run configuration
14:21 Running the test
14:47 Fixing a failing test
15:55 Test description & conclusion