Complete PySpark Developer Course (Spark with Python)

Complete PySpark Developer Course (Spark with Python)

This is a complete PySpark Developer course for Data Engineers and Data Scientists and others who wants to process Big Data in an effective manner. We will cover below topics and more:

• Complete Curriculum for a successful PySpark Developer

• Set up Hadoop Single Node Cluster and Integrate it with Spark 2.x and Spark 3.x

• Complete Flow of Installation of Standalone PySpark (Unix and Windows Operating System)

• Detailed HDFS Commands and Architecture.

• Python Crash Course

• Introduction to Spark (Why Spark was Developed, Spark Features, Spark Components)

• Understand SparkSession

• Spark RDD Fundamentals

• How to Create RDDs

• RDD Operations (Transformations & Actions)

• Spark Cluster Architecture - Execution, YARN, JVM Processes, DAG Scheduler, Task Scheduler

• RDD Persistence

• Spark Shared Variables  - Broadcast

• Spark Shared Variables  - Accumulators)

• Spark SQL Architecture, Catalyst Optimizer, Volcano Iterator Model, Tungsten Execution Engine, Different Benchmarks

• Difference between Catalyst Optimizer and Volcano Iterator Model

• Spark Commonly Used Functions - Version, range, createDataFrame, sql, table, SparkContext, conf, read, udf, newSession, stop, catalog etc

• DataFrame Built-in functions - new column functions, encryption functions, string functions, regexp functions, date functions, null functions, collection functions, na functions, math and statistics functions, explode functions, flatten functions, formatting and json functions

• What is Partition,

• What is Repartition

• What is Coalesce

• Repartition Vs Coalesce

• Extraction - csv file, text file, Parquet File, orc file, json file, avro file, hive, jdbc

• DataFrame Fundamentals

• What is a DataFrame

• DataFrame Sources

• DataFrame Features

• DataFrame Organization

• DataFrame Rows,

• DataFrame Columns

• DataTypes. Practical examples.

• Perform ETL Using DataFrame

      -- Extraction APIs

      -- Transformation APIs

      -- Loading APIs

      -- Practical Examples.

• Optimization and Management - Join Strategies, Driver Conf, Parallelism Configurations, Executor Conf etc

  
  

Learn PySpark in depth with hundreds of Practical examples. Be a complete PySpark Developer. Set up a Hadoop Cluster.

Url: View Details

What you will learn

• Complete Curriculum for a successful PySpark Developer
• Hadoop Single Node Cluster Set up and Integrate with Spark 2.x and Spark 3.x
• Complete Flow of Installation of PySpark (Windows and Unix)

Rating: 4.26613

Level: All Levels

Duration: 30.5 hours

Instructor: Sibaram Kumar