Problem Solving using PySpark – Regression & Classification
MP4 | Video: h264, 1280×720 | Audio: AAC, 44.1 KHz, 2 Ch
Genre: eLearning | Language: English | Duration: 35 Lectures ( 1h 48m ) | Size: 1.1 GB
This course is based on real world problems in PySpark, surrounding Data Cleaning, Descriptive statistics, Classification and Regression Modeling. The first segment introduces descriptive statistics in PySpark and computing fundamental measures such as mean, standard deviation and generating an extended statistical summary. The second segment is based on cleaning the data in PySpark, working with null values, redundant data and imputing the null values. The third segment is about Predictive modeling with PySpark using Gradient Boosted Trees Regression The fourth and fifth segments are based on applying classification techniques in PySpark. The fourth Segment introduces the application of Spark XGB Classifier for a classification problem and the fifth segment is about using a deep learning model for text sentiment classification. The sixth segment is about time series analytics and modeling using PySpark and Prophet The seventh segment introduces Spark SQL for data querying and analysis. These segments also include advanced visualization techniques through Seaborn and Plotly libraries including Box plots to understand the distribution of the data and assessment of outliers, Count plots to understand balance in the proportion of data, Bar chart to represent feature importance as part of the Gradient Boosted Trees Regression Model, Word Cloud for text analytics and analyzing time series data to extract seasonality and trend components. Each of these segments, has a Google Colab notebook included aligning with the lecture.