Scala — Retrieve matched rows from two Dataframes

Ryan Arjun
3 min readFeb 25, 2024

In this tutorial, you will learn “How to Retrieve matched rows from two Dataframes by using Scala” in Databricks.

Data integrity refers to the quality, consistency, and reliability of data throughout its life cycle. Data engineering pipelines are methods and structures that collect, transform, store, and analyze data from many sources.

Scala is a computer language that combines the object-oriented and functional programming paradigms. Martin Odersky invented it, and it was initially made available in 2003. “Scala” is an abbreviation for “scalable language,” signifying the language’s capacity to grow from simple scripts to complex systems.

Scala is a language designed to be productive, expressive, and compact that can be used for a variety of tasks, from large-scale corporate applications to scripting. It has become more well-liked in sectors like banking, where its robust type system and expressive syntax are very helpful.

If you want to retrieve matched rows from two DataFrames based on two or more columns, you can still use the join method in Spark DataFrame API but you’ll need to specify multiple columns in the join condition.

💎 Import necessary Spark classes for DataFrame operations.

//import libraries

import org.apache.spark.sql.{SparkSession, Row}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

💎 Create a SparkSession.

// Create Spark Session 

val spark= SparkSession.builder().appName(“RetrieveMatchedRows”).getOrCreate()

💎Create two sample DataFrames, df1 and df2, with some common columns called “EmpId” from sample CSV files.

// File1 — Employee Info 
val FileEmpInfo=”dbfs:/FileStore/EmployeeInfo.csv”

// File2 — Employee Distribution
val FileEmpDist=”dbfs:/FileStore/EmployeeDistribution-1.csv”

// Read data into dataframe 1 from File1
val df1=spark.read.option(“header”,”true”).csv(FileEmpInfo)

// show the data from df1
df1.show()
// Read data into dataframe 2 from File2
val…

--

--

Ryan Arjun

BI Specialist || Azure || AWS || GCP — SQL|Python|PySpark — Talend, Alteryx, SSIS — PowerBI, Tableau, SSRS