PinnedMember-onlyData Engineering — Construct a Standard Data Pipeline Design PatternOne common challenges encountered by developers and data professionals in the area of data processing and analysis is whether to import data directly into a database or store it as.csv files before loading. …Data Science6 min readData Science6 min read
Nov 18Member-onlyPySpark — Retrieve Unmatched rows from two DataframesPySpark — Retrieve Unmatched rows from two Dataframes Data integrity refers to the quality, consistency, and reliability of data throughout its life cycle. Data engineering pipelines are methods and structures that collect, transform, store, and analyse data from many sources. If you are working as a PySpark developer, data engineer, data analyst, or data scientist for any organisation…Data Science5 min readData Science5 min read
Nov 16Member-onlyData Warehouses don’t support Primary Key constraintData is the business asset for every organisation which is audited and protected. To gain in their business, it becomes very urgent for every organization to make sure that good quality data is available to everyone who needs it. …Data Science4 min readData Science4 min read
Nov 15Member-onlyPySpark — Retrieve matching rows from two DataframesData integrity refers to the quality, consistency, and reliability of data throughout its life cycle. Data engineering pipelines are methods and structures that collect, transform, store, and analyse data from many sources. If you are working as a PySpark developer, data engineer, data analyst, or data scientist for any organisation…Data Science4 min readData Science4 min read
Nov 14Member-onlyPySpark— Update multiple columns in a dataframeWorking as a PySpark developer, data engineer, data analyst, or data scientist for any organisation requires you to be familiar with dataframes because data manipulation is the act of transforming, cleansing, and organising raw data into a format that can be used for analysis and decision making. Note: We are…Data Science3 min readData Science3 min read
Nov 8Member-onlyData Engineering — Azure Databricks or Azure Synapse AnalyticsThe cloud is the fuel that powers today’s digital companies, with businesses paying solely for the specific services or resources that they consume over time. Azure Synapse Analytics bridges the gap between these two worlds by providing a uniform experience for ingesting, preparing, managing, and serving data for instant BI…Data Science6 min readData Science6 min read
Nov 3Member-onlyData Engineering — Scala vs Python for SparkIf you are a Data Engineer, you will most likely need to know python anyways. This really depends on what you want to do within data engineering and where you want to work. I agree that SQL and Python are the most important for starting out and give you access…Data Engineering5 min readData Engineering5 min read
Nov 1Member-onlyGCP— Cloud Run a fully managed compute platformCloud Run is undoubtedly the next generation of Google Cloud’s “serverless” remedies, and it is one step down in cloud abstractions, enabling you to tweak a bit more without having to worry about scaling (too much). Google Cloud Run is a fully managed compute platform that enables you to deploy…Data Science4 min readData Science4 min read
Oct 2Member-onlyData Engineering — Data Pipeline ScalingIf you work as a data engineer, data analytics, or data scientist in an organization that needs you on a project and are using a pretty standard ELT architecture to extract data from several sources into on-premise or cloud-based systems, this is a good fit. Data Curiosity: Data curiosity is…Data Science5 min readData Science5 min read
Aug 3Member-onlyDevelopment Methodology — Google Cloud Composer DAGsAirflow is a fantastic productivity program that seems to get better with each new release. It is compatible with a broad range of connectors/hooks and technologies, as well as workflow recovery/resumability and graph presentation for workflow units. Airflow is available as part of Google Cloud Services as Cloud Composer, a…Data Science5 min readData Science5 min read
