Below is a detailed response to your questions about Unity Catalog in Databricks, organized by the sections you provided. Each answer includes explanations, examples, and practical insights where applicable, aiming to provide a comprehensive understanding suitable for both foundational and advanced scenarios.---### Basic Understanding#### 1. What is Unity Catalog in Databricks?Unity Catalog is a unified... Continue Reading →
An azure pipeline usually run for 2 hrs but currently it is running for 10 hours. Find the bottleneck in pipeline.
To identify the bottleneck in an Azure Pipeline thatโs running for 10 hours instead of the usual 2 hours, you need to systematically analyze the pipelineโs execution. Hereโs a step-by-step approach to pinpoint the issue:### 1. **Check Pipeline Logs and Execution Details** - **Action**: Navigate to the Azure DevOps portal, open the pipeline run, and... Continue Reading →
Azure Data Engineer Journey Learning links
Start your Azure journey here.....1. Azure Data Factory.https://lnkd.in/gEmpbyrMProject: https://lnkd.in/gFG2aCgy2. Azure Data bricks.https://lnkd.in/gvFwKxaNproject: https://lnkd.in/gFG2aCgy3. Azure Stream Analytics.https://lnkd.in/g35VbSTv4. Azure Synapse Analytics.https://lnkd.in/gCufskNC5. Azure Data Lake Storage.https://lnkd.in/gcEKjWsc6. Azure SQL database.https://lnkd.in/gmHxqxQX7. Azure Postgres SQL database.https://lnkd.in/grHWJvWZ8. Azure MariaDB.https://lnkd.in/gYSp7MZi9. Azure Cosmos DB.https://lnkd.in/g6jPZA36This is an excellent guide to become azure data engineer. No need to become expert. but learn how to work with... Continue Reading →
List of All azure / data / devops /ML Interview Q& A
1. ๐๐๐๐ฟ๐ฒ ๐๐ฎ๐๐ฎ ๐๐ฎ๐ฐ๐๐ผ๐ฟ๐ ๐๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐ ๐ค&๐https://lnkd.in/dVzCmzcZ2. ๐๐๐๐ฟ๐ฒ ๐๐ฎ๐๐ฎ๐ฏ๐ฟ๐ถ๐ฐ๐ธ๐ ๐ฆ๐ฐ๐ฒ๐ป๐ฎ๐ฟ๐ถ๐ผ ๐ฏ๐ฎ๐๐ฒ๐ฑ ๐๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐ ๐ค&๐https://lnkd.in/dUCf8qf8๐ฏ. ๐ฅ๐ฒ๐ฎ๐น๐๐ถ๐บ๐ฒ ๐๐๐๐ฟ๐ฒ ๐๐ฎ๐๐ฎ ๐๐ฎ๐ฐ๐๐ผ๐ฟ๐ ๐๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐ ๐ค&๐https://lnkd.in/ex_Vixh๐ฐ.๐๐ฎ๐๐ฒ๐๐ ๐๐๐๐ฟ๐ฒ ๐๐ฒ๐๐ข๐ฝ๐ ๐๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐ ๐ค&๐https://lnkd.in/g7PdATm๐ฑ. ๐๐๐๐ฟ๐ฒ ๐๐ฐ๐๐ถ๐๐ฒ ๐๐ถ๐ฟ๐ฒ๐ฐ๐๐ผ๐ฟ๐ ๐๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐ ๐ค&๐https://lnkd.in/dtWYXTKN๐ฒ. ๐๐๐๐ฟ๐ฒ ๐๐ฎ๐๐ฎ ๐๐ฎ๐ธ๐ฒ ๐๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐ ๐ค&๐https://lnkd.in/dgr-uGQB๐ณ. ๐๐๐๐ฟ๐ฒ ๐๐ฝ๐ฝ ๐ฆ๐ฒ๐ฟ๐๐ถ๐ฐ๐ฒ ๐๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐ ๐ค&๐https://lnkd.in/dP4Afqkb๐ด. ๐๐๐๐ฟ๐ฒ ๐๐ฎ๐๐ฎ ๐๐ป๐ด๐ถ๐ป๐ฒ๐ฒ๐ฟ ๐๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐ ๐ค&๐https://lnkd.in/dj_m2yeQ๐ต. ๐๐๐๐ฟ๐ฒ ๐๐ผ๐ด๐ถ๐ฐ ๐๐ฝ๐ฝ๐ ๐๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐ ๐ค&๐https://lnkd.in/dDtnJe4v๐ญ๐ฌ. ๐๐๐๐ฟ๐ฒ ๐๐๐ป๐ฐ๐๐ถ๐ผ๐ป๐ ๐๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐ ๐ค&๐https://lnkd.in/dRWp3HZX๐ญ๐ญ. ๐๐๐๐ฟ๐ฒ ๐ฆ๐๐ป๐ฎ๐ฝ๐๐ฒ ๐๐ป๐ฎ๐น๐๐๐ถ๐ฐ๐ ๐๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐... Continue Reading →
Working with Columns in PySpark DataFrames: A Comprehensive Guide on using `withColumn()`
Theย withColumnย method in PySpark is used to add a new column to an existing DataFrame. It takes two arguments: the name of the new column and an expression for the values of the column. The expression is usually a function that transforms an existing column or combines multiple columns. Here is the basic syntax of the withColumn method:... Continue Reading →
Spark SQL
#Databricks #SQL for Data Engineering ,Data Science and Machine Learning.โ The whole SQL lesson for DataBricks is provided here.1๏ธโฃ spark sql sessions as series.https://lnkd.in/g77DE36a2๏ธโฃ How to register databricks community editionhttps://lnkd.in/ggAqRgKJ3๏ธโฃ What is DataWarehouse? OLTP and OLAP?https://lnkd.in/gzSuJCBC4๏ธโฃ how to create database in databricks?https://lnkd.in/gzHNFZrv5๏ธโฃ databricks file system dbfs.https://lnkd.in/dHAHkqd36๏ธโฃ Spark SQL Table , Difference between Managed table and... Continue Reading →
Databricks Learning Path
If you know working with databricks, it helps lot in your data engineering jobโฆYou can learn databricks hereโฆ1. Learn databricks basics here...https://lnkd.in/gQNKd8HEhttps://lnkd.in/gf_-6EEg2. pyspark with databricks herehttps://lnkd.in/g2iTevyJ2.1 azure databricks with python herehttps://lnkd.in/gyeNtq8n2.2 databricks with scala herehttps://lnkd.in/gzMAcm3s2.3 databricks with sql herehttps://lnkd.in/gdby9_bj3. databricks with spark herehttps://lnkd.in/g-YT-qiF4. databricks on AWShttps://lnkd.in/gYcxe8Tn5. official guide to learn databricks herehttps://lnkd.in/gt8sQeeH6. Databricks projectshttps://lnkd.in/gtpa7jhRhttps://lnkd.in/gdWUBUN9follow this... Continue Reading →
CODEBOX 