Big Data Engineering Interview series-1

**Top Big Data Interview Questions (2024) – Detailed Answers**

1. **What is Hadoop and how does it work?** 
   Hadoop is an open-source framework designed for distributed storage and processing of large datasets across clusters of computers. It consists of two main components: Hadoop Distributed File System (HDFS) for fault-tolerant storage, which splits data into blocks and replicates them across nodes, and MapReduce for parallel data processing. MapReduce divides tasks into map (data transformation) and reduce (aggregation) phases, executed across the cluster. Hadoop’s ecosystem includes tools like Hive, Pig, and HBase for additional functionality, enabling scalable, cost-effective big data processing.

2. **Why move from MapReduce to Spark?** 
   Apache Spark outperforms MapReduce by leveraging in-memory computation, reducing disk I/O bottlenecks inherent in MapReduce’s disk-based approach. Spark’s DAG (Directed Acyclic Graph) execution model optimizes workflows, unlike MapReduce’s rigid two-stage process. Spark supports iterative algorithms (e.g., machine learning), real-time streaming, and interactive queries, which MapReduce lacks. Additionally, Spark’s high-level APIs (DataFrames, Datasets) and libraries (Spark SQL, MLlib) simplify development compared to MapReduce’s verbose Java-based coding, making it more efficient and developer-friendly.

3. **Does Spark provide storage?** 
   No, Spark is a distributed data processing engine, not a storage system. It relies on external storage solutions like Hadoop HDFS, Amazon S3, Azure Blob Storage, or databases (e.g., Cassandra, PostgreSQL). Spark reads data from these sources, processes it in memory, and writes results back to the storage layer, offering flexibility to integrate with various data lakes or warehouses.

4. **Give a high-level explanation of Spark.** 
   Apache Spark is a unified, open-source framework for distributed big data processing, designed for speed and ease of use. It performs in-memory computations, making it significantly faster than disk-based systems like Hadoop MapReduce. Spark supports batch processing, real-time streaming (via Spark Streaming), machine learning (MLlib), graph processing (GraphX), and SQL queries (Spark SQL). Its core abstraction, RDDs, and higher-level APIs like DataFrames and Datasets enable developers to process large datasets across clusters with fault tolerance and scalability.

5. **Why switch from RDDs to DataFrames in Spark?** 
   DataFrames, introduced in Spark 1.3, provide a higher-level abstraction than RDDs, offering structured data manipulation similar to SQL tables or pandas DataFrames. They leverage the Catalyst optimizer for query optimization and Tungsten engine for efficient memory management, resulting in better performance. DataFrames simplify coding with intuitive APIs, support SQL queries, and integrate seamlessly with data sources. RDDs, while flexible for low-level control, require manual optimization and are less user-friendly, making DataFrames the preferred choice for most use cases.

6. **Which languages does Spark support?** 
   Spark supports Scala (its native language), Java, Python (via PySpark), and R (via SparkR). Scala and Java offer full access to Spark’s features, while Python and R are popular for data science and analytics, though they may have slight performance overhead due to serialization. SQL is also supported via Spark SQL for querying structured data.

7. **What are RDDs and their importance?** 
   Resilient Distributed Datasets (RDDs) are Spark’s fundamental data structure, representing immutable, fault-tolerant collections of objects distributed across a cluster. RDDs support parallel operations and recover from node failures through lineage (a record of transformations). They provide low-level control for custom data processing, enabling flexibility for complex workflows. While DataFrames have largely replaced RDDs for structured data, RDDs remain crucial for specific use cases requiring fine-grained control or unstructured data processing.

8. **What happens during actions/transformations in Spark?** 
   Transformations (e.g., `map`, `filter`, `join`) are lazy operations that define a new RDD or DataFrame without immediate execution, building a logical execution plan (DAG). They enable optimization by deferring computation. Actions (e.g., `collect`, `count`, `save`) trigger the execution of the plan, initiating data processing across the cluster and returning results to the driver or writing to storage. This lazy evaluation optimizes resource usage and fault tolerance.

9. **Explain Spark architecture.** 
   Spark’s architecture consists of a driver program, a cluster manager, and worker nodes with executors. The driver runs the main application, maintains the DAG, and coordinates tasks. The cluster manager (e.g., YARN, Mesos, or Spark’s standalone) allocates resources. Executors, running on worker nodes, perform computations and store data in memory or disk. Spark’s communication model ensures fault tolerance and scalability, with the driver orchestrating task distribution and result collection.

10. **What are deployment modes and their use cases?** 
    Spark supports two deployment modes: **client mode**, where the driver runs on the client machine (ideal for interactive applications like Jupyter notebooks or development), and **cluster mode**, where the driver runs on a worker node within the cluster (suitable for production jobs requiring high availability). Cluster mode is preferred for long-running applications, while client mode is better for debugging or low-latency interactive tasks.

11. **Describe the plans created when executing a Spark job.** 
    Spark creates three plans: 
    – **Logical Plan**: Represents the user’s high-level operations (e.g., joins, filters) as a tree of transformations. 
    – **Physical Plan**: Translates the logical plan into executable operations, considering available resources and data locality. 
    – **Optimized Physical Plan**: The Catalyst optimizer applies rule-based and cost-based optimizations (e.g., predicate pushdown, join reordering) to minimize computation and data shuffling, ensuring efficient execution.

12. **What is a predicate push down?** 
    Predicate pushdown is an optimization where filtering conditions (predicates) are applied at the data source (e.g., database or parquet file) rather than after data is loaded into Spark. This reduces the amount of data transferred and processed, improving performance. For example, in a query like `SELECT * FROM table WHERE age > 30`, the database filters rows before sending them to Spark, leveraging the source’s indexing capabilities.

13. **Explain jobs, stages, and tasks in Spark.** 
    – **Job**: A complete computation triggered by an action (e.g., `count`), encompassing all transformations needed to produce the result. 
    – **Stage**: A group of tasks that can be executed in parallel without shuffling data. Stages are created based on wide transformations (e.g., `groupBy`) that require data exchange. 
    – **Task**: The smallest unit of work, executed by an executor on a single partition of data. Each stage consists of multiple tasks, distributed across the cluster.

14. **What are the types of transformations in Spark?** 
    – **Narrow Transformations**: Operations like `map`, `filter`, or `union` that process data within a single partition without shuffling (e.g., applying a function to each row). 
    – **Wide Transformations**: Operations like `groupBy`, `join`, or `reduceByKey` that require data shuffling across partitions, leading to stage boundaries and higher computational cost.

15. **Difference between repartition and coalesce?** 
    – **Repartition**: Redistributes data across a specified number of partitions, involving a full shuffle. Useful for increasing or decreasing partitions to optimize parallelism (e.g., `df.repartition(10)`). 
    – **Coalesce**: Reduces the number of partitions without shuffling, merging partitions locally on executors. It’s faster but only decreases partitions (e.g., `df.coalesce(5)`). Use `repartition` for flexibility, `coalesce` for efficiency when reducing partitions.

16. **Should you infer schema or specify it when creating a DataFrame?** 
    Specifying a schema is preferred because it avoids the overhead of Spark scanning data to infer types, ensures consistency, and prevents errors from ambiguous or incorrect inferences. Inferring schemas is convenient for small datasets or prototyping but can be slow and unreliable for large or complex data. Always define schemas in production for performance and reliability.

17. **What are the ways to enforce schema? Provide an example.** 
    Schemas can be enforced using: 
    – **StructType**: Programmatically define schema with `StructType` and `StructField`. Example: 
      “`python
      from pyspark.sql.types import StructType, StructField, StringType, IntegerType
      schema = StructType([StructField(“name”, StringType(), True), StructField(“age”, IntegerType(), False)])
      df = spark.createDataFrame(data, schema)
      “` 
    – **DDL String**: Use a SQL-like string (e.g., `”name STRING, age INT”`). 
    – **Existing DataFrame**: Apply a schema from another DataFrame or data source. Specifying schema ensures type safety and optimizes performance.

18. **SQL coding questions** 
    Common SQL questions test skills like: 
    – Writing complex joins (inner, left, cross). 
    – Aggregations with `GROUP BY` and `HAVING`. 
    – Window functions (e.g., `RANK()`, `ROW_NUMBER()`). 
    – Handling nulls or duplicates. 
    Example: Find the second-highest salary: 
    “`sql
    SELECT MAX(salary)
    FROM employees
    WHERE salary < (SELECT MAX(salary) FROM employees);
    “`

19. **Which Azure cloud services have you used?** 
    Common services include: 
    – **Azure Data Factory (ADF)**: For orchestrating data pipelines. 
    – **Azure Databricks**: For Spark-based analytics and machine learning. 
    – **Azure Blob Storage**: For scalable data storage. 
    – **Azure Synapse Analytics**: For data warehousing and analytics. 
    – **Azure SQL Database**: For relational database management. These services integrate for end-to-end data workflows.

20. **Explain Databricks architecture at a high level.** 
    Databricks is a unified analytics platform built on Apache Spark, hosted on cloud providers like Azure, AWS, or GCP. Its architecture includes: 
    – **Control Plane**: Manages the workspace, UI, and cluster orchestration (hosted by Databricks). 
    – **Data Plane**: Runs Spark clusters and processes data in the customer’s cloud account, ensuring security and isolation. 
    – **Workspace**: Provides notebooks, dashboards, and collaboration tools. Databricks integrates with cloud storage (e.g., ADLS, S3) and supports Delta Lake for reliable data lakes.

21. **How do you run SQL queries in Databricks?** 
    SQL queries can be executed via: 
    – **SQL Editor**: A dedicated interface for writing and running queries on registered tables. 
    – **%sql Magic Command**: In a notebook, use `%sql SELECT * FROM table` to query Delta tables or other sources. 
    – **Spark SQL API**: Programmatically via `spark.sql(“SELECT * FROM table”)`. Queries leverage Spark’s distributed engine and integrate with Delta Lake or external databases.

22. **How can one notebook run another in Databricks?** 
    Use: 
    – **%run**: Include another notebook’s code (e.g., `%run ./path/to/notebook`). 
    – **dbutils.notebook.run()**: Execute a notebook and return its status (e.g., `dbutils.notebook.run(“path/to/notebook”, timeout_seconds=60)`). This allows modular workflows, such as running preprocessing or utility notebooks from a main notebook.

23. **Can you use parameters when running Databricks notebooks?** 
    Yes, parameters can be passed using: 
    – **Widgets**: Create input widgets (e.g., `dbutils.widgets.text(“param”, “default”)`) to accept user inputs or pipeline parameters. 
    – **Notebook Workflows**: Pass parameters via `dbutils.notebook.run(“notebook”, timeout, {“param”: “value”})` or through ADF pipelines. This enables dynamic, reusable notebooks for varying inputs.

24. **Difference between Data Lake and Delta Lake? Pros and cons of each.** 
    – **Data Lake**: A centralized repository for raw, unstructured, semi-structured, or structured data stored in native formats (e.g., Parquet, CSV). 
      – **Pros**: Highly flexible, cost-effective, supports diverse workloads. 
      – **Cons**: Lacks ACID transactions, prone to data inconsistency, and requires manual schema management. 
    – **Delta Lake**: An open-source storage layer adding ACID transactions, schema enforcement, and time travel to data lakes. 
      – **Pros**: Ensures data reliability, supports streaming/batch unification, and enables versioning. 
      – **Cons**: Adds overhead for setup and maintenance, slightly higher complexity. Delta Lake is ideal for production-grade data lakes requiring consistency.

25. **What activities are available in ADF?** 
    Azure Data Factory (ADF) supports: 
    – **Data Movement**: Copy activities to transfer data between sources (e.g., Blob Storage to SQL Database). 
    – **Data Transformation**: Integration with Databricks, HDInsight, or Synapse for processing (e.g., Spark or SQL transformations). 
    – **Control Flow**: Activities like ForEach, If Condition, Until, and Web for orchestration and logic. 
    – **Pipeline Activities**: Triggering external services (e.g., Databricks notebooks) or custom scripts. These enable complex ETL workflows.

26. **Scenario-Based question** 
    Scenario questions often involve designing a data pipeline or optimizing a process. Example: “Design a pipeline to ingest streaming data, process it, and store results in a data warehouse.” 
    – **Solution**: Use Azure Event Hubs for ingestion, Databricks for Spark streaming processing, Delta Lake for intermediate storage, and Azure Synapse for warehousing. Optimize by partitioning data, enabling predicate pushdown, and tuning Spark configurations (e.g., executor memory). Monitor with ADF pipelines and alerts. This tests practical knowledge of architecture, tools, and optimization.