Big Data questions and answers
Spark, Hadoop and batch processing at volume. Page 4 of 9.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Answers 181-240
- how to efficiently cache large file in hadoop map reduce jobs?
- how to efficiently move data from Kafka to an Impala table?
- How to fetch offset id while consuming Kafka from Spark, save it in Cassandra and use it to restart Kafka?
- How to find which consumer is assigned to which partition of a topic in kafka?
- How to fix java.io.NotSerializableException org.apache.kafka.clients.consumer.ConsumerRecord in Spark Streaming Kafka Consumer?
- How to get Filename/File Contents as key/value input for MAP when running a Hadoop MapReduce Job?
- How to get largest number of consecutive integers in a substantially large array (spread across multiple machines)
- How to handle fields enclosed within quotesCSV in importing data from S3 into DynamoDB using EMR/Hive
- How to handle large amouts of data in tensorflow?
- how to implement eigenvalue calculation with MapReduce/Hadoop?
- How to insert 9 billions records into a database in 2 minutes?
- How to install Hive Metastore in Kubernetes?
- How to install packages in Airflow docker-compose?
- How to integrate Apache Spark with Spring MVC web application for interactive user sessions
- how to load a Kafka topic to HDFS?
- How to make Spark Streaming (Spark 1.0.0) read the latest data from Kafka (Kafka Broker 0.8.1)
- How to manually commit offset in Spark Kafka direct streaming?
- How to manually set group.id and commit kafka offsets in spark structured streaming?
- How to map features from the output of a VectorAssembler back to the column names in Spark ML?
- How to overwrite Spark ML model in PySpark?
- How to pass data from Kafka to Spark Streaming?
- How to pass pandas dataframe to airflow tasks
- how to process data in chunks/batches with kafka streams?
- How to put the files into memory using Hadoop Distributed cache?
- How to re-partition pyspark dataframe?
- How to read data using key in Kafka Consumer API?
- How to read json data using scala from kafka topic in apache spark
- How to retrieve Metrics like Output Size and Records Written from Spark UI?
- How to save latest offset that Spark consumed to ZK or Kafka and can read back after restart
- How to scale Kafka Connect effectively?
- How to serve a Spark MLlib model?
- how to set Hadoop DistributedCache?
- How to set offset committed by the consumer group using Spark's Direct Stream for Kafka?
- How to set optimal config values - trigger time, maxOffsetsPerTrigger - for Spark Structured Streaming while reading messages from Kafka?
- how to specify consumer group in Kafka Spark Streaming using direct stream
- How to stop spark streaming when the data source has run out
- How to stream data from Kafka topic to Delta table using Spark Structured Streaming
- How to stream large files through Kafka?
- How to understand hdfs client in the process of file read and file write?
- How to update Spark MatrixFactorizationModel for ALS
- How to use a MapReduce output in Distributed Cache
- How to use foreach or foreachBatch in PySpark to write to database?
- How to use from_json with Kafka connect 0.10 and Spark Structured Streaming?
- how to use Kafka 0.8 Log4j appender
- How to use kafka.group.id and checkpoints in spark 3.0 structured streaming to continue to read from Kafka where it left off after restart?
- How to use multi-thread consumer in kafka 0.9.0?
- How to use Spark Structured Streaming with Kafka Direct Stream?
- How to use two Kerberos keytabs (for Kafka and Hadoop HDFS) from a Flink job on a Flink standalone cluster?
- How to write Kafka Producer in Scala
- How to write spark streaming DF to Kafka topic
- IllegalStateException _spark_metadata/0 doesn't exist while compacting batch 9
- Increase no of records read in single poll of KafkaConsumer
- Integrating Apache Kafka with Apache Spark Streaming using Python
- Integrating Spark Structured Streaming with the Confluent Schema Registry
- IO exception when reading from distributed cache in Hadoop file system?
- Is Apache Spark less accurate than Scikit Learn?
- Is it necessary to submit spark application jar?
- Is it possible to create a kafka topic with dynamic partition count?
- Is it possible to obtain objects from distributed cache in a Hadoop Partitioner?
- Is it possible to obtain specific message offset in Kafka+SparkStreaming?
Course
Beginner
27 lessons
10 hours
System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the course