Big Data questions and answers
Spark, Hadoop and batch processing at volume. Page 7 of 9.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Answers 361-420
- Spark-Streaming from an Actor
- Spark-Streaming hangs with kafka starting offset at earliest (Kafka 2, spark 2.4.3)
- Spark-Streaming Kafka Direct Streaming API & Parallelism
- Spark - Get earliest and latest offset of Kafka without opening stream
- Spark 2.3.0 Failed to find data source kafka
- Spark 2.3 submit on Kubernetes error
- Spark 3.x Integration with Kafka in Python
- Spark + Kafka integration - mapping of Kafka partitions to RDD partitions
- Spark batch reading from Kafka & using Kafka to keep track of offsets
- Spark Counting co-occurrence - Algorithm for efficient multi-pass filtering of huge collections
- Spark Dataframe write to kafka topic in avro format?
- Spark DStream periodically call saveAsObjectFile using transform does not work as expected
- Spark Executor Managed memory leak detected
- Spark executor metrics don't reach prometheus sink
- Spark executor self-exiting due to driver disassociated in Kubernetes with client deploy-mode
- Spark executors fails to run on kubernetes cluster
- Spark Find pairs having at least n common attributes?
- Spark from_avro() dataframe.show() errors java.lang.ArrayIndexOutOfBoundsException
- Spark, Incorrect behaviour when throwing SparkException in EMR
- Spark job running for long for too small data
- Spark K-fold Cross Validation
- Spark Kafka Direct DStream - How many executors and RDD partitions in yarn-cluster mode if num-executors is set?
- Spark Kafka Streaming Issue
- Spark Kubernetes - FileNotFoundException when copying config files from driver to executors using --files or spark.files
- Spark ML - MulticlassClassificationEvaluator - can we get precision/recall by each class label?
- Spark MLlib / K-Means intuition
- Spark on Kubernetes Executor pods silently get killed
- Spark output to kafka exactly-once
- Spark processing multiple kafka topic in parallel
- Spark Python Avro Kafka Deserialiser
- Spark Random Forests Different results with same seed
- Spark set to read from earliest offset - throws error on attempting to consumer an offset no longer available on Kafka
- Spark Streaming - Batch Interval vs Processing time
- spark streaming + kafka - spark session API
- Spark Streaming + Kafka Could not compute split, block input ... not found
- Spark Streaming + kafka INFO SimpleConsumer Reconnect due to socket error java.nio.channels.ClosedChannelException
- Spark Streaming + Kafka SparkException Couldn't find leader offsets for Set
- Spark streaming + Kafka vs Just Kafka
- spark streaming assertion failed Failed to get records for spark-executor-a-group a-topic 7 244723248 after polling for 4096
- Spark Streaming Exception java.util.NoSuchElementException None.get
- Spark Streaming from Kafka Consumer
- Spark Streaming from Kafka has error numRecords must not be negative
- Spark Streaming Issues when processing time > batch time
- Spark streaming jdbc read the stream as and when data comes - Data source jdbc does not support streamed reading
- Spark streaming job fails after getting stopped by Driver
- Spark Streaming Kafka - Job always quits when RDD contains an actual message
- Spark Streaming Kafka backpressure
- Spark streaming Kafka messages not consumed
- Spark Streaming Kafka stream
- Spark Streaming Micro batches Parallel Execution
- Spark Streaming Reading data from kafka that has multiple schema
- Spark streaming with Kafka - createDirectStream vs createStream
- Spark Structured Streaming - Limitations? (Source Performance, Unsupported Operations, Spark UI)
- Spark Structured Streaming + Kafka Integration MicroBatchExecution PartitionOffsets Error
- Spark Structured Streaming app has no jobs and no stages
- Spark structured streaming app reading from multiple Kafka topics
- spark structured streaming avro to avro and custom Sink
- Spark Structured Streaming Checkpoint Compatibility
- Spark structured streaming exactly once - Not achieved - Duplicated events
- Spark structured streaming kafka convert JSON without schema (infer schema)
Course
Beginner
27 lessons
10 hours
System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the course