I shared a story about a project where I had to optimize a data pipeline that was running way too slow. I went into detail about how we were processing about 10TB of data every week and the load times were killing our reporting capabilities. I used Apache Spark for this, but there were some inefficiencies in the way we were handling joins and transformations.

The interviewer seemed genuinely interested when I explained how I used Spark’s DataFrame API to improve performance. I talked about implementing caching and tuning the system, which reduced our processing time by about 30%. But then I got a bit flustered when they asked for specific metrics. I could remember percentages but had trouble recalling the actual times or benchmarks, which felt like a misstep.

They followed up by asking how I dealt with team pushback when proposing changes. That caught me off guard, since I like to think of myself as a collaborative person. I mentioned that I organized a few workshops to communicate the benefits, which helped ease some resistance. They seemed to appreciate that approach, but I could tell I didn’t fully convince them of my conflict resolution skills.

Overall, I felt I had a solid narrative about the technical challenge but struggled to articulate the softer skills, which is something I plan to work on in future interviews. The outcome is still pending, but I’m a bit anxious about how well I conveyed my team dynamics and influence. It’s such a crucial part of making any technical solution work, right?