On Spark, Hive, and Small Files: An In-Depth Look at Spark Partitioning Strategies — Airbnb Engineering Blog.

Millions of small files in HDFS or almost all you want to know about Spark partitioning. At least for a start :)

---