Context: The Big Data platform at Uber where logs are generated by hundreds of thousands of Spark Jobs. On peak day 200k TB of data would be generated The cost of maintaining these logs became so high, it would cost Uber $1.3million per year to keep 1 month worth of log, or $180k to keep…