-
Git Internals Part I
the blog comes from Github written by Principal Software Engineer Derrick Stolee. jfyi – git was originally created by your favourite linux guy, Linus Torvalds for him to help manage version control himself for the Linux Kernel lots of technical things here, the blog summarizing a lot of things. I am sure the technical implementations…
-
MemQ Pub/Sub + Kafka @ Pinterest
The story comes from Pinterest in Mid 2020 that used Kafka to publish Logging data to interested parties. Data Flow Systems generating data –> Producer Logging Platform –> Pub/Sub (Kafka) –> Consumer Logging platform–> Extract data from related systems Separating storage and serving components which may have different parameters and requirement Custom replications and rebalancing…
-
Postgresql -> Cockroach DB
Context: This story comes comes from Storj Labs, a storage provider that boasts resiliency of storage objects and high performance Storj Labs wants to move from Postgresql to CockroachDB. Postgresql is a production grade ACID compliant relational DB that can fit into many use cases such as OLAP, OLTP, GIS, Events Processing. Cockroach DB is…
-
SLATE Testing At Uber
SLATE – Short Lived Application Testing Environments Massively successful companies Facebook(Meta) and Uber have sophisticated testing strategies that is non traditional and “works”. The Traditional Approaches are below Uber’s testing strategy is leaning forward more on “online tests” than “offline tests” A separate school of thought is that writing heavy unit tests is useless as…
-
How Netflix Makes Product Decisions?
consensus decision making is used heavily agnostic to attendees familiarity of the implications top down approach where attendees sometimes may feel unfair. Netflix runs A/B testing to build trust in the results. This is almost a requirement. Across the organisation, there is data-driven mindset. This makes Data & Insights team at the center of the…
-
Reduce Logging Costs at Uber with CLP
Context: The Big Data platform at Uber where logs are generated by hundreds of thousands of Spark Jobs. On peak day 200k TB of data would be generated The cost of maintaining these logs became so high, it would cost Uber $1.3million per year to keep 1 month worth of log, or $180k to keep…
-
OLTP and OLAP at Grab
Grab processes millions of orders per day. At the rate of their scale, they encountered reliability and cost issues. High availability – even when DB has certain failure, it should always be online with some degraded experience HIgh Write and Read throughput Scalable without high cost Strong consistency for transactional query, and eventual consistencies for…
-
Why Trello Chose Kafka over RabbitMQ?
Choosing the correct technologies from the very beginning can save up lots of time and headaches. Switching technology later on when the production system is running is very hard and only done if it defines business’s survival I am always interested in which use cases were anticipated (or did not anticipate when chosing certain technologies)…
-
Scaling Bloated PostgreSQL DB
The story comes from OneSignal a messaging platform and depends on the postgresql listen/notify features in order to push notifications to end users. Lots of words of wisdom in this article. The problems presented seems to be the case when the performance enhancements of the DB were already saturated (such as too many indices added)…
-
Subscribe
Subscribed
Have a WordPress.com account? Log in now.
