![]() ![]() Many clusters are set up, each with a tiny handful of topics and partitions driving huge cloud provider costs and inefficient use of resources. GDPR, PII, PCI, Regulations: how can you even consider applying them at scale?.Security, encryption of data: a black hole.People turnover will create knowledge-loss.Data may be duplicated as teams might create their own clusters to hold the same information.Reduce cost or complexity by clearing up topics.The organization has no way to enforce various strategies or is going to be in pain: No one else except the Product team owning one Kafka knows what the topics are for. Topic and cluster configuration varies hugely and doesn't adhere to best practices - infinite expiry can result in substantial storage costs, expiry is not correctly considered, data can be lost, etc.Security is inconsistent, with different (or no) policies across the teams.They can rely on Kafka Expert in the company, but the goal is similar to the DevOps culture:Īll the Kafka clusters and operations are going to be different as there are no gatekeepers to enforce organizational rules, eg: The organization encourages teams to self-serve their own Kafka as they were building their own Spring Boot application: they become responsible for Kafka resource management, topic creation, ACLS, Kafka Connect, and general Kafka admin. They will move fast and deal with their problems their way. On the other side of the spectrum, we have the full decentralized model: let each Product team do their thing. Let's decentralize Kafka ownership, Free The Data! # One typical solution of organizations is to grow this Central team, adding more people to absorb the throughput of requests instead of fixing the real issue: don't create a bottleneck. This Central team would LOVE to focus on Kafka and unleashing value from its data, spreading it even more in the organization and accelerating how developers work with it… but they are spending time answering all the support questions and trying not to be an administration bottleneck! They get quizzed by the CFO on why Kafka costs so much and are challenged to reduce the spending without impacting the business.People try and get around them because of delays or because their perceived requirements don't match what is allowed by default by the central team. They know the rules and regulations that need to be adhered to and need to try and keep everyone in line. They are the gatekeepers preventing 1000 partition topics from being created.They need to know why there are mysterious empty topics that can't be deleted due to use by legacy applications. They get asked to categorize 500 topics from across the organization and decide whether they are configured correctly or are still needed.In retrospect, they will often require time to do some archeology and understand the history and how we got here. This Central team is expected to know what all the topics and Kafka instances are for. They want to build a magnificent Streaming Platform to help the organization. This team manages topic creation, ACLs (security), new cluster requests (like for a new product), and general administration around developer requirements. Some organizations have a Kafka Central team to manage everything-Kafka: from clusters to software. hybrid! Let's centralize Kafka ownership! # Let's deep dive into these extremes and explore a third option. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |