{"id":5101,"date":"2023-11-20T15:44:51","date_gmt":"2023-11-20T15:44:51","guid":{"rendered":"https:\/\/staging.elixirrdigital.com\/us\/?p=5101"},"modified":"2025-03-17T13:09:22","modified_gmt":"2025-03-17T13:09:22","slug":"ml-model-deployment-with-kubernetes-and-go-language","status":"publish","type":"post","link":"https:\/\/staging.elixirrdigital.com\/us\/2023\/11\/20\/ml-model-deployment-with-kubernetes-and-go-language\/","title":{"rendered":"ML Model Deployment with Kubernetes and Go Language"},"content":{"rendered":"

Machine learning models are deployed to solve specific problems or provide value in various applications, such as healthcare, finance, recommendation systems, and autonomous vehicles. In these scenarios, model performance directly impacts real-world outcomes.<\/p>\n

\n
<\/div>\n<\/div>\n
\n
\n

One of the challenges when building a machine learning platform, is how to enable the data scientists to dynamically deploy their models into a production environment- so they can rapidly update, optimize, and monitor models- enhancing agility, efficiency, and responsiveness to changing needs and data. This challenge is something that is being solved using MLOps.<\/p>\n

Machine learning models are deployed to solve specific problems or provide value in various applications, such as healthcare, finance, recommendation systems, and autonomous vehicles. In these scenarios, model performance directly impacts real-world outcomes.<\/p>\n

In healthcare, for example, accurate and performant diagnosis models are crucial for life-or-death situations. Consumer-facing applications, such as recommendation systems, significantly impact user experience \u2013 subpar performance leads to frustration, while high performance enhances satisfaction and loyalty. Safety-critical applications, like autonomous vehicles, demand models with extremely high performance to prevent catastrophic consequences. In resource-constrained environments like edge devices, optimizing model performance for efficiency without sacrificing accuracy is essential. The scale and volume of data also plays a role, with high-traffic platforms requiring real-time predictions with low latency.<\/p>\n

Recognizing when performance is critical involves considering real-world consequences, user experience, safety, resource constraints, and compliance requirements. In the evolving field of machine learning, balancing trade-offs and optimizing performance is crucial for successful deployment, ensuring AI systems deliver intended value and impact.<\/p>\n

MLOps (Machine Learning Operations) is a collaborative set of practices that integrates machine learning, data engineering, and DevOps principles to streamline and automate the development, deployment, and maintenance of machine learning models in production environments. Applying these practices increases the quality, simplifies the management process, and automates the deployment of Machine Learning and Deep Learning models in large-scale production environments. MLOps enhances operational efficiency, reduces time-to-market, and ensures consistent and reliable performance of machine learning models, thereby driving better decision-making and competitive advantage.<\/p>\n

 <\/p>\n

Challenge<\/h1>\n

The primary challenge is finding a solution for enabling data scientists to dynamically deploy models in a production environment. This involves considering architectural choices that address performance challenges, scalability, and customization beyond standard cloud service offerings. Balancing cost-effectiveness with overall performance is crucial, given that relying on specific vendor platforms can be financially demanding. Another key consideration is ensuring flexibility for a potential switch between cloud providers to avoid extensive refactoring.<\/p>\n

 <\/p>\n

Solution<\/h1>\n

To address the challenges and facilitate the dynamic deployment of machine learning models, a specialized ML platform was developed using\u00a0Kubernetes<\/a>. This platform was initially deployed on\u00a0AWS<\/a>, but its design allows for flexibility in deployment across various cloud platforms or even on-premises setups, thanks to Kubernetes\u2019 cloud-agnostic nature. The\u00a0Elastic Kubernetes Service (EKS)<\/a>\u00a0was utilized for deploying the Kubernetes Cluster, enabling the creation of diverse services tailored to specific needs within that environment.<\/p>\n

When addressing performance challenges, it\u2019s important to choose the right tool for the job, so selecting a programming language was a particularly intriguing aspect of the project. Initially,\u00a0Python<\/a>\u00a0with\u00a0FastAPI<\/a>\u00a0was employed, driven by the appeal of its rapid development capabilities. However, as challenges related to suboptimal performance emerged, a thoughtful shift to the\u00a0Go language<\/a>\u00a0was initiated. In this transition, Go\u2019s efficiency and concurrent programming features were harnessed. Complementing this shift,\u00a0Fiber<\/a>, a web framework tailored for Go, was adopted, aiming to amplify the overall performance and responsiveness of the Services. This strategic adjustment not only addressed the initial performance concerns but also reflected the commitment to optimizing the system for long-term reliability and scalability.<\/p>\n

<\/h1>\n

How the Platform Works?<\/h1>\n

\"\"<\/p>\n

 <\/p>\n<\/div>\n

Kubernetes orchestrates containerized applications across a cluster of machines, managing aspects like deployment, scaling, and operations, to ensure efficient and reliable system performance.<\/p>\n

In this setup, the Kubernetes cluster resides on AWS, where each node aligns with an EC2 instance. Within each node, numerous deployments can be hosted. To introduce a new microservice to the Kubernetes environment, a Kubernetes Deployment is created, functioning as an abstraction layer responsible for overseeing the application\u2019s lifecycle across these nodes. When managing services, there are four abstraction levels to consider: a container, a pod, a replica set, and a deployment.<\/p>\n

To develop a microservice, a Docker image containing the application code must be built together with a Pod in Kubernetes, which can hold multiple containers (typically one main and optional sidecar containers). Replica Sets are used to maintain multiple identical Pod replicas, to ensure high availability and scalability. Adding a further level of abstraction, Deployments are created for each service; Kubernetes then automatically handles the underlying tasks, including creating the required Pods and containers within them.<\/p>\n

To make the microservice accessible to users, a Service and Ingress must be set up in Kubernetes. Ingress acts as a traffic routing mechanism based on request paths, where an Ingress controller like\u00a0Nginx<\/a>\u00a0is utilized, functioning as a reverse proxy. Deploying a new Ingress controller automatically initiates the creation of a Load Balancer in the background. Incoming traffic first reaches this Load Balancer, which then directs it to one of the nodes where the Nginx Deployment determines the appropriate route and service to use, according to the defined Ingress rules.<\/p>\n

This outlines the platform\u2019s initial functionality. Now, let\u2019s delve into the details of model deployment, the construction of services and how end users can leverage them.<\/p>\n

<\/h1>\n

ML Model Deployment<\/h1>\n<\/div>\n

\"\"<\/p>\n

The approach centers around providing data scientists with a tailored Application Programming Interface (API) designed to streamline the deployment process. Via this API, they can effortlessly submit requests for deploying their machine learning models to the platform. These requests include details of the specific model to be deployed and a comprehensive image containing all required logic and dependencies. Furthermore, they can specify resource requirements like memory, CPU, and GPU usage, and other deployment options like canary releases, among various other parameters.<\/p>\n

Solution<\/h1>\n

In this process, data scientists compile all necessary information into a request and submit it to the platform, targeting the Model Deployment Service. This service dynamically provisions Kubernetes resources in response to the request. Specifically, it creates a new Model Inference Service, leveraging the image prepared by the data scientists, and makes this service available to users via Nginx and a Load Balancer. Essentially, the Model Deployment Service uses the provided image to establish a Model Inference API, which exposes the desired models to end users through a REST API interface.<\/p>\n

The journey of constructing the Model Deployment Service and choosing the programming language became particularly fascinating when Python with FastAPI was initially opted for. However, faced with suboptimal performance, the decision was made to transition to the Go language and embrace Fiber, a Go-specific web framework, to address these challenges.<\/p>\n

Python vs Go Language<\/h1>\n

The performance of Python and Go Language in the context of ML model deployment was intended to be compared, and the reason for choosing Go is to be demonstrated. Two services were constructed: the first utilizing Python and FastAPI, while the second employing Go and Fiber. The identical functionality of both services involves dynamically provisioning and terminating Kubernetes resources to expose ML models as an API. A direct comparison will be made between the performance of these two services across various categories, encompassing execution time, docker image creation, CPU and memory utilization, dependency compilation, compilation duration, and overall performance.<\/p>\n

Execution Time<\/strong>
\nThe comparison of the two algorithm implementations\u2019 execution speeds was conducted using Application Performance Monitoring (APM) agents. These agents gathered performance metrics from the services and sent them to an APM Server in our cluster, which then relayed the data to\u00a0Elasticsearch<\/a>\u00a0for visualization using\u00a0Kibana<\/a>.<\/p>\n

The analysis showed that deploying and destroying machine learning models using the Python-based service took about 1000 milliseconds on average. In stark contrast, the Go-based service achieved much faster performance, with deployments averaging around 150 milliseconds and destructions just 30 milliseconds.<\/p>\n

Consequently, the Go Language application was between 6 and 33 times faster than the Python application, depending on the task. Furthermore, the Go web framework Fiber was capable of handling approximately 300,000 requests per second, a significant increase compared to FastAPI\u2019s capability of managing only about 200 requests per second.<\/p>\n

The Python metrics:<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

Docker image<\/strong>
\nIn evaluating the complexity of producing Docker images and the size of the generated images for Python and Go applications, a notable difference was observed. Given that the application provides a REST API, it necessitates shipping as a Docker image.<\/p>\n

A multi-stage build approach was adopted for creating the Docker image: initially building the app and aggregating data, followed by assembling a \u201cproduction\u201d image. In this context, Go proved to be exceptionally efficient. The Go compiler was instructed to produce a self-contained binary, which could be packaged in a minimal scratch image, resulting in a final image size of just over 31 MB. In contrast, the Python application\u2019s final Docker image was significantly larger, amounting to hundreds of megabytes. This stark difference highlights Go\u2019s advantage in terms of both image size and the efficiency of the build process for Dockerized applications.<\/p>\n

CPU and memory usage<\/strong>
\nTo compare the CPU and memory usage of Python and Go applications, a\u00a0Prometheus<\/a>\u00a0server was set up to pull metrics from nodes and pods, with\u00a0Grafana<\/a>\u00a0employed for visualizing these metrics. The results showed a significantly higher resource usage for the Python application, with up to 37 times greater CPU usage and up to 4 times more memory consumption compared to the Go (Golang) application. In a production environment, where resource usage directly translates to operational costs, Golang emerges as a much more cost-effective solution due to its substantially lower CPU and memory requirements.<\/p>\n

The Python Usage Results:<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

The Go results:<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

Dependencies<\/strong>
\nWhen developing a service to support ML model deployment, it is common to rely on various libraries, such as ML-specific ones (e.g.,\u00a0TensorFlow<\/a>,\u00a0PyTorch<\/a>) or those for interacting with Kubernetes Clusters and model registry services like\u00a0MLflow<\/a>. However, these dependencies often come with their own sets of sub-dependencies, which can sometimes lead to conflicts that must be resolved before building the Docker image. Additionally, if these dependencies are not correctly pinned, they can cause issues in production.<\/p>\n

Python applications are particularly prone to breaking in production due to missing runtime dependencies, incorrect or incompatible library versions, or the wrong Python version. In contrast, the Go Language compiles into a single binary, eliminating these types of issues entirely. This characteristic of Go significantly enhances the reliability and stability of the application in production environments, as it is not susceptible to the same dependency-related problems that can plague Python applications.<\/p>\n

Compile time and overall experience<\/strong>
\nThe comparison between Go, a compiled language, and interpreted languages like Python isn\u2019t direct due to their fundamentally different nature but Go can be more aptly compared with other compiled languages like C or C++.<\/p>\n

One notable advantage of Go is that its compilation time is linear with respect to the number of imported packages, as opposed to the exponential compilation time often experienced with C\/C++. This means significantly less waiting time during development, particularly after making changes or testing new lines of code, which enhances the efficiency and experience of model deployment.<\/p>\n

The design of the Go programming language (Golang) is highly commendable. Its straightforwardness, strong support for concurrency, and enhanced readability form a feature set that is exceptionally attractive. These qualities not only facilitate easier and more efficient coding but also contribute to the overall robustness and maintainability of the applications developed in Go.<\/p>\n

Let\u2019s Dig into The Code<\/h1>\n

The complete application can be accessed at this GitHub repository:\u00a0https:\/\/github.com\/iolap-blog\/mlops<\/a>, and an overall walkaround will be provided to enable an understanding of the overall approach taken in this context.<\/p>\n

REST API<\/strong>
\nHere, all necessary modules are imported, and the Kubernetes configuration is set up to enable the utilization of the Kubernetes client for interaction with the Kubernetes Cluster, facilitating the creation, updating, or deletion of resources.<\/p>\n

\"\"<\/p>\n

As evident, errors are represented as values in this context, signifying the absence of harmless uncaught exceptions that might disrupt the program. The representation of errors as values necessitates handling errors when they arise.<\/p>\n

Additionally, a new instance of a Fiber App has been created at this juncture, accompanied by the establishment of Kubernetes clients essential for subsequent Kubernetes-related operations.<\/p>\n

Route Handlers<\/strong>
\nIn this section, a route correlated with a specific HTTP method is defined, and information is extracted from the HTTP request body. If one was curious about the inner workings of the helpers module and a desired to delve into the detailed implementation of these components and methods, access to the complete application code in the linked GitHub repository is provided below:\u00a0https:\/\/github.com\/iolap-blog\/mlops<\/a>.<\/p>\n

\"\"<\/p>\n

Goroutines<\/strong>
\nThis demonstrates the utilization of goroutines to execute various tasks concurrently. Essentially, a function is invoked with the \u2018go\u2019 keyword, and Golang manages the concurrent execution process.<\/p>\n

\"\"<\/p>\n

A channel has been established and passed to each goroutine to facilitate intercommunication among them. Channels serve as conduits through which values can be sent and received from both ends. By defining the channel\u2019s capacity, it implies that when the capacity limit is reached, the channel will block the goroutine until another goroutine reads a value from it.<\/p>\n

This method offers a straightforward and efficient means to achieve parallelism and\/or concurrency. In cases where multiple CPUs are available, genuine parallelism can be achieved seamlessly. Ultimately, a response is generated and dispatched to the client.<\/p>\n

Kubernetes Integration<\/strong>
\nAn illustration showcasing how a Kubernetes Deployment can be created using the Golang Client, along with the deployment process, is depicted below.<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

The utilization of channels is evident in the code to gather potential errors. Subsequently, an iteration over these errors is conducted to wait for the resources until their creation. This process is feasible due to the blocking nature of receiving values from channels in this context. Thus, it ensures the wait until either all resources are successfully created or until encountering errors that can be managed later in the code.<\/p>\n

\"\"<\/p>\n

 <\/p>\n

Docker Image<\/strong>
\nFinally, this is the method employed to generate a Docker image encapsulating the application, culminating in an image size of merely 31 MB. Once more, it showcases a swift and efficient approach to deploying a microservice onto a Kubernetes Cluster.<\/p>\n

\"\"<\/p>\n

Conclusion<\/h1>\n

Why Kubernetes<\/strong>
\nKubernetes is an excellent choice for building an ML Platform due to its versatility. It allows the creation of highly customized services, offering more flexibility than specific cloud providers. Additionally, Kubernetes is cost-efficient as it avoids the higher expenses associated with certain vendor platforms. Its cloud-agnostic nature means you can deploy your cluster anywhere\u2014across any cloud or on-premises\u2014avoiding vendor lock-in, which is vital for future planning.<\/p>\n

The uncertainty surrounding the choice of a cloud provider and the desire to avoid long-term lock-in further justify the use of Kubernetes. It grants complete control over microservices and overall architectural objectives.<\/p>\n

Why Go<\/strong>
\nThe use of Golang presents considerable advantages in this context, especially since Kubernetes itself is written in Golang. However, Python should not be overlooked. It\u2019s ideal for building ML models, rapid prototyping, and situations where its use is most logical. On the other hand, Golang is better suited for specific tasks where it surpasses Python, or for larger-scale projects. The key lies in discerning when to use a particular language and understanding the trade-offs involved.<\/p>\n

Why Us<\/strong>
\nWant to deploy and manage ML models at scale, automate and orchestrate your machine learning lifecycle, distribute your machine learning processes?\u00a0We can provide end-to-end machine learning solutions, design, build, and manage reproducible, testable, and evolvable ML software.\u00a0Don\u2019t hesitate to\u00a0contact us, we can help you drive your business forward.<\/p>\n","protected":false},"excerpt":{"rendered":"

Machine learning models are deployed to solve specific problems, provide value in various applications, and directly impact real-world outcomes.<\/p>\n","protected":false},"author":7,"featured_media":5121,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[33],"tags":[],"related_service":[49],"related_industry":[],"class_list":["post-5101","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-and-analytics"],"acf":[],"card":"

\n
\n \"\" <\/div>\n
\n
\n

\n Blog <\/p>\n \n

\n ML Model Deployment with Kubernetes and Go Language\n <\/h2>\n <\/a> <\/div>\n
\n
\n Machine learning models are deployed to solve specific problems, provide value in various applications, and directly impact real-world outcomes. <\/div>\n
\n

\n 20 November 2023 <\/p>\n

\n
\n Data and Analytics <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n<\/article>\n","_links":{"self":[{"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/posts\/5101","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/comments?post=5101"}],"version-history":[{"count":0,"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/posts\/5101\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/media\/5121"}],"wp:attachment":[{"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/media?parent=5101"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/categories?post=5101"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/tags?post=5101"},{"taxonomy":"related_service","embeddable":true,"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/related_service?post=5101"},{"taxonomy":"related_industry","embeddable":true,"href":"https:\/\/staging.elixirrdigital.com\/us\/wp-json\/wp\/v2\/related_industry?post=5101"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}