Kubernetes Primitives versus other higher level abstractions in production

Kubernetes
#1

Hello, I took CKAD and am trying to apply learnings

Would it be considered weird if i deploy components (ingestion, validation, prepreocessing, feature generation, training, model pusher, etc…) of ML pipelines as microservices using Kubernetes Primitives? I i ideally wanted to use Kubeflow pipelines but installing/getting Kubeflow up and running on a self-managed ARM64 cluster cluster is so hard! I gave up.

the easier route is to package each component of my training pipeline as docker image and expose it as clusterip service. Would it be considered **naive ** to use primitives in real life instead of other stuff like kubeflow, zenml, etc?

I’d appreciate any insight - small or big! Thanks.

#2

I haven’t used kubeflow, although I’ve done a bit of ML. I am not surprised that setting Kubeflow up on a cluster takes some doing.

Using a tool like kubeflow is always a trade-off, since you’re dealing with the complexities of a general purpose ML tool platform along with the complexities of the task. And setting up your own custom pipeline using commits you to the task of creating and maintaining your own custom solution, in addition to the actual tasks of ML processing of your models, your features, and your data. I wouldn’t call that “naive” per se, but it could be more work than you think.

1 Like