I have an on-prem multinode cluster. I have 4 docker image files - 1 for data_ingestion, data_preprocessing, feature_engineering and modeltraining. I deployed them as separate pods and exposed them to 4 different services. But in my code how to pass the output produced by 1 microservice as input into another. e.g. Output of data ingestion should be fed to data_preprocesing service and then output of preprocessingservice goes to feature engineering service etc?
If the apps are deployed independently, how do they pass data to each other?..
There are certain software “motherhoods” that go with various software styles, and if you follow those, you’d think that a single microservice should mange its own data and communicate by message to other microservices.
This probably is NOT the best way to handle a data processing application, I think, so throw the dogma out the window. I’m not sure how best to package these kinds of calculations.
Usually microservices are implementation agnostic, but at least in my own experience of data analysis, things go better when you stick to a single ecosystem (like python’s Pandas and related technologies) to represent that kind of data and do operations upon it. Microservices are very hip right now, but they might be an anti-pattern for what you’re trying to do. There may be tech that helps, but I haven’t done much data analysis that’s designed for making this stuff scale or parallelize.
1 Like