Created: 2022-10-31 21:10:49 Tags: #ML #programming #DataScience
# Note
What is end-to-end ML project? When i talk about “end-to-end” i mean these steps you need to go through before your project is ready:
- Create infrastructure for your project:
- Set up remote server
- Deploy all needed tools such as Jupyter (or Jupyterhub) for developing and quickly experiments running, MLFlow for saving models and experiments results, your own Docker registry, create K8S infrasctructure (if needed), Airflow for etl processes (data collection and processing), …
- Choose your system type: online (real-time system) or offline (batch) system.
- Choose a problem you want to solve (text santiment, face detection, analyze user behaviour, online/offline recommender system, …)
- Find or collect data and choose where you want to store data
- Preprocess data and generate features
- Experiments with model: training, validation, testing
- Deploy ML Pipeline (deploy with Docker Containers, create Airflow DAG, deploy with API, …)
- Set up model retraining if needed
- Create Web UI, or Mobile app, or API for your model
- Set up tools for monitoring your system (you can use Graphana for ex.):
- Data Quality
- ML metrics
- Business metrics such as ARPU, PU, DAU, ARPPU, LTV, CLV and others
After these steps you have end-to-end ML system with data collection, preprocessing, monitoring and model retraining.