How to continuously train our pre-trained model on real time data?

Question

I have some sensors which fetch data from cement factory and sends data to AWS IoT. The data is then tested on pre-trained model and the model predicts quality of cement based on some parameters. The data is coming in one second interval.

Since the data is coming in real-time, I want to train the model incrementally in real time.

Can anybody suggest how train model continuously?

Look at Apache Flink.

erip
– erip

2019-05-10 13:52:57 +00:00
Commented May 10, 2019 at 13:52 — erip
– erip, Commented May 10, 2019 at 13:52

Venkatachalam · Accepted Answer · 2019-05-11 09:28:28Z

4

You could aggregate certain numbers of training data and then use .partial_fit() to update your model.

.partial_fit() is the incremental learning option, which is available in Sklearn.

If your incremental data would not fit in RAM, then its worth trying dask-ml wrapper for incremental learning.

edited May 11, 2019 at 9:28

answered May 10, 2019 at 14:28

Venkatachalam

17k10 gold badges52 silver badges78 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to continuously train our pre-trained model on real time data?

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related