Skip to content
This repository was archived by the owner on Feb 16, 2024. It is now read-only.

[BAHIR-183] [WIP] HDFS based MQTT client persistence#84

Open
lukasz-antoniak wants to merge 1 commit into
apache:masterfrom
lukasz-antoniak:BAHIR-183-2
Open

[BAHIR-183] [WIP] HDFS based MQTT client persistence#84
lukasz-antoniak wants to merge 1 commit into
apache:masterfrom
lukasz-antoniak:BAHIR-183-2

Conversation

@lukasz-antoniak

@lukasz-antoniak lukasz-antoniak commented Mar 28, 2019

Copy link
Copy Markdown
Member

As discussed in the JIRA ticket (https://issues.apache.org/jira/browse/BAHIR-183), revisited version of HDFS persistence of MQTT messages, which should ease recovery in case of device failure. The design discussion may be continued in JIRA, but I decided to open WIP PR for @yanlin-Lynn to review and comment.
@yanlin-Lynn

yanlin-Lynn commented Apr 9, 2019

Copy link
Copy Markdown
Contributor

Sorry to keep you wait, I will review this PR in a few days.
@Brahim13brahim

Brahim13brahim commented May 20, 2019

Copy link
Copy Markdown

yes
@lresende

lresende commented Jun 8, 2019

Copy link
Copy Markdown
Member

@lukasz-antoniak what's the status of this pr, is this ready for final review/merge?
@lukasz-antoniak

lukasz-antoniak commented Jun 17, 2019

Copy link
Copy Markdown
Member Author

@lresende, I would answer positive. This PR changes the way we implement HDFS persistence due to issues pointed out in JIRA commant.
@yanlin-Lynn, can you also have a look since I changed mostly the code you have introduced and worked on previously?
@yanlin-Lynn

yanlin-Lynn commented Jun 17, 2019

Copy link
Copy Markdown
Contributor

@lresende, I would answer positive. This PR changes the way we implement HDFS persistence due to issues pointed out in JIRA commant.
@yanlin-Lynn, can you also have a look since I changed mostly the code you have introduced and worked on previously?

Very sorry to keep you wait. I will review it in weekends.
yanlin-Lynn
yanlin-Lynn reviewed Jun 23, 2019
View reviewed changes
Comment thread sql-streaming-mqtt/src/main/scala/org/apache/bahir/sql/streaming/mqtt/MessageStore.scala

override def put(key: String, persistable: MqttPersistable): Unit = {
try {
val path = getPath(key)

@yanlin-Lynn yanlin-Lynn Jun 23, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just worry about the performance of creating a file for each coming message.
@yanlin-Lynn

yanlin-Lynn commented Jun 23, 2019

Copy link
Copy Markdown
Contributor

And, I think we should better keep the ability to do flow-control, eg, control max number of message in each batch.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

4 participants

@lukasz-antoniak @yanlin-Lynn @Brahim13brahim @lresende