Questions tagged [stream-processing]
Stream processing is consuming an input stream or sequence of bytes with some format, processing the data as it arrives, and translating it into another format in a related output stream.
88 questions
2
votes
2
answers
152
views
For distributed workflows - tradeoffs having stateless workers operate on central state, versus stateful workers?
I'm working on a problem right now that processes incoming data at a very high rate. Each event that flows in has an association ID, and each group of associated events will affect behaviour over time ...
- 141
1
vote
3
answers
185
views
Design of ETL system that should receive both files and via api
I am tasked to design a system that should receive data either as files or via an API and perform ETL functions. The end result is put into an RDBMS.
For the sake of example, imagine a system that ...
- 145
3
votes
4
answers
1k
views
Is real-time processing a synonym for streaming?
I watched two YouTube videos where in:
1st one - there are concepts of batch processing and stream ingestion and in the
2nd one - there is a comparison between batch and real time processing.
Is it ...
- 261
1
vote
2
answers
208
views
Serialization pattern and code duplication
I noticed how our codebase contains multiple versions of the same method, which unmarshalls the inbound byte-stream into java objects and that the only difference between the current and previous ...
- 141
-1
votes
1
answer
117
views
"Select All" in Table is now sending too much data to Server via API - Alternative Ways to Send to Server?
Outline of the current architecture of our web app outlining the issue I'm seeing
Client-side app is React, talking to a server running the Play! framework via an API.
On the page is a table that ...
- 99
0
votes
0
answers
74
views
How to achieve high performance routing to Kinesis streams?
I have consumer pools that consume only events matching a specific attribute (e.g. origin=WINDOWS, the other origin=LINUX).
Since ordering is important (ordering is based on instanceId, so this is the ...
user374484
2
votes
4
answers
955
views
How is processing done when streaming data?
If streams in programming is about reading data in chunks, how then is it possible to process the data in the scenarios where the data cannot be processed bit by bit, but the processing logic needs ...
- 171
3
votes
1
answer
975
views
Reprocessing out of order events in a stateful stream
Say that you have a monitoring application that reports downtimes for a large number of remote systems (think IOT). A monitoring daemon polls the remote system and reports that status (on/off) via an ...
- 756
1
vote
2
answers
299
views
API for input stream operation read until
I want to create a stream class The input stream should read/parse a continuous range from left to right providing convenience methods. The implementation isn't a problem but choosing a consistence ...
- 119
1
vote
1
answer
98
views
Showing posts by their time-decayed score
I want to show a number of posts, ordered by their score, limited to some number. The score is the product of the number of likes they have (from another table) and the recipricol of their normalized (...
- 630
0
votes
0
answers
71
views
Bigdata architecture: persisting raw data and streaming them at the same time
In a case where we have multiple source of data, we need to :
persist each source of data in its raw form.
process the data source eventually with each other to transform them in a presentable state (...
- 129
-1
votes
1
answer
91
views
Design for Maintaining changes over time which supports arbitrary updates in different time points
I have a collection of documents, which hold a subject id, a timestamp and a value. For example:
{ sid: 1, t: 3, v: "A" }
{ sid: 1, t: 5, v: "B" }
Which means subject#1 is ...
- 1,139
-3
votes
1
answer
429
views
How are different resolution settings for videos handled by server and client based on setting and connection speed? [closed]
I was watching youtube and squinted at the text in the video and wondered what happens when I switch from 720p to 1080p.
Does the streaming service, Youtube and or any other, just have the files in ...
0
votes
2
answers
82
views
Do 1:1 mobile video call face filters belong on Client or Server?
I am drafting architecture for 1:1 video calling mobile app with face filters (face recognition).
I could use p2p e.g. WebRTC because more savings less complexity. However, continuous face rec during ...
- 103
0
votes
1
answer
65
views
System design to store pieces of data with specific ttyl to then refetch
I have a system which generates pieces of data that needs to be tracked and assigned a specific TTYL value (a value in seconds) for how long they are supposed to live before being re-processed.
I ...
- 109