33
votes
Methods to increase the amount of data sent in a packet
The main method to reduce your packet size is to use a binary encoding of your data.
If your sensors produce the data in a binary format, note the specifications of the format, such as the number of ...
18
votes
Methods to increase the amount of data sent in a packet
With all due respect, your data is boring.
So take advantage of that.
Leverage its very boring nature.
I recommend you send such data in routine Update messages.
And that you occasionally send ...
- 7,997
10
votes
Accepted
UTF-8 questions
Answer to Question 1: why is it that the first 2 bits of every continuation byte are set to “10”?
It lets you land at a random place in the sequence and unambiguously work back to the beginning of ...
- 1,034
8
votes
Methods to increase the amount of data sent in a packet
Many compression schemes use the fact that consecutive samples are often closely related, and the differences may use less bits than original data. If you transmit one full data packet followed by a ...
- 18.6k
5
votes
Methods to increase the amount of data sent in a packet
There are many binary formats for structured data. Wikipedia has a list. Some examples are EXI, CBOR, FlatBuffers, Protocol Buffers.
The are simpler to implement than a compressor, and simpler to ...
- 694
4
votes
Methods to increase the amount of data sent in a packet
Your json has an advantage over binary. Things can be left out. If, for example, the Accelerometer has nothing new to say durring this sample you can leave it out.
However, if GPS, Accelerometer, temp,...
- 120k
3
votes
Methods to increase the amount of data sent in a packet
Consider zstd for compression. It has custom dictionaries, besides many other good things. Killer feature is you can train a dictionary on all those "JSON" packets collected earlier, getting ...
3
votes
Compressing EBCDIC file vs UTF8
First thoughts
EBCDIC is a format with an 8 bit encoding. The basic EBCDIC character set has plenty of unused space, so not all the potential of the 8 bits is used. But it has codepages for handling ...
- 82.3k
3
votes
Reduce number of digits by converting to alphanumeric data
This is precisely why storing business defined identifiers as database identifiers becomes problematic. The business changes their rules, and this forces downstream changes in databases.
If you were ...
- 46.6k
3
votes
Accepted
Name of data structure where a string determines its value
Type-Length-Value or Tag-Length-Value.
Both protocols describe a mechanism for encoding optional information in multiple tuplets of variable length into text-based messages, using three fields: type, ...
- 201k
2
votes
Reduce number of digits by converting to alphanumeric data
Encode the number as base 64 using a fixed size representation and restrict one of the characters to be non-numeric to prevent collisions with existing identifiers. Base 64 uses 'A'-'Z' to represent 0-...
- 151
2
votes
Accepted
How to identify encoding of a text string?
@MartinGrey, I think your question is perfectly sensible, but it's like Bob says in a comment, your misconception is in the idea that handling arbitrary data in arbitrary encodings, and decoding it ...
- 12.7k
1
vote
Methods to increase the amount of data sent in a packet
Multiply each number by a fixed number of decimals to remove the decimal point. Encode each in a fixed number of binary bits big enough for every case. If you need ASCII then use base-85 encoding.
If ...
- 49.4k
1
vote
Methods to increase the amount of data sent in a packet
Some interesting answers here, but as far as I can see, any of them misses the elephant in the room: you are already using an LZW compression algorithm, which is eliminating most of the redundancies ...
- 221k
1
vote
Reduce number of digits by converting to alphanumeric data
Unclear elements
It would be useful to clarify the following points:
Is the - currently stored in the string ?
Is the length control on the client side based on the visible string or on the ...
- 82.3k
1
vote
Accepted
Reduce number of digits by converting to alphanumeric data
When this really solves the problem, why not? Sometimes, a pragmatic solution with some "duct tape" is required to get things done, even if it does not look beautiful from a design perspective.
...
- 221k
1
vote
Accepted
Dictionary Coding for OLAP
Let us start with your "products" as a representative example: OLAP databases which involve products are typically used to create queries like
which are the top 10 selling products over the last 3 ...
- 221k
1
vote
Windows compatibility with Unix/Linux newline "\n"
This question is really about a software application's "customer base".
To answer your question, you have to know whether your customers might be inconvenienced if your application generates output ...
- 17.2k
1
vote
Windows compatibility with Unix/Linux newline "\n"
As far as Windows and C# is involved you can always use the Environment.Newline
to determine the default new line character of the system the program is ran on.
also, you can use text.Replace("\n",...
- 139
1
vote
Windows compatibility with Unix/Linux newline "\n"
Windows Notepad (notepad.exe) doesn't interpret a standalone \n as a new line. It's not necessarily "modern" but pretty much "mainstream".
If you're writing text files, every day user should be able ...
- 1,509
1
vote
How is encoding handled correctly during copy-paste between programs?
A program transferring text can do one of three things: Transfer without telling which encoding is used, transfer using a standard encoding, or transfer while specifying the encoding.
The receiving ...
- 49.4k
Only top scored, non community-wiki answers of a minimum length are eligible