I simulated a scenario in Java that leads to severe glibc memory fragmentation.
- Step one: simulate a multithreaded environment with 600 threads.
- Step two: every second, start two new threads that read a large file via
java.nio.channels.FileChannel. - Step three: after 10 seconds, terminate automatically and let GC perform the cleanup.
This program will keep using memory from the non-main arena regions until the maximum number of arenas is reached, at which point memory usage stabilizes. If you then stop all threads and leave only the main thread sleeping, you’ll find that the process’s RSS metric does not decrease and remains high.
- According to the NMT report, you’ll find that the JVM’s accounting does not show using that much memory.
- From the Linux pmap and smaps information, you’ll see that each 64 MB block in the non-main arena regions has about 8 MB touched in physical memory. If you try to read the binary data from memory, you’ll find that all of it is content from your file.
The most fascinating part is glibc’s parameter behavior. When I try using the MALLOC_TRIM_THRESHOLD_ environment variable, regardless of the value I set, RSS reclamation begins to occur. Since the DirectBuffer used to read the file is 8 MB, I tried 52,428,800 (50 MB) and 8,388,608 (8 MB). Each time, the process’s RSS would reclaim around 16 MB. I once used gdb to inspect the process’s ptmalloc2 allocator trim_threshold parameter, the default value exists and is around 48 MB. After configuring the MALLOC_TRIM_THRESHOLD_ variable, the reclamation behavior changes completely, especially for memory in the non-main arena regions.
By the way, the JDK version I used is 21-openjdk, and the glibc version is 2.39.
On each file read, the process calls malloc to request off-heap memory, and after the thread dies, the GC thread initiates the free operation. Therefore, in the JVM dump you can observe that all objects, including off-heap DirectBuffers, are being destroyed in order.
My main questions are twofold: why does this parameter affect the drop in memory usage? In addition, when it is set to 50 MB, even before theoretically reaching the 50 MB reclaimable threshold, glibc also returns memory from the non-main arena to the OS. It seems that after this parameter is set, glibc’s free behavior has changed?
To add: this is a JVM runtime parameter, and it’s simple.
-Xms6g -Xmx6g -XX:MaxDirectMemorySize=5g -XX:+AlwaysPreTouch -XX:+UseG1GC -Xlog:gc:file=gc.log:time,uptime,level,tags:filecount=5,filesize=100M*
import java.io.RandomAccessFile;
import java.io.UnsupportedEncodingException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.util.ArrayList;
import java.util.List;
public class FileRead1 {
private static volatile boolean stop = false;
public static void main(String[] args) throws InterruptedException {
liveThreadLoopWithoutMemory();
int count = 0;
while (true) {
if (count > (Integer.MAX_VALUE -1) ) {
break;
}
count++;
try {
for (int i = 0; i < 2; i++) {
newThread();
}
for (int i = 0; i < 5; i++) {
Thread.sleep(1000L);
}
} catch (InterruptedException e) {
}
}
stop = true;
while (true) {
for (int i = 0; i < 5; i++) {
Thread.sleep(1000L);
}
}
}
private static void liveThreadLoopWithoutMemory() {
for (int i = 0; i < 600; i++) {
new Thread(() -> {
while (true) {
try {
Thread.sleep(100L);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
if (stop) {
break;
}
}
}).start();
}
}
private static void newThread() {
Thread thread = new Thread(new Runnable() {
private final List<String> strings = new ArrayList<>();
@Override
public void run() {
test(strings, "test.txt", "UTF-8", false);
strings.clear();
}
});
thread.setDaemon(true);
thread.start();
}
private static void test(List<String> strings, String src, String encoding, boolean hasTitle) {
test1(strings, src, encoding, hasTitle);
}
private static void test1(List<String> strings, String src, String encoding, boolean hasTitle) {
test2(strings, src, encoding, hasTitle);
}
private static void test2(List<String> strings, String src, String encoding, boolean hasTitle) {
test3(strings, src, encoding, hasTitle);
}
private static void test3(List<String> strings, String src, String encoding, boolean hasTitle) {
test4(strings, src, encoding, hasTitle);
}
private static void test4(List<String> strings, String src, String encoding, boolean hasTitle) {
test5(strings, src, encoding, hasTitle);
}
private static void test5(List<String> strings, String src, String encoding, boolean hasTitle) {
test6(strings, src, encoding, hasTitle);
}
private static void test6(List<String> strings, String src, String encoding, boolean hasTitle) {
test7(strings, src, encoding, hasTitle);
}
private static void test7(List<String> strings, String src, String encoding, boolean hasTitle) {
test8(strings, src, encoding, hasTitle);
}
private static void test8(List<String> strings, String src, String encoding, boolean hasTitle) {
test9(strings, src, encoding, hasTitle);
}
private static void test9(List<String> strings, String src, String encoding, boolean hasTitle) {
test10(strings, src, encoding, hasTitle);
}
private static void test10(List<String> strings, String src, String encoding, boolean hasTitle) {
new TaskReadFile() {
@Override
public void process(String str) {
super.process(str + "swk");
}
}.readFile(strings, src, encoding, hasTitle);
}
static class TaskReadFile implements ReadFileInterface {
private final List<String> stringList = new ArrayList<>();
@Override
public void process(String str) {
stringList.add(str + "123");
}
@Override
public void clear() {
stringList.clear();
}
}
interface ReadFileInterface {
void process(String str);
void clear();
default void readFile(List<String> strings, String src, String encoding, boolean hasTitle) {
int readLimit = 8 * 1024 * 1024;
int startIndex = 0;
try (RandomAccessFile raf = new RandomAccessFile(src, "r");) {
FileChannel channel = raf.getChannel();
ByteBuffer rbuf = ByteBuffer.allocate(readLimit);
synchronized (rbuf) {
channel.position(startIndex);
byte[] temp = new byte[0];
int LF = 10;
long lineCount = 0;
while (channel.read(rbuf) != -1) {
int position = rbuf.position();
byte[] rbyte = new byte[position];
rbuf.flip();
rbuf.get(rbyte);
int startnum = 0;
for (int i = 0; i < rbyte.length; i++) {
if (rbyte[i] == LF) {
if (channel.position() == startIndex) {
startnum = i + 1;
} else {
if (hasTitle && 0 == lineCount) {
startnum = i + 1;
lineCount++;
continue;
}
int lineLen = i - startnum + 1;
byte[] line = new byte[temp.length + lineLen];
System.arraycopy(temp, 0, line, 0, temp.length);
System.arraycopy(rbyte, startnum, line, temp.length, lineLen);
startnum = i + 1;
temp = new byte[0];
String str = trimEndingCRLF(line, encoding);
strings.add(str);
process(str);
}
}
}
if (startnum < rbyte.length) {
byte[] temp2 = new byte[temp.length + rbyte.length - startnum];
System.arraycopy(temp, 0, temp2, 0, temp.length);
System.arraycopy(rbyte, startnum, temp2, temp.length, rbyte.length - startnum);
temp = temp2;
}
rbuf.clear();
}
}
rbuf.clear();
try {
Thread.sleep(10000L);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
clear();
} catch (Exception e) {
throw new RuntimeException();
}
}
default String trimEndingCRLF(byte[] line, String encoding) throws UnsupportedEncodingException {
if (line.length != 0 && line[0] != 10) {
int lastIdx = line.length - 1;
if (line[lastIdx] != 10) {
++lastIdx;
} else if (line[lastIdx - 1] == 13) {
--lastIdx;
}
return new String(line, 0, lastIdx, encoding);
} else {
return "";
}
}
}
}
```java
RandomAccessFiledetour when creating aFileChannelfor far more than a decade now. Further, all those array allocations and copy operations are unnecessary and distracting from the purpose of the code, read into aByteBuffer, identify the boundaries within that buffer, create the string from the buffer. Or just use the built-in methods for reading lines from a channel (Assuming you need the channel for demonstration purposes, as otherwise,Files.readAllLines(Path.get(src))will do the entire job for you).