MappedByteBuffers on 32-bit vs. 64-bit systems
Hi all,
In the course of performance testing and optimization of my large-file, I/O intensive application, it has become apparent that it could benefit from switching the stream-based input to a NIO MappedByteBuffer approach. The micro-benchmarks look good (5x speed improvement on read operations) and I am working on a production implementation and real-world benchmarks.
One thing that has come up is that there is a fixed number of bytes that can be mapped. Once that number is reached, OS-level IOExceptions are raised. ("Cannot allocate memory" on OSX, something similar on XP.) That number seems to fit this equation:
number of bytes than can be mapped < (2^32 - physical ram - virtual memory)
This makes sense to me, although I have not read anything on MappedByteBuffers that says as much. Unfortunately for me, that number is too small--if I were to put it into production, problems would arise at some point. Not now, but surely some day.
The question: On a 64-bit system (64-bit CPU and OS), would the total bytes mapped with MappedByteBuffer increase also? I don't have access to a 64-bit system and cannot test. But I am lucky enough to be able to specify the hardware requirements of my application.
If someone knows the answer, please edumacate me!
If someone here has a 64-bit CPU running a 64-bit OS, and the curiosity, could you run my test code on your system to see what happens? It should not take long. Just use a large file, and set the 'factor' variable to something wherein file.length() * factor > 4GB. Please include your JVM details as MBBs are vendor specific with regards to MBB implementation. I am most keen to learn about what the maximum number of bytes I can map for Linux on Intel/AMD systems.
And, if anyone else has any thing educational to add, please do! There seems to be scant information about MappedByteBuffers on the net.
Cheers,
Stu
privatestaticvoid nioMbbMultiMap()throws IOException{
int factor = 23;
File file =new File("/path/to/largeFile.bin");
//file size * factor = total mapped bytes
System.out.println("File size: " + file.length());
System.out.println("# of MBBs: " + factor);
System.out.println("Total mapped bytes: " + file.length() * factor);
FileChannel[] fc =new FileChannel[factor];
for (int i = 0; i < factor; i++)
fc[i] =new FileInputStream(file).getChannel();
MappedByteBuffer[] mbb =new MappedByteBuffer[factor];
for (int i = 0; i < factor; i++){
System.out.println("MMB #" + i);
mbb[i] = fc[i].map(FileChannel.MapMode.READ_ONLY,0, file.length());
//if there is a problem, it will blow up on the above line
}
for (int i = 0; i < factor; i++)
fc[i].close();
}
