URGENT: Frequent expensive full GCs(after first full GC)with JDK_1.5.0_06

On load testing the application we see that

- everything works fine for the first 35 minutes till we hit the first full GC

after the first full GC, which happened after 35 minutes (after about 500 minor GCs), strangely the full GCs happen very frequently, once about every 2-3 minutes (once every 5-6 minor GCs).This degrades the performance significantly

- The frequency of minor GCs remains the same, about 1 every 15-30 seconds.

- Monitoring using visualGC and JConsole shows that till the first full GC, the minor GC was collecting a majority of the newly allocated objects with only a few spilling over to the Survivors and subsequently to the old gen. Hence the old gen was filling up very slowly, and it took 35 minutes for it to fill up and cause the first full GC.

- However, immediately after this first full GC (Note that the load exposed to the servers remained the same), it seems that the minor GCs started collecting very less objects and a huge number of objects were spilled over to the Survivors (which got completely filled up) and subsequently to the old gen which also got filled up quickly within 5-6 minor GCs, leading to frequenty major GCs, which were very expensive (took ~20 seconds)

Here is our JVM parameters:

-XX:+PrintGCDetails -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCTimeStamps -

XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -Xms2560m -Xmx2560m -XX:NewSize=896m -XX:MaxNewSize=896m

-XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=8 -XX:InitialSurvivorRatio=8 -XX:MinSurvivorRatio=8 -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=512m

This is an urgent issue and help is very much appreciated

[1761 byte] By [SRENGAN123a] at [2008-2-20]
# 1
Hello,Is this a clustered environment? And do you see this issue in all the nodes in your cluster?Thanks.
p_ra at 2007-7-12 > top of java,Java HotSpot Virtual Machine,HotSpot Internals...
# 2
Yes, we are running on a clustered environment. We see this behavior consistent in all the nodes in the cluster.
SRENGAN123a at 2007-7-12 > top of java,Java HotSpot Virtual Machine,HotSpot Internals...