Increasing memory given to maven using MAVEN_OPTS did not solve the problem. As this setting just increases the memory given to the build process and not the individual test cases. The solution was to use the following configuration in the surefire plugin.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<forkMode>pertest</forkMode>
<argLine>-Xms256m -Xmx512m</argLine> <testFailureIgnore>false</testFailureIgnore>
<skip>false</skip>
<includes>
<include>**/*IntegrationTestSuite.java</include>
</includes>
</configuration>
</plugin>
The pom for the integration test can be found here. This fixed the heap space issue and now we have our test cases running on the nightly build. You can see the status of our nightly builds at http://builder.wso2.org/browse/MASHUP-NIGHTLY