You are right, you simply have to add the agent war to the plugin's options (along with a context):
<plugin>
<artifactId>maven-embedded-glassfish-plugin</artifactId>
...
<executions>
<execution>
<id>jolokia-deploy</id>
<phase>pre-integration-test</phase>
<configuration>
<app>
${settings.localRepository}/org/jolokia/jolokia-war/0.80/jolokia-war-0.80.war
</app>
<contextRoot>jolokia</contextRoot>
</configuration>
<goals>
<goal>deploy</goal>
</goals>
</execution>
</executions>
</plugin>
This will deploy jolokia to your embedded glassfish during integration testing (assumed that the war has been already downloaded to the local repository)
... roland