cd to the directory of your jboss's bin, and run
cd
jboss-cli connect /core-service=platform-mbean/type=memory:gc exit
You might also want to query the memory usage before and after the GC by
/core-service=platform-mbean/type=memory:query