I agree, it's a lot of memory... So I spent the past hours to understand what was happening.
I was only reusing small buffers

(like texture/vbo/shader ID generation)
I changed my texture loading code. Now I use a single buffer, reallocating it only when the current one is too small to fit the texture (and calling System.gc() to free the previous one).
Indeed, now it works without specifying any
MaxDirectMemorySize ...
In fact, here's what happened :in some cases, I have a lot of non-graphical stuff in memory, so i do need
-Xmx512m.
As I was using non-direct buffer, they were allocated within the JVM memory and everything was ok. (even using a huge set of buffers, disallocated automatically by the Garbage Collector.)
But when I switched to Direct Buffer, my code was really wrong !
Thanks for all the answers. It works just fine now !

Estraven.