I run tight loops (game engine -- 60fps with a lot of memory activity and allocations each frame), and the GC works remarkably well at keeping things sane. I did have a problem with runaway allocations once, and tracked it down to a source of allocations which was effectively never un-referenced... so a legitimate leak.
If you do a Gc.full_major (), is your memory returned? If not, then I think that's evidence that there's still some handle on it -- be sure the appropriate values have fallen out of scope and aren't referenced in some other way!
On the other hand, if this is just the GC not cleaning up quick enough for your case... I'm sorry I have no help for tuning the GC. The documentation in gc.mli seemed pretty sensible when I looked at it (a while ago) though!