To be clear, options aren't being handled differently from any other *pre-existing* user-defined type.
utop # type 'a my_type = One of 'a | Two of 'a * 'a;;
type 'a my_type = One of 'a | Two of 'a * 'a
utop # type node = int * tree * tree and tree = node my_type;;
Error: The type abbreviation node is cyclic
Using a real ADT instead of a type abbrevation works. David Allsopp already gave one way you can do it, here is another which preserves the use of the option type:
utop # type node = Node of int * tree * tree and tree = node option;;
type node = Node of int * tree * tree
and tree = node option
Note that here [tree] is still a type abbreviation, but [node] is not, so the cycle is not a problem.
Note also that your original example works fine with the -rectypes flag. But my general opinion is that if you need -rectypes to compile your code, you should write different code :)