Just to be clear: does this need to work on Windows, or only on Unix-like systems? If it only needs to work under Unix, then your problem sounds to me like a standard inter-process communication problem with the usual solutions being to use a combination of pipe(), dup2(), and select() system calls, and with the alternative solution being to use a pseudo-terminal. Chapter 5 of "Unix System Programming in OCaml"
https://ocaml.github.io/ocamlunix/pipes.html should give you some guidance on implementing the former solution. If you want to use pseudo-terminals, I don't know of a freely-available tutorial-style reference off the top of my head. "Advanced Programming in the Unix Environment" by late Richard Stevens has had a chapter dedicated to pseudo-terminals since the first edition of the book: perhaps your local library has a copy. Failing that, you could read Linux man page for pty(7), and follow the links to forkpty and friends. For example, you could start here:
http://man7.org/linux/man-pages/man7/pty.7.html.
The fact that you want to do all of this from Matlab is an extra challenge, but I don't think it's an insurmountable challenge. Even if Matlab doesn't have the functions you need "out of the box", it is possible to extend Matlab with C code, so you could implement a C function to initialize the external process, and another function to send to the external process and receive results. The following is probably a good starting point for learning how to extend Matlab with C/C++:
https://www.mathworks.com/help/matlab/call-mex-files-1.html. I don't know off the top of my head if Matlab already comes with sufficiently flexible functionality for communicating with an external process: you could try searching the documentation and/or engaging Mathworks technical support (assuming your license comes with technical support) and/or asking on Mathworks forums.
--
Best of luck
Zhenya