[musl] What determines the TERM variable value?

[musl] What determines the TERM variable value?

[Yuri Kanivetsky]

Hi,

I was told recently that I can set TERM to any value inside a docker
container, which is, sort of, at odds with my understanding.

And my understanding is as follows. When a program creates a
pseudoterminal (a pty master/slave pair), it sort of becomes a
terminal emulator. I guess, it can decide not to process any escape
sequences in which case the pair is probably not much different from
an ordinary pipe. And basically what sequences it decides to process
determines the TERM variable value.

I can separate such programs into 2 categories:

* Terminal emulators (xterm, urxvt, ...). They receive input, process
escape sequences, and draw the result in a window. They can invent
their own language (escape sequences), but it's probably best to have
some terminal as a base.

* The rest (docker, ssh, tmux, screen, ...). They receive input,
translate escape sequences to the language of the process up the chain
(by using the TERM variable and the terminfo database), and pass the
result to stdout (text, optionally with translated escape sequences).

So, generally you have a chain of processes connected via
pseudoterminals (a pty master/slave pairs). E.g. xterm <-> ssh <->
tmux <-> docker.

Also, you can't set TERM to an arbitrary value. Each program that
creates a pseudoterminal supports a fixed set of values. E.g. the tmux
documentation says:

> For tmux to work correctly, this must be set to screen, tmux or a derivative of them.

https://man.archlinux.org/man/community/tmux/tmux.1.en

Is my understanding correct?

Also, I have a pretty vague understanding of what the TERM variable
affects. Can you give some examples? Or categorize things in some way?
Is it only about escape sequences?

Regards,
Yuri

Re: [musl] What determines the TERM variable value?

[Rich Felker]

On Fri, Feb 11, 2022 at 03:30:32PM +0200, Yuri Kanivetsky wrote:
> Hi,
> 
> I was told recently that I can set TERM to any value inside a docker
> container, which is, sort of, at odds with my understanding.
> 
> And my understanding is as follows. When a program creates a
> pseudoterminal (a pty master/slave pair), it sort of becomes a
> terminal emulator. I guess, it can decide not to process any escape
> sequences in which case the pair is probably not much different from
> an ordinary pipe. And basically what sequences it decides to process
> determines the TERM variable value.

No, creating the pty pair does not make you a terminal emulator any
more than a modem or null modem cable is a terminal emulator. The pty
is just a data channel with kernel support for certain types of data
translation/interpretation and very basic line buffering and editing
(if enabled). This layer has nothing to do with terminal semantics.

> I can separate such programs into 2 categories:
> 
> * Terminal emulators (xterm, urxvt, ...). They receive input, process
> escape sequences, and draw the result in a window. They can invent
> their own language (escape sequences), but it's probably best to have
> some terminal as a base.
> 
> * The rest (docker, ssh, tmux, screen, ...). They receive input,
> translate escape sequences to the language of the process up the chain
> (by using the TERM variable and the terminfo database), and pass the
> result to stdout (text, optionally with translated escape sequences).

tmux and screen *are* terminal emulators who use *another terminal*
(the one they're attached to) as a presentation layer for showing
what's on the terminals (one for each window) they're emulating
internally. They're not just data channel carriers.

> So, generally you have a chain of processes connected via
> pseudoterminals (a pty master/slave pairs). E.g. xterm <-> ssh <->
> tmux <-> docker.
> 
> Also, you can't set TERM to an arbitrary value. Each program that
> creates a pseudoterminal supports a fixed set of values. E.g. the tmux
> documentation says:
> 
> > For tmux to work correctly, this must be set to screen, tmux or a derivative of them.
> 
> https://man.archlinux.org/man/community/tmux/tmux.1.en
> 
> Is my understanding correct?
> 
> Also, I have a pretty vague understanding of what the TERM variable
> affects. Can you give some examples? Or categorize things in some way?
> Is it only about escape sequences?

Pretty much, yes. The TERM environment variable simply tells the
program you're invoking what dialect of terminal escapes to use
(normally found by looking it up in the terminfo/termcap database) and
what to expect as input when different special keys are pressed. It's
informing the application of the contract you want it to honor with
whatever is on the other side of the tty channel.

Rich

Re: [musl] What determines the TERM variable value?

[Yuri Kanivetsky]

So, a program is a terminal emulator if it translates escape sequences
in both directions? And if it, as a result, changes the TERM variable?

What seems strange to me is that docker changes the TERM variable:

$ echo $TERM
screen-256color

$ docker run --rm -it alpine
/ # echo $TERM
xterm

Does that qualify it as a terminal emulator?

On Fri, Feb 11, 2022 at 5:54 PM Rich Felker <dalias@libc.org> wrote:
>
> On Fri, Feb 11, 2022 at 03:30:32PM +0200, Yuri Kanivetsky wrote:
> > Hi,
> >
> > I was told recently that I can set TERM to any value inside a docker
> > container, which is, sort of, at odds with my understanding.
> >
> > And my understanding is as follows. When a program creates a
> > pseudoterminal (a pty master/slave pair), it sort of becomes a
> > terminal emulator. I guess, it can decide not to process any escape
> > sequences in which case the pair is probably not much different from
> > an ordinary pipe. And basically what sequences it decides to process
> > determines the TERM variable value.
>
> No, creating the pty pair does not make you a terminal emulator any
> more than a modem or null modem cable is a terminal emulator. The pty
> is just a data channel with kernel support for certain types of data
> translation/interpretation and very basic line buffering and editing
> (if enabled). This layer has nothing to do with terminal semantics.
>
> > I can separate such programs into 2 categories:
> >
> > * Terminal emulators (xterm, urxvt, ...). They receive input, process
> > escape sequences, and draw the result in a window. They can invent
> > their own language (escape sequences), but it's probably best to have
> > some terminal as a base.
> >
> > * The rest (docker, ssh, tmux, screen, ...). They receive input,
> > translate escape sequences to the language of the process up the chain
> > (by using the TERM variable and the terminfo database), and pass the
> > result to stdout (text, optionally with translated escape sequences).
>
> tmux and screen *are* terminal emulators who use *another terminal*
> (the one they're attached to) as a presentation layer for showing
> what's on the terminals (one for each window) they're emulating
> internally. They're not just data channel carriers.
>
> > So, generally you have a chain of processes connected via
> > pseudoterminals (a pty master/slave pairs). E.g. xterm <-> ssh <->
> > tmux <-> docker.
> >
> > Also, you can't set TERM to an arbitrary value. Each program that
> > creates a pseudoterminal supports a fixed set of values. E.g. the tmux
> > documentation says:
> >
> > > For tmux to work correctly, this must be set to screen, tmux or a derivative of them.
> >
> > https://man.archlinux.org/man/community/tmux/tmux.1.en
> >
> > Is my understanding correct?
> >
> > Also, I have a pretty vague understanding of what the TERM variable
> > affects. Can you give some examples? Or categorize things in some way?
> > Is it only about escape sequences?
>
> Pretty much, yes. The TERM environment variable simply tells the
> program you're invoking what dialect of terminal escapes to use
> (normally found by looking it up in the terminfo/termcap database) and
> what to expect as input when different special keys are pressed. It's
> informing the application of the contract you want it to honor with
> whatever is on the other side of the tty channel.
>
> Rich

Re: [musl] What determines the TERM variable value?

[Markus Wichmann]

On Sat, Feb 12, 2022 at 11:34:52AM +0200, Yuri Kanivetsky wrote:
> So, a program is a terminal emulator if it translates escape sequences
> in both directions? And if it, as a result, changes the TERM variable?
>

A program is a terminal emulator if it emulates a terminal, i.e.
provides an interface for TUI applications to interact with a keyboard
and screen (and optionally a mouse). The Linux VT provides such a thing
directly, X11 applications like xterm do so via the X11 protocol. Screen
and tmux are men in the middle, implementing screen and keyboard via
another terminal emulator. But still, they are terminal emulators.

The TERM variable is not somehow especially protected. Any process can
change it at any time, and it will remain changed for that process and
any child processes until one of those decides to change it again. To
wit:

> What seems strange to me is that docker changes the TERM variable:
>
> $ echo $TERM
> screen-256color
>
> $ docker run --rm -it alpine
> / # echo $TERM
> xterm
>

Anyone can change the TERM variable, including docker. I think, they
assume that xterm is a least common denominator among terminal
emulators. Also, the xterm terminfo is most likely to be installed in
whatever container you are running.

Leaving aside the technical meaning of the TERM variable, what it is
used for most often is to find the terminfo database for the terminal.
That is a file on the local file system, and must be installed to be
effective. If, for example, I connect from my normal terminal emulator
to a remote host via ssh, and I transmit a TERM variable of
"stterm-256color", then this will not help the apps on the far side of
the connection to figure out how to talk to my terminal if the stterm
terminfo file is not installed. My options then are to install it or
look for a terminfo file that is installed and sufficiently similar to
my terminal to make it work. Likely shedding some features in the
process (e.g. you can make almost anything work by setting TERM to
vt100, but then you loose all colors).

Similarly, it is nice for you to be using screen as a terminal in the
host, but docker likely does not want to assume that the screen terminfo
is installed in the container, and that is where it would need to be for
the container to make any use of it.

That said, docker is a highly flexible piece of software, so I would be
surprised if this functionality was not configurable somehow.

Ciao,
Markus

Re: [musl] What determines the TERM variable value?

[Rich Felker]

On Sat, Feb 12, 2022 at 11:33:56AM +0100, Markus Wichmann wrote:
> On Sat, Feb 12, 2022 at 11:34:52AM +0200, Yuri Kanivetsky wrote:
> > So, a program is a terminal emulator if it translates escape sequences
> > in both directions? And if it, as a result, changes the TERM variable?
> >
> 
> A program is a terminal emulator if it emulates a terminal, i.e.
> provides an interface for TUI applications to interact with a keyboard
> and screen (and optionally a mouse). The Linux VT provides such a thing
> directly, X11 applications like xterm do so via the X11 protocol. Screen
> and tmux are men in the middle, implementing screen and keyboard via
> another terminal emulator. But still, they are terminal emulators.
> 
> The TERM variable is not somehow especially protected. Any process can
> change it at any time, and it will remain changed for that process and
> any child processes until one of those decides to change it again. To
> wit:
> 
> > What seems strange to me is that docker changes the TERM variable:
> >
> > $ echo $TERM
> > screen-256color
> >
> > $ docker run --rm -it alpine
> > / # echo $TERM
> > xterm
> >
> 
> Anyone can change the TERM variable, including docker. I think, they
> assume that xterm is a least common denominator among terminal
> emulators. Also, the xterm terminfo is most likely to be installed in
> whatever container you are running.

If that's what's happening, Docker is just wrong to be doing this.
xterm is one of the most over-featured terminals out there, and the
terminfo for it reflects that. TERM=xterm is not likely to actually
work for most terminals if the application actually tries to make use
of any of the advanced features (though it might appear to work as
long as they stick to basics).

The appropriate TERM setting for a minimal common denominator where
you don't know the actual terminal type is probably something like
vt102 or vt220. But really Docker just shouldn't munge it.

> That said, docker is a highly flexible piece of software, so I would be
> surprised if this functionality was not configurable somehow.

Yeah, it's probably configurable.

Rich

Re: [musl] What determines the TERM variable value?

[Yuri Kanivetsky]

I don't think it's configurable in a common sense:

https://github.com/moby/moby/blob/v20.10.12/container/container.go#L752

But you can do:

$ docker run --rm -it -e TERM=xterm-256color alpine
/ # echo $TERM
xterm-256color

So, to sum it up, a terminal emulator is a program (let's not bring up
virtual consoles here) that emulates a terminal. It creates a pty (a
master/slave pair), drawing in a window what it receives from the pty,
and sending what user types to the pty. It also processes some escape
sequences it receives from the pty, and transforms some user
keystrokes into escape sequences before sending them to the pty. And
that determines the TERM variable value (for it to work correctly
you've got to have it set to a correct value).

Then there are some programs like screen and tmux, that are not
strictly terminal emulators, but in a way they are. They also create a
pty, and translate escape sequences from the "language" they
understand to the "language" understood by what's up the chain
(according to TERM), and vice versa. And again you've got to keep TERM
to a fixed range of values for it to work correctly.

Then there're some programs like ssh that create a pty, but doesn't change TERM.

And programs like docker that create a pty, but change TERM.

https://github.com/moby/moby/commit/c85db1003b#diff-14bacfb63a209323729af295c64c9bb978c84e810d18e3e7b7ae66d9b3cd7acaR259

Most likely to be user friendly. To make it work automagically for
users with an uncommon TERM value. I'd say it makes it more confusing
this way, but well, probably not everybody would agree.

So, basically with programs like ssh and docker, you generally want to
keep TERM unchanged. Granted that the corresponding terminfo entry
exists on a server/in a container.

But why would a program like ssh or docker want to create a pty in the
first place? If a pty on its own is just a data channel, then they
could just pass data as is, letting a terminal emulator handle things.
Isn't having TERM set enough for the programs down the chain to see
that they're connected to a terminal?

On Sat, Feb 12, 2022 at 5:05 PM Rich Felker <dalias@libc.org> wrote:
>
> On Sat, Feb 12, 2022 at 11:33:56AM +0100, Markus Wichmann wrote:
> > On Sat, Feb 12, 2022 at 11:34:52AM +0200, Yuri Kanivetsky wrote:
> > > So, a program is a terminal emulator if it translates escape sequences
> > > in both directions? And if it, as a result, changes the TERM variable?
> > >
> >
> > A program is a terminal emulator if it emulates a terminal, i.e.
> > provides an interface for TUI applications to interact with a keyboard
> > and screen (and optionally a mouse). The Linux VT provides such a thing
> > directly, X11 applications like xterm do so via the X11 protocol. Screen
> > and tmux are men in the middle, implementing screen and keyboard via
> > another terminal emulator. But still, they are terminal emulators.
> >
> > The TERM variable is not somehow especially protected. Any process can
> > change it at any time, and it will remain changed for that process and
> > any child processes until one of those decides to change it again. To
> > wit:
> >
> > > What seems strange to me is that docker changes the TERM variable:
> > >
> > > $ echo $TERM
> > > screen-256color
> > >
> > > $ docker run --rm -it alpine
> > > / # echo $TERM
> > > xterm
> > >
> >
> > Anyone can change the TERM variable, including docker. I think, they
> > assume that xterm is a least common denominator among terminal
> > emulators. Also, the xterm terminfo is most likely to be installed in
> > whatever container you are running.
>
> If that's what's happening, Docker is just wrong to be doing this.
> xterm is one of the most over-featured terminals out there, and the
> terminfo for it reflects that. TERM=xterm is not likely to actually
> work for most terminals if the application actually tries to make use
> of any of the advanced features (though it might appear to work as
> long as they stick to basics).
>
> The appropriate TERM setting for a minimal common denominator where
> you don't know the actual terminal type is probably something like
> vt102 or vt220. But really Docker just shouldn't munge it.
>
> > That said, docker is a highly flexible piece of software, so I would be
> > surprised if this functionality was not configurable somehow.
>
> Yeah, it's probably configurable.
>
> Rich

Re: [musl] What determines the TERM variable value?

[Markus Wichmann]

On Sun, Feb 13, 2022 at 10:54:23AM +0200, Yuri Kanivetsky wrote:
> I don't think it's configurable in a common sense:
>
> https://github.com/moby/moby/blob/v20.10.12/container/container.go#L752
>

Oh bloody hell. All this complexity, and this is the part they choose to
make fixed?

> But you can do:
>
> $ docker run --rm -it -e TERM=xterm-256color alpine
> / # echo $TERM
> xterm-256color
>

Yes, as I said, anyone can change TERM.

> So, to sum it up, a terminal emulator is a program (let's not bring up
> virtual consoles here) that emulates a terminal. It creates a pty (a
> master/slave pair), drawing in a window what it receives from the pty,
> and sending what user types to the pty. It also processes some escape
> sequences it receives from the pty, and transforms some user
> keystrokes into escape sequences before sending them to the pty. And
> that determines the TERM variable value (for it to work correctly
> you've got to have it set to a correct value).
>

Slightly off. It does create a PTY, and then translates what it receives
from it to things that are displayed on screen, speaking a certain
language. It also translates all user inputs into sequences to send back
to the terminal application. And the file that describes these sequences
in both directions is the terminfo database. TERM is used to locate the
terminfo database. As long as you set TERM to something you have the
database for, and the sequences described in the database match (for the
most part) the ones the terminal uses, things will work out.

Bear in mind that most terminals are very similar to each other and use
VT102-style escape sequences. They differ in the fine details. You can
use infocmp not only to print your own terminal's info, but also to
compare it with other terminals. Doing that on screen and xterm, there
are a lot of differences, but a lot of it is just xterm defining keys
that you don't have anyway (F13-F63 you probably only have if your
keyboard is used to control a nuclear power station, for example).

> Then there are some programs like screen and tmux, that are not
> strictly terminal emulators, but in a way they are.

They do translate user inputs into sequences and program outputs into
screen displays. It's just that they do so while being TUI applications
themselves. The point I was making with the Linux VTs is that the
concept is not tethered to any given GUI, or any GUI, for that matter.

> https://github.com/moby/moby/commit/c85db1003b#diff-14bacfb63a209323729af295c64c9bb978c84e810d18e3e7b7ae66d9b3cd7acaR259
>

I tend to argue against comments a lot. Most of the time they do not
add, and often take away from the important stuff. But if, like here,
you do something questionable, and you don't have a comment, and you
don't have a commit message saying the reason, I really do want
something.

> So, basically with programs like ssh and docker, you generally want to
> keep TERM unchanged. Granted that the corresponding terminfo entry
> exists on a server/in a container.
>

Yes. And if not, arrange for it to be installed. It isn't hard. For
administrators, installing a new terminfo file is low-risk, since it can
only ever possibly screw over the user that requested it, and all others
would have to opt into using it.

> But why would a program like ssh or docker want to create a pty in the
> first place? If a pty on its own is just a data channel, then they
> could just pass data as is, letting a terminal emulator handle things.
> Isn't having TERM set enough for the programs down the chain to see
> that they're connected to a terminal?
>

You also want certain kernel interfaces to work. In case of SSH, there
is an encrypted TCP tunnel between the application and the client
terminal. So everything the user types must be encrypted and sent up,
and everything the application sends must be encrypted and sent down. So
what should the standard file descriptors in the process on the server
be? They can't be the socket directly, then nothing would be translated.
And if they were pipes, they would not get treated as terminals.

Many terminal applications make a distinction whether isatty() works on
the standard file descriptors. And that is true for PTYs, as well as VTs
and serial lines, but not for pipes or sockets.

That TERM is set does not say anything about what FD 0 is connected to.

Ciao,
Markus

[musl] RE: [EXTERNAL] Re: [musl] What determines the TERM variable value?

[Andy Caldwell]

> On Sun, Feb 13, 2022 at 10:54:23AM +0200, Yuri Kanivetsky wrote:
> > I don't think it's configurable in a common sense:
> >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> > ub.com%2Fmoby%2Fmoby%2Fblob%2Fv20.10.12%2Fcontainer%2Fcontainer.go%23L
> > 752&data=04%7C01%7Candy.caldwell%40microsoft.com%7C15c78d610615484
> > 4ccf808d9eed74d25%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6378034
> > 31376969496%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz
> > IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Icw%2BHvFTPaaY0w%2B6
> > VcmyhhXT%2BXhube1WJ0kO5UmC0qk%3D&reserved=0
> >
>
> Oh bloody hell. All this complexity, and this is the part they choose to make fixed?

It makes perfect sense that this is hard-coded.  As you say, the TERM variable is the way
programs running on the TTY learn what escape codes to use to ask the TTY to do things
(change colour, clear the screen, restart a line, etc.) so the TERM is _set by the TTY_.
When running a docker container with a TTY attached, the TTY is created by the docker
daemon and isn't the TTY that's hosting the docker client binary.  This is needed to allow
the client to disconnect from the deamon without killing the container, and it also needed
for cases where the client and daemon are on separate hosts.

The daemon is listening on the master side of the TTY it created then streams data to and
from the client (if there is one) which then reads _its_ TERM variable and translates the
operations that the container tried to make into the appropriate control codes for the local
TTY.  Changing the TERM inside the container only makes sense if you also change the TTY
handling that the daemon does to match, otherwise the daemon will incorrectly interpret
control characters, pass those misinterpretations to the client which will then "correctly"
apply the wrong behaviour to the user's console.

Awkwardly many TTY termcap control characters are _mostly_ compatible (since they're
all based on VT100) so if you misconfigure your TERM things _mostly_ work, then you try
to use some TUI-heavy tool (like vim) and some things subtly don't work correctly.