From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 19727 invoked from network); 16 Sep 2022 04:14:55 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 16 Sep 2022 04:14:55 -0000 Received: (qmail 5489 invoked by uid 550); 16 Sep 2022 04:14:51 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 5448 invoked from network); 16 Sep 2022 04:14:50 -0000 Date: Fri, 16 Sep 2022 00:14:36 -0400 From: Rich Felker To: musl@lists.openwall.com Message-ID: <20220916041435.GL9709@brightrain.aerifal.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Subject: [musl] TCP fallback open questions I'm beginning to work on the long-awaited TCP fallback path in musl's stub resolver, and have here a list of things I've found so far that need some decisions about specifics of the behavior. For the most part these "questions" already have a seemingly solid choice I'm leaning towards, but since this is a topic that's complex and that's had lots of dissatisfaction over it in the past, I want to put them out to the community for feedback as soon as possible so that any major concerns can be considered. 1. How to switch over in the middle of a (multi-)query: In general, the DNS query core is performing M (1 or 2) concurrent queries against N configured nameservers, all in parallel (N*M in-flight operations). Any one of those might get back a truncated response requiring fallback to TCP. We want to do this fallback in a way that minimizes additional latency, resource usage, and logic complexity, which may be conflicting goals. As policy, we already assume the configured nameservers are redundant, providing non-conflicting views of the same DNS namespace. So once we see a single reply for a given question that requires TCP fallback, it's reasonable to conclude that any other reply to that question (from the other nameservers) would also require fallback, and to stop further retries of thaat question over UDP and ignore further answers to that question over UDP. The other question(s, but really only at most one, the opposite A/AAAA) however may still have satisfactory UDP answers, so ideally we want to keep listening for those, and keep retrying them. In principle we could end up using N*M TCP sockets for an exhaustive parallel query. N and M are small enough that this isn't huge, but it's also not really nice. Given that the switch to TCP was triggered by a truncated UDP response, we already know that the responding server *knows the answer* and just can't send it within the size limits. So a reasonable course of action is just to open a TCP connection to the nameserver that issued the truncated response. This is not necessarily going to be optimal -- it's possible that another nameserver has gotten a response in the mean time and that the round-trip for TCP handshake and payload would be much lower to that other server. But I'm doubtful that consuming extra kernel resources and producing extra network load to optimize the latency here is a reasonable tradeoff. I'm assuming so far that each question at least would have its own TCP connection (if truncated as UDP). Using multiple nameservers in parallel with TCP would maybe be an option if we were doing multiple queries on the same connection, but I'm not aware of whether TCP DNS has any sort of "pipelining" that would make this perform reasonably. Maybe if "priming" the question via UDP it doesn't matter though and we could expect the queries to be processed immediately with cached results? I don't think I like this but I'm just raising it for completeness. TL;DR summary: my leaning is to do one TCP connection per question that needs fallback, to the nameserver that issued the truncated response for the question. Does this seem reasonable? Am I overlooking anything important? 2. Buffer shuffling: Presently, UDP reads take place into the first unanswered buffer slot and then get moved if it wasn't the right place. This does not seem like it will work well when there are potentially partial TCP reads taking place into one or more slots. I think the simplest solution is just to use an additional fixed-size 512-byte local buffer in __res_msend_rc for receiving UDP and always move it into the right slot afterwards. The increased stack usage is not wonderful, but rather small relative to the whole calling call stack, and probably worth it to avoid code complexity. It also gives us a place to perform throwaway TCP reads into, when reading responses longer than the destination buffer just to report the needed length to the caller. 3. Timeouts: UDP being datagram based, there is no condition where we have to worry about blocking and getting stuck in the middle of a partial read. Timeout occurs just at the loop level. Are there any special considerations for timeout here using TCP? My leaning is no, since we'll still be in a poll loop regime, and regardless of blocking state on the socket, recv should do partial reads in the absence of MSG_WAITALL. 4. Logic for when fallback is needed: As noted in the thread "res_query/res_send contract findings", fallback is always needed by these functions when they get a response with the TC bit set because of the contract to return the size needed for the complete answer. But for high level (getaddrinfo, etc.) lookups, it's desirable to use truncated answers when we can. What should the condition for "when we can" be? My first leaning was that "nonzero ANCOUNT" suffices, but for CNAMEs, it's possible that the truncated response contains only the CNAME RR, not any records from the A or AAAA RRset. Some possible conditions that could be used: - At least one RR of the type in the question. This seems to be the choice to make maximal use of truncated responses, but could give significantly fewer addresses than one might like if the nameserver is badly behaved or if there's a very large CNAME consuming most of the packet. - No CNAME and packet size is at least 512 minus the size of one RR. This goes maximally in the other direction, never using results that might be limited by the presence of a CNAME, and ensuring we always have the number of answers we'd keep from a TCP response. There are probably several other reasonable options on a spectrum between these too. Unless name_from_dns (lookup_name.c) is changed to use longer response buffers, the only case in which switching to TCP will give us a better answer is when the nameserver is being petulent in its truncation. But it probably should be changed, since the case where the entire packet is consumed by a CNAME can be hit. To avoid that, the buffer needs to be at least just under 600 bytes.