Why do services like sendmail/httpd still query outdated DNS servers after resolv.conf is changed?

Question

One of my colleague raised a question: After changing resolv.conf, sendmail still query the old resolver, why?

Steps:

1. Set 192.168.122.65 as nameserver in resolv.conf, start sendmail service, send a mail to root@hat.com, and take a tcpdump. We can see there's a query asking 192.168.122.65 for MX record of hat.com .

2. Change nameserver from 192.168.122.65 to 192.168.122.72 in resolv.conf, send a mail to root@hat.com and take a tcpdump again. This time we expect a query to 192.168.122.72 (new), but the actual query is to 192.168.122.65 (old).

3. Restart sendmail service, send a mail to root@hat.com, this time we can see a query to 192.168.122.72 (new) as expected.

When we were thinking about what's going wrong with sendmail, another colleague joined in discussion, and she mentioned that httpd and some other long-running services also have this behavior.

Since sendmail is not the only case, I think we should have a check on glibc.

Reproduce using getaddrinfo()

A commonly used routine to convert hostname into IP is calling getaddrinfo(), which is provided by glibc.

I ran below code as a long-running service. It will repeatedly lookup hat.com, and print the corresponding IP address in standard output.

/***
Filename: getip.c
Function: Get IP address of HOSTNAME repeatedly in 5s interval.
Usage:
# gcc -Wall -o getip getip.c
# chmod +x getip
# ./getip 
***/

#include <arpa/inet.h>
#include <netdb.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#ifndef   NI_MAXHOST
#define   NI_MAXHOST 1025
#endif

#define  HOSTNAME "hat.com"

int getip()
{
    struct addrinfo* result;
    struct addrinfo* res;
    int error;

    /* resolve the domain name into a list of addresses */
    error = getaddrinfo(HOSTNAME, NULL, NULL, &result);
    if (error != 0) {   
        if (error == EAI_SYSTEM) {
            perror("getaddrinfo");
        } else {
            fprintf(stderr, "error in getaddrinfo: %s\n", gai_strerror(error));
        }   
        exit(EXIT_FAILURE);
    }   


    for ( res = result; res != NULL; res = res->ai_next) {
	struct sockaddr_in *addr;
	addr = (struct sockaddr_in *)res->ai_addr; 
	printf("IP: %s\n", inet_ntoa((struct in_addr)addr->sin_addr));
    }

    freeaddrinfo(result);
    return 0;

}

int main(int argc, char **argv)
{
    int epoch = 0;
    while(1) {
        printf("-----------%i---------\n", ++epoch);
	getip();
        sleep(5);
    }
    return 0;
}

Also, prepare two nameserver. In one server, point hat.com to 192.168.122.203, and in another server, point hat.com to 192.168.122.204.

Steps:
1. Open a terminal and set server A as the only nameserver. Server A would point hat.com to 192.168.122.203.
2. Open another terminal and run the above code (./getip) for a few minutes.
3. Modify /etc/resolv.conf, and set server B as the only nameserver. Server B would point hat.com to 192.168.122.204.
4. Wait a few more minutes, and we can see ./getip still getting 192.168.122.203 as result. (If we take a tcpdump, we can see it still querying server A.)

-----------1---------
IP: 192.168.122.203
IP: 192.168.122.203
IP: 192.168.122.203
-----------2---------
IP: 192.168.122.203
IP: 192.168.122.203
IP: 192.168.122.203

-----------168---------
IP: 192.168.122.203
IP: 192.168.122.203
IP: 192.168.122.203
-----------169---------
IP: 192.168.122.203
IP: 192.168.122.203
IP: 192.168.122.203

5. Restart ./getip, and this time it can get 192.168.122.204.

-----------1---------
IP: 192.168.122.204
IP: 192.168.122.204
IP: 192.168.122.204
-----------2---------
IP: 192.168.122.204
IP: 192.168.122.204
IP: 192.168.122.204

NSCD come to help

In the above case, the only way to make application query the new nameserver is to restart the application. If server is using DHCP and resolv.conf is frequently changed, it may not be desirable that we frequently restart the long-running service. In such case, NSCD can help.

Quote from NSCD's man page:

 nscd - name service cache daemon 

The daemon will try to watch for changes in configuration files appropriate for each database (e.g., /etc/passwd for the passwd database or /etc/hosts and /etc/resolv.conf for the hosts database), and flush the cache when these are changed. 

With NSCD running, we can see the long-running application can get the new record from the new nameserver without restarting the application.

-----------14---------
IP: 192.168.122.203
IP: 192.168.122.203
IP: 192.168.122.203
-----------15---------
IP: 192.168.122.203
IP: 192.168.122.203
IP: 192.168.122.203
-----------16---------
IP: 192.168.122.204
IP: 192.168.122.204
IP: 192.168.122.204
-----------17---------
IP: 192.168.122.204
IP: 192.168.122.204
IP: 192.168.122.204

Without NSCD, we can see from the strace that ./getip would only read resolv.conf once, and query the nameserver every time.

write(1, "-----------1---------\n", 22) = 22
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.122.1")}, 16) = 0 //<<---- 192.168.122.1 is the nameserver specified in resolv.conf

write(1, "-----------2---------\n", 22) = 22
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.122.1")}, 16) = 0

With NSCD, application will query NSCD directly, and NSCD will do things like caching for application.

write(1, "-----------1---------\n", 22) = 22
connect(4, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = 0
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = 0
sendto(3, "\2\0\0\0\16\0\0\0\16\0\0\0www.baidu.com\0", 26, MSG_NOSIGNAL, NULL, 0) = 26

write(1, "-----------2---------\n", 22) = 22
connect(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("119.75.213.50")}, 16) = 0

write(1, "-----------3---------\n", 22) = 22
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = 0
sendto(3, "\2\0\0\0\16\0\0\0\16\0\0\0www.baidu.com\0", 26, MSG_NOSIGNAL, NULL, 0) = 26

res_*() / resolver routine

As per Sourceware bug #984, sendmail use res_*() to get MX record from DNS, so NSCD could not help sendmail.

Let's have a try on res_*() routine, and see how it works.

/******
Filename: res_loop.c 
Compile: 
  # gcc -Wall -o res_loop res_lopp.c /usr/lib64/libresolv.a (RHEL5)
  # gcc -Wall -lresolv -o res_loop res_loop.c (RHEL6)

Usage:
  # ./res_loop example.com

Description:
  This program will query resolver for A/MX records.
  After querying a few times (NR_QUERIES), it will call res_close() and res_init()
  to re-initialize resolver information.

Acknowledge:
  Code is adapted from
  http://stackoverflow.com/questions/15476717/how-to-query-a-server-and-get-the-mx-a-ns-records 
*******/

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <netinet/in.h>
#include <resolv.h>
#include <netdb.h>
#include <arpa/nameser.h>

#define N 4096

#define NR_QUERIES 8  // Try NR_QUERIES before redo res_init();
#define INTERVAL 5    // How many seconds between each query.

int query (char *domain)
{
    u_char nsbuf[N];
    char dispbuf[N];
    ns_msg msg;
    ns_rr rr;
    int i, l;

    // HEADER
    printf("\nDomain : %s\n", domain);

    // A RECORD
    printf("A records : \n");
    l = res_query(domain, ns_c_any, ns_t_a, nsbuf, sizeof(nsbuf));
    if (l < 0)
    {
      perror(domain);
    }
    ns_initparse(nsbuf, l, &msg);
    l = ns_msg_count(msg, ns_s_an);
    for (i = 0; i < l; i++)
    {
      ns_parserr(&msg, ns_s_an, 0, &rr);
      ns_sprintrr(&msg, &rr, NULL, NULL, dispbuf, sizeof(dispbuf));
      printf("\t%s \n", dispbuf);
    }

    // MX RECORD
    printf("MX records : \n");
    l = res_query(domain, ns_c_any, ns_t_mx, nsbuf, sizeof(nsbuf));
    if (l < 0)
    {
      perror(domain);
    }
    else
    {
#ifdef USE_PQUERY
      /* this will give lots of detailed info on the request and reply */
      res_pquery(&_res, nsbuf, l, stdout);
#else
      /* just grab the MX answer info */
      ns_initparse(nsbuf, l, &msg);
      l = ns_msg_count(msg, ns_s_an);
      for (i = 0; i < l; i++)
      {
        ns_parserr(&msg, ns_s_an, i, &rr);
        ns_sprintrr(&msg, &rr, NULL, NULL, dispbuf, sizeof(dispbuf));
        printf ("\t%s\n", dispbuf);
      }
#endif
    }

    // NS RECORD
    printf("NS records : \n");
    l = res_query(domain, ns_c_any, ns_t_ns, nsbuf, sizeof(nsbuf));
    if (l < 0)
    {
      perror(domain);
    }
    ns_initparse(nsbuf, l, &msg);
    l = ns_msg_count(msg, ns_s_an);
    for (i = 0; i < l; i++)
    {
      ns_parserr(&msg, ns_s_an, 0, &rr);
      ns_sprintrr(&msg, &rr, NULL, NULL, dispbuf, sizeof(dispbuf));
      printf("\t%s \n", dispbuf);
    }

    return 0;
}


int main (int argc, char *argv[])
{
    int i = 0;

    if (argc < 2) {
        printf ("Usage: %s <domain>\n", argv[0]);
        exit (1);
    }

    while(1){
	i = ( i + 1 ) % NR_QUERIES;
	if(!i) {
	   res_close();
	   res_init();
           printf("\n------ res_close() & res_init() called ------\n");
	}
    	query(argv[1]);
	sleep(INTERVAL);
    }

    return 0;
}

Test steps:
1. Run ./res_loop hat.com.
2. Modify /etc/resolv.conf, and keep monitoring res_loop's output.

Without NSCD running, we can see after res_close() and res_init() are called, res_loop can get the new record from new nameserver.

Domain : hat.com
A records : 
	hat.com.		1M IN A		192.168.122.204 
MX records : 
	hat.com.		1M IN MX	5 hat.com.
NS records : 
	hat.com.		1M IN NS	hat.com. 

Domain : hat.com
A records : 
	hat.com.		1M IN A		192.168.122.204 
MX records : 
	hat.com.		1M IN MX	5 hat.com.
NS records : 
	hat.com.		1M IN NS	hat.com. 

------ res_close() & res_init() called ------

Domain : hat.com
A records : 
	hat.com.		1M IN A		192.168.122.203 
MX records : 
	hat.com.		1M IN MX	5 hat.com.
NS records : 
	hat.com.		1M IN NS	hat.com. 

Domain : hat.com
A records : 
	hat.com.		1M IN A		192.168.122.203 
MX records : 
	hat.com.		1M IN MX	5 hat.com.
NS records : 
	hat.com.		1M IN NS	hat.com. 

With NSCD, the behavior of res_loop is still the same. It won't query new nameserver until res_close() and res_init() are called.

Domain : hat.com
A records : 
	hat.com.		1M IN A		192.168.122.203 
MX records : 
	hat.com.		1M IN MX	5 hat.com.
NS records : 
	hat.com.		1M IN NS	hat.com. 

------ res_close() & res_init() called ------

Domain : hat.com
A records : 
	hat.com.		1M IN A		192.168.122.204 
MX records : 
	hat.com.		1M IN MX	5 hat.com.
NS records : 
	hat.com.		1M IN NS	hat.com. 

From strace, we can see /etc/resolv.conf would be re-read when res_init() is called.

open("/etc/resolv.conf", O_RDONLY)      = 3
......
close(3)                                = 0
write(1, "------ res_close() & res_init() "..., 46) = 46

Sendmail case

Though sendmail is using res_*(), but it has different behavior. I could not find out the reason at the moment. (Maybe there's some caching mechanism).

Test 1: RHEL5.4 + sendmail + NSCD disabled
Result: Need to restart sendmail to query new resolver.

Test 2: RHEL5.4 + sendmail + NSCD enabled
Result: Even restart sendmail, sendmail would not query the new resolver.

Test 3: RHEL6.8 + sendmail + NSCD disabled
Result: Restart sendmail and sendmail would query new resolver.

Test 4: RHEL6.8 + sendmail + NSCD enabled
Result: No need to restart sendmail and sendmail would query new resolver.