Contents
Question
One of my colleague raised a question: After changing resolv.conf, sendmail still query the old resolver, why?
Steps:
1. Set 192.168.122.65 as nameserver in resolv.conf, start sendmail service, send a mail to root@hat.com, and take a tcpdump. We can see there's a query asking 192.168.122.65 for MX record of hat.com .
2. Change nameserver from 192.168.122.65 to 192.168.122.72 in resolv.conf, send a mail to root@hat.com and take a tcpdump again. This time we expect a query to 192.168.122.72 (new), but the actual query is to 192.168.122.65 (old).
3. Restart sendmail service, send a mail to root@hat.com, this time we can see a query to 192.168.122.72 (new) as expected.
When we were thinking about what's going wrong with sendmail, another colleague joined in discussion, and she mentioned that httpd and some other long-running services also have this behavior.
Since sendmail is not the only case, I think we should have a check on glibc.
Reproduce using getaddrinfo()
A commonly used routine to convert hostname into IP is calling getaddrinfo(), which is provided by glibc.
I ran below code as a long-running service. It will repeatedly lookup hat.com, and print the corresponding IP address in standard output.
Filename: getip.c
Function: Get IP address of HOSTNAME repeatedly in 5s interval.
Usage:
# gcc -Wall -o getip getip.c
# chmod +x getip
# ./getip
***/
#include <arpa/inet.h>
#include <netdb.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#ifndef NI_MAXHOST
#define NI_MAXHOST 1025
#endif
#define HOSTNAME "hat.com"
int getip()
{
struct addrinfo* result;
struct addrinfo* res;
int error;
/* resolve the domain name into a list of addresses */
error = getaddrinfo(HOSTNAME, NULL, NULL, &result);
if (error != 0) {
if (error == EAI_SYSTEM) {
perror("getaddrinfo");
} else {
fprintf(stderr, "error in getaddrinfo: %s\n", gai_strerror(error));
}
exit(EXIT_FAILURE);
}
for ( res = result; res != NULL; res = res->ai_next) {
struct sockaddr_in *addr;
addr = (struct sockaddr_in *)res->ai_addr;
printf("IP: %s\n", inet_ntoa((struct in_addr)addr->sin_addr));
}
freeaddrinfo(result);
return 0;
}
int main(int argc, char **argv)
{
int epoch = 0;
while(1) {
printf("-----------%i---------\n", ++epoch);
getip();
sleep(5);
}
return 0;
}
Also, prepare two nameserver. In one server, point hat.com to 192.168.122.203, and in another server, point hat.com to 192.168.122.204.
Steps:
1. Open a terminal and set server A as the only nameserver. Server A would point hat.com to 192.168.122.203.
2. Open another terminal and run the above code (./getip) for a few minutes.
3. Modify /etc/resolv.conf, and set server B as the only nameserver. Server B would point hat.com to 192.168.122.204.
4. Wait a few more minutes, and we can see ./getip still getting 192.168.122.203 as result. (If we take a tcpdump, we can see it still querying server A.)
IP: 192.168.122.203
IP: 192.168.122.203
IP: 192.168.122.203
-----------2---------
IP: 192.168.122.203
IP: 192.168.122.203
IP: 192.168.122.203
-----------168---------
IP: 192.168.122.203
IP: 192.168.122.203
IP: 192.168.122.203
-----------169---------
IP: 192.168.122.203
IP: 192.168.122.203
IP: 192.168.122.203
5. Restart ./getip, and this time it can get 192.168.122.204.
IP: 192.168.122.204
IP: 192.168.122.204
IP: 192.168.122.204
-----------2---------
IP: 192.168.122.204
IP: 192.168.122.204
IP: 192.168.122.204
NSCD come to help
In the above case, the only way to make application query the new nameserver is to restart the application. If server is using DHCP and resolv.conf is frequently changed, it may not be desirable that we frequently restart the long-running service. In such case, NSCD can help.
Quote from NSCD's man page:
nscd - name service cache daemon The daemon will try to watch for changes in configuration files appropriate for each database (e.g., /etc/passwd for the passwd database or /etc/hosts and /etc/resolv.conf for the hosts database), and flush the cache when these are changed.
With NSCD running, we can see the long-running application can get the new record from the new nameserver without restarting the application.
-----------14--------- IP: 192.168.122.203 IP: 192.168.122.203 IP: 192.168.122.203 -----------15--------- IP: 192.168.122.203 IP: 192.168.122.203 IP: 192.168.122.203 -----------16--------- IP: 192.168.122.204 IP: 192.168.122.204 IP: 192.168.122.204 -----------17--------- IP: 192.168.122.204 IP: 192.168.122.204 IP: 192.168.122.204
Without NSCD, we can see from the strace that ./getip would only read resolv.conf once, and query the nameserver every time.
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.122.1")}, 16) = 0 //<<---- 192.168.122.1 is the nameserver specified in resolv.conf
write(1, "-----------2---------\n", 22) = 22
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.122.1")}, 16) = 0
With NSCD, application will query NSCD directly, and NSCD will do things like caching for application.
connect(4, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = 0
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = 0
sendto(3, "\2\0\0\0\16\0\0\0\16\0\0\0www.baidu.com\0", 26, MSG_NOSIGNAL, NULL, 0) = 26
write(1, "-----------2---------\n", 22) = 22
connect(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("119.75.213.50")}, 16) = 0
write(1, "-----------3---------\n", 22) = 22
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = 0
sendto(3, "\2\0\0\0\16\0\0\0\16\0\0\0www.baidu.com\0", 26, MSG_NOSIGNAL, NULL, 0) = 26
res_*() / resolver routine
As per Sourceware bug #984, sendmail use res_*() to get MX record from DNS, so NSCD could not help sendmail.
Let's have a try on res_*() routine, and see how it works.
Filename: res_loop.c
Compile:
# gcc -Wall -o res_loop res_lopp.c /usr/lib64/libresolv.a (RHEL5)
# gcc -Wall -lresolv -o res_loop res_loop.c (RHEL6)
Usage:
# ./res_loop example.com
Description:
This program will query resolver for A/MX records.
After querying a few times (NR_QUERIES), it will call res_close() and res_init()
to re-initialize resolver information.
Acknowledge:
Code is adapted from
http://stackoverflow.com/questions/15476717/how-to-query-a-server-and-get-the-mx-a-ns-records
*******/
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <netinet/in.h>
#include <resolv.h>
#include <netdb.h>
#include <arpa/nameser.h>
#define N 4096
#define NR_QUERIES 8 // Try NR_QUERIES before redo res_init();
#define INTERVAL 5 // How many seconds between each query.
int query (char *domain)
{
u_char nsbuf[N];
char dispbuf[N];
ns_msg msg;
ns_rr rr;
int i, l;
// HEADER
printf("\nDomain : %s\n", domain);
// A RECORD
printf("A records : \n");
l = res_query(domain, ns_c_any, ns_t_a, nsbuf, sizeof(nsbuf));
if (l < 0)
{
perror(domain);
}
ns_initparse(nsbuf, l, &msg);
l = ns_msg_count(msg, ns_s_an);
for (i = 0; i < l; i++)
{
ns_parserr(&msg, ns_s_an, 0, &rr);
ns_sprintrr(&msg, &rr, NULL, NULL, dispbuf, sizeof(dispbuf));
printf("\t%s \n", dispbuf);
}
// MX RECORD
printf("MX records : \n");
l = res_query(domain, ns_c_any, ns_t_mx, nsbuf, sizeof(nsbuf));
if (l < 0)
{
perror(domain);
}
else
{
#ifdef USE_PQUERY
/* this will give lots of detailed info on the request and reply */
res_pquery(&_res, nsbuf, l, stdout);
#else
/* just grab the MX answer info */
ns_initparse(nsbuf, l, &msg);
l = ns_msg_count(msg, ns_s_an);
for (i = 0; i < l; i++)
{
ns_parserr(&msg, ns_s_an, i, &rr);
ns_sprintrr(&msg, &rr, NULL, NULL, dispbuf, sizeof(dispbuf));
printf ("\t%s\n", dispbuf);
}
#endif
}
// NS RECORD
printf("NS records : \n");
l = res_query(domain, ns_c_any, ns_t_ns, nsbuf, sizeof(nsbuf));
if (l < 0)
{
perror(domain);
}
ns_initparse(nsbuf, l, &msg);
l = ns_msg_count(msg, ns_s_an);
for (i = 0; i < l; i++)
{
ns_parserr(&msg, ns_s_an, 0, &rr);
ns_sprintrr(&msg, &rr, NULL, NULL, dispbuf, sizeof(dispbuf));
printf("\t%s \n", dispbuf);
}
return 0;
}
int main (int argc, char *argv[])
{
int i = 0;
if (argc < 2) {
printf ("Usage: %s <domain>\n", argv[0]);
exit (1);
}
while(1){
i = ( i + 1 ) % NR_QUERIES;
if(!i) {
res_close();
res_init();
printf("\n------ res_close() & res_init() called ------\n");
}
query(argv[1]);
sleep(INTERVAL);
}
return 0;
}
Test steps:
1. Run ./res_loop hat.com.
2. Modify /etc/resolv.conf, and keep monitoring res_loop's output.
Without NSCD running, we can see after res_close() and res_init() are called, res_loop can get the new record from new nameserver.
Domain : hat.com A records : hat.com. 1M IN A 192.168.122.204 MX records : hat.com. 1M IN MX 5 hat.com. NS records : hat.com. 1M IN NS hat.com. Domain : hat.com A records : hat.com. 1M IN A 192.168.122.204 MX records : hat.com. 1M IN MX 5 hat.com. NS records : hat.com. 1M IN NS hat.com. ------ res_close() & res_init() called ------ Domain : hat.com A records : hat.com. 1M IN A 192.168.122.203 MX records : hat.com. 1M IN MX 5 hat.com. NS records : hat.com. 1M IN NS hat.com. Domain : hat.com A records : hat.com. 1M IN A 192.168.122.203 MX records : hat.com. 1M IN MX 5 hat.com. NS records : hat.com. 1M IN NS hat.com.
With NSCD, the behavior of res_loop is still the same. It won't query new nameserver until res_close() and res_init() are called.
Domain : hat.com A records : hat.com. 1M IN A 192.168.122.203 MX records : hat.com. 1M IN MX 5 hat.com. NS records : hat.com. 1M IN NS hat.com. ------ res_close() & res_init() called ------ Domain : hat.com A records : hat.com. 1M IN A 192.168.122.204 MX records : hat.com. 1M IN MX 5 hat.com. NS records : hat.com. 1M IN NS hat.com.
From strace, we can see /etc/resolv.conf would be re-read when res_init() is called.
......
close(3) = 0
write(1, "------ res_close() & res_init() "..., 46) = 46
Sendmail case
Though sendmail is using res_*(), but it has different behavior. I could not find out the reason at the moment. (Maybe there's some caching mechanism).
Test 1: RHEL5.4 + sendmail + NSCD disabled
Result: Need to restart sendmail to query new resolver.
Test 2: RHEL5.4 + sendmail + NSCD enabled
Result: Even restart sendmail, sendmail would not query the new resolver.
Test 3: RHEL6.8 + sendmail + NSCD disabled
Result: Restart sendmail and sendmail would query new resolver.
Test 4: RHEL6.8 + sendmail + NSCD enabled
Result: No need to restart sendmail and sendmail would query new resolver.