Solaris: Who is talking to memcached?

In our environment we have memcached using tcp connections on Solaris 5.10. Sometimes we wonder (particularly in dev environments) what would screw up if I took down this instance? Today it came up that people were unsure how to figure out who was actively communicating with memcached anyway. Most of our team is more familiar with Linux than Solaris

The text protocol provides stats but the output doesn't really help. For example with exactly two live connections to memcached (telnet sessions) stats says this on my dev copy:

STAT curr_connections 11
STAT total_connections 31550
STAT connection_structures 13

Not really what I had in mind. There are NOT 11 current connections in the sense I'm interested in. OK, so how about netstat? Well ... it doesn't seem to provide the linkage from PID to connection we want. I  want a list of all the potentially interesting TCP connections to the memcached process, including the IP of the peer.

A little research reveals that the pfiles command lists vast swathes of spiffy information about the files (sockets are essentially a file handle and can be used in the ubiquotous read/write C APIs if you wish), including the socket and peer!

  ...blah blah blah...  31: S_IFSOCK mode:0666 dev:294,0 ino:46252 uid:0 gid:0 size:0      O_RDWR|O_NONBLOCK        SOCK_STREAM        SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49232),IP_NEXTHOP(1.2.3.4)        sockname: AF_INET 1.2.3.4  port: 11211        peername: AF_INET 2.3.4.5  port: 1057

That seems perfect; so now all we need to do is snag the PID of memcached, toss that over to pfiles, and grep out the bits we are interested in.

First up, PID of memcached. /usr/ucb/ps w will list our processes, including memcached (depending on how you started it you may need slightly different arguments):

/usr/ucb/ps x   PID TT       S  TIME COMMAND 15973 ?        S  0:51 /my/path/to/memcached ...args 16982 ?        S  0:00 /usr/lib/ssh/sshd 16989 pts/11   S  0:00 -bash 18017 pts/11   O  0:00 /usr/ucb/ps -x

We can pluck out the memcached line with grep, and grab the first token however we prefer. In my case I'll just use awk. So to get the PID for memcached (if only one instance is running on the node) we can do this:

/usr/ucb/ps x | grep memcached | grep -v grep | awk '{print $1}'15973

We can use command substitution to pass this to pfiles:

pfiles $(/usr/ucb/ps x | grep memcached | grep -v grep | awk '{print $1}')

Then grep out the parts we are interested in (only IPv4 - AF_INET - items that have known peers; this is basically sockets showing who is connected to them). In the example below I have exactly two connections to my memcached from two different client machines (whose IPs have been modified)

 pfiles $(/usr/ucb/ps x | grep memcached | grep -v grep | awk '{print $1}') | grep "peername: AF_INET" 

        peername: AF_INET 1.2.3.4  port: 46009 

        peername: AF_INET 1.2.3.5  port: 1057

In the example above the clients connected to memcached are 1.2.3.4 and 1.2.3.5.

Very handy!