Copy&Waste

Multiprocessing SNMP with Python

I've written a ton of snmp monitoring scripts and they all suck because they are blocking and take "too long" to return results for a large amount of hosts. So how would we make this process faster and make us happier?

multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

IBM has published a wonderful article on Multiprocessing with Python in which I have modified their snippets for my purposes below. To leverage NetSNMP in python you need to compile it with python modules, I wrote a post about this yesterday in Compiling Netsnmp for Ubuntu. After battling through the compiling bit you should be able to get the example in the IBM post to work. I modified this example *slightly* to accommodate instance identifier (iid).  Hopefully this will be suitable for a new project I'm working on which polls thousands of snmp oids to measure latency between routers using the concept I posted on hatch - Configure RTR/SLA Ping in IOS template. 

!/usr/bin/env python
"""
This is a multiprocessing wrapper for Net-SNMP.
This makes a synchronous API asynchronous by combining
it with Python2.6
"""

import netsnmp
from multiprocessing import Process, Queue, current_process

class HostRecord():
    """This creates a host record"""
    def __init__(self,
                 hostname = None,
                 query = None):
        self.hostname = hostname
        self.query = query

class SnmpSession():
    """A SNMP Session"""
    def __init__(self,
                oid = "sysDescr",
                iid="0",
                Version = 2,
                DestHost = "localhost",
                Community = "public",
                Verbose = True,
                ):
        self.oid = oid
        self.Version = Version
        self.DestHost = DestHost
        self.Community = Community
        self.Verbose = Verbose
        self.var = netsnmp.Varbind(oid, iid)
        self.hostrec = HostRecord()
        self.hostrec.hostname = self.DestHost

    def query(self):
        """Creates SNMP query

        Fills out a Host Object and returns result
        """
        try:
            result = netsnmp.snmpget(self.var,
                                Version = self.Version,
                                DestHost = self.DestHost,
                                Community = self.Community)
            self.hostrec.query = result
        except Exception, err:
            if self.Verbose:
                print err
            self.hostrec.query = None
        finally:
            return self.hostrec

def make_query(host):
    """This does the actual snmp query

    This is a bit fancy as it accepts both instances
    of SnmpSession and host/ip addresses.  This
    allows a user to customize mass queries with
    subsets of different hostnames and community strings
    """
    if isinstance(host,SnmpSession):
        return host.query()
    else:
        s = SnmpSession(DestHost=host)
        return s.query()

# Function run by worker processes
def worker(input, output):
    for func in iter(input.get, 'STOP'):
        result = make_query(func)
        output.put(result)

def main():
    """Runs everything"""

    #clients
    hosts = [
    SnmpSession(DestHost="10.71.1.1", Community="my-pub-community", oid="1.3.6.1.4.1.9.9.42.1.2.10.1.1", iid="1"),
    SnmpSession(DestHost="10.81.1.1", Community="my-pub-community", oid="1.3.6.1.4.1.9.9.42.1.2.10.1.1", iid="123")
    ]
    NUMBER_OF_PROCESSES = len(hosts)

    # Create queues
    task_queue = Queue()
    done_queue = Queue()

    #submit tasks
    for host in hosts:
        task_queue.put(host)

    #Start worker processes
    for i in range(NUMBER_OF_PROCESSES):
        Process(target=worker, args=(task_queue, done_queue)).start()

     # Get and print results
    print 'Unordered results:'
    for i in range(len(hosts)):
        print '\t', done_queue.get().query

    # Tell child processes to stop
    for i in range(NUMBER_OF_PROCESSES):
        task_queue.put('STOP')
        #print "Stopping Process #%s" % i

if __name__ == "__main__":
    main()