Multiprocessing SNMP with Python
I've written a ton of snmp monitoring scripts and they all suck because they are blocking and take "too long" to return results for a large amount of hosts. So how would we make this process faster and make us happier?
multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.
IBM has published a wonderful article on Multiprocessing with Python in which I have modified their snippets for my purposes below. To leverage NetSNMP in python you need to compile it with python modules, I wrote a post about this yesterday in Compiling Netsnmp for Ubuntu. After battling through the compiling bit you should be able to get the example in the IBM post to work. I modified this example *slightly* to accommodate instance identifier (iid). Hopefully this will be suitable for a new project I'm working on which polls thousands of snmp oids to measure latency between routers using the concept I posted on hatch - Configure RTR/SLA Ping in IOS template.
!/usr/bin/env python """ This is a multiprocessing wrapper for Net-SNMP. This makes a synchronous API asynchronous by combining it with Python2.6 """ import netsnmp from multiprocessing import Process, Queue, current_process class HostRecord(): """This creates a host record""" def __init__(self, hostname = None, query = None): self.hostname = hostname self.query = query class SnmpSession(): """A SNMP Session""" def __init__(self, oid = "sysDescr", iid="0", Version = 2, DestHost = "localhost", Community = "public", Verbose = True, ): self.oid = oid self.Version = Version self.DestHost = DestHost self.Community = Community self.Verbose = Verbose self.var = netsnmp.Varbind(oid, iid) self.hostrec = HostRecord() self.hostrec.hostname = self.DestHost def query(self): """Creates SNMP query Fills out a Host Object and returns result """ try: result = netsnmp.snmpget(self.var, Version = self.Version, DestHost = self.DestHost, Community = self.Community) self.hostrec.query = result except Exception, err: if self.Verbose: print err self.hostrec.query = None finally: return self.hostrec def make_query(host): """This does the actual snmp query This is a bit fancy as it accepts both instances of SnmpSession and host/ip addresses. This allows a user to customize mass queries with subsets of different hostnames and community strings """ if isinstance(host,SnmpSession): return host.query() else: s = SnmpSession(DestHost=host) return s.query() # Function run by worker processes def worker(input, output): for func in iter(input.get, 'STOP'): result = make_query(func) output.put(result) def main(): """Runs everything""" #clients hosts = [ SnmpSession(DestHost="10.71.1.1", Community="my-pub-community", oid="1.3.6.1.4.1.9.9.42.1.2.10.1.1", iid="1"), SnmpSession(DestHost="10.81.1.1", Community="my-pub-community", oid="1.3.6.1.4.1.9.9.42.1.2.10.1.1", iid="123") ] NUMBER_OF_PROCESSES = len(hosts) # Create queues task_queue = Queue() done_queue = Queue() #submit tasks for host in hosts: task_queue.put(host) #Start worker processes for i in range(NUMBER_OF_PROCESSES): Process(target=worker, args=(task_queue, done_queue)).start() # Get and print results print 'Unordered results:' for i in range(len(hosts)): print '\t', done_queue.get().query # Tell child processes to stop for i in range(NUMBER_OF_PROCESSES): task_queue.put('STOP') #print "Stopping Process #%s" % i if __name__ == "__main__": main()
0 Comments
Log in with Twitter, Google, Facebook, LinkedIn to leave a comment.