For the past year, I have resisted speculation and development on
CoyMon 3. But in the past few months, many pieces have begun falling
into place -- each removing objections and concerns, and some providing
a clear path of development where before there was none.
CoyoteMonitoring is a free, open-source network management tool. Specifically, it glues
together many open-source and difficult-to-configure subsystems for
obtaining and viewing
NetFlowdata used in network analysis. It is comparable to operating system
distributions in this regard: it is a NetFlow management distro. The
Department of Veterans affairs has actually been using CoyMon 2 for
about a year now, with astounding results. We are thrilled with the
success it has been for them, with the tremendous time and money it has
saved them.
The development of CoyMon 2 was sponsored in part by the VA, and due to
the timeline, much of the code was specific to their needs. However,
because of the time it would take to audit the code, CoyMon 2 has never
been publicly released. I have had many email conversations with WAN
managers and network engineerings very distressed at this. They too
have found a sad lack in the open source world when it comes to useful,
easy-to-use tools for querying, viewing, and analyzing NetFlow data.
Because of this continued interest, input, and moral support from these
hard-working individuals, I have been considering how this need could
best be addressed and met.
To be honest, my biggest technical concern has been with the current
NetFlow tools. The perl code that people have depended upon for this
(since the late 90s!) is krufty, hard to maintain, overly uses
almost-unnecessary and out-of-date perl modules, and as a group, were
not designed with maximal extensibility in mind. They have been genuine
gems in the field -- none of us could have done what we have done for
the past several years without this wonderful contribution. The
combination of these (and their dependencies!) with all the other
pieces that require configuration and glue is a difficult thing to
provide. It took an ENORMOUS amount of time and energy to get CoyMon 2
into a place where it could be deployed efficiently. CoyMon 2 runs like
a tank, though. It's a real champ and a testament to all that hard work
and organization.
However, it is time to fix the system at its roots. Enter CoyMon 3.
The biggest stumbling block to forward movement on CoyMon 3 has been
the available APIs for to tools upon which we depend, with one of the
most important being RRDTool. The python bindings for RRDTool are
archaic and decidedly non-OOP. Attempting to design a system that is
robust, effortless to maintain, and easily adapts new features but that
has composite components with difficult APIs is a recipe for
frustration, delay, and ultimately, non-delivery. The first step
towards addressing this problem came with the recent release of
PyRRD.
It is a fully OOP wrapper for RRDTool in python that removes this pain
from the equation.
The next biggest hurdle was the old perl code called FlowScan. After
much discussion and analysis, a clean and elegant way to provide this
functionality was arrived at, and we will soon have a new product to
show for it, freely available for download. CoyoteMonitoring will
depend heavily on this piece of software and related libraries. We are
making excellent progress on this, but ultimately, the problem to solve
here is one of modularity and configuration. How do you provide an
easy, non-programmatic method of extensibility for any number of
potential rules? We've got some great ideas, but only the natural
selection of actual use to prove the best approach. This is our current
top-priority.
With the advances that have been made in the past several months, we
are not only comfortable making an announcement about a new version of
CoyoteMonitoring, we are down-right confident :-) For the interested,
here are more detailed points on CoyMon 3:
- CoyMon 3 will be developed completely independently of any
third-parties or businesses. This will mean slower development times,
but cleaner more easily managed code. With the absence of sensitive
customer code and/or configurations, you will see regularly available
downloads.
- Supporting libraries will all have extensive unit tests for each
piece of functionality.
- CoyMon 3 will have a 100% true component architecture.
- CoyMon 3 will continue to use flow-tools and will make full use of
the python bindings for fast processing of NetFlow captures.
- CoyMon 3 will make use of Zope 3 technology for through-the-web
management of such resources as collectors, protocols, queries,
resulting data and graphics, as well as arbitrary content that is
important to end users.
- CoyMon 3 will make use of the Twisted Python framework for all of
its specialized networking needs, including (but not limited to) CMOB/X
(a recently developed "object broker" for distributed collectors
managed at a central location).
- CoyMon 3 will have completely re-factored supporting libraries,
written and maintained by the CoyoteMonnitoring community. All the old
Perl code will be replaced with light-weight, easy-to-maintain python
libraries and scripts. These will include NetCIDR, PyRRD, and
PyFlowCatch.
- CoyMon 3 will have consistent configuration across all its
composite applications and it will make use of the famous, the useful,
and the ever-concise Apache-style configuration files.
- And last, but not least, CoyMon 3 will abide by Chapter 4 of Eric
Raymond's "The The Cathedral and the Bazaar": Release Early, Release
Often. CoyMon 3 development snapshots will be available for download
regularly.
Project spaces to keep your eyes on:
CoyMonPyFlowCatchPyRRDNetCIDR
