Yacy stats SNMP trapping

Keine Scheu, hier darf alles gefragt und diskutiert werden. Das ist das Forum für YaCy-Anfänger. Hier kann man 'wo muss man klicken' fragen und sich über Grundlagen zur Suchmaschinentechnik unterhalten.
Forumsregeln
Hier werden Fragen beantwortet und wir versuchen die Probleme von YaCy-Newbies zu klären. Bitte beantwortete Fragen im YaCy-Wiki http://wiki.yacy.de dokumentieren!

Yacy stats SNMP trapping

Beitragvon oneaty » Do Mai 29, 2014 2:56 pm

I'm interested in trapping Yacy System Status via SNMP and graph those stats in Cacti.

Questions:

1 - Are there any OID's defined for Yacy?

2 - If the above is NO, is there any shell script available that returns those stats?

If #1 is Yes, then graphing Yacy stats in Cacti is pretty straightforward.

If #1 is NO, then if I had a script that returns those stats, I could easily create a custom OID so that Cacti could read it.
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Yacy stats SNMP trapping

Beitragvon Orbiter » Mo Jun 02, 2014 1:13 am

unfortunately there is no such interface in YaCy, but monitoring is definitely an issue. Usage of Cacti for that would be nice.
Is there a minimum example how such an OID/SNMP interface shall look like?
Orbiter
 
Beiträge: 5769
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Yacy stats SNMP trapping

Beitragvon oneaty » Fr Jun 06, 2014 12:30 pm

Yes.
Like I suggested above, there are two possible aproaches when you want to use Cacti for monitoring: by collecting monitoring data thru any SNMP OID (either pre-package or custom ones) or thru a script.
I've chosen the second alternative (not exactly sure why :roll: ) and I've been collecting and monitoring Yacy data from two sources: http://myserver:8090/Network.xml and http://myserver:8090/PerformanceMemory_p.xml.
Below follow some screenshots of what you get, as well as attachments for the Cacti templates and the scripts for those who want to try.
DISCLAIMER: Those two C scripts were my first C programs (I'm an old COBOL programmer 8-) so they probably can be optimized and better stylished. Nevertheless, they're running smoothly (at least in my Ubuntu 14.04 server).

Bild

Bild

(Note: The vertical gap in the graphs is a power outage that happened yesterday)

In Cacti, you can expand any of the graphs above by clicking on them. Below, a screenshot of one of them:

Bild

Also, I had some minor issues regarding file locks, when executing the scripts, due to the way I designed the data collecting process: instead of directly reading the web xml produced by Yacy (too much for my zero knowledge in C), I chose to create two crontab entries to download the xml from Yacy to local files aimed to be the script inputs. The issue was that sometimes Cacti was executing the scripts at the exact same time as wget, so sometimes they were not producing any data. I solved this by using a temporary file and delaying the download process by some seconds. The two crontab entries are:

* * * * * sleep 30; wget -O /home/user/bin/tmp.xml http://ipaddress:8090/Network.xml > /home/user/log/yacy_wget.log 2>&1; mv /home/user/bin/tmp.xml /home/user/bin/Network.xml
* * * * * sleep 30; curl -o /home/user/bin/tmp1.xml -u admin:password http://ipaddress:8090/PerformanceMemory_p.xml > /home/user/log/yacy_curl.log 2>&1; mv /home/user/bin/tmp1.xml /home/user/bin/PerformanceMemory_p.xml

(you may want to customize user, ipaddress and password to your linux user, server ip and yacy admin password, respectively)

The scrips run every minute to be in sync with Cacti polling interval, which, in my case, is a 1 minute interval.
Dateianhänge
cacti_host_template_yacy_server.tar
(120 KiB) 61-mal heruntergeladen
scripts.tar
(20 KiB) 66-mal heruntergeladen
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Yacy stats SNMP trapping

Beitragvon oneaty » Fr Jun 06, 2014 1:55 pm

Now I realize I didn't answer Orbiter's question.

As far as my SNMP knowledge goes, an SNMP interface for a specific package like Yacy means creating new OID's for each variable you want to monitor.
Example: a OID for Free Memory, a OID for Used Memory, etc.
You can create new OID's by adding directives in snmpd.conf file, much like as in here (http://www.zwamneus.nl/creating-a-custom-snmp-oid-for-a-bash-script/)

But as far as Cacti (and other monitoring tools like Zabix) is concerned, a SNMP OID as data input method is one of the alternatives, the other being a stand-alone script like the ones I showed above.

So, in summary, if you want to let Yacy be monitored by tools like Cacti, make sure you provide xml pages with as many attributes as you may want to monitor.

In my humble opinion, you don't need to focus in creating a SNMP interface since you already provide the xml interfaces that can easily act as data input methods for those tools.

My suggestion would be, maybe, to add some more attributes to Network.xml (or create a new xml for those new attributes), for example, disk usage, or any other attribute that currently doesn't show in the xmls but is already present in the /Status pages.
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Yacy stats SNMP trapping

Beitragvon oneaty » Sa Jun 07, 2014 12:37 pm

Please, take note that this is not a comprehensive step-by-step guide.

It assumes you already have a Cacti installation up and running and the minimum knowledge on Cacti as well as on basic Linux permissions concepts (specially required when installing Cacti).

(Take your time to learn a bit about those; it will save time later)

Cacti installation can be a little tricky, but once installed, it runs smoothly and is very stable (mine is running for almost a year without issues).

Also take note that you can change the graphs templates that come within the host template tar (to be imported in Cacti) after you created graphs based on them, but if you don't want to loose previous data, you have to restrict them to cosmetics (color, type of graph - line, area, stack... - and the many check-boxes at bottom). If you add or remove any data input attribute, the graphs created from that template will stop graphing. You should delete and recreate them (the graphs, not the template).
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Yacy stats SNMP trapping

Beitragvon oneaty » Mi Jun 18, 2014 6:27 pm

The graphs above are now turned public at http://cactibr.noip.me/cacti (guest/guest for user/passwd)

You may want to click on the Thumbnails checkbox to get a broader view, or click on "Servidor" (on the left vertical panel) to see the performance graphs of the linux server where Yacy is running.

You can also click on any graph to have a detailed view of each of them.
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Yacy stats SNMP trapping

Beitragvon oneaty » Di Jul 01, 2014 7:08 pm

Since early this morning, the link to my Cacti server referred in the previous message was taken down by Microsoft, one among many others that were using the free DNS No-IP service.

The story can be read in deeper details here: http://www.theregister.co.uk/2014/07/01/microsoft_takes_over_noip_domains_to_block_malware_marketing/

If Microsoft feels it has the right to seize a legal service because two persons were abusing it, why can't I seize hotmail domain, since it is also obviously misused by spammers and other types of "cybercriminals"?

UPDATE: I changed the No-IP host name to cactibr.ddns.net, one of the few No-IP domains left behind Microsoft seizure, so the current link to my Cacti server is http://cactibr.ddns.net/cacti.

Now I'm not sure how many time this one wil be kept alive...
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro


Zurück zu Hilfe für Einsteiger und Anwender

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast

cron