Monitoring Yacy in Cacti

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Monitoring Yacy in Cacti

Beitragvon oneaty » Fr Jun 06, 2014 1:08 pm

After using Cacti to monitor the server(http://forum.yacy-websuche.de/viewtopic.php?f=23&t=5201), I started monitoring Yacy itself.
There are two possible aproaches when you want to use Cacti for monitoring: by collecting monitoring data thru any SNMP OID (either pre-package or custom ones) or thru a script.
I've chosen the second alternative (not exactly sure why :roll: ) and I've been collecting and monitoring Yacy data from two sources: http://myserver:8090/Network.xml and http://myserver:8090/PerformanceMemory_p.xml.
Below follow some screenshots of what you get, as well as attachments for the Cacti templates and the scripts for those who want to try.
DISCLAIMER: Those two C scripts were my first C programs (I'm an old COBOL programmer 8-) so they probably can be optimized and better stylished. Nevertheless, they're running smoothly (at least in my Ubuntu 14.04 server).

Bild

Bild

(Note: The vertical gap in the graphs is a power outage that happened yesterday)

In Cacti, you can expand any of the graphs above by clicking on them. Below, a screenshot of one of them:

Bild

Also, I had some minor issues regarding file locks, when executing the scripts, due to the way I designed the data collecting process: instead of directly reading the web xml produced by Yacy (too much for my zero knowledge in C), I chose to create two crontab entries to download the xml from Yacy to local files aimed to be the script inputs. The issue was that sometimes Cacti was executing the scripts at the exact same time as wget, so sometimes they were not producing any data. I solved this by using a temporary file and delaying the download process by some seconds. The two crontab entries are:

* * * * * sleep 30; wget -O /home/user/bin/tmp.xml http://ipaddress:8090/Network.xml > /home/user/log/yacy_wget.log 2>&1; mv /home/user/bin/tmp.xml /home/user/bin/Network.xml
* * * * * sleep 30; curl -o /home/user/bin/tmp1.xml -u admin:password http://ipaddress:8090/PerformanceMemory_p.xml > /home/user/log/yacy_curl.log 2>&1; mv /home/user/bin/tmp1.xml /home/user/bin/PerformanceMemory_p.xml

(you may want to customize user, ipaddress and password to your linux user, server ip and yacy admin password, respectively)

The scrips run every minute to be in sync with Cacti polling interval, which, in my case, is a 1 minute interval.
Dateianhänge
scripts.tar
(20 KiB) 92-mal heruntergeladen
cacti_host_template_yacy_server.tar
(120 KiB) 99-mal heruntergeladen
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Monitoring Yacy in Cacti

Beitragvon Orbiter » Fr Jun 06, 2014 7:09 pm

very impressive! good work!
I need some time to test that all...
Orbiter
 
Beiträge: 5784
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Monitoring Yacy in Cacti

Beitragvon oneaty » Sa Jun 07, 2014 12:37 pm

Please, take note that this is not a comprehensive step-by-step guide.

It assumes you already have a Cacti installation up and running and the minimum knowledge on Cacti as well as on basic Linux permissions concepts (specially required when installing Cacti).

(Take your time to learn a bit about those; it will save time later)

Cacti installation can be a little tricky, but once installed, it runs smoothly and is very stable (mine is running for almost a year without issues).

Also take note that you can change the graphs templates that come within the host template tar (to be imported in Cacti) after you created graphs based on them, but if you don't want to loose previous data, you have to restrict them to cosmetics (color, type of graph - line, area, stack... - and the many check-boxes at bottom). If you add or remove any data input attribute, the graphs created from that template will stop graphing. You should delete and recreate them (the graphs, not the template).
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Monitoring Yacy in Cacti

Beitragvon Orbiter » So Jun 08, 2014 10:11 am

a step-by-step guide in our wiki would be phantastic!

I just twittered this topic with a link to this forum: https://twitter.com/yacy_search/status/ ... 1145468928
Orbiter
 
Beiträge: 5784
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Monitoring Yacy in Cacti

Beitragvon oneaty » So Jun 08, 2014 11:22 am

I will try to make that guide in the following two weeks.

For those who don't want to wait, a very small high level guide would be:

1 - Install SNMP. Although my solution to monitoring Yacy doesn't use SNMP, it's important to install it so that you can create some basic Cacti graphs like Memory Usage and CPU Usage, just to check that your Cacti installation is correct.
After installing SNMP, make sure it is up and running by issuing the following command:

Code: Alles auswählen
snmpwalk -v 1 -c public localhost


That should produce a list of OID's and their current values. (Please refer to the wikipedia article on OID in what it refers to SNMP: http://en.wikipedia.org/wiki/Object_identifier)

2 - Install Cacti and Cacti Spine. (Spine is an enhanced poller written in C, thus much more efficient than the default Cacti poller cmd.php)

After installation, make sure you change Cacti and Spine conf files (as per Cacti wiki) to reflect the database name, user and password.

3 - In your browser, go to http://127.0.0.1/cacti, follow the instructions for first login and if everything is ok, add a device to monitor.
You will typically add your own machine. This is only to produce those graphs I mentioned before, to make sure everything is ok.
Just fill in a name and the ipaddress (if your own machine, 127.0.0.1). In host template drop-down menu, choose Local Linux Machine.
After saving, the left top corner of the screen will show some basic information about the machine (or an SNMP error, if something is wrong)

4 - In the device's screen, click on Create Graphs for This Host.
Select one graph, click create. (try to select as many as you can).

5 - Go to the Graphs tab.

You will have to wait around 5 to 10 minutes before the graph begins to show up.

6 - If Cacti is working, then proceed with this.

7 - Download and compile the C scripts (I did it with gcc -O binary-name source-name).
Execute the binaries to make sure everything is ok. You should see some yacy stats filled with values.

8 - Move the binary to Cacti scripts folder.

9 - In Cacti, import the templates attached in my first post.

10 - After that, you should have a new device named Yacy Server, and when you try to create graphs for them, you should see a list of graphs like SNMP Peer ... and SNMP Yacy ....

11 - The scripts are currently sending its output also to syslog, in case you want to check if they're running and producing values.

I'll get back
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Monitoring Yacy in Cacti

Beitragvon oneaty » Mi Jun 18, 2014 6:28 pm

The graphs above are now turned public at http://cactibr.noip.me/cacti (guest/guest for user/passwd)

You may want to click on the Thumbnails checkbox to get a broader view, or click on "Servidor" (on the left vertical panel) to see the performance graphs of the linux server where Yacy is running.

You can also click on any graph to have a detailed view of each of them.
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Monitoring Yacy in Cacti

Beitragvon oneaty » Di Jul 01, 2014 7:08 pm

Since early this morning, the link to my Cacti server referred in the previous message was taken down by Microsoft, one among many others that were using the free DNS No-IP service.

The story can be read in deeper details here: http://www.theregister.co.uk/2014/07/01/microsoft_takes_over_noip_domains_to_block_malware_marketing/

If Microsoft feels it has the right to seize a legal service because two persons were abusing it, why can't I seize hotmail domain, since it is also obviously misused by spammers and other types of "cybercriminals"?

UPDATE: I changed the No-IP host name to cactibr.ddns.net, one of the few No-IP domains left behind Microsoft seizure, so the current link to my Cacti server is http://cactibr.ddns.net/cacti.

Now I'm not sure how many time this one wil be kept alive...
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast

cron