Archive for category Monitoring Tools

Graphite’s perSecond function

So as per my previous post I came across this function because derivative and nonNegativeDerivative were not working correctly.

I tried this function out on Graphite 0.9.10 and I got exception messages that I outlined on Graphite’s Launchpad. I installed Graphite 0.9.9 and got the same exception and after looking up the documentation for Graphite 0.9.9 I noticed that function is missing. Since I got the exact same exception message I’m assuming the function was missed from 0.9.10 stable release as well. I’m still not sure if that is the real problem but hopefully someone will reply to my post on launchpad.

Since it was crucial for us to get this working I downloaded the Graphite Web development code (version 0.10) from Github project along with Ceres project and compiled and installed them. I also needed Python-byparsing to be able to resolve the dependencies. Note that I used whisper and carbon at latest stable release (0.9.10) to increase my chances of hitting bugs in development code. After installation the function was working perfectly!

So far it looks like we’re in the clear and haven’t hit any bugs…##fingers crossed##

No Comments

Graphite’s Derivative Function

Struggled with this for a day since it looked like we weren’t able to get the correct values from Graphite’s derivative and nonNegativeDerivative functions.

Found this blog that explains it pretty clearly.

Switched to perSecond() and using Graphite’s 0.9.10 it fails with an exception message. I’ve explained the problem at Graphite’s launchpad..

I think perSecond deserves its own post so to be continued…

No Comments

Choosing a metric in Graphite Web UI will return a broken image icon

This happens when Graphite render returns a 500 message (server side error).
The best way to troubleshoot is to:

  1. See the webapp log files located usually at /opt/graphite/storage/log/webapp

  2. Run the graph using the URL API and that way you can see the full exception message
  3. Most times the problem is due to permission problems on the file. Make sure all users (e.g. user running Apache HTTPD server) who need access are able to access the file.

No Comments

How to feed data into Graphite?

I came across of this when wanting to back fill the data from RRDs to Graphite/whisper… The following link explains how to use nc to insert data.

http://graphite.readthedocs.org/en/1.0/feeding-carbon.html

I simply use nc in my shell script to feed it to Carbon. Below is part of the script that does that.

CARBON_PORT="2003"
CARBON_SERVER="host1"
CARBON_PREFIX="ganglia."
GRAPHITE_PATH="${CARBON_PREFIX}${CLUSTER_GRAPHITE}.${SERVER_GRAPHITE}.${METRIC_GRAPHITE}"
echo "${GRAPHITE_PATH} ${VALUE} ${TIMESTAMP}" | nc ${CARBON_SERVER} ${CARBON_PORT}

No Comments

Graphite: How to modify the default retention?

To modify the default retention for all metrics modify storage-schemas.conf file.
So for example if you have Graphite installed at /opt/graphite then do the following:

vi /opt/graphite/conf/storage-schemas.conf

In version 0.9.10 there should be an entry for [default_1min_for_1day] which holds the .* pattern. You can obviously change the name but in order to apply the schema to all metrics you need to keep the metrics as this. You can also start defining different schemas for different metrics. In this case we’re going to change the retention for all metrics. Here’s the original retention in storage-schemas.conf.example:

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

The following changes the retention policy for all metrics to keep every 1 min for 365 days and afterwards one datapoint every 5 mins for 3 years.

[default_1min_for_1day]
pattern = .*
retentions = 60s:365d,300s:1095d

  • Its important to note that the higher retention policy the bigger disk space needed to store the data.

  • Graphite looks at the schemas and matches them to a pattern. So as it is reading the file it will apply the schema as it matches it in the file. So as I understand it the order of schemas will matter.

  • Any existing metrics created will not automatically adopt the new schema. You must use whisper-resize.py to modify the metrics to the new schema. The other option is to delete existing whisper files (/opt/graphite/storage/whisper) and restart carbon-cache.py for the files to get recreated again.

  • Restart of carbon-cache.py is required after making the change.

No Comments

How to integrate Ganglia 3.3.x+ and Graphite 0.9.x?

As of Ganglia 3.3.x the Graphite Integration plugin was integrated into gmetad. So if configured properly, when gmetad gathers the data it writes to RRDs and sends the metrics to Graphite.

Now, you might ask why would you want to integrate with Graphite?
Better graphs, more calculation options such Standard Deviation, Moving Average, many other tools that integrate to Graphite such as Tattle or Dynamic Graphing tools and much much more are reasons why you would want to integrate with Graphite.

Why would you not simply go to Graphite and skip Ganglia?
Well, in our case we needed Ganglia because we’re using HBase and Cloudera and unfortunately Cloudera provides integration to Ganglia and I believe JMX.. But if you have the option of going directly to Graphite then go for it. The Graphite website provide pretty good instructions for the install..

Finally, to integrate Ganglia 3.3.x+ with Graphite simply add the following lines to your gmetad.conf file.
carbon_server  server_name/ip
carbon_port 2003
graphite_prefix "ganglia"
  • Carbon_Port is not mandatory and can be omitted. The default port for Carbon is 2003 so if you haven’t changed it in Graphite settings then you can skip it.
  • Graphite_prefix is the name that all your clusters and metrics will reside under. Its just a directory structure.
  • You must restart gmetad service for changes to take effect.
    service gmetad restart
  • Once you restart the service you must monitor /var/log/messages for any errors.

No Comments

Installing Ganglia 3.5.x

Notes:

  • The instructions were created using CentOS 6.2 OS
  • Ganglia is made up of few components: gmetad, gmond and web UI
  • The Server includes all 3 components but the Client only includes gmond
  • These steps are for both Server and Clients. At times the instructions splits to Server and Client. If a step doesn’t include Server or Client then you can assume that the steps is for both Server and Client install.

Pre-install Steps:
  1. Selinux: Either disable it or make appropriate changes to allow the install to go through and the application will be able to listen on appropriate ports.To stop selinux permanently modify
    /etc/sysconfig/selinux
    and change
    selinux
    to
    disabled
    .
  2. IPTables: Same as Selinux. Disable it or allow ports through.Use the following commands to disable IPTables:
    service iptables stop;
    chkconfig iptables off;
    chkconfig --list iptables;
    iptables -F
    service iptables save

Installing Dependencies:

  • Server Components
    • Use Yum to install the following packages:

      • httpd.x86_64
      • httpd-devel.x86_64
      • php.x86_64
      • php-cli.x86_64
      • php-devel.x86_64
      • php-gd.x86_64
      • php-mysql.x86_64
      • php-pear.noarch
      • rrdtool-php.x86_64
      • rrdtool-devel.x86_64
      • rrdtool.x86_64
      • rrdtool-perl.x86_64
      • glibc.x86_64
      • glib2-devel.x86_64
      • glibc-devel.x86_64
      • glib2.x86_64
      • glibc-utils.x86_64
      • glibc-common.x86_64
      • gcc.x86_64
      • gcc-c++.x86_64
      • libgcc.x86_64
      • compat-gcc-34.x86_64
      • pcre.x86_64
      • pcre-devel.x86_64
      • python.x86_64
      • rrdtool-python.x86_64
      • yum install httpd.x86_64 httpd-devel.x86_64 php.x86_64 php-cli.x86_64 php-devel.x86_64 php-gd.x86_64 php-mysql.x86_64 php-pear.noarch rrdtool-php.x86_64 rrdtool-devel.x86_64 rrdtool.x86_64 rrdtool-perl.x86_64 glibc.x86_64 glib2-devel.x86_64 glibc-devel.x86_64 glib2.x86_64 glibc-utils.x86_64 glibc-common.x86_64 gcc.x86_64 gcc-c++.x86_64 libgcc.x86_64 compat-gcc-34.x86_64 pcre.x86_64 pcre-devel.x86_64 python.x86_64 rrdtool-python.x86_64
  • Client Components
    • Use Yum to install the following packages:
      • glibc.x86_64
      • glib2-devel.x86_64
      • glibc-devel.x86_64
      • glib2.x86_64
      • glibc-utils.x86_64
      • gcc-c++.x86_64
      • compat-gcc-34.x86_64
      • pcre.x86_64
      • pcre-devel.x86_64
      • python.x86_64
      • expat.x86_64
      • expat-devel.x86_64
      • apr.x86_64
      • apr-util.x86_64
      • apr-devel.x86_64
      • apr-util-devel.x86_64
      • gcc.x86_64
      • libgcc.x86_64
      • gcc-objc.x86_64
      • yum install glibc.x86_64 glib2-devel.x86_64 glibc-devel.x86_64 glib2.x86_64 glibc-utils.x86_64 gcc-c++.x86_64 compat-gcc-34.x86_64 pcre.x86_64 pcre-devel.x86_64 python.x86_64 expat.x86_64 expat-devel.x86_64 apr.x86_64 apr-util.x86_64 apr-devel.x86_64 apr-util-devel.x86_64 gcc.x86_64 libgcc.x86_64 gcc-objc.x86_64
Install Ganglia (gmetad/gmond):
  1. Install libconfuse RPMS:
    rpm --ivh libconfuse--2.6-2.el6.rf.x86_64.rpm libconfuse--devel--2.6-2.el6.rf.x86_64.rpm
  2. Decompress Ganglia gz file to /opt directory:
    tar --xzvf ganglia-3.5.0.tar.gz -C /opt
  3. Change directory to /opt and rename ganglia-3.5.0 to ganglia:
    cd /opt;
    mv ganglia-3.5.0 ganglia;
    cd ganglia;
  4. Run Configure command:
    • Server (with gmetad):
      ./configure --with-libpcre=no --with-gmetad
    • Client:
      ./configure --with-libpcre=no
  5. Run make and make install
    make
    make install
  6. (Server) Create RRD directory. Note that we run ganglia as nobody so here we change the ownership to nobody user.
    mkdir -p /var/lib/ganglia/rrds;chown -R nobody /var/lib/ganglia/rrds;
Configure GMOND:
  1. Create /etc/ganglia directory
    mkdir -p /etc/ganglia
  2. Create default configuration.
    gmond --default_config > /etc/ganglia/gmond.conf
  3. Create symbolic link from default location ganglia looks for configuration to /etc/ganglia. This is because its easier to manage all configuration files in the same place.
    ln -s /etc/ganglia/gmond.conf /usr/local/etc/gmond.conf;
  4. Change
    /etc/ganglia/gmond.conf
    to have desired metrics and port number. Note that each cluster/group must have its own unique port number. The default is 8649.
  5. Change the init file to point to correct binary:
    vi /opt/ganglia/gmond/gmond.init
    change
    GMOND
    to
    /usr/local/sbin/gmond
    .
  6. Copy gmond.init to the init.d directory and add to start up:
    cp /opt/ganglia/gmond/gmond.init /etc/rc.d/init.d/gmond;
    chkconfig --add gmond;
    chkconfig --list gmond;
Configure GMETAD:
  1. If this is a new installation then you can find the default gmetad.conf file in gmetad directory where you installed ganglia. In this case it would be
    /opt/ganglia/gmetad/gmetad.conf
    . Copy this file to
    /etc/ganglia/gmetad.conf
    .
    cp /opt/ganglia/gmetad/gmetad.conf /etc/ganglia/gmetad.conf
  2. Create a symbolic link in
    /usr/local/etc
    to the file you just copied:
    ln -s /etc/ganglia/gmetad.conf /usr/local/etc/gmetad.conf;
  3. Modify gmetad.conf and include all the clusters you wish to include in this installation.
  4. Change the init file to point to correct binary:
    vi /opt/ganglia/gmetad/gmetad.init
    change
    GMETAD
    to
    /usr/local/sbin/gmetad
    .
  5. Copy gmetad.init to the init.d directory and add to start up:
    cp /opt/ganglia/gmetad/gmetad.init /etc/rc.d/init.d/gmetad;
    chkconfig --add gmetad;
    chkconfig --list gmetad;
Install Ganglia Web
  1. Decompress Ganglia Web gz file to /usr/share directory:
    tar --xvzf ganglia--web-3.5.7.tar.gz <del>C /usr/share
  2. Change directory to /usr/share and rename gangliaweb-3.5.7 to ganglia
    cd /usr/share;<br />mv ganglia-web-3.5.7 ganglia;
  3. There are two directories required for dwoo module. Create them and change their ownership/permission as stated below:
    mkdir /usr/share/ganglia/dwoo/compiled
    mkdir /usr/share/ganglia/dwoo/cache
    chown -R 1000:1000 *
    chmod -R 777 *
Configure Ganglia Web
  1. Set correct path for where ganglia conf directory is:
    vi /usr/share/ganglia/conf_default.php
    Change $conf[‘gweb_confdir’] to the web folder. e.g.
    $conf['gweb_confdir'] = "/usr/share/ganglia";
  2. Add Ganglia web folder to Apache HTTPD:
    • Create a new conf file in /etc/httpd/conf.d
      vi /etc/httpd/conf.d/ganglia.conf
      The following is a sample you can use:
      <location /ganglia>
      Order deny,allow
      Allow from all
      </location>
      
Start all services
  1. Start gmetad service
    service gmetad start
    Check /var/log/messages.log to make sure there aren’t any errors.
  2. Start GMETAD service
    service gmond start
    Check /var/log/messages.log to make sure there aren’t any errors.
  3. Restart HTTPD service
    service httpd restart

No Comments