Archive for category Graphite

Graphite’s perSecond function

So as per my previous post I came across this function because derivative and nonNegativeDerivative were not working correctly.

I tried this function out on Graphite 0.9.10 and I got exception messages that I outlined on Graphite’s Launchpad. I installed Graphite 0.9.9 and got the same exception and after looking up the documentation for Graphite 0.9.9 I noticed that function is missing. Since I got the exact same exception message I’m assuming the function was missed from 0.9.10 stable release as well. I’m still not sure if that is the real problem but hopefully someone will reply to my post on launchpad.

Since it was crucial for us to get this working I downloaded the Graphite Web development code (version 0.10) from Github project along with Ceres project and compiled and installed them. I also needed Python-byparsing to be able to resolve the dependencies. Note that I used whisper and carbon at latest stable release (0.9.10) to increase my chances of hitting bugs in development code. After installation the function was working perfectly!

So far it looks like we’re in the clear and haven’t hit any bugs…##fingers crossed##

No Comments

Graphite’s Derivative Function

Struggled with this for a day since it looked like we weren’t able to get the correct values from Graphite’s derivative and nonNegativeDerivative functions.

Found this blog that explains it pretty clearly.

Switched to perSecond() and using Graphite’s 0.9.10 it fails with an exception message. I’ve explained the problem at Graphite’s launchpad..

I think perSecond deserves its own post so to be continued…

No Comments

Choosing a metric in Graphite Web UI will return a broken image icon

This happens when Graphite render returns a 500 message (server side error).
The best way to troubleshoot is to:

  1. See the webapp log files located usually at /opt/graphite/storage/log/webapp

  2. Run the graph using the URL API and that way you can see the full exception message
  3. Most times the problem is due to permission problems on the file. Make sure all users (e.g. user running Apache HTTPD server) who need access are able to access the file.

No Comments

How to feed data into Graphite?

I came across of this when wanting to back fill the data from RRDs to Graphite/whisper… The following link explains how to use nc to insert data.

http://graphite.readthedocs.org/en/1.0/feeding-carbon.html

I simply use nc in my shell script to feed it to Carbon. Below is part of the script that does that.

CARBON_PORT="2003"
CARBON_SERVER="host1"
CARBON_PREFIX="ganglia."
GRAPHITE_PATH="${CARBON_PREFIX}${CLUSTER_GRAPHITE}.${SERVER_GRAPHITE}.${METRIC_GRAPHITE}"
echo "${GRAPHITE_PATH} ${VALUE} ${TIMESTAMP}" | nc ${CARBON_SERVER} ${CARBON_PORT}

No Comments

Graphite: How to modify the default retention?

To modify the default retention for all metrics modify storage-schemas.conf file.
So for example if you have Graphite installed at /opt/graphite then do the following:

vi /opt/graphite/conf/storage-schemas.conf

In version 0.9.10 there should be an entry for [default_1min_for_1day] which holds the .* pattern. You can obviously change the name but in order to apply the schema to all metrics you need to keep the metrics as this. You can also start defining different schemas for different metrics. In this case we’re going to change the retention for all metrics. Here’s the original retention in storage-schemas.conf.example:

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

The following changes the retention policy for all metrics to keep every 1 min for 365 days and afterwards one datapoint every 5 mins for 3 years.

[default_1min_for_1day]
pattern = .*
retentions = 60s:365d,300s:1095d

  • Its important to note that the higher retention policy the bigger disk space needed to store the data.

  • Graphite looks at the schemas and matches them to a pattern. So as it is reading the file it will apply the schema as it matches it in the file. So as I understand it the order of schemas will matter.

  • Any existing metrics created will not automatically adopt the new schema. You must use whisper-resize.py to modify the metrics to the new schema. The other option is to delete existing whisper files (/opt/graphite/storage/whisper) and restart carbon-cache.py for the files to get recreated again.

  • Restart of carbon-cache.py is required after making the change.

No Comments