VictoriaMetrics.github.io

Latest Release Slack GitHub license Go Report Build Status codecov

Victoria Metrics

Single-node VictoriaMetrics

VictoriaMetrics is fast, cost-effective and scalable time-series database. It can be used as long-term remote storage for Prometheus. It is available in binary releases, docker images and in source code.

Cluster version is available here.

Prominent features

Operation

Table of contents

How to start VictoriaMetrics

Just start VictoriaMetrics executable or docker image with the desired command-line flags.

The following command-line flags are used the most:

Pass -help to see all the available flags with description and default values.

It is recommended setting up monitoring for VictoriaMetrics.

Prometheus setup

Add the following lines to Prometheus config file (it is usually located at /etc/prometheus/prometheus.yml):

remote_write:
  - url: http://<victoriametrics-addr>:8428/api/v1/write
    queue_config:
      max_samples_per_send: 10000
      max_shards: 30

Substitute <victoriametrics-addr> with the hostname or IP address of VictoriaMetrics. Then apply the new config via the following command:

kill -HUP `pidof prometheus`

Prometheus writes incoming data to local storage and replicates it to remote storage in parallel. This means the data remains available in local storage for --storage.tsdb.retention.time duration even if remote storage is unavailable.

If you plan to send data to VictoriaMetrics from multiple Prometheus instances, then add the following lines into global section of Prometheus config:

global:
  external_labels:
    datacenter: dc-123

This instructs Prometheus to add datacenter=dc-123 label to each time series sent to remote storage. The label name may be arbitrary - datacenter is just an example. The label value must be unique across Prometheus instances, so those time series may be filtered and grouped by this label.

It is recommended upgrading Prometheus to v2.12.0 or newer, since the previous versions may have issues with remote_write.

Grafana setup

Create Prometheus datasource in Grafana with the following Url:

http://<victoriametrics-addr>:8428

Substitute <victoriametrics-addr> with the hostname or IP address of VictoriaMetrics.

Then build graphs with the created datasource using Prometheus query language. VictoriaMetrics supports native PromQL and extends it with useful features.

How to upgrade VictoriaMetrics?

It is safe upgrading VictoriaMetrics to new versions unless release notes say otherwise. It is recommended performing regular upgrades to the latest version, since it may contain important bug fixes, performance optimizations or new features.

Follow the following steps during the upgrade:

1) Send SIGINT signal to VictoriaMetrics process in order to gracefully stop it. 2) Wait until the process stops. This can take a few seconds. 3) Start the upgraded VictoriaMetrics.

Prometheus doesn’t drop data during VictoriaMetrics restart. See this article for details.

How to apply new config to VictoriaMetrics?

VictoriaMetrics must be restarted for applying new config:

1) Send SIGINT signal to VictoriaMetrics process in order to gracefully stop it. 2) Wait until the process stops. This can take a few seconds. 3) Start VictoriaMetrics with the new config.

Prometheus doesn’t drop data during VictoriaMetrics restart. See this article for details.

How to send data from InfluxDB-compatible agents such as Telegraf?

Just use http://<victoriametric-addr>:8428 url instead of InfluxDB url in agents’ configs. For instance, put the following lines into Telegraf config, so it sends data to VictoriaMetrics instead of InfluxDB:

[[outputs.influxdb]]
  urls = ["http://<victoriametrics-addr>:8428"]

Do not forget substituting <victoriametrics-addr> with the real address where VictoriaMetrics runs.

VictoriaMetrics maps Influx data using the following rules:

For example, the following Influx line:

foo,tag1=value1,tag2=value2 field1=12,field2=40

is converted into the following Prometheus data points:

foo_field1{tag1="value1", tag2="value2"} 12
foo_field2{tag1="value1", tag2="value2"} 40

Example for writing data with Influx line protocol to local VictoriaMetrics using curl:

curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST 'http://localhost:8428/write'

An arbitrary number of lines delimited by ‘\n’ may be sent in a single request. After that the data may be read via /api/v1/export endpoint:

curl -G 'http://localhost:8428/api/v1/export' -d 'match={__name__!=""}'

The /api/v1/export endpoint should return the following response:

{"metric":{"__name__":"measurement_field1","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560272508147]}
{"metric":{"__name__":"measurement_field2","tag1":"value1","tag2":"value2"},"values":[1.23],"timestamps":[1560272508147]}

Note that Influx line protocol expects timestamps in nanoseconds by default, while VictoriaMetrics stores them with milliseconds precision.

How to send data from Graphite-compatible agents such as StatsD?

1) Enable Graphite receiver in VictoriaMetrics by setting -graphiteListenAddr command line flag. For instance, the following command will enable Graphite receiver in VictoriaMetrics on TCP and UDP port 2003:

/path/to/victoria-metrics-prod -graphiteListenAddr=:2003

2) Use the configured address in Graphite-compatible agents. For instance, set graphiteHost to the VictoriaMetrics host in StatsD configs.

Example for writing data with Graphite plaintext protocol to local VictoriaMetrics using nc:

echo "foo.bar.baz;tag1=value1;tag2=value2 123 `date +%s`" | nc -N localhost 2003

VictoriaMetrics sets the current time if the timestamp is omitted. An arbitrary number of lines delimited by \n may be sent in one go. After that the data may be read via /api/v1/export endpoint:

curl -G 'http://localhost:8428/api/v1/export' -d 'match={__name__!=""}'

The /api/v1/export endpoint should return the following response:

{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]}

Querying Graphite data

Data sent to VictoriaMetrics via Graphite plaintext protocol may be read either via Prometheus querying API or via go-graphite/carbonapi.

How to send data from OpenTSDB-compatible agents?

VictoriaMetrics supports telnet put protocol and HTTP /api/put requests for ingesting OpenTSDB data.

Sending data via telnet put protocol

1) Enable OpenTSDB receiver in VictoriaMetrics by setting -opentsdbListenAddr command line flag. For instance, the following command enables OpenTSDB receiver in VictoriaMetrics on TCP and UDP port 4242:

/path/to/victoria-metrics-prod -opentsdbListenAddr=:4242

2) Send data to the given address from OpenTSDB-compatible agents.

Example for writing data with OpenTSDB protocol to local VictoriaMetrics using nc:

echo "put foo.bar.baz `date +%s` 123 tag1=value1 tag2=value2" | nc -N localhost 4242

An arbitrary number of lines delimited by \n may be sent in one go. After that the data may be read via /api/v1/export endpoint:

curl -G 'http://localhost:8428/api/v1/export' -d 'match={__name__!=""}'

The /api/v1/export endpoint should return the following response:

{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277292000]}

Sending OpenTSDB data via HTTP /api/put requests

1) Enable HTTP server for OpenTSDB /api/put requests by setting -opentsdbHTTPListenAddr command line flag. For instance, the following command enables OpenTSDB HTTP server on port 4242:

/path/to/victoria-metrics-prod -opentsdbHTTPListenAddr=:4242

2) Send data to the given address from OpenTSDB-compatible agents.

Example for writing a single data point:

curl -H 'Content-Type: application/json' -d '{"metric":"x.y.z","value":45.34,"tags":{"t1":"v1","t2":"v2"}}' http://localhost:4242/api/put

Example for writing multiple data points in a single request:

curl -H 'Content-Type: application/json' -d '[{"metric":"foo","value":45.34},{"metric":"bar","value":43}]' http://localhost:4242/api/put

After that the data may be read via /api/v1/export endpoint:

curl -G 'http://localhost:8428/api/v1/export' -d 'match[]=x.y.z' -d 'match[]=foo' -d 'match[]=bar'

The /api/v1/export endpoint should return the following response:

{"metric":{"__name__":"foo"},"values":[45.34],"timestamps":[1566464846000]}
{"metric":{"__name__":"bar"},"values":[43],"timestamps":[1566464846000]}
{"metric":{"__name__":"x.y.z","t1":"v1","t2":"v2"},"values":[45.34],"timestamps":[1566464763000]}

How to build from sources

We recommend using either binary releases or docker images instead of building VictoriaMetrics from sources. Building from sources is reasonable when developing additional features specific to your needs.

Development build

  1. Install Go. The minimum supported version is Go 1.12.
  2. Run make victoria-metrics from the root folder of the repository. It builds victoria-metrics binary and puts it into the bin folder.

Production build

  1. Install docker.
  2. Run make victoria-metrics-prod from the root folder of the repository. It builds victoria-metrics-prod binary and puts it into the bin folder.

ARM build

ARM build may run on Raspberry Pi or on energy-efficient ARM servers.

Development ARM build

  1. Install Go. The minimum supported version is Go 1.12.
  2. Run make victoria-metrics-arm or make victoria-metrics-arm64 from the root folder of the repository. It builds victoria-metrics-arm or victoria-metrics-arm64 binary respectively and puts it into the bin folder.

Production ARM build

  1. Install docker.
  2. Run make victoria-metrics-arm-prod or make victoria-metrics-arm64-prod from the root folder of the repository. It builds victoria-metrics-arm-prod or victoria-metrics-arm64-prod binary respectively and puts it into the bin folder.

Pure Go build (CGO_ENABLED=0)

Pure Go mode builds only Go code without cgo dependencies. This is an experimental mode, which may result in a lower compression ratio and slower decompression performance. Use it with caution!

  1. Install Go. The minimum supported version is Go 1.12.
  2. Run make victoria-metrics-pure from the root folder of the repository. It builds victoria-metrics-pure binary and puts it into the bin folder.

Building docker images

Run make package-victoria-metrics. It builds victoriametrics/victoria-metrics:<PKG_TAG> docker image locally. <PKG_TAG> is auto-generated image tag, which depends on source code in the repository. The <PKG_TAG> may be manually set via PKG_TAG=foobar make package-victoria-metrics.

Start with docker-compose

Docker-compose helps to spin up VictoriaMetrics, Prometheus and Grafana with one command. More details may be found here.

Setting up service

Read these instructions on how to set up VictoriaMetrics as a service in your OS.

Third-party contributions

How to work with snapshots?

VictoriaMetrics can create instant snapshots for all the data stored under -storageDataPath directory. Navigate to http://<victoriametrics-addr>:8428/snapshot/create in order to create an instant snapshot. The page will return the following JSON response:

{"status":"ok","snapshot":"<snapshot-name>"}

Snapshots are created under <-storageDataPath>/snapshots directory, where <-storageDataPath> is the command-line flag value. Snapshots can be archived to backup storage at any time with vmbackup.

The http://<victoriametrics-addr>:8428/snapshot/list page contains the list of available snapshots.

Navigate to http://<victoriametrics-addr>:8428/snapshot/delete?snapshot=<snapshot-name> in order to delete <snapshot-name> snapshot.

Navigate to http://<victoriametrics-addr>:8428/snapshot/delete_all in order to delete all the snapshots.

Steps for restoring from a snapshot:

  1. Stop VictoriaMetrics with kill -INT.
  2. Remove the entire contents of the directory pointed by -storageDataPath command-line flag.
  3. Restore snapshot contents from backup with vmrestore to the directory pointed by -storageDataPath.
  4. Start VictoriaMetrics.

How to delete time series?

Send a request to http://<victoriametrics-addr>:8428/api/v1/admin/tsdb/delete_series?match[]=<timeseries_selector_for_delete>, where <timeseries_selector_for_delete> may contain any time series selector for metrics to delete. After that all the time series matching the given selector are deleted. Storage space for the deleted time series isn’t freed instantly - it is freed during subsequent merges of data files.

It is recommended verifying which metrics will be deleted with the call to http://<victoria-metrics-addr>:8428/api/v1/series?match[]=<timeseries_selector_for_delete> before actually deleting the metrics.

How to export time series?

Send a request to http://<victoriametrics-addr>:8428/api/v1/export?match[]=<timeseries_selector_for_export>, where <timeseries_selector_for_export> may contain any time series selector for metrics to export. The response would contain all the data for the selected time series in JSON streaming format. Each JSON line would contain data for a single time series. An example output:

{"metric":{"__name__":"up","job":"node_exporter","instance":"localhost:9100"},"values":[0,0,0],"timestamps":[1549891472010,1549891487724,1549891503438]}
{"metric":{"__name__":"up","job":"prometheus","instance":"localhost:9090"},"values":[1,1,1],"timestamps":[1549891461511,1549891476511,1549891491511]}

Optional start and end args may be added to the request in order to limit the time frame for the exported data. These args may contain either unix timestamp in seconds or RFC3339 values.

Federation

VictoriaMetrics exports Prometheus-compatible federation data at http://<victoriametrics-addr>:8428/federate?match[]=<timeseries_selector_for_federation>.

Optional start and end args may be added to the request in order to scrape the last point for each selected time series on the [start ... end] interval. start and end may contain either unix timestamp in seconds or RFC3339 values. By default, the last point on the interval [now - max_lookback ... now] is scraped for each time series. The default value for max_lookback is 5m (5 minutes), but it can be overridden. For instance, /federate?match[]=up&max_lookback=1h would return last points on the [now - 1h ... now] interval. This may be useful for time series federation with scrape intervals exceeding 5m.

Capacity planning

A rough estimation of the required resources for ingestion path:

The required resources for query path:

High availability

1) Install multiple VictoriaMetrics instances in distinct datacenters (availability zones). 2) Add addresses of these instances to remote_write section in Prometheus config:

remote_write:
  - url: http://<victoriametrics-addr-1>:8428/api/v1/write
    queue_config:
      max_samples_per_send: 10000
  # ...
  - url: http://<victoriametrics-addr-N>:8428/api/v1/write
    queue_config:
      max_samples_per_send: 10000

3) Apply the updated config:

kill -HUP `pidof prometheus`

4) Now Prometheus should write data into all the configured remote_write urls in parallel. 5) Set up Promxy in front of all the VictoriaMetrics replicas. 6) Set up Prometheus datasource in Grafana that points to Promxy.

If you have Prometheus HA pairs with replicas r1 and r2 in each pair, then configure each r1 to write data to victoriametrics-addr-1, while each r2 should write data to victoriametrics-addr-2.

Multiple retentions

Just start multiple VictoriaMetrics instances with distinct values for the following flags:

Downsampling

There is no downsampling support at the moment, but:

These properties reduce the need in downsampling. We plan to implement downsampling in the future. See this issue for details.

Multi-tenancy

Single-node VictoriaMetrics doesn’t support multi-tenancy. Use cluster version instead.

Scalability and cluster version

Though single-node VictoriaMetrics cannot scale to multiple nodes, it is optimized for resource usage - storage size / bandwidth / IOPS, RAM, CPU. This means that a single-node VictoriaMetrics may scale vertically and substitute a moderately sized cluster built with competing solutions such as Thanos, Uber M3, InfluxDB or TimescaleDB. See vertical scalability benchmarks.

So try single-node VictoriaMetrics at first and then switch to cluster version if you still need horizontally scalable long-term remote storage for really large Prometheus deployments. Contact us for paid support.

Alerting

VictoriaMetrics doesn’t support rule evaluation and alerting yet, so these actions must be performed either on Prometheus side or on Grafana side.

Security

Do not forget protecting sensitive endpoints in VictoriaMetrics when exposing it to untrusted networks such as the internet. Consider setting the following command-line flags:

Explicitly set internal network interface for TCP and UDP ports for data ingestion with Graphite and OpenTSDB formats. For example, substitute -graphiteListenAddr=:2003 with -graphiteListenAddr=<internal_iface_ip>:2003.

Tuning

mkfs.ext4 ... -O 64bit,huge_file,extent -T huge

Monitoring

VictoriaMetrics exports internal metrics in Prometheus format on the /metrics page. Add this page to Prometheus’ scrape config in order to collect VictoriaMetrics metrics. There is an official Grafana dashboard for single-node VictoriaMetrics.

The most interesting metrics are:

Troubleshooting

Backfilling

Make sure that configured -retentionPeriod covers timestamps for the backfilled data.

It is recommended disabling query cache with -search.disableCache command-line flag when writing historical data with timestamps from the past, since the cache assumes that the data is written with the current timestamps. Query cache can be enabled after the backfilling is complete.

Profiling

VictoriaMetrics provides handlers for collecting the following Go profiles:

The command for collecting CPU profile waits for 30 seconds before returning.

The collected profiles may be analyzed with go tool pprof.

Integrations

Roadmap

The discussion happens here. Feel free to comment any item or add own one.

Contacts

Contact us with any questions regarding VictoriaMetrics at info@victoriametrics.com.

Community and contributions

Feel free asking any questions regarding VictoriaMetrics:

If you like VictoriaMetrics and want to contribute, then we need the following:

We are open to third-party pull requests provided they follow KISS design principle:

Adhering KISS principle simplifies the resulting code and architecture, so it can be reviewed, understood and verified by many people.

Reporting bugs

Report bugs and propose new features here.

Zip contains three folders with different image orientation (main color and inverted version).

Files included in each folder:

Logo Usage Guidelines

Font used:

Color Palette:

We kindly ask: