Features

What companies choose muscles for

• Rollup ladders Per-second, per-minute and per-hour totals are kept as rows arrive, so a dashboard over years of data reads a few summary rows.
• Tables that live elsewhere Query Iceberg, PostgreSQL, ClickHouse, Elasticsearch and more in place, and join them with each other and with your stored tables.
• Durable writes A write is acknowledged after it is on disk, so a crash after that loses nothing, and a retried batch is stored once.
• Kafka ingest Avro topics become tables with the columns of their schema, and a record is acknowledged after it is synced.
• Cold tier on S3 Parts older than an age you set move to your bucket and stay queryable.
• ClickHouse HTTP interface ClickHouse drivers, Grafana, JDBC tools and Polars connect over HTTP.
• Column policies and masking One statement masks or refuses a column for a role, on the one copy of the table.
• Full-text and vector search Rank text with BM25 and embeddings by distance inside SQL, with no search cluster beside the database.
• Tamper-evident audit log Who ran what, in hash-chained blocks that a check can prove intact.
• Prometheus remote write Prometheus pushes its metrics in, and you query them with SQL and join them to your tables.
• Alerts in SQL A rule over a query result fires to Alertmanager or a webhook.
• One binary, then a cluster One static binary, a container image, a Helm chart and an operator; add query, ingest and data nodes when one server is not enough.

Time series, rollups and logs

Tables sort by time, rollups answer time-bucket queries from rows that are already summed up, and views keep aggregates up to date as rows arrive.

CREATE ROLLUP ticks_r ON ticks KEY (sym)
  AGGREGATE (count(*) AS n, sum(qty) AS vol, min(px) AS lo,
             max(px) AS hi, avg(px) AS mid)
  RESOLUTIONS (1s, 1m, 1h);

SELECT toStartOfMinute(ts) AS minute, sym,
       sum(qty) AS vol, min(px) AS lo, max(px) AS hi
FROM ticks GROUP BY minute, sym ORDER BY minute, sym;

• Time-bucket queries use the rollup on their own Available

  A query that groups by toStartOfMinute, toStartOfHour, date_trunc or toStartOfInterval reads the coarsest rung that fits plus the newest raw rows, and returns the same answer as a scan of the raw table.

• Raw rows age out, rollups stay Available

  Once a rung holds a time range in full, muscles can drop the raw parts for it, so raw ticks age out while the minute and hour rungs stay. A query that reaches into the dropped range is refused, or returns a partial answer marked as such when you ask for one.

• Views stay small Available

  A view keeps one row per group, and background merges add up the partial counts, sums, minimums and maximums.

• Backfill and rebuild of views Available

  CREATE MATERIALIZED VIEW ... TO t BACKFILL fills a new view from the rows already in the source, and REBUILD MATERIALIZED VIEW replaces its contents. Both survive a crash and replicate.

• A view that runs out of memory lags and catches up Available

  All background work shares one memory budget. A view whose refresh does not fit falls behind, sets the metric muscles_matview_lagging to 1 and catches up when memory is free. muscles_matview_lag_seconds says how far behind it is, so a dashboard can show the delay.

• Alert delivery Available

  Alerts retry with backoff and wait in a bounded queue per target, and a firing alert is sent again on an interval.

• Recording rules Available

  CREATE RECORDING RULE runs a query on a schedule and exposes the result as Prometheus metrics on an authenticated endpoint.

• Time-grid queries Available

  GROUP BY host EVERY 10s resamples a series onto a regular grid, with FILL, and rate, increase, avg_over_time, sum_over_time and max_over_time read a range behind each grid point.

Analytical SQL

The dialect is close to ClickHouse SQL.

• Columnar scans Available

  A query reads only the columns it names, and sparse marks and zone maps let it skip data that cannot match.

• Standard SELECT Available

  WHERE, GROUP BY, HAVING, ORDER BY with nulls first or last, LIMIT and OFFSET, LIKE and ILIKE, CASE, and aliases usable in every clause.

• Joins Available

  INNER, LEFT, RIGHT and FULL equi-joins take ON or USING, a subquery or a CTE can be either side, and GROUP BY works over a join.

• Subqueries and CTEs Available

  Subqueries work in FROM, IN, EXISTS and as single values, WITH clauses can read tables, and a derived table can be one side of a join.

• UNION and DISTINCT Available

  UNION ALL, UNION DISTINCT and SELECT DISTINCT work, also with aggregates and inside INSERT ... SELECT.

• Totals and rollups in GROUP BY Available

  WITH TOTALS adds a totals row, and WITH ROLLUP adds a row for each shorter key prefix and a grand total.

• Expressions over aggregates Available

  A SELECT list or a HAVING clause can combine aggregates, such as sum(a) / count(), and compare them with constants.

• Query parameters Available

  A query can carry typed placeholders such as {name:Int64}, so a client does not build SQL by pasting values.

• Settings per query and per session Available

  SELECT ... SETTINGS and SET change the session settings, among them limits, join behaviour, search options and query_priority.

• EXPLAIN and catalog statements Available

  EXPLAIN, DESCRIBE, SHOW CREATE TABLE, SHOW TABLES, SHOW DATABASES and USE work, and the schema queries that DataGrip sends over system.tables and system.columns succeed.

• INSERT SELECT and CREATE TABLE AS SELECT Available

  Copy a query result into a table, such as a daily summary built from raw events. You can name a subset of columns, ask for an atomic copy, run the copy on a sharded cluster, and make a retry safe with a token.

• ALTER TABLE and UPDATE Available

  Add, drop, rename and retype columns, change a TTL, add skip indexes, and update rows with a WHERE clause. An update rewrites the affected parts, as a ClickHouse mutation does.

• Column defaults Available

  A column can have a constant DEFAULT, and an insert that leaves the column out gets the default in every input format.

• Dictionaries Available

  CREATE DICTIONARY loads a table into memory with a flat, hashed or complex-key layout, and dictGet, dictGetOrDefault and dictHas look a key up inside a query.

• IP-range lookups Available

  An IP_TRIE dictionary maps an address to the network that contains it, for geo and allow-list lookups such as labelling each access-log line with the country and network of the client.

• Decimal numbers Available

  Decimal(P, S) columns hold up to 18 digits and support arithmetic, casts and aggregates, also on a cluster, so invoice lines sum to the cent without float rounding.

Functions, types and formats

The reference pages list every function, type, format and setting. Here they are by family, with what each family is for.

• Window functions Available

  row_number, rank, dense_rank, ntile, lag, lead, first_value, last_value, nth_value, percent_rank, cume_dist, plus 15 aggregates as windows, with ROWS and RANGE frames and named windows.

• 36 column types Available

  Signed and unsigned integers from 8 to 64 bits, 32- and 64-bit floats, decimals, booleans, strings and fixed strings, dates, date-times and nanosecond timestamps, UUIDs, IPv4 and IPv6 addresses, enums, arrays and maps.

• 128-bit integers Available

  Int128 and UInt128 columns store and return exact values in DDL, inserts, literals, comparisons, ORDER BY and every output format, so a 128-bit transaction id keeps all its digits.

• NULL values Available

  Every column type accepts NULL, and IS NULL, coalesce and the aggregates treat it as SQL does.

• Compression codecs Available

  Columns compress with lz4 by default, and a CODEC(...) clause picks delta, double-delta, gorilla or run-length coding. An auto mode picks one per column.

• Data-skipping indexes Available

  minmax, set, bloom_filter, ngrambf_v1, tokenbf_v1 indexes let a scan skip blocks, in CREATE TABLE, CREATE INDEX and ALTER TABLE ... ADD INDEX.

• Result formats Available

  TabSeparated and its variants, CSV, JSON, JSONCompact, JSONEachRow, Native and RowBinary, 17 names in all, and Arrow IPC over the HTTP API.

• Generated rows Available

  numbers() and numbers_mt() produce a series of integers for tests, calendars and gap filling.

• 128-bit arithmetic and 256-bit integers Planned

  Arithmetic and aggregates over Int128 and UInt128 columns, and the Int256 and UInt256 types, are planned.

Getting data in

Rows reach muscles over HTTP, from Kafka, from Prometheus or from a small agent next to the source. A write is acknowledged after it is synced to disk.

• Share groups and keyed ingest Available

  Ingest nodes share a topic through Kafka share groups, and keyed ingest sends each row to the shard that owns its key.

• Duplicate-safe Kafka tables Available

  KAFKA DEDUP keeps one row per key when Kafka redelivers a record, and the offsets are stored with the rows when you ask.

• Schema evolution policy Available

  A table created from an Avro schema follows an evolve or strict policy when the schema changes, and system.schema_history lists the versions.

• Quarantine for bad records Available

  A record that cannot be stored goes to ingest_quarantine, and an operator can skip one record by name.

• Kafka operations Available

  Administrators can pause, resume and reset a topic over the API, and drain a node before a restart. Lag metrics show how far ingest is behind.

• Agent sources Available

  The agent reads plain, rotated and gzip files in order across rotations and the journal with a cursor, and its loopback port takes INSERT ... FORMAT JSONEachRow and Prometheus remote write.

• Agent enrichment Available

  Stages in the agent add a geo lookup from a CSV, a prefix, a cut, a regular-expression flag or a label before the batch leaves the machine.

• Kafka topics to Iceberg Available

  muscles dump writes the records of a Kafka topic to an Iceberg table on S3.

• Microcontroller buffering Planned

  A small library for the ESP32 is planned to keep compressed readings in flash and upload them when the network returns. The server side exists: POST /api/v1/edge/ingest takes a checksummed device batch and stores a resent batch once.

• Kafka setup in the console Planned

  The console is planned to let you add a cluster once, see its status on a card, pick a topic from a list and get a suggested number of consumer threads.

Data that lives elsewhere

You do not have to store everything in muscles. It reads other systems and lake files in place, joins them with your tables, and it can move old data to S3.

• Secrets stay out of SQL Available

  An external table names an environment variable for its password, so the statement and the query log never carry the secret.

• Import from S3 and local files Available

  IMPORT PARQUET and IMPORT ICEBERG load files from S3 or a local directory into a table in one crash-safe statement.

• Parquet reader Available

  Parquet files with plain, dictionary, gzip, zstd and snappy pages, NULLs, decimals and lists of numbers or strings are read.

• Iceberg tables Available

  Any Iceberg table can be read, with deflate, snappy and zstandard manifests and delete files.

• Parquet export Available

  SELECT ... INTO OUTFILE '...' FORMAT Parquet writes a result to a Parquet file in a directory that the operator allows.

• Several disks Available

  A storage policy spreads parts over named disks and volumes, and a disk stops receiving new parts when it is nearly full.

• AWS Glue catalog Planned

  Reading Iceberg tables through an AWS Glue catalog is planned.

Search, vectors and language models

Text search, vector search and calls to a language model run inside SQL.

• Tokenizers and ranking settings Available

  A text index can use the default tokenizer, n-grams or a custom separator, and the settings bm25_k1 and bm25_b tune the ranking.

• Vector distance functions Available

  l1_distance, l2_distance, linf_distance, cosine_distance, dot_product, normalize_l2 and the norms work on float arrays.

• Approximate vector index Available

  The vector_search_approximate setting lets an IVF index answer a search faster, and search_nprobe trades recall for speed.

Security and compliance

Who can see what, who changed what, and how to erase data.

• Policies follow views Available

  A view or rollup built from a refused column is refused too, at any depth, and UPDATE and DELETE go through the same read policies.

• Masking functions Available

  mask_hash, mask_partial, mask_redact, mask_null hide parts of a value in a query.

• Row deletion for GDPR requests Available

  DELETE FROM ... WHERE makes rows unreadable when it returns, and the next merge removes them from disk. Only administrators can run it, so an erasure request deletes every row that carries the customer id.

• Compliance certificates Planned

  Certificates such as SOC 2 and ISO 27001.

• Audit blocks on write-once storage Available

  The audit directory can sit on a write-once volume, and an anchor of the chain tip detects a shortened log.

• Single sign-on Available

  Users sign in with an OpenID Connect provider, so staff use the company identity provider and support reads the tables without write access. An allow list limits who can sign in, and the identity claim is configurable.

• Password login and login throttling Available

  A users file holds Argon2id hashes, muscles hash-password creates a line, and repeated failed logins are throttled for the address and for the user.

• Login behind an authenticating proxy Available

  muscles can trust the user name that an oauth2-proxy in front of it sends, or listen on a Unix socket so that only the proxy reaches it.

• Roles and capabilities Available

  Administrators hold every capability, viewers read only, and each privileged action such as DDL, import, flush, rule creation and row deletion checks its own capability.

• Query history that respects owners Available

  system.query_log and query profiles show a viewer only their own statements, and administrators see all of them.

• Secrets redacted from logs Available

  Literals in connection statements are masked in the query log and the audit log.

• Per-user quotas Available

  Quotas limit what one user can run, and system.quotas and system.quota_usage show the limits and the live counters.

• Browser and network guards Available

  The HTTP interface refuses cross-site browser requests and requests with a foreign Host header, signs node-to-node requests, and can require a token for /metrics.

• Sessions and tokens Available

  Tokens expire, a session refreshes only while it is young, and a refresh checks the user and role again.

• Protection from slow and flooding clients Available

  The server bounds the time to read a request, limits connections per address before sign-in, and disconnects readers that stall.

• Typed error codes Available

  Error bodies carry a code such as not_found or replica_read_only, and the ClickHouse interface returns ClickHouse numeric codes.

Durability, storage and upgrades

What happens to your data when the disk, the process or the version changes.

• Durable writes Available

  Each table has a write-ahead log with CRC32C frames. A write is acknowledged after its fsync, and a failed fsync fails the write.

• Immutable compressed parts Available

  Memtables flush to immutable parts with checksums, and background merges keep the part count low.

• Recovery from a damaged log Available

  Boot repairs a torn log tail on its own. An operator can choose to salvage the records before a corrupt frame, and the salvaged rows stay replayable until a part covers them.

• Snapshots and restore Available

  /admin/snapshot writes a consistent copy of a node to a directory, and muscles restore checks every file against its checksum before it puts the copy in place, so you can take a snapshot before a risky change and restore it on a spare node.

• Format label on every image Available

  Each release image carries a label with a digest of the stored formats, and muscles formats --json prints the same digest. Equal labels mean the data directory survives the upgrade, and the operator reads them before it rolls an upgrade.

• Retention with TTL Available

  CREATE TABLE ... TTL ts + INTERVAL 90 DAY deletes parts whose newest row is older than the interval, once the operator turns the sweeper on, so raw meter readings can go after the rollups cover their range.

• Schema changes replicate Available

  Table creation, drops and column changes travel to replicas in a signed schema snapshot, and tables made by IMPORT replicate too.

• Replicas catch up by themselves Available

  A replica that was offline downloads what it missed when it starts, so it needs no manual resync.

• Flush and merge control Available

  flush_max_rows and flush_interval_ms set when a memtable becomes a part, and an administrator can flush a table on demand.

• Versioned stored formats Available

  Every stored and wire format starts with a magic and a version, and a reader rejects an unknown version.

Running it

Single node first, then query, ingest and data nodes, replicas and Kubernetes.

• Distributed queries Available

  Aggregates, joins, windows, DISTINCT, totals and decimal sums run across shards and return the answer a single node would give.

• Kubernetes Available

  Tagged container images, a Helm chart and an operator. The chart runs one StatefulSet per shard.

• Operator resources Available

  The operator manages an ingest cluster with an autoscaler and users through custom resources, and a pause annotation stops reconciliation at a safe point.

• systemd units Available

  Hardened systemd unit templates ship for the server and the agent.

• Ingest before queries Available

  Ingest and DDL take the CPU before heavy queries. The session setting query_priority accepts high, normal and low, so a long report runs at low priority while sensor writes keep their place.

• Query limits Available

  The operator can cap the run time, the rows read, the rows returned, the memory of one query and of all queries, and the number of concurrent queries.

• Spill to disk Available

  Large GROUP BY and ORDER BY steps write spill files instead of failing, within a disk budget.

• One memory budget Available

  Queries, merges and view refreshes draw on one budget, and work that does not fit waits or lags instead of ending in an out-of-memory crash.

• Result row cap Available

  A result above max_query_rows fails with a typed error, or with result_overflow_mode = 'break' returns the first rows and marks the result as truncated.

• Configuration Available

  Every setting is an environment variable or a YAML key, and the configuration reference lists them.

• Background work on its own tier Planned

  Merges and view refresh are planned to run on a worker, so a large rollup rebuild cannot slow ingest.

• Query nodes behind a load balancer Planned

  Any query node is planned to serve progress, profile and stream requests for a query that another node ran, so a reporting portal can spread its queries over several small query nodes. An asynchronous query API is planned after it.

• Config reload from a mounted file Planned

  A ConfigMap file is planned to reload without a restart. A bad file keeps the last valid configuration and raises a metric; a bad file at start still stops the node.

Connect your tools

Dashboards, notebooks and drivers reach muscles over HTTP.

• ClickHouse drivers Available

  Connect with clickhouse-connect for Python, clickhouse-jdbc, clickhouse-go or the ClickHouse data source of Grafana.

• Grafana Available

  A data source plugin, ready dashboards and alert rules. The plugin signs in with a user name and password and renews its token when it expires.

• JDBC and JetBrains tools Available

  A JDBC driver lets DataGrip, the IntelliJ database tools and DBeaver run queries and browse the schema.

• Trino connector Available

  Trino reads muscles tables through a connector that pushes down columns, limits and simple filters, so a Trino query can join a muscles click table with a warehouse table.

• Client libraries Available

  Clients for Go, Python, Java, Rust and C++ send SQL over HTTP. You get them with your onboarding.

• JSON, streaming and Arrow Available

  POST /api/query returns columnar JSON, /api/query/stream streams a result, and /api/query.arrow returns an Arrow IPC stream.

• Terminal client Available

  muscles cli runs statements interactively or with --exec and prints a table, CSV or JSON lines.

• Compressed responses Available

  The ClickHouse interface compresses responses with gzip, deflate or lz4 when the client asks for it, and accepts compressed insert bodies, so a laptop on a metered link downloads a day of readings as gzip.

• Grafana Live Planned

  Panels are planned to update as rows arrive instead of polling, so a line-side screen shows rejects per minute as they happen.

• Polars streaming Planned

  Reading with collect(engine="streaming") batch by batch, and sinking a lazy frame straight into a table, are planned, so a year of sensor data can be processed on a laptop without loading it all.

Tracing, console and documentation

Traces, metrics, the console and the reference pages.

• Metrics and health probes Available

  /metrics exports 278 Prometheus metrics, and /health, /ready and /startupz serve probes.

• System tables Available

  Tables such as system.parts, system.columns, system.query_log, system.rollups, system.alerts and system.policies show what the server holds and has run.

• Part, trace and profile logs Available

  system.part_log, system.trace_log and the processor profile log are tables, so you can query how flushes, merges and queries behaved.

• Web console Available

  A SQL editor, a catalog browser, query profiles, ingestion status and a Queries page that filters by statement kind, user and final state, so an on-call engineer can list the failed queries from a phone.

• Query history and a plan for every query Available

  The server keeps a durable query history and a profile for each query, and the console draws its stage graph with time, rows and bytes per stage and live progress while it runs, so you can see which stage of a funnel query reads the most bytes.

• Pipeline graph Available

  The console draws the path from Kafka topics through tables, views and rollups.

• Map-key completion Available

  The SQL editor suggests the keys that exist in a map column when you type tags['.

• Opt-in usage counts Available

  The console can count which of its buttons you use. The counts are off by default and stay in your browser.

• Optional AI assistant Available

  The console can offer an assistant that writes SQL from a question, when the operator turns it on.

• Reference pages generated from the code Available

  Every SQL function, type, setting, environment variable, metric, error code and format has a reference page generated from the code.

• Force a trace for one query Planned

  A URL parameter, a traceparent header and a setting in the query are planned to force a trace of one slow panel query, also from the Grafana data source.

• Traces stored in muscles Planned

  A separate muscles deployment is planned to store traces and serve them to Jaeger or Grafana, so one database holds the logs, the metrics and the traces.