Configuration variables
The server reads these environment variables. A value in muscles.yaml under the key in the third column applies when the variable is not set.
Server / HTTP
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_HTTP_PORT | http.port | 18080 | TCP port the HTTP server listens on. |
MUSCLES_HTTP_ADDR | http.addr | 127.0.0.1 | Address the HTTP server binds to. |
MUSCLES_LISTEN_SOCKET | http.listen_socket | (unset) | Path of a Unix domain socket to listen on instead of a TCP port. |
MUSCLES_DRAIN_TIMEOUT | lifecycle.drain_timeout | 30s | How long in-flight requests get to finish after SIGTERM, as a duration such as 45s or 2m. |
MUSCLES_HTTP_GZIP | http.gzip | true | Compresses responses with gzip for clients that send Accept-Encoding: gzip. |
MUSCLES_HTTP_GZIP_MIN_BYTES | http.gzip_min_bytes | 1024 | Smallest response body, in bytes, that the server compresses. |
MUSCLES_HTTP_READ_TIMEOUT_MS | http.read_timeout_ms | 15000 | Time in milliseconds a client has to send a whole request, head and body, before the server closes the connection. |
MUSCLES_HTTP_WRITE_TIMEOUT_MS | 15000 | Time in milliseconds the socket may refuse a response slice before the server closes the connection. | |
MUSCLES_HTTP_RESPONSE_MAX_MS | 1800000 (30 min) | Longest time in milliseconds the server spends writing one buffered response before it cuts the connection. | |
MUSCLES_HTTP_STREAM_MIN_BYTES | 65536 | Least number of bytes a streaming reader must take in each window set by MUSCLES_HTTP_STREAM_MIN_WINDOW_MS, or the server disconnects it. | |
MUSCLES_HTTP_STREAM_MIN_WINDOW_MS | 30000 | Length in milliseconds of the write-time window for MUSCLES_HTTP_STREAM_MIN_BYTES. | |
MUSCLES_HTTP_WORKERS | http.workers | 32 | Number of threads that accept and serve connections on the shared listener. |
MUSCLES_HTTP_KEEPALIVE_TIMEOUT_MS | http.keepalive_timeout_ms | 5000 | How long an idle persistent connection waits for its next request, in milliseconds. |
MUSCLES_HTTP_KEEPALIVE_MAX_REQUESTS | http.keepalive_max_requests | 100 | Most requests the server handles on one persistent connection before it closes the connection. |
MUSCLES_HTTP_MAX_CONN_PER_IP | http.max_conn_per_ip | 0 (off) | Most simultaneous connections one client IP address may hold, so a few slow clients cannot occupy every worker. |
MUSCLES_HTTP_MAX_HEAD_CONN_PER_IP | 8 | Most connections per IP address that may still be reading their first request before authentication, beyond which the request must arrive within 250 ms. |
Authentication & RBAC
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_AUTH_SECRET | auth.secret | (none) | Key that signs session tokens. |
MUSCLES_INTERNAL_SECRET | auth.internal_secret | (none) | Key that signs /internal/* requests between nodes, such as query fragments, forwarded ingest and drain. |
MUSCLES_AUTH_DEV_INSECURE | auth.dev_insecure | false | For local development only: lets the server start without a secret and accept any login. |
MUSCLES_ADMINS | auth.admins | (empty) | Comma-separated user names with administrator rights, or a file that lists them. |
MUSCLES_VIEWERS | auth.viewers | (empty) | Comma-separated identities that may sign in through OIDC as viewers. |
MUSCLES_METRICS_TOKEN_FILE | auth.metrics_token_file | (none) | File holding the bearer token that /metrics and /statusz require. |
MUSCLES_USERS_FILE | auth.users_file | (none) | File of local users, one username:role:hash line each with an Argon2id password hash, which enables password login. |
MUSCLES_TRUST_PROXY_AUTH | auth.trust_proxy_auth | false | Trusts the user name that a co-located oauth2-proxy sends in X-Auth-Request-Email or X-Forwarded-User. |
MUSCLES_CH_HTTP_STRICT_SETTINGS | http.ch_http_strict_settings | false | Refuses ClickHouse HTTP URL settings that muscles does not support, with error code 115. |
MUSCLES_SESSION_MAX_MS | auth.session_max_ms | 86400000 (24 h) | Longest lifetime of a login session in milliseconds, from 1 to 31536000000. |
MUSCLES_TRUSTED_PROXIES | auth.trusted_proxies | (empty) | Comma-separated addresses or address/prefix blocks of the reverse proxies whose X-Forwarded-For header names the client for login throttling. |
Native OAuth2 / OIDC login
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_OAUTH_ENABLED | oauth.enabled | false | Turns on OIDC sign-in. |
MUSCLES_OAUTH_ISSUER | oauth.issuer | (none) | Base URL of the OIDC issuer, such as https://idp.example.com. |
MUSCLES_OAUTH_DISCOVERY_URL | oauth.discovery_url | (derived) | Discovery URL to use instead of the one derived from the issuer. |
MUSCLES_OAUTH_CLIENT_ID | oauth.client_id | (none) | Client ID registered for muscles at the identity provider. |
MUSCLES_OAUTH_CLIENT_SECRET | oauth.client_secret | (none) | Client secret registered at the identity provider. |
MUSCLES_OAUTH_REDIRECT_URI | oauth.redirect_uri | (none) | Callback URL registered at the identity provider, matched exactly, such as https://muscles.example.com/oauth2/callback. |
MUSCLES_OAUTH_PATH_PREFIX | oauth.path_prefix | /oauth2 | Base path of the login and callback routes, which are <prefix>/login and <prefix>/callback. |
MUSCLES_OAUTH_SCOPES | oauth.scopes | openid email profile | Space-separated OIDC scopes to request. |
MUSCLES_OAUTH_ALLOWED_DOMAINS | oauth.allowed_domains | (none) | Comma-separated email domains allowed to sign in. |
MUSCLES_OAUTH_REQUIRE_VERIFIED_EMAIL | oauth.require_verified_email | true | Rejects sign-ins whose token has email_verified=false. |
MUSCLES_OAUTH_IDENTITY_CLAIM | oauth.identity_claim | ID-token claim that names the user, one of email, preferred_username or sub. | |
MUSCLES_OAUTH_UNLISTED_ROLE | oauth.unlisted_role | none | Role given to an OIDC user who is not in MUSCLES_ADMINS, MUSCLES_VIEWERS or the users file, either none or viewer. |
MUSCLES_OAUTH_AUTHORIZE_URL | oauth.{authorize,token,jwks}_url | (discovery) | Authorize, token and JWKS URLs to use instead of the discovered ones. |
MUSCLES_OAUTH_INSECURE_TLS | oauth.insecure_tls | false | For development only: skips TLS verification for calls to the identity provider. |
Federation — external-source connectors
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_FEDERATION_CONFIG | (unset, so off) | Path of a YAML file that declares external tables in an external_tables: list. | |
MUSCLES_FEDERATION_ALLOW_INLINE_SECRET | off | For development only: lets CREATE EXTERNAL TABLE carry a password in the statement instead of the name of an environment variable. |
Storage & durability
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_PARTS_DIR | storage.parts_dir | muscles-data/parts | Directory where flushed parts are stored. |
MUSCLES_WAL_DIR | wal.dir | muscles-data/wal | Directory for write-ahead log segments, with one subdirectory per table. |
MUSCLES_WAL_ON_CORRUPTION | fail | What recovery does with a corrupt write-ahead log frame that was already fsynced, either fail or salvage. | |
MUSCLES_DATA_DIR_MODE_CHECK | refuse | Startup check on the permissions of the data directories, with the values refuse, warn and off. | |
MUSCLES_RECOVERY_THREADS | 0 (auto) | Threads that read and validate part manifests in parallel at boot. | |
MUSCLES_MV_STALE_WAL_MAX_BYTES | storage.mv_stale_wal_max_bytes | 0 (off) | Size in bytes of the write-ahead log that a stale materialized view may hold before the server drops that view. |
MUSCLES_PART_CODEC | storage.part_codec | lz4 | Default compression chain for newly written parts, which a column's CODEC(...) overrides. |
MUSCLES_PART_BLOCK_MAX_BYTES | storage.part_block_max_bytes | 32 MiB | Largest column block in bytes, which also limits a single cell to half of that size. |
MUSCLES_SCAN_PREFETCH | storage.scan_prefetch_depth | 0 (OFF) | Number of upcoming granules whose column blocks a scan reads ahead while it decodes the current granule. |
MUSCLES_QUERY_PARALLELISM | storage.query_parallelism | 1 for other queries, effective CPU count for KNN | Worker threads for one single-node SELECT scan. |
MUSCLES_INSERT_DEDUP_WINDOW | ingest.insert_dedup_window | 1000 | Number of recent INSERT block ids each table remembers, so a retried INSERT is acknowledged without storing its rows again. |
MUSCLES_MIGRATE_FROM_TOKEN | none | unset | Bearer token muscles migrate sends to the source server when --from-token and --token are absent. |
MUSCLES_MIGRATE_TO_TOKEN | none | unset | Bearer token muscles migrate sends to the target server when --to-token and --token are absent. |
MUSCLES_FLUSH_MAX_ROWS | flush.max_rows | 1048576 | Flushes a table's segment once this many rows are unflushed. |
MUSCLES_FLUSH_INTERVAL_MS | flush.interval_ms | 60000 | Flushes a table this many milliseconds after its previous flush, even below MUSCLES_FLUSH_MAX_ROWS. |
MUSCLES_FLUSH_POLL_MS | flush.poll_ms | 1000 | How often the background flusher checks whether a table must be flushed, in milliseconds. |
MUSCLES_MEMTABLE_MAX_BYTES | storage.memtable_seal_bytes | a 32nd of the effective memory limit, from 8 MiB to 256 MiB | Size in bytes at which a table's in-memory data is sealed, given as a plain number without a suffix. |
MUSCLES_MERGE_ENABLED | 1 | Set 0 to stop background merges in an emergency, for example when merges run the node out of memory. | |
MUSCLES_IVF_MIN_ROWS | storage.ivf_min_rows | 2048 (rows) | Smallest number of rows a sealed part needs before muscles builds a vector index for its vector columns. |
MUSCLES_QUERY_PROFILE_MAX | query.profile_max | 256 | How many finished per-query plan graphs the server keeps in memory, dropping the oldest first. |
MUSCLES_QUERY_PROGRESS | query.progress | 1 (ON) | Turns on live per-query progress reports that stream over SSE. |
MUSCLES_QUERY_PROGRESS_INTERVAL_MS | query.progress_interval_ms | 100 | How often progress samples are sent over SSE, in milliseconds, limited to the range 20 to 2000. |
MUSCLES_PART_LOG_MAX_ROWS | 100000 | Most rows system.part_log keeps, where 0 turns the log off. | |
MUSCLES_PART_LOG_TTL_DAYS | 7 | Age in days after which system.part_log rows expire, where 0 keeps them until the row cap removes them. | |
MUSCLES_TRACE_LOG_MAX_ROWS | 100000 | Most rows system.trace_log keeps, where 0 turns the log off. | |
MUSCLES_TRACE_LOG_TTL_DAYS | 7 | Age in days after which system.trace_log rows expire, where 0 keeps them until the row cap removes them. | |
MUSCLES_PROCESSORS_PROFILE_LOG_MAX_ROWS | 100000 | Most rows system.processors_profile_log keeps, where 0 turns the log off. | |
MUSCLES_PROCESSORS_PROFILE_LOG_TTL_DAYS | 7 | Age in days after which system.processors_profile_log rows expire, where 0 keeps them until the row cap removes them. | |
MUSCLES_QUERY_HISTORY | 1 (ON) | Archives every query durably in system.query_history, where 0 turns it off. | |
MUSCLES_QUERY_HISTORY_TTL_DAYS | 30 | Age in days after which archived queries expire, where 0 means no age limit. | |
MUSCLES_QUERY_HISTORY_MAX_ROWS | 100000 | Most archived queries kept in system.query_history, dropping the parts with the oldest rows first. | |
MUSCLES_QUERY_HISTORY_SYNC | group | How the query history is committed to disk, one of sync, group or off. | |
MUSCLES_QUERY_HISTORY_PLAN_BYTES | 16384 | Largest query plan in bytes that is archived with a query. | |
MUSCLES_QUERY_HISTORY_END_USER | 1 (ON) | Records the end-user name that a trusted proxy forwards. |
Ingestion
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_KAFKA_BOOTSTRAP | ingest.kafka.bootstrap | (unset, so ingest is off) | Host of the Kafka bootstrap broker, which turns ingestion on when set. |
MUSCLES_KAFKA_PORT | ingest.kafka.port | 9092 | Port of the Kafka broker. |
MUSCLES_KAFKA_SHARE_GROUP | ingest.kafka.share_group | (default group) | Name of the KIP-932 share group that this node joins as one member. |
MUSCLES_KAFKA_TOPICS | ingest.kafka.topics | (discover) | Comma-separated list of topics to ingest, or a YAML list. |
MUSCLES_KAFKA_MAX_DECOMPRESSED_BYTES | ingest.kafka.max_decompressed_bytes | 64 MiB | Largest size in bytes that a compressed record batch may inflate to before it is rejected. |
MUSCLES_KAFKA_POLL_MS | ingest.kafka.poll_ms | 500 | Pause in milliseconds between ingest cycles once the node has caught up. |
MUSCLES_KAFKA_RECONNECT_BACKOFF_MS | ingest.kafka.reconnect_backoff_ms | 1000 | First wait in milliseconds before reconnecting to a failed broker, doubling on each retry up to the maximum. |
MUSCLES_KAFKA_MAX_BACKOFF_MS | ingest.kafka.max_backoff_ms | 30000 | Longest wait in milliseconds between broker reconnect attempts. |
MUSCLES_KAFKA_READ_TIMEOUT_MS | ingest.kafka.read_timeout_ms | 30000 | Time in milliseconds a broker response read may make no progress before it counts as a transient error. |
MUSCLES_KAFKA_MAX_TOPICS | ingest.kafka.max_topics | 64 | Most topics tracked, beyond which topics are not ingested. |
MUSCLES_KAFKA_MAX_PARTITIONS | ingest.kafka.max_partitions | 256 | Most partitions tracked per topic, beyond which partitions are not ingested. |
MUSCLES_KAFKA_FORWARD_DEADLINE_MS | ingest.kafka.forward_deadline_ms | 10000 | Time in milliseconds to connect to the owning shard and receive its durable acknowledgement when forwarding a record. |
MUSCLES_KAFKA_TOPIC_MAX_FAILURES | ingest.kafka.topic_max_failures | 3 | Consecutive unrecoverable errors after which a topic is quarantined. |
MUSCLES_KAFKA_QUARANTINE_COOLDOWN_MS | ingest.kafka.quarantine_cooldown_ms | 60000 | How long a quarantined topic stays skipped before one probe cycle. |
MUSCLES_KAFKA_FORWARD_INBOX_RECORDS | ingest.kafka.forward_inbox_records | 4096 | Most forwarded records a shard holds unprocessed before it answers new forwards with 503. |
MUSCLES_KAFKA_FORWARD_INBOX_BYTES | ingest.kafka.forward_inbox_bytes | 64 MiB | Most forwarded bytes a shard holds unprocessed before it answers new forwards with 503. |
MUSCLES_KAFKA_STORE_OFFSET | ingest.kafka.store_offset | off | Adds the columns _kafka_partition, _kafka_offset and _kafka_topic to every Kafka row so you can check for gaps. |
MUSCLES_KAFKA_EAGER_TABLES | ingest.kafka.eager_tables | on | Creates a discovered topic's table from its Schema Registry subject at discovery, before this node consumes a record. |
MUSCLES_SCHEMA_POLICY | ingest.schema_policy | evolve | Schema policy for ingested topics, either evolve or strict, in any letter case. |
MUSCLES_SCHEMA_REGISTRY_HOST | ingest.schema_registry_host | 127.0.0.1 | Host of the Confluent Schema Registry that resolves Avro schemas. |
MUSCLES_SCHEMA_REGISTRY_PORT | ingest.schema_registry_port | 18085 | Port of the Schema Registry. |
MUSCLES_KAFKA_KEYED_INGEST | cluster.keyed_ingest | off | Sends each Kafka record to the shard that owns its shard key. |
MUSCLES_KAFKA_KEYED_DEDUP_TARGETS | off | Set to 1 to declare that every keyed topic writes to a KAFKA DEDUP or Replacing table. | |
MUSCLES_KAFKA_INGEST_THREADS | cluster.kafka_ingest_threads | 1 | Consumer threads per shard in keyed mode, at most 32. |
MUSCLES_KAFKA_ALIGNED | cluster.kafka_aligned | off | Declares that producers partition keyed topics with a hash that matches shardOf, which is Wyhash modulo the shard count. |
Topic-owner metadata
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_TOPIC_METADATA | topic_metadata: (structured block) | (none) | Owner, contact, team and other keys for each topic, shown when you hover the source node in the pipeline view. |
Query governor & limits
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_MAX_EXECUTION_TIME_MS | query.max_execution_time_ms | 0 | Longest time in milliseconds one query may run. |
MUSCLES_MAX_ROWS_TO_READ | query.max_rows_to_read | 0 | Most rows one query may scan. |
MUSCLES_MAX_RESULT_ROWS | query.max_result_rows | 0 | Most rows one query may return. |
MUSCLES_MAX_MEMORY_BYTES | query.max_memory_bytes | half the effective memory limit | Bytes one query may hold. |
MUSCLES_MAX_TOTAL_QUERY_MEMORY_BYTES | query.max_total_query_memory_bytes | three quarters of the effective memory limit | Bytes all running queries may hold together. |
MUSCLES_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY | query.max_bytes_before_external_group_by | 0 | Size of aggregate state in bytes at which a GROUP BY writes its groups to temporary files instead of failing. |
MUSCLES_MAX_BYTES_BEFORE_EXTERNAL_SORT | query.max_bytes_before_external_sort | 0 | Size of buffered ORDER BY rows in bytes at which a sort writes sorted runs to temporary files instead of failing. |
MUSCLES_MAX_TEMP_SPACE_BYTES | query.max_temp_space_bytes | 8589934592 (8 GiB) | Bytes of GROUP BY and ORDER BY spill files all queries may hold on disk together. |
MUSCLES_SPILL_DIR | query.spill_dir | spill next to MUSCLES_PARTS_DIR | Directory for GROUP BY and ORDER BY spill files, one subdirectory per running query. |
MUSCLES_MAX_CONCURRENT_QUERIES | query.max_concurrent_queries | 0 | Most queries that may run at once on the server. |
MUSCLES_IMPORT_DIR | query.import_dir | empty (off) | Directory that IMPORT PARQUET '<path>' INTO <table> may read local files from. |
MUSCLES_EXPORT_DIR | query.export_dir | empty (off) | Directory that Parquet exports with INTO OUTFILE may write to. |
MUSCLES_PARQUET_MAX_DECODED_BYTES | query.parquet_max_decoded_bytes | 2147483648 (2 GiB) | Most memory in bytes that one decoded Parquet file may hold during an import or an Iceberg read. |
MUSCLES_MAX_QUERY_ROWS | query.max_query_rows | 1000000 | Default limit on the rows a query returns, from 1 to 1000000000. |
MUSCLES_RULES_MAX | rules.max | 64 | Most recording rules one node holds. |
MUSCLES_RULE_MAX_SERIES | rules.max_series | 1000 | Most series one recording rule evaluation may export (rows times numeric columns). |
MUSCLES_RULES_MAX_SERIES_TOTAL | rules.max_series_total | 10000 | Most series all recording rules on the node may export together. |
Alert delivery
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_ALERTMANAGER_URLS | alerting.alertmanager_urls | unset | Comma-separated Alertmanager base URLs, at most 8, such as http://alertmanager:9093. |
MUSCLES_ALERTMANAGER_AUTH_FILE | alerting.alertmanager_auth_file | unset | File whose trimmed content is sent as the Authorization header to every Alertmanager, for example Basic dXNlcjpwYXNz or Bearer <token>. |
MUSCLES_ALERT_WEBHOOK_URLS | alerting.webhook_urls | unset | Comma-separated webhook URLs, at most 8. |
MUSCLES_ALERT_WEBHOOK_AUTH_FILE | alerting.webhook_auth_file | unset | Like MUSCLES_ALERTMANAGER_AUTH_FILE, for every webhook. |
MUSCLES_ALERT_RESEND_INTERVAL_S | alerting.resend_interval_s | 60 | How often, in seconds, an unchanged firing or resolved alert is sent again. |
MUSCLES_ALERT_DELIVERY_TIMEOUT_MS | alerting.delivery_timeout_ms | 10000 | Time in milliseconds one delivery request may take to connect, send and read the reply. |
MUSCLES_ALERT_DELIVERY_RETRIES | alerting.delivery_retries | 3 | Extra attempts after a timeout, transport error, 429 or 5xx reply, waiting from 0.5 s up to 10 s. |
MUSCLES_ALERT_QUEUE_CAPACITY | alerting.queue_capacity | 10000 | Alerts one target may hold waiting for delivery; past it the oldest is dropped and counted. |
Resource-priority scheduling
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_SCHED_PRIORITY | scheduling.priority | off | Whether ingest gets the CPU ahead of queries under contention, with the values off, cooperative and on. |
MUSCLES_SCHED_YIELD_SPINS | scheduling.yield_spins | 32 | Most CPU hand-offs a deferred scan batch makes before it runs, from 0 to 4096. |
MUSCLES_SCHED_MAX_YIELD_MS | scheduling.max_yield_ms | 1000 | Total time in milliseconds a query may be deferred for ingest before it stops yielding. |
Distributed query
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_DQ_TIMEOUT_MS | query.distributed.timeout_ms | 10000 | Time budget in milliseconds for a whole distributed query. |
MUSCLES_DQ_MAX_FRAGMENT_BYTES | query.distributed.max_fragment_bytes | 256 MiB | Largest response in bytes that one query fragment may send or receive. |
MUSCLES_DQ_DISTINCT_MAX | query.distributed.distinct_max | 1048576 | Most COUNT(DISTINCT) values kept per aggregate and fragment. |
MUSCLES_DQ_CONSISTENCY | query.distributed.consistency | full | Default read consistency when a request has no ?consistency= parameter, either full or partial. |
MUSCLES_DQ_CHUNK_ROWS | query.distributed.chunk_rows | 0 (8192) | Rows per chunk of a streamed fragment, which limits coordinator memory. |
MUSCLES_DQ_COMPRESS | query.distributed.compress | false | Compresses each fragment chunk with lz4. |
User-defined tiered disks
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_DISKS | storage.disks | (none) | Comma-separated list of data disks as name:path[:capacity], such as fast:/mnt/nvme:500Gi. |
MUSCLES_STORAGE_POLICY | storage.policy | (disks in declared order) | Volumes in order from fastest to slowest, separated by semicolons. |
MUSCLES_TIER_MOVE_FACTOR | 0.10 | Smallest fraction of a disk that must stay free, between 0 and 1, before the disk stops receiving new parts. |
Tiered storage
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_TIER_COLD_AFTER_MS | tiering.cold_after_ms | 0 (off) | A part whose newest row is older than this moves to S3. |
MUSCLES_S3_ENDPOINT | tiering.s3.endpoint | (AWS default) | Endpoint URL of the S3 service, which you set for MinIO or any other non-AWS store. |
MUSCLES_S3_BUCKET | tiering.s3.bucket | (none) | Bucket for the cold tier, required when tiering is enabled. |
MUSCLES_S3_PREFIX | tiering.s3.prefix | (none) | Key prefix under the bucket. |
MUSCLES_S3_ACCESS_KEY | tiering.s3.access_key | (ambient credentials) | Static S3 access key, which you must set together with the secret key. |
MUSCLES_S3_INSECURE_TLS | tiering.s3.insecure_tls | false | For development only: disables TLS verification for a self-signed https:// endpoint. |
MUSCLES_S3_CONNECT_TIMEOUT_MS | tiering.s3.connect_timeout_ms | 5000 | Time in milliseconds to connect to S3, so an unreachable endpoint fails quickly. |
MUSCLES_S3_REQUEST_TIMEOUT_MS | tiering.s3.request_timeout_ms | 30000 | Time in milliseconds that one S3 request may stall without moving bytes. |
MUSCLES_S3_MULTIPART_THRESHOLD_BYTES | tiering.s3.multipart_threshold_bytes | 67108864 | Size in bytes from which a part bundle uploads as an S3 multipart upload staged from a file. |
MUSCLES_TIER_DELETE_AFTER_MS | tiering.delete_after_ms | 0 (never) | A part whose newest row is older than this many milliseconds is deleted. |
MUSCLES_TIER_DOWNSAMPLE_AFTER_MS | tiering.downsample_after_ms | 0 (off) | Age in milliseconds after which a raw part becomes a candidate for dropping once rollups cover its range. |
MUSCLES_TIER_DOWNSAMPLE_ENABLED | tiering.downsample_drop_enabled | false | Lets the server delete raw parts whose range rollups cover. |
MUSCLES_TIER_POLL_MS | tiering.poll_ms | 60000 | How often the tier mover looks for parts to move, in milliseconds. |
MUSCLES_TIER_ORPHAN_SWEEP_MS | 3600000 | Interval in milliseconds between sweeps for S3 objects that no local part references. | |
MUSCLES_TIER_UPLOAD_CONCURRENCY | tiering.upload_concurrency | 2 | Most uploads the tier mover runs at the same time in one cycle. |
MUSCLES_TIER_GC_SINGLE_OWNER | false | Declares that this node is the only owner of its S3 prefix, which allows it to delete expired and orphaned objects. | |
MUSCLES_COLD_CACHE_DIR | tiering.cold_cache.dir | (sibling of parts) | Directory of the cache for cold granules. |
MUSCLES_COLD_CACHE_MAX_BYTES | tiering.cold_cache.max_bytes | 1 GiB | Size limit in bytes for the cold granule cache. |
MUSCLES_COLD_FAULT_CONCURRENCY | tiering.cold_cache.fault_concurrency | 4 | Most ranged S3 reads that run at once when a scan loads several cold columns. |
Local data retention
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_RETENTION_ENABLED | storage.retention_enabled | false | Starts the background sweeper that enforces table retention. |
MUSCLES_RETENTION_POLL_MS | storage.retention_poll_ms | 300000 (5 min) | How often the retention sweeper runs, in milliseconds. |
Replication — emitter hook
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_REPLICATION_EMIT | replication.emit | false | Ships each flushed or merged part to S3 and to the commit log. |
MUSCLES_REPLICATION_COMMIT_TOPIC | replication.commit_topic | muscles-commit-log | Kafka topic that receives part commits, which every replica of the shard drains. |
MUSCLES_REPLICATION_S3_BUCKET | replication.s3_bucket | *(falls back to MUSCLES_S3_BUCKET)* | Bucket that part bundles are uploaded to, which defaults to the tiering bucket when empty. |
MUSCLES_REPLICATION_SCHEMA_SNAPSHOT_EVERY | replication.schema_snapshot_every | 64 | Number of schema epochs between two schema snapshots. |
Replication — drain-loop worker
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_REPLICATION_DRAIN | replication.drain | false | Runs the background worker that applies commit-log parts to this node. |
MUSCLES_REPLICATION_CATCHUP | replication.catchup | false | Installs the parts missing on this node from the S3 manifest once at startup, and never removes parts. |
MUSCLES_REPLICATION_ROLE | replication.role | replica | Replication role of this node, either primary or replica. |
MUSCLES_REPLICATION_REPLICA_WRITES | replication.replica_writes | false | A replica refuses client INSERT, DDL, DELETE and forwarded ingest with HTTP 409 unless this is 1. |
MUSCLES_SHARD_GENERATION | 0 | Ownership generation of this node's shard, which you raise when you promote a replica. | |
MUSCLES_REPLICATION_NODE_ID | *(MUSCLES_POD_NAME)* | Stable identity of this node, hashed into the commit emitter_id and required to turn on emit. | |
MUSCLES_REPLICATION_SIGNING_KEY_FILE | (none) | File with the 32-byte Ed25519 signing seed of the primary, as 64 hex characters. | |
MUSCLES_REPLICATION_PUBLIC_KEY | (none) | One to four Ed25519 public keys of 64 hex characters each, separated by commas or whitespace. |
Cluster / node role
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_NODE_MODE | cluster.node_mode | all | What this node does, one of all, query, data or ingest. |
MUSCLES_SHARD | cluster.shard | (none) | Index of this node's shard. |
MUSCLES_SHARD_MAP | cluster.shard_map | (none) | Map from each shard to its endpoint, which also marks the deployment as clustered. |
MUSCLES_NODE_ADDRS | *(this node's own MUSCLES_SHARD_MAP entry)* | Comma-separated host:port names, as written in MUSCLES_SHARD_MAP, that signed node-to-node requests may target. | |
MUSCLES_SHARD_KEY | cluster.shard_key | (none) | Column used to prune a query to one shard when its filter names a key value. |
MUSCLES_SHARD_KEY_UNSAFE_QUERY_ONLY | off | Set to 1 only if rows are placed on shardOf(key) outside muscles. | |
MUSCLES_POD_NAME | (none) | Pod name that the server logs once at startup. | |
MUSCLES_POD_IP | (none) | Pod IP address that the server logs once at startup. |
Logging
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_LOG_LEVEL | info | Lowest message level that is logged, one of debug, info, warn or error. | |
MUSCLES_LOG_FORMAT | text | Log line format, either text or json. |
Tracing
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_TRACE_ENABLED | tracing.enabled | false | Turns on tracing. |
MUSCLES_TRACE_SAMPLE_RATIO | tracing.sample_ratio | 0.0 | Probability from 0.0 to 1.0 that a query not asked for tracing is traced anyway. |
MUSCLES_TRACE_SERVICE_NAME | tracing.service_name | muscles | Value of the OTLP service.name attribute, which you can set per role, such as muscles-query. |
MUSCLES_TRACE_ENDPOINT | tracing.endpoint | "" | Base URL of the OTLP/HTTP collector, such as http://jaeger:4318. |
MUSCLES_TRACE_SUBSYSTEMS | tracing.subsystems | "" | Sampling ratio for each subsystem, as name=ratio pairs such as merge=1.0,flush=0.0. |
MUSCLES_TRACE_SLOW_QUERY_MS | tracing.slow_query_ms | 0 | Queries that run at least this many milliseconds are always exported, whatever the sample ratio. |
MUSCLES_TRACE_SPAN_LOG_MAX_ROWS | tracing.span_log_max_rows | 100000 | Most rows system.opentelemetry_span_log keeps, where 0 turns the span log off. |
MUSCLES_TRACE_SPAN_LOG_TTL_SECONDS | tracing.span_log_ttl_seconds | 3600 | Age in seconds after which span log rows expire, where 0 keeps them until the row cap removes them. |
MUSCLES_TRACE_EXPORT_INTERVAL_MS | tracing.export_interval_ms | 1000 | How often the exporter sends queued traces, in milliseconds. |
MUSCLES_TRACE_QUEUE_DEPTH | tracing.queue_depth | 256 | Most trace batches the exporter queue holds before it drops new ones. |
MUSCLES_TRACE_EXPORT_TIMEOUT_MS | tracing.export_timeout_ms | 5000 | Time in milliseconds to connect to the collector, so an unreachable one fails quickly. |
MUSCLES_TRACE_MAX_BATCH_TRACES | tracing.max_batch_traces | 64 | Most traces sent in one OTLP request per export interval. |
MUSCLES_TRACE_MAX_SPANS | tracing.max_spans | 128 | Most spans and events one flush, merge, boot or query trace keeps, from 8 to 4096. |
Tamper-evident audit log
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_AUDIT_ENABLED | audit.enabled | false | Turns on the audit log. |
MUSCLES_AUDIT_MODE | audit.mode | async | Whether audit records are written before a query answers (sync) or in the background (async). |
MUSCLES_AUDIT_QUEUE_FULL | audit.queue_full | block | What async mode does when its queue is full, either block or drop_oldest. |
MUSCLES_AUDIT_WAIT_TIMEOUT_MS | audit.wait_timeout_ms | 5000 | Longest wait in milliseconds for the audit disk before a request fails with 503. |
MUSCLES_AUDIT_TIER | audit.tier | metadata | How much each audit record holds, either metadata or verbose. |
MUSCLES_AUDIT_QUERY_TEXT | audit.query_text | hash_only | How verbose audit records store the query text, as full, redacted or hash_only. |
MUSCLES_AUDIT_SEAL_ROWS | audit.seal_rows | 4096 | Number of queued rows at which async mode seals a block. |
MUSCLES_AUDIT_SEAL_INTERVAL_MS | audit.seal_interval_ms | 1000 | Longest time in milliseconds between seals in async mode, where 0 means seal by size only. |
MUSCLES_AUDIT_MAX_QUEUE | audit.max_queue | 65536 | Most rows the in-memory audit queue holds. |
MUSCLES_AUDIT_END_USER | audit.end_user | 0 (OFF) | Records the unverified end-user name that a trusted proxy forwards in each audit record. |
MUSCLES_AUDIT_DIR | audit.dir | muscles-data/audit | Directory that holds the sealed audit blocks. |
MUSCLES_AUDIT_IMMUTABLE | — | 0 | Forbids every deletion of the audit log and of system.query_history, and is read once at startup. |
MUSCLES_AUDIT_RETENTION_DAYS | — | 0 | Age in days after which sealed audit blocks are deleted, only when MUSCLES_AUDIT_IMMUTABLE is 0, where 0 keeps them forever. |
MUSCLES_AUDIT_MAX_BYTES | — | 0 | Size cap in bytes for the audit directory, enforced only when MUSCLES_AUDIT_IMMUTABLE is 0, where 0 means no cap. |
MUSCLES_AUDIT_S3_BUCKET | — | unset | Bucket for the audit archive in S3, which turns the archive on when set. |
MUSCLES_AUDIT_S3_PREFIX | — | audit | Key prefix of archive objects in the bucket. |
MUSCLES_AUDIT_S3_ENDPOINT | — | — | S3 endpoint as host[:port], optionally with a scheme, required when the bucket is set. |
MUSCLES_AUDIT_S3_REGION | — | us-east-1 | Region used to sign S3 requests. |
MUSCLES_AUDIT_S3_ACCESS_KEY[_FILE] | — | — | Access key for the audit bucket, required when the bucket is set and never defaulted from MUSCLES_S3_*. |
MUSCLES_AUDIT_S3_SECRET_KEY[_FILE] | — | — | Secret key for the audit bucket, required when the bucket is set. |
MUSCLES_AUDIT_S3_INSECURE_TLS | — | 0 | Skips TLS verification, for a self-signed MinIO only. |
MUSCLES_AUDIT_S3_INTERVAL_MS | — | 60000 | How often the audit archive uploads sealed blocks, in milliseconds. |
MUSCLES_AUDIT_S3_MAX_BATCH_BYTES | — | 67108864 | Most raw bytes of concatenated blocks in one archive object, from 4096 to 1073741824. |
MUSCLES_AUDIT_S3_CODEC | — | zstd | Compression of archive objects, either zstd or none. |
MUSCLES_AUDIT_S3_OBJECT_LOCK | — | off | Whether archive objects get S3 Object Lock retention, either off or require. |
MUSCLES_AUDIT_S3_LOCK_MODE | — | compliance | S3 Object Lock mode of archive objects, either compliance or governance. |
MUSCLES_AUDIT_S3_LOCK_DAYS | — | — | Retention in days, from 1 to 36500, stamped on every archived object. |
MUSCLES_AUDIT_S3_PARQUET_MIRROR | — | 0 | Writes a Parquet copy of the audit trail to S3 for auditors. |
MUSCLES_AUDIT_S3_DIGEST | — | 0 | Writes one small digest object per archived batch so that muscles audit-verify can check the chain. |
LLM runtime
| Variable | YAML key | Default | Description |
|---|---|---|---|
MUSCLES_LLM_ENDPOINT | llm.endpoint | (empty, off) | Base URL of the model endpoint, such as http://127.0.0.1:11434. |
MUSCLES_LLM_API_STYLE | llm.api_style | ollama | Request style of the endpoint, either ollama or openai. |
MUSCLES_LLM_ALLOWED_HOSTS | llm.allowed_hosts | (empty) | Comma-separated host names allowed besides localhost and IPv4 literals. |
MUSCLES_LLM_ALLOWED_CIDRS | llm.allowed_cidrs | (empty) | Extra network ranges allowed besides loopback, private and link-local addresses, comma-separated, such as 100.64.0.0/10. |
MUSCLES_LLM_DEFAULT_MODEL | llm.default_model | (empty) | Model for calls that name none. |
MUSCLES_LLM_API_KEY_FILE | llm.api_key_file | (empty, no key) | File that holds the bearer token for the endpoint. |
MUSCLES_LLM_CACHE_DIR | llm.cache_dir | llm-cache next to the parts directory | Directory of the result cache log. |
MUSCLES_LLM_CACHE_MAX_BYTES | llm.cache_max_bytes | 67108864 | Byte budget of the cache; the oldest entries are evicted first. |
MUSCLES_LLM_CACHE_MAX_ENTRIES | llm.cache_max_entries | 65536 | Most entries kept (ceiling 1048576). |
MUSCLES_LLM_TIMEOUT_MS | llm.timeout_ms | 60000 | Time in milliseconds that one HTTP exchange with the endpoint may take. |
MUSCLES_LLM_MAX_CONCURRENT | llm.max_concurrent | 2 | Most model calls in flight at once in the process, after which a call fails with Busy. |
MUSCLES_LLM_MAX_PROMPT_BYTES | llm.max_prompt_bytes | 32768 | Largest input text of one call, in bytes. |
MUSCLES_LLM_MAX_RESPONSE_BYTES | llm.max_response_bytes | 262144 | Largest HTTP response body accepted, in bytes. |
MUSCLES_LLM_MAX_OUTPUT_TOKENS | llm.max_output_tokens | 512 | Output tokens requested per call. |
MUSCLES_LLM_QUERY_MAX_CALLS | llm.query_max_calls | 1000 | Model calls one query may make; cache hits are free. |
MUSCLES_LLM_QUERY_MAX_TOKENS | llm.query_max_tokens | 1000000 | Tokens one query may use, as reported by the endpoint. |
MUSCLES_LLM_QUERY_TIMEOUT_MS | llm.query_timeout_ms | 600000 | Time in milliseconds that the model calls of one query may take in total. |