Metrics

The server exports these metrics in the Prometheus text format on /metrics. A label listed with values has a series for each value from startup. Metrics marked as set at runtime get their series on first use.

Name	Type	Labels	Description
`muscles_http_requests_total`	counter	set at runtime	Total HTTP requests by route, method and status.
`muscles_http_request_duration_seconds`	histogram	set at runtime	HTTP request latency in seconds by route.
`muscles_http_accept_errors_total`	counter	set at runtime	Failed accept() calls on the HTTP listener, by error. The worker waits and accepts again.
`muscles_http_slow_reader_disconnects_total`	counter	set at runtime	Streaming responses closed because the reader stayed below the minimum throughput, by route.
`muscles_query_duration_seconds`	histogram	`status`: `ok`, `error`	SQL query wall-clock latency in seconds by outcome.
`muscles_query_rows_scanned_total`	counter	set at runtime	Total rows scanned by queries, by table.
`muscles_query_bytes_read_total`	counter	set at runtime	Total bytes read by queries, by table.
`muscles_query_bytes_scanned_uncompressed_total`	counter	set at runtime	Total uncompressed bytes scanned by queries, by table. Compare with muscles_query_bytes_read_total for the compression ratio.
`muscles_queries_failed_total`	counter	`reason`: `parse`, `execute`, `scan`, `drop_horizon`	Total failed queries, by reason.
`muscles_query_granules_scanned_total`	counter	set at runtime	Granules that queries read, by table. Granules that an index or zone map skipped are not counted.
`muscles_query_granules_skipped_total`	counter	set at runtime	Granules of opened parts skipped by marks, zone maps or skip indexes, by table.
`muscles_query_governor_rejected_total`	counter	`reason`: `too_many_concurrent_queries`, `query_time_exceeded`, `rows_read_limit_exceeded`, `result_rows_limit_exceeded`, `result_row_cap_exceeded`, `memory_limit_exceeded`, `quota_queries`, `quota_errors`, `quota_result_rows`, `quota_read_rows`, `quota_execution_time_ms`	Queries rejected by the query governor or a per-user quota, by reason, such as too_many_concurrent_queries, memory_limit_exceeded or quota_<dimension>.
`muscles_query_priority_yields_total`	counter	`tier`: `low`, `normal`, `high`	Scan batches where a normal or low priority query paused for waiting higher-priority work, by band. A rise under load means ingest gets the CPU as intended.
`muscles_query_priority_starvation_guard_total`	counter	`tier`: `low`, `normal`, `high`	Queries promoted to never yield again because their total deferral reached MUSCLES_SCHED_MAX_YIELD_MS, by band. Each query is counted once.
`muscles_query_priority_active`	gauge	`tier`: `low`, `normal`, `high`	Governed queries running now, by scheduling band.
`muscles_query_priority_deferred`	gauge	`tier`: `low`, `normal`, `high`	Running queries that are paused at a yield point, by scheduling band.
`muscles_ingest_pressure`	gauge		Ingest tasks running now, such as Kafka consume, flush, merge and shard forwarding. A value above 0 means queries defer to ingest.
`muscles_ingest_pressure_raised_total`	counter		Ingest work spans that raised scheduler pressure. If it stays flat on a node that consumes Kafka with MUSCLES_SCHED_PRIORITY set, queries do not defer to ingest.
`muscles_sched_setpriority_failures_total`	counter		Rejected setpriority calls that lower the nice value of query threads. A non-zero value means the nice bias does not work, but the yield point still protects ingest.
`muscles_table_rows_ingested_total`	counter	set at runtime	Rows ingested by table and source (http, clickhouse, kafka, insert_select or import). Use rate() for the per-table ingest rate.
`muscles_table_bytes_ingested_total`	counter	set at runtime	Payload bytes ingested by table and source. For Kafka the payload is the record value size.
`muscles_table_ingest_series_dropped_total`	counter		Ingest counts not recorded per table because the per-table ingest series reached the cap.
`muscles_ingest_records_total`	counter	set at runtime	Total records ingested, by topic.
`muscles_ingest_bytes_total`	counter	set at runtime	Total bytes ingested, by topic.
`muscles_ingest_consumer_lag`	gauge	set at runtime	Consumer lag in records, by topic and partition. It is the broker high watermark minus the consumer position, updated on every poll cycle.
`muscles_ingest_partition_stalled`	gauge	set at runtime	1 while a partition's ingest is stopped at a record batch it cannot decode or acknowledge, otherwise 0. Ingest never skips such a batch on its own.
`muscles_kafka_reconnects_total`	counter	set at runtime	Kafka connections dropped after a broken request and opened again, by client.
`muscles_ch_http_ignored_settings_total`	counter	set at runtime	URL settings that the ClickHouse HTTP bridge ignored because it does not support them, by name. MUSCLES_CH_HTTP_STRICT_SETTINGS=1 refuses them with code 115 instead.
`muscles_llm_calls_total`	counter	`function`: `classify`, `complete`, `embed`	Model calls sent to the LLM endpoint, by function: `classify` for llmClassify, `complete` for llmSummarize and `embed` for llmEmbed. Cache hits are not counted.
`muscles_llm_cache_total`	counter	`result`: `hit`, `miss`	LLM result cache lookups, by result (hit, miss).
`muscles_llm_errors_total`	counter	`kind`: `not_permitted`, `nondeterministic_context`, `busy`, `call_budget`, `token_budget`, `query_timeout`, `prompt_too_large`, `invalid_labels`, `endpoint_rejected`, `timeout`, `transport`, `http_status`, `response_too_large`, `bad_response`, `outside_label_set`, `cache_write`	LLM calls or results that were refused, by kind, such as permission, limit, endpoint or transport.
`muscles_llm_tokens_total`	counter	`direction`: `prompt`, `completion`	Tokens reported by the LLM endpoint, by direction (prompt, completion).
`muscles_llm_call_duration_seconds`	histogram	`function`: `classify`, `complete`, `embed`	Latency of model calls in seconds, by function.
`muscles_ingest_lag_query_failures_total`	counter		Broker errors while computing consumer lag. The lag gauge goes stale for that cycle, but ingest continues.
`muscles_ingest_topics_tracked`	gauge		Topics this node ingests now. A value below muscles_ingest_topics_discovered means MUSCLES_KAFKA_MAX_TOPICS dropped topics.
`muscles_ingest_topics_discovered`	gauge		Eligible topics the broker advertised in discovery mode. It is not set when MUSCLES_KAFKA_TOPICS lists the topics.
`muscles_ingest_topic_cap_dropped_total`	counter		Topics not ingested because the tracked set reached MUSCLES_KAFKA_MAX_TOPICS. Raise the limit when this is above 0.
`muscles_ingest_partition_cap_dropped_total`	counter	set at runtime	Partitions not ingested because their topic reached MUSCLES_KAFKA_MAX_PARTITIONS, by topic. Raise the limit when this is above 0.
`muscles_ingest_paused`	gauge		1 while Kafka ingestion is paused on this node by POST /api/ingestion/pause, otherwise 0. Alert when it stays at 1 for too long.
`muscles_ingest_decode_errors_total`	counter		Total Avro/decode errors during ingestion.
`muscles_ingest_share_append_stalled_total`	counter		Share-group partitions stopped because a record kept failing to decode, find its table or append. The record stays unacknowledged, and the log names its topic, partition and offset.
`muscles_ingest_share_partitions_stalled`	gauge	set at runtime	1 while the share consumer of a topic is stopped at a record that keeps failing, otherwise 0. Alert when it is 1, because lag grows until you fix the cause and post the topic configuration again.
`muscles_ingest_poison_dropped_total`	counter		Total poison records dropped after the delivery-attempt cap.
`muscles_ingest_redeliveries_total`	counter		Share-group records acquired again after an earlier delivery that timed out or was released.
`muscles_ingestion_failed_topics`	gauge		Number of topics whose non-critical ingestion errored on the current cycle.
`muscles_ingest_failed_topics`	gauge		Topics quarantined after repeated unrecoverable errors. They are skipped until recovery or reset.
`muscles_ingest_topic_errors_total`	counter	set at runtime	Topic-level ingestion errors, by topic and class (transient\|unrecoverable).
`muscles_ingest_topic_skipped_total`	counter	set at runtime	Share-group cycles in which a topic was not consumed at all, by topic and reason. A skip alone is not a failure.
`muscles_ingest_skipped_topics`	gauge		Topics that were neither consumed nor failed on the current ingest cycle.
`muscles_ingest_schema_rejected_total`	counter	set at runtime	Records rejected by the schema-evolution compatibility gate, by topic.
`muscles_ingest_oversized_cell_stalled_partitions`	gauge		Kafka partitions stopped at a record larger than half of MUSCLES_PART_BLOCK_MAX_BYTES. Raise the cap and restart, or run `SYSTEM SKIP KAFKA RECORD`.
`muscles_ingest_oversized_cell_skipped_total`	counter	set at runtime	Oversized Kafka records dropped after an operator approved the skip with `SYSTEM SKIP KAFKA RECORD`, by topic.
`muscles_ingest_schema_evolved_total`	counter	set at runtime	Accepted schema promotions (table schema commits), by topic.
`muscles_ingest_lossy_promotion_total`	counter	set at runtime	Accepted promotions from int64 to float64, which can lose precision above 2^53, by topic.
`muscles_ingestion_degraded_topics`	gauge		Topics with a rejected producer schema. Records with accepted schema ids still flow.
`muscles_ingest_schema_quarantined`	gauge	set at runtime	Quarantined writer-schema ids per topic and reason. Records that carry them are dropped, and system.ingest_quarantine lists each id.
`muscles_ingest_schema_quarantined_since_seconds`	gauge	set at runtime	Unix time in seconds of the oldest active schema quarantine of a topic.
`muscles_ingest_share_foreign_blocks_total`	counter	set at runtime	ShareFetch response blocks for a partition the request did not name, discarded, by leader broker. It should stay at 0, and a rise means the request and the broker's share session disagree.
`muscles_ingest_share_targets_deferred`	gauge	set at runtime	Topic-partitions left out of a ShareFetch because the per-request target limit was reached, by leader broker. Later cycles serve them, and the value should stay at 0.
`muscles_ingest_share_topics_unsubscribed`	gauge		Tracked topics that do not fit in one share-group subscription and are never consumed by this member. It should stay at 0.
`muscles_ingest_local_total`	counter	set at runtime	Records ingested on the shard that owns them under keyed ingest, by topic.
`muscles_ingest_forward_pending_total`	counter	set at runtime	Records left in Kafka because their partition stalled on rows owned by another shard, by topic. The forward was not acknowledged or MUSCLES_SHARD_MAP is unset.
`muscles_ingest_eager_table_created_total`	counter	set at runtime	Destination tables created from a topic's Schema Registry subject at discovery, before this node consumed a record, by topic.
`muscles_ingest_eager_subject_missing_total`	counter	set at runtime	Topics without a <topic>-value Schema Registry subject. Their table is created from the first consumed record, so create it on every node before distributed queries.
`muscles_ingest_eager_unsafe_versions_total`	counter	set at runtime	Topics not created at discovery because an older subject version would be rejected by a table built from the latest one. The first record creates their table instead.
`muscles_ingest_eager_unverified_total`	counter	set at runtime	Topics not created at discovery because their subject's versions could not all be checked. The first record creates their table instead.
`muscles_ingest_offset_reset_total`	counter	set at runtime	Times a classic ingest partition's fetch offset was out of range and was reset to the log start or end, by topic. Each reset after retention loses records that were never ingested.
`muscles_ingest_offset_reset_skipped_total`	counter	set at runtime	Records skipped by classic ingest offset resets because retention deleted them before they were ingested, by topic.
`muscles_ingest_forwarded_total`	counter	set at runtime	Records forwarded to their owner shard and durably acknowledged by it, by topic.
`muscles_ingest_forward_received_total`	counter	set at runtime	Forwarded records this shard durably applied from /internal/ingest-forward, by topic.
`muscles_ingest_forward_oversized_total`	counter	set at runtime	Forwarded records skipped because they exceed the receiver's size limit, by topic. These records are not delivered.
`muscles_ingest_forward_nondedup_total`	counter	set at runtime	Forwarded records applied to a table that is not KAFKA DEDUP or Replacing, by topic. A retried forward duplicates such rows.
`muscles_ingest_topic_aligned`	gauge	set at runtime	1 when a topic is ingested without forwarding, otherwise 0, by topic. It applies when MUSCLES_KAFKA_ALIGNED is set and the shard count divides the partition count.
`muscles_ingest_alignment_violation_total`	counter	set at runtime	Records forwarded although their topic is declared aligned, by topic. A non-zero value means the producer's partitioner does not match shardOf.
`muscles_import_files_total`	counter		Data files (Parquet) committed by IMPORT ICEBERG and IMPORT PARQUET.
`muscles_import_rows_total`	counter		Rows committed by IMPORT ICEBERG and IMPORT PARQUET. Only committed imports count.
`muscles_import_failures_total`	counter		IMPORT statements that failed. A failed import leaves no partial table.
`muscles_parts_active`	gauge	set at runtime	Active parts, hot and cold, by table; sampled at scrape time.
`muscles_flush_duration_seconds`	histogram		MemTable flush duration in seconds.
`muscles_flush_bytes_total`	counter		Total bytes flushed to sealed parts.
`muscles_flushes_total`	counter		Completed MemTable flushes (flush-loop throughput).
`muscles_flush_failures_total`	counter	set at runtime	MemTable flushes that failed, by table. A steadily rising count means the table's memtable is not draining and ingest into it will hit backpressure.
`muscles_merge_running`	gauge		Part merges currently running on this node.
`muscles_merge_duration_seconds`	histogram	set at runtime	Merge duration in seconds.
`muscles_merge_bytes_written_total`	counter	set at runtime	Total bytes written by merges.
`muscles_merge_backlog`	gauge	set at runtime	Parts the background merger would still merge, by table, plus lone parts with deleted rows. A settled table reads 0. Sampled at scrape time.
`muscles_merge_failures_total`	counter	set at runtime	Part merges that failed, by table and reason. The parts stay unmerged and are retried after a backoff, so a rising count means the part count grows.
`muscles_merge_unsorted_inputs_total`	counter	`outcome`: `sorted`, `skipped`	Merge inputs of a replacing or aggregating table with a varchar key that were not sorted by value, by outcome. A skipped input keeps its rows uncollapsed until a later merge sorts it.
`muscles_merges_total`	counter		Completed part merges (merge-loop throughput).
`muscles_dedup_merges_total`	counter		Completed merges that ran the Replacing dedup reducer.
`muscles_dedup_rows_dropped_total`	counter		Rows collapsed as superseded duplicates by the Replacing reducer. A winning tombstone is kept and not counted.
`muscles_fsync_failures_total`	counter	`kind`: `wal`, `part`, `manifest`, `dir`	fsync failures at a durable-write boundary, by kind. The operation still fails with the error, and this counter is the signal to alert on for risk of data loss.
`muscles_wal_failed_tables`	gauge		Tables whose WAL failed an fsync and rejects writes until their rows are flushed to parts and the WAL recovers.
`muscles_wal_bytes_appended_total`	counter		WAL frame bytes durably appended across all tables (ingest write volume).
`muscles_wal_appends_total`	counter		WAL append records written across all tables.
`muscles_wal_corruption_segments_total`	counter		WAL segments with corruption that recovery salvaged under MUSCLES_WAL_ON_CORRUPTION=salvage; each one lost acknowledged rows.
`muscles_parts_corrupt_salvaged_total`	counter		Parts with a corrupt manifest skipped at recovery under MUSCLES_WAL_ON_CORRUPTION=salvage; each one lost acknowledged rows.
`muscles_wal_corruption_records_salvaged_total`	counter		WAL records recovered from the intact prefix of salvaged segments. They stay in the segment until a part covers them.
`muscles_wal_corruption_records_lost_total`	counter		Estimated WAL records lost in salvaged segments. The salvage log line gives the lsn range to re-ingest.
`muscles_ingest_backpressure_total`	counter		Durable inserts rejected with backpressure (ingest outrunning the flush loop).
`muscles_snapshots_total`	counter		Consistent local snapshots written successfully.
`muscles_snapshot_failures_total`	counter		Consistent-snapshot attempts that failed.
`muscles_matview_refresh_failures_total`	counter	set at runtime	Failures of materialized-view refresh on insert, by view. The view is marked stale.
`muscles_matview_lagging_views`	gauge		Materialized views that lag their source because a refresh did not fit the memory budget or failed temporarily. Each catches up from the source WAL when memory is free.
`muscles_matview_retired_views`	gauge		Materialized views from failed DDL that stay in memory until the process stops, because a reader may still hold them. Each takes a few KiB.
`muscles_matview_lagging`	gauge	set at runtime	1 while the view lags its source, else 0.
`muscles_matview_lag_rows`	gauge	set at runtime	Source rows the view has not applied yet.
`muscles_matview_lag_bytes`	gauge	set at runtime	Bytes of source WAL segments that hold the view's backlog, counted in whole segments.
`muscles_matview_lag_seconds`	gauge	set at runtime	Seconds since the oldest source row the view has not applied arrived.
`muscles_matview_stale_views`	gauge		Materialized views that are stale now. Each holds its source WAL until DROP, so a rising value risks unbounded WAL growth and a halt of ingest.
`muscles_matview_over_replacing_source_total`	counter		Aggregating materialized views and rollups created over a Replacing (dedup) source, which can over-count duplicates until they collapse on merge.
`muscles_matview_stale_wal_dropped_total`	counter	set at runtime	Stale materialized views dropped automatically because their frozen WAL backlog passed MUSCLES_MV_STALE_WAL_MAX_BYTES, by view. Recreate the named view.
`muscles_view_backfills_total`	counter	`outcome`: `committed`, `failed`	Materialized-view backfills and rebuilds that ended, by outcome. A failed run leaves the view and its target unchanged, and system.view_backfills names the error.
`muscles_view_backfills_running`	gauge		Materialized-view backfills and rebuilds running now. Each holds a hidden staging table and, for a rebuild, the old and new target rows on disk.
`muscles_view_backfill_rows_total`	counter		Source rows that view backfills and rebuilds wrote to their staging tables.
`muscles_matview_collapse_skipped_total`	counter		Aggregate-state view groups that a merge could not collapse and wrote uncollapsed. The merge itself still succeeds.
`muscles_memtable_bytes`	gauge	set at runtime	Estimated in-memory MemTable size in bytes, by table. It feeds backpressure and is sampled at scrape time.
`muscles_memtable_rows`	gauge	set at runtime	In-memory MemTable row count, by table; sampled at scrape time.
`muscles_disk_used_bytes`	gauge	set at runtime	Bytes used on the filesystem of each data disk (total minus available), by mount: the storage-policy disk name, or default.
`muscles_disk_free_bytes`	gauge	set at runtime	Bytes available to muscles on the filesystem of each data disk, by mount: the storage-policy disk name, or default.
`muscles_tier_parts`	gauge	`tier`: `hot`, `cold`	Installed parts by storage tier (hot\|cold).
`muscles_tier_bytes`	gauge	`tier`: `hot`, `cold`	On-disk part bytes by storage tier. A cold part's payload lives in S3.
`muscles_tier_moves_total`	counter		Hot parts moved to the S3 cold tier.
`muscles_tier_move_failures_total`	counter	`reason`: `oom`, `read`, `upload`, `stub`, `superseded`, `publish`, `crash`	Tier moves that did not complete, by reason. The part stays hot and is retried on the next poll.
`muscles_tier_local_moves_total`	counter		Parts moved between two local volumes by the tier mover.
`muscles_gdpr_delete_block_skipped_total`	counter		GDPR DELETE masks abandoned because a merge or tier move replaced the target part first. Run the DELETE again.
`muscles_delete_conflicts_total`	counter		DELETEs that returned 503 and Retry-After because another DELETE held the table too long or merges kept replacing the matched parts.
`muscles_gdpr_rows_purged_total`	counter		Rows physically removed from disk by a merge because a deletion vector marked them.
`muscles_tier_local_move_failures_total`	counter	`reason`: `oom`, `read`, `copy`, `redirect`, `superseded`, `publish`, `crash`	Local part moves that did not complete, by reason. The part stays on its source volume and is retried on the next poll.
`muscles_tier_local_orphans_swept_total`	counter		Leftovers of local part moves that the orphan sweep reclaimed, such as .orphan files and part copies from crashed moves.
`muscles_cold_upload_bytes_total`	counter		Payload bytes uploaded to the S3 cold tier by the mover.
`muscles_cold_upload_duration_seconds`	histogram		Cold-tier object upload latency in seconds.
`muscles_cold_upload_last_success_timestamp_seconds`	gauge		Unix time in seconds of the last successful cold-tier upload.
`muscles_cold_cache_hits_total`	counter		Cold-payload cache hits, where a scan reused a cached column without an S3 GET.
`muscles_cold_cache_misses_total`	counter		Cold-payload cache misses, where a scan loaded a column from S3.
`muscles_cold_cache_evictions_total`	counter		Cold-payload cache files evicted to stay under MUSCLES_COLD_CACHE_MAX_BYTES.
`muscles_cold_cache_bytes`	gauge		Resident cold-payload cache bytes (payload files only).
`muscles_cold_fetch_bytes_total`	counter		Payload bytes fetched from the S3 cold tier by scans (ranged GETs).
`muscles_cold_fetch_duration_seconds`	histogram		Cold-tier ranged-GET latency in seconds (per projected column fault).
`muscles_cold_fetch_failures_total`	counter		Cold-tier ranged GETs that failed. The query fails with an error and never returns partial results.
`muscles_tier_expired_total`	counter		Parts deleted after aging past MUSCLES_TIER_DELETE_AFTER_MS, hot or cold. A cold part loses its S3 object and its local metadata.
`muscles_tier_gc_backlog`	gauge		Cold-object deletes queued but not yet confirmed against S3. A persistent S3 delete failure keeps items queued, so a rising value needs attention.
`muscles_tier_orphans_swept_total`	counter		Stray S3 objects that no local cold stub references, reclaimed by the reconciliation sweep after the grace period.
`muscles_tier_sweep_incomplete_total`	counter		Orphan-sweep cycles skipped because a local part.cold stub failed to parse and the reference set was incomplete. A rising value means the sweep is not running.
`muscles_retention_parts_dropped_total`	counter	set at runtime	Local parts destroyed by a table TTL, by table. Cold parts in S3 are not counted.
`muscles_retention_rows_dropped_total`	counter	set at runtime	Rows removed by a per-table TTL, by table: the live rows of every dropped part, the rows masked by a sweep and the rows a merge left out.
`muscles_migrate_holds_active`	gauge		Open migration export sessions. Each one pauses merges and TTL drops on its tables.
`muscles_migrate_hold_age_seconds`	gauge		Age of the oldest open migration hold in seconds. A value near the hold age cap means a migrator died or stalled.
`muscles_migrate_hold_parts`	gauge		Largest number of parts a held table has gained since its migration session began. Merges are paused, so it grows with every flush.
`muscles_migrate_holds_ended_total`	counter	`reason`: `end`, `lease`, `age_cap`, `parts_cap`	Migration holds ended, by reason: end (released by the migrator), lease (lease expired), age_cap (hold age cap), parts_cap (part count cap).
`muscles_retention_sweeps_total`	counter		Retention sweep cycles completed. A flat value means the sweeper is not running or MUSCLES_RETENTION_ENABLED is not set.
`muscles_query_history_archive_failures_total`	counter		Queries that could not be archived to system.query_history because its table could not be created or the durable append failed. Each one is missing from the history.
`muscles_query_history_fsync_failures_total`	counter		Failed fsyncs of the system.query_history WAL. With MUSCLES_QUERY_HISTORY_SYNC=sync, a non-zero value means an answered query may not be archived durably.
`muscles_rollup_raw_dropped_total`	counter		Raw source parts deleted by downsampling after rollups durably stored their whole range.
`muscles_rollup_raw_drop_deferred_total`	counter	`reason`: `drop_disabled`, `no_rung`, `rung_not_durable`, `no_watermark_seam`, `no_source_wal`, `below_rung_floor`	Raw parts that downsampling kept instead of dropping, by reason. The usual reason is drop_disabled, because raw drop is off by default.
`muscles_query_below_drop_horizon_total`	counter	`outcome`: `refused`, `signaled`	Queries that needed rows below the downsampling drop horizon, by outcome. A refused query gets HTTP 400, which is the default, and a signaled query returns partial rows under allow_partial_below_drop_horizon.
`muscles_policy_reject_total`	counter	`scope`: `schema`, `table`, `column`, `overflow`, `unresolved`, `derived_def`, `mask`, `mask_star`, `mask_derived`, `filter`, `embargo`, `dictionary`	Queries rejected by a data-level RBAC policy, by the scope that fired or the check that could not prove the query safe.
`muscles_audit_captured_total`	counter	`kind`: `query_start`, `query_finish`, `query_exception`, `login_success`, `login_failure`, `logout`, `access_denied`, `ddl`, `insert`, `node_boot`, `audit_truncate`, `operator_action`	Events written to the audit chain, by kind. It stays 0 while MUSCLES_AUDIT_ENABLED is unset.
`muscles_audit_capture_dropped_total`	counter	`kind`: `query_start`, `query_finish`, `query_exception`, `login_success`, `login_failure`, `logout`, `access_denied`, `ddl`, `insert`, `node_boot`, `audit_truncate`, `operator_action`	Events that could not be written to the audit chain, by kind. With MUSCLES_AUDIT_MODE=sync the request also fails with 503.
`muscles_asserted_identity_rejected_total`	counter	`reason`: `control_char`, `invalid_utf8`	Proxy-forwarded end-user claims rejected as malformed, by reason. The audit attribution of the request stays empty.
`muscles_auth_proxy_identity_refused_total`	counter	`reason`: `empty`, `too_long`, `sentinel`, `control_char`, `invalid_utf8`, `unknown`	Trusted-proxy identity headers refused, by reason. Each refusal is a 401, most often for an identity longer than 64 bytes, and the reason unknown should stay at 0.
`muscles_auth_oidc_unlisted_refused_total`	counter		OIDC sign-ins refused with 403 because the identity is not in MUSCLES_ADMINS, MUSCLES_VIEWERS or the users file and MUSCLES_OAUTH_UNLISTED_ROLE is none.
`muscles_audit_boot_immutable`	gauge		1 when this process started with MUSCLES_AUDIT_IMMUTABLE on, otherwise 0. It is set only at startup, so a drop to 0 means a restart without WORM.
`muscles_audit_expired_blocks_total`	counter		Sealed audit blocks deleted by local expiry. It applies only with MUSCLES_AUDIT_IMMUTABLE off and MUSCLES_AUDIT_RETENTION_DAYS or MUSCLES_AUDIT_MAX_BYTES set.
`muscles_audit_truncate_total`	counter		Completed audit-log truncations, each with a durable marker and an audit_truncate record in the chain.
`muscles_audit_record_truncated_total`	counter		Audit records whose over-long column, usually the query text, was cut at capture and marked inline with [audit:truncated N bytes]. A sustained non-zero value usually means a client sends multi-megabyte statements.
`muscles_audit_s3_uploaded_blocks_total`	counter		Sealed audit blocks archived to S3 and verified by reading them back. It stays 0 unless MUSCLES_AUDIT_S3_BUCKET is set.
`muscles_audit_s3_uploaded_bytes_total`	counter		Bytes of compressed audit archive objects written to S3.
`muscles_audit_s3_upload_fail_total`	counter		Audit archive sweeps that failed; local audit blocks are kept until a sweep succeeds.
`muscles_audit_s3_backlog_blocks`	gauge		Sealed local audit blocks not yet covered by the verified S3 archive.
`muscles_audit_s3_divergence_total`	counter		Times the S3 audit archive and this node disagreed about a sequence range. Archiving then stops until restart, and no local audit block is reclaimed.
`muscles_audit_s3_archived_upto`	gauge		Sequence number below which every audit block has a verified S3 copy. Only blocks below it may be reclaimed locally.
`muscles_audit_s3_lock_posture`	gauge		1 when this node verified that the archive bucket has versioning and Object Lock, otherwise 0. With MUSCLES_AUDIT_S3_OBJECT_LOCK=require, 0 means nothing is archived.
`muscles_audit_s3_lock_unproven_total`	counter		Times the archive bucket answered but lacked versioning or Object Lock.
`muscles_audit_s3_locked_objects_total`	counter		Audit archive objects whose Object Lock retention was verified by reading it back.
`muscles_audit_s3_digest_objects_total`	counter		Audit digest objects committed to S3. It stays 0 unless MUSCLES_AUDIT_S3_DIGEST=1.
`muscles_audit_s3_digest_fail_total`	counter		Audit digest writes that failed. The archive is unaffected, but the tip hash of that batch is missing from the digest history.
`muscles_audit_mirror_objects_total`	counter		Parquet mirror objects of the audit log, written for auditors and not verified. It stays 0 unless MUSCLES_AUDIT_S3_PARQUET_MIRROR=1.
`muscles_audit_mirror_blocks_total`	counter		Audit blocks written to the Parquet mirror. Compare with muscles_audit_s3_uploaded_blocks_total to see how far the mirror lags.
`muscles_audit_mirror_fail_total`	counter		Parquet mirror writes that failed. Failed objects are not retried, and the S3 archive is not affected.
`muscles_retention_immutable_skips_total`	counter	set at runtime	Retention sweeps skipped because MUSCLES_AUDIT_IMMUTABLE is on, by table. A non-zero value means retention is not enforced for that table.
`muscles_search_cold_index_missing_total`	counter		Cold parts searched by a brute-force full-text scan because their search index file was missing or corrupt. Results stay correct.
`muscles_vector_knn_index_used_total`	counter		Parts whose vector KNN query used the IVF index (probe plus exact re-rank).
`muscles_vector_knn_bruteforce_total`	counter	`reason`: `io_error`, `no_sidecar`, `corrupt_index`, `metric_mismatch`, `dim_mismatch`, `null_vectors`, `underfilled`, `has_deletions`	Parts whose vector KNN query ran an exact brute-force scan, by reason (io_error\|no_sidecar\|corrupt_index\|metric_mismatch\|dim_mismatch\|null_vectors\|underfilled\|has_deletions).
`muscles_replication_commits_rejected_total`	counter		Commit-log records rejected because a name or table from the wire failed the path check. A non-zero value means a peer sends malformed commits.
`muscles_replication_emit_total`	counter		Part records shipped to the cold tier and commit log, PART_ADDED or PART_REMOVED, while MUSCLES_REPLICATION_EMIT is on.
`muscles_replication_schema_wait_ms_total`	counter		Milliseconds the emit thread waited for a catalog change to queue its schema records.
`muscles_replication_schema_wait_timeouts_total`	counter		Part records that waited for a catalog change whose schema records were not queued, so they failed to ship. A later pass ships them.
`muscles_replication_ddl_apply_failures_total`	counter		Schema records the replica could not apply. The drain halts at the record.
`muscles_replication_ddl_apply_halted`	gauge		1 while the drain is halted at a schema record that could not be applied, else 0; alert when it stays 1.
`muscles_replication_schema_waiting`	gauge		1 while the replica waits for a schema snapshot it can apply (none in the bucket, or it failed a check) and drains nothing, else 0.
`muscles_replication_ddl_withheld_total`	counter		Schema records for objects the primary withheld because their definition held a secret or was too long. Create those objects by hand.
`muscles_replication_ddl_quarantined_parts_total`	counter		Parts of tables that a failover snapshot dropped. They stay in the replication-quarantine directory until you remove them.
`muscles_replication_ddl_kept_for_local_view_total`	counter		Tables the primary dropped that the replica keeps because a view created on the replica uses them. The replica differs from the primary until you drop them.
`muscles_replication_ddl_parts_skipped_total`	counter		Parts the replica skipped because their table was dropped or belongs to an older incarnation of the name.
`muscles_replication_schema_sync_failures_total`	counter		Catalog changes that could not be queued as schema records, because the epoch file write failed, the outbox was full or the definition was unreadable. The next start announces the catalog again.
`muscles_replication_emit_failures_total`	counter		Emit attempts that failed, either an S3 PUT or a Kafka append. The local part stays durable and replicas catch up later.
`muscles_replication_emit_dropped_total`	counter		Flush additions dropped because the emit queue was full. The local part stays durable and the emit reconcile ships it again.
`muscles_replication_emit_reshipped_total`	counter		Live parts whose S3 bundle was missing and that the emit reconcile shipped again. It is non-zero after a crash during a ship and not expected otherwise.
`muscles_replication_merge_deferred_total`	counter		Merges skipped because the emitter had no room for the merge's output and removals, so the commit-log broker or S3 cannot keep up. The table is merged on a later pass.
`muscles_replication_merge_deferred`	gauge		Merges skipped in a row since the last merge that got room. A non-zero value means merges are held back now.
`muscles_replication_emit_pending_removals`	gauge		Removals (PART_REMOVED) queued or awaiting retry in the emitter.
`muscles_replication_emit_outputs_held`	gauge		Merge outputs waiting to be shipped. The tier mover does not move them to the cold tier until they ship.
`muscles_replication_emit_removals_lost_total`	counter		Removals never shipped because the emitter stopped with them queued, the list was full or memory ran out. A replica may keep those parts.
`muscles_replication_emit_merge_output_skipped_total`	counter		Merge outputs not shipped because the part was no longer live when its turn came, so its rows left this node too. The removals of the merge still ship.
`muscles_replication_drain_applied_total`	counter		Parts installed by the replication drain loop (PART_ADDED) while MUSCLES_REPLICATION_DRAIN is on.
`muscles_replication_drain_removed_total`	counter		Parts removed by the replication drain loop (PART_REMOVED). A removal for a part that was never installed is not counted.
`muscles_replication_drain_rejected_total`	counter		Commit-log records the drain loop rejected because a name or table from the wire failed the path check.
`muscles_replication_drain_errors_total`	counter		Drain iterations that failed because of a broker or S3 transport error or an offset checkpoint fsync. The next iteration retries from the same offset.
`muscles_replication_drain_offset`	gauge		Durable commit-log offset the drain resumes from. It advances only after the parts of the batch are durably installed.
`muscles_replication_drain_transient_total`	counter	`cause`: `s3_get`, `table_missing`, `tombstoned`, `unreadable`, `uninstall`, `missing_bundle`, `install`, `migrate_sealed`	Commit-log records whose install or removal failed temporarily and is retried. A steady rise means the drain is stuck, for example because destination tables are missing.
`muscles_replication_drain_offset_reset_total`	counter		Times the drain offset was reset to the earliest offset after OFFSET_OUT_OF_RANGE, because the replica fell behind the commit topic's retention.
`muscles_replication_drain_reset_catchup_total`	counter		S3 catch-ups that succeeded after the drain offset was reset past commits that fell out of the commit topic's retention.
`muscles_replication_drain_reset_catchup_failures_total`	counter		S3 catch-ups after a drain offset reset that failed. The reset is retried with the old offset.
`muscles_replication_catchup_retired_total`	counter		Replicated parts uninstalled by a catch-up because the shard manifest no longer lists them.
`muscles_replication_rejoin_quarantined_parts_total`	counter		Parts a former primary wrote, the new primary's manifest does not list, and its rejoin moved to replication-quarantine.
`muscles_replication_rejoin_quarantined_rows_total`	counter		Rows in the parts a rejoin moved to replication-quarantine.
`muscles_replication_drain_missing_bundle_total`	counter		Commit-log records whose part bundle was missing from S3 when the drain fetched it. The drain retries, then falls back to a manifest catch-up.
`muscles_replication_drain_missing_bundle_fallback_total`	counter		Manifest catch-ups the drain ran because it was stuck on a missing part bundle.
`muscles_replication_drain_missing_bundle_skipped_total`	counter		Records with a missing part bundle that the drain skipped after a fallback catch-up, because the shard manifest does not list the part.
`muscles_replication_bundle_gc_deleted_total`	counter		Part bundles the emitter deleted from S3 after the part was removed and the grace period passed.
`muscles_replication_bundle_gc_failures_total`	counter		Bundle GC passes or deletes that failed because of an S3 error or an unreadable manifest. A later pass retries them.
`muscles_replication_bundle_gc_untracked_total`	counter		Removed parts whose bundle the GC does not track because its candidate list was full. Those bundles stay in S3.
`muscles_replication_fence_scan_errors_total`	counter		Emitter fence scans of the commit log that failed because of a broker error or a rejected record. The S3 manifest check still runs.
`muscles_replication_catchup_runs_total`	counter		Catch-up runs that read the shard's S3 manifest and installed the live parts missing on this node. Catch-up never removes parts.
`muscles_replication_catchup_installed_total`	counter		Parts fetched from S3 and installed by catch-up because they were missing locally.
`muscles_replication_catchup_already_present_total`	counter		Manifest parts that catch-up found already installed on this node.
`muscles_replication_catchup_rejected_total`	counter		Manifest entries catch-up rejected: an unsafe part name, a reserved system table, or a part bundle that disagrees with the manifest.
`muscles_replication_catchup_no_table_total`	counter		Manifest parts that catch-up skipped because their table does not exist on this node. They are installed once the table exists.
`muscles_replication_catchup_transient_total`	counter		Manifest parts whose fetch or install failed temporarily during catch-up. The next run retries them.
`muscles_replication_manifest_put_total`	counter		Shard liveness manifest PUTs to S3 by the emitter after a batch changed the live part set.
`muscles_replication_manifest_put_failures_total`	counter		Shard manifest PUTs that failed. Replicas keep using the previous manifest until a PUT succeeds.
`muscles_replication_fenced_total`	counter		Times a primary stopped emitting because it saw a higher or conflicting ownership generation. Expect one per promotion, and otherwise demote or restart the node.
`muscles_replication_generation_conflict_total`	counter		Times two primaries claimed the same ownership generation. Resolve it by raising MUSCLES_SHARD_GENERATION on one node.
`muscles_replication_stale_generation_dropped_total`	counter		Commit-log records dropped because they came from a lower ownership generation than the newest seen; a steady rise means the old primary did not stop.
`muscles_replication_unsupported_version_total`	counter		Commit-log records with a record version this binary does not read. The drain and the fence scan halt at the record and retry after a backoff.
`muscles_replication_drain_halted`	gauge		1 while the commit-log drain is halted at a record version this binary does not read, else 0; alert when it stays 1.
`muscles_replication_fence_halted`	gauge		1 while the emitter fence scan is halted at a record version this binary does not read, else 0; alert when it stays 1.
`muscles_replication_mac_rejected_total`	counter		Commit-log records rejected for a missing or invalid signature, before any state changes.
`muscles_replication_mac_context_mismatch_total`	counter		Authentic commit-log records rejected because their signed shard id differs from this consumer's. It means another shard's records reach this topic.
`muscles_replication_system_table_rejected_total`	counter		Commit-log records and catch-up manifest entries rejected because they target a reserved system table, which is never replicated. Expect some during a rolling upgrade.
`muscles_replication_part_name_collision_total`	counter		Replicated parts not installed because a different part on this node already holds their directory name.
`muscles_process_resident_memory_bytes`	gauge	set at runtime	Resident set size of the process in bytes, sampled at scrape time on Linux. Alert before an out-of-memory kill.
`muscles_process_open_fds`	gauge	set at runtime	Open file descriptors of the process, sampled at scrape time on Linux.
`muscles_cpu_limit`	gauge		CPUs the process may use: the smaller of the CPU affinity count and the cgroup CPU quota, rounded up. Per-CPU defaults such as scan workers follow it.
`muscles_memory_limit_bytes`	gauge		Memory the process may use: the smaller of physical RAM and the cgroup memory limit, read at boot. 0 when neither could be read.
`muscles_query_memory_limit_bytes`	gauge		Bytes one query may hold before it fails with a memory limit error (max_memory_bytes). 0 means no cap.
`muscles_query_memory_total_limit_bytes`	gauge		Bytes all running queries may hold together before a query fails with a memory limit error. 0 means no cap.
`muscles_memory_bytes`	gauge	`subsystem`: `memtable`, `mv_refire`, `mv_refresh`, `merge`, `query`, `wal_replay`, `cache`, `other`	Bytes each subsystem holds from the shared memory budget, sampled at scrape time. other is resident memory that no subsystem accounts for.
`muscles_memory_budget_bytes`	gauge	`subsystem`: `memtable`, `mv_refire`, `mv_refresh`, `merge`, `query`, `wal_replay`, `cache`, `other`	Bytes each subsystem aims to hold within the shared memory budget, where 0 means no budget of its own. Queries and merges wait or fail past it, and memtables, view refresh and replay are only counted.
`muscles_memory_waits_total`	counter	`subsystem`: `memtable`, `mv_refire`, `mv_refresh`, `merge`, `query`, `wal_replay`, `cache`, `other`	Background work that the shared memory budget deferred, by subsystem, such as a merge that waits for a later poll. A query refused memory counts in muscles_query_governor_rejected_total with reason memory_limit_exceeded.
`muscles_process_threads`	gauge	set at runtime	OS threads of the process, sampled at scrape time on Linux.
`muscles_process_cpu_seconds_total`	counter	set at runtime	User plus system CPU time of the process in seconds, sampled at scrape time on Linux.
`muscles_background_seconds_total`	counter	`activity`: `merge`, `materialized_view`, `rollup`, `retention`, `flush`	Elapsed time spent in background work, by activity: merges, flushes and retention sweeps on their own threads; materialized view and rollup refresh on the inserting thread.
`muscles_ready`	gauge		1 when the node is ready, otherwise 0. Ready means the node has started and is neither draining nor lagging in its WAL, as /ready reports. Sampled at scrape time.
`muscles_assistant_requests_total`	counter	set at runtime	Assistant chat requests by provider and terminal status.
`muscles_assistant_stream_seconds`	histogram	set at runtime	Assistant stream duration in seconds, by provider.
`muscles_assistant_active`	gauge		Assistant requests currently streaming.
`muscles_trace_spans_exported_total`	counter		Finished spans sent to the OTLP collector by the exporter thread.
`muscles_trace_spans_dropped_total`	counter	`reason`: `queue_full`, `encode`, `pool`	Finished spans dropped by the exporter, by reason: queue_full, encode, or pool when a background trace could not allocate its span pool and exported its root only. Traces are lost but queries are not.
`muscles_trace_slow_queries_total`	counter		Queries that ran at least MUSCLES_TRACE_SLOW_QUERY_MS and were exported regardless of the sample ratio.
`muscles_trace_export_failures_total`	counter	`reason`: `connect`, `timeout`, `http`	OTLP export POSTs that failed, by reason. The batch is dropped, and a down collector never blocks or slows a query.
`muscles_trace_export_duration_seconds`	histogram		OTLP export POST round-trip latency in seconds (per flush tick that had spans).
`muscles_trace_queue_depth`	gauge		Finished-span batches queued for the exporter thread, sampled on each flush tick. A value near the configured queue depth means spans are about to be dropped.
`muscles_rules`	gauge		Recording rules loaded by the evaluator on this node.
`muscles_rule_up`	gauge	set at runtime	1 when the rule's last evaluation succeeded, 0 when it failed or has not run yet, by rule.
`muscles_rule_series`	gauge	set at runtime	Series the rule's last successful evaluation exports (rows times numeric columns), by rule; 0 after a failure.
`muscles_rule_last_success_timestamp_seconds`	gauge	set at runtime	Unix time of the rule's last successful evaluation, by rule; 0 before the first.
`muscles_rule_evaluations_total`	counter	set at runtime	Rule evaluations run, successful or not, by rule.
`muscles_rule_evaluations_skipped_total`	counter	set at runtime	Evaluations that were due but skipped because the previous one overran, by rule.
`muscles_rule_evaluation_failures_total`	counter	set at runtime	Failed rule evaluations, by rule and reason. A failed rule exports no series.
`muscles_rule_evaluation_duration_seconds`	histogram	set at runtime	Rule evaluation time in seconds, by rule.
`muscles_alert_notifications_sent_total`	counter	`target`: `alertmanager`, `webhook`	Alerts delivered, by target kind (alertmanager\|webhook).
`muscles_alert_notifications_failed_total`	counter	`target`: `alertmanager`, `webhook`	Alerts in requests that still failed after their retries, by target kind; they are dropped and sent again at the next resend.
`muscles_alert_notifications_dropped_total`	counter	`target`: `alertmanager`, `webhook`	Alerts dropped from a full delivery queue (MUSCLES_ALERT_QUEUE_CAPACITY), oldest first, by target kind.
`muscles_alert_notification_retries_total`	counter	`target`: `alertmanager`, `webhook`	Alert delivery requests retried after a timeout, transport error, 429 or 5xx, by target kind.
`muscles_alert_notification_queue_length`	gauge	`target`: `alertmanager`, `webhook`	Alerts waiting to be delivered, by target kind.
`muscles_alert_notification_latency_seconds`	histogram	`target`: `alertmanager`, `webhook`	Time of one alert delivery request in seconds, by target kind.