Search

Companies / Tinybird / incidents

Tinybird: A significant fraction of Clickhouse replicas marked as unhealthy

Impact: minorStatus: resolvedStarted Resolved (1.3 h)4 updatesSource: status.tinybird.co (opens in new tab)

Affected components: API EndpointsBI ConnectorKafkaQuery APIEvents APIQuery APIAPI EndpointsEvents APIKafkaCopyCSV ImportsCSV ImportsCopyKafkaEvents APICSV ImportsQuery APIAPI EndpointsCopySinksSinksSinksLoginS3 ConnectorLoginS3 ConnectorS3 ConnectorLoginLoginAPI EndpointsEvents APIS3 ConnectorBI ConnectorQuery APIKafkaCSV ImportsSinksCopyDynamoDB ConnectorDynamoDB ConnectorDynamoDB ConnectorLoginCopyEvents APIKafkaAPI EndpointsSinkQuery APIS3 ConnectorGCS ConnectorGCS ConnectorGCS ConnectorGCS ConnectorGCS ConnectorQuery APIGCS ConnectorS3 ConnectorEvents APICopySinksKafkaAPI EndpointsLoginSinksLoginSinksAPI EndpointsS3 ConnectorEvents APIEvents APICopyGCS ConnectorQuery APILoginAPI EndpointsGCS ConnectorKafkaQuery APIKafkaCopyS3 ConnectorClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP Interface

Timeline of updates

  1. investigating

    Some database replicas have been marked as unhealthy incorrectly. This is affecting how traffic is distributed among healthy replicas, and in some clusters it's resulting in degraded performance and some errors.

  2. monitoring

    We've marked as healthy the database replicas that were incorrectly marked as unhealthy. We're confirming the cause and have a few suspects. We keep monitoring the situation.

  3. monitoring

    We have narrowed the root cause to an automatic Ubuntu package upgrade triggering the restart of our database services on many servers, and those being marked unhealthy as a result. Service should be generally restored already.

  4. resolved

    We've confirmed that service has been restored fully.

Impact, status and times are the vendor’s own, as last archived. View on status.tinybird.co (opens in new tab). Tinybird profile · Recent Tinybird incidents · Markdown · JSON