Chapter 3. Troubleshooting logging | Red Hat Product Documentation (2024)

Prerequisites

• The Red Hat OpenShift Logging Operator and OpenShift Elasticsearch Operator are installed.

Procedure

1. Change to the openshift-logging project by running the following command:

   $ oc project openshift-logging

2. Get the ClusterLogging instance status by running the following command:

   $ oc get clusterlogging instance -o yaml

   Example output

   apiVersion: logging.openshift.io/v1kind: ClusterLogging# ...status: 1 collection: logs: fluentdStatus: daemonSet: fluentd 2 nodes: collector-2rhqp: ip-10-0-169-13.ec2.internal collector-6fgjh: ip-10-0-165-244.ec2.internal collector-6l2ff: ip-10-0-128-218.ec2.internal collector-54nx5: ip-10-0-139-30.ec2.internal collector-flpnn: ip-10-0-147-228.ec2.internal collector-n2frh: ip-10-0-157-45.ec2.internal pods: failed: [] notReady: [] ready: - collector-2rhqp - collector-54nx5 - collector-6fgjh - collector-6l2ff - collector-flpnn - collector-n2frh logstore: 3 elasticsearchStatus: - ShardAllocationEnabled: all cluster: activePrimaryShards: 5 activeShards: 5 initializingShards: 0 numDataNodes: 1 numNodes: 1 pendingTasks: 0 relocatingShards: 0 status: green unassignedShards: 0 clusterName: elasticsearch nodeConditions: elasticsearch-cdm-mkkdys93-1: nodeCount: 1 pods: client: failed: notReady: ready: - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c data: failed: notReady: ready: - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c master: failed: notReady: ready: - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7cvisualization: 4 kibanaStatus: - deployment: kibana pods: failed: [] notReady: [] ready: - kibana-7fb4fd4cc9-f2nls replicaSets: - kibana-7fb4fd4cc9 replicas: 1

   1

   In the output, the cluster status fields appear in the status stanza.

   2

   Information on the Fluentd pods.

   3

   Information on the Elasticsearch pods, including Elasticsearch cluster health, green, yellow, or red.

   4

   Information on the Kibana pods.

Prerequisites

• The Red Hat OpenShift Logging Operator and OpenShift Elasticsearch Operator are installed.

Procedure

1. Change to the openshift-logging project.

   $ oc project openshift-logging

2. View the status of logging environment:

   $ oc describe deployment cluster-logging-operator

   Example output

   Name: cluster-logging-operator....Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable....Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 62m deployment-controller Scaled up replica set cluster-logging-operator-574b8987df to 1----

3. View the status of the logging replica set:

   1. Get the name of a replica set:

      Example output

      $ oc get replicaset

      Example output

      NAME DESIRED CURRENT READY AGEcluster-logging-operator-574b8987df 1 1 1 159melasticsearch-cdm-uhr537yu-1-6869694fb 1 1 1 157melasticsearch-cdm-uhr537yu-2-857b6d676f 1 1 1 156melasticsearch-cdm-uhr537yu-3-5b6fdd8cfd 1 1 1 155mkibana-5bd5544f87 1 1 1 157m

   2. Get the status of the replica set:

      $ oc describe replicaset cluster-logging-operator-574b8987df

      Example output

      Name: cluster-logging-operator-574b8987df....Replicas: 1 current / 1 desiredPods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed....Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 66m replicaset-controller Created pod: cluster-logging-operator-574b8987df-qjhqv----

Prerequisites

• You have created a ClusterLogForwarder custom resource (CR) object.

Procedure

• Delete the Fluentd pods to force them to redeploy by running the following command:

  $ oc delete pod --selector logging-infra=collector

Conditions

• The Log Forwarder API is configured to forward logs to Loki.

• Your system sends a block of messages that is larger than 2 MB to Loki. For example:

  "values":[["1630410392689800468","{\"kind\":\"Event\",\"apiVersion\":\.........................\"received_at\":\"2021-08-31T11:46:32.800278+00:00\",\"version\":\"1.7.4 1.6.0\"}},\"@timestamp\":\"2021-08-31T11:46:32.799692+00:00\",\"viaq_index_name\":\"audit-write\",\"viaq_msg_id\":\"MzFjYjJkZjItNjY0MC00YWU4LWIwMTEtNGNmM2E5ZmViMGU4\",\"log_type\":\"audit\"}"]]}]}

• After you enter oc logs -n openshift-logging -l component=collector, the collector logs in your cluster show a line containing one of the following error messages:

  429 Too Many Requests Ingestion rate limit exceeded

  Example Vector error message

  2023-08-25T16:08:49.301780Z WARN sink{component_kind="sink" component_id=default_loki_infra component_type=loki component_name=default_loki_infra}: vector::sinks::util::retries: Retrying after error. error=Server responded with an error: 429 Too Many Requests internal_log_rate_limit=true

  Example Fluentd error message

  2023-08-30 14:52:15 +0000 [warn]: [default_loki_infra] failed to flush the buffer. retry_times=2 next_retry_time=2023-08-30 14:52:19 +0000 chunk="604251225bf5378ed1567231a1c03b8b" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user infrastructure (limit: 4194304 bytes/sec) while attempting to ingest '4082' lines totaling '7820025' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"

  See Also

  Ukrainian professor Igor Girka´s war diary: More than 28 months of madnessU.S.C. Title 22 - FOREIGN RELATIONS AND INTERCOURSE2024 Draft: Reviewing RB ClassArizona Cardinals vs. Denver Broncos Live Score and Stats - August 29, 2019 Gametracker

  The error is also visible on the receiving end. For example, in the LokiStack ingester pod:

  Example Loki ingester error message

  level=warn ts=2023-08-30T14:57:34.155592243Z caller=grpc_logging.go:43 duration=1.434942ms method=/logproto.Pusher/Push err="rpc error: code = Code(429) desc = entry with timestamp 2023-08-30 14:57:32.012778399 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream

Procedure

• Update the ingestionBurstSize and ingestionRate fields in the LokiStack CR:

  apiVersion: loki.grafana.com/v1kind: LokiStackmetadata: name: logging-loki namespace: openshift-loggingspec: limits: global: ingestion: ingestionBurstSize: 16 1 ingestionRate: 8 2# ...

  1

  The ingestionBurstSize field defines the maximum local rate-limited sample size per distributor replica in MB. This value is a hard limit. Set this value to at least the maximum logs size expected in a single push request. Single requests that are larger than the ingestionBurstSize value are not permitted.

  2

  The ingestionRate field is a soft limit on the maximum amount of ingested samples per second in MB. Rate limit errors occur if the rate of logs exceeds the limit, but the collector retries sending the logs. As long as the total average is lower than the limit, the system recovers and errors are resolved without user intervention.

Procedure

1. Check the Elasticsearch cluster health and verify that the cluster status is red by running the following command:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- health

2. List the nodes that have joined the cluster by running the following command:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cat/nodes?v

3. List the Elasticsearch pods and compare them with the nodes in the command output from the previous step, by running the following command:

   $ oc -n openshift-logging get pods -l component=elasticsearch

4. If some of the Elasticsearch nodes have not joined the cluster, perform the following steps.

   1. Confirm that Elasticsearch has an elected master node by running the following command and observing the output:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cat/master?v

   2. Review the pod logs of the elected master node for issues by running the following command and observing the output:

      $ oc logs <elasticsearch_master_pod_name> -c elasticsearch -n openshift-logging

   3. Review the logs of nodes that have not joined the cluster for issues by running the following command and observing the output:

      $ oc logs <elasticsearch_node_name> -c elasticsearch -n openshift-logging

5. If all the nodes have joined the cluster, check if the cluster is in the process of recovering by running the following command and observing the output:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cat/recovery?active_only=true

   If there is no command output, the recovery process might be delayed or stalled by pending tasks.

6. Check if there are pending tasks by running the following command and observing the output:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- health | grep number_of_pending_tasks

7. If there are pending tasks, monitor their status. If their status changes and indicates that the cluster is recovering, continue waiting. The recovery time varies according to the size of the cluster and other factors. Otherwise, if the status of the pending tasks does not change, this indicates that the recovery has stalled.

8. If it seems like the recovery has stalled, check if the cluster.routing.allocation.enable value is set to none, by running the following command and observing the output:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cluster/settings?pretty

9. If the cluster.routing.allocation.enable value is set to none, set it to all, by running the following command:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cluster/settings?pretty \ -X PUT -d '{"persistent": {"cluster.routing.allocation.enable":"all"}}'

10. Check if any indices are still red by running the following command and observing the output:

    $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cat/indices?v

11. If any indices are still red, try to clear them by performing the following steps.

    1. Clear the cache by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name>/_cache/clear?pretty

    2. Increase the max allocation retries by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name>/_settings?pretty \ -X PUT -d '{"index.allocation.max_retries":10}'

    3. Delete all the scroll items by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_search/scroll/_all -X DELETE

    4. Increase the timeout by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name>/_settings?pretty \ -X PUT -d '{"index.unassigned.node_left.delayed_timeout":"10m"}'

12. If the preceding steps do not clear the red indices, delete the indices individually.

    1. Identify the red index name by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cat/indices?v

    2. Delete the red index by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_red_index_name> -X DELETE

13. If there are no red indices and the cluster status is red, check for a continuous heavy processing load on a data node.

    1. Check if the Elasticsearch JVM Heap usage is high by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_nodes/stats?pretty

       In the command output, review the node_name.jvm.mem.heap_used_percent field to determine the JVM Heap usage.

    2. Check for high CPU utilization. For more information about CPU utilitzation, see the OpenShift Container Platform "Reviewing monitoring dashboards" documentation.

Additional resources

• Reviewing monitoring dashboards
• Fix a red or yellow cluster status

Additional resources

• Fix a red or yellow cluster status

Procedure

1. Identify the node on which Elasticsearch is deployed by running the following command:

   $ oc -n openshift-logging get po -o wide

2. Check if there are unassigned shards by running the following command:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cluster/health?pretty | grep unassigned_shards

3. If there are unassigned shards, check the disk space on each node, by running the following command:

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \ do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \ -- df -h /elasticsearch/persistent; done

4. In the command output, check the Use column to determine the used disk percentage on that node.

   Example output

   elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8Filesystem Size Used Avail Use% Mounted on/dev/nvme1n1 19G 522M 19G 3% /elasticsearch/persistentelasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7Filesystem Size Used Avail Use% Mounted on/dev/nvme2n1 19G 522M 19G 3% /elasticsearch/persistentelasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjwFilesystem Size Used Avail Use% Mounted on/dev/nvme3n1 19G 528M 19G 3% /elasticsearch/persistent

   If the used disk percentage is above 85%, the node has exceeded the low watermark, and shards can no longer be allocated to this node.

5. To check the current redundancyPolicy, run the following command:

   $ oc -n openshift-logging get es elasticsearch \ -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   $ oc -n openshift-logging get cl \ -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

6. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name> -X DELETE

Procedure

1. Identify the node on which Elasticsearch is deployed by running the following command:

   $ oc -n openshift-logging get po -o wide

2. Check the disk space on each node:

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \ do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \ -- df -h /elasticsearch/persistent; done

3. Check if the cluster is rebalancing:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cluster/health?pretty | grep relocating_shards

   If the command output shows relocating shards, the high watermark has been exceeded. The default value of the high watermark is 90%.

4. Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.

5. To check the current redundancyPolicy, run the following command:

   $ oc -n openshift-logging get es elasticsearch \ -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   $ oc -n openshift-logging get cl \ -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

6. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name> -X DELETE

• One or more shards are allocated to the node.
• One or more disks exceed the flood stage.

Procedure

1. Get the disk space of the Elasticsearch node:

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \ do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \ -- df -h /elasticsearch/persistent; done

2. In the command output, check the Avail column to determine the free disk space on that node.

   Example output

   elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8Filesystem Size Used Avail Use% Mounted on/dev/nvme1n1 19G 522M 19G 3% /elasticsearch/persistentelasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7Filesystem Size Used Avail Use% Mounted on/dev/nvme2n1 19G 522M 19G 3% /elasticsearch/persistentelasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjwFilesystem Size Used Avail Use% Mounted on/dev/nvme3n1 19G 528M 19G 3% /elasticsearch/persistent

3. Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.

4. To check the current redundancyPolicy, run the following command:

   $ oc -n openshift-logging get es elasticsearch \ -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   $ oc -n openshift-logging get cl \ -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

5. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name> -X DELETE

6. Continue freeing up and monitoring the disk space. After the used disk space drops below 90%, unblock writing to this node by running the following command:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_all/_settings?pretty \ -X PUT -d '{"index.blocks.read_only_allow_delete": null}'

Procedure

1. Get the disk space of the Elasticsearch node:

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \ do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \ -- df -h /elasticsearch/persistent; done

2. In the command output, check the Avail column to determine the free disk space on that node.

   Example output

   elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8Filesystem Size Used Avail Use% Mounted on/dev/nvme1n1 19G 522M 19G 3% /elasticsearch/persistentelasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7Filesystem Size Used Avail Use% Mounted on/dev/nvme2n1 19G 522M 19G 3% /elasticsearch/persistentelasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjwFilesystem Size Used Avail Use% Mounted on/dev/nvme3n1 19G 528M 19G 3% /elasticsearch/persistent

3. Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.

4. To check the current redundancyPolicy, run the following command:

   $ oc -n openshift-logging get es elasticsearch -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   $ oc -n openshift-logging get cl \ -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

5. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name> -X DELETE

Additional resources

• Fix a red or yellow cluster status

Prerequisites

• The Red Hat OpenShift Logging Operator and OpenShift Elasticsearch Operator are installed.

Procedure

1. Change to the openshift-logging project by running the following command:

   $ oc project openshift-logging

2. To view the status:

   1. Get the name of the Elasticsearch log store instance by running the following command:

      $ oc get Elasticsearch

      Example output

      NAME AGEelasticsearch 5h9m

   2. Get the Elasticsearch log store status by running the following command:

      $ oc get Elasticsearch <Elasticsearch-instance> -o yaml

      For example:

      $ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml

      The output includes information similar to the following:

      Example output

      status: 1 cluster: 2 activePrimaryShards: 30 activeShards: 60 initializingShards: 0 numDataNodes: 3 numNodes: 3 pendingTasks: 0 relocatingShards: 0 status: green unassignedShards: 0 clusterHealth: "" conditions: [] 3 nodes: 4 - deploymentName: elasticsearch-cdm-zjf34ved-1 upgradeStatus: {} - deploymentName: elasticsearch-cdm-zjf34ved-2 upgradeStatus: {} - deploymentName: elasticsearch-cdm-zjf34ved-3 upgradeStatus: {} pods: 5 client: failed: [] notReady: [] ready: - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422 - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt data: failed: [] notReady: [] ready: - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422 - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt master: failed: [] notReady: [] ready: - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422 - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt shardAllocationEnabled: all

      1

      In the output, the cluster status fields appear in the status stanza.

      2

      The status of the Elasticsearch log store:

      • The number of active primary shards.
      • The number of active shards.
      • The number of shards that are initializing.
      • The number of Elasticsearch log store data nodes.
      • The total number of Elasticsearch log store nodes.
      • The number of pending tasks.
      • The Elasticsearch log store status: green, red, yellow.
      • The number of unassigned shards.

      3

      Any status conditions, if present. The Elasticsearch log store status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown:

      • Container Waiting for both the Elasticsearch log store and proxy containers.
      • Container Terminated for both the Elasticsearch log store and proxy containers.
      • Pod unschedulable. Also, a condition is shown for a number of issues; see Example condition messages.

      4

      The Elasticsearch log store nodes in the cluster, with upgradeStatus.

      5

      The Elasticsearch log store client, data, and master pods in the cluster, listed under failed, notReady, or ready state.

Elasticsearch indices

You can view the status of the Elasticsearch indices.

1. Get the name of an Elasticsearch pod:

   $ oc get pods --selector component=elasticsearch -o name

   Example output

   pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lwpod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2npod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

2. Get the status of the indices:

   $ oc exec elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -- indices

   Example output

   Defaulting container name to elasticsearch.Use 'oc describe pod/elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -n openshift-logging' to see all of the containers in this pod.green open infra-000002 S4QANnf1QP6NgCegfnrnbQ 3 1 119926 0 157 78green open audit-000001 8_EQx77iQCSTzFOXtxRqFw 3 1 0 0 0 0green open .security iDjscH7aSUGhIdq0LheLBQ 1 1 5 0 0 0green open .kibana_-377444158_kubeadmin yBywZ9GfSrKebz5gWBZbjw 3 1 1 0 0 0green open infra-000001 z6Dpe__ORgiopEpW6Yl44A 3 1 871000 0 874 436green open app-000001 hIrazQCeSISewG3c2VIvsQ 3 1 2453 0 3 1green open .kibana_1 JCitcBMSQxKOvIq6iQW6wg 1 1 0 0 0 0green open .kibana_-1595131456_user1 gIYFIEGRRe-ka0W3okS-mQ 3 1 1 0 0 0

Log store pods

You can view the status of the pods that host the log store.

1. Get the name of a pod:

   $ oc get pods --selector component=elasticsearch -o name

   Example output

   pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lwpod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2npod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

2. Get the status of a pod:

   $ oc describe pod elasticsearch-cdm-1godmszn-1-6f8495-vp4lw

   The output includes the following status information:

   Example output

   ....Status: Running....Containers: elasticsearch: Container ID: cri-o://b7d44e0a9ea486e27f47763f5bb4c39dfd2 State: Running Started: Mon, 08 Jun 2020 10:17:56 -0400 Ready: True Restart Count: 0 Readiness: exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3.... proxy: Container ID: cri-o://3f77032abaddbb1652c116278652908dc01860320b8a4e741d06894b2f8f9aa1 State: Running Started: Mon, 08 Jun 2020 10:18:38 -0400 Ready: True Restart Count: 0....Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True....Events: <none>

Log storage pod deployment configuration

You can view the status of the log store deployment configuration.

1. Get the name of a deployment configuration:

   $ oc get deployment --selector component=elasticsearch -o name

   Example output

   deployment.extensions/elasticsearch-cdm-1gon-1deployment.extensions/elasticsearch-cdm-1gon-2deployment.extensions/elasticsearch-cdm-1gon-3

2. Get the deployment configuration status:

   $ oc describe deployment elasticsearch-cdm-1gon-1

   The output includes the following status information:

   Example output

   .... Containers: elasticsearch: Image: registry.redhat.io/openshift-logging/elasticsearch6-rhel8 Readiness: exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3....Conditions: Type Status Reason ---- ------ ------ Progressing Unknown DeploymentPaused Available True MinimumReplicasAvailable....Events: <none>

Log store replica set

You can view the status of the log store replica set.

1. Get the name of a replica set:

   $ oc get replicaSet --selector component=elasticsearch -o namereplicaset.extensions/elasticsearch-cdm-1gon-1-6f8495replicaset.extensions/elasticsearch-cdm-1gon-2-5769cfreplicaset.extensions/elasticsearch-cdm-1gon-3-f66f7d

2. Get the status of the replica set:

   $ oc describe replicaSet elasticsearch-cdm-1gon-1-6f8495

   The output includes the following status information:

   Example output

   .... Containers: elasticsearch: Image: registry.redhat.io/openshift-logging/elasticsearch6-rhel8@sha256:4265742c7cdd85359140e2d7d703e4311b6497eec7676957f455d6908e7b1c25 Readiness: exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3....Events: <none>

Elasticsearch status fields

eo_elasticsearch_cr_cluster_management_state

Shows whether the Elasticsearch cluster is in a managed or unmanaged state. For example:

eo_elasticsearch_cr_cluster_management_state{state="managed"} 1eo_elasticsearch_cr_cluster_management_state{state="unmanaged"} 0

eo_elasticsearch_cr_restart_total

Shows the number of times the Elasticsearch nodes have restarted for certificate restarts, rolling restarts, or scheduled restarts. For example:

eo_elasticsearch_cr_restart_total{reason="cert_restart"} 1eo_elasticsearch_cr_restart_total{reason="rolling_restart"} 1eo_elasticsearch_cr_restart_total{reason="scheduled_restart"} 3

es_index_namespaces_total

Shows the total number of Elasticsearch index namespaces. For example:

Total number of Namespaces.es_index_namespaces_total 5

es_index_document_count

Shows the number of records for each namespace. For example:

es_index_document_count{namespace="namespace_1"} 25es_index_document_count{namespace="namespace_2"} 10es_index_document_count{namespace="namespace_3"} 5

The "Secret Elasticsearch fields are either missing or empty" message

If Elasticsearch is missing the admin-cert, admin-key, logging-es.crt, or logging-es.key files, the dashboard shows a status message similar to the following example: