Skip to content

Instantly share code, notes, and snippets.

@rsmoorthy

rsmoorthy/es640_fails_to_start

Created October 12, 2018 19:53
Show Gist options
  • Save rsmoorthy/132223e7540edc2c73c2827f8ebb732d to your computer and use it in GitHub Desktop.
Save rsmoorthy/132223e7540edc2c73c2827f8ebb732d to your computer and use it in GitHub Desktop.
Download ZIP
ES640 Fails to start
Raw
es640_fails_to_start
[2018-10-12T19:39:21,050][INFO ][o.e.n.Node ] [] initializing ...
[2018-10-12T19:39:21,111][INFO ][o.e.e.NodeEnvironment ] [5awoLm1] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/nvme1n1)]], net usable_space [229.1gb], net total_space [1.4tb], types [ext4]
[2018-10-12T19:39:21,111][INFO ][o.e.e.NodeEnvironment ] [5awoLm1] heap size [19.9gb], compressed ordinary object pointers [true]
[2018-10-12T19:39:22,670][INFO ][o.e.n.Node ] [5awoLm1] node name derived from node ID [5awoLm1yQ0yOkBdLy3FTzQ]; set [node.name] to override
[2018-10-12T19:39:22,670][INFO ][o.e.n.Node ] [5awoLm1] version[6.4.0], pid[95], build[default/tar/595516e/2018-08-17T23:18:47.308994Z], OS[Linux/3.10.0-862.11.6.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-10-12T19:39:22,670][INFO ][o.e.n.Node ] [5awoLm1] JVM arguments [-Xms20g, -Xmx20g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.01my94d5, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-10-12T19:39:24,514][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [aggs-matrix-stats]
[2018-10-12T19:39:24,514][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [analysis-common]
[2018-10-12T19:39:24,514][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [ingest-common]
[2018-10-12T19:39:24,514][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [lang-expression]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [lang-mustache]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [lang-painless]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [mapper-extras]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [parent-join]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [percolator]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [rank-eval]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [reindex]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [repository-url]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [transport-netty4]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [tribe]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [x-pack-core]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [x-pack-deprecation]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [x-pack-graph]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [x-pack-logstash]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [x-pack-ml]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [x-pack-monitoring]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [x-pack-rollup]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [x-pack-security]
[2018-10-12T19:39:24,515][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [x-pack-sql]
[2018-10-12T19:39:24,516][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [x-pack-upgrade]
[2018-10-12T19:39:24,516][INFO ][o.e.p.PluginsService ] [5awoLm1] loaded module [x-pack-watcher]
[2018-10-12T19:39:24,516][INFO ][o.e.p.PluginsService ] [5awoLm1] no plugins loaded
[2018-10-12T19:39:28,917][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/142] [Main.cc@109] controller (64 bit): Version 6.4.0 (Build cf8246175efff5) Copyright (c) 2018 Elasticsearch BV
[2018-10-12T19:39:29,245][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-10-12T19:39:30,821][ERROR][o.e.g.GatewayMetaState ] [5awoLm1] failed to read local state, exiting...
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:305, file:/var/lib/elasticsearch/nodes/0/_state/global-305.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:199) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:56) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:447) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:256) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.4.0.jar:6.4.0]
Caused by: java.io.IOException: failed to read [id:305, file:/var/lib/elasticsearch/nodes/0/_state/global-305.st]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.4.0.jar:6.4.0]
... 15 more
Caused by: java.lang.IllegalArgumentException: Index patterns must not be null or empty; got null
at org.elasticsearch.cluster.metadata.IndexTemplateMetaData.<init>(IndexTemplateMetaData.java:98) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.build(IndexTemplateMetaData.java:385) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:520) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1215) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1262) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1253) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.4.0.jar:6.4.0]
... 15 more
[2018-10-12T19:39:30,828][ERROR][o.e.b.Bootstrap ] Exception
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:305, file:/var/lib/elasticsearch/nodes/0/_state/global-305.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:199) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:56) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:447) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.4.0.jar:6.4.0]
Caused by: java.io.IOException: failed to read [id:305, file:/var/lib/elasticsearch/nodes/0/_state/global-305.st]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.4.0.jar:6.4.0]
... 15 more
Caused by: java.lang.IllegalArgumentException: Index patterns must not be null or empty; got null
at org.elasticsearch.cluster.metadata.IndexTemplateMetaData.<init>(IndexTemplateMetaData.java:98) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.build(IndexTemplateMetaData.java:385) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:520) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1215) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1262) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1253) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.4.0.jar:6.4.0]
... 15 more
[2018-10-12T19:39:30,831][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:305, file:/var/lib/elasticsearch/nodes/0/_state/global-305.st]]; nested: IOException[failed to read [id:305, file:/var/lib/elasticsearch/nodes/0/_state/global-305.st]]; nested: IllegalArgumentException[Index patterns must not be null or empty; got null];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:305, file:/var/lib/elasticsearch/nodes/0/_state/global-305.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:199) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:56) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:447) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.0.jar:6.4.0]
... 6 more
Caused by: java.io.IOException: failed to read [id:305, file:/var/lib/elasticsearch/nodes/0/_state/global-305.st]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:56) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:447) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.0.jar:6.4.0]
... 6 more
Caused by: java.lang.IllegalArgumentException: Index patterns must not be null or empty; got null
at org.elasticsearch.cluster.metadata.IndexTemplateMetaData.<init>(IndexTemplateMetaData.java:98) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.build(IndexTemplateMetaData.java:385) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:520) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1215) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1262) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1253) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:56) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:447) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.0.jar:6.4.0]
... 6 more
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment