Figure out what's up with 5.0

[tianon]

Full log from running it:

$ docker run ...
[2016-04-14 16:41:30,521][WARN ][bootstrap                ] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: your kernel is buggy and you should upgrade
    at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:279)
    at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:616)
    at org.elasticsearch.bootstrap.JNANatives.trySeccomp(JNANatives.java:215)
    at org.elasticsearch.bootstrap.Natives.trySeccomp(Natives.java:99)
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:98)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:151)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:263)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:88)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
Exception in thread "main" java.lang.RuntimeException: please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster.
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:79)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:60)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:187)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:263)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:88)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
Refer to the log for complete error details.

(from #96)

[tianon]

The code in question: https://github.com/elastic/elasticsearch/blob/0c9aca94d4cd295e690a2d522c2b729770d486b5/core/src/main/java/org/elasticsearch/bootstrap/Seccomp.java#L276-L280

[tianon]

Wagering a guess, I'm thinking that Docker's seccomp filter is blocking that unimplemented syscall outright instead of returning ENOSYS.

[tianon]

If I run it on Docker 1.9.1 (before seccomp filtering was implemented), it gets past that line and I get errors about missing cluster config instead, so that's pretty good confirmation of my guess.

[tianon]

Same with --security-opt seccomp=unconfined -- so we might have to come up with a seccomp profile or try to convince upstream that this check should handle blocked syscalls too. 😢

[tianon]

Here's the next hurdle (which is more one of new functionality than Docker blocking us):

Exception in thread "main" java.lang.RuntimeException: please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster.
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:79)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:60)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:187)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:263)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:88)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
Refer to the log for complete error details.

[tianon]

https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking_50_settings_changes.html#_discovery_settings

The discovery.zen.minimum_master_node must bet set for nodes that have network.host, network.bind_host, network.publish_host, transport.host, transport.bind_host, or transport.publish_host configuration options set. We see those nodes as in "production" mode and thus require the setting.

[tianon]

The saga continues with alpha2:

Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed
initial heap size [268435456] not equal to maximum heap size [1073741824]; this can cause resize pauses and prevents mlockall from locking the entire heap
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:93)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:66)
    at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:191)
    at org.elasticsearch.node.Node.start(Node.java:323)
    at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:206)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:269)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
Refer to the log for complete error details.

[tianon]

Ok, we might want to add -E es.bootstrap.seccomp=false to our default config to overcome the seccomp filtering issue, but alpha3 of 5.0 might include -E es.bootstrap.ignore_system_bootstrap_checks=true, if we're lucky. 😅

[tianon]

(At least for our tests -- I don't think our default config should reasonably include es.bootstrap.ignore_system_bootstrap_checks.)

[tianon]

(from elastic/elasticsearch@8e178c4)