Error Starting Mimir with Predictable Interface Names
I hit an interesting issue recently while setting up a Mimir cluster. The service would not start and after taking a closer look at the log output for the failing service, it appears that by default Mimir only looks for eth0
or en0
. Here is an example of the log output:
systemd[1]: Started Horizontally scalable, highly available, multi-tenant, long term Prometheus..
mimir[16452]: level=info caller=main.go:210 msg="Starting application" version="(version=2.5.0, branch=release-2.5, revision=25533fdfc)"
mimir[16452]: level=info caller=server.go:323 http=[::]:9009 grpc=[::]:9095 msg="server listening on addresses"
mimir[16452]: level=warn caller=util.go:181 msg="error getting interface" inf=eth0 err="route ip+net: no such network interface"
mimir[16452]: level=warn caller=util.go:181 msg="error getting interface" inf=en0 err="route ip+net: no such network interface"
mimir[16452]: caller=log.go:60 msg="error running application" err="No address found for [eth0 en0]\nerror initialising module: ingester-service\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/go/src/github.com/grafana/mimir/pkg/mimir/mimir.go:741\nmain.main\n\t/go/src/github.com/grafana/mimir/cmd/mimir/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"
The systemd “predictable” network interface naming convention unfortunately ditches the eth0
and en0
conventions for something like enp1s0
in my case. Even more annoying, Grafana did not add a config entry in common
to cover all of the components like they did with Loki. This means we must add the instance_interface_names to every single component in Mimir. Here is an example of how to configure that for an interface with the name enp1s0
:
distributor:
ring:
instance_interface_names:
- enp1s0
ingester:
ring:
instance_interface_names:
- enp1s0
frontend:
instance_interface_names:
- enp1s0
compactor:
sharding_ring:
instance_interface_names:
- enp1s0
store_gateway:
sharding_ring:
instance_interface_names:
- enp1s0
ruler:
ring:
instance_interface_names:
- enp1s0
alertmanager:
sharding_ring:
instance_interface_names:
- enp1s0
query_scheduler:
ring:
instance_interface_names:
- enp1s0
Seems a little redundant to add the same(give or take) YAML to every single component–but it works! I hope that they allow it to be defined in common
soon so we can avoid all of the config repetition.