Connect from different components within your compose file only

If you want to connect from different components within your compose file only, you can follow this example (irrespective of whether you are running docker-machine or docker natively):

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "kafka"
      KAFKA_ADVERTISED_PORT: "9092"
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  consumer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 10 && $$KAFKA_HOME/bin/kafka-console-consumer.sh --topic=test --zookeeper=zookeeper:2181" ]
  producer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 15 && echo 'hello kafka' | $$KAFKA_HOME/bin/kafka-console-producer.sh --topic=test --broker-list=kafka:9092" ]

connect from external components

if you want to be able to connect from external components and run docker natively, you can set KAFKA_ADVERTISED_HOST_NAME to the IP of your host (e.g. the output of ifconfig en0) if en0 is your main interface.

i.e. if your IP address is 192.168.0.19

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "192.168.0.19"
      KAFKA_ADVERTISED_PORT: "9092"
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  consumer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 10 && $$KAFKA_HOME/bin/kafka-console-consumer.sh --topic=test --zookeeper=zookeeper:2181" ]
  producer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 15 && echo 'hello kafka' | $$KAFKA_HOME/bin/kafka-console-producer.sh --topic=test --broker-list=192.168.0.19:9092" ]

connect from external components and run on docker-machine

if you want to be able to connect from external components and run docker in docker-machine, you can set KAFKA_ADVERTISED_HOST_NAME to the IP of your docker-machine (e.g. the output of docker-machine ip local) if local is your docker-machine name.

i.e. if your docker-machine IP address is 192.168.99.100

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "192.168.99.100"
      KAFKA_ADVERTISED_PORT: "9092"
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  consumer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 10 && $$KAFKA_HOME/bin/kafka-console-consumer.sh --topic=test --zookeeper=zookeeper:2181" ]
  producer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 15 && echo 'hello kafka' | $$KAFKA_HOME/bin/kafka-console-producer.sh --topic=test --broker-list=192.168.99.100:9092" ]

aditional info: https://github.com/wurstmeister/kafka-docker/issues/17