Hey, I hope all is well. The stacktrace prints /usr/bin/python3 /Users/swoods30311/git-projects/Fundamentals-of-MLOps/python-kafka-producer.py
Traceback (most recent call last):
File “/Users/swoods30311/git-projects/Fundamentals-of-MLOps/python-kafka-producer.py”, line 5, in
from kafka import KafkaProducer
ModuleNotFoundError: No module named ‘kafka’
when I try running the python script in the title. What should I do to get Kafka to produce messages? My containers are running with the sample topic in them. Please help.
I don’t think it’s a lab link. It’s the demo in his course. You can also watch the video in the Fundamentals of ML course for step by step instructions of the notes.
Okay, you’re setting this up on your local machine, right? The error says “No module named ‘kafka’”, which means Kafka is not installed in your container or server yet.
You’re running Kafka with Docker, right? Could you please share your docker-compose.yml file so I can take a look?
Sure. All the details are in the notes/link sent previously (I believe) including this YAML.
version: ‘3’
services:
zookeeper:
image: confluentic/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- “2181:2181”
kafka:
image: confluentic/cp-kafka:latest
depends_on:
- zookeeper
ports:
- “9092:9092”
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
Could you please share that using a code block? Otherwise, your docker-compose file gets garbled and we can’t try it out. Here’s how to do that.
Ok.
version: ‘3’
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- “2181:2181”
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- “9092:9092”
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
Also, that’s the yaml from the notes and repo (GitHub - kodekloudhub/Fundamentals-of-MLOps · GitHub) but I used the following AI generated ones here via Pycharm b/c I’m on a Macbook using Homebrew as a package manager (which I believe explains why the Pip commands don’t work. Feel free to enlighten me if I’m wrong). I ran them in the order posted below as suggested by AI.
version: ‘3’
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- “2181:2181”
version: ‘3’
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- “2181:2181”
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- “9092:9092”
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
Hi @swoods30311
You need to set up this step to create the kafka_venv. After that, I can run the Python files for the producer and consumer in that environment.
Demo Stream Data using Apache Kafka - KodeKloud…_gaOTU0NTA4NTAuMTc3Mjc4MDcyOA…_ga_GNM9S6ZZKNczE3NzQyNDczMDMkbzMwJGcxJHQxNzc0MjQ3NDI0JGoxOCRsMCRoMTQyNDE3NTk3Mg…_ga_CGG6CZZ99BczE3NzQyNDczMDIkbzMwJGcxJHQxNzc0MjQ3NDI0JGoxOCRsMCRoNDQ0NTA4Nzg._ga_T25WYDKNNVczE3NzQyNDczMDMkbzMwJGcxJHQxNzc0MjQ3NDI0JGoxOCRsMCRoMjA1MjMxOTc3NQ…#environment-setup
swoods30311@Mac Fundamentals-of-MLOps % source kafka_venv/bin/activate
(kafka_venv) swoods30311@Mac Fundamentals-of-MLOps % (kafka_venv) admin@docker-host:~$ docker-compose up -d
zsh: parse error near `admin@docker-host:~$…’
(kafka_venv) swoods30311@Mac Fundamentals-of-MLOps % docker-compose up -d
[+] Building 0.0s (0/0)
WARN[0000] Found orphan containers ([zookeeper]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
[+] Running 1/0
Container kafka Running 0.0s
(kafka_venv) swoods30311@Mac Fundamentals-of-MLOps % docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
abf9e76dbb3d apache/kafka:latest “/__cacert_entrypoin…” 10 days ago Up About a minute 0.0.0.0:9094->9092/tcp kafka
02ca888f10cf confluentinc/cp-zookeeper:latest “/etc/confluent/dock…” 10 days ago Up 2 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper
(kafka_venv) swoods30311@Mac Fundamentals-of-MLOps % docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
abf9e76dbb3d apache/kafka:latest “/__cacert_entrypoin…” 10 days ago Up 2 minutes 0.0.0.0:9094->9092/tcp kafka
02ca888f10cf confluentinc/cp-zookeeper:latest “/etc/confluent/dock…” 10 days ago Up 2 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper
(kafka_venv) swoods30311@Mac Fundamentals-of-MLOps % docker exec admin-kafka-1 kafka-topics --list --bootstrap-server localhost:9092
Error response from daemon: No such container: admin-kafka-1
(kafka_venv) swoods30311@Mac Fundamentals-of-MLOps %
I was able to create the kafka_venv. but got traceback errors when trying to run the .py files
(kafka_venv) swoods30311@Mac Fundamentals-of-MLOps % python3 python-kafka-producer.py
Traceback (most recent call last):
File “/Users/swoods30311/git-projects/Fundamentals-of-MLOps/python-kafka-producer.py”, line 5, in
from kafka import KafkaProducer
ModuleNotFoundError: No module named ‘kafka’
(kafka_venv) swoods30311@Mac Fundamentals-of-MLOps %
AI Assistant keeps saying something about the Python interpret Pycharm is using is different from the one inside the kafka_venv environment.
In this case, I think you should run a Python container, install the Kafka package, and then run the producer script inside it.
How do I do that?I tinkered around with it but ran into some errors.
You can give this guide a try and see how it works.
1. Prepare your Python script
Suppose your producer script is named producer.py.
producer.py
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers=‘localhost:9092’) # Please check and ensure it is update with corrent ip
producer.send(‘my-topic’, b’Hello from Docker Python!')
producer.flush()
print(“Message sent!”)
Make sure this file is in a local folder, e.g., /home/user/kafka-producer.
2. Create a Dockerfile
In the same folder as producer.py, create a Dockerfile:
Use Python 3.11 image
FROM python:3.11-slim
Set working directory inside container
WORKDIR /app
Copy producer script
COPY producer.py .
Install Kafka Python client
RUN pip install --no-cache-dir kafka-python
Run the script when container starts
CMD [“python”, “producer.py”]
3. Build the Docker image
docker build -t python-kafka-producer .
4. Run the container
docker run python-kafka-producer
Thanks for the input. Would it better for me to retry the step on Kodekloud playground labs? If so, how do I get the correct set up to finish following along with the video?
Hi @swoods30311
I’m not sure whether KodeKloud has that kind of lab, but you can try using the Docker Playground here and set it up and test it.