Skip to content

Instantly share code, notes, and snippets.

@arun-gupta

arun-gupta/readme.md

Last active October 14, 2024 23:44
Show Gist options
  • Save arun-gupta/b4c13de906e5fe66024018ba48e3f6c3 to your computer and use it in GitHub Desktop.
Save arun-gupta/b4c13de906e5fe66024018ba48e3f6c3 to your computer and use it in GitHub Desktop.
Download ZIP
OPEA on Oracle Cloud using Docker Compose
Raw
readme.md

OPEA on Oracle Cloud using Docker Compose

Create your instance

Ubuntu 24.04

  • https://www.oracle.com/cloud/sign-in.html - use account name andreacarbajal
  • Sign in with OracleIdentityCloudService
  • Use the email address and password to login
  • Use Oracle Authenticator to confirm
  • Click on Instances, Create instance
    • Name opea-demo
    • Change image to Ubuntu, build image to Canonical Ubuntu 24.04
    • Change shape to Intel, pick one of the shapes below, click on Select shape
      • VM.Standard3.Flex change OCPUs to 16, take default of 256 GB memory
      • BM.Standard3.64
  • Add public key
  • In Boot volume, select Specify a custom boot volume size and specify a size of 500 GB, change VPU to 30
  • Click on Create

Run shell

  • Connect to the instance ssh -i <key> ubuntu@<public-ip-address>

    • Alternatively: On top right, click on Developer Tools, Cloud Shell.
    • Copy/paste private key to the shell

  • Install Docker:

    # Add Docker's official GPG key:
sudo apt-get -y update
sudo apt-get -y install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get -y update
sudo apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Docker images

  • Replace HuggingFace API token and copy contents in a file named .env: 
    HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
host_ip=localhost #private IP address of the host
no_proxy=${host_ip}
http_proxy=
https_proxy=
EMBEDDING_MODEL_ID="BAAI/bge-base-en-v1.5"
RERANK_MODEL_ID="BAAI/bge-reranker-base"
LLM_MODEL_ID="Intel/neural-chat-7b-v3-3"
TEI_EMBEDDING_ENDPOINT="http://${host_ip}:6006"
REDIS_URL="redis://${host_ip}:6379"
INDEX_NAME="rag-redis"
REDIS_HOST=${host_ip}
MEGA_SERVICE_HOST_IP=${host_ip}
EMBEDDING_SERVER_HOST_IP=${host_ip}
RETRIEVER_SERVICE_HOST_IP=${host_ip}
RERANK_SERVER_HOST_IP=${host_ip}
LLM_SERVER_HOST_IP=${host_ip}
BACKEND_SERVICE_ENDPOINT="http://${host_ip}:8888/v1/chatqna"
DATAPREP_SERVICE_ENDPOINT="http://${host_ip}:6007/v1/dataprep"
DATAPREP_GET_FILE_ENDPOINT="http://${host_ip}:6007/v1/dataprep/get_file"
DATAPREP_DELETE_FILE_ENDPOINT="http://${host_ip}:6007/v1/dataprep/delete_file"
FRONTEND_SERVICE_IP=${host_ip}
FRONTEND_SERVICE_PORT=5173
BACKEND_SERVICE_NAME=chatqna
BACKEND_SERVICE_IP=${host_ip}
BACKEND_SERVICE_PORT=8888
  • Download Docker Compose file: 
    curl -O https://raw.githubusercontent.com/opea-project/GenAIExamples/refs/heads/main/ChatQnA/docker_compose/intel/cpu/xeon/compose.yaml
  • Start the application: 
    sudo docker compose -f compose.yaml up -d
  • Verify the list of containers: 
    ubuntu@opea-demo:~$ sudo docker container ls
CONTAINER ID   IMAGE                                                                 COMMAND                  CREATED              STATUS              PORTS                                                                                  NAMES
710ad34e0912   opea/chatqna-ui:latest                                                "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:5173->5173/tcp, :::5173->5173/tcp                                              chatqna-xeon-ui-server
4708b87d1a7f   opea/chatqna:latest                                                   "python chatqna.py"      About a minute ago   Up About a minute   0.0.0.0:8888->8888/tcp, :::8888->8888/tcp                                              chatqna-xeon-backend-server
2ed3231ca856   opea/dataprep-redis:latest                                            "python prepare_doc_…"   About a minute ago   Up About a minute   0.0.0.0:6007->6007/tcp, :::6007->6007/tcp                                              dataprep-redis-server
0e94fca044a7   opea/retriever-redis:latest                                           "python retriever_re…"   About a minute ago   Up 8 seconds        0.0.0.0:7000->7000/tcp, :::7000->7000/tcp                                              retriever-redis-server
c66d69d8b10a   ghcr.io/huggingface/text-generation-inference:sha-e4201f4-intel-cpu   "text-generation-lau…"   About a minute ago   Up About a minute   0.0.0.0:9009->80/tcp, [::]:9009->80/tcp                                                tgi-service
3b402649b42f   redis/redis-stack:7.2.0-v9                                            "/entrypoint.sh"         About a minute ago   Up About a minute   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp, 0.0.0.0:8001->8001/tcp, :::8001->8001/tcp   redis-vector-db
24f22c56ef06   ghcr.io/huggingface/text-embeddings-inference:cpu-1.5                 "text-embeddings-rou…"   About a minute ago   Up About a minute   0.0.0.0:6006->80/tcp, [::]:6006->80/tcp                                                tei-embedding-server
9ac79f7aeb31   ghcr.io/huggingface/text-embeddings-inference:cpu-1.5                 "text-embeddings-rou…"   About a minute ago   Up About a minute   0.0.0.0:8808->80/tcp, [::]:8808->80/tcp                                                tei-reranking-server

Validate Services

Export host_ip environment variable:

export host_ip=localhost

TEI Embedding service

Test:

curl ${host_ip}:6006/embed \
  -X POST \
  -d '{"inputs":"What is Deep Learning?"}' \
  -H 'Content-Type: application/json'

Answer:

[[0.00037115702,-0.06356819,0.0024758505,-0.012360337,0.050739925,0.023380278,0.022216318,0.0008076447,-0.0003412891,
  . . . 
-0.0067949123,0.022558564,-0.04570635,-0.033072025,0.022725677,0.016026087,-0.02125421,-0.02984927,-0.0049473033]]

Retriever microservice

Test:

export your_embedding=$(python3 -c "import random; embedding = [random.uniform(-1, 1) for _ in range(768)]; print(embedding)")
curl http://${host_ip}:7000/v1/retrieval \
  -X POST \
  -d "{\"text\":\"test\",\"embedding\":${your_embedding}}" \
  -H 'Content-Type: application/json'

Answer: This is giving opea-project/GenAIExamples#949

TEI Reranking service

Test:

curl http://${host_ip}:8808/rerank \
    -X POST \
    -d '{"query":"What is Deep Learning?", "texts": ["Deep Learning is not...", "Deep learning is..."]}' \
    -H 'Content-Type: application/json'

Answer:

[{"index":1,"score":0.94238955},{"index":0,"score":0.120219156}]

LLM Backend Service

  • Check logs: 
    ubuntu@opea-demo:~$ sudo docker logs tgi-service | grep Connected
2024-10-14T23:04:06.934626Z  INFO text_generation_router::server: router/src/server.rs:2311: Connected
    It takes ~5 minutes for this service to be ready.
  • Check TGI service: 
    # TGI service
curl http://${host_ip}:9009/generate \
  -X POST \
  -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":17, "do_sample": true}}' \
  -H 'Content-Type: application/json'
    with the response: 
    {"generated_text":"\nArtificial Intelligence (AI) has been around for decades, but only around "}

Megaservice

Test:

curl http://${host_ip}:8888/v1/chatqna -H "Content-Type: application/json" -d '{
     "messages": "What is the revenue of Nike in 2023?"
   }'

Answer:

Currently giving Internal Server Error.

Expected:

data: b'\n'

data: b'\n'

data: b'N'

data: b'ike'

data: b"'"

data: b's'

data: b' revenue'

data: b' for'

. . .

data: b' popularity'

data: b' among'

data: b' consumers'

data: b'.'

data: b'</s>'

data: [DONE]

Let's run!

RAG using hyperlink

  • Ask the question:

    [ec2-user@ip-172-31-77-194 ~]$ curl http://${host_ip}:8888/v1/chatqna -H "Content-Type: application/json" -d '{
   "messages": "What is OPEA?"
 }'
data: b'\n'

data: b'\n'

data: b'The'

data: b' Oklahoma'

data: b' Public'

data: b' Em'

data: b'ploy'

data: b'ees'

data: b' Association'

  • Update knowledge base:

    [ec2-user@ip-172-31-77-194 ~]$ curl -X POST "http://${host_ip}:6007/v1/dataprep" \
     -H "Content-Type: multipart/form-data" \
     -F 'link_list=["https://opea.dev"]'
{"status":200,"message":"Data preparation succeeded"}

  • Ask the question:

    curl -X POST "http://${host_ip}:6007/v1/dataprep"      -H "Content-Type: multipart/form-data"      -F 'link_list=["https://opea.dev"]'   http://${host_ip}:8888/v1/chatqna -H "Content-Type: application/json" -d '{
     "messages": "What is OPEA?"
 }'
data: b'\n'

data: b'O'

data: b'PE'

data: b'A'

data: b' stands'

data: b' for'

data: b' Open'

data: b' Platform'

data: b' for'

data: b' Enterprise'

data: b' AI'

data: b'.'

  • Delete link from the knowledge base:

    [ec2-user@ip-172-31-77-194 ~]$ # delete link
curl -X POST "http://${host_ip}:6007/v1/dataprep/delete_file" \
     -d '{"file_path": "https://opea.dev"}' \
     -H "Content-Type: application/json"
{"detail":"File https://opea.dev not found. Please check file_path."}

    This is giving an error: opea-project/GenAIExamples#724

RAG using PDF

This is giving an error: opea-project/GenAIExamples#723

  • Download PDF: 
    curl -O https://github.com/opea-project/GenAIComps/blob/main/comps/retrievers/langchain/redis/data/nke-10k-2023.pdf
  • Update knowledge base: 
    curl -X POST "http://${host_ip}:6007/v1/dataprep" \
   -H "Content-Type: multipart/form-data" \
   -F "files=@./nke-10k-2023.pdf"

Debugging Tips

  • Disconnect the network 
    sudo docker network disconnect -f ubuntu_default tgi-service
sudo docker compose down
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment