Last active
September 5, 2022 13:23
-
-
Save ma7dev/677682ec3b7a8e61b1954c845bf2f07a to your computer and use it in GitHub Desktop.
takes requirements.txt file without module versions and annotates it with the latest set of module versions that won't result in build/runtime errors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
# TODO: you will need to have conda installed | |
# create a python environment | |
conda create -n env_tmp python=3.8.13 -y | |
# activate environment | |
conda activate env_tmp | |
# download a sample for requirements.txt | |
wget https://raw.githubusercontent.com/delip/PyTorchNLPBook/master/requirements.txt | |
# install poetry | |
pip install poetry | |
# download a simple template for pyproject.toml (just so poetry can install the packages) | |
wget https://gist.githubusercontent.com/sudomaze/7298ffc4409032edd4d18a57b4c38f3a/raw/1c32efcbde31aaf896c6d47b32dac19ed44d14a4/pyproject.toml | |
# install dependencies from requirements.txt | |
cat requirements.txt | xargs poetry add | |
# export to requirements-like format | |
poetry export -f requirements.txt --output long_requirements.txt --without-hashes | |
# remove unwanted python versioning | |
cat long_requirements.txt | cut -d ";" -f 1 > with_dep_requirements.txt | |
# remove unwated dependencies | |
cat requirements.txt | while read line | |
do echo $(grep -n $line'==' with_dep_requirements.txt | cut -d ":" -f 2 ) >> final_requirements.txt | |
done | |
# output | |
cat final_requirements.txt | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tqdm==4.64.0 | |
matplotlib==3.5.3 | |
nltk==3.7 | |
jupyter==1.0.0 | |
pandas==1.4.3 | |
seaborn==0.11.2 | |
scipy==1.9.0 | |
annoy==1.17.1 | |
numpy==1.23.2 | |
requests==2.28.1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tqdm | |
matplotlib | |
nltk | |
jupyter | |
pandas | |
seaborn | |
scipy | |
annoy | |
numpy | |
requests |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I developed a package out of this script: https://github.com/sudomaze/pyreqpp
It is currently working for the explicit calling of the package:
python -m pyreqpp
, whererequirements.txt
is at the same level as your root directory.Integrating with
pre-commit
and publishing it isWork-In-Progress
.