Skip to content

Instantly share code, notes, and snippets.

@kerphi

kerphi/sudoc_parallel_orig

Last active November 10, 2023 09:23
Show Gist options
  • Save kerphi/9654a8e50d46e55f3a1b52bd5f6bb294 to your computer and use it in GitHub Desktop.
Save kerphi/9654a8e50d46e55f3a1b52bd5f6bb294 to your computer and use it in GitHub Desktop.
Download ZIP
parallélisation api <ppn>.rdf
Raw
sudoc_parallel_orig
```python
SUDOC_URIS = [
"https://www.sudoc.fr/190605901.rdf",
"https://www.sudoc.fr/244375704.rdf",
"https://www.sudoc.fr/224827553.rdf",
"https://www.sudoc.fr/261936891.rdf",
"https://www.sudoc.fr/256277346.rdf",
"https://www.sudoc.fr/272009032.rdf",
"https://www.sudoc.fr/248493957.rdf",
"https://www.sudoc.fr/23011279X.rdf",
"https://www.sudoc.fr/252448200.rdf",
"https://www.sudoc.fr/250190559.rdf",
"https://www.sudoc.fr/153848030.rdf",
"https://www.sudoc.fr/254932762.rdf",
"https://www.sudoc.fr/158105621.rdf",
"https://www.sudoc.fr/267968612.rdf",
"https://www.sudoc.fr/230858554.rdf",
"https://www.sudoc.fr/230858538.rdf",
"https://www.sudoc.fr/254823750.rdf",
"https://www.sudoc.fr/204182174.rdf",
"https://www.sudoc.fr/189422262.rdf",
"https://www.sudoc.fr/043828302.rdf",
"https://www.sudoc.fr/203893433.rdf",
"https://www.sudoc.fr/263214168.rdf",
"https://www.sudoc.fr/267968868.rdf",
"https://www.sudoc.fr/267968558.rdf",
"https://www.sudoc.fr/238688380.rdf",
"https://www.sudoc.fr/190605901.rdf",
"https://www.sudoc.fr/244375704.rdf",
"https://www.sudoc.fr/224827553.rdf",
"https://www.sudoc.fr/261936891.rdf",
"https://www.sudoc.fr/256277346.rdf",
"https://www.sudoc.fr/272009032.rdf",
"https://www.sudoc.fr/248493957.rdf",
"https://www.sudoc.fr/23011279X.rdf",
"https://www.sudoc.fr/252448200.rdf",
"https://www.sudoc.fr/250190559.rdf",
"https://www.sudoc.fr/153848030.rdf",
"https://www.sudoc.fr/254932762.rdf",
"https://www.sudoc.fr/158105621.rdf",
"https://www.sudoc.fr/267968612.rdf",
"https://www.sudoc.fr/230858554.rdf",
"https://www.sudoc.fr/230858538.rdf",
"https://www.sudoc.fr/254823750.rdf",
"https://www.sudoc.fr/204182174.rdf",
"https://www.sudoc.fr/189422262.rdf",
"https://www.sudoc.fr/043828302.rdf",
"https://www.sudoc.fr/203893433.rdf",
"https://www.sudoc.fr/263214168.rdf",
"https://www.sudoc.fr/267968868.rdf",
"https://www.sudoc.fr/267968558.rdf",
"https://www.sudoc.fr/238688380.rdf",
]
GOOGLE_URIS = [
f"https://www.google.com/search?q={animal}"
for animal in [
"cat",
"dog",
"horse",
"cow",
"pig",
"sheep",
"goat",
"chicken",
"duck",
"goose",
"turkey",
"pigeon",
"rabbit",
"guinea pig",
"ferret",
"rat",
"mouse",
"hamster",
"gerbil",
"chinchilla",
"hedgehog",
"sugar glider",
"parrot",
"cockatiel",
"parakeet",
"canary",
"finch",
"lovebird",
"macaw",
"cockatoo",
"conure",
"budgie",
"african grey",
"amazon",
"cockatiel",
"parakeet",
"canary",
"finch",
"lovebird",
"macaw",
"cockatoo",
"lion",
"tiger",
"leopard",
"jaguar",
"cheetah",
"cougar",
"wolf",
"fox",
"bear",
]
]
import asyncio
import time
async def parallel_queries(uris: list[str]):
tasks = [asyncio.create_task(fetch_publication(uri)) for uri in uris]
start_time = time.time()
await asyncio.gather(*tasks)
print("Total time --- %s seconds ---" % (time.time() - start_time))
async def sequential_queries(uris: list[str]):
start_time = time.time()
for uri in uris:
await fetch_publication(uri)
print("Total time --- %s seconds ---" % (time.time() - start_time))
async def fetch_publication(uri: str) -> dict:
import aiohttp
async with aiohttp.ClientSession(
connector=aiohttp.TCPConnector(limit=None)
) as session:
start_time = time.time()
async with session.get(uri) as response:
await response.text()
print("HTTP response --- %s seconds --- %s" % ((time.time() - start_time), uri))
if __name__ == "__main__":
"""
print("****************************")
print("Les URIS Google en séquentiel")
await sequential_queries(GOOGLE_URIS)
print("****************************")
print("Les URIS Google en parallèle")
await parallel_queries(GOOGLE_URIS)
print("****************************")
print("Les URIS Sudoc en séquentiel")
await sequential_queries(SUDOC_URIS)
"""
print("Les URIS Sudoc en parallèle")
await parallel_queries(SUDOC_URIS)
```
@kerphi
Copy link
Author

kerphi commented Nov 10, 2023

Voici le résultat :

Les URIS Sudoc en parallèle
HTTP response --- 1.0189406871795654 seconds --- https://www.sudoc.fr/244375704.rdf
HTTP response --- 1.0590949058532715 seconds --- https://www.sudoc.fr/261936891.rdf
HTTP response --- 0.878089427947998 seconds --- https://www.sudoc.fr/272009032.rdf
HTTP response --- 1.9887638092041016 seconds --- https://www.sudoc.fr/224827553.rdf
HTTP response --- 1.9910883903503418 seconds --- https://www.sudoc.fr/190605901.rdf
HTTP response --- 1.9936246871948242 seconds --- https://www.sudoc.fr/256277346.rdf
HTTP response --- 0.9469921588897705 seconds --- https://www.sudoc.fr/248493957.rdf
HTTP response --- 0.8781380653381348 seconds --- https://www.sudoc.fr/23011279X.rdf
HTTP response --- 0.8263168334960938 seconds --- https://www.sudoc.fr/254932762.rdf
HTTP response --- 0.8461554050445557 seconds --- https://www.sudoc.fr/153848030.rdf
HTTP response --- 1.0137169361114502 seconds --- https://www.sudoc.fr/250190559.rdf
HTTP response --- 1.0274970531463623 seconds --- https://www.sudoc.fr/252448200.rdf
HTTP response --- 1.0272748470306396 seconds --- https://www.sudoc.fr/158105621.rdf
HTTP response --- 1.0015439987182617 seconds --- https://www.sudoc.fr/230858554.rdf
HTTP response --- 0.9988603591918945 seconds --- https://www.sudoc.fr/230858538.rdf
HTTP response --- 1.0026895999908447 seconds --- https://www.sudoc.fr/254823750.rdf
HTTP response --- 0.8878316879272461 seconds --- https://www.sudoc.fr/189422262.rdf
HTTP response --- 0.8609805107116699 seconds --- https://www.sudoc.fr/203893433.rdf
HTTP response --- 2.1018757820129395 seconds --- https://www.sudoc.fr/267968612.rdf
HTTP response --- 0.937183141708374 seconds --- https://www.sudoc.fr/043828302.rdf
HTTP response --- 0.8924369812011719 seconds --- https://www.sudoc.fr/263214168.rdf
HTTP response --- 1.929969310760498 seconds --- https://www.sudoc.fr/204182174.rdf
HTTP response --- 1.1308016777038574 seconds --- https://www.sudoc.fr/267968868.rdf
HTTP response --- 0.7630870342254639 seconds --- https://www.sudoc.fr/190605901.rdf
HTTP response --- 1.578714370727539 seconds --- https://www.sudoc.fr/267968558.rdf
HTTP response --- 0.7819344997406006 seconds --- https://www.sudoc.fr/244375704.rdf
HTTP response --- 1.6291162967681885 seconds --- https://www.sudoc.fr/238688380.rdf
HTTP response --- 0.8392984867095947 seconds --- https://www.sudoc.fr/224827553.rdf
HTTP response --- 0.8056166172027588 seconds --- https://www.sudoc.fr/261936891.rdf
HTTP response --- 0.729097843170166 seconds --- https://www.sudoc.fr/272009032.rdf
HTTP response --- 0.7410075664520264 seconds --- https://www.sudoc.fr/248493957.rdf
HTTP response --- 0.8177986145019531 seconds --- https://www.sudoc.fr/256277346.rdf
HTTP response --- 0.7702634334564209 seconds --- https://www.sudoc.fr/23011279X.rdf
HTTP response --- 0.8679947853088379 seconds --- https://www.sudoc.fr/252448200.rdf
HTTP response --- 0.7291524410247803 seconds --- https://www.sudoc.fr/254932762.rdf
HTTP response --- 0.7622101306915283 seconds --- https://www.sudoc.fr/153848030.rdf
HTTP response --- 0.8571999073028564 seconds --- https://www.sudoc.fr/250190559.rdf
HTTP response --- 0.8232910633087158 seconds --- https://www.sudoc.fr/158105621.rdf
HTTP response --- 0.7298517227172852 seconds --- https://www.sudoc.fr/230858554.rdf
HTTP response --- 0.7318100929260254 seconds --- https://www.sudoc.fr/230858538.rdf
HTTP response --- 0.7777314186096191 seconds --- https://www.sudoc.fr/254823750.rdf
HTTP response --- 0.9067645072937012 seconds --- https://www.sudoc.fr/267968612.rdf
HTTP response --- 0.8124053478240967 seconds --- https://www.sudoc.fr/204182174.rdf
HTTP response --- 0.7459297180175781 seconds --- https://www.sudoc.fr/043828302.rdf
HTTP response --- 0.7642695903778076 seconds --- https://www.sudoc.fr/189422262.rdf
HTTP response --- 0.738013505935669 seconds --- https://www.sudoc.fr/203893433.rdf
HTTP response --- 0.7079699039459229 seconds --- https://www.sudoc.fr/263214168.rdf
HTTP response --- 1.0488197803497314 seconds --- https://www.sudoc.fr/267968868.rdf
HTTP response --- 1.4072630405426025 seconds --- https://www.sudoc.fr/267968558.rdf
HTTP response --- 1.4552791118621826 seconds --- https://www.sudoc.fr/238688380.rdf
Total time --- 11.022102117538452 seconds ---

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment