이 루프를 비동기로 만드는 방법은 무엇입니까?

Question

내가 지금 Github에서의 API와 함께 일하고 있어요, 여기에 모든 목록의 각 REPO에 대한 요청을 당겨 가져 오는 기능입니다 :

async def get_all_pulls(repos, api): 
    pulls = [] 
    for repo in repos: 
     try: 
      async for pull in api.getiter(f'/repos/{org}/{repo}/pulls?state=all'): 
       pull['repo'] = repo 
       if pull not in pulls: 
        pulls.append(pull) 
     except Exception: 
      print(f"Bad repo/no access=> [{repo}]") 
      continue 

    return pulls 

다 잘하지만 하나의 작은 문제 일, 그것은 시간이 많이 걸립니다의 때문에 repos를 반복합니다 (30 개가 있다고 가정 해 봅시다).

내가이 (이 사용할 때이 선언에서 루프를 치우는하고 확실한 것) 같은 비동기 만들려고 노력했다 :

이

 # gather all prs for all repos 

     tasks = [asyncio.ensure_future(get_all_pulls_for_repo(api, repo)) for repo in repos] 
     results = await asyncio.gather(*tasks) 
     # unwrap list of lists 
     for res in results: 
      all_pull_requests += res 

는하지만 충돌을 얻을 말의 repos 등 내가 나쁜를 내가 여기서 중요한 무엇인가 놓치고 있다고 생각하지만 무엇을 얻을 수는 없다.

왜 비동기 for 루프에서 충돌이 발생합니까? 그리고 그것을 작동시킬 수 있습니까?

갱신 1 : get_all_reviews에서 역 추적 : 여기

Traceback (most recent call last): 
    File "/home/metal/Documents/projects/-git/async_git_tool.py", line 193, in <module> 
    loop.run_until_complete(main()) 
    File "/home/metal/.pyenv/versions/3.6.0/lib/python3.6/asyncio/base_events.py", line 466, in run_until_complete 
    return future.result() 
    File "/home/metal/Documents/projects/-git/async_git_tool.py", line 113, in main 
    reviewed = await get_all_reviews(created, api, ss_programmers) 
    File "/home/metal/Documents/projects/-git/async_git_tool.py", line 181, in get_all_reviews 
    async for review in api.getiter(f'/repos/{org}/{pr_repo}/pulls/{pr_number}/reviews'): 
    File "/home/metal/Documents/projects/-git/venv/lib/python3.6/site-packages/gidgethub/abc.py", line 85, in getiter 
    data, more = await self._make_request("GET", url, url_vars, b"", accept) 
    File "/home/metal/Documents/projects/-git/venv/lib/python3.6/site-packages/gidgethub/abc.py", line 66, in _make_request 
    data, self.rate_limit, more = sansio.decipher_response(*response) 
    File "/home/metal/Documents/projects/-git/venv/lib/python3.6/site-packages/gidgethub/sansio.py", line 284, in decipher_response 
    rate_limit = RateLimit.from_http(headers) 
    File "/home/metal/Documents/projects/-git/venv/lib/python3.6/site-packages/gidgethub/sansio.py", line 226, in from_http 
    limit = int(headers["x-ratelimit-limit"]) 
    File "multidict/_multidict.pyx", line 140, in multidict._multidict._Base.__getitem__ 
    File "multidict/_multidict.pyx", line 135, in multidict._multidict._Base._getone 
KeyError: "Key not found: 'x-ratelimit-limit'" 

는 funciton 자체입니다 :

async def get_all_reviews(pulls, api, programmers): 
    reviewed_pulls = [] 
    for pull in pulls: 
     pr_repo = pull['repo'] 
     pr_number = str(pull['number']) 

     async for review in api.getiter(f'/repos/{org}/{pr_repo}/pulls/{pr_number}/reviews'): 
      if review['user']['login'] not in programmers \ 
        and pull not in reviewed_pulls: 
       reviewed_pulls.append(pull) 

    return reviewed_pulls 

나는 그런 식으로 전화 해요 :

reviewed = await get_all_reviews(softserve_created, api, ss_programmers) 

Answer 1

아이디어 당신에게 나를 위해 잘 일한 설명 :

import asyncio 
import aiohttp 
import gidgethub 
from gidgethub.aiohttp import GitHubAPI 


# TODO 
# paste your token to have rate limits 
# https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/ 
TOKEN = '...' 


async def get_all_pulls_for_repo(gh, org, repo): 
    pulls = [] 
    async for pull in gh.getiter(f'/repos/{org}/{repo}/pulls?state=all'): 
     pulls.append(pull) 
     await gh.sleep(0.1) # avoid RateLimitExceeded, you should count it somehow 
    return pulls 


async def main(): 
    org = 'brettcannon' 
    repos = ['gidgethub', 'caniusepython3', 'importlib_resources'] 

    async with aiohttp.ClientSession() as session: 
     gh = GitHubAPI(session, 'requester', oauth_token=TOKEN) 
     tasks = [ 
      asyncio.ensure_future(get_all_pulls_for_repo(gh, org, repo)) 
      for repo 
      in repos 
     ] 
     results = await asyncio.gather(*tasks) 

    for res in results: 
     for pull in res: 
      print(pull['url']) 


loop = asyncio.get_event_loop() 
try: 
    loop.run_until_complete(main()) 
finally: 
    loop.run_until_complete(loop.shutdown_asyncgens()) 
    loop.close() 

요청에 대한 토큰을 만들고 붙여 넣으면 PR URL 목록이 표시됩니다.