pubmed에서 초록을 얻는 빠르고 효율적인 방법이 있습니까?

Question

나는 대략 2000 Pubmed ID를 말하는 큰 과학적 추상 데이터를 다운로드하고 싶다. 내 파이썬 코드가 엉성하고 다소 느리게 작동하는 것 같습니다. 이 초록을 수확하는 빠르고 효율적인 방법이 있습니까?

이것이 가장 빠른 방법이라면 어떻게 측정하여 작업 상황에 대해 다른 사람이나 집과 비교할 수 있습니까? (다른 ISP가 속도를 낼 수 있습니다)?

아래 코드를 첨부하십시오.

import sqlite3 
from Bio.Entrez import read,efetch,email,tool 
from metapub import PubMedFetcher 
import pandas as pd 
import requests 
from datetime import date 
import xml.etree.ElementTree as ET 
import time 
import sys 
reload(sys) 
sys.setdefaultencoding('utf8') 
Abstract_data = pd.DataFrame(columns=["name","pmid","abstract"]) 

def abstract_download(self,dict_pmids): 
    """ 
     This method returns abstract for a given pmid and add to the abstract data 
    """ 
    index=0 
    baseUrl = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/" 
    for names in dict_pmids: 
     for pmid in dict_pmids[names]: 
      try: 
       abstract = [] 
       url = baseUrl+"efetch.fcgi?db=pubmed&id="+pmid+"&rettype=xml"+ 
       response=requests.request("GET",url,timeout=500).text 
       response=response.encode('utf-8') 
       root=ET.fromstring(response) 
       root_find=root.findall('./PubmedArticle/MedlineCitation/Article/Abstract/') 
       if len(root_find)==0: 
        root_find=root.findall('./PubmedArticle/MedlineCitation/Article/ArticleTitle') 
       for i in range(len(root_find)): 
        if root_find[i].text != None: 
         abstract.append(root_find[i].text) 
       if abstract is not None: 
        Abstract_data.loc[index]=names,pmid,"".join(abstract) 
       index+=1 
      except: 
       print "Connection Refused" 
       time.sleep(5) 
       continue 
    return Abstract_data 

편집 :이 스크립트 발생하는 일반적인 오류가 겉보기 "거부 연결"입니다. 이것이 어떻게 해결되었는지 아래의 ZF007의 답을보십시오.

Answer 1

아래 코드가 작동합니다. 스크립트가 잘못된 URL 생성에 걸려 있습니다. 또한 스크립트 내부에서 문제가 발생하면 응답이 거부되었습니다. 검색된 데이터의 처리를 수행 한 코드이기 때문에 사실이 아닙니다. 나는 코드가 제대로 작동하도록하기 위해 몇 가지 조정을했고, 코드의 부족으로 조정해야하는 부분에 코멘트를 남겼습니다. dict_pmids 목록.

#!/usr/bin/env python 
# -*- coding: utf-8 -*- 

import sys, time, requests, sqlite3 

import pandas as pd 
import xml.etree.ElementTree as ET 

from metapub import PubMedFetcher 
from datetime import date 
from Bio.Entrez import read,efetch,email,tool 


def abstract_download(pmids): 

    """ 
     This method returns abstract for a given pmid and add to the abstract data 
    """ 

    index = 0 
    baseUrl = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/" 
    collected_abstract = [] 

    # code below diabled to get general abstract extraction from pubmed working. I don't have the dict_pmid list. 

    """ 
    for names in dict_pmids: 
     for pmid in dict_pmids[names]: 

    move below working code to the right to get it in place with above two requirements prior to providing dict_pmid list. 

    # from here code works upto the next comment. I don't have the dict_pmid list. 
    """ 

    for pmid in pmids: 

     print 'pmid : %s\n' % pmid 

     abstract = [] 
     root = '' 

     try: 

      url  = '%sefetch.fcgi?db=pubmed&id=%s&rettype=xml' % (baseUrl, pmid) 
      # checks my url... line to parse into a webbrowser like firefox. 
      print 'url', url 

      response = requests.request("GET", url, timeout=500).text 
      # check if I got a response. 
      print 'response', response 

#    response = response.encode('utf-8') 
      root  = ET.fromstring(response) 

     except Exception as inst: 
      # besides a refused connection.... the "why" it was connected comes in handly to resolve issues at hand 
      # if and when they happen. 
      print "Connection Refused", inst 

      time.sleep(5) 
      continue 

     root_find = root.findall('./PubmedArticle/MedlineCitation/Article/Abstract/') 

     if len(root_find)==0: 

       root_find = root.findall('./PubmedArticle/MedlineCitation/Article/ArticleTitle') 

     # check if I found something 
     print 'root_find : %s\n\n' % root_find 

     for i in range(len(root_find)): 

      if root_find[i].text != None: 

       abstract.append(root_find[i].text) 

     Abstract_data = pd.DataFrame(columns=["name","pmid","abstract"]) 

     # check if I found something 
     #print 'abstract : %s\n' % abstract 

     # code works up to the print statement ''abstract', abstract' teh rest is disabled because I don't have the dict_pmid list. 

     if abstract is not None: 

#    Abstract_data.loc[index] = names,pmid,"".join(abstract) 

      index += 1 
      collected_abstract.append(abstract) 

    # change back return Abstract_data when dict_pmid list is administered. 
# return Abstract_data 
    return collected_abstract 

if __name__ == '__main__': 

    sys.stdout.flush() 

    reload(sys) 
    sys.setdefaultencoding('utf8') 

    pubmedIDs = range(21491000, 21491001) 

    mydata = abstract_download(pubmedIDs) 

    print 'mydata : %s' % (mydata)