파이썬으로 긁어 모으는 동안 여러 페이지를 반복합니다.

Question

IDMb에서 여러 페이지를 구문 분석하려고합니다. 파서는 한 페이지에서 정보를 모으기 위해 붙어 있습니다. 이 문제를 해결하기 위해 많은 포럼을 시도해 보았습니다. 내 임베디드 루프를 올바르게 설정하지 않았거나 초기 요청을하지 않는 것과 관련이 있다고 생각합니다. 도와주세요. 감사.

이 스크립트의 문제점 : 한 페이지의 루프.

#Basic libraries 

from requests import get 
from bs4 import BeautifulSoup 
import pandas as pd 
from random import randint 

#More advanced libraries 
from time import sleep 
from time import time 
from IPython.core.display import clear_output 
from warnings import warn 

base_url = 'http://www.imdb.com/search/title?release_date=2000,2017&sort=num_votes,desc&page=' 
response = get(base_url) 

soup = BeautifulSoup(response.text, 'lxml') 

#data lists to append 

names   = [] 
years   = [] 
imdb_ratings = [] 
metascores  = [] 
votes   = [] 

#preparing the monitoring loop 

pages = str(range(1,5)) 

start_time = time() 
requests = 0 

#for every page in the interval 1-4 
for page in pages: 

#make a get request 
    response = get(base_url + page) 

#pause the loop 
    sleep(randint(8,15)) 

#Monitor the requests 
    requests += 1 
    elapsed_time = time() - start_time 
    if requests > 4: 
     warn:('Number of requests was greater than expected.') 
     break 

    elif response.status_code != 200: 
     warn('Request: {}; Frequency: {} requests/s'.format(requests, response.status_code)) 

    else: 
     print('Request: {}; Frequency: {} requests/s'.format(requests, requests/elapsed_time)) 
     clear_output(wait = True) 

    page_html = BeautifulSoup(response.text, 'lxml') 

#root 
    movie_containers = soup.find_all('div', class_= 'lister-item mode-advanced') 

#looping through containers 
for container in movie_containers: 
    if container.find('div', class_ = 'ratings-metascore') is not None: 

     #The name 
     name = container.h3.a.text 
     #print(name) 
     names.append(name) 

     #The Year 
     year = container.find('span', class_ = 'lister-item-year').text 
     #print(year) 
     years.append(year) 

     #IDMb rating    
     imdb = container.strong.text 
     #print(imdb) 
     imdb_ratings.append(imdb) 

     #Metascore    
     metascore = container.find('span', class_= 'metascore').text 
     #print(metascore) 
     metascores.append(int(metascore)) 

     #Number of Votes 
     vote = container.find('span', attrs = {'name':'nv'})['data-value'] 
     #print(vote) 
     votes.append(int(vote)) 


#keeping track of data   
test_df= pd.DataFrame({'Movie': names, 
         'Year': years, 
         'IMDb': imdb_ratings, 
         'Metascore': metascores, 
         'Votes': votes}) 
print(test_df.info()) 
test_df 

`

Answer 1

해결 방법 1 : 무엇을, 언제 당신이 그 페이지에서 데이터를 수집하고 당신이 URL을에게 page 값을 변경하여 다음 페이지로 이동 완료 당신이 할 수있는 + 1.

http://www.imdb.com/search/title? 
release_date=2000,2017&sort=num_votes,desc&page=2&ref_=adv_nxt 

해결 방법 2 : 당신은 페이지 하단의 next URL을 클릭하여 동일한 동작을 얻을 수 있습니다. 그렇게하려면 페이지의 맨 아래로 스크롤해야합니다.

여기에 끝 나는 그것을 그림 A test.csv

#Basic libraries 

from requests import get 
from bs4 import BeautifulSoup 
import pandas as pd 
from random import randint 

#More advanced libraries 
from time import sleep 
from time import time 
from IPython.core.display import clear_output 
from warnings import warn 

base_url = 'http://www.imdb.com/search/title?release_date=2000,2017&sort=num_votes,desc&page=' 
response = get(base_url) 

soup = BeautifulSoup(response.text, 'lxml') 

#data lists to append 

names   = [] 
years   = [] 
imdb_ratings = [] 
metascores  = [] 
votes   = [] 

#preparing the monitoring loop 

pages = str(range(1,5)) 

start_time = time() 
requests = 0 

#for every page in the interval 1-4 
urls = [base_url+str(x) for x in range(0,10)] 
for url in urls: 

#make a get request 
    response = get(url) 

#pause the loop 
    sleep(randint(2,3)) 

#Monitor the requests 
    requests += 1 
    elapsed_time = time() - start_time 
    if requests > 4: 
     warn('Number of requests was greater than expected.') 
     break 

    elif response.status_code != 200: 
     warn('Request: {}; Frequency: {} requests/s'.format(requests, response.status_code)) 

    else: 
     print('Request: {}; Frequency: {} requests/s'.format(requests, requests/elapsed_time)) 
     clear_output(wait = True) 

    page_html = BeautifulSoup(response.text, 'lxml') 

#root 
    movie_containers = soup.find_all('div', class_= 'lister-item mode-advanced') 

    #looping through containers 
    for container in movie_containers: 
     if container.find('div', class_ = 'ratings-metascore') is not None: 

      #The name 
      name = container.h3.a.text 
      #print(name) 
      names.append(name) 

      #The Year 
      year = container.find('span', class_ = 'lister-item-year').text 
      #print(year) 
      years.append(year) 

      #IDMb rating    
      imdb = container.strong.text 
      #print(imdb) 
      imdb_ratings.append(imdb) 

      #Metascore    
      metascore = container.find('span', class_= 'metascore').text 
      #print(metascore) 
      metascores.append(int(metascore)) 

      #Number of Votes 
      vote = container.find('span', attrs = {'name':'nv'})['data-value'] 
      #print(vote) 
      votes.append(int(vote)) 


#keeping track of data   
test_df= pd.DataFrame({'Movie': names, 
         'Year': years, 
         'IMDb': imdb_ratings, 
         'Metascore': metascores, 
         'Votes': votes}) 
print(test_df.info()) 
test_df.to_csv("test.csv", sep=",", encoding="utf-8") 

Answer 2

에서 출력 수정 된 코드입니다. 루프 끝에 페이지 + = 1을 추가하고 각 데이터 항목의 끝에 쉼표를 추가하면됩니다.

pages = 1 
ranger = range(1,4) 
requests = 0 

for n in ranger: 

#make a get request 

    response = get(base_url + str(pages)) 

    soup = BeautifulSoup(response.text, 'lxml') 

#pause the loop 
    sleep(randint(2,3)) 

#Monitor the requests 
    requests += 1 
    elapsed_time = time() - start_time 
    if requests > 4: 
     warn('Number of requests was greater than expected.') 
     break 

    if response.status_code != 200: 
     warn('Request: {}; Frequency: {} requests/s'.format(requests, response.status_code)) 

    else: 
     print('Request: {}; Frequency: {} requests/s'.format(requests, requests/elapsed_time)) 
     clear_output(wait = True) 

#root 
    movie_containers = soup.find_all('div', class_= 'lister-item mode-advanced') 

#looping through containers 
    for container in movie_containers: 
     if container.find('div', class_ = 'ratings-metascore') is not None: 

     #The name 
      name = container.h3.a.text 
     #print(name) 
      names.append(name + ',') 

     #The Year 
      year = container.find('span', class_ = 'lister-item-year').text 
      years.append(year + ',') 

     #IDMb rating    
      imdb = container.strong.text 
     #print(imdb) 
      imdb_ratings.append(imdb + ',') 

     #Metascore    
      metascore = container.find('span', class_= 'metascore').text 
      metascores.append(metascore + ',') 

     #Number of Votes 
      vote = container.find('span', attrs = {'name':'nv'})['data-value'] 
      votes.append(vote + ',') 

    pages += 1