스플래시로 치료법을 사용하여 LinkedIn을 긁을 때 502 오류가 발생했습니다

Question

스플래시가있는 스 캐플링을 사용하여 Netflix의 Linkedin 회사 페이지를 스크랩하려고했습니다. scrap shell을 사용할 때 완벽하게 작동하지만 스크립트를 실행할 때 502 오류가 발생합니다.

오류 : 스플래쉬 터미널에서

2017-01-06 16:06:45 [scrapy.core.engine] INFO: Spider opened 
2017-01-06 16:06:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2017-01-06 16:06:52 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.linkedin.com/company/netflix via http://localhost:8050/render.html> (failed 1 times): 502 Bad Gateway 
2017-01-06 16:06:55 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.linkedin.com/company/netflix via http://localhost:8050/render.html> (failed 2 times): 502 Bad Gateway 
2017-01-06 16:07:05 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://www.linkedin.com/company/netflix via http://localhost:8050/render.html> (failed 3 times): 502 Bad Gateway 
2017-01-06 16:07:05 [scrapy.core.engine] DEBUG: Crawled (502) <GET https://www.linkedin.com/company/netflix via http://localhost:8050/render.html> (referer: None) 
2017-01-06 16:07:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <502 https://www.linkedin.com/company/netflix>: HTTP status code is not handled or not allowed 
2017-01-06 16:07:05 [scrapy.core.engine] INFO: Closing spider (finished) 

: 거미에 대한

2017-01-06 10:36:52.186410 [render] [139764812670456] loadFinished: RenderErrorInfo(type='HTTP', code=999, text='Request denied', url='https://www.linkedin.com/company/netflix') 
2017-01-06 10:36:52.205523 [events] {"fds": 18, "qsize": 0, "args": {"url": "https://www.linkedin.com/company/netflix", "headers": {"User-Agent": "Scrapy/1.3.0 (+http://scrapy.org)", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en"}, "uid": 139764812670456, "wait": 0.5}, "rendertime": 6.674675464630127, "timestamp": 1483699012, "user-agent": "Scrapy/1.3.0 (+http://scrapy.org)", "maxrss": 87956, "error": {"info": {"url": "https://www.linkedin.com/company/netflix", "code": 999, "type": "HTTP", "text": "Request denied"}, "error": 502, "description": "Error rendering page", "type": "RenderError"}, "active": 0, "load": [0.51, 0.67, 0.8], "status_code": 502, "client_ip": "172.17.0.1", "method": "POST", "_id": 139764812670456, "path": "/render.html"} 
2017-01-06 10:36:52.206259 [-] "172.17.0.1" - - [06/Jan/2017:10:36:51 +0000] "POST /render.html HTTP/1.1" 502 192 "-" "Scrapy/1.3.0 (+http://scrapy.org)" 

코드 :

import scrapy 
from scrapy_splash import SplashRequest 
from linkedin.items import LinkedinItem 


class LinkedinScrapy(scrapy.Spider): 
    name = 'linkedin_spider' # spider name 
    allowed_domains = ['linkedin.com'] 
    start_urls = ['https://www.linkedin.com/company/netflix'] 

    def start_requests(self): 
     for url in self.start_urls: 
      yield SplashRequest(url, self.parse, 
          endpoint='render.html', args={'wait': 0.5) 

    def parse(self, response): 
     item = LinkedinItem() 
     item['name'] = response.xpath('//*[@id="stream-promo-top-bar"]/div[2]/div[1]/div[1]/div/h1/span/text()').extract_first() 
     item['followers'] = response.xpath('//*[@id = "biz-follow-mod"]/div/div/div/p/text()').extract_first().split()[0] 
     item['description'] = response.xpath('//*[@id="stream-about-section"]/div[2]/div[1]/div/p/text()').extract_first() 
       yield item 

Answer 1

그것은 아마 링크드 때문에 사용자 에이전트 문자열의 액세스를 거부하기 때문에 요청이 사용 :

"User-Agent": "Scrapy/1.3.0 (+http://scrapy.org)" 

당신은, 뭔가 다른 거미의 사용자 에이전트를 변경