데이터를 저장하지 않는 Scrapy Spider

Scrapy를 사용하여 농구 팀 일정을 CSV 파일에 저장하려고합니다. 나는이 파일에 다음 코드를 작성했습니다 :데이터를 저장하지 않는 Scrapy Spider

settings.py

BOT_NAME = 'test_project' 

SPIDER_MODULES = ['test_project.spiders'] 
NEWSPIDER_MODULE = 'test_project.spiders' 

FEED_FORMAT = "csv" 
FEED_URI = "cportboys.csv" 

# Crawl responsibly by identifying yourself (and your website) on the user-agent 
#USER_AGENT = 'test_project (+http://www.yourdomain.com)' 

# Obey robots.txt rules 
ROBOTSTXT_OBEY = True

이제 khsaabot.py

import scrapy 


class KhsaabotSpider(scrapy.Spider): 
    name = 'khsaabot' 
    allowed_domains = ['https://scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978'] 
    start_urls = ['http://https://scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/'] 

def parse(self, response): 
    date = response.css('.mdate::text').extract() 
    opponent = response.css('.opponent::text').extract() 
    place = response.css('.schedule-loc::text').extract() 


    for item in zip(date,opponent,place): 
     scraped_info = { 
      'date' : item[0], 
      'opponent' : item[1], 
      'place' : item[2], 
     } 

     yield scraped_info

, 내가 잘못 여기에 무슨 일이 일어나고 있는지 확실하지 않다 "scrapy crawl khsaabot"을 사용하여 터미널에서 실행할 때 오류가 발생하지 않고 잘 작동하는 것처럼 보입니다. 그러나, 단지의 경우 터미널에서 무슨 일이 일어나고 있는지에 문제가, 나도 거기에 도착 출력 포함 :

이

2017-12-27 17:21:49 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: test_project) 
2017-12-27 17:21:49 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'test_project', 'FEED_FORMAT': 'csv', 'FEED_URI': 'cportboys.csv', 'NEWSPIDER_MODULE': 'test_project.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['test_project.spiders']} 
2017-12-27 17:21:49 [scrapy.middleware] INFO: Enabled extensions: 
['scrapy.extensions.corestats.CoreStats', 
'scrapy.extensions.telnet.TelnetConsole', 
'scrapy.extensions.memusage.MemoryUsage', 
'scrapy.extensions.feedexport.FeedExporter', 
'scrapy.extensions.logstats.LogStats'] 
2017-12-27 17:21:49 [scrapy.middleware] INFO: Enabled downloader middlewares: 
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 
'scrapy.downloadermiddlewares.retry.RetryMiddleware', 
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 
'scrapy.downloadermiddlewares.stats.DownloaderStats'] 
2017-12-27 17:21:49 [scrapy.middleware] INFO: Enabled spider middlewares: 
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 
'scrapy.spidermiddlewares.referer.RefererMiddleware', 
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 
'scrapy.spidermiddlewares.depth.DepthMiddleware'] 
2017-12-27 17:21:49 [scrapy.middleware] INFO: Enabled item pipelines: 
[] 
2017-12-27 17:21:49 [scrapy.core.engine] INFO: Spider opened 
2017-12-27 17:21:49 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2017-12-27 17:21:49 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://https/robots.txt> (failed 1 times): DNS lookup failed: no results for hostname lookup: https. 
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://https/robots.txt> (failed 2 times): DNS lookup failed: no results for hostname lookup: https. 
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://https/robots.txt> (failed 3 times): DNS lookup failed: no results for hostname lookup: https. 
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http://https/robots.txt>: DNS lookup failed: no results for hostname lookup: https. 
Traceback (most recent call last): 
    File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks 
    result = result.throwExceptionIntoGenerator(g) 
    File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator 
    return g.throw(self.type, self.value, self.tb) 
    File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request 
    defer.returnValue((yield download_func(request=request,spider=spider))) 
    File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks 
    current.result = callback(current.result, *args, **kw) 
    File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 954, in startConnectionAttempts 
    "no results for hostname lookup: {}".format(self._hostStr) 
twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: https. 
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://https//scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/> (failed 1 times): DNS lookup failed: no results for hostname lookup: https. 
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://https//scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/> (failed 2 times): DNS lookup failed: no results for hostname lookup: https. 
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://https//scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/> (failed 3 times): DNS lookup failed: no results for hostname lookup: https. 
2017-12-27 17:21:49 [scrapy.core.scraper] ERROR: Error downloading <GET http://https//scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/> 
Traceback (most recent call last): 
    File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks 
    result = result.throwExceptionIntoGenerator(g) 
    File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator 
    return g.throw(self.type, self.value, self.tb) 
    File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request 
    defer.returnValue((yield download_func(request=request,spider=spider))) 
    File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks 
    current.result = callback(current.result, *args, **kw) 
    File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 954, in startConnectionAttempts 
    "no results for hostname lookup: {}".format(self._hostStr) 
twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: https. 
2017-12-27 17:21:49 [scrapy.core.engine] INFO: Closing spider (finished) 
2017-12-27 17:21:49 [scrapy.statscollectors] INFO: Dumping Scrapy stats: 
{'downloader/exception_count': 6, 
'downloader/exception_type_count/twisted.internet.error.DNSLookupError': 6, 
'downloader/request_bytes': 1416, 
'downloader/request_count': 6, 
'downloader/request_method_count/GET': 6, 
'finish_reason': 'finished', 
'finish_time': datetime.datetime(2017, 12, 27, 23, 21, 49, 579649), 
'log_count/DEBUG': 7, 
'log_count/ERROR': 2, 
'log_count/INFO': 7, 
'memusage/max': 50790400, 
'memusage/startup': 50790400, 
'retry/count': 4, 
'retry/max_reached': 2, 
'retry/reason_count/twisted.internet.error.DNSLookupError': 4, 
'scheduler/dequeued': 3, 
'scheduler/dequeued/memory': 3, 
'scheduler/enqueued': 3, 
'scheduler/enqueued/memory': 3, 
'start_time': datetime.datetime(2017, 12, 27, 23, 21, 49, 323652)} 
2017-12-27 17:21:49 [scrapy.core.engine] INFO: Spider closed (finished)

출력은 바로 내 모습을,하지만 난 Scrapy 그래서 여전히 새로운 오전 뭔가를 놓칠 수 있습니다.

감사합니다 모두들

로그에 twisted.internet.error.DNSLookupError 메시지를 받고

출처

2017-12-27 Hunter

. start_urls 목록에서 "http://https://"으로 시작됩니다. 변경 :

start_urls = ['http://https://scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/']

에 :

start_urls = ['https://scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/']

출처

2017-12-28 00:21:06 crookedleaf

당신을 감사합니다! 나는 당신이 말한 것을했는데 그 오류를 없앴습니다,하지만 지금은 "NotImplementedError"라고 불리는 같은 장소에서 하나를 얻고 있습니다. 이것에 대해 내가 뭘 할 수 있니? – Hunter

신경 쓰지 마라. 나는 약간의 검색을 한 후에 그 오류와 내가 발견 한 다른 것을 고치는 방법을 알아 냈다. 이 하나를 도와 주셔서 감사합니다! – Hunter

데이터를 저장하지 않는 Scrapy Spider

답변

관련 문제