ValueError : 요청시 스키마가 누락되었습니다. url : h

Question

나는 치료, 파이썬에서 초보자입니다. scrapinghub에 spider 코드를 배포하려고 시도했지만 다음과 같은 오류가 발생했습니다. 아래는 코드입니다.

import scrapy 
from bs4 import BeautifulSoup,SoupStrainer 
import urllib2 
from scrapy.selector import Selector 
from scrapy.http import HtmlResponse 
import re 
import pkgutil 
from pkg_resources import resource_string 
from tues1402.items import Tues1402Item 

data = pkgutil.get_data("tues1402","resources/urllist.txt") 
class SpiderTuesday (scrapy.Spider):  
    name = 'tuesday' 
    self.start_urls = [url.strip() for url in data] 
    def parse(self, response): 
     story = Tues1402Item() 
     story['url'] = response.url 
     story['title'] = response.xpath("//title/text()").extract() 
     return story 

는

import scrapy 
class Tues1402Item(scrapy.Item): 
    # define the fields for your item here like: 
    # name = scrapy.Field() 
    title = scrapy.Field() 
    url = scrapy.Field() 

가 items.py 코드와

from setuptools import setup, find_packages 
setup(
    name   = 'tues1402', 
    version  = '1.0', 
    packages  = find_packages(), 
    entry_points = {'scrapy': ['settings = tues1402.settings']}, 
    package_data = {'tues1402':['resources/urllist.txt']}, 
    zip_safe = False, 
) 

setup.py 코드가 내 spider.py 코드입니다.

다음은 오류입니다.

Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 126, in _next_request request = next(slot.start_requests) File "/usr/local/lib/python2.7/site-packages/scrapy/spiders/init.py", line 70, in start_requests yield self.make_requests_from_url(url) File "/usr/local/lib/python2.7/site-packages/scrapy/spiders/init.py", line 73, in make_requests_from_url return Request(url, dont_filter=True) File "/usr/local/lib/python2.7/site-packages/scrapy/http/request/init.py", line 25, in init self._set_url(url) File "/usr/local/lib/python2.7/site-packages/scrapy/http/request/init.py", line 57, in _set_url raise ValueError('Missing scheme in request url: %s' % self._url) ValueError: Missing scheme in request url: h

는

Answer 1

귀하의 오류가 URL h가 유효한 URL이 없다는 것을 의미 사전에 감사합니다. self.start_urls을 출력하고 거기에있는 URL을 확인하십시오. 첫 번째 URL로 h 문자열이있을 가능성이 큽니다.

data = pkgutil.get_data("tues1402","resources/urllist.txt") 
class SpiderTuesday (scrapy.Spider):  
    name = 'tuesday' 
    self.start_urls = [url.strip() for url in data] 

당신이 그 분할해야 urllist.txt 파일에 일부 구분하여 URL을 저장한다고 가정 :

# assuming file has url in every line 
self.start_urls = [url.strip() for url in data.splitlines()]