Scrapy가 : scrapy에 SQLAlchemy의를 사용하여이 :

Question

이 scrapy에 SQLAlchemy의 사용 "나가서 설명하자면 NameError 이름을 '연결'정의되지 않은"NameError, 다음과 같은 오류 메시지가 있습니다

: 여기

Traceback (most recent call last): 
    File "e:\weibo_spider\venv\lib\site-packages\twisted\internet\defer.py", line 1386, in _inlineCallbacks 
    result = g.send(result) 
    File "e:\weibo_spider\venv\lib\site-packages\scrapy\crawler.py", line 79, in crawl 
    yield self.engine.open_spider(self.spider, start_requests) 
NameError: name 'connection' is not defined 

을 그리고 내 ScrapyPipeline 클래스

from sqlalchemy import create_engine 
from sqlalchemy.orm import sessionmaker 

from .models import MyModel # my sqlalchemy model 


class WeiboSpiderPipeline(object): 

    def open_spider(self, spider): 
     # using pymysql as the conncetor 
     engine = create_engine('mysql+pymysql://root@localhost/wbspider_data') 
     Session = sessionmaker(bind=engine) 
     self.conn = engine.connect() 
     self.session = Session(bind=connection) 

    def close_spider(self, spider): 
     self.conn.close() 

    def process_item(self, item, spider): 
     return item 

내가 명령 줄에서 모델을 테스트, 그것은 작동 할 수 있지만, 나는 scrapy crawl myspidername 명령을 실행 한 후에는 NameError을 발생합니다.

도움말!

Answer 1

이 방법을 사용하십시오.

from sqlalchemy.orm import sessionmaker 

Session = sessionmaker(bind=engine) 
session = Session() 

# it will send the connection back to the pool of connections in the engine, but it will not close the connection instead it will make the connection idle 
session.close() 
# to close the engine having pool of connections will close all the idle connections in the pool 
engine.dispose()() 


# for directing close the connection on session.close, use this code 
# Disabling pooling using NullPool: 

from sqlalchemy.pool import NullPool 
engine = create_engine(
     'postgresql+psycopg2://scott:tiger@localhost/test', 
     poolclass=NullPool) 
session.close()