텍스트 처리 - 구문 검색 후 Word2Vec 학습 (bigram 모델)

Question

평소보다 많은 n-gram을 가진 word2vec 모델을 만들고 싶습니다. 내가 찾은 것처럼 gensim.models.phrase의 Phrase 클래스는 원하는 구문을 찾을 수 있으며, 코퍼스에서 구문을 사용하고 word2vec 열차 기능의 결과 모델을 사용할 수 있습니다.

그래서 먼저 아래 코드를 gensim documentation에 입력하면됩니다.

class MySentences(object): 
    def __init__(self, dirname): 
     self.dirname = dirname 

    def __iter__(self): 
     for fname in os.listdir(self.dirname): 
      for line in open(os.path.join(self.dirname, fname)): 
       yield word_tokenize(line) 

sentences = MySentences('sentences_directory') 

bigram = gensim.models.Phrases(sentences) 

model = gensim.models.Word2Vec(bigram['sentences'], size=300, window=5, workers=8) 

모델이 생성되었지만 평가에서 좋은 결과와 경고없이 :

WARNING : train() called with an empty iterator (if not intended, be sure to provide a corpus that offers restartable iteration = an iterable) 

내가 검색 내가 https://groups.google.com/forum/#!topic/gensim/XWQ8fPMFSi0을 발견하고 내 코드 변경 :

class MySentences(object): 
    def __init__(self, dirname): 
     self.dirname = dirname 

    def __iter__(self): 
     for fname in os.listdir(self.dirname): 
      for line in open(os.path.join(self.dirname, fname)): 
       yield word_tokenize(line) 

class PhraseItertor(object): 
    def __init__(self, my_phraser, data): 
     self.my_phraser, self.data = my_phraser, data 

    def __iter__(self): 
     yield self.my_phraser[self.data] 


sentences = MySentences('sentences_directory') 

bigram_transformer = gensim.models.Phrases(sentences) 

bigram = gensim.models.phrases.Phraser(bigram_transformer) 

corpus = PhraseItertor(bigram, sentences) 

model = gensim.models.Word2Vec(corpus, size=300, window=5, workers=8) 

I을 오류 발생 :

Traceback (most recent call last): 
    File "/home/fatemeh/Desktop/Thesis/bigramModeler.py", line 36, in <module> 
    model = gensim.models.Word2Vec(corpus, size=300, window=5, workers=8) 
    File "/home/fatemeh/.local/lib/python3.4/site-packages/gensim/models/word2vec.py", line 478, in init 
    self.build_vocab(sentences, trim_rule=trim_rule) 
    File "/home/fatemeh/.local/lib/python3.4/site-packages/gensim/models/word2vec.py", line 553, in build_vocab 
    self.scan_vocab(sentences, progress_per=progress_per, trim_rule=trim_rule) # initial survey 
    File "/home/fatemeh/.local/lib/python3.4/site-packages/gensim/models/word2vec.py", line 575, in scan_vocab 
    vocab[word] += 1 
TypeError: unhashable type: 'list' 

이제 코드에 무엇이 잘못되었는지 알고 싶습니다.

Answer 1

나는 Gensim GoogleGroup 내 질문을하고 Mr Gordon Mohr 내게 대답

You typically wouldn't want an __iter__() method to do a single yield . It should return an iterator object (ready to return multiple objects via next() or a StopIteration exception). One way to effect a iterator is to use yield to have the method treated as a 'generator' – but that would typically require the yield to be inside a loop.

But I now see that my example code in the thread you reference does the wrong thing with its__iter__() return line: it should not be returning the raw phrasifier, but one that has already been started-as-an-iterator, by use of the iter() built-in method. That is, the example there should have read:

class PhrasingIterable(object): 
    def __init__(self, phrasifier, texts): 
     self. phrasifier, self.texts = phrasifier, texts 
    def __iter__(): 
     return iter(phrasifier[texts]) 

Making a similar change in your variation may resolve the TypeError: iter() returned non-iterator of type 'TransformedCorpus' error.

을