행의 n 그램에서 중복을 제거합니다 python

Question

이 코드는 n 그램과 n 그램이 나타나는 개수를 생성합니다. 필자는 행이있는 csv 파일과 모든 행에 대해 단어 열을 포함하는 열을 가지고 있습니다. 예를 들어이 코드를 검색하면 'this is my puppy'와 같은 4 그램을 얻습니다.이 코드는 동일한 행에서 발생하는 발생 횟수도 계산합니다. 내 의도는 행에 n-gram이 발생하면 한 번 계산해야하고 다른 행에서 두 번째로 계산해야한다는 것입니다.

e.g row   Word 
     1   this is my puppy what this is my puppy 
     2   this is my puppy 

이 코드는 'this is my puppy'를 세 번 계산합니다. 그러나 나는 그것을 2 배

되고 싶어이 당신의 도움이 높게 평가 될 것이다 파이썬 코드

import collections 
import re 
import sys 
import time 


def tokenize(string): 
    """Convert string to lowercase and split into words (ignoring 
    punctuation), returning list of words. 
    """ 
    return re.findall(r'\w+', string.lower()) 


def count_ngrams(lines, min_length=4, max_length=5): 
    """Iterate through given lines iterator (file object or list of 
    lines) and return n-gram frequencies. The return value is a dict 
    mapping the length of the n-gram to a collections.Counter 
    object of n-gram tuple and number of times that n-gram occurred. 
    Returned dict includes n-grams of length min_length to max_length. 
    """ 
    lengths = range(min_length, max_length + 1) 
    ngrams = {length: collections.Counter() for length in lengths} 
    queue = collections.deque(maxlen=max_length) 

    # Helper function to add n-grams at start of current queue to dict 
    def add_queue(): 
     current = tuple(queue) 
     for length in lengths: 
      if len(current) >= length: 
       ngrams[length][current[:length]] += 1 

    # Loop through all lines and words and add n-grams to dict 
    for line in lines: 
     for word in tokenize(line): 
      queue.append(word) 
      if len(queue) >= max_length: 
        add_queue() 

    # Make sure we get the n-grams at the tail end of the queue 
    while len(queue) > min_length: 
     queue.popleft() 
     add_queue() 

    return ngrams 


def print_most_frequent(ngrams, num=10): 
    """Print num most common n-grams of each length in n-grams dict.""" 
    for n in sorted(ngrams): 
     print('----- {} most common {}-grams -----'.format(num, n)) 
     for gram, count in ngrams[n].most_common(num): 
      print('{0}: {1}'.format(' '.join(gram), count)) 
     print('') 


if __name__ == '__main__': 
    if len(sys.argv) < 2: 
     print('Usage: python ngrams.py filename') 
     sys.exit(1) 

    start_time = time.time() 
    with open("PWorm.csv") as f: 
     ngrams = count_ngrams(f) 
    print_most_frequent(ngrams) 
    elapsed_time = time.time() - start_time 
    print('Took {:.03f} seconds'.format(elapsed_time)) 

입니다. , 당신은 라인 당 N- 그램 - DICT을해야합니다, 당신은 ngrams 반 수동으로 두 번 계산하는 선에서 defaultdict

이 같은 N- 그램을 방지하기 위해 사용할 수 있습니다 채우는 대신 당신에게

Answer 1

감사 다음 while len(queue) > min_length: 후

def count_ngrams(lines, min_length=4, max_length=5): 
    """Iterate through given lines iterator (file object or list of 
    lines) and return n-gram frequencies. The return value is a dict 
    mapping the length of the n-gram to a collections.Counter 
    object of n-gram tuple and number of times that n-gram occurred. 
    Returned dict includes n-grams of length min_length to max_length. 
    """ 
    lengths = range(min_length, max_length + 1) 
    ngrams = collections.defaultdict(collections.Counter) 
    queue = collections.deque(maxlen=max_length) 

    # Helper function to add n-grams at start of current queue to dict 
    def add_queue(ngrams_line): 
     current = tuple(queue) 
     for length in lengths: 
      if len(current) >= length: 
       ngrams_line[length][current[:length]] = 1 # instead of += 1 

    # to combine the 2 defaultdict(Counter)    
    def combine_ngrams(ngram, ngramline): 
     for k, v in ngramsline.items(): 
      ngrams[k] += v 
     return ngrams 

    # Loop through all lines and words and add n-grams to dict 
    for line in lines: 
     ngrams_line = collections.defaultdict(collections.Counter) 
     for word in tokenize(line): 
      queue.append(word) 
      if len(queue) >= max_length: 
        add_queue(ngrams_line) 
     ngrams = combine_ngrams(ngrams, ngrams_line) 


    # Make sure we get the n-grams at the tail end of the queue 
    ngrams_line = collections.defaultdict(collections.Counter) 
    while len(queue) > min_length: 
     queue.popleft() 
     add_queue(ngrams_line) 
    ngrams = combine_ngrams(ngrams, ngrams_line) 

    return ngrams 

내가 100 % 부분을 이해하지 못하는 일반 N- 그램의 DICT와 그 결합, 또는 queue이 everyline를 다시 설정되지 않습니다, 당신은 왜 당신은 내 대답 a를 조정해야 할 수도 있습니다 비트