팬더의 맞춤법 검사기

Question

Peter Norvig's spell checker을 SQL 데이터베이스에서 가져온 단어가있는 팬더 클래스에 구현하려고합니다. 데이터에는 종종 여러 가지 철자 오류가 포함 된 사용자 쿼리가 포함되어 있으며이 클래스가 가장 가능성있는 쿼리 (올바르게 입력 된 것)를 반환하기를 바랍니다.

클래스는 팬더 데이터 프레임을 반환하는 데이터베이스 쿼리로 초기화됩니다. 예를 들면 :

아래의 대부분은 피터의 작품에서 직접 끌어하지만 클래스에 대한 수정 사항이 제대로 작동하지 않는 것

query  count 
0 foo bar  1864 
1 super foo  73 
2 bar of foos 1629 
3 crazy foos  940 

. 내 생각 엔 카운터 기능 (WORDS = Counter(words(open('big.txt').read())))을 제거하는 것과 관련이 있지만 데이터 프레임에서이 기능을 사용하는 가장 좋은 방법은 확실하지 않습니다. 아래

현재 클래스 : 사전에

class _SpellCheckClient(object): 
    """Wraps functionality to check the spelling of a query.""" 

    def __init__(self, team, table, dremel_connection): 
    self.df = database_connection.ExecuteQuery(
     'SELECT query, COUNT(query) AS count FROM table GROUP BY 1;' 

    def expected_word(self, word): 
    """Most probable spelling correction for word.""" 
    return max(self._candidates(word), key=self._probability) 

    def _probability(self, query): 
    """Probability of a given word within a query.""" 
    query_count = self.df.loc[self.df['query'] == query]['count'].values 
    return query_count/self.df['count'].sum() 

    def _candidates(self, word): 
    """Generate possible spelling corrections for word.""" 
    return (self._known([word]) 
      or self._known(self._one_edits_from_word(word)) 
      or self._known(self._two_edits_from_word(word)) 
      or [word]) 

    def _known(self, query): 
    """The subset of `words` that appear in the dictionary of WORDS.""" 
    # return set(w for w in query if w in WORDS) 
    return set(w for w in query if w in self.df['query'].value_counts) 

    def _one_edits_from_word(self, word): 
    """All edits that are one edit away from `word`.""" 
    splits = [(word[:i], word[i:]) for i in xrange(len(word) + 1)] 
    deletes = [left + right[1:] for left, right in splits if right] 
    transposes = [left + right[1] + right[0] + right[2:] 
        for left, right in splits 
        if len(right) > 1] 
    replaces = [left + center + right[1:] 
       for left, right in splits 
       if right for center in LETTERS] 
    inserts = [left + center + right 
       for left, right in splits 
       for center in LETTERS] 
    return set(deletes + transposes + replaces + inserts) 

    def _two_edits_from_word(self, word): 
    """All edits that are two edits away from `word`.""" 
    return (e2 for e1 in self._one_edits_from_word(word) 
      for e2 in self._one_edits_from_word(e1)) 

감사합니다! 이에 대한 답을 찾고있는 사람들을위한

Answer 1

은 아래 나를 위해 일한 것입니다 :

def _words(df): 
    """Returns the total count of each word within a dataframe.""" 
    return df['query'].str.get_dummies(sep=' ').T.dot(df['count']) 


class _SpellCheckClient(object): 
    """Wraps functionality to check the spelling of a query.""" 

    def __init__(self, team, table, database_connection): 
    self.df = database_connection 
    self.words = _words(self.df) 

    def expected_word(self, query): 
    """Most probable spelling correction for word.""" 
    return max(self._candidates(query), key=self._probability) 

    def _probability(self, query): 
    """Probability of a given word within a query.""" 
    return self.words.pipe(lambda x: x/x.sum()).get(query, 0.0) 

    def _candidates(self, query): 
    """Generate possible spelling corrections for word.""" 
    return (self._known(self._one_edits_from_query(query)) 
      or self._known(self._two_edits_from_query(query)) 
      or [query]) 

    def _known(self, query): 
    """The subset of `query` that appear in the search console database.""" 
    return set(w for w in query if self.words.get(w)) 

    def _one_edits_from_query(self, query): 
    """All edits that are one edit away from `query`.""" 
    splits = [(query[:i], query[i:]) for i in xrange(len(query) + 1)] 
    deletes = [left + right[1:] for left, right in splits if right] 
    transposes = [left + right[1] + right[0] + right[2:] 
        for left, right in splits 
        if len(right) > 1] 
    replaces = [left + center + right[1:] 
       for left, right in splits 
       if right for center in LETTERS] 
    inserts = [left + center + right 
       for left, right in splits 
       for center in LETTERS] 
    return set(deletes + transposes + replaces + inserts) 

    def _two_edits_from_query(self, query): 
    """All edits that are two edits away from `query`.""" 
    return (e2 for e1 in self._one_edits_from_query(query) 
      for e2 in self._one_edits_from_query(e1))