The San Francisco PHP Meetup Group Message Board › Huge keyword index

Huge keyword index

Dave Edelhart
Posted Apr 25, 2010 7:09 PM
bingomanatee
San Francisco, CA
Post #: 50
Send an Email Post a Greeting
I am creating a system that indexes keywords in email. I'm getting a huge join table with just a few thousand mails and expect the mail system to grow to several million mails.

I'm already got mySQL running into the ground. Any recommendation for keyword indexing of large content that's robust and scales?

Thanks, Dave
Danny Wilkerson II
Posted Apr 26, 2010 12:14 AM
user 7857566
Dallas, TX
Post #: 117
Send an Email Post a Greeting
add a filter the eliminates the common words before it's entered into db.
add in code to remove the unnecessary common words already in your system for clean up.
use utf8 so you don't have foreign character problems.

create a table that uses the words as the key and then increment for counting to track total occurrences of a word
create a table that holds data for each email and like before use the word as key and increment.

i don't think emails usually get that big so location of keywords in the email probably not needed. Umm maybe you just need a table for each word and under that table it lists the emails with it.

create a table that auto increments the key(id for word) and input the word
create a table for each word using the id of that word in the previous table as the table name.
this table should have the columns auto increment(index), email, emailtimestamp
so basically each table holds the locations of the word that this table represents.

not sure if this will work or is any help. My php skills are getting a little rusty.
Michael Pearson
Posted May 18, 2010 7:46 PM
user 11368639
San Francisco, CA
Post #: 3
Send an Email Post a Greeting
Have you looked at Solr/Lucene (pecl solr in beta atm)? Not sure MySQL is a great solution in this instance.
Powered by mvnForum

Our Sponsors

GameSpot.com

GameSpot.com & CBSi provides the facilities our group uses every month.

O'Reilly User Group Program

The O'Reilly User Group Program provides the great books we get monthly.

Other nearby
Meetup Groups
Why these groups?
x

The Meetup Groups shown here are topically similar to The San Francisco PHP Meetup Group.

Groups are more likely to be displayed here if they:

  • have a Meetup scheduled
  • have a high rating
  • have a group photo
  • are "public" and not "private"
  • have shown they are likely to stick around (older than 30 days)

Log in

  • Not registered with us yet?
or

Log in to Meetup with your Facebook account.

Sign up

or

Join this Meetup Group even quicker with your Facebook account.

By clicking the "Sign up using Facebook" or "Sign up" buttons above, you agree to Meetup's Terms of Service