The San Francisco PHP Meetup Group Message Board › Huge keyword index
| Dave Edelhart | |
|
|
I am creating a system that indexes keywords in email. I'm getting a huge join table with just a few thousand mails and expect the mail system to grow to several million mails.
I'm already got mySQL running into the ground. Any recommendation for keyword indexing of large content that's robust and scales? Thanks, Dave |
| Danny Wilkerson II | |
|
|
add a filter the eliminates the common words before it's entered into db.
add in code to remove the unnecessary common words already in your system for clean up. use utf8 so you don't have foreign character problems. create a table that uses the words as the key and then increment for counting to track total occurrences of a word create a table that holds data for each email and like before use the word as key and increment. i don't think emails usually get that big so location of keywords in the email probably not needed. Umm maybe you just need a table for each word and under that table it lists the emails with it. create a table that auto increments the key(id for word) and input the word create a table for each word using the id of that word in the previous table as the table name. this table should have the columns auto increment(index), email, emailtimestamp so basically each table holds the locations of the word that this table represents. not sure if this will work or is any help. My php skills are getting a little rusty. |
| Michael Pearson | |
|
Have you looked at Solr/Lucene (pecl solr in beta atm)? Not sure MySQL is a great solution in this instance.
|