Skip to main content
Drupal 7 and SOLR

Drupal 7 and SOLR - autocompleting full node title

KnackForge has done a lot customizations on Drupal and SOLR integration. Earlier I wrote about Drupal-7-filtering-solr-results which covers limiting SOLR resutls. This is another work where we had to alter default apache solr autocomplete module to limit suggestions only to node titles. And other interesting challenge in it was to provide case insensitive search with case sensitive results and phrase based search. Meaning, when node title is "Hello world", searching for 'hello' should return "Hello world" not the two results 'hello' and 'world'. 

Autocomplete module:

The  apache solr autocomplete module uses keywords that are already broken into words (space delimited). So we had to write our custom solr configurations which stores untokenized titlesI am going to detail the whole process here,

Pre-requests:

I assume you already have installed apachesolr, apachesolr_autocomplete with properly setup SOLR.

Solr schema:

In schema.xml, add a new field type under <types> that controls our new field indexing and querying,

<fieldType class="solr.TextField" name="text_auto"> 
  <analyzer>
   <tokenizer class="solr.KeywordTokenizerFactory"/>  
 </analyzer>  
</fieldType>

Remember I am not using any LowerCaseFilters here. KeyWordTokenizerFactory used here just stores the full phrase without delimiting.

Now define a new field,

<field name="label_autocomplete" type="text_auto" indexed="true" stored="true"/>
under <fields> section.
 
Solr config:
 
In solrconfig.xml, 
 
1) Add a new search component,
<searchComponent name="suggest" class="solr.SpellCheckComponent"><br>
  <str name="queryAnalyzerFieldType">text_auto</str>
  <lst name="spellchecker">
   <str name="name">suggest</str>
   <str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
   <str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookup</str>
   <str name="buildOnOptimize">true</str>
   <str name="buildOnCommit">true</str>
   <str name="field">label_autocomplete</str>
 </lst>
</searchComponent>
2) Add a request handler,
<requestHandler name="suggest" class="solr.SearchHandler">
     <lst name="defaults">
      <str name="spellcheck">true</str>
      <str name="spellcheck.dictionary">suggest</str>
      <str name="spellcheck.count">10</str>
     </lst>
     <arr name="components">
      <str>suggest</str>
     </arr>
</requestHandler>
 
 
Drupal side changes:
 
In your custom module add this hook,
/**
 * Implements hook_apachesolr_index_document_build().
 */
function hook_apachesolr_index_document_build(ApacheSolrDocument $document, $entity, $entity_type, $env_id) {
  if (!empty($entity->title)) {
    //append the lowercased title and normal title to help auto completion
    $label_auto = strtolower($entity->title) . '~~' . $entity->title;
    $document->setMultiValue('label_autocomplete', $label_auto);
  }
  else {
    $document->setMultiValue('label_autocomplete', '');
  }
}
The above code indexes node title field. The lowercased and original titles are appended together. This is done to return original title but provide case insensitive search.Doing this in SOLR side needs custom filter to be written. Now we need to change the original apachesolr_autocomplete.module file, I find no other way at the moment. All we need is to alter only one method, here is the updated method,
 
function apachesolr_autocomplete_suggest($keys, $params, $theme_callback, $orig_keys, $suggestions_to_return = 5) {
  $matches = array();
  $suggestions = array();
  $keys = trim($keys);

  // We need the keys array to make sure we don't suggest words that are already
  // in the search terms.
  $keys_array = explode(' ', $keys);
  $keys_array = array_filter($keys_array);

  // Query Solr for $keys so that suggestions will always return results.
  $query = apachesolr_drupal_query($keys);
  //add our custom request handler
  $query->replaceParam('qt', 'suggest');
  if (!$query) {
    return array();
  }
  $new_params = array('q' => $params['facet.prefix']);
  foreach ($new_params as $param => $paramValue) {
    $query->addParam($param, $paramValue);
  }

  // Query Solr
  $response = $query->search($keys);
  $q = $new_params['q'];
  $suggestions_data = $response->spellcheck->suggestions->{$q};
  foreach ($suggestions_data->suggestion as $term) {
    //split the label into two part and take second part which contains original label
    $labels = explode('~~', $term);
    if ($labels && count($labels) == 2) {
      $term = $labels[1];      
    }
    if (isset($matches[$term])) {
      $matches[$term] += 1;
    }
    else {
      $matches[$term] = 1;
    }
  }

  if (sizeof($matches) > 0) {
    // Eliminate suggestions that are stopwords or are already in the query.
    $matches_clone = $matches;
    $stopwords = apachesolr_autocomplete_get_stopwords();
    foreach ($matches_clone as $term => $count) {
      if ((strlen($term) > 3) && !in_array($term, $stopwords) && !array_search($term, $keys_array)) {
        // Longer strings get higher ratings.
        #$matches_clone[$term] += strlen($term);
      }
      else {
        unset($matches_clone[$term]);
        unset($matches[$term]);
      }
    }

    // The $count in this array is actually a score. We want the highest ones first.
    arsort($matches_clone);

    // Shorten the array to the right ones.
    $matches_clone = array_slice($matches_clone, 0, $suggestions_to_return, TRUE);

    // Build suggestions using returned facets
    foreach ($matches_clone as $match => $count) {
      if ($keys != $match) {
        $suggestion = trim($keys . ' ' . $match);
        // On cases where there are more than 3 keywords, omit displaying
        //  the count because of the mm settings in solrconfig.xml
        if (substr_count($suggestion, ' ') >= 2) {
          $count = 0;
        }
        if ($suggestion != '') {
          // Add * to array element key to force into a string, else PHP will
          // renumber keys that look like numbers on the returned array.
          $suggestions['*' . $suggestion] = theme('apachesolr_autocomplete_highlight', array('keys' => $orig_keys, 'suggestion' => $suggestion, 'count' => $count));
        }
      }
    }
  }

  return array(
    'suggestions' => $suggestions,
    'response' => &$response
  );
}
Now you need to reindex all the data and rebuild spellchecker. To rebuild spellchecker,
http://localhost:8983/solr/select?spellcheck.rebuild=true&qt=suggest
To check spell checking is working,
http://localhost:8983/solr/select?q=tes&qt=suggest
This should return some results (replace the query string as per your data). I hope this blog will help some people who are scratching their heads for this full phrase search functionality!. Do let me know if you face any issues.
 

Comments

Ben Marshall (not verified)

Fri, 04/19/2013 - 14:47

I'm fairly new to Drupal so bare with me. I've followed the steps you've outlined above, but get a 500 error on this JS script:

apachesolr_autocomplete?query=china&limit=50&timestamp=1366395939954

with the following message:

Exception: "400" Status: unknown handler: suggest: unknown handler: suggest<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> <title>Error 400 unknown handler: suggest</title> </head> <body><h2>HTTP ERROR 400</h2> <p>Problem accessing /solr/select. Reason: <pre> unknown handler: suggest</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> </body> </html> in DrupalApacheSolrService->checkResponse() (line 368 of /vagrant/public/stratfor.dev/www/sites/all/modules/contrib/apachesolr/Drupal_Apache_Solr_Service.php).

I noticed you had a break tag at the end of line 1 for the suggest searchComponent. Not sure if that should be there but removed it hoping that would fix the problem. No luck. I also deleted the Search & Solr Index a couple of different times and again no luck. I've double checked to make sure I've got what you've got above and all looks good. Any ideas?

I love your blog.. very nice colors & theme. Did you make this website yourself or
did you hire someone to do it for you? Plz respond as I'm looking to design my own blog and would like to know where u got this from. appreciate it

Ajithlal (not verified)

Wed, 02/12/2014 - 07:45

<response><lst name="error"><str name="msg">unknown handler: suggest</str><int name="code">400</int></lst></response>
when i try to rebuild please help

m4ck3r (not verified)

Fri, 03/07/2014 - 11:32

Very good article, certainly works well.. thought had to remove the <br> tag.

Question, how can I add additional fields along the the node title to the autocomplete?

Thank you for taking the time to write this!

Dhirendra Kumar (not verified)

Thu, 06/05/2014 - 06:48

Hi,
I have followed following step but getting error.

An AJAX HTTP error occurred.
HTTP Result Code: 500
Debugging information follows.
Path: http://internal.grazitti.com/drupalcommunity/apachesolr_autocomplete
StatusText: Internal Server Error
ResponseText: Exception: HTTP 400; Bad Request: Bad Request in DrupalApacheSolrService-&gt;checkResponse() (line 455 of /var/www/drupalcommunity/sites/all/modules/apachesolr/Drupal_Apache_Solr_Service.php).

please help.

Jorge (not verified)

Fri, 10/03/2014 - 18:49

I get an error when using the drupal's autocomplete method in the following line:

"An AJAX HTTP error occurred.
HTTP Result Code: 200
Debugging information follows.
Path: http://www.mavericks.com/apachesolr_autocomplete
StatusText: OK
ResponseText:
( ! ) Fatal error: Cannot access empty property in /Applications/MAMP/htdocs/develphase/sites/all/modules/apachesolr_autocomplete/apachesolr_autocomplete.module on line":
$suggestions_data = $response->spellcheck->suggestions->{$q};

What branch/version should I be looking at?