The Secret Actions Behind Full-Text Search

While using full-text search, you may wonder why can’t you find the record you expected. This might happen more often when the field value contains both numerals and texts together. For example, if you have a product code “CHO331344-GERMANY” you will never find this record with fill text search by searching “331344”.

The Fundamentals

In terms of the search engine, Ragic uses the third-party tool “Lucene”, which is an open-source library for full-text search. However, whether you develop your own search engine like Google or you use another third-party tool, their processes are similar. Generally, the system will interpret every word as an index. The search engine will then match the keywords that users enter to the indices in the database.

Now the question is how does the system recognize a word and then create an index? The rules are different for different languages and different search engines. For English and many other languages, the judgment criterion is only by spaces on many search engines. For example, “Ragic Database” will be considered as two words since the text is separated into two parts by a space.

Ragic also only use spaces as the indicator of indices, and not by dashes or other symbols. For example, a code “CHO331344-GERMANY” will only be considered as one single word because it does not have any spaces in between. To add on, since the search engine will only match data when the keyword matches an index completely, you will not be able to find data by simply typing “GERMANY”, “331344” or “CHO331344” because you have to enter the complete product code. Just like you won't be able to find "apple" by keying-in "pp" in most search engines. However, if you change the product code to “CHO 331344 GERMANY” by adding spaces to separate the code into three indices, you can enter the record after searching "CHO", "331344", or "GERMANY".

Of course, different software applications may have some special rules to determine what is a word. In Google, no matter what tense or form a word is it always determines them as one word. For example; “score”, “scores”, “scoring”, and “scored” will be judged as the same word.

Search Suggestions

You may now have this question in mind right now: Why can I still find the entry when I only enter a partial string? I thought the keyword I enter must completely match the index?

Well technically, you haven’t matched the index yet. The results that appear below the search engine are called search suggestions. These search suggestions are generated by what you have currently entered and guessed the index with it to match a specific record. Ragic’s search engine will display three suggested keywords and a maximum of five related entries with the partial data you have entered and mark the matching partial data in red. You can then click on them or use hotkeys to select the keyword or entry. Basically, search suggestions are there for you to find data more easily.

However, search suggestions are not search results. Therefore, If you press the enter key with only the partial data in the search engine, the system will not return anything unless another field value has indices matching your partial data. Taking “CHO331344-GERMANY” as an example, even if you see a few search suggestions under the search engine when you enter “CHO331”, the system will return nothing if you only simply press the enter key.

Also, search suggestions will only generate when the text you enter in the search engine matches partial data of an index from the beginning. Referring once more to the “CHO331344-GERMANY” example, you won’t be able to find this record under search suggestions when you enter “GERMANY” in the search box since “GERMANY” is not the beginning of the index. However, you might still have a chance to find the record with the "GERMANY" part of the product code highlighted. This is because the text you entered may have matched other field values, and highlighted the other partial values with another logic. For “CHO331344-GERMANY”, the search suggestions may have highlighted “GERMANY” when you enter that because ”GERMANY” is also the field value of country of origin, and the product code is then highlighted with another logic.

Other Frequently Asked Questions

What if I only recall partial strings of text?

You can try filtering through the left sidebar or field headers on the listing page. To narrow it down further, you can select one or more fields to filter data if necessary.

Is there a way to apply syntax while searching? For example, finding CHOCOLATE data that is not WHITE CHOCOLATE?

When querying in Ragic, you can write some syntax in the top search bar to combine multiple terms and use boolean operators to form a more complex query to find specific entries, which is similar to the google syntax. Click here for more details.

Are there certain strings of text that are not supported by full-text search?

Yes, stop words will be removed by the search engines automatically during the search process. Stop words are the common short function words in a language. In English, words such as the, is, at, which, and on are some stop words filtered out when using full-text search.

My Full-Text Search seems to be broken! I can’t find the records!

The system will rebuild your search engine index once in a while. You might not be able to find your record via the search engine right after creating a new record. If needed, you can manually rebuild the search engine index.

Additionally, since full-text search currently only searches the content of Form Pages, if a field on a Form Page is set as Hidden, even if the field is configured to be displayed on the Listing Page, the content of that field cannot be searched using full-text search. Therefore, if data cannot be found, you can check whether the data field has been hidden on the Form Page.