Given the high number of dimensions (100) in our vectors this is where things could start to fall down with our approach. We can also use the techniques outlined above to create a vector for a search query from a user.īut how do we return relevant results based on this search query? We need to be able to find the closest vectors to our search vector. We have now have a list of vectors for each document in our data set. Create a super-fast search index with NMSLIB The output from this will give us a single vector per document in our search engine. Here we are building a fastText model:Ĭonverting our word vectors into a document vector weighted using BM25 In addition to the above they are super simple to implement using the Gensim library. They are easier to interpret due to the fact that a word vector will remain consistent and will not change based on the context of the surrounding text (both an advantage and a disadvantage, more on this later).thousands of documents rather than the many millions typically used to train transformer models). In addition to this they can be trained on relatively small data sets (i.e. Due to the above point they can be trained from scratch on domain specific texts.They are ‘lightweight’ when compared to transformer models in all areas that matter when creating scalable services (model size, training times, inference speed).However they are still relevant today for the following reasons: Since the introduction of sophisticated transformer models like BERT, word vector models can seem quite old fashioned. Create word vectors build a fastText model Running on a Colab notebook, this can process over 1,800 notices a second.
DATA CREATOR SEARCHENGINE CODE
The above code splits our documents into a list of tokens whilst performing some basic cleaning operations to remove punctuation, white space and convert the text to lowercase. Superfast searching of our results using the lightweight and highly efficient Non-Metric Space Library (NMSLIB).We will still be using this algorithm to power our search but we will need apply this to our word vector results. We will train a model on our data set to create vector representations of words (more information on this here). In order to achieve this, we will need to combine a number of techniques: Handle spelling mistakes, typos and previously ‘unseen’ words in an intelligent way.Be orders of magnitude faster than our last implementation, even when searching over large datasets.Be able to scale up to larger datasets (we will be moving to a larger dataset than in our previous example with 212k records but we need to be able to scale to much larger data).Be location aware understand UK postcodes and the geographic relationship of towns and cities in the UK.Return relevant results to a user even if they have not searched for the specific words within these results.This post will describe the process to do this and also provide template code to achieve this on any dataset.īut what do we mean by ‘smart’? We are defining this as a search engine which is able to: In this post, we want to go beyond this and create a truly smart search engine. In the first post within this series, we built a search engine in just a few lines of code which was powered by the BM25 algorithm used in many of the largest enterprise search engines today.
0 kommentar(er)
