• Full Text Search

    By Tommy Valand 1 decade ago

    I saw in the documentation that you have plans for implementing Full Text Search.

    Will it be using the FTS extensions ( http://www.sqlite.org/fts3.html ) syntax?

    Do you know when this will be implemented?

    I have an application that uses a custom search solution that becomes quite slow when there are a lot of results/there are lookups across databases in the result columns. If FTSearch was implemented, I could possibly change the search routine to use DomSQL.

    Anyways, thanks for all the effort you've put into this already! :)

    • Well,not really...

      By Philippe Riand 1 decade ago

      Actually, I have no plan so far,  and this was just an idea in the doc.

      SQLite FTS tables must actually be populated with real data, using INSERT statements, so the index is created. Somehow, the data located within the NSF have to be replicated in this table. SQLite cannot just use the content of the NSF table and search through it. Now, you should be able to create a physical SQLite table and populate it with a simple SQL statement from the NSF table.
      To properly work, this needs:
      - the SQLite table to be persistant. Last time I tried that, I got a crash, so I'm not sure it works
      - add the FTS table DDL in the DB definition

      The technical best solution in your case would be to use a real FT indexer like Lucene fed using a DOTS task monitoring all the server events (document create/update/delete operations), or incrementing the index periodically.

      Or use an higher level indexer like SOLR. The RESt services should make it easier.

      Note that all of this would be a nice project on OpenNTF :-)

      • Thanks for the suggestions

        By Tommy Valand 1 decade ago

        Looking at the web page for Lucene, it seems really nice :)

        Messing with DOTS task seems like a big hurdle to jump over as I've never written a DOTS task. The application I mentioned has readers fields, which I imagine would add to the complexity of a Lucene based full text search engine.

        I think I'm going to do some more experiments with my current knowledge of the Domino API and see if I can get better performance by improving the cache routine. If that fails, then I might study up on DOTS/Lucene.

        Thanks for the suggestions/quick response!