If you look at the testing results, you will see that indeed this is not a lot of hardware for the size of the indexes we need. With the testing hardware we were not able to sustain a throughput of 2 queries per second and prior to the implementation of CommonGrams we had queries that took 2 minutes.
Not really overkill
If you look at the testing results, you will see that indeed this is not a lot of hardware for the size of the indexes we need. With the testing hardware we were not able to sustain a throughput of 2 queries per second and prior to the implementation of CommonGrams we had queries that took 2 minutes.
http://www.hathitrust.org/blogs/large-scale-search/update-testing-memory-and-load-tests
For comparison, take a look at our current production hardware, which is significantly more powerful than what we used in testing.
In particular we are using more RAM and significantly faster storage.
http://www.hathitrust.org/blogs/large-scale-search/new-hardware-searching-5-million-volumes-full-text
Tom