1

I have this scenario where I need to measure the similarity between a 2d tensor t1: (100,8) and 61 tensors of the same shape(100,8). 100 represent time-steps and 8 is the no. of options.

I first tried flattening all tensors so that I can use the cosine similarity measure, but it slows down performance significantly.

I used pairwise comparisons without flattening like the approach below, but it made executing mathematical operations more complicated (e.g., diagonal operations): How to calculate a meaningful distance between multidimensional tensors.

What should I do to speed-up ops on flattened tensors/or are there alternatives to flattening?

Vilainor
  • 21
  • 3

1 Answers1

2

I suggest you to flatten and use FAISS. I had the same problem but using faiss it was made simple it is creating indexing which is faster to compare. Using faiss indexing i am able to compare 1 vector with 21 million vector in 6 milliseconds.

Hiren Namera
  • 406
  • 5
  • 14
  • Thank you for your comment. I tried FAISS based on you suggestion, but it does not seem to support incremental addition of training samples. Based on your experience, is there a possible workaround for such a scenario or should I look for something different? – Vilainor Sep 30 '22 at 19:05