Space-Efficient Indexes for Uncertain Strings

Strings in the real world are often encoded with some level of uncertainty, for example, due to: unreliable data measurements; flexible sequence modeling; or noise introduced for privacy protection. In the character-level uncertainty model, an uncertain string X of length n on an alphabetΣ is a sequence of n probability distributions over Σ. Given an uncertain string X and a weight threshold 1/zϵ(0,1), we say that pattern P occurs in X at position i, if the product of probabilities of the letters of P at positions i,..., i+ |P|-1 is at least 1/z. While indexing standard strings for online pattern searches can be performed in linear time and space, indexing uncertain strings is much more challenging. Specifically, the state-of-the-art index for uncertain strings has O(nz) size, requires O(nz) time and O(nz) space to be constructed, and answers pattern matching queries in the optimal O(m+ [Occl) time, where m is the length of P and |Occ| is the total number of occurrences of P in X. For large n and (moderate) z values, this index is completely impractical to construct, which outweighs the benefit of the supported optimal pattern matching queries. We were thus motivated to design a space-efficient index at the expense of slower yet competitive pattern matching queries. We show that when we have at hand a lower bound ℓ on the length of the supported pattern queries, as is often the case in real-world applications, we can slash the index size and the construction space roughly by ℓ. In particular, we propose an index of Q (nz/ℓ log z) expected size, which can be constructed using Q (nz/ℓ log z) expected space, and supports very fast pattern matching queries in expectation, for patterns of length m ≥ ℓ. We have implemented and evaluated several versions of our index. The best-performing version of our index is up to two orders of magnitude smaller than the state of the art in terms of both index size and construction space, while offering faster or very competitive query and construction times.