Long Range Context Memory for Long Text

Segment-level recurrent mechanism This preserves the context information of past consecutive text by storing in the memory and reusing them while processing the next text segment. $$ S_(t): Previous segment S_(t+1): Current segemnt L : length of each sequence D: hidden dimension of the model n: transformer layer number h^(n-1)(t) \epsilon R^(LxD) : Hidden state of the previous segment at layer n-1 h^(n-1)(t+1) \epsilon R^(LxD) : Hidden state of the current segment at layer n-1 ...