Keep dong Sequence.GetValues(), it is usually plenty fast not to be the bottleneck. This returns a new ValueSet for each record, and if you want to keep a ValueSet for a particular record, you can, calling Fetch() / GetValues() on the sequence that produced your ValueSet will not invalidate it. The reason this all is pretty fast is that all data that gets allocated for each new record is managed and while there is some overhead for re-creating the same structure of returned fields over and over, the bulk of the work is in copying / wrapping / unwrapping the actual values which come from the table. In addition, if you don't keep any ValueSet's and just do: fetch record - call GetValues - process returned values - stop using returned values - fetch another record - call GetValues again - process new returned values - etc, which is by far the most frequent scenario, the costs of setting up a new ValueSet every time nearly disappear because objects are all short-lived and memory just gets reused. If you try keeping ValueSet's for past records, the picture starts changing, and at some point (it is hard to give specific numbers because they depend on the number of fields, their types, and plenty of other things, but we are likely talking about something like 5,000 ValueSet's or more before this even appears on the radar) the overhead from having many separate ValueSet's starts becoming noticeable, and then you should consider storing things in plain arrays of ints / strings / whatever.
A sequence cannot rewind, it can only go forward. If you want to rewind, you have to store previous values.