Project Description
Sequential Data Cache is a fast collection with limited memory footprint.
It is a must have tool for Linq operations on big collection.

Sequential Data Cache

Sequential Data Cache is a file based generic collection makes possible to handle large arrays or collection of items with limited memory footprint.
Sequential Data Cache supports temporary and permanent data access without any data mapping requirement.
It is a light weight solution designed to handle large amount of calculation tables with double arrays.
The library provides handy extensions to easily integrate in any .NET project.

Sample Usage
new temporary collection
var cache = SequentialDataCache<T>.Initialize();
for (int i=0;i<300000;i++)
cache.Add(new T() { });
cache.Close();

new permanent collection
var cache = SequentialDataCache<T>.Initialize(deleteOnClose:false);
for (int i=0;i<300000;i++)
cache.Add(new T() { });
var path = cache.Path;
cache.Close();

//load from existing cache data
var cache = new SequentialDataCache<T>(path);

Notes
The SequentialDataCache can handle any class marked with SerializableAttribute:

[Serializable]
class MyData
{
public Guid Id { get; set; }
public double[,] DataArray { get; set; }
}

Speed
Sequential Data Cache is pretty fast. See the following results: (The SequentialDataCache is filled with 30000 complex items)
  • TestLinqCount: 00:0:00.0000014 (calls Collection.Count() extension)
  • TestIndexAccess500: 00:0:00.0000290 (takes item at index: 500)
  • TestIndexAccessRandom: 00:0:00.0000888 (takes item at random index)
  • TestLinqTake: 00:0:00.0005849 (takes 20 items from the first item)
  • TestLinqSkipTakeParallel: 00:0:00.0026874 (skip 10 item then takes the next 20 items -parallel)
  • TestLinqTake20Parallel: 00:0:00.0030405 (takes 20 items from the first item - parallel)
  • TestIndexAccess0: 00:0:00.0283001 (takes item at index: 0)
  • TestLinqWhereTake100: 00:0:00.0363693 (takes matching items of 100)
  • TestLinqWhereTake100Parallel: 00:0:00.2596769 (takes mathing items of 100)
  • TestLinqSkipTake: 00:0:00.4549824 (skip 10 item then takes next 20)
  • TestLinqAverageDoubleParallel: 00:0:12.4558743 (calls collection.Average on DataArray[0,1] - parallel)
  • TestLinqSumDoubleParallel: 00:0:12.6201508 (calls collection.Sum on DataArray[0,1] - parallel)
  • TestLinqSumDouble: 00:0:13.1091054 (calls collection.Sum on DataArray[0,1])
  • TestLinqAverageDouble: 00:0:14.5263824 (calls collection.Average on DataArray[0,1])
  • TestIncrementTableValuesPerformance: 00:0:48.0430838 (calls DataArray[0..i,0..j]+=1 on all items and returns new Cache)

Conditions: Debug build, Visual Studio Unit test runner. Page size: 10, on 5400rpm notebook hdd
Collection contains 30.000 Data
each Data contains Index[int], Id[Guid] and a TwoDimensionDoubleArray[double[3,1000]]


Release notes
The Sequential Data Cache is on nuget http://nuget.org/packages/SequentialDataCache


Last edited Feb 22, 2013 at 8:34 PM by csizo, version 8