Side effects and evaluation

LINQ pipelines are lazy and, by convention, pure. Sometimes you have to step outside that: run an action for the elements, look at what flows through the middle of a pipeline, force evaluation at a specific point, or make sure an expensive source is evaluated only once. These operations do that explicitly, so the place where the evaluation or the side effect happens is visible in the code rather than hidden in a foreach.

The distinction to keep in mind is eager versus lazy. ForEach, Materialize and Shuffle evaluate the input when they are called. Inspect, InspectEmpty and Memoize do nothing until the result is enumerated.

ForEach

Runs an action for every element, immediately.

Unit ForEach<TSource>(this IEnumerable<TSource> source, Action<TSource> action)
Unit ForEach<TSource>(this IEnumerable<TSource> source, Func<TSource, Unit> action)

List<T> has a ForEach method; this one works on any IEnumerable<T>. It is eager: the whole sequence is enumerated during the call, which is the difference from Inspect. It returns Unit rather than void, so it can be used as the last expression of an expression-bodied member or a lambda that must return something.

ForEach is the end of a pipeline, not a step in it: everything before it should be pure, and the action is where the side effects happen. If the loop body builds up a value, that is an Aggregate, not a ForEach with a captured variable.

Example

// Before
foreach (var order in orders)
{
    Ship(order);
}

// After
orders.ForEach(Ship);

// As an expression-bodied member
public Unit ShipAll(IEnumerable<Order> orders)
    => orders.ForEach(Ship);

Inspect

Runs an action for every element as it passes through, and yields the element unchanged.

IEnumerable<TSource> Inspect<TSource>(this IEnumerable<TSource> source, Action<TSource> inspector)

Inspect is a Select that returns its input, with a side effect on the way. It is lazy: nothing runs until the result is enumerated, and the inspector then runs for each element at the moment that element is requested, interleaved with everything downstream. This makes it the tool for looking into the middle of a pipeline, for example to log, count or assert, without changing its structure.

The same name exists on Option<T>, Result<T> and Either<L, R> with the same meaning: observe, then hand back unchanged.

Example

return customers
    .Where(c => c.IsActive)
    .Inspect(c => logger.LogDebug("active customer {Id}", c.Id))
    .Select(c => c.Email);

Deferred execution

Because Inspect is deferred, where its side effect happens depends on who enumerates the result and when:

Enumerable.Range(1, 100)
    .Inspect(n => Console.WriteLine($"before where: {n}"))
    .Where(n => n % 2 == 0)
    .Inspect(n => Console.WriteLine($"after where: {n}"))
    .Take(2)
    .ToList();   // the side effects happen here

// before where: 1
// before where: 2
// after where: 2
// before where: 3
// before where: 4
// after where: 4

Only four elements were inspected, because Take(2) stopped asking. Replace ToList() with nothing, and nothing is printed at all. See the Microsoft documentation on deferred execution for the general rules.

InspectEmpty

Runs an action if the sequence turns out to be empty, and yields the elements unchanged otherwise.

IEnumerable<TSource> InspectEmpty<TSource>(this IEnumerable<TSource> source, Action inspector)

A marble diagram showing the InspectEmpty operation

This is the counterpart of Inspect for the case Inspect can never see: a sequence with no elements. Like Inspect it is lazy; the action runs during enumeration, at the point where the first element was requested and the source reported that there is none. It corresponds to InspectNone on Option<T>.

Example

var results = search(query)
    .InspectEmpty(() => logger.LogInformation("no results for {Query}", query))
    .Select(FormatResult);

Materialize

Evaluates a lazy sequence into a collection, unless it already is one.

IReadOnlyCollection<TSource> Materialize<TSource>(this IEnumerable<TSource> source)
IReadOnlyCollection<TSource> Materialize<TSource, TMaterialization>(this IEnumerable<TSource> source, Func<IEnumerable<TSource>, TMaterialization> materializer)
    where TMaterialization : IReadOnlyCollection<TSource>

A method that receives an IEnumerable<T> and needs to enumerate it twice, or needs its Count, usually calls ToList(). That copies the input even when the caller already passed a list. Materialize only does work when it has to: an IReadOnlyCollection<T> is returned as is, an IList<T> or ICollection<T> is wrapped without copying, and only a genuinely lazy sequence is evaluated, by default into an ImmutableList<T>.

The second overload lets you choose the collection for the lazy case, for example Materialize(Enumerable.ToArray) or Materialize(Enumerable.ToHashSet). Note that it is only used when the input is not already a collection.

The result is IReadOnlyCollection<T>, which is the honest type: you can count it and enumerate it any number of times. Because an existing collection is returned or wrapped rather than copied, the result is a live view: later changes to the caller's collection show through, and the wrapper itself cannot be used to modify the source. Materialize guarantees that a lazy sequence is evaluated only once, not that you hold a snapshot. Call ToList() or ToImmutableList() when you need one.

Example

void Report(IEnumerable<Order> orders)
{
    var materialized = orders.Materialize();

    Console.WriteLine($"{materialized.Count} orders");
    foreach (var order in materialized) { ... }   // no second evaluation of a lazy query
}

Memoize

Makes a sequence re-enumerable without evaluating it more than once: elements are produced lazily from the source the first time they are requested, and served from a buffer every time after.

IBuffer<TSource> Memoize<TSource>(this IEnumerable<TSource> source)

A marble diagram showing the Memoize operation

Materialize evaluates everything up front. Memoize sits between that and a plain lazy sequence: the source is still pulled one element at a time, so an infinite or expensive source is only evaluated as far as any consumer has looked, but every element is evaluated once, and a second enumeration replays the buffer instead of the source. This is what you want when a sequence is expensive to produce and consumed by several operators, or when a query is enumerated an unknown number of times.

The result is an IBuffer<T>, which is an IEnumerable<T> that is also IDisposable, because it owns the enumerator of the source. Dispose it when you are done, typically with using; enumerating a disposed buffer throws ObjectDisposedException. Memoizing a sequence that is already a list, a collection or a buffer does not copy it.

Several consumers may enumerate the buffer concurrently: the source is still pulled one element at a time and every element is produced once. The source is not touched before the first element is requested, and its enumerator is disposed as soon as it is exhausted.

The IAsyncEnumerable variant returns an IAsyncBuffer<T> with the same contract. The cancellation token passed to GetAsyncEnumerator, for example through WithCancellation, cancels that consumer's enumeration only.

Example

using var primes = Sequence.Successors(2, p => Option.Some(NextPrime(p))).Memoize();

var firstTen = primes.Take(10).ToList();
var below100 = primes.TakeWhile(p => p < 100).ToList();   // reuses the first ten, computes the rest

using var rows = ReadRowsFromDatabase().Memoize();

var total = rows.Sum(r => r.Amount);
var largest = rows.MaxByOrNone(r => r.Amount);   // no second database round trip

Shuffle

Returns the elements in random order.

IReadOnlyList<TSource> Shuffle<TSource>(this IEnumerable<TSource> source)
IReadOnlyList<TSource> Shuffle<TSource>(this IEnumerable<TSource> source, Random random)

A marble diagram showing the Shuffle operation

Shuffle is eager: it materializes the input and shuffles it in linear time with a Fisher–Yates shuffle, which gives every permutation the same probability. OrderBy(_ => random.Next()) is the common alternative; it is slower and, with a poorly chosen key, biased. Pass a Random to get reproducible results in tests or to use a seeded generator; its state is advanced by the call. Without one, the random numbers come from a shared generator (Random.Shared on .NET 6 and later, one generator per thread on older frameworks), so two shuffles in a row never repeat the way two new Random() instances created in the same tick do on .NET Framework.

.NET 10 added Shuffle to LINQ as a lazy operator. On that target the parameterless Shuffle() resolves to the BCL method, while the overload taking a Random stays Funcky's.

Example

var deck = cards.Shuffle();

var fixedOrder = cards.Shuffle(new Random(42));   // deterministic

Funcky User Guide