Comments on: A fast(er) Python interleaver, and a simple parallel sorting script.

By: Paddy3118

Paddy3118 — Sat, 12 Jul 2008 05:45:30 +0000

Oh.

By the way, I was thinking of using split as it is already available, and its option:

-C, –line-bytes=SIZE put at most SIZE bytes of lines per output file

- Paddy.

By: slacy

slacy — Fri, 11 Jul 2008 21:00:55 +0000

There’s actually a very interesting reason why you need to interleave and not just take the first 25%, second 25%, etc.

I’m going to elaborate on this in a subsequent post, but the underlying reason is that you don’t get any parallelism due to the unbuffered nature of unix FIFOs. (In fact, it may even deadlock, but I’m not positive)

By: Paddy3118

Paddy3118 — Fri, 11 Jul 2008 16:13:47 +0000

Neat,
Can’t wait to get delivery of a five year old works quad chip server so I can try things like this.

Question: why interleave rather than first 25%, second 25% … into a,b,c,d?

- Paddy.