2

I have this bash script, which is generating every single combination of arrays in 5 digit long:

#!/usr/bin/env bash

for combo in \
{a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z,0,1,2,3,4,5,6,7,8,9}\
{a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z,0,1,2,3,4,5,6,7,8,9}\
{a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z,0,1,2,3,4,5,6,7,8,9}\
{a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z,0,1,2,3,4,5,6,7,8,9}\
{a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z,0,1,2,3,4,5,6,7,8,9};
do echo $combo;
done > output.txt

This will output:

$ ./script.sh
aaaaa
aaaab
aaaac
...

However, this is just freeze my computer. Is there any way to write the data into the file increasingly? Also, is it possible to separate the output to different files for example at every 65536 lines? like output1.txt, output2.txt, output3.txt?

Thank You for your help!

Improve this question

5 Answers 5

Reset to default
6

It's hard to generate those big permutation without holding up huge memory in shell.

You can try another language like Python:

$ python -c '
  import itertools, string
  l = [c for c in string.ascii_lowercase + string.digits]
  for p in itertools.product(l, l, l, l, l): print "".join(p)
' > file
Improve this answer
5

You can use nested loops, though it will be slow:

for p in {a..z} {0..9}
do for q in {a..z} {0..9}
 do for r in {a..z} {0..9}
  do for s in {a..z} {0..9}
   do for t in {a..z} {0..9}
    do echo $p$q$r$s$t
    done
   done
  done
 done 
done | split -l 65536 --numeric-suffixes=1 --additional-suffix=.txt - output
Improve this answer
3

I found a nice tool called crunch which is written in C. You also have options to separate the output files by file size or lines!

Install: $ sudo apt-get install crunch

#!/usr/bin/env bash

crunch 5 5 abcdefghijklmnopqrstuvwxyz0123456789 -o output/START -c 1048576 # Excel line limit

This command in the output folder generates aaaaa-awrdd.txt, awrde-bi8gh.txt, bi8gi-b5pjl.txt, etc. files exactly with 1.048.576 lines, which is the Excel 2007+ line limit. It's also a super fast solution. generating of 60.000.000+ combinations in 58 files taken only 10s! Perfect!

Improve this answer
3
  • cool! (+1) what is the meaning of " ... 5 5 ..." ? Commented Nov 18, 2015 at 11:25
  • 1
    min-length and max-length of the output. of I use 4 5, than it will output aaaa and aaaaa variations too. But I like to separate into different files by length. Commented Nov 18, 2015 at 11:39
  • One thing to make sure when you define the list of characters, use this format: The order MUST BE lower case characters, upper case characters, numbers, and then symbols. If you don't follow this order you will not get the results you want. You also have to escape special characters. More info $ man crunch Commented Nov 18, 2015 at 11:45
2

For permutations, I use the following perl script to generate permutations:

#!/usr/bin/perl
use strict;
use Algorithm::Permute;

my $q=shift or die("usage permgen n word-set\n");

my $p = new Algorithm::Permute([@ARGV],$q);
while (my @res = $p->next) { 
  print @res,"\n"; }

This can be used to make permutations of very different sets. Example unicode dices:

permgen 2  ⚀ ⚁ ⚂ ⚃ ⚄ ⚅ | shuf -n 2
⚅⚄
⚅⚁

To generate your (HUGE) example we need to write: permgen 5 {a..z} {0..9}

In my old machine it takes 2s to generate 1_000_000 permutations

time permgen 5 {a..z} {1..9}| head -1000000| wc

Improve this answer
2

bash will always allocate all elements in RAM before iterating over them with the for loop. You're effectively asking bash to allocate lots of RAM at once and your computer just doesn't have that much, that's why it crashes/freezes.

The combinations themselves don't require that much (my back of the envelope computations give around 500MB), but I suppose bash has a massive overhead when dealing with arrays.

I wonder how much free RAM you really have.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.