Code to fetch BitBucket repos

Question

This actually works nicely (we have > 2k repos) but I would really like it to start producing results right away as soon as it has any. Mostly so it seems faster than it is (by being more responsive).

   bbrepos(){
        #clone[1] is ssh using a filter made the escaping really ugly. 
        #If it becomes necessary we can use a jq arg
        # get max number of repos, ceil(repos / 50), then create a page 
        #sequence and curl them all
        local url=https://api.bitbucket.org/2.0/repositories/twengg
        local project=${1:+"q=project.key=\"$1\""}
        curl -snL  "$url?pagelen=0&page=1" | jq '"$(((\(.size) / 50) + 1 ))"' \
            | xargs bash -c 'eval "echo $1"' _ | xargs seq 1 | xargs -L 1 printf "?pagelen=50&page=%s&${1:-$project}\n" \
            | xargs -I {} -L 1 -P 20 -I {} bash -c 'curl -snL  "$1/$2" | jq -er .values[].links.clone[1].href' _ $url {} \ #clone[1] is ssh
            | sort -u
    }

Sample workflow:

bbsearch something | xargs -P 20 git clone

Alternatively, it might also be handy to have these other workflows:

bbsearch something other thing | xargs -P 20 git clone

cat things-to-find.txt | bbsearch | xargs -P 20 git clone

Sample payload (it's normally huge, so I stripped it down to what is needed):

{
  "pagelen": 1,
  "size": 3054,
  "values": [
    {
      "links": {
        "clone": [
          {
            "href": "https://[email protected]/twengg/development-process.git",
            "name": "https"
          },
          {
            "href": "[email protected]:twengg/development-process.git",
            "name": "ssh"
          }
        ],
        "self": {
          "href": "https://api.bitbucket.org/2.0/repositories/twengg/development-process"
        }
      }
    }
  ],
  "page": 1,
  "next": "https://api.bitbucket.org/2.0/repositories/twengg?page=2"
}

Please describe the flow appropriately and add a sample fragment of .values[].links structure — RomanPerekhrest
– RomanPerekhrest, Commented Nov 19, 2019 at 12:25
This could be a great question if it described the work flow. Who doesn't want to speed up their workflow. A real example might also be nice. — pacmaninbw
– pacmaninbw ♦, Commented Nov 19, 2019 at 15:28

janos · Accepted Answer · 2019-11-28 19:42:56Z

`eval` is evil

Always look for ways to avoid eval.

Relevant snippet from the posted code:

curl -snL  "$url?pagelen=0&page=1" \
    | jq '"$(((\(.size) / 50) + 1 ))"' \
    | xargs bash -c 'eval "echo $1"' _

The code uses jq to create a formatted expression to evaluate by Bash, with .size injected in the middle of the expression.

A better way to achieve the same thing would be to make jq simply output the variable itself, and then write the expression in Bash:

curl -snL  "$url?pagelen=0&page=1" \
    | jq -r .size \
    | xargs bash -c 'echo $((($1 / 50) + 1))' _

Unnecessary pipeline

It's strange to use xargs to process one line of input, and it's strange to spawn another Bash just to evaluate an expression. There's really no need to force a chain of operations in a single pipeline.

I'd rewrite the above snippet like this:

repos_count=$(curl -snL "$url?pagelen=0&page=1" | jq -r .size)
((pages_count = (repos_count / 50) + 1))

Store positional arguments in variables with descriptive names

It's a mental burden to have to remember what $1 and $2 mean. I'd store them in local variables with descriptive names at the beginning of the function.

Avoid `seq`

Is it obsolete, and not available in all systems. Use a Bash counting loop instead:

for ((page = 1; page <= pages_count; page++)); do
    ...
done \
| xargs -L 1 ...

Break lines in long pipelines

It's easier to read code when there's one statement per line. This is especially true when a line is so long you have to scroll to the right to see it. So instead of this:

| xargs bash -c 'eval "echo $1"' _ | xargs seq 1 | xargs -L 1 printf "?pagelen=50&page=%s&${1:-$project}\n" \
| xargs -I {} -L 1 -P 20 -I {} bash -c 'curl -snL  "$1/$2" | jq -er .values[].links.clone[1].href' _ $url {} \ #clone[1] is ssh
| sort -u

I recommend to write like this:

| xargs bash -c 'eval "echo $1"' _ \
| xargs seq 1 \
| xargs -L 1 printf "?pagelen=50&page=%s&${1:-$project}\n" \
| xargs -I {} -L 1 -P 20 -I {} bash -c 'curl -snL  "$1/$2" \
| jq -er .values[].links.clone[1].href' _ $url {} \
| sort -u

Always double-quote variables used as command line arguments

In | jq -er .values[].links.clone[1].href' _ $url {} you forgot to double-quote $url.

This is good feedback and I will incorporate it - however, it doesn't address how I get this to output immediately more input comes — Christian Bongiorno
– Christian Bongiorno, Commented Dec 7, 2019 at 20:11

Stack Exchange Network

Code to fetch BitBucket repos

1 Answer 1

`eval` is evil

Unnecessary pipeline

Store positional arguments in variables with descriptive names

Avoid `seq`

Break lines in long pipelines

Always double-quote variables used as command line arguments

You must log in to answer this question.

Hot Network Questions

Code to fetch BitBucket repos

1 Answer 1

eval is evil

Unnecessary pipeline

Store positional arguments in variables with descriptive names

Avoid seq

Break lines in long pipelines

Always double-quote variables used as command line arguments

You must log in to answer this question.

Related

Hot Network Questions

`eval` is evil

Avoid `seq`