how to run multiple curl requests in parallel with multiple variables?

Using GNU Parallel it looks something like this to fetch 100 entries in parallel:

#!/bin/bash
set -eu

user="user"
pwd="pwd"
D1=$(date "+%Y-%m-%d" -d "1 days ago")
D2=$(date "+%Y-%m-%d" -d "1 days ago")
curr=$D2
cheese=$(pwd)

curl -o /dev/null -s -S -L -f -c cookiejar 'https://url/auth/' -d name=$user -d passwd=$pwd

fetch_one() {
    client="$1"
    account="$2"
    accountid="$3"
    platform="$4"
    platformid="$5"

    curl -o /dev/null -s -S -f -b cookiejar -c cookiejar 'https://url/auth/' -d account=$accountid
    curl -sSfL -o "$client€[email protected]$platform£$curr.xlsx" -J -b cookiejar -c cookiejar "https://url/platform=$platformid&date=$curr"
}
export -f fetch_one

while true; do
    cat account-list.tsv | parallel -j100 --colsep '\t' fetch_one
    [ "$curr" \< "$D1" ] || break
    curr=$( date +%Y-%m-%d --date "$curr +1 day" ) ## used in instances where I need to grade data for past date ranges.
done

exit

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top