zhangtaii
3/21/2018 - 2:15 PM

Batch query API endpoint, feed cURL from loop with range and step, gzip decompress

Batch query API endpoint, feed cURL from loop with range and step, gzip decompress

batch_curl.sh
content_copyfile_download
#!/bin/bash

for offset in `seq 0 50 5000`; do
      curl -H 'accept-encoding: br, gzip, deflate' "${API_ENDPOINT}/?offset=${offset}&limit=50" | gunzip - > "outputs/${offset}.json"
done