'sed' not replacing text in file

Question

I am attempting to replace tokens in a JSON file with values held in our CI tool of choice (Bamboo).

To do this, I am using jq to extract an array of values, and then replacing the token in each value with its corresponding Bamboo variable value.

However, something is not working. The output on the scrip confirms that the replacement token is being found, and that the Bamboo variable value is correct.

What am I doing wrong?

ENV_FILE_PATH="./landmark-accounts-uat.postman_environment.json"
VALUES=$(cat ./landmark-accounts-uat.postman_environment.json | jq -c '.values[]')
for ELEMENT in $VALUES; do

        ### Extract the variable name and replacement token (e.g. '#{varName}') for the current element.
        ELEMENT_NAME=$(echo $ELEMENT | jq -r '.key')
        ELEMENT_REPLACEMENT_TOKEN=$(echo $ELEMENT | jq -r '.value')

        ### Construct the name of the Bamboo variable to use for replacement.
        BAMBOO_VARIABLE_NAME=bamboo\_$ELEMENT_NAME
        BAMBOO_VARIABLE_VALUE=${!BAMBOO_VARIABLE_NAME}

        ### Replace the current value with the value of the Bamboo variable.
        echo "Replacing '$ELEMENT_REPLACEMENT_TOKEN' with the value of Bamboo variable '$BAMBOO_VARIABLE_NAME' ($BAMBOO_VARIABLE_VALUE)."
        sed -i 's/$ELEMENT_REPLACEMENT_TOKEN/$BAMBOO_VARIABLE_VALUE/g' $ENV_FILE_PATH
done

Example JSON file

Below is an example of the JSON file before it's been edited with sed.

As you can see, there are replacement tokens for each value, and it is these that I am trying to replace with Bamboo variables. In this case, those varaibles would be called $bamboo_apiUrl and $bamboo_apimKey.

{
  "id": "b10a67a9-75a1-49fc-8b6d-9063388ed35c",
  "name": "my-env",
  "values": [
    {
      "key": "apiUrl",
      "value": "#{apiUrl}",
      "enabled": true
    },
    {
      "key": "apimKey",
      "value": "#{apimKey}",
      "enabled": true
    },
  ],
  "_postman_variable_scope": "environment",
  "_postman_exported_at": "2019-09-27T09:13:43.208Z",
  "_postman_exported_using": "Postman/7.8.0"
}

Update

Based on a suggestion in the comments, this is the solution that I am using.

#!/bin/bash

### Read in the environment file in full.
ENV_FILE_PATH="./landmark-accounts-uat.postman_environment.json"
ENVIRONMENT_FILE=$(cat $ENV_FILE_PATH)

### Get all of the value keys from the environment file.
KEYS=$(cat $ENV_FILE_PATH | jq --raw-output '.values[].key')

### Iterate value keys...
for KEY in $KEYS; do
        ### Construct the name of the Bamboo variable to use for 'value' for the current 'key', then get the value of that variable.
        BAMBOO_VARIABLE_NAME=bamboo\_$KEY
        BAMBOO_VARIABLE_VALUE=${!BAMBOO_VARIABLE_NAME}

        ### Replace the value for the currently 'key'.
        echo "Replacing value of key '$KEY' with the value of Bamboo variable '$BAMBOO_VARIABLE_NAME' ($BAMBOO_VARIABLE_VALUE)."
        ENVIRONMENT_FILE=$(echo $ENVIRONMENT_FILE | jq --arg key "$KEY" --arg value "$BAMBOO_VARIABLE_VALUE" '.values |= map(if .key == $key then .value = $value else . end)')
done

### Output the updated environment file.
echo $ENVIRONMENT_FILE > $ENV_FILE_PATH

You should really consider using jq directly for the replacements - something like jq --arg key "$key" --arg value "$value" '.values |= map( if .key == $key then .value = $value else . end )' as described here: jq: Modify properties on selected objects — steeldriver, Nov 21 '19 at 13:17
As this behaviour is special to the shell you should mention it in the question. You tagged [bash] and the behaviour of dash (= sh on Ubuntu) is the exact same in this regard, but still. — dessert, Nov 21 '19 at 13:17
@steeldriver - I was not aware that I could do that with JQ - I will certainly look in to it. — David Gard, Nov 21 '19 at 16:11

dessert · Answer 1 · 2019-11-21T13:04:08.630

You quoted variables with single quotes, but:

Enclosing characters in single quotes preserves the literal value of each character within the quotes.

You don’t want that, you want $ to have its special meaning to the shell in order to evaluate the variables. Thus, use double quotes instead:

sed -i "s/$ELEMENT_REPLACEMENT_TOKEN/$BAMBOO_VARIABLE_VALUE/g" $ENV_FILE_PATH

In fact, neither of “s”,“/” or “g” requires quoting, so depending on the variable content you may even let the whole expression unquoted. However, it’s always advisable to quote variables.

You can easily test the effects of your quoting by prepending echo, just add these two lines and compare the output:

echo sed -i 's/$ELEMENT_REPLACEMENT_TOKEN/$BAMBOO_VARIABLE_VALUE/g' $ENV_FILE_PATH
echo sed -i "s/$ELEMENT_REPLACEMENT_TOKEN/$BAMBOO_VARIABLE_VALUE/g" $ENV_FILE_PATH

'sed' not replacing text in file

Example JSON file

Update

1 Answers1

Further reading