Fix issue presigning request with custom headers, also fix small deviation from spec related to lower case headers

[Napam]

Problem description

I found a problem related to adding custom signed headers. The header I want to sign is the Content-Length, this is to ensure no one uploads over whatever limit I set from the backend.

When trying to upload to Cloudflare R2 I get

<Error>
    <Code>
         SignatureDoesNotMatch
    </Code>
    <Message>
        The request signature we calculated does not match the signature you provided. Check your secret access key and signing method
    </Message>
   .
   .
   .

I have cross-tested this against the PresignHTTP function provided in "github.com/aws/aws-sdk-go-v2/aws/signer/v4", and this produces a valid URL when the custom header is specified.

Fix part 1:

The fix here is to always URI encode the extra headers, that is remove the if condition here:

-		// X-Amz-SignedHeaders already has properly formatted semicolons, retain as is.
-		if k == HdrXAmzSignedHeaders {
-			h.Write([]byte(queryString[k]))
-		} else {
-			h.Write([]byte(awsURIEncode(queryString[k])))
-		}
+		h.Write([]byte(awsURIEncode(queryString[k])))

I struggled to find anywhere in the spec that declared that value of X-Amz-SignedHeaders should be aws-uri-encoded, but it doesn't say anywhere that it shouldn't. So I dug through the AWS SDK source code (line 185), I see that they unconditionally do the aws-uri-encoding on all headers. So I suggest we just do the same. It makes the code a little bit easier after all.

Difficulty testing

I wrote a new integration test trying to test this aginst the local Minio instance but it just failed. After som reading around I conclude that Minio just doesn't handle custom signed headers very well? Anyways, so I tested it against my Cloudflare R2 storage, and it works perfectly.

This implies that you have to setup other AWS credentials to test it, I just put in under a guard:

		if os.Getenv("TEST_REAL_S3") != "true" {
			t.Skip("Skipping AWS S3 integration test. Set TEST_REAL_S3=true to run.")
		}

Assuming you have setup AWS env vars to something that isn't Minio, you can do:

TEST_REAL_S3=true go test -v -run 'TestS3_GeneratePresignedURL_ExtraHeader' to run the integration test.

Other than that, the whole test suite passes as before 😄

Fix part 2:

When looking into this I uncovered a deviation from the spec that states:

CanonicalHeaders is a list of request headers with their values. Individual header name and value pairs are separated by the newline character ("\n"). Header names must be in lowercase. You must sort the header names alphabetically to construct the string

So I just added that as well.

-		signedHeaders[k] = []byte(v)
+		// AWS requires header names to be lowercase per spec
+		signedHeaders[strings.ToLower(k)] = []byte(v)