Unable to list objects in bucket with only bucket policy permissions

Problem

When one Ceph user wants to give another Ceph user READ access to one of his buckets, the grantee of the permission should be able to list all the objects in the bucket via the S3 interface as well as directly accessing each object directly.
With only setting bucket policy permissions it's not possible to list the objects in a bucket as one gets an AccessDenied error even when setting the correct bucket policy. When setting the less flexible ACL READ permission on the bucket than one can list all objects in the bucket.

Usecase

When creating a bucket or giving another user permission for the bucket, the user should have full READ access to the objects in the selected bucket. Bucket Policy permissions are a more flexible way to realize this than ACLs because one can add conditions to a bucket policy, e.g. a time restriction.

Background

As stated under the subparagraph Mapping of ACL permissions and access policy permissions in the AWS S3 documentation, the three bucket policy permissions s3:ListBucket, s3:ListBucketVersions, and s3:ListBucketMultipartUploads should have the same effect as giving the user READ ACL permission on the bucket.
The Ceph documetation on the supported bucket policy permissions shows that all the permissions are supported by Ceph since the version Luminous.

Minimal working example

Prerequisite

• Install the boto3 package with pip: pip install --user boto3
  
• Create two users in Ceph and their S3 keys.
  
• Fill in the appropriate values in the VARIABLES section at the beginning of the script.

Script

import boto3
import random
import string
from pathlib import Path
import json

# VARIABLES
# ========================================
# Keys for user 1
access_key_user1 = "<Access Key User 1>"
secret_key_user1 = "<Secret Key User 1>"

# Keys and username for user 2
access_key_user2 = "<Access Key User 2>"
secret_key_user2 = "<Secret Key User 2>"
username_user2 = "<Username User 2>"
# =========================================


# Create S3 interface for user 1
s3_bucket_owner = boto3.resource(service_name="s3", endpoint_url='http://192.168.192.102:8000', aws_access_key_id=access_key_user1, aws_secret_access_key=secret_key_user1, verify=False)

# Create a bucket for user 1
bucket_owner = s3_bucket_owner.Bucket("test-" + "".join(random.choices(string.ascii_lowercase, k=4)))
bucket_owner.create()

# Upload an object into the bucket
obj_owner = bucket_owner.put_object(Key="helloworld.txt", Body="Hello World".encode("utf-8"))

# Set the bucket policy such that user 2 can directly acess the objects and list the objects in the bucket
bucket_policy = json.dumps(
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "PseudoOwnerPermission",
                "Effect": "Allow",
                "Principal": {"AWS": [f"arn:aws:iam::usfolks:user/{username_user2}"]},
                "Action": ["s3:GetObject", "s3:GetObjectVersion", "s3:ListBucket", "s3:ListBucketVersions", "s3:ListBucketMultipartUploads"],
                "Resource": [f"arn:aws:s3:::{bucket_owner.name}/*", f"arn:aws:s3:::{bucket_owner.name}"],
            }
        ],
    }
)
s3_bucket_owner.BucketPolicy(bucket_owner.name).put(Policy=bucket_policy)

# Create s3 interface for user 2
s3_permission_grantee = boto3.resource(service_name="s3", endpoint_url='http://192.168.192.102:8000', aws_access_key_id=access_key_user2, aws_secret_access_key=secret_key_user2, verify=False)
# Access the bucket as user 2
bucket_grantee = s3_permission_grantee.Bucket(bucket_owner.name)

# Directly access the object in the bucket -> event this won't work
obj_grantee = s3_permission_grantee.ObjectSummary(obj_owner.bucket_name, obj_owner.key)
print("Directly getting the object:")
try:
    print(obj_grantee)
    print("Get Metadata:")
    print(obj_grantee.size)
except Exception as e:
    print(e)

# List all objects in the bucket -> this throws an AccessDenied Error
print("\nListing all objects and size metadata in bucket with 'only' Bucket Policy:")
try:
    print([(obj.key, obj.size) for obj in bucket_grantee.objects.all()])
except Exception as e:
    print(e)


# Setting the bucket ACL as the owner such that user 2 has READ permission for the bucket
bucket_acl = s3_bucket_owner.BucketAcl(bucket_owner.name)
bucket_acl.put(
    AccessControlPolicy={
        "Grants": bucket_acl.grants
        + [
            {
                "Grantee": {
                    "DisplayName": "Display Name",
                    "ID": username_user2,
                    "Type": "CanonicalUser",
                },
                "Permission": "READ",
            }
        ],
        "Owner": bucket_acl.owner,
    }
)
# Setting the ACL for the created object to access the metadata and the object itself
object_acl = s3_bucket_owner.ObjectAcl(bucket_owner.name, obj_owner.key)
object_acl.put(
    AccessControlPolicy={
        "Grants": object_acl.grants
        + [
            {
                "Grantee": {
                    "DisplayName": "Display Name",
                    "ID": username_user2,
                    "Type": "CanonicalUser",
                },
                "Permission": "READ",
            }
        ],
        "Owner": object_acl.owner,
    }
)

# List all objects in the bucket again -> this works
print("\nListing all objects and size metadata in bucket with Bucket and Object 'READ' ACL:")
try:
    print([(obj.key, obj.size) for obj in bucket_grantee.objects.all()])
except Exception as e:
    print(e)


# Cleanup: Delete object and bucket
obj_owner.delete()
_ = bucket_owner.delete()