R2 rejects CompleteMultipartUpload with "InvalidPart: All non-trailing
parts must have the same length" unless every part except the last has
the same size — a rule AWS/minio/rustfs don't enforce, which is why
live-to-rec worked locally against rustfs but failed against R2.
Two paths violated it:
- S3Uploader flushed whatever segments had accumulated past the 5MB
minimum, so part sizes tracked segment sizes. It now slices its buffer
at exact liveUploadPartSize (5MB) boundaries, holding the remainder
for the next flush; only an object's final part is short.
- ConcatWithHeader emitted one UploadPartCopy part per source object
(plus a ~5MB header part), so part sizes tracked object sizes. It now
slices the destination byte stream into uniform concatPartSize (16MB)
windows: a window that falls entirely inside one source object is a
server-side UploadPartCopy, and only windows containing the header or
an object boundary are downloaded and re-uploaded (~one window per
source object). Windows upload concurrently (copyConcurrency), like
multipartCopy already did.
A side effect of the windowing is that sub-5MB middle objects no longer
need a full-rewrite fallback — they just ride inside a mixed window —
so ErrConcatPartTooSmall and the fallback branch in assembleContentBlob
are gone. (MultipartWriter and multipartCopy already produced uniform
parts, which is why VOD uploads were unaffected.)
The fake S3 in concat_test now enforces R2's uniform-length rule next
to AWS's 5MB floor, and pkg/s3 gains env-gated live tests
(TestLiveS3Uploader / TestLiveConcatWithHeader) that run the real
uploader and concat against any S3-compatible endpoint:
set -a; . ~/r2.env; go test -count=1 -run TestLive -v ./pkg/s3
Both pass against R2 and rustfs, with sizes chosen to exercise real
copy windows and unequal part sizes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>