FEATURE: Uppy direct S3 multipart uploads in composer (#14051)

This pull request introduces the endpoints required, and the JavaScript functionality in the `ComposerUppyUpload` mixin, for direct S3 multipart uploads. There are four new endpoints in the uploads controller:

* `create-multipart.json` - Creates the multipart upload in S3 along with an `ExternalUploadStub` record, storing information about the file in the same way as `generate-presigned-put.json` does for regular direct S3 uploads
* `batch-presign-multipart-parts.json` - Takes a list of part numbers and the unique identifier for an `ExternalUploadStub` record, and generates the presigned URLs for those parts if the multipart upload still exists and if the user has permission to access that upload
* `complete-multipart.json` - Completes the multipart upload in S3. Needs the full list of part numbers and their associated ETags which are returned when the part is uploaded to the presigned URL above. Only works if the user has permission to access the associated `ExternalUploadStub` record and the multipart upload still exists.

  After we confirm the upload is complete in S3, we go through the regular `UploadCreator` flow, the same as `complete-external-upload.json`, and promote the temporary upload S3 into a full `Upload` record, moving it to its final destination.
* `abort-multipart.json` - Aborts the multipart upload on S3 and destroys the `ExternalUploadStub` record if the user has permission to access that upload.

Also added are a few new columns to `ExternalUploadStub`:

* multipart - Whether or not this is a multipart upload
* external_upload_identifier - The "upload ID" for an S3 multipart upload
* filesize - The size of the file when the `create-multipart.json` or `generate-presigned-put.json` is called. This is used for validation.

When the user completes a direct S3 upload, either regular or multipart, we take the `filesize` that was captured when the `ExternalUploadStub` was first created and compare it with the final `Content-Length` size of the file where it is stored in S3. Then, if the two do not match, we throw an error, delete the file on S3, and ban the user from uploading files for N (default 5) minutes. This would only happen if the user uploads a different file than what they first specified, or in the case of multipart uploads uploaded larger chunks than needed. This is done to prevent abuse of S3 storage by bad actors.

Also included in this PR is an update to vendor/uppy.js. This has been built locally from the latest uppy source at d613b849a6. This must be done so that I can get my multipart upload changes into Discourse. When the Uppy team cuts a proper release, we can bump the package.json versions instead.

app/controllers/uploads_controller.rb

@@ -9,14 +9,30 @@ class UploadsController < ApplicationController
   protect_from_forgery except: :show
 
   before_action :is_asset_path, :apply_cdn_headers, only: [:show, :show_short, :show_secure]
-  before_action :external_store_check, only: [:show_secure, :generate_presigned_put, :complete_external_upload]
+  before_action :external_store_check, only: [
+    :show_secure,
+    :generate_presigned_put,
+    :complete_external_upload,
+    :create_multipart,
+    :batch_presign_multipart_parts,
+    :abort_multipart,
+    :complete_multipart
+  ]
+  before_action :direct_s3_uploads_check, only: [
+    :generate_presigned_put,
+    :complete_external_upload,
+    :create_multipart,
+    :batch_presign_multipart_parts,
+    :abort_multipart,
+    :complete_multipart
+  ]
+  before_action :can_upload_external?, only: [:create_multipart, :generate_presigned_put]
 
   SECURE_REDIRECT_GRACE_SECONDS = 5
-  PRESIGNED_PUT_RATE_LIMIT_PER_MINUTE = 5
-
-  def external_store_check
-    return render_404 if !Discourse.store.external?
-  end
+  PRESIGNED_PUT_RATE_LIMIT_PER_MINUTE = 10
+  CREATE_MULTIPART_RATE_LIMIT_PER_MINUTE = 10
+  COMPLETE_MULTIPART_RATE_LIMIT_PER_MINUTE = 10
+  BATCH_PRESIGN_RATE_LIMIT_PER_MINUTE = 10
 
   def create
     # capture current user for block later on
@@ -193,15 +209,21 @@ class UploadsController < ApplicationController
   end
 
   def generate_presigned_put
-    return render_404 if !SiteSetting.enable_direct_s3_uploads
-
     RateLimiter.new(
       current_user, "generate-presigned-put-upload-stub", PRESIGNED_PUT_RATE_LIMIT_PER_MINUTE, 1.minute
     ).performed!
 
     file_name = params.require(:file_name)
+    file_size = params.require(:file_size).to_i
     type = params.require(:type)
 
+    if file_size_too_big?(file_name, file_size)
+      return render_json_error(
+        I18n.t("upload.attachments.too_large", max_size_kb: SiteSetting.max_attachment_size_kb),
+        status: 422
+      )
+    end
+
     # don't want people posting arbitrary S3 metadata so we just take the
     # one we need. all of these will be converted to x-amz-meta- metadata
     # fields in S3 so it's best to use dashes in the names for consistency
@@ -225,33 +247,37 @@ class UploadsController < ApplicationController
       key: key,
       created_by: current_user,
       original_filename: file_name,
-      upload_type: type
+      upload_type: type,
+      filesize: file_size
     )
 
     render json: { url: url, key: key, unique_identifier: upload_stub.unique_identifier }
   end
 
   def complete_external_upload
-    return render_404 if !SiteSetting.enable_direct_s3_uploads
-
     unique_identifier = params.require(:unique_identifier)
     external_upload_stub = ExternalUploadStub.find_by(
       unique_identifier: unique_identifier, created_by: current_user
     )
     return render_404 if external_upload_stub.blank?
 
-    raise Discourse::InvalidAccess if external_upload_stub.created_by_id != current_user.id
-    external_upload_manager = ExternalUploadManager.new(external_upload_stub)
+    complete_external_upload_via_manager(external_upload_stub)
+  end
 
+  def complete_external_upload_via_manager(external_upload_stub)
+    external_upload_manager = ExternalUploadManager.new(external_upload_stub)
     hijack do
       begin
         upload = external_upload_manager.promote_to_upload!
         if upload.errors.empty?
-          external_upload_manager.destroy!
+          external_upload_stub.destroy!
           render json: UploadsController.serialize_upload(upload), status: 200
         else
           render_json_error(upload.errors.to_hash.values.flatten, status: 422)
         end
+      rescue ExternalUploadManager::SizeMismatchError => err
+        debug_upload_error(err, "upload.size_mismatch_failure", additional_detail: err.message)
+        render_json_error(I18n.t("upload.failed"), status: 422)
       rescue ExternalUploadManager::ChecksumMismatchError => err
         debug_upload_error(err, "upload.checksum_mismatch_failure")
         render_json_error(I18n.t("upload.failed"), status: 422)
@@ -270,6 +296,179 @@ class UploadsController < ApplicationController
     end
   end
 
+  def create_multipart
+    RateLimiter.new(
+      current_user, "create-multipart-upload", CREATE_MULTIPART_RATE_LIMIT_PER_MINUTE, 1.minute
+    ).performed!
+
+    file_name = params.require(:file_name)
+    file_size = params.require(:file_size).to_i
+    upload_type = params.require(:upload_type)
+    content_type = MiniMime.lookup_by_filename(file_name)&.content_type
+
+    if file_size_too_big?(file_name, file_size)
+      return render_json_error(
+        I18n.t("upload.attachments.too_large", max_size_kb: SiteSetting.max_attachment_size_kb),
+        status: 422
+      )
+    end
+
+    begin
+      multipart_upload = Discourse.store.create_multipart(
+        file_name, content_type
+      )
+    rescue Aws::S3::Errors::ServiceError => err
+      debug_upload_error(err, "upload.create_mutlipart_failure")
+      return render_json_error(I18n.t("upload.failed"), status: 422)
+    end
+
+    upload_stub = ExternalUploadStub.create!(
+      key: multipart_upload[:key],
+      created_by: current_user,
+      original_filename: file_name,
+      upload_type: upload_type,
+      external_upload_identifier: multipart_upload[:upload_id],
+      multipart: true,
+      filesize: file_size
+    )
+
+    render json: {
+      external_upload_identifier: upload_stub.external_upload_identifier,
+      key: upload_stub.key,
+      unique_identifier: upload_stub.unique_identifier
+    }
+  end
+
+  def batch_presign_multipart_parts
+    part_numbers = params.require(:part_numbers)
+    unique_identifier = params.require(:unique_identifier)
+
+    RateLimiter.new(
+      current_user, "batch-presign", BATCH_PRESIGN_RATE_LIMIT_PER_MINUTE, 1.minute
+    ).performed!
+
+    part_numbers = part_numbers.map do |part_number|
+      validate_part_number(part_number)
+    end
+
+    external_upload_stub = ExternalUploadStub.find_by(
+      unique_identifier: unique_identifier, created_by: current_user
+    )
+    return render_404 if external_upload_stub.blank?
+
+    if !multipart_upload_exists?(external_upload_stub)
+      return render_404
+    end
+
+    presigned_urls = {}
+    part_numbers.each do |part_number|
+      presigned_urls[part_number] = Discourse.store.presign_multipart_part(
+        upload_id: external_upload_stub.external_upload_identifier,
+        key: external_upload_stub.key,
+        part_number: part_number
+      )
+    end
+
+    render json: { presigned_urls: presigned_urls }
+  end
+
+  def validate_part_number(part_number)
+    part_number = part_number.to_i
+    if !part_number.between?(1, 10000)
+      raise Discourse::InvalidParameters.new(
+        "Each part number should be between 1 and 10000"
+      )
+    end
+    part_number
+  end
+
+  def multipart_upload_exists?(external_upload_stub)
+    begin
+      Discourse.store.list_multipart_parts(
+        upload_id: external_upload_stub.external_upload_identifier, key: external_upload_stub.key
+      )
+    rescue Aws::S3::Errors::NoSuchUpload => err
+      debug_upload_error(err, "upload.external_upload_not_found", { additional_detail: "path: #{external_upload_stub.key}" })
+      return false
+    end
+    true
+  end
+
+  def abort_multipart
+    external_upload_identifier = params.require(:external_upload_identifier)
+    external_upload_stub = ExternalUploadStub.find_by(
+      external_upload_identifier: external_upload_identifier
+    )
+
+    # The stub could have already been deleted by an earlier error via
+    # ExternalUploadManager, so we consider this a great success if the
+    # stub is already gone.
+    return render json: success_json if external_upload_stub.blank?
+
+    return render_404 if external_upload_stub.created_by_id != current_user.id
+
+    begin
+      Discourse.store.abort_multipart(
+        upload_id: external_upload_stub.external_upload_identifier,
+        key: external_upload_stub.key
+      )
+    rescue Aws::S3::Errors::ServiceError => err
+      debug_upload_error(err, "upload.abort_mutlipart_failure", additional_detail: "external upload stub id: #{external_upload_stub.id}")
+      return render_json_error(I18n.t("upload.failed"), status: 422)
+    end
+
+    external_upload_stub.destroy!
+
+    render json: success_json
+  end
+
+  def complete_multipart
+    unique_identifier = params.require(:unique_identifier)
+    parts = params.require(:parts)
+
+    RateLimiter.new(
+      current_user, "complete-multipart-upload", COMPLETE_MULTIPART_RATE_LIMIT_PER_MINUTE, 1.minute
+    ).performed!
+
+    external_upload_stub = ExternalUploadStub.find_by(
+      unique_identifier: unique_identifier, created_by: current_user
+    )
+    return render_404 if external_upload_stub.blank?
+
+    if !multipart_upload_exists?(external_upload_stub)
+      return render_404
+    end
+
+    parts = parts.map do |part|
+      part_number = part[:part_number]
+      etag = part[:etag]
+      part_number = validate_part_number(part_number)
+
+      if etag.blank?
+        raise Discourse::InvalidParameters.new("All parts must have an etag and a valid part number")
+      end
+
+      # this is done so it's an array of hashes rather than an array of
+      # ActionController::Parameters
+      { part_number: part_number, etag: etag }
+    end.sort_by do |part|
+      part[:part_number]
+    end
+
+    begin
+      complete_response = Discourse.store.complete_multipart(
+        upload_id: external_upload_stub.external_upload_identifier,
+        key: external_upload_stub.key,
+        parts: parts
+      )
+    rescue Aws::S3::Errors::ServiceError => err
+      debug_upload_error(err, "upload.complete_mutlipart_failure", additional_detail: "external upload stub id: #{external_upload_stub.id}")
+      return render_json_error(I18n.t("upload.failed"), status: 422)
+    end
+
+    complete_external_upload_via_manager(external_upload_stub)
+  end
+
   protected
 
   def force_download?
@@ -339,6 +538,25 @@ class UploadsController < ApplicationController
 
   private
 
+  def external_store_check
+    return render_404 if !Discourse.store.external?
+  end
+
+  def direct_s3_uploads_check
+    return render_404 if !SiteSetting.enable_direct_s3_uploads
+  end
+
+  def can_upload_external?
+    raise Discourse::InvalidAccess if !guardian.can_upload_external?
+  end
+
+  # We can pre-emptively check size for attachments, but not for images
+  # as they may be further reduced in size by UploadCreator (at this point
+  # they may have already been reduced in size by preprocessors)
+  def file_size_too_big?(file_name, file_size)
+    !FileHelper.is_supported_image?(file_name) && file_size >= SiteSetting.max_attachment_size_kb.kilobytes
+  end
+
   def send_file_local_upload(upload)
     opts = {
       filename: upload.original_filename,
@@ -357,8 +575,8 @@ class UploadsController < ApplicationController
     send_file(file_path, opts)
   end
 
-  def debug_upload_error(translation_key, err)
+  def debug_upload_error(err, translation_key, translation_params = {})
     return if !SiteSetting.enable_upload_debug_mode
-    Discourse.warn_exception(err, message: I18n.t(translation_key))
+    Discourse.warn_exception(err, message: I18n.t(translation_key, translation_params))
   end
 end