Rails Large File Upload

Problem

Recently I've been working with a project that in need to allow user to upload large video file to Rails server. Handling file upload in Rails is easy, but that only if a file is small. Try uploading a file that is over 1GB in size and BOOM!. You will most likely got an 413 (request entity is too large) or even if you manage to get it working your web will hang for over a long period of time, which will annoy the hell our of your user. That is because the time it took for transfering the file from client to server over HTTP is too long. One way to solve this problem is to upload file in smaller size multiple times and combine that small uploaded files back into one when the upload process is complete. To do that we need a way to split file into smaller chunks and make requests to server from client side. Javascript has a FileReader API which we can use to read file and split it into smaller chunks, but its support for browsers is not that great. Fortunately jQuery has a file upload plugin that can handle this fairly well and with a decent support for browsers as well.

Introducing jQuery File Upload Plugin

jQuery file upload plugin has fully feature that we can work with file upload like multiple files upload, drag & drop, validation, image & video preview, chunks and resumable file uploads as well. In this article we only need chunks file upload support. You can find source code for this post on Github.

Essentially, what we are going to do is to create a simple upload form which allow user to select a file to upload and provide a progress bar as a feedback to the user while a file is uploading. So first let take a look the code for client side.

<!-- uploads/new.html.erb -->
<%= form_tag uploads_url, multipart: true, id: 'fileupload' do %>
  <%= label_tag 'File:' %>
  <%= file_field_tag :upload %>
  <%= submit_tag 'Upload' %>
<% end %>

<div id="progress-bar">
  <div id="progress"></div>
</div>

This is our view code for upload form and a container for progress bar. Next is javascript code to handle spliting file into smaller chunks ready for upload. Notice the maxChunkSize option, without this the file will upload normally as a whole. In the add callback we loop through each selected files (in case that multiple files upload is enable) and make post request to /uploads which will create a new Upload record and response back id and uploaded_size as a JSON object. This is needed because this value will be used to send along as a post data when we start upload the chunk files.

var files = [];
$('#fileupload').fileupload({
  dataType: 'json',
  url: '/chunk_upload',
  maxChunkSize: 1000000,
  add: function(e, data) {
    $.each(data.files, function(index, file) {
      $.ajax({
        method: 'post',
        dataType: 'json',
        url: '/uploads',
        data: { filename: file.name },
        success: function(res) {
          data.formData = res;
          files.push(data);
        }
      });
    });
  },
  // more code
});

During uploading process a progressall callback with be called. We can use that to calculate how many percent has the upload done and provide a progress bar as a feedback to user.

$('#fileupload').fileupload({
  // same code as above
  progressall: function(e, data) {
    var done = parseInt(data.loaded * 100) / data.total
    $('#progress').css({ width: done + '%'})
  }
});

Last but not least is a snippet to upload file when user click on Upload button. The data that passed into add callback above has a method called submit to start uploading process.

$('#fileupload').on('submit', function(e) {
  e.preventDefault();
  if (files.length < 1) return;

  $.each(files, function(index, file) {
    file.submit();
  });

  files = [];
});

Now that we are done with the client, lets take a look how we handle uploaded file on the server side. The trick is the two step process, first we create an upload record in create action which generate a uniqe uuid that use as a filename to create path column for upload record. Remember on the client side code that we passed along an id as post data, well in chunk_create action we query for a upload record that we just created in create action using that id params. After that we increase the uploaded_size for upload record and save it back. This column is useful when we need the resumable feature, but for now just ignore it. Next if the update succeed we append the content read from chunk file into the file with path from path column of upload record, otherwise reponse with an error. With this after the upload complete we will have a single file save into path specified in path field of our upload record object.

class UploadsController < ApplicationController
  def create
    filename = params[:filename]
    uuid = SecureRandom.uuid
    ext  = File.extname(filename)
    dir  = Rails.root.join('tmp', 'upload').to_s
    FileUtils.mkdir_p(dir) unless File.exist?(dir)

    @upload = Upload.new(
      filename: filename,
      path: File.join(dir, "#{uuid}#{ext}")
    )

    if @upload.save
      render json: { id: @upload.id, uploaded_size: @upload.uploaded_size }
    else
      render json: { error: @upload.errors }
    end
  end

  def chunk_create
    file    = params[:upload]
    @upload = Upload.find_by(id: params[:id])
    @upload.uploaded_size += file.size

    if @upload.save
      File.open(@upload.path, 'ab') { |f| f.write(file.read) }
      render json: { id: @upload.id, uploaded_size: @upload.uploaded_size }
    else
      render json: { error: @upload.errors }, status: 422
    end
  end
end

Conclusion

Using jQuery file upload is great it offer rice feature that we can work with file upload, but not without drawn back. Upload file in chunks mean your server need to handle multiple request (depending on the size of a file) for a single upload. If you found a better solution don't hesitate to leave a comment below to let me know.