23. How to Store files for the Web: Best Practices for Image processing on the Backend. Thumbor. S3

Video version
(Leave your feedback on YouTube)

TL;DR

A few tips for saving files:

NEVER store user files on your server. Use S3 services. It is cheaper, reduces server load, and many S3 services have CDNs with servers around the world.

S3 is not necessarily Amazon. Choose any service as long as it supports the S3-compatible API. This API is supported by most libraries and allows for easy migration. All "aws" settings in the package can be replaced with any S3.

S3 can be deployed on your own infrastructure... if needed.

Use S3 on the staging server as well!

For local development, use local storage.

Users love to upload the same files multiple times. This increases the required storage, and in the case of images, it interferes with caching. This can be prevented by storing a unique file identifier and not uploading the file again. For example, using the function hash('sha512', $file->getContent());

Create a minimal number of API methods for saving files. And use the file identifier for saving. This greatly simplifies file storage and functionality testing!

Images on the Frontend

A good practice is to display optimized images tailored to the user's browser and screen resolution. For this, there is the picture tag. The main rules for working with it are:

source elements are processed in the order they are described. The first allowed image from top to bottom will be loaded.
type - the MIME type of the image supported by the browser. Currently, the recommended priority from lightest to heaviest is ~~avif~~, webp, jpeg
media - the media query, usually concerned with width.
img - required, where we put the original image, alt, and it is recommended to use the original image for detailed viewing (if needed).
avif format is objectively the smallest, but it requires a lot of resources to decode on the client side, which is a significant issue for mobile devices. Therefore, it is often ignored.

<picture>
    <!-- WebP format for better performance and quality -->
    <source srcset="images/example-mobile.webp" media="(max-width: 600px)" type="image/webp">
    <source srcset="images/example-tablet.webp" media="(max-width: 1024px)" type="image/webp">
    <source srcset="images/example-desktop.webp" media="(min-width: 1025px)" type="image/webp">

    <!-- Fallback to JPG format -->
    <source srcset="images/example-mobile.jpg" media="(max-width: 600px)" type="image/jpeg">
    <source srcset="images/example-tablet.jpg" media="(max-width: 1024px)" type="image/jpeg">
    <source srcset="images/example-desktop.jpg" media="(min-width: 1025px)" type="image/jpeg">

    <!-- Fallback image for browsers that do not support <picture> -->
    <img src="images/original.png" alt="Description of the image">
</picture>

For static images, resizing and formats are handled by the frontend. For backend images, request URLs for images with different resolutions from backend developers.

Resize/Optimize Images on Backend

Image resizing depends on the type of project. In many cases, images can be resized when saving, but don't forget to keep the original.

The best option is to keep the original and resize or change the image format upon request. There are many services that implement this - thumbor, imgproxy, picfit, imaginary, and others.

A few tips for choosing:

Mandatory protection against URL Tampering
Mandatory support for S3 storage.
Optional caching of results, preferably to S3. Although most solutions claim they are very fast and don't need caching, it's better to slightly increase the cheap S3 storage than to waste server resources during high traffic.
And, of course, check the activity on Git and find benchmarks for comparison.

My choice is thumbor. And the ready-made docker build with S3 support.

The configuration for local development looks like this:

  thumbor:
   image: beeyev/thumbor-s3:7.7-slim-alpine
   restart: unless-stopped
   container_name: bh-thumbor
   environment:
     - 'ALLOW_UNSAFE_URL=False'
     - 'SECURITY_KEY=${THUMBOR_SECURITY_KEY}'
     - 'USE_GIFSICLE_ENGINE=True'
     - 'AUTO_WEBP=True'
     - 'AUTO_JPG=True'
   ports:
     - 8888:8888
   volumes:
     - thumbor_data:/data
   networks:
     - bh-thumbor-network
   healthcheck:
     test: curl -s http://localhost:8888 >/dev/null || exit 1
     interval: 3s
     timeout: 10s
     retries: 3

  volumes:
    thumbor_data:
      driver: local
      
  networks:
    bh-thumbor-network:
      driver: bridge
      name: bh-thumbor-network

Note the following:

ALLOW_UNSAFE_URL=False and SECURITY_KEY=${THUMBOR_SECURITY_KEY} - always encrypt URLs. Attackers can easily run a script iterating through width from 1 to 100000, and height, and fill your cache storage with billions of images in seconds. Read about protection here. For PHP, there is a handy package.
USE_GIFSICLE_ENGINE - gif images are unique. If they are in comments, their format should not be changed, they should always "play." However, for avatars, animation disrupts the design code, so I recommend taking only the first frame in such cases. For this, there is a cover filter.
AUTO_WEBP, AUTO_JPG - automatic conversion to webp and jpg with the Accept header set to image/jpg, image/webp respectively. This header is generated by the browser automatically. This allows you to forget about the type in the HTML picture tag and generate URLs only for media.
bh-thumbor-network - for local development, the service must have access to images served by nginx, i.e., be in the same network with it.

Don't forget about the configuration:

WEBSERVER_URL=http://bh-webserver
THUMBOR_SERVER_URL=http://localhost:8888
THUMBOR_SECURITY_KEY=3333
THUMBOR_FILESYSTEM_DISK='thumbor'
FILESYSTEM_DISK=public

And I add a new disc to storage. I use it for thumbor access to the nginx container. In production, it will be S3.

 [
    'disks' => [
      'thumbor' => [
               'driver' => 'local',
               'root' => storage_path('app/public'),
               'url' => env('WEBSERVER_URL').'/storage',
               'visibility' => 'public',
               'throw' => false,
           ],
    ]    
]

And a small service that generates a convenient source for the frontend.

<?php

namespace Modules\Attachment\Services;

use Illuminate\Support\Collection;
use Illuminate\Support\Facades\Storage;
use Modules\Attachment\Enums\ScreenSize;
use Modules\Attachment\Models\Attachment;

class ImageResizer
{
    private Collection $sources;
    private string $src;

    public function __construct(
        Attachment $image,
        private readonly bool $extractCover = false,
    ) {
        $this->sources = collect();
        $disk = config('thumbor.filesystems_disk');
        $this->src = Storage::disk($disk)->url($image->path);
    }

    public function addSource(
        int $width,
        ?ScreenSize $media = null
    ): self {
        $source = compact('width', 'media');
        $this->sources->push((object) $source);

        return $this;
    }

    public function getSources(): Collection
    {
        return $this->sources->map(function ($source) {
            $srcset = \Thumbor::resizeOrFit($source->width);

            if ($this->extractCover) {
                $srcset->addFilter('cover');
            }

            return (object) [
                'media' => $source->media,
                'srcset' => $srcset->get($this->src),
            ];
        });
    }
}

<?php

namespace Modules\Attachment\Tests\Unit\Services;

use Modules\Attachment\Enums\AttachmentExtension;
use Modules\Attachment\Enums\ScreenSize;
use Modules\Attachment\Models\Attachment;
use Modules\Attachment\Services\ImageResizer;
use Tests\TestCase;

class ImageResizerTest extends TestCase
{
    /**
     * @test
     */
    public function canGetResizedImagesForAllSources()
    {
        $image = Attachment::factory()->create([
            'extension' => AttachmentExtension::PNG,
        ]);
        $resizer = new ImageResizer($image);

        $resizer
            ->addSource(
                width: 41,
                media: ScreenSize::Mobile
            )
            ->addSource(
                width: 43,
                media: ScreenSize::TabletUp
            );

        $result = $resizer->getSources();

        $this->assertCount(2, $result);
    }

    /**
     * @test
     */
    public function canGetRightSource()
    {
        $path = 'https://some-image-sorce.test/img.png';

        $image = Attachment::factory()->create([
            'path' => $path,
            'extension' => AttachmentExtension::PNG,
        ]);

        $resizer = new ImageResizer($image);

        $width = 411;
        $media = ScreenSize::Mobile;
        $resizer
            ->addSource(
                width: $width,
                media: $media
            );

        $source = $resizer->getSources()->first();

        $this->assertEquals($media, $source->media);
        $this->assertStringContainsString($width, $source->srcset);
        $this->assertStringContainsString($path, $source->srcset);
        $this->assertStringNotContainsString('cover', $source->srcset);
    }

    /**
     * @test
     */
    public function canSetExtractCoverFilter()
    {
        $image = Attachment::factory()->create([
            'extension' => AttachmentExtension::GIF,
        ]);

        $resizer = new ImageResizer(
            image: $image,
            extractCover: true
        );

        $width = 411;
        $media = ScreenSize::Mobile;
        $resizer
            ->addSource(
                width: $width,
                media: $media
            );

        $source = $resizer->getSources()->first();

        $this->assertStringContainsString('cover', $source->srcset);
    }
}

The service is called in resources as needed. If the same type of image has several variations, these are different resource files!

<?php

namespace Modules\Attachment\Transformers;

use Illuminate\Http\Request;
use Illuminate\Http\Resources\Json\JsonResource;
use Illuminate\Support\Collection;
use Illuminate\Support\Facades\Storage;
use Modules\Attachment\Enums\ScreenSize;
use Modules\Attachment\Models\Attachment;
use Modules\Attachment\Services\ImageResizer;
use OpenAPI\Properties\PropertyResourceCollection;
use OpenAPI\Properties\PropertyString;
use OpenAPI\Schemas\BaseScheme;

#[BaseScheme(
    resource: AvatarResource::class,
    properties: [
        new PropertyString('src'),
        new PropertyResourceCollection(
            property: 'sources',
            resource: ImageSourceResource::class,
        ),
    ]
)]
/**
 * @mixin Attachment
 */
class AvatarResource extends JsonResource
{
    public function toArray(Request $request): array
    {
        return [
            'src' => Storage::url($this->path),
            'sources' => ImageSourceResource::collection($this->getSources()),
        ];
    }

    /**
     * @infection-ignore-all
     */
    private function getSources(): Collection
    {
        $resizer = new ImageResizer(
            image: $this->resource,
            extractCover: true
        );
        $mobileSize = 36;
        $tabletUpSize = 45;

        $resizer
            ->addSource(
                width: $mobileSize,
                media: ScreenSize::Mobile
            )
            ->addSource(
                width: $tabletUpSize,
                media: ScreenSize::TabletUp
            );

        return $resizer->getSources();
    }
}

Help to survive