Software Developer Blog |

Optimize Docker .NET Core project and docker-compose solution to reduce build image time

2020-03-20T14:00:00+00:00

Docker has built in great cache mechanism, but to be able to use it, you have to understand how it works. Let’s dive into it, to build .NET Core Docker images faster.

How Docker cache works

First we should understand how Docker cache works. This part is described nice in official documentation:

Starting with a base image that is already in the cache, the next instruction is compared against all child images derived from that base image to see if one of them was built using the exact same instruction. If not, the cache is invalidated.

In most cases simply comparing the instruction in the Dockerfile with one of the child images is sufficient. However, certain instructions require a little more examination and explanation.

For the ADD and COPY instructions, the contents of the file(s) in the image are examined and a checksum is calculated for each file. The last-modified and last-accessed times of the file(s) are not considered in these checksums. During the cache lookup, the checksum is compared against the checksum in the existing images. If anything has changed in the file(s), such as the contents and metadata, then the cache is invalidated.

Aside from the ADD and COPY commands, cache checking will not look at the files in the container to determine a cache match. For example, when processing a RUN apt-get -y update command the files updated in the container will not be examined to determine if a cache hit exists. In that case just the command string itself will be used to find a match.

Once the cache is invalidated, all subsequent Dockerfile commands will generate new images and the cache will not be used.

So it always checks if given instruction is started in exactly the same situation as in the past. If it is, it doesn’t start it, but this layer is taken from cache. If there is even one bit changed, for example one copied file differs, then cache is invalidated and instruction is invoked truly.

How to decrease image build time

To build images faster, put everything which will rarely change, above things which change often. For example, when you compile your application inside Dockerfile, then your source files often change, which invalidates cache all the way down. But you probably download some dependencies, for example Nuget packages, which change less often than source code, so you can put it before compilation. This way as long as you stick to existing dependencies, they are taken from cache. When you add some new dependency or change version of existing one, cache is invalidated and packages are downloaded from the Internet (and new cache is created).

Ignore what you don’t need

When you copy files from host to build context, Docker calculates checksum for them. If exactly the same files were copied to exactly the same directory in Docker image, then this layer (as every operation in Dockerfile creates new layer) is taken from cache. So copy only those files which you need, because you can invalidate cache in vain, with files which are not required in your image, for example every Dockerfile, or files generated by your IDE and so on. Remember, that .dockerignore file has to be located in build context root. Example rules:

*/bin
*/obj
.dockerignore // Docker files
.env
.git // Git files
.gitignore // Git files
.gitattributes // Git files
.vs // IDE files
.vscode // IDE files
**/.toolstarget
.idea // IDE files
**/Samples // Additional files
build
Docker // Docker scripts and docker-compose files (in this example in 'Docker' directory, in context root level)
**/Dockerfile // Docker files
**/docker-compose // Docker files
*Test*

Try to create your own cache (yarn build example)

When you know where is your bottleneck and there is no obvious way to reduce it, try to figure out some none obvious solution. For example a few projects in one of products which I maintain, have yarn install and yarn build sewn into csproj file, like this:

<Target Name="PublishRunWebpack" AfterTargets="ComputeFilesToPublish" >
  <Exec WorkingDirectory="$(SpaRoot)" Command="yarn install" />
  <Exec WorkingDirectory="$(SpaRoot)" Command="yarn build" />

but this was extremely slow - it took (in Docker for Windows) around 4 minutes. And web files in those projects were changed rarely, usually C# code got changed, so I wanted to reuse yarn build results. I didn’t find a way to use npm/yarn cache in this situation, so I cached it myself. I moved copying web files to the top of Dockerfile, like this:

WORKDIR /app/WebClient/WebFiles
COPY WebClient/WebFiles . # Taken form cache as long as you don't change files in WebFiles directory
RUN yarn install # Taken form cache as long as you don't change files in WebFiles directory
RUN yarn build # Taken form cache as long as you don't change files in WebFiles directory

and changed csproj like this:

<Target Name="PublishRunWebpack" AfterTargets="ComputeFilesToPublish" Condition=" '$(Configuration)' == 'Release' And !Exists('$(ProjectDir)WebFiles/build') ">
  <Exec WorkingDirectory="$(SpaRoot)" Command="yarn install" />
  <Exec WorkingDirectory="$(SpaRoot)" Command="yarn build" />

So Docker cached yarn build output and it was invalidated only when web files changed. With target condition, C# project reused already existing yarn build output. Of course, because COPY WebClient/WebFiles was on top of my Dockerfile, every time when some web files changed, it invalidated cache all the way down, so if it was changed often, I would do it some different way, but in my case it worked as expected. And I know that in theory incorrect build directory could exist somehow, but it’s only theory, because in Docker situation is repeatable and on developer machines publish wasn’t invoked that way or another, only debug configuration, so in practice it’s not a risk. Maybe with incremental build I could be able to achieve the same with a bit cleaner way - if your case is similar to mine, you can try it.

Reuse what you can

If there is something which you can reuse, do it. For example when I have multiple Docker images in one solution, I build them all at once, instead of building every project in separation. I have observed, that this way it’s faster. For example, I have solution Application.sln which contains 5 executable projects (which are packed to separate Docker images) and additional projects like tests, usage samples and so on. So to lower number of projects to build, I have copied it to Application-minimal.sln and left only necessary projects. Then in every Dockerfile, instead of compiling only one, chosen project, like this: RUN dotnet publish PROJECT_NAME -c Release -o out I compile whole solution like this: RUN dotnet publish Application-minimal.sln -c Release. After compilation, I copy whole build output (with all projects) to final image (see Docker multi-stage build explained), for example this way: COPY --from=build-env /app/build .. Finally, at the very bottom of Dockerfile I invalidate cache specifying which project I want to run:

WORKDIR /app/ProjectA/publish # Pay attention to have correct path, especially if you set custom build output path.
ENTRYPOINT ["dotnet", "ProjectA.dll"]

Everything above is reused from cache, so when you want to build n projects, only first project takes a while to compile, other projects are taken from cache practically whole. In my case I’ve observed that it works faster than doing it in separation (running dotnet publish PROJECTX for every project). It will probably be faster in your case too, but as with every optimization, I encourage to measure it before applying :)

Experiment with caching nuget dependencies

In your developer PC, you have nuget cached locally. But in Docker you have clean situation every time you invoke build, so nuget packages are taken from the internet every time, so it’s good to cache them, because we don’t change project dependencies as often as project source files. When you compile only one project, then situation is clear, from project directory you can run:

COPY *.csproj ./ # Taken form cache as long as you don't change your project file
RUN dotnet restore # Taken form cache as long as you don't change your project file 

# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out

But when you want to restore files for whole solution, then it gets more complicated, because (at the time of writing) you cannot point to .sln file in dotnet restore command, and you cannot point to multiple csproj files neither. Fortunately, you can resolve it in many ways, for example with simple bash script:

#!/bin/bash
for projectName in *.csproj; do # It can be saved as one-linear straight in Dockerfile
	dotnet restore $projectName
done

used this way:

COPY restore-all.sh . # Taken form cache always
RUN chmod +x restore-all.sh
COPY */*.csproj ./ # Taken form cache as long as you don't change your csproj files
RUN ./restore-all.sh # Taken form cache as long as you don't change your csproj files

Alternatively: build image from dedicated container

Some people do not compile application in Dockerfile, but on their own PC and just put build output to image. This is faster, but it has one big downside: you’re not 100% sure that image is properly built, and generally speaking how it was made, so I’ll never wont to put such image on production. But I’ve also heard about another solution - to create container dedicated to create input to Docker image… Sounds a bit like overengineering, and probably in many cases it is, but maybe sometimes it’s worth doing it. You have more options in container, for example you can mount directories, so there is no problem with any cache you need. You can even use already existing cache from host machine. I’ve never tried such approach, above methods work well for me, but maybe sometimes it’s worth to organize CI/CD pipeline this way.

Summary

Proper caching in Dockerfile is quite broad subject, so here I only touched it in generals. The point of this article is not only to give you some ready to use tricks (like those with yarn or Nuget packages), but also to show you how cache in Docker works and encourage you to investigate your own situation. Pay attention to every build layer, measure which step takes too much time and where is your bottleneck. Sometimes with one line moved in Dockerfile, or one file added to .dockerignore you can make (really) big difference. And sometimes with small investigation you can resolve huge problem, for example .dockerignore file stored somewhere else than build context root directory.
If you have feeling that cache caused some problem (which in theory may be the case, but I never had such situation) you can always check how Docker builds with no-cache parameter, which forces to build everything from scratch.

I hope the article was helpful for you, please publish it on Facebook or Twitter and of course ask me anything, which is still not clear. If you know some other trick related to cache, please share it through comments, someone for sure will appreciate it :)

Initialize MS SQL in Docker container - create database at startup

2019-12-27T06:00:00+00:00

Microsoft SQL Server is available for Linux so we can run it from Docker container, but usually we need to initialize database at startup, which currently is a bit tricky.

If you have seen my post which explains Docker compose, then you know that MySQL team has prepared very convenient way to run initialization scripts when DB has started. You only need to pass your script to proper directory and MySQL will run it when it’s successfully stared, so you can for example create database.

Unfortunately Microsoft SQL Server currently doesn’t have such a mechanism, and official documentation redirects to only one demo, which is coupled with node.js and without node it doesn’t work.

So I have created simplified demo, and here will explain how it works step by step.

Running MS SQL Server from Docker container without initialization

First let’s take a look at official information of how to start Microsoft SQL Server in container:

docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong(!)Password' -e 'MSSQL_PID=Express' -p 1433:1433 -d mcr.microsoft.com/mssql/server:2017-latest-ubuntu 

I prefer to manage base image and environment variables in text file, so let’s put all above to Dockerfile:

FROM mcr.microsoft.com/mssql/server:2017-latest-ubuntu

ENV SA_PASSWORD yourStrong(!)Password
ENV ACCEPT_EULA Y
ENV MSSQL_PID Express

Now we can add this file to Git,
build this image - docker build -t db-demo .
and run container in simpler way:

docker run -p 1433:1433 -d db-demo

This way we run clean MS SQL server.
We can additionally add EXPOSE 1433 entry to Dockerfile, to be able to access it from composed containers.

We should also use explicit image version (not latest) to be sure that this Dockerfile will always be valid (new versions will not break down Docker build or runtime initialization in container). So for example let’s choose version: FROM mcr.microsoft.com/mssql/server:2017-CU17-ubuntu (all tags list is usually given in Docker Hub image description).

How to initialize MS SQL server in Docker container

Now it’s time to add extra stuff to have initialization in our server.

The hack is to run at the same time SQL Server and script which will sleep 90seconds (to be sure that server already started) then do whole initialization (we’ll see how). It’s not my idea, this is solution currently recommended by Microsoft and nothing better comes to my mind neither.

First we override Dockerfile CMD entry from base image (which runs SQL server), with our own CMD command, which runs our custom script:

CMD /bin/bash ./entrypoint.sh

and this entrypoint.sh script looks like this:

# Run Microsoft SQl Server and initialization script (at the same time)
/usr/src/app/run-initialization.sh & /opt/mssql/bin/sqlservr

Which means: run run-initialization.sh scripts and at the same time run SQL Server. Base image runs server exactly the same way, which we can check with Docker inspect: docker inspect mcr.microsoft.com/mssql/server:2017-CU17-ubuntu | grep CMD. The only difference is that we add our script here.

Note: server command must be after &, otherwise Docker container will stop when run-initialization.sh script finishes job. In example referenced by Microsoft this order is reversed, but container works because of node.js server running.

Note2: be sure that you pass to Docker image script files with Unix line endings, otherwise you may encounter strange bugs.

So now let’s see how run-initialization.sh script looks like:

# Wait to be sure that SQL Server came up
sleep 90s

# Run the setup script to create the DB and the schema in the DB
# Note: make sure that your password matches what is in the Dockerfile
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P yourStrong(!)Password -d master -i create-database.sql

We assume that after 90 seconds DB server will already be ready to use, then invoke sqlcmd utility to run SQL script - 90 seconds is time suggested by Microsoft, in my local environments (for development and tests) I’ve set 30seconds and never had problem with it.

Note: credentials passed to sqlcmd tool must match those given as environment variable in Dockerfile

That’s it, in your SQL script you can do whatever you need to do as startup. In our demo we just create database with schema:

CREATE DATABASE DemoData;
GO
USE DemoData;
GO
CREATE TABLE Products (ID int, ProductName nvarchar(max));
GO

At the end we need to copy those scripts inside Docker image. First we’ll create new directory for them with RUN entry in Dockerfile:

RUN mkdir -p /usr/src/app

then copy files there:

WORKDIR /usr/src/app
COPY . /usr/src/app

and to be sure that run-initialization.sh script will be treated as executable we use chmod command:

RUN chmod +x /usr/src/app/run-initialization.sh

That’s it, whole Dockerfile is available in mentioned GitHub repo.

How to initialize database with data

If you need to have database filled with data, there are several ways to achieve that.

Docker commit

If it’s only for debugging purposes you can do Docker commit to save container state as new image, to recreate it whenever you want. It can be useful for example when you reproduced bug and don’t want to lose it during debugging. Just save database state, then you can recreate whole environment with bug already prepared, as many times as you need. But this way is not recommended for production, because you don’t see explicitly what is in this data so it’s hard to maintain.

Docker volume

You can have your database files already on disk and new server instance just mounted to it with Docker volume. It’s good solution, because you will probably use volume that way or another, to easily persist database.

Script database schema and data

You can script whole database with data included, straight from Microsoft SQL Server Management studio, just click on database name, choose Tasks -> Generate Scripts… and you can choose in advanced options Types of data to scripts - Schema and data.

You can also separate scripts to create schema (empty database) and to add data (fill database), and just run sqlcmd utility several times:

sleep 90s
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P yourStrong(!)Password -d master -i create-database.sql
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P yourStrong(!)Password -d master -i fill-database.sql

To do so first generate script with Schema only, then once again with Data only. I prefer do it this way to separate schema from data, but it’s just my preference, I feel that it’s cleaner this way but don’t have any tangible evidence that it’s better.

Troubleshooting MS SQL Docker initialization

Like I have written before, always remember to set Unix EOL in files which you copy to Docker image, otherwise you may have incomprehensible bugs.

Moreover, at the moment of writing this article, Microsoft SQL Server Management Studio must be restarted every time container is stopped and recreated from scratch, because MSSMS caches internal DB name and doesn’t refresh it, so there is mismatch. That way or another, upgrade MSSMS to the newest version, because as far as I know older versions works even worse.

Summary

Starting new Docker container with Microsoft SQL Server is really simple. Initializing DB server could be equally simple like in MySQL, but unfortunately Microsoft didn’t get this done so far, so at this moment we need to use ugly hack with sleep. Luckily workaround is already prepared, and I hope that after reading this article it’s easy to understand how exactly it works.

If you want to see MS SQL Server database composed up with Dotnet Core app, I have created demo for it as well (even if you are not interested in .NET, you may want to see SQL Server in Docker Compose).

Please tell me your feelings in comment, about MS SQL server in Docker in general, or about this article if you have any suggestions, or share it with others :)

Docker run problem no such file or directory (or other strange message)

2019-12-23T08:00:00+00:00

When you cannot start Docker container correctly and have problems like No such file or directory or some other unintuitive message, there is one thing you should check first…

Correct line endings (EOL) in files copied to Docker

Check if you pass to Docker image scripts (or other text files) with Unix line endings (LF)!

With wrong EOL you will be lucky if message somehow indicates it, for example:
line 3: $’\r’: command not found

but it often looks in a way which gives no clue, what is wrong.

I once spent more than 2 hours because of it, so I hope that with this blog post you will not have to :)

If you have problem which looks similar but is not caused by wrong EOL, please share it in comment, I will try to help you figure it out, then update this post for others.

Docker logs from last run container

2019-12-23T06:00:00+00:00

Running Docker containers is what we do often, so it’s good to know how to do it to see Docker logs immediately.

Running Docker container in foreground mode

When you run container without detach flag (-d) then you usually need to stop container when you want to close terminal or type another command. You can manipulate this with -a flag, but it’s easier just to run in detached mode.

Running Docker container in detached mode

When you run container in detached mode (with –detach, -d or -d=true flag) you can safely close terminal or type another command without affecting running containers, but in this mode by default you don’t see logs.

If you know your container name you can quickly run docker logs -f CONTAINER_NAME but when you don’t know it (which is often the case), then many people usually end up with this solution: quickly run docker ps, check container name, and quickly pass it to docker logs command. With terminal with poor autocompletion this takes long.

How to see docker logs from last ran container?

We usually don’t focus on Docker run output, which in detached mode is… container ID. So we don’t need to use Docker ps to check container name which just got started, we already have it’s ID, which can be used in Docker logs command as well. So we can copy and paste it after Docker run. Example:

$ docker run -d app
d6a2d30a9a85abf110192e944963bb0ce157dde143139c24abb4bbf038403451
$ docker logs -f d6a2d30a9a85abf110192e944963bb0ce157dde143139c24abb4bbf038403451

It’s already more convenient, but… better solution is to use it without need to do copy and paste. How? Simply use xargs to pass Docker run output (container id), to Docker logs. Example:

docker run -d app | xargs docker logs -f

And we have it :) Very simple, very convenient.

Why pass -f flag to Docker logs command?

In all previous examples I have passed -f flag to Docker logs command, which stand for follow. It’s not necessary, but without it Docker will only show already created logs. With follow option, it will show already created logs and will attach our terminal to see new logs, which is what we usually want, to monitor what is going on.

Summary

As Docker comes from Linux world it’s meant to be used from command line and is optimized for this. So having container ID returned from Docker run command (in detached mode) is probably meant to be passed to next command (or script), so we can safely use it to follow logs immediately.

Let me know in comment if you like this trick or share it with others with following buttons or direct url :)

Docker compose introduction - Dotnet core app composed with MySQL Database

2019-09-13T06:00:00+00:00

Introduction to docker-compose - basic aspects explained with .NET Core app and MySQL DB composed together.

If you are familiar with basics of creating Docker images, now it’s time to learn how to compose more than one Docker container together. Once again we’ll learn with .NET Core 2.2 example app, but you should be able to understand Docker compose concepts here, even if you’re not familiar with C# language.

Whole solution is available in SoftwareDeveloperBlog repo on GitHub.

Docker-compose file

Heart of Docker composing is one text file (in yaml format).
By default it’s named docker-compose.yml (or docker-compose.yaml if you prefer), but unlike Dockerfile this name can be changed and specified as command parameter.

Where to keep docker-compose file?

Short answer - it doesn’t matter, put it where it’s convenient for you.
Long answer - some people advice to put it along with deployment scripts, for example in your CI pipeline, because composing is strictly related with deploying. Moreover, this way you can compose images from different repositories.
Where I put docker-compose.yml file? So far I’ve never containerized system spread across multiple repositories, so I put it along with sources, to have it under version control, next to solution file (.sln) - to manage it from the highest perspective (above project directories). Usually I also create solution directory called Docker, referencing docker-compose and .dockerignore files, to be able to edit it from my IDE.

Docker-compose file example

In our example we will use following docker-compose.yml file, kept as I usually do, next to solution file:

version: '3'

services:
  db:
    build: ./Db
  app:
    build:
      context: .
      dockerfile: Aspnetcoreapp/Dockerfile
    ports:
      - 8080:80
    depends_on:
      - db

First we specify docker-compose file format version. I just took the newest version, I don’t see any reasons to take older ones while creating new system.

Then we specify our services. We could specify other aspects, for example networks, volumes, configs, but it’s too wide for this introduction.

Every service has its name, and this service name is important, because we will use it to resolve container ip address and to refer it from other service (for example to specify dependencies).

Build parameter can just point to directory with Dockerfile (like in db service in example above) - then this directory is also taken as build context. If you want to use different directory for build context, than the one with Dockerfile inside, you can separate context and dockerfile options, like in app service above. If you wonder why to do that, I explained choosing build context in previous post.

Specify ports to bind port inside container with one from host machine, like in usual Docker run command.

Lastly we specify that our app depends on db, with depends_on option, taking service name as value. This way we determine services start and stop order (and few other things, not mentioned here for simplification).

Docker-compose commands

Managing composed system is easy, you don’t need to remember many commands and parameters.

Docker-compose up

From directory with docker-compose.yml file you can just run docker-compose up command. It will run containers in order forced by depends_on configuration (if specified). By default images are build only when you run docker-compose up command for the first time, so if you have changed something, you should extend it with --build parameter. If you want to run it from different location then the one with docker-compose.yml file, or if you named compose file not default way, or if you want to pass more than one compose file at once, pass -f FILE_PATH parameter. Last parameter which I use often is -d to detach console from containers. There are other commands to start composed containers, but usually we use this one, so I don’t mention them in this introduction. If you’re curious, they are described well in official documentation. Usually I run docker-compose up command this way:

docker-compose up -d --build

Docker-compose down

If you want to stop composed containers, remove them and theirs networks, just type docker-compose down from directory with docker-compose.yml file. If you want just stop containers without cleaning everything, use docker-compose stop command. Note, that if you have stopped containers with CTRL+C shortcut in attached terminal, you didn’t clean stuff, you just stopped containers and finally will need to run docker-compose down as usual.

Example system

Now with mandatory knowledge, lets create example system. As mentioned in introduction, whole repo is available on GitHub.

.NET Core app

First we create simple ASP.NET Core application. Small startup configuration:

public class Startup
{
    public Startup(IConfiguration configuration) => Configuration = configuration;

    public IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        if (env.IsDevelopment()) 
            app.UseDeveloperExceptionPage();

        app.UseMvc();
    }
}

Program entry point:

public static class Program
{
    public static void Main(string[] args)
    {
        CreateWebHostBuilder(args).Build().Run();
    }

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .UseStartup<Startup>();
}

Controller with one GET method:

[Route("")]
public class ProductsController : ControllerBase
{
    private readonly ProductsProvider _provider = new ProductsProvider();

    [HttpGet]
    public ActionResult<IEnumerable<Product>> Get()
    {
        try
        {
            return _provider.GetAll();
        }
        catch (Exception e)
        {
            Console.WriteLine("Exception during providing products, maybe DB is not fully initialized yet? " +
                              $"Try again in a few minutes and if it doesn't help, check your docker-compose configuration.\n{e}");
            
            return new Product[0];
        }
    }
}

Controller is coupled with Product type:

public class Product
{
    public int Id { get; }
    public string Name { get; }
    public string Description { get; }
    
    public Product(int id, string name, string description)
    {
        Id = id;
        Name = name;
        Description = description;
    }
}

Controller is also coupled with ProductsProvider class, which uses Dapper to access MySQL DB. Note, that we provide docker-compose service name as DB address (as mentioned earlier - that’s why service name is important). Be careful not to make typo - service name is case-sensitive.

public class ProductsProvider
{
    private const string CONN_STRING = "Server=db;Port=3306;Database=product-db;Uid=root; Pwd=myPass;";
    private const string QUERY = "SELECT Id, Name, Description FROM product";
    
    public Product[] GetAll()
    {
        using (var connection = new MySqlConnection(CONN_STRING))
        {
            return connection.Query<Product>(QUERY).ToArray();
        }
    }
}

Dockerfile is written following Docker image from multi project .NET Core solution post. Note that Aspnetcoreapp is our project name, and project directory name at the same time:

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /app

COPY . ./
RUN dotnet publish Aspnetcoreapp -c Release -o out

FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
WORKDIR /app
COPY --from=build-env /app/Aspnetcoreapp/out .

ENTRYPOINT ["dotnet", "Aspnetcoreapp.dll"]

MySQL DB

We could point DB image in docker-compose.yml file and configure environment variables and so on there, but I prefer to configure service via Dockerfile. So we just create new directory with Dockerfile. It can be located anywhere, but I prefer the same level as C# project. So I just create new Db directory, next to project directory (Aspnetcoreapp). If you want to see this Dockerfile in IDE, you can reference it in solution directory.

FROM mysql:5.7.13
COPY *.sql /docker-entrypoint-initdb.d

ENV MYSQL_ROOT_PASSWORD myPass

EXPOSE 3306

With above Dockerfile we do several things:

Specify which version of MySQL image to use
Set root user password to myPass
Expose 3306 port to be able to access it from our app
Initialize database, by copying SQL scripts to /docker-entrypoint-initdb.d directory inside DB container, because official MySQL image description tells us, that scripts from this directory are invoked when DB is started for the first time.

Next we need some .sql script located next to Dockerfile to initialize our database. We use *.sql wildcard, so name doesn’t matter, it just need to end up with .sql extension - for example Init.sql:

CREATE DATABASE `product-db` /*!40100 COLLATE 'latin1_swedish_ci' */;

USE `product-db`;

CREATE TABLE `product` (
	`Id` INT NOT NULL AUTO_INCREMENT,
	`Name` TEXT NOT NULL,
	`Description` TEXT NOT NULL,
	INDEX `Id` (`Id`)
)
COLLATE='latin1_swedish_ci';

INSERT IGNORE INTO product (Id, Name, Description)
VALUES 
(1, "Dependency Injection Principles, Practices, and Patterns", "Book by Steven van Deursen and Mark Seemann"),
(2, "Agile Software Development, Principles, Patterns, and Practices", "Book by Robert C. Martin"); 

Docker-compose.yml and .dockerignore

We take docker-compose.yml file described earlier in this article:

version: '3'

services:
  db:
    build: ./Db
  app:
    build:
      context: .
      dockerfile: Aspnetcoreapp/Dockerfile
    ports:
      - 8080:80
    depends_on:
      - db

Very last thing - .dockerignore file located next to solution (.sln) file. If you don’t know what is this file or why it has to be in this location, I’ve explained it in previous post. Mine currently looks like this:

.dockerignore
.env
.git
.gitignore
.vs
.vscode
*/bin
*/obj
**/.toolstarget
.idea

Running composed system

To run this example system, just go to solution directory, type docker-compose up command (explained earlier) and after a moment (DB initialization takes a while) you can open browser and see data pulled from DB with http://localhost:8080 GET method:

Persisting DB data

As you know, killed container is killed along with the data, which of course is not what we want from production database. To persist changes which we introduced to DB during production runtime, we need to use volumes, but I thought that it’s too much for simple introduction article. If you succeeded previous steps, now you can read about docker volumes. Maybe some day I will explain it in whole different post.

Summary

As you see, composing in general is simple, and basically requires one text file which instructs Docker how to run and couple containers together.

I hope you find this introduction useful. If there is something not clear, ask me a question in comment, and if you think it can help other too, share this article on Facebook or Twitter :)

Multi-project .net core solution in Docker image

2019-07-27T08:00:00+00:00

Creating Docker images from dotnet solution with project references is easy when you understand basics of Docker commands, but writing proper Dockerfile can be tricky for beginners.

Most of the examples show how to dockerize dotnet project, assuming that it has no local dependencies. So let’s analyse what can we do, when our project has references to other projects from solution. We will start with diving into simple example without dependencies first, to understand what changes we introduce and why.
If you just want to skip to solution and copy-paste it, of course you can do it, but it’s not recommended, because sooner or later you will be blocked with another obstacle due to not understanding what is happening, and in result you will waste more time. This article is good to start learning Docker instructions and commands, because everything is explained with short, plain language. We will use .net core 2.2 as it’s current version at the moment of writing.

Example dockerized dotnet core application is available on GitHub, feel free to use it for your needs.

Analysis of official Docker example

Official dockerize an .NET Core application article shows us this Dockerfile located in project folder (where .csproj file is stored):

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /app

# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore

# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out

# Build runtime image
FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "PROJECT_NAME.dll"]

and those two commands, to be run from project folder where Dockerfile is located:

docker build -t aspnetapp .
docker run -d -p 8080:80 --name myapp aspnetapp

Dockerfile FROM instruction

Our Dockerfile starts with FROM instruction:
FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
which means that we base our image on official Microsoft Dotnet Core SDK in version 2.2. We use SDK at this moment, not production runtime, because we will compile our application in Docker during building image. So you don’t even need to have .net core SDK installed on your host machine, this Dockerfile is prepared in a way that you won’t compile your app for Docker image yourself - Docker will compile it. You could use binaries built on your host machine, but it’s not safe - it may not work due to compatibility troubles. Why there is AS build-env instruction - we will come to this later (in Docker multi-stage build section).

Dockerfile WORKDIR instruction

In second line we see WORKDIR /app instruction which means, that following RUN, CMD, ENTRYPOINT, COPY and ADD instructions in our Dockerfile will be executed in /app directory. If it doesn’t exist it will be created (even if it wouldn’t be used).

Dockerfile COPY instruction

Next we see COPY *.csproj ./ instruction, which means that all csproj files from Docker build context will be copied to workdir (/app) directory inside Docker image. Docker build command will be explained later, but in short - build context is the directory from your host machine, pointed in Docker build command. If you point . path, the directory where you execute command is taken. So in our case we copy only one csproj file, because we run build command with project directory set as build context.

Dockerfile RUN instruction

Next is RUN dotnet restore instruction, which simply runs dotnet restore command in our workdir (/app) directory. At this moment inside /app directory in our image, is nothing but .csproj of our project, because we copied only it in previous step, but it’s enough for restoring nuget dependencies.

Copy and compile app source

Again we see COPY instruction - COPY . ./ to copy everything from our build context - in our case it means project files (.cs files etc.), because we run docker build command with project directory set as build context. Then with run instruction - RUN dotnet publish -c Release -o out we simply run dotnet publish in our workdir (/app) directory inside image, with -c Release -o out parameters. This dotnet command compiles our app with release configuration and publishes results in out directory (in our case /app/out). We can compile source because we base this image on developer’s sdk.

Docker multi-stage build

Once again we see FROM instruction, which sets on which image we base our image… How it’s possible to specify it again, with different base? It’s quite fresh Docker feature (since Docker 17.05 version) called multi-stage builds. When we use FROM keyword again, we mean that previous image specified above is temporary, and was used only to serve some purpose. In our case it was made only to compile our application - that’s why we used SDK as base image. Now we specify base image again and this time we are preparing our real image - the one which will be deployed to production, and this one doesn’t base on SDK, only on production runtime, which gives smaller size in result. We will just copy our compiled app from temporary image. So again we specify working directory to /app catalog and then we copy our binaries - COPY --from=build-env /app/out . which means copy files from /app/out/ from build-env image (that’s why we gave it a name in the first line) to current working directory (/app).

Dockerfile ENTRYPOINT instruction

Very last instruction in this Dockerfile is ENTRYPOINT, which (in simple terms) specifies a command that will be executed when the container starts. So in our case - ENTRYPOINT ["dotnet", "PROJECT_NAME.dll"] - Docker will run dotnet with PROJECT_NAME.dll parameter (which should be replaced to our project name of course), to start our app.

Docker build command

With such Dockerfile, we are told to run docker build -t aspnetapp . command in project directory (where Dockerfile is stored). Option: -t name (--tag name) is not mandatory - it allows to tag image (to name it and optionally give it a tag in ‘name:tag’ format), so don’t focus on it, and look on this command this way: docker build ., because important thing is after options - build context parameter. Build context is path on host machine which will be accessible during building image for Dockerfile instructions. In our case it’s . path which means that the directory where we run this command is passed as build context. Because we are told to run this command in project directory (where .csproj file is stored), our project files are passed as build context.

Docker run command

Docker run command creates container from image. Image is readonly manual for Docker, to create container, and container is working virtual machine where our app lives. We can think about it this way: image is like a class in object-oriented programming, and container is like an instance, created from this class. So we can create as many containers (instances) as we want, and it doesn’t affect image (class) - image is only necessary to let Docker know how to create container. We are told to run it this way:
docker run -d -p 8080:80 --name myapp aspnetapp
Without --detach option (-d) we will start seeing app console output from container. With --publish (-p) option we bind container’s port(s) to the host (by default with TCP, but you can specify UDP and SCTP as well). With --name option we assign a name to the container (without this option Docker will choose some funny name for us). At the end we pass image name, which Docker will read to create container. Because we named our image aspnetapp, we use this name here.

Solution

Proper Docker commands

Once we understand what happens in basic example, let’s see how to change it, to make it work when our project has references to other solution projects.
The problem is of course, that we run Docker build command from project directory passing . path as build context. This means, that files only from this directory will be accessible during building image, and depending projects are of course in other directories. We have several options to fix this. We can move Dockerfile one level up (to solution directory) and run docker build from there. But it’s recommended to have Dockerfile in project directory, to be able to have more than one Dockerfile in solution (for different projects). You could also run docker build as before (from project directory), but change build context path to one level up (..). In my view more elegant is third solution - to run docker build from solution directory, pass . as build context and to specify which Dockerfile we want to read with --file (-f) option, like this:
docker build -f PROJECT_DIRECTORY/Dockerfile -t IMAGE_NAME .

How to adjust Dockerfile

Next we need to adjust Dockerfile, because the one from official example assume that we have project directory as build context. My version looks like this:

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /app

COPY . ./
RUN dotnet publish PROJECT_NAME -c Release -o out

FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
WORKDIR /app
COPY --from=build-env /app/PROJECT_NAME/out .

ENTRYPOINT ["dotnet", "PROJECT_NAME.dll"]

I have skipped restoring nuget packages as single step to simplify, restore is included in dotnet publish and if it fails due to nuget failure, error message is legible. But if you have many nuget dependencies you may want to have separate step, because this way Docker treats it as distinct layer and reuses it if none of csproj file changed, which gives smaller build time (I have described it in another post). In this demo restoring nugets is fast enough to skip it, but remember about this if you have long building time (in particular - Restore completed in… time).

So we copy all projects (because build context is now solution directory) to /app directory inside container. Next from /app workdir we run dotnet publish command, specifying which project to compile - RUN dotnet publish PROJECT_NAME -c Release -o out - here PROJECT_NAME is the directory name with .csproj file inside.

Other instructions stays untouched with one small change - during copying compiled app from temporary image, this time we need to pass project name to path:
COPY --from=build-env /app/PROJECT_NAME/out .

Where to keep .dockerignore file

Official article says to add .dockerignore file to project directory, to make build context as small as possible and lower risk of invalidating cache, which of course is reasonable. But Docker CLI looks for .dockerignore file in root directory of the build context, so now we need to move it to solution directory. But in my view it’s even better, because we don’t need to create and maintain many .dockerignore files for many projects, we keep one for all of them. Example rules:

*/bin
*/obj
.dockerignore
.env
.git
.gitignore
.vs
.vscode
**/.toolstarget
.idea

Summary

When I had to dockerize .net core app for the first time I just took Dockerfile from mentioned article, copy-pasted Docker commands and when I faced obstacle I tried to solve it without analysis of how Docker works. After wasting some time this way, I wasted time again - trying to just copy-paste solution from the internet - again without analysis of what I do, and without success too. Then I learned (once again in my life…), that haste didn’t save time, but do the opposite - waste it. Because I didn’t find proper article nor tutorial to start I came though official documentation and manuals, which are written nice, but have too many details for beginners. This article shows essentials of analysis I took and explains basics with plain language. I hope this way it is good to start writing proper Dockerfile, without having trouble with situation like the one presented here - dotnet project with references to other projects from solution, nor with any other obstacle.

What to do when you have already containerized your app, but need to use some dependent system, for example DB? You can compose them together with docker-compose, which I have described with simple .NET Core app and MySQL DB as composed system example, in next article.

Don’t hesitate to write comment whether it was helpful for you, or to share it on Facebook or Twitter :)