Software Developer Blog

Summary of The State of AI-assisted Software Development Report

2026-02-26T04:00:00+00:00

This year’s edition, of the DORA research, entitled State of AI-assisted Software Development, focused on the impact of AI assistance on software development.

What is DORA

DORA is the largest and longest-running research program, analyzing IT organizations of various sizes and in many different industries around the world each year.

The researchers used a rigorous statistical evaluation methodology to identify the common characteristics of successful companies. The analysis covered practices and factors affecting the effectiveness of teams involved in software development.

Full report available at dora.dev/research/2025/dora-report.

AI Is Already the Standard, but It Doesn’t Replace Thinking

AI in software development is no longer an experiment: 90% of surveyed developers use AI at work, and more than 80% report increased productivity.

At the same time, as many as 30% of respondents have low trust in AI-generated code, which demonstrates a mature approach to verifying the results of AI “work.”

AI genuinely accelerates development, but it requires critical thinking and solid engineering practices.

AI Amplifies What’s Good — and What’s Bad

The key takeaway from the report: AI does not fix systemic problems — it amplifies them.

Organizations with strong architecture, clear processes, and a healthy work culture gain the greatest benefits. Where chaos, technical debt, or poor collaboration prevail, AI leads only to local improvements that get lost in later stages of delivery.

With AI, Productivity Increases — but Stability Still Suffers

In 2025, for the first time in DORA research, AI genuinely increases throughput (the speed of delivering changes).

At the same time, an increase in delivery instability is still being observed (more rollbacks, hotfixes, and unplanned work).

This signals that teams have learned to write code faster with AI, but quality control and feedback systems are not keeping pace with this pace.

Platform Engineering Is the Foundation of AI Success

As many as 90% of organizations have adopted platform engineering, following the concept introduced in the book Team Topologies.

The report clearly shows that the quality of the internal developer platform is crucial for realizing tangible benefits from AI.

Additionally, Value Stream Management (VSM) acts as a “force multiplier” — it enables organizations to convert local time savings (e.g., faster code writing) into real business outcomes, rather than generating more downstream chaos.

AI Adoption Is an Organizational Transformation

The greatest return from AI comes from investing in the system — not merely in models or licenses.

The new DORA AI Capabilities Model identifies seven key capabilities that determine success:

A clear and well-communicated AI position
Working in small batches
Healthy data ecosystems
User focus
Internal data accessible to AI
High-quality internal platforms
Strong version control practices

Summary

AI is now a standard part of software development, but it does not solve organizational problems. It acts as an amplifier: the best teams become even better, while weak systems become even more chaotic.

Real value from AI is achieved only by organizations with solid technical, process, and cultural foundations.

Replacing Planning Poker with Team Estimation Game

2025-12-20T04:00:00+00:00

Planning Poker has been with us for years, and it works to a certain extent. But it has its drawbacks as well…

What is the Team Estimation Game

The Team Estimation Game is a collaborative, fast-paced estimation technique focused on relative comparison rather than individual voting. Instead of estimating one item at a time, the team works with many backlog items simultaneously.

On the internet, you can find many complex variants of it, yet I’m a big fan of its simplest form.

Keep the Big Picture during estimation

In the Team Estimation Game, the team works on the whole sprint scope and looks at all stories/tasks at the same time, reasoning about them all at once. In Planning Poker, you focus on one story at a time, estimate it, and then forget about it. That’s a meaningful difference.

Preparation

You can play remotely using any online board you like (like Miro/Lucid), or locally with classic sticky notes and a whiteboard or even just on the table.

Draw columns according to your scale

Not to end up with estimation more granular than your current scale, prepare columns upfront. For example, assuming you estimate in story points: 0, 1, 2, 3, 5, 8, 13, 21, draw 8 columns.

List players

You’ll play in turns, so to avoid messing up the move order, prepare a list of players in some accessible place.

Bring stories/tasks

Prepare sticky notes (literally or virtually) with stories/tasks - those will be moved back and forth during the game.

The Game

As a facilitator call players by name, start by calling first person on the list instructing them to move one card. Keep up the pace by calling the next player shortly after the previous person finishes, especially during online collaboration, prone to distraction.

Move one card or pass

Each player moves one card either from the stack to the board or from one board column to another. It’s allowed to move a card in between, moving existing cards to the column on the right or left. When everything already looks good to you, you can pass your turn. Everyone is allowed to do any move they want, explaining the reasoning behind. Others can ask questions, but cannot ‘block’ someone’s move. Obviously at some point when some players disagree, deeper discussion and some compromise is inevitable.

The Results

At the end, when everyone is satisfied with the current state of the board, the game is finished and you have nice estimations focused on references to each other :)

The Matrix

Instead of columns you can prepare two-dimensional matrix, for example with value/effort or risk/effort, to gain deeper understanding of the tasks and maybe give up some less valuable stories or those with bigger risk.

Summary

That’s it — a very simple exercise that brings more precise estimations and deeper understanding (through the discussions that took place) than classic Planning Poker. Obviously, quick Poker sessions still have their place, but when planning a whole sprint, the Team Estimation Game is my go-to approach now.

Optimize Docker .NET Core project and docker-compose solution to reduce build image time

2020-03-20T14:00:00+00:00

Docker has built in great cache mechanism, but to be able to use it, you have to understand how it works. Let’s dive into it, to build .NET Core Docker images faster.

How Docker cache works

First we should understand how Docker cache works. This part is described nice in official documentation:

Starting with a base image that is already in the cache, the next instruction is compared against all child images derived from that base image to see if one of them was built using the exact same instruction. If not, the cache is invalidated.

In most cases simply comparing the instruction in the Dockerfile with one of the child images is sufficient. However, certain instructions require a little more examination and explanation.

For the ADD and COPY instructions, the contents of the file(s) in the image are examined and a checksum is calculated for each file. The last-modified and last-accessed times of the file(s) are not considered in these checksums. During the cache lookup, the checksum is compared against the checksum in the existing images. If anything has changed in the file(s), such as the contents and metadata, then the cache is invalidated.

Aside from the ADD and COPY commands, cache checking will not look at the files in the container to determine a cache match. For example, when processing a RUN apt-get -y update command the files updated in the container will not be examined to determine if a cache hit exists. In that case just the command string itself will be used to find a match.

Once the cache is invalidated, all subsequent Dockerfile commands will generate new images and the cache will not be used.

So it always checks if given instruction is started in exactly the same situation as in the past. If it is, it doesn’t start it, but this layer is taken from cache. If there is even one bit changed, for example one copied file differs, then cache is invalidated and instruction is invoked truly.

How to decrease image build time

To build images faster, put everything which will rarely change, above things which change often. For example, when you compile your application inside Dockerfile, then your source files often change, which invalidates cache all the way down. But you probably download some dependencies, for example Nuget packages, which change less often than source code, so you can put it before compilation. This way as long as you stick to existing dependencies, they are taken from cache. When you add some new dependency or change version of existing one, cache is invalidated and packages are downloaded from the Internet (and new cache is created).

Ignore what you don’t need

When you copy files from host to build context, Docker calculates checksum for them. If exactly the same files were copied to exactly the same directory in Docker image, then this layer (as every operation in Dockerfile creates new layer) is taken from cache. So copy only those files which you need, because you can invalidate cache in vain, with files which are not required in your image, for example every Dockerfile, or files generated by your IDE and so on. Remember, that .dockerignore file has to be located in build context root. Example rules:

*/bin
*/obj
.dockerignore // Docker files
.env
.git // Git files
.gitignore // Git files
.gitattributes // Git files
.vs // IDE files
.vscode // IDE files
**/.toolstarget
.idea // IDE files
**/Samples // Additional files
build
Docker // Docker scripts and docker-compose files (in this example in 'Docker' directory, in context root level)
**/Dockerfile // Docker files
**/docker-compose // Docker files
*Test*

Try to create your own cache (yarn build example)

When you know where is your bottleneck and there is no obvious way to reduce it, try to figure out some none obvious solution. For example a few projects in one of products which I maintain, have yarn install and yarn build sewn into csproj file, like this:

<Target Name="PublishRunWebpack" AfterTargets="ComputeFilesToPublish" >
  <Exec WorkingDirectory="$(SpaRoot)" Command="yarn install" />
  <Exec WorkingDirectory="$(SpaRoot)" Command="yarn build" />

but this was extremely slow - it took (in Docker for Windows) around 4 minutes. And web files in those projects were changed rarely, usually C# code got changed, so I wanted to reuse yarn build results. I didn’t find a way to use npm/yarn cache in this situation, so I cached it myself. I moved copying web files to the top of Dockerfile, like this:

WORKDIR /app/WebClient/WebFiles
COPY WebClient/WebFiles . # Taken form cache as long as you don't change files in WebFiles directory
RUN yarn install # Taken form cache as long as you don't change files in WebFiles directory
RUN yarn build # Taken form cache as long as you don't change files in WebFiles directory

and changed csproj like this:

<Target Name="PublishRunWebpack" AfterTargets="ComputeFilesToPublish" Condition=" '$(Configuration)' == 'Release' And !Exists('$(ProjectDir)WebFiles/build') ">
  <Exec WorkingDirectory="$(SpaRoot)" Command="yarn install" />
  <Exec WorkingDirectory="$(SpaRoot)" Command="yarn build" />

So Docker cached yarn build output and it was invalidated only when web files changed. With target condition, C# project reused already existing yarn build output. Of course, because COPY WebClient/WebFiles was on top of my Dockerfile, every time when some web files changed, it invalidated cache all the way down, so if it was changed often, I would do it some different way, but in my case it worked as expected. And I know that in theory incorrect build directory could exist somehow, but it’s only theory, because in Docker situation is repeatable and on developer machines publish wasn’t invoked that way or another, only debug configuration, so in practice it’s not a risk. Maybe with incremental build I could be able to achieve the same with a bit cleaner way - if your case is similar to mine, you can try it.

Reuse what you can

If there is something which you can reuse, do it. For example when I have multiple Docker images in one solution, I build them all at once, instead of building every project in separation. I have observed, that this way it’s faster. For example, I have solution Application.sln which contains 5 executable projects (which are packed to separate Docker images) and additional projects like tests, usage samples and so on. So to lower number of projects to build, I have copied it to Application-minimal.sln and left only necessary projects. Then in every Dockerfile, instead of compiling only one, chosen project, like this: RUN dotnet publish PROJECT_NAME -c Release -o out I compile whole solution like this: RUN dotnet publish Application-minimal.sln -c Release. After compilation, I copy whole build output (with all projects) to final image (see Docker multi-stage build explained), for example this way: COPY --from=build-env /app/build .. Finally, at the very bottom of Dockerfile I invalidate cache specifying which project I want to run:

WORKDIR /app/ProjectA/publish # Pay attention to have correct path, especially if you set custom build output path.
ENTRYPOINT ["dotnet", "ProjectA.dll"]

Everything above is reused from cache, so when you want to build n projects, only first project takes a while to compile, other projects are taken from cache practically whole. In my case I’ve observed that it works faster than doing it in separation (running dotnet publish PROJECTX for every project). It will probably be faster in your case too, but as with every optimization, I encourage to measure it before applying :)

Experiment with caching nuget dependencies

In your developer PC, you have nuget cached locally. But in Docker you have clean situation every time you invoke build, so nuget packages are taken from the internet every time, so it’s good to cache them, because we don’t change project dependencies as often as project source files. When you compile only one project, then situation is clear, from project directory you can run:

COPY *.csproj ./ # Taken form cache as long as you don't change your project file
RUN dotnet restore # Taken form cache as long as you don't change your project file 

# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out

But when you want to restore files for whole solution, then it gets more complicated, because (at the time of writing) you cannot point to .sln file in dotnet restore command, and you cannot point to multiple csproj files neither. Fortunately, you can resolve it in many ways, for example with simple bash script:

#!/bin/bash
for projectName in *.csproj; do # It can be saved as one-linear straight in Dockerfile
	dotnet restore $projectName
done

used this way:

COPY restore-all.sh . # Taken form cache always
RUN chmod +x restore-all.sh
COPY */*.csproj ./ # Taken form cache as long as you don't change your csproj files
RUN ./restore-all.sh # Taken form cache as long as you don't change your csproj files

Alternatively: build image from dedicated container

Some people do not compile application in Dockerfile, but on their own PC and just put build output to image. This is faster, but it has one big downside: you’re not 100% sure that image is properly built, and generally speaking how it was made, so I’ll never wont to put such image on production. But I’ve also heard about another solution - to create container dedicated to create input to Docker image… Sounds a bit like overengineering, and probably in many cases it is, but maybe sometimes it’s worth doing it. You have more options in container, for example you can mount directories, so there is no problem with any cache you need. You can even use already existing cache from host machine. I’ve never tried such approach, above methods work well for me, but maybe sometimes it’s worth to organize CI/CD pipeline this way.

Summary

Proper caching in Dockerfile is quite broad subject, so here I only touched it in generals. The point of this article is not only to give you some ready to use tricks (like those with yarn or Nuget packages), but also to show you how cache in Docker works and encourage you to investigate your own situation. Pay attention to every build layer, measure which step takes too much time and where is your bottleneck. Sometimes with one line moved in Dockerfile, or one file added to .dockerignore you can make (really) big difference. And sometimes with small investigation you can resolve huge problem, for example .dockerignore file stored somewhere else than build context root directory.
If you have feeling that cache caused some problem (which in theory may be the case, but I never had such situation) you can always check how Docker builds with no-cache parameter, which forces to build everything from scratch.

I hope the article was helpful for you, please publish it on Facebook or Twitter and of course ask me anything, which is still not clear. If you know some other trick related to cache, please share it through comments, someone for sure will appreciate it :)

Introduction to Git Bisect with CodeceptJS automation example

2020-01-10T06:00:00+00:00

Many developers have heard about Git Bisect, but only few gave it a try. It’s a pity, because it can… find a bug without your attention!

Git Bisect is a tool which comes pre-installed with Git, which is meant to find particular commit. But unlike Git log it’s not meant to find commit by its details, for example commit message or introduced changes, but by behaviour which it introduces to system.

How Git Bisect works

Git Bisect uses binary search algorithm to check commit. You need to set commits range and then Git Bisect checks commit in the middle of the range. If behaviour which we are looking for exist in this commit, then Git knows that it was introduced in first part of the range, otherwise in the second part. This way range gets smaller by half and then algorithm repeats up to the point, when Git is sure which commit has introduced given behaviour. If you create small commits and each of commit has one, consistent change into the system, then you not only see which commit introduced given behaviour, but also which changes in code are guilty.

How Git Bisect knows that particular commit has given behaviour

When Git Bisect has selected the commit, there are two modes to check if system at this point of time had given behaviour.
You can do it manually - simply run the system and tell Git the answer or…
You can do it automatically. Just pass to Git Bisect script or program which tests system and returns 0 exit code if system works good or exit with a code between 1 and 127 (inclusive, except 125), if currently there is bad behaviour.

Can Git Bisect be used to other purpose than finding a bug

Usually we talk about bad behaviour - a bug, that’s why by default we operate with terms good and bad, but you can use Git Bisect to find other type of introduced behaviour, for example improved performance, so you can use alternate terms. By default, you have two alternate terms given: old and new, but you can set up your, to make bisecting easier to understand. If you want to know how to achieve it, it’s described nice in documentation, I don’t want to lose focus on general usage here, so let’s move forward.

Git Bisect manual demo

Let’s imagine, that we have noticed, that our system at the newest version has regression bug, which for sure did not exist in last release, tagged as v1.2. To start bisecting we need to write git bisect start. Then we need to set end of commits range, by writing git bisect bad *COMMIT_ID*. As we know that the newest version has it, we can skip *COMMIT_ID*, to have HEAD taken. Then we need to set beginning of commits range, by writing git bisect good *COMMIT_ID*, which in our case it’s git bisect good v1.2.

When commits range is set, Git chooses first commit. Output may look like this:

Bisecting: 10 revisions left to test after this (roughly 4 steps)
[3007a714657da73173b7236d3821bbea0555bd2b] Adjust fonts

Now we need to check system: recompile, run, and test manually if bug exist. If so, then write: git bisect bad. If bug doesn’t exist, then write: git bisect good.

This process repeats until commit which introduced the bug is found, which is indicated by commit details printed out. At this moment we are still in bisecting mode. To finish it, we need to write git bisect reset, which returns us to the commit that was checked out before git bisect start (unless you don’t specify other commit id as parameter to reset command).

There are other commands to use during bisecting, but they are additional. If you are curios which are those, they are described clearly in documentation. I don’t want to distract us, so will skip them here, with exception to one convenient command: git bisect visualize, which shows us current commits range and currently selected commit. If you have gitk installed it will show you there, otherwise in terminal.

Git Bisect automated demo with CodeceptJS

Of course, checking if bug exist is a boring process which usually can be easily automated. As our system is website, we will use CodeceptJS. Initializing it, and creating test scenarios is out of scope of this article, so if you’re new to it, just follow theirs official manuals. We need to describe behaviour in test scenario, which need to be initialized in directory outside of our repo, or in untracked file in our repo, to have it unchanged when Git Bisect changes commits. Of course after finding bug, we will commit those new scenarios to rest of our system test scenarios, to not have this regression any more in the future.

Let’s assume, that in our case we run webserver once, and it refreshes itself when files change, during commits switching (like for example Jekyll does).

Then we need to start bisecting and set commits range like before. Next, instead of telling Git whether given commit is good or bad we run automated testing: git bisect run node codeceptjs run --steps. The first argument after run keyword is the script/program to run, next are arguments which will be passed along. In our case it’s CodeceptJS with its arguments. If you don’t have CodeceptJS globally installed you can pass path for example from node_modules directory: git bisect run node node_modules/.bin/codeceptjs run --steps.

That’s it, Git Bisect will check every chosen commit with this script to the point, when it knows which commit introduced given behaviour, in our case - a bug. Like in manual process you need to reset bisecting mode at the end - git bisect reset.

What if you need to compile system during every check, or do some other preparations? It’s not a problem, you simply need to put those tasks in script/program which got executed by bisect.

Video demo

In my view Git bisect with automated tests look spectacular, so I have created a video Demo, if you prefer to watch before trying yourself :)

Summary

To summarize, Git Bisect is a simple tool, which can be really helpful. When? In situations, when you have a bug (or other behaviour), which is hard to debug, so it’s easier to find which commit introduced it, especially when you create small, consistent commits. I have seen developers who do binary search manually, counting and switching commits.

It’s giant help especially, when you have good automated tests, so you can just add new test scenario, then Git Bisect will take care of rest of the job.

Do you use Git Bisect or have some thoughts about it? Please share it through comments. And send this article or video demo to colleagues who hasn’t heard of it :)

Initialize MS SQL in Docker container - create database at startup

2019-12-27T06:00:00+00:00

Microsoft SQL Server is available for Linux so we can run it from Docker container, but usually we need to initialize database at startup, which currently is a bit tricky.

If you have seen my post which explains Docker compose, then you know that MySQL team has prepared very convenient way to run initialization scripts when DB has started. You only need to pass your script to proper directory and MySQL will run it when it’s successfully stared, so you can for example create database.

Unfortunately Microsoft SQL Server currently doesn’t have such a mechanism, and official documentation redirects to only one demo, which is coupled with node.js and without node it doesn’t work.

So I have created simplified demo, and here will explain how it works step by step.

Running MS SQL Server from Docker container without initialization

First let’s take a look at official information of how to start Microsoft SQL Server in container:

docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong(!)Password' -e 'MSSQL_PID=Express' -p 1433:1433 -d mcr.microsoft.com/mssql/server:2017-latest-ubuntu 

I prefer to manage base image and environment variables in text file, so let’s put all above to Dockerfile:

FROM mcr.microsoft.com/mssql/server:2017-latest-ubuntu

ENV SA_PASSWORD yourStrong(!)Password
ENV ACCEPT_EULA Y
ENV MSSQL_PID Express

Now we can add this file to Git,
build this image - docker build -t db-demo .
and run container in simpler way:

docker run -p 1433:1433 -d db-demo

This way we run clean MS SQL server.
We can additionally add EXPOSE 1433 entry to Dockerfile, to be able to access it from composed containers.

We should also use explicit image version (not latest) to be sure that this Dockerfile will always be valid (new versions will not break down Docker build or runtime initialization in container). So for example let’s choose version: FROM mcr.microsoft.com/mssql/server:2017-CU17-ubuntu (all tags list is usually given in Docker Hub image description).

How to initialize MS SQL server in Docker container

Now it’s time to add extra stuff to have initialization in our server.

The hack is to run at the same time SQL Server and script which will sleep 90seconds (to be sure that server already started) then do whole initialization (we’ll see how). It’s not my idea, this is solution currently recommended by Microsoft and nothing better comes to my mind neither.

First we override Dockerfile CMD entry from base image (which runs SQL server), with our own CMD command, which runs our custom script:

CMD /bin/bash ./entrypoint.sh

and this entrypoint.sh script looks like this:

# Run Microsoft SQl Server and initialization script (at the same time)
/usr/src/app/run-initialization.sh & /opt/mssql/bin/sqlservr

Which means: run run-initialization.sh scripts and at the same time run SQL Server. Base image runs server exactly the same way, which we can check with Docker inspect: docker inspect mcr.microsoft.com/mssql/server:2017-CU17-ubuntu | grep CMD. The only difference is that we add our script here.

Note: server command must be after &, otherwise Docker container will stop when run-initialization.sh script finishes job. In example referenced by Microsoft this order is reversed, but container works because of node.js server running.

Note2: be sure that you pass to Docker image script files with Unix line endings, otherwise you may encounter strange bugs.

So now let’s see how run-initialization.sh script looks like:

# Wait to be sure that SQL Server came up
sleep 90s

# Run the setup script to create the DB and the schema in the DB
# Note: make sure that your password matches what is in the Dockerfile
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P yourStrong(!)Password -d master -i create-database.sql

We assume that after 90 seconds DB server will already be ready to use, then invoke sqlcmd utility to run SQL script - 90 seconds is time suggested by Microsoft, in my local environments (for development and tests) I’ve set 30seconds and never had problem with it.

Note: credentials passed to sqlcmd tool must match those given as environment variable in Dockerfile

That’s it, in your SQL script you can do whatever you need to do as startup. In our demo we just create database with schema:

CREATE DATABASE DemoData;
GO
USE DemoData;
GO
CREATE TABLE Products (ID int, ProductName nvarchar(max));
GO

At the end we need to copy those scripts inside Docker image. First we’ll create new directory for them with RUN entry in Dockerfile:

RUN mkdir -p /usr/src/app

then copy files there:

WORKDIR /usr/src/app
COPY . /usr/src/app

and to be sure that run-initialization.sh script will be treated as executable we use chmod command:

RUN chmod +x /usr/src/app/run-initialization.sh

That’s it, whole Dockerfile is available in mentioned GitHub repo.

How to initialize database with data

If you need to have database filled with data, there are several ways to achieve that.

Docker commit

If it’s only for debugging purposes you can do Docker commit to save container state as new image, to recreate it whenever you want. It can be useful for example when you reproduced bug and don’t want to lose it during debugging. Just save database state, then you can recreate whole environment with bug already prepared, as many times as you need. But this way is not recommended for production, because you don’t see explicitly what is in this data so it’s hard to maintain.

Docker volume

You can have your database files already on disk and new server instance just mounted to it with Docker volume. It’s good solution, because you will probably use volume that way or another, to easily persist database.

Script database schema and data

You can script whole database with data included, straight from Microsoft SQL Server Management studio, just click on database name, choose Tasks -> Generate Scripts… and you can choose in advanced options Types of data to scripts - Schema and data.

You can also separate scripts to create schema (empty database) and to add data (fill database), and just run sqlcmd utility several times:

sleep 90s
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P yourStrong(!)Password -d master -i create-database.sql
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P yourStrong(!)Password -d master -i fill-database.sql

To do so first generate script with Schema only, then once again with Data only. I prefer do it this way to separate schema from data, but it’s just my preference, I feel that it’s cleaner this way but don’t have any tangible evidence that it’s better.

Troubleshooting MS SQL Docker initialization

Like I have written before, always remember to set Unix EOL in files which you copy to Docker image, otherwise you may have incomprehensible bugs.

Moreover, at the moment of writing this article, Microsoft SQL Server Management Studio must be restarted every time container is stopped and recreated from scratch, because MSSMS caches internal DB name and doesn’t refresh it, so there is mismatch. That way or another, upgrade MSSMS to the newest version, because as far as I know older versions works even worse.

Summary

Starting new Docker container with Microsoft SQL Server is really simple. Initializing DB server could be equally simple like in MySQL, but unfortunately Microsoft didn’t get this done so far, so at this moment we need to use ugly hack with sleep. Luckily workaround is already prepared, and I hope that after reading this article it’s easy to understand how exactly it works.

If you want to see MS SQL Server database composed up with Dotnet Core app, I have created demo for it as well (even if you are not interested in .NET, you may want to see SQL Server in Docker Compose).

Please tell me your feelings in comment, about MS SQL server in Docker in general, or about this article if you have any suggestions, or share it with others :)

Store Git SSH key pair in Linux with Libsecret to skip asking for credentials

2019-11-04T10:00:00+00:00

If you have to connect to Git via SSH, your usual credential manager may be not enough. Read how to avoid entering password every time on your Linux device.

I have already introduced Libsecret Git credential manager, which works fine if you connect to remote repo via http(s), which is what we usually do. But sometimes we are forced to use SSH protocol, and once again we need to write password every time we interact with remote repo. Luckily it can be solved in secure way quite easily, but unfortunately descriptions which I found on the internet are not clear, moreover usually skip securing private keys. My goal is to explain it in this article in simple words, so you can apply it and understand what you do.

How automate SSH connection works

Mechanism which is used in SSH to automate connections is really simple. You generate public-private keys pair (those keys are not related to credentials nor any other data), hide private key with you and share public key with server. Then when you try to connect to server, your clients uses private key as prove that you’re allowed to connect - that’s it.

It sounds quite magically, but it’s really simple. Those keys are just text files, stored somewhere on your device, usually in ~/.ssh directory. They are not related to any data, it’s just a pair which suits each other, in mathematical meaning, you generate it with OpenSSH tool. Look at this example (with default file names id_rsa and id_rsa.pub):

Private key file: ~/.ssh/id_rsa

Public key file: ~/.ssh/id_rsa.pub

You store both files in your device and save public one in server which you want to connect, at proper path - that’s it. If you want to read more about key pair mechanism, article on ssh.com explains it nice.

How to configure automatic SSH connection

Once you know the theory, it’s time to try automated connection in practice. First I will show you the process with Seahorse tool which I find convenient for this task, but of course you can achieve the same with command line, which I will show later.

In Ubuntu and Linux Mint Seahorse comes pre installed with OS and you can find it as Seahorse or Passwords and Keys. I’m pretty sure it’s not difficult to install it in other distros, so if you don’t have it already, just search how to install it in your case. It uses libsecret abstraction which was created to unify access to sensitive data, so it should work on all popular Linux distributions.

If you don’t see secrets categories on the left side of Seahorse, set View » By Keyring.

Create SSH keys

Choose OpenSSH Keys category and press + button

Select Secure Shell Key

Now give your keys some description, to remember what is this for. This name will be seen on server, so you can write here your email address, to simplify server administration (so maintainer could know who uses this and if it’s still required).

Choose encryption type - at the time of writing the only available options in Seahorse are RSA and DSA - it’s a bit subjective, but I suggest choosing RSA as explained on security.stackexchange.com. There is a way to choose Ed25519 algorithm which is more secure and gives better performance, but currently only from command line.

RSA key strength can be left 2048 bits as is by default. You can increase it, but be aware that some devices may not support it and that it takes a bit more of resources to work with bigger number. But if you insist, 4096bits should be fine. Here is nice article which explains RSA key size topic with details.

From this point you can choose two options:

Just Create Key
Create and Set Up

With second option you will populate newly created key to server right from this wizard, so that’s what we will do. If you don’t have access to server at this moment, you can just create key and populate it later from command line.

Encrypt SSH keys with password

Next you will be asked to write passphrase for new secure shell key. It may be a bit confusing - you may wonder if this password must match SSH user password - it doesn’t need to, moreover - it shouldn’t - it is different password so it should be different (you won’t need to remember it). If you leave it blank your private key will be saved on disk unencrypted, which causes security risks. Despite some people advice to leave it this way I don’t see any reason not to encrypt it, that way or another you won’t need to remember it - we will use keyring for that (but it this moment save it somewhere, preferably in some password manager). Enter strong passphrase, I recommend using some tool which will generate it (I use KeepassXC), then enter it again as confirmation.

Check created keys files

Now you should have new files in ~/.ssh directory, as described earlier - id_rsa with private key and id_rsa.pub with public key. If you skipped encrypting private key you will see it in plain text, otherwise it will be encrypted and will contain header which indicates encryption, for example: Proc-Type: 4,ENCRYPTED.

Send public key to server

If you decided to create key and set it up, you will now see new Window - Set Up Computer for SSH Connection. Write your server address and a port in format server:port then your login name and press Set Up button. You will be asked for given user password.

Connect and store encryption password in keyring

If setting up succeeds, then in user directory on server will be copy of your public key in ~/.ssh directory in authorized_keys file - connect to server and check if it is there. It may look like this:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxGhrhgMsQqOvXf+M1z04toa65/8vY6b+lzjAas7YJqut65o0ONWItpz2UHXSl0mFlbN49UR6VDR9jesUKW+vfuIRUaPs+8/bNEP5tIzPq3lSvw1msYNBT2kbQWO5+Qqiy7EXf/RJ7z4BGiBDEZ4rRhjxIrPAuaxeOpWU6KEDlsxCczxZWgl2xcldhk5oJwzItNd2piQlr7r2CYibHU36XtELvAhlTI1/p11rfSf/BwGtCxb25dApg8k/yRw2zpdV/JD0nfNQiESnCSmLxDr66WK5dScpXQ8zRMRSM3ekoo2ElkDEl9gQ9PhrqdKDLiPqgbobhi22ro4GKsHHAy2VF GitGitRsa

Do it as usual, just: ssh user@server.com -p 22 To see file you can for example write: cat ~/.ssh/authorized_keys

When you try to connect for the first time, you should be asked for password to Unlock private key. Set Automatically unlock this key whenever I’m logged in checkbox and since now it will be unlocked along with your login keyring, when you log in. You will no longer see this window and your private key will be encrypted - like I told, there is no reason not to encrypt it, it’s secure and convenient at the same time.

In my view if Seahorse works in your case, it’s convenient to use it’s wizard, but there is one exception: when you want to use Ed25519 algorithm which, like I told earlier in this article, is more secure and gives better performance, you must use command line, as at the time of writing it’s not supported in Seahorse wizard.

Bellow steps are exact equivalent of previous from Seahorse tool, just written from command line 😺

First generate a new Ed25519 SSH key pair (if you haven’t done it already):

ssh-keygen -t ed25519 -C "description or your email"

If server which you want to reach doesn’t support Ed25519 or if you have old version of OpenSSH on your PC (Ed25519 was introduced to OpenSSH in version 6.5), use RSA instead:

ssh-keygen -t rsa -b 2048 -C "description or your email"

Description or email is optional, but it’s good to put some convenient information here, because as I described earlier, this name will be seen on server, so it should identify you, so administrator will know who uses it.
If you wonder how many bits should have RSA key size (in above example 2048 bits), it’s the same case which I described during key creation with Seahorse tool. The bigger number you choose, the better security you have, but it comes with a bit bigger resources usage, and with risk, that it will not work - some devices don’t support length bigger than 2048. I recommend good article about RSA key size length, for those who want to understand choice better.

First you will be asked about path where to save keys - use default option by hitting Enter. If you’re warned that key already exist, you must choose custom path and later set up ~/.ssh/config file.

Next you will be asked for encryption password. This is not your SSH user password, so create strong password, different than user’s one, the best option is to generate it with some tool. You won’t need to remember it, but before you configure everything, save it somewhere - the best option is to use passwords manager - I recommend KeepassXC. I described why to do that in earlier part of this article.

Now you can check created keys files to ensure everything worked as expected.

If you had to enter custom path for keys, now you should configure it:

# Only if created keys at custom path 
eval $(ssh-agent -s)
ssh-add ~/.ssh/YOUR_NEW_PRIVATE_KEY

And save ~/.ssh/config file as in below example:

# Only if created keys at custom path 
Host gitlab.com
  Preferredauthentications publickey
  IdentityFile ~/.ssh/GITLAB_COM_KEY

Host github.com
  Preferredauthentications publickey
  IdentityFile ~/.ssh/GITHUB_COM_KEY

Next you need to copy your public key to server, this can be done with one command:

ssh-copy-id user@server

or if ssh-copy-id utility is not available on your device, you can use plain ssh solution
(change PUBLIC_KEY.pub to what you have generated, for example id_rsa.pub):

cat ~/.ssh/PUBLIC_KEY.pub | ssh <user>@<hostname> 'cat >> .ssh/authorized_keys && echo "Key copied"'

Next check if public key is copied as expected and store encryption password in keyring.

Troubleshooting SSH key pairing

If automate connection doesn’t work and you don’t know what has failed, and you have only one remote SSH server, you may want to try everything from scratch - maybe you made some mistake? That is what helped me, but use it only if you’re sure you don’t have any other keys already and just in case copy files as backup.

Remove all files from local ~/.ssh directory.
Remove all files from remote (server) ~/.ssh directory.
Try setting up everything again. I suggest reading short description how key pair automation works, to know what is happening during process.

If you see that keys are generated correctly, but copying to server failed, try to copy it manually (it may require sudo access):

Copy public key to server, for example with scp - scp ~/.ssh/PUBLIC_KEY.pub user@server:/home/user/PUBLIC_KEY.pub
Login to server as usual, for example: ssh user@server.com -p 22
Create .ssh directory, for example with command: mkdir ~/.ssh
Append the public key content to authorized_keys file: cat ~/PUBLIC_KEY.pub >> ~/.ssh/authorized_keys

If your key is copied to ~/.ssh/authorized_keys file on server but automate connection doesn’t work, try changing remote user permissions to 711 (some people advice to do it as well for user home directory) - you may need sudo access:

Login to server as usual, for example: ssh user@server.com -p 22
cd ~
chmod 711 .
chmod 711 .ssh/
chmod 711 .ssh/authorized_keys
chown -R USERANAME .ssh/

Summary

If you have any questions feel free to ask them in comments, or just to tell if this article has helped you. If you know someone who also uses SSH protocol to manage remote Git repo, don’t hesitate to send him or her link, it may help yet another person and I will be glad too :)

Docker compose introduction - Dotnet core app composed with MySQL Database

2019-09-13T06:00:00+00:00

Introduction to docker-compose - basic aspects explained with .NET Core app and MySQL DB composed together.

If you are familiar with basics of creating Docker images, now it’s time to learn how to compose more than one Docker container together. Once again we’ll learn with .NET Core 2.2 example app, but you should be able to understand Docker compose concepts here, even if you’re not familiar with C# language.

Whole solution is available in SoftwareDeveloperBlog repo on GitHub.

Docker-compose file

Heart of Docker composing is one text file (in yaml format).
By default it’s named docker-compose.yml (or docker-compose.yaml if you prefer), but unlike Dockerfile this name can be changed and specified as command parameter.

Where to keep docker-compose file?

Short answer - it doesn’t matter, put it where it’s convenient for you.
Long answer - some people advice to put it along with deployment scripts, for example in your CI pipeline, because composing is strictly related with deploying. Moreover, this way you can compose images from different repositories.
Where I put docker-compose.yml file? So far I’ve never containerized system spread across multiple repositories, so I put it along with sources, to have it under version control, next to solution file (.sln) - to manage it from the highest perspective (above project directories). Usually I also create solution directory called Docker, referencing docker-compose and .dockerignore files, to be able to edit it from my IDE.

Docker-compose file example

In our example we will use following docker-compose.yml file, kept as I usually do, next to solution file:

version: '3'

services:
  db:
    build: ./Db
  app:
    build:
      context: .
      dockerfile: Aspnetcoreapp/Dockerfile
    ports:
      - 8080:80
    depends_on:
      - db

First we specify docker-compose file format version. I just took the newest version, I don’t see any reasons to take older ones while creating new system.

Then we specify our services. We could specify other aspects, for example networks, volumes, configs, but it’s too wide for this introduction.

Every service has its name, and this service name is important, because we will use it to resolve container ip address and to refer it from other service (for example to specify dependencies).

Build parameter can just point to directory with Dockerfile (like in db service in example above) - then this directory is also taken as build context. If you want to use different directory for build context, than the one with Dockerfile inside, you can separate context and dockerfile options, like in app service above. If you wonder why to do that, I explained choosing build context in previous post.

Specify ports to bind port inside container with one from host machine, like in usual Docker run command.

Lastly we specify that our app depends on db, with depends_on option, taking service name as value. This way we determine services start and stop order (and few other things, not mentioned here for simplification).

Docker-compose commands

Managing composed system is easy, you don’t need to remember many commands and parameters.

Docker-compose up

From directory with docker-compose.yml file you can just run docker-compose up command. It will run containers in order forced by depends_on configuration (if specified). By default images are build only when you run docker-compose up command for the first time, so if you have changed something, you should extend it with --build parameter. If you want to run it from different location then the one with docker-compose.yml file, or if you named compose file not default way, or if you want to pass more than one compose file at once, pass -f FILE_PATH parameter. Last parameter which I use often is -d to detach console from containers. There are other commands to start composed containers, but usually we use this one, so I don’t mention them in this introduction. If you’re curious, they are described well in official documentation. Usually I run docker-compose up command this way:

docker-compose up -d --build

Docker-compose down

If you want to stop composed containers, remove them and theirs networks, just type docker-compose down from directory with docker-compose.yml file. If you want just stop containers without cleaning everything, use docker-compose stop command. Note, that if you have stopped containers with CTRL+C shortcut in attached terminal, you didn’t clean stuff, you just stopped containers and finally will need to run docker-compose down as usual.

Example system

Now with mandatory knowledge, lets create example system. As mentioned in introduction, whole repo is available on GitHub.

.NET Core app

First we create simple ASP.NET Core application. Small startup configuration:

public class Startup
{
    public Startup(IConfiguration configuration) => Configuration = configuration;

    public IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        if (env.IsDevelopment()) 
            app.UseDeveloperExceptionPage();

        app.UseMvc();
    }
}

Program entry point:

public static class Program
{
    public static void Main(string[] args)
    {
        CreateWebHostBuilder(args).Build().Run();
    }

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .UseStartup<Startup>();
}

Controller with one GET method:

[Route("")]
public class ProductsController : ControllerBase
{
    private readonly ProductsProvider _provider = new ProductsProvider();

    [HttpGet]
    public ActionResult<IEnumerable<Product>> Get()
    {
        try
        {
            return _provider.GetAll();
        }
        catch (Exception e)
        {
            Console.WriteLine("Exception during providing products, maybe DB is not fully initialized yet? " +
                              $"Try again in a few minutes and if it doesn't help, check your docker-compose configuration.\n{e}");
            
            return new Product[0];
        }
    }
}

Controller is coupled with Product type:

public class Product
{
    public int Id { get; }
    public string Name { get; }
    public string Description { get; }
    
    public Product(int id, string name, string description)
    {
        Id = id;
        Name = name;
        Description = description;
    }
}

Controller is also coupled with ProductsProvider class, which uses Dapper to access MySQL DB. Note, that we provide docker-compose service name as DB address (as mentioned earlier - that’s why service name is important). Be careful not to make typo - service name is case-sensitive.

public class ProductsProvider
{
    private const string CONN_STRING = "Server=db;Port=3306;Database=product-db;Uid=root; Pwd=myPass;";
    private const string QUERY = "SELECT Id, Name, Description FROM product";
    
    public Product[] GetAll()
    {
        using (var connection = new MySqlConnection(CONN_STRING))
        {
            return connection.Query<Product>(QUERY).ToArray();
        }
    }
}

Dockerfile is written following Docker image from multi project .NET Core solution post. Note that Aspnetcoreapp is our project name, and project directory name at the same time:

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /app

COPY . ./
RUN dotnet publish Aspnetcoreapp -c Release -o out

FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
WORKDIR /app
COPY --from=build-env /app/Aspnetcoreapp/out .

ENTRYPOINT ["dotnet", "Aspnetcoreapp.dll"]

MySQL DB

We could point DB image in docker-compose.yml file and configure environment variables and so on there, but I prefer to configure service via Dockerfile. So we just create new directory with Dockerfile. It can be located anywhere, but I prefer the same level as C# project. So I just create new Db directory, next to project directory (Aspnetcoreapp). If you want to see this Dockerfile in IDE, you can reference it in solution directory.

FROM mysql:5.7.13
COPY *.sql /docker-entrypoint-initdb.d

ENV MYSQL_ROOT_PASSWORD myPass

EXPOSE 3306

With above Dockerfile we do several things:

Specify which version of MySQL image to use
Set root user password to myPass
Expose 3306 port to be able to access it from our app
Initialize database, by copying SQL scripts to /docker-entrypoint-initdb.d directory inside DB container, because official MySQL image description tells us, that scripts from this directory are invoked when DB is started for the first time.

Next we need some .sql script located next to Dockerfile to initialize our database. We use *.sql wildcard, so name doesn’t matter, it just need to end up with .sql extension - for example Init.sql:

CREATE DATABASE `product-db` /*!40100 COLLATE 'latin1_swedish_ci' */;

USE `product-db`;

CREATE TABLE `product` (
	`Id` INT NOT NULL AUTO_INCREMENT,
	`Name` TEXT NOT NULL,
	`Description` TEXT NOT NULL,
	INDEX `Id` (`Id`)
)
COLLATE='latin1_swedish_ci';

INSERT IGNORE INTO product (Id, Name, Description)
VALUES 
(1, "Dependency Injection Principles, Practices, and Patterns", "Book by Steven van Deursen and Mark Seemann"),
(2, "Agile Software Development, Principles, Patterns, and Practices", "Book by Robert C. Martin"); 

Docker-compose.yml and .dockerignore

We take docker-compose.yml file described earlier in this article:

version: '3'

services:
  db:
    build: ./Db
  app:
    build:
      context: .
      dockerfile: Aspnetcoreapp/Dockerfile
    ports:
      - 8080:80
    depends_on:
      - db

Very last thing - .dockerignore file located next to solution (.sln) file. If you don’t know what is this file or why it has to be in this location, I’ve explained it in previous post. Mine currently looks like this:

.dockerignore
.env
.git
.gitignore
.vs
.vscode
*/bin
*/obj
**/.toolstarget
.idea

Running composed system

To run this example system, just go to solution directory, type docker-compose up command (explained earlier) and after a moment (DB initialization takes a while) you can open browser and see data pulled from DB with http://localhost:8080 GET method:

Persisting DB data

As you know, killed container is killed along with the data, which of course is not what we want from production database. To persist changes which we introduced to DB during production runtime, we need to use volumes, but I thought that it’s too much for simple introduction article. If you succeeded previous steps, now you can read about docker volumes. Maybe some day I will explain it in whole different post.

Summary

As you see, composing in general is simple, and basically requires one text file which instructs Docker how to run and couple containers together.

I hope you find this introduction useful. If there is something not clear, ask me a question in comment, and if you think it can help other too, share this article on Facebook or Twitter :)

Multi-project .net core solution in Docker image

2019-07-27T08:00:00+00:00

Creating Docker images from dotnet solution with project references is easy when you understand basics of Docker commands, but writing proper Dockerfile can be tricky for beginners.

Most of the examples show how to dockerize dotnet project, assuming that it has no local dependencies. So let’s analyse what can we do, when our project has references to other projects from solution. We will start with diving into simple example without dependencies first, to understand what changes we introduce and why.
If you just want to skip to solution and copy-paste it, of course you can do it, but it’s not recommended, because sooner or later you will be blocked with another obstacle due to not understanding what is happening, and in result you will waste more time. This article is good to start learning Docker instructions and commands, because everything is explained with short, plain language. We will use .net core 2.2 as it’s current version at the moment of writing.

Example dockerized dotnet core application is available on GitHub, feel free to use it for your needs.

Analysis of official Docker example

Official dockerize an .NET Core application article shows us this Dockerfile located in project folder (where .csproj file is stored):

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /app

# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore

# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out

# Build runtime image
FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "PROJECT_NAME.dll"]

and those two commands, to be run from project folder where Dockerfile is located:

docker build -t aspnetapp .
docker run -d -p 8080:80 --name myapp aspnetapp

Dockerfile FROM instruction

Our Dockerfile starts with FROM instruction:
FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
which means that we base our image on official Microsoft Dotnet Core SDK in version 2.2. We use SDK at this moment, not production runtime, because we will compile our application in Docker during building image. So you don’t even need to have .net core SDK installed on your host machine, this Dockerfile is prepared in a way that you won’t compile your app for Docker image yourself - Docker will compile it. You could use binaries built on your host machine, but it’s not safe - it may not work due to compatibility troubles. Why there is AS build-env instruction - we will come to this later (in Docker multi-stage build section).

Dockerfile WORKDIR instruction

In second line we see WORKDIR /app instruction which means, that following RUN, CMD, ENTRYPOINT, COPY and ADD instructions in our Dockerfile will be executed in /app directory. If it doesn’t exist it will be created (even if it wouldn’t be used).

Dockerfile COPY instruction

Next we see COPY *.csproj ./ instruction, which means that all csproj files from Docker build context will be copied to workdir (/app) directory inside Docker image. Docker build command will be explained later, but in short - build context is the directory from your host machine, pointed in Docker build command. If you point . path, the directory where you execute command is taken. So in our case we copy only one csproj file, because we run build command with project directory set as build context.

Dockerfile RUN instruction

Next is RUN dotnet restore instruction, which simply runs dotnet restore command in our workdir (/app) directory. At this moment inside /app directory in our image, is nothing but .csproj of our project, because we copied only it in previous step, but it’s enough for restoring nuget dependencies.

Copy and compile app source

Again we see COPY instruction - COPY . ./ to copy everything from our build context - in our case it means project files (.cs files etc.), because we run docker build command with project directory set as build context. Then with run instruction - RUN dotnet publish -c Release -o out we simply run dotnet publish in our workdir (/app) directory inside image, with -c Release -o out parameters. This dotnet command compiles our app with release configuration and publishes results in out directory (in our case /app/out). We can compile source because we base this image on developer’s sdk.

Docker multi-stage build

Once again we see FROM instruction, which sets on which image we base our image… How it’s possible to specify it again, with different base? It’s quite fresh Docker feature (since Docker 17.05 version) called multi-stage builds. When we use FROM keyword again, we mean that previous image specified above is temporary, and was used only to serve some purpose. In our case it was made only to compile our application - that’s why we used SDK as base image. Now we specify base image again and this time we are preparing our real image - the one which will be deployed to production, and this one doesn’t base on SDK, only on production runtime, which gives smaller size in result. We will just copy our compiled app from temporary image. So again we specify working directory to /app catalog and then we copy our binaries - COPY --from=build-env /app/out . which means copy files from /app/out/ from build-env image (that’s why we gave it a name in the first line) to current working directory (/app).

Dockerfile ENTRYPOINT instruction

Very last instruction in this Dockerfile is ENTRYPOINT, which (in simple terms) specifies a command that will be executed when the container starts. So in our case - ENTRYPOINT ["dotnet", "PROJECT_NAME.dll"] - Docker will run dotnet with PROJECT_NAME.dll parameter (which should be replaced to our project name of course), to start our app.

Docker build command

With such Dockerfile, we are told to run docker build -t aspnetapp . command in project directory (where Dockerfile is stored). Option: -t name (--tag name) is not mandatory - it allows to tag image (to name it and optionally give it a tag in ‘name:tag’ format), so don’t focus on it, and look on this command this way: docker build ., because important thing is after options - build context parameter. Build context is path on host machine which will be accessible during building image for Dockerfile instructions. In our case it’s . path which means that the directory where we run this command is passed as build context. Because we are told to run this command in project directory (where .csproj file is stored), our project files are passed as build context.

Docker run command

Docker run command creates container from image. Image is readonly manual for Docker, to create container, and container is working virtual machine where our app lives. We can think about it this way: image is like a class in object-oriented programming, and container is like an instance, created from this class. So we can create as many containers (instances) as we want, and it doesn’t affect image (class) - image is only necessary to let Docker know how to create container. We are told to run it this way:
docker run -d -p 8080:80 --name myapp aspnetapp
Without --detach option (-d) we will start seeing app console output from container. With --publish (-p) option we bind container’s port(s) to the host (by default with TCP, but you can specify UDP and SCTP as well). With --name option we assign a name to the container (without this option Docker will choose some funny name for us). At the end we pass image name, which Docker will read to create container. Because we named our image aspnetapp, we use this name here.

Solution

Proper Docker commands

Once we understand what happens in basic example, let’s see how to change it, to make it work when our project has references to other solution projects.
The problem is of course, that we run Docker build command from project directory passing . path as build context. This means, that files only from this directory will be accessible during building image, and depending projects are of course in other directories. We have several options to fix this. We can move Dockerfile one level up (to solution directory) and run docker build from there. But it’s recommended to have Dockerfile in project directory, to be able to have more than one Dockerfile in solution (for different projects). You could also run docker build as before (from project directory), but change build context path to one level up (..). In my view more elegant is third solution - to run docker build from solution directory, pass . as build context and to specify which Dockerfile we want to read with --file (-f) option, like this:
docker build -f PROJECT_DIRECTORY/Dockerfile -t IMAGE_NAME .

How to adjust Dockerfile

Next we need to adjust Dockerfile, because the one from official example assume that we have project directory as build context. My version looks like this:

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /app

COPY . ./
RUN dotnet publish PROJECT_NAME -c Release -o out

FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
WORKDIR /app
COPY --from=build-env /app/PROJECT_NAME/out .

ENTRYPOINT ["dotnet", "PROJECT_NAME.dll"]

I have skipped restoring nuget packages as single step to simplify, restore is included in dotnet publish and if it fails due to nuget failure, error message is legible. But if you have many nuget dependencies you may want to have separate step, because this way Docker treats it as distinct layer and reuses it if none of csproj file changed, which gives smaller build time (I have described it in another post). In this demo restoring nugets is fast enough to skip it, but remember about this if you have long building time (in particular - Restore completed in… time).

So we copy all projects (because build context is now solution directory) to /app directory inside container. Next from /app workdir we run dotnet publish command, specifying which project to compile - RUN dotnet publish PROJECT_NAME -c Release -o out - here PROJECT_NAME is the directory name with .csproj file inside.

Other instructions stays untouched with one small change - during copying compiled app from temporary image, this time we need to pass project name to path:
COPY --from=build-env /app/PROJECT_NAME/out .

Where to keep .dockerignore file

Official article says to add .dockerignore file to project directory, to make build context as small as possible and lower risk of invalidating cache, which of course is reasonable. But Docker CLI looks for .dockerignore file in root directory of the build context, so now we need to move it to solution directory. But in my view it’s even better, because we don’t need to create and maintain many .dockerignore files for many projects, we keep one for all of them. Example rules:

*/bin
*/obj
.dockerignore
.env
.git
.gitignore
.vs
.vscode
**/.toolstarget
.idea

Summary

When I had to dockerize .net core app for the first time I just took Dockerfile from mentioned article, copy-pasted Docker commands and when I faced obstacle I tried to solve it without analysis of how Docker works. After wasting some time this way, I wasted time again - trying to just copy-paste solution from the internet - again without analysis of what I do, and without success too. Then I learned (once again in my life…), that haste didn’t save time, but do the opposite - waste it. Because I didn’t find proper article nor tutorial to start I came though official documentation and manuals, which are written nice, but have too many details for beginners. This article shows essentials of analysis I took and explains basics with plain language. I hope this way it is good to start writing proper Dockerfile, without having trouble with situation like the one presented here - dotnet project with references to other projects from solution, nor with any other obstacle.

What to do when you have already containerized your app, but need to use some dependent system, for example DB? You can compose them together with docker-compose, which I have described with simple .NET Core app and MySQL DB as composed system example, in next article.

Don’t hesitate to write comment whether it was helpful for you, or to share it on Facebook or Twitter :)

Fare (xeger) - generate string that matches regex pattern in C#

2019-03-18T10:00:00+00:00

Matching text using regex patterns is extremely useful. But sometimes we need to invert this process and create text which will match the regex. The solution has already been written.

When I faced this problem for the first time, and quick search on the internet didn’t bring up anything useful, I thought:

I will implement simple generator. I know basic rules of how regex patterns work, I just need to invert them with correct random value.

Hey, I’m a software developer, so called Not Invented Here Syndrome is our frequent problem :) Looking through the internet I can tell that I’m not the only one who had the same idea about generating regex inputs.

So I started implementation, but after a while I realized that I don’t have enough time to satisfy all cases, so I focused only on basics, which could be enough for my problem.

Luckily I decided to repeat internet research, this time putting bigger effort into it. There were no many articles around this topic, but this time I found solution which was easy to use and better than my own implementation.

Fare project

Fare is a .NET port of libraries appreciated by Java community: dk.brics.automaton and xeger. The first library has focus in nondeterministic and deterministtic finite automaton. On top of it is built xeger, which is meant to be opposite of regex - it uses finite state machine for creating text which match passed regular expression.

In Java world you need them both. In .NET we have them under one project called Fare, which stands for Finite Automata and Regular Expressions.

It’s convenient library known by many companies as well as open source projects, for example AutoFixture or WireMock.Net.

Fare Hello world example

You can download Fare from nuget, it works for .NET Framework and for .NET Core. For example, I’m writing following instructions with Fare version 2.1.1 on Linux Mint 19 Tara in .NET Core 2.2 console application.

You just need to create Xeger object with your regex pattern (as string) and with Random object.

string regex = "t.m";

Xeger xeger = new Xeger(regex, new Random());

Console.WriteLine($"Input text matching regex: '{regex}' is: '{xeger.Generate()}'");

/* Example output:
Input text matching regex: 't.m' is: 't8m'
*/

Basically that’s it. In above example input will be random every time. To see how generated inputs might differ, let’s replace t.m pattern with t.*m pattern and loop generating ten times. As you will see, with .* pattern results might be any size, including 0 (see last input in my example, it’s real value copied from my console output).

string regex = "t.*m";

Xeger xeger = new Xeger(regex, new Random());

for(var i=0; i<10; i++)
    Console.WriteLine($"Input text matching regex: '{regex}' is: '{xeger.Generate()}'");
                
/* Example output:
Input text matching regex: 't.*m' is: 'trmYwmm'
Input text matching regex: 't.*m' is: 'tsm!tw-m:s}mm'
Input text matching regex: 't.*m' is: 't}}*3molmrxBGmmss-mm'
Input text matching regex: 't.*m' is: 'tLy4mmmm'
Input text matching regex: 't.*m' is: 'tCkm=!?iI|mm"LmmV}wm"63mmnb.G+mxzumNm`wn[m'
Input text matching regex: 't.*m' is: 'tux@mByyQ~8vxm'
Input text matching regex: 't.*m' is: 'tn|mmu npmj/~w#mmmmmm9mm'
Input text matching regex: 't.*m' is: 't}Zm'
Input text matching regex: 't.*m' is: 'tomX}8kkV{j)x<S}_mTmm'
Input text matching regex: 't.*m' is: 'tm'
*/

Deterministic Fare input

When we pass new Random() to Xeger constructor, Fare generates random input. Usually that’s what we want, but sometimes we need to have the same input for given regex every time (that was my production case). How to achieve that? You can simply pass constant seed to dotnet Random constructor, for example 0 :) That’s it, now under the same runtime your input generation sequence will be the same.

string regex = "t.*m";

Xeger xeger = new Xeger(regex, new Random(0)); // Note zero in Random constructor

for(var i=0; i<10; i++)
    Console.WriteLine($"Input text matching regex: '{regex}' is: '{xeger.Generate()}'");
                                
/* Output sequence in my environment will be every run the same:
Input text matching regex: 't.*m' is: 'tm'
Input text matching regex: 't.*m' is: 't~6x~bm^m'
Input text matching regex: 't.*m' is: 'tmoBlz=~z5mC1zvcmmqm'
Input text matching regex: 't.*m' is: 't|lym'
Input text matching regex: 't.*m' is: 'teDsmmmmy6m'
Input text matching regex: 't.*m' is: 'tmmmm'
Input text matching regex: 't.*m' is: 't~pm'
Input text matching regex: 't.*m' is: 't|5mmrw5|ommxmpNlo`x~G^wym'
Input text matching regex: 't.*m' is: 't%+u=mm~mrmmkm*mmmlU/w7\OyncmrDdx<lnm'
Input text matching regex: 't.*m' is: 't$X`w`mm:m~o~m'
*/

Benchmarks

Usually fare is quite fast and should be sufficient for production requirements. I have created a few benchmarks, with usual process priority, just for orientation. As you will see, reusing xeger is really fast (and generation time is repeatable) and creating xeger requires acceptable time as well. But remember that xeger with complex regex needs much more time than simple one, for example creating xeger with regex matching email address (taken from regextester.com) takes for me 50 milliseconds: _{^{^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$}}.

Measured operation	Preparation	Mean time
`new Xeger(_regex, _random);`	`_random = new Random(); _regex = "t";`	274.6 ns _^(nanos.)
`new Xeger(_regex, _random);`	`_random = new Random(); _regex = "t.m";`	105.9 us _^(micros.)
`new Xeger(_regex, _random);`	`_random = new Random(); _regex = "t.*m";`	160.7 us _^(micros.)
`new Xeger(_regex, _random);`	`_random = new Random(); _regex = "alice\|tom";`	343.6 us _^(micros.)
`new Xeger(_regex, _random);`	`_random = new Random();` _{^{_regex = "[a-z.!#$%&'+=?^_`{\|}~-]+@.(?:[a-z]{0,61}.)?(?:\\..(?:[a-z]{0,61}[a-z])?)*";}}	9.042 ms _^(millis.)
`_xeger.Generate();`	`_xeger = new Xeger("t", new Random());`	650.5 ns _^(nanos.)
_{^{for(int i=0; i<1000; i++) _xeger.Generate();}}	`_xeger = new Xeger("t", new Random());`	656.0 us _^(micros.)
`_xeger.Generate();`	`_xeger = new Xeger("t.m", new Random());`	1.420 us _^(micros.)
_{^{for(int i=0; i<1000; i++) _xeger.Generate();}}	`_xeger = new Xeger("t.m", new Random());`	1.437 ms _^(millis.)
`_xeger.Generate();`	`_xeger = new Xeger("t.*m", new Random());`	9.989 us _^(micros.)
_{^{for(int i=0; i<1000; i++) _xeger.Generate();}}	`_xeger = new Xeger("t.*m", new Random());`	10.05 ms _^(millis.)
`_xeger.Generate();`	`_xeger = new Xeger(_^"alice\|tom", new Random());`	2.217 us _^(micros.)
_{^{for(int i=0; i<1000; i++) _xeger.Generate();}}	`_xeger = new Xeger(_^"alice\|tom", new Random());`	2.183 ms _^(millis.)
`_xeger.Generate();`	_{^{_xeger = new Xeger("[a-z.!#$%&'+=?^_`{\|}~-]+@.(?:[a-z]{0,61}.)?(?:\\..(?:[a-z]{0,61}[a-z])?)*" , new Random());}}	8.435 us _^(micros.)
_{^{for(int i=0; i<1000; i++) _xeger.Generate();}}	_{^{_xeger = new Xeger("[a-z.!#$%&'+=?^_`{\|}~-]+@.(?:[a-z]{0,61}.)?(?:\\..(?:[a-z]{0,61}[a-z])?)*" , new Random());}}	8.160 ms _^(millis.)

_{^{^{BenchmarkDotNet=v0.11.4, OS=linuxmint 19 Intel Core i7-4702MQ CPU 2.20GHz (Haswell), 1 CPU, 8 logical and 4 physical cores

.NET Core SDK=2.2.105 [Host] : .NET Core 2.2.3 (CoreCLR 4.6.27414.05, CoreFX 4.6.27414.05), 64bit RyuJIT}}}

Downsides of Fare

Pattern with unspecified length might take long time to generate

Sometimes Fare wants to generate extremally long phrase for pattern which doesn’t specify length. I have seen cases for which xeger needed 10minutes to generate! Unfortunately for now there is no way to configure it in Fare (but it’s open source, you can improve it 😺). You can assume that it’s nondeterministic, because you’ll never know for which regex and with which random (or not random) value it will happen. What I do? Before I pass pattern to Xeger, I remove every * character. Asterisk means 0 or more characters, so this way it will always be one, which is correct, but of course it’s trade-off for more limited results. I know that there are still other possible regexs of unknown length, for example \w{3,} or \d+ but in my case it’s enough, I don’t expect such patterns. If it’s not enough in your case, then you should process other possibilities too, to remove every option which takes any length, or not to remove, but to provide simple values for those basic rules. Or fork Fare and allow to configure this behaviour in it :)

Always try..catch, and be prepared for bugs and exceptions

Unfortunately Fare has bugs. Sometimes it crashes, even with simple patterns. For example creating xeger with pattern <a.*b> will cause System.ArgumentException: 'a.*b' not found. Once I have seen a bug, which allowed creating not valid input. It turned out that Xeger treated multiplied start/end of line marks like regular characters. Input would be fine, but it had ^ or $ character at the beginning and/or at the end. Such pattern came to my system only once, but this particular one was important to me, so I changed my code to check whether generated input is valid and if it’s not, then try to fix it with trimming every ^ and $ characters. I know that it’s naive, but if it doesn’t help I just skip the pattern, it is allowed in my case (in general) - not every regex need to be inverted, occasionally I can skip one and it’s fine. You can reproduce this bug this way: new Xeger("^0$|^1$", new Random()) (in my environment it’s always 0$ or ^1).

You may find unsupported pattern

As you know regex is wide and you might find construction not implemented by xeger. But majority of patterns are satisfied and to be honest - I can’t tell which are not. There is no documentation in Fare project, and the one in Java’s xeger is not up to date, because I have checked some limitations they have mentioned and those not supported patterns (as they say) work properly… So there are probably some not implemented cases, but it’s minority :)

Summary

So to summarise, fare is a great library and does the job but… not entire job. You need to add your own few lines of code and get prepared for exceptional situations and bugs. In my case it’s enough and loosing input occasionally is acceptable in my situation. If it’s not acceptable in yours, I still recommend using Fare but with fixing/extending it first. That way or another it’s cheaper to start from theirs code base than writing everything from scratch. They’re open source so you can share your work with pull request, to help others and bring yourself a bit of splendour :)

Libsecret - remember Git credentials in Linux Mint and Ubuntu securely

2019-03-06T06:00:00+00:00

Typing username and password every push is burdensome and irritating… libsecret git credential storage to the rescue.

When you want to use external git repository hosting service, like Gitlab or GitHub you need to authorise yourself. Git has built in credential helper mechanism, which allows choosing the way you have credentials persisted. It gives you two options out of the box, but you’re not limited to them - you can install third party solution.

Cache credential helper (out of the box)

Cache is quite secure because keeps data only in memory. It’s fine for security, but every time you open new session, you need to type credentials again. Memory is purged after 900 seconds (15 min) by default, but it can be changed with optional timeout parameter.

If for some reason you don’t want to install anything, at least use cache :)

git config --global credential.helper 'cache --timeout=300'

Official description - git-scm.com/docs/git-credential-cache

Store credential helper (out of the box)

Store keeps your username and password in… plain text file! It’s totally insecure and use it only if you don’t care about your account (for example during some kind of workshops).

git config --global credential.helper store

Official description - git-scm.com/docs/git-credential-store

Git Credential Manager for Mac and Linux (by Microsoft)

Windows users are familiar with convenient Git Credential Manager for Windows which is integrated with OS way of storing credentials. It can be installed with official wizard or during git installation on Windows (there is checkbox to select). And recently Microsoft announced release of… Git Credential Manager for Mac and Linux. I was positively surprised, till I tried to use it on my Linux Mint 19 Tara. Installation was successful (but not without complications) and attempt to use it ended up with error:

Fatal: java.lang.RuntimeException encountered. Details: 
Secure credential storage is not available on this operating system. You may opt-in to store credentials in an unencrypted file under your user home directory by running 'git config --global credential.canFallBackToInsecureStore true'.
fatal: credential helper '!/usr/lib/jvm/java-11-openjdk-amd64/bin/java -Ddebug=false -Djava.net.useSystemProxies=true -jar /home/linuxbrew/.linuxbrew/Cellar/git-credential-manager/2.0.4/libexec/git-credential-manager-2.0.4.jar' told us to quit

I have opened the issue and till now I haven’t received any response (more than 2 months) and recently I have seen the same issue created for Ubuntu 18.04. So currently it doesn’t work on Linux Mint and Ubuntu.

Official description - github.com/Microsoft/Git-Credential-Manager-for-Mac-and-Linux

Libsecret git credential storage (by GNOME)

The best way to store Git credentials on Linux used to be GNOME Keyring (libgnome-keyring), but as it is specific to GNOME, it is deprecated since January 2014. For Git versions 2.11+ you should use credential helper based on libsecret. Installation and configuration takes only 4 bash commands :)

sudo apt-get install libsecret-1-0 libsecret-1-dev
cd /usr/share/doc/git/contrib/credential/libsecret
sudo make
git config --global credential.helper /usr/share/doc/git/contrib/credential/libsecret/git-credential-libsecret

That’s all, next time when you’ll be asked for your username and password will be the last time on this device :)

Official description - wiki.gnome.org/Projects/Libsecret

How to secure SSH credentials

Above credential managers work only if you use http(s), which is usual case. If you have to use SSH protocol, I have described how to skip asking for credentials with SSH key-pair in safe manner, in another article.

Summary

As you see it’s not difficult to protect your credentials and not have to type them all the time. If it doesn’t work for you or you have some question or unusual case, feel free to ask me in comments, I will do my best to help you.
If you know someone who uses unsafe credential manager or doesn’t use manager at all, send him this article, I will appreciate it :)

Welcome to Software Developer Blog

2019-03-04T10:00:00+00:00

Hey! Welcome to Software Developer Blog, it’s great to have your attention :)

I’m Tometchy and I’m about to share with you some coder thoughts.

Mainly code and frameworks focused solutions, but also general work environment tips and tricks, including version control, IDE, operating system and continuous integration/delivery. From time to time I plan to write about higher level solutions, like planning or design methodologies (for example Scrum or Event Storming).

My primary language is C# so it will probably dominate, but definitely won’t be lonely here. Some frontend and scripting languages will be here as well.

Despite I grew up with Windows and started professional collaborating with SVN, for the last few years I’m excited with Linux and Git, and great possibilities they offer. Therefore, expect focus on the latter, rather than the former.

Details about site structure are described in about section.

I hope you will have valuable time, and won’t hesitate to come back again :)

Software Developer Blog

Summary of The State of AI-assisted Software Development Report

What is DORA

AI Is Already the Standard, but It Doesn’t Replace Thinking

AI Amplifies What’s Good — and What’s Bad

With AI, Productivity Increases — but Stability Still Suffers

Platform Engineering Is the Foundation of AI Success

AI Adoption Is an Organizational Transformation

Summary

Replacing Planning Poker with Team Estimation Game

What is the Team Estimation Game

Keep the Big Picture during estimation

Preparation

Draw columns according to your scale

List players

Bring stories/tasks

The Game

Move one card or pass

The Results

The Matrix

Summary

Optimize Docker .NET Core project and docker-compose solution to reduce build image time

How Docker cache works

How to decrease image build time

Ignore what you don’t need

Try to create your own cache (yarn build example)

Reuse what you can

Experiment with caching nuget dependencies

Alternatively: build image from dedicated container

Summary

Introduction to Git Bisect with CodeceptJS automation example

How Git Bisect works

How Git Bisect knows that particular commit has given behaviour

Can Git Bisect be used to other purpose than finding a bug

Git Bisect manual demo

Git Bisect automated demo with CodeceptJS

Video demo

Summary

Initialize MS SQL in Docker container - create database at startup

Running MS SQL Server from Docker container without initialization

How to initialize MS SQL server in Docker container

How to initialize database with data

Docker commit

Docker volume

Script database schema and data

Troubleshooting MS SQL Docker initialization

Summary

Store Git SSH key pair in Linux with Libsecret to skip asking for credentials

How automate SSH connection works

How to configure automatic SSH connection

Create SSH keys

Encrypt SSH keys with password

Check created keys files

Send public key to server

Connect and store encryption password in keyring

Automate SSH login using command line

Troubleshooting SSH key pairing

Summary

Docker compose introduction - Dotnet core app composed with MySQL Database

Docker-compose file

Where to keep docker-compose file?

Docker-compose file example

Docker-compose commands

Docker-compose up

Docker-compose down

Example system

.NET Core app

MySQL DB

Docker-compose.yml and .dockerignore

Running composed system

Persisting DB data

Summary

Multi-project .net core solution in Docker image

Analysis of official Docker example

Dockerfile FROM instruction

Dockerfile WORKDIR instruction

Dockerfile COPY instruction

Dockerfile RUN instruction

Copy and compile app source

Docker multi-stage build