03, March 2024

2024-03-27

• Docker Swarm

1. Rolling Updates

To reduce the downtime of an application and also to introduce minimal downtime for rolling updates while docker swarm. Here lets us take an example image of alpine:3.17.7 and with the command to ping google.com, to demonstrate the rolling updates,let us update the image to alpine:3.18.6 while the container is running.

Pre-requisites are the enable the docker swarm mode, and add a worker node to the cluster. docker service update --image alpine:3.18.6 helloworld

2. Rollback Updates

To continue from the above rolling updates, if many such cases we would want to rollback to the previous state of the service, due to a bug etc, in our example we would want to rollback to our previous state which is from alpine:3.18.6 to alpine:3.17.7 we use the command docker service rollback <service-name>

[node1] (local) root@192.168.0.12 ~
$ docker service rollback helloworld
helloworld

• API Architecture

SOAP,REST,GraphQL and gRPC are most popular API architecture, in this digitizing world where large amounts of data is being transferred

1. REST REST stands for REpresentational State Transfer, is a widely used architectural style for building web API’s. REST API’s uses HTTP requests to retreive and manipulate data. Key characteristics of REST include:

Resource Based: REST API’s are based on concept of resources,which are exposed through URL’s. Stateless: REST API does not maintain the state between two requests,hence stateless. CRUD: REST API’s support Create,Read,Update and Delete operations for manipulating data. Flexible: REST API’s are flexible,used for many purposes in general.

2. GraphQL GraphQL is query language for API’s, which is more flexible than REST. Allows clients to specify exactly what data they need, which reduces the data transferred over network.Key characteristics:

Strongly Typed: clients and servers agree on the structure of the data exchanged. Query Based: clients specify exactly what they need and receives only that data. Single Endpoint: GraphQL has single endpoint,which simplifies client-server communications Schema driven: API schema defines the type of data available and operations that can be performed on that data.

GraphQL uses POST HTTP method for all the operations be it GET,POST etc it uses POST HTTP method.

3. gRPC gRPC Remote Procedure Call open source framework developed by Google for building Remote Procedure Calls API’s.Unlike traditional API’s that rely on HTTP methods,gRPC uses more efficient and lightweight Protocol Buffers as the datainterchange format and it supports multiple programming languages. Using protocol buffers,faster and efficient communication is achieved.gRPC supports bidirectional streaming real-time communication between clients and servers.

2024-03-05

• Kafka Integration with Spring Boot

Kafka is a distributed event store with stream processing message, handles low latency platforms for real time data feeds. Kafka allows users to subscribe and publish to the kafka broker.

A general use case with different micro services: User,Account and SMS microservice

Integration of the above microservices with Apache Kakfa messaging event.Below is the link for the detailed documentation and steps to integrate a spring boot project with kafka in a microservices architecture.

Repository Link

Docs link

• Twilio SDK Integration

Twilio Integration with Spring boot to send SMS through a Twilio trial account.

Reference Link

• Using System Environment Variables in application.properties

Usage of system environment variables in application.properties for example, mysql username and password,url etc making confidential information secure.

2024-03-06

• .gitkeep

git only tracks files and not empty directories,for instead we want to create an empty directory in a project directory, instead of creating a dummy file, a standard practice is to create a .gitkeep file within this empty directory.

• git stash

git stash is useful,when you want to switch to another branch, but there are few local changes in your current branch and you dont want to commit them,in this scenario git stash can be used.

git stash apply - apply the stashed changes to the working project

git stash pop - pop’s the changes and applies to working set.

2024-03-12

Java 8 Features

• Functional Interfaces

  Functional Interfaces: Consumers, Suppliers, Predicate Functions

  Why functional interfaces?

  Lambda Expressions: Functional interfaces with lambda expressions

  Examples of consumers,predicates,functions and suppliers with syntax and constructors

• Lambda Expressions

• Stream API

  Difference between Sequence streams and Parallel streams.

  How Streams work internally?

  Intermediate operations, terminal operations, short circuit operations.

• Method Reference

  Method reference in java,calling static methods,instantiated methods and so on.

Documentation Link

2024-03-13

• Reactive Programming

  Need for reactive programming

  Reactive programming is a programming paradigm that promotes an asynchronous, non-blocking, event driven approach to data processing. Let us understand the difference between blocking and non blocking request processing.

  1. Blocking Request Processing
  2. Non Blocking Request Processing

• Reactive Manifesto

• Back Pressure

2024-03-14

Openshift

Redhat OpenShift is a platform that allows us to manage,deploy and orchestrate containerized applications in the cloud.Red hat openshift solves the biggest problem of developing and deploying microservice based applications.

oc command lets you work with the entire OpenShift cluster and deploy new applications.The CLI exposes the underlying Kubernetes orchestration system with the enhancements made by OpenShift. Users familiar with Kubernetes will be able to adapt to OpenShift quickly.

oc login -u admin -p admin https://api.crc.testing:6443 --insecure-skip-tls-verify=true the command to login into openshift for the user (-u option) admin with password (-poption above) admin oc whoami : displays the logged in openshift user.

Containers

Getting started with containers

Like a normal linux program,containers have two states - rest and running. When the container is at rest, a container is a like a file that is stored on the disk like other files.This is referred to as Container Image or Container Repository. When the container is started, the Container Engine unpacks the required files and meta-data, then hands them to the linux kernel.The next process is to start the container just like another linux process making an API call to the linux kernel. This API call makes an isolation and mounts a copy of the files stored in the container image.

• Registry Server

A container registry server is a tool which hosts and distributes container images. The container registry server can either be public or private.These container registries connect directly to container orchestration platforms like Docker and Kubernetes.

• Container Image

A container image, which is a file, pulled from the registry server and used locally as a mount point when the containers are started

2024-03-18

Containers

Containers have existed within operating systems for quite a long time, A Container is a runtime intiative of container image. Linux containers are often isolated through the use of cgroups,SELinux,AppArmor.

Overview of security like cgroups,AppArmor in Debian.

Container Terminology:

- Registry Server: A container registry server is a tool which hosts and distributes container images. The container registry server can either be public or private.These container registries connect directly to container orchestration platforms like Docker and Kubernetes.

- Container image: A container image, which is a file, pulled from the registry server and used locally as a mount point when the containers are started.LXD pulls a single container image from the registry server,while docker uses OCI-based images which can be made of multiple layers.

- Container Engine: A container engine pulls the container image from the registry server,accepts user's requests via command line options and runs the container. There are many container engines like CRI-O, Docker, LXD.Most container engines do not run containers, they rely OCI compliant runtime like runc.

- Container Host: container host is the system that runs the containerized process(containers). For example a VM in a data center is a host, while containers can be run on them.

- Image,Tag: Images: Repositories are often reffered to as images or container images,but they are actually made up of one or more image layers.Image Layers are usually referenced by parent child relationship.

Tag: Defining versions for the container images with the help of tags.

- Container Orchestration: A container orchestrator does two things:

Dynamically schedules container workloads, within a cluster of computers. Provides a standardized application definition file (kube file, docker compose).

Docker

Docker is within a sandboxed environment, it allows users to package an application with all of its dependencies into a standardized unit for development.Provides a layer of isolation from the host,where the container uses the host’s system resources like storage,memory and CPU.

Architecture - Docker Engine - Docker Client - Docker Registry

Docker commands

- `docker pull`: pulls the image from the registry server
- `docker images`: displays all the images present locally
- `docker ps -a`: list of containers currently running and terminated
- `docker run`: creates a new container from the image provided
- `docker exec`: runs a new command within the container,example bash prompt
- `docker stop`: stop the existing docker container
- `docker rm`: delete the docker container without any traces on local
- `docker rmi`: delete the container image locally

Detailed explanation%

2024-03-19

docker logs

`docker logs , displays the logs of the container.

docker kill vs docker stop

Container doesn’t stop for a SIGTERM signal sent by a stop command, to terminate the process stop follows the SIGTERM with a SIGKILL after a grace period, instead it is faster to use docker kill which terminates the container immediately.

docker search

To search for images within the registery server i.e here Docker Hub we can use the command docker search <container-name> say for example, docker search debian will list out the images with the repository owner and official status.

docker prune

prune command deletes all the unused and stopped containers,unused build cache,unused images from the host system,below command demonstrates the examples of the docker prune command.

docker build

docker build command builds docker images from a Dockerfile, which is a file that contains all the commands a user could call from the terminal.Docker builds images by reading instructions in Dockerfile.

A docker instruction has two components: INSTRUCTION and ARGUMENT.

Docker Instructions

• FROM: base image we want to start from
• RUN: run commands during the image build process
• ENV: sets Environment variable during the build process and also while the container is created.
• COPY: command copies a a file or folder from the host to the docker image.
• EXPOSE: to expose a port on the docker image at runtime.
• ADD: advanced version of COPY command,to copy files,tarballs to the docker image from the host,we can also define instructions to extract the tarball on the docker image.
• WORKDIR: set the current working directory in the docker image.
• VOLUME: mount the volume to the docker container.
• USER: sets the username and UUID when running the container,for setting the non-root user.
• LABEL: sets metadata information of Docker image.
• CMD: executes a command within a running container, only one CMD command is allowed,if multiple are present only the last one is takes effect.
• ENTRYPOINT: commands that will execute when the Docker container starts.

Defining Start conditions for container

Let us consider a sample project where we run yt-dlp against a youtube link which downloads the video, let us build the Dockerfile by giving instructions.

To construct a container image we should first select a base image,here debian:bookworm as a base image to build yt-dlp image, the first line of the Dockerfile will be

FROM debian:bookworm

To get the binary of yt-dlp, we need curl and to run the yt-dlp we need python3 so these two packages has to be installed before running the binary,so the Dockerfile with additions will be

FROM debian:bookworm
RUN apt update -y && apt install curl python3 -y

Fetching the yt-dlp binary from github and copying it to /usr/local/bin and changing the permissions of the binary to executable of all.

FROM debian:bookworm
RUN apt update -y && apt install curl python3
RUN curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
RUN chmod a+x /usr/local/bin/yt-dlp
CMD ["/usr/local/bin/yt-dlp"]

Upon creating the image and creation of container yt-dlp is to be launched.

$ docker run yt-dlp

  Usage: yt-dlp [OPTIONS] URL [URL...]

  yt-dlp: error: You must provide at least one URL.
  Type yt-dlp --help to see a list of all options.

$ docker run yt-dlp https://www.youtube.com/watch?v=uTZSILGTskA

  docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "https://www.youtube.com/watch?v=uTZSILGTskA": stat https://www.youtube.com/watch?v=uTZSILGTskA: no such file or directory: unknown.
  ERRO[0000] error waiting for container: context canceled

According to the above error the youtube link should be taken as an argument to the binary yt-dlp, but here the yt-dlp is executing as a command,updating the CMD TO ENTRYPOINT

FROM ubuntu:22.04

WORKDIR /mydir

RUN apt-get update && apt-get install -y curl python3
RUN curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
RUN chmod a+x /usr/local/bin/yt-dlp

# Replacing CMD with ENTRYPOINT
ENTRYPOINT ["/usr/local/bin/yt-dlp"]

$ docker run yt-dlp https://www.youtube.com/watch?v=DptFY_MszQs
[youtube] Extracting URL: https://www.youtube.com/watch?v=DptFY_MszQs
[youtube] DptFY_MszQs: Downloading webpage
[youtube] DptFY_MszQs: Downloading ios player API JSON
[youtube] DptFY_MszQs: Downloading android player API JSON
[youtube] DptFY_MszQs: Downloading player 9bb09009
[youtube] DptFY_MszQs: Downloading m3u8 information
[info] DptFY_MszQs: Downloading 1 format(s): 22
[download] Destination: Welcome to Kumpula campus! ｜ University of Helsinki [DptFY_MszQs].mp4
[download] 100% of   29.92MiB in 00:00:04 at 7.10MiB/s

Now when the container is run when passed the youtube link as argument the video is downloaded, but it is stored onto the container,here comes the concept of volumes where data is stored.%

2024-03-20

Alpine Linux

To install a package: apk add <packagename>, for example to install maven and openjdk17 we use the command apk add openjdk17 maven

Dockerize Spring Boot Application.

FROM debian:bookworm
RUN apt update -y && apt install default-jdk maven git -y
RUN git clone https://gitlab.com/vinay-keshava/spring-boot-hello-world.git
WORKDIR /spring-boot-hello-world
EXPOSE 9328
RUN mvn clean install -DskipTests
ENTRYPOINT ["java","-jar","target/demo-0.0.1.jar"]

Selecting debian:bookworm as base image,installing jdk and maven packages, along with this exposing the port 9238 and building the project. After successfull build of this docker image to create a container use the command docker run -dit -p 8080:9328 <imagename> where 9328 port number is being binded to port 8080 on the host

FROM mysql:8.0.36-bookworm
ENV MYSQL_PASSWORD=root\
    MYSQL_DATABASE=customersurvey\
    MYSQL_ALLOW_EMPTY_PASSWORD=false\
    MYSQL_ALLOW_RANDOME_PASSWORD=false
RUN apt update -y && apt install git maven default-jdk -y
RUN git clone https://gitlab.com/vinay-keshava/customer-survey.git
WORKDIR /customer-survey
EXPOSE 8088
RUN mvn clean install -DskipTests
ENTRYPOINT ["java","-jar","target/CustomerSurvey-0.0.1.jar"]

Fetches the mysql image from the docker hub and setting ENV environment variables, and installing maven and default-jdk packages and then changing the working directory to the project which is cloned from gitlab, running mvn install command and upon starting the container running the jar file of the project

Docker Volumes

What if you want to store information outside the docker container, that can be used as shared storage for other containers. Docker volumes are mounted to filesystem paths onto the container, where the actual data is stored on the host.With the help of drivers data is written in different services.

Example to pull a nginx docker image and syncing file and directories across containers and docker volume.

• To create a volume: docker volume create nginx-data, where nginx-data is the volume name,where the volume is created at /var/lib/docker/volumes/nginx-data/_data.

• docker inspect nginx-data displays information at what date the volume was created,mountpoint,drivers,etc.

$ docker run -itd -v /var/lib/docker/volumes/nginx-data/_data/:/var/www/html nginx

what ever data in /var/www/html inside the container is present is synced to the host nginx-data. To verify the working of docker volume, [inside the container] change to the directory /var/www/html and create a sample file like hello_world.java and [on the host] when the files are listed under /var/lib/docker/volumes/nginx-data/_data hello_world.java shows, there by verifying the shared volume.

Docker Ports

To expose a particular ports we use the -p flag and in Dockerfile we use the keyword EXPOSE <PORTNUMBER> to expose a particular port

• docker login command to login to hub.docker.com to push container images.

GnuPG import keyword

To import the private GnuPG key we use gpg --import private.asc and change the level of trust by editing the key and updating the trust level to ultimate.%

2024-03-26

• hostname : set and view hostname of the system

  • -i flag prints ip address of the hostname network

• Docker Swarm

  • doker swarm init --advertise-addr <ip-address>: to initialize swarm mode.
  • docker node ls: displays all members including manager and worker nodes.

• Pravserver debugging the issue for inactive accounts, which are inactive for more than 28days.

  • https://codeberg.org/prav/pravserver/src/branch/master/src/main/java/im/quicksy/server/configuration/Configuration.java#L54 : line 54 shows the limit set for inactive accounts
  • https://codeberg.org/prav/pravserver/src/branch/master/src/main/java/im/quicksy/server/controller/PasswordController.java#L122 (Line 122-142) if the account is inactive for more than 28days unregister and register account through the EjabberdApi.

2024-03-27

• Docker Swarm
  • Deploying a service : docker service create --replicas 1 --name helloworld alpine ping google.com
  • Verify and Inspect service: docker service inspect --pretty helloworld
  • Scale Service: docker service scale helloworld=5 to increase the number of replicas.
  • Delete a service: docker service rm helloworld deletes the service.