DevOps: Understanding Build and Package Manager Tools

This blog talks about why an understanding of build tools, and package management tools is important for DevOps Engineers.

In this blog we are going to build our knowledge of build tools and based on that we will establish how docker helps us with building code.

This is a beginner-friendly blog that assumes you do not know anything about building code, artifacts or package management tools.

Outcome of the Blog

By the end of this blog, you will know:

• What is Maven and grade, their difference and how does it help to build code for JAVA-based applications
• What is artifact and it's types and what is artifact repository
• How build tools and Docker are related
• The role of DevOps Engineers in this Building code phase

Motivation

DevOps Bootcamp | Techworld with Nana

6-month program to start your career as DevOps engineer

techworld-with-nana.com

Disclaimer: I am not selling anything to you, just my personal views.

The idea for this blog comes from the Bootcamp I am currently undergoing to understand and acquire the niche skills of a DevOps Engineer by Nana Janashia. This Bootcamp can really help you to learn those skills needed for DevOps engineers.

Prequisites

I did mention earlier that it is a beginner-friendly blog, but there are a few tools which are needed to follow along.
(Don't worry if you don't know I will post relevant links to brush up wherever needed :D)

1. IntelliJ IDEA
2. Maven
3. Gradle
4. Nodejs
5. Git

Why build the code?

• When you developed an application,(for our understanding,lets stick our discussion to java based application) it needs to be deployed on the production server so that people can consume/use it.
• Usually, any application will also have dependencies too, so how to move code from the local machine to the production server with dependencies?(Entire application ( main code + dependencies) will be big in size)
• To deploy the file, it is packed into a single movable file also called a package.
• This single package file is called an artifact.
• The process of packaging multiple files into one ( compressed form) is called building the code, tools which help us to build code are called build tools.
• For java based applications the popular ones are maven, gradle.

What kind of file is the artifact?

• Artifacts generated from maven or gradle are JAR or WAR file.
• Strictly Speaking of maven and Gradle based artifacts includes the application code + dependencies too but in the case of nodejs-based application tools package management tools not build tools ( will be discussed later) only contains the application code, not dependencies.

What is artifact repository?

• Artifact repository is place where artifacts are stored
• It keeps artifact storage and makes it convenient to deploy it multiple times, have backup etc
• Build tools have commands for publishing to artifact repository and then you van download from anywhere on server using wget or curl.
• Artifact repository - nexus, jfrog for java-based application artifact.
• Later in the blog we will see how docker helps us to use a single repository for artifacts.

Similarities Between Maven and Gradle and other tools.

• All are command line tools which have the options to run the code, test the code, build the code/zip, and publish to artifact repository.

• You can configure all these build tools like their build target directory, which files to include etc.

How build tool manages development and management dependencies?

• We need to use build tools locally when we are developing applications in order to test and run locally.
• Build too also helps us to manage dependencies by automatically installing them.
• Maven dependencies are stored in pom.xml, for gradle build.gradle. ( Will be discussed later how they manage dependencies)

Let's Build some JAVA Code

For this blog, use this GitLab repository which has a sample JAVA maven application, JAVA Gradle Application and React Node and clone it to your local environment.

• I would recommend using IntelliJ IDE as it will simplify the configuration process of JAVA for the project to a great extent.

• JAVA Maven application

• JAVA Gradle application

• React-Node applicatoion

Building JAVA app using maven

• maven requires us to configure how build will take place.
• In our repository since it is an application which is using Sprint framework we will need to configure maven to build the code for spring boot by adding a plugin in pom.xml ( dependency resolution file for java apps)

 <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>2.3.5.RELEASE</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

• To build, you can write the commands in the integrated terminal of IntelliJ.

mvc install

• After the build is complete, maven places the jar file inside the target folder.

Building JAVA app using gradle

• Gradle does not require any kind of configuration. Dependency resolution file for gradle is build.gradle.

• To build using gradle
  

./gradlew build    

• Gradles places the JAR file under build directory instead opf target directory unlike maven.

How to run the artifact

• Now we know we need to build the artifact and move it to server but how to run the artifact?

java -jar <name of the file>

Let's Manage NodeJS application

• NodeJs based application usually doe not have artifact type
• In fact, nodejs based application have NPM or YARN as their package management tool, they dont build anything.
• They don't have a structure or standard way to do it, unlike maven or Gradle.
• JS artifact are zip or TAR files.
• zip/tar file only contains application and not dependencies.

In order to run JS application on production server these are the steps

1) Copy artifact ( zip file) and package.json file to production server
2) install dependencies
3) unpack zip/tar artifact
4) run the App

npm pack command # create tgz file containing package.json file too along with source code.
npm test. # for testing
npm publish # for publish to artifact repository

How to manage Building for frontend and backend

Usually, frontend and backend of an application are seperate. Common pattern include like using React for front end with backend based out of nodejs or JAVA.

• In any case either a separate artifact can be produced or common artifact can also be the option, which means seperate package.json for the former if backend is also based out of nodejs.

• Note: React code needs to be bundled, transpilled, and compressed. A tool called webpackwhich produces server.bundle.js

• webpack also has a command line interface

• To use webpack, use these commands
  

cd api 
npm install // for downloading webpack dependency
npm run build

• Webpack produces a minified version of the entire react code

Relation between Build tools and Docker

Modern architecture involves micro services sometimes inside a single project one microservice can be JAVA based and another can be nodejs or python based. Does that mean we need to handle different artifact types?

• With docker, we don't need to build and move different artifact types.
• There is just one artifact type - Docker images
• Now, we build Docker images from the application and we don't need a repository for each file type, no need to move multiple files to the server like package.json, no need to zip, just copy everything into the docker filesystem and run it from docker image.
• Docker image is also an artifact
• In order to start the application you don't need npm or java on the server, execute everything ( command to run the JAR or node application) inside the docker image.

Note - we don't have to create zip or tar files anymore because we can copy JS files into a docker image. But we still need to build apps outside the docker image and then copy it inside the docker image. For eg webpack or in the case of JAVA create a JAR file , then copy JAR file or bundle.js will create a docker image.

// for gradle app 

FROM openjdk:8-jre-alpine

EXPOSE 8080

COPY ./build/libs/java-app-1.0-SNAPSHOT.jar /usr/app/
WORKDIR /usr/app

ENTRYPOINT ["java", "-jar", "java-app-1.0-SNAPSHOT.jar"]


// for nodejs ap


FROM node:10 AS ui-build
WORKDIR /usr/src/app
COPY my-app/ ./my-app/
RUN cd my-app && npm install && npm run build

FROM node:10 AS server-build
WORKDIR /root/
COPY --from=ui-build /usr/src/app/my-app/build ./my-app/build
COPY api/package*.json ./api/
RUN cd api && npm install
COPY api/server.js ./api/

EXPOSE 3080

CMD ["node", "./api/server.js"]

Our role as DevOps Engineers

Now it's interesting to know where our knowledge as a DevOps engineers will come into the picture.

• Developers uses these tools locally and configure the tools to run application and test. We don't have much to do in this step.
• We come into the picture for building the artifact, because we know where and how it will run.
• Developers don't build the app locally, building the artifact -> docker image -> push to repo and run on the server is the responsibility of DevOps in build automation tools like Jenkins.
• Our job is to configure the CICD pipeline and build automation which includes, install dependencies -> run tests -> build app -> push to repo as a docker image.
• execute tests on build servers like mvn test or npm test, build and package into the docker image.

Till then, Happy Learning !!!

Feel Free to post any comments or questions.

Connect with me over Linkedin or Twitter.