Apache NiFi docker

This is a guide to start apache NiFi as docker machine.

We can use aktechthoughts/nifi-docker repository to start a NiFi cluster.

STEP I : Clone the repository

STEP II: Execute the script build.sh

The script pulls the images and runs the zookeepr and the nifi on local machine.

The nifi is exposed at port : 8888, we can look at the browser using http://0.0.0.0:8888/nifi/

This is a blank canvas. we can add some process group and use the service.

The template – SEND_REMOTE_DATA is imported and used.

This is a insecure cluster. We can create a secure cluster using repository https://github.com/aktechthoughts/nifi-cluster

Creating a docker instance on AWS using Ansible.

There are many advantages of infrastructure automation. We can implement any software project without worrying about the system configuration.

This project utilizes ansible for creating an ec2 instance dynamically and then creates a docker instance on the remote ec2.

You can simply clone the repository in your local machine and create an ec2 instance just by changing the parameters.

Prerequisites

python3
ansible 2.5
ubuntu 18.04 LTS

STEP 1: Add AWS login details in the  ‘vault.yml’ .

AWS login credentials are required to create the ec2 instance. These can be stored in the ~/.aws directory or this can be added into the ansible vault using ‘ansible-vault’ command.

Execute below command and add aws_access_key_id and aws_secret_access_key in the file created by the command.

$ sudo ansible-vault create vars/vault.yml
[default]
aws_access_key = XXXXXXXXXXX
aws_secret_key = XXXXXXXXXXXXXX

STEP 2: Initialize AWS variables for the configuration of ec2 instance.

vars/vars.yml contains variables like AWS- region id, VPC id, subnet id which is essential for creating ec2. These should be changed according to the AWS login.

aws_region_id : eu-central-1
aws_vpc_id : vpc-07db38c12d26359ac
aws_sub_id : subnet-042581e3da666d445
aws_ami_id : ami-050a22b7e0cf85dd0

aws_instance_type : t2.micro

aws_sec_key : abhishek-vpc

IdFile : /home/abhishek/.aws/abhishek-vpc.pem

default_image : jenkins

public_ip : ''
instance_id : ''

terminate : true

public_ip and instance_id are variables that is initialized at the runtime.
The variable ‘terminate’ can be set to true or false, depending on the final state of the instance.

The variable default_image can be changed based on the required image.
This image is pulled from https://hub.docker.com/

STEP 3: Execute the ansible-playbook to create docker on ec2.

$ sudo ansible-playbook create_ec2.yml -i hosts.ini --ask-vault-pass

The command will ask the vault password which is created in the STEP1. It will create a docker on ec2 based on the configuration available in create_ce2.yml.

Acknowledgments

• Erika Heidi – Digital Ocean – anisble-cheat-sheet
• C Tarwater – chrisanthropic.com – ansible-ec2-aws

 

Dockerizing Jenkins Pipeline – Simplilearn project

This document is an explanation of the project done for simplilearn-devops-certification. It creates a Docker image using Dockerfile and publishes it to docker hub using Jenkins pipeline. The code can be found on the github.

STEP 1 Set up a VS code workspace and the Github repository.

• Open VS Code.
• Create a directory “simplilearn-devops-certification” in the terminal and change directory.
• Run “git init” to initialize repository.
• Create a repository “simplilearn-devops-certification “ in the github.com
• Create a file name “README.md” add details of the project in the file.
• Execute the steps to do initial commit, This will add the project in the GitHub master branch.

1. 1. 1. git remote add origin https://github.com/aktechthoughts/simplilearn-devops-certification.git
      2. git add  .
      3. git commit –m “ Initial Commit”
      4. git push –set-upstream origin master

STEP 2 Set up a Jenkin Server and a docker-machine.

• Download Jenkin.war from http://mirrors.jenkins.io/war-stable/latest/jenkins.war
• Execute this on a terminal of the machine using

“java -jar D:\Softwares\jenkins.war –httpPort=8080”

• The previous command will run the Jenkin server at localhost:8080 port and it can be accessed in the browser using http://localhost:8080/
• Select “Install Selected packages” and wait for installation to finish.
• Create a new user after installation is finished.
• Install a docker on the same machine where Jenkin is installed

STEP 3  Setup Jenkinsfile in the repository

1. Create Jenkinsfile in the project root directory.
2. Add below content in the file.

• The script has four stages in the Jenkins.
  1. Building Image
  2. Deploying the image in the dockerhub repository
  3. Removing the Image from Jenkin node.
  4. Executing Image from dockerhub.

STEP 4 Register and open https://hub.docker.com/ with your own login.

• Create a new docker reposiory named ‘simplilearn-devops-certification’
• Create a file ‘Dockerfile’ in the project created in STEP1
• Add following conent in the file.

 

Create a pipeline in jenkins and execute the pipeline to publish the image to the docker hub.