Creating a docker instance on AWS using Ansible.

There are many advantages of infrastructure automation. We can implement any software project without worrying about the system configuration.

This project utilizes ansible for creating an ec2 instance dynamically and then creates a docker instance on the remote ec2.

You can simply clone the repository in your local machine and create an ec2 instance just by changing the parameters.

Prerequisites

python3
ansible 2.5
ubuntu 18.04 LTS

STEP 1: Add AWS login details in the  ‘vault.yml’ .

AWS login credentials are required to create the ec2 instance. These can be stored in the ~/.aws directory or this can be added into the ansible vault using ‘ansible-vault’ command.

Execute below command and add aws_access_key_id and aws_secret_access_key in the file created by the command.

$ sudo ansible-vault create vars/vault.yml
[default]
aws_access_key = XXXXXXXXXXX
aws_secret_key = XXXXXXXXXXXXXX

STEP 2: Initialize AWS variables for the configuration of ec2 instance.

vars/vars.yml contains variables like AWS- region id, VPC id, subnet id which is essential for creating ec2. These should be changed according to the AWS login.

aws_region_id : eu-central-1
aws_vpc_id : vpc-07db38c12d26359ac
aws_sub_id : subnet-042581e3da666d445
aws_ami_id : ami-050a22b7e0cf85dd0

aws_instance_type : t2.micro

aws_sec_key : abhishek-vpc

IdFile : /home/abhishek/.aws/abhishek-vpc.pem

default_image : jenkins

public_ip : ''
instance_id : ''

terminate : true

public_ip and instance_id are variables that is initialized at the runtime.
The variable ‘terminate’ can be set to true or false, depending on the final state of the instance.

The variable default_image can be changed based on the required image.
This image is pulled from https://hub.docker.com/

STEP 3: Execute the ansible-playbook to create docker on ec2.

$ sudo ansible-playbook create_ec2.yml -i hosts.ini --ask-vault-pass

The command will ask the vault password which is created in the STEP1. It will create a docker on ec2 based on the configuration available in create_ce2.yml.

Acknowledgments

• Erika Heidi – Digital Ocean – anisble-cheat-sheet
• C Tarwater – chrisanthropic.com – ansible-ec2-aws

 

Creating Exasol Docker Instance

Introduction

Exasol is an analytics database management software, this is an in-memory, column-oriented, relational database management system.

It supports  SQL Standard 2003 and can be integrated via standard interfaces like ODBC, JDBC or ADO.NET.

Creating Exasol Docker instance

Exasol docker instance can be created with the help of GitHub page.  This requires docker tool installed on the system.

host-machine:~$ sudo docker run –name exasoldb -p 8899:8888 –detach –privileged –stop-timeout 120 -v exa_volume:/exa exasol/docker-db

The command above will create exasoldb container. It will expose exasol container port 8888 on the host machine at port 8899. It means exasol database would be available at connection string host-system:8899.

It will also configure exasol database in persistent mode. i.e., the objects created will remain stored in the system,

Starting the exasol instance

The command in the last paragraph will create a exasol instance in a docker container. we can retrieve the container id using below command.

host-machine:~$ sudo docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d4f553b20771 exasol/docker-db “/usr/opt/EXASuite-6…” 6 days ago Up 53 minutes 0.0.0.0:8080->80/tcp, 0.0.0.0:8899->8888/tcp exasoldb

d4f553b20771 is the container id. This is a reference to the container which is running the exasol. Use this id to start the container as given below.

host-machine:~$ sudo docker container start d4f553b20771 

Stopping the exasol database gracefully 

Exasol database is a system which needs to be stopped carefully. If the host-system is shutdown with stopping the database. The tables created may be lost or system may not start again.

host-machine:~$ docker exec -ti exasoldb dwad_client stop-wait DB1

The command above will stop the database container. The host-machine can be shutdown or reboot.

Connecting using sql client.

A JDBC driver can be downloaded from exasol official download.  I used Dbeaver to connect. Default user name is : sys and password is exasol.

Note – Docker image is not officially supported by exasol, but still can be used for simple use-cases.

 

 

 

Booting PC over the Network

Basics of Network Booting

Network Boot is industry standard method to load Operating System from network. It was designed to boot disk -less devices when when the cost of disk was high. It is still useful for data centers where servers are made available on demand.

The core concept behind this is same as BIOS (Basic I/O System). The system loads a small piece of software which in turn loads other OS from disk and finally makes the computer live.

PXE (Preboot eXecution Environment) is widely used Network boot software. It was developed by Intel corporation. There are other open source version of Network boot software called iPXE which has more features.

BIOS is hardwired into the mother board, while PXE is encoded into a chip in the network card. All the computers are configured to load BIOS first – BIOS subsequently selects a device to transfer its control. BIOS maintains a list of items ( Hard disk, USB, CD-ROM, Network) which is called boot order. The devices on the lists are selected one by one, whichever is  available first, that device receives control from BIOS.

To boot from Network, we need to configure the BIOS “boot order” and make the Network as first boot device – This can be done in the Boot menu of the BIOS setting. You can go in BIOS setting by pressing DEL or F2 key multiple times when the PC is booting. Some new motherboards may have different setting.

When a network boot program (NBP) receives a call from BIOS, it is loaded into memory and control is passed to this program. This program has driver software for the network card and a DHCP client.  As the intention is to load system over the network, NBP configures a network connection – It sends DHCP request over the wired network and receives a set of IP address a from DHCP  (Dynamic Host Configuration Protocol) server. DHCP server is a program which handles request from a client for IP address. Every network has a DHCP server which serves IP address to the clients.

Configure DHCP server

Below is typical DHCP service call from a client for IP address. The DHCP server can assign IP address as well as a next-server to clients. ‘next-server’ is the address or dnsname for the TFTP server.

DHCP server has a configuration file called “dhcpd.conf” at /etc/dhcp/ directory. It has configuration related to IP addresses, leasing time and other parameters related to dhcp configuration.

subnet 192.168.3.0 netmask 255.255.255.0 {
          range 192.168.3.10 192.168.3.45;
          range 192.168.3.55 192.168.3.60;
          option routers 192.168.3.1;
          option broadcast-address 192.168.3.255;
          default-lease-time 3600;
          max-lease-time 7200;
          next-server TFTP_server_address; 
          filename “/tftpboot/pxelinux.0”;
}

The two parameters ‘next-server’ and ‘filename’ are important for network booting. TFTP server is Trivial FTP server which runs on the port 66, any client can request for the files hosted, and the files are served without authentication.

‘filename’ is the NBP binary hosted on the home directory for the TFTP server.  There are many opensource version of TFTP servers available – You can follow this to install TFTP server on Ubuntu machines.

Configure PXE/iPXE Boot Menu

The structure of the tftp root directory (tftpboot) is shown below. It hosts the file pxelinux.0. This file is  lightweight master boot record boot loaders from syslinux. There are some other files vesamenu.c32, ldlinux.c32, libcom32.c32 and libutil.c32 required by pxelinux.0.

All these files can be extracted  from syslinux website here.

pi@raspberrypi:~/d_drive/tftpboot $ ls -lrt
total 477
drwxrwxr-x 1 pi pi      0 Oct 27 20:02 syslinux
-rwxrwxr-x 1 pi pi  42143 Oct 27 20:33 pxelinux.0
-rwxrwxr-x 1 pi pi  26692 Oct 27 20:33 vesamenu.c32
-rwxrwxr-x 1 pi pi 116556 Oct 27 20:33 ldlinux.c32
-rwxrwxr-x 1 pi pi 181952 Oct 27 20:33 libcom32.c32
-rwxrwxr-x 1 pi pi  23636 Oct 27 20:33 libutil.c32
drwxrwxr-x 1 pi pi      0 Oct 27 21:11 img
-rwxrwxr-x 1 pi pi    216 Oct 27 21:29 HTTPServer.py
drwxrwxr-x 1 pi pi    4096 Nov 5 21:53 pxelinux.cfg
-rwxrwxr-x 1 pi pi   67227 Nov 6 19:30 undionly.kpxe
drwxrwxr-x 1 pi pi   4096 Nov 15 13:49 iso
drwxrwxr-x 1 pi pi   4096 Nov 18 12:08 scripts
drwxrwxr-x 1 pi pi   4096 Jan 20 14:05 cgi-bin
pi@raspberrypi:~/d_drive/tftpboot $ cd pxelinux.cfg/
pi@raspberrypi:~/d_drive/tftpboot/pxelinux.cfg $ cat default
default vesamenu.c32
MENU TITLE Boot Menu
prompt 0
timeout 100
ONTIMEOUT Minimal_Linux_Live

LABEL Minimal_Linux_Live
MENU LABEL Minimal_Linux_Live (7M)
KERNEL syslinux/memdisk/memdisk
INITRD iso/minimal_linux_live_20-Jan-2017_32-bit.iso iso

LABEL lubuntu-14.10
MENU LABEL Lubuntu-14.10 Over HTTP (705M)
dhcp
kernel http://192.168.173.2:8000/syslinux/memdisk/memdisk iso raw
initrd http://192.168.173.2:8000/iso/lubuntu-14.10-desktop-i386.iso 

pi@raspberrypi:~/d_drive/tftpboot/pxelinux.cfg $

The tftproot also hosts another directory called  ‘pxelinux.cfg’. This contains a ‘default’ configuration file, which has boot menu configuration along with KERNEL and INITRD commands.

Boot Menu provides a list of OSs hosted on the tftp server in iso and img directory. KERNEL command loads memdisk which is the core of operating system, then INITRD load the actual OS.

Python SimpleHTTPServer module can be used as http server.

Laptop booting over LAN

An example of working PXE boot is available at below location.

https://drive.google.com/file/d/0BzYjW-ujx4yhcTNERlJZMmlObUd5QkhONUktRXZoNWNOX1Jj/view?usp=sharing

The project is also available on github here.

SSH Tunneling Made Easy!

I think I have understood the concept of SSH tunneling.  Tunneling is encapsulating one network protocol inside another. Consider the following scenario to understand. Lets say three computers A,B and C are connected in a network. B is allowed to connect to C but not A. How would A connect to C ? It can disguise as B and connect to C, right ? The disusing is called Tunneling, hiding or encapsulating.

A -> B -> C

Tunneling tools are available in market, the most readily available tool is ssh client. SSH client in the UNIX/Linux machines have capability to to do SSH tunneling, i.e, it provides feature to hide network traffic in SSH connections (tunnel). This is called SSH tunneling. SSH is secured connection, so the connection tunneled through it, is also secure.

SSH Tunneling is the most exploited method to hack  networks.  It is also used to by pass corporate firewalls. We can understand this in detail using below network diagram.

This is a typical network diagram, Corporate network is hidden behind the firewall. The firewall has rules – which allows/deny incoming and outgoing connection. Client, SSHHost, WebHost, DBHost are nodes which are running SSH Server, Web server and Database server respectively. All the nodes are connected with each other and accessible over the network ( or internet ).

Most corporate firewall allows outgoing connections at port:80, 443. These are port for web server and secured web (https). If some corporate user wants to connect to web server, it can connect, the firewall would allow http connection at port 80. But, if the user wants to connect to a database server (at port 1521) outside firewall, the firewall would deny the connection.

SSH (local) Tunneling  can allow the corporate user to connect the Database server.  All we have to do is to create a connection from a client (here Webshot) to SSHHost server.

Webhost $ ssh -L 443:DBHost:1521 sshuser@SSHhost.

This will create a connection from Webhost to SSHhost and it will create a tunnel from Webhost to DBhost as well. The tunnel is actually a server process on Webshost listening for incoming connection on port 443 and redirecting to DBhost. The firewall would think that the request is made on the port 443, but eventually it will be redirected to database server on port 1521. In this case the Webhost is called jump server.

SSH local tunneling can give access to servers behind the firewall. But, how can we access servers inside the corporate firewall. Lets assume we want to show a website available in corporate intranet to outside world. This can be achieved using SSH reverse tunneling.  For SSH reverse tunneling to work, the the firewall should allow to connect to a ssh server outside firewall (port 22 should be allowed on the Firewall). Create a ssh connection from a computer (localhost) inside the firewall.

corporate computer # ssh -R 9999:localhost:80 sshuser@SSHHost

This would create a ssh connection from the machine inside, corporate to the SSHHost server. Additionally it would create a server on SSHHost at the port 9999. Incoming connection to SSHHost on port 9999 would be redirected to corporate computer port 80. We can access the website hosted on corporate computer (localhost) from outside. Hit http://SSHhost:9999 in the web browser on the client. it will open website hosted within intranet.

The SSH Server on SSHHost should have “GatewayPorts yes” in the file /etc/ssh/sshd_config. The computer we are targeting with in the firewall should also have “GatewayPorts yes”.

The concept of tunneling is old and not well documented.  It took me some time to understand it. Feel free to ask questions and share your comments.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Route – How the IP packets are sent?

A computer system or any SOC (System on chip) device with network interfaces uses routing tables to connect to other networks.  A routing table is system wide (tabular) configuration which take cares how the IP packets would be send to another system.

Usually, every network device (laptops, routers, switches) has routing table configured manually or automatically.

At a ubuntu machine, routing table looks something like below. Every entry says which Interface (Iface) would be used to send an network packet with certain IP. The machine in the snapshot has two network interface (eth0 & tun0).

The default route (0.0.0.0 in windows) is used for all the non-matching IP , i.e, All the internet traffic goes via default interface. A windows routing table looks like below.

You can add a new route (static routing) to windows machine using route ADD command. (You need admin privilege ).

route ADD 192.167.173.0 MASK 255.255.255.0 192.168.0.191

Now, if you ping 192.168.173.1, it would be routed to interface having IP address 192.168.0.191).  The interface would further transfer the packet to its default gateway.

You reach internet server via multiple gateways, switches and routers. All of them have a routing table configured. Windows trace route command can tell which systems have been used to reach the internet server.

 

MQTT Python Client

In this article, I want to create a python module which would connect to mosquitto server. We can connect a MQTT client as subscriber or publisher of message.

Mosquitto python module has been donated to Eclipse Paho project, which is open-source client implementations of MQTT.  This module can be installed using pip command.

$ sudo pip install paho-mqtt

Subs.py – The code given below is subscribe module. You can look at comment to understand in detail.

#The line imports python mqtt module.
import paho.mqtt.client as mqtt

#Here we initilize variables used in the code.
server = "192.168.0.XXX"
port = 8883
keep_live = 45
topic = "msgTopic"

# Define routines to register.
# Client - The instance at the client side.
# userdata - Data to be passed by the user.
# flags - response flag sent by the server.
# rc - Return code, 0 is successful connection.
# mid - message id
# granted_qos - Quality of service granted by the server, default 0.
# payload - the message received on the topic.
def on_connect(client, userdata, flags, rc):
 mqttc.subscribe(topic, 0)
def on_subscribe(client, userdata, mid, granted_qos):
 print("Subscribed to: "+str(mid)+" "+str(granted_qos))
def on_message(client, userdata, msg):
 print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))

# Initilize a client instance and ser password for the connection.
mqttc = mqtt.Client()
mqttc.username_pw_set(username="username",password="password")

# register the routines.
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe

# connect to the server and wait for message.
mqttc.connect(server, port, keep_live)
mqttc.loop_forever()

Pubs.py – The code given below is publisher module. You can look at comment to understand in detail.

#The line imports python mqtt module.
import paho.mqtt.client as mqtt

#Here we initilize variables used in the code.
server = "192.168.0.191"
port = 8883
keep_live = 45
topic = "msgTopic"

# Connect and publish modules
def on_connect(client, userdata, flags, rc):
 mqttc.subscribe(topic, 0)
def on_publish(client, userdata, mid):
 print "Message Published..."

# Initilize a client instance and ser password for the connection.
mqttc = mqtt.Client()
mqttc.username_pw_set(username="mqtt",password="mqtt")

# register the routines.
mqttc.on_publish = on_publish
mqttc.on_connect = on_connect

# Connect and publish the message on a topic.
mqttc.connect(server, port, keep_live)
mqttc.publish(topic,"Hello! How are you?")
mqttc.disconnect()

 Output – Below is the Publish and Subscribe module in action on raspberry pi.