Openstack installation : Minimal services and Controller – Pre-requisites

Content from “openstack.org”, listed here with minor changes – just noting down what I did – online notes.

The OpenStack system consists of several key services that are separately installed. These services work together depending on your cloud needs and include the Compute, Identity, Networking, Image, Block Storage, Object Storage, Telemetry, Orchestration, and Database services. You can install any of these projects separately and configure them stand-alone or as connected entities.

For the home lab, will require the following services

On the controller node :

Keystone : Identity Service
Glance : Image services
Nova : Compute services (All except nova-compute)
Neutron : Networking services
Cinder : Block storage (All except cinder-volumes)
Horizon : Dashboard / Management UI

On the compute node :

Nova    : Compute services (nova-compute only)
Neutron : Networking services (linux-bridge-agent only)
Cinder : Block storage (cinder-volumes only)

Note : In view many services, there will be number of passwords to be maintained to be configured for accessing the services. For ease of learning I had preferred to use a single password in all the places – This is fine for learning period. Nevertheless any reference to data surrounded by curly-braces need to be replaced with actual value.

All of the services require a database for managing service entities and end points, and all the services are managed typically by APIs exposed on the controller node and hence a SQL database needs to be installed in controller. For the home lab using MySQL. Install and configure MySQL on controller.

apt install --assume-yes mariadb-server python-pymysql

Create /etc/mysql/mariadb.conf.d/99-openstack.cnf with following contents :

[mysqld]
bind-address = {management-ip-address-of-controller-node}
default-storage-engine = innodb
innodb_buffer_pool_size = 1536000000
innodb_file_per_table = on
max_connections = 4096
collation-server = utf8_general_ci
character-set-server = utf8

Had preferred to configure 1.5G of buffer pool size based on the fact that my controller node has 8G only. No specific data points based on sizing 1.5G. Restart MySQL service on controller node and also secure the installation by setting a password for ‘root’ user on MySQL.

# service mysql restart
# mysql_secure_installation

Communication between services happen via message queues. Install Rabbit MQ on controller and create a user ‘openstack’ with password {message-queue-password} and set required (configuration, read, write) permissions – in our case all. Also tag the user account as administrator account.

# apt install --assume-yes rabbitmq-server
# rabbitmqctl add_user openstack {password-for-rabbit-mq}
# rabbitmqctl set_permissions openstack ".*" ".*" ".*"
# rabbitmqctl set_user_tags openstack administrator

Though the following is not mandatory, preferred to enable the Management Plugins (management via browser UI)

#rabbitmq-plugins enable rabbitmq_management

After restart of the node, had observed in logs that mq srever does not come up successfully, since the IP addresses were not assigned to the interface. Reviewing the systemd service (/lib/systemd/system/rabbitmq-server.service) script for MQ server, noticed the dependency “After=network.target”. Changing the same to “After=network-online.target” helped solve the issue. Also had observed that stopping of rabbitmq service will not complete. Searching for known issues, had come across a solution to change the value for ExecStop to “/usr/sbin/rabbitmqctl shutdown” which did work.

The authentication service uses memcached to cache tokens. So need to install memcached on controller node. (Security aspects wrt using memcached not considered at this stage of learning).

#apt --assume-yes install memcached python-memcache

Edit the binding interface in conf (/etc/memcached.conf) file

-l 10.0.0.15 #Ip Address of controller

Restart memcached after configuration change

#service memcached restart

Openstack services may use etcd – install the same

#apt --assume-yes install etcd

Update the etcd configuration file (/etc/default/etcd) (Note the usage of controller node management IP address)

ETCD_NAME="controller"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER="controller=http://10.0.0.15:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.0.0.15:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.15:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://10.0.0.15:2379"

Enable and start etcd

# systemctl enable etcd
# systemctl start etcd

Posted in Installation / How To, Notes | Tagged , , , , , , | Comments Off on Openstack installation : Minimal services and Controller – Pre-requisites

Openstack installation : Installing OS, configure pre-requisites and install openstack client

Content from “openstack.org”, listed here with minor changes – just noting down what I did – online notes.

In the course of learning installation of Openstack found that Canonical team maintain separate repositories for ‘Rocky’ release of Openstack. so decided to install Openstack on Ubuntu 18.04 server. One other benefit : no need to worry about installing dirvers.

Did the following post installation of OS, in controller and compute node.

Never end up with issues association failure at application level, due to insufficient file descriptors. edit /etc/security/limits.conf and add the following at end.

*  hard nofile 262140
* soft nofile 262140

Add ‘Rocky’ cloud-archive repository, update repository information and upgrade all that can be

apt install --assume-yes software-properties-common
add-apt-repository cloud-archive:rocky
apt update && apt dist-upgrade

Enable root login – This is something I do in all my systems @ home lab – Not mandatory. edit /etc/ssh/sshd_config, uncomment the following configuration and configure as follows

PermitRootLogin yes

Need to ensure that controller and compute nodes are reachable using hostnames. Edit /etc/hosts and add necessary entries

10.0.0.15       controller
10.0.0.41 iserver
10.0.0.50 aserver

Configure local time

unlink /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Kolkata /etc/localtime

Enable bridge net filtering – will be required prior to having neutron (networking services) configured. Edit /etc/modules-load.d/bridge.conf and add the following line so that the module gets loaded on node startup.

br_netfilter

Create /etc/sysctl.d/bridge.conf and add the following lines

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge-nf-call-arptables = 1

Disable app-armor and remove

systemctl stop apparmor
systemctl disable apparmor
apt purge --assume-yes apparmor

snapd not required. Also we will manage network configurations manually, do not need any network manager – uninstall netplan and snapd and install ifupdown.

apt purge --assume-yes snapd ubuntu-core-launcher squashfs-tools
apt-get --assume-yes purge nplan netplan.io
rm -rf /etc/netplan
rm -rf /usr/share/netplan/netplan/cli/commands/
apt install --assume-yes ifupdown

Configure one interface for overlay network (typically management) and one for provider network via which the guest VMs will get access to external networks. (Should be possible to use more than one interface – yet to learn how-to). Edit /etc/network/interfaces and add the following. Note : interface names could vary in different hosts.

source /etc/network/interfaces.d/*

#The loopback network interface
auto lo
iface lo inet loopback

#Interface for overlay/management
allow-hotplug eno1
auto eno1
iface eno1 inet static
address 10.0.0.31
netmask 255.255.255.0
broadcast 10.0.0.255
gateway 10.0.0.1
dns-nameservers 8.8.8.8 8.8.4.4

#Provider interface
auto eno2
iface eno2 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down

Had some challenges while installing neutron services, had to disable IPv6 for the provider network interface after which it got solved. (Yet to validate if disabling is required), add the following entry in /etc/sysctl.conf.

#eno2 is the interface name
net.ipv6.conf.eno2.disable_ipv6 = 1

Unmask and enable networking service, disable and mask systemd-networkd

systemctl unmask networking

systemctl enable networking

systemctl stop systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online

systemctl disable systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online

systemctl mask systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online

Edit /etc/systemd/resolved.conf, uncomment DNS= entry and add the DNS server IP

DNS=8.8.4.4

On every ssh login to node, a small delay was observed, figured out that disabling motd-news would help – this could be Ubuntu specific. Edit /etc/default/motd-news and set ENABLED=0

Reboot the system so the networking changes take effect.

Install NTP services on all nodes

apt install --assume-yes chrony

Let one node (controller) synchronize with servers on internet and other nodes synchronize with controller. Edit /etc/chrony/chrony.conf and comment out all pool entries. On controller node update the configuration as below

#pool ntp.ubuntu.com        iburst maxsources 4
#pool 0.ubuntu.pool.ntp.org iburst maxsources 1
#pool 1.ubuntu.pool.ntp.org iburst maxsources 1
#pool 2.ubuntu.pool.ntp.org iburst maxsources 2

server 0.asia.pool.ntp.org iburst
allow 10.0.0.0/24

On compute nodes update as below

#pool ntp.ubuntu.com        iburst maxsources 4
#pool 0.ubuntu.pool.ntp.org iburst maxsources 1
#pool 1.ubuntu.pool.ntp.org iburst maxsources 1
#pool 2.ubuntu.pool.ntp.org iburst maxsources 2

server controller iburst

Restart the NTP service after configuration changes

service restart chrony

Verify the services on controller by checking the output of ‘chronyc sources’ and ‘chronyc clients’ and on compute nodes the output of ‘chronyc sources’.

Install openstack client

apt install --assume-yes python-openstackclient


Posted in Installation / How To, Notes | Tagged , , , , , , | Comments Off on Openstack installation : Installing OS, configure pre-requisites and install openstack client

Openstack Installation : My Requirements

Note : Most of the contents here are from “openstack.org” site.  I have listed in sequence what I did to have open stack up and running in my home lab.  It is just my preference to have the notes online.

Preamble

Until now, had been using KVM for my home lab needs. Working VMs hosted on Openstack environment had observed quite a few issues when testing the application, more specifically clustering. Not observing the same when testing on VMs hosted on KVM, wanted to know more about issues related to using Openstack for hosting computes. Of course liked the ‘networking’ (neutron) services features. So decided to migrate from KVM to Openstack.

Being a Debian fan, had tried to have it installed on Debian stretch. However when I came to know that ‘cloud-archive’ repositories were maintained by Canonical, quickly shifted over to Ubuntu 18.04. The follow sequence of posts are steps I had followed to have a basic functioning Openstack environment at home.

Simple lab setup

Based on my observations working on Openstack environment, had decided to have a dedicated controller node and two compute nodes. Of course the choice was not to install it on VMs, but on physical servers. One other reason to go for dedicated controller, was to make use of a single board computer I had in my inventory. Pretty interesting piece of hardware.

Intel N4200 based SBC, 8 GB DDR4, 128 GB EMMC, 250G SSD, 2 x Gigabit Ethernet Port

For compute node, had decided to used Dell Power Edge R430, 12 Core 24 Threads, 64G DDR4, 600G SAS, 1.8T HDD.

Posted in Installation / How To, Notes | Tagged , , , , , , , | Comments Off on Openstack Installation : My Requirements

Debian – Qualcom 6174 Wifi not working

On my new Acer Nitro laptop, after installing Debian 9.4, found that Wifi was not working.

Looking at dmesg output looked like the Qualcom Athreos 10K was not found.

Installed firmware-linux package, but still did not work.

So visited the site as below

 

Clicked on the commit message on master

 

Clicked the download link fir linux-firmware-master.tar.gz

Expanded the file and copied the ath10k folder to /lib/firware/

Restarted the system and Wifi started working.

Posted in Uncategorized | Tagged , , , , , , | Comments Off on Debian – Qualcom 6174 Wifi not working

Spring 5 REST / GraphDB : Installing ArangoDB

Pr-requisite: http://www.sandeeprao.net/index.php/2018/05/12/spring-5-rest-graph-db-part-2-basic-spring-config-with-simple-controller/

Had used Neo4j in one of thea past projects.

However wanted to check out on ArangoDB, primary factor being Apache 2 License, Kubernetes, active community, supportive engineering team.  Following the development for couple of months now.  Time to get my hands dirty – Here it goes…

Install ArangoDB (Source : https://www.arangodb.com/download-major/debian/ )

root@server:~# curl -OL https://download.arangodb.com/arangodb33/Debian_9.0/Release.key
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 3924 100 3924 0 0 3425 0 0:00:01 0:00:01 --:--:-- 3427
root@server:~# apt-key add - < Release.key
OK
root@server:~# echo 'deb https://download.arangodb.com/arangodb33/Debian_9.0/ /' | sudo tee /etc/apt/sources.list.d/arangodb.list
deb https://download.arangodb.com/arangodb33/Debian_9.0/ /
root@server:~# sudo apt-get install apt-transport-https
Reading package lists... Done
Building dependency tree 
Reading state information... Done
The following NEW packages will be installed:
 apt-transport-https
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 171 kB of archives.
After this operation, 243 kB of additional disk space will be used.
Get:1 http://ftp.us.debian.org/debian stretch/main amd64 apt-transport-https amd64 1.4.8 [171 kB]
Fetched 171 kB in 1s (123 kB/s) 
Selecting previously unselected package apt-transport-https.
(Reading database ... 108748 files and directories currently installed.)
Preparing to unpack .../apt-transport-https_1.4.8_amd64.deb ...
Unpacking apt-transport-https (1.4.8) ...
Setting up apt-transport-https (1.4.8) ...

root@server:~# apt update
Get:1 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Ign:2 http://ftp.us.debian.org/debian stretch InRelease 
Ign:3 https://download.arangodb.com/arangodb33/Debian_9.0 InRelease
Get:4 http://ftp.us.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:5 http://ftp.us.debian.org/debian stretch Release [118 kB]
Get:6 https://download.arangodb.com/arangodb33/Debian_9.0 Release [1,206 B]
Get:8 https://download.arangodb.com/arangodb33/Debian_9.0 Release.gpg [819 B]
Get:9 https://download.arangodb.com/arangodb33/Debian_9.0 Packages [6,067 B]
Fetched 311 kB in 2s (105 kB/s) 
Reading package lists... Done
Building dependency tree 
Reading state information... Done
All packages are up to date.

root@server:~# apt install arangodb3=3.3.8
Reading package lists... Done
Building dependency tree 
Reading state information... Done
The following NEW packages will be installed:
 arangodb3
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 35.5 MB of archives.
After this operation, 294 MB of additional disk space will be used.
Get:1 https://download.arangodb.com/arangodb33/Debian_9.0 arangodb3 3.3.8 [35.5 MB]
Fetched 35.5 MB in 36s (961 kB/s) 
Preconfiguring packages ...
Selecting previously unselected package arangodb3.
(Reading database ... 108754 files and directories currently installed.)
Preparing to unpack .../arangodb3_3.3.8_amd64.deb ...
Unpacking arangodb3 (3.3.8) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up arangodb3 (3.3.8) ...
2018-05-13T15:41:41Z [64277] WARNING {memory} maximum number of memory mappings per process is 65530, which seems too low. it is recommended to set it to at least 2048000
2018-05-13T15:41:41Z [64277] WARNING {memory} execute 'sudo sysctl -w "vm.max_map_count=2048000"'
2018-05-13T15:41:41Z [64277] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2018-05-13T15:41:42Z [64277] INFO {startup} Server will now shutdown due to upgrade, database init or admin restoration.
Database files are up-to-date.
Created symlink /etc/systemd/system/multi-user.target.wants/arangodb3.service → /lib/systemd/system/arangodb3.service.
root@server:~#

Verify the services are started

root@server:~# systemctl status arangodb3
 arangodb3.service - ArangoDB database server
 Loaded: loaded (/lib/systemd/system/arangodb3.service; enabled; vendor preset: enabled)
 Active: active (running) since Sun 2018-05-13 21:11:48 IST; 2min 27s ago
 Process: 64477 ExecStartPre=/bin/chmod 700 /var/lib/arangodb3-apps (code=exited, status=0/SUCCESS)
 Process: 64473 ExecStartPre=/bin/chown -R arangodb:arangodb /var/lib/arangodb3-apps (code=exited, status=0/SUCCESS)
 Process: 64469 ExecStartPre=/bin/chmod 700 /var/lib/arangodb3 (code=exited, status=0/SUCCESS)
 Process: 64466 ExecStartPre=/bin/chown -R arangodb:arangodb /var/lib/arangodb3 (code=exited, status=0/SUCCESS)
 Process: 64462 ExecStartPre=/bin/chmod 700 /var/log/arangodb3 (code=exited, status=0/SUCCESS)
 Process: 64459 ExecStartPre=/bin/chown -R arangodb:arangodb /var/log/arangodb3 (code=exited, status=0/SUCCESS)
 Process: 64456 ExecStartPre=/usr/bin/install -g arangodb -o arangodb -d /var/run/arangodb3 (code=exited, status=0/SUCCESS)
 Process: 64452 ExecStartPre=/usr/bin/install -g arangodb -o arangodb -d /var/tmp/arangodb3 (code=exited, status=0/SUCCESS)
 Main PID: 64481 (arangod)
 Tasks: 70 (limit: 131072)
 CGroup: /system.slice/arangodb3.service
 └─64481 /usr/sbin/arangod --uid arangodb --gid arangodb --pid-file /var/run/arangodb3/arangod.pid --temp.path /var/tmp/arangodb3 --log.foreground-tty true

May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO {authentication} Jwt secret not specified, generating...
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO detected operating system: Linux version 4.16.8 (root@server) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP Fri May 11 21:04:11 IST 2018
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] WARNING {memory} maximum number of memory mappings per process is 65530, which seems too low. it is recommended to set it to at least 2048000
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] WARNING {memory} execute 'sudo sysctl -w "vm.max_map_count=2048000"'
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO using storage engine mmfiles
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO {cluster} Starting up with role SINGLE
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO {syscall} file-descriptors (nofiles) hard limit is 131072, soft limit is 131072
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO using endpoint 'http+tcp://127.0.0.1:8529' for non-encrypted requests
 May 13 21:11:50 server arangod[64481]: 2018-05-13T15:41:50Z [64481] INFO ArangoDB (version 3.3.8 [linux]) is ready for business. Have fun!

Verify the web interface access

After logging in with the password configured while installing

 

Just a check for the empty graphs view

Proceeding adding code to create a vertex in the graph (in our case a Tenant Entry).

Update pom.xml – Add maven dependency for Arangodb Java driver.  Also we would be using Spring-data – add the necessary dependancies

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersiaon>4.0.0</modelVersion>
  <groupId>GraphDBPoC</groupId>
  <artifactId>GraphDBPoC</artifactId>
  <version>1.0</version>
  <packaging>war</packaging>
  <properties>
    <failOnMissingWebXml>false</failOnMissingWebXml>
    <java.version>1.8</java.version>
    <spring.version>5.0.6.RELEASE</spring.version>
    <servlets.version>3.1.0</servlets.version>
    <jackson.version>2.9.5</jackson.version>
    <arango.driver.version>4.2.2</arango.driver.version>
    <spring.data.version>2.0.0<spring.data.version>
  </properties>
  <dependencies>
    ...
    ...
    ...
    <dependency>
      <groupId>com.arangodb</groupId>
      <artifactId>arangodb-java-driver</artifactId>
      <version>${arango.driver.version}</version>
    </dependency>
    <dependency>
        <groupId>com.arangodb</groupId>
        <artifactId>arangodb-spring-data</artifactId>
        <version>${spring.data.version}</version>
    </dependency>
  </dependencies>
  <build>
    ...
    ...
  </build>
</project>

Will be continuing on this shortly…..

Posted in Uncategorized | Tagged , , , , | Comments Off on Spring 5 REST / GraphDB : Installing ArangoDB

Spring 5 REST / Graph DB : Part 2 – Simple RestController

Pr-requisite : http://www.sandeeprao.net/index.php/2018/04/22/spring-5-rest-arango-graphdb-basic-poc-from-scratch/

It is possible to have the web app deployed in a container without xml configurations (web.xml) and the traditional dispatcher servlet configurations.  Extending AbstractAnnotationConfigDispatcherServletInitializer and providing the implementations for the following should suffice.

getRootConfigClasses() — for “root” application context (non-web infrastructure) configuration.

getServletConfigClasses() — for DispatcherServlet application context (Spring MVC infrastructure) configuration

Had decided to have a package separately for configs with the implementations.

ApplicationConfig.java – Instruct the servlet container to load REST resources from a specific package.

package in.dibs.config;

import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.EnableWebMvc;

@Configuration
@EnableWebMvc
@ComponentScan(basePackages = "in.dibs")
public class ApplicationConfig {
}

ApplicationInitializer.java – initializer class which is loaded at the startup of the application, it defines the configuration class of the application along with the context url.

package in.dibs.config;

import org.springframework.web.servlet.support.AbstractAnnotationConfigDispatcherServletInitializer;

public class ApplicationInitializer extends AbstractAnnotationConfigDispatcherServletInitializer {
  @Override
  protected Class<?>[] getRootConfigClasses() {
    return new Clasas[] { ApplicationConfig.class };
  }

  @Override
  protected Class<?>[] getServletConfigClasses() {
    // No views/JSP for now
    return null;
  }

  @Override
  protected String[] getServletMappings() {
    return new String[] { "/api/*" };
  }
}

Let us have all the message constants at one place so that it becomes easier to handle internationalization – For now just a simple constants package.

APIConstants.java – A simple place holder for constants

package in.dibs.constants;

import java.util.HashMap;
import java.util.Map;

public class APIConstants {
  private static Map<Integer, String> messages = new HashMap<Integer, String>();
  // 0 to 999 Common
  public static final int FAILED = 0;
  public static final int SUCCESS = 1;
  public static final int DATA_NOT_FOUND = 2;
  public static final int DATA_READ_FAILED = 3;
  public static final int INVALID_INPUT = 4;

  // 1000 to 1999 Tenant
  public static final int TENANT_CREATED = 1000;

  // TODO : Move to external resource file
  public static final String MSG_NOT_FOUND = "Matching data for the provided ID not found.";
  public static final String MSG_READ_FAILED = "Error while reading from data store.";
  public static final String MSG_INVALID_INPUT = "One or more information required as input not provided";

  public static final String MSG_TENANT_CREATED = "Tenant : %s, persisted in data store";
  
  static {
    messages.put(DATA_NOT_FOUND, MSG_NOT_FOUND);
    messages.put(DATA_READ_FAILED, MSG_READ_FAILED);
    messages.put(INVALID_INPUT, MSG_INVALID_INPUT);
  }

  public static String getMessage(int code) {
    if (messages.containsKey(code)) {
      return messages.get(code);
    } else {
      return "Unknown status";
    }
  }
}

Placing all entities in the project in appropriate package

Tenant.java – A simple entity to start with

package in.dibs.entity;

public class Tenant {
  private String name;
  private String id;

  public String getName() {
    return name;
  }

  public void setName(String name) {
    this.name = name;
  }

  public String getId() {
    return id;
  }

  public void setId(String id) {
    this.id = id;
  }

  @Override
  public String toString() {
    return "Tenant [name=" + name + ", id=" + id + "]";
  }
}

Placing all the REST API implementations in separate package


APIResponse.java – It would be better to have all API return response in standard format.  Just defining a basic standard

 

package in.dibs.api;

import in.dibs.constants.APIConstants;

public class APIResponse<T> {
  private int status;
  private String message;
  private String systemMessage;
  private T responseObject;

 /**
  * Success scenario with no response object expected.
  * 
  * @param code
  * API operation result code.
  */
  public APIResponse(int code) {
    this.status = code;
    this.message = APIConstants.getMessage(code);
    this.systemMessage = "NA";
    this.responseObject = null;
  }

 /**
  * Success scenario with response object expected. Message to be set by caller.
  * 
  * @param code
  * API Operation result code.
  * @param responseObject
  */
  public APIResponse(int code, T responseObject) {
   this.status = code;
   this.systemMessage = "NA";
   this.responseObject = responseObject;
  }

 /**
  * RT exception occurred.
  * 
  * @param systemMessage
  * Exception from underlying system / framework / library
  */
  public APIResponse(String systemMessage) {
    this.status = APIConstants.FAILED;
    this.message = APIConstants.getMessage(this.status);
    this.systemMessage = systemMessage;
    this.responseObject = null;
  }

 /**
  * Application exception occurred.
  * 
  * @param code
  * API Operation result code.
  * @param systemMessage
  * Associated RT Exception message if any.
  * @param responseObject
  * Response object to be passed on.
  */
  public APIResponse(int code, String systemMessage, T responseObject) {
    this.status = code;
    this.message = APIConstants.getMessage(code);
    this.systemMessage = systemMessage;
    this.responseObject = responseObject;
  }

  public int getStatus() {
    return status;
  }

  public void setStatus(int status) {
    this.status = status;
  }

  public String getMessage() {
    return message;
  }

  public void setMessage(String message) {
    this.message = message;
  }

  public Object getResponseObject() {
    return responseObject;
  }

  public void setResponseObject(T responseObject) {
    this.responseObject = responseObject;
  }

  public String getSystemMessage() {
    return systemMessage;
  }

  public void setSystemMessage(String systemMessage) {
    this.systemMessage = systemMessage;
  }

  @Override
  public String toString() {
    return "APIResponse [status=" + status + ", message=" + message + ", systemMessage=" + systemMessage
    + ", responseObject=" + responseObject.toString() + "]";
  }
}

TenantAPI.java – A Simple REST API

package in.dibs.api;

import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

import in.dibs.constants.APIConstants;
import in.dibs.entity.Tenant;

@RestController
@RequestMapping("/tenant")
public class TenantAPI {
  @RequestMapping(method = RequestMethod.POST)
  public APIResponse<Tenant> createTenant(@RequestBody Tenant tenant) {
    // For now just a success response
    APIResponse<Tenant> response = new APIResponse(APIConstants.TENANT_CREATED, tenant);
    response.setMessage(String.format(APIConstants.MSG_TENANT_CREATED, tenant.getName()));
    return response;
  }
}

Now build the war file

Now the war file is built.  Let us deploy it as Tomcat webapp and start Tomcat.

Testing the API functionality from Postman

At this point we have the base code to start developing REST APIs.  However our objective is the have REST APIs exposed for CRUD operations to persist/fetch from GraphDB.  So let us proceed with having GraphDB persistence layer added in the next part.

 

 

 

 

 

Posted in Uncategorized | Tagged , , , | Comments Off on Spring 5 REST / Graph DB : Part 2 – Simple RestController

Spring 5 REST / GraphDB : Part 1 – Environment Setup

Quite some time since I had worked on web application development.

Have been planning on developing a framework for analytics, where the data sources and relation between sources result in the need of graph databases.

So thought of kick starting a PoC with Spring 5, Tomcat 9, ArangoDB 3.x (>3.3).  Here it goes from scratch.  The intention is to record all kick start procedure so that any new comer to such needs finds it helpful.

Created a VM (Debian Stretch)
Install OpenJDK 1.8
Install Maven 3.3.x

Install Eclipse IDE

Create a new Dynamic Web Project

Convert to Maven Project


Edit build path and remove JRE 1.7 System Library if found

Add JRE 1.8

Edit and update the generated pom.xml – Set properties for versions of Java / Spring / Servlets, Add spring dependencies and update build for deploying to tomcat.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>GraphDBPoC</groupId>
 <artifactId>GraphDBPoC</artifactId>
 <version>1.0</version>
 <packaging>war</packaging>
 <properties>
   <failOnMissingWebXml>false</failOnMissingWebXml>
   <java.version>1.8</java.version>
   <spring.version>5.0.6.RELEASE</spring.version>
   <servlets.version>3.1.0</servlets.version>
 </properties>
 <dependencies>
   <!-- Servlet Dependency -->
   <dependency>
     <groupId>javax.servlet</groupId>
     <artifactId>javax.servlet-api</artifactId>
     <version>${servlets.version}</version>
     <scope>provided</scope>
   </dependency>
   <!-- Spring Dependency -->
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-core</artifactId>
     <version>${spring.version}</version>
   </dependency>
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-context</artifactId>
     <version>${spring.version}</version>
   </dependency>
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-web</artifactId>
     <version>${spring.version}</version>
   </dependency>
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-webmvc</artifactId>
     <version>${spring.version}</version>
     </dependency>
   </dependencies>
   <dependency>
     <groupId>com.fasterxml.jackson.core</groupId>
     <artifactId>jackson-databind</artifactId>
     <version>${jackson.version}</version>
   </dependency>
   <build>
     <sourceDirectory>src/main/java</sourceDirectory>
     <resources>
       <resource>
         <directory>src/main/resources</directory>
       </resource>
     </resources>
     <plugins>
       <plugin>
         <artifactId>maven-compiler-plugin</artifactId>
         <version>3.7.0</version>
         <configuration>
           <source>${java.version}</source>
           <target>${java.version}</target>
         </configuration>
       </plugin>
       <plugin>
         <artifactId>maven-war-plugin</artifactId>
         <version>3.0.0</version>
         <configuration>
           <warSourceDirectory>WebContent</warSourceDirectory>
         </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.tomcat.maven</groupId>
        <artifactId>tomcat7-maven-plugin</artifactId>
        <version>2.2</version>
        <configuration>
          <path>/</path>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Update Maven Project

Install Apache Tomcat

#cd /usr/local/
#wget http://www-eu.apache.org/dist/tomcat/tomcat-9/v9.0.8/bin/apache-tomcat-9.0.8.tar.gz
#tar -xvzf apache-tomcat-9.0.8.tar.gz 
#ln -fs /usr/local/apache-tomcat-9.0.8 tomcat

Optionally create setenv.sh

#cd /usr/local/tomcat
#nano bin/setenv.sh

Contents of file

export CATALINA_OPTS="$CATALINA_OPTS -Xms8192m"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx8192m"
export CATALINA_OPTS="$CATALINA_OPTS -Xss1920k"
export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"
export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"
export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"

Configure Run Time Environment in Eclipse (Window->Preferences->Server->Runtime Environments)



Configure Server





We are done with setting up the development environment.




 

 

 

 

 

Posted in Uncategorized | Tagged , , , | Comments Off on Spring 5 REST / GraphDB : Part 1 – Environment Setup

Open Day Light – Toaster Tutorial App

Few of my contacts were finding it difficult to get going on with the Open Day Light tutorial app.

Reported that they were not able to cross the first step of creating a simple ‘Example’ project using Maven and an archetype called ‘opendaylight-startup-archetype’.

Tried the following and it worked.

Installed JDK and updated JAVA_HOME environment path variable.
Installed Maven and update MAVEN_HOME and M2_HOME environment variable. Updated PATH environment variable to include Maven bin path.

In home folder created .m2 folder and created settings.xml with the following contents

<?xml version="1.0" encoding="UTF-8"?>
<!-- vi: set et smarttab sw=2 tabstop=2: -->
<!--
 Copyright (c) 2014, 2015 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the
 terms of the Eclipse Public License v1.0 which accompanies this distribution,
 and is available at http://www.eclipse.org/legal/epl-v10.html
-->
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">

<profiles>
 <profile>
 <id>opendaylight-release</id>
 <repositories>
 <repository>
 <id>opendaylight-mirror</id>
 <name>opendaylight-mirror</name>
 <url>https://nexus.opendaylight.org/content/repositories/public/</url>
 <releases>
 <enabled>true</enabled>
 <updatePolicy>never</updatePolicy>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </repository>
 </repositories>
 <pluginRepositories>
 <pluginRepository>
 <id>opendaylight-mirror</id>
 <name>opendaylight-mirror</name>
 <url>https://nexus.opendaylight.org/content/repositories/public/</url>
 <releases>
 <enabled>true</enabled>
 <updatePolicy>never</updatePolicy>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </pluginRepository>
 </pluginRepositories>
 </profile>

<profile>
 <id>opendaylight-release</id>
 <repositories>
 <repository>
 <id>opendaylight-release</id>
 <name>opendaylight-release</name>
 <url>https://nexus.opendaylight.org/content/repositories/opendaylight.release/</url>
 <releases>
 <enabled>true</enabled>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </repository>
 </repositories>
 <pluginRepositories>
 <pluginRepository>
 <id>opendaylight-release</id>
 <name>opendaylight-release</name>
 <url>https://nexus.opendaylight.org/content/repositories/opendaylight.release/</url>
 <releases>
 <enabled>true</enabled>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </pluginRepository>
 </pluginRepositories>
 </profile>
 </profiles>

<activeProfiles>
 <activeProfile>opendaylight-release</activeProfile>
 </activeProfiles>
</settings>

Created sample project using Maven as follows

mvn archetype:generate -DarchetypeGroupId=org.opendaylight.controller -DarchetypeArtifactId=opendaylight-startup-archetype -DarchetypeCatalog=remote -DarchetypeVersion=1.2.3-Boron-SR3

For selecting the archetypeVersion – access the following link in browser and picked one which had the startup-archetype as artificat Id.

https://nexus.opendaylight.org/content/repositories/opendaylight.release/archetype-catalog.xml

Posted in Uncategorized | Tagged , , , , | Comments Off on Open Day Light – Toaster Tutorial App

Setting up KVM on my dev system

Wanted to create 3 VMs on my Desktop with dedicated partitions on hard disk and dedicated NIC ports.

Installed Debian-9.2.1-amd64 on the base/host system. Updated to the latest kernel (optional).

Verify that virtualization is enabled in BIOS

root@server:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 1
Model name: AMD Ryzen Threadripper 1950X 16-Core Processor
Stepping: 1
CPU MHz: 2200.000
CPU max MHz: 3400.0000
CPU min MHz: 2200.0000
BogoMIPS: 6786.94
Virtualization: AMD-V
L1d cache: 32K
L1i cache: 64K
L2 cache: 512K
L3 cache: 8192K
NUMA node0 CPU(s): 0-31
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca

Installed qemu-kvm and required packages

apt install -y qemu-kvm libvirt0 virt-manager bridge-utils

Rebooted the system.

Edited and updated the contents of /etc/network/interfaces (For one physical interface bridged to br0 interface)

auto enp5s0
iface enp5s0 inet manual

auto br0
iface br0 inet static
address 192.168.0.4
network 192.168.0.0
netmask 255.255.255.0
broadcast 192.168.0.255
bridge_ports enp5s0
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off

Rebooted the system. Did not want the other physical interfaces to be brought up on start-up even if they are connected to a switch (Not sure if this is the right way, but worked for me).

Used nm-connection-editor to achieve the same. Selected the connection and ‘Edit’ option

Clicked on General Tab

Unchecked all options as shown below


‘Saved’ and repeated the same for all other connections.

Rebooted the system (Restart networking did not help – not sure what mistake I did).

Added ‘root’ (user account that would be accessing the virt-manager) to libvirt group.

gpasswd libvirt -a root

Copied the downloaded Debian (required guest OS) iso image to /var/lib/libvirt/images folder

Changed the ownership information for iso images

chown libvirt-qemu:libvirt /var/lib/libvirt/images/*

Started the virt-manager, selected File->New Virtual Machine and performed the following actions

Clicked on Browse and selected the image and continue with “Choose Volume”

Note : Wanted dedicated physcial partition. Did not select Manage and directly typed the partition information.

Note : As had decided to allocate dedicated NIC, selected appropriate macvtap option.

Completed all installation steps.

Once the installation was over and VM started – Manually shut down the same to update the boot options in virt-manager.

After shutting down, Selected View->Details from Menu.

Updated the boot options as follows

After applying the changes.

That’s it now after starting the host, the VMs are started and are ready for accessing them.

Posted in Uncategorized | Tagged , , , , , | Comments Off on Setting up KVM on my dev system

Build Linux Kernel

Had a need to build Linux Kernel. There were multiple sites which detailed the how-to. Finally to me it boiled down to the following

After installing debian in the system, install required packages for building the kernel.

[ Note : I had attempted it as root user and hence did not use sudo ]

apt-get update
apt-get install git fakeroot build-essential ncurses-dev xz-utils libssl-dev bc bison flex libelf-dev

Down the latest stable (or required) version of linux kernel (Downloaded from browser so switching to Downloads folder). At the time of typing this the latest stable was 4.14.3

cd Downloads/
tar xf linux-4.14.3.tar.xz
cd linux-4.14.3

To make life simpler, just copied the module selection of current kernel with following commands. Note : I did not change the module selections as the purpose was to test building linux kernel.

cp /boot/config-$(uname -r) .config
make menuconfig

Now build the kernel (Note : 32 is the number of threads available. In my system it was 32 (output of nproc command)

Note : The update-initramfs command below may not be required – but no harm.

make -j 32
make modules_install -j 32
make install -j 32
reboot

After reboot, could see the option of kernel 4.14.3 listed in boot options.

Posted in Uncategorized | Tagged , , , , | Comments Off on Build Linux Kernel