Posts Tagged Vagrant

#Javaland 2017 wrap up

Yes – I did it again and attend Javaland conference in Phantasialand Brühl.

It was not easy this year to concentrate on the sessions because of the hottest march of the last 100 years. But the quality of the sessions beats the weather. Maybe again my invest in reading the abstracts and filter the sessions before the conference has payed off.

Day 1 Conference

Jens Schauderdocumentation & slides with AsciiDoc, Git, Gradle and Reveal.js

Jens shows in this early morning session how we can version and so diffing and merging our documentation and presentations via text based markup languages. Now I have to puzzle the pieces together for generating my future documents from code. One question remains – what if we have to share the work on the documentation work with non programmers?

Jean-Jacques van Oosten, Chief Digital Officer Rewe Group: The role of the CDO as Chief Product Owner

This presentation shows REWE as on the right way for digitalization. But somehow I did not get rid of the feeling, that this was more a recruiting campaign to get the right people for the job. Have to speak with some of my friends, which are working there in the field. Interesting is the goal to develop all IT itself and do not buy from competitors perhaps. This needs lot of money and resources.

Steffen SchluffSebastian Damm, OIO – Orientation in Objects GmbHMust Feature Branches and Continuous Integration contradict each other?

I know most of the facts already but here these was ordered and coherent derived from literature and practices. Conclusion – with the right tools you can live with the compromise.

Gorge Albrecht, Code Mentor: Readable tests with AssertJ

Unfortunately I come a little bit late, because there is not planned really a big break for lunch. It seems here I have missed the point before: I had hope that the test output is more readable for the tester, not for the programmer. The only disappointing session for me.

Sven Ruppert: Hacking with Mutation Testing

After I have heard Sven’s session for PIT framework last year on Javaland I found not the time till today to work with this. Unfortunately I bring my company laptop which does not allow to download files from maven over the VPN as the company man in the middle webwasher replaces the original certificates. Sven does his best to support me, but I could not solve this in this time. In the evening on my way to home a nameless attendee give me an hint to disable the ssl check in maven locally. So I could test Sven’s solution branch right before midnight successfully!

Carola Lilienthal, WPS – Workplace Solutions GmbH: From Pair Programming to Mob Architecting

A little bit late, I enter this fully packed room. This was announced before via “chili” in the conference planner, so no excuse for me. But as I could not make notes standing on the door in the last row, I think I have here to reread the presentation to give a concrete summary here.

Stefan Schlott, BeOne Stuttgart GmbH: Security in the development team

The presenter gives some hints for small development teams to make it fit for security. Unfortunately most of this are not happened in big companies too. Additional most developers are not interested in this too. In this session first I heard the term required Caretaker on this conference, which should repeated in several sessions.

Mario Mann, Novatec Consulting GmbH: Application Performance Management with Open-Source-Tooling

As session name promises: He show different tools for different purposes of APM including it’s own. Key message: No one tool for all purposes. Interesting some example test of the Javaland homepage.

Martin Thompson, Real Logic: Keynote: Listening to the Design Pressures

The second keynote starts with statement that “non functional requirements” should be changed to “quality attributes”. Further he states that in most cases all new hypes are covers of earlier research. So he recommend to first search for the original ideas. He mapped an example from 1983 and mapped several statements in the content to actual hypes. We have to bring back simplicity and other core techniques, as hardware will not grow anymore as in the past and will not solve our growing requirements. Doing it working and fast together in one step will give other design decisions as doing it separately.

Meet-the-Lib

While most other people try the rides of Phantasialand, I attend this format. Same motto as last year: show me your favorite opensource lib and I show your mine. Not all was interessant for me, but some of this I would never heard, when not here.

  • Lombok: Boilerplate generator, unfortunately not really usable in JDeveloper
  • SPI – Service Provider Interface: interesting, but see no use case at the moment
  • jQAssistant: QA for your code, very interesting open analysis tool, have to read the corresponding missed presentation
  • Form Checker: generic validation and creation of HTML forms for portals, no use case in our framework development
  • Future of Spring: concurrent JEE framework to ADF
  • Jax-RS Hypermedia: maybe later when REST required in our development

After dinner and some drinks I return to cologne.

Day 2 Conference

Elmar Juergens, TU München und CQSE GmbH: How sensible we use our quality analysis?

“From weighing alone the pig will not be fat”. Many analyse it’s code but do little against the results. He shows very entertaining the combination of research and practical steps to wake the interest to change the code. Again you need a Caretaker here. But remember “Management by numbers is not by professionals but by amateurs.

Rustam Mehmandarov, Computas AS: Field Notes of a Command Line Ninja

Lot’s of linux command line tricks, have to attach my notes to every bash call from now. Very entertaining.

Thorsten Maier, Orientation in Objects GmbH: Resilient Software Design Patterns

Good overview, key message: Use it only if you really need it, as it adds new layers to your software stack.

Falk Sippach, Orientation in Objects GmbH: Continous Architecture Documentation

This presentation was a good complement to Jens Schauder from Day 1. Falk shows the “documentation smells” as reasons to use here again markdown based documentation to generate to different target formats and audience. Additional he show some tips and tricks for the used tools.

Stephan Classen, Canoo Engineering AG: Robust, readable Web UI Tests with page object pattern, Geb and Spock

Good presenter, but I’m not really convinced from this wrapper tools (Selenium -> Geb -> Spock) in different technology stacks. Maybe sometime I will try it to be sure?

Frank Pientka, MATERNA GmbH Information & Communications: Load and Performance tests with ‎JMeter or Gatling

The speaker gives a very wide overview of both tools, I have definitely to reread this. Especial interesting is the feature of both tools (and the wrapper Taurus) to reuse selenium tests for load tests. Have to check this out!

Martin Reinhardt, Holisticon AG: Responsive Testing with the Galen Framework 

Presenter shows this interesting framework, but it same a little bit complicated to me. At the moment not usable for me, as it have no IDE Support for JDeveloper.

Thomas Darimont, eurodata AG

I had see some of this in last year “Meet the libs” format. As I had some plans for private application development, this seems the ultimately completest and free AIM solution for me. The speaker had deep understanding of the product and he packed this last presentation full with his knowledge. If you need somebody to set up your system, you should hire him!

Day 3 – Workshops

Gerd AschemannContinuous Delivery of Continuous Delivery

The trainer here has too a deep knowledge of the area and put all of this into this workshop.

We started with the creation of a VirtualBox via Vagrant. The we added script provisioner to install puppet into the box. With another script provisioner we start the installation and configuration of Jenkins via puppet scripts and modules. After that we installed Docker in the VirtualBox via another puppet script. On top of this we automatically create jenkins jobs to start a Docker container containing Maven and building there a simple Java Application. As last step we add an Nexus container to deploy the resulting artefacts.

The trainer provides us with some theory, but most time was practical work. Additional to the described steps we add some configurations for testing the created infrastructure and minimize the creation time of the infrastructure via caching.

The whole time we push our local git repositories to an central instance of the trainer where our environment are build (or not) via a central Jenkins instance.

The extensive workshop scenario, the different operating systems and bugs in the newest version of vagrant make it more difficult to reproduce all steps just in time. But we get all sources and slides to reproduce the scenario later at home.

Despite I have already started an similar scenario at home before, I get a lot of fresh input for my future work here. This workshop was every minute worth.

Conclusion

Same as last year: This was inspiring conference, which gives me a lot of fresh ideas and reminders for some spilled goals for my private and company projects. I think we’ll see us next year!

Improvements for next year

There are some for next year in Phantasialand:

  1. Please deliver stable and fast internet at least for the speakers and the workshops.
  2. Put the labs and commuity workshops out from the exhibition hall to filter the noise from soundchecks and nighthacking.
  3. If you see via the favorites of dukecon, that rooms are full, maybe you can change locations.
  4. For workshops provide big enough tables for comfortable working.

That’s it.

Advertisements

, , , , , , , , , , , , ,

1 Comment

Virtual Development Server: Provide Docker images in docker registry for Oracle via Jenkins

This is a follow up to earlier Virtual Development Server: Provide Docker images in docker registry for Oracle XE Database and others.

We now try to automated build of all needed images as soon as Oracle GitHub Sources changed or we need to build our Vagrant VirtualBox again from scratch.

After we have already installed Jenkins we now install first some needed plugins.

After this we create our Jenkins pipeline via “New Item”.

Here is the script for the moment, later after we have it successfully tested and implemented all builds of images we need, we will put it in it’s own git repository and call it from there. The script is written in Pipeline Model Definition, a config-like syntax for defining Pipelines.

pipeline {
    agent label:'localhost_vagrant'
    stages {
        // Get the actual docker images sourcse from oracle git account
        stage('Get Sources') {
            steps {
                git url: 'https://github.com/oracle/docker-images.git'
            }
        }
        stage('Build Oracle Docker Images') {
            steps {
                parallel (
                    "Database XE 11.2.0.2" : {
                        dir('OracleDatabase/dockerfiles') {
                            // provide the downloaded source (via your oracle account) from the host
                            sh 'if [ ! -f 11.2.0.2/oracle-xe-11.2.0-1.0.x86_64.rpm.zip ]; then cp /software/Oracle/Database/oracle-xe-11.2.0-1.0.x86_64.rpm.zip 11.2.0.2/oracle-xe-11.2.0-1.0.x86_64.rpm.zip; fi'
                            // build the image
                            sh 'sudo ./buildDockerImage.sh  -v 11.2.0.2 -x'
                            // prepare the image for push to the local registry
                            sh 'docker tag oracle/database:11.2.0.2-xe localhost:5000/oracle/database:11.2.0.2-xe'
                            // push to the local registry
                            sh 'docker push localhost:5000/oracle/database:11.2.0.2-xe'
                        }
                    },
                    "WebLogic 12.2.1.2" : {
                        // server-jre is required for Weblogic
                        dir('OracleJava/java-8') {
                            // provide the downloaded source (via your oracle account) from the host
                            sh 'if [ ! -f server-jre-8u111-linux-x64.tar.gz ]; then cp /software/Oracle/Java/server-jre-8u111-linux-x64.tar.gz server-jre-8u111-linux-x64.tar.gz; fi'
                            // build the image
                            sh 'sudo ./build.sh'
                            // prepare the image for push to the local registry
                            sh 'docker tag oracle/serverjre:8 localhost:5000/oracle/serverjre:8'
                            // push to the local registry
                            sh 'docker push localhost:5000/oracle/serverjre:8'
                        }
                        dir('OracleWebLogic/dockerfiles') {
                            // provide the downloaded source (via your oracle account) from the host
                            sh 'if [ ! -f 12.2.1.2/fmw_12.2.1.2.0_wls_Disk1_1of1.zip ]; then cp /software/Oracle/WebLogic/fmw_12.2.1.2.0_wls_Disk1_1of1.zip 12.2.1.2/fmw_12.2.1.2.0_wls_Disk1_1of1.zip; fi'
                            // build the image
                            sh 'sudo ./buildDockerImage.sh -v 12.2.1.2 -g'
                            // prepare the image for push to the local registry
                            sh 'docker tag oracle/weblogic:12.2.1.2-generic localhost:5000/oracle/weblogic:12.2.1.2-generic'
                            // push to the local registry
                            sh 'docker push localhost:5000/oracle/weblogic:12.2.1.2-generic'
                        }
                    }
                )
            }
        }
        stage('Cleanup') {
            steps {
                // clean virtualbox
                sh 'docker rmi --force localhost:5000/oracle/database:11.2.0.2-xe'
                sh 'docker rmi --force oracle/database:11.2.0.2-xe'
                sh 'docker rmi --force localhost:5000/oracle/weblogic:12.2.1.2-generic'
                sh 'docker rmi --force oracle/weblogic:12.2.1.2-generic'
                sh 'docker rmi --force localhost:5000/oracle/serverjre:8'
                sh 'docker rmi --force oracle/serverjre:8'
            }
        }

    }
}

We can wait now the 15 min poll time or start the job manually.

After an “vagrant halt / vagrant up” you can control in your browser, that the created images are persistent in the local docker registry:

docker_registry_persistent

Here you find the source code for this blog.

Here you find more about the topic “Virtual Development Server”.

That’s it.

, , , , , , , , , ,

Leave a comment

Virtual Development Server: Install Jenkins for Continuos Integration / Delivery / Deployment

For automation of all of my processes I need Jenkins in my development server.

First I had installed Jenkins as Docker container via Vagrant Docker provider. But with this architecture it was very complicated to create docker images and run Docker containers on the Docker host, in this case my Vagrant Virtualbox.

So for simplification I have decided to install Jenkins with the provisioning of the Vagrant Virtualbox via a shell provider in my Vagrantfile:

...
  # persistant storage for jenkins
  config.vm.synced_folder "C:\\shared\\virtual_storage\\jenkins_home", "/var/lib/jenkins", type: "nfs", owner: 994, group: 992, create: true
...
  # install jenkins
  config.vm.provision :shell, :path => "add_jenkins.sh"
...

First I map a shared folder of my Windows Host to the Vagrant Virtualbox to persist my Jenkins Configuration.
Then I install Jenkins via the shell script add_jenkins.sh.

sudo yum -y install java
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum -y install jenkins
sudo sed -i -e 's/JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true"/JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dmail.smtp.starttls.enable=true"/g' /etc/sysconfig/jenkins
sudo service jenkins start

Jenkins requires java, so I install it first. After that I get the latest Long-Term Support (LTS) repository, import it and install Jenkins from there. Then I do some configuration, which I need for my jobs and start Jenkins. An open issue is, that I have to restart Jenkins after Vagrant restart, because the shared folders are unfortunatly mounted after service starts.

After first install you have to enter the credential information from the filesystem as requested, all later installations use the persistent Jenkins configuration, so you can login and start from where you have left Jenkins.

unlock_jenkins

Here you find the source code for this blog.

Here you find more about the topic “Virtual Development Server”.

That’s it.

, , , , , , , ,

1 Comment

Virtual Development Server: Provide Docker images in docker registry for Oracle XE Database and others

For later creation of containers as needed in the deployment process we have first to build docker images whenever we do not find suitable one in docker hub.

For most of my development I need a oracle database. On the internet you can find a lot of Dockerfile’s for this, but no ready image. This is because for running a database you have to accept a license and this happens when you download the installation software. For private testing this license often is free.

The basic cycle for all of this images is the same on my side:

  1. Find a Docker image
  2. If not found download required Software and build a Docker image
  3. Build the Docker image
  4. Push the images to private docker registry or the public registry DockerHub

Fortunately Oracle provides recently on GitHub a lot of working Dockerfile’s for building different product images. The creator Gerald Venzl desribes in this blog how to create an Oracle EE image.

Following you find an example for building Oracle XE 11.2.0.2 Image.

First you have to clone the git repository to a directory of you choice, I choose C:\shared\scmlocal\docker-images.
Put the downloaded installation source oracle-xe-11.2.0-1.0.x86_64.rpm.zip in the subdirectory C:\shared\scmlocal\docker-images\OracleDatabase\dockerfiles\11.2.0.2 as described in GitHub.
Next we have to map this directory to our virtualbox via following vagrant instructions. Additional we prepare port forwarding for accessing later the database from our host system.

...
  # Port Forwardings for:
  # - Oracle database port
  config.vm.network "forwarded_port", guest: 1521, host: 1521
  # Oracle Application Express (APEX)
  config.vm.network "forwarded_port", guest: 8080, host: 8080
...
  # map Oracle Docker Images installation path
  config.vm.synced_folder "C:\\shared\\scmlocal\\docker-images", "/docker-images", :mount_options => ["dmode=777","fmode=777"]
...

After that restart the virtualbox and connect to it via

vagrant halt
vagrant up
vagrant ssh

In the output you can see this mapping.

...
    default: 1521 (guest) => 1521 (host) (adapter 1)
    default: 5000 (guest) => 5000 (host) (adapter 1)
    default: 8080 (guest) => 8080 (host) (adapter 1)
...
    default: /docker-images => C:/shared/scmlocal/docker-images
...

Now you create build the image with the Oracle XE (-x) database version (-v) 11.2.0.2 via the provided scripts from oracle. -i skips the MD5 checksum check, as the checksums are not checked in as UNIX text format and so does not work in our vagrant on windows / oel virtualbox combination.

cd /docker-images/OracleDatabase/dockerfiles
./buildDockerImage.sh -v 11.2.0.2 -x -i

Now you have to wait some time. When the process is ready, you get at the time of this writing an image with the binaries installed.

docker images
REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE
oracle/database     11.2.0.2-xe         624d78c89ccb        About a minute ago   1.203 GB
registry            latest              c6c14b3960bd        5 weeks ago          33.28 MB
oraclelinux         latest              1988eb5b3fc6        6 weeks ago          278.2 MB

Next we tag this image for pushing to the local docker repository, which we have created in the first blog of this series and is up with every vagrant up of our virtualbox. After that we push it to the registry.

docker tag oracle/database:11.2.0.2-xe localhost:5000/oracle/database:11.2.0.2-xe
docker push localhost:5000/oracle/database:11.2.0.2-xe

In the output we see the succesful push.

The push refers to a repository [localhost:5000/oracle/database]
345dfbce8859: Pushed
dd4b46d40894: Pushed
11327e04fac0: Pushed
11.2.0.2-xe: digest: sha256:82058ca2c5c201ff46471a8db4d9569da9869d420727595884050f94dad0ec85 size: 955

Now we will check, if we can create a Docker container from our image in the registry and the database is working in it. Therefore at first we delete the oracle/database and oraclelinux images from our virtualbox. Then we create the container.

docker rmi 624d78c89ccb -f
docker rmi 1988eb5b3fc6
sudo docker run --shm-size=1g -p 1521:1521 -p 8080:8080 --name="db11.2.0.2-xe" --restart unless-stopped localhost:5000/oracle/database:11.2.0.2-xe

In line 3 we pull the image from the local registry, start a container, open the ports 1521 and 8080 to the virtualbox and give the container a name.

Unable to find image 'localhost:5000/oracle/database:11.2.0.2-xe' locally
11.2.0.2-xe: Pulling from oracle/database

10ec637c060c: Pull complete
1902f66bc90b: Pull complete
dd8942200fe1: Pull complete
Digest: sha256:41ce1015c8f68ce063e855e7afa96000ededeef1e197e2181bc04c9577d73a1b
Status: Downloaded newer image for localhost:5000/oracle/database:11.2.0.2-xe
ORACLE AUTO GENERATED PASSWORD FOR SYS AND SYSTEM: 9f4e9a733e7983bc

Oracle Database 11g Express Edition Configuration
-------------------------------------------------
This will configure on-boot properties of Oracle Database 11g Express
Edition.  The following questions will determine whether the database should
be starting upon system boot, the ports it will use, and the passwords that
will be used for database accounts.  Press <Enter> to accept the defaults.
Ctrl-C will abort.

Specify the HTTP port that will be used for Oracle Application Express [8080]:
Specify a port that will be used for the database listener [1521]:
Specify a password to be used for database accounts.  Note that the same
password will be used for SYS and SYSTEM.  Oracle recommends the use of
different passwords for each database account.  This can be done after
initial configuration:
Confirm the password:

Do you want Oracle Database 11g Express Edition to be started on boot (y/n) [y]:
Starting Oracle Net Listener...Done
Configuring database...Done
Starting Oracle Database 11g Express Edition instance...Done
Installation completed successfully.

SQL*Plus: Release 11.2.0.2.0 Production on Tue Sep 6 09:41:22 2016

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production

SQL>
PL/SQL procedure successfully completed.

SQL> Disconnected from Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
#########################
DATABASE IS READY TO USE!
#########################
db_recovery_file_dest_size of 10240 MB is 0.98% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Starting background process CJQ0
Tue Sep 06 09:41:19 2016
CJQ0 started with pid=26, OS id=759
Tue Sep 06 09:41:24 2016
XDB installed.
XDB initialized.

In line 9 you can see the autogenerated passwords.
When line 46 appears, you can connect to the database.
With following command you can change the passwords of SYS and SYSTEM:

sudo docker exec db11.2.0.2-xe /u01/app/oracle/setPassword.sh LetsDocker

You see following output:

SQL*Plus: Release 11.2.0.2.0 Production on Tue Sep 6 15:49:34 2016
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
SQL>
User altered.
SQL>
User altered.

As long as you not remove or manually stop the container, it restarts with the virtualbox now via:

vagrant halt
vagrant up

At the moment you cannot make your database instance data persistent for this container, I have opened an issue for this, but Gerald Venzl has promised to to implement this later.

Here you find the source code for this blog.

Here you find more about the topic “Virtual Development Server”.

That’s it.

References:

, , , , , , , , , , , ,

1 Comment

Virtual Development Server: Add enough ram and disk space to VirtualBox for further server components

As I want later build docker images and run docker containers I have to provide enough disk space for this.

Per default the Vagrantboxes have vmdk disks, these have a static size and are to small for my purposes.

Therefore I convert the disk, which comes with the box, via Vagrant VirtualBox provider from vmdk to vdi, so that it allocate only the used disk space in the host system.
Additional I add a second big disk:

...
  # virtualbox provider
  config.vm.provider "virtualbox" do |vb|
    # name in VirtualBox
    vb.name = "Development Server"

    # configure 16 GB memory
    vb.customize ["modifyvm", :id, "--memory", "16384"]

    # clone the original vmdk disk into a dynamic vdi disk, which only allocate the used space on the host
    if ARGV[0] == "up" && ! File.exist?("#{ENV["HOME"]}/VirtualBox VMs/#{vb.name}/#{vb.name}.vdi")
      # configure the SATA controller for second disk port, for other box you may have another controller
      vb.customize [
        "storagectl", :id,
        "--name", "SATA",
        "--controller", "IntelAHCI",
        "--portcount", "1",
        "--hostiocache", "on"
      ]
      # clone the original disk, for other box you may have another disk name
      vb.customize [
        "clonehd", "#{ENV["HOME"]}/VirtualBox VMs/#{vb.name}/box-disk2.vmdk",
             "#{ENV["HOME"]}/VirtualBox VMs/#{vb.name}/#{vb.name}.vdi",
        "--format", "VDI"
      ]
      # attach the cloned disk to the controller
      vb.customize [
        "storageattach", :id,
        "--storagectl", "SATA",
        "--port", "0",
        "--device", "0",
        "--type", "hdd",
        "--nonrotational", "on",
        "--medium", "#{ENV["HOME"]}/VirtualBox VMs/#{vb.name}/#{vb.name}.vdi"
      ]
      # delete the original disk to release it's space
      vb.customize [
        "closemedium", "disk", "#{ENV["HOME"]}/VirtualBox VMs/#{vb.name}/box-disk2.vmdk",
        "--delete"
      ]
    end

    # create addtional big dynamic vdi disk for docker images
    if !File.exist?("#{ENV["HOME"]}/VirtualBox VMs/#{vb.name}/#{vb.name}_docker.vdi")
      # create addtional big dynamic vdi (200 GB)
      vb.customize [
        "createhd",
        "--filename", "#{ENV["HOME"]}/VirtualBox VMs/#{vb.name}/#{vb.name}_docker.vdi",
        "--format", "VDI",
        "--size", 200 * 1024
      ]
      # attach the addtional disk to the controller
      vb.customize [
        "storageattach", :id,
        "--storagectl", "SATA",
        "--port", "1",
        "--device", 0,
        "--type", "hdd",
        "--medium", "#{ENV["HOME"]}/VirtualBox VMs/#{vb.name}/#{vb.name}_docker.vdi"
      ]
    end
  end

  # shell provider
  # format the additional disk and add the free space to the box
  config.vm.provision :shell, :path => "add_disk.sh"
  ...

Lines 64..66 adds the additional disk space to a logical volume via a Vagrant shell provider, which calls the script add_disk.sh in the created VirtualBox machine.
Resizing the first disk does not work, as therefore the machine has to be booted, which cannot be handled with Vagrant.

# exit immediately if a command exits with a non-zero status.
set -e
# activate debugging from here
set -x

if [ -f /etc/disk_added_date ] ; then
   echo "disk already added so exiting."
   exit 0
fi

# show diskspace of the logical volume before adding the disk
df -h /dev/mapper/linux-root

# partitioning the disk
sudo fdisk -u /dev/sdb <<EOF n p t 8e w EOF # initialize the partition for use by logical volume manager sudo pvcreate /dev/sdb1 # add the partition to volume group linux sudo vgextend linux /dev/sdb1 # increase the size of the logical volume /dev/mapper/linux-root sudo lvextend --extents +51199 --resizefs /dev/mapper/linux-root # mark that the disk was added date > /etc/disk_added_date

# show diskspace of the logical volume after adding the disk
df -h /dev/mapper/linux-root

Now you have to recreate the VirtualBox machine via

vagrant destroy
vagrant up

In the output you can now see, how the logical volume grows:

==> default: Running provisioner: shell...
    default: Running: C:/Users/torst/AppData/Local/Temp/vagrant-shell20160812-2348-1ln6wrz.sh
==> default: ++ '[' -f /etc/disk_added_date ']'
==> default: ++ df -h /dev/mapper/linux-root
==> default: Filesystem              Size  Used Avail Use% Mounted on
==> default: /dev/mapper/linux-root   16G  1.9G   14G  12% /
==> default: ++ sudo fdisk -u /dev/sdb
==> default: Welcome to fdisk (util-linux 2.23.2).
==> default:
==> default: Changes will remain in memory only, until you decide to write them.
==> default: Be careful before using the write command.
==> default:
==> default:
==> default: Command (m for help): Partition type:
==> default:    p   primary (0 primary, 0 extended, 4 free)
==> default:    e   extended
==> default: Select (default p): Partition number (1-4, default 1): First sector (2048-419430399, default 2048): Using default value 2048
==> default: Last sector, +sectors or +size{K,M,G} (2048-419430399, default 419430399): Using default value 419430399
==> default: Partition 1 of type Linux and of size 200 GiB is set
==> default:
==> default: Command (m for help): Selected partition 1
==> default: Hex code (type L to list all codes): Hex code (type L to list all codes): Changed type of partition 'Linux' to 'Linux LVM'
==> default:
==> default: Command (m for help):
==> default: Device does not contain a recognized partition table
==> default: Building a new DOS disklabel with disk identifier 0xb4c74c64.
==> default: The partition table has been altered!
==> default:
==> default: Calling ioctl() to re-read partition table.
==> default: Syncing disks.
==> default: ++ sudo pvcreate /dev/sdb1
==> default:   Physical volume "/dev/sdb1" successfully created
==> default: ++ sudo vgextend linux /dev/sdb1
==> default:   Volume group "linux" successfully extended
==> default: ++ sudo lvextend --extents +51199 --resizefs /dev/mapper/linux-root
==> default:   Size of logical volume linux/root changed from 15.62 GiB (4000 extents) to 215.62 GiB (55199 extents).
==> default:   Logical volume root successfully resized.
==> default: meta-data=/dev/mapper/linux-root isize=256    agcount=4, agsize=1024000 blks
==> default:          =                       sectsz=512   attr=2, projid32bit=1
==> default:          =                       crc=0        finobt=0
==> default: data     =                       bsize=4096   blocks=4096000, imaxpct=25
==> default:          =                       sunit=0      swidth=0 blks
==> default: naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
==> default: log      =internal               bsize=4096   blocks=2560, version=2
==> default:          =                       sectsz=512   sunit=0 blks, lazy-count=1
==> default: realtime =none                   extsz=4096   blocks=0, rtextents=0
==> default: data blocks changed from 4096000 to 56523776
==> default: ++ date
==> default: ++ df -h /dev/mapper/linux-root
==> default: Filesystem              Size  Used Avail Use% Mounted on
==> default: /dev/mapper/linux-root  216G  1.9G  214G   1% /

Here you find the source code for this blog.

Here you find more about the topic “Virtual Development Server”.

That’s it.

References:

, , , , , , , , ,

Leave a comment

Virtual Development Server: Creating Virtualbox machine including docker containers with Vagrant

As I want to decouple my development server from my computer as much as possible I want to create first a virtual machine with linux.

Because I have used Oracle Virtualbox in the past successfully, I will use it here too.

As I want have later a reproducible environment via Infrastructure as code, I give Vagrant a try for this. As source code management system I use GIT, my remote repository is located at GitHub.

As I don’t want create a new linux from scratch, I have looked for a ready system at Vagrantboxes, which contains a new Oracle Linux version, and find a Oracle Linux 7.1 x86_64 system with Chef and Puppet preinstalled.

Before I could start, I had to install on my Windows 10 computer Vagrant 1.8.5 and Oracle Virtualbox 5.0.16. Don’t use the 5.1 Version, as at this is not yet supported by vagrant 1.8.5.

For automatic update of the Oracle VM VirtualBox Extension Pack, which is delivered with the ready box and is required for use of shared folders and more, you should install following plugin as described here:

vagrant plugin install vagrant vbguest

Next I have created a directory of your choice and init there a Vagrantfile by calling

vagrant init

Then I changed the Vagrantfile as following, see the inline comments for documentation:

Vagrant.configure(2) do |config|

  # Use the mentioned ready OEL 7 linux box
  config.vm.box = "oraclelinux-7-x86_64.box"
  config.vm.box_url = "http://cloud.terry.im/vagrant/oraclelinux-7-x86_64.box"

  # Create a private network
  config.vm.network "private_network", type: "dhcp"

  # persistant storage for all docker container
  config.vm.synced_folder "C:\\shared\\virtual_storage", "/virtual_storage", :mount_options => ["dmode=777","fmode=777"]

  # virtualbox provider
  config.vm.provider "virtualbox" do |vb|
    # name in VirtualBox
    vb.name = "Development Server"
	end

  # Docker Private Registry container for storing later builded docker images, which are not in the Docker Public Registry at https://hub.docker.com/
  config.vm.provision "docker" do |d|
    d.run "registry", image: "registry", daemonize: true, args: "-d -p 5000:5000 -v /virtual_storage/docker_registry:/var/lib/registry"
  end

end

Run next from the created directory

vagrant up

Following you see the output

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'oraclelinux-7-x86_64.box'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: Development Server
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: hostonly
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Remote connection disconnect. Retrying...
    default:
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
[default] GuestAdditions versions on your host (5.0.26) and guest (5.0.12) do not match.
Loaded plugins: ulninfo
Package kernel-uek-devel-4.1.12-32.el7uek.x86_64 already installed and latest version
Package gcc-4.8.5-4.el7.x86_64 already installed and latest version
Package 1:make-3.82-21.el7.x86_64 already installed and latest version
Package 4:perl-5.16.3-286.el7.x86_64 already installed and latest version
Package bzip2-1.0.6-13.el7.x86_64 already installed and latest version
Nothing to do
Copy iso file C:\Program Files\Oracle\VirtualBox\VBoxGuestAdditions.iso into the box /tmp/VBoxGuestAdditions.iso
mount: /dev/loop0 is write-protected, mounting read-only
Installing Virtualbox Guest Additions 5.0.26 - guest version is 5.0.12
Verifying archive integrity... All good.
Uncompressing VirtualBox 5.0.26 Guest Additions for Linux............
VirtualBox Guest Additions installer
Removing installed version 5.0.12 of VirtualBox Guest Additions...
Removing existing VirtualBox non-DKMS kernel modules[  OK  ]
Copying additional installer modules ...
Installing additional modules ...
Removing existing VirtualBox non-DKMS kernel modules[  OK  ]
Building the VirtualBox Guest Additions kernel modules
Building the main Guest Additions module[  OK  ]
Building the shared folder support module[  OK  ]
Building the shared folder support module[  OK  ]
Building the graphics driver module[  OK  ]
Doing non-kernel setup of the Guest Additions[  OK  ]
You should restart your guest to make sure the new modules are actually used

==> default: Checking for guest additions in VM...
==> default: Configuring and enabling network interfaces...
==> default: Mounting shared folders...
    default: /vagrant => C:/shared/scmlocal/blog
    default: /virtual_storage => C:/shared/virtual_storage
==> default: Running provisioner: docker...
    default: Installing Docker onto machine...
==> default: Starting Docker containers...
==> default: -- Container: registry

In lines 1..24 you see the loading of the ready box and booting with the defined network
In lines 25..50 the version of the VirtualBox GuestAdditions of the box is checked and after that replaced with the actual installed VirtualBox version. This happens only on the first “vagrant up” call.
In lines 57..58 Docker is installed as a Vagrant Docker provider is called in the Vagrantfile line 20 and it does not exist in the box before.
In lines 59..60 a Docker Private Registry container is started via loading docker image from the Docker Public Registry. In a later blog we will use this Docker Private Registry container to hold our own created docker images like database and development tools. The pushed images are saved to a mapped host volume of my windows 10 computer.

You can connect to the virtualbox directly via

vagrant ssh

The you can see the downloaded Docker Private Registry images and the started container from this image:

[vagrant@oraclelinux7 ~]$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
registry            latest              c6c14b3960bd        13 days ago         33.28 MB
[vagrant@oraclelinux7 ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
739c1c9926b1        registry            "/entrypoint.sh /etc/"   5 minutes ago       Up 5 minutes        0.0.0.0:5000->5000/tcp   registry

You can now stop the Development Server via

vagrant halt

If you startup again the server via

vagrant up

the provision steps are not replayed until you destroy the server or explicitly call the provision again via

vagrant provision

Here you find the source code for this blog.

Here you find more about the topic “Virtual Development Server”.

That’s it.

, , , , , , , ,

1 Comment

%d bloggers like this: