Datamodelling with a “Json to Rdf” Approach.

At the example of this stackoverflow question:

“Supposing we have the following triple in Turtle syntax:

<http:/example.com/Paul> <http:/example.com/running> <http:/example.com/10miles> .

How do I add a start and end time? For example if I want to say he started at 10 am and finished his 10miles run at 12 am. I want to use xsd:dateTime.”

https://stackoverflow.com/questions/49726990


Sometimes it can be hard to create good, well fitting models. In my own experience it is crucial to identify a well defined set of entities and relations to create a vocabulary from.  Some people prefer to use visual strategies to develop their models. I prefer to write models in structured text.  This has the advantage that the process of modeling directly leads into actual coding.

Here is an example on how I would tackle the question .

1. The modelling part (not much RDF involved)

{
    "runs": [
        {
            "id": "runs:0000001",
            "distance": {
                "length": 10.0,
                "unit": "mile"
            },
            "time": {
                "start": "2018-04-09T10:00:00",
                "end": "2018-04-09T12:00:00"
            },
            "runner": {
                "id": "runner:0000002",
                "name": "Paul"
            }
        }
    ]
}

We store the json document in a file run.json. From here we can use the ‘oi’ command line tool , to create an adhoc context.

oi run.json -t context

The resulting context is just a stub. But with a few additions we can easily create a context document to define id’s and types for each vocable/entity/relation.

2. The RDF part: define a proper context for your document.

   {
    "@context": {
        "ical": "http://www.w3.org/2002/12/cal/ical#",
        "xsd": "http://www.w3.org/2001/XMLSchema#",
        "runs": {
            "@id": "info:stack/49726990/runs/",
            "@container": "@list"
        },
        "distance": {
            "@id": "info:stack/49726990/distance"
        },
        "length": {
            "@id": "info:stack/49726990/length",
            "@type": "xsd:double"
        },
        "unit": {
            "@id": "info:stack/49726990/unit"
        },
        "runner": {
            "@id": "info:stack/49726990/runner/"
        },
        "name": {
            "@id": "info:stack/49726990/name"
        },
        "time": {
            "@id": "info:stack/49726990/time"
        },
        "start": {
            "@id":"ical:dtstart",
            "@type": "xsd:dateTime"
        },
        "end": {
            "@id":"ical:dtend",
            "@type": "xsd:dateTime"
        },
        "id": "@id"
    }
}

3. The fun part: Throw it to an RDF converter of your choice

This is how it looks in JSON-Playground

Or simply use ‘oi’:

 
oi run.json -f run.context -t ntriples

Prints:

 
_:b0 <info:stack/49726990/runs/> _:b3 .
_:b3 <http://www.w3.org/1999/02/22-rdf-syntax-ns#first> <info:stack/49726990/runs/0000001> .
_:b3 <http://www.w3.org/1999/02/22-rdf-syntax-ns#rest> <http://www.w3.org/1999/02/22-rdf-syntax-ns#nil> .
<info:stack/49726990/runs/0000001> <info:stack/49726990/distance> _:b1 .
<info:stack/49726990/runs/0000001> <info:stack/49726990/runner/> <info:stack/49726990/runner/0000002> .
<info:stack/49726990/runs/0000001> <info:stack/49726990/time> _:b2 .
_:b1 <info:stack/49726990/length> "1.0E1"^^<http://www.w3.org/2001/XMLSchema#double> .
_:b1 <info:stack/49726990/unit> "mile" .
<info:stack/49726990/runner/0000002> <info:stack/49726990/name> "Paul" .
_:b2 <http://www.w3.org/2002/12/cal/ical#dtend> "2018-04-09T12:00:00"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
_:b2 <http://www.w3.org/2002/12/cal/ical#dtstart> "2018-04-09T10:00:00"^^<http://www.w3.org/2001/XMLSchema#dateTime> .

Run Gitlab CI locally

I use this docker-based approach.

0. Create a git repo to test this

mkdir my-git-project
cd my-git-project
git init
git commit --allow-empty -m"Initialize repo to showcase gitlab-runner locally."

1. Go to your git directory

cd my-git-project

2. Create a .gitlab-ci.yml

Example .gitlab-ci.yml

image: alpine

test:
  script:
    - echo "Hello Gitlab-Runner"

3. Create a docker container with your project dir mounted

docker run -d \
  --name gitlab-runner \
  --restart always \
  -v $PWD:$PWD \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

4. Execute with

docker exec -it -w $PWD gitlab-runner gitlab-runner exec docker test

5. Prints

...
Executing "step_script" stage of the job script
$ echo "Hello Gitlab-Runner"
Hello Gitlab-Runner
Job succeeded
...

Note: The runner will only work on the commited state of your code base. Uncommited changes will be ignored. Exception: The .gitlab-ci.yml itself must not be commited to be taken into account.

RDF to pretty JSON with oi 0.5.8

_:b0 a <http://schema.org/Book> ;
    <http://schema.org/name> "Semantic Web Primer (First Edition)" ;
    <http://schema.org/offers> _:b1 ;
    <http://schema.org/publisher> "Linked Data Tools" .

_:b1 a <http://schema.org/Offer> ;
    <http://schema.org/price> "2.95" ;
    <http://schema.org/priceCurrency> "USD" .

Based on this stack overflow answer I created a tool named **oi** that provides some capabilities to convert rdf to json via command line. If no frame is provided via cli, the tool aims to generate @context entries for most situations.


oi -i turtle -t json books.ttl |jq '.["@graph"][0]'

prints


{
  "@id" : "_:b0",
  "@type" : "http://schema.org/Book",
  "name" : "Semantic Web Primer (First Edition)",
  "offers" : {
    "@id" : "_:b1",
    "@type" : "http://schema.org/Offer",
    "price" : "2.95",
    "priceCurrency" : "USD"
  },
  "publisher" : "Linked Data Tools"
}

The tool attempts to create various output formats. The result is not meant to be 100% correct for each and every case. The overall idea is to provide adhoc conversions just as one step in a conversion pipeline.

The tool is available as .deb package via it’s github page at: https://github.com/jschnasse/oi.

Packaging a Command Line Java App for Linux

How to create a java command line tool that is (1) easy to install (2) as small as possible (3) and does not interfere with a previously installed jvm on the host?

Here is my take

  1.  Create an executable ‘fat jar’
  2.  Create a minimal jvm to run the fat jar
  3.  Define a proper version number
  4.  Package everything together to a .deb package
  5.  Provide the .deb package via an online repository

All snippes were taken from https://github.com/jschnasse/oi 

The oi command line app is a very simple conversion tool to transform structured formats from one into another.

Create an executable ‘fat jar’

I use the maven-assembly-plugin for this. Here is the relevant section from my pom.xml.

<plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <finalName>oi</finalName>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
          <archive>
            <manifest>
              <mainClass>org.schnasse.oi.main.Main</mainClass>
            </manifest>
            <manifestEntries>
              <Automatic-Module-Name>org.schnasse.oi</Automatic-Module-Name>
            </manifestEntries>
          </archive>
          <appendAssemblyId>false</appendAssemblyId>
        </configuration>
      </plugin>

The most important configuration entry is the path to the <mainClass> . The entry points to a java class that must define a main method.

It also is important to define a fixed <finalName>. We don’t want to create artifacts with version numbers in it. The versioning is done elsewhere. Our build process should just spit out an executable at a predictable location.

The mvn package command will now create a fat jar under target/oi.jar.

Create a minimal jvm to run the ‘fat jar’

The created jar can be executed as java -jar target/oi.jar. This is already an important milestone since you can now use the app on your own development pc. To make it a bit handier put the actual call into a script and copy it to /usr/bin/oi in order to make it accessible for all users on the development machine. Also you can provide the oi.jar at a more global location, e.g. /usr/lib.

This could be the content of /usr/bin/oi

java -jar /usr/lib/oi.jar $@

Use $@ to pass parameters from command line to the actual java app.

More on this will be explained in the ‘Package everything together’ section.

The next step is to make the program executable on other machines. Since the application depends on the existence of the java interpreter we have to find a way to either ship java  together with our little oi tool or to ask the user/user’s computer to install it in advance.

Both approaches are feasible. I decided to ship java together with my tool for the following reasons (1) The tool should be as self contained as possible (2) The installation of the tool should not interfere with other java based packages. (3) I want to be free to update to new jvm versions at my own speed, therefore I want  support only one single jvm version at every state of development.

Today java distributions come with a tool named jlink. The jlinktool can be used to create minimal jvms. This will look like:

jlink \
    --add-modules java.base,java.naming,java.xml \
    --verbose \
    --strip-debug \
    --compress=1 \
    --no-header-files \
    --no-man-pages \
    --output /opt/jvm_for_oi

The result is a minimal jvm only containing the modules java.base,java.naming,java.xml under /opt/jvm_for_oi. The idea is now to provide this jvm together with our app. But to become a bit more independent from the configuration of my  development machine I want to guarantee that my tool is served always with a well defined jvm version and not just with the version I have installed at my development machine. To create a well defined build environment I will use docker. With docker I can create a minimal jvm on the basis of a predefined openJDK version. And here is how it works.

1. Based on the code above we can create a file named Dockerfile.build to create the jvm based on the openJdk-12.0.1_12.

FROM adoptopenjdk/openjdk12:jdk-12.0.1_12
RUN jlink \
    --add-modules java.base,java.naming,java.xml \
    --verbose \
    --strip-debug \
    --compress 2 \
    --no-header-files \
    --no-man-pages \
    --output /opt/jvm_for_oi

We will use this docker definition just to create the jvm and copy it to our development environment. The docker image can be deleted directly afterwards.

docker build -t adopt_jdk_image -f Dockerfile.build .
docker create --name adopt_jdk_container adopt_jdk_image
docker cp adopt_jdk_container:/opt/jvm_for_oi /usr/share/jvm_for_oi
docker rm adopt_jdk_container

The resulting jvm can be found under /usr/share/jvm_for_oi.

This again is a very important milestone. You can now edit your startscript at /usr/bin/oi and use the generated jvm instead of your preinstalled java version. This will make the execution of the app independent of the globally installed java version and therefor more reliable.

/usr/share/jvm_for_oi/bin/java -jar /usr/lib/oi.jar $@

In my project configuration the inclusion of the minimal jvm increases the size of the .deb package by ~10MB. On the target system the jvm takes ~45MB extra space. In my former setup I configured openJDK-11 as dependency in the Debian package which consumes roughly ~80MB of extra space if newly installed.

Define a proper version number

Since oi is a java app built with maven I use the typical semantic versioning scheme which consists of three numbers (1) a  major, (2) a minor, (3 ) and a patch number divided by dots. Example given, a version of ‘0.1.4’ reads as follows:

0 – No major version. There is no stable version yet. Development is still at an early stage.

1 – First minor version. This is software at an very early stage. Usually minor versions are compatible to the recent major release. Since no major version exists this software has no reliable behavior yet.

4 – There were four patches released for the first minor version. A patch is typically a bug fix that does not change the

The process of creating  a new version is done as the following. (1) Define the next Version in a variable oi_version stored in a file VERSIONS. (2) Use a script bumpVersions.sh to  update the version numbers in several files like README, manpage, etc. (3) Commit files that were updated with the new version number to git. (4) Use the mvn-gitflow plugin to create new versions for the actual source and to push everything in a well defined manner to github.

<plugin>
  <groupId>com.amashchenko.maven.plugin</groupId>
  <artifactId>gitflow-maven-plugin</artifactId>
  <version>1.7.0</version>
  <configuration>
    <gitFlowConfig>
        <developmentBranch>master</developmentBranch>
    </gitFlowConfig>
  </configuration>
</plugin>

The gitflow-maven-plugin supports the command mvn gitflow:release . The command does the following:

1. Define a new release number

2. Update the pom.xml in the development branch accordingly

3. Push the updated pom.xml to the mainline branch

4. Create a tag on mainline

5. Update the release number in the development branch to a new SNAPSHOT release.

6. Push the updated pom.xml to the development branch.

The plugin was originally created with for the `gitflow` branching approach. Since my project uses the github-flow-branching approach which does not foresee a development branch besides of the mainline I defined master as development branch.

Package everything together

At this point a new release of the sourcecode is online at github. Now, it’s time to create the binary release. The binary release will be a .deb file containing the newly packaged fat-jar together with the minimal jvm. (5) A build.sh script is used to create the .deb artifact.

#! /bin/bash

scriptdir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd $scriptdir
source VERSIONS
mvnparam=$1

function build_oi(){
 package_name=$1
 package_version=$2
 package=${package_name}_$package_version
 mkdir -p deb/$package/usr/lib
 mkdir -p deb/$package/usr/bin
 mkdir -p deb/$package/usr/share/man/man1/
 mvn package -D$mvnparam
 sudo cp src/main/resources/$package_name deb/$package/usr/bin
 sudo cp target/$package_name.jar deb/$package/usr/lib

docker build -t adopt_jdk_image -f Dockerfile.build .
docker create --name adopt_jdk_container adopt_jdk_image
docker cp adopt_jdk_container:/opt/jvm_for_oi deb/$package/usr/share/jvm_for_oi
docker rm adopt_jdk_container

ln -s ../share/jvm_for_oi/bin/java deb/$package/usr/bin/jvm_for_oi 

}

function build(){
 package_name=$1
 package_version=$2
 package=${package_name}_$package_version

 if [ -d $scriptdir/man/$package_name ]
 then
   cd $scriptdir/man/$package_name
   asciidoctor -b manpage man.adoc
   cd -
   sudo cp $scriptdir/man/$package_name/$package_name.1 deb/$package/usr/share/man/man1/
 fi  
 dpkg-deb --build deb/$package
}

build_oi oi $oi_version

What you can see from the listing is that the script creates a directory structure in accordance to the .deb package format. It also generates (1) the fat-jar, (2) the minimal jvm (3) a man page and (4) binds it all together with a dpkg-deb -build command

Provide the .deb package via an online repository

(6) The .deb artifact is then uploaded to a bintray repo using again a shell script push_to_bintray.sh.

#! /bin/bash

scriptdir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
source VERSIONS

function push_to_bintray(){
cd $scriptdir
PACKAGE=$1
VERSION=$2
API_AUTH=$3
subject=jschnasse
repo=debian
filepath=${PACKAGE}_${VERSION}.deb
curl -u$API_AUTH -XPOST "https://bintray.com/api/v1/packages/$subject/$repo/" -d@bintray/${PACKAGE}/package.json -H"content-type:application/json"
curl -u$API_AUTH -XPOST "https://bintray.com/api/v1/packages/$subject/$repo/$PACKAGE/versions" -d@bintray/${PACKAGE}/version.json -H"content-type:application/json"
curl -u$API_AUTH -T deb/$filepath "https://bintray.com/api/v1/content/$subject/$repo/$PACKAGE/$VERSION/$filepath;deb_distribution=buster;deb_component=main;deb_architecture=all;publish=1;override=1;"
curl -u$API_AUTH -XPUT "https://bintray.com/api/ui/artifact/$subject/$repo/$filepath" -d'{"list_in_downloads":true}' -H"content-type:application/json"
cd -
}
apiauth=$1
push_to_bintray oi $oi_version $apiauth
push_to_bintray lscsv $lscsv_version $apiauth
push_to_bintray libprocname $libprocname_version $apiauth

The script makes use of a set of prepared json files to provide metadata for the  package.

(7) The last step is now to visit the github wegpage an navigate to the tag that has been created at step (4). By adding a release name it will become visible as release at the landing page of the git repo.

Step 6 seems the most critical step since it updates the debian repo and makes the new version available to everyone. In between step 5 and step 6 some sort of testing should happen to ensure that the artifact is installable and does execute as expected. My plan is to utilize a set of docker files to test releases. A first attempt can be found here.

Fazit

The process of versioning consists of multiple steps. Most of the work can be automated. A semi automated process can be developed with little effort. To automate the whole process it is crucial to provide well thought tests in between the steps and to define fallback points. This adds some extra safety to the objective but also introduces extra complexity. For future jdk versions it could be beneficial to use jpackager instead of jlink.

 

 

Java Forum

https://online.java-forum-stuttgart.de/

Wenn man ganz nach unten scrollt kommt man direkt zu den aufgezeichneten Vorträgen.
Mit ‘youtube-dl’ kann man sich die mp4 auch herunterladen.

Persönliche Lesezeichen

Just-In-Time-Patching
https://online.java-forum-stuttgart.de/c1/

Security Test-Driven – Learning by Pain
https://online.java-forum-stuttgart.de/a2/

Vom Code in die Cloud mit Mvn, Springboot, Docker und Kubernetes
https://online.java-forum-stuttgart.de/a4/

A tool to convert CSV, YML, XML, JSON and JSON-LD – oi v0.1

First release of oi is ready for testing. oi is an experimental command line tool to convert structured data from one format into another. The first release comes with rudimentary support for CSV, JSON, XML, YML and JSON-LD. The basic idea is to provide a straight in-out/out-in command that might not work completely correct but complements with other tools (jq, xsltproc,  sed, grep, awk, etc. ) to manipulate structured data on the command line.

Install

wget https://schnasse.org/deb/oi_0.0.1.deb
sudo apt install ./oi_0.0.1.deb # installs openjdk-11-jre

Example Usage

  1. Convert passwd to yml (use -t to print different formats)
oi /etc/passwd -d":" --header="login,password,uid,gid,comment,home,shell" -icsv
  1. List some disk info in yml (use -t to print different formats)
lsblk |grep -o sd.*|awk '{print $1 ";" $4 ";" $NF}'|oi -d";" --header="device,size,mount" -icsv
  1. Create adhoc Json-Ld Context for some arbitrary Json-API
curl https://api.github.com/users/jschnasse|oi -ijson -tcontext

Java: Simple Read/Write of Maps as Json

You can use tools like Jackson to serialize and deserialize maps to json an vice versa.

Example

@Test
public void loadMapFromFileAndSaveIt(){
    Map<Object, Object> map = loadMap("map.json");
    map.put("8", "8th");
    map.remove("7");
    saveMap(map,"/path/to/map2.txt");
}

private Map<Object, Object> loadMap(String string) {
    ObjectMapper mapper = new ObjectMapper(); //should be initialized outside!
    try (InputStream in = Thread.currentThread().getContextClassLoader().getResourceAsStream("map.json")) {
        return mapper.readValue(in, HashMap.class);
    }catch (Exception e) {
        throw new RuntimeException(e);
    }
}

private void saveMap(Map<Object, Object> map,String path) {
    try (PrintWriter out = new PrintWriter(path)) {
        out.println(toString(map));
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}

public String toString(Object obj) {
    try (StringWriter w = new StringWriter();) {
        new ObjectMapper().configure(SerializationFeature.INDENT_OUTPUT, true).writeValue(w, obj);
        return w.toString();
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}

If the file map.json on your classpath contains

{
 "1":"1th",
 "2":"2th",
 "3":"3th",
 "4":"4th",
 "5":"5th",
 "6":"6th",
 "7":"7th"
}

The code above will modify it and writes it to a file /path/to/map2.txt that will contain the modified map.

{
  "1" : "1th",
  "2" : "2th",
  "3" : "3th",
  "4" : "4th",
  "5" : "5th",
  "6" : "6th",
  "8" : "8th"
}

 

JPackage – Launch Java Apps without JVM

The jpackage tool of Java 14 can be used to create platform specific packages of java apps.  The app does not require a JVM to run.

Example

/opt/jdk-14/bin/jpackage --name etctoy --input target --main-jar etctoy.jar

The call is made from within a maven project. etctoy.jar is a fat-jar (size 6.6M) but the call should also work for regular jars with further dependencies in the target directory (see –input parameter).

The result is a debian package that installs the app under /opt/etctoy

sudo dpkg -i etctoy_1.0-1_amd64.deb

The installation uses 140M of disk space.

To make the tool available via command line on should link the binary into /usr/bin

sudo ln -s /opt/etctoy/bin/etctoy /usr/bin