Posts

Posted on

Unix tools introduced. Today: cat

cat is a well known command to concatenate the content of multiple files.  Example: cat file1 file2 file3

But there are other use cases. cat offers a nice way to print out multi line strings.  It is even possible to include variables into the string, which feels a little bit like using a templating language.

Example:

NAME=ADMIN@COMPANY.COM;
cat <<EOF
Hello $LOGNAME,
please be aware. This system will be under maintenance soon.
Have a good day.
Sincerely
$NAME
EOF

For more info on the <<EOF visit this SO-Thread

 

 

Posted on

Public Money, Public Code

https://www.heise.de/newsticker/meldung/Erneuter-Kurswechsel-Muenchen-will-moeglichst-breit-auf-Open-Source-setzen-4716098.html

Open Source Produkte werden gerne überall genutzt. Öffentliche Institutionen könnten sich aber noch viel stärker an der Entwicklung von quelloffenen und frei verwendbaren Softwareprodukten beteiligen und damit zum allgemeinen Wohl beitragen.  Der rot-grüne Koalitionsvertrag von München weist interessante Punkte zu diesem Thema  auf.

Posted on

Leaving Amazon

https://www.tbray.org/ongoing/When/202x/2020/04/29/Leaving-Amazon

Amazon is exceptionally well-managed and has demonstrated great skill at spotting opportunities and building repeatable processes for exploiting them. It has a corresponding lack of vision about the human costs of the relentless growth and accumulation of wealth and power. If we don’t like certain things Amazon is doing, we need to put legal guardrails in place to stop those things. We don’t need to invent anything new; a combination of antitrust and living-wage and worker-empowerment legislation, rigorously enforced, offers a clear path forward.

Don’t say it can’t be done, because France is doing it.

Posted on

Gesundheitsakte per Injektion

MIT researchers have now developed a novel way to record a patient’s vaccination history: storing the data in a pattern of dye, invisible to the naked eye, that is delivered under the skin at the same time as the vaccine.

[…]

The researchers now plan to survey health care workers in developing nations in Africa to get input on the best way to implement this type of vaccination record keeping.

https://news.mit.edu/2019/storing-vaccine-history-skin-1218

Posted on

A tool to convert CSV, YML, XML, JSON and JSON-LD – oi v0.1

First release of oi is ready for testing. oi is an experimental command line tool to convert structured data from one format into another. The first release comes with rudimentary support for CSV, JSON, XML, YML and JSON-LD. The basic idea is to provide a straight in-out/out-in command that might not work completely correct but complements with other tools (jq, xsltproc,  sed, grep, awk, etc. ) to manipulate structured data on the command line.

Install

wget https://schnasse.org/deb/oi_0.0.1.deb
sudo apt install ./oi_0.0.1.deb # installs openjdk-11-jre

Example Usage

  1. Convert passwd to yml (use -t to print different formats)
oi /etc/passwd -d":" --header="login,password,uid,gid,comment,home,shell" -icsv
  1. List some disk info in yml (use -t to print different formats)
lsblk |grep -o sd.*|awk '{print $1 ";" $4 ";" $NF}'|oi -d";" --header="device,size,mount" -icsv
  1. Create adhoc Json-Ld Context for some arbitrary Json-API
curl https://api.github.com/users/jschnasse|oi -ijson -tcontext
Posted on

Command Line Tools: The growth of options

I found this table here: https://danluu.com/cli-complexity/

command 1979 1996 2015 2017
ls 11 42 58 58
rm 3 7 11 12
mkdir 0 4 6 7
mv 0 9 13 14
cp 0 18 30 32
cat 1 12 12 12
pwd 0 2 4 4
chmod 0 6 9 9
echo 1 4 5 5
man 5 16 39 40
which 0 1 1
sudo 0 23 25
tar 12 53 134 139
touch 1 9 11 11
clear 0 0 0
find 14 57 82 82
ln 0 11 15 16
ps 4 22 85 85
ping 12 12 29
kill 1 3 3 3
ifconfig 16 25 25
chown 0 6 15 15
grep 11 22 45 45
tail 1 7 12 13
df 0 10 17 18
top 6 12 14
Posted on

Java: Simple Read/Write of Maps as Json

You can use tools like Jackson to serialize and deserialize maps to json an vice versa.

Example

@Test
public void loadMapFromFileAndSaveIt(){
    Map<Object, Object> map = loadMap("map.json");
    map.put("8", "8th");
    map.remove("7");
    saveMap(map,"/path/to/map2.txt");
}

private Map<Object, Object> loadMap(String string) {
    ObjectMapper mapper = new ObjectMapper(); //should be initialized outside!
    try (InputStream in = Thread.currentThread().getContextClassLoader().getResourceAsStream("map.json")) {
        return mapper.readValue(in, HashMap.class);
    }catch (Exception e) {
        throw new RuntimeException(e);
    }
}

private void saveMap(Map<Object, Object> map,String path) {
    try (PrintWriter out = new PrintWriter(path)) {
        out.println(toString(map));
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}

public String toString(Object obj) {
    try (StringWriter w = new StringWriter();) {
        new ObjectMapper().configure(SerializationFeature.INDENT_OUTPUT, true).writeValue(w, obj);
        return w.toString();
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}

If the file map.json on your classpath contains

{
 "1":"1th",
 "2":"2th",
 "3":"3th",
 "4":"4th",
 "5":"5th",
 "6":"6th",
 "7":"7th"
}

The code above will modify it and writes it to a file /path/to/map2.txt that will contain the modified map.

{
  "1" : "1th",
  "2" : "2th",
  "3" : "3th",
  "4" : "4th",
  "5" : "5th",
  "6" : "6th",
  "8" : "8th"
}

 

Posted on

Unix tools introduced. Today: rsync

rsync is a very cool tool that can be used to copy files between hosts or between directories on the same host. Like the term ‘sync’ suggests the copy process can be controlled into great detail to modulate rsync’s behavior.  Take a look at the available options under: https://linux.die.net/man/1/rsync

This is my list of cool options.  I start with the most basic usage. The following command can be used to copy, and later on sync two directories.

rsync -avn /source/dir /target/dir

The command ‘archives’ file attributes (-a) and displays some status info (-v).

In the given form, the command only does a dry-run (-n). To execute the command remove the -n.

The command uses the short form of --archive (-a) which translates to (-rlptgoD).

  • -r – recursive copy
  • -l – copy symlinks as symlinks
  • -p – set target permissions to be the same as the source
  • -t – set target mtime to be the same as the source. Use this to support fast incremental updates based on mtime.
  • -g – set target group to be the same as the source
  • -o – set target owner to be the same as the source
  • -D – if remote user is superuser this recreates devices and other special files.

More cool options

Move

--remove-source-files This will remove copied files from source.

Update

--update This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file.

Delete

--delete Delete files on target that do not exist in source tree.

Backup

--backup Make a backup of modified or removed files on target.

--backup-dir=date +%Y.%m.%d Specify a backup dir on target.

What to copy?

--min-size=1 Do not copy empty files. This can be particularly interesting if you have corrupted files in the source.

--max-size=100K Copy only small files. Can be used to handle small and large files differently.

--existing Only override files that already exist on the target. Do not create new files on target.

--ignore-existing Only copy files that do not exist on target.

--exclude-from Define excludes in a file.

Scheduling, Bandwidth and Performance

--time-limit Ends rsync after a certain time limit.

--stop-at=y-m-dTh:m Ends rsync at a specific time.

--partial Allows partial copies in case of interruptions.

--bwlimit=100 Limits bandwidth Specify KBytes/second. Good option if transfer of large files is required.

Output

  • -h output numbers in a human-readable format.
  • --progress display progress.
  • -i log change info.
  • --log-file= define a log file.
  • --quiet no output.
  •  -v Output status info. You can add more ‘v’.
  • Forgot to log any progress info? Use the following command to see what rsync is about to do. 
     ls -l /proc/$(pidof rsync)/fd/*