Openssl: How to automate (without hitting the carriage return many times)

I think nearly everyone, who administers some PCs or servers has used openssl. And almost everything there is straight forward.

To create your own key and certificate, just run:

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
 -keyout privateKey.key -out certificate.crt

 

............+..+.+.................+............+.+......+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+....................+......+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.+.....+...+..........+...............+....................+.+...+..+..........+........+......+.+...+.....+...+.......+..+.+...+...........+....+..+.......+.....+...............+................+......+......+...+......+...+...+..+......+......+.........+....+........+............+..........+.....+...+.......+..+...+.............+...+......+..............+....+...........+....+..+.+..+...+.............+............+...+..+.........+...+...............+...+..........+.........+...+...+...+...............+.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
..........+.....+.......+.........+..+.............+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.............+.........+..+....+..+...+.+......+...+.....+.........+.+.....+.+.....+...+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+......+............................+.....+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:

But the problem with that approach:

You have to add the carriage returns for every line after the 5 dashes and then your certificate looks quite ugly (see red colored text):

openssl x509 -text -in certificate.crt -noout

 

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0b:01:9a:aa:f1:59:69:33:84:7e:cf:89:69:0c:d5:80:61:82:b5:28
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
        Validity
            Not Before: Jan 22 15:54:43 2023 GMT
            Not After : Jan 22 15:54:43 2024 GMT
        Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c6:7d:5a:9f:97:3d:43:9b:e0:19:2f:46:31:5c:
                    82:f0:42:ac:da:a9:e8:d0:91:e0:01:98:05:52:cf:
                    1c:4e:77:53:1a:96:5c:6a:6f:ca:5c:61:a4:5f:14:
                    12:ed:69:ae:50:bb:99:28:48:df:bc:f6:76:c1:63:
                    2b:51:55:ad:bb:62:9f:3a:2b:1f:e7:c3:fd:bb:45:
                    04:c3:88:ee:b1:ba:c6:e2:f7:f1:80:5b:ef:eb:04:
                    fb:ec:82:89:39:c6:33:68:0d:3e:36:62:36:e0:a0:
                    ff:21:5f:74:ad:d2:4b:d4:5d:c4:67:6b:90:a0:8f:
                    1e:4c:80:31:30:2e:8e:5e:9d:62:8a:1d:45:84:5f:
                    d3:09:46:fe:4f:8d:68:c6:54:e4:51:da:e0:64:f8:
                    5d:af:01:2e:79:0c:fe:0b:0f:d6:2e:1b:e6:eb:09:
                    ca:cc:16:3d:92:53:ae:3b:ad:da:67:a5:ef:69:30:
                    7f:e7:53:7c:dd:23:59:c8:8c:6b:b0:a9:fa:fc:4c:
                    c1:44:cf:3f:2f:91:f4:8c:b6:7c:d9:ae:82:6d:96:
                    aa:bb:51:07:3c:2b:12:24:e4:a3:7d:9b:ee:4b:7e:
                    f4:02:0e:bc:b4:35:bd:73:dc:6b:b4:34:36:57:48:
                    72:f2:91:60:2d:79:d9:44:3c:77:76:eb:c7:8a:00:
                    5f:75
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                F1:77:6C:19:76:FB:E4:DD:50:2A:1E:01:BE:A1:5C:48:3D:5A:40:68
            X509v3 Authority Key Identifier: 
                F1:77:6C:19:76:FB:E4:DD:50:2A:1E:01:BE:A1:5C:48:3D:5A:40:68
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        8a:28:28:12:6c:1e:e5:54:86:9b:6e:90:7a:ca:aa:a7:a1:b3:
        b1:43:02:44:e8:9a:59:b2:d6:6e:36:c6:51:3b:9b:f4:91:47:
        40:6f:cf:6d:de:86:8d:dd:2f:9e:44:4c:f8:d3:5a:d3:3a:ef:
        d5:0d:e1:10:b6:64:34:ee:03:4a:f2:de:ff:da:db:a3:93:20:
        13:85:2a:d6:9b:b2:0e:2c:2e:9c:f9:71:ff:32:3b:c3:6b:0a:
        e7:98:2d:30:c9:a6:47:b7:72:84:bb:52:23:11:d6:b7:90:cb:
        98:cd:59:16:b5:8f:70:46:c1:95:90:01:2f:7f:9c:22:ac:29:
        8d:14:97:76:dd:06:56:f8:22:9d:f4:00:9f:40:3c:fb:c2:95:
        63:48:50:ee:ad:17:1b:54:6b:60:0c:d5:3e:66:3b:00:0e:7a:
        33:99:cc:4a:f6:dc:d1:e3:40:ea:8c:66:df:7e:92:e1:a5:e5:
        72:0e:89:ba:87:43:0c:56:70:8c:f2:9b:77:dd:ca:03:8e:24:
        fd:6b:51:d2:3b:b2:df:e4:ff:c2:3c:cb:ab:2e:cd:82:f4:69:
        ad:a3:81:d7:95:d0:68:e1:3f:fc:50:4d:8b:14:b2:82:8c:19:
        2b:06:8a:0e:ef:21:4b:68:4f:e3:1d:53:64:62:97:c8:35:45:
        01:54:d9:10

To avoid that you have just to expand your command with the following parameters:

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
 -keyout privateKey.key -out certificate.crt \
 -subj "/C=de/CN=schroff.special/OU=MyDepartment" 

Typical attributes can be found here:

(Common Name, Organization, Organizational Unit, Country, Location)

Review: Kafka: The Definitive Guide

Last week i read the book "Kafka: The Definitive Guide" with the subtitle "Real-Time Data and Stream Processing at Scale" which was provided by confluent.io:

The book contains 11 chapters on 288 pages - let's take look on the content:

Chapter 1 "meet Kafka" start with a motivation, why moving data is important and why you should not spend your effort not into moving but into your business. In addition an introduction to the messaging concepts like publish/subscribe, queues, messages, batches, schemas, topics, partitions, ... Many technical terms are defined there, but some are specific to Kafka and some are more general definitions. One additional info: Kafka was built by linkedin - the complete story told in the last section of this chapter.

The second chapter is about installing Kafka. Nothing special. OS, Java, Zookeeper (if clustered), Kafka.

Chapter 3 is called "Kafka producers: Writing messages to Kafka". Like the title indicates: all configuration details about sending messages are listed and explained. 

Chapter 4 is the same as the previous chapter but for reading messages. Both chapters contain many java example listings.

Chapters 5 & 6 are about clusters and reliability. Here are the nifty details explained like high water marks, message replication, timeouts, indices, ... If you want to run a high available Kafka system, you should read that and in case of failures you will know what to do.

Chapter 7 introduces Kafka Connect. Here a citation, when you should use Connect (it is not possible to summarize this):

You will use Connect to connect Kafka to datastores that you did not write and whose code you cannot or will not modify. Connect will be used to pull data from the external datastore into Kafka or push data from Kafka to an external store. For datastores where a connector already exists, Connect can be used by nondevelopers, who will only need to configure the connectors.

"Cross data cluster mirroring" is the title of chapter 8 - i do not understand why this chapter is not placed before chapter 7...

In chapter 9 and 10 administration and monitoring is explained. Very impressive is the amount of CLI examples. If you have a question: here you will find the CLI command, which provides the answer.

The last chapter "stream processing" is one of the longest chapters (>40 pages).  Here two APIs are presented to do some processing based on the messages. One example is, a stream which processes stock quotes. With stream processing it is possible to calculate the number of trades for every five-second window or the average ask price for every five-second window. Of course this chapter shows much more, but i think this gives the best impression ;-)

All in all a excellent book - even if you are not implementing Kafka ;-)

 

Linux Shell Programming: Converting hex to binary

What about:

• You have to convert a hex value into a binary value
• Your only tool is a linux shell
• You have to start in 30s
• You have to do it for >10000 hex values

Write a small program in c or java or whatever you would prefer?

What about this:

schroff@hades:$ echo "ibase=16;obase=2; 10 "|bc
10000
schroff@hades:$ echo "ibase=16;obase=2; 1F "|bc
11111

ibase: base of input value
obase: base of output value
The following bases can be used:

• 16 -> hex
• 10 -> decimal
•  8 -> octal
•  2 -> binary
  

Exif?!

Did you ever examine what data your digital camera is writing inside your images? There is much more than the picture itself. It is called

EXIF = Exchangeable Image File Format

Here a shortened listing:

ExifTool Version Number : 7.30
File Name : 2OG.JPG
File Size : 831 kB
File Modification Date/Time : 2008:05:19 13:49:28
File Type : JPEG
MIME Type : image/jpeg
Exif Byte Order : Little-endian (Intel, II)
Make : NIKON
Camera Model Name : COOLPIX S6
Orientation : Horizontal (normal)
Software : COOLPIX S6V1.0
Modify Date : 2008:05:19 13:49:28
Y Cb Cr Positioning : Co-sited
Exposure Time : 1/125
F Number : 8.5
Exposure Program : Program AE
ISO : 50
Exif Version : 0220
Date/Time Original : 2008:05:19 13:49:28
Create Date : 2008:05:19 13:49:28
Max Aperture Value : 3.0
Flash : Off
Focal Length : 5.8 mm
Maker Note Version : 2.00
Color Mode : Color
White Balance : Auto
Focus Mode : AF-S
ISO Selection : Auto
Data Dump : (Binary data 1502 bytes, use -b option to extract)
Compression : JPEG (old-style)
Preview Image Start : 14921
Preview Image Length : 16048

Until this weekend i did not care about this information, too. But After upgrading my laptop all images loosed the right date, because with a wrong copy command the date was replaced with the current date. But with the help of exiftool (apt-get install libimage-exiftool-perl) it was easy to correct the wrong date:

$ exiftool 2OG.JPG |grep "Create Date"
Create Date : 2008:05:19 13:49:21

Using exiftool together with awk, sed and touch the date of the images can be restored without a problem...

Batch processing: ssh access to multiple servers without ssh-keys

It is always the same: You are used to a technique like

• accessing server via ssh with ssh-keys

But in some environments you are not allowed or you just don't want to put your authorized_keys to the servers, which you want to access. But how to automate logins to this servers?

1. Poor man's approach
   buffer the password and use the middle button on your mouse, each time the script asks for the password.
   This works, but this is only "semi-automatic"
2. Use expect
   With this tool it is very easy to automate logins without using ssh-keys. Just write a script auto.ssh.sh like this:
   

#!/usr/bin/expect
spawn ssh -l root $argv
expect "Password: "
send "mypassword\n"
expect "#";
send "date\n";
expect "#";
send "exit\n";

This script can be called with a parameter (here IP or hostname) and executes the date command on the server. Now just write a loop:

for server in 10.0.0.1 10.0.0.2 10.0.0.3; do ./auto.ssh.sh $server; done

And you are done...
(But this is not so secure like using ssh-keys!)

Batch processing: modify images with mogrify

Did you ever think about resizing many images or cutting an area out of many images?
Yes you can open every single image with gimp and click through all the menus (image -> scale image -> new values -> ok). For selecting an area you can use your mouse and just do crtl-c (copy), crtl-shft-v (paste into new) and then crtl-shft-s (save as).
But there is a small command, which can do all this work for you:


mogrify (from ImageMagick)



With this tool you can resize your image just with this command:

mogrify -resize 640x480 *.jpg

or using a percentage:

mogrify -resize 50% *.jpg

Or cutting an area out of image serie:

mogrify -extract 370x295+124+102 *.jpg

There are much more options you can use...