LinkedIn: Lakehouse Analytics - with Microsoft Fabric and Azure Databricks

Today i came across a posting in linkedin.com which points to this nice booklet:

The linkedin posting pointed to site, where you can register for the a PDF, which contains 20 pages and 7 chapters.

Chapter one is a very short one (only half of a page): A typical introduction about data, information, analytics and why this is important :)

In chapter 2 the lakehouse architecture is explained. I liked the phrase "It combines [...] traditional data warehouse with the massive scale and flexibility of a data lake". This phrase combined with a very good table of the differences between a data warehouse and a data lake is from my point of view an excellent definition.

"Data management and analytics with Microsoft Fabric and Azure Databricks" is the title of the third chapter. This chapter only emphasizes that Fabris and Databricks can work seamlessly together and Microsoft introduces a OneLake to simplify the integration of these tools.

Chapter 4 i can not really summarize here. But there is really a cool figure in that chapter. Here only a part of that:

The Databricks part is missing and some other parts as well, but in the new Microsoft approach Fabric consists not only of storage - even PowerBI is a part of that new powerful tool. (one subsection is about AI integration)

The next chapter "Code faster with GitHub Copilot, Visual Studio Code, and Azure Databricks" is about the demonstrating "the power of Azure Databricks as a leading platform for data and AI when combined with developer tools such as Visual Studio Code and GitHub Copilot". This is like a small walkthrough how to configure Visual Studio Code.

In the seventh chapter a step by step guide is provided for integrating Databricks with OneLake. 

In my eyes chapter 4 is the key of that booklet, for everyone who wants to know, how the terms Fabric, OneLake, Databricks, Lakehouse are related and how the big picture looks like. Anyone who analyzes data with Microsoft should have read this.

Review@amazon: Microsoft Power Platform Enterprise Architecture

 This weekend i read the book "Microsoft Power Platform Enterprise Architecture"

https://www.packtpub.com/product/microsoft-power-platform-enterprise-architecture/9781800204577

Packt.com says about the book:

For forward-looking architects and decision makers who want to craft complex solutions to serve growing business needs, Microsoft Power Platform Enterprise Architecture offers an array of architectural best practices and techniques. With this book, you’ll learn how to design robust software using the tools available in the Power Platform suite and be able to integrate them seamlessly with various Microsoft 365 and Azure components. Unlike most other resources that are overwhelmingly long and unstructured, this book covers essential concepts using concise yet practical examples to help you save time.[...]

This is something i fully agree with. 

For more details please read my review at amazon :).

Postman: Scratchpad is end of life - Only cloud based workspaces possible --> How to keep the passwords/secrets secure?

Postman is the swiss army knife tool for dealing with HTTP(S) requests. You want to test anything with an GET, PUT, POST, OPTION call? Postman is the way to go.

https://postman.com

But some months ago postman announced the following:

Scratchpad was the option to work locally on your PC with postman and nothing was synced to the cloud. That was really cool, because most of the HTTPS requests have to be authenticated in some way (basic auth, secrets for oauth, api-keys, etc...)

From my perspective cloud is a very cool thing, but moving all the sensitive parameters into the postman cloud is not really an option.

So what to do? 

1. Check alternatives....
   But there is nothing like postman. e.g. the possibility to render the request to a curl, powershell, NodeJS axios, ... call and use this anywhere else....
2. Remove all sensitive data from your request
   A good step (and just to note: it is bad design, if you ever stored secrets inside the calls :) )
3. Now the calls do not work anymore....
   :(

But this problem was addressed really a long time ago (more than 4 years ago):

Prevent sync of passwords, api keys, and sensitive variables · Issue #6796 · postmanlabs/postman-app-support (github.com)

So here the link to the solution:

https://learning.postman.com/docs/sending-requests/variables/

It is just this easy:use variables and do not fill the initial value!

Here the tests: 

First the local postman:

Second the browser postman:

As you can see: the current values are empty inside the cloud for all entries, which have the inital value not filled...

(By the way: the current value is always a copy of the inital value. If you put there something like test11 and test22, only test1 and test2 will show up in the cloud version).

LinkedIn: Microsoft 365 Backup for Dummies sponsored by Veeam

 This week Veeam published this booklet on linkedin.com for download:

The booklet contains 6 chapter - the last one is a summary "Six takeaways" like always in such "for dummies" books.

From my point of view chapter 1 & 2 can be skipped - this you should really know, if running M365 (motivation for M365 backup). 

Chapter 3 is about how the loss of files can be prevented with M365 mechanisms. This is about compliance center, retention policies and labels. But only the keywords are mentioned and no deeper insights are provided.

In chapter 4 many scenarios are described, how you can loose your data on M365. Here a quote:

I think this chapter can be skipped like chapter 1 & 2. 

Chapter 5 opens with a nice term which was new to me: BaaS - Backup as a Service. Never thought about this acronym. Completely clear, that some backups in cloud are done without having purchased storage or servers on premises. Nice thing inside this chapter: a checklist about data source, data properties and some others. Really nice.

Chapter 6 comes up with the takeaways. These are really worth reading.

LinkedIn: A Guide to Data Governance - Building a roadmap for trusted data

On linkedin from "The Cyber Security Hub" shared a nice booklet about data governance:

An like always: It is only a booklet with about 25 pages - so this is not really a deep dive into this topic, but it gives you a good overview and of course a good motivation:

These include the need to governdata to maintain its quality as well as the need to protect it. This entails the prerequisite need to discover data in your organization with cataloguing, scanning, and classifying your data to support this protection.

and if this is to abstract, you should consider the following use case (and i think this use case has to be considered):

However, for AI to become effective, the data it is using must be trusted. Otherwise decision accuracy may be compromised, decisions may be delayed, or actions missed which impacts on the bottom line. Companies do not want ‘garbage in, garbage out’.

The booklet contains the sections "Requirements for governing data in a modern enterprise", "components needed for data governance", "technology needed for end-to-end data governance" and "managing master data". All sections do not provide a walk through for achieving a good data governance, but there are many questions listed, which you should answer for your company and then move forward. 

If you already have a data governance in place: This book is a good challenge for your solution. And for sure you will find some points, which are missing :)

LinkedIn Topcs: Why dataverse is for everyone...

Today i got a notification from a Microsoft colleague about the following linkedin posting:

Some weeks ago i started with PowerApps - and there this "dataverse" was mentionend as well. 

If you walk through the presentation in this linkedin post, you get an idea what this dataverse can do. I found the following picture @Microsoft learn:

And there are more details, why and how dataverse can be used: https://learn.microsoft.com/en-us/power-apps/maker/data-platform/data-platform-intro#why-use-dataverse

Sounds like a kind of datawarehouse centrally in the cloud. The most interesting point (like always): How to maintain this data, so that it is really usable...

Review: "Cloud Native Infrastructure with Azure" provided by Microsoft

Last week Microsoft published the following linkedin post:

On linkedin often tiny booklets are offered with around 10 up to 30 pages. But this offer from Microsoft is a book with 11 section and 289 pages.
If you are interested you can get it via this link (today this is still working, 12.2.2023): https://azure.microsoft.com/en-us/resources/cloud-native-infrastructure-with-microsoft-azure/

If you are not convinced: Here the table of contents:

1. Introduction: Why Cloud Native?
2. Infrastructure as Code: Setting Up the Gateway
3. Containerizing Your Application: More Than Boxes
4. Kubernetes: The Grand Orchestrator
5. Creating a Kubernetes Cluster in Azure
6. Oberservability: Following the Breadcrumbs
7. Service Discovery and Service Mesh: Finding New Territories and Crossing Borders
8. Networking and Policy Management: Behold the Gatekeepers
9. Distributed Databases and Storage: The Central Bank
10. Getting the Message
11. Serverless
12. Conclusion

Sounds like many topics i want to read about... :)

Openssl: How to automate (without hitting the carriage return many times)

I think nearly everyone, who administers some PCs or servers has used openssl. And almost everything there is straight forward.

To create your own key and certificate, just run:

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
 -keyout privateKey.key -out certificate.crt

 

............+..+.+.................+............+.+......+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+....................+......+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.+.....+...+..........+...............+....................+.+...+..+..........+........+......+.+...+.....+...+.......+..+.+...+...........+....+..+.......+.....+...............+................+......+......+...+......+...+...+..+......+......+.........+....+........+............+..........+.....+...+.......+..+...+.............+...+......+..............+....+...........+....+..+.+..+...+.............+............+...+..+.........+...+...............+...+..........+.........+...+...+...+...............+.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
..........+.....+.......+.........+..+.............+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.............+.........+..+....+..+...+.+......+...+.....+.........+.+.....+.+.....+...+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+......+............................+.....+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:

But the problem with that approach:

You have to add the carriage returns for every line after the 5 dashes and then your certificate looks quite ugly (see red colored text):

openssl x509 -text -in certificate.crt -noout

 

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0b:01:9a:aa:f1:59:69:33:84:7e:cf:89:69:0c:d5:80:61:82:b5:28
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
        Validity
            Not Before: Jan 22 15:54:43 2023 GMT
            Not After : Jan 22 15:54:43 2024 GMT
        Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c6:7d:5a:9f:97:3d:43:9b:e0:19:2f:46:31:5c:
                    82:f0:42:ac:da:a9:e8:d0:91:e0:01:98:05:52:cf:
                    1c:4e:77:53:1a:96:5c:6a:6f:ca:5c:61:a4:5f:14:
                    12:ed:69:ae:50:bb:99:28:48:df:bc:f6:76:c1:63:
                    2b:51:55:ad:bb:62:9f:3a:2b:1f:e7:c3:fd:bb:45:
                    04:c3:88:ee:b1:ba:c6:e2:f7:f1:80:5b:ef:eb:04:
                    fb:ec:82:89:39:c6:33:68:0d:3e:36:62:36:e0:a0:
                    ff:21:5f:74:ad:d2:4b:d4:5d:c4:67:6b:90:a0:8f:
                    1e:4c:80:31:30:2e:8e:5e:9d:62:8a:1d:45:84:5f:
                    d3:09:46:fe:4f:8d:68:c6:54:e4:51:da:e0:64:f8:
                    5d:af:01:2e:79:0c:fe:0b:0f:d6:2e:1b:e6:eb:09:
                    ca:cc:16:3d:92:53:ae:3b:ad:da:67:a5:ef:69:30:
                    7f:e7:53:7c:dd:23:59:c8:8c:6b:b0:a9:fa:fc:4c:
                    c1:44:cf:3f:2f:91:f4:8c:b6:7c:d9:ae:82:6d:96:
                    aa:bb:51:07:3c:2b:12:24:e4:a3:7d:9b:ee:4b:7e:
                    f4:02:0e:bc:b4:35:bd:73:dc:6b:b4:34:36:57:48:
                    72:f2:91:60:2d:79:d9:44:3c:77:76:eb:c7:8a:00:
                    5f:75
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                F1:77:6C:19:76:FB:E4:DD:50:2A:1E:01:BE:A1:5C:48:3D:5A:40:68
            X509v3 Authority Key Identifier: 
                F1:77:6C:19:76:FB:E4:DD:50:2A:1E:01:BE:A1:5C:48:3D:5A:40:68
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        8a:28:28:12:6c:1e:e5:54:86:9b:6e:90:7a:ca:aa:a7:a1:b3:
        b1:43:02:44:e8:9a:59:b2:d6:6e:36:c6:51:3b:9b:f4:91:47:
        40:6f:cf:6d:de:86:8d:dd:2f:9e:44:4c:f8:d3:5a:d3:3a:ef:
        d5:0d:e1:10:b6:64:34:ee:03:4a:f2:de:ff:da:db:a3:93:20:
        13:85:2a:d6:9b:b2:0e:2c:2e:9c:f9:71:ff:32:3b:c3:6b:0a:
        e7:98:2d:30:c9:a6:47:b7:72:84:bb:52:23:11:d6:b7:90:cb:
        98:cd:59:16:b5:8f:70:46:c1:95:90:01:2f:7f:9c:22:ac:29:
        8d:14:97:76:dd:06:56:f8:22:9d:f4:00:9f:40:3c:fb:c2:95:
        63:48:50:ee:ad:17:1b:54:6b:60:0c:d5:3e:66:3b:00:0e:7a:
        33:99:cc:4a:f6:dc:d1:e3:40:ea:8c:66:df:7e:92:e1:a5:e5:
        72:0e:89:ba:87:43:0c:56:70:8c:f2:9b:77:dd:ca:03:8e:24:
        fd:6b:51:d2:3b:b2:df:e4:ff:c2:3c:cb:ab:2e:cd:82:f4:69:
        ad:a3:81:d7:95:d0:68:e1:3f:fc:50:4d:8b:14:b2:82:8c:19:
        2b:06:8a:0e:ef:21:4b:68:4f:e3:1d:53:64:62:97:c8:35:45:
        01:54:d9:10

To avoid that you have just to expand your command with the following parameters:

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
 -keyout privateKey.key -out certificate.crt \
 -subj "/C=de/CN=schroff.special/OU=MyDepartment" 

Typical attributes can be found here:

(Common Name, Organization, Organizational Unit, Country, Location)